GitHub PushEvent at 2014-06-13 19:39:41
Repository: spark
Actor: markhamstra
Payload: {"shas":[["234a3786ac8c4ece22083be67c340ac0e4484e01","9585396914bf96a14f5c1901ae0dc4a228f856dd@radialpoint.com","Spark 1916\n\nThe changes could be ported back to 0.9 as well.\nChanging in.read to in.readFully to read the whole input stream rather than the first 1020 bytes.\nThis should ok considering that Flume caps the body size to 32K by default.\n\nAuthor: David Lemieux <david.lemieux@radialpoint.com>\n\nCloses #865 from lemieud/SPARK-1916 and squashes the following commits:\n\na265673 [David Lemieux] Updated SparkFlumeEvent to read the whole stream rather than the first X bytes.\n(cherry picked from commit 0b769b73fb7ae314325857138a2d3138ed157908)\n\nSigned-off-by: Patrick Wendell <pwendell@gmail.com>","David Lemieux",false],["aef6390cdbbbcdc58d927b5498f206708aadf917","48d3a847ef684d87177a99cd6c21cdf466f2e98a@qq.com","[SPARK-1712]: TaskDescription instance is too big causes Spark to hang\n\nAuthor: witgo <witgo@qq.com>\n\nCloses #694 from witgo/SPARK-1712_new and squashes the following commits:\n\n0f52483 [witgo] review commit\n83ce29b [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n52e6752 [witgo] reset test SparkContext\n63636b6 [witgo] review commit\n44a59ee [witgo] review commit\n3b6d48c [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n926bd6a [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n9a5cfad [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n03cc562 [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\nb0930b0 [witgo] review commit\nb1174bd [witgo] merge master\nf76679b [witgo] merge master\n689495d [witgo] fix scala style bug\n1d35c3c [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n062c182 [witgo] fix small bug for code style\n0a428cf [witgo] add unit tests\n158b2dc [witgo] review commit\n4afe71d [witgo] review commit\n9e4ffa7 [witgo] review commit\n1d35c7d [witgo] fix hang\n7965580 [witgo] fix Statement order\n0e29eac [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n3ea1ca1 [witgo] remove duplicate serialize\n743a7ad [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n86e2048 [witgo] Merge branch 'master' of https://github.com/apache/spark into SPARK-1712_new\n2a89adc [witgo] SPARK-1712: TaskDescription instance is too big causes Spark to hang\n\nConflicts:\n\tcore/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala\n\tcore/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala","witgo",false],["a92900c5aca96d0609682ae9d9893c7d9e0f6327","9d468ac3e51a0dd55cd858437e2237db1fbaae1b@gmail.com","SPARK-1188: Do not re-use objects in the EdgePartition/EdgeTriplet iterators.\n\nThis avoids a silent data corruption issue (https://spark-project.atlassian.net/browse/SPARK-1188) and has no performance impact by my measurements. It also simplifies the code. As far as I can tell the object re-use was nothing but premature optimization.\n\nI did actual benchmarks for all the included changes, and there is no performance difference. I am not sure where to put the benchmarks. Does Spark not have a benchmark suite?\n\nThis is an example benchmark I did:\n\ntest(\"benchmark\") {\n  val builder = new EdgePartitionBuilder[Int]\n  for (i <- (1 to 10000000)) {\n    builder.add(i.toLong, i.toLong, i)\n  }\n  val p = builder.toEdgePartition\n  p.map(_.attr + 1).iterator.toList\n}\n\nIt ran for 10 seconds both before and after this change.\n\nAuthor: Daniel Darabos <darabos.daniel@gmail.com>\n\nCloses #276 from darabos/spark-1188 and squashes the following commits:\n\n574302b [Daniel Darabos] Restore \"manual\" copying in EdgePartition.map(Iterator). Add comment to discourage novices like myself from trying to simplify the code.\n4117a64 [Daniel Darabos] Revert EdgePartitionSuite.\n4955697 [Daniel Darabos] Create a copy of the Edge objects in EdgeRDD.compute(). This avoids exposing the object re-use, while still enables the more efficient behavior for internal code.\n4ec77f8 [Daniel Darabos] Add comments about object re-use to the affected functions.\n2da5e87 [Daniel Darabos] Restore object re-use in EdgePartition.\n0182f2b [Daniel Darabos] Do not re-use objects in the EdgePartition/EdgeTriplet iterators. This avoids a silent data corruption issue (SPARK-1188) and has no performance impact in my measurements. It also simplifies the code.\nc55f52f [Daniel Darabos] Tests that reproduce the problems from SPARK-1188.\n\n(cherry picked from commit 78236334e4ca7518b6d7d9b38464dbbda854a777)\nSigned-off-by: Reynold Xin <rxin@apache.org>","Daniel Darabos",false],["563bfe1dd8f504cd65f9ad1fb64563511b568ce2","83c986bd2081d3fd42a4602027c9d1ec5f2ebb59@cse.ohio-state.edu","SPARK-1935: Explicitly add commons-codec 1.5 as a dependency (for branch-0.9).\n\nThis is for branch 0.9.\n\nAuthor: Yin Huai <huai@cse.ohio-state.edu>\n\nCloses #912 from yhuai/SPARK-1935-branch-0.9 and squashes the following commits:\n\nd7f0f7c [Yin Huai] Explicitly add commons-codec 1.5 as a dependency.","Yin Huai",false],["e03af416cf6947359135299e5550108ecb8001b2","c4a2c8bf37ab1b5dce5d3215d2087e497cded4eb@cloudera.com","SPARK-1917: fix PySpark import of scipy.special functions\n\nhttps://issues.apache.org/jira/browse/SPARK-1917\n\nAuthor: Uri Laserson <laserson@cloudera.com>\n\nCloses #866 from laserson/SPARK-1917 and squashes the following commits:\n\nd947e8c [Uri Laserson] Added test for scipy.special importing\n1798bbd [Uri Laserson] SPARK-1917: fix PySpark import of scipy.special\n\nConflicts:\n\tpython/pyspark/tests.py","Uri Laserson",false],["41e7853fceee444d25e449188fb2bc321925fe46","341565cfe74330d2db49baed916af36bb841e1fc@jadedpixel.com","[SPARK-1468] Modify the partition function used by partitionBy.\n\nMake partitionBy use a tweaked version of hash as its default partition function\nsince the python hash function does not consistently assign the same value\nto None across python processes.\n\nAssociated JIRA at https://issues.apache.org/jira/browse/SPARK-1468\n\nAuthor: Erik Selin <erik.selin@jadedpixel.com>\n\nCloses #371 from tyro89/consistent_hashing and squashes the following commits:\n\n201c301 [Erik Selin] Make partitionBy use a tweaked version of hash as its default partition function since the python hash function does not consistently assign the same value to None across python processes.\n\n(cherry picked from commit 8edc9d0330c94b50e01956ae88693cff4e0977b2)\nSigned-off-by: Matei Zaharia <matei@databricks.com>","Erik Selin",false],["6634a348adfcda4cd50c50bc61927b6ca41f4f5f","54b4f217b63553fdd3d11dc2de54a6d99e9a9f7d@gopivotal.com","SPARK-1790: Update EC2 scripts to support r3 instance types\n\nAuthor: Varakhedi Sujeet <svarakhedi@gopivotal.com>\n\nCloses #960 from sujeetv/ec2-r3 and squashes the following commits:\n\n3cb9fd5 [Varakhedi Sujeet] SPARK-1790: Update EC2 scripts to support r3 instance\n(cherry picked from commit 11ded3f66f178e4d8d2b23491dd5e0ea23bcf719)\n\nConflicts:\n\n\tec2/spark_ec2.py","Varakhedi Sujeet",false],["51f677eb9a4fb2cbef8c62b69b4af3ff02560ac5","c481c5e387de8107aa9fe99c851e11e1ff44d0c3@databricks.com","SPARK-2043: ExternalAppendOnlyMap doesn't always find matching keys\n\nThe current implementation reads one key with the next hash code as it finishes reading the keys with the current hash code, which may cause it to miss some matches of the next key. This can cause operations like join to give the wrong result when reduce tasks spill to disk and there are hash collisions, as values won't be matched together. This PR fixes it by not reading in that next key, using a peeking iterator instead.\n\nAuthor: Matei Zaharia <matei@databricks.com>\n\nCloses #986 from mateiz/spark-2043 and squashes the following commits:\n\n0959514 [Matei Zaharia] Added unit test for having many hash collisions\n892debb [Matei Zaharia] SPARK-2043: don't read a key with the next hash code in ExternalAppendOnlyMap, instead use a buffered iterator to only read values with the current hash code.\n\n(cherry picked from commit b45c13e7d798f97b92f1a6329528191b8d779c4f)\nSigned-off-by: Matei Zaharia <matei@databricks.com>","Matei Zaharia",false],["1d3aab96120c6770399e78a72b5692cf8f61a144","24f70c800c13ae23cbad3ee3edbc742a6ecfb50e@dbtsai.com","[SPARK-1870] Made deployment with --jars work in yarn-standalone mode.\n\nPorted from 1.0 branch to 0.9 branch. Sent secondary jars to distributed cache of all containers and add the cached jars to classpath before executors start.\n\nAuthor: DB Tsai <dbtsai@dbtsai.com>\n\nCloses #1013 from dbtsai/branch-0.9 and squashes the following commits:\n\nc5696f4 [DB Tsai] fix line too long\nb085f10 [DB Tsai] Make sure that empty string is filtered out when we get secondary jars\n3cc1085 [DB Tsai] changed from var to val\nab94aa1 [DB Tsai] Code formatting.\n0956af9 [DB Tsai] Ported SPARK-1870 from 1.0 branch to 0.9 branch","DB Tsai",false],["cc95d978a1b2dab6ed3dd8f73ccf8c299b16fdc0","a1a405b82800b492a7268cba6bc58270d2b527de@apache.org","Spark 1384 - Fix spark-shell on yarn access to secure hdfs - branch-0.9 only\n\nAuthor: Thomas Graves <tgraves@apache.org>\n\nCloses #287 from tgravescs/SPARK-1384 and squashes the following commits:\n\nae9162a [Thomas Graves] SPARK-1384 - fix spark-shell on yarn access to secure HDFS","Thomas Graves",false],["706e38fd7f9a0a6dd152ab7014ed35d068c355b7","072f9da45395737c9a1b2c02c185e754199f4da0@gmail.com","[SPARK-1998] SparkFlumeEvent with body bigger than 1020 bytes are not re...\n\nflume event sent to Spark will fail if the body is too large and numHeaders is greater than zero\n\nAuthor: joyyoj <sunshch@gmail.com>\n\nCloses #951 from joyyoj/master and squashes the following commits:\n\nf4660c5 [joyyoj] [SPARK-1998] SparkFlumeEvent with body bigger than 1020 bytes are not read properly\n(cherry picked from commit 29660443077619ee854025b8d0d3d64181724054)\n\nSigned-off-by: Patrick Wendell <pwendell@gmail.com>","joyyoj",true],["69d5f87a6a3be89696af29fcad99a17305358e4e","47f947204c0a430d1f3d77ee35599c230fda5709@gmail.com","Merge branch 'branch-0.9' of github.com:apache/spark into master-csd","Mark Hamstra",true]],"size":12,"ref":"refs/heads/master-csd","head":"69d5f87a6a3be89696af29fcad99a17305358e4e"}