GitHub PushEvent at 2014-05-29 07:43:39
Repository: spark
Actor: asfgit
Payload: {"shas":[["a92900c5aca96d0609682ae9d9893c7d9e0f6327","9d468ac3e51a0dd55cd858437e2237db1fbaae1b@gmail.com","SPARK-1188: Do not re-use objects in the EdgePartition/EdgeTriplet iterators.\n\nThis avoids a silent data corruption issue (https://spark-project.atlassian.net/browse/SPARK-1188) and has no performance impact by my measurements. It also simplifies the code. As far as I can tell the object re-use was nothing but premature optimization.\n\nI did actual benchmarks for all the included changes, and there is no performance difference. I am not sure where to put the benchmarks. Does Spark not have a benchmark suite?\n\nThis is an example benchmark I did:\n\ntest(\"benchmark\") {\n  val builder = new EdgePartitionBuilder[Int]\n  for (i <- (1 to 10000000)) {\n    builder.add(i.toLong, i.toLong, i)\n  }\n  val p = builder.toEdgePartition\n  p.map(_.attr + 1).iterator.toList\n}\n\nIt ran for 10 seconds both before and after this change.\n\nAuthor: Daniel Darabos <darabos.daniel@gmail.com>\n\nCloses #276 from darabos/spark-1188 and squashes the following commits:\n\n574302b [Daniel Darabos] Restore \"manual\" copying in EdgePartition.map(Iterator). Add comment to discourage novices like myself from trying to simplify the code.\n4117a64 [Daniel Darabos] Revert EdgePartitionSuite.\n4955697 [Daniel Darabos] Create a copy of the Edge objects in EdgeRDD.compute(). This avoids exposing the object re-use, while still enables the more efficient behavior for internal code.\n4ec77f8 [Daniel Darabos] Add comments about object re-use to the affected functions.\n2da5e87 [Daniel Darabos] Restore object re-use in EdgePartition.\n0182f2b [Daniel Darabos] Do not re-use objects in the EdgePartition/EdgeTriplet iterators. This avoids a silent data corruption issue (SPARK-1188) and has no performance impact in my measurements. It also simplifies the code.\nc55f52f [Daniel Darabos] Tests that reproduce the problems from SPARK-1188.\n\n(cherry picked from commit 78236334e4ca7518b6d7d9b38464dbbda854a777)\nSigned-off-by: Reynold Xin <rxin@apache.org>","Daniel Darabos",true]],"size":1,"ref":"refs/heads/branch-0.9","head":"a92900c5aca96d0609682ae9d9893c7d9e0f6327"}