GitHub PushEvent at 2014-11-27 01:30:08
Repository: spark
Actor: dbtsai
Payload: {"shas":[["5af53ada65f62e6b5987eada288fb48e9211ef9d","fd726277649d3b2e3ee5d332dca6c89739b68aa5@gmail.com","[SPARK-732][SPARK-3628][CORE][RESUBMIT] eliminate duplicate update on accmulator\n\nhttps://issues.apache.org/jira/browse/SPARK-3628\n\nIn current implementation, the accumulator will be updated for every successfully finished task, even the task is from a resubmitted stage, which makes the accumulator counter-intuitive\n\nIn this patch, I changed the way for the DAGScheduler to update the accumulator,\n\nDAGScheduler maintains a HashTable, mapping the stage id to the received <accumulator_id , value> pairs. Only when the stage becomes independent, (no job needs it any more), we accumulate the values of the <accumulator_id , value> pairs, when a task finished, we check if the HashTable has contained such stageId, it saves the accumulator_id, value only when the task is the first finished task of a new stage or the stage is running for the first attempt...\n\nAuthor: CodingCat <zhunansjtu@gmail.com>\n\nCloses #2524 from CodingCat/SPARK-732-1 and squashes the following commits:\n\n701a1e8 [CodingCat] roll back change on Accumulator.scala\n1433e6f [CodingCat] make MIMA happy\nb233737 [CodingCat] address Matei's comments\n02261b8 [CodingCat] rollback  some changes\n6b0aff9 [CodingCat] update document\n2b2e8cf [CodingCat] updateAccumulator\n83b75f8 [CodingCat] style fix\n84570d2 [CodingCat] re-enable  the bad accumulator guard\n1e9e14d [CodingCat] add NPE guard\n21b6840 [CodingCat] simplify the patch\n88d1f03 [CodingCat] fix rebase error\nf74266b [CodingCat] add test case for resubmitted result stage\n5cf586f [CodingCat] de-duplicate on task level\n138f9b3 [CodingCat] make MIMA happy\n67593d2 [CodingCat] make if allowing duplicate update as an option of accumulator","CodingCat",true]],"size":1,"ref":"refs/heads/master","head":"5af53ada65f62e6b5987eada288fb48e9211ef9d"}