mesos/spark — pull request #315 by squito (2012-11-30 07:00:48+00:00)
Title: Counters

This is a first pass at addressing https://spark-project.atlassian.net/browse/SPARK-603
I don't think this is ready to be merged in, but ready for a discussion of api & implementation (it works, at least)

Two major changes here:
1) Every RDD can set a "completion function" -- a function that will be run whenever the RDD is actually generated. When execution actually happens, the completion function is run for *all* RDDs that were executed (eg., 3 maps in a row can specify 3 completion functions, all are executed). By default, the value of all accumulators are printed out.
2) "Counters" are a simple accumulator that are automatically created for every job that is run, and that can be easily incremented.

I don't really like the name "completionFunction", but not sure what else to call it. Not really sure what will happen when if task is killed and gets re-run -- I think it will only get called once, but I'm not positive.

Getting Counters to work was much harder than I expected. b/c the closure doesn't contain a reference to the accumulator, they aren't serialized via normal accumulator serialization, so required special logic.