apache/spark — pull request #2438 by nfergu (2014-09-17 22:10:02+00:00)
Title: [SPARK-3051] Support looking-up named accumulators in a registry

This proposal builds on SPARK-2380 (Support displaying accumulator values in the web UI) to allow
named accumulables to be looked-up in a "registry", as opposed to having to be passed to every
method that need to access them. See the JIRA ticket (SPARK-3051) for some more details, including
an example use case.

An AccumulableRegistry object is provided to allow accumulables to be looked-up by name at task
execution time. This requires named accumulables to be broadcast to the executors before the task
is executed. This is taken care of in the DAGScheduler in much the same way as for Tasks.

Accumulables were already stored in thread-local variables in the Accumulators object, so exposing
these in the registry was simply a matter of wrapping this object, and keying the accumulables by
name (they were previously keyed only by ID).

Note that Accumulables cannot be looked-up from the registry in the driver program; they can only be
obtained while an operation is being performed on an RDD (a task is executing).

One important thing to note in the implementation is that it is important that we we deserialize
any named accumulators that have been broadcast before those that are explicitly passed with the task
as we want the explicitly-passed ones to override the broadcast ones (otherwise the explicitly passed
ones would not work). This may be a little brittle, but there are tests (in AccumulatorSuite) that will break
if this is ever changed.