apache/spark — pull request #221 by concretevitamin (2014-03-25 06:16:16+00:00)
Title: Add getListenerBus() in SparkContext.

Motivation:

We had a job that collects and analyzes statistics collected by a custom SparkListener. What we realized was that it was possible for our main job to finish first, when the listener event poster thread is still processing events. (In our particular case -- a small job consisting of linear algebra operations -- an average of ~43ms is needed for the main job to wait.) A simple solution to this seems to be gaining access to the listener bus, and thus do something like `sc.getListenerBus().waitUntilEmpty(10000)`.