apache/spark — pull request #42 by andrewor14 (2014-02-28 01:36:15+00:00)
Title: [WIP] [SPARK-1132] Persisting Web UI through refactoring the SparkListener interface

The fleeting nature of the Spark Web UI has long been a problem reported by many users: The existing Web UI disappears as soon as the associated application terminates. This is because SparkUI is tightly coupled with SparkContext, and cannot be instantiated independently from it. To solve this, some state must be saved to persistent storage while the application is still running.

The approach taken by this PR involves persisting the UI state through SparkListenerEvents. This requires a major refactor of the SparkListener interface because existing events (1) maintain deep references, making de/serialization is difficult, and (2) do not encode all the information displayed on the UI.

The new architecture is as follows - The SparkUI registers a central gateway listener with the SparkContext for events. For each event this gateway listener receives, it logs the event to a file and relays it to all of its children listeners (e.g. ExecutorsListener) used by the UI. Each of these children listeners then construct the appropriate information from these events and supplies it to its parent UI (e.g. ExecutorsUI), which renders the associated page(s) on demand. Then, after the SparkContext has stopped, the SparkUI can be revived by replaying all the logged events to the children listeners through the gateway listener.

This patch is WIP and still expects additional features. The main TODO's at this point include adding support for logging into HDFS, handling long running jobs (perhaps through checkpointing), and performance testing. To try this patch out, run your Spark application to completion as usual, then run ```bin/spark-class org.apache.spark.ui.UIReloader /tmp/spark-```. Your revived Spark Web UI awaits on port 14040.

More details can be found in the commit messages, comments within the code, and the [design doc](https://spark-project.atlassian.net/secure/attachment/12900/PersistingSparkWebUI.pdf). Comments and feedback are most welcome.