apache/spark — pull request #204 by andrewor14 (2014-03-22 05:36:41+00:00)
Title: [SPARK-1276] Add a HistoryServer to render persisted UI

Currently, a persisted UI can only be rendered through the standalone Master. This greatly limits the use case of the new feature of being able to log the details of a Spark application as events, since many people also run Spark on Yarn / Mesos.

This PR introduces a new entity called the HistoryServer, which, given a log directory, keeps track of all completed applications independently of a Spark Master. Unlike Master, the HistoryServer needs not be running while the application is still running. It is relatively light-weight in that it only maintains static information of applications after-the-fact.

To quickly test it out, generate event logs with ```spark.eventLog.enabled=true``` and run ```sbin/start-history-server.sh ```. Your HistoryServer awaits on port 18080.

A few other changes introduced in this PR include refactoring the WebUI interface, which is beginning to have a lot of duplicate code now that we add more functionality to it. Two new SparkListenerEvents have been introduced (SparkListenerApplicationStart/End) to keep track of application name and start/finish times. This PR also clarifies the semantics of the ReplayListenerBus introduced in #42.

A potential TODO in the future (not part of this PR) is to render live event logging applications in addition to just completed applications. This is useful if an application fails, in which case our current HistoryServer does not render the associated UI unless the user manually signals application completion. Processing the event logs in this case becomes significantly more complicated, however, because we must deal with multiple levels of streams that may each have arbitrary behavior if we want to avoid processing the entire file over and over again.

Comments and feedback are most welcome.