mesos/spark — pull request #659 by pwendell (2013-06-25 04:23:14+00:00)
Title: Refactoring, Consolidation, and Updates to the Web UI

This patch performs an internal re-factoring of the Spark Web UI and adds a new UI component that exposes stage progress and metrics.

The re-factoring removes the dependency on Spray and instead uses Scala's built in XML support and a lightweight server based on Jetty. The changes also consolidate code related to the spark ui in a "ui" package which can server as a point for future expansion. Certain very common rendering operations (such as enumerating a list of items) are factored out into utility functions.

The expansion adds a stage-level monitoring feature which exposes the progress of ongoing stages and statistical metrics for both ongoing and historical stages. It is meant to to be a modest first step towards a more comprehensive workflow UI. Nonetheless, it exposes some very useful previously hidden metrics, such as distribution data about tasks within a job, and an adjacent view of the service time and locality properties of given tasks.

So far this has been tested using a local standalone cluster with a variety of job conditions, including failures. I'm in the process of doing more rigorous tests, but wanted to put this up for review.