apache/spark — pull request #290 by tdas (2014-04-01 20:58:41+00:00)
Title: [SPARK-1386] Web UI for Spark Streaming [WIP]

When debugging Spark Streaming applications it is necessary to monitor certain metrics that are not shown in the Spark application UI. For example, what is average processing time of batches? What is the scheduling delay? Is the system able to process as fast as it is receiving data? How many records I am receiving through my receivers?

While the StreamingListener interface introduced in the 0.9 provided some of this information, it could only be accessed programmatically. A UI that shows information specific to the streaming applications is necessary for easier debugging. This PR introduces such a UI. It shows various statistics related to the streaming application. Here is a screenshot of the UI running on my local machine.
http://i.imgur.com/Sf9TnG5.png

This is still a WIP. The UI current runs on a different port (6060). We do not want to make everyone open a new port in their firewalls, so we would like to integrate this UI into the Spark UI running at 4040. This is still to be done.