[SPARK-4145] Web UI job pages

This PR adds two new pages to the Spark Web UI:

- A jobs overview page, which shows details on running / completed / failed jobs.
- A job details page, which displays information on an individual job's stages.

The jobs overview page is now the default UI homepage; the old homepage is still accessible at /stages.

In some situations, the jobs page may display "No information on stage" for some stages of pending jobs that have not begun executing.  This is due to some limitations in how JobProgressListener finds out about stages.  We can address this later as part of a separate scheduler PR.

### Screenshots

#### New UI homepage

![image](https://cloud.githubusercontent.com/assets/50748/4989750/801ff1ce-6932-11e4-8d1a-1d2a3c527227.png)


#### Job details page

(This is effectively a per-job version of the stages page that can be extended later with other things, such as DAG visualizations)

![image](https://cloud.githubusercontent.com/assets/50748/4837239/79ee6b1e-5fd2-11e4-839d-6cbb42fb01c0.png)

One idea I had was to just label the column as Job ID if none of the jobs
are part of a job group.  Is that easy to do?

On Tue, Nov 11, 2014 at 2:15 PM, Josh Rosen <notifications@github.com>
wrote:

> In core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala:
>
> > +/** Page showing list of all ongoing and recently finished jobs */
> > +private[ui] class AllJobsPage(parent: JobsTab) extends WebUIPage("") {
> > +  private val sc = parent.sc
> > +  private val listener = parent.listener
> > +
> > +  private def getSubmissionTime(job: JobUIData): Option[Long] = {
> > +    for (
> > +      firstStageId <- job.stageIds.headOption;
> > +      firstStageInfo <- listener.stageIdToInfo.get(firstStageId);
> > +      submitTime <- firstStageInfo.submissionTime
> > +    ) yield submitTime
> > +  }
> > +
> > +  private def jobsTable(jobs: Seq[JobUIData]): Seq[Node] = {
> > +    val columns: Seq[Node] = {
> > +      <th>Job Id (Job Group)</th>
>
> Job groups are perhaps an obscure feature, but they can be useful in
> environments where multiple users interact with a shared SparkContext (such
> as a job server).
>
> Do you think there's a clearer way to label this that's less confusing?
> Job groups might be an uncommon feature that most users aren't aware of /
> don't need.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/3009/files#r20185389>.
>