apache/spark — pull request #2696 by JoshRosen (2014-10-07 21:48:43+00:00)
Title: [SPARK-2321] [WIP] Stable pull-based progress / status API

This is a WIP pull request that proposes a stable, pull-based progress / status API for Spark (see [SPARK-2321](https://issues.apache.org/jira/browse/SPARK-2321)). For now, I'd like to discuss the basic implementation, API names, and overall interface design. Once we arrive at a good design, I'll go back and add additional methods to expose more information via these API.

#### Design goals:

- Pull-based API
- Usable from Java / Scala / Python
- Can be extended to expose more information without introducing binary incompatibilities.
- Returns immutable objects.
- Don't leak any implementation details, preserving our freedom to change the implementation.

#### Implementation:

- Add public methods (`getJobInfo`, `getStageInfo`) to SparkContext to allow status / progress information to be retrieved.
- Add public interfaces (`SparkJobInfo`, `SparkStageInfo`) for our API return values. These interfaces consist entirely of Java-style getter methods. The interfaces are currently implemented in Java. I decided to explicitly separate the interface from its implementation (`SparkJobInfoImpl`, `SparkStageInfoImpl`) in order to prevent users from constructing these responses themselves.
- Move the construction of the web UI's SparkListeners into SparkContext and out of the web UI tabs themselves. This allows us to re-use these listeners in the implementation of this status API. There are a few reasons why this listener re-use makes sense:
- The status API and web UI are guaranteed to show consistent information.
- These listeners are already well-tested.
- The same garbage-collection / information retention configurations can apply to both this API and the web UI.
- Extend JobProgressListener to maintain `jobId -> Job` and `stageId -> Stage` mappings.

#### Open questions:

- I'm open to naming suggestions / improvements.
- There's a "job -> stages -> tasks" hierarchy; how should we expose this in the API responses? Should `SparkJobInfo` only expose stages' ids, requiring consumers to call `getStageInfo` to get information on stages? If it returns `SparkStageInfo` objects, then should it enforce "consistent snapshot" semantics such that once I get a `SparkJobInfo` I have a complete, immutable snapshot of the state of the job's stages and tasks at the time that I call `getJobInfo()`? That would be nice, but potentially expensive for large jobs.
- By implementing the interfaces in Java, we're limited in terms of how much Scala syntactic sugar we can provide for Scala users. For example, paren-less accessor methods don't seem to work on Java interfaces, so this current API means that Scala users have to write

```scala
sc.getJobInfo(0).getStages()(0)
```

to get information on the first stage in a job. On the other hand, this prevents us from introducing incompatibilities if we ever choose to overload the `getStages` method (or any other ones).

#### TODOs:

- [ ] Unit tests
- [ ] Examples
- [ ] Python API
- [ ] Documentation of new configuration options