apache/incubator-spark — pull request #559 by tdas (2014-02-07 22:53:46+00:00)
Title: Improved NetworkReceiver in Spark Streaming to allow for a more graceful StreamingContext shutdown

Current version of StreamingContext.stop() directly killed all the receivers without waiting for data received to be persisted and processed. This leads to a large window of data loss. This PR primarily addresses that. It also fixes the NetworkReceiver API to be more stable while allowing future optimizations. This has been a semi-public API (though not explicitly mentioned anywhere) and is not heavily used. However, for future stability, this API change is important.

The detailed changes are as follows
- Redefined NetworkReceiver public API for future stability. If any one has written their own NetworkReceiver using the [custom receiver guidelines](http://spark.incubator.apache.org/docs/latest/streaming-custom-receivers.html), then he/she can migrate very easily. Instead of creating the `blockGenerator` object and using `blockGenerator += data` to push received data to Spark, one can now calls `store(data)`. The core interface of `onStart()` and `onStop()` remains the same.
- Changed all the receivers to use the new API.
- Added an additional flag to `StreamingContext.stop()` to specify whether to shutdown gracefully. Without setting the flag, the system will shutdown immediately (current default behavior). With the flag set, the system may take time to shutdown to ensure data received by the system is processed completely.
- Clearly defined the behavior of `StreamingContext.start()` and `stop()` - calling stop() twice or calling stop() before start() will give no exception. However, calling start() twice or start() after stop() throws an exception as they are a strong indicator of incorrect control flow.