apache/incubator-spark — pull request #394 by tdas (2014-01-12 10:26:53+00:00)
Title: Better error handling in Spark Streaming and more API cleanup

Earlier errors in jobs generated by Spark Streaming (or in the generation of jobs) could not be caught from the main driver thread (i.e. the thread that called StreamingContext.start()) as it would be thrown in different threads. With this change, after `ssc.start`, one can call `ssc.waitForStop()` which will be block until the ssc is closed, or there is an exception. This makes it easier to debug.

This change also adds ssc.stop() where you can stop StreamingContext without stopping the SparkContext.

Also fixes the bug that came up with PRs #393 and #381. MetadataCleaner default value has been changed from 3500 to -1.

And changed a lot of protected[streaming] to private[streaming].