mesos/spark — pull request #305 by woggling (2012-11-16 08:59:27+00:00)
Title: Set default uncaught exception handler to exit

This patch sets a global default uncaught exception handler in SparkEnv which logs the exception and then exits with exit status 1, or an exit status of 2 if an exception occurs while trying to log the exception. This should prevent utility threads in Spark from failing (for example, due to OutOfMemoryErrors or IOExceptions caused by running out disk space) but merely causing Spark overall to stall.

This patch assumes that no Spark code or user code relies on uncaught exceptions causing normal thread termination. I looked at where Spark created utility threads and didn't see any obvious instances of this problem, though I wonder what PipedRDD is supposed to do in the case of a broken pipe.

I'm a bit uncomfortable setting this JVM-wide setting rather than something that might be limited to Spark only. An alternate approach would be to create utility threads within Spark using some wrapper for Thread that sets an uncaught exception handler (or wraps run() appropriately), but this would be more code and some cases would likely be missed.