mesos/spark — pull request #377 by stephenh (2013-01-15 17:52:37+00:00)
Title: Call executeOnCompleteCallbacks in a finally block.

I had a map operation causing runtime exceptions (due to a bug in my code), but it ended up hanging the whole job because the commons-httpclient in Hadoop's NativeS3FileSystem ran out of connections.

I saw the recently-added executeOnCompleteCallback to call reader.close(), which looks great, but just needs to be called in a finally.

After this change, my console no longer hangs, but successfully gets the "Failed to run" job error.

I didn't see an existing ResultTaskTest or ResultTaskSuite, so assume adding a test for this sort of change is a good idea. My only hesitation is that I don't think I really need a full SparkContext spun up, but RDD requires one, and it seemed non-trivial to try and stub/fake out.