mesos/spark — pull request #648 by shivaram (2013-06-13 22:02:29+00:00)
Title: Shuffle fixes and cleanup

This set of patches contains fixes for issues uncovered while debugging a large shuffle using netty (SPARK-757). It contains two major groups of changes:

Netty shuffle fetcher changes:
a. FileClient now uses a connection timeout that can be set using a flag. Default is 60s.
b. Errors during a shuffle fetch are now handled correctly -- We use a variable to track if the callback for adding a fetch result has been invoked and add a failed result (with size -1) if the fetch failed.
c. Minor cleanup of BlockFetcherIterator to handle zero-sized local and remote blocks.

DiskStore changes:
a. The JavaSerializer adds a 9-byte header on initialization which is bad if a file is otherwise empty during a large shuffle. This patch postpones the opening of the file until the first write happens in the DiskBlockWriter.
b. if a ShuffleMap task is rescheduled on the same machine as a previously failed task, we may have a partial shuffle file present. This change deletes a shuffle file if it already exists.

Thanks to @etrain for help with debugging.