mesos/spark — pull request #408 by woggling (2013-01-23 09:47:32+00:00)
Title: Use MapOutputTracker generation for failures in DAGScheduler and test failures more

Use MapOutputTracker generation instead of a blacklist of nodes in DAGScheduler to handle failed shuffle results from failed nodes.

Also, increment MapOutputTracker generation for all shuffle tasks, avoiding situations where an map output list with holes is cached. (This fix is suboptimal since it may result in us clearing cached map statuses more often than necessary and more often than the existing code; alternatives welcome.)

Also, add several tests to DistributedSuite which inject failures, including repeated node failures involving shuffles.