mesos/spark — pull request #669 by jason-dai (2013-07-03 12:30:44+00:00)
Title: Shuffle consolidation

In Spark, it is common practice (and usually preferred) to launch a large number of small tasks, which unfortunately can create an even larger number of very small shuffle files – one of the major shuffle performance issues. To address this issue, this change will combine multiple such files (for the same reduce partition or bucket) into one single large shuffle file. Please see [SPARK-751](https://spark-project.atlassian.net/browse/SPARK-751) for the detailed design.