mesos/spark — pull request #635 by jason-dai (2013-05-31 08:19:05+00:00)
Title: Consolidations of shuffle files from different map tasks

A Spark job can have many small tasks, which in turn generates many small shuffle files - one of the major shuffle performance issues. This change combines multiple such files (for the same reducer) into one single file as follows (using the shuffle manager introduced by Reynold's https://github.com/mesos/spark/pull/587):

1) On each slave, the shuffle manager maintains a shuffle block pool for each shuffle

2) Each pool maintains a list of shuffle block group; a ShuffleMapTask acquires a free group when it needs to write its results, and returns the group when it's done

3) Each group maintains a list of writers, each for a different bucket (reduce partition)