apache/incubator-spark — pull request #303 by andrewor14 (2013-12-27 08:13:58+00:00)
Title: External Sorting for Aggregator and CoGroupedRDDs

The target issue for this patch is the out-of-memory exceptions triggered by aggregate operations such as reduce, groupBy, join, cogroup. The existing AppendOnlyMap used by these operations resides purely in memory, and grows with the size of the input data until the amount of allocated memory is exceeded. Under large workloads, this problem is aggravated by the fact that OOM frequently occurs only after a very long (> 1 hour) map phase, in which case the entire job must be restarted.

The solution is to spill the contents of this map to disk once a certain memory threshold is exceeded. This functionality is provided by ExternalAppendOnlyMap, which additionally sorts this buffer before writing it out to disk, and later merges these buffers back in sorted order.

Under normal circumstances in which OOM is not triggered, ExternalAppendOnlyMap is simply a wrapper around AppendOnlyMap and incurs little overhead. Only when the memory usage is expected to exceed the given threshold does ExternalAppendOnlyMap spill to disk.

By default, ExternalAppendOnlyMap is turned off, but can be enabled via `spark.shuffle.externalSorting`. Additionally, the spill threshold is specified by `spark.shuffle.buffer.mb`, and allowed a margin of safety through `spark.shuffle.buffer.fraction`.

Still forthcoming are user documentation and automatic testing.