NICTA/scoobi — Issue #55: In-mapper combining
Author: blever
Created: 2012-03-08 01:15:02+00:00
State: open
Labels: Improvement

Scoobi has `Combine` as a primitive node type in its logical graphs/plans. The advantage of "combining" is to increase local aggregation so as to reduce disk and network I/O. It also helps solve the reduce straggler problem - that is, the reducers that get a highly-distribued skew of values for a particular key (e.g. "the" in word-count). Scoobi implements the `Combine` node as a combination of the Hadoop Combiner and Hadoop Reducer classes. The disadvantage of Hadoop's "combiner" feature is that it is provided only as a hint to the framework and may not necessarily run.

Instead, combining can be move inside the "mapper" by using a Map container per map task. In-mapper combining can also further improve on Hadoop combining - the output of the map doesn't need to be written to the local disk, serialised, etc. Experiment with different techniques for determining when the Map becomes too large and needs to be flushed.