Yelp/mrjob — Issue #240: add support for sorting values
Author: davidmarin
Created: 2011-10-31 21:01:57+00:00
State: open
Labels: Feature

It would be useful if we could get Hadoop to sort the values passed to any reducer. An example use case is `mrjob.examples.mr_next_word_stats` (in the `development` branch) where we want to know for any given word, the *percentage* of time various words follow it.

To do this without dumping a bunch of stuff into memory, we need to guarantee that the reducers which key on the first word in the pair get the total number of times the first word appears before any information about following words.

Users should be able to trigger this behavior simply by setting `SORT_VALUES = True` in their `MRJob` class.

This is surprisingly difficult and counter-intuitive to get Hadoop to do.