RevolutionAnalytics/RHadoop — Issue #43: finish call in mapreduce
Author: piccolbo
Created: 2012-01-27 07:22:54+00:00
State: open
Labels: complexity-medium, importance-medium, pkg-rmr, type-improvement

it could be useful to have mapreduce execute a function after the last input record in map or reduce or combine. Introduced in HADOOP-20, they are branded there as simple cleanup functions, but I think they can be used to implement a number of map-side programming patterns as found in Cascading. The simplest one can think of is a almost map side counter, where the map is an accumulator and the map-finish function would output only the total of the count for that mapper. This way the reducer has only one input record per mapper. This can be achieved also with combiners, but with the current list interface one could run out of memory. Instead with the map-finish approach one can do it in constant memory,