RevolutionAnalytics/RHadoop — Issue #40: prevent map and reduce functions from using stdin and out
Author: piccolbo
Created: 2012-01-11 20:29:55+00:00
State: open
Labels: complexity-medium, importance-medium, pkg-rmr, type-improvement

stdin and stdout have special meaning for streaming, in that they are used to communicate between streaming and map and reduce executables. Even stderr can have a special meaning (it is used to implement counters and statuses) but writing other things to stderr doesn't cause damage. The problem is not when map and reduce are little function of which we know the ins and outs and do not use stdin and stdout, but when we use libraries of which we may not know whether they do IO under some circumstance and we may not be able to change or fix it if they do. So taking a page from hadoopy, it would be nice to divert or redirect all output from within the mapper (say with sink) so that only the blessed streaming stream can be produced (from the map.driver and record.writer specifically). Same for input even if I think that is a less solvable or common occurrence.