Yelp/mrjob — Issue #295: combiners in local mode don't work on Windows
Author: davidmarin
Created: 2011-11-29 20:26:39+00:00
State: open
Labels: Bug

`mrjob/examples/mr_word_freq_count.py` doesn't work on Windows because of the way we handle the shell pipeline, we actually construct a command line like `mapper | sort | combiner` and pass it to the shell.

I've gotten part way through fixing this, but it seems like the underlying problem is that `mrjob.util.cmd_line()` only really works with UNIX (it uses `pipes.quote()`, and `pipes` is a Unix Python module). Probably the best fix is to set up a pipeline using `subprocess` and avoid the shell entirely.