Yelp/mrjob — Issue #288: sorting step fails on Windows
Author: jretz
Created: 2011-11-28 00:10:46+00:00
State: open
Labels: Bug

The command line `sort` utility on Windows can only take a single input file. It can take input from stdin, but there is no `cat` utility (there is `type`, but again it only takes a single file) so piping would probably have to come from python.

```
C:\Documents and Settings\jimmy\Desktop\mrjob-0.3.0-dev\mrjob\examples>python mr_wc.py --runner=local mr_wc.py
using configs in C:\Documents and Settings\jimmy/.mrjob
This config path is deprecated and will stop working in mrjob 0.4. use ~/.mrjob.conf instead.
no configs for runner type 'local'; returning {}
creating tmp directory c:\docume~1\jimmy\locals~1\temp\mr_wc.jimmy.20111127.234937.453000
> 'C:\Python25\python.exe' mr_wc.py --step-num=0 --mapper 'c:\docume~1\jimmy\locals~1\temp\mr_wc.jimmy.20111127.234937.453000/input_part-00000'
writing to c:\docume~1\jimmy\locals~1\temp\mr_wc.jimmy.20111127.234937.453000\step-0-mapper_part-00000
> 'C:\Python25\python.exe' mr_wc.py --step-num=0 --mapper 'c:\docume~1\jimmy\locals~1\temp\mr_wc.jimmy.20111127.234937.453000/input_part-00001'
writing to c:\docume~1\jimmy\locals~1\temp\mr_wc.jimmy.20111127.234937.453000\step-0-mapper_part-00001
Counters from step 1:
(no counters found)
> sort 'c:\docume~1\jimmy\locals~1\temp\mr_wc.jimmy.20111127.234937.453000\step-0-mapper_part-00000' 'c:\docume~1\jimmy\locals~1\temp\mr_wc.jimmy.20111127.234937.453000\step-0-mapper_part-00001'
writing to c:\docume~1\jimmy\locals~1\temp\mr_wc.jimmy.20111127.234937.453000\step-0-mapper-sorted
STDERR: Input file specified two times.
STDERR:
Counters from step 1:
(no counters found)
Traceback (most recent call last):
File "mr_wc.py", line 47, in
MRWordCountUtility.run()
File "c:\python25\lib\site-packages\mrjob-0.3.0_dev-py2.5.egg\mrjob\job.py", line 498, in run
mr_job.execute()
File "c:\python25\lib\site-packages\mrjob-0.3.0_dev-py2.5.egg\mrjob\job.py", line 514, in execute
self.run_job()
File "c:\python25\lib\site-packages\mrjob-0.3.0_dev-py2.5.egg\mrjob\job.py", line 564, in run_job
runner.run()
File "c:\python25\lib\site-packages\mrjob-0.3.0_dev-py2.5.egg\mrjob\runner.py", line 487, in run
self._run()
File "c:\python25\lib\site-packages\mrjob-0.3.0_dev-py2.5.egg\mrjob\local.py", line 189, in _run
step_type='S', num_tasks=1) # ignore locale
File "c:\python25\lib\site-packages\mrjob-0.3.0_dev-py2.5.egg\mrjob\local.py", line 429, in _invoke_step
self._wait_for_process(proc, step_num)
File "c:\python25\lib\site-packages\mrjob-0.3.0_dev-py2.5.egg\mrjob\local.py", line 594, in _wait_for_process
(proc['args'], returncode))
Exception: Command ['sort', 'c:\\docume~1\\jimmy\\locals~1\\temp\\mr_wc.jimmy.20111127.234937.453000\\step-0-mapper_part-00000', 'c:\\docume~1\\jimmy\\locals~1\\temp\\mr_wc.jimmy.20111127.234937.453000\\step-0-mapper_part-00001'] returned non-zero exit status 1
```