Yelp/mrjob — pull request #577 by unshift (2012-11-14 22:46:27+00:00)
Title: Fix prefixing already-prefixed ls output

When `lsr`'ing a fully-qualified HDFS path, hdfs:// prefixes are returned (using Cloudera CDH4):

$ hadoop fs -lsr hdfs:///tmp
lsr: DEPRECATED: Please use 'ls -R' instead.
drwxr-xr-x - mapred hdfs 0 2012-11-13 20:21 hdfs:///tmp/mapred
drwx------ - mapred hdfs 0 2012-11-14 17:36 hdfs:///tmp/mapred/system

This was causing the `lsr` call to double-prefix paths when `cat`ing my output:

```
HADOOP: Job complete: job_201211141612_0018
HADOOP: Output: hdfs:///user/mperry/tmp/mrjob/mr.mperry.20121114.222915.615989/output
STDOUT: packageJobJar: [/tmp/hadoop-mperry/hadoop-unjar5877860083449559295/] [] /tmp/streamjob8356065975532485730.jar tmpDir=null
Counters from step 1:
(no counters found)
Streaming final output from hdfs:///user/mperry/tmp/mrjob/mr.mperry.20121114.222915.615989/output
STDERR: -cat: java.net.UnknownHostException: hdfs

STDERR: Usage: hadoop fs [generic options] -cat [-ignoreCrc] ...

removing tmp directory /tmp/mr.mperry.20121114.222915.615989
deleting hdfs:///user/mperry/tmp/mrjob/mr.mperry.20121114.222915.615989 from HDFS
Traceback (most recent call last):
File "./mr.py", line 17, in
MRWordCounter.run()
File "/home/mperry/mrjob/mrjob/job.py", line 483, in run
mr_job.execute()
File "/home/mperry/mrjob/mrjob/job.py", line 501, in execute
super(MRJob, self).execute()
File "/home/mperry/mrjob/mrjob/launch.py", line 146, in execute
self.run_job()
File "/home/mperry/mrjob/mrjob/launch.py", line 210, in run_job
for line in runner.stream_output():
File "/home/mperry/mrjob/mrjob/runner.py", line 477, in stream_output
for line in self._cat_file(filename):
File "/home/mperry/mrjob/mrjob/fs/composite.py", line 71, in _cat_file
for line in self._do_action('_cat_file', path):
File "/home/mperry/mrjob/mrjob/util.py", line 414, in read_file
for line in f:
File "/home/mperry/mrjob/mrjob/fs/hadoop.py", line 180, in stream
raise IOError("Could not stream %s" % filename)
IOError: Could not stream hdfs://hdfs:///user/mperry/tmp/mrjob/mr.mperry.20121114.222915.615989/output/part-00000
```

This patch makes that prefixing optional, and fixes my use case (and continues passing unit tests)