Yelp/mrjob — pull request #674 by davidmarin (2013-07-21 03:37:43+00:00)
Title: Support non-hdfs URIs (patched)

Here's what I had in mind with my comments on #668. I kept `fully_qualify_hdfs_path()` as a function, and renamed the option to `hadoop_uri_scheme` to be more precise and less confusing.

Now that I've done this and thought about it a bit more, I'm a bit puzzled as to what your use case is.

I guess it's not clear that you can do something like `--hdfs-scratch-dir maprfs:///path/to/tmp/dir`. Maybe a better solution would be to rename/alias the option to `--hadoop-scratch-dir`.

It's true there's no way to currently no way to use `maprfs://user/$USER` as the temp directory (that is, automatically have a different temp dir for each user somewhere other than on HDFS), but we could add [os.path.expandvars()](http://docs.python.org/2/library/os.path.html#os.path.expandvars) capability to the `--hadoop-scratch-dir` option.