Yelp/mrjob — Issue #236: actually stream data from stdin
Author: davidmarin
Created: 2011-10-24 19:44:11+00:00
State: open
Labels: Feature

`MRJob` current accepts data from stdin, but all the runners actually dump stdin to a file because it makes the code simpler.

At least one user (Scott Sweeney) wants to pipe a lot of data into `MRJob` (in `local` mode), so we should re-think this.

What we can do:
* **inline** mode can start streaming immediately
* **local** mode can start streaming immediately (even if there are multiple mappers, we can round-robin)
* **hadoop** mode can use `hadoop fs -put - ...` to stream your data into HDFS
* **emr** mode can, in theory, stream data to S3

Inline mode and hadoop mode should be pretty straightforward. Local mode might be trickier but is do-able.

Streaming to S3 is not well-supported by `boto` for files larger than about 5GB; you have to split them into pieces, and boto doesn't support streaming multipart uploads (it has no problem doing multipart uploads from files). The logical thing would be to add that feature to boto first, and then have mrjob use it. Also, if your md5 checksum fails, you're kind of out of luck because you don't have a copy of the data you just streamed.

One caveat is that it's valid to pass stdin several times to the same job (e.g. `mr_your_job.py - - -`), in which case we still need to dump it to a file. Kind of a pointless thing to support, so we could drop that in the next major release (v0.4).

We might also want to think about whether it's useful to support streams in a more general way (e.g. from unix sockets), and treat files as a "special" (though overwhelmingly common) case. We can use `os.path.isfile()` to check if something is actually a file.