Yelp/mrjob — pull request #508 by irskep (2012-07-17 17:35:08+00:00)
Title: Refactor of the way steps are defined and passed between job and runner

This branch fixes #441, #478, #479, #499, and #501. **It is only 70% complete** but if there are any major issues, I'd rather know now than after I've done the rest of the work.

**Things that work**

* `--steps` passes a JSON dict with a documented format
* `local` mode supports new format (lots of testing)
* `emr` mode supports new format (nominal but incomplete testing)
* `inline` mode supports new format but intentionally ignores non-script steps
* Command steps (`mapper_cmd`, etc.)
* Pre-task filters (`mapper_filter`, etc.)
* `pick_protocols()` had to be rewritten to account for non-script steps
* New examples. `mr_cmd` lets you write MapReduce jobs on a single shell line. `mr_grep` uses `grep` to filter lines by an extended regular expression.

**Things that don't work**

* Anything to do with the `hadoop` runner. It is completely broken right now.
* Certain kinds of quotes in command strings in the `emr` runner. I know how to fix it but haven't done it yet (see `mrjob.util.bash_wrap`)
* There is no `combiner_filter` due to an implementation detail that has since been worked around, so that might go back in, but I don't consider it a priority.

The docs could use a little more love, but they do convey all the information.