Yelp/mrjob — Issue #233: Run portions of a job
Author: nuance
Created: 2011-10-14 23:13:18+00:00
State: open

My development flow for most multi-step jobs is something like this:

cat input | python my_job.py --step-num 0 --mapper > step-1-mapper
cat step-1-mapper | sort > step-1-sorted
(you get the idea)

I'll sometimes find bugs in step 1 only after running step 2 or 3, which means I have to re-run all preceding steps. It'd be really convenient if I could tell mrjob to run steps 1-2 and output the intermediate result.