Yelp/mrjob — Issue #262: Support for changing protocols between dynamically created steps
Author: l33twolf
Created: 2011-11-10 22:29:11+00:00
State: open

i'm using the latest development code and I am having trouble running iterative jobs where i can't know the number of steps i will need upfront and my initial protocol is different than my "internal" (between dynamically created steps) input protocol (RawValueProtocol vs. PickleProtocol) and my final output format is different than the output format of dynamically created steps (again, raw vs. pickle).

Ideally, I'd love a built in way to have a job runner that dynamically creates more steps (or not) based on some post-run condition of the previous step (existence of a file, something in the output, etc).

Right now I'm trying to do this manually with multiple calls to make_runner().

I can override my MRJob instance (called job) job.args attribute and successfully point the input of my next step to the output of the previous one, but for some reason no matter what I change (job.options.input_protocol, job.INPUT_PROTOCOL, job.DEFAULT_INPUT_PROTOCOL) i can't get dynamically created steps from subsequent calls to make_runner on this job to change to use PickleProtocol.