bwhite/hadoopy — Issue #53: Make hadoopy.run entry point allow for jobconfs and command line execution
Author: bwhite
Created: 2012-02-07 15:01:54+00:00
State: open

Allowing jobconfs to be specified in the job itself is useful for jobs when....
1. There is a modified partitioner where the semantics externally are complex and correct execution requires it.
2. Extra memory is required in the java stack (I run into this often when processing videos, currently when you launch it you set this value but it is nicer to hide the boilerplate)
3. To be able to launch the job with no additional jobconfs (useful for the next modification)

Allow running jobs on the command line such as...
python your_job.py launch_frozen input_path output_path

where launch_frozen can also be launch or launch_local (i.e., any of the launch tools)

This modification lowers the single job barrier significantly by not requiring another file.