klbostee/dumbo — pull request #75 by jso (2013-08-14 21:02:13+00:00)
Title: new "parallel unix" backend for running jobs over multiple local processors

I wrote this backend to enable local dumbo jobs to leverage multiple processor cores.

Minimal usage example, which will run 4 mappers in parallel and then run 4 reducers:
dumbo -input [infiles] -output [outpath] -punix yes -tmpdir [tmppath] -nmappers 4 -nreducers 4

Along the way, I also added a few additional options and features.

These new options are backend-agnostic:
* inputfile: which is used instead of "input". This is helpful for when the input files are too long for the command line buffer, i.e. for big jobs.
* shell: to specify the shell to be executed instead of the default '/bin/sh'

These are specific to the new parallel unix backend:
* punix: flag to enable this backend
* nmappers: the number of mappers to run at the same time
* nreducers: the number of reducers to use, which all run simultaneously
* tmpdir: local file system path to store temporary files
* permapper: the number of input files to be handled by each mapper process. This reduces the turnover rate of creating new processes, which can be beneficial for reducing file system load if you are running dumbo on a clustered file system (optional)