Yelp/mrjob — Issue #280: explicitly control number of subprocesses in local mode
Author: davidmarin
Created: 2011-11-18 19:58:07+00:00
State: open
Labels: Feature

Local mode currently reads `mapreduce.job.maps` and `mapreduce.job.reduces`, which tell us the suggested number of map/reduce tasks. Local mode then launches one process per task.

Where this goes wrong is if you have more input files than suggested tasks, local mode is forced to have at least one task per input file, so you end up launching more subprocesses than expected.

There should be an explicit way to control the number of subprocesses in local mode, and some sort of queue in case we have more tasks than subprocesses.