Yelp/mrjob — Issue #310: roll back #66 (ec2_instance_type trickiness)
Author: davidmarin
Created: 2011-12-14 21:24:52+00:00
State: open
Labels: Bug

When you run a job flow with multiple instances on EC2, the master node doesn't actually run tasks; it just coordinates the other nodes. In #66, we thought we could save a little money by not having the `ec2_instance_type` option apply to the master node when there are multiple instances.

After deploying v0.3.0 at Yelp and getting a variety of sporadic errors having to do with jobs not being able to contact the task tracker, HDFS running out of memory, etc., it became clear that the master node is still doing a lot of work.

We should roll back this change, both to save other people the chaos we experienced, and because it's easier to understand `ec2_instance_type` if it just applies to all instance types.

(It will continue to be possible to explicitly set different master and slave instance types and save money that way.)