Yelp/mrjob — Issue #263: allow saving inter-step output to s3 for emr jobs
Author: l33twolf
Created: 2011-11-11 18:39:18+00:00
State: open

in the current dev branch, _s3_step_output_uri and _s3_step_input_uri both make use of hdft:// paths for intermediate data between steps.

this is fine if a cluster is running with only master and core type instances. but if we wanted to add task instances (which can be dynamically added/removed from a running cluster) we no longer have access to hdfs on these nodes and cannot store to/load from hdfs on them.

potentially we could decide this if there are any task instances running when a step is created, but it may be better to allow this as a command line option since task instances can be added at any time and it would be likely that someone would know ahead of time if they were planning on resizing their cluster in the future.