Yelp/mrjob — Issue #317: copy_s3_to_scratch option
Author: davidmarin
Created: 2011-12-20 23:00:48+00:00
State: open
Labels: Feature

Shivkumar Shivaji is working on a patch to copy keys from S3 to our scratch "directory" (also on S3) before running the job.

S3 is unlike most filessystems in that you can make a copy of the latest version of a key's contents at the same time someone is writing new contents to the key. So this feature would allow us to say, make a database dump to the same key on S3, and always be able to use that database as input even when a new dump is taking place, without EMR getting confused.