Yelp/mrjob — Issue #391: terminate_idle_job_flows should lock pooled job flows
Author: davidmarin
Created: 2012-03-09 22:21:48+00:00
State: open
Labels: Bug

We haven't seen this in practice, but there is a potential race condition where a job attempts to join a pooled job flow at the same time `mrjob.tools.emr.terminate_idle_job_flows` is terminating it. `terminate_idle_job_flows` should try to lock the job flow first (using our S3 locking) before terminating it, and if it can't, it should assume that its in use.

We should double-check that our S3 locks expire. Would be really bad for an idle job flow to end up in a permanently locked state.