Yelp/mrjob — Issue #346: describe_all_job_flows() API issues
Author: davidmarin
Created: 2012-01-30 19:15:30+00:00
State: open
Labels: Bug

`mrjob.emr.describe_all_job_flows()` gets around EMR's 500-job-flow limit by walking backward in time, using the earliest create time returned as the `created_before` for the next query.

Where this goes wrong is, the first time we query EMR (without setting `created_before` and or `created_after`, it returns *all* running job flows (regardless of create time) and then the most recently created jobs, which means we can set `created_before` too early for the next query, and skip a lot of job flows.

We need to read the relevant docs carefully, and come up with an appropriate fix.