Yelp/mrjob — Issue #239: audit_usage script should handle pooling
Author: davidmarin
Created: 2011-10-31 20:54:11+00:00
State: open
Labels: Feature

Currently, `mrjob.tools.emr.audit_usage` only looks at the name and owner of job flows, but not the jobs in the job flows.

This will make it difficult to get meaningful stats when using pooling, since any job can go into any free job flow.

Fortunately, `mrjob` already puts useful information into the "Name" field that can be used to identify the job.

In general, jobs should be responsible for the time they used to run, plus any idle time afterwards. In addition, whichever job's name is on the job flow (this will be the first job under pooling) is responsible for time between when the job flow started, and when the first job started running.

It'd also be nice to have stats by pool name.

We'll do this once Yelp starts using pooling in development and have some meaningful examples to test on.