dmwm/WMCore — pull request #4536 by dballesteros7 (2013-04-04 21:17:38+00:00)
Title: Include performance estimates in Condor JDLs

On the way to better provisioning, some of the splitters are able
to "predict" how long a job will take based on time per event estimates,
how much disk it will require based on estimates of output size per event
and how much memory the job will require. This information is included in
the condor JDLs in the following values:

+MaxWallTimeMins : Time in minutes, used already by CRAB2, coordinating with the FE admins if central production is already set to use it.
request_memory : Standard condor attribute to request memory during the matchmaking. It's in MB
request_disk : Standard condor attribute to request disk space during the matchmaking. It's in KB

The list of algorithms which support estimates follows:

EventAwareLumiBased, EventBased, LumiBased, FileBased: Provides estimates for all 3 values, this covers all Processing/Production/Skim tasks.
WMBSMergeBySize, ParentlessMergeBySize: Provide very accurate estimates for the disk space. This covers all merge tasks.

The tasks which provide no estimates are:

Harvest, Cleanup, LogCollect : All these tasks are very fast and with very little requirements.
We may tailor some defaults for them in the future.

The initial idea included changing the estimates for time per event, size per event
and memory dynamically using couchDB information. However after giving some thought
I realized that with the current schema this is not useful and can be harmful. The case
is when the second block is split after only fast jobs have run, these will have
low time per event and memory usage which will trigger bad pilots for future jobs
and won't allow for correction.

The estimates will be only reliable in a model where we first fully process at least a block
before splitting the rest of the workload. This would give some guarantee that the sample
used for the performance information is somewhat representative.

Fixes #4472