dmwm/WMCore — pull request #4127 by dballesteros7 (2012-10-03 15:40:39+00:00)
Title: Smart lumi splitting

The purpose of this algorithm is to dynamically split jobs by lumis
trying to make the jobs have a target number of events which is
ultimately the unit that determines job running time, disk space used
and memory constraints.
However this differentiates from the EventBased algorithm since
it doesn't split indiviual lumis, the basic unit for the splitting is the
lumi and every job is guaranteed to have at least one full lumi, this implies
that the number of events in a job will never be the exact target
but on average all jobs have about that number.
Also this includes the option to define a hard limit on the events
a job can process, if a file has a single lumi and more events
that such limit that job is created and marked as bad. This means
that this job will be moved from created to createdfailed by the
JobCreator.

The ErrorHandler is now configured to immediatly exhaust createfailed
jobs. The reason is that the createfailed state didn't make much sense,
the transitions defined it to go from createfailed to cooloff and then to
new. But a job that is in createfailed, had to actually be created otherwise
it wouldn't exist in WMBS. I changed the transitions to allow from created to createfailed and changed the transitions from createdcooloff/paused to new
for created instead which seems to be the logical step, since jobs in "new" state are not picked up by any piece of code.
However for now the createcooloff state is unreachable as the ErrorHandler will always take createfailed and send them to exhausted.

The algorithm takes the following inputs:

- events_per_job : This is the main parameter and defines the target events we want in each job. The algorithm will split lumis according to average events per lumi in each file
- max_events_per_lumi: Hard limit that defines jobs to fail when there are too many events in a lumi
- halt_on_file_boundaries, splitOnRun, etc..: Usual LumiBased algorithm parameters

There are two commits here:

- First is all client-side and adds the support for the algorithm
- Second is reqmgr-side which allows to set the splitting to requests