dmwm/WMCore — pull request #4425 by dballesteros7 (2013-02-15 23:24:49+00:00)
Title: Open requests feature

Allows a request with an input dataset to continue acquiring and processing new
blocks after assignment. It works by continuously polling DBS for new blocks in the
dataset, and when found these are splitted as usual and queued in the global queue.

It changes the ReqMgr running state to running-open and running-closed, where
running-open means that the request is still looking for new blocks. running-closed
is equivalent to the old running state.

The criteria to move a request from running-open to running-closed depends on a predefined
timeout, which comes with the request schema. If not given the request will be closed
as soon as it starts running. The request can also be closed at any time
through the web interface or the REST API. Once a request is closed, it will continue
process the already acquired inputs and will stop polling for blocks.

The timeout counts from the last time a block was found (either closed or open).
This feature can only be used with workflows which take a dataset as input,
namely ReDigi, ReReco, MontecarloFromGEN.

Finally, this also fixes the reported issue when a single open block in a dataset
can prevent the request from being acquired. The new behavior is to ignore open blocks
in the initial splitting, and the request will be acquired as long as there is at least
one closed blocks. The continuous polling will only acquire closed blocks.

Also zero event blocks with no site will be ignored, all these ignored blocks will be accounted
on the RejectedInputs field of the global workqueue element.

Fixes #4400

@sfoulkes, @zdenekmaxa please review. WorkQueue and ReqMgr must be updated with this for the feature to work.