dmwm/WMCore — pull request #4554 by dballesteros7 (2013-04-17 22:09:40+00:00)
Title: Priority handling improvement

The purpose of this patch is to improve the priority
handling in the ReqMgr/WorkQueue/WMAgent looking forward
to the new global priority feature in HTCondor. It is now
possible for priorities to cross schedd boundaries
and be published to the negotiatior, this is
available in the 7.9.xx series.

The changes are the following:

- ReqMgr
Push priority changes to the WorkQueue by updating
the available elements in the global queue and the spec.
This effectively updates the priority value for any
future split in the local queues and in any new global
element created.

- WMBS schema
The priority is now added as a column in wmbs_workflow table,
this priority is filled by the WMBSHelper the first time a workflow
is injected into WMBS by the WorkQueue. The value used is the one
available in the spec.

- Local WorkQueue changes
The scope of these changes is to prevent priority inversion due
to the different queues available in the WorkQueue/WMAgent/Batch system
chain. The WorkQueue now considers priority of running/pending jobs
and queued elements before acquiring and injecting work. This means
that if all the pending jobs are of lower priority than an incoming
element then this will not be prevented from being acquired and injected.
E.g. The WMAgent has 100 jobs in condor, in pending status for site A, from a workflow
with priority 500. Also in the local queue there is 100 jobs for site A from a workflow
with priority 500. The element in the local queue is not injected because there are already
100 jobs of equal priority in the batch system. Now, someone creates and assigns a workflow
to site A with priority 1000. Then that element will be acquired into the LQ
and injected in WMBS because the 100 job slots are occupied only by lower priority jobs.

- JobSubmitter
The JobSubmitter now order the workflows with jobs to submit by priority and timestamp,
this means that when slots are available the high priority workflows will be submitted
first (after considering the task priorities). In case of equal priority, then the
older workflow is submitted first.

- BossAir/CondorPlugin
The priority of the workflow is now propagated to condor through BossAir, the
value passed to condor is a linear combination of the sub-task type priority
and the request priority. This ensures that Merge/LogCollect/Cleanup/Harvest/Skim
jobs are submitted faster regardless of priority to avoid missing data
due to temporary files.
Also the sub task path and the sub task type (e.g. Processing) is passed
to the condor JDL as this allows more efficients updates later.

- Resource control
Make sub-task type priorities equal for all sites,
the granularity creates more inconveniences down the road
and currently there is no use case for them.
Simplifies the priority system in submission.

- JobUpdater (New component)

The JobUpdater is a new component which makes sure that
the information in the jobs already submitted to the batch
system are up to date. At the moment this only
concern the value of priority through the CondorPlugin.

Additionally in the case of priority, it also ensures that
the value is in synch with WMBS and the available elements
in the Local Queue.

Fixes #4509.

Authors:

Diego Ballesteros (@dballesteros7)
Brian Bockelman (@bbockelm)