dmwm/WMCore — pull request #4551 by dballesteros7 (2013-04-16 19:55:09+00:00)
Title: Priority handling improvement

The purpose of this patch is to improve the priority
handling in the ReqMgr/WorkQueue/WMAgent looking forward
to the new global priority feature in HTCondor. It is now
possible for priorities to cross schedd boundaries
and be published to the negotiatior, this is
available in the 7.9.xx series.

The changes are the following:

- Global WorkQueue
Implement a polling function that checks the currently
available elements (i.e. not acquired by a local queue)
and keeps their priority in synchrony with the
value in the ReqMgr. This allows priority changes
in requests beyond acquired state to have some effect.

- WMBS schema
The priority is now added as a column in wmbs_workflow table,
this priority is filled by the WMBSHelper the first time a workflow
is injected into WMBS by the WorkQueue. The value used is the one
available in the spec.

- Local WorkQueue changes
The scope of these changes is to prevent priority inversion due
to the different queues available in the WorkQueue/WMAgent/Batch system
chain. The WorkQueue now considers priority of running/pending jobs
and queued elements before acquiring and injecting work. This means
that if all the pending jobs are of lower priority than an incoming
element then this will not be prevented from being acquired and injected.
E.g. The WMAgent has 100 jobs in condor, in pending status for site A, from a workflow
with priority 500. Also in the local queue there is 100 jobs for site A from a workflow
with priority 500. The element in the local queue is not injected because there are already
100 jobs of equal priority in the batch system. Now, someone creates and assigns a workflow
to site A with priority 1000. Then that element will be acquired into the LQ
and injected in WMBS because the 100 job slots are occupied only by lower priority jobs.

- JobSubmitter
The JobSubmitter now order the workflows with jobs to submit by priority and timestamp,
this means that when slots are available the high priority workflows will be submitted
first (after considering the task priorities). In case of equal priority, then the
older workflow is submitted first.

- BossAir/CondorPlugin
The priority of the workflow is now propagated to condor through BossAir, the
value passed to condor is a linear combination of the sub-task type priority
and the request priority. This ensures that Merge/LogCollect/Cleanup/Harvest/Skim
jobs are submitted faster regardless of priority to avoid missing data
due to temporary files.

This is part of #4509. Authors:

Diego Ballesteros (@dballesteros7)
Brian Bockelman (@bbockelm)