Make robust merge jobs

Add the skipBadFiles flag to the input source in merge jobs
in order to avoid failing the merge for a single missing file.
When a successful merge job comes back with skipped files
these are added to the ACDC collection for the merge task
for posterior recovery.

Additionally, now CMSSW steps will be declared failed
if no output files are produced. This way the scenario
where all files were skipped is not counted as a success and also
allows to catch glitches when the cmsRun process fails to process the input
but stil returns with exit code 0.

This is only enabled for ParentlessMergeBySize.

Fixes #4476
Fix cases where performance parameters are None

TaskChain requests are bugged in current cmsweb, not
propagating valid values for the performance parameters.

Make sure it doesn't crash the JobCreator.
Priority handling improvement

The purpose of this patch is to improve the priority
handling in the ReqMgr/WorkQueue/WMAgent looking forward
to the new global priority feature in HTCondor. It is now
possible for priorities to cross schedd boundaries
and be published to the negotiatior, this is
available in the 7.9.xx series.

The changes are the following:

- ReqMgr
  Push priority changes to the WorkQueue by updating
  the available elements in the global queue and the spec.
  This effectively updates the priority value for any
  future split in the local queues and in any new global
  element created.

- WMBS schema
  The priority is now added as a column in wmbs_workflow table,
  this priority is filled by the WMBSHelper the first time a workflow
  is injected into WMBS by the WorkQueue. The value used is the one
  available in the spec.

- Local WorkQueue changes
  The scope of these changes is to prevent priority inversion due
  to the different queues available in the WorkQueue/WMAgent/Batch system
  chain. The WorkQueue now considers priority of running/pending jobs
  and queued elements before acquiring and injecting work. This means
  that if all the pending jobs are of lower priority than an incoming
  element then this will not be prevented from being acquired and injected.
  E.g. The WMAgent has 100 jobs in condor, in pending status for site A, from a workflow
  with priority 500. Also in the local queue there is 100 jobs for site A from a workflow
  with priority 500. The element in the local queue is not injected because there are already
  100 jobs of equal priority in the batch system. Now, someone creates and assigns a workflow
  to site A with priority 1000. Then that element will be acquired into the LQ
  and injected in WMBS because the 100 job slots are occupied only by lower priority jobs.

- JobSubmitter
  The JobSubmitter now order the workflows with jobs to submit by priority and timestamp,
  this means that when slots are available the high priority workflows will be submitted
  first (after considering the task priorities). In case of equal priority, then the
  older workflow is submitted first.

- BossAir/CondorPlugin
  The priority of the workflow is now propagated to condor through BossAir, the
  value passed to condor is a linear combination of the sub-task type priority
  and the request priority. This ensures that Merge/LogCollect/Cleanup/Harvest/Skim
  jobs are submitted faster regardless of priority to avoid missing data
  due to temporary files.
  Also the sub task path and the sub task type (e.g. Processing) is passed
  to the condor JDL as this allows more efficients updates later.

- Resource control
  Make sub-task type priorities equal for all sites,
  the granularity creates more inconveniences down the road
  and currently there is no use case for them.
  Simplifies the priority system in submission.

- JobUpdater (New component)

  The JobUpdater is a new component which makes sure that
  the information in the jobs already submitted to the batch
  system are up to date. At the moment this only
  concern the value of priority through the CondorPlugin.

  Additionally in the case of priority, it also ensures that
  the value is in synch with WMBS and the available elements
  in the Local Queue.

Fixes #4509.

Authors:

Diego Ballesteros (@dballesteros7)
Brian Bockelman (@bbockelm)
Disregard thresholds for jobs with multiple possible running sites. For monitoring purposes, report one random site in the possible site list as the 'pending' site.
Record location changes in WMBS and Couch

Whenever a job starts running, check the CMS site
where it started and store in WMBS and the transition
record in the JobDump in CouchDB. This ensures that
WMStats has the proper monitoring information and
runtime errors are attributed to the right site.

Fixes #3971
Fixes #3972
Remove site lists for second-level tasks

Fixes #4491
Align Proxy to the slc6 ui
Remove unused etc/ReqMgrConfig.py

This configuration file is IMO not used anywhere. Only considered in the
scenario in which ReqMgr gets deployed along with WMAgent deploy script,
I don't think we run this anywhere. @sfoulkes, please review.
ReqMgr - ProcessingString, ProcessingVersion

ProcessingString, ProcessingVersion were missing in the get request response
details. This information is stored only on spec (nor on Oracle or
CouchDB). It may be stored in CouchDB if the request defined this at the
injection time. Added now in the response.
Fixes #4561
Merge pull request #4563 from zdenekmaxa/reqmgr-processing-version-string-in-request-details-2

ReqMgr - ProcessingString, ProcessingVersion
0.9.62

Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
Extend LHEStepZero support

Include a new option that indicates if we are processing
lhe input files instead of forking a generator in cmsRun,
this way the input source options are set correctly
to read n events per job.

Fixes #2090
Merge pull request #4560 from zdenekmaxa/reqmgr-remove-unused-config-file

Remove unused etc/ReqMgrConfig.py
Removed GroupUser couchapp from ReqMgr unittests

ReqMgr unittests pushing GroupUser couchapp was not necessary, removed. The
couchapp is not used in ReqMgr at all.
Merge pull request #4565 from zdenekmaxa/reqmgr-remove-groupuser-couchapp-references

Removed GroupUser couchapp from ReqMgr unittests
0.9.63

Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
Fix ASO installation scripts.
Support a many-to-many CMS site to SE relation

Make sure that data is read properly from siteDB when building
lists of sites from a SE, taking into account that a se can be
shared by many CMS sites.

Update the test emulator such that the unit test works when
disabling the emulator in it.
refactor wmstats

commit 69b2674fc96f615d16d564bd51755b34b0324f60
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Thu Mar 21 10:57:11 2013 -0500

    remove resubmission

commit 6118abade8a756e9d9ed76e9bde906d1c4e7d074
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Mon Mar 18 11:49:43 2013 -0500

    fix run_closed ETA

commit b22b791611bfb1226a3a6b1df559bd1bd7ff815a
Author: Seangchan Ryu <sryu@fnal.gov>

commit 5ff010448618dc9014efbe294b73e42dca151433
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Wed Mar 13 10:55:49 2013 -0500

    acdc helper work in progress

commit ec226ffcbb97bbf04aa7d2fb6741b8a167783e06
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Wed Mar 6 14:48:09 2013 -0600

    acdc submission helper work in progress
Cascade close-out

Implement the option to close-out/announce requests and cascade the
status change to all resubmission children for the request.

For this there is a new view which allows the ReqMgr service
to construct the resubmission tree starting from a "original" request.

Fixes #4477