Make robust merge jobs

Add the skipBadFiles flag to the input source in merge jobs
in order to avoid failing the merge for a single missing file.
When a successful merge job comes back with skipped files
these are added to the ACDC collection for the merge task
for posterior recovery.

Additionally, now CMSSW steps will be declared failed
if no output files are produced. This way the scenario
where all files were skipped is not counted as a success and also
allows to catch glitches when the cmsRun process fails to process the input
but stil returns with exit code 0.

This is only enabled for ParentlessMergeBySize.

Fixes #4476
Fixed version of MyProxy module UI free, with unit test

Signed-off-by: cinquo <m.cinquilli@gmail.com>
Fix cases where performance parameters are None

TaskChain requests are bugged in current cmsweb, not
propagating valid values for the performance parameters.

Make sure it doesn't crash the JobCreator.
Add ASO databases.
Priority handling improvement

The purpose of this patch is to improve the priority
handling in the ReqMgr/WorkQueue/WMAgent looking forward
to the new global priority feature in HTCondor. It is now
possible for priorities to cross schedd boundaries
and be published to the negotiatior, this is
available in the 7.9.xx series.

The changes are the following:

- ReqMgr
  Push priority changes to the WorkQueue by updating
  the available elements in the global queue and the spec.
  This effectively updates the priority value for any
  future split in the local queues and in any new global
  element created.

- WMBS schema
  The priority is now added as a column in wmbs_workflow table,
  this priority is filled by the WMBSHelper the first time a workflow
  is injected into WMBS by the WorkQueue. The value used is the one
  available in the spec.

- Local WorkQueue changes
  The scope of these changes is to prevent priority inversion due
  to the different queues available in the WorkQueue/WMAgent/Batch system
  chain. The WorkQueue now considers priority of running/pending jobs
  and queued elements before acquiring and injecting work. This means
  that if all the pending jobs are of lower priority than an incoming
  element then this will not be prevented from being acquired and injected.
  E.g. The WMAgent has 100 jobs in condor, in pending status for site A, from a workflow
  with priority 500. Also in the local queue there is 100 jobs for site A from a workflow
  with priority 500. The element in the local queue is not injected because there are already
  100 jobs of equal priority in the batch system. Now, someone creates and assigns a workflow
  to site A with priority 1000. Then that element will be acquired into the LQ
  and injected in WMBS because the 100 job slots are occupied only by lower priority jobs.

- JobSubmitter
  The JobSubmitter now order the workflows with jobs to submit by priority and timestamp,
  this means that when slots are available the high priority workflows will be submitted
  first (after considering the task priorities). In case of equal priority, then the
  older workflow is submitted first.

- BossAir/CondorPlugin
  The priority of the workflow is now propagated to condor through BossAir, the
  value passed to condor is a linear combination of the sub-task type priority
  and the request priority. This ensures that Merge/LogCollect/Cleanup/Harvest/Skim
  jobs are submitted faster regardless of priority to avoid missing data
  due to temporary files.
  Also the sub task path and the sub task type (e.g. Processing) is passed
  to the condor JDL as this allows more efficients updates later.

- Resource control
  Make sub-task type priorities equal for all sites,
  the granularity creates more inconveniences down the road
  and currently there is no use case for them.
  Simplifies the priority system in submission.

- JobUpdater (New component)

  The JobUpdater is a new component which makes sure that
  the information in the jobs already submitted to the batch
  system are up to date. At the moment this only
  concern the value of priority through the CondorPlugin.

  Additionally in the case of priority, it also ensures that
  the value is in synch with WMBS and the available elements
  in the Local Queue.

Fixes #4509.

Authors:

Diego Ballesteros (@dballesteros7)
Brian Bockelman (@bbockelm)
ReqMgr2 project startup

-REST service based on WMCore.REST (new REST API)
-some example plus request list, query REST API
-configuration defined, called from deployment/reqmgr2/config.py
-startup of web GUI implemented in JavaScript, calling service's REST APIs
-couchapp ReqMgr (after initial clean up remained 1 view) shared between
ReqMgr1 and ReqMgr2 (validation, update already previously implemented),
necessary view will be addded
-CouchDB database (reqmgr_workload_cache) behind, shared with ReqMgr1

Incorporated comments from @ticonn collected on #4542
Incorporated comments from @geneguvo:
    -only 1 major config.py under deployment
    -regexp exported into a dedicated module

Added a check in WMCore.REST.Main, since it relies on private patched version
of cherrypy.
Merge pull request #4555 from zdenekmaxa/reqmgr2-startup-2

ReqMgr2 project startup
0.9.61

Signed-off-by: Zdenek Maxa <zdenek.maxa@cern.ch>
Merge pull request #4553 from HassenRiahi/add-aso-databases

Add ASO databases.
Merge pull request #4544 from cinquo/master

Add MyProxy module only depending on openssl, removing UI
Revert "Fixed version of MyProxy module UI free, with unit test"

This reverts commit 5d7ffd4be1bd5d379c11200b33edd37902e86b51.
Disregard thresholds for jobs with multiple possible running sites. For monitoring purposes, report one random site in the possible site list as the 'pending' site.
Record location changes in WMBS and Couch

Whenever a job starts running, check the CMS site
where it started and store in WMBS and the transition
record in the JobDump in CouchDB. This ensures that
WMStats has the proper monitoring information and
runtime errors are attributed to the right site.

Fixes #3971
Fixes #3972
Remove site lists for second-level tasks

Fixes #4491
Align Proxy to the slc6 ui
Remove unused etc/ReqMgrConfig.py

This configuration file is IMO not used anywhere. Only considered in the
scenario in which ReqMgr gets deployed along with WMAgent deploy script,
I don't think we run this anywhere. @sfoulkes, please review.
ReqMgr - ProcessingString, ProcessingVersion

ProcessingString, ProcessingVersion were missing in the get request response
details. This information is stored only on spec (nor on Oracle or
CouchDB). It may be stored in CouchDB if the request defined this at the
injection time. Added now in the response.
Fixes #4561
Merge pull request #4563 from zdenekmaxa/reqmgr-processing-version-string-in-request-details-2

ReqMgr - ProcessingString, ProcessingVersion
0.9.62

Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
Extend LHEStepZero support

Include a new option that indicates if we are processing
lhe input files instead of forking a generator in cmsRun,
this way the input source options are set correctly
to read n events per job.

Fixes #2090