fix cmsrun config for endofrun dqm harvesting

EndOfRun DQM harvest jobs need a special cmsRun configuration.
ReqMgr cloned requests priority fix

When cloning a request which had changed priority, the newly cloned request
priority is on default values. Fixes goes via MakeRequest.createRequest() to
database DAOs Request/New.py
Also reqmgr.py checks the case.

Modified default value SoftTimeout on Ops request.
0.9.39

Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
ReqMgr CustodialSite bug fix.

CustodialSite is an argument not used for long. (Non)CustodialSites have
superseded this, however no one bothered to remove all dead functionality and
parameter definition. On cloned requests, the default value (None) of
CustodialSite request parameter appared and PhEDExInjector was failing on
that. This patch removes CustodialSite from ReqMgr only.
ReqMgr testing enhanced by TaskChain requests
Get the correct times out of a report if any

The function to retrieve the start and end time out of a FJR had a
bug where a None time would make it return always a None stopTime.

Fix it and also don't allow the ErrorHandler to get stuck if
the jobs don't have running time information.
Fix unit tests breakage due to Specs_t

Make sure that all unit tests in Specs_t use the temp directory
to store sandboxes and emulators for DBS

Fixes #4363
check components are alive using pid and also thread status
add script to unregister agent when agent is retired or inactive
Fixes from the testing of DBS3 upload.
0.9.40

Signed-off-by: Zdenek Maxa <zdenek.maxa@cern.ch>
Fix a typo in the CleanCouchPoller code

It can cause unexpected errors, when an error occurs
ReqMgr localhost testing, running
add support to stageout from HLT Cloud at CERN

The VMs on the HLT Cloud at CERN don't have access to the eos environment.
Change the stageout plugin to only use generic xrootd commands.

Patch was developed by Stephen Gowdy, I did test it in a replay though.
prevent task archiver crashing when central couch server is down
Fix how we delete documents in couchdb

A weird hitch in couch's delete semantics is that if you don't use the
DELETE verb (i.e. you commit with _bulk_docs and set _deleted=true), it
stores the whole data in the delete tombstone. See:

https://issues.apache.org/jira/browse/COUCHDB-1141

Since (AFAIK), we don't ever use _deleted_revs=true to look at things,
it means that we have tons of old data sitting around in couch that's
basically invisible. This patch strips the document before comitting it
so the only thing lefts are _id, _rev and _deleted
Enforce integer processing versions

This extends the support in WMWorkload and WMTask to include a
processing string along with the processing version and acquisition era.
The processing version is now enforced to be an integer and the lfns
are built as following:

<acquisitionEra>-<processingString>-v<processingVersion>

The job reports contain all these three elements for each output file
however the DBSBuffer will only use the processing version and acquisition
era.

The ReqMgr changes to allow the specification of these parameters in the
assign page, also updated the tests and templates. Include a test template
for TaskChain with multiple tasks, multiple acquisition eras and versions.

Fixes #4007

- added assert into reqmgr.py to check for correct request state
- modified template values of AcquisitionEra parameter
Fixing delete stuff
Fixes for problems uncovered during the DBS2/DBS3 consistency check:
  - Block size and file count is now correct.
  - Physics group is not correct.

This also adds a legitimate unit test which validates all data injected
and tests the special T0 DBS2/DBS3 upload only mode.
Bugfix Smart error handler

Bugfix for extreme cases of null reports and also
a few cases where jobs could get stuck in jobfailed.