When killing a job for timeout, check if a report exist before writing the fake-one
Implement checksums in LCGImpl backend module

Add support for verifying the adler32 checksum when available
through the lcg-ls -l command. Also add general support
for passing the locally calculated checksums of the files
to the StageOutImpl objects, this will enable the checksum
verification in the rfcp-CERN backend plugin as well.

For the LCGImpl plugin the checksum is obtained as follows:

echo "$LCG_OUTPUT" | sed -nr 's/^.*\s([a-f0-9]{8})\s*\([aA][dD][lL][eE][rR]32\)\s*$/\\1/p'

Where LCG_OUTPUT=`lcg-ls -l -b -D srmv2 <SURL> 2>/dev/null`

The expected format of LCG_OUTPUT is a multi-line string where one of the lines has the following format:

* Checksum: <8-hex character string> (ADLER32)

Or

* Checksum: <8-hex character string> (adler32)

If this line is not present then it is assumed that the checksum is not supported and the check is skipped,
but if a checksum is present then it will be compared and will fail with exit code 63011 if it doesn't match.
Error code is the same as for size mismatch.

Fixes #4185
Make component sleeping configurable

Factor out the sleeping part of the component of the component harness.
In some cases, simply sleeping isn't the best idea, this change makes it
so subclasses of BaseWorkerThread can choose how they sleep.
Check that files contains at least one good lumi before adding them to WWMBS. Fixes #4267
add output dataset
Allow nullauth for WorkloadSummary in local couch

Check for null auth and allow it,
also allow the update of existing documents
this is to provide support for multiple agents
and eventually cleanup. The TaskArchiver already takes care
of not doing funny stuff and uploading stuff twice
unless it's necessary.
Merge pull request #4279 from ticoann/add_output_datasets

add output dataset
Minor bugfix for the resource control script

The update threshold option had the pending and running
slots for a task swapped.
Remove restriction of autoapprove on custodial

Removing an unnecessary restriction I added that prevented
custodial subscriptions from being auto approve as well.
The Tier-0 needs it since we auto approve the custodial
subscriptions to CERN.
Adds a real LogCollect fwjr to the TaskArchiver test and changes to support it better. Changes in the summary document. fixes #4214
ConfigCacheUrl, extented reqmgr.py Ops script

* ConfigCacheUrl implemented for all worklflows types (allows to load
  ConfigCache document from an arbitrary CouchDB server, not necessarily
  the one coupled with ReqMgr (whose URL ReqMgr passes in CouchURL
  argument)).
  ConfigCacheUrl, if specified, overrides CouchURL. Is backwards compatible,
  if not specified, CouchURL is used as before. reqmgr.py Ops ReqMgr client
  script enhanced - more workflows template arguments defined
  (test/data/ReqMgr/requests)
  Fixes #4269

* Ops ReqMgr client script allows to create and assign requests in one go or
  separately. Approve is always coupled with create. If necessary can be
  separated too, however, if create succeeds, approve must too. Unfortunately,
  assignment lacks REST API so the client talks to a webpage.
  Ideally (for ReqMgr2), REST API shall support all possible status
  transitions, this will then be more easily exposed to reqmgr.py.
  Fixes #4247

 * Add /store/hidata to the lfn base allowed in the assign page of the
   ReqMgr. Fixes 4287

 * minor reqmgr.py bugfix, template default values agreed by the Ops, tested
 * by the Ops on a private VM instance
Merge pull request #4293 from zdenekmaxa/reqmgr-ops-scrips-02-1

ConfigCacheUrl, extented reqmgr.py Ops script
Allow ConfigCacheIDs for DQM harvesting

Add the option in the ReqMgr and handle the specific tweaks
for harvesting

Fixes #4262
Merge pull request #4291 from samircury/logarchives-listing-workflowsummary

Adds PFN of logArchives@CERN per task to the summary
Allow requests to run at sites without the data

Include a flag at assignment time that
allows ops to decide if a request should run
in all CMS sites filtered by whitelist/blacklist
instead of where the data is. This assumes Ops knows
what is doing (sites have xrootd to get the data).
Include REST API for missing lumis in the summary

Add a show function which works as a REST API
to get the missing lumis for a completed workflow from
the workload summary.

URL is ugly at the moment:

/_design/WorkloadSummary/_show/lumiMissingByWorkflow/<workflowName>?outputDataset=<string>

The outputDataset parameter is optional and is used to filter the output. The format of the output
is:

{<outputDataset:string> : { <run:string> : [ [<startLumi:int>, <endLumi:init>], ...] , ... }, ...}

Note that for summaries produced by agents without the TaskArchiver changes in this patch, the outputDataset won't
be returned as a key, instead it will be the task where the lumis were lost.

Fixes #4216
Implement summary of failure per site

Include two type of histograms with the number of failed jobs
per site. One given per site at a workflow level, and other
per step and exit code.

This is included in the summary using the new Histogram data structure
that allows generic histograms to be built with a simple interface.
This strutcture takes care of statistic calculation and generation
of a dictionary suitable for couchdb, also is meant to provide
support for multiple agents which will require merging summaries.

Fixes #4213
* Adds histograms for failing jobs per task/step under the label of errorsHistogram
* Improves TaskArchiver test adding the feature where half of the jobs get the FWJR of a succeeding job, just to have variety in the job set and simulate prod situations where there are failed jobs, but those are not all jobs.
* Adds jobTime as a histogram key, so now we will actually generate histograms in the summary of this test, before we weren't

Fixes #4215
Merge pull request #4164 from mmascher/poller-errors

When killing a job for timeout, check if a report exist before writing t...
Add /store/hidata as a possible LFN base for single request assignment
and add tests to the Lexicon tests verifying that works as well.