add unittest for couch view
log error message instead of stoping the process when request is not exist in wmstats
when status is updated
Implement general DQM Harvesting in WMAgent

Changes:

  - Disable stageOut in DQM step. We don't want to store the file, it is only for the DQM GUI.
  - Integrate the Harvest splitting algorithm in the splitting page in ReqMgr
  - If there is a proxy available under X509_USER_PROXY, the Harvest
    splitter will use it and ship it in the job pickle, if CondorPlugin
    finds such proxy then it will set it in the jdl
  - The DQM executor now supports 3 possible input proxies:
    - A proxy in the sandbox
    - A proxy in the environment (X509_USER_PROXY)
    - A proxy whose path is defined in self.upload.proxy
  - Given that Harvesting works with an scenario then StdBase must support getting an scenario
    even when a ConfigCache is provided, to ensure this then self.procScenario was moved up to
    StdBase and it is obtained from ProcScenario. Any support of Scenario as the key in the input
    arguments was dropped. Also the ReqMgr doesn't delete a scenario when it finds a config cache.
  - All merge tasks created using addMergeTask that have DQM output will
    have a DQM Harvest task attached by default (can be changed passing
    doHarvesting = False). This task will use the upload proxy
    in self.dqmUploadProxy, the url in self.dqmUploadUrl and the dqm
    sequences in self.dqmSequences. dqmSequences is now a StdBase attribute.
  - Merge for DQM changed a bit, it accepts the same merge size and event parameters
    as the other data tiers but still requires that the merges are not performed
    across run.

What is missing:
  - Support in harvesting for multiple runs per file

Include DQM Harvesting options in the ReqMgr

Modify the create request schema to support the DQM harvesting
for both ReReco and ReDigi workflows

Deactivate run splitting restrictions

Set the splitting algorithms to process and merge
different runs together by default, even for DQM.
simplify and cleanup DQMUpload code
Include RequestName in the Condor JDL

Changes:

- Modify the ListForSubmitter DAO to fetch the workflow name
- Pass the request name in the RunJob objects
- Include the line +WMAgent_RequestName in the condor JDL
Create an inactive doc in the old active db and not in the actual active one.
WorkQueue: Handle open blocks.

Fix #3980

Periodically check open blocks for new files and inject into wmbs. Once
block closed in dbs close fileset in wmbs.

Signed-off-by: stuartw <stuart.wakefield@gmail.com>
add support for Alca Harvesting

Add AlcaHarvest step which goes through the analysis files
the CMSSW job wrote out and checks for sqlite condition files.
If it finds any, it does create metadata text files and copies
each condition file and it's corresponding metadata text file
to a special directory (POSIX path) and then adds both files
to the FJR under a fake 'Sqlite' output module.

Also make a change in StdBase to support output from
AlcaSkimming with data tier 'ALCAPROMPT'. This is needed
to support AlcaHarvesting.
have TaskArchiver accept missing working directory

Currently, the TaskArchiver will print an error message and send
an alert if it cannot delete the working directory for a task due
to that working directory not existing.

If the input for a task never gets any events, the task itself is
never initialized by the JobCreator and therefor it's working
directory is never created in the first place. This is normal
for these cases and not an error.

Downgrade the error message to a debug message and remove the alert.

While modifying this code anyways, make the consistency check more
specific: If the working directory does not exist, handle it in
the described way. If it exists, but is not a directory, bail out
with an exception because at that point something is screwed up.
Merge pull request #4046 from hufnagel/taskarchiver-workdir

Have TaskArchiver accept missing working directory.
Merge pull request #4045 from hufnagel/alcaharvest

Add support for Alca Harvesting.
Merge pull request #4034 from hufnagel/dqm-upload

Simplify and cleanup runtime DQMUpload code.
Merge pull request #4021 from ticoann/missing_request

Log error messages instead of stopping the process when a WMStats document for a request doesn't exist in the database.
Merge pull request #4041 from stuartw/patch/WorkQueue/open-block

WorkQueue: Handle open blocks.
Merge pull request #4040 from dballesteros7/jdl-requestname

Include RequestName in the Condor JDL.
Merge pull request #4037 from HassenRiahi/FixInactiveDocInsertion

Create the inactive doc in the old active db.
Merge pull request #4033 from dballesteros7/dqm-harvest-2

Implement general DQM Harvesting in WMAgent.
fix build problems due to global cherrypy in setup_test

Cherrypy is not avalaible to the build system, therefore
the global import at the top of setup_test crashes the
build. Moved it to a local import where it is used.
Merge pull request #4054 from hufnagel/build-fail

Fix build problems due to global cherrypy in setup_test.