dmwm/WMCore — pull request #4032 by dballesteros7 (2012-08-22 19:47:29+00:00)
Title: Implement general DQM Harvesting in WMAgent

This is the version without changes to the specs.

Changes:

- Disable stageOut in DQM step. We don't want to store the file, it is only for the DQM GUI.
- Integrate the Harvest splitting algorithm in the splitting page in ReqMgr
- If there is a proxy available under X509_USER_PROXY, the Harvest
splitter will use it and ship it in the job pickle, if CondorPlugin
finds such proxy then it will set it in the jdl
- The DQM executor now supports 3 possible input proxies:
- A proxy in the sandbox
- A proxy in the environment (X509_USER_PROXY)
- A proxy whose path is defined in self.upload.proxy
- Given that Harvesting works with an scenario then StdBase must support getting an scenario
even when a ConfigCache is provided, to ensure this then self.procScenario was moved up to
StdBase and it is obtained from ProcScenario. Any support of Scenario as the key in the input
arguments was dropped. Also the ReqMgr doesn't delete a scenario when it finds a config cache.
- All merge tasks created using addMergeTask that have DQM output will
have a DQM Harvest task attached by default (can be changed passing
doHarvesting = False). This task will use the upload proxy
in self.dqmUploadProxy, the url in self.dqmUploadUrl and the dqm
sequences in self.dqmSequences. dqmSequences is now a StdBase attribute.
- Merge for DQM changed a bit, it accepts the same merge size and event parameters
as the other data tiers but still requires that the merges are not performed
across run.

Include DQM Harvesting options in the ReqMgr

Modify the create request schema to support the DQM harvesting
for both ReReco and ReDigi workflows

Deactivate run splitting restrictions

Set the splitting algorithms to process and merge
different runs together by default, even for DQM.