dmwm/WMCore — pull request #39 by dballesteros7 (2012-07-22 07:16:18+00:00)
Title: Integrate DQM harvesting in the WMAgent workflows

Please review.

Changes:

- Disable stageOut in DQM step. We don't want to store the file, it is only for the DQM GUI.
- Integrate the Harvest splitting algorithm in the splitting page in ReqMgr
- If there is a proxy available under X509_USER_PROXY, the Harvest
splitter will use it and ship the path in the job pickle, if CondorPlugin
finds such proxy then it will set it in the jdl
- The DQM executor now supports 3 possible input proxies:
- A proxy in the sandbox
- A proxy in the environment (X509_USER_PROXY)
- A proxy which path is defined in self.upload.proxy
- Given that Harvesting works with an scenario then StdBase must support getting an scenario
even when a ConfigCache is provided, to ensure this then self.procScenario was moved up to
StdBase and it is obtained from ProcScenario. Any support of 'Scenario' as the key in the input
arguments was dropped. Also the ReqMgr doesn't delete a scenario when it finds a config cache.
- All merge tasks created using addMergeTask that have DQM output will
have a DQM Harvest task attached by default (can be changed passing
doHarvesting = False). This task will use the upload proxy
in self.dqmUploadProxy, the url in self.dqmUploadUrl and the dqm
sequences in self.dqmSequences. dqmSequences is now a StdBase attribute.
- Merge for DQM changed a bit, it accepts the same merge size and event parameters
as the other data tiers but still requires that the merges are not performed
across run.

What is missing:
- Support in harvesting for multiple runs per file
- Support for putting more blocks in the workqueue after a workflow has been created, this is needed for Tier1PromptReco/PromptSkimming to work under the rare cases of late blocks.