Handle files with multiple runs in Harvest

Modifiy the Harvest JobSplitter algorithm to handle multiple
runs in a file, it adds the run/lumi information to the job mask.

Note that this patch doesn't remove the special do-not-mix-runs
behavior of the agent, that will go in another patch which
will need https://github.com/dmwm/WMCore/pull/39 to
be applied.
Fix setup_test for zero-case runs
Switch from dumpPython to pickle.

Fix some simple mistakes in PSet tweaking. Unit test for multicore
Fixes #4009
More informative component logs
Fixing a flaky test
Merge pull request #4026 from PerilousApricot/testing/update-failing-tests

Fixing a flaky test
add unittest for couch view
log error message instead of stoping the process when request is not exist in wmstats
when status is updated
Whitelisting failing tests

Traceback (most recent call last):
  File "/jenkins/deploy/0.9.10/sw/slc5_amd64_gcc461/external/python/2.6.8/lib/python2.6/unittest.py", line 270, in run
    self.setUp()
  File "/jenkins/workspace/WMCore-UnitTests-try/jobSlice/2/label/wmcore-unit-test-slaves/code/test/python/WMCore_t/WorkQueue_t/WorkQueue_t.py", line 89, in setUp
    WorkQueueTestCase.setUp(self)
  File "/jenkins/workspace/WMCore-UnitTests-try/jobSlice/2/label/wmcore-unit-test-slaves/code/test/python/WMCore_t/WorkQueue_t/WorkQueueTestCase.py", line 45, in setUp
    self.testInit.setupCouch(self.globalQDB, *self.couchApps)
  File "/jenkins/install/lib/python2.6/site-packages/WMQuality/TestInitCouchApp.py", line 98, in setupCouch
    self.couch.create(dropExistingDb=self.dropExistingDb)
  File "/jenkins/install/lib/python2.6/site-packages/WMQuality/TestInitCouchApp.py", line 49, in create
    self.couchServer.createDatabase(self.dbName)
  File "/jenkins/install/lib/python2.6/site-packages/WMCore/Database/CMSCouch.py", line 777, in createDatabase
    self.put("/%s" % urllib.quote_plus(dbname))
  File "/jenkins/install/lib/python2.6/site-packages/WMCore/Services/Requests.py", line 125, in put
    encode, decode, contentType)
  File "/jenkins/install/lib/python2.6/site-packages/WMCore/Database/CMSCouch.py", line 108, in makeRequest
    getattr(e, "reason", None), data)
  File "/jenkins/install/lib/python2.6/site-packages/WMCore/Database/CMSCouch.py", line 131, in checkForCouchError
    raise CouchPreconditionFailedError(reason, data, result)
CouchPreconditionFailedError: CouchPreconditionFailedError - reason: Precondition Failed, data: {} result: None
Merge pull request #4027 from PerilousApricot/testing/update-failing-tests

Whitelisting failing testsv2
remove job unit test dependency on removed dao

The NewestStateChangeDAO was only used in the
Periodic and EndOfRun job splitting algorithms
and was removed when these algorithms was
removed. Seems like I forgot to remove the
corresponding unit test for this DAO.
Merge pull request #4024 from dballesteros7/verbose-components

Adding logging statements to components logs so that it is easier to tell when a component dies.
Merge pull request #4030 from hufnagel/job-unittest

remove job unit test dependency on removed dao
Merge pull request #4020 from PerilousApricot/bugfix/jenkins-empty-tests

Fix setup_test for zero-case runs
Fix unit tests for UnpackUserTarball_t

Fixes #3680
Implement general DQM Harvesting in WMAgent

Changes:

  - Disable stageOut in DQM step. We don't want to store the file, it is only for the DQM GUI.
  - Integrate the Harvest splitting algorithm in the splitting page in ReqMgr
  - If there is a proxy available under X509_USER_PROXY, the Harvest
    splitter will use it and ship it in the job pickle, if CondorPlugin
    finds such proxy then it will set it in the jdl
  - The DQM executor now supports 3 possible input proxies:
    - A proxy in the sandbox
    - A proxy in the environment (X509_USER_PROXY)
    - A proxy whose path is defined in self.upload.proxy
  - Given that Harvesting works with an scenario then StdBase must support getting an scenario
    even when a ConfigCache is provided, to ensure this then self.procScenario was moved up to
    StdBase and it is obtained from ProcScenario. Any support of Scenario as the key in the input
    arguments was dropped. Also the ReqMgr doesn't delete a scenario when it finds a config cache.
  - All merge tasks created using addMergeTask that have DQM output will
    have a DQM Harvest task attached by default (can be changed passing
    doHarvesting = False). This task will use the upload proxy
    in self.dqmUploadProxy, the url in self.dqmUploadUrl and the dqm
    sequences in self.dqmSequences. dqmSequences is now a StdBase attribute.
  - Merge for DQM changed a bit, it accepts the same merge size and event parameters
    as the other data tiers but still requires that the merges are not performed
    across run.

What is missing:
  - Support in harvesting for multiple runs per file

Include DQM Harvesting options in the ReqMgr

Modify the create request schema to support the DQM harvesting
for both ReReco and ReDigi workflows

Deactivate run splitting restrictions

Set the splitting algorithms to process and merge
different runs together by default, even for DQM.
Implement general DQM Harvesting in WMAgent

Changes:

  - Disable stageOut in DQM step. We don't want to store the file, it is only for the DQM GUI.
  - Integrate the Harvest splitting algorithm in the splitting page in ReqMgr
  - If there is a proxy available under X509_USER_PROXY, the Harvest
    splitter will use it and ship it in the job pickle, if CondorPlugin
    finds such proxy then it will set it in the jdl
  - The DQM executor now supports 3 possible input proxies:
    - A proxy in the sandbox
    - A proxy in the environment (X509_USER_PROXY)
    - A proxy whose path is defined in self.upload.proxy
  - Given that Harvesting works with an scenario then StdBase must support getting an scenario
    even when a ConfigCache is provided, to ensure this then self.procScenario was moved up to
    StdBase and it is obtained from ProcScenario. Any support of Scenario as the key in the input
    arguments was dropped. Also the ReqMgr doesn't delete a scenario when it finds a config cache.
  - All merge tasks created using addMergeTask that have DQM output will
    have a DQM Harvest task attached by default (can be changed passing
    doHarvesting = False). This task will use the upload proxy
    in self.dqmUploadProxy, the url in self.dqmUploadUrl and the dqm
    sequences in self.dqmSequences. dqmSequences is now a StdBase attribute.
  - Merge for DQM changed a bit, it accepts the same merge size and event parameters
    as the other data tiers but still requires that the merges are not performed
    across run.

What is missing:
  - Support in harvesting for multiple runs per file

Include DQM Harvesting options in the ReqMgr

Modify the create request schema to support the DQM harvesting
for both ReReco and ReDigi workflows

Deactivate run splitting restrictions

Set the splitting algorithms to process and merge
different runs together by default, even for DQM.
Merge pull request #4032 from dballesteros7/dqm-harvest-1

Implement general DQM Harvesting in WMAgent.
Fix MonitoringDAO BossAir test
Fix workqueue mc splitting to normally use 1 lumi section per job.

Previsouly defaulted to 1000 events per lumi section.

Signed-off-by: stuartw <stuart.wakefield@gmail.com>