Reduce default couchdb queue size for ChangeState

1000 seems to be too much in the busiest machine (e.g. vocms96)
0.9.17

Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
Merge branch 'master' of github.com:dmwm/WMCore
0.9.18

Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
Merge branch 'master' of github.com:dmwm/WMCore
0.9.19

Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
Merge the Tier1PromptReco and PromptReco specs

They do the same thing, no need to mantain so much duplicated code.
I tested this against the T0 unit tests besides the usual WMCore tests.

Also added a few optional parameters to make it easier to setup
a testing environment to run the tests which involve skims

Fixes #3994
Fix TaskArchiver not cleaning outputs

In the last task archiver fixes I missed loading a workflow
when loading the subscription before deleting it which
made the workflow have empty output maps.

Modified the unit test to catch this
Merge pull request #4079 from dballesteros7/reduce-couch-queue

Reduce default couchdb queue size for ChangeState
Merge pull request #4125 from dballesteros7/fix-taskarchiver-bug

Fix TaskArchiver not cleaning outputs
Make Reqmgr url configurable in integration test
Fix integration test reqmgr url
Remove hardcoded wq url in integration test
Fix string formatting error
New splitting algorithm

The purpose of this algorithm is to dynamically split jobs by lumis
trying to make the jobs have a target number of events which is
ultimately the unit that determines job running time, disk space used
and memory constraints.
However this differentiates from the EventBased algorithm since
it doesn't split indiviual lumis, the basic unit for the splitting is the
lumi and every job is guaranteed to have at least one full lumi, this implies
that the number of events in a job will never be the exact target
but on average all jobs have about that number.
Also this includes the option to define a hard limit on the events
a job can process, if a file has a single lumi and more events
that such limit that job is created and marked as bad. This means
that this job will be moved from created to createdfailed by the
JobCreator.

The ErrorHandler is now configured to immediatly exhaust createfailed
jobs. The reason is that the createfailed state didn't make much sense,
the transitions defined it to go from createfailed to cooloff and then to
new. But a job that is in createfailed, had to actually be created otherwise
it wouldn't exist in WMBS. I changed the transitions to allow from created to createfailed and changed the transitions from createdcooloff/paused to new
for created instead which seems to be the logical step, since jobs in "new" state are not picked up by any piece of code.
However for now the createcooloff state is unreachable as the ErrorHandler will always take createfailed and send them to exhausted.

The algorithm takes the following inputs:

- events_per_job : This is the main parameter and defines the target events we want in each job. The algorithm will split lumis according to average events per lumi in each file
- max_events_per_lumi: Hard limit that defines jobs to fail when there are too many events in a lumi
- halt_on_file_boundaries, splitOnRun, etc..: Usual LumiBased algorithm parameters
Support for the new splitting algorithm in reqmgr

Reqmgr modifications to include the new algorithm
Bugfix for validation of well-formed dqmHarvest

Validation depends on having a self.procScenario in stdBase
this holds in all cases but ACDC workflows, this bug
prevents from creating ACDC workflows of requests
with DQM harvesting included through the webinterface (it is still
possible through script by passing a dummy 'scenario' parameter).

This patch changes the validation to check the steps in the harvesting task
and make sure the scenario was defined, this covers all workflows including
ACDC.
combine taskarchiver and cleanup manager to handle the case when reqmgr is not used
(need to be tested.)
Adding page to visualize requests which are done and summarized
Merge pull request #4130 from samircury/workload-summary-couchapp

Adding page to visualize requests which are done and summarized