Merge the Tier1PromptReco and PromptReco specs

They do the same thing, no need to mantain so much duplicated code.
I tested this against the T0 unit tests besides the usual WMCore tests.

Also added a few optional parameters to make it easier to setup
a testing environment to run the tests which involve skims

Fixes #3994
New splitting algorithm

The purpose of this algorithm is to dynamically split jobs by lumis
trying to make the jobs have a target number of events which is
ultimately the unit that determines job running time, disk space used
and memory constraints.
However this differentiates from the EventBased algorithm since
it doesn't split indiviual lumis, the basic unit for the splitting is the
lumi and every job is guaranteed to have at least one full lumi, this implies
that the number of events in a job will never be the exact target
but on average all jobs have about that number.
Also this includes the option to define a hard limit on the events
a job can process, if a file has a single lumi and more events
that such limit that job is created and marked as bad. This means
that this job will be moved from created to createdfailed by the
JobCreator.

The ErrorHandler is now configured to immediatly exhaust createfailed
jobs. The reason is that the createfailed state didn't make much sense,
the transitions defined it to go from createfailed to cooloff and then to
new. But a job that is in createfailed, had to actually be created otherwise
it wouldn't exist in WMBS. I changed the transitions to allow from created to createfailed and changed the transitions from createdcooloff/paused to new
for created instead which seems to be the logical step, since jobs in "new" state are not picked up by any piece of code.
However for now the createcooloff state is unreachable as the ErrorHandler will always take createfailed and send them to exhausted.

The algorithm takes the following inputs:

- events_per_job : This is the main parameter and defines the target events we want in each job. The algorithm will split lumis according to average events per lumi in each file
- max_events_per_lumi: Hard limit that defines jobs to fail when there are too many events in a lumi
- halt_on_file_boundaries, splitOnRun, etc..: Usual LumiBased algorithm parameters
Support for the new splitting algorithm in reqmgr

Reqmgr modifications to include the new algorithm
Start finishing subscriptions from the top

Before the latest changes the subscriptions were finished from
the bottom because they were deleted on the spot. However
now that the deletions only take place at the very end when everything
is finished we can actually remove the requirement that the child
subscriptions are finished to mark the parent as finished. This will
allow us to have a view of the progress of a workflow in terms of its
finished subscriptions, e.g. a workflow finished the DataProcessing subscription
and now it's just waiting on the merge. Also the more we can skim from this
dreaded query the better.
Modify LFN counters in extension workflows

Add the initial_lfn_counter parameter to the splitting of the
production and merge tasks in MonteCarlo and LHEStepZero workflows
if the FirstEvent/Lumi is different from 1. The initial_lfn_counter
is calculated as what the last previous job count should have been
based on the current events_per_job and FirstEvent.

Fixes #4151
Store phedex subscription information in workloads

Modify the WMTask and WMWorkload classes to store subscription
information for each outputModule/dataset in each task.

This can be specified during assign, the reqmgr web interface now shows
another set of site lists for subscription configuration. The subscription
configuration can be totally unrelated to the site whitelists.

The options that can be configured for a subscription are:

- Custodial sites
- Non custodial sites
- Sites where the request can be auto approved
- Priority

The option to customize subscription information per task is not exposed in the ReqMgr
for now is only intended for the Tier-0 use case.

This a ReqMgr-only patch
Improve the PhEDEx subscriber

The PhEDEx subscriber component now uses the spec information
to make the subscriptions regardless of where the files ended up.

The specs are copied to a directory in the JobAccountant and its
path is stored in the DBSBuffer workflow table under the column "spec".

The PhEDEx subscriber can operate in 2 modes:

- Unsafe: Subscribes custodial as Move as soon as files appear in DBSBuffer
- Safe: Subscribes custodial as Replica first, and when the workflow is complete as Move

Also implemented a PhEDEx subscriber unit test using the PhEDEx emulator and fixed
the broken JobAccountant_t test.

This is a client-only patch, if an agent is updated but the reqmgr doesn't have
the previous commit then the PhEDEx subscriber won't do anything. But it is stable,
i.e. it won't crash.
Merge pull request #4174 from dballesteros7/increase-lfn-counter

Modify LFN counters in extension workflows
Merge pull request #4171 from dballesteros7/phedex-fancy-subs

Phedex fancy subs
Merge pull request #4167 from dballesteros7/taskarchiver-reverse-order

Start finishing subscriptions from the top
Merge pull request #4127 from dballesteros7/smart-lumi-splitting

Smart lumi splitting
Merge pull request #4118 from dballesteros7/merge-promptreco-specs

Merge the Tier1PromptReco and PromptReco specs
0.9.22

Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>
Retry request submission in integration tests - get round intermittent problems