handle when couch has problem. keep accountant going and retry logcollect accounting
Add TransientOutputModules to TaskChain

Same functionality as in ReReco, some modules can be kept unmerged if
necessary in order to reduce the output but allow subsequent processing.

The parameter per task is TransientOutputModules, also KeepOutput is still
valid and can be used to wipe all the output in the task to unmerged. If
KeepOutput is False then TransientOutputModules doesn't have any effect.

Fixes #4474
fix missing input files
Merge pull request #4499 from zdenekmaxa/reqmgr-fixing-database-consistency-issue-4388-1

First round of ReqMgr CouchDB values fixes.
Merge pull request #4500 from zdenekmaxa/reqmgr-fixing-database-consistency-issue-4388-2

Second round of ReqMgr CouchDB values fixes.
ReqMgr GetRequest DAO not propagating all values

Fixes #3437
Merge pull request #4501 from zdenekmaxa/reqmgr-fix-dao-getrequest-incomplete-issue-3437

ReqMgr GetRequest DAO not propagating all values
Fix xrootd option for ACDC

Allow the trust site whitelist as location option to work for ACDC workflows
the ACDC files will have locations according to the site whitelist and blacklist.

Also improve an ACDC view making it compatible to newer versions of couchDB where
documents can't be modified in a view (it was sorting in place a element of the doc
inside the view).

Fixes #4496
fix firefox in Mac and Windows
Merge pull request #4503 from ticoann/firefox_hack

fix firefox in Mac and Windows
ReqMgr CouchApp validation function

Fixes #4464
Merge pull request #4505 from zdenekmaxa/reqmgr2-couchapp-validation-function-issue-4464-2

ReqMgr CouchApp validation function
Track pileup locations

The WorkQueue now tracks the location of pileup datasets
in a workload. The handling of these datasets it's similar
to the parent data, it is required that all pileup datasets
are at a site before acquiring the work for that site. However,
since the pileup is set at a dataset-level instead of blocks.
It is required that just a block is at the site for the dataset
to be considered "at the site". If the pileup is not at any
of the valid sites (after white/black lists) then the work
will wait in Available until the datasets/blocks are transferred.

Fixes #3733
Merge pull request #4507 from dballesteros7/tackPileUp

Track pileup locations
Merge pull request #4502 from dballesteros7/fixXrootdForACDC

Fix xrootd option for ACDC
Block close settings per request

Move the settings that determine when to close a produced block in
DBS to the request level instead of an agent-wide parameter.

These settings are:

- BlockCloseMaxWaitTime: Maximum time that a block will be left open since its creation.
- BlockCloseMaxFiles: Maximum number of files in a block, when reached the block is closed.
- BlockCloseMaxEvents: Maximum number of events in a block, when reached the block is closed.
- BlockCloseMaxSize: Maximum size of a block, when reached the block is closed.

Breakdown of the changes:

- At the RequestManager and workload definition level:
  - The WMWorkload contains the 4 block closing settings in the properties section (top level).
  - These are set to sensible defaults that are also modified per request type. E.g. LHEStepZero workloads have 5 files per block only.
  - The values can be modified through the assign page, these parameters are optional. The identifiers of the parameters were given above.

- At the local workqueue level:

  - The WMBSHelper now installs all tasks that will potentially produce merge output in the dbsbuffer workflow table, this includes the block closing settings.

- DBSBuffer3 schema changes:

  - The dbsbuffer_workflow table now contains 4 columns, one for each setting.
  - The InsertWorkflow DAO is updated to require the settings with any workflow injection
  - A new UpdateSpec DAO is provided to updated the spec path in the dbsbuffer_workflow table

- JobAccountant changes:

  - Since the workflows are injected at the WorkQueue level then workflows are not injected by default when files arrive,
    however if no workflow is found, the JobAccountant can still take care of this (so far the only use for this is unit testing).

- DBSUpload and DBSBuffer changes (2 & 3)

  - Every file record used in the DBSUpload pollers now contains the block closing settings
  - New blocks take the block closing settings from the first file they add and then use it to check when the block should be closed
  - Any component setting has been removed

- Unit tests
  - New emulator for the DBSInterface module for DBS2, this can be used to test the full dbsUploader without having a valid DBS instance
  - New mock API for the DBS3 Api, this can be used to the test the full dbsUploader (DBS3) without having a valid DBS instance
  - Extend unit tests for the dbsUploader to test new settings per workflow in DBS2 and DBS3
Add additional dependency for workqueue

Needed for tracking the pileup datasets
Merge pull request #4511 from dballesteros7/fix-workqueue-dependency

Add additional dependency for workqueue
Merge pull request #4481 from ticoann/fix_logcollect_archive

handle when couch has problem. keep accountant going and retry logcollect accounting
Merge pull request #4493 from ticoann/fix_input_files

fix missing input files