handle when couch has problem. keep accountant going and retry logcollect accounting
Add TransientOutputModules to TaskChain

Same functionality as in ReReco, some modules can be kept unmerged if
necessary in order to reduce the output but allow subsequent processing.

The parameter per task is TransientOutputModules, also KeepOutput is still
valid and can be used to wipe all the output in the task to unmerged. If
KeepOutput is False then TransientOutputModules doesn't have any effect.

Fixes #4474
fix missing input files
Fix xrootd option for ACDC

Allow the trust site whitelist as location option to work for ACDC workflows
the ACDC files will have locations according to the site whitelist and blacklist.

Also improve an ACDC view making it compatible to newer versions of couchDB where
documents can't be modified in a view (it was sorting in place a element of the doc
inside the view).

Fixes #4496
Track pileup locations

The WorkQueue now tracks the location of pileup datasets
in a workload. The handling of these datasets it's similar
to the parent data, it is required that all pileup datasets
are at a site before acquiring the work for that site. However,
since the pileup is set at a dataset-level instead of blocks.
It is required that just a block is at the site for the dataset
to be considered "at the site". If the pileup is not at any
of the valid sites (after white/black lists) then the work
will wait in Available until the datasets/blocks are transferred.

Fixes #3733
Merge pull request #4507 from dballesteros7/tackPileUp

Track pileup locations
Merge pull request #4502 from dballesteros7/fixXrootdForACDC

Fix xrootd option for ACDC
Block close settings per request

Move the settings that determine when to close a produced block in
DBS to the request level instead of an agent-wide parameter.

These settings are:

- BlockCloseMaxWaitTime: Maximum time that a block will be left open since its creation.
- BlockCloseMaxFiles: Maximum number of files in a block, when reached the block is closed.
- BlockCloseMaxEvents: Maximum number of events in a block, when reached the block is closed.
- BlockCloseMaxSize: Maximum size of a block, when reached the block is closed.

Breakdown of the changes:

- At the RequestManager and workload definition level:
  - The WMWorkload contains the 4 block closing settings in the properties section (top level).
  - These are set to sensible defaults that are also modified per request type. E.g. LHEStepZero workloads have 5 files per block only.
  - The values can be modified through the assign page, these parameters are optional. The identifiers of the parameters were given above.

- At the local workqueue level:

  - The WMBSHelper now installs all tasks that will potentially produce merge output in the dbsbuffer workflow table, this includes the block closing settings.

- DBSBuffer3 schema changes:

  - The dbsbuffer_workflow table now contains 4 columns, one for each setting.
  - The InsertWorkflow DAO is updated to require the settings with any workflow injection
  - A new UpdateSpec DAO is provided to updated the spec path in the dbsbuffer_workflow table

- JobAccountant changes:

  - Since the workflows are injected at the WorkQueue level then workflows are not injected by default when files arrive,
    however if no workflow is found, the JobAccountant can still take care of this (so far the only use for this is unit testing).

- DBSUpload and DBSBuffer changes (2 & 3)

  - Every file record used in the DBSUpload pollers now contains the block closing settings
  - New blocks take the block closing settings from the first file they add and then use it to check when the block should be closed
  - Any component setting has been removed

- Unit tests
  - New emulator for the DBSInterface module for DBS2, this can be used to test the full dbsUploader without having a valid DBS instance
  - New mock API for the DBS3 Api, this can be used to the test the full dbsUploader (DBS3) without having a valid DBS instance
  - Extend unit tests for the dbsUploader to test new settings per workflow in DBS2 and DBS3
Add additional dependency for workqueue

Needed for tracking the pileup datasets
Merge pull request #4511 from dballesteros7/fix-workqueue-dependency

Add additional dependency for workqueue
Merge pull request #4481 from ticoann/fix_logcollect_archive

handle when couch has problem. keep accountant going and retry logcollect accounting
Merge pull request #4493 from ticoann/fix_input_files

fix missing input files
Merge pull request #4471 from dballesteros7/blockParametersPerRequest

Block close settings per request
Include performance parameter in workload

First part of #4472, include the performance information that is now already
required but only stored in the schema. The performance information is stored by
task and can be different depending on the type of task.

Tasks like skims and second step TaskChain use their own estimate which is optional
and defaults to the top level value.

This is only the ReqMgr and Workload definition changes. After this we need the changes at the
agent level in the splitters but that is independent of cmsweb schedule.
Fix the initial location for pileup

It was set to the SEs instead of the CMS site names. Correct that.
Merge pull request #4512 from dballesteros7/fix-pileup-initial-location

Fix the initial location for pileup
Merge pull request #4508 from dballesteros7/RunningTimePrediction

Include performance parameter in workload
Merge pull request #4488 from dballesteros7/TaskChainDropOutputModules

Add TransientOutputModules to TaskChain
Squashed commit of the following:

commit e1f1265bbdb45b23e88900808803da789f17c27a
Merge: 44aedc1 146d10f
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Mon Apr 1 15:47:37 2013 -0500

    Merge branch 'acdc_support' into pull_april

commit 146d10fd49de01a170986dd4fbb299eeef0f5bbb
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Mon Apr 1 15:45:49 2013 -0500

    add acdc_support

commit f1f4a0c08013de1fec5c6d83e920239283cbfd3e
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Thu Mar 28 12:00:41 2013 -0500

    fix runnning-closed ETA

commit 463029de145708648806060666fcc7093116082d
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Wed Mar 27 13:25:30 2013 -0500

    add control fix

commit 69b2674fc96f615d16d564bd51755b34b0324f60
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Thu Mar 21 10:57:11 2013 -0500

    remove resubmission

commit 6118abade8a756e9d9ed76e9bde906d1c4e7d074
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Mon Mar 18 11:49:43 2013 -0500

    fix run_closed ETA

commit b22b791611bfb1226a3a6b1df559bd1bd7ff815a
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Wed Mar 13 16:31:53 2013 -0500

    acdc

commit 5ff010448618dc9014efbe294b73e42dca151433
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Wed Mar 13 10:55:49 2013 -0500

    acdc helper work in progress

commit ec226ffcbb97bbf04aa7d2fb6741b8a167783e06
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Wed Mar 6 14:48:09 2013 -0600

    acdc submission helper work in progress
Merge pull request #4513 from ticoann/pull_april1

wmstats april change