dmwm/WMCore — pull request #4471 by dballesteros7 (2013-03-13 20:40:50+00:00)
Title: Block close settings per request

Move the settings that determine when to close a produced block in
DBS to the request level instead of an agent-wide parameter.

These settings are:

- BlockCloseMaxWaitTime: Maximum time that a block will be left open since its creation.
- BlockCloseMaxFiles: Maximum number of files in a block, when reached the block is closed.
- BlockCloseMaxEvents: Maximum number of events in a block, when reached the block is closed.
- BlockCloseMaxSize: Maximum size of a block, when reached the block is closed.

Breakdown of the changes:

- At the RequestManager and workload definition level:
- The WMWorkload contains the 4 block closing settings in the properties section (top level).
- These are set to sensible defaults that are also modified per request type. E.g. LHEStepZero workloads have 5 files per block only.
- The values can be modified through the assign page, these parameters are optional. The identifiers of the parameters were given above.

- At the local workqueue level:

- The WMBSHelper now installs all tasks that will potentially produce merge output in the dbsbuffer workflow table, this includes the block closing settings.

- DBSBuffer3 schema changes:

- The dbsbuffer_workflow table now contains 4 columns, one for each setting.
- The InsertWorkflow DAO is updated to require the settings with any workflow injection
- A new UpdateSpec DAO is provided to updated the spec path in the dbsbuffer_workflow table

- JobAccountant changes:

- Since the workflows are injected at the WorkQueue level then workflows are not injected by default when files arrive,
however if no workflow is found, the JobAccountant can still take care of this (so far the only use for this is unit testing).

- DBSUpload and DBSBuffer changes (2 & 3)

- Every file record used in the DBSUpload pollers now contains the block closing settings
- New blocks take the block closing settings from the first file they add and then use it to check when the block should be closed
- Any component setting has been removed

- Unit tests
- New emulator for the DBSInterface module for DBS2, this can be used to test the full dbsUploader without having a valid DBS instance
- New mock API for the DBS3 Api, this can be used to the test the full dbsUploader (DBS3) without having a valid DBS instance
- Extend unit tests for the dbsUploader to test new settings per workflow in DBS2 and DBS3

@sfoulkes, please review.