billing: improve the insert queue logic

Revised version of http://rb.dcache.org/r/5841 (discarded)

Currently the way which billing data gets inserted via DataNucleus uses a twin mechanism:  on the one hand, the database API put() calls makePersistent on an open transaction, then asks for conditions to be checked (max inserts or timeout, whichever is reached first), and if so, commits the transaction.  At the same time, a daemon runs which periodically checks to see if those conditions have been met and to commit if so.

This is an unnecessarily complicated logic and can be simplified by making use of the following two features:

1.  java.util.concurrent.BlockingQueue
2.  the built-in batching provided by DataNucleus itself (makePersistentAll).

The database call to put now simply adds the data object to the queue.  A separate thread waits for the queue to be non-empty, then drains it and commits all the objects inside the current critical section.

For added efficiency, the patch implements four separate queues which are mapped to the four separate billing message types/tables; insert is thus specialized via method overloading so that DataNucleus need not have to filter the objects into distinct batched commit statements.

The additional abstraction layer is for flexibility, in case some other façade to the queueing structure is desired.

Several new properties (see below) expose tuning choices.

A separate patch for 2.2 will be posted.

Testing:

Basic functionality tested.

Junit test modified to eliminate meaningless test case; other tests pass.

Stress tests which directly insert to the JDO layer were run with various settings to diagnose stability and performance.

Full testing with billing message processing (using a test cell which produces random billing messages) done on live dcache installation.

Target: master
Request: 2.7
Request: 2.6
Patch: http://rb.dcache.org/r/5992/diff/#index_header
Require-notes: yes
Require-book: yes
Acked-by: Tigran

RELEASE NOTES:
BOOK:

An improved insert strategy for the billing database has been implemented.  The following changes to billing properties should be noted:

billingMaxInsertsBeforeCommit
billing.db.inserts.max-before-commit
billingMaxTimeBeforeCommitInSecs
billing.db.inserts.timeout-before-commit
billing.db.inserts.timeout-before-commit.unit

(one-of?org.dcache.services.billing.db.impl.DirectQueueDelegate)billing.db.inserts.queue-delegate.type=org.dcache.services.billing.db.impl.DirectQueueDelegate

billing.db.inserts.max-queue-size=100000

billing.db.inserts.max-batch-size=1000

(one-of?true|false)billing.db.inserts.drop-messages-at-limit=true

It should be noted that, in conjunction with http://rb.dcache.org/r/5983, which modifies the database triggers, insertion should now perform without message loss (and without running out of memory) under most conditions (3000Hz or less).