xrootd: Make xrootdPlugins unscoped for door

The consequences of accidentally configuring xrootdPlugins in
dcache.conf can be quite big: Access to the door would be
unauthenticated, and possible name space mapping plugins would
not be applied.

To reduce the risk of this problem, and since we removed the
scoping operator in 2.7 anyway, this patch makes xrootdPlugins
unscoped for the door. Pools continue to use the scoped xrootdPlugins
property, but the patch introduces the pool.mover.xrootd.plugins
name as an alternative. This is the same name that was introduced
in 2.7.

Target: 2.6
Require-notes: yes
Require-book: yes
(2.6) billing: improve the insert queue logic

Revised version of http://rb.dcache.org/r/5841 (discarded)

Currently the way which billing data gets inserted via DataNucleus uses a twin mechanism:  on the one hand, the database API put() calls makePersistent on an open transaction, then asks for conditions to be checked (max inserts or timeout, whichever is reached first), and if so, commits the transaction.  At the same time, a daemon runs which periodically checks to see if those conditions have been met and to commit if so.

This is an unnecessarily complicated logic and can be simplified by making use of the following two features:

1.  java.util.concurrent.BlockingQueue
2.  the built-in batching provided by DataNucleus itself (makePersistentAll).

The database call to put now simply adds the data object to the queue.  A separate thread waits for the queue to be non-empty, then drains it and commits all the objects inside the current critical section.

For added efficiency, the patch implements four separate queues which are mapped to the four separate billing message types/tables; insert is thus specialized via method overloading so that DataNucleus need not have to filter the objects into distinct batched commit statements.

The additional abstraction layer is for flexibility, in case some other façade to the queueing structure is desired.

Several new properties (see below) expose tuning choices.

Testing:

Basic functionality tested.

Junit test modified to eliminate meaningless test case; other tests pass.

Stress tests which directly insert to the JDO layer were run with various settings to diagnose stability and performance.

Full testing with billing message processing (using a test cell which produces random billing messages) done on live dcache installation.

Target: 2.6
Patch: http://rb.dcache.org/r/5992/diff/#index_header
Require-notes: yes
Require-book: yes
Acked-by: Tigran
Committed: c799209b1b548d6e2c32baf5a6179c06b95dbef3

RELEASE NOTES:
BOOK:

An improved insert strategy for the billing database has been implemented.  The following changes to billing properties should be noted:

billingMaxInsertsBeforeCommit
billing.db.inserts.max-before-commit
billingMaxTimeBeforeCommitInSecs
billing.db.inserts.timeout-before-commit
billing.db.inserts.timeout-before-commit.unit

(one-of?org.dcache.services.billing.db.impl.DirectQueueDelegate)billing.db.inserts.queue-delegate.type=org.dcache.services.billing.db.impl.DirectQueueDelegate

billing.db.inserts.max-queue-size=100000

billing.db.inserts.max-batch-size=1000

(one-of?true|false)billing.db.inserts.drop-messages-at-limit=true

It should be noted that, in conjunction with http://rb.dcache.org/r/5983, which modifies the database triggers, insertion should now perform without message loss (and without running out of memory) under most conditions (3000Hz or less).
namespace: relax HSM URI validation rules

dcache requires hsm type and hsm instance are provided.
Ignore all other rules as it's OPAQUE to us.

Ticket: #7192
Acked-by: Gerd Behrmann
Target: master, 2.7, 2,6
Require-book: no
Require-notes: no
Merge pull request #221 from gbehrmann/fix/2.6/xrootdplugins

xrootd: Make xrootdPlugins unscoped for door
Merge pull request #216 from kofemann/iss7192-2.6

namespace: relax HSM URI validation rules
Merge pull request #215 from alrossi/fix/2.6/billing-insert-queueing

(2.6) billing: improve the insert queue logic
nfsv4: use concurrent hash map for mover bookkeeping

Acked-by: Gerd Behrmann
Require-notes: no
Require-book: no
target: master, 2.6
(cherry picked from commit aeb2f49c4dcc9d86e953870491501731c125c66d)
Merge pull request #222 from kofemann/nfs-race-2.6

nfsv4: use concurrent hash map for mover bookkeeping