(2.6) webadmin: eliminate system-specific dir for unpacking of war file

Version of http://rb.dcache.org/r/6002 for 2.6.

See above-cited patch for details.

Target: 2.6
Patch: http://rb.dcache.org/r/6003/
Require-book: no
Require-notes: yes
Bug: http://rt.dcache.org/Ticket/Display.html?id=7979
Acked-by: Paul

Testing:  as with http://rb.dcache.org/r/6002.

RELEASE NOTES:

Fixes bug where on some systems automatic cleanup of temporary directories was eliminating the webapp application files and thus causing the webadmin pages to fail to be found.
Renamed .gitignore to .empty-dir in conformity with dCache usage.
Merge pull request #195 from kofemann/iss8020-2.6

Iss8020 2.6
chimera: Fix major performance issue on file creation with ACL enabled

Chimera uses a trigger on insert into t_dirs to copy the ACLs of the
parent directory to the newly created inode. The stored procedure used
for this uses character(36) to represent PNFS IDs. This causes a
problem with postgresql and it refuses to use the indexes when the
type of the value you search for is different from the type of the
column - even if one is character(36) and the other character varying(36).

The consequence is that every insert into t_dirs causes a linear
scan of t_acl - something that becomes slow and cpu intensive once
you have millions of entries.

Insertion time into t_dirs dropped from around 2 seconds to 2 milliseconds
when we applied this patch at NDGF.

Implementation note: I replaced the existing changeset as it uses
create or replace anyway. That way new sites will never see the buggy
implementation, and old sites will execute the change set as it has
a new id.

Target: trunk
Request: 2.7
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6049/
(cherry picked from commit 780a19ac41c08faad5992925129e909b7ed41f94)
billing (plots): adjust upper bound of aggregate plots

Because of the fact that the aggregate tables are now updated only once a day for the data from the previous day, it does not make sense to $

Testing:

Redeployed and observed plots.

Target: 2.6
Patch: http://rb.dcache.org/r/6034/
Require-notes: no
Require-book: no
Acked-by: Karsten
Committed: dde0fcadfa2094d8ee1ad4af217fcec15be00897
pnfsmanager: Use correct queue for create entry and map messages

Messages are put on a pnfs manager queue by pnfs id or path. This is to
ensure that we don't perform concurrent operations on a name space
object and two enforce a happens-before relationship on operations on
a name space object (ie to avoid messages from being processed out of
order). It is also essential for request folding to work correctly.

PnfsCreateEntryMessage and PnfsMapPathMessage however use their own path
field rather than the one in the PnfsMessage base class. Therefore a
random queue was chosen for these messages.

This patch fixes this by also binding the path in the base class. A future
patch for master should remove the redudant fields.

Target: trunk
Request: 2.7
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6051/
(cherry picked from commit f9f050dc52516bf04e8ba1850ad70dde8ffd0276)
http: fix context based pages

regression introduced by 8b7c60e

Acked-by: Albert Rossi
Ticket: #8024
Target: master
Require-book: no
Require-notes: no
xrootd: Make xrootdPlugins unscoped for door

The consequences of accidentally configuring xrootdPlugins in
dcache.conf can be quite big: Access to the door would be
unauthenticated, and possible name space mapping plugins would
not be applied.

To reduce the risk of this problem, and since we removed the
scoping operator in 2.7 anyway, this patch makes xrootdPlugins
unscoped for the door. Pools continue to use the scoped xrootdPlugins
property, but the patch introduces the pool.mover.xrootd.plugins
name as an alternative. This is the same name that was introduced
in 2.7.

Target: 2.6
Require-notes: yes
Require-book: yes
Merge pull request #209 from kofemann/iss8024-2.6

Iss8024 2.6
Merge pull request #193 from alrossi/fix/2.6/unpackdir

Fix/2.6/unpackdir
Merge pull request #197 from gbehrmann/fix/2.6/rb6049

chimera: Fix major performance issue on file creation with ACL enabled
Merge pull request #201 from alrossi/fix/2.6/plot-upper-bound

Fix/2.6/plot upper bound
Merge pull request #206 from gbehrmann/fix/2.6/rb6051

Fix/2.6/rb6051
(2.6) billing: improve the insert queue logic

Revised version of http://rb.dcache.org/r/5841 (discarded)

Currently the way which billing data gets inserted via DataNucleus uses a twin mechanism:  on the one hand, the database API put() calls makePersistent on an open transaction, then asks for conditions to be checked (max inserts or timeout, whichever is reached first), and if so, commits the transaction.  At the same time, a daemon runs which periodically checks to see if those conditions have been met and to commit if so.

This is an unnecessarily complicated logic and can be simplified by making use of the following two features:

1.  java.util.concurrent.BlockingQueue
2.  the built-in batching provided by DataNucleus itself (makePersistentAll).

The database call to put now simply adds the data object to the queue.  A separate thread waits for the queue to be non-empty, then drains it and commits all the objects inside the current critical section.

For added efficiency, the patch implements four separate queues which are mapped to the four separate billing message types/tables; insert is thus specialized via method overloading so that DataNucleus need not have to filter the objects into distinct batched commit statements.

The additional abstraction layer is for flexibility, in case some other façade to the queueing structure is desired.

Several new properties (see below) expose tuning choices.

Testing:

Basic functionality tested.

Junit test modified to eliminate meaningless test case; other tests pass.

Stress tests which directly insert to the JDO layer were run with various settings to diagnose stability and performance.

Full testing with billing message processing (using a test cell which produces random billing messages) done on live dcache installation.

Target: 2.6
Patch: http://rb.dcache.org/r/5992/diff/#index_header
Require-notes: yes
Require-book: yes
Acked-by: Tigran
Committed: c799209b1b548d6e2c32baf5a6179c06b95dbef3

RELEASE NOTES:
BOOK:

An improved insert strategy for the billing database has been implemented.  The following changes to billing properties should be noted:

billingMaxInsertsBeforeCommit
billing.db.inserts.max-before-commit
billingMaxTimeBeforeCommitInSecs
billing.db.inserts.timeout-before-commit
billing.db.inserts.timeout-before-commit.unit

(one-of?org.dcache.services.billing.db.impl.DirectQueueDelegate)billing.db.inserts.queue-delegate.type=org.dcache.services.billing.db.impl.DirectQueueDelegate

billing.db.inserts.max-queue-size=100000

billing.db.inserts.max-batch-size=1000

(one-of?true|false)billing.db.inserts.drop-messages-at-limit=true

It should be noted that, in conjunction with http://rb.dcache.org/r/5983, which modifies the database triggers, insertion should now perform without message loss (and without running out of memory) under most conditions (3000Hz or less).
namespace: relax HSM URI validation rules

dcache requires hsm type and hsm instance are provided.
Ignore all other rules as it's OPAQUE to us.

Ticket: #7192
Acked-by: Gerd Behrmann
Target: master, 2.7, 2,6
Require-book: no
Require-notes: no
Merge pull request #221 from gbehrmann/fix/2.6/xrootdplugins

xrootd: Make xrootdPlugins unscoped for door
Merge pull request #216 from kofemann/iss7192-2.6

namespace: relax HSM URI validation rules
Merge pull request #215 from alrossi/fix/2.6/billing-insert-queueing

(2.6) billing: improve the insert queue logic
nfsv4: use concurrent hash map for mover bookkeeping

Acked-by: Gerd Behrmann
Require-notes: no
Require-book: no
target: master, 2.6
(cherry picked from commit aeb2f49c4dcc9d86e953870491501731c125c66d)
Merge pull request #222 from kofemann/nfs-race-2.6

nfsv4: use concurrent hash map for mover bookkeeping