billing: Fix regression in billing flat file format rendering

Addresses a regression in rendering the configurable billing format
strings. A bug prevented these formats from being respected. Note
that the format strings are now preprocessed by the dCache
configuration system, which means that escape sequences are
processed. You may have to update custom format strings (in
$$ sequences have to be replaced by $$$$).

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: yes
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/6065/
Refactor BootLoader

The patch moves the part for loading the dCache configuration into
the new LayoutBuilder class.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6059/
pnfsmanager: Drop unused fields and methods

These fields were only kept for backwards compatibility with
version of dCache no longer supported.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6052/
poolmanager: Delete unused messages

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6054/
Remove redundant and verbose code constructs

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6055/
doors: allow Transfer to accept a subject to talk to namespace

allow transfer to use a different subject for namespace and pool
operations. Update NFS door to use it.

Acked-by: Dmitry Litvintsev
Target: master
Require-book: no
Require-notes: no
srm: fix srmExtendFileLifeTime implementation

Addresses several issues:

- Improves error messages

- Uses standard compliant return codes

- Fixes seconds vs milliseconds bug in SURL lifetime implementation
  (which isn't supported by dCache, so an insignificant bug).

- Cleaned up the code (lots of public fields, unused fields, cut'n'paste
  comments that didn't belong, etc).

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6050/
billing: Add bloom filter indexer and search tool

Billing files are a valuable tool for forensic analysis. It's good practice
to keep them indefinitly. Unfortunately, billing files may grow large, making
it necessary to compress them. Compressed billing files in turn are very
expensive to search, making analysis more difficult.

This patch introduces a tool set for building a bloom filter for billing files.
The bloom filter can be used as a compact and fast probabilistic index. The
indexes are not exact, having a small false positive rate. However as an index
this only means that we sometimes have to search a billing file in vein, not
that we miss any entries.

The /usr/sbin/dcache-billing-indexer tool is used to index and compress
billing files. For the Debian package, I added a cron.daily file that invokes
this script with the -yesterday -compress options. This means that for Debian
packages we compress and index yesterdays billing file. I don't have the OS
to test an RPM build and I suggest somebody else adds a similar file for RPM
based distros. If no billing files are found (like on a non-billing host),
nothing happens. The tool has an -all option to (re)index all billing files.

The /usr/bin/dcache script is extended with a single new command to search
billing files, using the index if available.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6075/
billing: Add bloom filter indexer and search tool

Follow-up to 3c4a069810832c4b7ffcc9fbf72c8fbdc8796299: Adds new man
page to RPM spec file.

Target: trunk
Require-notes: no
Require-book: no
Fix white space bug in argument parser

When parsing arguments that have already been split, white space
should not be treated as a word separator or stripped.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6072/
billing: improve the insert queue logic

Revised version of http://rb.dcache.org/r/5841 (discarded)

Currently the way which billing data gets inserted via DataNucleus uses a twin mechanism:  on the one hand, the database API put() calls makePersistent on an open transaction, then asks for conditions to be checked (max inserts or timeout, whichever is reached first), and if so, commits the transaction.  At the same time, a daemon runs which periodically checks to see if those conditions have been met and to commit if so.

This is an unnecessarily complicated logic and can be simplified by making use of the following two features:

1.  java.util.concurrent.BlockingQueue
2.  the built-in batching provided by DataNucleus itself (makePersistentAll).

The database call to put now simply adds the data object to the queue.  A separate thread waits for the queue to be non-empty, then drains it and commits all the objects inside the current critical section.

For added efficiency, the patch implements four separate queues which are mapped to the four separate billing message types/tables; insert is thus specialized via method overloading so that DataNucleus need not have to filter the objects into distinct batched commit statements.

The additional abstraction layer is for flexibility, in case some other façade to the queueing structure is desired.

Several new properties (see below) expose tuning choices.

A separate patch for 2.2 will be posted.

Testing:

Basic functionality tested.

Junit test modified to eliminate meaningless test case; other tests pass.

Stress tests which directly insert to the JDO layer were run with various settings to diagnose stability and performance.

Full testing with billing message processing (using a test cell which produces random billing messages) done on live dcache installation.

Target: master
Request: 2.7
Request: 2.6
Patch: http://rb.dcache.org/r/5992/diff/#index_header
Require-notes: yes
Require-book: yes
Acked-by: Tigran

RELEASE NOTES:
BOOK:

An improved insert strategy for the billing database has been implemented.  The following changes to billing properties should be noted:

billingMaxInsertsBeforeCommit
billing.db.inserts.max-before-commit
billingMaxTimeBeforeCommitInSecs
billing.db.inserts.timeout-before-commit
billing.db.inserts.timeout-before-commit.unit

(one-of?org.dcache.services.billing.db.impl.DirectQueueDelegate)billing.db.inserts.queue-delegate.type=org.dcache.services.billing.db.impl.DirectQueueDelegate

billing.db.inserts.max-queue-size=100000

billing.db.inserts.max-batch-size=1000

(one-of?true|false)billing.db.inserts.drop-messages-at-limit=true

It should be noted that, in conjunction with http://rb.dcache.org/r/5983, which modifies the database triggers, insertion should now perform without message loss (and without running out of memory) under most conditions (3000Hz or less).
namespace: relax HSM URI validation rules

dcache requires hsm type and hsm instance are provided.
Ignore all other rules as it's OPAQUE to us.

Ticket: #7192
Acked-by: Gerd Behrmann
Target: master, 2.7, 2,6
Require-book: no
Require-notes: no
core: fix broken commit 187b136
billing: add commands for recording queue insert data

Adds commands to billing cell to be able to monitor insert queue statistics and to dump threads.

NOTE:  I have decided not to request back-porting to 2.7 or 2.6 for this patch.

TESTING:

[cmssrv153.fnal.gov] (billing) admin > log commit statistics on
on
[cmssrv153.fnal.gov] (billing) admin > log commit statistics off
off

billing log contains:
16 Sep 2013 11:51:32 (billing) [alm-admin-118] insert queue (last 0, current 0, change 0/minute)
16 Sep 2013 11:51:32 (billing) [alm-admin-118] commits (last 0, current 0, change 0/minute)
16 Sep 2013 11:51:32 (billing) [alm-admin-118] dropped (last 0, current 0, change 0/minute)
16 Sep 2013 11:51:32 (billing) [alm-admin-118] total memory 505282560; free memory 482253512

this repeated every minute until turned off.

Target: master
Patch: http://rb.dcache.org/r/5994/
Require-notes: yes
Require-book: yes
Acked-by: Karsten

RELEASE NOTES:
BOOK:

You can now obtain statistics (printed to billing log and pinboard) via the dcache admin command:
display insert statistics <on/off>

Turning this command on logs the following once a minute:

insert queue (last 0, current 0, change 0/minute)
commits (last 0, current 0, change 0/minute)
dropped (last 0, current 0, change 0/minute)
total memory 505282560; free memory 482253512

"insert queue" refers to how many messages actually were put on the queue; "commits" are the number of messages committed to the database; dropped are the number of lost messages. "last" refers to the figures at the last iteration.  For insert queue, this is the actual size of the queue; for commits and dropped, these are cumulative totals.

You can also generate a Java thread dump by issuing the "dump threads" command.
dcache-webadmin: reformat html pages

Wicket mark-up was messy.  All html resources for webadmin have been reformatted for regularity and readability.

(The extra indentation is easier to maintain in my HTML editor.)

Testing: Redeployed and checked webadmin pages.

Target: master
Request: 2.7
Patch: http://rb.dcache.org/r/6077
Require-notes: no
Require-book: no
Acked-by: Tigran
dcache-webadmin: fix page update

This patch addresses several RT tickets stemming from the same problem, one which has already been corrected for the CellServices page (see http://rb.dcache.org/r/5807).

This has to do with the way Wicket PropertyModels were provided for pages which maintain lists which need to be updated through the service layer.  In most of these cases, the list was set in the constructor but was never updated during page/form refresh.  The changes adopted here follow the same general pattern of replacing the field accessed by the model with a getter method which also takes care of calling the service to update the collection in question.

Testing:

Redeployed and checked updating by enabling/disabling pools.

Target: master
Request: 2.7
Patch: http://rb.dcache.org/r/6080
Require-notes: yes
Require-book: no
Bug: http://rt.dcache.org/Ticket/Display.html?id=7929
Bug: http://rt.dcache.org/Ticket/Display.html?id=7964
Bug: http://rt.dcache.org/Ticket/Display.html?id=7983
Acked-by: Gerd
Acked-by: Tigran

RELEASE NOTES:
Fixes an issue with some of the tables in the webadmin service which were not refreshing.  Update correctly occurs now on reload, resubmission of the form through link clicking
Revert "dcache-webadmin: fix page update"

This reverts commit cc7166ad47b6aeada7f2c9ef5ef0a0183d93bb8a.

Need to resolve issue with PoolUsage form.

Will repost when fixed.
Update example layouts

Addresses the problem that the example layout files contains services
and properties that no longer exist or have been deprecated.

Addresses the problem that the head.conf example layout file is a
rather bad example of how to place services in domains. Too many sites
have copied this layout, believing it is a good setup for a head node.
The problem is addressed by removing this layout.

Addresses the problem that our default layout is single, even though
best practice recommendation is to use the host name. The default is
now the host name. This affects sites on upgrade if they relied on the
old default.

Addresses the problem that we ship example files in a configuration
directory. Instead these are now placed in /usr/share/dcache/examples/layouts.

Release notes should probably contain a note reminding RPM users that if
they called their layout file either single, head or pool, that they
should check that RPM didn't rename their layout file.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6076/
Make message type final in InfoMessage

The setter was only used during initialization. This patch makes the
field final and sets the value through the constructor. This makes it
simpler to enumerate all possible values in a parser for billing files.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6074/
srm: Improve error codes and messages in srmReleaseFiles implementation

Addresses issues in srmReleaseFiles, ensuring that spec compliant return
codes are used, and better error messages are returned to the client.

Addresses issues in which release files in some casses would trigger
IllegalStateTransition exceptions and cause the release file operation
to fail.

The patch refactors some URI handling and request token handling
to reduce the number of times we need to do convert between types.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6086/