TransferManager: fix error-handling

An earlier patch (5ed882c6cd56bd9cdc50a9017a2cb74047e00607) switched from
using CellAdapter#sendMessage to the recently introduced call-back mechanism.
This separates the normal and error response handling into separate call-back
methods.  Unfortunately the patch didn't also refactor the error handling
in TransferManagerHandler so the existing error handling remained in the
successful code-path, where it was never called.

This patch moves the error handling to the error call-back method.  It
tidies up the error messages and reduces the logging somewhat.  The retry
if the delete of a file fails is refactored slightly.

Target: master
Patch: http://rb.dcache.org/r/6889/
Acked-by: Gerd Behrmann
Enable IPv6 stack by default

The patch enables the use of the IPv6 network stack if available. This
means that dCache can be used with IPv6 sockets.

The JVM still defaults to preferring IPv4 addresses, even when using the
IPv6 stack.

On installation the old behavior can be reactivated by configuring dcache.java.options.extra=-Djava.net.preferIPv4Stack=true.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6901/
ftp: Implement delayed passive and enable redirect FTP for IPv6

Implements the Globus delayed passive extension. Delayed passive
allows dCache to redirect passive data channels to pools without
using GridFTP 2 GET and PUT commands. Delayed passive can be used
with IPv6, which is not the case for GET and PUT.

Pools that do not support IPv6 or pools older than 2.9 will fall
back to using the door as a proxy if an IPv6 data channel is
requested.

Target: trunk
Require-notes: yes
Require-book: yes
Patch: http://rb.dcache.org/r/6895/
Patch: http://rb.dcache.org/r/6900/
Patch: http://rb.dcache.org/r/6902/
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Update libraries to the latest releases

Spring 3.0.4
xrootd4j 1.3.2
HikariCP 1.3.5
commons-compress 1.8

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6903/
pool: Let HsmStorageHandler2#store accept more than one file

This pushes the batching granularity all the way down to HsmStorageHandler2.

Target: master
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6870/
Generify CDC executor decorators

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6887/
pool: Rewrite of HSM interface

The patch introduces a new SPI for tape drivers. One possible implementation
of this SPI is included for calling out to legacy HSM scripts. Thus backwards
compatibility is maintained, with only minor behavioural changes.

The main class of the new HSM subsystem is NearlineStorageHandler.  It replaces
the old HsmStorageHandler2 class. NearlineStorageHandler handles store, fetch
and remove requests. It passes on those requests to a NearlineStorageProvider
implementation. NearlineStorageProvider is implemented by nearline storage
providers (drivers) and implementations are discovered through Java's
ServiceLoader facility.

NearlineStorageHandler interacts with the provider by submitting request
objects represented by the three interfaces StageRequest, FlushRequest and
RemoveRequest. These interfaces provide interfaces to report completion or
failure, but also for reporting transition from queued to activate state and
for space allocation. NearlineStorageHandler uses those callbacks to implement
the usual pre-flush lookups, checksum policies, allocate space, etc. A provider
thus does not interact with the rest of dCache and is unaware of the details of
such interactions and policies. Queuing on the other hand is fully implemented
by the provider. This is to give the provider as much flexibility to optimize
interaction with the tape system as possible. The provider may have no queueing
at all and submit all requests to the tape system, it may have detailed
knowledge about tape layout and reorder requests itself, or it may have a
simple queueing scheme like we had before. The point is that this is left to
the provider. Similarly, for staging the point of space allocation is left to
the provider, which allows the provider to request more stages from tape than
will currently fit on the pool, allocating space in the order in which files
arrive from tape.

The functionality of the old HsmStorageHandler2 class has thus been split
between NearlineStorageHandler and the provider.

Since queueing is part of the provider, the maximum number of concurrent tasks
is thus also part of the provider and may not exist for some providers. Thus it
is no longer reported to pool manager for flush and stage queues.

The HsmSet class has been refactored to allow the provider to be specified when
definining new nearline storages. The new 'hsm create' command must be called
before 'hsm set' and accepts the provider name. Care has been taken to maintain
backwards compatibility with old pool setup files, both for defining storages
and for setting concurreny limits.

A default provider has been implemented. This provider uses FIFO queuing and
callouts to an external HSM script. It is thus very similar to the old
HsmStorageHandler2. Old tape integration scripts can be reused without
modification.

Three additional providers are provided for purposes of demonstration and testing,
but may also have pratical use:

- The copy provider copies files to another directory (like our standard hsmcp
  scripts).
- The link provider creates hard links in another directory (achieving a
  similar effect to the copy provider, but without the physical copy and
  limited to the same file system).
- The tar provider is incomplete, but serves as a demo of a provider that works
  on several files at once: It bundles multiple files into a tarball and reorders
  staging by tarball. It doesn't support request cancellation or file removal.

Due to the changes in the hsm commands, it is advisable to regenerate the pool
setup file after upgrade (run 'save' in the pool).

Observable changes in behaviour:

- Concurrency of the script provider is configured through options rather than
  the dedicated 'xx set max active' commands.
- Concurrency settings are interpreted per nearline storage instance per pool,
  not as a sum for all nearline storages on a pool.
- Timeouts are interpreted for the entire request period, not just for the
  period during which there is a callout to the script. This is not set in
  stone, but the point being that a client doesn't care whether we are queueing
  or waiting for a script: SRM in particular defines a timeout for the stage and
  after that the client doesn't care about the result anymore. I hope that
  eventually we can include the timeout in the stage request from SRM, thus
  avoiding to configure the timeout in the pool.
- When the timeout is adjusted, this also affects existing requests (although
  the script provider doesn't adjust it's internal timeout when the script was
  already started, but this is subject to change).
- Changed the default of cleaner.ensble.hsm to true, meaning HSM clean is
  enabled by default (defaulting to not rely on some external component seems
  like the more sensible choice).
- Rather than as-hoc return codes, most HSM failures are translated to the
  default CacheException return code. The script provider obviously still uses
  the exit code of the script as the return code.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
scripts: fix host-credential generation

The system-test includes a script for generating a host
credential.  It currently contains BASH-isms and Debian-isms,
which this patch fixes.

Target: master
Request: 2.8
Request: 2.7
Patch: http://rb.dcache.org/r/6908/
Acked-by: Gerd Behrmann
releases: update dCache version to v2.10
pool: wake-up sweeper if new entry is added

To avoid busy-loop sweeper uses Thread#sleep(). But, it makes
sense to wake-up if list of file is updated.

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no