chimera: fix behavior on create of '.(fset)' files

the fset/pset files are special commands and we
do not allow to create them. Nevertheless, current
behavior is to throw IOError.

With this change will will throw Exists which will force client
to fall-back to regular open-for-write.

Fixes fset over NFSv4.1

Acked-by: Albert Rossi
Acked-by: Paul Millar
Target: master, 2.8, 2.7, 2.6
Require-book: no
Require-notes: no
libs: use nfs 0.8.0

a major update of NFS code with bugfix and fuctional changes:

  - simplify data types for performace
  - BETA level support for read/write with NFSv4.0
  - improved spec compliance
  - use inodes generation for change detection
  - added lookup and attribute cache
  - new oncrpc-2.2 with error handling of bad xdr streams

all fixes for 0.7 branch included as well.
The most visible change for dCache is better NFSv4.0, which allows
IO for v4.0-only clients, like mac osx 10.9, and VfsCache.

Changelog for nfs4j-0.7.0..nfs4j-0.8.0
    * [0cb05a2] core: fix data file creation in simple-server
    * [8b534f7] vfs: chimera: let inode to call setStat()
    * [1454aa5] nfs4: implement missing RECLAIM_COMPLETE
    * [4710b27] nfs4: fix open CLAIM_PREVIOUS
    * [d56193f] pom: enforce  maven-release-plugin 2.4.2
    * [1c916c5] vfs: add a special option to hide dcache from dcap clients
    * [1c89149] vfs: fix regression introduced in 1c916c5
    * [39fcca6] nfs4: refactor/optimize data structures
    * [45f0a62] nfs4: simplify nfstime4 structure
    * [9b8420d] nfs4: fix millis to nfstime4 conversion
    * [fefb630] nfs4: simplify data types fattr4_{files,space} and Co.
    * [00118a8] nfs4: fix nfstime4 encoding
    * [6d05b98] nfs4: simplify bitmap4
    * [44e4d54] nfs4: fix lookupp on the root of the tree
    * [0222c7d] nfs4: fix handling of illegal operations
    * [dc38de7] nfs4: client: add read-no-state command
    * [7e865b5] nfs4: simplify data types
    * [ab1a807] nfs4: fix secinfo_no_name operation
    * [2b161a3] nfs4: fix behavior on undefined operation
    * [59046c9] nfs4: fix parent lookup behavior on a symlink
    * [1119117] nfs4: fix open_claim after 'reclaim complete'
    * [1fd3d92] nfs4: fix regular open before 'reclaim complete'
    * [cde8648] nfsv4: fix regression introduced in 7e865b5
    * [57153a6] nfs4: fix create_session of expired clients
    * [f0b201e] nfsv41: add support for IO without pNFS
    * [f6dfa96] nfsv4: some operation must be solo in the compound
    * [eddcec4] nfsv4: DESTROY_CLIENTID must return BUSY on existing sessions
    * [b73fc63] nfsv: fix client principal for non GSS credentials
    * [9854b6d] nfsv4: open_owner clientid used for v4.0 only
    * [9c342a9] vfs: chimera: translate exceptions to nfs errors
    * [7ec5cb3] nfsv4: add a special operation factory for local IO
    * [333c6af] nfs4: simplify data types
    * [3ae2ea2] vfs: update interface to match chimera changes
    * [15e274e] src: remove jna based nss-id-mapping
    * [7a54d56] vfs: fix export idx on get parent
    * [cddf937] nfs4: verify seqid in OPEN_CONFIRM as per RFC
    * [ad27d17] nfsv41: add check for max number of operations in a COMPOUND
    * [b61060a] nfs4: commit allowed only for file objects
    * [2efcd7d] nfs4: consolidate object name checks into a single place
    * [2345ce5] nfs4: allow IO only on file objects
    * [c0e40b4] nfs4: validate nanos in nfstime4
    * [fe439ec] nfs4: validate readdir cookie
    * [8d2e6ac] pom: update junit to support single test run from IDE
    * [61d15cb] nfs4: do strict utf8 check
    * [f9156b2] nfs4: limit max session slots
    * [46540c3] nfs4: respect client's requested back channel reply cache size
    * [8c0d4ab] nfs4: remove redundant List object
    * [852f4d0] nfs4: remove redundant autoboxing
    * [20c12d4] nfs4: fix hard link constrains
    * [f8f0999] nfs40: fix setclientid
    * [aea34e8] nfs40: fix open_confirm with bad stateid
    * [ee87871] nfs41: implement putpubfh
    * [f91b989] nfs4: fix secinfo
    * [0d746c1] nfs4: allow open only for regular files
    * [202c4b9] nfs4: verify/nverify should check that all requested attributes evaluated
    * [1bfaaf7] nfs4: hidden and system are unsupported file attributes
    * [c7ad329] nfs4: verify/nverify must return error if rdattr_error is used
    * [f0f11e5] nfs4: add bitmap4#isEmpty() to check is any bit set
    * [7ba3337] nfs: propagate nfs exception on readdir
    * [d3ac4a1] vfs: chimera: do extra access check only for regular inodes
    * [119f722] nfs4: introduce stateid4#equalsWithSeq()
    * [b3efb2c] nfs40: update stateid on read and write
    * [ff190f3] nfs4: check state's sequence id when updating lease time
    * [009a32c] nfs4: do not set size on non file objects
    * [3d1a845] nfs: add a possibility to shutdown Cache's cleanup thread
    * [b6a6e12] nfs4: add utility method to check for stateless stateids
    * [93e8059] vfs: fix io on levels
    * [41def7d] nfs4: fix change_info on open-create
    * [c73e09b] vfs: call stat only when it's needed
    * [414c947] nfs: remove useless rpc program check
    * [71d00a6] libs: update to oncrpc4j 2.2
    * [ab2118c] nfs4: validate attributes on setattr
    * [a4461fa] vfs: update cache decorator to cache stat information
    * [d2d365b] vfs: use backend filesystem file id as a cache key
    * [25ed237] pom: enforce utf8 for source files
    * [61015d9] nfs: reject readdir requests if cached listing not found
    * [be616ab] nfs4: set the file mode with create operation
    * [306ea49] vfs: introduce AclCheckable interface to VirtualFileSystem
    * [3084d3a] nfs4: use inode generation to indicate file/direcory change

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: yes
nfs: use vfs cache

make use of attribute caching. Three new options are added:

    nfs.namespace-cache.time
    nfs.namespace-cache.unit
    nfs.namespace-cache.size

The default value for caching time is 3 sec, which an optimal for most workloads.
The default cache size (number of cached objects) is zero, which is effectively, switching off
the caching.

Each caches entry takes 120 bytes of additional memory.

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: yes
TransferManager: fix error-handling

An earlier patch (5ed882c6cd56bd9cdc50a9017a2cb74047e00607) switched from
using CellAdapter#sendMessage to the recently introduced call-back mechanism.
This separates the normal and error response handling into separate call-back
methods.  Unfortunately the patch didn't also refactor the error handling
in TransferManagerHandler so the existing error handling remained in the
successful code-path, where it was never called.

This patch moves the error handling to the error call-back method.  It
tidies up the error messages and reduces the logging somewhat.  The retry
if the delete of a file fails is refactored slightly.

Target: master
Patch: http://rb.dcache.org/r/6889/
Acked-by: Gerd Behrmann
Enable IPv6 stack by default

The patch enables the use of the IPv6 network stack if available. This
means that dCache can be used with IPv6 sockets.

The JVM still defaults to preferring IPv4 addresses, even when using the
IPv6 stack.

On installation the old behavior can be reactivated by configuring dcache.java.options.extra=-Djava.net.preferIPv4Stack=true.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6901/
ftp: Implement delayed passive and enable redirect FTP for IPv6

Implements the Globus delayed passive extension. Delayed passive
allows dCache to redirect passive data channels to pools without
using GridFTP 2 GET and PUT commands. Delayed passive can be used
with IPv6, which is not the case for GET and PUT.

Pools that do not support IPv6 or pools older than 2.9 will fall
back to using the door as a proxy if an IPv6 data channel is
requested.

Target: trunk
Require-notes: yes
Require-book: yes
Patch: http://rb.dcache.org/r/6895/
Patch: http://rb.dcache.org/r/6900/
Patch: http://rb.dcache.org/r/6902/
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Update libraries to the latest releases

Spring 3.0.4
xrootd4j 1.3.2
HikariCP 1.3.5
commons-compress 1.8

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6903/
pool: Let HsmStorageHandler2#store accept more than one file

This pushes the batching granularity all the way down to HsmStorageHandler2.

Target: master
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6870/
Generify CDC executor decorators

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6887/
pool: Rewrite of HSM interface

The patch introduces a new SPI for tape drivers. One possible implementation
of this SPI is included for calling out to legacy HSM scripts. Thus backwards
compatibility is maintained, with only minor behavioural changes.

The main class of the new HSM subsystem is NearlineStorageHandler.  It replaces
the old HsmStorageHandler2 class. NearlineStorageHandler handles store, fetch
and remove requests. It passes on those requests to a NearlineStorageProvider
implementation. NearlineStorageProvider is implemented by nearline storage
providers (drivers) and implementations are discovered through Java's
ServiceLoader facility.

NearlineStorageHandler interacts with the provider by submitting request
objects represented by the three interfaces StageRequest, FlushRequest and
RemoveRequest. These interfaces provide interfaces to report completion or
failure, but also for reporting transition from queued to activate state and
for space allocation. NearlineStorageHandler uses those callbacks to implement
the usual pre-flush lookups, checksum policies, allocate space, etc. A provider
thus does not interact with the rest of dCache and is unaware of the details of
such interactions and policies. Queuing on the other hand is fully implemented
by the provider. This is to give the provider as much flexibility to optimize
interaction with the tape system as possible. The provider may have no queueing
at all and submit all requests to the tape system, it may have detailed
knowledge about tape layout and reorder requests itself, or it may have a
simple queueing scheme like we had before. The point is that this is left to
the provider. Similarly, for staging the point of space allocation is left to
the provider, which allows the provider to request more stages from tape than
will currently fit on the pool, allocating space in the order in which files
arrive from tape.

The functionality of the old HsmStorageHandler2 class has thus been split
between NearlineStorageHandler and the provider.

Since queueing is part of the provider, the maximum number of concurrent tasks
is thus also part of the provider and may not exist for some providers. Thus it
is no longer reported to pool manager for flush and stage queues.

The HsmSet class has been refactored to allow the provider to be specified when
definining new nearline storages. The new 'hsm create' command must be called
before 'hsm set' and accepts the provider name. Care has been taken to maintain
backwards compatibility with old pool setup files, both for defining storages
and for setting concurreny limits.

A default provider has been implemented. This provider uses FIFO queuing and
callouts to an external HSM script. It is thus very similar to the old
HsmStorageHandler2. Old tape integration scripts can be reused without
modification.

Three additional providers are provided for purposes of demonstration and testing,
but may also have pratical use:

- The copy provider copies files to another directory (like our standard hsmcp
  scripts).
- The link provider creates hard links in another directory (achieving a
  similar effect to the copy provider, but without the physical copy and
  limited to the same file system).
- The tar provider is incomplete, but serves as a demo of a provider that works
  on several files at once: It bundles multiple files into a tarball and reorders
  staging by tarball. It doesn't support request cancellation or file removal.

Due to the changes in the hsm commands, it is advisable to regenerate the pool
setup file after upgrade (run 'save' in the pool).

Observable changes in behaviour:

- Concurrency of the script provider is configured through options rather than
  the dedicated 'xx set max active' commands.
- Concurrency settings are interpreted per nearline storage instance per pool,
  not as a sum for all nearline storages on a pool.
- Timeouts are interpreted for the entire request period, not just for the
  period during which there is a callout to the script. This is not set in
  stone, but the point being that a client doesn't care whether we are queueing
  or waiting for a script: SRM in particular defines a timeout for the stage and
  after that the client doesn't care about the result anymore. I hope that
  eventually we can include the timeout in the stage request from SRM, thus
  avoiding to configure the timeout in the pool.
- When the timeout is adjusted, this also affects existing requests (although
  the script provider doesn't adjust it's internal timeout when the script was
  already started, but this is subject to change).
- Changed the default of cleaner.ensble.hsm to true, meaning HSM clean is
  enabled by default (defaulting to not rely on some external component seems
  like the more sensible choice).
- Rather than as-hoc return codes, most HSM failures are translated to the
  default CacheException return code. The script provider obviously still uses
  the exit code of the script as the return code.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
scripts: fix host-credential generation

The system-test includes a script for generating a host
credential.  It currently contains BASH-isms and Debian-isms,
which this patch fixes.

Target: master
Request: 2.8
Request: 2.7
Patch: http://rb.dcache.org/r/6908/
Acked-by: Gerd Behrmann
releases: update dCache version to v2.10
SRM: add support for 3rd-party HTTP copy

This patch adds support for pulling or pushing data to a remote server via HTTP.
pool: switch to using HttpClient