pnfsmanager: Use ISO8601 compatible date for metadataof output

metadataof now includes the year in dates.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6379/
Fix PoolRemoveFilesMessage#toString

There is an erroneous comma at the beginning of the file list.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6378/
libs: update nfs4j to 0.7.0

fixes compatibility with rhel7 and linux kernels > 3.11
provides functionality required by proxy IO

Changelog for nfs4j-0.6.3..nfs4j-0.7.0
    * [ffcf8a9] vfs: update API interface to match new changes in chimera
    * [3252e04] nfsv3: fix creation of symbolic link
    * [f3a8db4] nfsv3: fix hard link creation (similar to http://rb.dcache.org/r
    * [a3642de] src: keep style check happy
    * [89346e7] nfsv4: remove dead code
    * [6e7cd2a] vfs: chimera: avoid extra db lookup
    * [cc22d5c] test: do not use obsolete ip host name
    * [5958f97] nfsv4: add state dispose listener
    * [1e946d2] core: implement missing SECINFO_NO_NAME
    * [03201b8] pom: update liquibase-core to 3.0.8

Acked-by: Gerd Behrmann
Target: master, 2.7, 2.6
Require-book: no
Require-notes: no
nfs4: implement proxy-io on read

Linux client too often falls back to IO through MDS (door).
This ends up with IO error on the client side as dCache does
not supports it.

This change set aims to add proxy IO by using internally DCAP
protocol. The door-pool communication can be changed later to
other protocol.

This is for master, nevertheless, 2.7 and 2.6 should have it as well.

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
spacemanager: Fix use of global property in spacemanager configuration

Spacemanager did not respect the spacemanager.enable.space-reservation
property.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6388/
Add preallocation field to PoolAcceptFileMessage

Adds a field for preallocation to the pool accept message - I imagine the pool
could make use of this information to preallocate space during mover startup or
reject a mover if not enough garbage collectible space is available up front.
In the short term this will be used by space manager for implicit space
reservation.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6386/
spacemanager: Refactor message dispatching code

Ensures that message forwarding and reply logic is all places in
messageToForward/messageArrived, all error injection logic in processMessage,
and all other methods are ignorant to how and when messages are sent.

All improves error handling a bit.

The layering achieved by this class makes subsequent refactoring of the code
simpler.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6389/
Upgrade dependencies

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6371/
spacemanager: Postpone DB update to intercepted pool accept message

A recent patch moved the DB update on transfer to the intercepted
pool selection reply. This patch moves those updates even further
back so that they now happen in the intercepted pool accept request,
which triggers the creation of a mover in the pool.

There are several benefits to this change:

- If pool selection took a long time and the door no longer exists
  (client gave up), then we now do not create any temporary reservations.
  Before a temporary reservation would be created, but since there
  would not have been a DoorTransferFinishedMessage or any other
  message to indicate a failure, the temporary reservation would not
  be cleaned until it expired.

- In case a component does a pool selection without starting a mover
  (eg older versions of pin manager or SRM would do this), doing the
  reservation update in the pool selection is clearly wrong.

- More messages of transfers without space reservation can now
  bypass queuing in space manager. This is because the LinkGroup entry
  in storage info is now passed to the pool and can be used as a
  simple test whehter additional processing in space manager is
  required.

- We are one step closer to being able do pool selection in doors: since
  space manager no longer updates its database on pool selection, we could
  load all information needed to select a linkgroup into memory, serialize
  it and transfer it to doors (just like we do with pool manager's
  PoolMonitor).

Temporary reservations created by space manager (when reservation
for non-srm transfers is enabled or a default space token is defined for
the target directory) are no longer bound to the path - the reservations
are created directly with a pnfs id and the path field is null (this
change is admin visible).  This has two benefits:

- There is no risk of duplicate paths. A failed upload will not block
  a subsequent retry even if the retry is performed before the delete
  notification reaches space manager.

- In case of failures, space manager can recognize which reservation it
  created itself (those without a path) and delete those rather than
  unpin the pnfs id (a reservation without a path would not be useable
  to any future transfer anyway as there is no way to bind that transfer
  to the reservation; the only point of having the path in space manager
  is so SRM can create entries before a PNFS ID has been generated).

One consequence of the patch is that space manager once again intercepts
the pool accept message. In return the pool selection reply is no
longer intercepted, thus this change should not affect latency.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6360/
spacemanager: Do not delete SRM managed reservation entries on file deletion

SRM creates file entries in space reservations prior to asking a client to
upload a file through some transfer protocol. It is valid for a client to retry
a failed upload without renegotiating the transfer with the SRM (our own srm
client does this). However when the name space entry is deleted after a failed
upload, the space manager marks the file entries as deleted and possibly
deletes the file entry from its tables (depends on whether a disk replica was
registered at that time). Thus when the client retries the upload, the upload
will fail or a temporary reservation will be created and used instead of the
one managed by the SRM.

This patch addresses this problem by not deleting entries/marking entries as
deleted in state RESERVED or TRANSFERRING unless they do not have a path
associated with them (entries without a path cannot be reused anyway). Instead
such entries are reset to RESERVED and the associated PNFS ID is removed.

The patch also cleans up some related code.

Target: trunk
Require-notes: yes
Require-book: no
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7799
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6361/
spacemanager: Mover DB schema management to Liquibase

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6368/
spacemanager: Drop 'fix missing size' command

The command was used to recover from a bug in dCache 1.9.1 and
is no longer needed.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litinvtsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6372/
spacemanager: Maintain aggregate columns with database triggers

Moves the logic to maintain aggregate columns in srmlinkgroup and srmspace from
Java code to database triggers. Adds not-null constraints to several space
manager database columns. This may take a little while to apply on large
databases.

The schema modifications are not compatible with older versions. Should a
downgrade be needed, the schema modifications have to be rolled back using
'dcache database rollback <tag> SrmSpaceManager@<domain>' or 'dcache database
rollback <date> SrmSpaceManager@<domain>'. The former requires that a tag was
created before upgrade (this is recommended). Should the system be downgraded
without rolling back the schema changes, the space accounting will be
incorrect.

Target: master
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6370/
spacemanager: Remove dead code and other minor cleanup

All identified by IntelliJ.

Target: master
Require-notes: no
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6373/
spacemanager: More code cleanup

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6374/
spacemanager: Make database connection pool configurable

Moves BoneCP instantiation into Spring and exposes connection pool parameters
in configuration files.

New configuration properties:

spacemanager.db.connections.partition-count=3
spacemanager.db.connections.max-per-partition=10
spacemanager.db.connections.min-per-partition=1

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6375/
spacemanager: Do not use ThreadManager

Space manager uses a regular Java thread pool instead. Introduces
the following new configuration property:

spacemanager.limits.threads=30

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6376/
spacemanager: Use PoolMonitor to extract link groups

Since PoolMonitor is imported every 30 seconds anyway, we don't
need to ask pool manager for link groups anymore.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6377/
spacemanager: Add missing rollback in liquibase

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Dmitry Litvtinsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6392/
nfs4: fix IO on special files

regression introduced in 93ec46a3a1be602c0e47d54abd8b0cb12ebb730f

Acked-by: Albert Rossi
Target: master, 2.7
Require-book: no
Require-notes: no