nfsv41: simplify internal Transfer class

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
dcache-core (billing): modify changesets to do concurrent index creation

As suggested in support ticket 7947: slow start on create index in billing.

Note that two modifications were necessary:

1.  Because CREATE INDEX CONCURRENTLY is not supported by the Liquibase createIndex task, these were changed to raw <sql> calls.
2.  Because this command cannot run inside a transaction, each of these changeset is marked runInTransaction=false, with cognizance taken of the risks.

The old changesets were eliminated and replaced with new ones having a different id.

Testing:

Deployed new changesets.  Eliminated all indices from billing database and booted billing domain.

Here are the stats for the largest tables (the fine-grained data):

billing=> select count(*) from billinginfo;
  count
---------
 8070858
(1 row)

billing=> select count(*) from storageinfo;
  count
---------
 1862998
(1 row)

billing=> select count(*) from doorinfo;
  count
---------
 1902624
(1 row)

billing=> select count(*) from hitinfo;
 count
-------
 58028
(1 row)

11 Dec 2013 14:02:58 (billing) [] Continuing past: liquibase.precondition.core.PreconditionContainer@5303eb4 despite precondition error:
 ...
11 Dec 2013 14:24:36 (System) [] created : billing

(Total initialization time of about 22 minutes with 8M, 2M and 2M rows in the three main tables).

Target: master
Request: 2.7
Patch: http://rb.dcache.org/r/6343/diff/#index_header
Require-notes: yes
Require-book: no
Acked-by: Tigran
Bug: http://rt.dcache.org/Ticket/Display.html?id=7947

RELEASE NOTES:

Indices on the billing database are now created concurrently, so database (re)initialization should be sped up by a modest linear factor.
nfsv41: remove unused field in NFS4ProtocolInfo

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
poolmanager: Add named error codes for no-pool-configured and no-pool-online

Improves error messages when poolmanager is unable to find pools for a
transfer. In particular the 'no write pool configured' and 'no write pool
available' messages are simplified when sent to the FTP client, and the log
messages no longer use the legacy StorageInfo#toString method (instead only the
information that is actually involved in pool selection is logged).

Having two different error codes is important as one error is transient and the
other is not. Read pool selection currently uses some additional error codes
which should probably be changed to one of the two new values, but I leave that
for a future patch.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6358/
Upgrade to Spring Framework 4

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6364/
Upgrade to liquibase 3

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6365/
Upgrade to DataNucleus 3.3

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/6367/
pnfsmanager: Use ISO8601 compatible date for metadataof output

metadataof now includes the year in dates.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6379/
Fix PoolRemoveFilesMessage#toString

There is an erroneous comma at the beginning of the file list.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6378/
libs: update nfs4j to 0.7.0

fixes compatibility with rhel7 and linux kernels > 3.11
provides functionality required by proxy IO

Changelog for nfs4j-0.6.3..nfs4j-0.7.0
    * [ffcf8a9] vfs: update API interface to match new changes in chimera
    * [3252e04] nfsv3: fix creation of symbolic link
    * [f3a8db4] nfsv3: fix hard link creation (similar to http://rb.dcache.org/r
    * [a3642de] src: keep style check happy
    * [89346e7] nfsv4: remove dead code
    * [6e7cd2a] vfs: chimera: avoid extra db lookup
    * [cc22d5c] test: do not use obsolete ip host name
    * [5958f97] nfsv4: add state dispose listener
    * [1e946d2] core: implement missing SECINFO_NO_NAME
    * [03201b8] pom: update liquibase-core to 3.0.8

Acked-by: Gerd Behrmann
Target: master, 2.7, 2.6
Require-book: no
Require-notes: no
nfs4: implement proxy-io on read

Linux client too often falls back to IO through MDS (door).
This ends up with IO error on the client side as dCache does
not supports it.

This change set aims to add proxy IO by using internally DCAP
protocol. The door-pool communication can be changed later to
other protocol.

This is for master, nevertheless, 2.7 and 2.6 should have it as well.

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
spacemanager: Fix use of global property in spacemanager configuration

Spacemanager did not respect the spacemanager.enable.space-reservation
property.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6388/
Add preallocation field to PoolAcceptFileMessage

Adds a field for preallocation to the pool accept message - I imagine the pool
could make use of this information to preallocate space during mover startup or
reject a mover if not enough garbage collectible space is available up front.
In the short term this will be used by space manager for implicit space
reservation.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6386/
spacemanager: Refactor message dispatching code

Ensures that message forwarding and reply logic is all places in
messageToForward/messageArrived, all error injection logic in processMessage,
and all other methods are ignorant to how and when messages are sent.

All improves error handling a bit.

The layering achieved by this class makes subsequent refactoring of the code
simpler.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6389/
Upgrade dependencies

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6371/
spacemanager: Postpone DB update to intercepted pool accept message

A recent patch moved the DB update on transfer to the intercepted
pool selection reply. This patch moves those updates even further
back so that they now happen in the intercepted pool accept request,
which triggers the creation of a mover in the pool.

There are several benefits to this change:

- If pool selection took a long time and the door no longer exists
  (client gave up), then we now do not create any temporary reservations.
  Before a temporary reservation would be created, but since there
  would not have been a DoorTransferFinishedMessage or any other
  message to indicate a failure, the temporary reservation would not
  be cleaned until it expired.

- In case a component does a pool selection without starting a mover
  (eg older versions of pin manager or SRM would do this), doing the
  reservation update in the pool selection is clearly wrong.

- More messages of transfers without space reservation can now
  bypass queuing in space manager. This is because the LinkGroup entry
  in storage info is now passed to the pool and can be used as a
  simple test whehter additional processing in space manager is
  required.

- We are one step closer to being able do pool selection in doors: since
  space manager no longer updates its database on pool selection, we could
  load all information needed to select a linkgroup into memory, serialize
  it and transfer it to doors (just like we do with pool manager's
  PoolMonitor).

Temporary reservations created by space manager (when reservation
for non-srm transfers is enabled or a default space token is defined for
the target directory) are no longer bound to the path - the reservations
are created directly with a pnfs id and the path field is null (this
change is admin visible).  This has two benefits:

- There is no risk of duplicate paths. A failed upload will not block
  a subsequent retry even if the retry is performed before the delete
  notification reaches space manager.

- In case of failures, space manager can recognize which reservation it
  created itself (those without a path) and delete those rather than
  unpin the pnfs id (a reservation without a path would not be useable
  to any future transfer anyway as there is no way to bind that transfer
  to the reservation; the only point of having the path in space manager
  is so SRM can create entries before a PNFS ID has been generated).

One consequence of the patch is that space manager once again intercepts
the pool accept message. In return the pool selection reply is no
longer intercepted, thus this change should not affect latency.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6360/
spacemanager: Do not delete SRM managed reservation entries on file deletion

SRM creates file entries in space reservations prior to asking a client to
upload a file through some transfer protocol. It is valid for a client to retry
a failed upload without renegotiating the transfer with the SRM (our own srm
client does this). However when the name space entry is deleted after a failed
upload, the space manager marks the file entries as deleted and possibly
deletes the file entry from its tables (depends on whether a disk replica was
registered at that time). Thus when the client retries the upload, the upload
will fail or a temporary reservation will be created and used instead of the
one managed by the SRM.

This patch addresses this problem by not deleting entries/marking entries as
deleted in state RESERVED or TRANSFERRING unless they do not have a path
associated with them (entries without a path cannot be reused anyway). Instead
such entries are reset to RESERVED and the associated PNFS ID is removed.

The patch also cleans up some related code.

Target: trunk
Require-notes: yes
Require-book: no
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7799
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6361/
spacemanager: Mover DB schema management to Liquibase

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6368/
spacemanager: Drop 'fix missing size' command

The command was used to recover from a bug in dCache 1.9.1 and
is no longer needed.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litinvtsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6372/
spacemanager: Maintain aggregate columns with database triggers

Moves the logic to maintain aggregate columns in srmlinkgroup and srmspace from
Java code to database triggers. Adds not-null constraints to several space
manager database columns. This may take a little while to apply on large
databases.

The schema modifications are not compatible with older versions. Should a
downgrade be needed, the schema modifications have to be rolled back using
'dcache database rollback <tag> SrmSpaceManager@<domain>' or 'dcache database
rollback <date> SrmSpaceManager@<domain>'. The former requires that a tag was
created before upgrade (this is recommended). Should the system be downgraded
without rolling back the schema changes, the space accounting will be
incorrect.

Target: master
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6370/