webdav: support redirecting to next replica for NOT FOUND or FORBIDDEN responses

To support the concept of federation, clients must survive catalogues or other
starting points containing incomplete or out-of-date information.  In particular,
a starting point could redirect a client requesting some data to a storage system
that no longer contains that data or that applies additional constraints on which
users may read that file.

A normal WebDAV implementation would return an error to the client, so failing
the request.  This is undesirable as, no matter how active the starting point is
at maintaining its state, there is always the possibility for the information to
become stale.

The solution adopted by CERN is to encode a stack of replicas in the query-string
part of the request.  With WebDAV, this is not used for anything, so can carry
this extra information.

This patch adds support for redirecting the client to the next replica, if there
is an error and an additional replica is known.

Target: master
Request: 2.7
Request: 2.6
Patch: http://rb.dcache.org/r/6359
Acked-by: Gerd Behrmann
spacemanager: Fix capitalization bug in liquibase changeset

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6436/
spacemanager: Fix syntax error in SQL expression

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvtintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6437/
spacemanager: Propagate SpaceException error to SRM

SRM relies on receiving the exception object in failure replies.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6439/
chimera: Add more documentation and error checking to chimera shell

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6444/
chimera: Fix help output of chimera CLI

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6440/
spacemanager: Rewrite command line interface

The patch more or less rewrites all space manager related admin commands.

The following commands are removed:

ls
ls file space tokens
listInvalidSpaces
listFilesInSpace
removeFilesFromSpace
remove file
reserve
release
update space reservation

The following commands are added:

ls spaces
ls files
reserve space
update space
release space
release file

The new commands should offer greater flexibility and more concise output.
Documentation has been added for all commands. Options for almost all
commands have changed.

Note that although commands have been rewritten to make use of the
annotation syntax, the point of this patch is the changed user experience,
not the change of implementation.

Internal database keys like the link group id and the file reservation id
are no longer exposed through the user interface. Group (which confusingly
could be both group and user name) and role fields are presented as a
unified owner, which may be a user name or an FQAN. Various list commands
try to hide irrellevant information, dynamically adding columns to the output
suitable for the requested information.

The code was refactored such that the CLI appears in its own class. The link
group import code has been moved to its own classs too. Commands with blocking
code (ie database operations) are now executed on a thread pool to prevent
blocking the message thread.

The RESERVED state of files has been renamed to ALLOCATED to better reflect
how it is used and to be consistent with the terminology of the aggregation
field in space reservations.

The database interface has been extended with a criterion API to allow
flexible querying.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvtintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6415/
spacemanager: Refresh pool monitor on 'update link groups'

The patch adds a workaround for triggering a pool monitor updated
when update link groups is called.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6438/
spacemanager: Fix expiration output on 'ls files' command

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6445/
spacemanager: Delete flushed file reservations when name space entry is deteled

Space manager marks file reservations as deleted when the name space entry
is deleted. The actual file reservation is not removed until space manager
is notified that the file is removed from the pool. For files flushed to
tape there may not be a disk copy and thus such a notification is never sent.
The consequence is that reservations for flushed files are never removed.

This patch fixes the problem by deleting the file reservation of flushed files
as soon as the name space entry is deleted. A liquibase changeset is added to
delete old reservations for deleted flushed files.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6435/
chimera: tigger action with FsInode_PSET#setStat()

requires corresponding nfs changes

Ticket: #8201
Acked-by: Albert Rossi
Acked-by: Gerd Behrmann
Target: master, 2.6, 2.7
Require-book: no
Require-notes: no
pool: Fixing rep ls output, RT 8203

Output of rep ls <pnfsID that exists> <pnfsID that does not exist> <another pnfsID that exists> would show:
<pnfsID that exists> <C-------X--L(0)[0]> 2255071 si={test:disk}
CacheException(rc=10007;msg=Entry not in repository : <pnfsID that does not exist would show>)
<another pnfsID that exists> <C-------X--L(0)[0]> 2255071 si={test:disk}

Now it show this:

[wla000617wpa.desy.de] (pool_write) admin > rep ls 0000B27FEE232A7840A7ADCBEBEE57886730 00000DC04577984D4FCD9610F44B19CF3BD7 00000DC04577984D4FCD9610F44B19CF3B23 00000DC04577984D4FCD9610F44B19CF3B45
0000B27FEE232A7840A7ADCBEBEE57886730 <C-------X--L(0)[0]> 2255071 si={test:disk}
00000DC04577984D4FCD9610F44B19CF3BD7 <C-------X--L(0)[0]> 10002434 si={test:disk}
CacheException(rc=10007;msg=Entry not in repository : 00000DC04577984D4FCD9610F44B19CF3B23)
CacheException(rc=10007;msg=Entry not in repository : 00000DC04577984D4FCD9610F44B19CF3B45)

Ticket: 8203
Acked-by: Gerd
Target: trunk
Require-book: no
Require-notes: yes
nfs4: fix orphan proxy movers

Check client state before starting a proxy mover.
This prevents orphan movers when client recovery failed
or stateid provided by client is not valid.

Acked-by: Gerd Behrmann
Target: master, 2.7
Require-book: no
Require-notes: no
Drop chimera-cli script

Has been replaced by chimera shell.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6449/
spacemanager: Add missing update of deleted flag

Also refactored File class to use a boolean for this flag.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6451/
spacemanager: Don't accept output retention policy

Since we don't implement output in dCache, there is no
point in allowing such reservations to be created.

Target: trunk
Require-notes: no
Require-book: no
Acjed-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6453/
spacemanager: Fix help output of reserve space

@Argument help is currently not rendered as part of the help
output. This patch moves the mentioning of the optional suffix
to the main description.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6454/
spacemanager: Minor UI improvements

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6455/
pool: Follow up cd0c482af5338249c064f0cbe889bdc2c1790c0f - Make rep ls output more efficient

This is a follow up patch to https://github.com/dCache/dcache/commit/cd0c482af5338249c064f0cbe889bdc2c1790c0f
The initial commit message did not say that commit cd0c482af5338249c064f0cbe889bdc2c1790c0f actually prevents
rep ls from getting interrupted by a FileNotFoundException.

This patch just improves the data structure and resourcces needs.

Changed some internal data structure to reduce resource needs.

Ticket: 8203
Acked-by: Gerd (verbally)
Target: master, 2.6, 2.7
Require-book: no
Require-notes: no
Patch: http://rb.dcache.org/r/<id>/
backoff-controller: update unit-tests, refactor code

The current unit-tests rely on the code requesting a delay of x
milliseconds resulting in an actual delay of, at most, (x+100)ms.

Operating systems make no such guarantee and we have had a few cases
where dCache builds have failed because of this.

This patch refactors backoff-controller and the unit-tests to remove this
assumption.  The backoff-controller was simplified.  The superfluous builder
was removed.  The unit-tests updated to remove dependency on exponential-
backoff algorithm, add assertions of the overall response and cover more
possible behaviours.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/6463/