srm-client: Do not retry on permanent FTP failures

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6705/
Allow retry on transient transfer failures

This patch fixes a minor regression introduced in 2.8 that prevents retry
on a transient error. The patch also removes retry on a permanent error.

Target: trunk
Request: 2.8
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6707/
ftp: Make root path behaviour configurable

Our FTP door is different from all other doors in dCache by
exposing a user-specific root path. This patch introduces
the ftp.root property. It mimics that of webdav.root and
xrootd.root by defining which directory should form the root
of the door. If set to the empty string (which is the default
value), the existing behaviour is maintained.

As before, the user's home directory is used to define the
initial working directory of a new FTP session.

Also fixes a bug that could prevent the user's settings from
being applied if logging in multiple times in the same FTP
session.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6686/
srm: Make FTP TURLs absolute

RFC 1738 defines that FTP URLs are interpreted relative to
the initial working directory of the FTP session, which may
be different from the root. In dCache the user's home directory
is the initial working directory. The TURL generated by the
SRM thus doesn't refer to the correct directory if dCache has
been configured with home directories.

The spec states that an absolute path should be encoded with
starting as %2F. That's because the spec requires the client
to split the path at / and use each segment in a CWD command
(plain FTP doesn't have a concept of paths; path names are
an extension defined in RFC 3659). Ie gsiftp://localhost//foo/bar
just means to do

CWD
CWD foo

and then STOR or RETR bar. gsiftp://localhost/%2Ffoo/bar means

CD /foo

and then STOR or RETR bar.

However our own srmcp client doesn't deal with the URI encoding
and just sends %2F as part of the directory name.

Globus Toolkit (globus-url-copy) doesn't follow the spec either,
since it prefixes the path in gsiftp://localhost/foo/bar with an
extra slash, thus with Globus it isn't possible to refer to
something relative to the user's home directory.

Both srmcp and globus-url-copy do not interpret the empty
path element according to the spec since they just pass the
entire path on to the server. Ie. for gsiftp://localhost//foo/bar,
srmcp sends /foo to the server, while globus-url-copy sends
//foo. At the FTP level the double slash is undefined, but our
FTP door collapses it into a single slash.

The conclusion from the above is that we cannot use the spec
compliant encoding as it would break compatibility with our
own client. We can prefix the path with an extra slash, which
works with both Globus and srmcp. It will not work with a
spec compliant client, however the current code doesn't work
with a spec complaint client anyway, so the situation doesn't
become worse.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6685/
pnfsmanager: Add support for temporary upload paths

This patch adds operations to pnfs manager for creating, commiting and
cancelling temporary upload paths. Such paths can be used to write a new file
to dCache without exposing the existence of the file untilt the upload
completed. A temporary upload path is revocable. These features make it ideal
for generating unique write TURLs for use with SRM, although other use cases
exist (eg. xrootd POSC semantics could be achieved using a temporary upload
path).

How to construct a temporary upload path is a detail of the name space provider
implementation. ChimeraNameSpaceProvider uses a dedicated upload directory (the
default value being /upload) with mode 0711 to create a temporary directory
with a cryptographically strong UUID. Tags of the real upload directory are
copied to the temporary directory, thus ensuring that the file gets the right
storage class, access latency, retention policy, etc. While creating the upload
path, an optional expected file size, access latency, retention policy and
space token can be injected. These too are stored as tags in the temporary
upload directory.

Chimera is extended to allow a directory with a set of tags to be created
without copying the parent tags. Unfortunately the Chimera postgresql driver
copies the tags in a trigger, which makes that optimization ineffective. One
could argue that this functionality doesn't belong in a trigger and instead
should have been implemnted with a stored procedure for creating directories
with inherited tags - the driver could then have called the stored procedure
explicitly rather than relying on the trigger. This is left as a future
optimization.

The actual upload to the temporary path is a regular upload, with the exception
that the expected file size and real path are injeced into the CreateFile reply
from pnfs manager.

The upload directory is automatically created the first time this feature is
used. A site not using a protocl relying on unique upload paths will never see
the upload directory.

Since doors may export a different directory as their name space root, the
unqiue write paths are only useable if the upload directory is visible through
the doors. A non-default root path is a primary reason for changing the default
upload path. A typical value may be /pnfs/example.org/data/upload, with
example.org being the site domain name. This can be adjusted by setting
dcache.upload-directory.

Since doors only allow access to a user's root directory, doors have been
extended with an authorization exception allowing access to the upload
directory. Thus a non-default upload directory has to be configured on both
pnfs manager and all doors.

Ownership of the temporary upload directory mimics that of the file that is to
be created. The temporary upload path is thus writable to the same user that
requested the path to be created. One may imagine refining this in the future
to allow anonymous protocols to be used for writing to the upload path (eg to
create single use write URIs that one can use with protocols without
authorization, or that one can give to somebody else to upload a file to [a
form of delegation]).

The changes were originally part of the unique write TURL patch, but have been
split out for easier reviewing. The code in the patch is mostly unused at this
stage.

Target: trunk
Require-notes: yes
Require-book: yes
srm: Generate unique write TURLs

The patch adds unique write TURLs for SRM. Unique TURLs are generated by
asking pnfs manager to create a temporary upload path. After upload, pnfs
manager commits the new file to its final name and location.

Since access latency, retention policy, size and space token are now stored in
the name space as part of the temporary upload path, there is no longer any
need to create a file reservation in space manager bound to the path.  Besides
simplifying the control flow, this solves the problem that the fields in
PutFileRequest that kept the state about whether we marked space as being used
were not persistent. This was a cause of the "Already have 1 record(s)" error
message.

Implicit space reservations as performed by the SRM are no longer needed and
srm.enable.space-reservation.implicit is obsolete. Instead space manager has to
be configured to allow uploads to link groups without prior reservations.  This
is achived using the spacemanager.enable.unreserved-uploads-to-linkgroups
setting.  The setting spacemanager.enable.reserve-space-for-non-srm-transfers
is deprecated. The property srm.enable.space-reservation.strict is obsolete
too.

Since there is a risk of lost notification from PnfsManager to SpaceManager,
SpaceManager has gained code to check whether name space entries of files in
the TRANSFERRING state still exists. Such checks should be done for all states,
however the risk is the biggest for files in TRANSFERRING.

The AbstractStorageElement interface is modified in several ways. The most
obvious change is that putDone and abortPut now call through to the storage
implementation. The TURL computation functions now take a fileId rather than
the SURL. The fileId is opaque to the SRM code, and the dCache storage
implementation now uses it to store a transfer path rather than a PNFS ID.

PnfsManager no longer injects default access latency and retention policy for
files in a directory with a WriteToken tag. Such values are injected by
SpaceManager. However explicit AccessLatency and RetentionPolicy tags are still
respected. SpaceManager will fail the upload in case of conflicts between those
tags.

Since the upload now hapens to a different path, I added billing notification
from SRM on upload. The billing entry from the transfer door contains the turl
path (I considered injecting the real path, but since the client could try
uploading several times and since the upload isn't persistent until putDone is
called, I think using the turl path is better). The SRM is assymetric here as
downloads are not reported to billing (they would just duplicate existing
information) - it is however easy to add if there is a good use case for it.

Target: trunk
Require-book: yes
Require-notes: yes
spacemanager: Remove obsolete features

Due to the introduction of unique write TURLs, several space manager features
are no longer needed. This patch removes those featurs.

The removed features are all a result of no longer needing the ALLOCATED state
of a file reservation. Thus there is no need to bind file reservations to a
path, no need to track deleted paths, and no need to track lifetime of a file
reservation (the lifetime is now that of the PNFS ID). As a consequence, the
"Already have 1 record(s)" code path no longer exists.

The CLI has been updated to remove the options and output related to the above
features.

The Use and CancelUse messages are no longer supported.

Database schema changes: The path, expiration time, and deleted columns of
srmspacefile have been deleted, together with their indexes. The ALLOCATED
state of srmspacefile records no longer exists.

Target: trunk
Require-notes: yes
Require-book: yes
srm: Replace pin and unpin callbacks with ListenableFuture

This replaces some of the bulky callback interfaces in the SRM with
Guava's ListenableFuture. The same approach was used for the recently
introduced prepareToPut method.

Target: trunk
Require-notes: no
Require-book: no