pnfsmanager: Add support for temporary upload paths

This patch adds operations to pnfs manager for creating, commiting and
cancelling temporary upload paths. Such paths can be used to write a new file
to dCache without exposing the existence of the file untilt the upload
completed. A temporary upload path is revocable. These features make it ideal
for generating unique write TURLs for use with SRM, although other use cases
exist (eg. xrootd POSC semantics could be achieved using a temporary upload
path).

How to construct a temporary upload path is a detail of the name space provider
implementation. ChimeraNameSpaceProvider uses a dedicated upload directory (the
default value being /upload) with mode 0711 to create a temporary directory
with a cryptographically strong UUID. Tags of the real upload directory are
copied to the temporary directory, thus ensuring that the file gets the right
storage class, access latency, retention policy, etc. While creating the upload
path, an optional expected file size, access latency, retention policy and
space token can be injected. These too are stored as tags in the temporary
upload directory.

Chimera is extended to allow a directory with a set of tags to be created
without copying the parent tags. Unfortunately the Chimera postgresql driver
copies the tags in a trigger, which makes that optimization ineffective. One
could argue that this functionality doesn't belong in a trigger and instead
should have been implemnted with a stored procedure for creating directories
with inherited tags - the driver could then have called the stored procedure
explicitly rather than relying on the trigger. This is left as a future
optimization.

The actual upload to the temporary path is a regular upload, with the exception
that the expected file size and real path are injeced into the CreateFile reply
from pnfs manager.

The upload directory is automatically created the first time this feature is
used. A site not using a protocl relying on unique upload paths will never see
the upload directory.

Since doors may export a different directory as their name space root, the
unqiue write paths are only useable if the upload directory is visible through
the doors. A non-default root path is a primary reason for changing the default
upload path. A typical value may be /pnfs/example.org/data/upload, with
example.org being the site domain name. This can be adjusted by setting
dcache.upload-directory.

Since doors only allow access to a user's root directory, doors have been
extended with an authorization exception allowing access to the upload
directory. Thus a non-default upload directory has to be configured on both
pnfs manager and all doors.

Ownership of the temporary upload directory mimics that of the file that is to
be created. The temporary upload path is thus writable to the same user that
requested the path to be created. One may imagine refining this in the future
to allow anonymous protocols to be used for writing to the upload path (eg to
create single use write URIs that one can use with protocols without
authorization, or that one can give to somebody else to upload a file to [a
form of delegation]).

The changes were originally part of the unique write TURL patch, but have been
split out for easier reviewing. The code in the patch is mostly unused at this
stage.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6690/
Auto indented file to clean up inconsistencies

Patch: none
srm: Generate unique write TURLs

The patch adds unique write TURLs for SRM. Unique TURLs are generated by
asking pnfs manager to create a temporary upload path. After upload, pnfs
manager commits the new file to its final name and location.

Since access latency, retention policy, size and space token are now stored in
the name space as part of the temporary upload path, there is no longer any
need to create a file reservation in space manager bound to the path.  Besides
simplifying the control flow, this solves the problem that the fields in
PutFileRequest that kept the state about whether we marked space as being used
were not persistent. This was a cause of the "Already have 1 record(s)" error
message.

Implicit space reservations as performed by the SRM are no longer needed and
srm.enable.space-reservation.implicit is obsolete. Instead space manager has to
be configured to allow uploads to link groups without prior reservations.  This
is achived using the spacemanager.enable.unreserved-uploads-to-linkgroups
setting.  The setting spacemanager.enable.reserve-space-for-non-srm-transfers
is deprecated. The property srm.enable.space-reservation.strict is obsolete
too.

Since there is a risk of lost notification from PnfsManager to SpaceManager,
SpaceManager has gained code to check whether name space entries of files in
the TRANSFERRING state still exists. Such checks should be done for all states,
however the risk is the biggest for files in TRANSFERRING.

The AbstractStorageElement interface is modified in several ways. The most
obvious change is that putDone and abortPut now call through to the storage
implementation. The TURL computation functions now take a fileId rather than
the SURL. The fileId is opaque to the SRM code, and the dCache storage
implementation now uses it to store a transfer path rather than a PNFS ID.

PnfsManager no longer injects default access latency and retention policy for
files in a directory with a WriteToken tag. Such values are injected by
SpaceManager. However explicit AccessLatency and RetentionPolicy tags are still
respected. SpaceManager will fail the upload in case of conflicts between those
tags.

Since the upload now hapens to a different path, I added billing notification
from SRM on upload. The billing entry from the transfer door contains the turl
path (I considered injecting the real path, but since the client could try
uploading several times and since the upload isn't persistent until putDone is
called, I think using the turl path is better). The SRM is assymetric here as
downloads are not reported to billing (they would just duplicate existing
information) - it is however easy to add if there is a good use case for it.

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6639/
spacemanager: Remove obsolete features

Due to the introduction of unique write TURLs, several space manager features
are no longer needed. This patch removes those featurs.

The removed features are all a result of no longer needing the ALLOCATED state
of a file reservation. Thus there is no need to bind file reservations to a
path, no need to track deleted paths, and no need to track lifetime of a file
reservation (the lifetime is now that of the PNFS ID). As a consequence, the
"Already have 1 record(s)" code path no longer exists.

The CLI has been updated to remove the options and output related to the above
features.

The Use and CancelUse messages are no longer supported.

Database schema changes: The path, expiration time, and deleted columns of
srmspacefile have been deleted, together with their indexes. The ALLOCATED
state of srmspacefile records no longer exists.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6691/
http: add minimal HEAD support in the mover

some clients do send HEAD after redirect

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no