nfs: return NFSERR_IO is we detect broken file on tape

currently we return NFSERR_LAYOUTTRYLATER and client
tries later. Stop infinite loop as we cant get file
anyway.

Acked-by: Gerd Behrmann
Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
http: adjust unit test to multirage changes

Acked-by: Paul Millar
Target: master, 2.8, 2.7, 2.6
Require-book: no
Require-notes: no
pnfsmanager: Add support for temporary upload paths

This patch adds operations to pnfs manager for creating, commiting and
cancelling temporary upload paths. Such paths can be used to write a new file
to dCache without exposing the existence of the file untilt the upload
completed. A temporary upload path is revocable. These features make it ideal
for generating unique write TURLs for use with SRM, although other use cases
exist (eg. xrootd POSC semantics could be achieved using a temporary upload
path).

How to construct a temporary upload path is a detail of the name space provider
implementation. ChimeraNameSpaceProvider uses a dedicated upload directory (the
default value being /upload) with mode 0711 to create a temporary directory
with a cryptographically strong UUID. Tags of the real upload directory are
copied to the temporary directory, thus ensuring that the file gets the right
storage class, access latency, retention policy, etc. While creating the upload
path, an optional expected file size, access latency, retention policy and
space token can be injected. These too are stored as tags in the temporary
upload directory.

Chimera is extended to allow a directory with a set of tags to be created
without copying the parent tags. Unfortunately the Chimera postgresql driver
copies the tags in a trigger, which makes that optimization ineffective. One
could argue that this functionality doesn't belong in a trigger and instead
should have been implemnted with a stored procedure for creating directories
with inherited tags - the driver could then have called the stored procedure
explicitly rather than relying on the trigger. This is left as a future
optimization.

The actual upload to the temporary path is a regular upload, with the exception
that the expected file size and real path are injeced into the CreateFile reply
from pnfs manager.

The upload directory is automatically created the first time this feature is
used. A site not using a protocl relying on unique upload paths will never see
the upload directory.

Since doors may export a different directory as their name space root, the
unqiue write paths are only useable if the upload directory is visible through
the doors. A non-default root path is a primary reason for changing the default
upload path. A typical value may be /pnfs/example.org/data/upload, with
example.org being the site domain name. This can be adjusted by setting
dcache.upload-directory.

Since doors only allow access to a user's root directory, doors have been
extended with an authorization exception allowing access to the upload
directory. Thus a non-default upload directory has to be configured on both
pnfs manager and all doors.

Ownership of the temporary upload directory mimics that of the file that is to
be created. The temporary upload path is thus writable to the same user that
requested the path to be created. One may imagine refining this in the future
to allow anonymous protocols to be used for writing to the upload path (eg to
create single use write URIs that one can use with protocols without
authorization, or that one can give to somebody else to upload a file to [a
form of delegation]).

The changes were originally part of the unique write TURL patch, but have been
split out for easier reviewing. The code in the patch is mostly unused at this
stage.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6690/
Auto indented file to clean up inconsistencies

Patch: none
srm: Generate unique write TURLs

The patch adds unique write TURLs for SRM. Unique TURLs are generated by
asking pnfs manager to create a temporary upload path. After upload, pnfs
manager commits the new file to its final name and location.

Since access latency, retention policy, size and space token are now stored in
the name space as part of the temporary upload path, there is no longer any
need to create a file reservation in space manager bound to the path.  Besides
simplifying the control flow, this solves the problem that the fields in
PutFileRequest that kept the state about whether we marked space as being used
were not persistent. This was a cause of the "Already have 1 record(s)" error
message.

Implicit space reservations as performed by the SRM are no longer needed and
srm.enable.space-reservation.implicit is obsolete. Instead space manager has to
be configured to allow uploads to link groups without prior reservations.  This
is achived using the spacemanager.enable.unreserved-uploads-to-linkgroups
setting.  The setting spacemanager.enable.reserve-space-for-non-srm-transfers
is deprecated. The property srm.enable.space-reservation.strict is obsolete
too.

Since there is a risk of lost notification from PnfsManager to SpaceManager,
SpaceManager has gained code to check whether name space entries of files in
the TRANSFERRING state still exists. Such checks should be done for all states,
however the risk is the biggest for files in TRANSFERRING.

The AbstractStorageElement interface is modified in several ways. The most
obvious change is that putDone and abortPut now call through to the storage
implementation. The TURL computation functions now take a fileId rather than
the SURL. The fileId is opaque to the SRM code, and the dCache storage
implementation now uses it to store a transfer path rather than a PNFS ID.

PnfsManager no longer injects default access latency and retention policy for
files in a directory with a WriteToken tag. Such values are injected by
SpaceManager. However explicit AccessLatency and RetentionPolicy tags are still
respected. SpaceManager will fail the upload in case of conflicts between those
tags.

Since the upload now hapens to a different path, I added billing notification
from SRM on upload. The billing entry from the transfer door contains the turl
path (I considered injecting the real path, but since the client could try
uploading several times and since the upload isn't persistent until putDone is
called, I think using the turl path is better). The SRM is assymetric here as
downloads are not reported to billing (they would just duplicate existing
information) - it is however easy to add if there is a good use case for it.

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6639/
spacemanager: Remove obsolete features

Due to the introduction of unique write TURLs, several space manager features
are no longer needed. This patch removes those featurs.

The removed features are all a result of no longer needing the ALLOCATED state
of a file reservation. Thus there is no need to bind file reservations to a
path, no need to track deleted paths, and no need to track lifetime of a file
reservation (the lifetime is now that of the PNFS ID). As a consequence, the
"Already have 1 record(s)" code path no longer exists.

The CLI has been updated to remove the options and output related to the above
features.

The Use and CancelUse messages are no longer supported.

Database schema changes: The path, expiration time, and deleted columns of
srmspacefile have been deleted, together with their indexes. The ALLOCATED
state of srmspacefile records no longer exists.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6691/
http: add minimal HEAD support in the mover

some clients do send HEAD after redirect

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
srm: Replace pin and unpin callbacks with ListenableFuture

This replaces some of the bulky callback interfaces in the SRM with
Guava's ListenableFuture. The same approach was used for the recently
introduced prepareToPut method.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6708/
Rename one-way CellStuf#send to notify

This both better describes what is happening (no reply expected),
and it avoids naming conflicts with a future patch I am working on.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6727/
Refactor CellStub

The history of CellStub: In the old days we had the very same piece of code
repeated throughout dCache. Whenever we wanted to send a message for an RPC
style interaction, we would build CellMessage, send and wait for a reply, check
the reply for null (a timeout), for having the proper type, and rethrow any
error if the reply indicated an error.  CellStub was born to encapsulate this
repeating pattern.

CellStub also served as a way to encapsulate communication behaviour for a
particular destination. Rather than having a cell address, a timeout, and
possibly other parameters for every other service a component may need to talk
to, CellStub encapsulates all this and can be injected as a single reference.

CellStub isolated the dCache code from the low level details of the cell
communication. The idea was to simplify unit testing and to ease a migration to
JMS.

Later CellStub gained support for asynchronous callback based messaging,
exposing a higher level callback interface with processing aware of the
semantics of the Message class.

Lately the class has gained support for rate limiting the traffic it generates.

I feel CellStub has grown a little too much with to many responsibilities.  At
the same time it has several limitations and at a few places in dCache we have
code to work around these limitations.

This patch decouples the part of CellStub that deals with encapsulating
destination parameters and cell communication from the higher level semantics
of the Message class. In this first patch only the asynchronous message
processing is touched.

Rather than MessageCallback, CellStub now returns a Guava ListenableFuture.
The future provides the reply as is, thus there is no translation of failures
encoded in Message to exceptions. Cell exceptions, timeouts, or exception
replies are considered failures and the future will rethrow these as
exceptions.

Modelling the structure of Futures#addCallback, CellStub#addCallback adds
support for the old MessageCallback interface. Like before, it will interpret
errors encoded in the Message class and report those as failures.
MessageCallback is now limited to Message payload. Other payload has to use
Futures#addCallback instead.

Although CellStub still has all the same features as before, part of the Message
processing has been separated from the message passing code. ListenableFuture
also has the benefit that the callback can happen on another executor. The patch
doesn't update dCache code to make use of these features. That is left to
future patches.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6730/
Eliminate some direct use of AbstractCellComponent#send

Use CellStub where appropriate.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6741/
Add transparent retry support to async use of CellStub

Synchronous message passing has support an option to transparently
resend messages on NoRouteToCellException. This is to be used for
destinations that are known to be required in a dCache installation
(like PnfsManager) when no sensible alternative action exists.

Support has been limited to the synchronous sendAndWait methods. This
patch adds support for the callback based sendMessage calls. On failure
to deliver a message, the message is not resend right away to avoid
a tight retry loop. Instead the message is send while performing
other cell background tasks like processing cell message timeouts.

The change is made in anticipation of a future patch that refactors
the sendAndWait calls to be build upon the async infrastructure of
CellStub.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6739/
Delete obsolete code

The code calls into CellNucleus#sendAndWait and thus makes further
refactoring more difficult. Since the code is unused, this patch
removes it.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6747/
Layer synchronous CellStub use on top of async code

Rather than duplicating the same logic for synchronous and asynchronous
use cases, this patch changes CellStub to use the exact same asyncronous
behaviour in both cases. CellStub#sendAndWait is now a simple wrapper
around an asynchronous send with a blocking get on the Future.

Part of the superfluous synchronous code has been removed, simplifying
CellEndpoint and CellAdapter.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6740/
Remove direct calls of CellEndpoint#sendAndWait outside cells module

CellStub is used instead, which in many places leads to more compact code.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6742/
Avoid direct use of CellAdapter#sendMessage

Also removes unused message passing methods in CellAdapter.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6747/
Get rid of more uses of CellNucleus#send methods

Depending on the situation the code is either deleted, ported to CellStub,
or ported to CellEndpoint.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6757/
srm: fix race condition if stop is called too soon after start

The WorkSupplyService fetches information that is protected by
the Scheduler monitor.  If the stop method is called while the
WorkSupplyService is not resting then the thread calling stop
enters the Scheduler monitor and waits for the WorkSupplyService
service to end, resulting in a deadlock.

This patch fixes this by moving the WorkSupplyService activation
and deactivation so it happens outside the Scheduler monitor.

Target: master
Acked-by: Dmitry Litvintsev
Patch: http://rb.dcache.org/r/6750/
Request: 2.8
libraries: update bouncy-castle

Update our hard-coded bouncy-castle to v1.45

The opensaml jar dependency is built against bouncy-castle v1.45, as
is jglobus and our current voms library.

Patch: http://rb.dcache.org/r/6728/
Acked-by: Tigran Mkrtchyan
Acked-by: Gerd Behrmann
Acked-by: Albert Rossi
delegation: add support for GridSite

This patch adds support for GridSite delegation v2.0.0.  The endpoint is
run as part of the SRM service and has the path /srm/delegation and
allows a user to delegate a credential, discover the remaining lifetime
of any delegated credential and destroy delegated credentials.  Those
credentials delegated via SRM and httpg have the delegated ID of 'gsi'.
Using this, a client may delete credentials delegated via SRM
operations.

Also included is a delegation client that allows scripted and
console-based interaction with GridSite endpoints.

Target: master
Requires-notes: true
Requires-book: true
Patch: http://rb.dcache.org/r/6628
Acked-by: Gerd Behrmann