http: add minimal HEAD support in the mover

some clients do send HEAD after redirect

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
srm: Replace pin and unpin callbacks with ListenableFuture

This replaces some of the bulky callback interfaces in the SRM with
Guava's ListenableFuture. The same approach was used for the recently
introduced prepareToPut method.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6708/
Rename one-way CellStuf#send to notify

This both better describes what is happening (no reply expected),
and it avoids naming conflicts with a future patch I am working on.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6727/
Refactor CellStub

The history of CellStub: In the old days we had the very same piece of code
repeated throughout dCache. Whenever we wanted to send a message for an RPC
style interaction, we would build CellMessage, send and wait for a reply, check
the reply for null (a timeout), for having the proper type, and rethrow any
error if the reply indicated an error.  CellStub was born to encapsulate this
repeating pattern.

CellStub also served as a way to encapsulate communication behaviour for a
particular destination. Rather than having a cell address, a timeout, and
possibly other parameters for every other service a component may need to talk
to, CellStub encapsulates all this and can be injected as a single reference.

CellStub isolated the dCache code from the low level details of the cell
communication. The idea was to simplify unit testing and to ease a migration to
JMS.

Later CellStub gained support for asynchronous callback based messaging,
exposing a higher level callback interface with processing aware of the
semantics of the Message class.

Lately the class has gained support for rate limiting the traffic it generates.

I feel CellStub has grown a little too much with to many responsibilities.  At
the same time it has several limitations and at a few places in dCache we have
code to work around these limitations.

This patch decouples the part of CellStub that deals with encapsulating
destination parameters and cell communication from the higher level semantics
of the Message class. In this first patch only the asynchronous message
processing is touched.

Rather than MessageCallback, CellStub now returns a Guava ListenableFuture.
The future provides the reply as is, thus there is no translation of failures
encoded in Message to exceptions. Cell exceptions, timeouts, or exception
replies are considered failures and the future will rethrow these as
exceptions.

Modelling the structure of Futures#addCallback, CellStub#addCallback adds
support for the old MessageCallback interface. Like before, it will interpret
errors encoded in the Message class and report those as failures.
MessageCallback is now limited to Message payload. Other payload has to use
Futures#addCallback instead.

Although CellStub still has all the same features as before, part of the Message
processing has been separated from the message passing code. ListenableFuture
also has the benefit that the callback can happen on another executor. The patch
doesn't update dCache code to make use of these features. That is left to
future patches.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6730/
Eliminate some direct use of AbstractCellComponent#send

Use CellStub where appropriate.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6741/
Add transparent retry support to async use of CellStub

Synchronous message passing has support an option to transparently
resend messages on NoRouteToCellException. This is to be used for
destinations that are known to be required in a dCache installation
(like PnfsManager) when no sensible alternative action exists.

Support has been limited to the synchronous sendAndWait methods. This
patch adds support for the callback based sendMessage calls. On failure
to deliver a message, the message is not resend right away to avoid
a tight retry loop. Instead the message is send while performing
other cell background tasks like processing cell message timeouts.

The change is made in anticipation of a future patch that refactors
the sendAndWait calls to be build upon the async infrastructure of
CellStub.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6739/
Delete obsolete code

The code calls into CellNucleus#sendAndWait and thus makes further
refactoring more difficult. Since the code is unused, this patch
removes it.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6747/
Layer synchronous CellStub use on top of async code

Rather than duplicating the same logic for synchronous and asynchronous
use cases, this patch changes CellStub to use the exact same asyncronous
behaviour in both cases. CellStub#sendAndWait is now a simple wrapper
around an asynchronous send with a blocking get on the Future.

Part of the superfluous synchronous code has been removed, simplifying
CellEndpoint and CellAdapter.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6740/
Remove direct calls of CellEndpoint#sendAndWait outside cells module

CellStub is used instead, which in many places leads to more compact code.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6742/
Avoid direct use of CellAdapter#sendMessage

Also removes unused message passing methods in CellAdapter.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6747/
Get rid of more uses of CellNucleus#send methods

Depending on the situation the code is either deleted, ported to CellStub,
or ported to CellEndpoint.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6757/
srm: fix race condition if stop is called too soon after start

The WorkSupplyService fetches information that is protected by
the Scheduler monitor.  If the stop method is called while the
WorkSupplyService is not resting then the thread calling stop
enters the Scheduler monitor and waits for the WorkSupplyService
service to end, resulting in a deadlock.

This patch fixes this by moving the WorkSupplyService activation
and deactivation so it happens outside the Scheduler monitor.

Target: master
Acked-by: Dmitry Litvintsev
Patch: http://rb.dcache.org/r/6750/
Request: 2.8
libraries: update bouncy-castle

Update our hard-coded bouncy-castle to v1.45

The opensaml jar dependency is built against bouncy-castle v1.45, as
is jglobus and our current voms library.

Patch: http://rb.dcache.org/r/6728/
Acked-by: Tigran Mkrtchyan
Acked-by: Gerd Behrmann
Acked-by: Albert Rossi
delegation: add support for GridSite

This patch adds support for GridSite delegation v2.0.0.  The endpoint is
run as part of the SRM service and has the path /srm/delegation and
allows a user to delegate a credential, discover the remaining lifetime
of any delegated credential and destroy delegated credentials.  Those
credentials delegated via SRM and httpg have the delegated ID of 'gsi'.
Using this, a client may delete credentials delegated via SRM
operations.

Also included is a delegation client that allows scripted and
console-based interaction with GridSite endpoints.

Target: master
Requires-notes: true
Requires-book: true
Patch: http://rb.dcache.org/r/6628
Acked-by: Gerd Behrmann
srm-client: fix RPM packaging

Previous commit introduced an additional file without updating the
RPM packaging.  This patch fixes this oversight.

Target: master
Patch: http://rb.dcache.org/r/6764/
Acked-by: Tigran Mkrtchyan
srm: refactor SchedulerFactory

This patch modifies SchedulerFactory so it has a single concern:
creating Scheduler instances.  The current implementation of
SchedulerFactory has two other concerns: creating all necessary
schedulers and maintaining a collection of schedulers. These
additional activities are moved elsewhere.

Creating all necessary schedulers has moved to the spring XML file.

Most of the behaviour associated with multiple schedules has moved
into the new SchedulerContainer class and the process of submitting a
Job to a scheduler is achieved via this new class.

One design goal is that SchedulerContainer should not 'leak' Scheduler
instances.  The getScheduler-like methods has been removed: both from
SchedulerContainer/SchedulerFactory and from SRM class.  Activity that
iterates over all the Schedulers has moved to SchedulerContainer.
Activity that targets a specific scheduler is implemented with a
method that accepts a Class<? extends Job> argument to target a
specific Scheduler.

Note that the current code has the Job classes being responsible for
adding themselves to a scheduler.  To try to keep the patch short,
this mode of operation is maintained; however, Job is supplied with a
Scheduler rather than discovering the scheduler itself.

The handler classes must be able to add a Job to a scheduler.  The
patch achieves this via the SRM object passed to the handler's
constructor.  Future patch may refactor this so the object to which a
Job is submitted is supplied on the constructor.

Another future direction is to make Scheduler an interface that the
current Scheduler class implements.  The SchedulerContainer would also
implement this interface and the similarity between SchedulerContainer
and Scheduler becomes clearer.

Another beneficial side-effect from this patch is that
SchedulerFactory is no longer a singleton.

Target: master
Requires-book: no
Requires-notes: no
Patch: http://rb.dcache.org/r/6321/
Acked-by: Gerd Behrmann
srm: remove unused state

A job is never assigned the state RESTORED (perhaps as the state doesn't make
sense).  Because of this, the code that depends on a Job being in state RESTORED
is never called.

This patch removes this dead-wood

Target: master
Patch: http://rb.dcache.org/r/6650
Acked-by: Gerd Behrmann
Acked-by: Dmitry Litvintsev
logback: correct MDC dependency in appender

MDC dependency should be to slf4j, not log4j.
The error has gone unnoticed because we do use
log4j-over-slf4j, but the import should not
be tightly coupled to it.

Target: master
Patch: http://rb.dcache.org/r/6777/
Require-notes: no
Require-book: no
Acked-by: Paul
remote logging: add empty string defense to remote server wrapper

See patch 6719 and logback bug report: http://jira.qos.ch/browse/LOGBACK-959

As a workaround which allows us to update to logback 1.1.1, I have added
defensive code to the String setters which converts empty strings to nulls,
which are properly handled.

I have also changed returns to System.exit, since we require this component
to run in its own domain, and have logged exceptions which may affect the
logging system itself using System.err.

Testing: Before changes, the logback 1.1.1 libraries throw the NullPointer exception;
with the changes, the server boots correctly and an alarm can be sent.

Target: master
Patch: http://rb.dcache.org/r/6775/
Require-notes: no
Require-book: no
Acked-by: Gerd
chimera: allow updating the creation time

A recent patch introduced the possibility to update the creation time
via FTP.  Unfortunately, the creation time wasn't actually updated.
This patch fixes this.

Target: master
Patch: http://rb.dcache.org/r/6790/
Acked-by: Tigran Mkrtchyan