systemtest: regenerate host credentials, add user credentials

Patch adds new host credentials so the system-test continues to work
out-of-the-box.  The patch also adds user-certificates generated by
the same Disposable CA.  This should allow people without an X.509
certificate to use system-test without first regenerating credentials.

Target: master
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
webdav: work-around race-condition in StringTemplate library

StringTemplate has a lazy-load for parsing the template file.  It
turns out that this is very racy, with at least two race-conditions.

Once race-condition has the effect that if two threads attempt to
fetch a template at about the same time with the template file not
loaded then the template being requested by the second thread (the one
that lost the race, so not loading the template file) will be never be
available.

Since dCache uses STGroupFile object to render only one template, this
means that if two threads attempt to fetch the template close enough
together then the template is never available and null is returned.
Because of another problem, this triggers a NPE.

A symptom of this problem is that dCache (> 2.6) will log a single
entry

   "redefinition of template <name>"

where <name> is the template that is no longer available.

The work-around is to force the loading and parsing the template file
by the thread creating the cell.  This removes the race-condition and
the file is loaded correctly.

There is another race-condition that results in multiple threads
attempting to load and parse the file.  This will result in dCache
logging a line like:

   "redefinition of template <name>"

for each template and for each additional thread that attempts to load
the file. While noisy, this should only happen (if at all) for "a
while" after initialising and does not affect the correctness of the
rendered response.

Target: master
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Bug-report: http://rt.dcache.org/Ticket/Display.html?id=8315
Bug-report: (private correspondance with DESY people)
Patch: http://rb.dcache.org/r/6986/
Acked-by: Gerd Behrmann
webdav: generate error page when template couldn't be found

The webdav door uses StringTemplate to generate HTML pages for
directory listing and error pages, using hard-coded template
names that the corresponding template files must provide.

One reason for using StringTemplate is to allow site customisation
of the output.  However, dCache behaves badly if the hard-coded
template name isn't found; rather than logging a meaningful
error, a NPE is triggered.

This patch provides a hard-coded "broken dCache" page that describes
the problem so that the admin can fix their own site.

Target: master
Patch: http://rb.dcache.org/r/6981/
Acked-by: Tigran Mkrtchyan
billing: do not generate billing message if file based store is disabled

do not log billing messages to pinboard

Acked-by: Albert Rossi
Target: master
Require-book: no
Require-notes: yes
hsm scripts: fix command and options order

the order of command and options as they are used currently, breaks the
use of several hsm scripts.
This patch reorders the components to be

<command> put|get -si=... [-o1=.., ...]

Ticket: 8317
Acked-by:
Target: trunk
Target: 2.9
Require-book: no
Require-notes: no
srm: Fix transition ordering for restored requests

The transition time isn't precise enough to serve as a key or sorting.
Instead we use the element id which we know is allocated sequentially.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6972/
info: add JSON serialisation

This patch adds support for JSON serialisation of the information gathered by the info service.

Along with the existing methods for obtaining output in different formats, the web interface has been
improved to support the client choosing which format.  The default format remains XML, but the query
parameter 'format' and Accept HTTP request header are both consulted to allow an alternative format,
with the query parameter taking precedence.

httpd-side caching has been restructured to use a Guava cache and now applies to all requests,
keeping the last ten distict queries in memory.

Target: master
Patch: http://rb.dcache.org/r/6974/
Acked-by: Tigran Mkrtchyan
pool: Log P2P fatal errors with stack trace

We have observed class initialization errors at NDGF, but the code
doesn't log a stack trace for subclasses of Error. This patch solves
that problem.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6988/
system-test: Pick up DN of dCache user cert if available

If the system-test user cert is installed, that DN is authorized in addition
to the DN of your primary cert.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6989/
pool: Fix error handling in staging code

A regression in 2.9. Causes pools to disable in case of staging
an existing file.

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6990/
pool: Improve how we deal with unexpected files in pools

We recently observed all kinds of weird behaviour related to whether
some files did or did not exist on a pool. We don't quite know how
we got into that mess, but we do know that dCache was not particularly
good at recoverying from the situation.

Essentially, an unexpefted file in the data directory or an unexpected
entry in the pool would cause pools to disable themselves or go into
retry loops, respectively.

This patch implements a number of defensive moves. MetaDataCache now
deletes replicas it didn't load during startup and didn't create itself,
eg files that got injected by staging after dCache timed out. If dCache
receives a request to create an entry that already exists, we now
defensively register the pool location; this mirrors the opposite situation
in which somebody tries to open a file that does not exist on a pool - in
that situation we already defensively deregistered the pool location.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6991/
cells: Fix message counting in case of queue overflow

In case of message queue overflow, the message counters were not
decremented as they should.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6992/
Drop messages when TTL is exceeded

The purpose of having a message TTL is to allow messages to be
discarded when the TTL is exceeded. We have used this sparingly
in a couple of services, but there is no reason why we cannot
apply this logic to all services.

We have often observed that dCache doesn't recover well from
an overload situation and I hope this change improves on that.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6994/
fix type in gplazma ldap default value

gplazma.ldap.organization default value contains a typo

This patch fixes the typo and changes the default values to strings that
make clear that these have to be changed on order to use the ldap
plugin.

Acked-by:
Target: trunk
Require-book: no
Require-notes: no
nfs: make cell export configurable

make door well known

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
srm: Fix request expiration

Fixes an issue in which request expiration caused requests to be marked
as completed with success rather than as failed. The same issue would
also lead to messages about illegal state transitions from Failed to
Done in the log file.

The Job base class supports expiring jobs. When a job is already in
READY or TRANSFERRING, it doesn't fail the job - instead it is marked
as done successfully. I can only guess as to the motivation for this:
Probably becaues for get requests, the TURL is still usable and if
the client got the TURL then the transfer is likely going to succeed.

For put, the put container request had an override for the expiration
to fail the request no matter the state (as long as it is not final
already). However the expiration check is applied to both container
requests and file level requests. The consequence was that although
the container request was marked as failed, the file level request
would be marked as done successfully if expired before the contain
request. Consequently the file was not deleted.

For some requests, the SRM is quite clear that the expiration should
result in SRM_REQUEST_TIMED_OUT. For other requests (eg get) the
text says it should be returned, although the return code is not
listed as a possible return code of srmPrepareToGet. For ls, the
possibility of a timeout is not mentioned at all. I decided to
consider these as flaws in the spec and return SRM_REQUEST_TIMED_OUT
for all expired requests.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.9
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6997/
srm: Fix request cancellation due to SURL deletion

An srmRm cancels requests on the SURL that is deleted. The code however just
changed the state to CANCELED rather that calling abort and thus for put
requests the put wasn't properly aborted - in particular the upload directory
isn't deleted, resulting in those leaking.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.9
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6998/
Revert "srm: Drop fail fast for protocol checks"

This reverts commit 0f5a4d579476094c12e393bd14fdccb90ba99826.
chimera: Fix duplicate key supression

The postgresql driver for chimera has logic to avoid duplicate key
errors in postgresql while still throwing the SQLException on the
client side.

Unfortunately, the error code used is inconsistent with the error
code used in other logic that supresses the duplicate key error.

This patch resolves the problem: I checked the other places the
modified methods are called, and code that doesn't rely on the
isDuplicateKeyError just checks whther the error code starts with
'23'. This will still work after the change.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6999/
xrootd: split protocol name form the client IP in XrootdProtocolInfo

Acked-by: Paul Millar
Target: master, 2.8, 2.9
Require-book: no
Require-notes: yes