systemtest: regenerate host credentials, add user credentials

Patch adds new host credentials so the system-test continues to work
out-of-the-box.  The patch also adds user-certificates generated by
the same Disposable CA.  This should allow people without an X.509
certificate to use system-test without first regenerating credentials.

Target: master
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
webdav: work-around race-condition in StringTemplate library

StringTemplate has a lazy-load for parsing the template file.  It
turns out that this is very racy, with at least two race-conditions.

Once race-condition has the effect that if two threads attempt to
fetch a template at about the same time with the template file not
loaded then the template being requested by the second thread (the one
that lost the race, so not loading the template file) will be never be
available.

Since dCache uses STGroupFile object to render only one template, this
means that if two threads attempt to fetch the template close enough
together then the template is never available and null is returned.
Because of another problem, this triggers a NPE.

A symptom of this problem is that dCache (> 2.6) will log a single
entry

   "redefinition of template <name>"

where <name> is the template that is no longer available.

The work-around is to force the loading and parsing the template file
by the thread creating the cell.  This removes the race-condition and
the file is loaded correctly.

There is another race-condition that results in multiple threads
attempting to load and parse the file.  This will result in dCache
logging a line like:

   "redefinition of template <name>"

for each template and for each additional thread that attempts to load
the file. While noisy, this should only happen (if at all) for "a
while" after initialising and does not affect the correctness of the
rendered response.

Target: master
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Bug-report: http://rt.dcache.org/Ticket/Display.html?id=8315
Bug-report: (private correspondance with DESY people)
Patch: http://rb.dcache.org/r/6986/
Acked-by: Gerd Behrmann
Merge pull request #541 from paulmillar/fix/2.8/stringtemplate-race

Fix/2.8/stringtemplate race
srm: Fix transition ordering for restored requests

The transition time isn't precise enough to serve as a key or sorting.
Instead we use the element id which we know is allocated sequentially.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6972/
(cherry picked from commit f6c4fc5aef90652c2fa5199dc63d845fd3b47ce2)
pool: Log P2P fatal errors with stack trace

We have observed class initialization errors at NDGF, but the code
doesn't log a stack trace for subclasses of Error. This patch solves
that problem.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6988/
(cherry picked from commit 3a22b10f971e3a78372636c9c18930acfd223a28)
Merge pull request #549 from gbehrmann/fix/2.8/rb6988

Fix/2.8/rb6988
Merge pull request #546 from gbehrmann/fix/2.8/rb6972

Fix/2.8/rb6972
pool: Improve how we deal with unexpected files in pools

We recently observed all kinds of weird behaviour related to whether
some files did or did not exist on a pool. We don't quite know how
we got into that mess, but we do know that dCache was not particularly
good at recoverying from the situation.

Essentially, an unexpefted file in the data directory or an unexpected
entry in the pool would cause pools to disable themselves or go into
retry loops, respectively.

This patch implements a number of defensive moves. MetaDataCache now
deletes replicas it didn't load during startup and didn't create itself,
eg files that got injected by staging after dCache timed out. If dCache
receives a request to create an entry that already exists, we now
defensively register the pool location; this mirrors the opposite situation
in which somebody tries to open a file that does not exist on a pool - in
that situation we already defensively deregistered the pool location.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6991/
(cherry picked from commit 69fbf87d1dedc111fbfaf9321e80e2222ccafd5d)
cells: Fix message counting in case of queue overflow

In case of message queue overflow, the message counters were not
decremented as they should.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6992/
(cherry picked from commit 360a2302a77cde3eac52e5d30f3fd9ae33b98872)

Conflicts:
	modules/cells/src/main/java/dmg/cells/nucleus/CellNucleus.java
Merge pull request #554 from gbehrmann/fix/2.8/rb6991

Fix/2.8/rb6991
Merge pull request #559 from gbehrmann/fix/2.8/rb6992

Fix/2.8/rb6992
chimera: Fix duplicate key supression

The postgresql driver for chimera has logic to avoid duplicate key
errors in postgresql while still throwing the SQLException on the
client side.

Unfortunately, the error code used is inconsistent with the error
code used in other logic that supresses the duplicate key error.

This patch resolves the problem: I checked the other places the
modified methods are called, and code that doesn't rely on the
isDuplicateKeyError just checks whther the error code starts with
'23'. This will still work after the change.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6999/
(cherry picked from commit ee903925e9ea4513b3c553847ad120f62dab5ab3)

Conflicts:
	modules/chimera/src/main/java/org/dcache/chimera/PgSQLFsSqlDriver.java
Merge pull request #562 from gbehrmann/fix/2.8/rb6999

Fix/2.8/rb6999
xrootd: split protocol name form the client IP in XrootdProtocolInfo

Acked-by: Paul Millar
Target: master, 2.8, 2.9
Require-book: no
Require-notes: yes
(cherry picked from commit 0eb1af0b1c867fe736220ba1dbbf581232c4aefc)
Signed-off-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
core: split protocol name form the client IP in HttpProtocolInfo

Acked-by: Gerd Behrmann
Target: master, 2.8, 2.9
Require-book: no
Require-notes: no
(cherry picked from commit 418c8ff1fd3c16e8274f26250fe09d3587e44092)
Signed-off-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
(cherry picked from commit e534f7551b93855eb37abe06326592fadaa39a01)
Signed-off-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Merge pull request #566 from kofemann/fix-proto-2.8

split protocol name form the client IP in ProtocolInfos
srm: Avoid duplicate status code update on status query

Container requests calculate their status code based on the status of
their file requests: However once they reached a final state, that
final state should be used rather than update the state several times.

When requests expire, I have observed illegal state transition errors
caused by trying to update the state several times.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/7000/
(cherry picked from commit 39b2c1153e02882c4b4ae4309d37bffdffbe8f87)
httpd: Fix a couple of bugs

Target: trunk
Require-notes: no
Require-book: no
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/7002/
(cherry picked from commit ed1975e03a69065d1fe2b9af5f2570341e9bd13e)

Conflicts:
	modules/dcache/src/main/java/org/dcache/cells/UniversalSpringCell.java
Merge pull request #569 from gbehrmann/fix/2.8/rb7000

Fix/2.8/rb7000
Merge pull request #572 from gbehrmann/fix/2.8/rb7002

Fix/2.8/rb7002