srm: fix IllegalStateTransition for COPY requests

If the user cancels a COPY request then the file request is moved to
state CANCELLED; however, if the copy then succeeds anyway, the SRM
attempts to move the job to state DONE.  This cannot succeed since
CANCELLED is a final state.

Fix this by first checking whether the job is already in a final
state.  Also tidy up some messages.

Target: master
Request: 2.10
Requires-notes: no
Requires-book: no
Acked-by: Tigran Mkrtchyan
Patch: https://rb.dcache.org/r/7147
srm: fix deserialise of TExtraInfo

The SRM saves the content of TExtraInfo for COPY requests, but fails
to deserialise it correctly if the user supplied no TExtraInfo.
This patch fixes that.

Target: master
Request: 2.10
Requires-book: no
Requires-notes: no
Acked-by: Tigran Mkrtchyan
Patch: https://rb.dcache.org/r/7147/
srm: fix NPE when checking delegated credential

The SRM keeps "the best" delegated credential, if the user supplies one.
If the user doesn't delegate a credential then the current code throws
a NPE.  This is fixed with this patch.

Target: master
Request: 2.10
Requires-notes: no
Requires-book: no
Acked-by: Tigran Mkrtchyan
Patch: https://rb.dcache.org/r/7147/
Merge pull request #702 from paulmillar/fix/2.10/srm-illegal-state-transition

srm: fix IllegalStateTransition for COPY requests
Merge pull request #703 from paulmillar/fix/2.10/srm-fix-extra-info

srm: fix deserialise of TExtraInfo
Merge pull request #704 from paulmillar/fix/2.10/srm-no-delegation

srm: fix NPE when checking delegated credential
[maven-release-plugin] prepare release 2.10.0
[maven-release-plugin] prepare for next development iteration
Don't use legacy properties

Fixes a couple of places that still relied on old property names.

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7167/
(cherry picked from commit ce761048c861c0bff59e927247c90d35cf1a8239)
srm,dcap: Use configured vomsdir and capath

The code for extracting vomsdir and the capath from certificates ignored the
configured paths. The code also created internal refresh threads with the wrong
diagnostic context.

Introduces srm.authn.vomsdir.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
(cherry picked from commit 7f39db9755cd151cbce2ac38f6ecdea3df5c40d3)
pool: Fix several minor issues in nearline storage subsystem

The pool now cancels flush and stage requests when the file is deleted.

Nearline storage configuration is now applied atomically rather than
one attribute at a time.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7172/
(cherry picked from commit c272c484ebfd4a95ab81b6e88908fed3ae771f78)
webdav: Make threading and TCP connection limits configurable

With the increase use of our webdav door, we see a need to tune
certain aspects of the webdav door that are currently hardcoded.

This patch exposes those aspects to the configuration.

Since I suspect that other sites will run into similar problems,
I suggest merging this into dCache 2.10.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7173/
(cherry picked from commit 797b966b4723b12f033d2c516d41fc292926a25f)
webdav: Kill mover in case transfer times out

Rather than leaving movers to timeout (and filling a transfer slot
in the meantime), this patch sends a notification to pools to kill
the mover in case the transfer times out in the door.

This approach is already used by other doors. I submit this for
inclusion in 2.10 as more sites will begin to see issues with the
webdav door as the use of HTTP increases.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7174/
(cherry picked from commit 06cf58197cabbad56b480ccca77759f937544758)
srm: Restore estimated wait time update

Commit 5211277 removed some "dead wood" (as the patch description called
it), but it turns out that some of the wood wasn't quite dead - just old.

Since we all grow older, this patch brings justice to the old trees and
restores the polling backoff that used to be part of the SRM. Clients will
once again reduce the polling frequency for requests that seem to take
a long time.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7175/
(cherry picked from commit c99aa7b18214d079dfe75e78d55794ecd46eb7d9)
poolmanager: Fix excessive p2p and stage alive checks

Stage and pool to pool requests in pool manager periodically check
whether the request is still alive on the pool. This is done by
listing the requests on the pool and checking whether the PNFS ID
of the file to stage or replicate is in the output.

For pools with many requests (like a stage pool), this will lead
to all requests being listed for every request. Thus O(n*n) entries
are transferred, which leads to problems on pools with many requests.

This patch is meant as an intermediate solution that can be backported
to stable branches. Rather than letting every request query the
list itself, a single thread will now do this collectively for all
requests waiting for a pool to pool transfer or stage.

A more long term solution should involve using a binary API to query
this information from the pool (or possibly let the pool broadcast this
information instead - could be considered together with a refactoring
of the way we collect information about active transfers).

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7177/
(cherry picked from commit f9f9ccdba6dcb0ef02f39c5e8c0678408740e671)
webdav: Fix certificate handling for HTTP third-party copy

The third-party copy support in the webdav manually extracts the certificate
chain from the request. It uses GLite VOMS validator component to extract the
primary FQAN. The VomsValidator requires a local vomsdir configuration, thus
requiring replicating configuration we already got in gPlazma.

Since the CopyFilter that implements third-party copy is invoked after the
SecurityFilter that communicates with gPlazma, there should be no reason why
CopyFilter couldn't simply access the current Subject (as returned by gPlazma)
and extract the DN and FQAN from it. This avois having a vomsdir setup on the
webdav door. This change is implemented by this patch.

The patch also updates the capath configuration in the mover used for
third-party copy. The mover would expand the *entire* dCache configuration to
subsequently extract a single property. It would do this on every transfer.
This is updated to only expand the single property that is needed, thus
reducing the per transfer overhead.

The patch fixes a synchronization issue in the mover in which the loaded CA
certs are stored in a static TrustStore, however access of that TrustStore was
synchronized locally on the mover instance, not on something shared by the
movers (like the mover class).

The patch fixes an issue caused by using per pool CA directories. Since the
mover cached this once, all pools of a domain would use the CA directory
configured for the first mover started. This is solved by using a Guava
LoadingCache. The cache also solves another problem: That of not reloading
the certificates. This is now achieved by expiring cache entries after
one minute.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7171/
(cherry picked from commit deb59961e3a9e89b94724b8c4935cfa14d02b92a)
srm: Fix race condition in srmPrepareToGet

There is a race condition in the pin call in which the callback happens
while the request is still in state RUNNING, but the main run method
already broke off to the branch that will switch the request to ASYNCWAIT.
In that case the request will not get rescheduled, meaning it will
stay in ASYNCWAIT with no further action until it expires or is aborted.

The patch fixes the problem and also updates the bring online implementation
to match the prepare to get logic. In particular that also means that
protocol support is checked for the entire request and not on a per file
level.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7176
(cherry picked from commit 8d1ce5367cc28f8b9e910a4a5c05dfaba49ee087)
poolmanager: Avoid request leak

When a read pool request waits for a file to be replicated or staged, it
registers a callback for the reply from the pool.

We have observed a situation, in which there were many more entries in the hash
table of callbacks than there were requests. The only code path I could find that
could explain this behaviour is when resubmitting a stage request. I am not
certain when exactly this would happen, but it is the only scenario I can see
how entries in the map could leak.

The patch also removes the 'rc destroy' command. The implementation was incomplete
and dangerous to use.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7179/
(cherry picked from commit abdb8d47fb31af99b539a32369b06d94f5fbe4da)

Conflicts:
	modules/dcache/src/main/java/diskCacheV111/poolManager/RequestContainerV5.java
poolmanager: Fix 'request clumping limit reached' failures

Pool manager has an incomplete feature in which a read pool selection is at
most used for a certain number of requests. This limit is hard coded to one,
meaning that as soon as two clients ask for the same file, the second will
receive an error. For some protocols the door silently retries. For other
protocols, the error is propagated to the client.

This patch changes the behaviour in two ways:

The limit is raised to 20, meaning up to 20 clients will receive the same pool
selection. The reason not to put this to infinity is because we still want to
trigger load based replication. 20 is an arbitrary number, that reduces the
risk of triggering the clumping warning, while also protecting against starting
too many movers on a single pool without pool manager getting a chance to
intervene.

The error code is changed from one indicating an out of resource problem to
one that simply indicates that the request is out of date - the latter causes
the door to refresh information about the file and resubmit the pool selection.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7184/
Patch: https://rb.dcache.org/r/7180/
(cherry picked from commit 38e3ac367a6ac7696bed1b863388830dfe46194c)
pool: Fix concurrency settings for script nearline storage

Adjusting the thread pool size did not have an immediate effect. This
patch resolves the problem.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7181/
(cherry picked from commit b89cd06d4d41c0f6b32f7d944a871a913bdb014a)