srm,dcap: Use configured vomsdir and capath

The code for extracting vomsdir and the capath from certificates ignored the
configured paths. The code also created internal refresh threads with the wrong
diagnostic context.

Introduces srm.authn.vomsdir.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
pool: Fix several minor issues in nearline storage subsystem

The pool now cancels flush and stage requests when the file is deleted.

Nearline storage configuration is now applied atomically rather than
one attribute at a time.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7172/
webdav: Make threading and TCP connection limits configurable

With the increase use of our webdav door, we see a need to tune
certain aspects of the webdav door that are currently hardcoded.

This patch exposes those aspects to the configuration.

Since I suspect that other sites will run into similar problems,
I suggest merging this into dCache 2.10.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7173/
webdav: Kill mover in case transfer times out

Rather than leaving movers to timeout (and filling a transfer slot
in the meantime), this patch sends a notification to pools to kill
the mover in case the transfer times out in the door.

This approach is already used by other doors. I submit this for
inclusion in 2.10 as more sites will begin to see issues with the
webdav door as the use of HTTP increases.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7174/
srm: srm,dcap: Use configured vomsdir and capath

Fix build failure after commit 7f39db9755cd151cbce2ac38f6ecdea3df5c40d3.

Patch: none
srm: Restore estimated wait time update

Commit 5211277 removed some "dead wood" (as the patch description called
it), but it turns out that some of the wood wasn't quite dead - just old.

Since we all grow older, this patch brings justice to the old trees and
restores the polling backoff that used to be part of the SRM. Clients will
once again reduce the polling frequency for requests that seem to take
a long time.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7175/
poolmanager: Fix excessive p2p and stage alive checks

Stage and pool to pool requests in pool manager periodically check
whether the request is still alive on the pool. This is done by
listing the requests on the pool and checking whether the PNFS ID
of the file to stage or replicate is in the output.

For pools with many requests (like a stage pool), this will lead
to all requests being listed for every request. Thus O(n*n) entries
are transferred, which leads to problems on pools with many requests.

This patch is meant as an intermediate solution that can be backported
to stable branches. Rather than letting every request query the
list itself, a single thread will now do this collectively for all
requests waiting for a pool to pool transfer or stage.

A more long term solution should involve using a binary API to query
this information from the pool (or possibly let the pool broadcast this
information instead - could be considered together with a refactoring
of the way we collect information about active transfers).

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7177/
webdav: Fix certificate handling for HTTP third-party copy

The third-party copy support in the webdav manually extracts the certificate
chain from the request. It uses GLite VOMS validator component to extract the
primary FQAN. The VomsValidator requires a local vomsdir configuration, thus
requiring replicating configuration we already got in gPlazma.

Since the CopyFilter that implements third-party copy is invoked after the
SecurityFilter that communicates with gPlazma, there should be no reason why
CopyFilter couldn't simply access the current Subject (as returned by gPlazma)
and extract the DN and FQAN from it. This avois having a vomsdir setup on the
webdav door. This change is implemented by this patch.

The patch also updates the capath configuration in the mover used for
third-party copy. The mover would expand the *entire* dCache configuration to
subsequently extract a single property. It would do this on every transfer.
This is updated to only expand the single property that is needed, thus
reducing the per transfer overhead.

The patch fixes a synchronization issue in the mover in which the loaded CA
certs are stored in a static TrustStore, however access of that TrustStore was
synchronized locally on the mover instance, not on something shared by the
movers (like the mover class).

The patch fixes an issue caused by using per pool CA directories. Since the
mover cached this once, all pools of a domain would use the CA directory
configured for the first mover started. This is solved by using a Guava
LoadingCache. The cache also solves another problem: That of not reloading
the certificates. This is now achieved by expiring cache entries after
one minute.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7171/
srm: Fix race condition in srmPrepareToGet

There is a race condition in the pin call in which the callback happens
while the request is still in state RUNNING, but the main run method
already broke off to the branch that will switch the request to ASYNCWAIT.
In that case the request will not get rescheduled, meaning it will
stay in ASYNCWAIT with no further action until it expires or is aborted.

The patch fixes the problem and also updates the bring online implementation
to match the prepare to get logic. In particular that also means that
protocol support is checked for the entire request and not on a per file
level.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7176
pool: Order nearline requests

Orders the outpur of 'rh ls', 'st ls', and 'rm ls' by creation time.

Target: trunk
Require-notes: yes
Require-book: yes
poolmanager: Avoid request leak

When a read pool request waits for a file to be replicated or staged, it
registers a callback for the reply from the pool.

We have observed a situation, in which there were many more entries in the hash
table of callbacks than there were requests. The only code path I could find that
could explain this behaviour is when resubmitting a stage request. I am not
certain when exactly this would happen, but it is the only scenario I can see
how entries in the map could leak.

The patch also removes the 'rc destroy' command. The implementation was incomplete
and dangerous to use.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7179/
poolmanager: Fix 'request clumping limit reached' failures

Pool manager has an incomplete feature in which a read pool selection is at
most used for a certain number of requests. This limit is hard coded to one,
meaning that as soon as two clients ask for the same file, the second will
receive an error. For some protocols the door silently retries. For other
protocols, the error is propagated to the client.

This patch changes the behaviour in two ways:

The limit is raised to 20, meaning up to 20 clients will receive the same pool
selection. The reason not to put this to infinity is because we still want to
trigger load based replication. 20 is an arbitrary number, that reduces the
risk of triggering the clumping warning, while also protecting against starting
too many movers on a single pool without pool manager getting a chance to
intervene.

The error code is changed from one indicating an out of resource problem to
one that simply indicates that the request is out of date - the latter causes
the door to refresh information about the file and resubmit the pool selection.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7184/
Patch: https://rb.dcache.org/r/7180/
pool: Fix concurrency settings for script nearline storage

Adjusting the thread pool size did not have an immediate effect. This
patch resolves the problem.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7181/