Don't use legacy properties

Fixes a couple of places that still relied on old property names.

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7167/
(cherry picked from commit ce761048c861c0bff59e927247c90d35cf1a8239)
srm,dcap: Use configured vomsdir and capath

The code for extracting vomsdir and the capath from certificates ignored the
configured paths. The code also created internal refresh threads with the wrong
diagnostic context.

Introduces srm.authn.vomsdir.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
(cherry picked from commit 7f39db9755cd151cbce2ac38f6ecdea3df5c40d3)
pool: Fix several minor issues in nearline storage subsystem

The pool now cancels flush and stage requests when the file is deleted.

Nearline storage configuration is now applied atomically rather than
one attribute at a time.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7172/
(cherry picked from commit c272c484ebfd4a95ab81b6e88908fed3ae771f78)
webdav: Make threading and TCP connection limits configurable

With the increase use of our webdav door, we see a need to tune
certain aspects of the webdav door that are currently hardcoded.

This patch exposes those aspects to the configuration.

Since I suspect that other sites will run into similar problems,
I suggest merging this into dCache 2.10.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7173/
(cherry picked from commit 797b966b4723b12f033d2c516d41fc292926a25f)
webdav: Kill mover in case transfer times out

Rather than leaving movers to timeout (and filling a transfer slot
in the meantime), this patch sends a notification to pools to kill
the mover in case the transfer times out in the door.

This approach is already used by other doors. I submit this for
inclusion in 2.10 as more sites will begin to see issues with the
webdav door as the use of HTTP increases.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7174/
(cherry picked from commit 06cf58197cabbad56b480ccca77759f937544758)
srm: Restore estimated wait time update

Commit 5211277 removed some "dead wood" (as the patch description called
it), but it turns out that some of the wood wasn't quite dead - just old.

Since we all grow older, this patch brings justice to the old trees and
restores the polling backoff that used to be part of the SRM. Clients will
once again reduce the polling frequency for requests that seem to take
a long time.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7175/
(cherry picked from commit c99aa7b18214d079dfe75e78d55794ecd46eb7d9)
poolmanager: Fix excessive p2p and stage alive checks

Stage and pool to pool requests in pool manager periodically check
whether the request is still alive on the pool. This is done by
listing the requests on the pool and checking whether the PNFS ID
of the file to stage or replicate is in the output.

For pools with many requests (like a stage pool), this will lead
to all requests being listed for every request. Thus O(n*n) entries
are transferred, which leads to problems on pools with many requests.

This patch is meant as an intermediate solution that can be backported
to stable branches. Rather than letting every request query the
list itself, a single thread will now do this collectively for all
requests waiting for a pool to pool transfer or stage.

A more long term solution should involve using a binary API to query
this information from the pool (or possibly let the pool broadcast this
information instead - could be considered together with a refactoring
of the way we collect information about active transfers).

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7177/
(cherry picked from commit f9f9ccdba6dcb0ef02f39c5e8c0678408740e671)
webdav: Fix certificate handling for HTTP third-party copy

The third-party copy support in the webdav manually extracts the certificate
chain from the request. It uses GLite VOMS validator component to extract the
primary FQAN. The VomsValidator requires a local vomsdir configuration, thus
requiring replicating configuration we already got in gPlazma.

Since the CopyFilter that implements third-party copy is invoked after the
SecurityFilter that communicates with gPlazma, there should be no reason why
CopyFilter couldn't simply access the current Subject (as returned by gPlazma)
and extract the DN and FQAN from it. This avois having a vomsdir setup on the
webdav door. This change is implemented by this patch.

The patch also updates the capath configuration in the mover used for
third-party copy. The mover would expand the *entire* dCache configuration to
subsequently extract a single property. It would do this on every transfer.
This is updated to only expand the single property that is needed, thus
reducing the per transfer overhead.

The patch fixes a synchronization issue in the mover in which the loaded CA
certs are stored in a static TrustStore, however access of that TrustStore was
synchronized locally on the mover instance, not on something shared by the
movers (like the mover class).

The patch fixes an issue caused by using per pool CA directories. Since the
mover cached this once, all pools of a domain would use the CA directory
configured for the first mover started. This is solved by using a Guava
LoadingCache. The cache also solves another problem: That of not reloading
the certificates. This is now achieved by expiring cache entries after
one minute.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7171/
(cherry picked from commit deb59961e3a9e89b94724b8c4935cfa14d02b92a)
srm: Fix race condition in srmPrepareToGet

There is a race condition in the pin call in which the callback happens
while the request is still in state RUNNING, but the main run method
already broke off to the branch that will switch the request to ASYNCWAIT.
In that case the request will not get rescheduled, meaning it will
stay in ASYNCWAIT with no further action until it expires or is aborted.

The patch fixes the problem and also updates the bring online implementation
to match the prepare to get logic. In particular that also means that
protocol support is checked for the entire request and not on a per file
level.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7176
(cherry picked from commit 8d1ce5367cc28f8b9e910a4a5c05dfaba49ee087)
poolmanager: Avoid request leak

When a read pool request waits for a file to be replicated or staged, it
registers a callback for the reply from the pool.

We have observed a situation, in which there were many more entries in the hash
table of callbacks than there were requests. The only code path I could find that
could explain this behaviour is when resubmitting a stage request. I am not
certain when exactly this would happen, but it is the only scenario I can see
how entries in the map could leak.

The patch also removes the 'rc destroy' command. The implementation was incomplete
and dangerous to use.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7179/
(cherry picked from commit abdb8d47fb31af99b539a32369b06d94f5fbe4da)

Conflicts:
	modules/dcache/src/main/java/diskCacheV111/poolManager/RequestContainerV5.java
poolmanager: Fix 'request clumping limit reached' failures

Pool manager has an incomplete feature in which a read pool selection is at
most used for a certain number of requests. This limit is hard coded to one,
meaning that as soon as two clients ask for the same file, the second will
receive an error. For some protocols the door silently retries. For other
protocols, the error is propagated to the client.

This patch changes the behaviour in two ways:

The limit is raised to 20, meaning up to 20 clients will receive the same pool
selection. The reason not to put this to infinity is because we still want to
trigger load based replication. 20 is an arbitrary number, that reduces the
risk of triggering the clumping warning, while also protecting against starting
too many movers on a single pool without pool manager getting a chance to
intervene.

The error code is changed from one indicating an out of resource problem to
one that simply indicates that the request is out of date - the latter causes
the door to refresh information about the file and resubmit the pool selection.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7184/
Patch: https://rb.dcache.org/r/7180/
(cherry picked from commit 38e3ac367a6ac7696bed1b863388830dfe46194c)
pool: Fix concurrency settings for script nearline storage

Adjusting the thread pool size did not have an immediate effect. This
patch resolves the problem.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7181/
(cherry picked from commit b89cd06d4d41c0f6b32f7d944a871a913bdb014a)
build: add work-around for JDK bug and PowerMock

When mocking recalcitrant classes, the unit-tests uses powermock
to generate and inject the necessary byte-code to allow testing
independent of the underlying platform.

With OpenJDK v1.7.0_60, powermock-generated byte-code functions
correctly.  With OpenJDK v1.7.0_65, this generated byte-code
fails will errors like:

    java.lang.VerifyError: Bad <init> method call from inside of a branch
    Exception Details:
      Location:
        org/dcache/gplazma/plugins/NsswitchTest$SimpleIntStorage.<init>(Lorg/dcache/gplazma/plugins/NsswitchTest;)V @39: invokespecial
      Reason:
        Error exists in the bytecode
      Bytecode:
        0000000: 2a2b b500 142a 4d12 16b8 001c 03bd 000f
        0000010: 121e b800 22b8 0028 3a04 1904 b200 2ca5
        0000020: 000e 2a01 c000 2eb7 0031 a700 092c b700
        0000030: 3301 57b1
      Stackmap Table:
        full_frame(@45,{UninitializedThis,Object[#13],UninitializedThis,Top,Object[#15]},{})
        full_frame(@51,{Object[#2],Object[#13],Object[#2],Top,Object[#15]},{})

            at sun.reflect.GeneratedSerializationConstructorAccessor7.newInstance(Unknown Source)
            at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
            at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:40)
            at org.powermock.reflect.internal.WhiteboxImpl.newInstance(WhiteboxImpl.java:229)
            at org.powermock.reflect.Whitebox.newInstance(Whitebox.java:139)
            at org.dcache.gplazma.plugins.NsswitchTest.setUp(NsswitchTest.java:86)

This problem has been reported and will hopefully be fixed at some
point, but meanwhile, this patch adds a work-around.

Target: master
Patch: https://rb.dcache.org/r/7185/
Acked-by: Gerd Behrmann
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Merge pull request #730 from gbehrmann/fix/2.10/rb7181

pool: Fix concurrency settings for script nearline storage
Merge pull request #722 from gbehrmann/fix/2.10/rb7179

poolmanager: Avoid request leak
Merge pull request #727 from gbehrmann/fix/2.10/rb7180

poolmanager: Fix 'request clumping limit reached' failures
Merge pull request #719 from gbehrmann/fix/2.10/rb7176

srm: Fix race condition in srmPrepareToGet
Merge pull request #718 from gbehrmann/fix/2.10/rb7171

webdav: Fix certificate handling for HTTP third-party copy
Merge pull request #716 from gbehrmann/fix/2.10/rb7177

poolmanager: Fix excessive p2p and stage alive checks
Merge pull request #714 from gbehrmann/fix/2.10/rb7175

srm: Restore estimated wait time update