Don't use legacy properties

Fixes a couple of places that still relied on old property names.

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7167/
Upgrade to DataNucleus 4

Second attempt at upgrading to DataNucleus 4. In contrast to the previous patch,
this patch fixes a compatibility issue in AuthorizationRecord when upgrading
from an earlier version of dCache using postgresql.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7164/
srm,dcap: Use configured vomsdir and capath

The code for extracting vomsdir and the capath from certificates ignored the
configured paths. The code also created internal refresh threads with the wrong
diagnostic context.

Introduces srm.authn.vomsdir.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
pool: Fix several minor issues in nearline storage subsystem

The pool now cancels flush and stage requests when the file is deleted.

Nearline storage configuration is now applied atomically rather than
one attribute at a time.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7172/
webdav: Make threading and TCP connection limits configurable

With the increase use of our webdav door, we see a need to tune
certain aspects of the webdav door that are currently hardcoded.

This patch exposes those aspects to the configuration.

Since I suspect that other sites will run into similar problems,
I suggest merging this into dCache 2.10.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7173/
webdav: Kill mover in case transfer times out

Rather than leaving movers to timeout (and filling a transfer slot
in the meantime), this patch sends a notification to pools to kill
the mover in case the transfer times out in the door.

This approach is already used by other doors. I submit this for
inclusion in 2.10 as more sites will begin to see issues with the
webdav door as the use of HTTP increases.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7174/
srm: srm,dcap: Use configured vomsdir and capath

Fix build failure after commit 7f39db9755cd151cbce2ac38f6ecdea3df5c40d3.

Patch: none
srm: Restore estimated wait time update

Commit 5211277 removed some "dead wood" (as the patch description called
it), but it turns out that some of the wood wasn't quite dead - just old.

Since we all grow older, this patch brings justice to the old trees and
restores the polling backoff that used to be part of the SRM. Clients will
once again reduce the polling frequency for requests that seem to take
a long time.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7175/
poolmanager: Fix excessive p2p and stage alive checks

Stage and pool to pool requests in pool manager periodically check
whether the request is still alive on the pool. This is done by
listing the requests on the pool and checking whether the PNFS ID
of the file to stage or replicate is in the output.

For pools with many requests (like a stage pool), this will lead
to all requests being listed for every request. Thus O(n*n) entries
are transferred, which leads to problems on pools with many requests.

This patch is meant as an intermediate solution that can be backported
to stable branches. Rather than letting every request query the
list itself, a single thread will now do this collectively for all
requests waiting for a pool to pool transfer or stage.

A more long term solution should involve using a binary API to query
this information from the pool (or possibly let the pool broadcast this
information instead - could be considered together with a refactoring
of the way we collect information about active transfers).

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7177/
webdav: Fix certificate handling for HTTP third-party copy

The third-party copy support in the webdav manually extracts the certificate
chain from the request. It uses GLite VOMS validator component to extract the
primary FQAN. The VomsValidator requires a local vomsdir configuration, thus
requiring replicating configuration we already got in gPlazma.

Since the CopyFilter that implements third-party copy is invoked after the
SecurityFilter that communicates with gPlazma, there should be no reason why
CopyFilter couldn't simply access the current Subject (as returned by gPlazma)
and extract the DN and FQAN from it. This avois having a vomsdir setup on the
webdav door. This change is implemented by this patch.

The patch also updates the capath configuration in the mover used for
third-party copy. The mover would expand the *entire* dCache configuration to
subsequently extract a single property. It would do this on every transfer.
This is updated to only expand the single property that is needed, thus
reducing the per transfer overhead.

The patch fixes a synchronization issue in the mover in which the loaded CA
certs are stored in a static TrustStore, however access of that TrustStore was
synchronized locally on the mover instance, not on something shared by the
movers (like the mover class).

The patch fixes an issue caused by using per pool CA directories. Since the
mover cached this once, all pools of a domain would use the CA directory
configured for the first mover started. This is solved by using a Guava
LoadingCache. The cache also solves another problem: That of not reloading
the certificates. This is now achieved by expiring cache entries after
one minute.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7171/
srm: Fix race condition in srmPrepareToGet

There is a race condition in the pin call in which the callback happens
while the request is still in state RUNNING, but the main run method
already broke off to the branch that will switch the request to ASYNCWAIT.
In that case the request will not get rescheduled, meaning it will
stay in ASYNCWAIT with no further action until it expires or is aborted.

The patch fixes the problem and also updates the bring online implementation
to match the prepare to get logic. In particular that also means that
protocol support is checked for the entire request and not on a per file
level.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7176
pool: Order nearline requests

Orders the outpur of 'rh ls', 'st ls', and 'rm ls' by creation time.

Target: trunk
Require-notes: yes
Require-book: yes
poolmanager: Avoid request leak

When a read pool request waits for a file to be replicated or staged, it
registers a callback for the reply from the pool.

We have observed a situation, in which there were many more entries in the hash
table of callbacks than there were requests. The only code path I could find that
could explain this behaviour is when resubmitting a stage request. I am not
certain when exactly this would happen, but it is the only scenario I can see
how entries in the map could leak.

The patch also removes the 'rc destroy' command. The implementation was incomplete
and dangerous to use.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7179/
poolmanager: Fix 'request clumping limit reached' failures

Pool manager has an incomplete feature in which a read pool selection is at
most used for a certain number of requests. This limit is hard coded to one,
meaning that as soon as two clients ask for the same file, the second will
receive an error. For some protocols the door silently retries. For other
protocols, the error is propagated to the client.

This patch changes the behaviour in two ways:

The limit is raised to 20, meaning up to 20 clients will receive the same pool
selection. The reason not to put this to infinity is because we still want to
trigger load based replication. 20 is an arbitrary number, that reduces the
risk of triggering the clumping warning, while also protecting against starting
too many movers on a single pool without pool manager getting a chance to
intervene.

The error code is changed from one indicating an out of resource problem to
one that simply indicates that the request is out of date - the latter causes
the door to refresh information about the file and resubmit the pool selection.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7184/
Patch: https://rb.dcache.org/r/7180/
pool: Fix concurrency settings for script nearline storage

Adjusting the thread pool size did not have an immediate effect. This
patch resolves the problem.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7181/
build: add work-around for JDK bug and PowerMock

When mocking recalcitrant classes, the unit-tests uses powermock
to generate and inject the necessary byte-code to allow testing
independent of the underlying platform.

With OpenJDK v1.7.0_60, powermock-generated byte-code functions
correctly.  With OpenJDK v1.7.0_65, this generated byte-code
fails will errors like:

    java.lang.VerifyError: Bad <init> method call from inside of a branch
    Exception Details:
      Location:
        org/dcache/gplazma/plugins/NsswitchTest$SimpleIntStorage.<init>(Lorg/dcache/gplazma/plugins/NsswitchTest;)V @39: invokespecial
      Reason:
        Error exists in the bytecode
      Bytecode:
        0000000: 2a2b b500 142a 4d12 16b8 001c 03bd 000f
        0000010: 121e b800 22b8 0028 3a04 1904 b200 2ca5
        0000020: 000e 2a01 c000 2eb7 0031 a700 092c b700
        0000030: 3301 57b1
      Stackmap Table:
        full_frame(@45,{UninitializedThis,Object[#13],UninitializedThis,Top,Object[#15]},{})
        full_frame(@51,{Object[#2],Object[#13],Object[#2],Top,Object[#15]},{})

            at sun.reflect.GeneratedSerializationConstructorAccessor7.newInstance(Unknown Source)
            at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
            at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:40)
            at org.powermock.reflect.internal.WhiteboxImpl.newInstance(WhiteboxImpl.java:229)
            at org.powermock.reflect.Whitebox.newInstance(Whitebox.java:139)
            at org.dcache.gplazma.plugins.NsswitchTest.setUp(NsswitchTest.java:86)

This problem has been reported and will hopefully be fixed at some
point, but meanwhile, this patch adds a work-around.

Target: master
Patch: https://rb.dcache.org/r/7185/
Acked-by: Gerd Behrmann
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
srm: generate SRM v2.2 Axis-generated code as part of build process

Previously, SRM v2.2 code was generated by Axis v1.3 and committed
to VCS.  After this, some changes were made to the code, mostly
superficial.  Specifically:

  o  axis-generated single-line if-statements were changed to dCache
     style (commit 38841d52)

  o  the Serializable classes had their serialVersionUID added
     explicitly (commit d09fa438)

  o  Axis-generated redundant field-member initialisers were
     removed (commit 3e4d9b29)

  o  The TStatusCode class had an isProcessing method added
     (commit 0cd9d2d1)

This patch removes the auto-generated code and uses a maven plugin
to generate the code as part of the maven build process.  A patch
introduces the isProcessing method, but all other changes above are
lost.

Target: master
Requires-book: no
Requires-notes: no
Patch: https://rb.dcache.org/r/7186/
Acked-by: Gerd Behrmann
Fix several issues in recent alarms changes

The patch fixes several issues introduced in ff5e0423492a4931b188e48e2599962f1013a377:

- dcache-common imports hikaricp, which means that several components that
  don't make use of databases now pull in database related code (like the srm client).

- Missing @Override annotations in AlarmEnabledDataSource

- Redundant shutdown method in AlarmEnabledDataSource (use close)

- Infinite loop in uncaught exception handler (on IOException that isn't an
  out of file descriptor error, the code would loop forever)

- Complex code before kill on fatal JVM error (which in case of OOM may lead to
  another OOM rather than a shutdown)

To fix the HikariCP dependency I had to update to HikariCP 2.0.1.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7188/
cells: Fix deadlock during startup

The patch fixes a regression introduced in 2.8. The regression may cause a
deadlock in cells if they receive a message during startup.  This may happen
when restarting individual services in a busy dCache. It is unlikely to happen
when "cold starting" dCache.

The problem is caused by AbstractCell running most cell initialization on the
cell message thread (to ensure the correct context is used). If a message is
received before this task is created, the message will block the message thread
waiting for the cell startup to complete, however the cell startup cannot
complete because the initialization task cannot be executed due to the message
thread being blocked.

The patch solves this by returning a NoRouteToCellException when receiving a
message before cell startup has completed. Thus the blocking behaviour is
removed and the deadlock is avoided.

An unused shutdown gate is removed too.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7187/
Clean up Spring instantiation of database connection pools

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7189/