alarms: convert definitions into marked alarms

Replaces the remaining server-side alarm definitions
currently shipped with an alarm marked at the point
of emission (the database-connection-related alarms
were taken care of by patch 7138).

Target: master
Patch: https://rb.dcache.org/r/7137/
Require-book: yes
Require-notes: yes
Acked-by: Tigran

BOOK:
RELEASE NOTES:

All predefined alarms which were previously shipped
as alarm definitions are now embedded in the code.
These are also given specific types, severity levels
and even unique keys, just as the xml definition
allows.

The procedure for generating a custom alarm definition
has remained unchanged, so sites can continue to augment the
number of errors to be marked as alarms via the alarms-
definition.xml file.

The web page filter box for string matches
no longer carries as a drop-down all predefined
alarms, since there are now many alarm types
that are "hidden" in the code itself. What is
displayed as a drop-down list are any and all
types currently in the data store (hence if
a known alarm type does not appear in that list,
it indicates that no alarm of that type has
been received or at least remains recorded in
the store).

The dcache alarm send command option '-l' now
refers to the Severity (CRITICAL, HIGH, MEDIUM
or LOW) and not to the logging level.

As a rule, one can expect that any alarms envisaged
by the developers will be of the embedded type,
and any alarms defined in the alarms-definition.xml
file will by subject to verification by the site
administrators (since they depend on logging
statements, care must be taken to make sure
they continue to match).

Because the alarms-definition.xml file is no
longer used to create the web page drop-down
list, a copy of it is no longer necessary on
the node running the httpd service. The
corresponding path property has been dropped
from the httpd.properties (defaults).
alarms: eliminate httpd.alarms.db.xml.path

Since it is not possible to stream an xml
file by giving the DataNucleus connector
a remote URL, the constraint that httpd
and alarms share a local (or mounted)
file partition for alarms data still
holds when the alarms.db.type is 'xml'.

The alarms.db.xml.path property is
a convenience so as not to have to define
the entire URL in case one wishes to
change the location of the store xml
file from the default.

The immutable httpd version of this
property was thus merely a mirror
defined for use in spring injection
(for internal use only).

However, it should be noted that,
subsequent to changes made in 02/2014
to the webadmin spring injection,
the data stores are now entirely
configured via spring profiles;
hence, the internal xml path field no
longer serves any purpose in
the dCache DataNucleusAlarmStore class.

This patch eliminates the (immutable)
httpd path property and also the
internal xml path field.

Testing:

Deployed, tested the various type
options on the httpd end (rdbms, xml, off).

Target: master
Patch: https://rb.dcache.org/r/7143/
Require-notes: no
Require-book: no
Acked-by: Tigran
alarms: eliminate httpd.alarms.db.xml.path -- missing deletion of batch check

This seems to have been caught while I was testing
but I neglected to update the patch accordingly.

It needs to be eliminated or httpd will not start now
(after commit 4a66f6c982a90024141e19b842d94ddfca005cdf).

Target: master
Patch: https://rb.dcache.org/r/7158/
Require-notes: no
Require-book: no
Acked-by: Gerd
Don't use legacy properties

Fixes a couple of places that still relied on old property names.

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7167/
Upgrade to DataNucleus 4

Second attempt at upgrading to DataNucleus 4. In contrast to the previous patch,
this patch fixes a compatibility issue in AuthorizationRecord when upgrading
from an earlier version of dCache using postgresql.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7164/
srm,dcap: Use configured vomsdir and capath

The code for extracting vomsdir and the capath from certificates ignored the
configured paths. The code also created internal refresh threads with the wrong
diagnostic context.

Introduces srm.authn.vomsdir.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
pool: Fix several minor issues in nearline storage subsystem

The pool now cancels flush and stage requests when the file is deleted.

Nearline storage configuration is now applied atomically rather than
one attribute at a time.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7172/
webdav: Make threading and TCP connection limits configurable

With the increase use of our webdav door, we see a need to tune
certain aspects of the webdav door that are currently hardcoded.

This patch exposes those aspects to the configuration.

Since I suspect that other sites will run into similar problems,
I suggest merging this into dCache 2.10.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7173/
webdav: Kill mover in case transfer times out

Rather than leaving movers to timeout (and filling a transfer slot
in the meantime), this patch sends a notification to pools to kill
the mover in case the transfer times out in the door.

This approach is already used by other doors. I submit this for
inclusion in 2.10 as more sites will begin to see issues with the
webdav door as the use of HTTP increases.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7174/
srm: srm,dcap: Use configured vomsdir and capath

Fix build failure after commit 7f39db9755cd151cbce2ac38f6ecdea3df5c40d3.

Patch: none
srm: Restore estimated wait time update

Commit 5211277 removed some "dead wood" (as the patch description called
it), but it turns out that some of the wood wasn't quite dead - just old.

Since we all grow older, this patch brings justice to the old trees and
restores the polling backoff that used to be part of the SRM. Clients will
once again reduce the polling frequency for requests that seem to take
a long time.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7175/
poolmanager: Fix excessive p2p and stage alive checks

Stage and pool to pool requests in pool manager periodically check
whether the request is still alive on the pool. This is done by
listing the requests on the pool and checking whether the PNFS ID
of the file to stage or replicate is in the output.

For pools with many requests (like a stage pool), this will lead
to all requests being listed for every request. Thus O(n*n) entries
are transferred, which leads to problems on pools with many requests.

This patch is meant as an intermediate solution that can be backported
to stable branches. Rather than letting every request query the
list itself, a single thread will now do this collectively for all
requests waiting for a pool to pool transfer or stage.

A more long term solution should involve using a binary API to query
this information from the pool (or possibly let the pool broadcast this
information instead - could be considered together with a refactoring
of the way we collect information about active transfers).

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7177/
webdav: Fix certificate handling for HTTP third-party copy

The third-party copy support in the webdav manually extracts the certificate
chain from the request. It uses GLite VOMS validator component to extract the
primary FQAN. The VomsValidator requires a local vomsdir configuration, thus
requiring replicating configuration we already got in gPlazma.

Since the CopyFilter that implements third-party copy is invoked after the
SecurityFilter that communicates with gPlazma, there should be no reason why
CopyFilter couldn't simply access the current Subject (as returned by gPlazma)
and extract the DN and FQAN from it. This avois having a vomsdir setup on the
webdav door. This change is implemented by this patch.

The patch also updates the capath configuration in the mover used for
third-party copy. The mover would expand the *entire* dCache configuration to
subsequently extract a single property. It would do this on every transfer.
This is updated to only expand the single property that is needed, thus
reducing the per transfer overhead.

The patch fixes a synchronization issue in the mover in which the loaded CA
certs are stored in a static TrustStore, however access of that TrustStore was
synchronized locally on the mover instance, not on something shared by the
movers (like the mover class).

The patch fixes an issue caused by using per pool CA directories. Since the
mover cached this once, all pools of a domain would use the CA directory
configured for the first mover started. This is solved by using a Guava
LoadingCache. The cache also solves another problem: That of not reloading
the certificates. This is now achieved by expiring cache entries after
one minute.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7171/
srm: Fix race condition in srmPrepareToGet

There is a race condition in the pin call in which the callback happens
while the request is still in state RUNNING, but the main run method
already broke off to the branch that will switch the request to ASYNCWAIT.
In that case the request will not get rescheduled, meaning it will
stay in ASYNCWAIT with no further action until it expires or is aborted.

The patch fixes the problem and also updates the bring online implementation
to match the prepare to get logic. In particular that also means that
protocol support is checked for the entire request and not on a per file
level.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7176
pool: Order nearline requests

Orders the outpur of 'rh ls', 'st ls', and 'rm ls' by creation time.

Target: trunk
Require-notes: yes
Require-book: yes
poolmanager: Avoid request leak

When a read pool request waits for a file to be replicated or staged, it
registers a callback for the reply from the pool.

We have observed a situation, in which there were many more entries in the hash
table of callbacks than there were requests. The only code path I could find that
could explain this behaviour is when resubmitting a stage request. I am not
certain when exactly this would happen, but it is the only scenario I can see
how entries in the map could leak.

The patch also removes the 'rc destroy' command. The implementation was incomplete
and dangerous to use.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7179/
poolmanager: Fix 'request clumping limit reached' failures

Pool manager has an incomplete feature in which a read pool selection is at
most used for a certain number of requests. This limit is hard coded to one,
meaning that as soon as two clients ask for the same file, the second will
receive an error. For some protocols the door silently retries. For other
protocols, the error is propagated to the client.

This patch changes the behaviour in two ways:

The limit is raised to 20, meaning up to 20 clients will receive the same pool
selection. The reason not to put this to infinity is because we still want to
trigger load based replication. 20 is an arbitrary number, that reduces the
risk of triggering the clumping warning, while also protecting against starting
too many movers on a single pool without pool manager getting a chance to
intervene.

The error code is changed from one indicating an out of resource problem to
one that simply indicates that the request is out of date - the latter causes
the door to refresh information about the file and resubmit the pool selection.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7184/
Patch: https://rb.dcache.org/r/7180/
pool: Fix concurrency settings for script nearline storage

Adjusting the thread pool size did not have an immediate effect. This
patch resolves the problem.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7181/
build: add work-around for JDK bug and PowerMock

When mocking recalcitrant classes, the unit-tests uses powermock
to generate and inject the necessary byte-code to allow testing
independent of the underlying platform.

With OpenJDK v1.7.0_60, powermock-generated byte-code functions
correctly.  With OpenJDK v1.7.0_65, this generated byte-code
fails will errors like:

    java.lang.VerifyError: Bad <init> method call from inside of a branch
    Exception Details:
      Location:
        org/dcache/gplazma/plugins/NsswitchTest$SimpleIntStorage.<init>(Lorg/dcache/gplazma/plugins/NsswitchTest;)V @39: invokespecial
      Reason:
        Error exists in the bytecode
      Bytecode:
        0000000: 2a2b b500 142a 4d12 16b8 001c 03bd 000f
        0000010: 121e b800 22b8 0028 3a04 1904 b200 2ca5
        0000020: 000e 2a01 c000 2eb7 0031 a700 092c b700
        0000030: 3301 57b1
      Stackmap Table:
        full_frame(@45,{UninitializedThis,Object[#13],UninitializedThis,Top,Object[#15]},{})
        full_frame(@51,{Object[#2],Object[#13],Object[#2],Top,Object[#15]},{})

            at sun.reflect.GeneratedSerializationConstructorAccessor7.newInstance(Unknown Source)
            at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
            at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:40)
            at org.powermock.reflect.internal.WhiteboxImpl.newInstance(WhiteboxImpl.java:229)
            at org.powermock.reflect.Whitebox.newInstance(Whitebox.java:139)
            at org.dcache.gplazma.plugins.NsswitchTest.setUp(NsswitchTest.java:86)

This problem has been reported and will hopefully be fixed at some
point, but meanwhile, this patch adds a work-around.

Target: master
Patch: https://rb.dcache.org/r/7185/
Acked-by: Gerd Behrmann
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
srm: generate SRM v2.2 Axis-generated code as part of build process

Previously, SRM v2.2 code was generated by Axis v1.3 and committed
to VCS.  After this, some changes were made to the code, mostly
superficial.  Specifically:

  o  axis-generated single-line if-statements were changed to dCache
     style (commit 38841d52)

  o  the Serializable classes had their serialVersionUID added
     explicitly (commit d09fa438)

  o  Axis-generated redundant field-member initialisers were
     removed (commit 3e4d9b29)

  o  The TStatusCode class had an isProcessing method added
     (commit 0cd9d2d1)

This patch removes the auto-generated code and uses a maven plugin
to generate the code as part of the maven build process.  A patch
introduces the isProcessing method, but all other changes above are
lost.

Target: master
Requires-book: no
Requires-notes: no
Patch: https://rb.dcache.org/r/7186/
Acked-by: Gerd Behrmann