pool: Fix race condition that leads to orphaned files

Using FileNotInCacheException cause the chimera registration to be cleared.
Clearly we don't want this when the file is actually in the pool and just
happens to be in a state in which it isn't accessible.

This patch changes the exception to LockedCacheException.

This is an alternative to patch: https://rb.dcache.org/r/5801

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7103/
(cherry picked from commit ade506eb0c921bdeb4284d42e7c247794b3e5d1a)
Merge pull request #677 from kschwank/2.9

Suppress nfs' file protocol from being published
Merge pull request #688 from gbehrmann/fix/2.9/rb7103

pool: Fix race condition that leads to orphaned files
[maven-release-plugin] prepare release 2.9.6
[maven-release-plugin] prepare for next development iteration
delegation-shell: generate delegated credential of correct type

There are several types of proxy credential.  Currently the delegation shell generates
a fixed type.

In practise, the chain of proxy certificates must be of the same type or some services
will reject the chain.

This patch updates the delegation client so it generates a certificate of the same
type as the most recent (top-most) proxy certificate.

Target: master
Patch: https://rb.dcache.org/r/7112/
Acked-by: Gerd Behrmann
Request: 2.9
Merge pull request #689 from paulmillar/fix/2.9/delegation-shell-proxy-type

delegation-shell: generate delegated credential of correct type
deb: Add python-psycopg2 as a recommended dependency

Needed by dcache-star.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7091/
(cherry picked from commit efa2de674ac1ca5aa1b18d0aba186ee98c4ead22)
alarms: Fix checksum alarm filter.

The filter provided for checksum alarms generated by the pool scrubber
is out of sync with the actual logging statement in the scrubber.

This patch fixes the regex for the filter to match.

Target: 2.9
Require-book: no
Require-notes: yes
Acked-by: Karsten
Acked-by: Paul

RELEASE NOTES

Fixes a bug present since April 2013 (but up to now undiscovered by us) preventing
reporting of checksum alarms from the pool scrubber.
Merge pull request #691 from gbehrmann/fix/2.9/rb7091

deb: Add python-psycopg2 as a recommended dependency
Merge pull request #694 from alrossi/fix/2.9/checksum_alarm_filter

alarms: Fix checksum alarm filter.
srm-client: by default only delegate for srmcp with 3rd-party copies

Currently, the SRM client will delegate for all calls.  This patch
updates the default behaviour so it doesn't delegate, except for
srmcp involving an SRM-endpoint and some remote end-point.

Patch: https://rb.dcache.org/r/7031/
Acked-by: Karsten Schwank
Requires-book: no
Requires-notes: yes
Merge pull request #698 from paulmillar/fix/2.9/fix/srm-client-delegation

srm-client: by default only delegate for srmcp with 3rd-party copies
srm,dcap: Use configured vomsdir and capath

The code for extracting vomsdir and the capath from certificates ignored the
configured paths. The code also created internal refresh threads with the wrong
diagnostic context.

Introduces srm.authn.vomsdir.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
(cherry picked from commit 7f39db9755cd151cbce2ac38f6ecdea3df5c40d3)

Conflicts:
	modules/dcache/src/main/resources/diskCacheV111/srm/srm.xml
pool: Fix several minor issues in nearline storage subsystem

The pool now cancels flush and stage requests when the file is deleted.

Nearline storage configuration is now applied atomically rather than
one attribute at a time.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7172/
(cherry picked from commit c272c484ebfd4a95ab81b6e88908fed3ae771f78)
srm: Restore estimated wait time update

Commit 5211277 removed some "dead wood" (as the patch description called
it), but it turns out that some of the wood wasn't quite dead - just old.

Since we all grow older, this patch brings justice to the old trees and
restores the polling backoff that used to be part of the SRM. Clients will
once again reduce the polling frequency for requests that seem to take
a long time.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7175/
(cherry picked from commit c99aa7b18214d079dfe75e78d55794ecd46eb7d9)

Conflicts:
	modules/srm-server/src/main/java/org/dcache/srm/request/ContainerRequest.java
poolmanager: Fix excessive p2p and stage alive checks

Stage and pool to pool requests in pool manager periodically check
whether the request is still alive on the pool. This is done by
listing the requests on the pool and checking whether the PNFS ID
of the file to stage or replicate is in the output.

For pools with many requests (like a stage pool), this will lead
to all requests being listed for every request. Thus O(n*n) entries
are transferred, which leads to problems on pools with many requests.

This patch is meant as an intermediate solution that can be backported
to stable branches. Rather than letting every request query the
list itself, a single thread will now do this collectively for all
requests waiting for a pool to pool transfer or stage.

A more long term solution should involve using a binary API to query
this information from the pool (or possibly let the pool broadcast this
information instead - could be considered together with a refactoring
of the way we collect information about active transfers).

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7177/
(cherry picked from commit f9f9ccdba6dcb0ef02f39c5e8c0678408740e671)
poolmanager: Avoid request leak

When a read pool request waits for a file to be replicated or staged, it
registers a callback for the reply from the pool.

We have observed a situation, in which there were many more entries in the hash
table of callbacks than there were requests. The only code path I could find that
could explain this behaviour is when resubmitting a stage request. I am not
certain when exactly this would happen, but it is the only scenario I can see
how entries in the map could leak.

The patch also removes the 'rc destroy' command. The implementation was incomplete
and dangerous to use.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7179/
(cherry picked from commit abdb8d47fb31af99b539a32369b06d94f5fbe4da)

Conflicts:
	modules/dcache/src/main/java/diskCacheV111/poolManager/RequestContainerV5.java

(cherry picked from commit ca2afba19a8255fc1bc401f871a83cfabf44c981)
poolmanager: Fix 'request clumping limit reached' failures

Pool manager has an incomplete feature in which a read pool selection is at
most used for a certain number of requests. This limit is hard coded to one,
meaning that as soon as two clients ask for the same file, the second will
receive an error. For some protocols the door silently retries. For other
protocols, the error is propagated to the client.

This patch changes the behaviour in two ways:

The limit is raised to 20, meaning up to 20 clients will receive the same pool
selection. The reason not to put this to infinity is because we still want to
trigger load based replication. 20 is an arbitrary number, that reduces the
risk of triggering the clumping warning, while also protecting against starting
too many movers on a single pool without pool manager getting a chance to
intervene.

The error code is changed from one indicating an out of resource problem to
one that simply indicates that the request is out of date - the latter causes
the door to refresh information about the file and resubmit the pool selection.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7184/
Patch: https://rb.dcache.org/r/7180/
(cherry picked from commit 38e3ac367a6ac7696bed1b863388830dfe46194c)
pool: Fix concurrency settings for script nearline storage

Adjusting the thread pool size did not have an immediate effect. This
patch resolves the problem.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7181/
(cherry picked from commit b89cd06d4d41c0f6b32f7d944a871a913bdb014a)