poolmanager: Avoid request leak

When a read pool request waits for a file to be replicated or staged, it
registers a callback for the reply from the pool.

We have observed a situation, in which there were many more entries in the hash
table of callbacks than there were requests. The only code path I could find that
could explain this behaviour is when resubmitting a stage request. I am not
certain when exactly this would happen, but it is the only scenario I can see
how entries in the map could leak.

The patch also removes the 'rc destroy' command. The implementation was incomplete
and dangerous to use.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7179/
(cherry picked from commit abdb8d47fb31af99b539a32369b06d94f5fbe4da)

Conflicts:
	modules/dcache/src/main/java/diskCacheV111/poolManager/RequestContainerV5.java

(cherry picked from commit ca2afba19a8255fc1bc401f871a83cfabf44c981)
poolmanager: Fix 'request clumping limit reached' failures

Pool manager has an incomplete feature in which a read pool selection is at
most used for a certain number of requests. This limit is hard coded to one,
meaning that as soon as two clients ask for the same file, the second will
receive an error. For some protocols the door silently retries. For other
protocols, the error is propagated to the client.

This patch changes the behaviour in two ways:

The limit is raised to 20, meaning up to 20 clients will receive the same pool
selection. The reason not to put this to infinity is because we still want to
trigger load based replication. 20 is an arbitrary number, that reduces the
risk of triggering the clumping warning, while also protecting against starting
too many movers on a single pool without pool manager getting a chance to
intervene.

The error code is changed from one indicating an out of resource problem to
one that simply indicates that the request is out of date - the latter causes
the door to refresh information about the file and resubmit the pool selection.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7184/
Patch: https://rb.dcache.org/r/7180/
(cherry picked from commit 38e3ac367a6ac7696bed1b863388830dfe46194c)
build: add work-around for JDK bug and PowerMock

When mocking recalcitrant classes, the unit-tests uses powermock
to generate and inject the necessary byte-code to allow testing
independent of the underlying platform.

With OpenJDK v1.7.0_60, powermock-generated byte-code functions
correctly.  With OpenJDK v1.7.0_65, this generated byte-code
fails will errors like:

    java.lang.VerifyError: Bad <init> method call from inside of a branch
    Exception Details:
      Location:
        org/dcache/gplazma/plugins/NsswitchTest$SimpleIntStorage.<init>(Lorg/dcache/gplazma/plugins/NsswitchTest;)V @39: invokespecial
      Reason:
        Error exists in the bytecode
      Bytecode:
        0000000: 2a2b b500 142a 4d12 16b8 001c 03bd 000f
        0000010: 121e b800 22b8 0028 3a04 1904 b200 2ca5
        0000020: 000e 2a01 c000 2eb7 0031 a700 092c b700
        0000030: 3301 57b1
      Stackmap Table:
        full_frame(@45,{UninitializedThis,Object[#13],UninitializedThis,Top,Object[#15]},{})
        full_frame(@51,{Object[#2],Object[#13],Object[#2],Top,Object[#15]},{})

            at sun.reflect.GeneratedSerializationConstructorAccessor7.newInstance(Unknown Source)
            at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
            at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:40)
            at org.powermock.reflect.internal.WhiteboxImpl.newInstance(WhiteboxImpl.java:229)
            at org.powermock.reflect.Whitebox.newInstance(Whitebox.java:139)
            at org.dcache.gplazma.plugins.NsswitchTest.setUp(NsswitchTest.java:86)

This problem has been reported and will hopefully be fixed at some
point, but meanwhile, this patch adds a work-around.

Target: master
Patch: https://rb.dcache.org/r/7185/
Acked-by: Gerd Behrmann
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Merge pull request #724 from gbehrmann/fix/2.6/rb7179

poolmanager: Avoid request leak
Merge pull request #729 from gbehrmann/fix/2.6/rb7180

poolmanager: Fix 'request clumping limit reached' failures
[maven-release-plugin] prepare release 2.6.32
[maven-release-plugin] prepare for next development iteration
Update chimera-cli

Gives some more hints when invoked wrongly.
This is perhaps not a very useful patch since the command is dropped in 2.10, but I hope you like it anyway.
I hope I chose the correct branch.
Merge pull request #736 from onnozweers/patch-1

Update chimera-cli
bugfix: fix reverseMap not throwing NoSuchPrincipalExceptions

Instead of throwing a NoSuchPrincipalException on failed reverse
mappings the Ldap plugin would just return an empty set.
This fixed this misbehaviour.

Ticket: http://rt.dcache.org/Ticket/Display.html?id=8422
Acked-by: Paul
Target: 2.6
Require-book: no
Require-notes: yes
Merge pull request #741 from kschwank/bugfix/ldapReverseMap8422/2.6

bugfix: fix reverseMap not throwing NoSuchPrincipalExceptions
poolmanager: Suppress stack trace in case of unmatch units

The patch suppresses stack traces like the following:

03 Aug 2014 06:40:09 (PoolManager) [door:NFS-fax-dcap-proxy@faxDomain:1407040808696 NFS-fax-dcap-proxy PoolMgrSelectReadPool 0000D36F8581AC7043118452477B188E6EC1] Read pool selection failed
java.lang.IllegalArgumentException: Unit not matched : 0:0:0:0:0:0:0:1
        at diskCacheV111.poolManager.PoolSelectionUnitV2.match(PoolSelectionUnitV2.java:545) ~[dcache-core-2.9.6.jar:2.9.6]
        at diskCacheV111.poolManager.PoolMonitorV5$PnfsFileLocation.selectReadPool(PoolMonitorV5.java:234) ~[dcache-core-2.9.6.jar:2.9.6]
        at diskCacheV111.poolManager.RequestContainerV5$PoolRequestHandler.askIfAvailable(RequestContainerV5.java:2008) [dcache-core-2.9.6.jar:2.9.6]
        at diskCacheV111.poolManager.RequestContainerV5$PoolRequestHandler.stateEngine(RequestContainerV5.java:1488) [dcache-core-2.9.6.jar:2.9.6]
        at diskCacheV111.poolManager.RequestContainerV5$PoolRequestHandler.stateLoop(RequestContainerV5.java:1266) [dcache-core-2.9.6.jar:2.9.6]
        at diskCacheV111.poolManager.RequestContainerV5$PoolRequestHandler.access$2300(RequestContainerV5.java:691) [dcache-core-2.9.6.jar:2.9.6]
        at diskCacheV111.poolManager.RequestContainerV5$PoolRequestHandler$RunEngine.run(RequestContainerV5.java:1197) [dcache-core-2.9.6.jar:2.9.6]
        at org.dcache.util.FireAndForgetTask.run(FireAndForgetTask.java:31) [dcache-core-2.9.6.jar:2.9.6]
        at org.dcache.util.CDCExecutorServiceDecorator$1.run(CDCExecutorServiceDecorator.java:104) [dcache-core-2.9.6.jar:2.9.6]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_65]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7194/
(cherry picked from commit 1adf0d0b7cb9c262ab9b07009c53d35b58a6fe65)
Merge pull request #750 from gbehrmann/fix/2.6/rb7194

poolmanager: Suppress stack trace in case of unmatch units
gplazma: Fix JVM implementation dependency in unit test

The ListCommand outputs plugins in hash table order - this ordering is
implementation dependent, which in turn causes the
ListCommandTests.testDetailedList unit test to fail with some JVMs (eg Oracle
JDK 8).

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7196/
(cherry picked from commit c0a68c5a29d48d06dcb10ead9261aadeac074c6d)
Make JDK byte code verification bug workaround Java 8 compatible

A workaround for a JDK bug that affects PowerMock only work with Java 7.
The -UseSplitVerifier option is no longer supported in Java 8. As an
alternative workaround, this patch disables the byte code verifier
entirely. Since this is only used with our own unit tests, the impact
is low.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: no
Requier-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7197/
(cherry picked from commit d06ed79d0bd6f1e387fcd10ae3a62b32895d0d35)
info: Fix HashMap ordering assumption in unit test

The unit test assumes a particular serialized form of the state tree
nodes, which isn't guaranteed when iterating over a HashMap. This causes
the unit tests to fail with some JDKs, in particular JDK 8.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7198/
(cherry picked from commit 14797ddac174e77e1aabf2a1873e5878d3d311f2)
Merge pull request #755 from gbehrmann/fix/2.6/rb7196

gplazma: Fix JVM implementation dependency in unit test
Merge pull request #760 from gbehrmann/fix/2.6/rb7197

Make JDK byte code verification bug workaround Java 8 compatible
Merge pull request #765 from gbehrmann/fix/2.6/rb7198

info: Fix HashMap ordering assumption in unit test
pool: Resolve high memory usage and other issues in sweeper

Space sweeper while reclaiming loads the meta data entry of the
files it will remove. This consumes a lot of memory, in particular
when the entire pool is purged, causing OOM in the worst case.

This patch resolves this issue. The patch also resolves an issue
with the 'sweeper ls' command which would block the message thread.

Finally, log messages indicating the end of long running operations
in sweeper and for 'rep rmclass' have been added. This allows the
admin to detect when these operations have finished. Other log messages
have been refined too, including lowering the log level of several
messages.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7195/
(cherry picked from commit 467ad69ceaa2d5f7cdb4dd8ce21f36015381bdf4)

Conflicts:
	modules/dcache/src/main/java/org/dcache/pool/classic/SpaceSweeper2.java

(cherry picked from commit 8aaa6da3e174714e6f0433a12a598d62bf7bc7f2)