dcache,nfs,dcap,srm: Drop duplicate LoginStrategy cache

DCAP had its own simplified cache for LoginStrategy. This patch replaces
it with the generic implementation. The generic implementation is moved
out of the login service package, as the cache is unrelated to the login
service.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7329/
dcache: Use Mockito for CacheLoginStrategy unit tests

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7330/
test: Upgrade to mockito 1.10.4

It's alive!

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7331/
dcache: Do not cache gPlazma timeouts

The gPlazma cache used by NFS and SRM caches both positive and negative
replies. Unfortunately it also caches failures to communicate with gPlazma.
This means that a transient timeout would be cached too, thus increasing the
effect of the transient error.

This patch resolves this problem by not caching timeouts (which include failure
to communicate with gPlazma).

Other errors are still cached - the correctness of this is debatable when
gPlazma does not distinguish between transient and permanent errors.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7332/
dcache: Let login cache propagate IllegalArgumentException

IllegalArgumentException is part of the contract of LoginStrategy, yet
the cache fails to propagate those correctly.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7333/
pool: Fix HSM cancellation

A regression may cause a Berkeley DB operation to be interrupted, resulting in
an invalid pool state:

com.sleepycat.je.ThreadInterruptedException: (JE 6.0.11) /pool/meta java.lang.InterruptedException THREAD_INTERRUPTED: InterruptedException may cause incorrect internal state, unable to continue. Environment is invalid and must be closed.
        at com.sleepycat.je.latch.LatchImpl.acquireExclusive(LatchImpl.java:62) ~[je-6.0.11.jar:6.0.11]
        at com.sleepycat.je.tree.IN.latch(IN.java:539) ~[je-6.0.11.jar:6.0.11]
        at com.sleepycat.je.dbi.CursorImpl.latchBIN(CursorImpl.java:441) ~[je-6.0.11.jar:6.0.11]
        at com.sleepycat.je.dbi.CursorImpl.removeCursorAndPerformCacheEviction(CursorImpl.java:588) ~[je-6.0.11.jar:6.0.11]
        at com.sleepycat.je.dbi.CursorImpl.close(CursorImpl.java:572) ~[je-6.0.11.jar:6.0.11]
        at com.sleepycat.je.dbi.CursorImpl.close(CursorImpl.java:555) ~[je-6.0.11.jar:6.0.11]
        at com.sleepycat.je.Cursor.close(Cursor.java:518) ~[je-6.0.11.jar:6.0.11]
        at com.sleepycat.collections.CurrentTransaction.closeCursor(CurrentTransaction.java:484) ~[je-6.0.11.jar:6.0.11]
        at com.sleepycat.collections.DataCursor.close(DataCursor.java:215) ~[je-6.0.11.jar:6.0.11]
        at com.sleepycat.collections.StoredContainer.closeCursor(StoredContainer.java:422) ~[je-6.0.11.jar:6.0.11]
        at com.sleepycat.collections.StoredContainer.getValue(StoredContainer.java:310) ~[je-6.0.11.jar:6.0.11]
        at com.sleepycat.collections.StoredMap.get(StoredMap.java:240) ~[je-6.0.11.jar:6.0.11]
        at org.dcache.pool.repository.meta.db.CacheRepositoryEntryImpl.getStorageInfo(CacheRepositoryEntryImpl.java:148) ~[dcache-core-2.10.3.jar:2.10.3]
        at org.dcache.pool.repository.meta.db.CacheRepositoryEntryImpl.getFileAttributes(CacheRepositoryEntryImpl.java:156) ~[dcache-core-2.10.3.jar:2.10.3]
        at org.dcache.pool.repository.v5.CacheEntryImpl.<init>(CacheEntryImpl.java:35) ~[dcache-core-2.10.3.jar:2.10.3]
        at org.dcache.pool.repository.v5.CacheRepositoryV5.setState(CacheRepositoryV5.java:837) ~[dcache-core-2.10.3.jar:2.10.3]
        at org.dcache.pool.repository.v5.CacheRepositoryV5.destroyWhenRemovedAndUnused(CacheRepositoryV5.java:955) ~[dcache-core-2.10.3.jar:2.10.3]
        at org.dcache.pool.repository.v5.ReadHandleImpl.close(ReadHandleImpl.java:52) ~[dcache-core-2.10.3.jar:2.10.3]
        at org.dcache.pool.nearline.NearlineStorageHandler$FlushRequestImpl.failed(NearlineStorageHandler.java:604) ~[dcache-core-2.10.3.jar:2.10.3]
        at org.dcache.pool.nearline.AbstractBlockingNearlineStorage$Task.run(AbstractBlockingNearlineStorage.java:230) ~[dcache-core-2.10.3.jar:2.10.3]
        at org.dcache.util.CDCExecutorServiceDecorator$1.run(CDCExecutorServiceDecorator.java:104) [dcache-core-2.10.3.jar:2.10.3]
        at org.dcache.util.BoundedExecutor$Worker.run(BoundedExecutor.java:211) [dcache-core-2.10.3.jar:2.10.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
        at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
Caused by: java.lang.InterruptedException: null
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1244) ~[na:1.7.0_51]
        at java.util.concurrent.locks.ReentrantLock.tryLock(ReentrantLock.java:445) ~[na:1.7.0_51]
        at com.sleepycat.je.latch.LatchImpl.acquireExclusive(LatchImpl.java:55) ~[je-6.0.11.jar:6.0.11]

This patch fixes the problem by preventing cancellation from killing the state
update.

The patch fixes a secondary problem in the same code: Cancellation could lead
to a minor memory leak due to requests not being removed from a map.

Target: trunk
Request: 2.10
Require-notes: yes
Require-nook: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7334/
pool: Prevent interruption of replica deletion during HSM flush

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7335/
pool: Decouple migration task from migration job

This change allows the task abstraction to be used outside a job.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7336/
pool: Allow multiple replicas for migration jobs

Migration module is idempotent, meaning running the same job several times has
the same result as running it once. Thus even when running a copy job twice,
only a single copy is created. This is generally desirable, as it allows jobs
to be rerun on failure without tracking lots of state. In some cases it may
however be desirable to create more than one target replica. This patch adds
such a feature without compromising the idempotence of migration jobs.

The new -replicas option enables the creation of migration jobs that create
more than one target copy. This is fully integrated with discovery, validation
and update of existing replicas. In the current design, only a single copy is
created at a time, although one may consider increasing parallellism in a
future update.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Albert Rossir <arossi@fnal.gov>
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7327/
pool: Add JavaDoc to migration JobDefinition

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7337/
pool: Fix minor errors in migration module documentation

Target: trunk
Require-notes: no
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7338/
xrootd: Upgrade to xrootd4j 1.3.5

Fixes problems with kXR_set requests.

Ticket: http://rt.dcache.org/Ticket/Display.html?id=8468
Ticket: http://rt.dcache.org/Ticket/Display.html?id=8429
Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7344/
ftp: Fix root path validation for upload directory

The ftp door can be configured to use a static root rather than the user's
root path. If the static root is not a prefix of the user's root directory,
the door should only allow access to the upload directory. The logic that
enforces this was broken, resulting in such upload attempts failing.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7347/
spacemanager: fix forbidden property message

Patch https://rb.dcache.org/r/7168/ committed as 8ea811d576, marks
deprecated properties as forbidden.  Unfortunately the message for
some spacemanager properties was incorrect and following the advice
leaves sites broken.

This patch updates the comment.

Target: master
Requires-notes: no
Requires-book: no
Patch: https://rb.dcache.org/r/7309
Acked-by: Gerd Behrmann
system-test: add 'start' profile

Add a maven profile option to automatically start (or restart) dCache
as part of the system-test build.

Target: master
Requires-book: false
Requires-notes: false
Patch: https://rb.dcache.org/r/7314/
Acked-by: Karsten Schwank
cell: Declassify eval failure as a bug

Currently, this is claimed to be a bug:

[Gerds-MacBook-Pro.local] (System) admin > eval sdsd
There is a bug here, please report to support@dcache.org: (3) Eval Result : sdsd

After applying this patch, we get:

[Gerds-MacBook-Pro.local] (System) admin > eval sds
(3) Eval Result : sds

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7353/
cell: Prevent interpreter stack overflow from killing the domain

This no longer causes a dCache crash:

cd System
define env x
exec env x
.
exec env x

Now it simply outputs an error message stating that a stack overflow
happened. Before it caused the domain to restart.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7352/
srm: Fix listing of failed, done and cancelled jobs

Both the condition and the ORDER BY clause of the SQL query were
faulty.

The faulty expression can be traced all the way back to 2007 - isn't
that wonderful?

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7350/
pool: Respect LAN port range for internal srmcp transfers

When copying files internally within the same dCache using server side srmCopy,
the actual transfer is done through DCAP. The mover implementing this did not
respect the LAN port range. This patch resolves that problem.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: yes
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7348/
srm-client: Output file level errors for sync requests

srm clients fail to output file level error information if the initial srmPrepareToPut,
srmPrepareToGet or srmCopy request fails. This would be the common way for synchroneous
process to fail. Originally these requests were always asynchronous and this problem
was not visible to the client.

This patch fixes this for srmCopy, srmPrepareToGet and srmPrepareToPut.

Without the patch one could observe:

Gerds-MacBook-Pro:dcache-git behrmann$ srmcp file:////bin/bash srm://localhost/private/test2
srm copy of at least one file failed or not completed
srm client error:
java.io.IOException: srmPrepareToPut submission failed, unexpected or failed status : SRM_FAILURE explanation= File requests have failed.

with the patch one now gets:

Gerds-MacBook-Pro:dcache-git behrmann$ srmcp file:////bin/bash srm://localhost/private/test2
Fri Oct 10 13:35:55 CEST 2014: retrieval of surl srm://localhost/private/test2 failed, status = SRM_AUTHORIZATION_FAILURE explanation=Access denied: /private/test2
srm copy of at least one file failed or not completed

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7349/