Make use of improved diamond operator

Java 8 is more liberal in the use of the diamon operator.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7560/
Use lambda expressions when suitable

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7561/
spacemanager: Make use of lambda expressions and other minor fixes

All identified by IntelliJ.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7562/
Make use of streams API

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7563/
pool: add Io mode into mover's status line

we got a request from our admins to add READ/WRITE info
to the output of 'mover ls'.

Acked-by: Gerd Behrmann
Target: master, 2.10, 2.11
Require-book: no
Require-notes: yes
admin: Fix error reporting

admin shell erroneously logs remote RuntimeExceptions as bugs. This is
a regression introduced in version 2.9 due to UserAdminShell rethrowing
remote errors.

CellCommandInterpreter already does a local classification of runtime
exceptions into bugs and non-bugs (depending on the throws clause of
the message handler). RuntimeExceptions are propagated and logged
by CellNucleus as uncaught exceptions.

This patch changes the CommandInterpreter to propagate bugs as
CommandPanicException. The admin service is updated to properly
report these as bugs. Likewise, CommandThrowableExceptions are
no longer treated as fatal as these simply represent an regular
error reported by a command. The dispatch code is update to wrap
non-bug remote errors in CommandThrowableException.

Finally, the wording of the bug reporting code in the admin shell
is adjusted for grammar and precision.

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7556/
pool: Fix shutdown of nearline storage subsystem

Ensures proper shutdown of the nearline storage subsystem. There is an
existing shutdown call to every nearline storage instance initiated
by the HsmSet class, however this is

- asynchronous

- has no dependency to the repository

This means there is no guarantee that requests get cancelled before
the repository is closed. This leads to errors like these when shutting
down a pool with acive stage requests:

04 dec. 2014 00:02:32 (pool_read) [admin d7ab5ce2-4ff7-4cfa-883e-3bfb41a5c313] Uncaught exception in thread pool-48-thread-1
java.lang.IllegalStateException: Can't open a cursor Database was closed.
        at com.sleepycat.je.Database.checkOpen(Database.java:2200) ~[je-6.1.5.jar:6.1.5]
        at com.sleepycat.je.Database.openCursor(Database.java:824) ~[je-6.1.5.jar:6.1.5]
        at com.sleepycat.collections.CurrentTransaction.openCursor(CurrentTransaction.java:415) ~[je-6.1.5.jar:6.1.5]
        at com.sleepycat.collections.MyRangeCursor.openCursor(MyRangeCursor.java:53) ~[je-6.1.5.jar:6.1.5]
        at com.sleepycat.collections.MyRangeCursor.<init>(MyRangeCursor.java:29) ~[je-6.1.5.jar:6.1.5]
        at com.sleepycat.collections.DataCursor.init(DataCursor.java:170) ~[je-6.1.5.jar:6.1.5]
        at com.sleepycat.collections.DataCursor.<init>(DataCursor.java:58) ~[je-6.1.5.jar:6.1.5]
        at com.sleepycat.collections.StoredContainer.putKeyValue(StoredContainer.java:319) ~[je-6.1.5.jar:6.1.5]
        at com.sleepycat.collections.StoredMap.put(StoredMap.java:279) ~[je-6.1.5.jar:6.1.5]
        at org.dcache.pool.repository.meta.db.CacheRepositoryEntryImpl.storeStateIfDirty(CacheRepositoryEntryImpl.java:270) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at org.dcache.pool.repository.meta.db.CacheRepositoryEntryImpl.setState(CacheRepositoryEntryImpl.java:190) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at org.dcache.pool.repository.v5.CacheRepositoryV5.setState(CacheRepositoryV5.java:853) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at org.dcache.pool.repository.v5.WriteHandleImpl.fail(WriteHandleImpl.java:406) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at org.dcache.pool.repository.v5.WriteHandleImpl.close(WriteHandleImpl.java:429) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at org.dcache.pool.nearline.NearlineStorageHandler$StageRequestImpl.done(NearlineStorageHandler.java:955) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at org.dcache.pool.nearline.NearlineStorageHandler$StageRequestImpl.failed(NearlineStorageHandler.java:918) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at org.dcache.pool.nearline.AbstractBlockingNearlineStorage$Task.processRequest(AbstractBlockingNearlineStorage.java:268) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at org.dcache.pool.nearline.AbstractBlockingNearlineStorage$Task.run(AbstractBlockingNearlineStorage.java:242) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at org.dcache.util.CDCExecutorServiceDecorator$1.run(CDCExecutorServiceDecorator.java:104) [dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at org.dcache.util.BoundedExecutor$Worker.run(BoundedExecutor.java:211) [dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_25]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_25]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]

I considered whether one should update CacheRepositoryV5 to close all open
descriptors on shutdown, but that would be rather bad: If the descriptor
is closed, there is nothing prevent the nearline storage from still writing
the file.

Instead the nearline storage now properly cancel all requests and also
enforces that no new requests are enqueued once the nearline storage was
shut down. This is done with a two step procedure, first initiating
cancellation followed by a blocking call waiting for requests to terminate
(with a timeout).

We could also simply have represed the above error and relied on pool
startup verification to delete the file, but I think it is cleaner
to properly cancel the requests.

The patch also fixes a race in request cancellation (there was an open
todo comment in the code).

The desired of avoid coarse grained locking unfortunately makes the code
quite a bit more complicated.

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7558
dcap: check for url in some commands

some dcap commands can be executed only with URL suntax

Acked-by: Gerd Behrmann
Target: master, 2.11, 2.10 ( in cobmination with 14d25b2)
Require-book: no
Require-notes: no
logback: revert commit 1cc57b1efaced76d22e5b561f861c6daf981f249

Not all layout tags had to be replaced by encoder tags, which now even caused problems.
This patch reverts this situation. The review that was acked before was this:
https://rb.dcache.org/r/7554

packages/system-test/src/main/skel/etc/logback.xml was deleted by a later patch after, which
makes it unnessecary to revert this files's changes.

Acked-by: Gerd
Target: master
Require-book: no
Require-notes: no
Patch: http://rb.dcache.org/r/7570/
ssh: Allow empty lines and comments marked by hash in authorized_keys2

Previously authorized_keys2 file could neither have empty lines nor entries like these:

Ticket: 8552
Acked-by: Gerd
Target: master
Require-book: no
Require-notes: no
Patch: http://rb.dcache.org/r/7569/
Port billing indexer to Java 8

Mostly this patch is about using the new date API of Java 8.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7447/
alarms: Detach alarms appender on shutdown

Otherwise alarms may be attemped to be logged to a datasource that is
already shut down - this causes hanging shutdown issues, in particular
when alarms shares a domain with other services.

Target: trunk
Request: 2.11
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7553/
alarms: Fix several resource leak and shutdown problems

- Fixes issues with transactions not being rolled back in case of
  failures.

- Fixes issues with shutting persistence manager factory while
  writing to it (caused JDO exceptions stating that there were
  still active transactions).

Note that the AppenderBase base class already checks whether the
appender is started and synchronizes on the object when calling
append().

Target: trunk
Request: 2.11
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7557/
pool: Simplify synchronization in nearline storage handler

To allow parallellizaation in nearline storage request creation,
2.11 introduced the use of a Guava Striped object to lock a part
of the request key space. This was necessary as we didn't want
to rely on Java 8 in dCache 2.11. With master now using Java 8,
the code can be simplified by using ConcurrentHashMap's ability
to compute the value to be inserted on demand when not already
present. The map imeplementation locks hash table buckets, which
is potentially more fine-grained as the Striped object used.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7559/
Fix typo in PnfsManager

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7564/
ftp: Some simple refactorings identified by IntelliJ

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7565/
spacemanager: Move dcache.enable.space-reservation to proper file

It was originally declared in spacemanager.properties because it inherited
from a now deprecated space manager property. Since that's no longer the
case, we may just as well move it to the "normal" place.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Milar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7576/
spacemanager: Fix null constraints and other schema migration issues

The old space manager allowed null values for several fields. According
to the code, these fields should not actually become null, however at
least one site reported seeing records with null values. Either this
is from an undetected bug in the old code or because older versions of
dCache created records with null values. Such null values cause the schema
migration to fail. This patch resolves the issue by adding a default value
for some of the null fields.

For the remanining fields that could be null in the schema but which shouldn't
be null, the patch adds an explicit check to instruct sites to contact us.

The patch also adds a check for site local computed indexes on expiration
time - at least one site had this.

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Request: 2.8
Require-notes: yes
Require-book: no
Ticket: http://rt.dcache.org/Ticket/Display.html?id=8567
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7579/