admin: Propopage NPE to user

I ran into a bug in migration module that caused an NPE to be thrown. Unfortunately
the admin shell return no message at all. This was because the exception was wrapped
in a CacheException with an empty message string.

This patch addresses this problem at several levels: Unexpected exceptions are
now wrapped with message being Throwable#toString (given that we don't know which
except is being wrapped it is reasonable to include the exception type in the error
message), RuntimeExceptions and Errors are propagated unwrapped, and the log output
in admin is increased from warning to error.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7361/
pool: Fix pool selection bugs in migration module

There were two bugs in migration module:

- Random pool selection would select pools even when they are full.

- Proportional pool selection would trigger a NPE as it would return
  null when all pools are full.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7362/
cell: Log exceptions within the correct cell context

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7363/
pool: Make migration task cancellation more robust

A recent bug as a side effect caused task cancellation to fail
because cancelCopy threw an NPE. As a result the task got stuck
in this state. This patch moves the timeout initiation to be the
first action when entering the Cancelling state. That way the task
would eventually have timed out.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7364/
webadmin: fix exit login in billing refresh loop

Currently the thread which refreshes the billing plots
will exit if it encounters an unspecified error from
the billing service. But this behavior does not
take into account slow start-up of domains (i.e.,
the billing service may not be there yet but will
be eventually).

The patch fixes the logic to treat NoRouteToCell
exceptions differently by waiting for a short time
and retrying.

Testing done: On deployed service without billing,
and then with billing booted.

Target: master
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Patch: https://rb.dcache.org/r/7306
Acked-by: Gerd
Require-note: yes
Require-book: no

RELEASE NOTES:
Fixes a bug where a simple timeout (NoRouteToCell) causes an
exit from the billing service refresh loop requiring a restart
of the domain in order to reconnect.
webadmin: make pool queue data enum Java 8 compatible

The static method seems to cause problems for java 8 compilation.
Evidently there is something different about the availability of
of the enum instance reference inside the class definition.

But the better solution is just a shared instance method anyway.
Is there a need to change this in the backports? I imagine
systems won't be updating to java 8 unless installing 2.12 ...

Testing:

This compilation error goes away with the patch:

[ERROR] /Developer/Work/fnal/dCache/dcache/modules/dcache-webadmin/src/main/java/org/dcache/webadmin/model/businessobjects/rrd4j/PoolQueuePlotData.java:[90,24] no suitable method found for getSourceName(<anonymous org.dcache.webadmin.model.businessobjects.rrd4j.PoolQueuePlotData.PoolQueueHistogram>)
method org.dcache.webadmin.model.businessobjects.rrd4j.RoundRobinDatabaseDatasource.getSourceName() is not applicable
(actual and formal argument lists differ in length)
method <anonymous org.dcache.webadmin.model.businessobjects.rrd4j.PoolQueuePlotData$PoolQueueHistogram$1>.getSourceName() is not applicable
(actual and formal argument lists differ in length)

Target: master
Patch: https://rb.dcache.org/r/7370
Acked-by: Gerd
Require-book: no
Require-notes: no

Note that the pull requests for rb 7346 have been updated to take care of this.
pool: Fix ISE in 'rep ls'

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7369/
pool: Fix ISE in CacheEntryImpl#toString

Fixes

18 Sep 2014 12:57:38 (pdc_kth_se_034) [admin] Uncaught exception in thread sweeper-free
java.lang.IllegalStateException: Attribute is not defined: STORAGEINFO
        at org.dcache.vehicles.FileAttributes.guard(FileAttributes.java:150) ~[dcache-core-2.8.4.jar:2.8.4]
        at org.dcache.vehicles.FileAttributes.getStorageInfo(FileAttributes.java:296) ~[dcache-core-2.8.4.jar:2.8.4]
        at org.dcache.pool.repository.v5.CacheEntryImpl.toString(CacheEntryImpl.java:139) ~[dcache-core-2.8.4.jar:2.8.4]
        at java.lang.String.valueOf(String.java:2854) ~[na:1.7.0_51]
        at java.lang.StringBuilder.append(StringBuilder.java:128) ~[na:1.7.0_51]
        at org.dcache.pool.classic.SpaceSweeper2.reclaim(SpaceSweeper2.java:335) ~[dcache-core-2.8.4.jar:2.8.4]
        at org.dcache.pool.classic.SpaceSweeper2.access$000(SpaceSweeper2.java:39) ~[dcache-core-2.8.4.jar:2.8.4]
        at org.dcache.pool.classic.SpaceSweeper2$1.run(SpaceSweeper2.java:209) ~[dcache-core-2.8.4.jar:2.8.4]
Waiting for data... (interrupt to abort)

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7368/
pool: Fix race leading to false positices in pool size health checks

The pool would register free space before actually deleting the file
from the file system. Thus for a brief moment, the pool would appear
to have more free space that is available on disk.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7367/
spacemanager: Suppress transient deadlock resolution errors

When having non-trivial concurrent transactions, it is not unusual that
these transactions may conflict and cause deadlocks. An RDBMS will detect
such deadlocks and role back one or more of the involved transactions
to allow the rest to complete. An application is supposed to retry such
a failed transaction.

Space manager does catch and retry such transient failures. At the time
I didn't know which transient failures to expect, so I limited it to three
retries and added log output on each. After we have run the new space
manager in production, I see that DeadlockLoserDataAccessException happens
during load. This exception is absolutely expected from the RDBMS and
indiciates that the present transaction lost in conflicting updates and
has to be retried. The RDBMS should ensure that there is progress (ie
no livelock in which all transactions always fail and are retried), so
it is safe to retry such a failed transaction until it succeeds.

Therefore this patch removes the retry limit for this failure and also
lowers the log output to debug.

I will as a separate activity investigate whether some of the observed
deadlocks are avoidable (using normal deadlock prevention techniques)
and thus improve throughput.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Require-notes: yes
Require-book: no
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7371/
pool: Fix pool size health check in case of asynchronous release of space

We have observed that operating sometimes don't report freed space
right away when deleting files. This was observed on Linux (using ext4
and Mac OS X), but the behaviour can likely be observed on many other
systems.

The pool periodically checks the free space on the pool partition and
compares it with its own internal account of free space. If the partition
does not have enough free space, the pool size is reduced accordingly.

If the operating system does not report free space right after a file
has been deleted, it may happen that the pool's internal account object
has registered free space that isn't yet reported as free by the operating
system. In that case, the periodic health check may falsely reduce the
pool size.

The patch fixes this by introducing a 60 second grace period after
the pool deletes a file. During this period, the pool size health
check is suppressed.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7366/
chimera: fix unit tests

Two unit tests are racy because they assume at least 1 ms has elapsed
between two calls to stat.  This patch has the testing thread sleep
for a millisecond to ensure this is true.

The patch also alters how the directory listing helper handles invalid
situations, with it throwing an exception rather than logging an error.
This allows the unit-test that checks this behaviour to do so without
logging the error.

Target: master
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
poolmanager: Shut down request container thread pool during shutdown

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7376/
Minor adjustment to NoRouteToCell error in LoginBrokerHandler

Better matches the error we use in other places.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7377/
billing: Improve shutdown for db backend

The billing db would always print a warning on shutdown caused by
an expected thread interrupt. This patch elliminates that warning.
The patch also removes unnecessary checks for null values as well
as failures to react correctly to thread interrupt (when the thread
calling close is interrupted, it shouldn't continue blocking on
yet another Thread#join call). Finally, the patch changes serial
shutdown of consumer threads to shutting them down concurrently.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7378/
replica: Proper shutdown of database and avoid database singleton

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Albert Rossir <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7379/
replica: Shut down message processing thread on cell shutdown

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7380/
chimera: Avoid coupling to hikari and alarms in JdbcFs

Both the Hikari CP DataSource and the alarm DataSource wrapper
implement Closeable.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7381/