webadmin: fix exit login in billing refresh loop

Currently the thread which refreshes the billing plots
will exit if it encounters an unspecified error from
the billing service. But this behavior does not
take into account slow start-up of domains (i.e.,
the billing service may not be there yet but will
be eventually).

The patch fixes the logic to treat NoRouteToCell
exceptions differently by waiting for a short time
and retrying.

Testing done: On deployed service without billing,
and then with billing booted.

Target: master
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Patch: https://rb.dcache.org/r/7306
Acked-by: Gerd
Require-note: yes
Require-book: no

RELEASE NOTES:
Fixes a bug where a simple timeout (NoRouteToCell) causes an
exit from the billing service refresh loop requiring a restart
of the domain in order to reconnect.
webadmin: make pool queue data enum Java 8 compatible

The static method seems to cause problems for java 8 compilation.
Evidently there is something different about the availability of
of the enum instance reference inside the class definition.

But the better solution is just a shared instance method anyway.
Is there a need to change this in the backports? I imagine
systems won't be updating to java 8 unless installing 2.12 ...

Testing:

This compilation error goes away with the patch:

[ERROR] /Developer/Work/fnal/dCache/dcache/modules/dcache-webadmin/src/main/java/org/dcache/webadmin/model/businessobjects/rrd4j/PoolQueuePlotData.java:[90,24] no suitable method found for getSourceName(<anonymous org.dcache.webadmin.model.businessobjects.rrd4j.PoolQueuePlotData.PoolQueueHistogram>)
method org.dcache.webadmin.model.businessobjects.rrd4j.RoundRobinDatabaseDatasource.getSourceName() is not applicable
(actual and formal argument lists differ in length)
method <anonymous org.dcache.webadmin.model.businessobjects.rrd4j.PoolQueuePlotData$PoolQueueHistogram$1>.getSourceName() is not applicable
(actual and formal argument lists differ in length)

Target: master
Patch: https://rb.dcache.org/r/7370
Acked-by: Gerd
Require-book: no
Require-notes: no

Note that the pull requests for rb 7346 have been updated to take care of this.
pool: Fix ISE in 'rep ls'

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7369/
pool: Fix ISE in CacheEntryImpl#toString

Fixes

18 Sep 2014 12:57:38 (pdc_kth_se_034) [admin] Uncaught exception in thread sweeper-free
java.lang.IllegalStateException: Attribute is not defined: STORAGEINFO
        at org.dcache.vehicles.FileAttributes.guard(FileAttributes.java:150) ~[dcache-core-2.8.4.jar:2.8.4]
        at org.dcache.vehicles.FileAttributes.getStorageInfo(FileAttributes.java:296) ~[dcache-core-2.8.4.jar:2.8.4]
        at org.dcache.pool.repository.v5.CacheEntryImpl.toString(CacheEntryImpl.java:139) ~[dcache-core-2.8.4.jar:2.8.4]
        at java.lang.String.valueOf(String.java:2854) ~[na:1.7.0_51]
        at java.lang.StringBuilder.append(StringBuilder.java:128) ~[na:1.7.0_51]
        at org.dcache.pool.classic.SpaceSweeper2.reclaim(SpaceSweeper2.java:335) ~[dcache-core-2.8.4.jar:2.8.4]
        at org.dcache.pool.classic.SpaceSweeper2.access$000(SpaceSweeper2.java:39) ~[dcache-core-2.8.4.jar:2.8.4]
        at org.dcache.pool.classic.SpaceSweeper2$1.run(SpaceSweeper2.java:209) ~[dcache-core-2.8.4.jar:2.8.4]
Waiting for data... (interrupt to abort)

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7368/
pool: Fix race leading to false positices in pool size health checks

The pool would register free space before actually deleting the file
from the file system. Thus for a brief moment, the pool would appear
to have more free space that is available on disk.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7367/
spacemanager: Suppress transient deadlock resolution errors

When having non-trivial concurrent transactions, it is not unusual that
these transactions may conflict and cause deadlocks. An RDBMS will detect
such deadlocks and role back one or more of the involved transactions
to allow the rest to complete. An application is supposed to retry such
a failed transaction.

Space manager does catch and retry such transient failures. At the time
I didn't know which transient failures to expect, so I limited it to three
retries and added log output on each. After we have run the new space
manager in production, I see that DeadlockLoserDataAccessException happens
during load. This exception is absolutely expected from the RDBMS and
indiciates that the present transaction lost in conflicting updates and
has to be retried. The RDBMS should ensure that there is progress (ie
no livelock in which all transactions always fail and are retried), so
it is safe to retry such a failed transaction until it succeeds.

Therefore this patch removes the retry limit for this failure and also
lowers the log output to debug.

I will as a separate activity investigate whether some of the observed
deadlocks are avoidable (using normal deadlock prevention techniques)
and thus improve throughput.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Require-notes: yes
Require-book: no
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7371/
pool: Fix pool size health check in case of asynchronous release of space

We have observed that operating sometimes don't report freed space
right away when deleting files. This was observed on Linux (using ext4
and Mac OS X), but the behaviour can likely be observed on many other
systems.

The pool periodically checks the free space on the pool partition and
compares it with its own internal account of free space. If the partition
does not have enough free space, the pool size is reduced accordingly.

If the operating system does not report free space right after a file
has been deleted, it may happen that the pool's internal account object
has registered free space that isn't yet reported as free by the operating
system. In that case, the periodic health check may falsely reduce the
pool size.

The patch fixes this by introducing a 60 second grace period after
the pool deletes a file. During this period, the pool size health
check is suppressed.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7366/
chimera: fix unit tests

Two unit tests are racy because they assume at least 1 ms has elapsed
between two calls to stat.  This patch has the testing thread sleep
for a millisecond to ensure this is true.

The patch also alters how the directory listing helper handles invalid
situations, with it throwing an exception rather than logging an error.
This allows the unit-test that checks this behaviour to do so without
logging the error.

Target: master
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6