unittests: increase timeouts and fix race in DiskSpaceAllocatorTest

Various unit tests have timeouts to allow the test to fail if it gets
stuck.  While such tests are short-lived and the existing timeouts work
fine for desktop environments, when testing under heavy load the thread
scheduling within the OS can lead to severe delays and the tests could
fail.  For this reason, the timeouts are bumped up to 1 minute.  Tests
should aim to complete easily within one second (number picked out of
the air), so one minute should allow at least 2 orders of magnitude
leeway.

This patch also removes two unnecessary assertions.  These assertions
add nothing against an existing, earlier test and introduce a race
condition for a possible failure mode.

The test in question has a race in attempting to ensure the testing
thread blocks.  This is hard to fix without rewriting the code being
tested and, since it doesn't affect the test result, the code is left
as-is.

Target: master
Patch: https://rb.dcache.org/r/7308/
Acked-by: Tigran Mkrtchyan
Requires-notes: no
Requires-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
srm: Fix asynchroneous job storage leak

The asynchroneous job storage fails to preserve the force flag when
collapsing several save calls. In case srm.persistence.enable.history
is false (the default), this could lead to incomplete data in the
database. This in case could lead to incomplete cleanup of upload
directories after an SRM restart.

The patch also renames the boolean parameter to something that I find
less confusing.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7307/
srm-client: Fix logback configuration

Ever since common-cli was introduced, the console logback configuration
gets included in the srm package. This causes conflict to barf on startup
as it now got two configuration files (the one from the console package
and its own).

This patch resolves the issue. The patch also removes a couple of
unused and obsolete files from the srm client's conf directory
and removes the conf directory from the class path.

We may eventually port the srm client to use the console package, but
that is left for a future patch.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7311/
dcap: fix interaction with Spacemanager

we have to set subject in message to pool as well, as
space manager processes PoolAcceptFileMessage as well.

Ticket: #8456
Acked-by: Dmitry Litvintsev
Target: master, 2.10
Require-book: no
Require-notes: no
chimera: squash usedFiles() and usedSize() into a single query

kill two birds with one stone

Acked-by: Dmitry Litvintsev
Target: master
Require-book: no
Require-notes: no
alarms version 2: move database cleanup from webadmin to service

It no longer makes sense that the cleanup of
old alarms be done from inside the webadmin servlet. This
option should be part of the alarm server.

The configuration properties have been shifted over to alarm.properties,
and the query utilities have been relocated to dcache core.

Target: master
Require-book: yes
Require-notes: yes
Acked-by: Karsten
chimera: fix NPE on '.(parent)()' for root inode

Acked-by: Albert Rossi
Acked-by: Dmitry Litvintsev
Target: master, 2.10, 2.9, 2.8, 2.7, 2.6
Require-book: no
Require-notes: no
pool: Fix xrootd vector read limits

Fixes a bug that caused problems with ROOT 6: dCache provided
too large limits for the readv_ior_max and readv_iov_max
properties.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Ticket: http://rt.dcache.org/Ticket/Display.html?id=8461
Require-notes: ues
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7313/
info: Generate NAS by links rather than access

Info generates Normalized Access Storages - a non-overlapping grouping of pools
that share some properties.

In the past, how pools were grouped was based on which storage and dcache units
could do what kind of operation on the pool. Pools with the same access such
access properties were put into the same NAS. This did however result in NAS
that were rather difficult to understand: They had weird names that encoded the
access rules, and the list of links for the NAS did not apply to all pools in
the NAS. Also, the definition ignored link groups, which meant that pools
belonging to different link groups could be put into the same NAS if they
otherwise shared the same access properties.

This patch updates the NAS generator. It now groups those pools that are
accessible through the same set of links. NAS names are now an ordered list of
link names, which makes the relation ship to the pool manager configuration
much more obvious. The links of a NAS provide access to all the pools in the
NAS.

To the reviewer (Paul in particular): Please double check the updated unit
tests.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7305/
srm: Fall back to synchronous saving in case of overload

Requests are stored asynchronously in the database to avoid that the
save affects request processing. When the database cannot keep up, such
as may happen during an inrush of bulk bring-online requests, saves
to the database are dropped. A code comment claims that this only
affects monitoring, however the effect of SRM restarts, both as observed
by the client and wrt clean-up of upload directories. It also affects
debugging as information on past requests will be incomplete (something
one may be particularly interested in after an overload situation).

Saves are naturally classified into major and minor saves: Those that
merely log state transitions and those that contain updates to the
information about a request.

This patch changes the logic so that only minor updates are dropped. Major
updates are stored synchronously in case the asynchronous save queue
is full. This creates natural back pressure by slowing down request
processing.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7310/
hsm: Increase concurrency in nearline storage interface

When receving many HSM requests at once (in particular stage), the strict
synchronization in the nearline storage interface causes these requests to
serialize. In particular for stage requests this may be a problem, as the
replica entries are created in this phase.

This patch increases the concurrency in the nearline storage interface.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7183/
commons: fix TimeUtils nanos to duration string conversion

Ached-by: Karsten Schwank
Target: master
Require-book: no
Require-notes: no
commons: add TimeUtils#unitStringOf()

to provide short string form for a given time unit.

Ached-by: Karsten Schwank
Target: master
Require-book: no
Require-notes: no
commons: update RequestExecutionTimeGauges to accept units

Update nfs door/mover to use correct units, e.g. 'ns'.
Fix gauge's min value and format.

Acked-by: Albert Rossi
Target: master
Require-book: no
Require-notes: no
commons: remove period based gauges

The RequestExecutionTimeGauge has a possibility to have
period based average. This functionality can be replaced
by having two gauges - one to full time period and
second for a desired time period.

Acked-by: Albert Rossi
Target: master
Require-book: no
Require-notes: no
cells: fix IAE in 'show pinboard'

we get IllegalArgumentException if number of entries in the pinboard
less that we have to skip:

java.lang.IllegalArgumentException: number to skip cannot be negative
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:125) ~[guava-17.0.jar:na]
	at com.google.common.collect.Iterables.skip(Iterables.java:846) ~[guava-17.0.jar:na]
	at dmg.util.Pinboard.dump(Pinboard.java:44) ~[cells-2.11.0-SNAPSHOT.jar:2.11.0-SNAPSHOT]
	at dmg.cells.nucleus.CellAdapter$ShowPinboardCommand.call(CellAdapter.java:676) ~[cells-2.11.0-SNAPSHOT.jar:2.11.0-SNAPSHOT]
	at dmg.cells.nucleus.CellAdapter$ShowPinboardCommand.call(CellAdapter.java:657) ~[cells-2.11.0-SNAPSHOT.jar:2.11.0-SNAPSHOT]
	at org.dcache.util.cli.AnnotatedCommandExecutor.execute(AnnotatedCommandExecutor.java:128) ~[common-cli-2.11.0-SNAPSHOT.jar:2.11.0-SNAPSHOT]
	at org.dcache.util.cli.CommandInterpreter$CommandEntry.execute(CommandInterpreter.java:260) ~[common-cli-2.11.0-SNAPSHOT.jar:2.11.0-SN

Acked-by: Gerd Behrmann
Target: master, 2.10
Require-book: no
Require-notes: no
dcache,nfs,dcap,srm: Drop duplicate LoginStrategy cache

DCAP had its own simplified cache for LoginStrategy. This patch replaces
it with the generic implementation. The generic implementation is moved
out of the login service package, as the cache is unrelated to the login
service.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7329/
dcache: Use Mockito for CacheLoginStrategy unit tests

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7330/
test: Upgrade to mockito 1.10.4

It's alive!

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7331/
dcache: Do not cache gPlazma timeouts

The gPlazma cache used by NFS and SRM caches both positive and negative
replies. Unfortunately it also caches failures to communicate with gPlazma.
This means that a transient timeout would be cached too, thus increasing the
effect of the transient error.

This patch resolves this problem by not caching timeouts (which include failure
to communicate with gPlazma).

Other errors are still cached - the correctness of this is debatable when
gPlazma does not distinguish between transient and permanent errors.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7332/