common: Fix race condition in AtomicCounter unit test

The patch modifies two unit tests. The first slept, although the
race between the two threads did not lead to failing tests (the
test thus was non-deterministic, but not non-determinate).

The second test could fail on certain interleavings of threads.
This is resolved by hooking into the base class at a specific
point to synchronize the execution of the two threads involved.
This too elliminates the sleep from the test.

Finally, the unit test is moved to the correct package and
Maven module.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: no
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7293/
alarms (common, dcache, dcache-webadmin, skel):  version 2.

This combined set of patches provides improvements to and
several new features for the alarm system.  The changes
are rather significant, but should not prevent downgrading
to previous versions. Interaction with version 1 should
be feasible, since what goes over the wire is still
a logback event, though alarm data inconsistencies could result.
If desirable, a back-port to the Golden Release 2.10 would be
possible when 2.11+ has been proven sufficiently stable.

1) common interfaces and properties

Provides a new set of interfaces for the alarms system, along
with revised property definitions.

AlarmDefinitions have been moved out of dcache core and
into commons.

Definitions are a special class of Alarm. Internal
alarms now must belong to the enum PredefinedAlarm.

Severity has been replaced by AlarmPriority.
Interfaces for the implementation of two internal maps
supporting the mapping of Alarms to AlarmPriority
and for custom definitions are provided.

2) DAO and map implementations

Maps: Implements priority map using properties file as
persistent backup. Implements definitions map
as before using JDOM and xml file as persistent
backup. Provides message vehicle for remote
request of the priority map.

DAO: LogEntry has now been entirely decoupled from
both the Alarms interfaces as well as ch.qos.logback
specifics. The Severity field and methods have
been removed. A liquibase changeset to remove
this attribute and its index from the schema
is provided.

3) Logback

Logback server: rather than wrapping the SocketServer
and giving it an entirely independent configuration
(thus preventing alarms from running in any
domain with other services), the server now uses
a Receiver which is a lifecycle component it adds
to the existing configuration. The extra guards in
the alarms.batch file have consequently been removed,
and logback-server.xml has been deleted.

Logback appender: now directly configures from
the relevant dcache properties and adds
the two embedded appenders (history and email).
The part of the appender which glues the logback-specific
event structure to the LogEntry and Alarms interfaces
has been extracted out into a package utility,
LoggingEventConverter. However, there is still
a binding between this appender and datanucleus
inasmuch as the PersistenceManagerFactory remains
injected through Spring.

The way in which an event is classified as an alarm
now is slight different. If the event is Marked
but no type is provided, it will follow the same
branch as unmarked events in trying to find a
regex match from any provided definitions; however,
failing a match, the untyped but marked event is
considered still to be an alarm, and is marked
GENERIC. The new structure also obviates the
need for an receiving filter based on the Alarm marker.

The RemoteMDCFilter no longer maintains its own
static host name lookup, but uses NetworkUtils.

4) Admin and Shell Commands

dCache Shell Commands: some refactoring of code in order to
reuse for admin commands; eliminated
use of URI to express optional source/origin
metadata. Added a new list command to
show names of internally defined alarms.
Add, modify and remove still work as before.

dCache Admin Commands: The send, add/modify/remove (definition)
commands from the shell are also reflected
here, as well as commands to control
priority mappings, and for saving and reloading
both priority and definitions.

5) Embedded Alarms

Changes calls to MarkerFactory to support new API.
Also incorporates changes from discarded patch
https://rb.dcache.org/r/7169/:

    Current trunk now logs the following when a pool starts:

    22 Jul 2014 12:33:52 (PoolManager) [pool_0 PoolManagerPoolUp] Pool pool_0 changed from mode disabled() to disabled(fetch,store,stage,p2p-client,p2p-server)
    22 Jul 2014 12:33:54 (PoolManager) [pool_0 PoolManagerPoolUp] Pool pool_0 is DOWN
    22 Jul 2014 12:33:57 (PoolManager) [pool_0 PoolManagerPoolUp] Pool pool_0 changed from mode disabled(fetch,store,stage,p2p-client,p2p-server) to disabled(store,stage,p2p-client)
    22 Jul 2014 12:33:57 (PoolManager) [pool_0 PoolManagerPoolUp] Pool pool_0 changed from mode disabled(store,stage,p2p-client) to enabled

    The 'Pool pool_0 is DOWN' message was added with patch 7137.

    However, this ALARM seems to capture disabled states which should
    not generate alarms, especially at pool start-up.

    The logic has been refined to limit the alarm to when a pool
    mode has actually changed to either disabled STRICT or DEAD,
    excluding on initialization.

    In particular: I have taken the general alarms out of the
    PoolManagerV5 messageArrived. It makes more sense to distinguish
    etween alarms at the source or origin of the problem.

    Thus:

    The watchdog thread is responsible for the POOL_DOWN
    alarm sent went the heartbeat ping is not returned.
    Specific pool failures on init, access, etc., which
    cause a shutdown raise alarms.
    Initial disabling and shutdown, as well as changes
    in state initiated through a user command, do not
    raise alarms.

6) Webadmin DAO

Changes to webadmin DAO to support modified scheme (without
Severity).

Entry types getter also eliminated since types are now
known via messaging with the alarm service.

7) Webadmin Presentation Layer

Changes to support filtering on priorty in memory,
as it is not part of the persistent DAO. These involve
changes to the AlarmTableProvider comparators and to
the AlarmDisplayService, which refreshes the provider
by sending a message to the alarm service in order to
fetch the alarm/priorty mapping.

The query panel drop-down list now holds the priority
values (rather than Severity), and the auto-complete
combo is based on the listing refreshed from the
alarm service. Hence httpd no longer requires the
presence of any alarms configuration/properties files
locally (though if the service is XML rather than
RDMBS, the store.xml file must still be accessible
locally).

Severity does not appear in the display table, but
the filtering of the query automatically sorts
by level of priority first (by default). Thus
when both alarms and non-alarms are displayed,
first CRITICAL, then HIGH, etc., alarms, and then
simple ERROR, WARN, etc. statements will appear.

8) Removes code which is no longer used.

TESTING DONE:

All previously existing JUnit tests have been modified
and rerun.

system-test built, booted, manually tested.

Integration tests conducted on my desktop (SL 6.3)
and testbed nodes (SL 5.3).  Interactive testing of
the commands were done.  The PoolManager heartbeat
message was also temporarily converted to an alarm.
Larger system testing was conducted using standalone
components.  Stress tests were conducted for both
a singleton alarm service domain (with no other
services) as well as including the alarm service in
another domain (billing and dCache/head node).
The stress test consisted of sending error log events
at approximately 100Hz with an exception trace of
depth 25, with 26 server-side regex definitions
guaranteed not to match the error message but set
to search the entire depth of the trace for each
message.  The service was able to sustain approximately
94Hz over the course of 100000+ events (about 15 minutes).
Writes were simultaneously done while the alarm
service was running inside billing.  No memory
issues were detected.

Target: master
Patch: https://rb.dcache.org/r/7255/
Patch: https://rb.dcache.org/r/7257/
Patch: https://rb.dcache.org/r/7258/
Patch: https://rb.dcache.org/r/7259/
Patch: https://rb.dcache.org/r/7260/
Patch: https://rb.dcache.org/r/7261/
Patch: https://rb.dcache.org/r/7262/
Patch: https://rb.dcache.org/r/7263/
Acked-by: Tigran
Require-notes: yes
Require-book:  yes

I will provide the book chapter changes
and the release notes.
srm: Fix return code and logging on "login errors"

A user recently reported that he got the following error:

ERROR MSG:	[FTS] FTS State [Failed] FTS Retries [0] Reason [SOURCE Failed to get source file size: srm-ifce err: Invalid argument, err: [SE][Ls][SRM_AUTHENTICATION_FAILURE] httpg://srm.ndgf.org:8443/srm/managerv2: Authentication failed (server log contains additional information)] Duration [1]

The server log however did not contain additional information, as that
information was only logged at info (which is disabled by default).

This patch refines the exceptions thrown by related code in different
error cases and updates how this is logged and reported to the client.
Thus the client will get more precise error codes and the server log
will contain more relevant information.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
srm: Log common misconfiguration preventing upload

Whether a transfer protocol is supported is checked in several places:

- At the request level (per requirement in the SRM spec)

- At the TURL computation level (as that's where we know the final path)

- In PutFileRequest (redundantly)

The redundant check in PutFileRequest is removed. The logging for
failed TURL computation is removed - this information is already
registered as a state transition on the job. A new log message is
added that logs common misconfiguration in which the path being
accessed isn't exposed by any of the doors supporting the requested
protocol. This would typically be the upload directory.

Such a misconfiguration now logs:

10 sep. 2014 23:46:18 (SRM-Gerds-MacBook-Pro) [hMA:1:srm2:prepareToPut:-2147482647:-2147482646] No door for [gsiftp] provides access to /upload/eefaa8f5-ea76-460e-b172-57ea8c69eb5f/a.

Target: trunk
Require-notes: yes
Require-book: yes
Request: 2.10
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7294/
srm: Minor refactoring of ls command

The 'ls' and 'ls completed' commands are merged, and the 'ls completed'
command is replaced by 'ls -completed=<n>'.

The command line interface is moved from thed dcache-srm module to the
srm-server module and used for both dCache and the standalone server.

List formatting related methods are moved from the SRM class to
SrmCommandLineInterface.

This patch is in anticipation of further extensions to the list command.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7264/
unittests: increase timeouts and fix race in DiskSpaceAllocatorTest

Various unit tests have timeouts to allow the test to fail if it gets
stuck.  While such tests are short-lived and the existing timeouts work
fine for desktop environments, when testing under heavy load the thread
scheduling within the OS can lead to severe delays and the tests could
fail.  For this reason, the timeouts are bumped up to 1 minute.  Tests
should aim to complete easily within one second (number picked out of
the air), so one minute should allow at least 2 orders of magnitude
leeway.

This patch also removes two unnecessary assertions.  These assertions
add nothing against an existing, earlier test and introduce a race
condition for a possible failure mode.

The test in question has a race in attempting to ensure the testing
thread blocks.  This is hard to fix without rewriting the code being
tested and, since it doesn't affect the test result, the code is left
as-is.

Target: master
Patch: https://rb.dcache.org/r/7308/
Acked-by: Tigran Mkrtchyan
Requires-notes: no
Requires-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
srm: Fix asynchroneous job storage leak

The asynchroneous job storage fails to preserve the force flag when
collapsing several save calls. In case srm.persistence.enable.history
is false (the default), this could lead to incomplete data in the
database. This in case could lead to incomplete cleanup of upload
directories after an SRM restart.

The patch also renames the boolean parameter to something that I find
less confusing.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7307/
srm-client: Fix logback configuration

Ever since common-cli was introduced, the console logback configuration
gets included in the srm package. This causes conflict to barf on startup
as it now got two configuration files (the one from the console package
and its own).

This patch resolves the issue. The patch also removes a couple of
unused and obsolete files from the srm client's conf directory
and removes the conf directory from the class path.

We may eventually port the srm client to use the console package, but
that is left for a future patch.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7311/
dcap: fix interaction with Spacemanager

we have to set subject in message to pool as well, as
space manager processes PoolAcceptFileMessage as well.

Ticket: #8456
Acked-by: Dmitry Litvintsev
Target: master, 2.10
Require-book: no
Require-notes: no
chimera: squash usedFiles() and usedSize() into a single query

kill two birds with one stone

Acked-by: Dmitry Litvintsev
Target: master
Require-book: no
Require-notes: no
alarms version 2: move database cleanup from webadmin to service

It no longer makes sense that the cleanup of
old alarms be done from inside the webadmin servlet. This
option should be part of the alarm server.

The configuration properties have been shifted over to alarm.properties,
and the query utilities have been relocated to dcache core.

Target: master
Require-book: yes
Require-notes: yes
Acked-by: Karsten
chimera: fix NPE on '.(parent)()' for root inode

Acked-by: Albert Rossi
Acked-by: Dmitry Litvintsev
Target: master, 2.10, 2.9, 2.8, 2.7, 2.6
Require-book: no
Require-notes: no
pool: Fix xrootd vector read limits

Fixes a bug that caused problems with ROOT 6: dCache provided
too large limits for the readv_ior_max and readv_iov_max
properties.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Ticket: http://rt.dcache.org/Ticket/Display.html?id=8461
Require-notes: ues
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7313/
info: Generate NAS by links rather than access

Info generates Normalized Access Storages - a non-overlapping grouping of pools
that share some properties.

In the past, how pools were grouped was based on which storage and dcache units
could do what kind of operation on the pool. Pools with the same access such
access properties were put into the same NAS. This did however result in NAS
that were rather difficult to understand: They had weird names that encoded the
access rules, and the list of links for the NAS did not apply to all pools in
the NAS. Also, the definition ignored link groups, which meant that pools
belonging to different link groups could be put into the same NAS if they
otherwise shared the same access properties.

This patch updates the NAS generator. It now groups those pools that are
accessible through the same set of links. NAS names are now an ordered list of
link names, which makes the relation ship to the pool manager configuration
much more obvious. The links of a NAS provide access to all the pools in the
NAS.

To the reviewer (Paul in particular): Please double check the updated unit
tests.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7305/
srm: Fall back to synchronous saving in case of overload

Requests are stored asynchronously in the database to avoid that the
save affects request processing. When the database cannot keep up, such
as may happen during an inrush of bulk bring-online requests, saves
to the database are dropped. A code comment claims that this only
affects monitoring, however the effect of SRM restarts, both as observed
by the client and wrt clean-up of upload directories. It also affects
debugging as information on past requests will be incomplete (something
one may be particularly interested in after an overload situation).

Saves are naturally classified into major and minor saves: Those that
merely log state transitions and those that contain updates to the
information about a request.

This patch changes the logic so that only minor updates are dropped. Major
updates are stored synchronously in case the asynchronous save queue
is full. This creates natural back pressure by slowing down request
processing.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7310/
hsm: Increase concurrency in nearline storage interface

When receving many HSM requests at once (in particular stage), the strict
synchronization in the nearline storage interface causes these requests to
serialize. In particular for stage requests this may be a problem, as the
replica entries are created in this phase.

This patch increases the concurrency in the nearline storage interface.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7183/
commons: fix TimeUtils nanos to duration string conversion

Ached-by: Karsten Schwank
Target: master
Require-book: no
Require-notes: no
commons: add TimeUtils#unitStringOf()

to provide short string form for a given time unit.

Ached-by: Karsten Schwank
Target: master
Require-book: no
Require-notes: no
commons: update RequestExecutionTimeGauges to accept units

Update nfs door/mover to use correct units, e.g. 'ns'.
Fix gauge's min value and format.

Acked-by: Albert Rossi
Target: master
Require-book: no
Require-notes: no
commons: remove period based gauges

The RequestExecutionTimeGauge has a possibility to have
period based average. This functionality can be replaced
by having two gauges - one to full time period and
second for a desired time period.

Acked-by: Albert Rossi
Target: master
Require-book: no
Require-notes: no