scripts: fix java invocation

Patch https://rb.dcache.org/r/7168/ committed as 8ea811d5 prevents
dCache from starting as the 'pool.dcap.port' is now marked
forbidden yet the patch failed to update the reference to this
property in the dCache command-line.

This patch fixes the command-line to use the updated property name.

Target: master
Patch: https://rb.dcache.org/r/7271/
Acked-by: Gerd Behrmann
Requires-book: no
Requires-notes: no
xrootd: allow empty xrootd.authz.read-paths and xrootd.authz.write-paths

The default value for xrootd.authz.write-paths is the empty string, but
the "check -strong" prevents this default value from working.

This patch removes the strong check for both paths.

Target: master
Requires-book: no
Requires-notes: no
Acked-by: Gerd Behrmann
Patch: https://rb.dcache.org/r/7272/
httpd: Use service specific property for list of login brokers

Part of the httpd service used dcache.service.loginbroker rather
than httpd.service.loginbroker. This patch resolves the issue.

The patch introduces httpd.setup to define the setup file of the
httpd service, thus making the reference to the file explicit in
the configuration.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7266/
configuration: adjust references to forbidden properties

The recent change that made deprecated properties either forbidden or
obsolete failed to update some references to these now-forbidden
properties.

This patch fixes these.  It also fixes one typo when defining a
forbidden property, tries to rationalise the use of ${} in comments and
literal property values are placed in single quotes.The SRM description
for QoS plugins has been expanded.

As ReviewBoard was misbehaving, review was via GitHub

Target: master
Acked-by: Gerd Behrmann
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
billing plots: use 'transferred' rather than 'size' for bytes

This bug has gone undiscovered. The plotting for dCache
bytes read and written has been accessing aggregate table
values for fullsize rather than transfersize. While this
is OK for writes, reads will not be correctly represented
(we found this on our CMS Tier1 dcache, where
it was off by a factor of 10).

RELEASE NOTES:

Fixes a bug in the billing plots causing incorrect values for dCache bytes read.

Target: master
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-book: no
Require-notes: yes
Acked-by: Dmitry
Patch: https://rb.dcache.org/r/7290
srm: Fix deadlock like bug

The CellStub refactoring led to more things being executor on the message
delivery thread, the point being to reduce the number of threads messages have
to hop through to reach the real destination. In this SRM this has however lead
to a deadlock like situation during file upload: If the file is aborted by
the client while the upload directory is prepared, the SRM will issue an abort
once the callback from the creation of the upload directory comes back. The
abort will send a message to the pnfs manager from the message delivery thread,
which blocks the entire cell - until the abort times out (so not a true
deadlock).

The proper fix is more elaborate: Eventually I would like all the SRM callbacks
to execute on the request scheduler's executor. This requires quite some more
refactoring, which obviously isn't applicable to stable branches. Hence the
present patch simply puts those callbacks back on a dedicated thread pool -
similar to the old behaviour.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7292/
common: Fix race condition in AtomicCounter unit test

The patch modifies two unit tests. The first slept, although the
race between the two threads did not lead to failing tests (the
test thus was non-deterministic, but not non-determinate).

The second test could fail on certain interleavings of threads.
This is resolved by hooking into the base class at a specific
point to synchronize the execution of the two threads involved.
This too elliminates the sleep from the test.

Finally, the unit test is moved to the correct package and
Maven module.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: no
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7293/
alarms (common, dcache, dcache-webadmin, skel):  version 2.

This combined set of patches provides improvements to and
several new features for the alarm system.  The changes
are rather significant, but should not prevent downgrading
to previous versions. Interaction with version 1 should
be feasible, since what goes over the wire is still
a logback event, though alarm data inconsistencies could result.
If desirable, a back-port to the Golden Release 2.10 would be
possible when 2.11+ has been proven sufficiently stable.

1) common interfaces and properties

Provides a new set of interfaces for the alarms system, along
with revised property definitions.

AlarmDefinitions have been moved out of dcache core and
into commons.

Definitions are a special class of Alarm. Internal
alarms now must belong to the enum PredefinedAlarm.

Severity has been replaced by AlarmPriority.
Interfaces for the implementation of two internal maps
supporting the mapping of Alarms to AlarmPriority
and for custom definitions are provided.

2) DAO and map implementations

Maps: Implements priority map using properties file as
persistent backup. Implements definitions map
as before using JDOM and xml file as persistent
backup. Provides message vehicle for remote
request of the priority map.

DAO: LogEntry has now been entirely decoupled from
both the Alarms interfaces as well as ch.qos.logback
specifics. The Severity field and methods have
been removed. A liquibase changeset to remove
this attribute and its index from the schema
is provided.

3) Logback

Logback server: rather than wrapping the SocketServer
and giving it an entirely independent configuration
(thus preventing alarms from running in any
domain with other services), the server now uses
a Receiver which is a lifecycle component it adds
to the existing configuration. The extra guards in
the alarms.batch file have consequently been removed,
and logback-server.xml has been deleted.

Logback appender: now directly configures from
the relevant dcache properties and adds
the two embedded appenders (history and email).
The part of the appender which glues the logback-specific
event structure to the LogEntry and Alarms interfaces
has been extracted out into a package utility,
LoggingEventConverter. However, there is still
a binding between this appender and datanucleus
inasmuch as the PersistenceManagerFactory remains
injected through Spring.

The way in which an event is classified as an alarm
now is slight different. If the event is Marked
but no type is provided, it will follow the same
branch as unmarked events in trying to find a
regex match from any provided definitions; however,
failing a match, the untyped but marked event is
considered still to be an alarm, and is marked
GENERIC. The new structure also obviates the
need for an receiving filter based on the Alarm marker.

The RemoteMDCFilter no longer maintains its own
static host name lookup, but uses NetworkUtils.

4) Admin and Shell Commands

dCache Shell Commands: some refactoring of code in order to
reuse for admin commands; eliminated
use of URI to express optional source/origin
metadata. Added a new list command to
show names of internally defined alarms.
Add, modify and remove still work as before.

dCache Admin Commands: The send, add/modify/remove (definition)
commands from the shell are also reflected
here, as well as commands to control
priority mappings, and for saving and reloading
both priority and definitions.

5) Embedded Alarms

Changes calls to MarkerFactory to support new API.
Also incorporates changes from discarded patch
https://rb.dcache.org/r/7169/:

    Current trunk now logs the following when a pool starts:

    22 Jul 2014 12:33:52 (PoolManager) [pool_0 PoolManagerPoolUp] Pool pool_0 changed from mode disabled() to disabled(fetch,store,stage,p2p-client,p2p-server)
    22 Jul 2014 12:33:54 (PoolManager) [pool_0 PoolManagerPoolUp] Pool pool_0 is DOWN
    22 Jul 2014 12:33:57 (PoolManager) [pool_0 PoolManagerPoolUp] Pool pool_0 changed from mode disabled(fetch,store,stage,p2p-client,p2p-server) to disabled(store,stage,p2p-client)
    22 Jul 2014 12:33:57 (PoolManager) [pool_0 PoolManagerPoolUp] Pool pool_0 changed from mode disabled(store,stage,p2p-client) to enabled

    The 'Pool pool_0 is DOWN' message was added with patch 7137.

    However, this ALARM seems to capture disabled states which should
    not generate alarms, especially at pool start-up.

    The logic has been refined to limit the alarm to when a pool
    mode has actually changed to either disabled STRICT or DEAD,
    excluding on initialization.

    In particular: I have taken the general alarms out of the
    PoolManagerV5 messageArrived. It makes more sense to distinguish
    etween alarms at the source or origin of the problem.

    Thus:

    The watchdog thread is responsible for the POOL_DOWN
    alarm sent went the heartbeat ping is not returned.
    Specific pool failures on init, access, etc., which
    cause a shutdown raise alarms.
    Initial disabling and shutdown, as well as changes
    in state initiated through a user command, do not
    raise alarms.

6) Webadmin DAO

Changes to webadmin DAO to support modified scheme (without
Severity).

Entry types getter also eliminated since types are now
known via messaging with the alarm service.

7) Webadmin Presentation Layer

Changes to support filtering on priorty in memory,
as it is not part of the persistent DAO. These involve
changes to the AlarmTableProvider comparators and to
the AlarmDisplayService, which refreshes the provider
by sending a message to the alarm service in order to
fetch the alarm/priorty mapping.

The query panel drop-down list now holds the priority
values (rather than Severity), and the auto-complete
combo is based on the listing refreshed from the
alarm service. Hence httpd no longer requires the
presence of any alarms configuration/properties files
locally (though if the service is XML rather than
RDMBS, the store.xml file must still be accessible
locally).

Severity does not appear in the display table, but
the filtering of the query automatically sorts
by level of priority first (by default). Thus
when both alarms and non-alarms are displayed,
first CRITICAL, then HIGH, etc., alarms, and then
simple ERROR, WARN, etc. statements will appear.

8) Removes code which is no longer used.

TESTING DONE:

All previously existing JUnit tests have been modified
and rerun.

system-test built, booted, manually tested.

Integration tests conducted on my desktop (SL 6.3)
and testbed nodes (SL 5.3).  Interactive testing of
the commands were done.  The PoolManager heartbeat
message was also temporarily converted to an alarm.
Larger system testing was conducted using standalone
components.  Stress tests were conducted for both
a singleton alarm service domain (with no other
services) as well as including the alarm service in
another domain (billing and dCache/head node).
The stress test consisted of sending error log events
at approximately 100Hz with an exception trace of
depth 25, with 26 server-side regex definitions
guaranteed not to match the error message but set
to search the entire depth of the trace for each
message.  The service was able to sustain approximately
94Hz over the course of 100000+ events (about 15 minutes).
Writes were simultaneously done while the alarm
service was running inside billing.  No memory
issues were detected.

Target: master
Patch: https://rb.dcache.org/r/7255/
Patch: https://rb.dcache.org/r/7257/
Patch: https://rb.dcache.org/r/7258/
Patch: https://rb.dcache.org/r/7259/
Patch: https://rb.dcache.org/r/7260/
Patch: https://rb.dcache.org/r/7261/
Patch: https://rb.dcache.org/r/7262/
Patch: https://rb.dcache.org/r/7263/
Acked-by: Tigran
Require-notes: yes
Require-book:  yes

I will provide the book chapter changes
and the release notes.
srm: Fix return code and logging on "login errors"

A user recently reported that he got the following error:

ERROR MSG:	[FTS] FTS State [Failed] FTS Retries [0] Reason [SOURCE Failed to get source file size: srm-ifce err: Invalid argument, err: [SE][Ls][SRM_AUTHENTICATION_FAILURE] httpg://srm.ndgf.org:8443/srm/managerv2: Authentication failed (server log contains additional information)] Duration [1]

The server log however did not contain additional information, as that
information was only logged at info (which is disabled by default).

This patch refines the exceptions thrown by related code in different
error cases and updates how this is logged and reported to the client.
Thus the client will get more precise error codes and the server log
will contain more relevant information.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
srm: Log common misconfiguration preventing upload

Whether a transfer protocol is supported is checked in several places:

- At the request level (per requirement in the SRM spec)

- At the TURL computation level (as that's where we know the final path)

- In PutFileRequest (redundantly)

The redundant check in PutFileRequest is removed. The logging for
failed TURL computation is removed - this information is already
registered as a state transition on the job. A new log message is
added that logs common misconfiguration in which the path being
accessed isn't exposed by any of the doors supporting the requested
protocol. This would typically be the upload directory.

Such a misconfiguration now logs:

10 sep. 2014 23:46:18 (SRM-Gerds-MacBook-Pro) [hMA:1:srm2:prepareToPut:-2147482647:-2147482646] No door for [gsiftp] provides access to /upload/eefaa8f5-ea76-460e-b172-57ea8c69eb5f/a.

Target: trunk
Require-notes: yes
Require-book: yes
Request: 2.10
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7294/
srm: Minor refactoring of ls command

The 'ls' and 'ls completed' commands are merged, and the 'ls completed'
command is replaced by 'ls -completed=<n>'.

The command line interface is moved from thed dcache-srm module to the
srm-server module and used for both dCache and the standalone server.

List formatting related methods are moved from the SRM class to
SrmCommandLineInterface.

This patch is in anticipation of further extensions to the list command.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7264/
unittests: increase timeouts and fix race in DiskSpaceAllocatorTest

Various unit tests have timeouts to allow the test to fail if it gets
stuck.  While such tests are short-lived and the existing timeouts work
fine for desktop environments, when testing under heavy load the thread
scheduling within the OS can lead to severe delays and the tests could
fail.  For this reason, the timeouts are bumped up to 1 minute.  Tests
should aim to complete easily within one second (number picked out of
the air), so one minute should allow at least 2 orders of magnitude
leeway.

This patch also removes two unnecessary assertions.  These assertions
add nothing against an existing, earlier test and introduce a race
condition for a possible failure mode.

The test in question has a race in attempting to ensure the testing
thread blocks.  This is hard to fix without rewriting the code being
tested and, since it doesn't affect the test result, the code is left
as-is.

Target: master
Patch: https://rb.dcache.org/r/7308/
Acked-by: Tigran Mkrtchyan
Requires-notes: no
Requires-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
srm: Fix asynchroneous job storage leak

The asynchroneous job storage fails to preserve the force flag when
collapsing several save calls. In case srm.persistence.enable.history
is false (the default), this could lead to incomplete data in the
database. This in case could lead to incomplete cleanup of upload
directories after an SRM restart.

The patch also renames the boolean parameter to something that I find
less confusing.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7307/
srm-client: Fix logback configuration

Ever since common-cli was introduced, the console logback configuration
gets included in the srm package. This causes conflict to barf on startup
as it now got two configuration files (the one from the console package
and its own).

This patch resolves the issue. The patch also removes a couple of
unused and obsolete files from the srm client's conf directory
and removes the conf directory from the class path.

We may eventually port the srm client to use the console package, but
that is left for a future patch.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7311/
dcap: fix interaction with Spacemanager

we have to set subject in message to pool as well, as
space manager processes PoolAcceptFileMessage as well.

Ticket: #8456
Acked-by: Dmitry Litvintsev
Target: master, 2.10
Require-book: no
Require-notes: no