dcache alarms: add properties and fuller documentation for configuring SMTP appender

The instructions in the current logback-server.xml are
incomplete in terms of how to set up the appender to
talk to a normal SMTP server.

This patch is a revision of http://rb.dcache.org/r/6945
which exposes all of the configuration through the
standard dCache property framework.

This means that the patch is more invasive,
with code slightly modified in a few places.

A separate patch adding the fuller settings to
logback-server.xml without property changes
has already been posted for backport and merged
in (2.6, 2.7, 2.8 and 2.9).

Testing: Used new properties to configure send through gmail SMTP
to two fnal email recipients.

Target: master
Require-book: yes
Require-notes: yes
Acked-by: Karsten

RELEASE NOTES:  Fuller documentation for the configuration
of the alarm server's logback appenders (particularly for
email notification) has been provided.  The logback
settings are now fully configurable through the properties
framework.
gplazma-xacml: Add handling of GID returned by GUMS in gPlazma2 XACML plugin

    Patch: https://rb.dcache.org/r/7061/
    Acked-by: Albert Rossi <arossi@fnal.gov>
    Target: trunk
    Require-book: no
    Require-notes: yes

RELEASE NOTES:
gplazma-xacml: Add handling of GID returned by GUMS in gPlazma2 XACML plugin
Fix issue with incorrect determination of whether or not a string is encoded
by replacing URLDecoder with import org.springframework.web.util.UriUtils
as the former does not handle '+' character.

    Ticket: #8356
    Acked-by: Albert Rossi <arossi@fnal.gov>
    Target: trunk
    Request: 2.9
    Request: 2.8
    Request: 2.7
    Request: 2.6
    Require-book: no
    Require-notes: yes

RELEASE NOTES

Fix bug effecting ability to stage files from Enstore if file names
contain '+' character.
Fix ArrayIndexOutOfBoundsException when processing
multiple surls invoking srmget.

        Ticket: #8083
        Acked-by: Christian Bernardt <christian.bernardt@desy.de>
        Target: trunk
        Request: 2.9
        Request: 2.8
        Request: 2.7
        Request: 2.6
	Request: 2.2
        Require-book: no
        Require-notes: yes

    RELEASE NOTES

Fix ArrayIndexOutOfBoundsException when processing
multiple surls invoking srmget. Addresses RT #8083
pool: Fix regressions in new HSM subsystem

The argument order for remove callouts was wrong.

Logging was a bit too quiet.

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7072/
ftp: fix LIST, NLST and MLSD cmd-channel output

A recent patch, committed as b72f2b5283c7a1f3bdd7b7f2c7e5313802a1dff2,
added support for the delayed passive mode.  Unfortunately, it
also broke the LIST, NLST and MLSD output as it omits the 150
response for clients that use active mode.

Although our functional tests failed to detect the problem, Globus
(formally known as GlobusOnline) fails if the 150 message is missing.

This patch adds the missing message, fixing Globus

Request: 1.9
Patch: https://rb.dcache.org/r/7075/
Acked-by: Gerd Behrmann
Acked-by: Dmitry Litvintsev
Target: master
Require-notes: yes
Require-book: no
srm: Cancel upload path if request was aborted

If the request is aborted while pnfs manager creates the
unique upload path, the SRM would fail to release the
upload path.

This patch fixes this problem.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.9
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7077
srm: Undo external operations if the request has expired

Much of our code registers a callback or usses blocking RPC style
when communicating with other services. This code may time out. If
the external operation happens to be successfull anyway, then the
message is discarded unless a message handler exists for that
message.

This patch adds three such message handlers for the srm. It deals
with unique upload path creation, pinning and space reservation.
In each case the resource that was created is removed. Common to
these three requests is that the resource created is uniquely
identified. This is different from, eg, a directory that was
created - we cannot delete the exisence of the directory may
already have been noticed by others.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7078/
ftp: Fix indentation

Target: trunk
Patch: none
ftp: Close passive port on failure

globus-url-copy hangs on proxied passive mode connections for which
the mover is killed. Although the proxy adapter is shut down and an
error is reported on the control channel, the fact that the server
socket for the passive mode is not closed and thus has an unaccepted
connection causes globus-url-copy to hang.

This patch closes the passive port on transfer failure. The mode state
is set to an invalid state to force the client to reset the data
channel using either PORT or PASV (or one of the variations).

I am not request backport further than 2.9 as I expect that we will
soon backport the entire door to get delayed passive and IPv6 functionality
in older releases.

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7079/
nfs: remove double locking

do not use synchronized block and lock at the same time.

Acked-by: Albert Rossi
Target: master
Require-book: no
Require-notes: no
nfs: use new admin command interface

Acked-by: Albert Rossi
Target: master
Require-book: no
Require-notes: no
nfs: add a command to reset pool's device id

device id used to map pool with data server know by client.
In RHEL6 and clones, if sore some reason pools get blacklisted,
the we have to restart door, pool or client. Added command
allowes door to 'forget' the pool and generate new id.

Acked-by: Albert Rossi
Target: master
Require-book: no
Require-notes: no
pool: log with error if checksum scanner gets an IO error

Ticket: #8181
Acked-by: Karsten Schwank
Target: master, 2.9, 2.8, 2.7, 2.6
Require-book: no
Require-notes: no
webadmin: Simplify titel initialization of base page

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7089/
Intern several strings on message deserialization

Serialization/deserialization of messages with an object stream reset
in between messages has the side effect of completely decoupling those
object graphs. Thus elements (like strings) that were shared before,
are no longer shared when deserializing. This wastes a lot of memory
for string fields with a small universe of possible values.

This patch adds code to internalize some string fields during deserialization.
This is only done for fields for which the value would typically come from
a relatively small domain.

The particular messages touched by this patch related to the active
transfers page in webadmin and the patch thus reduces the memory footprint
of that information.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7088/
webadmin: Reimplement active transfers page

Reimplements most of the active transfers page of webadmin, including the
information gathering logic, transfer cancellation, pagination, sorting,
filtering, etc.

Queries to gather information about active transfers are no longer
sequential: Both queries to login brokers, doors, and pools are
done concurrently. This should lead to much more up to date information
on large and busy systems.

Transfer cancellation is no longer done per pool. Instead all selected
transfers are aborted concurrently.

The amount of state copied into each view is much smaller. Rather than
repeatedly copying the transfer state into ActiveTransferBean,
ActiveTransfersBean is used as the data model in the MoverDAO.

The entire list of active transfers is no longer copied into each
instance of the page. This copy could quickly lead to problems to
memory consumption, in particular when several clients opened the
page at the same time. Instead the shared list of active transfers
is exposed as a data provider. The data in this provider is
rendered by a Wicket DataTable component.

Pagenation, sorting and filtering are implemented server side. This
has the benefit that when a very large number of transfers are ongoing,
not all of them need to be rendered at once. Sorting and filtering
is pagenation aware. Since these things are implemented server side,
the pagenation, sorting, filtering and selection state survives
automatic form reloading.

The selection state has been decoupled from the data model, meaning
that there is no need for the SelectableWrapper hack.

I followed Wicket recomendations and did *not* introduce a separte panel.
It is easy enough to refactor once the same view is needed somewhere
else. Until then a separate panel just clutters the code.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7084/
cells: remove System time dependency in unit-test

The CellNameServiceRegistryTests class provides unit-tests for the
CellNameServiceReigstry.  One of these tests verifies that expired
well-known cells registration is removed successfully, leaving the
existing cells intact.

This patch refacts CellNameServiceRegistry slighly to allow a
subclass to adjust its concept of time.  This is used to provide a
reliable simulated flow of time in the unit-tests, removing the
need for a sleep statement.

Target: master
Patch: https://rb.dcache.org/r/7090/
Acked-by: Tigran Mkrtchyan
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
webadmin: fix regression in alarm deletion filter

Patch 6344 introduced a regression in the webadmin alarms
management which prohibits the deletion of entries.

See http://rb.dcache.org/r/6344.

which forced a range limit on the number of
entries returned by a query.  JDO does not allow
a range to be specified this way on a deletion query,
but this case was overlooked.

This patch remedies the oversight.

Testing:

Deployed with patch; successfully deleted checked entries from table.

Target: master
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-book: no
Require-notes: yes
Acked-by: Karsten
Acked-by: Paul

RELEASE NOTES:  Fixes a regression in the webadmin alarms
page preventing the deletion of entries from the alarm
table.
dcache-webadmin: fix pool queue plot grid placeholder image

The pool queue plot grid is computed to find the best square fit up to 8 X 8 of plots.
This means that there will remainders (blanks) for all but the square cases.
Previously the placeholder was an empty symbol denoting no pool, but this particular
image was not the right size and at any rate was rather ugly.

The patch replaces the previous image with a simple blank image the same size as the plots.

Target:  master
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: no
Require-book: no
Acked-by: Dmitry