srm: Fix lock congestion

To support SRM_FILE_BUSY, both get and ls requests scan for active
put requests. This causes lock congestion as reading the SURL of
the put request acquires a read lock, while abort and put done calls
are processed with a write lock (and these calls trigger callouts
to pnfsmanager).

The read lock to access the surl is however unneeded as surl is a
final field. This patch removes the read-lock for surl access. The
patch also removes the read-lock on several other final fields, as
well as marking several fields that are not modified as final.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7054/
pinmanager,srm,spacemanager,xrootd: Fix pool monitor expiration

The pool manager state (aka pool monitor) is cached in a number of
other services. A recent change modified the cache logic to trigger
in-thread asynchronous updates rather than dedicated an entire thread
to the update. The Guava cache class is configured to trigger an update
after 20 seconds, however the update is asynchronous, meaning that the
first access after 20 seconds will still get the old data. There is a
total lifetime of 10 minutes, meaning that once the entry is 10 minutes
old, it is purged and a blocking update is done instead (i.e. you
never get data older than 10 minutes).

Unfortunately the cached pool state in pool monitor is considered
invalid after 5 minutes and the pool is then treated as offline. On
a lightly loaded system this causes some strange effects.

The immediate fix is to reduce the max lifetime to 3 minutes (allowing
some slack up to the 5 minute lifetime of the pool data). An async.
update is still triggered after 20 seconds.

Longer term, the pool information should be channelled directly from
pools to the services that need it.

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7055/
xrootd,nfs: Add support for space management

Xrootd and NFS doors fail to include the space manager in pool manager
requests, thus failing to support space management. Even though the
protocols don't have explicit support for space management, they may
still be used with the WriteToken tag or with implicit reservations.

If space manager is disabled, the message flow is unaffected. Even if
space manager is enabled, read requests or writes to unmanaged space
are simply passed through in space manager (which when space manager
is placed in the same domain as pool manager is fairly cheap).

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7049/
Request: 2.9
Request: 2.8
Request: 2.7
dcache-star: create valid xml even on db connection failure

the dcache-star script creates invalid xml output if the connection to
the database failes.
This patch ensures that valid xml is created under those circumstances,
too.

Ticket: 8251
Acked-by: Tigran
Target: master
Require-book: no
Require-notes: no
http: fix multipart response size

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
config: do not set default values for kdc and realm

if vales are not set, java will read them form /etc/krb5.conf

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
system-test: fix web font corruption

A recent patch added bootstrap support, which includes embedded fonts
for icons.  While other packages handled these additional files correctly,
the system-test deployed "filtered" versions of the font files, resulting
in browsers rejecting the fonts and deplaying the characters with a
regular font.

This patch ensures that the web-fonts, CSS and javascript files are not
affected by dCache filtering.

Target: master
Patch: https://rb.dcache.org/r/7059/
Acked-by: Karsten Schwank
Acked-by: Gerd Behrmann
dcache alarms: add properties and fuller documentation for configuring SMTP appender

The instructions in the current logback-server.xml are
incomplete in terms of how to set up the appender to
talk to a normal SMTP server.

This patch is a revision of http://rb.dcache.org/r/6945
which exposes all of the configuration through the
standard dCache property framework.

This means that the patch is more invasive,
with code slightly modified in a few places.

A separate patch adding the fuller settings to
logback-server.xml without property changes
has already been posted for backport and merged
in (2.6, 2.7, 2.8 and 2.9).

Testing: Used new properties to configure send through gmail SMTP
to two fnal email recipients.

Target: master
Require-book: yes
Require-notes: yes
Acked-by: Karsten

RELEASE NOTES:  Fuller documentation for the configuration
of the alarm server's logback appenders (particularly for
email notification) has been provided.  The logback
settings are now fully configurable through the properties
framework.
gplazma-xacml: Add handling of GID returned by GUMS in gPlazma2 XACML plugin

    Patch: https://rb.dcache.org/r/7061/
    Acked-by: Albert Rossi <arossi@fnal.gov>
    Target: trunk
    Require-book: no
    Require-notes: yes

RELEASE NOTES:
gplazma-xacml: Add handling of GID returned by GUMS in gPlazma2 XACML plugin
Fix issue with incorrect determination of whether or not a string is encoded
by replacing URLDecoder with import org.springframework.web.util.UriUtils
as the former does not handle '+' character.

    Ticket: #8356
    Acked-by: Albert Rossi <arossi@fnal.gov>
    Target: trunk
    Request: 2.9
    Request: 2.8
    Request: 2.7
    Request: 2.6
    Require-book: no
    Require-notes: yes

RELEASE NOTES

Fix bug effecting ability to stage files from Enstore if file names
contain '+' character.
Fix ArrayIndexOutOfBoundsException when processing
multiple surls invoking srmget.

        Ticket: #8083
        Acked-by: Christian Bernardt <christian.bernardt@desy.de>
        Target: trunk
        Request: 2.9
        Request: 2.8
        Request: 2.7
        Request: 2.6
	Request: 2.2
        Require-book: no
        Require-notes: yes

    RELEASE NOTES

Fix ArrayIndexOutOfBoundsException when processing
multiple surls invoking srmget. Addresses RT #8083
pool: Fix regressions in new HSM subsystem

The argument order for remove callouts was wrong.

Logging was a bit too quiet.

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7072/
ftp: fix LIST, NLST and MLSD cmd-channel output

A recent patch, committed as b72f2b5283c7a1f3bdd7b7f2c7e5313802a1dff2,
added support for the delayed passive mode.  Unfortunately, it
also broke the LIST, NLST and MLSD output as it omits the 150
response for clients that use active mode.

Although our functional tests failed to detect the problem, Globus
(formally known as GlobusOnline) fails if the 150 message is missing.

This patch adds the missing message, fixing Globus

Request: 1.9
Patch: https://rb.dcache.org/r/7075/
Acked-by: Gerd Behrmann
Acked-by: Dmitry Litvintsev
Target: master
Require-notes: yes
Require-book: no
srm: Cancel upload path if request was aborted

If the request is aborted while pnfs manager creates the
unique upload path, the SRM would fail to release the
upload path.

This patch fixes this problem.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.9
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7077
srm: Undo external operations if the request has expired

Much of our code registers a callback or usses blocking RPC style
when communicating with other services. This code may time out. If
the external operation happens to be successfull anyway, then the
message is discarded unless a message handler exists for that
message.

This patch adds three such message handlers for the srm. It deals
with unique upload path creation, pinning and space reservation.
In each case the resource that was created is removed. Common to
these three requests is that the resource created is uniquely
identified. This is different from, eg, a directory that was
created - we cannot delete the exisence of the directory may
already have been noticed by others.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7078/
ftp: Fix indentation

Target: trunk
Patch: none
ftp: Close passive port on failure

globus-url-copy hangs on proxied passive mode connections for which
the mover is killed. Although the proxy adapter is shut down and an
error is reported on the control channel, the fact that the server
socket for the passive mode is not closed and thus has an unaccepted
connection causes globus-url-copy to hang.

This patch closes the passive port on transfer failure. The mode state
is set to an invalid state to force the client to reset the data
channel using either PORT or PASV (or one of the variations).

I am not request backport further than 2.9 as I expect that we will
soon backport the entire door to get delayed passive and IPv6 functionality
in older releases.

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7079/
nfs: remove double locking

do not use synchronized block and lock at the same time.

Acked-by: Albert Rossi
Target: master
Require-book: no
Require-notes: no
nfs: use new admin command interface

Acked-by: Albert Rossi
Target: master
Require-book: no
Require-notes: no
nfs: add a command to reset pool's device id

device id used to map pool with data server know by client.
In RHEL6 and clones, if sore some reason pools get blacklisted,
the we have to restart door, pool or client. Added command
allowes door to 'forget' the pool and generate new id.

Acked-by: Albert Rossi
Target: master
Require-book: no
Require-notes: no