ssh2: Honor admin.colors.enable

    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by: Gerd
    Target: master
    Require-book: no
    Require-notes: no

Squashed commit of the following:

commit 8e3db9a3f6cd0c9b8ce6f340f67eb3ecf3651163
Merge: 3887dc4 3e02f94
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 18:33:45 2013 +0200

    Merge color fix branch with current master because not all changes got committed with the last push.

    Sorry.

    ssh2: Honor admin.colors.enable
    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by: Gerd
    Target: master
    Require-book: no
    Require-notes: no

commit 3887dc446e7d388659ffb651e5a0a9e1db67f868
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 14:15:39 2013 +0200

    ssh2: Honor admin.colors.enable

    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by: Gerd
    Target: master
    Require-book: no
    Require-notes: no

commit 3e02f94eebb02bddf81441a9dd7ac41344577e74
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 14:15:39 2013 +0200

    Change parameter to no type definition and switch line to above cell related properties

commit 89bd221adafba600ce35c5f103f65a6736c408e6
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 12:22:09 2013 +0200

    Remove line that sets colors from environment param and not from within dCache

commit 0def3e296a3018c173062457b7256a4ac207ef30
Author: Christian Bernardt <christian.bernardt@desy.de>
Date:   Mon Jul 15 11:41:13 2013 +0200

    ssh2: Honor admin.colors.enable

    Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

    Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
    Acked-by:
    Target: trunk
    Require-book: no
    Require-notes: no
Merge branch 'master' of github.com:dCache/dcache
Merge branch 'master' of github.com:dCache/dcache
Minor updates to third party libraries

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5877/
srm: whitespace-only changes to Spring XML file
nfs: start LoginBrokerHandler only if NFSv4.1 is enabled

As NFS v3 and v4 served by the same component, register
NFS door at login broker only if v4.1 is enabled.

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
scripts: don't create pool with deprecated properties

The recent changes to the properties resulted in properties 'name' and
'path' being deprecated in favour of 'pool.name' and 'pool.path'.  Most
references were updated, but a couple were missed.

This patch fixes those missed properties to use the new naming scheme.

Target: master
Patch: http://rb.dcache.org/r/5896/
Acked-by: Tigran Mkrtchyan
Requires-book: no
Requires-notes: no
Fix more deprecated properties

When creating a new pool, a wait-for-files configuration is added. This generated
wait-for-files uses the deprecated property 'path'.  If the admin configures the pool's path
with the new property ('pool.path') then '${pool}' expands to the empty string and the pool
doesn't start.

This patch fixes this by using the new property.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5897/
Acked-by: Tigran Mkrtchyan
srm: Use ISO 8601 format for transition time stamp with millisecond resolution

While debugging some issues at NDGF, I was frustrated by the lack of
precision in the timestamp of SRM transitions. Turns out the timestamps
are stored with millisecond precision, it's just the formatting doesn't
include milliseconds.

This patch switches the format to ISO 8601 compliant timestamps with
milliseconds.

Also turned on transition logging in system-test.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5892/
pool: Add rate limit for pnfsmanager messages

Our admins have pointed out, that using sweeper purge or sweeper free
on a pool can easily cause an outtage in dCache: The huge number of
deregistrations sent to pnfsmanager can create a long queue which causes
timeouts in other services. Indeed, we recently had a surge of FTS
failures due to slow srmLs responses caused by 'sweeper free' calls
on a pool.

This patch addresses this problem by adding a rate limiter to pnfsmanager
requests from a pool. Rather than an adhoc solution in the pool, the patch
extends CellStub with the ability to limit the number of concurrent
requests and the ability to apply a rate limit. Thus any service that makes
use of CellStub can now easily limit how much load it induces on other
services. At the moment this is however only used in pools.

I added support for both limiting the number of concurrent requests and the
rate. According to Little's law, either has the same expressive power in a
stable system, although to convert one to the the other one has to know
the request latency. In an unstable system (consistently higher inflow than
outflow), the two are not equivalent. A rate limiter would keep sending
requests to the remote service at the configured rate, while limiting the
concurrency would lead to the sender being throttled. On the other hand,
a rate limiter can be applied to both RPC style messages and notification
(messages without a reply). Hence both have benefits and drawbacks.

The defaults in CellStub are such that no limit is applied. For pools, a
rate limit of 250 Hz pnfs manager requests is the default.

New property introduced by this patch: pool.service.pnfsmanager.rate.

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5878/
srm: Add failure counters for SRMv2 interface and refine counter formatting

The srm (and many other services) collects stats of requests and failures.
However for the SRMv2 interface failures were not counted. This is rectified
by this patch.

The patch also adjusts the formatting of the request gauges as seen in
the admin shell: The sampling period is printed in seconds rather than
milliseconds (the high resolution is not needed, IMO, and it makes it
hard to read), thousand separators are added to several columns, and a
couple of spelling errors are fixed.

While at it, I removed some dead code in the SRM.

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5895/
doors: fix support for binding to a specific interface

There are a number of problems with how doors handle configuration that describes to which
network interface they should bind, if any.  First, the default value (the empty string) is
not accepted by xrootd and webdav seems to interpret this as "bind to localhost".  Second, the
admin interface (a.k.a. "ssh2") doesn't support binding at all and always listens on all
interfaces.

This patch fixes these problems.  It also changes the place-holder value for wildcard from the
empty string to 'any'.

FOR RELEASE NOTES:

The keyword 'any' is used when specifying a door should listen on any interface (previously it
was the empty string).  The default value has been adjusted accordingly.

Requires-notes: yes
Requires-book: yes
Target: master
Patch: http://rb.dcache.org/r/5898/
Acked-by: Gerd Behrmann
spacemanager,transfermanagers: Don't rely on SRM db properties

Several service have historically piggybacked on the SRM db
settings. Nowadays each service has its own DB properties, however
spacemanager and transfermanagers defines it's properties in
terms of the SRM properties, thus causing an unwanted coupling.

We still want backwards compatibility and thus let spacemanager
and transfermanagers default to the deprecated (legacy) srm
properties, however the new properties should be decoupled. This
structure matches what is already done in pinmanager.

At the same time, both spacemanager and transfermanagers did
not actually respect it's own properties: That's because they
used the JDBC URL defined for the SRM and these didn't make
use of properties like spacemanager.db.host and spacemanager.db.name.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5901/
srm: Drop properties marked obsolete or forbidden in 2.6

We cannot keep them forever and for other services we dropped
old properties in 2.7.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5902/
pnfsmanager: Use pnfsmanager.enable.* for properties that enable features

Changes

pnfsmanager.folding -> pnfsmanager.enable.folding
pnfsmanager.inherit-file-ownership -> pnfsmanager.enable.inherit-file-ownership
pnfsmanager.full-path-permission-check -> pnfsmanager.enable.full-path-permission-check
pnfsmanager.acl -> pnfsmanager.enable.acl

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5903/
SRM: fix ping

SRM includes code that checks if a reponse is successful.  It does
this by invoking the getReturnStatus that Axis creates, via
reflection.

Unfortunately, not all response objects contain this method (the
response from a ping operation is the only example).  The current
code that evaluates whether a response is successful results in a
NoSuchMethodException being thrown for such operations.  This
exception is treated as a failure, resulting in ping no longer
working.

This patch fixes this problem by catching the NoSuchMethodException
and treating such operations as always succeeding.  (Catching the
exception like this is not nice, but seems the least ugly way of
discovering if the response object contains the desired method).

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5904/
Acked-by: Gerd Behrmann
pool: update nfs mover to fit into current design

after last mover code restructuring, NFS mover was adopted
not in the optimal way. This change makes NFS mover to naturally
fit into new design.

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
Patch: http://rb.dcache.org/r/5893/
spacemanager: Fix forwarding when disabled and add fast forward path

Addresses a bug in which PoolIoFileMessage was not forwarded to pool
manager when spacemanager was disabled.

Addresses a performance issue in which spacemanager congestion caused
by heavy write activity slows down read requests, even though read
requests are simply forwarded by pool manager. The issue is that the
forwarding of messages related to a read request happens on a thread
pool (manager by ThreadManager) that is also used by write requests.
This patch addresses the problem by forwarding such message immediately
rather than scheduling them on a different thread pool.

Note that read requests may still suffer from write congestion as
DoorTransferFinished must be intercepted by space manager and we
cannot distinguish between read and write requests without a
database lookup.

Target: trunk
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7902
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5886/
srm: fix file ownership queries

Currently, the SRM relies on down-casting the supplied SRMUser object to an
AuthorizationRecord to discover the uid and gid of this user.  This is used either
when querying whether a file is owned by some user or the user is a member of the
group that owns this file.

Since we recently moved away from AuthorizationRecord and used DcacheUser as the
concrete class that provides SRMUser, the ability to discover uid and gid of a user
was lost.

This patch fixes the problem by downcasting to the correct class.

A problem not fixed by this patch is that group-ownership is checked only against
the user's primary gid (all groups should be checked).  A future patch will fix
this.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5905
Acked-by: Dmitry Litvintsev
Acked-by: Gerd Behrmann
*dcap, *ftp, ssh1: fix error handling

LoginManager is agnostic to the underlying protocol and delegates
this responsibility to some StreamEngine.  The StreamEngine is
instantated via reflection, which may result in an
InvocationTargetException being thrown, which wraps the underlying
cause of the problem.

There are two problems with the current code: first, if the underlying
problem is an Error (e.g. OutOfMemoryError) then this is not
propagated and dCache will try to continue in this ill-defined state;
second, the error message logged is that of the ITE, which does
not include a description of the underlying problem.

This patch fixes these two problems.

Target: master
Request: 2.6
Request: 2.2
Patch: http://rb.dcache.org/r/5906
Acked-by: Tigran Mkrtchyan