srm: Use ISO 8601 format for transition time stamp with millisecond resolution

While debugging some issues at NDGF, I was frustrated by the lack of
precision in the timestamp of SRM transitions. Turns out the timestamps
are stored with millisecond precision, it's just the formatting doesn't
include milliseconds.

This patch switches the format to ISO 8601 compliant timestamps with
milliseconds.

Also turned on transition logging in system-test.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5892/
pool: Add rate limit for pnfsmanager messages

Our admins have pointed out, that using sweeper purge or sweeper free
on a pool can easily cause an outtage in dCache: The huge number of
deregistrations sent to pnfsmanager can create a long queue which causes
timeouts in other services. Indeed, we recently had a surge of FTS
failures due to slow srmLs responses caused by 'sweeper free' calls
on a pool.

This patch addresses this problem by adding a rate limiter to pnfsmanager
requests from a pool. Rather than an adhoc solution in the pool, the patch
extends CellStub with the ability to limit the number of concurrent
requests and the ability to apply a rate limit. Thus any service that makes
use of CellStub can now easily limit how much load it induces on other
services. At the moment this is however only used in pools.

I added support for both limiting the number of concurrent requests and the
rate. According to Little's law, either has the same expressive power in a
stable system, although to convert one to the the other one has to know
the request latency. In an unstable system (consistently higher inflow than
outflow), the two are not equivalent. A rate limiter would keep sending
requests to the remote service at the configured rate, while limiting the
concurrency would lead to the sender being throttled. On the other hand,
a rate limiter can be applied to both RPC style messages and notification
(messages without a reply). Hence both have benefits and drawbacks.

The defaults in CellStub are such that no limit is applied. For pools, a
rate limit of 250 Hz pnfs manager requests is the default.

New property introduced by this patch: pool.service.pnfsmanager.rate.

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5878/
srm: Add failure counters for SRMv2 interface and refine counter formatting

The srm (and many other services) collects stats of requests and failures.
However for the SRMv2 interface failures were not counted. This is rectified
by this patch.

The patch also adjusts the formatting of the request gauges as seen in
the admin shell: The sampling period is printed in seconds rather than
milliseconds (the high resolution is not needed, IMO, and it makes it
hard to read), thousand separators are added to several columns, and a
couple of spelling errors are fixed.

While at it, I removed some dead code in the SRM.

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5895/
doors: fix support for binding to a specific interface

There are a number of problems with how doors handle configuration that describes to which
network interface they should bind, if any.  First, the default value (the empty string) is
not accepted by xrootd and webdav seems to interpret this as "bind to localhost".  Second, the
admin interface (a.k.a. "ssh2") doesn't support binding at all and always listens on all
interfaces.

This patch fixes these problems.  It also changes the place-holder value for wildcard from the
empty string to 'any'.

FOR RELEASE NOTES:

The keyword 'any' is used when specifying a door should listen on any interface (previously it
was the empty string).  The default value has been adjusted accordingly.

Requires-notes: yes
Requires-book: yes
Target: master
Patch: http://rb.dcache.org/r/5898/
Acked-by: Gerd Behrmann