pnfsmanager: Fix compatibility with 2.6 pools

2.6 pools send PnfsGetStorageInfoMessage upon issuing the 'rh restore' command.
This message has been removed and thus pool compatibility with 2.6 is broken.
This patch adds a minimal version of that message back to ensure compatibility
with old pools.

Target: trunk
Request: 2.8
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6696/
srm-client: Do not retry on permanent FTP failures

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6705/
Allow retry on transient transfer failures

This patch fixes a minor regression introduced in 2.8 that prevents retry
on a transient error. The patch also removes retry on a permanent error.

Target: trunk
Request: 2.8
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6707/
ftp: Make root path behaviour configurable

Our FTP door is different from all other doors in dCache by
exposing a user-specific root path. This patch introduces
the ftp.root property. It mimics that of webdav.root and
xrootd.root by defining which directory should form the root
of the door. If set to the empty string (which is the default
value), the existing behaviour is maintained.

As before, the user's home directory is used to define the
initial working directory of a new FTP session.

Also fixes a bug that could prevent the user's settings from
being applied if logging in multiple times in the same FTP
session.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6686/
srm: Make FTP TURLs absolute

RFC 1738 defines that FTP URLs are interpreted relative to
the initial working directory of the FTP session, which may
be different from the root. In dCache the user's home directory
is the initial working directory. The TURL generated by the
SRM thus doesn't refer to the correct directory if dCache has
been configured with home directories.

The spec states that an absolute path should be encoded with
starting as %2F. That's because the spec requires the client
to split the path at / and use each segment in a CWD command
(plain FTP doesn't have a concept of paths; path names are
an extension defined in RFC 3659). Ie gsiftp://localhost//foo/bar
just means to do

CWD
CWD foo

and then STOR or RETR bar. gsiftp://localhost/%2Ffoo/bar means

CD /foo

and then STOR or RETR bar.

However our own srmcp client doesn't deal with the URI encoding
and just sends %2F as part of the directory name.

Globus Toolkit (globus-url-copy) doesn't follow the spec either,
since it prefixes the path in gsiftp://localhost/foo/bar with an
extra slash, thus with Globus it isn't possible to refer to
something relative to the user's home directory.

Both srmcp and globus-url-copy do not interpret the empty
path element according to the spec since they just pass the
entire path on to the server. Ie. for gsiftp://localhost//foo/bar,
srmcp sends /foo to the server, while globus-url-copy sends
//foo. At the FTP level the double slash is undefined, but our
FTP door collapses it into a single slash.

The conclusion from the above is that we cannot use the spec
compliant encoding as it would break compatibility with our
own client. We can prefix the path with an extra slash, which
works with both Globus and srmcp. It will not work with a
spec compliant client, however the current code doesn't work
with a spec complaint client anyway, so the situation doesn't
become worse.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6685/
spacemanager: Fix database sequence initialization

The default range for sequences starts at 1. This caused failures
with the schema changeset when migrating from a database where the
largest id was smaller than 1.

Target: trunk
Request: 2.8
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6711/
pool: Fix pf command regression

A recent patch changed a number of variables and fields from String to
FsPath. Inadvertently the return message of the 'pf' command in the pool
was changed to, leading to Serialization errors.

Target: trunk
Request: 2.8
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6713/
system-test: Explicitly disable GridFTP 2 in tests

Different versions of globus-url-copy have different defaults for
GridFTP 2 support. This patch adds explicit arguments to the test
to disable GridFTP 2 in tests that are supposed to test GridFTP 1.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6714/
xrootd,httpd,pool: Upgrade to Netty 3.9.0

Despite the change in minor version, this is mostly a
bug fix release.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6717/
Upgrade third party libraries

Note that logback has been upgraded too. There was a comment
noting that we did not upgrade earlier due to issues with remote
logging, but there was no link to a specific bug report and thus
it is difficult to know it the issue is fixed. Logback release
notes do however say that issues related to remote logging have
been fixed.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6716/
core: update PoolPassiveIoFileMessage to provide boot verifier

update nfs code to detect pool reboots based on verifier if available.

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
nfs: do not label pnfsid and client ip in NDC

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
http: fix mocking of Netty in unit-tests

The recently updated Netty brings new internal behaviour.  Unfortunately,
the mocking necessary to unit-test pool behaviour means these internals
are exposed as sufficiently mocked Netty is needed.

This patch fixes the problem by adding a mocked HttpHeaders object
that Netty now requires.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/6718/
Acked-by: Tigran Mkrtchyan
logback: revert to 1.0.12

Update of logback from 1.0.12 to 1.1.1 causes the alarmserver to throw this exception.  The same configuration works fine in 1.0.12.

java.lang.NullPointerException
        at ch.qos.logback.core.subst.Parser.T(Parser.java:75)
        at ch.qos.logback.core.subst.Parser.E(Parser.java:50)
        at ch.qos.logback.core.subst.Parser.parse(Parser.java:46)
        at ch.qos.logback.core.subst.NodeToStringTransformer.tokenizeAndParseString(NodeToStringTransformer.java:55)
        at ch.qos.logback.core.subst.NodeToStringTransformer.handleVariable(NodeToStringTransformer.java:96)
        at ch.qos.logback.core.subst.NodeToStringTransformer.compileNode(NodeToStringTransformer.java:72)
        at ch.qos.logback.core.subst.NodeToStringTransformer.transform(NodeToStringTransformer.java:60)
        at ch.qos.logback.core.subst.NodeToStringTransformer.substituteVariable(NodeToStringTransformer.java:48)
        at ch.qos.logback.core.util.OptionHelper.substVars(OptionHelper.java:117)
        at ch.qos.logback.core.joran.spi.InterpretationContext.subst(InterpretationContext.java:159)
        at ch.qos.logback.core.joran.action.NestedBasicPropertyIA.body(NestedBasicPropertyIA.java:87)
        at ch.qos.logback.core.joran.spi.Interpreter.callBodyAction(Interpreter.java:295)
        at ch.qos.logback.core.joran.spi.Interpreter.characters(Interpreter.java:175)
        at ch.qos.logback.core.joran.spi.EventPlayer.play(EventPlayer.java:57)
        at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:149)
        at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:135)
        at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
        at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:76)
        at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:68)
        at org.dcache.alarms.server.LogEntryServerWrapper.startUp(LogEntryServerWrapper.java:181)
        ...

Target: master
Patch: http://rb.dcache.org/r/6719
Require-notes: no
Require-book: no
Acked-by: Paul
ftp: add support for HELP and SITE HELP commands

Both the ncftp and Globus clients parse the output of HELP commands
to determine which additional commands are available.  Both use
broadly the same approach: each response line is searched for
references to any of the SITE commands the client understands.  The
search is a simple substring search, looking for the command in
upper case, excluding the initial 'SITE '.  The ncftp client parses
the output of 'HELP SITE' and the Globus client parses the output
of 'SITE HELP'.

This patch is to allow support for additional commands.

Target: master
Patch: http://rb.dcache.org/r/6701
Acked-by: Gerd Behrmann
ftp: add logging to access log file

Add support for logging FTP interactions to the access log.  Each
client command is logged along with the corresponding reply from
dCache.  Base64-encoded context-handling and user's passwords are
not logged.  Encoded commands and replies are indicated as
encrypted and the plain-text version is logged.

Requires-notes: yes
Requires-book: yes
Target: master
Patch: http://rb.dcache.org/r/6584/
Acked-by: Gerd Behrmann
ftp: fix printing of UNIX.Mode fact

The format of the UNIX.Mode fact is not documented.  The ncftp client
expacts that a value starting '0' is in octal and other values are in
decimal.  The Globus server follows this convention by returning an
octal number that starts '0'.

Currently dCache returns an octal number without the leading '0'.
This patch fixes this by appending the '0'.

Target: master
Patch: http://rb.dcache.org/r/6639/
Acked-by: Tigran Mkrtchyan
srm: Fix compatibility with PostgreSQL 8.4

PostgreSQL 8.4 doesn't support the standards compliant syntax
used in a recent change.

This patch moves the logic into dCache instead.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/6723/
http: fix multipart reply

fixes compatibility with libneon ( davix and ROOT )

after second reading of http://tools.ietf.org/html/rfc2616
each Range block should not contain Content-length, but
complete requests must.

Acked-by: Gerd Behrmann
Target: master, 2.8, 2.7, 2.6
Require-book: no
Require-notes: no
alarms: add logging and peaceful exit for logback startup errors

If there is an runtime exception from the configuration of the
SimpleSocketServer it is currently not handled, causing
repeated restart of the domain.

This patch catches and logs the exception, but does not rethrow it.

Target: master
Patch: http://rb.dcache.org/r/6720
Require-notes: no
Require-book: no
Acked-by: Gerd