pool: Use FireAndForgetTask instead of custom class

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6676/
pool: Fix logging context for thread pool

Fixes the logging context in a thread pool used by migration tasks,
pool to pool tasks, and sticky bit expiration.

Target: trunk
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6675/
billing: Initialize all fields in billing records

Plenty of billing records have fields that are logged as 'unknown' due to
missing values. This patch injects those missing values.

I added file path to PoolIoFileMessage to allow the pool to inject the path
into the billing record. Although the paht is known as part of StorageInfo,
StorageInfo is the wrong place to store the path: The path is not an attribute
of a file. A file can have many paths pointing to it.  A path is rather an
attribute of a particular access (the file is accessed through a path) and thus
the path should be part of the request messages. For the same reason a path
should not become an attribute in FileAttributes.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6689/
Convert tabs to spaces

Patch: none
srm-client: Improve startup time

The patch cleans up the srm script and adds the '-XX:+TieredCompilation
-XX:TieredStopAtLevel=1' java options for improved startup speed. On my laptop
this has reduced the time required to run test --srm against system-test with
approximately 40 seconds.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6692/
Improve startup speed of tools

The options for client utilities defaulted to running the JVM in interpreted
mode. This patch steps up the compilation level to allow simple JIT
compilation. This removes more than 1 second from starting the chimera utility
on my laptop, and more than 35 seconds from the build time of the system-test
module.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6693/
system-test: Enabled tiered compilation

This change improves startup speed for the dCacheDomain in system-test setups.
Although dCache in general is not sensitive to startup speed, system-test is
restarted many times during development and startup has an effect on developer
productivity.

Since the process isn't necessarily short lived like our client tools, I did
not limit the compilation level.

On my laptop this cut about 5 seconds from process start to the last pool being
enabled.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6694/
ftp: add support for the CREATE fact

Now that dCache supports querying the creation time of a file, we
can support the CREATE fact in FTP door.

Target: master
Patch: http://rb.dcache.org/r/6695/
Acked-by: Gerd Behrmann
ssh1: Fix typo in service batch file

The typo prevented the use of a history file.

Target: trunk
Request: 2.8
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6697/
ssh1: Fix session instantiation

A recent regression prevented ssh1 login due to a class signature mismatch.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6698/
srm: Avoid duplicate key violations on startup

Postgresql logs warnings about duplicate key violations on startup. This patch
alters the schema initiatlization to avoid such key violations.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6700/
chimera: Avoid integrity constraint violations on registering locations

When running 'pnfs register' on a pool to ensure that all files are registered,
PostgreSQL logs a warning about every file that was already registered:

2014-03-08 14:36:30 CET DETAIL:  Key (ipnfsid, itype, ilocation)=(000096B89DDF7E7E432A8CB4ED0F28543F69, 1, pool_1) already exists.
2014-03-08 14:36:30 CET STATEMENT:  INSERT INTO t_locationinfo VALUES($1,$2,$3,$4,$5,$6,$7)

More over, the transaction rollbacks appear to generate quite a lot of churn
between our master and slave instances.

This patch using a trick we already use when inserting new names in t_dirs:
Only insert when there is no existing record with the same key.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6699/
ftp: Give io thread pool a name

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6702/
ftp: Fix race condition in command dispatching

Fixes a race condition that lead to hanging FTP sessions.

A FutureTask is not thread safe in the sense that it cannot be
submitted concurrently for several executions.

I must admit, I have no idea why the code was like it was. The future
must have been part of an earlier revision of the code.

Target: trunk
Request: 2.8
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6704/
pnfsmanager: Fix compatibility with 2.6 pools

2.6 pools send PnfsGetStorageInfoMessage upon issuing the 'rh restore' command.
This message has been removed and thus pool compatibility with 2.6 is broken.
This patch adds a minimal version of that message back to ensure compatibility
with old pools.

Target: trunk
Request: 2.8
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6696/
srm-client: Do not retry on permanent FTP failures

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6705/
Allow retry on transient transfer failures

This patch fixes a minor regression introduced in 2.8 that prevents retry
on a transient error. The patch also removes retry on a permanent error.

Target: trunk
Request: 2.8
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6707/
ftp: Make root path behaviour configurable

Our FTP door is different from all other doors in dCache by
exposing a user-specific root path. This patch introduces
the ftp.root property. It mimics that of webdav.root and
xrootd.root by defining which directory should form the root
of the door. If set to the empty string (which is the default
value), the existing behaviour is maintained.

As before, the user's home directory is used to define the
initial working directory of a new FTP session.

Also fixes a bug that could prevent the user's settings from
being applied if logging in multiple times in the same FTP
session.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6686/
srm: Make FTP TURLs absolute

RFC 1738 defines that FTP URLs are interpreted relative to
the initial working directory of the FTP session, which may
be different from the root. In dCache the user's home directory
is the initial working directory. The TURL generated by the
SRM thus doesn't refer to the correct directory if dCache has
been configured with home directories.

The spec states that an absolute path should be encoded with
starting as %2F. That's because the spec requires the client
to split the path at / and use each segment in a CWD command
(plain FTP doesn't have a concept of paths; path names are
an extension defined in RFC 3659). Ie gsiftp://localhost//foo/bar
just means to do

CWD
CWD foo

and then STOR or RETR bar. gsiftp://localhost/%2Ffoo/bar means

CD /foo

and then STOR or RETR bar.

However our own srmcp client doesn't deal with the URI encoding
and just sends %2F as part of the directory name.

Globus Toolkit (globus-url-copy) doesn't follow the spec either,
since it prefixes the path in gsiftp://localhost/foo/bar with an
extra slash, thus with Globus it isn't possible to refer to
something relative to the user's home directory.

Both srmcp and globus-url-copy do not interpret the empty
path element according to the spec since they just pass the
entire path on to the server. Ie. for gsiftp://localhost//foo/bar,
srmcp sends /foo to the server, while globus-url-copy sends
//foo. At the FTP level the double slash is undefined, but our
FTP door collapses it into a single slash.

The conclusion from the above is that we cannot use the spec
compliant encoding as it would break compatibility with our
own client. We can prefix the path with an extra slash, which
works with both Globus and srmcp. It will not work with a
spec compliant client, however the current code doesn't work
with a spec complaint client anyway, so the situation doesn't
become worse.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6685/
spacemanager: Fix database sequence initialization

The default range for sequences starts at 1. This caused failures
with the schema changeset when migrating from a database where the
largest id was smaller than 1.

Target: trunk
Request: 2.8
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6711/