spacemanager: Fix database sequence initialization

The default range for sequences starts at 1. This caused failures
with the schema changeset when migrating from a database where the
largest id was smaller than 1.

Target: trunk
Request: 2.8
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6711/
pool: Fix pf command regression

A recent patch changed a number of variables and fields from String to
FsPath. Inadvertently the return message of the 'pf' command in the pool
was changed to, leading to Serialization errors.

Target: trunk
Request: 2.8
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6713/
system-test: Explicitly disable GridFTP 2 in tests

Different versions of globus-url-copy have different defaults for
GridFTP 2 support. This patch adds explicit arguments to the test
to disable GridFTP 2 in tests that are supposed to test GridFTP 1.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6714/
xrootd,httpd,pool: Upgrade to Netty 3.9.0

Despite the change in minor version, this is mostly a
bug fix release.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6717/
Upgrade third party libraries

Note that logback has been upgraded too. There was a comment
noting that we did not upgrade earlier due to issues with remote
logging, but there was no link to a specific bug report and thus
it is difficult to know it the issue is fixed. Logback release
notes do however say that issues related to remote logging have
been fixed.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6716/
core: update PoolPassiveIoFileMessage to provide boot verifier

update nfs code to detect pool reboots based on verifier if available.

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
nfs: do not label pnfsid and client ip in NDC

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
http: fix mocking of Netty in unit-tests

The recently updated Netty brings new internal behaviour.  Unfortunately,
the mocking necessary to unit-test pool behaviour means these internals
are exposed as sufficiently mocked Netty is needed.

This patch fixes the problem by adding a mocked HttpHeaders object
that Netty now requires.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/6718/
Acked-by: Tigran Mkrtchyan
logback: revert to 1.0.12

Update of logback from 1.0.12 to 1.1.1 causes the alarmserver to throw this exception.  The same configuration works fine in 1.0.12.

java.lang.NullPointerException
        at ch.qos.logback.core.subst.Parser.T(Parser.java:75)
        at ch.qos.logback.core.subst.Parser.E(Parser.java:50)
        at ch.qos.logback.core.subst.Parser.parse(Parser.java:46)
        at ch.qos.logback.core.subst.NodeToStringTransformer.tokenizeAndParseString(NodeToStringTransformer.java:55)
        at ch.qos.logback.core.subst.NodeToStringTransformer.handleVariable(NodeToStringTransformer.java:96)
        at ch.qos.logback.core.subst.NodeToStringTransformer.compileNode(NodeToStringTransformer.java:72)
        at ch.qos.logback.core.subst.NodeToStringTransformer.transform(NodeToStringTransformer.java:60)
        at ch.qos.logback.core.subst.NodeToStringTransformer.substituteVariable(NodeToStringTransformer.java:48)
        at ch.qos.logback.core.util.OptionHelper.substVars(OptionHelper.java:117)
        at ch.qos.logback.core.joran.spi.InterpretationContext.subst(InterpretationContext.java:159)
        at ch.qos.logback.core.joran.action.NestedBasicPropertyIA.body(NestedBasicPropertyIA.java:87)
        at ch.qos.logback.core.joran.spi.Interpreter.callBodyAction(Interpreter.java:295)
        at ch.qos.logback.core.joran.spi.Interpreter.characters(Interpreter.java:175)
        at ch.qos.logback.core.joran.spi.EventPlayer.play(EventPlayer.java:57)
        at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:149)
        at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:135)
        at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
        at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:76)
        at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:68)
        at org.dcache.alarms.server.LogEntryServerWrapper.startUp(LogEntryServerWrapper.java:181)
        ...

Target: master
Patch: http://rb.dcache.org/r/6719
Require-notes: no
Require-book: no
Acked-by: Paul
ftp: add support for HELP and SITE HELP commands

Both the ncftp and Globus clients parse the output of HELP commands
to determine which additional commands are available.  Both use
broadly the same approach: each response line is searched for
references to any of the SITE commands the client understands.  The
search is a simple substring search, looking for the command in
upper case, excluding the initial 'SITE '.  The ncftp client parses
the output of 'HELP SITE' and the Globus client parses the output
of 'SITE HELP'.

This patch is to allow support for additional commands.

Target: master
Patch: http://rb.dcache.org/r/6701
Acked-by: Gerd Behrmann
ftp: add logging to access log file

Add support for logging FTP interactions to the access log.  Each
client command is logged along with the corresponding reply from
dCache.  Base64-encoded context-handling and user's passwords are
not logged.  Encoded commands and replies are indicated as
encrypted and the plain-text version is logged.

Requires-notes: yes
Requires-book: yes
Target: master
Patch: http://rb.dcache.org/r/6584/
Acked-by: Gerd Behrmann
ftp: fix printing of UNIX.Mode fact

The format of the UNIX.Mode fact is not documented.  The ncftp client
expacts that a value starting '0' is in octal and other values are in
decimal.  The Globus server follows this convention by returning an
octal number that starts '0'.

Currently dCache returns an octal number without the leading '0'.
This patch fixes this by appending the '0'.

Target: master
Patch: http://rb.dcache.org/r/6639/
Acked-by: Tigran Mkrtchyan
srm: Fix compatibility with PostgreSQL 8.4

PostgreSQL 8.4 doesn't support the standards compliant syntax
used in a recent change.

This patch moves the logic into dCache instead.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/6723/
http: fix multipart reply

fixes compatibility with libneon ( davix and ROOT )

after second reading of http://tools.ietf.org/html/rfc2616
each Range block should not contain Content-length, but
complete requests must.

Acked-by: Gerd Behrmann
Target: master, 2.8, 2.7, 2.6
Require-book: no
Require-notes: no
pnfsmanager: Add support for temporary upload paths

This patch adds operations to pnfs manager for creating, commiting and
cancelling temporary upload paths. Such paths can be used to write a new file
to dCache without exposing the existence of the file untilt the upload
completed. A temporary upload path is revocable. These features make it ideal
for generating unique write TURLs for use with SRM, although other use cases
exist (eg. xrootd POSC semantics could be achieved using a temporary upload
path).

How to construct a temporary upload path is a detail of the name space provider
implementation. ChimeraNameSpaceProvider uses a dedicated upload directory (the
default value being /upload) with mode 0711 to create a temporary directory
with a cryptographically strong UUID. Tags of the real upload directory are
copied to the temporary directory, thus ensuring that the file gets the right
storage class, access latency, retention policy, etc. While creating the upload
path, an optional expected file size, access latency, retention policy and
space token can be injected. These too are stored as tags in the temporary
upload directory.

Chimera is extended to allow a directory with a set of tags to be created
without copying the parent tags. Unfortunately the Chimera postgresql driver
copies the tags in a trigger, which makes that optimization ineffective. One
could argue that this functionality doesn't belong in a trigger and instead
should have been implemnted with a stored procedure for creating directories
with inherited tags - the driver could then have called the stored procedure
explicitly rather than relying on the trigger. This is left as a future
optimization.

The actual upload to the temporary path is a regular upload, with the exception
that the expected file size and real path are injeced into the CreateFile reply
from pnfs manager.

The upload directory is automatically created the first time this feature is
used. A site not using a protocl relying on unique upload paths will never see
the upload directory.

Since doors may export a different directory as their name space root, the
unqiue write paths are only useable if the upload directory is visible through
the doors. A non-default root path is a primary reason for changing the default
upload path. A typical value may be /pnfs/example.org/data/upload, with
example.org being the site domain name. This can be adjusted by setting
dcache.upload-directory.

Since doors only allow access to a user's root directory, doors have been
extended with an authorization exception allowing access to the upload
directory. Thus a non-default upload directory has to be configured on both
pnfs manager and all doors.

Ownership of the temporary upload directory mimics that of the file that is to
be created. The temporary upload path is thus writable to the same user that
requested the path to be created. One may imagine refining this in the future
to allow anonymous protocols to be used for writing to the upload path (eg to
create single use write URIs that one can use with protocols without
authorization, or that one can give to somebody else to upload a file to [a
form of delegation]).

The changes were originally part of the unique write TURL patch, but have been
split out for easier reviewing. The code in the patch is mostly unused at this
stage.

Target: trunk
Require-notes: yes
Require-book: yes
srm: Generate unique write TURLs

The patch adds unique write TURLs for SRM. Unique TURLs are generated by
asking pnfs manager to create a temporary upload path. After upload, pnfs
manager commits the new file to its final name and location.

Since access latency, retention policy, size and space token are now stored in
the name space as part of the temporary upload path, there is no longer any
need to create a file reservation in space manager bound to the path.  Besides
simplifying the control flow, this solves the problem that the fields in
PutFileRequest that kept the state about whether we marked space as being used
were not persistent. This was a cause of the "Already have 1 record(s)" error
message.

Implicit space reservations as performed by the SRM are no longer needed and
srm.enable.space-reservation.implicit is obsolete. Instead space manager has to
be configured to allow uploads to link groups without prior reservations.  This
is achived using the spacemanager.enable.unreserved-uploads-to-linkgroups
setting.  The setting spacemanager.enable.reserve-space-for-non-srm-transfers
is deprecated. The property srm.enable.space-reservation.strict is obsolete
too.

Since there is a risk of lost notification from PnfsManager to SpaceManager,
SpaceManager has gained code to check whether name space entries of files in
the TRANSFERRING state still exists. Such checks should be done for all states,
however the risk is the biggest for files in TRANSFERRING.

The AbstractStorageElement interface is modified in several ways. The most
obvious change is that putDone and abortPut now call through to the storage
implementation. The TURL computation functions now take a fileId rather than
the SURL. The fileId is opaque to the SRM code, and the dCache storage
implementation now uses it to store a transfer path rather than a PNFS ID.

PnfsManager no longer injects default access latency and retention policy for
files in a directory with a WriteToken tag. Such values are injected by
SpaceManager. However explicit AccessLatency and RetentionPolicy tags are still
respected. SpaceManager will fail the upload in case of conflicts between those
tags.

Since the upload now hapens to a different path, I added billing notification
from SRM on upload. The billing entry from the transfer door contains the turl
path (I considered injecting the real path, but since the client could try
uploading several times and since the upload isn't persistent until putDone is
called, I think using the turl path is better). The SRM is assymetric here as
downloads are not reported to billing (they would just duplicate existing
information) - it is however easy to add if there is a good use case for it.

Target: trunk
Require-book: yes
Require-notes: yes
spacemanager: Remove obsolete features

Due to the introduction of unique write TURLs, several space manager features
are no longer needed. This patch removes those featurs.

The removed features are all a result of no longer needing the ALLOCATED state
of a file reservation. Thus there is no need to bind file reservations to a
path, no need to track deleted paths, and no need to track lifetime of a file
reservation (the lifetime is now that of the PNFS ID). As a consequence, the
"Already have 1 record(s)" code path no longer exists.

The CLI has been updated to remove the options and output related to the above
features.

The Use and CancelUse messages are no longer supported.

Database schema changes: The path, expiration time, and deleted columns of
srmspacefile have been deleted, together with their indexes. The ALLOCATED
state of srmspacefile records no longer exists.

Target: trunk
Require-notes: yes
Require-book: yes
srm: Replace pin and unpin callbacks with ListenableFuture

This replaces some of the bulky callback interfaces in the SRM with
Guava's ListenableFuture. The same approach was used for the recently
introduced prepareToPut method.

Target: trunk
Require-notes: no
Require-book: no
spacemanager: Fix serialization issue in File class

FsPath isn't serializable and thus File isn't serializable either.
File is never serialized, so the fix is to stop implementing the
Serializable interface.

Target: trunk
Require-notes: no
Require-book: no