srm: fix status of all-failed container jobs

The status of a container job is updated as a side-effect of running
the ContainerRequest#getRequestStatus method.  Unfortunately, if
all the file requests have failed then a very long, multi-line
status is recorded, which includes a copy of the previous
status.

This patch fixes the status so that, even under these circumstances,
the status line is a short string.

Target: master
Requires-notes: no
Requires-book: no
srm: fix retry and failure messages for scheduled jobs

The state-change history of a job is needlessly verbose, which masks
what was the real problem.

This patch tidies up the output so the underlying message is recorded.

Target: master
Require-notes: no
Require-book: no
Patch: https://rb.dcache.org/dashboard/
Acked-by: Karsten Schwank
srm: make timestamps easier to read

The current format for time-stamps in SRM output follows ISO 8601, but is
needlessly verbose.

The timezone designator is optional: if ommitted, local timezone is assumed.
This only matters if the information is shared across multiple timezones and
the source is ambigous.  Since log files also include the SRM's cell name,
which is unique within a dCache inistance, I believe such ambiguity is
sufficiently unlikely that it doesn't justify the extra three characters in the
output.

According to Wikipedia, ISO 8601 says:

    4.3.2 NOTE: By mutual agreement of the partners in information interchange,
          the character [T] may be omitted in applications where there is no
          risk of confusing a date and time of day representation with others
          defined in this International Standard.

Since (personally) I find the 'T' makes the output harder to read and a single
space easier, this patch removes the 'T' and replaces it with a single space.

I'm not sure if this is in complete agreement with the above note, but I
believe there's no real risk of misinterpretation the fields.

Target: master
Revert "Update to DataNucleus 4"

This reverts commit 5e710729cf24782c596d6a2baf8b5974d7ef51fd.

After above commit, S2 tests started to fail.  The underlying
problem is presented as a org.datanucleus.exceptions.NucleusUserException
with the following message

    Could not create new "sequence" "pin_sequence" since autoCreate
    flags do not allow it.

For the S2 tests, the message is always the same and appears to
originate from pin-manager.

I've reverting this patch at Gerd's request.

04 Jul 2014 12:20:18 (SRM-sisyphus) [131.169.72.210:53318 45M:33:srm2:prepareToGet:-2147482625:-2147482624] Uncaught exception in thread pool-11-thread-9
libs: revet luquibase to 3.1.2

The 3.2.0 version produces incorrect checksums:

https://liquibase.jira.com/browse/CORE-1950

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
fix .gitignore to work for Eclipse on MacOS X

The intention being to exclude (ignore) all Eclipse generated files,
simply specifying the name of the file in the .gitignore file is
sufficient (it will be excluded from any and all directories).

The directory + glob + name seems to work on Linux, but does not
on my Mac OS X.9 laptop. The solution adopted here works on both.

You may want to check if something similar can be done for IntelliJ
and Maven.

Target: master
Patch: https://rb.dcache.org/r/7132/
Acked-by: Tigran
Require-notes: no
Require-book: no
chimera: fix liquibase changeset to fail peacefully

Fixes http://rt.dcache.org/Ticket/Display.html?id=8379.

Target: master
Require-notes: no
Require-book: no
Acked-by: Tigran
webdav: add support for remote copy

Add support for the COPY command for use with third-party copies.  This follows the protocol
described here:

    https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm/WebDAV/Extensions#ThirdPartyCopies

Although the document describes only HTTPS used as a transport, this patch includes a framework
where many different transports may be supported.  The patch adds support for gsiftp.  Support
for https is delayed on a separate patch (that provides HTTPS push support) being accepted.
Once that patch is in dCache, adding WebDAV support for https transfers is trivial.

Periodic performance reports are sent to the client, describing the progress of the transfer.
To achieve this, the patch adds the ability to query the transfermanager service to discover
the current status of a transfer.  This may prove useful in the SRM third-party transfer,
which also allows for feedback to the client.

As per the above protocol description, if the client disconnects then the transfer is cancelled.

Target: master
Patch: https://rb.dcache.org/r/7113/
Acked-by: Tigran Mkrtchyan
Requires-book: yes
Requires-notes: yes
srm,pool: improve support for 3rd-party HTTP copy

This patch improves SRM's existing support for HTTP 3rd-party copy.
With this patch, both push and pull requests are supported and both
unencrypted (http) and encrypted (https) targets will work.  For
encrypted connections where the pool is pushing a file to a third
party, a client X.509 credential will be used, if supplied.

Both transfer directions will attempt to check data integrity.  For
pull requests, all data integrety information comes from the headers
supplied in the GET response.  For push requests, data integrety
information comes from the headers in a HEAD response after a
successful transfer.

For push requests, if a PUT request is successful but the subsequent
HEAD request reveals data corruption then the pool will attempt to
delete the file with a DELETE request.  If this fails then the error
message is updated.  For pull requests no additional cleanup steps are
needed.

There are two levels of data integrety check: weak and strong.  Weak
data integrety is satisfied when

 o  remote service and dCache server agree on the content length,

 o  none of the checksums supplied by the remote server (if any)
    disagree with the checksums known by dCache.

Strong data integrety is satisfied when, in addition to satisfying
the weak conditions, the transfer also satisfies the following:

 o  there is at least one checksum supplied by the remote server
    that agrees with a known by dCache for this file.

Checksums are discovered via RFC-3230.  For pull requests the
checksums come as part of the GET response headers; for pull request
they come as part of the subsequent HEAD response headers.

The mover accepts a boolean argument 'requires-verification'.  If
set to false then weak data integrety is used; if true then strong
data integrety is required.

The mover also accepts zero or more (arbitrary) HTTP headers when
requesting a file be transferred.  This allows the requester to
customise the HTTP request that the pool sends to the remote server.
Such headers could include authorisation information or some other
information to steer how the remote server handles the HTTP request.

The SRM Copy request includes a list of arbitrary key-value pairs
called ExtraInfo.  This allows the client to specify additional
information that is platform-specific.  When processing such
remote-copy requests, the generic SRM-server layer now passes this
ExtraInfo to the AbstractStorageElement plugin.  The SRM
CopyFileRequest database storage has been updated to store this
information so it is available when restarting the SRM.  The same
approach may be used in the future for other requests, if necessary.

The dCache AbstractStorageElement plugin has been updated to use
ExtraInfo information to control the behaviour:

  "verify"    if set, must be either 'true' or 'false'.  If 'true' then
              strong data integrety is required for a successful
              transfer; if 'false' then weak data integrety is
              sufficient.  If not specified then a configurable default
              value is used.

  "header-"   any ExtraInfo element that starts "header-" are converted
              to an HTTP header.  The HTTP header key is the ExtraInfo
              key without the initial "header-"; the HTTP header value
              is the ExtraInfo value.  The HTTP headers are then used
              when making GET or PUT requests.

The srm-client has been updated to allow the user to specify arbitrary
ExtraInfo information.

Target: master
Acked-by: Tigran Mkrtchyan
Acked-by: Karsten Schwank
https://rb.dcache.org/r/7096/
Fix broken commit
webdav: allow users to choose between showing file's content inline and downloading it

This patch adjusts the directory listing to be more "browser friendly".
There are two parts to this: updating the GET operation and updating the template.

When a pool processes a GET or HEAD request, it generates a Content-Disposition header.  This
is intended to provide the client with a hint under which filename the file should be saved,
but has a side-effect of forcing the pool to suggest either showing the content in the browser
('inline') or saved as a file ('attachment').  The current behaviour is to always sent
'attachment'.

There are situations where a user may wish to view the content in the browser (assuming the
browser supports viewing this content-type) that this behaviour precludes.

With this patch, an optional query parameter 'action' may be specified.  If the action
parameter has the value 'show' then dCache will return a Content-Disposition of 'inline'; if
action is 'download' then the a Content-Disposition is 'attachment'.  If the action parameter
isn't one of these two, or no action parameter is specied then 'attachment' is used.

This patch also updates the html rendering of a directory with the following new behaviour:

  selecting the filename will deliver the file's contents with Content-Disposition
  of 'inline' (i.e. "viewing the file")

  an additional 'download' icon is shown next to each file's filename; selecting
  this will deliver the file's contents with Content-Disposition of 'attachment'.
  This also includes the HTML5 'download' attribute that provides the same hint
  from HTML.

  both the filename and the download icon now have titles (rendered as tooltips in
  most browsers) that describe the effect of selecting the links.

Target: master
Patch: https://rb.dcache.org/r/7058/
Acked-by: Gerd Behrmann
Require-notes: yes
Require-book: yes
Add support for webdav 3rd-party transfer using http and https

This patch also fixes that dCache doesn't send its version as part of the User-Agent string and that
the user-supplied headers were missing from pool HTTP requests.

Patch: https://rb.dcache.org/r/7142
Acked-by: Tigran Mkrtchyan
Requires-book: yes
Requires-notes: yes
nfs4: use only address string in ProtocolInfo#toString() method

Target: master
Acked-By: Paul Millar
Require-book: no
Require-notes: no
releases: update dCache version to v2.11
srm: fix IllegalStateTransition for COPY requests

If the user cancels a COPY request then the file request is moved to
state CANCELLED; however, if the copy then succeeds anyway, the SRM
attempts to move the job to state DONE.  This cannot succeed since
CANCELLED is a final state.

Fix this by first checking whether the job is already in a final
state.  Also tidy up some messages.

Target: master
Request: 2.10
Requires-notes: no
Requires-book: no
Acked-by: Tigran Mkrtchyan
Patch: https://rb.dcache.org/r/7147
srm: fix deserialise of TExtraInfo

The SRM saves the content of TExtraInfo for COPY requests, but fails
to deserialise it correctly if the user supplied no TExtraInfo.
This patch fixes that.

Target: master
Request: 2.10
Requires-book: no
Requires-notes: no
Acked-by: Tigran Mkrtchyan
Patch: https://rb.dcache.org/r/7147/
srm: fix NPE when checking delegated credential

The SRM keeps "the best" delegated credential, if the user supplies one.
If the user doesn't delegate a credential then the current code throws
a NPE.  This is fixed with this patch.

Target: master
Request: 2.10
Requires-notes: no
Requires-book: no
Acked-by: Tigran Mkrtchyan
Patch: https://rb.dcache.org/r/7147/
srm: fix NPE when checking delegated credential

The SRM keeps "the best" delegated credential, if the user supplies one.
If the user doesn't delegate a credential then the current code throws
a NPE.  This is fixed with this patch.

Target: master
Request: 2.10
Requires-notes: no
Requires-book: no
Acked-by: Tigran Mkrtchyan
Patch: https://rb.dcache.org/r/7147/
Merge pull request #699 from paulmillar/fix/2.10/srm-delegated-credential

srm: fix NPE when checking delegated credential
srm: fix deserialise of TExtraInfo

The SRM saves the content of TExtraInfo for COPY requests, but fails
to deserialise it correctly if the user supplied no TExtraInfo.
This patch fixes that.

Target: master
Request: 2.10
Requires-book: no
Requires-notes: no
Acked-by: Tigran Mkrtchyan
Patch: https://rb.dcache.org/r/7147/