Revert "Update to DataNucleus 4"

This reverts commit 5e710729cf24782c596d6a2baf8b5974d7ef51fd.

After above commit, S2 tests started to fail.  The underlying
problem is presented as a org.datanucleus.exceptions.NucleusUserException
with the following message

    Could not create new "sequence" "pin_sequence" since autoCreate
    flags do not allow it.

For the S2 tests, the message is always the same and appears to
originate from pin-manager.

I've reverting this patch at Gerd's request.

04 Jul 2014 12:20:18 (SRM-sisyphus) [131.169.72.210:53318 45M:33:srm2:prepareToGet:-2147482625:-2147482624] Uncaught exception in thread pool-11-thread-9
libs: revet luquibase to 3.1.2

The 3.2.0 version produces incorrect checksums:

https://liquibase.jira.com/browse/CORE-1950

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
fix .gitignore to work for Eclipse on MacOS X

The intention being to exclude (ignore) all Eclipse generated files,
simply specifying the name of the file in the .gitignore file is
sufficient (it will be excluded from any and all directories).

The directory + glob + name seems to work on Linux, but does not
on my Mac OS X.9 laptop. The solution adopted here works on both.

You may want to check if something similar can be done for IntelliJ
and Maven.

Target: master
Patch: https://rb.dcache.org/r/7132/
Acked-by: Tigran
Require-notes: no
Require-book: no
chimera: fix liquibase changeset to fail peacefully

Fixes http://rt.dcache.org/Ticket/Display.html?id=8379.

Target: master
Require-notes: no
Require-book: no
Acked-by: Tigran
webdav: add support for remote copy

Add support for the COPY command for use with third-party copies.  This follows the protocol
described here:

    https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm/WebDAV/Extensions#ThirdPartyCopies

Although the document describes only HTTPS used as a transport, this patch includes a framework
where many different transports may be supported.  The patch adds support for gsiftp.  Support
for https is delayed on a separate patch (that provides HTTPS push support) being accepted.
Once that patch is in dCache, adding WebDAV support for https transfers is trivial.

Periodic performance reports are sent to the client, describing the progress of the transfer.
To achieve this, the patch adds the ability to query the transfermanager service to discover
the current status of a transfer.  This may prove useful in the SRM third-party transfer,
which also allows for feedback to the client.

As per the above protocol description, if the client disconnects then the transfer is cancelled.

Target: master
Patch: https://rb.dcache.org/r/7113/
Acked-by: Tigran Mkrtchyan
Requires-book: yes
Requires-notes: yes
srm,pool: improve support for 3rd-party HTTP copy

This patch improves SRM's existing support for HTTP 3rd-party copy.
With this patch, both push and pull requests are supported and both
unencrypted (http) and encrypted (https) targets will work.  For
encrypted connections where the pool is pushing a file to a third
party, a client X.509 credential will be used, if supplied.

Both transfer directions will attempt to check data integrity.  For
pull requests, all data integrety information comes from the headers
supplied in the GET response.  For push requests, data integrety
information comes from the headers in a HEAD response after a
successful transfer.

For push requests, if a PUT request is successful but the subsequent
HEAD request reveals data corruption then the pool will attempt to
delete the file with a DELETE request.  If this fails then the error
message is updated.  For pull requests no additional cleanup steps are
needed.

There are two levels of data integrety check: weak and strong.  Weak
data integrety is satisfied when

 o  remote service and dCache server agree on the content length,

 o  none of the checksums supplied by the remote server (if any)
    disagree with the checksums known by dCache.

Strong data integrety is satisfied when, in addition to satisfying
the weak conditions, the transfer also satisfies the following:

 o  there is at least one checksum supplied by the remote server
    that agrees with a known by dCache for this file.

Checksums are discovered via RFC-3230.  For pull requests the
checksums come as part of the GET response headers; for pull request
they come as part of the subsequent HEAD response headers.

The mover accepts a boolean argument 'requires-verification'.  If
set to false then weak data integrety is used; if true then strong
data integrety is required.

The mover also accepts zero or more (arbitrary) HTTP headers when
requesting a file be transferred.  This allows the requester to
customise the HTTP request that the pool sends to the remote server.
Such headers could include authorisation information or some other
information to steer how the remote server handles the HTTP request.

The SRM Copy request includes a list of arbitrary key-value pairs
called ExtraInfo.  This allows the client to specify additional
information that is platform-specific.  When processing such
remote-copy requests, the generic SRM-server layer now passes this
ExtraInfo to the AbstractStorageElement plugin.  The SRM
CopyFileRequest database storage has been updated to store this
information so it is available when restarting the SRM.  The same
approach may be used in the future for other requests, if necessary.

The dCache AbstractStorageElement plugin has been updated to use
ExtraInfo information to control the behaviour:

  "verify"    if set, must be either 'true' or 'false'.  If 'true' then
              strong data integrety is required for a successful
              transfer; if 'false' then weak data integrety is
              sufficient.  If not specified then a configurable default
              value is used.

  "header-"   any ExtraInfo element that starts "header-" are converted
              to an HTTP header.  The HTTP header key is the ExtraInfo
              key without the initial "header-"; the HTTP header value
              is the ExtraInfo value.  The HTTP headers are then used
              when making GET or PUT requests.

The srm-client has been updated to allow the user to specify arbitrary
ExtraInfo information.

Target: master
Acked-by: Tigran Mkrtchyan
Acked-by: Karsten Schwank
https://rb.dcache.org/r/7096/
Fix broken commit
webdav: allow users to choose between showing file's content inline and downloading it

This patch adjusts the directory listing to be more "browser friendly".
There are two parts to this: updating the GET operation and updating the template.

When a pool processes a GET or HEAD request, it generates a Content-Disposition header.  This
is intended to provide the client with a hint under which filename the file should be saved,
but has a side-effect of forcing the pool to suggest either showing the content in the browser
('inline') or saved as a file ('attachment').  The current behaviour is to always sent
'attachment'.

There are situations where a user may wish to view the content in the browser (assuming the
browser supports viewing this content-type) that this behaviour precludes.

With this patch, an optional query parameter 'action' may be specified.  If the action
parameter has the value 'show' then dCache will return a Content-Disposition of 'inline'; if
action is 'download' then the a Content-Disposition is 'attachment'.  If the action parameter
isn't one of these two, or no action parameter is specied then 'attachment' is used.

This patch also updates the html rendering of a directory with the following new behaviour:

  selecting the filename will deliver the file's contents with Content-Disposition
  of 'inline' (i.e. "viewing the file")

  an additional 'download' icon is shown next to each file's filename; selecting
  this will deliver the file's contents with Content-Disposition of 'attachment'.
  This also includes the HTML5 'download' attribute that provides the same hint
  from HTML.

  both the filename and the download icon now have titles (rendered as tooltips in
  most browsers) that describe the effect of selecting the links.

Target: master
Patch: https://rb.dcache.org/r/7058/
Acked-by: Gerd Behrmann
Require-notes: yes
Require-book: yes
Add support for webdav 3rd-party transfer using http and https

This patch also fixes that dCache doesn't send its version as part of the User-Agent string and that
the user-supplied headers were missing from pool HTTP requests.

Patch: https://rb.dcache.org/r/7142
Acked-by: Tigran Mkrtchyan
Requires-book: yes
Requires-notes: yes
nfs4: use only address string in ProtocolInfo#toString() method

Target: master
Acked-By: Paul Millar
Require-book: no
Require-notes: no
releases: update dCache version to v2.11
srm: fix IllegalStateTransition for COPY requests

If the user cancels a COPY request then the file request is moved to
state CANCELLED; however, if the copy then succeeds anyway, the SRM
attempts to move the job to state DONE.  This cannot succeed since
CANCELLED is a final state.

Fix this by first checking whether the job is already in a final
state.  Also tidy up some messages.

Target: master
Request: 2.10
Requires-notes: no
Requires-book: no
Acked-by: Tigran Mkrtchyan
Patch: https://rb.dcache.org/r/7147
srm: fix deserialise of TExtraInfo

The SRM saves the content of TExtraInfo for COPY requests, but fails
to deserialise it correctly if the user supplied no TExtraInfo.
This patch fixes that.

Target: master
Request: 2.10
Requires-book: no
Requires-notes: no
Acked-by: Tigran Mkrtchyan
Patch: https://rb.dcache.org/r/7147/
srm: fix NPE when checking delegated credential

The SRM keeps "the best" delegated credential, if the user supplies one.
If the user doesn't delegate a credential then the current code throws
a NPE.  This is fixed with this patch.

Target: master
Request: 2.10
Requires-notes: no
Requires-book: no
Acked-by: Tigran Mkrtchyan
Patch: https://rb.dcache.org/r/7147/
srm: fix NPE when checking delegated credential

The SRM keeps "the best" delegated credential, if the user supplies one.
If the user doesn't delegate a credential then the current code throws
a NPE.  This is fixed with this patch.

Target: master
Request: 2.10
Requires-notes: no
Requires-book: no
Acked-by: Tigran Mkrtchyan
Patch: https://rb.dcache.org/r/7147/
Merge pull request #699 from paulmillar/fix/2.10/srm-delegated-credential

srm: fix NPE when checking delegated credential
srm: fix deserialise of TExtraInfo

The SRM saves the content of TExtraInfo for COPY requests, but fails
to deserialise it correctly if the user supplied no TExtraInfo.
This patch fixes that.

Target: master
Request: 2.10
Requires-book: no
Requires-notes: no
Acked-by: Tigran Mkrtchyan
Patch: https://rb.dcache.org/r/7147/
Merge pull request #700 from paulmillar/fix/2.10/srm-extra-info

srm: fix deserialise of TExtraInfo
alarms: add fuller support for Markers

Recently it was discovered that one of the alarm definitions
(for checksum) was out of sync with the code because the logging
statement it was based on had been modified and this modification
had gone undiscovered (see patch https://rb.dcache.org/r/7115).

The bug brought home the point that reliance on the remote-server
filters can be fragile. While it is desirable to continue to
support these filters, a more robust policy in terms of
development would be to make all such filters optional and
generated by admins. This would mean that all predefined
alarms (introduced by developers) should code them directly
via the alarm Marker provided for.

Currently, however, all such alarms end up being generic, and
their severity level is determined by the logging level at
which they are emitted. This situation is not optimal, as
developer-defined alarms should be capable of the same
finer-grained distinctions (severity and type) as are
permitted by the XML definitions processed on the remote end.

In anticipation of a subsequent patch converting all the
currently predefined alarms into alarms emitted directly
using a marker, this patch extends the internal alarms
API in order to support more fully the definition of
such code-driven alarms.

The main changes introduced by this patch are as follows:

1)  Fuller API for producing an Alarm Marker is provided.
    The main method now takes three parameters: severity,
    type, and key. An alarm marker thus consists of
    a principal marker (ALARM), plus two required
    attribute markers (severity, type) and one optional
    (key, allowing one to determine whether two alarms
    of this type are to be considered identical) whose
    actual values are submarkers of each.

    The creation of markers has now been moved to a static
    AlarmMarkerFactory, and this, plus the basic interfaces
    have been moved to dcache-common for availability
    everywhere.

2)  Extraction of these attributes to create a LogEntry
    has been modified accordingly.

3)  The dcache alarm send command has also been adjusted
    so that -l now denotes Severity level, not logging level.
    All alarms generated by this command are now sent at
    the error level; they are all still without a key,
    meaning each instance is considered to be a different
    alarm. The man page has also been updated.

4)  Because incoming alarms that have already been
    marked as such (and are not in need of being
    matched to a definition) now carry fuller type
    information, the current webadmin alarms page
    also needed modification. The drop-down box
    allowing one to filter the table entries
    currently carries the list of all alarms
    defined in the alarms-definition.xml. This
    has always had the disadvantage of requiring
    a copy of this file to be on the http server
    node as well as the node where the remote
    alarm server is running. But since this
    list no longer represents all the currently
    defined alarms, the usefulness of displaying
    that drop-down list based on that file has
    diminished.

    A compromise has been adopted here: instead
    of trying to display all the possible alarms,
    we have opted to display a list of all types
    which are "known" to the underlying store:
    that is, all types which have thus far been
    sent. One may thus suppose that if a particular
    type of alarm does not appear in the list,
    it has not been received (yet).

    In order to support the extraction of this
    list through the DataNucleus layer, we have
    used a JPA-like (JPQL) query run each time
    a load of the combo-box is called. Since there
    is already an index on the type attribute,
    the query should not cause severe response
    problems.

    We have also adjusted a few minor inconsistencies
    in the logging inside the DataNucleusAlarmStore
    and have eliminated the webadmin-specific
    alarms definition file path, since it is now
    unused on the httpd end.

5)  I have also eliminated the ChecksumAlarm simulator;
    this class was never really used and is in any case
    not necessary.)

Testing:

With RDBMS connector:

Tested production of alarm using dcache alarm send.
Checked the webadmin alarms page for proper loading
of the combo box.

With XML connector:

Same procedure. Works.

RELEASE NOTES

The "-l" option on the dcache alarm send command now takes a Severity level
rather than logging level.

Target: master
Patch: https://rb.dcache.org/r/7127/
Require-book: no
Require-notes: yes
Acked-by: Tigran
network utils: add constant for canonical host name

This patch is an add-on to 7127.  Since a number of alarm keys
need to include the host name in the set of identifiers,
the static initializer has been extended to make this
readily accessible as a static final field.

Target: master
Patch: https://rb.dcache.org/r/7140/
Require-notes: no
Require-book: no
Acked-by: Tigran