libs: prevent JNA jar be pulled with NFS code

Ticket: #8020
Acked-by: Paul Millar
Target: master, 2.6, 2.7
packaging: fix deb and tgz packaging

This patch addresses two issues: Debian building fails for tagged versions
and tar packages are missing a '-1' in their name.

The Debian packaging takes care that the '-SNAPSHOT' in the project version
becomes '~SNAPSHOT' in the debian version; however, for a tagged version,
there is no "-SNAPSHOT" in the project version, so the regex substitution
fails.  This patch makes this optional.

Currently, tar.gz releases are built like dcache-2.7.0.tar.gz.  We publish
tgz release with filenames like 'dcache-2.7.0-1.tar.gz'.  This patch alters
the build so the resulting file may be used directly.

Target: master
Request: 2.7
Patch: http://rb.dcache.org/r/6046/
Acked-by: Tigran Mkrtchyan
Requires-notes: no
Requires-book: no
    Enstore Chimera Info extractor
    Enstore storageinfo URI contains original_name as a part of the query
    Special characters in the file name cause failure to create an URI with
    subsequent failure to stage the file in leaving stage request in suspended
    state.
    This patch addresses it:
      1) introduces uri_encode/uri_decode strorage procedures
      2) modifies ChimeraEnstoreStorageInfoExtractor class
         to handle both encoded and un-encoded file names

    Acked-by: Tigran Mkrtchyan
    Patch: http://rb.dcache.org/r/6043/
    Target: trunk
    Request: 2.2
    Request: 2.6
    Require-book: no
    Require-notes: yes
logback: revert to previous version (broken remote logging)

https://github.com/dCache/dcache/commit/94e3ae116b6f54ed90f80b07a05f7d836f8b97bc

in which the logback library version was bumped from 1.0.12 to 1.0.13 breaks alarms because remote logging hangs.

Until we ascertain why the socket appender does not work with the new version, this patch switches back to the previous version.

Update:  this is a known bug which is not yet resolved:

http://jira.qos.ch/browse/LOGBACK-896?page=com.atlassian.jira.plugin.system.issuetabpanels:changehistory-tabpanel

Testing:
Before patch, dcache alarm send command hangs.
After patch, it succeeds.

Target: master
Request: 2.7
Require-book: no
Require-notes: no
Acked-by: Gerd
chimera: Fix major performance issue on file creation with ACL enabled

Chimera uses a trigger on insert into t_dirs to copy the ACLs of the
parent directory to the newly created inode. The stored procedure used
for this uses character(36) to represent PNFS IDs. This causes a
problem with postgresql and it refuses to use the indexes when the
type of the value you search for is different from the type of the
column - even if one is character(36) and the other character varying(36).

The consequence is that every insert into t_dirs causes a linear
scan of t_acl - something that becomes slow and cpu intensive once
you have millions of entries.

Insertion time into t_dirs dropped from around 2 seconds to 2 milliseconds
when we applied this patch at NDGF.

Implementation note: I replaced the existing changeset as it uses
create or replace anyway. That way new sites will never see the buggy
implementation, and old sites will execute the change set as it has
a new id.

Target: trunk
Request: 2.7
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6049/
srm: Fix authorization for srmv1 requests

Addresses an authorization regression in SRMv1 support.

When AuthorizationRecord was replaced by DcacheUser, SRMv1 authorization
broke. The SRMv1 code relies on SRMUser#equals to determine whether
the request query or update came from the same user that created the
request in the first place. Since DcacheUser doesn't implement equals,
this check always fails.

Rather than implement DcacheUser#equals, the SRM code is updated to
compare the IDs of the users instead. This SRM specific user identifier
is what is stored in the database, and in contrast to the DcacheUser
record, the ID can be restored from disk. The ID also ignores principals
like the Origin, which means that a request query or update will be
authorized even when issued from a different client host.

The patch only affects srmv1. Apparantly srmv2 doesn't do any
authorization checks.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6023/
srm: Inject data source and transaction manager from Spring

The patch gets rid of the JdbcConnectionPool class. The BoneCP
dataSource and transaction manager are instantiated by Spring
and injected into the classes that operate on them.

Besides simplifying the code, the patch has the admin visible
consequence that only one connection pool is created in the
SRM. Previously there was one for the authentication store and
one for the remaining SRM tables. These are now served by a
single BoneCP connection pool (the wonders of using
standardized interfaces...).

New properties are:

srm.db.connections.partition-count=1
srm.db.connections.max-per-partition=50
srm.db.connections.min-per-partition=1

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5985/
srm: Treat TReturnStatus as immutable

Replaces most TReturnStatus instantiations to treat the object as
immutable. Since TReturnStatus is auto-generated, it isn't possible
to actually make it immutable, but our code becomes cleaner if we
treat it as if it was.

I deliberately didn't change any of the status messages, except when
it was obviously wrong. Subsequent patches will focus on the status
messages.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5997/
srm: Clean up SRMServerV2

- Make fields private and final
- Use class name as name of the logger and make logger static and final
- Catch specific excptions
- Improve error messages, both as logged and returned to the client
- Use varargs when possible
- Treat srmPing differently than other requests

srmPing is special as it cannot return a failure. Therefore the failure
code had a special case to return a regular ping reply even in case of
failure (eg authentication failure). Rather than trying to force srmPing
into the regular handler processing, this patch adds an implementation
of srmPing directly in SRMServerV2 and completely bypasses the usual
request processing. Besides speeding up srmPing processing, this allows
us to remove the workaround in the failure path.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6026/
srm-server: Minor cleanup in Job class

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6027/
srm: Fix srmAbortFiles and srmAbortRequest implementations

srmAbortFiles and srmAbortRequest are not at all compliant with the
SRM 2.2 spec. One symptom is that one may get failurs like

23 Sep 2013 23:06:57 (SRM-dcache-vm) [127.0.0.1:39844 t+Q:1:srm2:abortFiles:-2147482647] Illegal State Transition : Illegal state transition from Done to Canceled

when aborting requests that have already completed.

Another big fault is that the SURL return status used is that of
the file request, whereas the spec defines it to be the return
status of the abort operation, with only three legal return
values.

Finally, the return status of the entire request doesn't reflect the
return status of the individual SURLs.

These problems are addresses by this patch. A utility method for computing
a summary return status for the request has been added and a number of
otherwise unrelated classes are modified to make use of the utility method.

Target: trunk
Require-book: no
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6040/
dcache, system-tests: change .gitignore to .empty-dir

Rename .gitignore placeholder for empty var/httpd directory to conform with dCache usage.

Target: master
Request: 2.7
Patch: http://rb.dcache.org/r/6039/
Require-notes: no
Require-book: no
Acked-by: Gerd
billing (plots): adjust upper bound of aggregate plots

Because of the fact that the aggregate tables are now updated only once a day for the data from the previous day, it does not make sense to round the current date to the next 24-hour-period to give the upper bound for the plots of those tables.

Testing:

Redeployed and observed plots.

Target: trunk
Request: 2.7
Request: 2.6
Patch: http://rb.dcache.org/r/6034/
Require-notes: no
Require-book: no
Acked-by: Karsten
Fix bug in SpreadAndWait

A recent refactoring of SpreadAndWait introduced a bug in the
next method.

Target: trunk
Request: 2.7
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6056/
pnfsmanager: Fix null value in diagnostic context

Adresses the problem that several name space messages add 'null'
to the diagnostic context.

PnfsId is an optional field in PnfsMessage and we thus have to check
for a null value.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6053/
pnfsmanager: Use correct queue for create entry and map messages

Messages are put on a pnfs manager queue by pnfs id or path. This is to
ensure that we don't perform concurrent operations on a name space
object and two enforce a happens-before relationship on operations on
a name space object (ie to avoid messages from being processed out of
order). It is also essential for request folding to work correctly.

PnfsCreateEntryMessage and PnfsMapPathMessage however use their own path
field rather than the one in the PnfsMessage base class. Therefore a
random queue was chosen for these messages.

This patch fixes this by also binding the path in the base class. A future
patch for master should remove the redudant fields.

Target: trunk
Request: 2.7
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6051/
http: fix context based pages

regression introduced by 8b7c60e

Acked-by: Albert Rossi
Ticket: #8024
Target: master
Require-book: no
Require-notes: no
billing: fix invalid property handling

fix regression introduced by 8549a34.
The changed property required code update.

Acked-by: Albert Rossi
Target: master, 2.7
Require-book: no
Require-notes: no
SrmSpaceManager: adapt to UniversalSpringCell instatntiation

Adapts SrmSpaceManager to be instantiated using UniversalSpringCell,
replaces debug with trace,
fixes issues with messageForwarding,
fixed CellPath#insert methods

RELEASE NOTES:

introduced the follwing variables:

spacemanager.service.pnfsmanager.timeout=300
spacemanager.service.pnfsmanager.timeout.unit=SECONDS

spacemanager.service.poolmanager.timeout=300
spacemanager.service.poolmanager.timeout.unit=SECONDS

spacemanager.db.password.file=${srmPasswordFile}

Acked-by: Gerd Behrmann
Require-notes: yes
Require-book: no
Target: trunk
Patch: http://rb.dcache.org/r/5864/
billing: Fix regression in billing flat file format rendering

Addresses a regression in rendering the configurable billing format
strings. A bug prevented these formats from being respected. Note
that the format strings are now preprocessed by the dCache
configuration system, which means that escape sequences are
processed. You may have to update custom format strings (in
$$ sequences have to be replaced by $$$$).

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: yes
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/6065/