webadmin: eliminate system-specific dir for unpacking of war file

The directory where the webapp war should be unpacked is currently set to /var/tmp.  This location, however, is susceptible on some systems to periodic deletion, in which case the web application fails to be found by the server.

For this version, we have simply switched this location to /var/lib/dcache/httpd.

Separate patch for 2.6 will be posted.

Testing:

Built rpm and ran instance, checking for presence of webapp directory in /var/lib/dcache/httpd, and for sanity of webadmin pages.

Target: master
Request: 2.7
Patch: http://rb.dcache.org/r/6002
Require-book: no
Require-notes: yes
Bug: http://rt.dcache.org/Ticket/Display.html?id=7979
Acked-by: Paul

RELEASE NOTES:

Fixes bug where on some systems automatic cleanup of temporary directories was eliminating the webapp application files and thus causing the webadmin pages to fail to be found.
srmclient: Fix failure detection

Addresses an issue in failure reporting for failed SRMv1 requests.

SRM v1 on master is currently broken. Our srmclient however provides a secondary
and non-informative failure:

srmcp -1 file:////bin/bash srm://localhost:8443/disk/test-1379612127-1/a
Thu Sep 19 19:35:50 CEST 2013: incorrect number of RequestFileStatusesin RequestStatus expected 1 received 0
Thu Sep 19 19:35:50 CEST 2013: java.io.IOException: incorrect number of RequestFileStatusesin RequestStatus expected 1 received 0
srm copy of at least one file failed or not completed

This is caused by the SRM client checking the number of file-level status
entries before it checks the status of the response itself. This patch resolves
the issue. The client now produces:

srmcp -1 file:////bin/bash srm://localhost:8443/disk/test-1379612984-1/a
Thu Sep 19 19:50:04 CEST 2013: rs.state = Failed rs.error = getRequestStatus(): request #-2147483645 owned by [GidPrincipal[1000,primary], UidPrincipal[1000], S:/127.0.0.1, UserNamePrincipal[behrmann], /O=Grid/O=NorduGrid/OU=nordu.net/CN=Gerd Behrmann] read-write / does not belong to user [GidPrincipal[1000,primary], UidPrincipal[1000], S:/127.0.0.1, UserNamePrincipal[behrmann], /O=Grid/O=NorduGrid/OU=nordu.net/CN=Gerd Behrmann] read-write /
Thu Sep 19 19:50:04 CEST 2013: java.io.IOException: rs.state = Failed rs.error = getRequestStatus(): request #-2147483645 owned by [GidPrincipal[1000,primary], UidPrincipal[1000], S:/127.0.0.1, UserNamePrincipal[behrmann], /O=Grid/O=NorduGrid/OU=nordu.net/CN=Gerd Behrmann] read-write / does not belong to user [GidPrincipal[1000,primary], UidPrincipal[1000], S:/127.0.0.1, UserNamePrincipal[behrmann], /O=Grid/O=NorduGrid/OU=nordu.net/CN=Gerd Behrmann] read-write /
srm copy of at least one file failed or not completed

I am aware of the double logging (caused by logging and throwing an
exception), however that's a different problem and not addressed by
this patch.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6019/
srm-client: Added missing srm-abort-files options in help text

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6028/
srm: Introduce srm-common module and rename existing srm modules

Addresses bloat in the srmclient packages.

srmclient relies on srm for protocol and utility classes. The dependency
causes the srmclient packages to include lots of transitive dependencies
not actually needed by the client. Those dependencies increase the size
of the RPM and increase the startup time of the clients as the classpath
is longer.

This patch moves those classes to a new srm-common module. The srmclient
module is renamed to srm-client and the srm module to srm-server; this
follows the naming scheme used by other modules.

The patch also restricts the scope of some powermock components to
testing. This avoid that these are included in the final product.

The effect of the pathc is that the RPM shrinks from a bit above 16 MB
to sligthly below 8 MB.

The patch removes a few old build files and build instructions that
no longer apply.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6024/
Upgrade BoneCP and Guava

BoneCP and Guava have to be upgraded together as the new BoneCP uses the new
Guava, and the new Guava is incompatible with the old BoneCP.

BoneCP 0.7 contains a bug in which it doesn't restore the auto-commit flag
on recycled connections. We work arround that problem in our code by carefully
resetting the flag before 'closing' the connection, however that only works
as long as you don't mix your code with eg DataNucleus. DataNucleus changes
the auto commit flag and thus you run into problems if your own code relies
on connections haven't auto commit enabled by default (as required by the
JDBC spec).

The problem is fixed in BoneCP 0.8, although that is only available in a release
candidate. rc3 has however been adopted by the Play project, which I consider
a high profile endorsement.

0.8-rc3 contains a bug in that the default auto-commit value is false rather
than true as required by the standard. I add a custom bonecp-config.xml to
work around that problem. The config also enables logging of slow SQL statements,
using a conservative 10 second threshold (this is the only thing worth
mentioning in the release notes).

The BoneCP setReleaseHelperThreads method is deprecated in BoneCP 0.8 and I
removed all calls to it.

Guava removes the Ranges class, which means a number of classes had to be
updated to compensate.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: http://rb.dcache.org/dashboard/
libs: prevent JNA jar be pulled with NFS code

Ticket: #8020
Acked-by: Paul Millar
Target: master, 2.6, 2.7
packaging: fix deb and tgz packaging

This patch addresses two issues: Debian building fails for tagged versions
and tar packages are missing a '-1' in their name.

The Debian packaging takes care that the '-SNAPSHOT' in the project version
becomes '~SNAPSHOT' in the debian version; however, for a tagged version,
there is no "-SNAPSHOT" in the project version, so the regex substitution
fails.  This patch makes this optional.

Currently, tar.gz releases are built like dcache-2.7.0.tar.gz.  We publish
tgz release with filenames like 'dcache-2.7.0-1.tar.gz'.  This patch alters
the build so the resulting file may be used directly.

Target: master
Request: 2.7
Patch: http://rb.dcache.org/r/6046/
Acked-by: Tigran Mkrtchyan
Requires-notes: no
Requires-book: no
    Enstore Chimera Info extractor
    Enstore storageinfo URI contains original_name as a part of the query
    Special characters in the file name cause failure to create an URI with
    subsequent failure to stage the file in leaving stage request in suspended
    state.
    This patch addresses it:
      1) introduces uri_encode/uri_decode strorage procedures
      2) modifies ChimeraEnstoreStorageInfoExtractor class
         to handle both encoded and un-encoded file names

    Acked-by: Tigran Mkrtchyan
    Patch: http://rb.dcache.org/r/6043/
    Target: trunk
    Request: 2.2
    Request: 2.6
    Require-book: no
    Require-notes: yes
logback: revert to previous version (broken remote logging)

https://github.com/dCache/dcache/commit/94e3ae116b6f54ed90f80b07a05f7d836f8b97bc

in which the logback library version was bumped from 1.0.12 to 1.0.13 breaks alarms because remote logging hangs.

Until we ascertain why the socket appender does not work with the new version, this patch switches back to the previous version.

Update:  this is a known bug which is not yet resolved:

http://jira.qos.ch/browse/LOGBACK-896?page=com.atlassian.jira.plugin.system.issuetabpanels:changehistory-tabpanel

Testing:
Before patch, dcache alarm send command hangs.
After patch, it succeeds.

Target: master
Request: 2.7
Require-book: no
Require-notes: no
Acked-by: Gerd
chimera: Fix major performance issue on file creation with ACL enabled

Chimera uses a trigger on insert into t_dirs to copy the ACLs of the
parent directory to the newly created inode. The stored procedure used
for this uses character(36) to represent PNFS IDs. This causes a
problem with postgresql and it refuses to use the indexes when the
type of the value you search for is different from the type of the
column - even if one is character(36) and the other character varying(36).

The consequence is that every insert into t_dirs causes a linear
scan of t_acl - something that becomes slow and cpu intensive once
you have millions of entries.

Insertion time into t_dirs dropped from around 2 seconds to 2 milliseconds
when we applied this patch at NDGF.

Implementation note: I replaced the existing changeset as it uses
create or replace anyway. That way new sites will never see the buggy
implementation, and old sites will execute the change set as it has
a new id.

Target: trunk
Request: 2.7
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6049/
srm: Fix authorization for srmv1 requests

Addresses an authorization regression in SRMv1 support.

When AuthorizationRecord was replaced by DcacheUser, SRMv1 authorization
broke. The SRMv1 code relies on SRMUser#equals to determine whether
the request query or update came from the same user that created the
request in the first place. Since DcacheUser doesn't implement equals,
this check always fails.

Rather than implement DcacheUser#equals, the SRM code is updated to
compare the IDs of the users instead. This SRM specific user identifier
is what is stored in the database, and in contrast to the DcacheUser
record, the ID can be restored from disk. The ID also ignores principals
like the Origin, which means that a request query or update will be
authorized even when issued from a different client host.

The patch only affects srmv1. Apparantly srmv2 doesn't do any
authorization checks.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6023/
srm: Inject data source and transaction manager from Spring

The patch gets rid of the JdbcConnectionPool class. The BoneCP
dataSource and transaction manager are instantiated by Spring
and injected into the classes that operate on them.

Besides simplifying the code, the patch has the admin visible
consequence that only one connection pool is created in the
SRM. Previously there was one for the authentication store and
one for the remaining SRM tables. These are now served by a
single BoneCP connection pool (the wonders of using
standardized interfaces...).

New properties are:

srm.db.connections.partition-count=1
srm.db.connections.max-per-partition=50
srm.db.connections.min-per-partition=1

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5985/
srm: Treat TReturnStatus as immutable

Replaces most TReturnStatus instantiations to treat the object as
immutable. Since TReturnStatus is auto-generated, it isn't possible
to actually make it immutable, but our code becomes cleaner if we
treat it as if it was.

I deliberately didn't change any of the status messages, except when
it was obviously wrong. Subsequent patches will focus on the status
messages.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5997/
srm: Clean up SRMServerV2

- Make fields private and final
- Use class name as name of the logger and make logger static and final
- Catch specific excptions
- Improve error messages, both as logged and returned to the client
- Use varargs when possible
- Treat srmPing differently than other requests

srmPing is special as it cannot return a failure. Therefore the failure
code had a special case to return a regular ping reply even in case of
failure (eg authentication failure). Rather than trying to force srmPing
into the regular handler processing, this patch adds an implementation
of srmPing directly in SRMServerV2 and completely bypasses the usual
request processing. Besides speeding up srmPing processing, this allows
us to remove the workaround in the failure path.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6026/
srm-server: Minor cleanup in Job class

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6027/
srm: Fix srmAbortFiles and srmAbortRequest implementations

srmAbortFiles and srmAbortRequest are not at all compliant with the
SRM 2.2 spec. One symptom is that one may get failurs like

23 Sep 2013 23:06:57 (SRM-dcache-vm) [127.0.0.1:39844 t+Q:1:srm2:abortFiles:-2147482647] Illegal State Transition : Illegal state transition from Done to Canceled

when aborting requests that have already completed.

Another big fault is that the SURL return status used is that of
the file request, whereas the spec defines it to be the return
status of the abort operation, with only three legal return
values.

Finally, the return status of the entire request doesn't reflect the
return status of the individual SURLs.

These problems are addresses by this patch. A utility method for computing
a summary return status for the request has been added and a number of
otherwise unrelated classes are modified to make use of the utility method.

Target: trunk
Require-book: no
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6040/
dcache, system-tests: change .gitignore to .empty-dir

Rename .gitignore placeholder for empty var/httpd directory to conform with dCache usage.

Target: master
Request: 2.7
Patch: http://rb.dcache.org/r/6039/
Require-notes: no
Require-book: no
Acked-by: Gerd
billing (plots): adjust upper bound of aggregate plots

Because of the fact that the aggregate tables are now updated only once a day for the data from the previous day, it does not make sense to round the current date to the next 24-hour-period to give the upper bound for the plots of those tables.

Testing:

Redeployed and observed plots.

Target: trunk
Request: 2.7
Request: 2.6
Patch: http://rb.dcache.org/r/6034/
Require-notes: no
Require-book: no
Acked-by: Karsten
Fix bug in SpreadAndWait

A recent refactoring of SpreadAndWait introduced a bug in the
next method.

Target: trunk
Request: 2.7
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6056/
pnfsmanager: Fix null value in diagnostic context

Adresses the problem that several name space messages add 'null'
to the diagnostic context.

PnfsId is an optional field in PnfsMessage and we thus have to check
for a null value.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6053/