nfs4: fix IO on special files

regression introduced in 93ec46a3a1be602c0e47d54abd8b0cb12ebb730f

Acked-by: Albert Rossi
Target: master, 2.7
Require-book: no
Require-notes: no
chimera: fix get locality to work with id-type FsInode

The current implementation of ".(get)(filename)(locality)" relies on an FsInode type which embeds the filename and the "locality" argument.  Since the filename is arbitrarily long, the encoded handle length could exceed the file handle length specified by NFS4.

The solution is to change the handling of this command (file) to use the id FsInode type and to encode the operation directly into it.  The id (pnfsid) is of fixed length, as is the code for the op.

A new FsInode type (PLOC) has been created for this purpose.  Several auxiliary methods and private classes which are no longer necessary have also been eliminated.

Testing Done:

The new code allows this operation to work successfully with both v4 and with the smaller handle size of v3 as well (both were tested).

Target: master
Request: 2.7
Patch: http://rb.dcache.org/r/6404
Require-book: no
Require-notes: yes
Acked-by: Tigran

RELEASE NOTES:  Fixes a bug in the ".(get)(filename)(locality)" preventing successful completion of the call when the filename is very long.
chimera: add ".(get)(filename)(checksum)" command

Adds command

".(get)(filename)(checksum)"
OR
".(get)(filename)(checksums)"

which returns a comma-delimited list of type:value pairs for all checksums stored in the database.

Note that the FileSystemProvider API was extended with a method to retrieve a set of org.dcache.util.Checksum objects, and the FsSqlDriver similarly was provided with the necessary wrapper call to the database.

Testing: Works on both nfs3 and fns4.1.

Target: master
Request: 2.7
Patch: http://rb.dcache.org/r/6405
Requires-book: yes
Requires-notes: yes
Depends-on: http://rb.dcache.org/r/6404
Acked-by: Tigran

BOOK:
RELEASE NOTES:

A new NFS dot-command has been added to retrieve all the existing checksums for a given file (see body of description).
admin: fix batch file of acm cell

cells arguments should be quoted

Acked-by: Albert Rossi
Target: master, 2.7
Require-book: no
Require-notes: no
cells: Catch UnresolvedAddressException

We have observed the following error in our logs:

13 Jan 2014 13:26:20 (lm) [] Uncaught exception in thread TunnelConnector
java.nio.channels.UnresolvedAddressException: null
        at sun.nio.ch.Net.checkAddress(Net.java:127) ~[na:1.7.0_45]
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640) ~[na:1.7.0_45]
        at java.nio.channels.SocketChannel.open(SocketChannel.java:184) ~[na:1.7.0_45]
        at dmg.cells.network.LocationManagerConnector.connect(LocationManagerConnector.java:103) ~[cells-2.7.4.jar:2.7.4]
        at dmg.cells.network.LocationManagerConnector.run(LocationManagerConnector.java:153) ~[cells-2.7.4.jar:2.7.4]
        at dmg.cells.nucleus.CellNucleus$1.run(CellNucleus.java:674) ~[cells-2.7.4.jar:2.7.4]
        at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_45]

This patch fixes this problem by translating the exception to an
IOException with additional information about the address that could
not be resovled.

Target: trunk
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Use ajc to compile aspects

The AspectJ language is cleaner than the annotation based style. This
patch adds an invocation of the AspectJ compile to the build process.
The patch also changes EofExceptionAspect from the annotation style to
the AspectJ language.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6380/
spacemanager: Add changeset to install plpgsql

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/6403/
Per ApplicationContext @Transactional configuration

Spring's AspectJ @Transactional implementation is limited to a singleton that
only allows one ApplicationContext to configure the aspect, thus only
supporting a single transaction manager for the entire ClassLoader.

This patch introduces a modified version of Spring's
AnnotationTransactionAspect, which instantiates the aspect for each
transactional bean. A BeanPostProcesser is used to inject the appropriate
transaction manager into the aspect instances.

With this patch we can begin to use @Transactional outside pin manager (and
have multiple pin manager services in the same dCache domain).

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6382/
nfs: fix compatibility with 2.6

post 2.6 nfs code has changes package name from

org.dcache.chimera.nfs => org.dcache.nfs
Nevertheless one class is used inside
a message sent to a pool and brakes
compatibility with.

this change restores compatibility.

Notice, this makes 2.7.6 not compatible in mixed
setup.

Acked-by: Gerd Behrmann
Target: master, 2.7
Require-book: no
Require-notes: no
Fix gitignore
spacemanager: Use JdbcTemplate and annotation based transaction demarcation

JdbcTemplate is similar to our custom DBManager class. JdbcTemplate is however
much more feature complete. It is also much more well known. This patch replaces
DBManager with JdbcTemplate in space manager.

The patch also changes space manager to Spring's annotation based transaction
demarcation. This removes the need to pass around a Connection object between
methods. It makes it easier to ensure that all processing done for a message is
in a single transaction.

Note that I in some cases have annotated methods with @Transactional even
though they are already called from a transactional context. By default Spring
will simply let the method be part of the outer transaction. I use this for
methods that would not be correct if run outside a transactional context (eg
methods that do select for update).

The patch changes the space manager schema such that primary keys are flagged
as auto increment fields. Thus we no longer need to maintain the counter logic
ourself (the counter logic is tricky in the pressence of transactions as it
must not be rolled back in case of an aborted transaction). How this is
implemented is however database specific and the changeset thus contains some
special cases for PostgreSQL. The liquibase changelog defines rollback logic.
Upon downgrading dCache, the schema changes must be rolled back.

The patch adds unique constraint to pnfsid field of srmspacefile table. An
index on this field already existed, so making it unique does not add any extra
overhead. The column type is changed to varchar(36) to match the type used in
the rest of dCache.

Since JdbcTemplate translates all SQLException instances to a more detailed
exception hierarchy, the patch also updates lots of exception handlers to
catch the more detailed exceptions.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6395/
spacemanager: Convert tabs to spaces.

Require-notes: no
Require-book: no
Patch: none
spacemanager: Split space manager into service layer and data access layer

Introduces a clean data access object interface, separating the space manager
into a data access layer and a service layer.

Since all JDBC related code is now in a single class, I have chosen to inline
SQL expressions. I feel this makes the code easier to read as one does not
need to lookup the expressions all the time.

Given the size of the change, it is difficult not to sneak in some minor
janitorial changes in that is touched anyway.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6396/
transfermanagers: Send subject and updated file attributes to pool

In contrast to other doors, transfermanager does not inject the subject
and updated file attributes intp the pool io file message it sends to pool.
Space manager in master requires this information and thus cannot work
with transfermanagers unless the correct information is injected.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6413/
poolmanager: Remove link group attributes

Link group attributes were not used. The following commands have
been removed from the command line interface:

psu remove linkGroup attribute
psu set linkGroup attribute

The latter of these still exists in the code as a noop to alove existing
poolmanager.conf files to be read.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6418/
Reduce class path length on liquibase invocation

Reduces system-test build time by approx. 25 seconds on my laptop.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6421/
build: fix RPM builds

The rpmbuild tool requires certain directories to exist and will fail
rather than attempting to create them.  The 'clean' phase of maven will
remove these directories.

This patch adds support to ensure the directories exist.

NB. This patch changes the location of rpmbuild-related directories
from path/to/target/ (e.g. packages/fhs/target/RPMS) to
path/to/target/rpmbuild (e.g. packages/fhs/target/rpmbuild/RPMS).
There is an obvious knock-on effect to the location of resulting RPM
files.

Target: master
Request: 2.7 (as needed)
Request: 2.6 (as needed)
Request: 2.2 (as needed)
Patch: http://rb.dcache.org/r/6427/
Acked-by: Gerd Behrmann
cns: make unit tests less sensitive to speed of computer

A recent build failed because the CNS unit-tests rely on a test taking
no longer than 100 ms to complete.  Under normal circumstances, this
is reasonable; however, the build machine was suffering high IO load
and didn't managed to complete the unit test in the allotted time
window.

While the build machine shouldn't be suffering from this IO-load,
equally, our unit-tests shouldn't fail on slow machines.

This patch doesn't fix the dependency on system time; rather, it
increases the time window to 1 minute.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/6432/
Acked-by: Gerd Behrmann
dcache-chimera: Add interactive chimera shell

Adds an interactive shell for chimera. May also be used in non-interactive mode
with a command provided on the command line. In the latter form it supplants
the older chimera-cli utility.

Note that in some cases the argument order has changed compared to the
chimera-cli utility. The argument order now matches the order found in common
Posix utilities.

The chimera-cli utility should be documented as deprecated and eventually
removed from dCache.

The system-test populate script has been updated to use the new utility, which
avoids means the chimera skeleton can be created with a single invocation,
cutting down on the build time.

Target: trunk
Require-book: yes
Require-notes yes
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6423/
spacemanager: Retry on transient errors

Besides allowing the space manager to recover from various transient errors,
this change also enables us to use serializable transaction isolation.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6397/