nfs4: fix IO on special files

regression introduced in 93ec46a3a1be602c0e47d54abd8b0cb12ebb730f

Acked-by: Albert Rossi
Target: master, 2.7
Require-book: no
Require-notes: no
chimera: fix get locality to work with id-type FsInode

The current implementation of ".(get)(filename)(locality)" relies on an FsInode type which embeds the filename and the "locality" argument.  Since the filename is arbitrarily long, the encoded handle length could exceed the file handle length specified by NFS4.

The solution is to change the handling of this command (file) to use the id FsInode type and to encode the operation directly into it.  The id (pnfsid) is of fixed length, as is the code for the op.

A new FsInode type (PLOC) has been created for this purpose.  Several auxiliary methods and private classes which are no longer necessary have also been eliminated.

Testing Done:

The new code allows this operation to work successfully with both v4 and with the smaller handle size of v3 as well (both were tested).

Target: master
Request: 2.7
Patch: http://rb.dcache.org/r/6404
Require-book: no
Require-notes: yes
Acked-by: Tigran

RELEASE NOTES:  Fixes a bug in the ".(get)(filename)(locality)" preventing successful completion of the call when the filename is very long.
chimera: add ".(get)(filename)(checksum)" command

Adds command

".(get)(filename)(checksum)"
OR
".(get)(filename)(checksums)"

which returns a comma-delimited list of type:value pairs for all checksums stored in the database.

Note that the FileSystemProvider API was extended with a method to retrieve a set of org.dcache.util.Checksum objects, and the FsSqlDriver similarly was provided with the necessary wrapper call to the database.

Testing: Works on both nfs3 and fns4.1.

Target: master
Request: 2.7
Patch: http://rb.dcache.org/r/6405
Requires-book: yes
Requires-notes: yes
Depends-on: http://rb.dcache.org/r/6404
Acked-by: Tigran

BOOK:
RELEASE NOTES:

A new NFS dot-command has been added to retrieve all the existing checksums for a given file (see body of description).
admin: fix batch file of acm cell

cells arguments should be quoted

Acked-by: Albert Rossi
Target: master, 2.7
Require-book: no
Require-notes: no
cells: Catch UnresolvedAddressException

We have observed the following error in our logs:

13 Jan 2014 13:26:20 (lm) [] Uncaught exception in thread TunnelConnector
java.nio.channels.UnresolvedAddressException: null
        at sun.nio.ch.Net.checkAddress(Net.java:127) ~[na:1.7.0_45]
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640) ~[na:1.7.0_45]
        at java.nio.channels.SocketChannel.open(SocketChannel.java:184) ~[na:1.7.0_45]
        at dmg.cells.network.LocationManagerConnector.connect(LocationManagerConnector.java:103) ~[cells-2.7.4.jar:2.7.4]
        at dmg.cells.network.LocationManagerConnector.run(LocationManagerConnector.java:153) ~[cells-2.7.4.jar:2.7.4]
        at dmg.cells.nucleus.CellNucleus$1.run(CellNucleus.java:674) ~[cells-2.7.4.jar:2.7.4]
        at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_45]

This patch fixes this problem by translating the exception to an
IOException with additional information about the address that could
not be resovled.

Target: trunk
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Use ajc to compile aspects

The AspectJ language is cleaner than the annotation based style. This
patch adds an invocation of the AspectJ compile to the build process.
The patch also changes EofExceptionAspect from the annotation style to
the AspectJ language.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6380/
spacemanager: Add changeset to install plpgsql

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/6403/
Per ApplicationContext @Transactional configuration

Spring's AspectJ @Transactional implementation is limited to a singleton that
only allows one ApplicationContext to configure the aspect, thus only
supporting a single transaction manager for the entire ClassLoader.

This patch introduces a modified version of Spring's
AnnotationTransactionAspect, which instantiates the aspect for each
transactional bean. A BeanPostProcesser is used to inject the appropriate
transaction manager into the aspect instances.

With this patch we can begin to use @Transactional outside pin manager (and
have multiple pin manager services in the same dCache domain).

Target: trunk
Require-notes: no
Require-book: no
spacemanager: Use JdbcTemplate and annotation based transaction demarcation

JdbcTemplate is similar to our custom DBManager class. JdbcTemplate is however
much more feature complete. It is also much more well known. This patch replaces
DBManager with JdbcTemplate in space manager.

The patch also changes space manager to Spring's annotation based transaction
demarcation. This removes the need to pass around a Connection object between
methods. It makes it easier to ensure that all processing done for a message is
in a single transaction.

Note that I in some cases have annotated methods with @Transactional even
though they are already called from a transactional context. By default Spring
will simply let the method be part of the outer transaction. I use this for
methods that would not be correct if run outside a transactional context (eg
methods that do select for update).

The patch changes the space manager schema such that primary keys are flagged
as auto increment fields. Thus we no longer need to maintain the counter logic
ourself (the counter logic is tricky in the pressence of transactions as it
must not be rolled back in case of an aborted transaction). How this is
implemented is however database specific and the changeset thus contains some
special cases for PostgreSQL. The liquibase changelog defines rollback logic.
Upon downgrading dCache, the schema changes must be rolled back.

The patch adds unique constraint to pnfsid field of srmspacefile table. An
index on this field already existed, so making it unique does not add any extra
overhead. The column type is changed to varchar(36) to match the type used in
the rest of dCache.

Since JdbcTemplate translates all SQLException instances to a more detailed
exception hierarchy, the patch also updates lots of exception handlers to
catch the more detailed exceptions.

Target: trunk
Require-notes: yes
Require-book: no
Convert tabs to spaces.

Require-notes: no
Require-book: no
Patch: none
spacemanager: Split space manager into service layer and data access layer

Introduces a clean data access object interface, separating the space manager
into a data access layer and a service layer.

Since all JDBC related code is now in a single class, I have chosen to inline
SQL expressions. I feel this makes the code easier to read as one does not
need to lookup the expressions all the time.

Given the size of the change, it is difficult not to sneak in some minor
janitorial changes in that is touched anyway.

Target: trunk
Require-notes: no
Require-book: no
spacemanager: Retry on transient errors

Besides allowing the space manager to recover from various transient errors,
this change also enables us to use serializable transaction isolation.

Target: trunk
Require-notes: no
Require-book: no
Fix text width calculation for ANSI formatted help output

ANSI escape sequences have to be ignored to get the correct
width.

Target: trunk
Require-notes: no
Require-book: no
transfermanagers: Send subject and updated file attributes to pool

In contrast to other doors, transfermanager does not inject the subject
and updated file attributes intp the pool io file message it sends to pool.
Space manager in master requires this information and thus cannot work
with transfermanagers unless the correct information is injected.

Target: trunk
Require-book: no
Require-notes: no
spacemanager: Rewrite command line interface

The patch more or less rewrites all space manager related admin commands.

The following commands are removed:

ls
ls file space tokens
listInvalidSpaces
listFilesInSpace
removeFilesFromSpace
remove file
reserve
release
update space reservation

The following commands are added:

ls spaces
ls files
reserve space
update space
release space
release file

The new commands should offer greater flexibility and more concise output.
Documentation has been added for all commands. Options for almost all
commands have changed.

Note that although commands have been rewritten to make use of the
annotation syntax, the point of this patch is the changed user experience,
not the change of implementation.

Internal database keys like the link group id and the file reservation id
are no longer exposed through the user interface. Group (which confusingly
could be both group and user name) and role fields are presented as a
unified owner, which may be a user name or an FQAN. Various list commands
try to hide irrellevant information, dynamically adding columns to the output
suitable for the requested information.

The code was refactored such that the CLI appears in its own class. The link
group import code has been moved to its own classs too. Commands with blocking
code (ie database operations) are now executed on a thread pool to prevent
blocking the message thread.

The RESERVED state of files has been renamed to ALLOCATED to better reflect
how it is used and to be consistent with the terminology of the aggregation
field in space reservations.

The database interface has been extended with a criterion API to allow
flexible querying.

Target: trunk
Require-notes: yes
Require-book: yes
spacemanager: Delete flushed file reservations when name space entry is deteled

Space manager marks file reservations as deleted when the name space entry
is deleted. The actual file reservation is not removed until space manager
is notified that the file is removed from the pool. For files flushed to
tape there may not be a disk copy and thus such a notification is never sent.
The consequence is that reservations for flushed files are never removed.

This patch fixes the problem by deleting the file reservation of flushed files
as soon as the name space entry is deleted. A liquibase changeset is added to
delete old reservations for deleted flushed files.

Target: trunk
Require-notes: yes
Require-book: no
Reduce class path length on liquibase invocation

Reduces system-test build time by approx. 25 seconds on my laptop.

Target: trunk
Require-notes: no
Require-book: no
dcache-chimera: Add interactive chimera shell

Adds an interactive shell for chimera. May also be used in non-interactive mode
with a command provided on the command line. In the latter form it supplants
the older chimera-cli utility.

Note that in some cases the argument order has changed compared to the
chimera-cli utility. The argument order now matches the order found in common
Posix utilities.

The chimera-cli utility should be documented as deprecated and eventually
removed from dCache.

The system-test populate script has been updated to use the new utility, which
avoids means the chimera skeleton can be created with a single invocation,
cutting down on the build time.

Target: trunk
Require-book: yes
Require-notes yes