core: fix checksum paresing in Storageinfos

Ticket: #7750
Acked-by: Paul Millar
Target: trunk
Require-book: no
Require-notes: no
pom: use custom jglobus2 libraries

changes since rc3:

      [0f8de4f] pom: point to dcache repository
      [3d9f9aa] pom: revert to bc-1.43
      [8388c50] fix string to principal parsing
      [78c7f50] pom: add explicit version numbers for plugins

the official 2.0.5 release missing Paul's serialization fix and 8388c50

Acked-by: Paul Millar
Target: trunk, 2.5
Require-book: no
Require-notes: no
pom: fix bad version number introduced by 07c09b31d2
Restore pinboard default log level to info

module: logback.xml

pinboard default log level accidentally set back to debug.  This patch restores it to info.

Patch: http://rb.dcache.org/r/5419
Acked-by: Dmitry
Target: trunk
Require-book: no
Require-notes: no
pool: Revert change to ListenableFuture

Commit d180e90c3a56710797b239d80dcf396bd13b6f7c replaced the use of the
CompletionHandler interface with Guava's ListenableFuture.  The motivation for
this change was that CompletionHandler is an interface for async I/O in NIO 2
as thus isn't intended as a generic callback interface for async operations in
general. Future and ListenableFuture on the other hand are.

I have however discovered that ListenableFuture is seriously flawed when used
for Movers like I did. When a Future is cancelled, it immediately transitions
to the cancelled state and its get method will throw CancellationException. If
the task is already running, it will not wait for the task to complete before
considering the Future to be in a final state. When extended with listeners in
Guava, these semantics are inherited. Ie. as soon as the ListenableFuture is
cancelled, any listeners are invoked with a failure even though the mover has
not yet terminated. One consequence of that is that we would invoke post
processing even though the mover is still running.

Although I don't like the use of CompletionHandler for this use, I decided to
revert the code back to using CompletionHandler - at least until a better
solution is found. For consistency I use the same interface for the post
processing.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5434/
Fix NPE in GenericStorageInfo#equals

GenericStorageInfo has a constructor that allows it to be created
without any HSMs, yet the equals method will fail with an NPE if
such objects are compared.

This patch fixes this problem and also simplifies the remaining
check in the equals method.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5398/
pool: Allow ReplicaDescriptor#getFileAttributes on closed descriptor

The patch achieves two things. First it avoid repeated loads of
FileAttributes from the meta data store during a file transfer.
It does this by storing the file attributes inside the
ReplicaDescriptor instance. Second, it allows these file attributes
to be queried even after the descriptor is closes (which will be
usefull in another patch).

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5399
pool: Fix corruption error message

A file size mismatch was reported as a checksum mismatch.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5426/
pool: Make use of AbstractCell in NFS execution service

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5427/
Add throwable to a couple of more CacheException subclasses

Exceptions are recommended to have constructors that accept a cause.
Note that cause is a Throwable, not an Exception.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5433/
Add checkum to error message when parsing fails

Patch http://rb.dcache.org/r/5420/ fixed a bug related to checksum
parsing, however the patch didn't fix the fact that the error message
produced omited the problematic checksum string.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5435/
Make CDC AutoCloseable

Makes the CDC AutoCloseable and thus usable with the try-with-resource
construct.

This change was suggested by Paul.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5440/
Exclude Spring internal beans from 'bean ls' output

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5441/
pnfsmanager: Include legacy attributes in storageinfoof CLI output

Fixes a regression introduced in the 2.6 development cycle.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5442/
Revert "pool: Revert change to ListenableFuture"

This reverts commit 22075efe9d1fe2f4e51e7156e80db932209da55d.

I committed the wrong version of the patch.
pool: Revert change to ListenableFuture

Commit d180e90c3a56710797b239d80dcf396bd13b6f7c replaced the use of the
CompletionHandler interface with Guava's ListenableFuture.  The motivation for
this change was that CompletionHandler is an interface for async I/O in NIO 2
as thus isn't intended as a generic callback interface for async operations in
general. Future and ListenableFuture on the other hand are.

I have however discovered that ListenableFuture is seriously flawed when used
for Movers like I did. When a Future is cancelled, it immediately transitions
to the cancelled state and its get method will throw CancellationException. If
the task is already running, it will not wait for the task to complete before
considering the Future to be in a final state. When extended with listeners in
Guava, these semantics are inherited. Ie. as soon as the ListenableFuture is
cancelled, any listeners are invoked with a failure even though the mover has
not yet terminated. One consequence of that is that we would invoke post
processing even though the mover is still running.

Although I don't like the use of CompletionHandler for this use, I decided to
revert the code back to using CompletionHandler - at least until a better
solution is found. For consistency I use the same interface for the post
processing.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5434/
pool: Allow mover queues with zero capacity

PoolV4#mover_set_max_active allows the queue capacity to be set
to zero, yet the AdjustableSemaphore rejects this value (even
though the default capacity for a newly created semaphore is
zero).

Target: trunk
Require-book: no
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5425/
pool: Move scheduling state into SimpleIoScheduler

Scheduling state is kept in PoolIORequest although it is exclusively
maintained and used by the scheduler.

This patch moves scheduling related fields (scheduling state, scheduling
related time stamps) to PrioritizedRequest, an inner class of
SimpleIoScheduler. Thus scheduling information is now represented in
and maintained by the scheduler.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5424/
Change webadmin alarms DAO to do lazy initialization for XML database

module: webadmin (alarms)

The current initialization of the web app uses a greedy startup of the underlying DAO for alarms;
if this is an XML plugin (the default), the file may not be there if the alarmserver has not already been started;
moreover, the pages are created regardless of whether the deployment is actually making use of the alarm server.

The code currently tries to work around this by waiting 30 seconds, and then providing a dummy NOP implementation
of the DAO API if the file is still not there.  This is not an optimal solution.

This patch changes the DAO implementation to lazily initialize the DataNucleus layer.  This also remedies the
situation (rare, but possible) in which the app is started with an XML file but in which the alarmserver not
only gets turned off but the file gets deleted.

Patch: http://rb.dcache.org/r/5407
Acked-by: Gerd
Target: trunk
Require-book: no
Require-notes: yes

RELEASE NOTES:
The case where an alarmserver is started against an already running dcache installation, or where it is stopped and
the xml file store is moved or deleted, is not correctly handled.
Add p2p attribute to MoverInfoMessage

module: core class pool, mover info

Part 1.

In the current billing logs and data tables there is no coherent way to track migration/pool-to-pool operations as distinct from door-initiated uploads and downloads.   This has been done partially at Fermilab (with respect to the text logs) by using a regex filter on the accounting entries; however, the plots generated from the database have never reflected this distinction.  Moreover, a change from 1.9.12+ to 2 which entailed switching to Http from dcap for the p2p protocol also brought with it a change to how the "initiator" is recorded (formerly, simply as <undefined>) which invalidates the matching in the Fermi script but which makes it less certain as to whether any general pattern will now catch the p2p cases.

This patch initiates a few changes which enable the tracking p2p transfers as distinct from door-initiated uploads and downloads.

Several of the relevant mover and pool messages have been modified to include a 'p2p' field, which is also included in the template for creating the accounting log lines (see billing.properties); the information is passed on by the pool when the billing message is sent.

Note: I realize that there is a patch pending which deprecates PoolIORequest.  I have posted this patch anyway, with the understanding that the appropriate modifications will be made when those changes are committed.

Acked-by: Gerd
Target: trunk
Patch: http://rb.dcache.org/r/5413
Require-book: no
Require-notes: yes

RELEASE NOTES:

These should be combined with those of the subsequent patches which actually implement the code for the plots to display this new data.  On the basis of this patch alone, no changes will be visible, though the MoverInfoMessage format could be altered to include the new p2p attribute (see the comments in the billing.properties file).

Notes: the new field has not been added to the default billing string format for backward compatibility.  It needs to be added by hand if you wish the logs to reflect the difference between p2p and door-initiated transfers.  The initiator field now begins with either "pool:" or "door:", respectively.

Testing done in conjunction with dependent patches.