pool: Move scheduling state into SimpleIoScheduler

Scheduling state is kept in PoolIORequest although it is exclusively
maintained and used by the scheduler.

This patch moves scheduling related fields (scheduling state, scheduling
related time stamps) to PrioritizedRequest, an inner class of
SimpleIoScheduler. Thus scheduling information is now represented in
and maintained by the scheduler.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5424/
Change webadmin alarms DAO to do lazy initialization for XML database

module: webadmin (alarms)

The current initialization of the web app uses a greedy startup of the underlying DAO for alarms;
if this is an XML plugin (the default), the file may not be there if the alarmserver has not already been started;
moreover, the pages are created regardless of whether the deployment is actually making use of the alarm server.

The code currently tries to work around this by waiting 30 seconds, and then providing a dummy NOP implementation
of the DAO API if the file is still not there.  This is not an optimal solution.

This patch changes the DAO implementation to lazily initialize the DataNucleus layer.  This also remedies the
situation (rare, but possible) in which the app is started with an XML file but in which the alarmserver not
only gets turned off but the file gets deleted.

Patch: http://rb.dcache.org/r/5407
Acked-by: Gerd
Target: trunk
Require-book: no
Require-notes: yes

RELEASE NOTES:
The case where an alarmserver is started against an already running dcache installation, or where it is stopped and
the xml file store is moved or deleted, is not correctly handled.
Add p2p attribute to MoverInfoMessage

module: core class pool, mover info

Part 1.

In the current billing logs and data tables there is no coherent way to track migration/pool-to-pool operations as distinct from door-initiated uploads and downloads.   This has been done partially at Fermilab (with respect to the text logs) by using a regex filter on the accounting entries; however, the plots generated from the database have never reflected this distinction.  Moreover, a change from 1.9.12+ to 2 which entailed switching to Http from dcap for the p2p protocol also brought with it a change to how the "initiator" is recorded (formerly, simply as <undefined>) which invalidates the matching in the Fermi script but which makes it less certain as to whether any general pattern will now catch the p2p cases.

This patch initiates a few changes which enable the tracking p2p transfers as distinct from door-initiated uploads and downloads.

Several of the relevant mover and pool messages have been modified to include a 'p2p' field, which is also included in the template for creating the accounting log lines (see billing.properties); the information is passed on by the pool when the billing message is sent.

Note: I realize that there is a patch pending which deprecates PoolIORequest.  I have posted this patch anyway, with the understanding that the appropriate modifications will be made when those changes are committed.

Acked-by: Gerd
Target: trunk
Patch: http://rb.dcache.org/r/5413
Require-book: no
Require-notes: yes

RELEASE NOTES:

These should be combined with those of the subsequent patches which actually implement the code for the plots to display this new data.  On the basis of this patch alone, no changes will be visible, though the MoverInfoMessage format could be altered to include the new p2p attribute (see the comments in the billing.properties file).

Notes: the new field has not been added to the default billing string format for backward compatibility.  It needs to be added by hand if you wish the logs to reflect the difference between p2p and door-initiated transfers.  The initiator field now begins with either "pool:" or "door:", respectively.

Testing done in conjunction with dependent patches.
Fix regression introduced in Junit test by patch 5407

module: webadmin alarm tests

Fixed Junit test: needed to force service to return true for connected in order for mockito to work.

This is stopping the current build ...

Patch: http://rb.dcache.org/r/5447
Acked-by: Dmitry
Acked-by: Tigran
Target: trunk
Require-book: no
Require-notes: no
Merge branch 'master' of https://github.com/dCache/dcache
Upgrade to Jetty 7.6

The version of Jetty we use is no longer supported. This patch
upgrades to the latest 7.6 release.

SSL support has been drastically updated in Jetty and as a
result our code for async GSI handling no longer works. I tried
to port the code to Jetty 7.6, but the changes are so substantial
that the port amounts to a rewrite.

Since the async connector never worked without issues, I decided
to drop support for the async connector instead. We may consider
adding it back in the future after moving to Jetty 8 or 9 and
possibly making better use of the JGlobus 2 architecture.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5253/
Fix 'dcache version' regression

We recently decoupled version number handling from SVN. That change introduced
a regression in that the 'dcache version' command no longer prints the
version number of dCache.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
Patch: http://rb.dcache.org/r/5445/
pool: Move post processing to post execution service

PoolIORequest currently both holds transfer properties and
properties independent of any particular transfer.

This patch seperates the post processing related functionality
out of PoolIORequest and into the post execution service.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5429/
pool: Move transfer error handling to execution service

This patch moves the bulk of the transfer related code
out of PoolIoRequest and into the execution service. The
goal is to eventually get rid of the PoolIoRequest class
by moving all functionality to other classes.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5430/
pool: Add flag to create data file when repository entry is created

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5444/
pool: Get rid of PoolIORequest

A mover represents the pool side of a file transfer. It implements
the MoverProtocol interface. As part of scheduling and executing a
mover, the pool wraps the mover at least three times:

- PoolIOTransfer (and its subclasses) wrap the mover and the
  ReplicaDescriptor.

- PoolIORequest wraps PoolIOTransfer and adds billing and door
  notification.

- PrioritizedRequest wraps PoolIORequest and adds scheduling
  related fields.

More over, for NFS the mover itself is just an adapter for the
NFS code.

There is far too much wrapping going on. The encapsulation represented
by the above classes is no longer valid as internal state was exposed
in the refactoring that was done in dCache 1.9.11. The current situation
is difficult to understand and tightly coupled.

This patch eliminates PoolIORequest and the IoProcessable interface.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5431/
pool: Decouple scheduler from execution services

The SimpleIoScheduler is currently responsible for looking up
the execution services for a transfer as part of its scheduling
logic.

In the interest of moving towards a design without the legacy
MoverProtocol interface, this patch moves the execution service
lookup up into the PoolV4 class. The execution services are
now associated with the PoolIOTransfer class just like the
legacy MoverProtocol is. The scheduler is decoupled from the
execution services and only invoke the execute and postprocess
methods of the PoolIOTransfer.

As I am injecting the execution service factor into PoolV4, I
also cleaned up the setters of PoolV4 to use the @Required
annotation.

Eventually my goal is to move the instantiation of the legacy mover
classes into LegacyExecutionService, the only class that should
depend on this old interface. This is one of the reason why the
execution service lookup has to be moved up to happen before
the task is inserted into the queue.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5390/
cells: Let CellPath#revert return a copy

CellPath#revert reverts the path inline. In almost all
cases this means one should clone the CellPath object
before reverting it.

This patch updates revert to return a reverted copy of
the path rather than modifying the path itself. This
protects against accidentally modifying shared cell
path.

The patch also modifies CellMessage to clone the injected
CellPath. This too is to avoid that shared CellPath objects
get modified.

Targe: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5393/
pool: Add Mover interface

A new interface, Mover, is introduced and implemented by the PoolIOTransfer
class. The pool uses PoolIOTransfer to represent a transfer in the pool. In
the UI we usually refer to this entity as "the mover" (it is this object
that we queue and kill, not the legacy MoverProtocol).

The change should be reviewed bearing in mind that I want to get rid of
the dependency on the MoverProtocol interface.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5436/
pool: Rename and parameterize execution services

The execution services are inconsistenly named. Some are called
ExecutorService, some ExecutionService and the NFS one is misspelled
as ExcecutionService.

The patch renames these as variations of TransferService, which IMO
better capture the purpose of the interfaces.

I parameterized the interfaces to avoid coupling the interface
to any particular mover implementation, yet allow implementations to
access mover specific properties.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5437/
pool: Decouple pool from MoverProtocol

The pool has traditionally used the MoverProtocol interface to encapsulate
protocol specific logic for transfers. In dCache 1.9.11 a new transfer service
abstraction was introduced, which allows protocol specific state to be captured
outside of any one transfer.

The first thing a pool does when receiving a transfer request from a door is to
create a MoverProtocol instance. This despite the fact that the core aspect of
this interface, the MoverProtocol#runIO method, was only used by a single
transfer service.

This patch refactors this and decouples the pool from the MoverProtocol
interface. MoverProtocol is only used by TransferService implementations that
choose to use it - but one can implement an execution service without any
MoverProtocol instance.

A MoverFactory interface is introduced that can create Mover instances.
Protocol specific MoverFactory implementations can create protocol specific
Mover instances. TransferServices serves to provide MoverFactory instances for
particular protocols. The available factories are injected through Spring (we
can make this pluggable in the future). This replaces the old movermap logic
that was previously found in PoolV4. The movermap still exists for legacy
MoverProtocol implementations, but is now maintained by
MoverProtocolTransferService.

The NFS transfer service still uses its MoverProtocol bridge and thus also
creates instances of MoverProtocolMover, however this is simply because I
wanted to minize changes to the NFS code (I cannot test it). I expect that now
that MoverProtocol is no longer required, we can update the NFS code to get rid
of it.

I have to stress that the overall design is still as introduced in 1.9.11.  We
have Movers which are "executed" by TransferServices and PostTransferServices,
we have a scheduler that takes care of queuing, killing and activing the
transfer and post transfer steps asynchronously.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5450/
pool: Remove the unused MoverProtocol#wasChanged method

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5401/
pool: Restore mover CDC during kill

Otherwise postprocessing in case of kills will not contain the correct
CDC.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5446/
pool: Fix NPE in HTTP mover

When a transfer doing a chunked upload is killed, the following
NPE is caught when several chunks are in the queue already.

10 apr 2013 12:28:50 (pool_write) [] Uncaught exception in thread http-disk-83
java.lang.NullPointerException: null
        at java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:332) ~[na:1.7.0_11]
        at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:987) ~[na:1.7.0_11]
        at org.dcache.pool.movers.AbstractNettyServer.close(AbstractNettyServer.java:235) ~[dcache-core-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
        at org.dcache.http.HttpPoolRequestHandler.close(HttpPoolRequestHandler.java:620) ~[dcache-core-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
        at org.dcache.http.HttpPoolRequestHandler.doOnChunk(HttpPoolRequestHandler.java:551) ~[dcache-core-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSH
OT]
        at org.dcache.http.HttpRequestHandler.messageReceived(HttpRequestHandler.java:63) ~[dcache-core-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
        at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:48) ~[netty-3.2.7
.Final.jar:na]
        at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:149) ~[netty-3.2.7.Final.jar:na]
        at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:275) ~[netty-3.2.7.Final.jar:na]
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69) [netty-3.2.7.Final.jar:na]
        at org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.j
ava:316) [netty-3.2.7.Final.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_11]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_11]
        at org.dcache.util.CDCThreadFactory$1.run(CDCThreadFactory.java:42) [dcache-core-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_11]

The problem is caused by falsely closing the mover several times. The
error message generated is also misleading.

Target: trunk
Require-notes: no
Require-book: no
Request: 2.2
Request: 2.5
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5443/
pool: Port xrootd and http movers to thread-less architecture

Benefits:

- Get rid of idle thread per transfer problem

- Make protocol specific configuration much cleaner

- Have per pool Netty instances and thread pools rather than
  per domain

The last point is important to note as it means that xrootdMoverXXX and
httpMoverXXX properties are now interpreted per pool, and not per
domain.

For xrootd I also cleaned up the redirect handling. The pool now
send a single socket address to the door, and backwards compatibility
with pools older than 2.5 years has been dropped (don't know the
exact version number but dCache's that old are not supported anymore).

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5448/