ftp: Reorganize ftp classes

Moves FTP related classes to org.dcache.ftp, with appropriate
subpackages.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6299/
Clean service shutdown order in LoginManager

Ensures that the LoginCellFactory and executors are not shut
down before we are done using them.

Changes the LoginCellFactory interface to inherit from
Guava's Service class, thus exposing methods for both
starting and stopping a factory.

LoginManager shutdown logic is altered to shut down child
instances during LoginManager shutdown. Since the
LoginCellFactory is shut down, shared resources would be
deallocated when LoginManager terminates, thus possible
rendering child instances dead.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6309/
dcache-webadmin: fix comparison semantics

Java 1.7 now enforces comparison contracts on sort, and will throw an IllegalArgument exception if the implementation of compareTo violates it.

Several comparisons used in the webadmin beans did not adhere to this contract.  For instance, something like:

a.name.compareTo(b.name) + a.domain.compareTo(b.domain) is incorrect, since this could possibly result in 0 if the two parts evaluate to oppositely signed 1s, when in fact 0 should be reserved for equality.

This patch reimplements all compareTo in the bean classes, modernizing them with a call to Guava's ComparisonChain.

Testing:

Reran junit tests.  Deployed and rechecked.

Target: master
Request: 2.6
Patch: http://rb.dcache.org/r/6307/
Bug: http://rt.dcache.org/Ticket/Display.html?id=8145
Require-notes: yes
Require-book: no
Acked-by: Tigran

RELEASE NOTES:

Fixes bug which throws java.lang.IllegalArgumentException: Comparison method violates its general contract! when data is being sorted in certain webadmin pages.
srm: add missing information from "ls queues" command

The 'ls queues' command prints detailed information about the current
status of different queues within the SRM.  The output claims to
show the current status of download (get) requests, but one part of
this output is actually showing 3rd-party copy activity.

This patch fixes the simple typo to show the correct information.

Target: master
Request: 2.7
Request: 2.6
Request: 2.2
Patch: http://rb.dcache.org/r/6316/
Acked-by: Dmitry Litvintsev
namespace: add support for link creation

update chimera providet

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
pnfsmanager: add support for symlink creation

Acked-by: Karsten Schwank
Target: master
Require-book: no
Require-notes: no
chimera: avoid extra db lookups on create and remove

Acked-by: Karsten Schwank
Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
srm: refactor part of Scheduler to use guava Service

This patch takes the code that tries to keep the executors busy and
makes a Service out of it, based on AbstractExecutionThreadService.
The core logic remains unchanged, but it is collected into a
class.

The goal is to refactor the different sub-components that Scheduler
uses into Service classes and use a ServiceManager to manage their
lifecycle.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/6313/
Acked-by: Gerd Behrmann
webdav: log why request is rejected with permission denied

Certain configuration can lead to a user's request being rejected
inside the webdav door.  These are currently either not logged or
logged with limited information.  Either way, it is hard to
understand why a request is failing.

Target: master
Request: 2.7
Request: 2.6
Request: 2.2
Patch: http://rb.dcache.org/r/6319/
Acked-by: Gerd Behrmann
scripts: fix dcache script for Solaris machines

Patch http://rb.dcache.org/r/6293/ (committed as 19c2b522)
fixed a problem with the configuration cache not being
invalidated.

Unfortunately, in fixing a Mac-specific issue, the patch
introduced a Solaris-specific issue; specifically, that
Solaris machines do not support any options to the hostname
command.  The result is that calls like 'hostname -s' will
set the hostname to '-s'.

This patch fixes the problem by using shell's built in
filtering.

Target: master
Ticket: https://github.com/dCache/dcache/issues/335
Request: 2.7
Request: 2.6
Request: 2.2
Patch: http://rb.dcache.org/r/6315/
Acked-by: Gerd Behrmann
nfsv41: use sort timeout between door and pool manager

NFS client will backoff if it will notice that server is
not responsive. Use short timeouts to keep the pipe full.

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
pool: Fix race condition in migration module

Addresses an IllegalStateException in migration module.

The following exception can occur:

java.lang.IllegalStateException: Unexpected source pool list: Exactly one item was expected, but it contained 0
        at org.dcache.pool.migration.PoolListFilter.getSource(PoolListFilter.java:116) ~[dcache-core-2.6.17.jar:2.6.17]
        at org.dcache.pool.migration.PoolListFilter.evaluate(PoolListFilter.java:126) ~[dcache-core-2.6.17.jar:2.6.17]
        at org.dcache.pool.migration.PoolListFilter.isExcluded(PoolListFilter.java:101) ~[dcache-core-2.6.17.jar:2.6.17]
        at org.dcache.pool.migration.PoolListFilter.getPools(PoolListFilter.java:76) ~[dcache-core-2.6.17.jar:2.6.17]
        at org.dcache.pool.migration.Task.getPools(Task.java:195) ~[dcache-core-2.6.17.jar:2.6.17]
        at org.dcache.pool.migration.Task.setLocations(Task.java:266) ~[dcache-core-2.6.17.jar:2.6.17]
        at org.dcache.pool.migration.Task.access$000(Task.java:38) ~[dcache-core-2.6.17.jar:2.6.17]
        at org.dcache.pool.migration.Task$1.success(Task.java:253) ~[dcache-core-2.6.17.jar:2.6.17]
        at org.dcache.pool.migration.Task$1.success(Task.java:249) ~[dcache-core-2.6.17.jar:2.6.17]
        at org.dcache.cells.AbstractMessageCallback.success(AbstractMessageCallback.java:31) ~[dcache-core-2.6.17.jar:2.6.17]
        at org.dcache.cells.CellStub$CellCallback.answerArrived(CellStub.java:416) ~[dcache-core-2.6.17.jar:2.6.17]
        at dmg.cells.nucleus.CellNucleus$CallbackTask.innerRun(CellNucleus.java:983) ~[cells-2.6.17.jar:2.6.17]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:941) ~[cells-2.6.17.jar:2.6.17]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_17]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_17]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_17]

The problem occurs if migration module queries information about itself
from pool manager before pool manager has registered that information.
In that case a valid, but empty, reply from pool manager is received
and a pre-condition check in the migration module fails.

Target: trunk
Request: 2.7
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6283/
ftp: Add LoginCellProvider for FTP

Adding a dedicating FTP provider for LoginCellFactory
allows the shared executor to be bound to the lifetime
of the LoginManager.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6310/
ftp: Fix deadlock introduced in recent refactoring

Recent changes to the FTP door introduced a deadlock:

"pool-116-thread-7":
        at org.dcache.ftp.door.AbstractFtpDoorV1.setTransfer(AbstractFtpDoorV1.java:2380)
        - waiting to lock <0x00000007c5eade98> (a org.dcache.ftp.door.GsiFtpDoorV1)
        at org.dcache.ftp.door.AbstractFtpDoorV1$FtpTransfer.transferCompleted(AbstractFtpDoorV1.java:1022)
        - locked <0x00000007c5eb3860> (a org.dcache.ftp.door.AbstractFtpDoorV1$FtpTransfer)
        at org.dcache.ftp.door.AbstractFtpDoorV1$FtpTransfer.finished(AbstractFtpDoorV1.java:972)
        at org.dcache.util.Transfer.finished(Transfer.java:464)
        - locked <0x00000007c5eb3860> (a org.dcache.ftp.door.AbstractFtpDoorV1$FtpTransfer)
        at org.dcache.ftp.door.AbstractFtpDoorV1.messageArrived(AbstractFtpDoorV1.java:1444)
        at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.dcache.cells.CellMessageDispatcher$ShortReceiver.deliver(CellMessageDispatcher.java:289)
        at org.dcache.cells.CellMessageDispatcher.call(CellMessageDispatcher.java:202)
        at org.dcache.cells.AbstractCell.messageArrived(AbstractCell.java:859)
        at dmg.cells.nucleus.CellAdapter.messageArrived(CellAdapter.java:981)
        at dmg.cells.nucleus.CellNucleus$DeliverMessageTask.innerRun(CellNucleus.java:1037)
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:946)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
"Thread-160":
        at org.dcache.ftp.door.AbstractFtpDoorV1$FtpTransfer.abort(AbstractFtpDoorV1.java:1060)
        - waiting to lock <0x00000007c5eb3860> (a org.dcache.ftp.door.AbstractFtpDoorV1$FtpTransfer)
        at org.dcache.ftp.door.AbstractFtpDoorV1$FtpTransfer.abort(AbstractFtpDoorV1.java:1036)
        at org.dcache.ftp.door.AbstractFtpDoorV1.shutdown(AbstractFtpDoorV1.java:1363)
        - locked <0x00000007c5eade98> (a org.dcache.ftp.door.GsiFtpDoorV1)
        at diskCacheV111.doors.LineBasedDoor$CommandQueue.stop(LineBasedDoor.java:343)
        - locked <0x00000007c5ed1290> (a diskCacheV111.doors.LineBasedDoor$CommandQueue)
        at diskCacheV111.doors.LineBasedDoor.run(LineBasedDoor.java:176)
        at java.lang.Thread.run(Thread.java:744)

This deadlock is caused by unnecessary synchronization in the shutdown
method. This synchronization did not exist in the old code.

The patch also adds an accessor for the transfer field, thus solving
potential race conditions.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6318/
ftp: Fix race that lead to erroneous error message in log file

Removes the following error message:

05 dec 2013 14:35:34 (httpd) [door:httpd@dCacheDomain:1386250529077 door:httpd@dCacheDomain:1386250530581 door:httpd@dCacheDomain:1386250530980 door:httpd@dCacheDomain:13
86250531049 door:httpd@dCacheDomain:1386250531677 door:httpd@dCacheDomain:1386250531744 door:httpd@dCacheDomain:1386250533468 door:httpd@dCacheDomain:1386250533660 door:h
ttpd@dCacheDomain:1386250534760] Cannot abort transfer that already completed: 451 Aborting transfer due to session termination

This error is a result of a race in which door shutdown and
the transfer termination sequence conincide. The patch
rearranges the termination sequence to mark the transfer
as completed before providing the final reply to the client.
The patch also removes the error logging.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6320/
ftp: Fix race conditions caused by message reordering

Since DoorTransferFinished and GftpTransferStarted take different
routes, it is possible for these messages to be delivered out of
order. As a result the FTP door produces a 226 reply earlier than
is allowed.

This patch introduces an abstract base class for asynchronous
redirected transfers. That class deals with out of order
notifications. As a side effect, a small pause during transfer
abortion has been removed.

The patch also replaces several 426 replies with 451. 451 is for
internal errors while 426 is for client initiated transfer
abort.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6322/
commons: use long to store updates count of a gauge

to prevent integer overflow

Ticket: #8158
Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
srm: refactor SchedulerFactory

This patch modifies SchedulerFactory so it has a single concern:
creating Scheduler instances.  The current implementation of
SchedulerFactory has two other concerns: creating all necessary
schedulers and maintaining a collection of schedulers. These
additional activities are moved elsewhere.

Creating all necessary schedulers has moved to the SRM class.
Ultimately this should go into the spring XML file, but I didn't want
to do too much in one patch.

Most of the behaviour associated with multiple schedules has moved
into the new SchedulerContainer class and the process of submitting a
Job to a scheduler is achieved via this new class.

One design goal is that SchedulerContainer should not 'leak' Scheduler
instances.  The getScheduler-like methods has been removed: both from
SchedulerContainer/SchedulerFactory and from SRM class.  Activity that
iterates over all the Schedulers has moved to SchedulerContainer.
Activity that targets a specific scheduler is implemented with a
method that accepts a Class<? extends Job> argument to target a
specific Scheduler.

Note that the current code has the Job classes being responsible for
adding themselves to a scheduler.  To try to keep the patch short,
this mode of operation is maintained; however, Job is supplied with a
Scheduler rather than discovering the scheduler itself.

The handler classes must be able to add a Job to a scheduler.  The
patch achieves this via the SRM object passed to the handler's
constructor.  Future patch may refactor this so the object to which a
Job is submitted is supplied on the constructor.

Another future direction is to make Scheduler an interface that the
current Scheduler class implements.  The SchedulerContainer would also
implement this interface and the similarity between SchedulerContainer
and Scheduler becomes clearer.

Another beneficial side-effect from this patch is that
SchedulerFactory is no longer a singleton.

Target: master
Requires-book: no
Requires-notes: no
Patch: http://rb.dcache.org/r/6321/
Acked-by: Gerd Behrmann
ftp: Refactor command queue into sequential executor

The FTP door uses a custom CommandQueue to provide the
ability to sequentially process commands using a shared
thread pool. This patch refactors this code into a more
generic SequntialExecutor class. It implements the
ExecutorService interface and is semantically similar to
a single thread executor. However the thread used for
processing tasks is allocated on demands from a shared
Executor.

The selective enabling of interrupts has been removed
from the FTP door. This was a workaround for bad exception
handling in old versions of the door. The workaround is
not easily ported to SequentialExecutor and is therefore
removed.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6325/
srm: fix Copy scheduling

Recent refactoring resulted in srmCopy operations failing with the message:

    Scheduler for class class org.dcache.srm.request.CopyRequest is not
    supported.

This patch fixes srmCopy and the error message.

Target: master
Requires-book: no
Requires-notes: no