srm: Get rid of ThreadManager

SRM is updated to use a regular Executor for message processing. Introduces the
following new configuration properties:

srm.cell.limits.message.threads.min
srm.cell.limits.message.threads.max
srm.cell.limits.message.threads.max-idle-time
srm.cell.limits.message.threads.max-idle-time.unit
srm.cell.limits.message.queue.max

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6784/
Get rid of callback executor in CellNucleus

A bit of history: In the old days CellNucleus ran all callbacks in the thread
that added the message event to the queue - that's usually the sending thread.
This worked fine if the callbacks didn't do anything heavy; and it was
absolutely essential that they didn't do anything blocking.

Later this was extended with an option to instead spawn a new thread for each
callback invocation. This didn't have the downsides described above, but added
overhead by adding thread instantiation and a context switch.

Some years ago this was refactored with a callback executor in CellNucleus. The
default was single threaded, but multi-threaded callbacks could be injected.
This reduced the thread creation overhead, but now the context switch cost was
incurred for every callback. Since many callbacks simply set a flag or schedule
another task on another executor, this context switch is unnecessary.

Recently the synchronous message handling code was refactored to be build on
top of the asynchronous code. This introduced a deadlock potential, since a
thread issuing sendAndWait could be unable to receive the reply notification as
the callback would execute on a thread pool blocked by the sendAndWait call.

This removes the entire callback thread handling from CellNucleus. Instead it
uses a design inspired by Guava: Every callback registration also registers an
executor on which to run the callback. Thus the caller is responsible to choose
the callback policy, and can select whether to run the callback inline with the
message delivery thread, run it on a single thread executor or an a multi
thread executor. sendAndWait uses the MoreExecutors#sameThreadExecutor to
receive the signal independently from any executors, thus avoiding the
deadlock.

Other code is updated to provide an executor when registering the callback.
Often this is the sameThreadExecutor (similar to how it was in the old days)
since the callback doesn't do much more that set a flag, release a semaphor, or
maybe send another message with a new callback.

One problem is the SRM which has cheap callbacks, except for the getJob call:
That could potentially load the Job from the database, but since we lock all
jobs in memory, this will not happen in the current code. I plan to refactor
the SRM to execute the callbacks under the control of the Scheduler, but this
has not been done yet.

The patch obsoletes the following properties:

pool.cell.limits.callback.threads.min
pool.cell.limits.callback.threads.max
pool.cell.limits.callback.threads.max-idle-time
pool.cell.limits.callback.threads.max-idle-time.unit
pool.cell.limits.callback.queue.max

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6783/
poolmanager: Get rid of ThreadPool interface

Use a regular Executor instead. The following properties are now
obsolete:

threadPool
poolmanager.plugins.thread-pool

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6785/
pool: Controlled termination of thread pools during pool shutdown

Also adds additional dependencies between pool components to control
the shutdown order.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6789/
srm: Add error message to an IllegalArgumentException

After upgrading to 2.9, we suffered from a hard to find IllegalStateException.
This patch adds an error message to the exception.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6803/
srm: Improve database initialization and shutdown

Fixes several problems:

- Move database initialization to the init-method to match what
  is cleaned up in the destroy-method (otherwise an instantiated
  but uninitialized SRM object would not shut down the db executors).

- Protect against resource leaks in case of initialization failure
  (if the init-method throws an exception, the destroy-method is
  not called).

- Avoid waiting twice during executor shutdown if the first wait
  timed out.

- Catch InterruptedException and set the interrupted flag on the
  current thread. This allows the rest of the shutdown to
  complete.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6804/
srm: fix restore from database

Many (but not all) ContainerRequests are not scheduled and rely on
the scheduling of the member FileRequest jobs.  These receive a
scheduler for their FileRequest jobs, which they cannot use should
they wish to schedule themselves.

The current restore-after-restart code mistakenly tries to reschedule
these ContainerRequest jobs with their member FileRequest scheduler.
This fails as the container jobs are the wrong type for the scheduler.

This patch fixes this issue by suppressing the default behaviour of
scheduling jobs for container jobs.

Target: master
Patch: http://rb.dcache.org/r/6815
Acked-by: Gerd Behrmann
srm: add description of job states

In SRM, jobs have a defined state; however, the semantics of these
states isn't described and isn't always clear.

This patch adds description of the states.

Target: master
Patch: http://rb.dcache.org/r/6806/
Acked-by: Gerd Behrmann
chimera: fix NPE if mkdir fails with permission denied

If a mkdir operation fails then the error message contains the path.
The method to calculate the path throws a NPE if the subdirectory is
in the (apparent) root directory.

This patch fixes that problem.

Target: master
Patch: http://rb.dcache.org/r/6805
Acked-by: Tigran Mkrtchyan
srm: allow jobs in RQUEUED and READY states to survive restart

Jobs in RQUEUED and READY states require no further processing from
backend before presenting to client, or are waiting for client to
move the job into a final state.  Therefore it is safe to allow
such jobs to survive the SRM restart.

Target: master
srm: Fix scheduling period for expiration task

I believe the intention was to schedule the task every 60 seconds, not every
60 milliseconds.

Target: trunk
Request: 2.8
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6816/
xrootd: Shutdown channel factory on shutdown

This will release resources allocated by Netty.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6812/
srm: Break cyclic dependencies

I observed the fulling NPE during SRM startup:

> 26 Mar 2014 20:20:37 (System) [] Bug found, please report the following information to support@dcache.org
> java.lang.RuntimeException: URL [file:/root/dcache-2.9.0-SNAPSHOT/share/services/srm.batch]: line 262: java.lang.NullPointerException
>         at dmg.cells.nucleus.CellShell.execute(CellShell.java:1653) ~[cells-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
>         at dmg.cells.nucleus.CellShell.execute(CellShell.java:1563) ~[cells-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
>         at org.dcache.boot.Domain.executeBatchFile(Domain.java:270) ~[dcache-core-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
>         at org.dcache.boot.Domain.executeService(Domain.java:216) ~[dcache-core-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
>         at org.dcache.boot.Domain.start(Domain.java:129) ~[dcache-core-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
>         at org.dcache.boot.BootLoader.main(BootLoader.java:122) ~[dcache-core-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
> java.lang.NullPointerException: null
>         at diskCacheV111.srm.dcache.UnpinCompanion.unpinFile(UnpinCompanion.java:142) ~[dcache-core-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
>         at diskCacheV111.srm.dcache.Storage.unPinFile(Storage.java:1034) ~[dcache-core-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
>         at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) ~[na:na]
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
>         at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
>         at org.dcache.commons.stats.MonitoringProxy.invoke(MonitoringProxy.java:47) ~[dcache-common-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
>         at com.sun.proxy.$Proxy28.unPinFile(Unknown Source) ~[na:na]
>         at org.dcache.srm.request.GetFileRequest.stateChanged(GetFileRequest.java:476) ~[srm-server-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
>         at org.dcache.srm.request.Job.setState(Job.java:344) ~[srm-server-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
>         at org.dcache.srm.request.Job.onSrmRestart(Job.java:1111) ~[srm-server-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
>         at org.dcache.srm.scheduler.SchedulerContainer.restoreJobsOnSrmStart(SchedulerContainer.java:115) ~[srm-server-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
>         at org.dcache.srm.request.sql.DatabaseJobStorageFactory.restoreJobsOnSrmStart(DatabaseJobStorageFactory.java:144) ~[srm-server-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
>         at org.dcache.srm.SRM.start(SRM.java:314) ~[srm-server-2.9.0-SNAPSHOT.jar:2.9.0-SNAPSHOT]
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_51]
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_51]
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
>         at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1681) ~[spring-beans-4.0.2.RELEASE.jar:4.0.2.RELEASE]
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1620) ~[spring-beans-4.0.2.RELEASE.jar:4.0.2.RELEASE]
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1549) ~[spring-beans-4.0.2.RELEASE.jar:4.0.2.RELEASE]
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539) ~[spring-beans-4.0.2.RELEASE.jar:4.0.2.RELEASE]
>         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) ~[spring-beans-4.0.2.RELEASE.jar:4.0.2.RELEASE]
>         at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:304) ~[spring-beans-4.0.2.RELEASE.jar:4.0.2.RELEASE]
>         at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[spring-beans-4.0.2.RELEASE.jar:4.0.2.RELEASE]
>         at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:300) ~[spring-beans-4.0.2.RELEASE.jar:4.0.2.RELEASE]
>         at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195) ~[spring-beans-4.0.2.RELEASE.jar:4.0.2.RELEASE]

This happens due to cyclic initialization dependencies, which causes requests to
be restored before Storage is fully initialized.

The patch resolves this by breaking the initialization dependencies. This involved
moving some code out of the Storage class, which has the nice side effect of
increasing cohesion.

I deleted an older info provider webservice that wasn't used and which I guess
nobody knew about.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6817/
srm: Added missing import

Target: trunk
Patch: none
srm: fix restore so jobs for other schedulers are not restored

The dCache may have multiple SRM instances and jobs are associated
with a specific scheduler.

Currently on startup, jobs may be restored that are associated with
other SRM instances.  This may lead to the job being processed
multiple times, leading to inconsistent results.

This patch alters the behaviour so only jobs for the named scheduler
will be started.  It also adds a check that jobs that have been
assigned a scheduler are submitted only to that scheduler.

Target: master
Patch: http://rb.dcache.org/r/6826/
Acked-by: Gerd Behrmann
srm: fix unit-tests after recent update

A recent commit broke unit tests.  This patch restores them to a
working condition by adding missing information to the mocked
objects.

Target: master
srm: fix bring-online rescheduling on restart

Several states of BringOnlineFileRequest were not handled correctly.
This patch fixes this.

Target: master
Patch: http://rb.dcache.org/r/6807/
Acked-by: Gerd Behrmann