srm-client: fix RPM packaging

Previous commit introduced an additional file without updating the
RPM packaging.  This patch fixes this oversight.

Target: master
Patch: http://rb.dcache.org/r/6764/
Acked-by: Tigran Mkrtchyan
srm: refactor SchedulerFactory

This patch modifies SchedulerFactory so it has a single concern:
creating Scheduler instances.  The current implementation of
SchedulerFactory has two other concerns: creating all necessary
schedulers and maintaining a collection of schedulers. These
additional activities are moved elsewhere.

Creating all necessary schedulers has moved to the spring XML file.

Most of the behaviour associated with multiple schedules has moved
into the new SchedulerContainer class and the process of submitting a
Job to a scheduler is achieved via this new class.

One design goal is that SchedulerContainer should not 'leak' Scheduler
instances.  The getScheduler-like methods has been removed: both from
SchedulerContainer/SchedulerFactory and from SRM class.  Activity that
iterates over all the Schedulers has moved to SchedulerContainer.
Activity that targets a specific scheduler is implemented with a
method that accepts a Class<? extends Job> argument to target a
specific Scheduler.

Note that the current code has the Job classes being responsible for
adding themselves to a scheduler.  To try to keep the patch short,
this mode of operation is maintained; however, Job is supplied with a
Scheduler rather than discovering the scheduler itself.

The handler classes must be able to add a Job to a scheduler.  The
patch achieves this via the SRM object passed to the handler's
constructor.  Future patch may refactor this so the object to which a
Job is submitted is supplied on the constructor.

Another future direction is to make Scheduler an interface that the
current Scheduler class implements.  The SchedulerContainer would also
implement this interface and the similarity between SchedulerContainer
and Scheduler becomes clearer.

Another beneficial side-effect from this patch is that
SchedulerFactory is no longer a singleton.

Target: master
Requires-book: no
Requires-notes: no
Patch: http://rb.dcache.org/r/6321/
Acked-by: Gerd Behrmann
srm: remove unused state

A job is never assigned the state RESTORED (perhaps as the state doesn't make
sense).  Because of this, the code that depends on a Job being in state RESTORED
is never called.

This patch removes this dead-wood

Target: master
Patch: http://rb.dcache.org/r/6650
Acked-by: Gerd Behrmann
Acked-by: Dmitry Litvintsev
logback: correct MDC dependency in appender

MDC dependency should be to slf4j, not log4j.
The error has gone unnoticed because we do use
log4j-over-slf4j, but the import should not
be tightly coupled to it.

Target: master
Patch: http://rb.dcache.org/r/6777/
Require-notes: no
Require-book: no
Acked-by: Paul
remote logging: add empty string defense to remote server wrapper

See patch 6719 and logback bug report: http://jira.qos.ch/browse/LOGBACK-959

As a workaround which allows us to update to logback 1.1.1, I have added
defensive code to the String setters which converts empty strings to nulls,
which are properly handled.

I have also changed returns to System.exit, since we require this component
to run in its own domain, and have logged exceptions which may affect the
logging system itself using System.err.

Testing: Before changes, the logback 1.1.1 libraries throw the NullPointer exception;
with the changes, the server boots correctly and an alarm can be sent.

Target: master
Patch: http://rb.dcache.org/r/6775/
Require-notes: no
Require-book: no
Acked-by: Gerd
chimera: allow updating the creation time

A recent patch introduced the possibility to update the creation time
via FTP.  Unfortunately, the creation time wasn't actually updated.
This patch fixes this.

Target: master
Patch: http://rb.dcache.org/r/6790/
Acked-by: Tigran Mkrtchyan
loginbroker: Fix login broker registration retry

Fixes a problem in which doors would log "Failed to send update to LoginBroker"
in a busy retry loop when unable to register with LoginBroker. This typically
happens during shutdown.

The intention of the code was to switch to an eager retry loop with a one
second sleep between retries, however the code switches the update period to
'normal' update before attempting to send an update and then switches back to
'eager' mode. Changing the scheduling time causes the task to be rescheduled,
but rescheduling schedules a new update with an initial delay of zero, thus
causing an immediate retry. Thus failure to deliver an update initiates a
tight retry loop without any delays.

Also fixes a bug in which the 'lb set update' command was not respected.

Target: trunk
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6787/
Get rid of support for synchronous messages in CellNucleus

There is still a sendAndWait method in CellNucleus to support legacy
code, but it is implemented on top of the asynchronous logic.

sendAndWait is gone from CellAdapter and CellEndpoint. Higher level
code should use CellStub if synchronous behaviour is desired.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6758
Mock CellEndpoint rather than CellAdapter in HsmRestoreTest

Now that most of our code no longer depends directly on CellNucleus and
CellAdapter, we only have to mock CellEndpoint.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6760/
pnfsmanager: Clean up cleaner by making use of new CellStub features

A workaround for CellStub limitations is no longer necessary.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6765/
logback: add comment about bugs in 1.1.1 to poms

We are not quite ready to update to 1.1.1 yet because of the
issue which persists with respect to main not exiting in the
alarm send command.

I have added comments to the poms indicating the bug reports
I filed with logback.

Target: master
Patch: http://rb.dcache.org/r/6786/
Require-notes: no
Require-book: no
Acked-by: Gerd
ftp: add read-only support for ctime and atime

This patch adds support for the UNIX.ctime and UNIX.atime facts,
which allows clients to know a file's ctime and atime respectively.

Target: master
Patch: http://rb.dcache.org/r/6794/
Acked-by: Gerd Behrmann
pool: Fix lock problems in xrootd and http movers

When the pool is full, a blocking preallocate call could block
simple 'mover ls' queries and thus cause the entire pool to appear
unresponsive.

Target: trunk
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6795/
pool: Avoid memory exhaustion problems on write with xrootd and http

The http and xrootd movers use an OrderedMemoryAwayThreadPoolExecutor. This
thread pool limits memory usage per client, however it relies on being able to
estimate the size of the messages in the queue. It knows how to calculate the
size for ChannelBuffer objects, however it doesn't know how to do this for most
other objects. Since we store protocol messages in the queue, the limit doesn't
have the intended effect. The consequence is that a slow disk or full pool can
cause the queue to fill up and trigger OOM.

The patch solves this by moving the ExecutorHandler to the front of the
pipeline such that only ChannelBuffers are stored in the queue.

Target: trunk
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6796/
srm: Restore RESTORED state

A recent patch removed the RESTORED state from the SRM. This however causes
problems upgrading existing systems as they are unable to map value of state
fields in the database to the enum in the code. This patch fixes the regression
by adding the enum value back.

The patch also fixes a cut'n'paste error message in AccessLatency.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6800/
billing: Make indexer case insensitive

The billing indexer turns the billing output patterns into regular expressions
and uses those when parsing billing files. Literal text in the pattern are
translated to literal text in the regular expression.

Unfortunately, at some point we altered the capitalization of 'Unknown' in the
output pattern. This causes the billing indexer to fail to match older billing
files.

The patch changes the pattern to case insensitive matching. The generated bloom
filter is still case sensitive.

It is recomended that existing indexes are reindexed using the
'/usr/sbin/dcache-billing-indexer -all' command.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.8
Request: 2.7
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6798/
srm: Terminate db threads on shutdown

The patch ensures that db related threads are terminated when the SRM
shuts down. This both addresses warnings during dCache shutdown about
unterminated threads, and it addresses attempted database accesses
after the connection pool has been shut down.

The database garbage collector threads have been replaced by a single
scheduled executor. It is single threaded to reduce the load the
low priority GC activity puts on the database.

The thread pools have been configured to use named threads.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6788/
poolmanager: Get rid of ThreadPool interface

Use a regular Executor instead. The following properties are now
obsolete:

threadPool
poolmanager.plugins.thread-pool

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6785/
Revert "poolmanager: Get rid of ThreadPool interface"

This reverts commit d06eb680ecb31c79edca2889724824b9aeccc7ca.
nfs4: update stateid on read with proxy adapter

since we do support proxy-io with nfsv4.0 let be spec complaint and
renew the state.

RELEASE NOTES:
with proxy IO on READ dcache can support NFSv4.0 clients such as RHEL5 or mac osx 10.9 (OS X Mavericks)

Acked-by: Gerd Behrmann
Target: master
Require-notes: yes
Require-book: no