srm: Align SRM scheduler with SRM request states

An SRM request usually goes through the states SRM_REQUEST_QUEUED,
SRM_REQUEST_INPROGRESS, SRM_FILE_PINNED/SRM_SPACE_AVAILABLE to
SRM_SUCCESS.

Internally the SRM scheduler used a somewhat more complicated flow with several
queuing phases: An intial queuing phase before any processing, called TQUEUED,
followed by a number of synchronous or asynchronous operations that each go
through another queuing phase called PRIORITYTQUEUED. Both types of queuing
were attempting to introduce fairness between request owners.

Since most requests are processing asynchronously, neither of those queuing
phases did have much of an effect. When a request is processed, it quickly
submits a request to some other component in dCache and enters the ASYNCWAIT
state. After this the thread pool is free to process the next request, meaning
that any queued request is quickly submitted for execution, and all requests
are active at the same time.

A notable exception to the previous paragraph are srm copy jobs, as these are
implemented synchronously. Thus the tuning parameters for the scheduler have a
quite different effect on srm copy than on put or get.

This patch simplifies the SRM scheduler to more align with the basic states at
the SRM protocol level. The interface and state names are deliberately
unchanged to keep the patch small.

A request now enters an initial queuing phase indicated by the PENDING and
TQUEUED states (as before). When capacity allows, such a queued job is chosen
(taken fairness contraints into account) and moved to the INPROGRESS phase
indicated by the PRIORITYTQUEUED, RUNNING, RUNNINGWITHOUTTHREAD, ASYNCWAIT
states. At this point the requests run on a standard Java Executor thread pool,
with no additional fairness considerations or limits. Once completed, the
request enters RQUEUED like before.

The change is the simplification of PRIORITYTQUEUED to run of a standard Java
Executor. Throttling happens when to move the request from TQUEUED to
PRIORITYTQUEUED (ie from QUEUED to INPROGRESS). This is only done when RQUEUED
has sufficient capacity to hold the additional requests, and if the limit for
INPROGRESS requests hasn't been reached.

The meaning of configuration parameters has changed slightly to map to the
externally visible states rather than the implementation specific internal
states. I kept the existing property names, although a future patch should
update the names to something more suitable. The semantics are close enough
to what we had before that existing configurations should work fine.

An issue still is that requests that do not use RQUEUED/READY (such as list
requests) still are subject to the RQUEUED capacity check. This is more a
consequence of the needless configuration properties for such requests and
should be fixed in a future patch (setting the internal default to MAX_VALUE
and removing the properties for requests without those states).

The request retry cycle has been changed to reset jobs to the TQUEUED state
when retrying. Those on retry jobs reenter the initial queued phase, and
jobs waiting for retry do not count as INPROGRESS.

As mentioned above, I deliberately kept most of the mechanics of the scheduler
to keep the patch small. It is obvious that after this patch, the scheduler
can be simplified, and much of the actual request execution can be decoupled
from the scheduler (as it now uses a simple Executor rather than the full
scheduler logic).

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6953/