srm: Split standalone configuration from generic configuration

Large parts of the Configuration class is only used by the standalone server.
This patch moves that part to the subclass specific to the standalone server.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6959/
core: split protocol name form the client IP in HttpProtocolInfo

Acked-by: Gerd Behrmann
Target: master, 2.8, 2.9
Require-book: no
Require-notes: no
pool: Remove legacy hsm initialization from default pool setup

Otherwise pool startup spams the log files.

Target: trunk
Request: 2.9
Require-notees: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6958/
statistics: Fix NPE

Fixes

java.lang.NullPointerException: null
        at diskCacheV111.services.PoolStatisticsV0.printTotal(PoolStatisticsV0.java:1219) ~[dcache-core-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at diskCacheV111.services.PoolStatisticsV0.prepareDailyHtml(PoolStatisticsV0.java:1151) ~[dcache-core-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at diskCacheV111.services.PoolStatisticsV0.prepareDailyHtmlFiles(PoolStatisticsV0.java:1056) ~[dcache-core-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at diskCacheV111.services.PoolStatisticsV0.access$1100(PoolStatisticsV0.java:119) ~[dcache-core-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at diskCacheV111.services.PoolStatisticsV0$HourlyRunner.run(PoolStatisticsV0.java:441) ~[dcache-core-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$3.run(CellNucleus.java:707) [cells-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]

If there is no data, we cannot print a total.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6963/
srm: Report correct state for RQUEUED and TQUEUED

An SRM request goes through the states SRM_REQUEST_QUEUED and
SRM_REQUEST_INPROGRESS, to SRM_FILE_PINNED, SRM_SPACE_AVAILABLE and
SRM_SUCCESS.

Internally a request passes from PENDING to TQUEUED, to alternating
between ASYNCWAIT, PRIORITYTQUEUED, RUNNING, RUNNINGWITHOUTTHREAD, to
RQUEUED, READY and DONE.

There were a number of errors in mapping the internal state to the
external state: RQUEUED was sometimes reported as SRM_REQUEST_QUEUED,
even though this is rather the last step of SRM_REQUEST_INPROGRESS.
TQUEUED on the other hand was often reported as SRM_REQUEST_INPROGRESS,
even though it is the state before any processing has been done and
thus should be reported as SRM_REQUEST_QUEUED.

Restore from database was also changed to reset TQUEUED back to
PENDING, since such jobs have not begun processing.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paulmillar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6951/
srm: Align SRM scheduler with SRM request states

An SRM request usually goes through the states SRM_REQUEST_QUEUED,
SRM_REQUEST_INPROGRESS, SRM_FILE_PINNED/SRM_SPACE_AVAILABLE to
SRM_SUCCESS.

Internally the SRM scheduler used a somewhat more complicated flow with several
queuing phases: An intial queuing phase before any processing, called TQUEUED,
followed by a number of synchronous or asynchronous operations that each go
through another queuing phase called PRIORITYTQUEUED. Both types of queuing
were attempting to introduce fairness between request owners.

Since most requests are processing asynchronously, neither of those queuing
phases did have much of an effect. When a request is processed, it quickly
submits a request to some other component in dCache and enters the ASYNCWAIT
state. After this the thread pool is free to process the next request, meaning
that any queued request is quickly submitted for execution, and all requests
are active at the same time.

A notable exception to the previous paragraph are srm copy jobs, as these are
implemented synchronously. Thus the tuning parameters for the scheduler have a
quite different effect on srm copy than on put or get.

This patch simplifies the SRM scheduler to more align with the basic states at
the SRM protocol level. The interface and state names are deliberately
unchanged to keep the patch small.

A request now enters an initial queuing phase indicated by the PENDING and
TQUEUED states (as before). When capacity allows, such a queued job is chosen
(taken fairness contraints into account) and moved to the INPROGRESS phase
indicated by the PRIORITYTQUEUED, RUNNING, RUNNINGWITHOUTTHREAD, ASYNCWAIT
states. At this point the requests run on a standard Java Executor thread pool,
with no additional fairness considerations or limits. Once completed, the
request enters RQUEUED like before.

The change is the simplification of PRIORITYTQUEUED to run of a standard Java
Executor. Throttling happens when to move the request from TQUEUED to
PRIORITYTQUEUED (ie from QUEUED to INPROGRESS). This is only done when RQUEUED
has sufficient capacity to hold the additional requests, and if the limit for
INPROGRESS requests hasn't been reached.

The meaning of configuration parameters has changed slightly to map to the
externally visible states rather than the implementation specific internal
states. I kept the existing property names, although a future patch should
update the names to something more suitable. The semantics are close enough
to what we had before that existing configurations should work fine.

An issue still is that requests that do not use RQUEUED/READY (such as list
requests) still are subject to the RQUEUED capacity check. This is more a
consequence of the needless configuration properties for such requests and
should be fixed in a future patch (setting the internal default to MAX_VALUE
and removing the properties for requests without those states).

The request retry cycle has been changed to reset jobs to the TQUEUED state
when retrying. Those on retry jobs reenter the initial queued phase, and
jobs waiting for retry do not count as INPROGRESS.

As mentioned above, I deliberately kept most of the mechanics of the scheduler
to keep the patch small. It is obvious that after this patch, the scheduler
can be simplified, and much of the actual request execution can be decoupled
from the scheduler (as it now uses a simple Executor rather than the full
scheduler logic).

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6953/
srm: Simplify unpin callbacks

The callback was essentially only used for logging, and in three
of the four cases the unpin call was blocking already and didn't
need a callback.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6955/
fhs: Use gzip compression for deb package

Ubuntu 14.04 defaults to xv which isn't supported by Ubuntu 10.10.
To restore compatibility, this patch explicitly sets the compression
format to gzip.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: no
Require-book: no
Silence log messages commonly associated with client disconnects

When an FTP client disconnects prematurely, the FTP door deletes the incomplete
file and log the disconnect. Other services log errors too due to the missing
file and missing door:

06 May 2014 12:57:38 (PnfsManager) [extraload_ndgf_org_001 PnfsFlag 00006E16928BADB447EE8B10BDB9F90D47A4] Exception in updateFlag: CacheException(rc=10001;msg=No such file or directory: 00006E16928BADB447EE8B10BDB9F90D47A4)
06 May 2014 12:57:38 (SrmSpaceManager) [door:GFTP-bunsen-5413@dCacheDomain:1399373677151 extraload_ndgf_org_001 DoorTransferFinished 00006E16928BADB447EE8B10BDB9F90D47A4] Failed to forward message: {uoid=<1399373852975:39246>;path=[*@dCacheDomain:PoolManager@dCacheDomain:SrmSpaceManager@dCacheDomain:>GFTP-bunsen-5413@dCacheDomain];msg=GFTP-bunsen-5413@dCacheDomain not found}

These are expected failurs in case of a disconnected FTP client. Since the door
already logged the fact, there is no reason for other services to log these
errors.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6971/
ssh1: Remove service from dCache

Code will be removed later.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
srm: Simplify scheduler configuration

The SRM scheduler has traditionally been quite cryptic to anybody not well
versed in the Scheduler internals - and not many people were. This patch
attempts to rectify this situation.

The patch introduces three configuration properties:

srm.request.max-requests   - max number of non-final requests
srm.request.max-inprogress - max number of requests being prepared
srm.request.max-transfers  - max number of turls

In addition, request specific versions of these properties are introduced.

The srm.request.{thread-queue-size,max-waiting-requests,ready-queue-size}
properties are obsolete.  The srm.request.max-ready-requests property is
deprecated. It's value is used for the new max-transfers setting.

The semantics of the max-by-same-user setting changes slightly in that it
now controls the transition from QUEUED to INPROGRESS and all requests
INPROGRESS and transferring count towards the limit.

There are no individual limits for the number of requests in TQUEUED or
RQUEUED. I think it is easier to understand a limit on the total number of
requests, and we shouldn't really care whether a request is in TQUEUED or
RQUEUED.

At least to me, these settings are far easier to understand: There is a limit
on the total number of requests before we begin to reject new requests; there
is a limit on how many requests we prepare concurrently (the
SRM_REQUEST_INPROGRESS status of SRM) and thus how much load we induce on other
services; and there is a limit on the number of concurrent client transfers
(number of turls).

The load reported to login broker has changed. It is now the total number of
requests in SRM_REQUEST_QUEUED or SRM_REQUESTS_INPROGRESS divided by the
max-inprogress setting. Thus when load is above 100%, new requests are queued.

The patch also applies some minor refactoring to the Scheduler and
ModifiableQueue classes.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6965/
srm: Minor cleanup of job scheduling

- Initialize Scheduler defaults to 0 to fail fast in case
  of missing configuration.

- Don't set scheduler id twice

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6969/
scripts: fix credentials script to work for non-Linux machines

Some commands proved to be dependent on the OS and OpenSSL version on
the running machines.  These are tidied up.

The install procedure for trusting the generated certificates has been
made more robust.

Target: master
Patch: http://rb.dcache.org/r/6967/
Acked-by: Albert Rossi
pnfsmanager: Fix expected size extraction and tag caching

The expected file size tag is not extracted correctly when creating new
files. The tag caching also wasn't as effective as intended: To fix
this I had to add caching of the parent tag.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.9
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6975/
srm: Drop fail fast for protocol checks

For both get and put the requested protocol is compared to supported
protocols before creating the schedulable request objects. I guess
this was supposed to provide a fail-fast path. However it means we

- Don't leave behind the request object for subsequent analysis.

- Do a non-trivial operation in the Jetty thread when we have
  a perfectly fine scheduler to deal with this.

- Duplicate checks as the Storage class already checks this while
  selecting a door.

This patch gets rids of the duplication. I submit this for 2.9 as is
resolves a regression (in the old code the protocols were fetched
differently).

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6977/
srm: Schedule resubmit of pin request to avoid deadlock

Since the callback happens on the delivery thread, we shouldn't do
anything heavy in it (such as doing a protocol support check that
may involve querying login broker - something that would cause a
deadlock as the reply would not be deliverable).

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6978/
srm: Fix deadlock in pinning

Another case of me being too optimistic about how much or how little
happens in a message callback. In this case pinning uses the remote
pool monitor, which may tricker a fetch from pool manager.

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6979/
core: remove obsolete code

as pointed out by Al, after HSM interface refactoring, the
old sheduler is not used any more.

Acked-by: Albert Rossi
Target: master
Require-book: no
Require-notes: no
httpd: log bugs with stacktrace

Log a stack-trace and return appropriate HTTP response status code if
a bug is detected.

Target: master
Patch: http://rb.dcache.org/r/6973/
Acked-by: Tigran Mkrtchyan
systemtest: regenerate host credentials, add user credentials

Patch adds new host credentials so the system-test continues to work
out-of-the-box.  The patch also adds user-certificates generated by
the same Disposable CA.  This should allow people without an X.509
certificate to use system-test without first regenerating credentials.

Target: master
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6