systemtest: add support for generating user credentials

Add support for the --user option.  This will use the disposable
CA to generate user credentials in addition to the host credentials.

The existing generated certificate's CN and subjectAltName (email) are
used as default CN and subjectAltName, if present, otherwise the git
'user.name' and 'user.email' config options are used.  Either may be
overwritten by specifying command-line options.

The script will store these credentials in ~/.globus.  If there is
no existing default credential then it will set the freshly created
credential as the default; but, otherwise, it will leave the existing
default credential alone.

If --generate is specified and existing user credentials are detected
then the --user option is assumed.

Target: master
Patch: http://rb.dcache.org/r/6954/
Acked-by: Gerd Behrmann
alarms, billing: fix liquibase character varying and cleanup.

The latest liquibase (all 3.x versions) has a problem - it generated
invalid SQL:

    https://liquibase.jira.com/browse/CORE-1853

The solution is to update all changesets to use 'VARCHAR(xxx)' instead of
'CHARACTER VARYING(xxx)'.

The affected changesets all seem to be in alarms and billing.

This patch, in addition to applying these necessary changes, has
also cleaned up the logic on index creation:

1. errors should be fail-fast, so onError=WARN have been eliminated
   from these table creation routines.
2. each index is given a separate changeset with a separate pre-condition.
3. the pre-existence of the table with which the indices are associated
   is assumed from the failure logic and sequence of changesets itself.

We have also eliminated a changeset from alarms which creates an unused
table.

The older affected changesets have been eliminated and their newer
versions all given new names/ids.  While this will leave changeset
entries in the database changeset log table, the changes should
be idempotent inasmuch as the preconditions will prevent them from
running if the tables and indices already exist.

Inasmuch as the changes here actually do constitute fixes to
the liquibase logic, I believe this justifies backporting
the entire patch at least to 2.9.

Testing Done: deployed over fresh installation and pre-existing installation of the databases.

Target: master
Request: 2.9
Require-book: no
Require-notes: yes
Acked-by: Dmitry

RELEASE NOTES:  fixes a regression in liquibase scripts for alarms and billing
due to a bug in updated version (3.x) producing incorrect SQL for 'character varying'.
srm: fix retry of copy-pull

When srmCopy copies a file from a remote site it issues a prepareToPut and putDone
when the transfer is successful.  If the transfer is unsuccessful then an abortPut
is called, which destroys the upload directory.

The current behaviour attempts to reuse the transfer TURL when retrying.  Since
abortPut has been called, this directory no longer exists and the transfer-manager
will fail the request as it discovers this.

This patch fixes the problem by issuing prepareToPut when retrying.

Target: master
Patch: http://rb.dcache.org/r/6956/
Acked-by: Gerd Behrmann
Request: 2.9
srm: Split standalone configuration from generic configuration

Large parts of the Configuration class is only used by the standalone server.
This patch moves that part to the subclass specific to the standalone server.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6959/
core: split protocol name form the client IP in HttpProtocolInfo

Acked-by: Gerd Behrmann
Target: master, 2.8, 2.9
Require-book: no
Require-notes: no
pool: Remove legacy hsm initialization from default pool setup

Otherwise pool startup spams the log files.

Target: trunk
Request: 2.9
Require-notees: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6958/
statistics: Fix NPE

Fixes

java.lang.NullPointerException: null
        at diskCacheV111.services.PoolStatisticsV0.printTotal(PoolStatisticsV0.java:1219) ~[dcache-core-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at diskCacheV111.services.PoolStatisticsV0.prepareDailyHtml(PoolStatisticsV0.java:1151) ~[dcache-core-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at diskCacheV111.services.PoolStatisticsV0.prepareDailyHtmlFiles(PoolStatisticsV0.java:1056) ~[dcache-core-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at diskCacheV111.services.PoolStatisticsV0.access$1100(PoolStatisticsV0.java:119) ~[dcache-core-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at diskCacheV111.services.PoolStatisticsV0$HourlyRunner.run(PoolStatisticsV0.java:441) ~[dcache-core-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$3.run(CellNucleus.java:707) [cells-2.10.0-SNAPSHOT.jar:2.10.0-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]

If there is no data, we cannot print a total.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6963/
srm: Report correct state for RQUEUED and TQUEUED

An SRM request goes through the states SRM_REQUEST_QUEUED and
SRM_REQUEST_INPROGRESS, to SRM_FILE_PINNED, SRM_SPACE_AVAILABLE and
SRM_SUCCESS.

Internally a request passes from PENDING to TQUEUED, to alternating
between ASYNCWAIT, PRIORITYTQUEUED, RUNNING, RUNNINGWITHOUTTHREAD, to
RQUEUED, READY and DONE.

There were a number of errors in mapping the internal state to the
external state: RQUEUED was sometimes reported as SRM_REQUEST_QUEUED,
even though this is rather the last step of SRM_REQUEST_INPROGRESS.
TQUEUED on the other hand was often reported as SRM_REQUEST_INPROGRESS,
even though it is the state before any processing has been done and
thus should be reported as SRM_REQUEST_QUEUED.

Restore from database was also changed to reset TQUEUED back to
PENDING, since such jobs have not begun processing.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paulmillar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6951/
srm: Align SRM scheduler with SRM request states

An SRM request usually goes through the states SRM_REQUEST_QUEUED,
SRM_REQUEST_INPROGRESS, SRM_FILE_PINNED/SRM_SPACE_AVAILABLE to
SRM_SUCCESS.

Internally the SRM scheduler used a somewhat more complicated flow with several
queuing phases: An intial queuing phase before any processing, called TQUEUED,
followed by a number of synchronous or asynchronous operations that each go
through another queuing phase called PRIORITYTQUEUED. Both types of queuing
were attempting to introduce fairness between request owners.

Since most requests are processing asynchronously, neither of those queuing
phases did have much of an effect. When a request is processed, it quickly
submits a request to some other component in dCache and enters the ASYNCWAIT
state. After this the thread pool is free to process the next request, meaning
that any queued request is quickly submitted for execution, and all requests
are active at the same time.

A notable exception to the previous paragraph are srm copy jobs, as these are
implemented synchronously. Thus the tuning parameters for the scheduler have a
quite different effect on srm copy than on put or get.

This patch simplifies the SRM scheduler to more align with the basic states at
the SRM protocol level. The interface and state names are deliberately
unchanged to keep the patch small.

A request now enters an initial queuing phase indicated by the PENDING and
TQUEUED states (as before). When capacity allows, such a queued job is chosen
(taken fairness contraints into account) and moved to the INPROGRESS phase
indicated by the PRIORITYTQUEUED, RUNNING, RUNNINGWITHOUTTHREAD, ASYNCWAIT
states. At this point the requests run on a standard Java Executor thread pool,
with no additional fairness considerations or limits. Once completed, the
request enters RQUEUED like before.

The change is the simplification of PRIORITYTQUEUED to run of a standard Java
Executor. Throttling happens when to move the request from TQUEUED to
PRIORITYTQUEUED (ie from QUEUED to INPROGRESS). This is only done when RQUEUED
has sufficient capacity to hold the additional requests, and if the limit for
INPROGRESS requests hasn't been reached.

The meaning of configuration parameters has changed slightly to map to the
externally visible states rather than the implementation specific internal
states. I kept the existing property names, although a future patch should
update the names to something more suitable. The semantics are close enough
to what we had before that existing configurations should work fine.

An issue still is that requests that do not use RQUEUED/READY (such as list
requests) still are subject to the RQUEUED capacity check. This is more a
consequence of the needless configuration properties for such requests and
should be fixed in a future patch (setting the internal default to MAX_VALUE
and removing the properties for requests without those states).

The request retry cycle has been changed to reset jobs to the TQUEUED state
when retrying. Those on retry jobs reenter the initial queued phase, and
jobs waiting for retry do not count as INPROGRESS.

As mentioned above, I deliberately kept most of the mechanics of the scheduler
to keep the patch small. It is obvious that after this patch, the scheduler
can be simplified, and much of the actual request execution can be decoupled
from the scheduler (as it now uses a simple Executor rather than the full
scheduler logic).

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6953/
srm: Simplify unpin callbacks

The callback was essentially only used for logging, and in three
of the four cases the unpin call was blocking already and didn't
need a callback.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6955/
fhs: Use gzip compression for deb package

Ubuntu 14.04 defaults to xv which isn't supported by Ubuntu 10.10.
To restore compatibility, this patch explicitly sets the compression
format to gzip.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: no
Require-book: no
Silence log messages commonly associated with client disconnects

When an FTP client disconnects prematurely, the FTP door deletes the incomplete
file and log the disconnect. Other services log errors too due to the missing
file and missing door:

06 May 2014 12:57:38 (PnfsManager) [extraload_ndgf_org_001 PnfsFlag 00006E16928BADB447EE8B10BDB9F90D47A4] Exception in updateFlag: CacheException(rc=10001;msg=No such file or directory: 00006E16928BADB447EE8B10BDB9F90D47A4)
06 May 2014 12:57:38 (SrmSpaceManager) [door:GFTP-bunsen-5413@dCacheDomain:1399373677151 extraload_ndgf_org_001 DoorTransferFinished 00006E16928BADB447EE8B10BDB9F90D47A4] Failed to forward message: {uoid=<1399373852975:39246>;path=[*@dCacheDomain:PoolManager@dCacheDomain:SrmSpaceManager@dCacheDomain:>GFTP-bunsen-5413@dCacheDomain];msg=GFTP-bunsen-5413@dCacheDomain not found}

These are expected failurs in case of a disconnected FTP client. Since the door
already logged the fact, there is no reason for other services to log these
errors.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6971/
ssh1: Remove service from dCache

Code will be removed later.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
srm: Simplify scheduler configuration

The SRM scheduler has traditionally been quite cryptic to anybody not well
versed in the Scheduler internals - and not many people were. This patch
attempts to rectify this situation.

The patch introduces three configuration properties:

srm.request.max-requests   - max number of non-final requests
srm.request.max-inprogress - max number of requests being prepared
srm.request.max-transfers  - max number of turls

In addition, request specific versions of these properties are introduced.

The srm.request.{thread-queue-size,max-waiting-requests,ready-queue-size}
properties are obsolete.  The srm.request.max-ready-requests property is
deprecated. It's value is used for the new max-transfers setting.

The semantics of the max-by-same-user setting changes slightly in that it
now controls the transition from QUEUED to INPROGRESS and all requests
INPROGRESS and transferring count towards the limit.

There are no individual limits for the number of requests in TQUEUED or
RQUEUED. I think it is easier to understand a limit on the total number of
requests, and we shouldn't really care whether a request is in TQUEUED or
RQUEUED.

At least to me, these settings are far easier to understand: There is a limit
on the total number of requests before we begin to reject new requests; there
is a limit on how many requests we prepare concurrently (the
SRM_REQUEST_INPROGRESS status of SRM) and thus how much load we induce on other
services; and there is a limit on the number of concurrent client transfers
(number of turls).

The load reported to login broker has changed. It is now the total number of
requests in SRM_REQUEST_QUEUED or SRM_REQUESTS_INPROGRESS divided by the
max-inprogress setting. Thus when load is above 100%, new requests are queued.

The patch also applies some minor refactoring to the Scheduler and
ModifiableQueue classes.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6965/
srm: Minor cleanup of job scheduling

- Initialize Scheduler defaults to 0 to fail fast in case
  of missing configuration.

- Don't set scheduler id twice

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6969/
scripts: fix credentials script to work for non-Linux machines

Some commands proved to be dependent on the OS and OpenSSL version on
the running machines.  These are tidied up.

The install procedure for trusting the generated certificates has been
made more robust.

Target: master
Patch: http://rb.dcache.org/r/6967/
Acked-by: Albert Rossi
pnfsmanager: Fix expected size extraction and tag caching

The expected file size tag is not extracted correctly when creating new
files. The tag caching also wasn't as effective as intended: To fix
this I had to add caching of the parent tag.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.9
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6975/
srm: Drop fail fast for protocol checks

For both get and put the requested protocol is compared to supported
protocols before creating the schedulable request objects. I guess
this was supposed to provide a fail-fast path. However it means we

- Don't leave behind the request object for subsequent analysis.

- Do a non-trivial operation in the Jetty thread when we have
  a perfectly fine scheduler to deal with this.

- Duplicate checks as the Storage class already checks this while
  selecting a door.

This patch gets rids of the duplication. I submit this for 2.9 as is
resolves a regression (in the old code the protocols were fetched
differently).

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6977/
srm: Schedule resubmit of pin request to avoid deadlock

Since the callback happens on the delivery thread, we shouldn't do
anything heavy in it (such as doing a protocol support check that
may involve querying login broker - something that would cause a
deadlock as the reply would not be deliverable).

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6978/
srm: Fix deadlock in pinning

Another case of me being too optimistic about how much or how little
happens in a message callback. In this case pinning uses the remote
pool monitor, which may tricker a fetch from pool manager.

Target: trunk
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6979/