pool: wake-up sweeper if new entry is added

To avoid busy-loop sweeper uses Thread#sleep(). But, it makes
sense to wake-up if list of file is updated.

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
pool: small cleanups in sweeper

No logic changes

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
poolmanager: remove wrandom partition type

Acked-by: Gerd Behrmann
Target: master
Require-book: yes
Require-notes: yes
poolmanager,pool: Fix precision underflow in WASS

Fixes a problem in which a large number of writers on a pool (such as what you
would see on a tape read pool) would cause the scale of a double to underflow,
causing the weigthed acailable space to become zero.  If all pools have such
high load, no pool would be availble for selection and pool manager would
report that no pool is available for staging.

The fix uses the fact that only the relative weighted available space is
relevant. Since the weighted space is calculated by dividing unweighted space
with two to the power of the load, we can subtract a constant from the load of
all pools and maintain the relative available space of each pool. The patch
first finds the lowest load of all non-full pools in the selection and
subtracts that from the load of all pools, thus normalizing the calculation.

Other pools may have even high load, still causing the precision of a double to
be insufficient, however the likelihood of selection such a pool is virtuallt
zero anyway, so this barely affects the write distribution. The normalization
does however ensure that at least one pool does not underflow (assuming
non-full pools exist at all) and thus that a pool can be selected.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Dmitry Litvtinsev <litvinse@fnal.gov>
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7993
nfs: make cache time unit property name guideline compliant

Acked-by: Gerd Behrmann
Target: master, 2.9
Require-book: no
Require-notes: yes
nfs4: do not log stacktrace on IO errors

there are no reasons to log  full stack trace on IO errors

Acked-by: Paul Millar
Target: master, 2.9
databases: update hikari minimum connection configuration

Commit 004d4d0c updated the version of Hikari to v1.3.5.  This new version of Hikari removed
support for maintaining a minimum number of database connections.  Attempts to configure this
minimum number of connections result in a warning being logged:

    The minimumPoolSize property has been retired, remove it from your pool configuration to avoid
    this warning.

Hikari now supports maintaining a minimum number of idle connections.  This behaviour is
substantially different from the previous minimum number of connections, therefore this patch
forbids setting the old *.db.connections.min properties and new *.db.connections.idle properties
are introduced.

This patch also adds missing annotations to help sites migrate their database pool configuration
from BoneCP to Hikari.

Target: master
Request: 2.9
Patch: http://rb.dcache.org/r/6933/
Acked-by: Gerd Behrmann
Acked-by: Tigran Mkrtchyan
nfs: do not use CDC as try-with-resource

in a cunstruction like:

  try (CDC ignored = new CDC()) {
     ....
  } catch (Exception e) {
    log.error(e);
  }

the catch block will not have a correct context.
Use try-catch-finally instead.

Acked-by: Gerd Behrmann
Target: master, 2.9, 2.8, 2.7
Require-book: no
Require-notes: no
gplazma-nis: pulgin must fail if no matching principal found

Acked-by: Paul Millar
Target: master, 2.9
Require-book: no
Require-notes: no
poolmanager: Fix infinite loop in lru partition type

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6938/
srm: Fix race condition in listing

The ListDirectoryHandler only worked if replies arrived in sequence
and sequentially. For singe threaded cells this worked fine, however
srm now uses multithreaded message delivery and thus listing broke.

The fix updates the handler to deal with concurrent and out of order
reply delivery.

Target: trunk
Request: 2.9
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6946/
pool: Report correct client IP to billing for passive FTP transfers

The pool incorrectly reported the FTP proxy address of the FTP door
even when a direct connection between the client and the pool was
established.

I am undecided how far back to fix this. Please advice.

Target: trunk
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6940/
systemtest: add support for generating user credentials

Add support for the --user option.  This will use the disposable
CA to generate user credentials in addition to the host credentials.

The existing generated certificate's CN and subjectAltName (email) are
used as default CN and subjectAltName, if present, otherwise the git
'user.name' and 'user.email' config options are used.  Either may be
overwritten by specifying command-line options.

The script will store these credentials in ~/.globus.  If there is
no existing default credential then it will set the freshly created
credential as the default; but, otherwise, it will leave the existing
default credential alone.

If --generate is specified and existing user credentials are detected
then the --user option is assumed.

Target: master
Patch: http://rb.dcache.org/r/6954/
Acked-by: Gerd Behrmann
alarms, billing: fix liquibase character varying and cleanup.

The latest liquibase (all 3.x versions) has a problem - it generated
invalid SQL:

    https://liquibase.jira.com/browse/CORE-1853

The solution is to update all changesets to use 'VARCHAR(xxx)' instead of
'CHARACTER VARYING(xxx)'.

The affected changesets all seem to be in alarms and billing.

This patch, in addition to applying these necessary changes, has
also cleaned up the logic on index creation:

1. errors should be fail-fast, so onError=WARN have been eliminated
   from these table creation routines.
2. each index is given a separate changeset with a separate pre-condition.
3. the pre-existence of the table with which the indices are associated
   is assumed from the failure logic and sequence of changesets itself.

We have also eliminated a changeset from alarms which creates an unused
table.

The older affected changesets have been eliminated and their newer
versions all given new names/ids.  While this will leave changeset
entries in the database changeset log table, the changes should
be idempotent inasmuch as the preconditions will prevent them from
running if the tables and indices already exist.

Inasmuch as the changes here actually do constitute fixes to
the liquibase logic, I believe this justifies backporting
the entire patch at least to 2.9.

Testing Done: deployed over fresh installation and pre-existing installation of the databases.

Target: master
Request: 2.9
Require-book: no
Require-notes: yes
Acked-by: Dmitry

RELEASE NOTES:  fixes a regression in liquibase scripts for alarms and billing
due to a bug in updated version (3.x) producing incorrect SQL for 'character varying'.
srm: fix retry of copy-pull

When srmCopy copies a file from a remote site it issues a prepareToPut and putDone
when the transfer is successful.  If the transfer is unsuccessful then an abortPut
is called, which destroys the upload directory.

The current behaviour attempts to reuse the transfer TURL when retrying.  Since
abortPut has been called, this directory no longer exists and the transfer-manager
will fail the request as it discovers this.

This patch fixes the problem by issuing prepareToPut when retrying.

Target: master
Patch: http://rb.dcache.org/r/6956/
Acked-by: Gerd Behrmann
Request: 2.9
SRM: add support for 3rd-party HTTP copy

This patch adds support for pulling or pushing data to a remote server via HTTP.