defaults: add common explanation of annotations

We don't explain what the annotations in mean in the defaults file.  This
has led to confusion with people misunderstanding the annotation and
mistakenly copying the annotation with configuring dCache.

This patch adds a common information about what the annotations mean.  It
also refactors the common explanation so the current duplication is reduced
(but not eliminated).

Target: master
Ticket: http://rt.dcache.org/Ticket/Display.html?id=8166
Patch: http://rb.dcache.org/r/6346/
Acked-by: Karsten Schwank
Request: 2.7
Request: 2.6
Request: 2.2
spacemanager: Split selectPool method

The selectPool method had to nearly unoverlapping flows. This
patch splits the method into two methods, one for the request
flow and one for the reply flow.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6332/
spacemanager: Fix typo in property name

Deprecates spacemanager.enable.reserve-space-for-non-srm-tranfers and
introduces spacemanager.enable.reserve-space-for-non-srm-transfers.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6340/
defaults: fix typo in common header
cells: Further refine LoginManager startup and shutdown

Addresses flaws in server socket recreation in case of failures,
makes the code more DRY, uses thread pool for rejected connections,
kills children early, and waits for them to die before terminating
the login cell factory, removes legacy socket shutdown hack.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6326/
ftp: Use shared thread pool for various processing threads

This patch makes the ftp door use the shared thread pool for
the I/O thread, for the message processing thread, and the
the callback thread.

The patch also updates AbstractCell to use the Cell's message
thread. This saves one additional thread instantiation for
each cell instance.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6331/
spacemanager: Use imported PoolMonitor to select link group

Space manager in some cases has to ask PoolManager to select a
link group. This patch changes the code to use an imported copy
of the PoolMonitor to achieve the same effect. This reduces
load on PoolManager and reduces latency in SpaceManager.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/6351/
spacemanager: Postpone DB update on pool selection

Addresses issues in automatic space reservation for non-srm transfers.

Space manager intercepts both the select pool request and the request pool
reply. In the existing code, space manager selects a link group and creates or
updates a space reservation in the select pool request. If pool selection fails
in pool manager, space manager uses the intercepted reply message to attempt to
rollback what it did when intercepting the request.

The existing code was suboptimal for several reasons. First, the rollback done
when pool selection failed is incomplete, as it only works for srm managed
reservations (see the ticket). Second, we could just as well wait with the DB
update until we get the successfull reply rather than attempting to role back
the changes.

This patch changes the code such that the DB isn't updated until the reply
is received.

Target: trunk
Require-notes: yes
Require-book: no
Ticket: http://rt.dcache.org/Ticket/Display.html?id=8163
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6352/
spacemanager: Reduce latency for some pool selection messages

Some pool selection messages do not require any processing in
space manager and can be forwarded as is. This patch changes
the logic to bypass the additional queueing in those cases.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Acked-by: Albert Rossir <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/6353/
spacemanager: Allow unreserved transfers even when space reservation for non-srm transfers is enabled

Space manager can be configured to create space reservations for non-srm
transfers. The problem is that as soon as this feature is enabled, all
authenticated non-srm transfers that support space manager must go to a link
group. It isn't possible to mix reserved and unreserved transfers.

This patch addresses this problem by letting space manager fall back to an
unreserved transfer. This fallback already exists if the transfer is
unauthenticated or when a protocol without space- manager support is used.

The patch thus changes the semantics of the property
spacemanager.enable.reserve-space-for-non-srm-transfers. Whereas before
enabling this property would cause some transfers that succeeded before to
fail, the property now only adds additional ways in which a transfer can
succeed. This is easily demonstrated using the system-test package.

Without this patch and with reserve-space-for-non-srm-transfers disabled,
uploading to the reserved directory with gridftp would fail (e.g. 'test
--gridftp -d reserved'), while upload to the disk directory would succeed (e.g.
'test --gridftp -d disk'). If reserve-space-for-non-srm-transfers is enabled,
the success status of these two transfers would be reversed (the former would
now succeed, the latter would now fail). With this patch, enabling
reserve-space-for-non-srm-transfers will cause both uploads to succeed. The
patch thus also changes the default system-test setup to enabel space
reservation for non-srm transfers.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6356
poolmanager: Replace StorageInfo by FileAttributes in PSU

Also avoid the use of the legacy StoageInfo#toString method.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Albert Rossir <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/6357/
gplazma: ldap: make user home and root configurable

The LDAP plugin has hard coded values for the home and root directory as
they are presented to the user. The root directory is set to / and the
home directory is set to the home directory as stored in the LDAP
server. For operating in "cloud" mode this is not what we want, because
we want every user to see his home directory as the root directory.

This patch makes the behaviour configurable by introducing two new
properties gplazma.ldap.home-dir and gplazma.ldap.root-dir that can be
used to configure dcache to rather run in "cloud"-mode than in the usual
way. In those two properties it is now possible to use keywords (e.g.,
"%homeDirectory%" that will be substituted by the corresponding
attribute value in LDAP. For example %homeDirectory% will be replaced by
the value of attribute "homeDirectory" (in the peopleTree).

Acked-by: Paul
Target: trunk
Require-book: yes
Require-notes: yes
pom: scala-reflect depenencies

1) Currently we are using a Milestone release of scala test rather than
the most recent actual release. With this patch this is changed to use
the release 2.0 of scalatest_2.10.

2) scalatest has a hard coded dependency to scala-reflect 2.10.0
The current version of scala-reflect is together with scala-library and
scala compiler version 2.10.3.
This patch adds a dependency entry for scala-reflect 2.10.3 to
dependencyManagement.

Acked-by: Gerd
Target: trunk
Require-book: no
Require-notes: no
nfsv41: simplify internal Transfer class

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
dcache-core (billing): modify changesets to do concurrent index creation

As suggested in support ticket 7947: slow start on create index in billing.

Note that two modifications were necessary:

1.  Because CREATE INDEX CONCURRENTLY is not supported by the Liquibase createIndex task, these were changed to raw <sql> calls.
2.  Because this command cannot run inside a transaction, each of these changeset is marked runInTransaction=false, with cognizance taken of the risks.

The old changesets were eliminated and replaced with new ones having a different id.

Testing:

Deployed new changesets.  Eliminated all indices from billing database and booted billing domain.

Here are the stats for the largest tables (the fine-grained data):

billing=> select count(*) from billinginfo;
  count
---------
 8070858
(1 row)

billing=> select count(*) from storageinfo;
  count
---------
 1862998
(1 row)

billing=> select count(*) from doorinfo;
  count
---------
 1902624
(1 row)

billing=> select count(*) from hitinfo;
 count
-------
 58028
(1 row)

11 Dec 2013 14:02:58 (billing) [] Continuing past: liquibase.precondition.core.PreconditionContainer@5303eb4 despite precondition error:
 ...
11 Dec 2013 14:24:36 (System) [] created : billing

(Total initialization time of about 22 minutes with 8M, 2M and 2M rows in the three main tables).

Target: master
Request: 2.7
Patch: http://rb.dcache.org/r/6343/diff/#index_header
Require-notes: yes
Require-book: no
Acked-by: Tigran
Bug: http://rt.dcache.org/Ticket/Display.html?id=7947

RELEASE NOTES:

Indices on the billing database are now created concurrently, so database (re)initialization should be sped up by a modest linear factor.
nfsv41: remove unused field in NFS4ProtocolInfo

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no