dcache-core: fix statistics totals for top, year and month

http://rb.dcache.org/r/5768 attempted to fix an NPE when the the topmost element of the statistics tree had not yet been generated.

The fix applied there, however, changed the loop invariant inadvertently, causing the summary pages to reflect the same stats for all nodes (based on the oldest node).

This patch restores the original invariant and places a guard against the NPE after the loop, where it does not interfere.

Testing:

redeployed, eliminated current total.raw files and index.html files and regenerated html tree from the admin interface, observing correct totals.

Target: master
Request: 2.7
Request: 2.6
Request: 2.2
Patch: http://rb.dcache.org/r/6335
Require-book: no
Require-notes: yes
Bug: http://rt.dcache.org/Ticket/Display.html?id=8147
Acked-by: Gerd

RELEASE NOTES:

Fixes bug in the statistics page which caused the summary tables for top, year and month to report uniformly the data for the oldest date.
dcache-core (statistics): fix yet another potential NPE in tree creation

If the user runs the admin command create html <yyyy> or create html <yyyy> <mm> and those directories do not exist, an NPE is thrown when the list of subfiles is sorted, since the list ends up being a NULL array.

It costs little to ensure the presence of those directories in each of the three methods, so dir.mkdirs() has been added there.

Testing:

Running commands in absence of the statistics directory tree now avoids the NPE and creates the necessary empty directories.

Target: master
Request: 2.7
Request: 2.6
Request: 2.2
Patch: http://rb.dcache.org/r/6341/
Require-book: no
Require-notes: yes
Acked-by: Tigran

RELEASE NOTES:

Fixes potential Null Pointer Exception thrown when create html <yyyy> or create html <yyyy> <mm> is run and those directories do not yet exist.
dcache-webadmin: wget xml statepath (Info page) fails with HTML 500 error

All pages which extend BasePage require a "title" property.  Info was missing the properties file.

Testing:

Before (example from ticket):

[mol@lxplus0443 ~]$ wget http://cmsdcacheweb-kit.gridka.de:2288/webadmin/info?statepath=pools/f01-065-105-e_1D_cms
--2013-11-08 13:07:41-- http://cmsdcacheweb-kit.gridka.de:2288/webadmin/info?statepath=pools/f01-065-105-e_1D_cms
Resolving cmsdcacheweb-kit.gridka.de (cmsdcacheweb-kit.gridka.de)... 192.108.45.36
Connecting to cmsdcacheweb-kit.gridka.de (cmsdcacheweb-kit.gridka.de)|192.108.45.36|:2288... connected.
HTTP request sent, awaiting response... 500 Server Error
2013-11-08 13:07:41 ERROR 500: Server Error.

After:

info.xml is successfully fetched.

Target: master
Patch: http://rb.dcache.org/r/6342/
Request: 2.7
Request: 2.6
Require-book: no
Require-notes: yes
Bug: http://rt.dcache.org/Ticket/Display.html?id=8097
Acked-by: Tigran

RELEASE NOTES:

Fixes bug (HTML 500 is reported) preventing wget of Info.xml.
dcache-webadmin: put limit on entries in alarms table

In order to avoid OOM.

This is hard-coded as effectively the capacity to hold more than 10000 entries in memory should not be required.  One can always adjust the numbers or refine the query in order to retrieve subsets of a larger set.

Testing: Rechecked interactively.

Target: master
Request: 2.7
Request: 2.6
Patch: http://rb.dcache.org/r/6344/
Bug: http://rt.dcache.org/Ticket/Display.html?id=8084
Require-book: no
Require-notes: no
Acked-by: Karsten
transfermanager: white-space only fix
transfermanager: formatting patch

Adjust formatting to be more consistent with other dCache code.

Target: master
Requires-notes: no
Requires-book: bo
logback shell: add set pattern command

logback supports configuration of the log pattern. To change the
pattern in dcache it is necessary to configure the pattern in the
configuration files and restart.
This patch adds commands to get and set the pattern for the appenders.

Acked-by: Al
Target: trunk
Require-book: yes
Require-notes: yes
nfs: do not create an instance of StripingPattern per pool

as we do not support different strategies, instantiated only once.

Acked-by: Karsten Schwank
Target: master
Require-book: no
Require-notes: no
nfs: remove obsolete comment

as we already sending the complete list of addresses.

Acked-by: Karsten Schwank
Target: master
Require-book: no
Require-notes: no
nfs: introduce PoolDeviceMap for pool to deviceid mapping

Acked-by: Karsten Schwank
Target: master
Require-book: no
Require-notes: no
defaults: add common explanation of annotations

We don't explain what the annotations in mean in the defaults file.  This
has led to confusion with people misunderstanding the annotation and
mistakenly copying the annotation with configuring dCache.

This patch adds a common information about what the annotations mean.  It
also refactors the common explanation so the current duplication is reduced
(but not eliminated).

Target: master
Ticket: http://rt.dcache.org/Ticket/Display.html?id=8166
Patch: http://rb.dcache.org/r/6346/
Acked-by: Karsten Schwank
Request: 2.7
Request: 2.6
Request: 2.2
spacemanager: Split selectPool method

The selectPool method had to nearly unoverlapping flows. This
patch splits the method into two methods, one for the request
flow and one for the reply flow.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6332/
spacemanager: Fix typo in property name

Deprecates spacemanager.enable.reserve-space-for-non-srm-tranfers and
introduces spacemanager.enable.reserve-space-for-non-srm-transfers.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6340/
defaults: fix typo in common header
cells: Further refine LoginManager startup and shutdown

Addresses flaws in server socket recreation in case of failures,
makes the code more DRY, uses thread pool for rejected connections,
kills children early, and waits for them to die before terminating
the login cell factory, removes legacy socket shutdown hack.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6326/
ftp: Use shared thread pool for various processing threads

This patch makes the ftp door use the shared thread pool for
the I/O thread, for the message processing thread, and the
the callback thread.

The patch also updates AbstractCell to use the Cell's message
thread. This saves one additional thread instantiation for
each cell instance.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6331/
spacemanager: Use imported PoolMonitor to select link group

Space manager in some cases has to ask PoolManager to select a
link group. This patch changes the code to use an imported copy
of the PoolMonitor to achieve the same effect. This reduces
load on PoolManager and reduces latency in SpaceManager.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/6351/
spacemanager: Postpone DB update on pool selection

Addresses issues in automatic space reservation for non-srm transfers.

Space manager intercepts both the select pool request and the request pool
reply. In the existing code, space manager selects a link group and creates or
updates a space reservation in the select pool request. If pool selection fails
in pool manager, space manager uses the intercepted reply message to attempt to
rollback what it did when intercepting the request.

The existing code was suboptimal for several reasons. First, the rollback done
when pool selection failed is incomplete, as it only works for srm managed
reservations (see the ticket). Second, we could just as well wait with the DB
update until we get the successfull reply rather than attempting to role back
the changes.

This patch changes the code such that the DB isn't updated until the reply
is received.

Target: trunk
Require-notes: yes
Require-book: no
Ticket: http://rt.dcache.org/Ticket/Display.html?id=8163
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6352/
spacemanager: Reduce latency for some pool selection messages

Some pool selection messages do not require any processing in
space manager and can be forwarded as is. This patch changes
the logic to bypass the additional queueing in those cases.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Acked-by: Albert Rossir <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/6353/
spacemanager: Allow unreserved transfers even when space reservation for non-srm transfers is enabled

Space manager can be configured to create space reservations for non-srm
transfers. The problem is that as soon as this feature is enabled, all
authenticated non-srm transfers that support space manager must go to a link
group. It isn't possible to mix reserved and unreserved transfers.

This patch addresses this problem by letting space manager fall back to an
unreserved transfer. This fallback already exists if the transfer is
unauthenticated or when a protocol without space- manager support is used.

The patch thus changes the semantics of the property
spacemanager.enable.reserve-space-for-non-srm-transfers. Whereas before
enabling this property would cause some transfers that succeeded before to
fail, the property now only adds additional ways in which a transfer can
succeed. This is easily demonstrated using the system-test package.

Without this patch and with reserve-space-for-non-srm-transfers disabled,
uploading to the reserved directory with gridftp would fail (e.g. 'test
--gridftp -d reserved'), while upload to the disk directory would succeed (e.g.
'test --gridftp -d disk'). If reserve-space-for-non-srm-transfers is enabled,
the success status of these two transfers would be reversed (the former would
now succeed, the latter would now fail). With this patch, enabling
reserve-space-for-non-srm-transfers will cause both uploads to succeed. The
patch thus also changes the default system-test setup to enabel space
reservation for non-srm transfers.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6356