Squashed commit of the following:

commit b6faa1aa898dc62854696d6719ca61a8cea8d6f9
Author: Dmitry Litvintsev <litvinse@fnal.gov>
Date:   Tue Jul 23 13:26:40 2013 -0500

    srm: fixed a left over from srm in transfermanagers.batch

    what happened

    Ticket: #number
    Acked-by:
    Target: trunk
    Require-book: no
    Require-notes: no

commit 9b6bfd5898543adde2503eb85f83e38d4c3eec38
Author: Dmitry Litvintsev <litvinse@fnal.gov>
Date:   Tue Jul 23 13:11:54 2013 -0500

    srm: adapt to new naming convention

    what happened

    Ticket: #number
    Acked-by:
    Target: trunk
    Require-book: no
    Require-notes: no

commit 9b82f2e5eb26dc2051a6ab9312cd13a2e5ecf140
Author: Dmitry Litvintsev <litvinse@fnal.gov>
Date:   Fri Jul 19 19:46:47 2013 -0500

    srm: new property policy

    what happened

    Ticket: #number
    Acked-by:
    Target: trunk
    Require-book: no
    Require-notes: no

commit 65d6daf918500a0504c46799456ef1fbef7a3333
Author: Dmitry Litvintsev <litvinse@fnal.gov>
Date:   Thu Jul 18 22:31:39 2013 -0500

    srm: new propery policy

    what happened

    Ticket: #number
    Acked-by:
    Target: trunk
    Require-book: no
    Require-notes: no
SRM: tidy up how JDC is used and how it interacts with NDC

The SRM has a class 'JDC' that has roughly the same functionality as
the cells CDC class, namely to be a container for all diagnostic
information.  Both CDC and JDC make use of the NDC class, which
provides a stack of diagnostic information in which items may be
pushed, popped, reset and the state saved and restored later on.  Also,
both use the same key-value storage: MDC.  (In fact, correct flow of
diagnostic information from SRM to dCache relies on both using MDC with
the same key, which is not clear from casual inspection.)

When a request start from within SRM it is assigned a session-ID, as
with other doors.  Like other doors, the SRM pushes the session-ID into
the NDC, so logged messages contain this useful information.  Note
that, although the stored session-ID and the string pushed into the NDC
have the same value, they are stored independently.

Unlike other doors, the SRM may alter the session-ID while processing a
request.  For example, an SRM request that may involve multiple files
will have different session-IDs when working on different files within
the same request.  The file-specific session-ID is derived from the
SRM-request session-ID by appending a counter.

Updating the stored session-ID is supported; however, NDC does not
support altering an already-stored value.

To work around this, the JDC pops the existing value and pushes a new
value.  This is fragile as it only works if no other diagnostic
information is pushed after the session-ID.

This patch updates JDC so it allows popping an arbitrary number of
items off the NDC, updating the session-ID and pushed the previously
popped items back onto the NDC.

The patch also updates JDC to make use of the try-with-resource
pattern.  Some methods that alter the diagnostic context return a JDC
object that, when closed, revert these (and any subsequent) changes.
This is similar to how CDC was updated to follow this pattern.

The SRM code is updated to make use of this new pattern; it also fixes
some places where SRMv1 code failed to set the session.

The Axis thread is updated so that the session-ID is present in logged
messages.  This is particularly useful for the srmGetStatus-family of
requests.

Finally, the patch updates the format of the session-ID.  The current
format is unnecessarily verbose.  The new format is <epoc>:<counter>,
where <epoc> is a one to three Base64 characters (i.e., from
[a-zA-Z0-9+/]) and <counter> is the number of SOAP requests since
start-up.  The <epoc> is chosen so it is very (very) likely different
between successive SRM instances.  The result is the session-ID is
noticeable shorter (initially 3--5 characters compared to 8), making
logged lines easier to read.

Target: master
Requires-notes: no
Requires-book: no
gplazma2-htpasswd: Add htpasswd support

This plugin adds support for the htpasswd file - a file format
used by many web servers.

The plugin only supports MD5 hashes. SHA is not supported
at the moment.

The htpasswd can be managed with the htpasswd utility available on
most OSes.

system-test is updated to use htpasswd + authzdb for the admin
account if the htpasswd utility is available.

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: http://rb.dcache.org/r/5759/
pool: Add callout to health check script

dCache pools periodically perform simple I/O tests to verify
that a pool is still working. We can however not check the
general health status of the host system, eg. whether a disk
in the array has failed.

This patch adds the ability to call an external health check
program. If the program exits with a non-zero exit code, the
pool health is considered degraded and will cause the pool to
be marked read-only or disabled.

The command is configurable through the pool.check-health-command
property.

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5775/
pool,poolmanager: Factor WASS code out of partition and selection strategy code

Addresses code duplication between pool manager and migration module
code.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5788/
doors: Fix parsing of cipher flags

Addresses the problem that the empty string was not accepted for
*.security.ciphers configuraiton properties.

Target: trunk
Request: 2.6
Request: 2.2-sha2
Require-book: no
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5795/
chimera-cli: fix usage help for writetag

Acked-by: Paul Millar
Target: trunk
Require-book: no
Require-notes: no
srm,transfermanagers,pnfsmanager: Fix problems introduced in 29685a5

A recent patch to rename all SRM properties broke pnfsmanager by
removing the aclEnabled property.

The patch also broke hsqldb support by changing db properties even though
db.* are still the authorative properties as long as our scripts have
not been updated.

This patch all updates the system-test configuration to use the new
properties, and moves transfermanagers properties to their own
properties file.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5798/
httpd: Comply with new configuration policy; convert HttpServiceCell to UniversalSpringCell

Besides the properties conversion, this patch also converts the HttpServiceCell to use UniversalSpringCell.  This was done in order to avoid unnecessary rewriting of properties in the batch file which would then simply be discarded with the final changes.

Note that in order to provide to one of the httpd aliases the domain context, a new interface was added (DomainContextAware) and this interface is checked during the bean post-processing, similarly to EnvironmentAware.

Target: master
Patch: http://rb.dcache.org/r/5760/
Require-book: yes
Require-notes: yes
Acked-by: Gerd

Testing done:

Domain was booted and logs were checked; web-page functionality was verified.

RELEASE NOTES:
BOOK:

poolqplots.properties has been consolidated with httpd.properties and the separate file eliminated

NEW:

httpd.cell.name

httpd.enable.spacemanager=${dcache.enable.spacemanager}
httpd.enable.authn=${authenticated}
httpd.net.port=${httpdPort}
httpd.net.https-port=${httpsPort}
httpd.authz.admin-gid=${webadminAdminGid}
httpd.authn.keystore=${keyStore}
httpd.authn.keystore.password=${keyStorePassword}
httpd.authn.truststore=${trustStore}
httpd.authn.truststore.password=${trustStorePassword}

httpd.limits.idle-time=${httpdMaxIdleTimeInMs}
(one-of?MILLISECONDS|SECONDS|MINUTES|HOURS|DAYS)httpd.limits.idle-time.unit=MILLISECONDS
httpd.limits.threads=${httpdMaxThreads}

httpd.service.jetty.default-webapp=${httpdDefaultWebapp}
httpd.service.jetty.webapps.dir=${webadminWebappsPath}
httpd.service.jetty.webapps.tmp-dir=${webadminWarunpackdir}
httpd.service.html.dcache-instance-name=${webadminDCacheInstanceName}

httpd.service.billing=${dcache.service.billing}
httpd.service.loginbroker=${dcache.service.loginbroker}
httpd.service.gplazma=${dcache.service.gplazma}
httpd.service.pnfsmanager=${dcache.service.pnfsmanager}
httpd.service.poolmanager=${dcache.service.poolmanager}

httpd.service.pool.timeout=4000
(one-of?MILLISECONDS|SECONDS|MINUTES)httpd.service.pool.timeout.unit=MILLISECONDS
httpd.service.billing.timeout=60000
(one-of?MILLISECONDS|SECONDS|MINUTES)httpd.service.billing.timeout.unit=MILLISECONDS
httpd.service.poolmonitor-collector.timeout=4000
(one-of?MILLISECONDS|SECONDS|MINUTES)httpd.service.poolmonitor-collector.timeout.unit=MILLISECONDS
httpd.service.poolmonitor-collector.period=4000
(one-of?MILLISECONDS|SECONDS|MINUTES)httpd.service.poolmonitor-collector.period.unit=MILLISECONDS
httpd.service.activetransfers-collector.timeout=5000
(one-of?MILLISECONDS|SECONDS|MINUTES)httpd.service.activetransfers-collector.timeout.unit=MILLISECONDS
httpd.service.activetransfers-collector.period=60000
(one-of?MILLISECONDS|SECONDS|MINUTES)httpd.service.activetransfers-collector.period.unit=MILLISECONDS
httpd.service.cellstatus-collector.timeout=5000
(one-of?MILLISECONDS|SECONDS|MINUTES)httpd.service.cellstatus-collector.timeout.unit=MILLISECONDS
httpd.service.cellstatus-collector.period=10000
(one-of?MILLISECONDS|SECONDS|MINUTES)httpd.service.cellstatus-collector.period.unit=MILLISECONDS
httpd.service.spacetoken-collector.timeout=5000
(one-of?MILLISECONDS|SECONDS|MINUTES)httpd.service.spacetoken-collector.timeout.unit=MILLISECONDS
httpd.service.spacetoken-collector.period=10000
(one-of?MILLISECONDS|SECONDS|MINUTES)httpd.service.spacetoken-collector.period.unit=MILLISECONDS
httpd.service.restorehandler-collector.timeout=5000
(one-of?MILLISECONDS|SECONDS|MINUTES)httpd.service.restorehandler-collector.timeout.unit=MILLISECONDS
httpd.service.restorehandler-collector.period=10000
(one-of?MILLISECONDS|SECONDS|MINUTES)httpd.service.restorehandler-collector.period.unit=MILLISECONDS
httpd.enable.alarm-cleaner=${webadmin.alarm.cleaner.enabled}
httpd.service.alarm-cleaner.timeout=${webadmin.alarm.cleaner.timeout}
(one-of?MILLISECONDS|SECONDS|MINUTES|HOURS|DAYS)httpd.alarm-cleaner.timeout.unit=DAYS
httpd.service.alarm-cleaner.delete-entries-before=${webadmin.alarm.cleaner.delete-threshold}
(one-of?MILLISECONDS|SECONDS|MINUTES|HOURS|DAYS)httpd.service.alarm-cleaner.delete-entries-before.unit=DAYS

httpd.service.alarms.db.config.path=${webadmin.alarm.dao.properties}
httpd.service.alarms.db.xml.path=${alarms.db.xml.path}
httpd.service.alarms.db.driver=${alarms.db.driver}
httpd.service.alarms.db.url=${alarms.db.url}
httpd.service.alarms.db.user=${alarms.db.user}
httpd.service.alarms.db.password=${alarms.db.password}
httpd.service.alarms.definitions.path=${alarms.definitions.path}
(one-of?true|false|${generatePlots})httpd.enable.billing-plots=${generatePlots}

httpd.service.billing.plot.type=${billingPlotsExportType}
httpd.service.billing.plot.refresh=${webadmin.billing-plot.refresh-threshold}
(one-of?MILLISECONDS|SECONDS|MINUTES|HOURS|DAYS)httpd.service.billing.plot.refresh.unit=MINUTES
httpd.service.billing.plot.height=${webadmin.billing-plot.height}
httpd.service.billing.plot.width=${webadmin.billing-plot.width}
(one-of?CONNECTED|OUTLINE|FILLED|${webadmin.billing-plot.default-style})httpd.service.billing.plot.style=${webadmin.billing-plot.default-style}
(one-of?lin|log|${webadmin.billing-plot.default-scale})httpd.service.billing.plot.scale=${webadmin.billing-plot.default-scale}

(one-of?true|false|${poolqplots.enabled})httpd.enable.pool-queue-plots=${poolqplots.enabled}
httpd.service.pool-queue.plot.refresh=${poolqplots.refresh-interval}
(one-of?MILLISECONDS|SECONDS|MINUTES|HOURS|DAYS|${poolqplots.refresh-interval-unit})httpd.service.pool-queue.plot.refresh.unit=${poolqplots.refresh-interval-unit}
httpd.service.pool-queue.db.version=${poolqplots.rrdb.db.version}
httpd.service.pool-queue.db.heartbeat-factor=${poolqplots.rrdb.plot.heartbeat-factor}
httpd.service.pool-queue.plot.type=${poolqplots.rrdb.image-type}
httpd.service.pool-queue.plot.width=${poolqplots.rrdb.plot.width}
httpd.service.pool-queue.plot.x-span=${poolqplots.rrdb.plot.span-size}
(one-of?MINUTES|HOURS|DAYS|${poolqplots.rrdb.plot.span-unit})httpd.service.pool-queue.plot.x-span.unit=${poolqplots.rrdb.plot.span-unit}
httpd.service.pool-queue.plot.min-time-step=${poolqplots.rrdb.plot.step-size}
(one-of?MINUTES|HOURS|DAYS|${poolqplots.rrdb.plot.step-unit})httpd.service.pool-queue.plot.min-time-step.unit=${poolqplots.rrdb.plot.step-unit}
(one-of?MINUTES|HOURS|DAYS|${poolqplots.rrdb.plot.major-unit})httpd.service.pool-queue.plot.major.unit=${poolqplots.rrdb.plot.major-unit}
httpd.service.pool-queue.plot.grid.spacing.major=${poolqplots.rrdb.plot.major-unit-count}
(one-of?MINUTES|HOURS|DAYS|${poolqplots.rrdb.plot.minor-unit})httpd.service.pool-queue.plot.minor.unit=${poolqplots.rrdb.plot.minor-unit}
httpd.service.pool-queue.plot.grid.spacing.minor=${poolqplots.rrdb.plot.minor-unit-count}
httpd.service.pool-queue.plot.right-margin=${poolqplots.rrdb.plot.right-margin}
httpd.service.pool-queue.plot.label-span=${poolqplots.rrdb.plot.label-span}
(one-of?MINUTES|HOURS|DAYS|${poolqplots.rrdb.plot.label-unit})httpd.service.pool-queue.plot.label.unit=${poolqplots.rrdb.plot.label-unit}
httpd.service.pool-queue.plot.x-label.spacing=${poolqplots.rrdb.plot.label-unit-count}
httpd.service.pool-queue.plot.x-label.format=${poolqplots.rrdb.plot.x-label-format}
httpd.service.pool-queue.plot.y-label=${poolqplots.rrdb.plot.y-label}

DEPRECATED:

httpdPort=2288
authenticated=false
httpsPort=8444
webadminAdminGid=0
httpdMaxIdleTimeInMs=30000
httpdMaxThreads=100
httpdDefaultWebapp=/org/dcache/admin/webadmin/jettycell/resources/webdefault.xml
webadminWebappsPath=${dcache.paths.classes}/webapps
webadminWarunpackdir=${dcache.paths.unpack}
webadminDCacheInstanceName=${host.name}
webadmin.alarm.cleaner.enabled=false
webadmin.alarm.cleaner.timeout=7
webadmin.alarm.cleaner.delete-threshold=14
webadmin.alarm.dao.properties=${alarms.db.config.path}
generatePlots=false
billingPlotsExportType=png
webadmin.billing-plot.refresh-threshold=5
webadmin.billing-plot.height=800
webadmin.billing-plot.width=1200
webadmin.billing-plot.default-style=CONNECTED
webadmin.billing-plot.default-scale=log
poolqplots.enabled=false
poolqplots.refresh-interval=5
poolqplots.refresh-interval-unit=MINUTES
poolqplots.rrdb.db.version=2
poolqplots.rrdb.plot.heartbeat-factor=1.5
poolqplots.rrdb.image-type=png
poolqplots.rrdb.plot.height=800
poolqplots.rrdb.plot.width=1200
poolqplots.rrdb.plot.span-size=15
poolqplots.rrdb.plot.span-unit=DAYS
poolqplots.rrdb.plot.step-size=5
poolqplots.rrdb.plot.step-unit=MINUTES
poolqplots.rrdb.plot.major-unit=DAYS
poolqplots.rrdb.plot.major-unit-count=1
poolqplots.rrdb.plot.minor-unit=DAYS
poolqplots.rrdb.plot.minor-unit-count=1
poolqplots.rrdb.plot.right-margin=3
poolqplots.rrdb.plot.label-span=24
poolqplots.rrdb.plot.label-unit=DAYS
poolqplots.rrdb.plot.label-unit-count=2
poolqplots.rrdb.plot.x-label-format=E MMM dd YYYY HH:mm
poolqplots.rrdb.plot.y-label=Threads

OBSOLETE/FORBIDDEN:

(obsolete)httpdEnablePoolCollector=PoolCollector is now always enabled
(obsolete)httpd.static-content.plots.subdir=No longer used
(obsolete)collectorTimeout=use specific service timeout
(obsolete)transfersCollectorUpdate=no longer used

(forbidden)images=use httpd.static-content.images instead
(forbidden)styles=use httpd.static-content.styles instead
(forbidden)webadminAuthenticated = use httpd.enable.authn instead
(forbidden)httpd/port = use httpd.net.port instead
(forbidden)webadminHttpPort = use httpd.net.port instead
(forbidden)webadminHttpsPort = use httpd.net.https-port instead
httpd: update Wicket libraries to version 6.9.1

Move to the latest version of Wicket.

There were a number of API changes which required minor tweaks throughout.  Also, something seems to have changed in the interaction between Java and the HTML markup to require certain specific css settings which were not previously required.

Testing done:

Reran all unit tests. Deployed instance and checked functioning and presentation webadmin interface and pages.

Target: master
Patch: http://rb.dcache.org/r/5781/
Require-book: no
Require-notes: no
Acked-by: Gerd
httpd (webadmin): Fix rerendering of cellInfo page

In Wicket, the BasePage instances is only created once (or at least
not on every page load). The basic structure of a page is cached,
independent of what caching settings have otherwise been defined
for a page.

The CellServices class, which is the implementation of the cellInfo
page, not only builds the basic page structure in the constructor,
it also builds the data model that is driving the rendering. The
CellStatus data is taken out of a shared map object. The shared map
is periodically updated by background threads, however the data is
only ever taken out of the map in the constructor of CellServices.
Wicket, when rendering the page on page reload, will use its model
abstraction to access the model of the CellServices page, however
since it is never updated, the rendering never changes - after the
first load of the cellInfo page, it stays the same (at least for
that browser session).

This patch fixes the problem by replacing the field with a getter.
The getter fetches the data from the shared cache, thus rendering
a different result on every page load.

Target: trunk
Request: 2.6
Request: 2.2 (?)
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5807/
admin: Add pcells subsystem to ssh2

The pcells subsystem allows the pcells GUI to establish a binary
channel to dCache.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5792/
core: don't use reflection to invoke getPnfsId

Many messages  in dCache do have getPnfsId method.
This information is stored in diagnostic context. Unfortunately,
to invoke getPnfsId() reflection is used as there is no common
way to get this information.

This change modifies all messages which have getPnfsId() and
override getDiagnosticContext() to provide the same information,
but without reflection.

Acked-by: http://rb.dcache.org/r/5640/
Acked-by: Gerd Behrmann
Target: trunk
Require-book: no
Require-notes: no