pool: Add missing attributes in replicateOnArrival request

Addresses an issue in the pool's replicateOnArrival feature. If enabled it
failed with an error stating that required attributes where missing.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5715/
gsi-pam: Remove service

The gsi-pam service appears to be a predecessor of gPlazma. The service
can be used by LoginManager to authenticate users. Since all our doors
support gPlazma, gsi-pam should be dropped as a service - we didn't
support the options to configure LoginManager to use gsi-pam anyway. Also,
it appears that LoginManager only uses the authenticator when configured
with an SSL tunnel, which non of our services use.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5721/
libs: Update to latest versions

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5724/
libs: Upgrade datanucleus

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5726/
info: switch to using UniversalSpringCell

This is a (long overdue) patch to update the info cell to use
UniversalSpringCell.  Most of the wiring is now handled in a spring XML
file.

The patch tries to strike a balance between being complete and trying
to do too much, so that the patch is review-able.  For example,
messages that request serialised data (from web-server) are handled via
the CellMessageReceiver interface but the DataGatheringActivities send
requests for data and processing the responses through a custom
framework.

Target: trunk
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5689/
Acked-by: Gerd Behrmann
Acked-by: Albert Rossi
pool: Comply with new configuration policy

Besides changing the property names, the patch alters the replicate
feature: The arguments to 'set replication' have changed. Destination
mode is no longer accepted as it was never implemented. The replication
notification is now generated no matter whether a file arrived by
restore or from a client. In previous versions the replication was not
triggered when restoring from tape. This however meant that the
information in our book regarding replication on restore would be wrong.

New properties

pool.name
pool.path
pool.queues
pool.lfs
pool.size
pool.tags
pool.enable.repository-check
pool.enable.remove-precious-files-on-delete
pool.cell.name
pool.cell.limits.message.threads.min
pool.cell.limits.message.threads.max
pool.cell.limits.message.threads.max-idle-time
pool.cell.limits.message.threads.max-idle-time.unit
pool.cell.limits.message.queue.max
pool.cell.limits.callback.threads.min
pool.cell.limits.callback.threads.max
pool.cell.limits.callback.threads.max-idle-time
pool.cell.limits.callback.threads.max-idle-time.unit
pool.cell.limits.callback.queue.max
pool.cell.wait-for-files
pool.plugins.meta
pool.plugins.sweeper
pool.mover.nfs.rpcsec_gss
pool.mover.dcap.port
pool.mover.ftp.mmap
pool.mover.ftp.read-ahead
pool.mover.ftp.allow-incoming-connections
pool.mover.xrootd.disk-threads
pool.mover.xrootd.socket-threads
pool.mover.xrootd.memory-per-connection
pool.mover.xrootd.memory
pool.mover.xrootd.frame-size
pool.mover.xrootd.plugins
pool.mover.xrootd.timeout.idle
pool.mover.xrootd.timeout.idle.unit
pool.mover.http.disk-threads
pool.mover.http.socket-threads
pool.mover.http.memory-per-connection
pool.mover.http.memory
pool.mover.http.chunk-size
pool.mover.http.timeout.idle
pool.mover.http.timeout.idle.unit
pool.mover.http.timeout.connect
pool.mover.http.timeout.connect.unit
pool.service.pnfsmanager
pool.service.pnfsmanager.timeout
pool.service.pnfsmanager.timeout.unit
pool.service.billing
pool.service.pool.timeout
pool.service.pool.timeout.unit
pool.service.poolmanager
pool.service.poolmanager.timeout
pool.service.poolmanager.timeout.unit
pool.service.pinmanager
pool.service.pinmanager.timeout
pool.service.pinmanager.timeout.unit
pool.destination.flush-notification
pool.destination.heartbeat
pool.destination.replicate
pool.destination.replicate.ip

Deprecated

name
path
poolIoQueue
lfs
maxDiskSpace
tags
checkRepository
replicateOnArrival
allowCleaningPreciousFiles
waitForFiles
metaDataRepository
sweeper
pool.dcap.port
gsiftpAllowMmap
gsiftpReadAhead
xrootdMoverDiskThreads
xrootdMoverSocketThreads
xrootdMoverMaxMemoryPerConnection
xrootdMoverMaxMemory
xrootdMoverMaxFrameSize
xrootdPlugins
httpMoverDiskThreads
httpMoverSocketThreads
httpMoverConnectionMaxMemory
httpMoverMaxMemory
httpMoverChunkSize
httpMoverClientIdleTimeout
httpMoverConnectTimeout
flushMessageTarget
poolupDestination

Removed

waitForRepositoryReady
removeUnexistingEntriesOnFlush
metaDataRepositoryImport
httpMoverMaxChunkSize

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5720/
gplazma: Comply with new configuration policy

The option to implicitly embed gPlazma as a 'module' has been
removed. Instead one can now add a local gplazma service to a
domain and choose not to export it by setting gplazma.cell.export
to false. This has the same effect.

The ability for plain ftp to not use gPlazma at all and use a
local kpwd file has been removed. The same effect can be achieved
by configuring a local gplazma service and let the ftp door use it.

New properties

gplazma.cell.name
gplazma.cell.limits.threads
gplazma.cell.export
gplazma.service.pnfsmanager

Obsolete properties

kpwdFile
useGPlazmaAuthorizationModule
useGPlazmaAuthorizationCell

Removed properties

gplazma.version

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5722/
broadcast: Comply with new configuration policy

Also updates space manager and replica manager enable switches. Note
that the allows values for the new *.enable.spacemanager and
*.enable.replica properties is limited to true and false.

New properties

broadcast.cell.name
broadcast.service.billing
broadcast.service.pinmanager
broadcast.service.poolmanager
broadcast.service.spacemanager
broadcast.service.replica
broadcast.enable.replica
broadcast.enable.spacemanager
dcache.enable.replica
dcache.enable.spacemanager
dcap.enable.spacemanager
ftp.enable.spacemanager
webdav.enable.spacemanager

Deprecated

replicaManager
srmSpaceManagerEnabled

Removed

dcap.service.spacemanager.enabled
ftp.service.spacemanager.enabled
webdav.service.spacemanager.enabled

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5727/
Update SpreadAndWait to use CellStub

This will make it easier to inject timeout configuration into
SpreadAndWait.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5728/
topo,cns: Comply with new configuration policy

Updates all broker related properties to comply with the
new configuration policy.

New properties

cns.cell.name
cns.broker.scheme
cns.broker.amq.url
cns.broker.openmq.host
cns.broker.openmq.port
cns.broker.openmq.reconnect
cns.broker.openmq.reconnect
dcache.broker.scheme
dcache.broker.domain
dcache.broker.host
dcache.broker.port
dcache.broker.messaging.port
dcache.broker.client.port
dcache.broker.cells.config
dcache.broker.amq.host
dcache.broker.amq.port
dcache.broker.amq.ssl.port
dcache.broker.amq.url
dcache.broker.openmq.host
dcache.broker.openmq.port
dcache.broker.openmq.reconnect
dcache.broker.openmq.reconnect
topo.cell.name
topo.limits.refresh
topo.limits.refresh
topo.service.system.timeout
topo.service.system.timeout.unit

Deprecated

broker.scheme
broker.domain
broker.host
broker.port
broker.messaging.port
broker.client.port
broker.cells.config
broker.amq.host
broker.amq.port
broker.amq.ssl.port
broker.amq.url
broker.openmq.host
broker.openmq.port
broker.openmq.reconnect.interval
topoWaitTime
topoTimeout

Removed

useFilesystem
srmVacuum
srmVacuumPeriod
java
java_options
bufferSize
tcpBufferSize
maintenanceLibPath
maintananceLibAutogeneraetPaths
maintenanceLogoutTime
srmVersion
permissionHandler
PermissionHandlerDataSource
user
pidDir
logArea
logMode
classpath
librarypath
kerberosRealm
kerberosKdcList
authLoginConfig
messageBroker
serviceLocatorHost
serviceLocatorPort
amqHost
amqPort
amqSSLPort
amqUrl
ourHomeDir
portBase

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5729/
hoppingmanager: Comply with new configuration policy

The hopping service was renamed to hoppingmanager.

New properties

hoppingmanager.cell.name
hoppingmanager.cell.export
hoppingmanager.config

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5731/
hsqldb: Update to version 2.3.0

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5734/
info: fix broken import

A previous patch modified the package of CellInfoProvider.
Unfortunately the patch that moved info to use UniversalSpringCell was
not updated accordingly so, when committed, it broke master.

This patch fixes that problem.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5737/
Acked-by: Gerd Behrmann
(skel): time-unit corrections to billing migration script

The scripts contained in skel/share/migration which are intended to aid in migrating preexisting aggregated billing data or in aggregating the data from the main fine-grained tables have two errors in them.

1.  migrate_from_preexistent.sql does not account for the time-unit differences between the original dc_tm_daily tables, which store the data in seconds, and the new tables, which preserve the millisecond unit of the base table.

2.  migrate_from_messageinfo.sql neglected to exclude data from failed operations (errorcode = 0).

This patch repairs these two issues.

Testing done:

Before the patches, stored and displayed data did not correspond with previous installations, nor did the two scripts produce similar results.  After the patch, the results of both migrations are consonant with prior data.

Target: master
Request: 2.6
Request: 2.2
Patch: http://rb.dcache.org/r/5732/
Require-book: no
Require-notes: yes
Acked-by: Gerd

RELEASE NOTES:

Fixes two bugs in the migration scripts.  One which neglected to convert from seconds to milliseconds (migrate_from_preexistent.sql) for connection time values, the other which neglected to exclude data from failed operations (error code = 0: migrate_from_messageinfo.sql).

Note for those who already have applied the faulty script: the migration from the preexistent tables does not really affect anyone outside of Fermi.  The other migration error generates differences which are probably negligible.