pool: Comply with new configuration policy

Besides changing the property names, the patch alters the replicate
feature: The arguments to 'set replication' have changed. Destination
mode is no longer accepted as it was never implemented. The replication
notification is now generated no matter whether a file arrived by
restore or from a client. In previous versions the replication was not
triggered when restoring from tape. This however meant that the
information in our book regarding replication on restore would be wrong.

New properties

pool.name
pool.path
pool.queues
pool.lfs
pool.size
pool.tags
pool.enable.repository-check
pool.enable.remove-precious-files-on-delete
pool.cell.name
pool.cell.limits.message.threads.min
pool.cell.limits.message.threads.max
pool.cell.limits.message.threads.max-idle-time
pool.cell.limits.message.threads.max-idle-time.unit
pool.cell.limits.message.queue.max
pool.cell.limits.callback.threads.min
pool.cell.limits.callback.threads.max
pool.cell.limits.callback.threads.max-idle-time
pool.cell.limits.callback.threads.max-idle-time.unit
pool.cell.limits.callback.queue.max
pool.cell.wait-for-files
pool.plugins.meta
pool.plugins.sweeper
pool.mover.nfs.rpcsec_gss
pool.mover.dcap.port
pool.mover.ftp.mmap
pool.mover.ftp.read-ahead
pool.mover.ftp.allow-incoming-connections
pool.mover.xrootd.disk-threads
pool.mover.xrootd.socket-threads
pool.mover.xrootd.memory-per-connection
pool.mover.xrootd.memory
pool.mover.xrootd.frame-size
pool.mover.xrootd.plugins
pool.mover.xrootd.timeout.idle
pool.mover.xrootd.timeout.idle.unit
pool.mover.http.disk-threads
pool.mover.http.socket-threads
pool.mover.http.memory-per-connection
pool.mover.http.memory
pool.mover.http.chunk-size
pool.mover.http.timeout.idle
pool.mover.http.timeout.idle.unit
pool.mover.http.timeout.connect
pool.mover.http.timeout.connect.unit
pool.service.pnfsmanager
pool.service.pnfsmanager.timeout
pool.service.pnfsmanager.timeout.unit
pool.service.billing
pool.service.pool.timeout
pool.service.pool.timeout.unit
pool.service.poolmanager
pool.service.poolmanager.timeout
pool.service.poolmanager.timeout.unit
pool.service.pinmanager
pool.service.pinmanager.timeout
pool.service.pinmanager.timeout.unit
pool.destination.flush-notification
pool.destination.heartbeat
pool.destination.replicate
pool.destination.replicate.ip

Deprecated

name
path
poolIoQueue
lfs
maxDiskSpace
tags
checkRepository
replicateOnArrival
allowCleaningPreciousFiles
waitForFiles
metaDataRepository
sweeper
pool.dcap.port
gsiftpAllowMmap
gsiftpReadAhead
xrootdMoverDiskThreads
xrootdMoverSocketThreads
xrootdMoverMaxMemoryPerConnection
xrootdMoverMaxMemory
xrootdMoverMaxFrameSize
xrootdPlugins
httpMoverDiskThreads
httpMoverSocketThreads
httpMoverConnectionMaxMemory
httpMoverMaxMemory
httpMoverChunkSize
httpMoverClientIdleTimeout
httpMoverConnectTimeout
flushMessageTarget
poolupDestination

Removed

waitForRepositoryReady
removeUnexistingEntriesOnFlush
metaDataRepositoryImport
httpMoverMaxChunkSize

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5720/
gplazma: Comply with new configuration policy

The option to implicitly embed gPlazma as a 'module' has been
removed. Instead one can now add a local gplazma service to a
domain and choose not to export it by setting gplazma.cell.export
to false. This has the same effect.

The ability for plain ftp to not use gPlazma at all and use a
local kpwd file has been removed. The same effect can be achieved
by configuring a local gplazma service and let the ftp door use it.

New properties

gplazma.cell.name
gplazma.cell.limits.threads
gplazma.cell.export
gplazma.service.pnfsmanager

Obsolete properties

kpwdFile
useGPlazmaAuthorizationModule
useGPlazmaAuthorizationCell

Removed properties

gplazma.version

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5722/
broadcast: Comply with new configuration policy

Also updates space manager and replica manager enable switches. Note
that the allows values for the new *.enable.spacemanager and
*.enable.replica properties is limited to true and false.

New properties

broadcast.cell.name
broadcast.service.billing
broadcast.service.pinmanager
broadcast.service.poolmanager
broadcast.service.spacemanager
broadcast.service.replica
broadcast.enable.replica
broadcast.enable.spacemanager
dcache.enable.replica
dcache.enable.spacemanager
dcap.enable.spacemanager
ftp.enable.spacemanager
webdav.enable.spacemanager

Deprecated

replicaManager
srmSpaceManagerEnabled

Removed

dcap.service.spacemanager.enabled
ftp.service.spacemanager.enabled
webdav.service.spacemanager.enabled

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5727/
Update SpreadAndWait to use CellStub

This will make it easier to inject timeout configuration into
SpreadAndWait.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5728/
topo,cns: Comply with new configuration policy

Updates all broker related properties to comply with the
new configuration policy.

New properties

cns.cell.name
cns.broker.scheme
cns.broker.amq.url
cns.broker.openmq.host
cns.broker.openmq.port
cns.broker.openmq.reconnect
cns.broker.openmq.reconnect
dcache.broker.scheme
dcache.broker.domain
dcache.broker.host
dcache.broker.port
dcache.broker.messaging.port
dcache.broker.client.port
dcache.broker.cells.config
dcache.broker.amq.host
dcache.broker.amq.port
dcache.broker.amq.ssl.port
dcache.broker.amq.url
dcache.broker.openmq.host
dcache.broker.openmq.port
dcache.broker.openmq.reconnect
dcache.broker.openmq.reconnect
topo.cell.name
topo.limits.refresh
topo.limits.refresh
topo.service.system.timeout
topo.service.system.timeout.unit

Deprecated

broker.scheme
broker.domain
broker.host
broker.port
broker.messaging.port
broker.client.port
broker.cells.config
broker.amq.host
broker.amq.port
broker.amq.ssl.port
broker.amq.url
broker.openmq.host
broker.openmq.port
broker.openmq.reconnect.interval
topoWaitTime
topoTimeout

Removed

useFilesystem
srmVacuum
srmVacuumPeriod
java
java_options
bufferSize
tcpBufferSize
maintenanceLibPath
maintananceLibAutogeneraetPaths
maintenanceLogoutTime
srmVersion
permissionHandler
PermissionHandlerDataSource
user
pidDir
logArea
logMode
classpath
librarypath
kerberosRealm
kerberosKdcList
authLoginConfig
messageBroker
serviceLocatorHost
serviceLocatorPort
amqHost
amqPort
amqSSLPort
amqUrl
ourHomeDir
portBase

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5729/
hoppingmanager: Comply with new configuration policy

The hopping service was renamed to hoppingmanager.

New properties

hoppingmanager.cell.name
hoppingmanager.cell.export
hoppingmanager.config

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5731/
hsqldb: Update to version 2.3.0

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5734/
info: fix broken import

A previous patch modified the package of CellInfoProvider.
Unfortunately the patch that moved info to use UniversalSpringCell was
not updated accordingly so, when committed, it broke master.

This patch fixes that problem.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5737/
Acked-by: Gerd Behrmann
(skel): time-unit corrections to billing migration script

The scripts contained in skel/share/migration which are intended to aid in migrating preexisting aggregated billing data or in aggregating the data from the main fine-grained tables have two errors in them.

1.  migrate_from_preexistent.sql does not account for the time-unit differences between the original dc_tm_daily tables, which store the data in seconds, and the new tables, which preserve the millisecond unit of the base table.

2.  migrate_from_messageinfo.sql neglected to exclude data from failed operations (errorcode = 0).

This patch repairs these two issues.

Testing done:

Before the patches, stored and displayed data did not correspond with previous installations, nor did the two scripts produce similar results.  After the patch, the results of both migrations are consonant with prior data.

Target: master
Request: 2.6
Request: 2.2
Patch: http://rb.dcache.org/r/5732/
Require-book: no
Require-notes: yes
Acked-by: Gerd

RELEASE NOTES:

Fixes two bugs in the migration scripts.  One which neglected to convert from seconds to milliseconds (migrate_from_preexistent.sql) for connection time values, the other which neglected to exclude data from failed operations (error code = 0: migrate_from_messageinfo.sql).

Note for those who already have applied the faulty script: the migration from the preexistent tables does not really affect anyone outside of Fermi.  The other migration error generates differences which are probably negligible.
cleaner: Add hsqldb support

Addresses the issue that the cleaner was not functional when used with
hsqldb, and thus didn't work in system-test. The trigger to populate
the trash table was missing.

This patch adds a trigger. HSQLDB unfortunately does not support data
manipulation statements in BEFORE DELETE triggers and thus cannot
populate the trash on delete from t_inodes (AFTER DELETE is too late
as the delete cascades to t_locationinfo before the trigger is run,
and thus the data is already gone). Instead the patch adds an
AFTER DELETE trigger on the t_locationinfo table itself. A trigger
condition is used to limit the trigger to deletion of an inode and
thus excludes regular deletes of replicas on pool and tapes.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5735/
cleaner: Comply with new configuration policy

Also added a couple of delete tests to the system-test test
script.

New properties

cleaner.cell.name
cleaner.cell.export
cleaner.limits.threads
cleaner.limits.period
cleaner.limits.period.unit
cleaner.service.pool.timeout
cleaner.service.pool.timeout.unit
cleaner.service.pool.retry-period
cleaner.service.pool.retry-period.unit
cleaner.destination.remove-notification
cleaner.limits.batch-size
cleaner.enable.hsm
cleaner.limits.hsm-batch-size
cleaner.service.pool.hsm-timeout
cleaner.service.pool.hsm-timeout.unit
cleaner.db.host
cleaner.db.name
cleaner.db.user
cleaner.db.password
cleaner.db.driver
cleaner.db.url
cleaner.db.schema.auto

Deprecated

cleaner.period
cleaner.pool-reply-timeout
cleaner.pool-retry
cleaner.report-cell
cleaner.max-files-in-message
cleaner.enable.hsm
cleaner.hsm.max-files-in-message
cleaner.hsm.pool-reply-timeout

Obsolete

cleaner.hsm

Removed

cleaner.trash.dir
cleanerArchive
cleanerDB
cleanerPoolTimeout
cleanerProcessFilesPerRun
cleanerRecover
cleanerRefresh
hsmCleaner
hsmCleanerFlush
hsmCleanerRecover
hsmCleanerRepository
hsmCleanerRequest
hsmCleanerScan
hsmCleanerTimeout
hsmCleanerTrash
hsmCleanerQueue
trash
cleaner.hsm.trash.dir
cleaner.hsm.repository.dir
cleaner.book-keeping.dir
cleaner.archive
cleaner.hsm.max-concurrent-requests

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5738/
info: Comply with new configuration policy

There are plenty of hardcoded cell names in the info service, but info
service has to be refactored further to parameterize these. This is not
done in this patch.

New properties

info.cell.name
info.cell.export
info.net.port
info.net.listen
info.net.backlog

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5740/
missing-files: Comply with new configuration policy

New properties

missing-files.cell.name
missing-files.cell.export
missing-files.plugins
missing-files.limits.threads
dcache.service.missing-files
dcache.service.missing-files.timeout
dcache.service.missing-files.timeout.unit
dcache.enable.missing-files

Deprecated

missing-files.plugin.list
missing-files.thread-count
missing-files.name
missing-files.timeout
missing-files.timeout.unit
missing-files.enabled

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5741/