srm: Minimize use of AuthorizationRecord

AuthorizationRecord is a legacy class from before gPlazma 2 existed.
Since the introduction of gPlazma 2, we have tried to use Subject
to represent users everywhere in dCache. SRM is the last component
in which this was not the case. This patch addresses this inconsistency.

AuthorizationRecord is not removed completely as the SRM relies
on storing the user identity in the database together with user
requests. To maintain this functionality and preserve backwards
compatibility, AuthorizationRecord is still used for this purpose.

The SRM itself uses the SRMUser interface to abstract from the
exact representation of a user. This patch introduces the DcacheUser
class, which wraps a Subject to expose it as an SRMUser. The
dCache specific SRM backend is updated to extract the Subject
directly from DcacheUser. DcacheUserPersistenceManager acts as a
bridge to persist and load DcacheUser instances to and from the
database. Internally, DcacheUserPersistenceManager however simply
interfaces with the legacy AuthRecordPersistenceManager. This is
the only place the legacy AuthorizationRecord class is used.

The custom authorization caching of the SRM has been replaced by
a CachingLoginStrategy.

The wiring related to the above classes has been moved into the
Spring XML file.

The only admin visible change is that the output format of the SRM
ls admin commands has changed slightly. It now shows a list of
principals as an identification of the request owner. If output
of these commands is used in the book, the book must be updated.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5691/
system-test: Fixed several bashisms in populate script

system-test failed to compile on systems on which /bin/sh does not
point to bash (eg Debian and Ubuntu).

Also reformatted output to fit in 80-character terminal.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5752/
httpd: Parameterize war unpack directory

Addresses the issue that the webapp unpack directory is /var/tmp in
tar and system-test builds. For these builds the unpack directory
should be within the var directory of the installation directory.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5753/
system-test: build authorized-keys file

The admin interface supports authentication via SSH pub/priv keys,
which are often stored in well-known locations.

This patch takes the user's public keys, if found, and uses them to
build an authorized-keys file for the admin interface.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5754/
Acked-by: Gerd Behrmann
fix servlet classpath clash

The servlet API jar files are signed.  Recent versions of java (or
possibly bug http://bugs.sun.com/view_bug.do?bug_id=6805618 ) means
that, on some platforms, all the classes in a package must be signed
by the same certificate.  A failure results in an error message and
dCache failing to startup.

Currently, dCache includes a Jetty-supplied 'javax.servlet' jar file
(v3.0), and a 'javax.servlet'-supplied 'servlet-api' jar file (v2.5).
These are signed by different certificates and, since the v3.0 file
contains additional classes, if v2.5 precedes v3.0 in the classpath
then the classloader will find classes in the same package with
different signatures.

The previous version of Jetty depended on servlet-api jar file and,
while this too had two jar files (a Jetty-supplied and a
'javax.servlet'-supplied) since the two jar files have the same set of
classes, the classloader will take javax.servlet.* classes from one
jar file, independent of classpath order.

This patch fixes this problem by excluding the references to
servlet-api dependency in axis and axisg jar files.

Target: master
Requires-nodes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5757
Acked-by: Gerd Behrmann
Add DB driver to the list of classes to load

The chimera-cli command was overhauled to make it significantly faster.
One technique used was to limit the classes in the classpath to only
those needed.

Unfortunately, this work excluded the database-driver jar. The current
chimera-cli works for system-test, as the plugin mechanism ensures the
database-driver jar is included; however, the PostgreSQL driver is not
included, resulting in chimera-cli no longer working for machines using
PostgreSQL.  The error message is a stack-trace that starts:

[root@bes-srm dcache]# chimera-cli ls /
Exception in thread "main" java.lang.ClassNotFoundException:
org/postgresql/Driver

[reported in user-forum]

Target: master
Request: 2.6
Requires-notes: yes
Requires-book: no
Patch: http://rb.dcache.org/r/5755/
Acked-by: Gerd Behrmann
info: fix concurrent modification

Gerd reported seeing this:

06 jun 2013 21:44:33 (info) [] Uncaught exception in thread Thread-79
java.util.ConcurrentModificationException: null
        at java.util.HashMap$HashIterator.remove(HashMap.java:912) ~[na:1.7.0_11]
        at org.dcache.services.info.gathers.MessageHandlerChain.flushOldMetadata(MessageHandlerChain.java:230) ~[dcache-core-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.dcache.services.info.gathers.MessageHandlerChain.getMetricTTL(MessageHandlerChain.java:173) ~[dcache-core-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.dcache.services.info.gathers.MessageHandlerChain.getMetricTTL(MessageHandlerChain.java:37) ~[dcache-core-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at org.dcache.services.info.gathers.CellMessageHandlerSkel.answerArrived(CellMessageHandlerSkel.java:150) ~[dcache-core-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$CallbackTask.innerRun(CellNucleus.java:971) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:929) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_11]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_11]
        at dmg.cells.nucleus.CellNucleus$1.run(CellNucleus.java:657) [cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_11]

The problem is triggered by having multiple messenger threads and the
class not being thread-safe.

This patch updates the class to make it thread-safe

Target: trunk
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5756/
Acked-by: Gerd Behrmann
ssh2: Honor admin.colors.enable

Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
Acked-by:
Target: trunk
Require-book: no
Require-notes: no
Remove line that sets colors from environment param and not from within dCache
Change parameter to no type definition and switch line to above cell related properties
ssh2: Honor admin.colors.enable

Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
Acked-by: Gerd
Target: master
Require-book: no
Require-notes: no
Merge color fix branch with current master because not all changes got committed with the last push.

Sorry.

ssh2: Honor admin.colors.enable
Ssh2 admin interface did not honor admin.colors.enable, this patch resolve this problem.

Ticket: http://rt.dcache.org/Ticket/Display.html?id=7820
Acked-by: Gerd
Target: master
Require-book: no
Require-notes: no
loginbroker: Comply with new configuration policy

Drops the service srm-loginbroker. All doors now register with
a single login broker called LoginBroker and SRM and all
monitoring services query this one LoginBroker to locate doors.
SRM is updated to exclude srm as a transfer protocol, thus
making it safe to register all doors in the same login broker.

The dcache.service.loginbroker property is updated to take
a comma separated list of loginbrokers to register with or
to query. httpd now uses this property to configure which
loginbrokers to query.

New properties

loginbroker.cell.name
srm.protocols.disallowed.get
srm.protocols.disallowed.put
srm.protocols.preferred

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5742/
cleaner: Clean up cleaner.properties

Besides cleaning up the comments, this patch turns the previously
obsoleted property cleaner.hsm into a deprecated property.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5744/
spacemanager: Allow previously obsoleted values for deprecated srmSpaceManagerEnabled

This patch adds legacy values for srmSpaceManaerEnabled. Thus one can
once again use true, on, yes, enabled, false, off, no, disabled, but
only when using the legacy property.

Also fixed a bug in srm.batch introduced when previously deprecating
srmSpaceManagerEnabled.

Require-book: no
Require-notes: yes
Target: trunk
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5748/
replica: Restored legacy values of deprecated replicaManager property

When renaming the configuration properties of replica manager, the
replicaManager was marked deprecated and the allowed values changed
to true and false rather than yes and no. This patch restored the
legacy values such that yes and no can be used. The new
dcache.enable.replica property still requires the values true and
false.

The default value of the property has been changed from yes/true to
no/false, as this better reflect most dCache instances.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5749/
alarms: Comply with new configuration policy

RELEASE NOTES:
BOOK:

Deprecated:

alarms.server.config
alarms.store.db.type
alarms.store.path
alarms.store.db.rdbms
alarms.store.db.driver
alarms.store.db.host
alarms.store.db.user
alarms.store.db.pass
alarms.store.db.properties
alarms.store.db.url

New:

alarms.cell.name=alarms
alarms.log.config.path=${alarms.server.config}
(one-of?off|error|warn|info|debug|${alarms.server.log.level})alarms.log.root-level=${alarms.server.log.level}
(one-of?off|xml|rdbms|${alarms.store.db.type})alarms.db.type=off
alarms.db.xml.path=${alarms.store.path}
alarms.db.rdbms.type=${alarms.store.db.rdbms}
alarms.db.driver=${alarms.store.db.driver}
alarms.db.host=${$alarms.store.db.host}
alarms.db.user=${$alarms.store.db.user}
alarms.db.password=${alarms.store.db.pass}
alarms.db.name=alarms
alarms.db.config.path=${alarms.store.db.properties}
alarms.db.url=${alarms.store.db.url}
alarms.db.schema.auto
alarms.db.schema.changelog

Obsolete:
alarms.store.db.changelog

Target: master
Require-notes: yes
Require-book: yes
Patch: http://rb.dcache.org/r/5743
Acked-by: Gerd
Fix CellStub timeout handling

A recent change to CellStub introduced a time unit property. Thus
the interpretation of getTimeout() changed, as the value was no
longer guaranteed to be in milliseconds. This patch updates the
code that uses getTimeout() and either lets it use getTimeoutInMillis()
instead, or updates it to be time unit aware. The patch also fixes
several calls to setTimeout().

Thanks to Dmitry for finding this.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/dashboard/
ssh2: factory-out command factory

Reworking how username is delivered to the dCache shell means that
the CommandFactory may be instantiated by UniversalSpringCell, which
simplifies Ssh2Admin.

It also solves a potential race-condition should two users attempt to
login at the same time with different usernames.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5761/
Acked-by: Gerd Behrmann
billing: Comply with new configuration policy

RELEASE NOTES:
BOOK:

NEW:

billing.cell.name
billing.enable.text=${billing.enable.text.when-disable-is-${billingDisableTxt}}
billing.enable.db=${billing.enable.db.when-toDb-is-${billingToDb}}

billing.service.poolmanager.timeout=${poolConnectTimeout}
billing.service.poolmanager.timeout.unit=MILLISECONDS

billing.text.dir=${billingLogsDir}
billing.text.flat-dir
billing.text.format.mover-info-message=${billing.format.MoverInfoMessage}
billing.text.format.remove-file-info-message=${billing.format.RemoveFileInfoMessage}
billing.text.format.door-request-info-message=${billing.format.DoorRequestInfoMessage}
billing.text.format.storage-info-message=${billing.format.StorageInfoMessage}

billing.db.config.path=${billingInfoAccessPropertiesFile}
billing.db.inserts.max-before-commit=${billingMaxInsertsBeforeCommit}
billing.db.inserts.timeout-before-commit={billingMaxTimeBeforeCommitInSecs}
billing.db.inserts.timeout-before-commit.unit=SECONDS
billing.db.driver=${billingDbDriver}
billing.db.url=${billingDbUrl}
billing.db.host=${billingDbHost}
billing.db.user=${billingDbUser}
billing.db.password=${billingDbPass}
billing.db.password.file=${billingDbPgPassFileName}
billing.db.name=${billingDbName}

billing.db.connections.max-per-partition=30
billing.db.connections.min-per-partition=3
billing.db.schema.auto=${updateBillingDb}
billing.db.schema.changelog=${billing.db.schema.changelog-when-${billing.enable.db}}

DEPRECATED:

billingLogsDir
billingDisableTxt
billingToDb

poolConnectTimeout

billing.format.MoverInfoMessage
billing.format.RemoveFileInfoMessage
billing.format.DoorRequestInfoMessage
billing.format.StorageInfoMessage

updateBillingDb
billingInfoAccessPropertiesFile
billingMaxInsertsBeforeCommit
billingMaxTimeBeforeCommitInSecs
billingDbDriver
billingDbUrl
billingDbHost
billingDbUser
billingDbPass
billingDbPgPassFileName
billingDbName

OBSOLETE:

billingInfoAccess
billingChangelog

TO BE REMOVED (to httpd.properties):

generatePlots
billingPlotsExportType
webadmin.billing-plot.refresh-threshold
webadmin.billing-plot.height
webadmin.billing-plot.width
webadmin.billing-plot.default-style
webadmin.billing-plot.default-scale

Testing done:

For the camel-case conversion, deployed and issued gridftp write, checked that billing text message was logged properly.

Target: master
Patch: http://rb.dcache.org/r/5750/
Require-notes: yes
Require-book: yes
Acked-by: Gerd