ssh-admin: quieten noisy logging

We deliver dCache without an authorized_keys2 file, yet
attempts to log in when a user presents an SSH key results in
logging like:

03 Jul 2013 14:11:40 (ssh2Admin) [] The authorized_keys2 file was not found. Please check if it exists in: /home/paul/git/dCache/packages/system-test/target/dcache/etc/admin/authorized_keys2
03 Jul 2013 14:11:40 (ssh2Admin) [] Could not find key: RSA Public Key
            modulus: b724ef9e0182b02db76ea20a0f674d83 [many hex digits follow ...]
    public exponent: 23
in /home/paul/git/dCache/packages/system-test/target/dcache/etc/admin/authorized_keys2.
03 Jul 2013 14:11:40 (ssh2Admin) [] The authorized_keys2 file was not found. Please check if it exists in: /home/paul/git/dCache/packages/system-test/target/dcache/etc/admin/authorized_keys2
03 Jul 2013 14:11:40 (ssh2Admin) [] Could not find key: DSA Public Key
            y: 51e74d6deb9b0b87456522d56f4fb952c21ec0 [many hex digits follow ...]
in /home/paul/git/dCache/packages/system-test/target/dcache/etc/admin/authorized_keys2.

This is wrong as the authorized_keys2 file is not required and
the login attempt was actually successful, using username+password
authentication.

This patch reduces the above messages to a single message per failure to
read the authorized_keys2 file (rather than two) and logs this message at
debug level, rather than error.

Some log messages were also rephrased to avoid unnecessary verbosity.

Target: master
Requires-book: no
Requires-notes: no
Patch: http://rb.dcache.org/r/5696/
Acked-by: Gerd Behrmann
Fix unit-tests for maven-2

Gerd reported that a recent patch (a019dc43) broke unit-tests.  This
was surprising as, before submitting, the patch was tested and all
unit-tests passed.

The patch introduced new dependencies, including
'powermock-api-mockito'.  This dependency, in turn, declares a
dependency on mockito-all, which is the Mockito jar that includes all
its dependencies within the jar file.  One of these dependencies is
Hamcrest; i.e., the mockito-all jar contains class files from some
unspecified (but probably v1.1) version of Hamcrest.

Unit tests were compiled against the declared version of Hamcrest
(v1.3).  When run within the JUnit framework, the classes loaded will
depend on the classpath order.  When mockito-all precedes the Hamcrest
jar, some unit tests were in error.  These are tests that use methods
in Hamcrest that have a different signature between v1.1 and v1.3.
Therefore, these tests would throw a NoSuchMethodException when run.

It seems that, for Maven v3, the Hamcrest v1.3 jar appeared before the
mockito-all jar whereas, for Maven v2, it is the other way around.
Therefore, the patch worked when tested, but only by chance.

This patch fixes this problem by excluding the mockito-all dependency
and including the mockito-core dependency.  The Mockito classes in the
latter has the same dependencies as the former, but this is expressed
through the pom.xml file.  For both compilation and running within
JUnit, the Hamcrest dependency is satisfied by the Hamcrest v1.3 jar
file.

Target: trunk
Requires-notes: no
Requires-book: no
Acked-by: Gerd Behrmann
Patch: http://rb.dcache.org/r/5704/
ssh-admin: fix race condition when shutting down

When an administrator logs out of the SSH admin interface, occasionally
an error message is logged:

    (ssh2Admin) [] Something went wrong cleaning up the console: Already closed

This problem comes about from a race condition in the
ConsoleReaderCommand class.  The finally-clause in the run method calls
the onExit ExitCallback then immediately attempts to send a newline
character and flush the console.

Unfortunately, the onExit method sends the SSH-protocol EOF and
exit-code messages and schedules closing down the channel.  This
creates a race between the scheduled closing of the channel and the
ConsoleReaderCommand's sending a final newline and (itself) closing the
connection.  Mostly ConsoleReaderCommand wins the race, in which case
there's no problem; however, if it looses the race then the cleanUp
attempts to write to an output that is already closed.

This patch resolves this issue by moving the onExit call until after
the newline has been sent.  It also rewords the error message to be
slightly less verbose.

FOR RELEASE NOTES:

A rare race condition when logging out of the admin interface has been
fixed.  If the problem is triggered then the message "Something went
wrong cleaning up the console: Already closed" is recorded in the log
file.

Target: master
Requires-book: no
Requires-notes: yes
Patch: http://rb.dcache.org/r/5699
Acked-by: Gerd Behrmann
pool: Fix memory leak in xrootd and httpd movers

Addresses a slow memory leak that occurs whenever more than one httpd
or more than one xrootd transfer is started on a pool. Also reduces
http/xrootd server component startup overhead.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5695/
cells: Fix race conditions in cell lookup

When cells are killed, they are moved from one map to another map. This operation
is not atomic and methods that look in both places need to be synchronized.

This addresses a null pointer exception experienced in the DCAP door:

java.lang.NullPointerException: null
        at diskCacheV111.doors.DCapDoorInterpreterV3.ac_get_door_info(DCapDoorInterpreterV3.java:733) ~[dcache-dcap-2.2.13.jar:2.2.13]
        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_19]
        at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_19]
        at dmg.util.CommandInterpreter.execute(CommandInterpreter.java:609) ~[cells-2.2.13.jar:2.2.13]
        at dmg.util.CommandInterpreter.command(CommandInterpreter.java:486) ~[cells-2.2.13.jar:2.2.13]
        at dmg.cells.nucleus.CellAdapter.executeLocalCommand(CellAdapter.java:1020) ~[cells-2.2.13.jar:2.2.13]
        at dmg.cells.nucleus.CellAdapter.messageArrived(CellAdapter.java:940) ~[cells-2.2.13.jar:2.2.13]
        at dmg.cells.nucleus.CellNucleus$DeliverMessageTask.innerRun(CellNucleus.java:1051) ~[cells-2.2.13.jar:2.2.13]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:949) ~[cells-2.2.13.jar:2.2.13]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_19]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_19]
        at dmg.cells.nucleus.CellNucleus$1.run(CellNucleus.java:654) [cells-2.2.13.jar:2.2.13]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_19]

Target: trunk
Request: 2.6
Request: 2.2
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7896
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5700/
pool: Remove dead code for compatibility with very old pools

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5693/
spacemanager: Don't rely on AuthorizationRecord

Several spacemanager messages still used AuthorizationRecord rather than the
simpler Subject class. Space manager itself internally also used
AuthorizationRecord.

This patch updates space manager to use Subject directly. All spacemanager
related messages are updated to send Subject rather than AuthorizationRecord.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5684/
system-test: Fix webdav protocol configuration

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5710/
cells: Log stack trace if cell constructor throws undeclared exception

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5709/
system-test: refactor custom host certificate support

This patch refactors and hopefully simplifies the host credential
support within system-test.

Although the patch modifies the hostCredential and, to a minor
extent, the populate script, the default system-test build is not
changed.  A host credential is stored in git that support 'localhost'
connections.  This is used by default.  Therefore the system-test
continues to work without storing any files externally.

As currently, the hostCredential utility can create host credentials,
either 'localhost'-only or with SubjectAlternativeName that match all
discovered possible FQDNs for the host.  The 'localhost'-only
certificate allows easy renewing of the git-stored credential.  The
SAN certificate is to allow some clients that use the Globus GSI
library to work correctly (gSOAP-based ones, but potentially others,
too).

The difference is that, in addition to storing the certificate in the
target/dcache directory, it is also stored in ~/.globus directory.
The populate script has been updated so that, if it detects a stored
host credential in ~/.globus, this is used in preference to the
git-stored credential.

Previously hostCredential required the --install option in addition
to the --generate option to trust the generated certificate.  Now,
this is implied: the script ensures that the generated certificate is
trusted.

The --clean command is added.  This removes all files in ~/.globus
directory that the command installs.  After running this command, the
system-test build will use the git-stored host credential.

The --install command installs all unknown certificates and
signing-policy files in ~/.globus/certificates.  This is similar to
what is does currently, but now it does this so that that installed
files may be deleted later on (by --clean)

The populate script continues to check if dCache-trusted certificates
are trusted by the user.  The output if one or more files are missing
has changed: rather than listing the missing files with instructions,
the developer is told to use 'hostCredential --trust' to fix the
problem.

If a dCache-trusted certificate is trusted by the developer, the
populate script checks that they are byte-wise identical files.  This
test is useful if the Disposable CA changes (e.g.,
'localhost'-certificate is renewed) as the filenames will remain the
same but have different contents.

Target: trunk
Requires-note: no
Requires-book: no
Patch: http://rb.dcache.org/r/5705
Acked-by: Gerd Behrmann
Add support in system-test for using user's trust-store

In general, Globus supports two client trust-stores; the system-wide
trust-store (/etc/grid-security/certificates) and a user-specific
trust-store (~/.globus/certificates).  Currently system-test project
copies only the system-wide trust-store.

This can be a problem for people that use the user-specific
trust-store (e.g., they do not have write privileges to the system
trust-store).

This patch adds support for building the test instance's trust-store
by copying certificates from both the system-wide and the user's
trust-store.

Target: trunk
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5707/
Acked-by: Gerd Behrmann
dcap: Comply with new configuration property policy

The patch replaces all dcap services with a single dcap service that
has a configurable authentication protocol. This avoids lots of
duplication between the dcap services.

Also fixes a couple of unreleased bugs in the ftp properties (became
apparent when I added property checks to the stage.fragment).

New properties

dcap.cell.name
dcap.authn.protocol
dcap.authn.passwd
dcap.authn.kerberos.service-principle-name
dcap.authn.kerberos.key-distribution-center-list
dcap.authn.hostcert.cert
dcap.authn.hostcert.key
dcap.authn.capath
dcap.authn.vomsdir
dcap.security.ciphers
dcap.authz.truncate
dcap.authz.anonymous-operations
dcap.authz.readonly
dcap.authz.staging
dcap.authz.staging.pep
dcap.net.port
dcap.mover.queue
dcap.mover.overwrite
dcap.limits.clients
dcap.service.loginbroker.family
dcap.service.loginbroker.version
dcap.service.pnfsmanager
dcap.service.gplazma
dcap.service.billing
dcap.service.spacemanager
dcap.service.pinmanager
dcap.service.poolmanager
ftp.authz.staging
ftp.authz.staging.pep

Deprecated

dCapPort
dCapGsiPort
dCapAuthPort
dCapKerberosPort
gsidcap.security.ciphers
truncate
dcapAnonymousAccessLevel
dcapReadOnly
dcapIoQueue
gsidcapIoQueue
kerberosdcapIoQueue
dcapIoQueueOverwrite
gsidcapIoQueueOverwrite
kerberosdcapIoQueueOverwrite
dcapMaxLogin
gsidcapMaxLogin
kerberosdcapMaxLogin
dcapPasswdFile

Forbidden

kerberos.service-principle-name

Removed

gsidcap.service.loginbroker
gsidcap.service.loginbroker.update-period
gsidcap.service.loginbroker.update-period.unit
gsidcap.service.loginbroker.update-treshold
kerberosdcap.service.loginbroker
kerberosdcap.service.loginbroker.update-period
kerberosdcap.service.loginbroker.update-period.unit
kererosdcap.service.loginbroker.update-treshold
authdcap.service.loginbroker
authdcap.service.loginbroker.update-period
authdcap.service.loginbroker.update-period.unit
authdcap.service.loginbroker.update-treshold
ftp.authz.stage
ftp.authz.stage.pep

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5698/
nfs: Comply with new configuration policy

Collapses nfsv3 and nfsv41 services into a single new service called
nfs. Use the nfs.version property to select between the different
NFS versions.

New properties

nfs.cell.name
nfs.version
nfs.net.port
nfs.mover.queue
nfs.service.loginbroker
nfs.service.loginbroker.update-period
nfs.service.loginbroker.update-period.unit
nfs.service.loginbroker.update-threshold
nfs.service.loginbroker.family
nfs.service.loginbroker.version
nfs.service.poolmanager
nfs.service.poolmanager.timeout
nfs.service.poolmanager.timeout.unit
nfs.service.pool.timeout
nfs.service.pool.timeout.unit
nfs.service.gplazma
nfs.service.gplazma.timeout
nfs.service.gplazma.timeout.unit
nfs.service.pnfsmanager
nfs.service.billing
nfs.db.host
nfs.db.name
nfs.db.user
nfs.db.password
nfs.db.url
nfs.db.schema
nfs.db.dialect

Deprecated

nfs.port
nfsIoQueue
nfs.published.name
nfs.poolmanager.timeout
nfs.pool.timeout
nfs.gplazma.timeout

Removed

nfsv41.service.loginbroker
nfsv41.service.loginbroker.update-period
nfsv41.service.loginbroker.update-period.unit
nfsv41.service.loginbroker.update-threshold

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5701/
Drop obsolete services

Drops the obsolete services acl, dummy-prestager and webadmin. All these
service definitions did was to log an error message.

Target: trunk
Require-book: no
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5702/
poolmanager: Comply with new configuration policy

New properties

poolmanager.cell.name
poolmanager.plugins.selection-unit
poolmanager.plugins.thread-pool
poolmanager.plugins.quota-manager
poolmanager.service.broadcast
poolmanager.service.pnfsmanager
poolmanager.service.billing
poolmanager.service.pool.timeout
poolmanager.service.pool.timeout.unit
poolmanager.authz.staging
poolmanager.authz.staging.pep

Deprecated

selectionUnit
threadPool
quotaManager
poolStatusRelay

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5703/
pnfsmanager: Comply with new configuration policy

New properties

pnfsmanager.cell.name
pnfsmanager.plugins.storage-info-extractor
pnfsmanager.limits.threads-per-group
pnfsmanager.limits.thread-groups
pnfsmanager.limits.list-threads
pnfsmanager.limits.list-chunk-size
pnfsmanager.limits.log-slow-threshold
pnfsmanager.limits.queue-length
pnfsmanager.folding
pnfsmanager.inherit-file-ownership
pnfsmanager.full-path-permission-check
pnfsmanager.acl
pnfsmanager.default-retention-policy
pnfsmanager.default-access-latency
pnfsmanager.db.connections.partition-count
pnfsmanager.db.connections.max-per-partition
pnfsmanager.db.connections.min-per-partition
pnfsmanager.db.host
pnfsmanager.db.name
pnfsmanager.db.user
pnfsmanager.db.password
pnfsmanager.db.driver
pnfsmanager.db.url
pnfsmanager.db.schema.changelog
pnfsmanager.db.dialect
pnfsmanager.atime-gap

Deprecated

hsmStorageInfoExtractor
pnfsNumberOfThreads
pnfsNumberOfThreadGroups
pnfsNumberOfListThreads
pnfsListChunkSize
pnfsLogSlowThreshold
pnfsQueueMaxSize
pnfsFolding
pnfsInheritFileOwnership
pnfsVerifyAllLookups
aclEnabled
DefaultRetentionPolicy
DefaultAccessLatency

Removed

namespaceProvider
pnfs
pnfsInfoExtractor
pnfsNumberOfLocationThreads
pnfsDbUser
pnfsDbPassword
pnfsPasswordFile
cacheInfo
companionDatabaseHost

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5706/
replica: Comply with new configuration policy

Drops the 'debug' admin shell command.

New properties:

replica.cell.name
replica.poolgroup
replica.limits.pool-watchdog-period
replica.limits.pool-watchdog-period.unit
replica.limits.wait-db-update-timeout
replica.limits.wait-db-update-timeout.unit
replica.limits.excluded-files-expiration-timeout
replica.limits.excluded-files-expiration-timeout.unit
replica.limits.delay-db-start-timeout
replica.limits.delay-db-start-timeout.unit
replica.limits.wait-replicate-timeout
replica.limits.wait-replicate-timeout.unit
replica.limits.wait-reduce-timeout
replica.limits.wait-reduce-timeout.unit
replica.limits.adjust-start-timeout
replica.limits.adjust-start-timeout.unit
replica.limits.workers
replica.limits.min
replica.limits.max
replica.enable.check-pool-host
replica.enable.same-host-replica
replica.db.host
replica.db.name
replica.db.user
replica.db.password
replica.db.password.file
replica.db.driver
replica.db.url
replica.db.schema.auto

Deprecated

resilientGroupName
replicaPoolWatchDogPeriod
replicaWaitDBUpdateTimeout
replicaExcludedFilesExpirationTimeout
replicaDelayDBStartTimeout
replicaAdjustStartTimeout
replicaWaitReplicateTimeout
replicaWaitReduceTimeout
replicaMaxWorkers
replicaMin
replicaMax
replicaCheckPoolHost
replicaEnableSameHostReplica
replicaManagerDatabaseHost
replicaDbName
replicaDbUser
replicaDbPassword
replicaPasswordFile
replicaDbJdbcDriver

Obsolete

replicaDebug

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5708/
pool: Allow multiple pool groups in migration target

This was a feature request from our admins.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5711/
Remove meaningless logging on replica-manager startup

ReplicaManager logs some meaningless values at warning value.  An
example is:

    05 Jul 2013 22:56:52 (System) [] ReplicaManager   version: $Id$
    05 Jul 2013 22:56:52 (System) [] DCacheController version: $Id$
    05 Jul 2013 22:56:52 (System) [] resilientGroupName=ResilientPools

The final line erroneously includes a new line.

This patch simply removes all references to the $Id$, fixes the
resilientGroupName logging and demotes it to info-level.

Target: trunk
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5713/
Acked-by: Gerd Behrmann
dcap: better handling of client-supplied unresolvable hostname

The dcap protocol has the client send its hostname to the server when
opening a file in 'active' mode.

Under some circumstances, our current client becomes confused and
sends a name that the server cannot resolve. (This happened to me when
using a laptop without Internet connectivity, but I'm unsure how
exactly to reproduce this.)  The current code assumes the
client-supplied hostname can be resolved.  As a result, when the name
cannot be resolved, an NPE is generated, which is logged in the
server:

06 Jul 2013 15:33:15 (DCap-gsi-vedrfolnir-/C=DE/O=GermanGrid/OU=DESY/CN=Alexander Paul Millar-101) [] Uncaught exception in thread Thread-130
java.lang.NullPointerException: null
        at diskCacheV111.doors.DCapDoorInterpreterV3$IoHandler.getIoDoorEntry(DCapDoorInterpreterV3.java:1966) ~[dcache-dcap-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at diskCacheV111.doors.DCapDoorInterpreterV3.ac_get_door_info(DCapDoorInterpreterV3.java:803) ~[dcache-dcap-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_21]
        at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_21]
        at dmg.util.command.AcCommandExecutor.execute(AcCommandExecutor.java:157) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.util.CommandInterpreter$CommandEntry.execute(CommandInterpreter.java:385) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.util.CommandInterpreter.execute(CommandInterpreter.java:281) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.util.CommandInterpreter.command(CommandInterpreter.java:215) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellAdapter.executeLocalCommand(CellAdapter.java:1020) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellAdapter.messageArrived(CellAdapter.java:939) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$DeliverMessageTask.innerRun(CellNucleus.java:1020) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:929) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_21]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_21]
        at dmg.cells.nucleus.CellNucleus$1.run(CellNucleus.java:657) [cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_21]

When this happens, the client waits and retries.  The retry will use
the same hostname, so will generate a fresh NPE stack-track.  The
client (at least, dccp) appears not to time out; therefore the
transfer will appear stuck until the user (or batch system)
intervenes.

The server should be robust against such client-side mistakes.  Dcap
clients allow the user to override which hostname is sent, which both
introduces the possibility of human error and allows the situation to
be simulated.

This patch allows the server to better handle this failure mode.  A
CacheException is thrown, which is propagated to the client and
terminates the transfer immediately.  The problem is logged at debug
level.

Target: master
Requires-book: no
Requires-notes: no
Patch: http://rb.dcache.org/r/5717/
Acked-by: Gerd Behrmann