system-test: Fix webdav protocol configuration

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5710/
cells: Log stack trace if cell constructor throws undeclared exception

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5709/
system-test: refactor custom host certificate support

This patch refactors and hopefully simplifies the host credential
support within system-test.

Although the patch modifies the hostCredential and, to a minor
extent, the populate script, the default system-test build is not
changed.  A host credential is stored in git that support 'localhost'
connections.  This is used by default.  Therefore the system-test
continues to work without storing any files externally.

As currently, the hostCredential utility can create host credentials,
either 'localhost'-only or with SubjectAlternativeName that match all
discovered possible FQDNs for the host.  The 'localhost'-only
certificate allows easy renewing of the git-stored credential.  The
SAN certificate is to allow some clients that use the Globus GSI
library to work correctly (gSOAP-based ones, but potentially others,
too).

The difference is that, in addition to storing the certificate in the
target/dcache directory, it is also stored in ~/.globus directory.
The populate script has been updated so that, if it detects a stored
host credential in ~/.globus, this is used in preference to the
git-stored credential.

Previously hostCredential required the --install option in addition
to the --generate option to trust the generated certificate.  Now,
this is implied: the script ensures that the generated certificate is
trusted.

The --clean command is added.  This removes all files in ~/.globus
directory that the command installs.  After running this command, the
system-test build will use the git-stored host credential.

The --install command installs all unknown certificates and
signing-policy files in ~/.globus/certificates.  This is similar to
what is does currently, but now it does this so that that installed
files may be deleted later on (by --clean)

The populate script continues to check if dCache-trusted certificates
are trusted by the user.  The output if one or more files are missing
has changed: rather than listing the missing files with instructions,
the developer is told to use 'hostCredential --trust' to fix the
problem.

If a dCache-trusted certificate is trusted by the developer, the
populate script checks that they are byte-wise identical files.  This
test is useful if the Disposable CA changes (e.g.,
'localhost'-certificate is renewed) as the filenames will remain the
same but have different contents.

Target: trunk
Requires-note: no
Requires-book: no
Patch: http://rb.dcache.org/r/5705
Acked-by: Gerd Behrmann
Add support in system-test for using user's trust-store

In general, Globus supports two client trust-stores; the system-wide
trust-store (/etc/grid-security/certificates) and a user-specific
trust-store (~/.globus/certificates).  Currently system-test project
copies only the system-wide trust-store.

This can be a problem for people that use the user-specific
trust-store (e.g., they do not have write privileges to the system
trust-store).

This patch adds support for building the test instance's trust-store
by copying certificates from both the system-wide and the user's
trust-store.

Target: trunk
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5707/
Acked-by: Gerd Behrmann
dcap: Comply with new configuration property policy

The patch replaces all dcap services with a single dcap service that
has a configurable authentication protocol. This avoids lots of
duplication between the dcap services.

Also fixes a couple of unreleased bugs in the ftp properties (became
apparent when I added property checks to the stage.fragment).

New properties

dcap.cell.name
dcap.authn.protocol
dcap.authn.passwd
dcap.authn.kerberos.service-principle-name
dcap.authn.kerberos.key-distribution-center-list
dcap.authn.hostcert.cert
dcap.authn.hostcert.key
dcap.authn.capath
dcap.authn.vomsdir
dcap.security.ciphers
dcap.authz.truncate
dcap.authz.anonymous-operations
dcap.authz.readonly
dcap.authz.staging
dcap.authz.staging.pep
dcap.net.port
dcap.mover.queue
dcap.mover.overwrite
dcap.limits.clients
dcap.service.loginbroker.family
dcap.service.loginbroker.version
dcap.service.pnfsmanager
dcap.service.gplazma
dcap.service.billing
dcap.service.spacemanager
dcap.service.pinmanager
dcap.service.poolmanager
ftp.authz.staging
ftp.authz.staging.pep

Deprecated

dCapPort
dCapGsiPort
dCapAuthPort
dCapKerberosPort
gsidcap.security.ciphers
truncate
dcapAnonymousAccessLevel
dcapReadOnly
dcapIoQueue
gsidcapIoQueue
kerberosdcapIoQueue
dcapIoQueueOverwrite
gsidcapIoQueueOverwrite
kerberosdcapIoQueueOverwrite
dcapMaxLogin
gsidcapMaxLogin
kerberosdcapMaxLogin
dcapPasswdFile

Forbidden

kerberos.service-principle-name

Removed

gsidcap.service.loginbroker
gsidcap.service.loginbroker.update-period
gsidcap.service.loginbroker.update-period.unit
gsidcap.service.loginbroker.update-treshold
kerberosdcap.service.loginbroker
kerberosdcap.service.loginbroker.update-period
kerberosdcap.service.loginbroker.update-period.unit
kererosdcap.service.loginbroker.update-treshold
authdcap.service.loginbroker
authdcap.service.loginbroker.update-period
authdcap.service.loginbroker.update-period.unit
authdcap.service.loginbroker.update-treshold
ftp.authz.stage
ftp.authz.stage.pep

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5698/
nfs: Comply with new configuration policy

Collapses nfsv3 and nfsv41 services into a single new service called
nfs. Use the nfs.version property to select between the different
NFS versions.

New properties

nfs.cell.name
nfs.version
nfs.net.port
nfs.mover.queue
nfs.service.loginbroker
nfs.service.loginbroker.update-period
nfs.service.loginbroker.update-period.unit
nfs.service.loginbroker.update-threshold
nfs.service.loginbroker.family
nfs.service.loginbroker.version
nfs.service.poolmanager
nfs.service.poolmanager.timeout
nfs.service.poolmanager.timeout.unit
nfs.service.pool.timeout
nfs.service.pool.timeout.unit
nfs.service.gplazma
nfs.service.gplazma.timeout
nfs.service.gplazma.timeout.unit
nfs.service.pnfsmanager
nfs.service.billing
nfs.db.host
nfs.db.name
nfs.db.user
nfs.db.password
nfs.db.url
nfs.db.schema
nfs.db.dialect

Deprecated

nfs.port
nfsIoQueue
nfs.published.name
nfs.poolmanager.timeout
nfs.pool.timeout
nfs.gplazma.timeout

Removed

nfsv41.service.loginbroker
nfsv41.service.loginbroker.update-period
nfsv41.service.loginbroker.update-period.unit
nfsv41.service.loginbroker.update-threshold

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5701/
Drop obsolete services

Drops the obsolete services acl, dummy-prestager and webadmin. All these
service definitions did was to log an error message.

Target: trunk
Require-book: no
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5702/
poolmanager: Comply with new configuration policy

New properties

poolmanager.cell.name
poolmanager.plugins.selection-unit
poolmanager.plugins.thread-pool
poolmanager.plugins.quota-manager
poolmanager.service.broadcast
poolmanager.service.pnfsmanager
poolmanager.service.billing
poolmanager.service.pool.timeout
poolmanager.service.pool.timeout.unit
poolmanager.authz.staging
poolmanager.authz.staging.pep

Deprecated

selectionUnit
threadPool
quotaManager
poolStatusRelay

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5703/
pnfsmanager: Comply with new configuration policy

New properties

pnfsmanager.cell.name
pnfsmanager.plugins.storage-info-extractor
pnfsmanager.limits.threads-per-group
pnfsmanager.limits.thread-groups
pnfsmanager.limits.list-threads
pnfsmanager.limits.list-chunk-size
pnfsmanager.limits.log-slow-threshold
pnfsmanager.limits.queue-length
pnfsmanager.folding
pnfsmanager.inherit-file-ownership
pnfsmanager.full-path-permission-check
pnfsmanager.acl
pnfsmanager.default-retention-policy
pnfsmanager.default-access-latency
pnfsmanager.db.connections.partition-count
pnfsmanager.db.connections.max-per-partition
pnfsmanager.db.connections.min-per-partition
pnfsmanager.db.host
pnfsmanager.db.name
pnfsmanager.db.user
pnfsmanager.db.password
pnfsmanager.db.driver
pnfsmanager.db.url
pnfsmanager.db.schema.changelog
pnfsmanager.db.dialect
pnfsmanager.atime-gap

Deprecated

hsmStorageInfoExtractor
pnfsNumberOfThreads
pnfsNumberOfThreadGroups
pnfsNumberOfListThreads
pnfsListChunkSize
pnfsLogSlowThreshold
pnfsQueueMaxSize
pnfsFolding
pnfsInheritFileOwnership
pnfsVerifyAllLookups
aclEnabled
DefaultRetentionPolicy
DefaultAccessLatency

Removed

namespaceProvider
pnfs
pnfsInfoExtractor
pnfsNumberOfLocationThreads
pnfsDbUser
pnfsDbPassword
pnfsPasswordFile
cacheInfo
companionDatabaseHost

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5706/
replica: Comply with new configuration policy

Drops the 'debug' admin shell command.

New properties:

replica.cell.name
replica.poolgroup
replica.limits.pool-watchdog-period
replica.limits.pool-watchdog-period.unit
replica.limits.wait-db-update-timeout
replica.limits.wait-db-update-timeout.unit
replica.limits.excluded-files-expiration-timeout
replica.limits.excluded-files-expiration-timeout.unit
replica.limits.delay-db-start-timeout
replica.limits.delay-db-start-timeout.unit
replica.limits.wait-replicate-timeout
replica.limits.wait-replicate-timeout.unit
replica.limits.wait-reduce-timeout
replica.limits.wait-reduce-timeout.unit
replica.limits.adjust-start-timeout
replica.limits.adjust-start-timeout.unit
replica.limits.workers
replica.limits.min
replica.limits.max
replica.enable.check-pool-host
replica.enable.same-host-replica
replica.db.host
replica.db.name
replica.db.user
replica.db.password
replica.db.password.file
replica.db.driver
replica.db.url
replica.db.schema.auto

Deprecated

resilientGroupName
replicaPoolWatchDogPeriod
replicaWaitDBUpdateTimeout
replicaExcludedFilesExpirationTimeout
replicaDelayDBStartTimeout
replicaAdjustStartTimeout
replicaWaitReplicateTimeout
replicaWaitReduceTimeout
replicaMaxWorkers
replicaMin
replicaMax
replicaCheckPoolHost
replicaEnableSameHostReplica
replicaManagerDatabaseHost
replicaDbName
replicaDbUser
replicaDbPassword
replicaPasswordFile
replicaDbJdbcDriver

Obsolete

replicaDebug

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5708/
pool: Allow multiple pool groups in migration target

This was a feature request from our admins.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5711/
Remove meaningless logging on replica-manager startup

ReplicaManager logs some meaningless values at warning value.  An
example is:

    05 Jul 2013 22:56:52 (System) [] ReplicaManager   version: $Id$
    05 Jul 2013 22:56:52 (System) [] DCacheController version: $Id$
    05 Jul 2013 22:56:52 (System) [] resilientGroupName=ResilientPools

The final line erroneously includes a new line.

This patch simply removes all references to the $Id$, fixes the
resilientGroupName logging and demotes it to info-level.

Target: trunk
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5713/
Acked-by: Gerd Behrmann
dcap: better handling of client-supplied unresolvable hostname

The dcap protocol has the client send its hostname to the server when
opening a file in 'active' mode.

Under some circumstances, our current client becomes confused and
sends a name that the server cannot resolve. (This happened to me when
using a laptop without Internet connectivity, but I'm unsure how
exactly to reproduce this.)  The current code assumes the
client-supplied hostname can be resolved.  As a result, when the name
cannot be resolved, an NPE is generated, which is logged in the
server:

06 Jul 2013 15:33:15 (DCap-gsi-vedrfolnir-/C=DE/O=GermanGrid/OU=DESY/CN=Alexander Paul Millar-101) [] Uncaught exception in thread Thread-130
java.lang.NullPointerException: null
        at diskCacheV111.doors.DCapDoorInterpreterV3$IoHandler.getIoDoorEntry(DCapDoorInterpreterV3.java:1966) ~[dcache-dcap-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at diskCacheV111.doors.DCapDoorInterpreterV3.ac_get_door_info(DCapDoorInterpreterV3.java:803) ~[dcache-dcap-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_21]
        at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_21]
        at dmg.util.command.AcCommandExecutor.execute(AcCommandExecutor.java:157) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.util.CommandInterpreter$CommandEntry.execute(CommandInterpreter.java:385) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.util.CommandInterpreter.execute(CommandInterpreter.java:281) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.util.CommandInterpreter.command(CommandInterpreter.java:215) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellAdapter.executeLocalCommand(CellAdapter.java:1020) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellAdapter.messageArrived(CellAdapter.java:939) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$DeliverMessageTask.innerRun(CellNucleus.java:1020) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:929) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_21]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_21]
        at dmg.cells.nucleus.CellNucleus$1.run(CellNucleus.java:657) [cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_21]

When this happens, the client waits and retries.  The retry will use
the same hostname, so will generate a fresh NPE stack-track.  The
client (at least, dccp) appears not to time out; therefore the
transfer will appear stuck until the user (or batch system)
intervenes.

The server should be robust against such client-side mistakes.  Dcap
clients allow the user to override which hostname is sent, which both
introduces the possibility of human error and allows the situation to
be simulated.

This patch allows the server to better handle this failure mode.  A
CacheException is thrown, which is propagated to the client and
terminates the transfer immediately.  The problem is logged at debug
level.

Target: master
Requires-book: no
Requires-notes: no
Patch: http://rb.dcache.org/r/5717/
Acked-by: Gerd Behrmann
admin: remove some dead code

Target: master
Requires-book: no
Requires-notes: no
Acked-by: Gerd Behrmann
Patch: http://rb.dcache.org/r/5716/
httpd: fix engine-based responses that don't implement getInfo

On starting system-test, I observed errors like:

06 Jul 2013 08:42:19 (System) [] context:httpdSetup: line 18: Command failed ((3) java.lang.NoSuchMethodException:
diskCacheV111.services.web.PoolInfoObserverEngineV2.getInfo(java.io.PrintWriter) from ac_set_alias_$_3_16; caused by:
diskCacheV111.services.web.PoolInfoObserverEngineV2.getInfo(java.io.PrintWriter))

This error is printed for all the classes declared through the 'alias'
command as having type 'class', except HttpPoolMgrEngineV3.  This
class is not logged because it's the only HttpResponseEngine to
implement the getInfo method.

The effect of these error messages is that the corresponding aliases
fail to work; i.e., URIs with the path starting /billing,
/flushManager, /pools, /api and /info no longer return the expected
content.

The problem seems to come from patch http://rb.dcache.org/r/5690/
(committed on master as 0039ddbe9de626aff2d06f8a1900689b59f5b26c).
This mostly focused on fixing logging; however, it also includes a few
bug-fixes.  One such bug-fix was for the AliasEntry class, used by the
httpd service.  This class does have a bug (the ResponseEngineHandler
instance is used when invoking the getInfo method, rather than the
HttpResponseEngine instance), however the above patch fails to fix it;
instead it breaks the Engine-based httpd services as described above.

Rather than simply reverting the broken part of the patch, this patch
removes the use reflection altogether.  Instead, the CellInfoProvider
interface is used to mark those HttpResponseEngine classes that
provide cell information.  To achieve this, the CellInfoProvider is
moved from the 'dcache' project to the 'cells' project.  A number of
classes have minor updates that correspond to this change.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5714/
Acked-by: Gerd Behrmann
cells: Fix deadlock

Addresses a regression introduced in 45850f5d50d867772d91b8526d9b7210ae9aaa55. The
regression introduces a deadlock caused by low-level class loading logic in AspectJ
using the logging system, and our logging system looks up cells through getCell(),
a method made synchronized by the above commit.

The solution adopted is to not have two cell maps. Rather we have a set of killed
cells to track the fact that a cell was killed.

Target: trunk
Request: 2.6
Request: 2.2
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5718/
Switch voms-api version to 2.0.6

We are using an unsupported and buggy voms-api version

RELEASE NOTES: Switch to voms-api to 2.0.6 fixing "Certificate verification: Maximum certification path length exceeded." exception problem

Patch: <ReviewBoard URL>
Require-notes: yes
Require-book:  no
Bug: http://rt.dcache.org/Ticket/Display.html?id=7848
Acked-by: <name of person>
Target: master
Request: 2.2
Request: 2.6
Merge-req:

The problem I have, I can not test the solution myself as it only happens with certain proxy certificates, see bug ticket for details.

I have however tested that the update does not break g2 and s2 while I did not test xrootd as I have no idea about it.

G2 testing: http://svn.dcache.org/build/view/emi_certification/job/g2-Certification_oneMachine/243/,

mvn building fine, while testing switched on,

S2 test went through fine except for one test, which I would see as a typical S2 timeout problem: http://svn.dcache.org/build/view/emi_certification/job/S2-certification_oneMachine/21/
pool: Add missing attributes in replicateOnArrival request

Addresses an issue in the pool's replicateOnArrival feature. If enabled it
failed with an error stating that required attributes where missing.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5715/
gsi-pam: Remove service

The gsi-pam service appears to be a predecessor of gPlazma. The service
can be used by LoginManager to authenticate users. Since all our doors
support gPlazma, gsi-pam should be dropped as a service - we didn't
support the options to configure LoginManager to use gsi-pam anyway. Also,
it appears that LoginManager only uses the authenticator when configured
with an SSL tunnel, which non of our services use.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5721/
libs: Update to latest versions

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5724/