dcap: better handling of client-supplied unresolvable hostname

The dcap protocol has the client send its hostname to the server when
opening a file in 'active' mode.

Under some circumstances, our current client becomes confused and
sends a name that the server cannot resolve. (This happened to me when
using a laptop without Internet connectivity, but I'm unsure how
exactly to reproduce this.)  The current code assumes the
client-supplied hostname can be resolved.  As a result, when the name
cannot be resolved, an NPE is generated, which is logged in the
server:

06 Jul 2013 15:33:15 (DCap-gsi-vedrfolnir-/C=DE/O=GermanGrid/OU=DESY/CN=Alexander Paul Millar-101) [] Uncaught exception in thread Thread-130
java.lang.NullPointerException: null
        at diskCacheV111.doors.DCapDoorInterpreterV3$IoHandler.getIoDoorEntry(DCapDoorInterpreterV3.java:1966) ~[dcache-dcap-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at diskCacheV111.doors.DCapDoorInterpreterV3.ac_get_door_info(DCapDoorInterpreterV3.java:803) ~[dcache-dcap-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_21]
        at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_21]
        at dmg.util.command.AcCommandExecutor.execute(AcCommandExecutor.java:157) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.util.CommandInterpreter$CommandEntry.execute(CommandInterpreter.java:385) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.util.CommandInterpreter.execute(CommandInterpreter.java:281) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.util.CommandInterpreter.command(CommandInterpreter.java:215) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellAdapter.executeLocalCommand(CellAdapter.java:1020) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellAdapter.messageArrived(CellAdapter.java:939) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$DeliverMessageTask.innerRun(CellNucleus.java:1020) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:929) ~[cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_21]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_21]
        at dmg.cells.nucleus.CellNucleus$1.run(CellNucleus.java:657) [cells-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_21]

When this happens, the client waits and retries.  The retry will use
the same hostname, so will generate a fresh NPE stack-track.  The
client (at least, dccp) appears not to time out; therefore the
transfer will appear stuck until the user (or batch system)
intervenes.

The server should be robust against such client-side mistakes.  Dcap
clients allow the user to override which hostname is sent, which both
introduces the possibility of human error and allows the situation to
be simulated.

This patch allows the server to better handle this failure mode.  A
CacheException is thrown, which is propagated to the client and
terminates the transfer immediately.  The problem is logged at debug
level.

Target: master
Requires-book: no
Requires-notes: no
Patch: http://rb.dcache.org/r/5717/
Acked-by: Gerd Behrmann
admin: remove some dead code

Target: master
Requires-book: no
Requires-notes: no
Acked-by: Gerd Behrmann
Patch: http://rb.dcache.org/r/5716/
httpd: fix engine-based responses that don't implement getInfo

On starting system-test, I observed errors like:

06 Jul 2013 08:42:19 (System) [] context:httpdSetup: line 18: Command failed ((3) java.lang.NoSuchMethodException:
diskCacheV111.services.web.PoolInfoObserverEngineV2.getInfo(java.io.PrintWriter) from ac_set_alias_$_3_16; caused by:
diskCacheV111.services.web.PoolInfoObserverEngineV2.getInfo(java.io.PrintWriter))

This error is printed for all the classes declared through the 'alias'
command as having type 'class', except HttpPoolMgrEngineV3.  This
class is not logged because it's the only HttpResponseEngine to
implement the getInfo method.

The effect of these error messages is that the corresponding aliases
fail to work; i.e., URIs with the path starting /billing,
/flushManager, /pools, /api and /info no longer return the expected
content.

The problem seems to come from patch http://rb.dcache.org/r/5690/
(committed on master as 0039ddbe9de626aff2d06f8a1900689b59f5b26c).
This mostly focused on fixing logging; however, it also includes a few
bug-fixes.  One such bug-fix was for the AliasEntry class, used by the
httpd service.  This class does have a bug (the ResponseEngineHandler
instance is used when invoking the getInfo method, rather than the
HttpResponseEngine instance), however the above patch fails to fix it;
instead it breaks the Engine-based httpd services as described above.

Rather than simply reverting the broken part of the patch, this patch
removes the use reflection altogether.  Instead, the CellInfoProvider
interface is used to mark those HttpResponseEngine classes that
provide cell information.  To achieve this, the CellInfoProvider is
moved from the 'dcache' project to the 'cells' project.  A number of
classes have minor updates that correspond to this change.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5714/
Acked-by: Gerd Behrmann
cells: Fix deadlock

Addresses a regression introduced in 45850f5d50d867772d91b8526d9b7210ae9aaa55. The
regression introduces a deadlock caused by low-level class loading logic in AspectJ
using the logging system, and our logging system looks up cells through getCell(),
a method made synchronized by the above commit.

The solution adopted is to not have two cell maps. Rather we have a set of killed
cells to track the fact that a cell was killed.

Target: trunk
Request: 2.6
Request: 2.2
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5718/
Switch voms-api version to 2.0.6

We are using an unsupported and buggy voms-api version

RELEASE NOTES: Switch to voms-api to 2.0.6 fixing "Certificate verification: Maximum certification path length exceeded." exception problem

Patch: <ReviewBoard URL>
Require-notes: yes
Require-book:  no
Bug: http://rt.dcache.org/Ticket/Display.html?id=7848
Acked-by: <name of person>
Target: master
Request: 2.2
Request: 2.6
Merge-req:

The problem I have, I can not test the solution myself as it only happens with certain proxy certificates, see bug ticket for details.

I have however tested that the update does not break g2 and s2 while I did not test xrootd as I have no idea about it.

G2 testing: http://svn.dcache.org/build/view/emi_certification/job/g2-Certification_oneMachine/243/,

mvn building fine, while testing switched on,

S2 test went through fine except for one test, which I would see as a typical S2 timeout problem: http://svn.dcache.org/build/view/emi_certification/job/S2-certification_oneMachine/21/
pool: Add missing attributes in replicateOnArrival request

Addresses an issue in the pool's replicateOnArrival feature. If enabled it
failed with an error stating that required attributes where missing.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5715/
gsi-pam: Remove service

The gsi-pam service appears to be a predecessor of gPlazma. The service
can be used by LoginManager to authenticate users. Since all our doors
support gPlazma, gsi-pam should be dropped as a service - we didn't
support the options to configure LoginManager to use gsi-pam anyway. Also,
it appears that LoginManager only uses the authenticator when configured
with an SSL tunnel, which non of our services use.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5721/
libs: Update to latest versions

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5724/
libs: Upgrade datanucleus

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5726/
info: switch to using UniversalSpringCell

This is a (long overdue) patch to update the info cell to use
UniversalSpringCell.  Most of the wiring is now handled in a spring XML
file.

The patch tries to strike a balance between being complete and trying
to do too much, so that the patch is review-able.  For example,
messages that request serialised data (from web-server) are handled via
the CellMessageReceiver interface but the DataGatheringActivities send
requests for data and processing the responses through a custom
framework.

Target: trunk
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5689/
Acked-by: Gerd Behrmann
Acked-by: Albert Rossi
pool: Comply with new configuration policy

Besides changing the property names, the patch alters the replicate
feature: The arguments to 'set replication' have changed. Destination
mode is no longer accepted as it was never implemented. The replication
notification is now generated no matter whether a file arrived by
restore or from a client. In previous versions the replication was not
triggered when restoring from tape. This however meant that the
information in our book regarding replication on restore would be wrong.

New properties

pool.name
pool.path
pool.queues
pool.lfs
pool.size
pool.tags
pool.enable.repository-check
pool.enable.remove-precious-files-on-delete
pool.cell.name
pool.cell.limits.message.threads.min
pool.cell.limits.message.threads.max
pool.cell.limits.message.threads.max-idle-time
pool.cell.limits.message.threads.max-idle-time.unit
pool.cell.limits.message.queue.max
pool.cell.limits.callback.threads.min
pool.cell.limits.callback.threads.max
pool.cell.limits.callback.threads.max-idle-time
pool.cell.limits.callback.threads.max-idle-time.unit
pool.cell.limits.callback.queue.max
pool.cell.wait-for-files
pool.plugins.meta
pool.plugins.sweeper
pool.mover.nfs.rpcsec_gss
pool.mover.dcap.port
pool.mover.ftp.mmap
pool.mover.ftp.read-ahead
pool.mover.ftp.allow-incoming-connections
pool.mover.xrootd.disk-threads
pool.mover.xrootd.socket-threads
pool.mover.xrootd.memory-per-connection
pool.mover.xrootd.memory
pool.mover.xrootd.frame-size
pool.mover.xrootd.plugins
pool.mover.xrootd.timeout.idle
pool.mover.xrootd.timeout.idle.unit
pool.mover.http.disk-threads
pool.mover.http.socket-threads
pool.mover.http.memory-per-connection
pool.mover.http.memory
pool.mover.http.chunk-size
pool.mover.http.timeout.idle
pool.mover.http.timeout.idle.unit
pool.mover.http.timeout.connect
pool.mover.http.timeout.connect.unit
pool.service.pnfsmanager
pool.service.pnfsmanager.timeout
pool.service.pnfsmanager.timeout.unit
pool.service.billing
pool.service.pool.timeout
pool.service.pool.timeout.unit
pool.service.poolmanager
pool.service.poolmanager.timeout
pool.service.poolmanager.timeout.unit
pool.service.pinmanager
pool.service.pinmanager.timeout
pool.service.pinmanager.timeout.unit
pool.destination.flush-notification
pool.destination.heartbeat
pool.destination.replicate
pool.destination.replicate.ip

Deprecated

name
path
poolIoQueue
lfs
maxDiskSpace
tags
checkRepository
replicateOnArrival
allowCleaningPreciousFiles
waitForFiles
metaDataRepository
sweeper
pool.dcap.port
gsiftpAllowMmap
gsiftpReadAhead
xrootdMoverDiskThreads
xrootdMoverSocketThreads
xrootdMoverMaxMemoryPerConnection
xrootdMoverMaxMemory
xrootdMoverMaxFrameSize
xrootdPlugins
httpMoverDiskThreads
httpMoverSocketThreads
httpMoverConnectionMaxMemory
httpMoverMaxMemory
httpMoverChunkSize
httpMoverClientIdleTimeout
httpMoverConnectTimeout
flushMessageTarget
poolupDestination

Removed

waitForRepositoryReady
removeUnexistingEntriesOnFlush
metaDataRepositoryImport
httpMoverMaxChunkSize

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5720/
gplazma: Comply with new configuration policy

The option to implicitly embed gPlazma as a 'module' has been
removed. Instead one can now add a local gplazma service to a
domain and choose not to export it by setting gplazma.cell.export
to false. This has the same effect.

The ability for plain ftp to not use gPlazma at all and use a
local kpwd file has been removed. The same effect can be achieved
by configuring a local gplazma service and let the ftp door use it.

New properties

gplazma.cell.name
gplazma.cell.limits.threads
gplazma.cell.export
gplazma.service.pnfsmanager

Obsolete properties

kpwdFile
useGPlazmaAuthorizationModule
useGPlazmaAuthorizationCell

Removed properties

gplazma.version

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5722/
broadcast: Comply with new configuration policy

Also updates space manager and replica manager enable switches. Note
that the allows values for the new *.enable.spacemanager and
*.enable.replica properties is limited to true and false.

New properties

broadcast.cell.name
broadcast.service.billing
broadcast.service.pinmanager
broadcast.service.poolmanager
broadcast.service.spacemanager
broadcast.service.replica
broadcast.enable.replica
broadcast.enable.spacemanager
dcache.enable.replica
dcache.enable.spacemanager
dcap.enable.spacemanager
ftp.enable.spacemanager
webdav.enable.spacemanager

Deprecated

replicaManager
srmSpaceManagerEnabled

Removed

dcap.service.spacemanager.enabled
ftp.service.spacemanager.enabled
webdav.service.spacemanager.enabled

Target: trunk
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5727/
Update SpreadAndWait to use CellStub

This will make it easier to inject timeout configuration into
SpreadAndWait.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5728/
topo,cns: Comply with new configuration policy

Updates all broker related properties to comply with the
new configuration policy.

New properties

cns.cell.name
cns.broker.scheme
cns.broker.amq.url
cns.broker.openmq.host
cns.broker.openmq.port
cns.broker.openmq.reconnect
cns.broker.openmq.reconnect
dcache.broker.scheme
dcache.broker.domain
dcache.broker.host
dcache.broker.port
dcache.broker.messaging.port
dcache.broker.client.port
dcache.broker.cells.config
dcache.broker.amq.host
dcache.broker.amq.port
dcache.broker.amq.ssl.port
dcache.broker.amq.url
dcache.broker.openmq.host
dcache.broker.openmq.port
dcache.broker.openmq.reconnect
dcache.broker.openmq.reconnect
topo.cell.name
topo.limits.refresh
topo.limits.refresh
topo.service.system.timeout
topo.service.system.timeout.unit

Deprecated

broker.scheme
broker.domain
broker.host
broker.port
broker.messaging.port
broker.client.port
broker.cells.config
broker.amq.host
broker.amq.port
broker.amq.ssl.port
broker.amq.url
broker.openmq.host
broker.openmq.port
broker.openmq.reconnect.interval
topoWaitTime
topoTimeout

Removed

useFilesystem
srmVacuum
srmVacuumPeriod
java
java_options
bufferSize
tcpBufferSize
maintenanceLibPath
maintananceLibAutogeneraetPaths
maintenanceLogoutTime
srmVersion
permissionHandler
PermissionHandlerDataSource
user
pidDir
logArea
logMode
classpath
librarypath
kerberosRealm
kerberosKdcList
authLoginConfig
messageBroker
serviceLocatorHost
serviceLocatorPort
amqHost
amqPort
amqSSLPort
amqUrl
ourHomeDir
portBase

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5729/
hoppingmanager: Comply with new configuration policy

The hopping service was renamed to hoppingmanager.

New properties

hoppingmanager.cell.name
hoppingmanager.cell.export
hoppingmanager.config

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5731/
hsqldb: Update to version 2.3.0

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5734/
info: fix broken import

A previous patch modified the package of CellInfoProvider.
Unfortunately the patch that moved info to use UniversalSpringCell was
not updated accordingly so, when committed, it broke master.

This patch fixes that problem.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5737/
Acked-by: Gerd Behrmann
(skel): time-unit corrections to billing migration script

The scripts contained in skel/share/migration which are intended to aid in migrating preexisting aggregated billing data or in aggregating the data from the main fine-grained tables have two errors in them.

1.  migrate_from_preexistent.sql does not account for the time-unit differences between the original dc_tm_daily tables, which store the data in seconds, and the new tables, which preserve the millisecond unit of the base table.

2.  migrate_from_messageinfo.sql neglected to exclude data from failed operations (errorcode = 0).

This patch repairs these two issues.

Testing done:

Before the patches, stored and displayed data did not correspond with previous installations, nor did the two scripts produce similar results.  After the patch, the results of both migrations are consonant with prior data.

Target: master
Request: 2.6
Request: 2.2
Patch: http://rb.dcache.org/r/5732/
Require-book: no
Require-notes: yes
Acked-by: Gerd

RELEASE NOTES:

Fixes two bugs in the migration scripts.  One which neglected to convert from seconds to milliseconds (migrate_from_preexistent.sql) for connection time values, the other which neglected to exclude data from failed operations (error code = 0: migrate_from_messageinfo.sql).

Note for those who already have applied the faulty script: the migration from the preexistent tables does not really affect anyone outside of Fermi.  The other migration error generates differences which are probably negligible.
cleaner: Add hsqldb support

Addresses the issue that the cleaner was not functional when used with
hsqldb, and thus didn't work in system-test. The trigger to populate
the trash table was missing.

This patch adds a trigger. HSQLDB unfortunately does not support data
manipulation statements in BEFORE DELETE triggers and thus cannot
populate the trash on delete from t_inodes (AFTER DELETE is too late
as the delete cascades to t_locationinfo before the trigger is run,
and thus the data is already gone). Instead the patch adds an
AFTER DELETE trigger on the t_locationinfo table itself. A trigger
condition is used to limit the trigger to deletion of an inode and
thus excludes regular deletes of replicas on pool and tapes.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5735/