dcache-webadmin: eliminate clojure dependency

    Get rid of stack trace reporting error.  deps.js is
    unused.

    Target: master
    Request: 2.11
    Request: 2.10
    Request: 2.9
    Request: 2.8
    Request: 2.7
    Requires-book: no
    Requires-notes: yes
    Acked-by:  Paul
srm-server: add servlet webapp-2.3.dtd file

The SRM server makes use of an XML file that follows the Sevlet 2.3
deployment descriptor.  When receiving the first SRM operation, Jetty
will parse the deployment descriptor, which requires the corresponding
DTD.

Previously (with Jetty 8) the DTD came as part of the servlet-api-2.x
jar file.  Jetty 9 depends on servlet-api 3.1, which does not contain
the deployment descriptor DTD:

    https://java.net/jira/browse/SERVLET_SPEC-115

Therefore, when it receives the first SRM operation, dCache will
download the DTD from java.sun.com.  For most sites, this is merely
undesirable, but if the dCache instance is not connected to the
Internet then this is problematic as a stack-trace is logged with the
root cause of:

    java.net.UnknownHostException: java.sun.com

and the SRM does not work.

This patch works around the problem by embedding the DTD in the
srm-server jar file.

Target: master
Patch: https://rb.dcache.org/r/7459
Acked-by: Gerd Behrmann
Request: 2.11
Require-notes: no
Require-book: no
dcache-webadmin: change output field of cell admin page to monospace font

Many of the dCache services format the output into columns and that only works with a mono space font.

Target: master
Request: 2.11
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Gerd
Acked-by: Paul
Require-notes: no
Require-book: no
webadmin: remove redundant head element in alarms panel html

Not necessary, should have been removed numerous versions ago.

Target: master
Request: 2.11
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Acked-by: Paul
Require-book: no
Require-notes: no
webadmin: Fix ClassCastException in cell admin

Fixes

07 nov. 2014 23:32:44 (httpd) [] RuntimeException while executing runnable org.dcache.cells.CellStub$1@135dac1f with executor com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService@12afd1d3
java.lang.ClassCastException: java.lang.String cannot be cast to diskCacheV111.vehicles.Message
        at org.dcache.cells.CellStub$1.run(CellStub.java:503) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-17.0.jar:na]
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) [guava-17.0.jar:na]
        at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) [guava-17.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185) [guava-17.0.jar:na]
        at org.dcache.cells.CellStub$CallbackFuture.set(CellStub.java:594) [dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at org.dcache.cells.CellStub$CallbackFuture.answerArrived(CellStub.java:616) [dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$CallbackTask.innerRun(CellNucleus.java:1050) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:1005) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-17.0.jar:na]
        at dmg.cells.nucleus.CellNucleus.addToEventQueue(CellNucleus.java:813) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellGlue.sendMessage(CellGlue.java:557) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus.sendMessage(CellNucleus.java:371) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellAdapter.messageArrived(CellAdapter.java:831) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$DeliverMessageTask.innerRun(CellNucleus.java:1095) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:1005) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_20]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_20]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_20]

The problem was hidden by the use of a raw type and thus the compiler didn't
reject the code.

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7478/
Limit MessageCallback to Message

Could have prevented https://rb.dcache.org/r/7478/

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7479/
doors: Add pnfs ID to billing remove entries

Doors reform billing when files are deleted, but many only included the path.
Since the path is ambiguous, this patch adds PnfsId to those notifications. In
a few cases we even have the file size available and add that too. The client
IP address is also added.

Please advice whether this should be backported to 2.10.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7476/
httpd: Integrate handlers into spring wiring

The httpd service used a partial Spring wiring model where the bulk of the
service was still manually and dynamically instantiated. To integrate with cell
messaging, info extraction, admin shell, etc, the instantiation code tried to
glue the two models together, with lots of indirection and coupling. Worse, it
didn't actually work for admin shell integration. Thus the 'set update' command
was not actually exposed in the admin shell.

This patch reworks the entire service. The handlers are passed to Spring for
initialization, allowing them to hook into all the cell services without extra
glue code.

Life cycle management is moved into the HandleDelegator, just like for all
other Jetty handler collections.

Webadmin WAR files are no longer unpacked, resolving the issue with stale files
on upgrade. The httpd.container.webapps.tmp-dir property is now obsolete.

The arguments of the 'set webapp' command in httpd has changed: It now takes
the path to the WAR file, not the path to the directory containing the WAR
file.  Sites that have customized their httpd alias layout will likely have to
update their configuraiton.

The output of the info command has changed. The list of defined aliases is no
longer shown. The same information can be obtained with the existing 'ls alias'
command.

The changes are unfortunately so intrusive that backporting these fixes to
stable branches is unrealistic.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7436/
gplazma2-htpasswd: Avoid deprecated Guava classes

The plugin was incompatible with Guava 18 due to use of deprecated
classes that are removed in the new release.

This patch ports the plugin to Java 8.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7465/
jetty: update webdefault.xml to adhere to new specification

The current xml stanza concerning security constraints does
not observe the new requirement that all HTTP methods be
specified.

This patch updates the xml namespace to point to the newer xsd,
and adds the missing omission stanza.

Testing:

Before application of patch, the following type of warning appears in the log:

05 Nov 2014 11:45:13 (httpd) [] ServletContext@o.e.j.w.WebAppContext@1db9683d{/webadmin,file:/var/lib/dcache/httpd/webadmin/webapp/,STARTING}{/webadmin.war} has uncovered http methods for path: /

The patch silences this warning.

Target: master
Patch: https://rb.dcache.org/r/7488
Request: 2.11
Acked-by: Paul
Require-notes: no
Require-book: no
Update to HikariCP 2.2.4

Required getting rid of DriverManagerDataSource as it doesn't support
setLoginTimeout.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7477/
xrootd,pool: Fix xrootd and http logging context

Several executors didn't use a CDC resetting executor, causing the CDC
to be from whatever transfer first created the thread.

Target: trunk
Request: 2.11
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7507/
pool: Fix cleaning of stale locations

dCache allows stale file locations. These are automatically cleaned when the
file is attempted to be accessed. The current implementation has a race if the
asynchronous notification to pnfs manager to remove the location is interleaved
with another thread creating that same file on the pool: In this case the
registration of the new location may succeed before the notification to clean
the location, thus leaving an orphaned file on the pool. The window for this is
very small and I therefore to not request the fix to be backported.

The patch fixes the problem by creating a temporary meta data entry for the
stale file. The entry is then immediately put into a REMOVED state, thus
causing the stale information to be cleaned. The temporary entry prevents other
threads from creating the file on the pool, thus preventing the race.

The fix comes at a cost of creating the meta data entry, but stale locations
should be sufficiently rare that the overhead is insignificant.

The patch fixes another problem affecting volatile pools: The notification
didn't include the flag to remove the name space entry once the last location
was removed.

The patch leaves another race open: The notification to clean the location sent
by the code removing the meta data entry is also asynchronous. Although the
code holds a lock on the meta data entry, the lock is eventually removed when
the entry is deleted. The pool could subsequently create a new entry or the
same file and send a registration message to the name space. If that message
were to be received before the notification to clean the previous location, we
would again have an orphaned file. For this race to be possible, messages
should be able to overtake each other. Without retries, messages between a pair
of pools should be ordered, which should make this race impossible. Fixes this
potential race would require changing the asynchronous notification to a
blocking request, and this would seriously affect sweeper performance and thus
uploads to full pools.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7502/
cells: Refine TTL discard message

A recent addition has added automatic discard of messages when their TTL is
exceeded. dCache logs a warning when this happens. Our admins have noted that
in particular for String messages, the warning is not particularly useful, as
only the message type is logged.

Since String messages are already a special case in dCache, I think it makes
sense to also log when they are discarded in a special way. This patch logs the
(possibly truncated) content of the string.

The patch also adds the message age and ttl to the log message.

Given how minor the change is and given that the log message was recently
introduced and may trigger support requests, I request that this change is
backported.

Target: trunk
Request: 2.11
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7503/
dcache-webadmin: revert rrd data source names

The following releases

2.10.8
2.9.12
2.8.16
2.7.21
2.6.36

introduced improvements to the pool queue plots code
which also unfortunately altered the data source names
(mainly in order to eliminate NaNs from the underlying
store). This results in an incompatibility with
existing .rrd files preventing plot generation (see
the exception trace below).

While the round-robin datastore is transient (the old
default was 15 days of queue information, now changed to 2)
and can be eliminated and then recreated afresh, this
patch restores the old names, so that upgrades from before
these releases to the current will avoid this issue.
It turns out that the storing of NaNs has no impact
on the principal issue the original patch was trying
to address (faulty stacked rendering of histogram bars).

Those who have already moved to the indicated releases
will see the stack trace again and will have
to remove the .rrd files.  Those upgrading from
a version prior to those indicated will experience
no such error.

Target: master
Patch: https://rb.dcache.org/r/7434
Acked-by: Tigran
Require-book: no
Require-notes: yes

RELEASE NOTES:  Those upgrading to

2.10.8
2.9.12
2.8.16
2.7.21
2.6.36

Will experience an error message regarding the pool-queue plots reporting "Unknown datasource name".
The problem can be fixed in the following ways:

1) No upgrade.  Do:

dcache stop httpdDomain
rm -f /var/lib/dcache/plots/*.rrd
dcache start httpDomain

2) Upgrade to this release.

If a fix such as 1) was applied, and then an upgrade is done, 1) will have to be reapplied.

Upgrading from a version preceding the listed versions to this one should experience no
such error.
pool: Avoid interrupting Berkeley DB in migration module server

Berkeley DB for Java doesn't like thread interrupts, and the pool tries to
never interrupt threads while they update the meta data database. We however
missed at least one case:

17 Nov 2014 11:11:10 (ore_ndgf_org_001) [ore_ndgf_org_002 PoolMigrationCopyReplica 00007D0EC04343684CF6A5F78B289361CD7B] Fault occured in repository: Internal repository error
diskCacheV111.util.CacheException: No transaction is active
        at org.dcache.pool.repository.meta.db.CacheRepositoryEntryImpl.setSticky(CacheRepositoryEntryImpl.java:225) ~[dcache-core-2.10.10.jar:2.10.10]
        at org.dcache.pool.repository.v5.CacheRepositoryV5.setSticky(CacheRepositoryV5.java:890) [dcache-core-2.10.10.jar:2.10.10]
        at org.dcache.pool.repository.v5.CacheRepositoryV5.setSticky(CacheRepositoryV5.java:600) [dcache-core-2.10.10.jar:2.10.10]
        at org.dcache.pool.migration.MigrationModuleServer$Request.run(MigrationModuleServer.java:301) [dcache-core-2.10.10.jar:2.10.10]
        at org.dcache.util.CDCExecutorServiceDecorator$1.run(CDCExecutorServiceDecorator.java:104) [dcache-core-2.10.10.jar:2.10.10]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_71]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_71]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_71]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) [na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]

This patch fixes this problem.

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7509/
Fix NPE in httpd service

The patch fixes the following NPE:

17 nov. 2014 11:52:48 (poolCollector) [] RuntimeException while executing runnable com.google.common.util.concurrent.Futures$5@2437e8e6 with executor com.google.common.util.concurrent.MoreExecuto
rs$SameThreadExecutorService@3f726b8a
java.lang.NullPointerException: null
        at java.util.TreeMap.rotateLeft(TreeMap.java:2220) ~[na:1.8.0_25]
        at java.util.TreeMap.fixAfterInsertion(TreeMap.java:2287) ~[na:1.8.0_25]
        at java.util.TreeMap.put(TreeMap.java:582) ~[na:1.8.0_25]
        at diskCacheV111.services.web.PoolCellQueryContainer.put(PoolCellQueryContainer.java:57) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at diskCacheV111.services.web.PoolInfoObserverV3$2.onSuccess(PoolInfoObserverV3.java:140) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at diskCacheV111.services.web.PoolInfoObserverV3$2.onSuccess(PoolInfoObserverV3.java:134) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at com.google.common.util.concurrent.Futures$5.run(Futures.java:1231) ~[guava-17.0.jar:na]
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-17.0.jar:na]
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) [guava-17.0.jar:na]
        at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) [guava-17.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185) [guava-17.0.jar:na]
        at org.dcache.cells.CellStub$CallbackFuture.set(CellStub.java:594) [dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at org.dcache.cells.CellStub$CallbackFuture.answerArrived(CellStub.java:616) [dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$CallbackTask.innerRun(CellNucleus.java:1050) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:1005) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-17.0.jar:na]
        at dmg.cells.nucleus.CellNucleus.addToEventQueue(CellNucleus.java:813) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellGlue.sendMessage(CellGlue.java:557) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus.sendMessage(CellNucleus.java:371) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellAdapter.messageArrived(CellAdapter.java:831) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$DeliverMessageTask.innerRun(CellNucleus.java:1095) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:1005) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_25]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_25]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]

The patch also changes a few return types from Set to List, as the code
is more concerned about the ordering that uniqueness (and the items in
the list are still unique). Changed a TreeMap to a more efficient
HashMap.

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7510/
Upgrade to latest version of third party libs

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7491/
Fix NPE in cell initialization

Cells block delivery of messages until they have been started. Prior to being
started the message is returned to the sender. The code however suffers from a
race resulting in the following exception:

17 nov. 2014 15:01:17 (SrmSpaceManager) [] Uncaught exception in thread SrmSpaceManager-0
java.lang.NullPointerException: null
        at dmg.cells.nucleus.CellAdapter.getCellName(CellAdapter.java:325) ~[cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellAdapter.messageArrived(CellAdapter.java:784) ~[cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$DeliverMessageTask.innerRun(CellNucleus.java:1095) ~[cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:1005) ~[cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_25]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_25]
        at dmg.cells.nucleus.CellNucleus$3.run(CellNucleus.java:696) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]

The cause is the classic problem of leaking a reference to an object in its
constructor. In this case CellNucleus registers itself with CellGlue inside
its constructor, meaning that others may deliver messages to it even before
the final nucleus member in CellAdapter has been initialized.

This minimal patch moves the CellGlue registration into a new CellNucleus#start
method. The method is called right after the constructor returns. The patch
is kept small to allow it to be backported, however a subsequent patch for trunk
will change the code further to not start the CellNucleus until the cell has
been started, thus solving the problem of early message delivery.

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Request: 2.8
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7511/
cells: Reduce coupling between CellAdapter and CommandInterpreter

Uses delegation rather inheritance.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7512/