xrootd,pool: Fix xrootd and http logging context

Several executors didn't use a CDC resetting executor, causing the CDC
to be from whatever transfer first created the thread.

Target: trunk
Request: 2.11
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7507/
pool: Fix cleaning of stale locations

dCache allows stale file locations. These are automatically cleaned when the
file is attempted to be accessed. The current implementation has a race if the
asynchronous notification to pnfs manager to remove the location is interleaved
with another thread creating that same file on the pool: In this case the
registration of the new location may succeed before the notification to clean
the location, thus leaving an orphaned file on the pool. The window for this is
very small and I therefore to not request the fix to be backported.

The patch fixes the problem by creating a temporary meta data entry for the
stale file. The entry is then immediately put into a REMOVED state, thus
causing the stale information to be cleaned. The temporary entry prevents other
threads from creating the file on the pool, thus preventing the race.

The fix comes at a cost of creating the meta data entry, but stale locations
should be sufficiently rare that the overhead is insignificant.

The patch fixes another problem affecting volatile pools: The notification
didn't include the flag to remove the name space entry once the last location
was removed.

The patch leaves another race open: The notification to clean the location sent
by the code removing the meta data entry is also asynchronous. Although the
code holds a lock on the meta data entry, the lock is eventually removed when
the entry is deleted. The pool could subsequently create a new entry or the
same file and send a registration message to the name space. If that message
were to be received before the notification to clean the previous location, we
would again have an orphaned file. For this race to be possible, messages
should be able to overtake each other. Without retries, messages between a pair
of pools should be ordered, which should make this race impossible. Fixes this
potential race would require changing the asynchronous notification to a
blocking request, and this would seriously affect sweeper performance and thus
uploads to full pools.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7502/
cells: Refine TTL discard message

A recent addition has added automatic discard of messages when their TTL is
exceeded. dCache logs a warning when this happens. Our admins have noted that
in particular for String messages, the warning is not particularly useful, as
only the message type is logged.

Since String messages are already a special case in dCache, I think it makes
sense to also log when they are discarded in a special way. This patch logs the
(possibly truncated) content of the string.

The patch also adds the message age and ttl to the log message.

Given how minor the change is and given that the log message was recently
introduced and may trigger support requests, I request that this change is
backported.

Target: trunk
Request: 2.11
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7503/
dcache-webadmin: revert rrd data source names

The following releases

2.10.8
2.9.12
2.8.16
2.7.21
2.6.36

introduced improvements to the pool queue plots code
which also unfortunately altered the data source names
(mainly in order to eliminate NaNs from the underlying
store). This results in an incompatibility with
existing .rrd files preventing plot generation (see
the exception trace below).

While the round-robin datastore is transient (the old
default was 15 days of queue information, now changed to 2)
and can be eliminated and then recreated afresh, this
patch restores the old names, so that upgrades from before
these releases to the current will avoid this issue.
It turns out that the storing of NaNs has no impact
on the principal issue the original patch was trying
to address (faulty stacked rendering of histogram bars).

Those who have already moved to the indicated releases
will see the stack trace again and will have
to remove the .rrd files.  Those upgrading from
a version prior to those indicated will experience
no such error.

Target: master
Patch: https://rb.dcache.org/r/7434
Acked-by: Tigran
Require-book: no
Require-notes: yes

RELEASE NOTES:  Those upgrading to

2.10.8
2.9.12
2.8.16
2.7.21
2.6.36

Will experience an error message regarding the pool-queue plots reporting "Unknown datasource name".
The problem can be fixed in the following ways:

1) No upgrade.  Do:

dcache stop httpdDomain
rm -f /var/lib/dcache/plots/*.rrd
dcache start httpDomain

2) Upgrade to this release.

If a fix such as 1) was applied, and then an upgrade is done, 1) will have to be reapplied.

Upgrading from a version preceding the listed versions to this one should experience no
such error.
pool: Avoid interrupting Berkeley DB in migration module server

Berkeley DB for Java doesn't like thread interrupts, and the pool tries to
never interrupt threads while they update the meta data database. We however
missed at least one case:

17 Nov 2014 11:11:10 (ore_ndgf_org_001) [ore_ndgf_org_002 PoolMigrationCopyReplica 00007D0EC04343684CF6A5F78B289361CD7B] Fault occured in repository: Internal repository error
diskCacheV111.util.CacheException: No transaction is active
        at org.dcache.pool.repository.meta.db.CacheRepositoryEntryImpl.setSticky(CacheRepositoryEntryImpl.java:225) ~[dcache-core-2.10.10.jar:2.10.10]
        at org.dcache.pool.repository.v5.CacheRepositoryV5.setSticky(CacheRepositoryV5.java:890) [dcache-core-2.10.10.jar:2.10.10]
        at org.dcache.pool.repository.v5.CacheRepositoryV5.setSticky(CacheRepositoryV5.java:600) [dcache-core-2.10.10.jar:2.10.10]
        at org.dcache.pool.migration.MigrationModuleServer$Request.run(MigrationModuleServer.java:301) [dcache-core-2.10.10.jar:2.10.10]
        at org.dcache.util.CDCExecutorServiceDecorator$1.run(CDCExecutorServiceDecorator.java:104) [dcache-core-2.10.10.jar:2.10.10]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_71]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_71]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_71]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) [na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]

This patch fixes this problem.

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7509/
Fix NPE in httpd service

The patch fixes the following NPE:

17 nov. 2014 11:52:48 (poolCollector) [] RuntimeException while executing runnable com.google.common.util.concurrent.Futures$5@2437e8e6 with executor com.google.common.util.concurrent.MoreExecuto
rs$SameThreadExecutorService@3f726b8a
java.lang.NullPointerException: null
        at java.util.TreeMap.rotateLeft(TreeMap.java:2220) ~[na:1.8.0_25]
        at java.util.TreeMap.fixAfterInsertion(TreeMap.java:2287) ~[na:1.8.0_25]
        at java.util.TreeMap.put(TreeMap.java:582) ~[na:1.8.0_25]
        at diskCacheV111.services.web.PoolCellQueryContainer.put(PoolCellQueryContainer.java:57) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at diskCacheV111.services.web.PoolInfoObserverV3$2.onSuccess(PoolInfoObserverV3.java:140) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at diskCacheV111.services.web.PoolInfoObserverV3$2.onSuccess(PoolInfoObserverV3.java:134) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at com.google.common.util.concurrent.Futures$5.run(Futures.java:1231) ~[guava-17.0.jar:na]
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-17.0.jar:na]
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) [guava-17.0.jar:na]
        at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) [guava-17.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185) [guava-17.0.jar:na]
        at org.dcache.cells.CellStub$CallbackFuture.set(CellStub.java:594) [dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at org.dcache.cells.CellStub$CallbackFuture.answerArrived(CellStub.java:616) [dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$CallbackTask.innerRun(CellNucleus.java:1050) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:1005) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-17.0.jar:na]
        at dmg.cells.nucleus.CellNucleus.addToEventQueue(CellNucleus.java:813) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellGlue.sendMessage(CellGlue.java:557) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus.sendMessage(CellNucleus.java:371) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellAdapter.messageArrived(CellAdapter.java:831) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$DeliverMessageTask.innerRun(CellNucleus.java:1095) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:1005) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_25]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_25]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]

The patch also changes a few return types from Set to List, as the code
is more concerned about the ordering that uniqueness (and the items in
the list are still unique). Changed a TreeMap to a more efficient
HashMap.

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7510/
Upgrade to latest version of third party libs

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7491/
Fix NPE in cell initialization

Cells block delivery of messages until they have been started. Prior to being
started the message is returned to the sender. The code however suffers from a
race resulting in the following exception:

17 nov. 2014 15:01:17 (SrmSpaceManager) [] Uncaught exception in thread SrmSpaceManager-0
java.lang.NullPointerException: null
        at dmg.cells.nucleus.CellAdapter.getCellName(CellAdapter.java:325) ~[cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellAdapter.messageArrived(CellAdapter.java:784) ~[cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$DeliverMessageTask.innerRun(CellNucleus.java:1095) ~[cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:1005) ~[cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_25]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_25]
        at dmg.cells.nucleus.CellNucleus$3.run(CellNucleus.java:696) [cells-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]

The cause is the classic problem of leaking a reference to an object in its
constructor. In this case CellNucleus registers itself with CellGlue inside
its constructor, meaning that others may deliver messages to it even before
the final nucleus member in CellAdapter has been initialized.

This minimal patch moves the CellGlue registration into a new CellNucleus#start
method. The method is called right after the constructor returns. The patch
is kept small to allow it to be backported, however a subsequent patch for trunk
will change the code further to not start the CellNucleus until the cell has
been started, thus solving the problem of early message delivery.

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Request: 2.8
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7511/
cells: Reduce coupling between CellAdapter and CommandInterpreter

Uses delegation rather inheritance.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7512/
cells: Get rid of thread local holding the current message

The same functionality can be achieved by returning a Reply from message
handlers. The Reply functionality had to be adjusted slightly to cover an
interesting usecase in multicaster.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7513/
cells: Drop support for auto starting CellAdapter

Starting CellAdapter from its constructor suffers from the mistake of leaking a
reference to CellAdapter (through CellNucleus) before it is constructed.  This
patch removes support for starting the CellAdapter in its constructor - most
subclasses didn't make use of this feature anyway.

In the current patch subclasses call start instead, which obviously suffers from
the same problems as if CellAdapter had called it. Subsequent patches will
address that problem.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7514/
pool: Improve xrootd error message on missing UUID

The existing error when the client doesn't present a valid UUID is misleading
as it usually states that the open request is not supported when it reallity
the request is not authorized due to the lack of the token.

This patch improves the error message and return code.

Target: trunk
Request: 2.10
Request: 2.11
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7515/
dcap: refactor PnfsSessionHandler to unify permission check and url handling

fixes issue with permission check with prestage over mounted NFS

Acked-by: Gerd Behrmann
Target: master, 2.11, 2.10
Ticket: #8518, #8532
Require-book: no
Require-notes: no
pool: Restore flush and stage stats in info

Resolves a regression in which statistics about the number of HSM
requests and HSM timeouts is no longer provided in the info output.

Also fixes a minor bug in how active HSM jobs are counted: A cancelled
task is still active until the driver has ended the job.

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7520/
xrootd: Propagate mover errors to dCache

The patch ensures that failures to close a file or unexpected exceptions
are propagated to the pool and logged in billing. Before this patch, xrootd
transfers were always registered as succesful in billing.

The precise error depends on whether the file was opened with the POSC flag
(Persist On Successful Close) or not. If it was, the file is known to be corrupt
from the point of the view of the client and FileCorruptedCacheException is
reported. If not, a generic error is reported.

The current implementation does not actually delete the file on POSC failures.
This is left for a future patch.

The XrootdProtocolInfo is extended with a flag to inject POSC semantics
independent of whether the client specified this flag. This may be used
in doors to enable a POSC-on-default feature mimicking a similar feature
in the SLAC xrootd implementation. This feature is however not supported
by the door yet.

The patch adds logic to disconnect the client when a server side failure
is triggered. Previously this failure was only logged and the error was
ignored.

Backport to 2.10 is requested because the current behaviour fails to log
errors in billing.

Target: trunk
Request: 2.11
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Karsten Schwank <karsten.schwank@desy.d>
Patch: https://rb.dcache.org/r/7499/
update schema version from 3.0 to 3.1
datanucleus: use persistence units

Based on the 2.11 release notes, a patch to embed the alarms service in dCacheDomain of system-test
was committed. The release notes claimed that there no longer was a requirement to run it in a separate domain.

Embedding it however broke the tests because the EnhancementHelper.registerClass() call for the XML database (alarms)
rejects other classes using the datastore identity, even though not destined for storage in that database.

The issue here, to quote the author of the code, is as follows:

"JDO comes with an auto-notify feature that will inform ALL interested parties when an enhanced class is loaded (so they can load up metadata for it). Since you have multiple PMF's then the MetaDataManager of each PMF are informed and they try to load it (since it is simply an enhanced class - can be used with any persistence environment that supports JDO). Since one PMF is for XML and it only accepts particular metadata then it rejects that class (for that MetaDataManager) by exception. DN 4.1 code simply logs a warning (instead of exception) that the class is not available for that PMF (so the user can fix their metadata if they want it to be available there)."

For fuller discussion, see http://www.datanucleus.org/servlet/forum/viewthread_thread,7815.

While an update to 4.1 will remove the failure, the better solution is the one we have adopted:
the suggested partitioning of all enhanced classes by persistence units (already used for the pinmanager).

Also makes the 'off' url for alarms a non-empty string; otherwise an NPE occurs during injection.

Target: trunk
Patch: https://rb.dcache.org/r/7517
Request: 2.11
Require-book: no
Require-notes: yes (in case 2.11 has been used already)
Acked-by: Gerd

RELEASE NOTES:  Fixes error when alarms XML database and TransferManager share the same domain.
spacemanager: Fix various error recovery scenarios

The patch fixes several related issues in space manager:

- The code used to delete a file reservation when the mover reported
  a failure. However whether such a file is actually deleted depends
  on what the door does. If the door deletes it, the space manager
  will be notified and the file can be deleted at that point.

  The special case that exists is if the mover failure was caused by the
  pool being unable to register the location in the name space. When
  such a file is deleted it would only get a name space delete notification,
  and no pool remove file notification. To account for this case, the
  pool leaves a file for which the mover reported an error in the
  transferring state. If the name space entry is deleted, such an entry
  is removed. If the file is not deleted, the other fixes of this
  patch will address it eventually.

- The code used to assume that a file that was in transferring
  would either get a DoorTransferFinished notification or be deleted.
  There is however no guarantee that this happens for non-srm uploads,
  such as when the door doesn't delete on failed upload. The client
  may even think the transfer is successfull if no control channel is
  used (eg http and xrootd). Such a file is perfectly readable, but
  stayed in transferring in space manager.

  This patch addresses this problem by extending the background check
  of files that have stayed in transferring for more than 24 hours:
  rather than just deleting reservations for deleted files, the code
  now also detects flushed and completed files and updates the reservation
  accordingly.

- The patch changes how we react to notifications that a name space entry
  is deleted: Before, we deleted the reservation if the file was marked
  flushed. For other states we assumed we would eventually get a pool
  remove file notification: However if the pool failed to register the
  location, we will never get such a notification. This scenario is
  quite likely if the name space entry is deleted during upload.

  The patch addresses this problem by deleting a transferring reservation on
  the name space delete notification. The downside is that we have some
  space allocated for the file on the pool that isn't explicitly linked
  to a reservation in space manager, but since the name space entry got
  deleted, the file on the pool will soon get deleted too.

- The patch also refines how we react to notifications for flushed files: if the
  reservation was in transferring, it is now considered complete.  Before the
  flush notification was simply ignored. This could both happen for lost
  DoorTransferFinished notification, of if the flush notification managed to
  overtake the DoorTransferFinished notification (unlikely).

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: https://rb.dcache.org/r/7519/
pnfsmanager: Propagate cleaner errors to RemoteNameSpaceProvider

The NameSpaceProvider interface is not asynchronous and cleaning
a pool location through this interface should propagate errors
back to the client.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7501/
info: Reduce delay between messages

The info service takes a long time to populate its state after a restart, in
particular on production scale systems.

This patch reduces the delay between queries. The overall frequency is not
increased, but the delay between two queries is reduced to 100 ms (from 2, 5,
or 10 seconds, depending on the query). As these queries are relatively
light-weight, 100 ms should be plenty to keep the message passing overhead
down.

I request backport to 2.10 as we always have issues with our Nagios probes
after a restart.

Target: trunk
Request: 2.11
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7535/