pool: Restore flush and stage stats in info

Resolves a regression in which statistics about the number of HSM
requests and HSM timeouts is no longer provided in the info output.

Also fixes a minor bug in how active HSM jobs are counted: A cancelled
task is still active until the driver has ended the job.

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7520/
xrootd: Propagate mover errors to dCache

The patch ensures that failures to close a file or unexpected exceptions
are propagated to the pool and logged in billing. Before this patch, xrootd
transfers were always registered as succesful in billing.

The precise error depends on whether the file was opened with the POSC flag
(Persist On Successful Close) or not. If it was, the file is known to be corrupt
from the point of the view of the client and FileCorruptedCacheException is
reported. If not, a generic error is reported.

The current implementation does not actually delete the file on POSC failures.
This is left for a future patch.

The XrootdProtocolInfo is extended with a flag to inject POSC semantics
independent of whether the client specified this flag. This may be used
in doors to enable a POSC-on-default feature mimicking a similar feature
in the SLAC xrootd implementation. This feature is however not supported
by the door yet.

The patch adds logic to disconnect the client when a server side failure
is triggered. Previously this failure was only logged and the error was
ignored.

Backport to 2.10 is requested because the current behaviour fails to log
errors in billing.

Target: trunk
Request: 2.11
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Karsten Schwank <karsten.schwank@desy.d>
Patch: https://rb.dcache.org/r/7499/
update schema version from 3.0 to 3.1
datanucleus: use persistence units

Based on the 2.11 release notes, a patch to embed the alarms service in dCacheDomain of system-test
was committed. The release notes claimed that there no longer was a requirement to run it in a separate domain.

Embedding it however broke the tests because the EnhancementHelper.registerClass() call for the XML database (alarms)
rejects other classes using the datastore identity, even though not destined for storage in that database.

The issue here, to quote the author of the code, is as follows:

"JDO comes with an auto-notify feature that will inform ALL interested parties when an enhanced class is loaded (so they can load up metadata for it). Since you have multiple PMF's then the MetaDataManager of each PMF are informed and they try to load it (since it is simply an enhanced class - can be used with any persistence environment that supports JDO). Since one PMF is for XML and it only accepts particular metadata then it rejects that class (for that MetaDataManager) by exception. DN 4.1 code simply logs a warning (instead of exception) that the class is not available for that PMF (so the user can fix their metadata if they want it to be available there)."

For fuller discussion, see http://www.datanucleus.org/servlet/forum/viewthread_thread,7815.

While an update to 4.1 will remove the failure, the better solution is the one we have adopted:
the suggested partitioning of all enhanced classes by persistence units (already used for the pinmanager).

Also makes the 'off' url for alarms a non-empty string; otherwise an NPE occurs during injection.

Target: trunk
Patch: https://rb.dcache.org/r/7517
Request: 2.11
Require-book: no
Require-notes: yes (in case 2.11 has been used already)
Acked-by: Gerd

RELEASE NOTES:  Fixes error when alarms XML database and TransferManager share the same domain.
spacemanager: Fix various error recovery scenarios

The patch fixes several related issues in space manager:

- The code used to delete a file reservation when the mover reported
  a failure. However whether such a file is actually deleted depends
  on what the door does. If the door deletes it, the space manager
  will be notified and the file can be deleted at that point.

  The special case that exists is if the mover failure was caused by the
  pool being unable to register the location in the name space. When
  such a file is deleted it would only get a name space delete notification,
  and no pool remove file notification. To account for this case, the
  pool leaves a file for which the mover reported an error in the
  transferring state. If the name space entry is deleted, such an entry
  is removed. If the file is not deleted, the other fixes of this
  patch will address it eventually.

- The code used to assume that a file that was in transferring
  would either get a DoorTransferFinished notification or be deleted.
  There is however no guarantee that this happens for non-srm uploads,
  such as when the door doesn't delete on failed upload. The client
  may even think the transfer is successfull if no control channel is
  used (eg http and xrootd). Such a file is perfectly readable, but
  stayed in transferring in space manager.

  This patch addresses this problem by extending the background check
  of files that have stayed in transferring for more than 24 hours:
  rather than just deleting reservations for deleted files, the code
  now also detects flushed and completed files and updates the reservation
  accordingly.

- The patch changes how we react to notifications that a name space entry
  is deleted: Before, we deleted the reservation if the file was marked
  flushed. For other states we assumed we would eventually get a pool
  remove file notification: However if the pool failed to register the
  location, we will never get such a notification. This scenario is
  quite likely if the name space entry is deleted during upload.

  The patch addresses this problem by deleting a transferring reservation on
  the name space delete notification. The downside is that we have some
  space allocated for the file on the pool that isn't explicitly linked
  to a reservation in space manager, but since the name space entry got
  deleted, the file on the pool will soon get deleted too.

- The patch also refines how we react to notifications for flushed files: if the
  reservation was in transferring, it is now considered complete.  Before the
  flush notification was simply ignored. This could both happen for lost
  DoorTransferFinished notification, of if the flush notification managed to
  overtake the DoorTransferFinished notification (unlikely).

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: https://rb.dcache.org/r/7519/
pnfsmanager: Propagate cleaner errors to RemoteNameSpaceProvider

The NameSpaceProvider interface is not asynchronous and cleaning
a pool location through this interface should propagate errors
back to the client.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7501/
info: Reduce delay between messages

The info service takes a long time to populate its state after a restart, in
particular on production scale systems.

This patch reduces the delay between queries. The overall frequency is not
increased, but the delay between two queries is reduced to 100 ms (from 2, 5,
or 10 seconds, depending on the query). As these queries are relatively
light-weight, 100 ms should be plenty to keep the message passing overhead
down.

I request backport to 2.10 as we always have issues with our Nagios probes
after a restart.

Target: trunk
Request: 2.11
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7535/
httpd: Drop info pages from webadmin

The info pages exist because originally webadmin was developed as a complete
replacement for httpd, meaning everything httpd could do, webadmin had to do
too, even if that functionallity wasn't really a part of the admin tool.

The info for info-provider is just as easy to expose through the classic
info link - info-provider was actually never updated to use webadmin.

Even the UI in webadmin for accessing the info xml is inferior to just
load the old page in the browser.

This patch removes both the Info and InfoXml pages from webadmin.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7536/
pool: Fix ISE when flushing deleted file

Prior to flushing a file, the pool checks if the name space entry still exists. If
not, it deletes the replica rather than flushing it to tape. Release 2.10.7 introduced
a regression that causes this code path to throw an ISE:

11:41:01 [pool-28-thread-1] [1f45dec7-f415-4dea-80cb-57c825b51486] Flush of 000084C99C4AC70E4CC38279851A856FABB8 failed with: CacheException(rc=10001;msg=No such file or directory: 000084C99C4AC70E4CC38279851A856FABB8).
11:41:01 [pool-12-thread-2] [1f45dec7-f415-4dea-80cb-57c825b51486] remove entry for: 000084C99C4AC70E4CC38279851A856FABB8
11:41:02 [pool-10-thread-1] [door:GFTP-Gerds-MacBook-Pro-5-AAUHz0KyLFg GFTP-Gerds-MacBook-Pro-5-AAUHz0KyLFg PoolAcceptFile 000084C99C4AC70E4CC38279851A856FABB8] Unexpected failure during state change notificationjava.lang.IllegalStateException: Handle is closed
    at org.dcache.pool.repository.v5.ReadHandleImpl.close(ReadHandleImpl.java:48) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at org.dcache.pool.nearline.NearlineStorageHandler$FlushRequestImpl.failed(NearlineStorageHandler.java:628) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at org.dcache.pool.nearline.AbstractBlockingNearlineStorage$Task.cancel(AbstractBlockingNearlineStorage.java:200) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at org.dcache.pool.nearline.AbstractBlockingNearlineStorage.cancel(AbstractBlockingNearlineStorage.java:71) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at org.dcache.pool.nearline.NearlineStorageHandler$AbstractRequest.cancel(NearlineStorageHandler.java:365) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at org.dcache.pool.nearline.NearlineStorageHandler$AbstractRequestContainer.cancel(NearlineStorageHandler.java:446) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at org.dcache.pool.nearline.NearlineStorageHandler.stateChanged(NearlineStorageHandler.java:279) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at org.dcache.pool.repository.v5.StateChangeListeners$1.run(StateChangeListeners.java:68) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_25]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_25]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]

11:41:02 [pool-12-thread-2] [1f45dec7-f415-4dea-80cb-57c825b51486] File not found in name space; removed 000084C99C4AC70E4CC38279851A856FABB8.

This is caused by the pre-flush check deleting the replica followed by failing the flush
request. Deleting the replica subsequently causes a notification to be sent to the HSM
subsystem, which causes the same flush request to be cancelled. The script driver reports
a second failure, thus causing the replica handler to be closed twice - hence the ISE.

The patch fixes the regression.

The log message is also cleaned up to better state what happened:

12:07:39 [pool-28-thread-1] [84967682-8766-4e66-80c3-3199f047b6b2] Flush of 00005466F5DC5AEA4263B3FA8E6EECCCF045 failed with: CacheException(rc=10001;msg=File not found in name space during pre-flush check.).
12:07:39 [pool-12-thread-2] [84967682-8766-4e66-80c3-3199f047b6b2] remove entry for: 00005466F5DC5AEA4263B3FA8E6EECCCF045

Target: trunk
Request: 2.11
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7506/
pnfsmanager: update atime on get attribute if desired

PnfsManager still controls the time interval during which updates are ignored.

Acked-by: Gerd Behrmann
Target: master
Require-book: yes
Require-notes: yes
chimera: make Stat aware of defined attributes

update FsSqlDriver to generate SQL statement for required
attributes only.

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
chimera-provider: do not get file attributes prior update

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: yes