dCache/dcache — pull request #1160 by gbehrmann (2014-12-06 13:07:10+00:00)
Title: pool: Fix shutdown of nearline storage subsystem

Ensures proper shutdown of the nearline storage subsystem. There is an
existing shutdown call to every nearline storage instance initiated
by the HsmSet class, however this is

- asynchronous

- has no dependency to the repository

This means there is no guarantee that requests get cancelled before
the repository is closed. This leads to errors like these when shutting
down a pool with acive stage requests:

04 dec. 2014 00:02:32 (pool_read) [admin d7ab5ce2-4ff7-4cfa-883e-3bfb41a5c313] Uncaught exception in thread pool-48-thread-1
java.lang.IllegalStateException: Can't open a cursor Database was closed.
at com.sleepycat.je.Database.checkOpen(Database.java:2200) ~[je-6.1.5.jar:6.1.5]
at com.sleepycat.je.Database.openCursor(Database.java:824) ~[je-6.1.5.jar:6.1.5]
at com.sleepycat.collections.CurrentTransaction.openCursor(CurrentTransaction.java:415) ~[je-6.1.5.jar:6.1.5]
at com.sleepycat.collections.MyRangeCursor.openCursor(MyRangeCursor.java:53) ~[je-6.1.5.jar:6.1.5]
at com.sleepycat.collections.MyRangeCursor.(MyRangeCursor.java:29) ~[je-6.1.5.jar:6.1.5]
at com.sleepycat.collections.DataCursor.init(DataCursor.java:170) ~[je-6.1.5.jar:6.1.5]
at com.sleepycat.collections.DataCursor.(DataCursor.java:58) ~[je-6.1.5.jar:6.1.5]
at com.sleepycat.collections.StoredContainer.putKeyValue(StoredContainer.java:319) ~[je-6.1.5.jar:6.1.5]
at com.sleepycat.collections.StoredMap.put(StoredMap.java:279) ~[je-6.1.5.jar:6.1.5]
at org.dcache.pool.repository.meta.db.CacheRepositoryEntryImpl.storeStateIfDirty(CacheRepositoryEntryImpl.java:270) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
at org.dcache.pool.repository.meta.db.CacheRepositoryEntryImpl.setState(CacheRepositoryEntryImpl.java:190) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
at org.dcache.pool.repository.v5.CacheRepositoryV5.setState(CacheRepositoryV5.java:853) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
at org.dcache.pool.repository.v5.WriteHandleImpl.fail(WriteHandleImpl.java:406) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
at org.dcache.pool.repository.v5.WriteHandleImpl.close(WriteHandleImpl.java:429) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
at org.dcache.pool.nearline.NearlineStorageHandler$StageRequestImpl.done(NearlineStorageHandler.java:955) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
at org.dcache.pool.nearline.NearlineStorageHandler$StageRequestImpl.failed(NearlineStorageHandler.java:918) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
at org.dcache.pool.nearline.AbstractBlockingNearlineStorage$Task.processRequest(AbstractBlockingNearlineStorage.java:268) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
at org.dcache.pool.nearline.AbstractBlockingNearlineStorage$Task.run(AbstractBlockingNearlineStorage.java:242) ~[dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
at org.dcache.util.CDCExecutorServiceDecorator$1.run(CDCExecutorServiceDecorator.java:104) [dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
at org.dcache.util.BoundedExecutor$Worker.run(BoundedExecutor.java:211) [dcache-core-2.12.0-SNAPSHOT.jar:2.12.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_25]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]

I considered whether one should update CacheRepositoryV5 to close all open
descriptors on shutdown, but that would be rather bad: If the descriptor
is closed, there is nothing prevent the nearline storage from still writing
the file.

Instead the nearline storage now properly cancel all requests and also
enforces that no new requests are enqueued once the nearline storage was
shut down. This is done with a two step procedure, first initiating
cancellation followed by a blocking call waiting for requests to terminate
(with a timeout).

We could also simply have represed the above error and relied on pool
startup verification to delete the file, but I think it is cleaner
to properly cancel the requests.

The patch also fixes a race in request cancellation (there was an open
todo comment in the code).

The desired of avoid coarse grained locking unfortunately makes the code
quite a bit more complicated.

Target: trunk
Request: 2.11
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan
Patch: https://rb.dcache.org/r/7558
(cherry picked from commit ff1c133b948b8cd803836efd2ae5b9cb2494a769)