Bug: CancellationException when shutting down the 3.11.16 node
Status: Resolved
Priority: Normal
Resolution: Won't Fix
Created: 13/Sep/23 02:50
Resolved: 13/Sep/23 11:16

I encountered the following exception when I tried to shutdown the cassandra node after drain operation.
{code:java}
INFO  [RMI TCP Connection(2)-127.0.0.1] 2023-09-07 12:59:43,440 StorageService.java:1679 - DRAINED
ERROR [MigrationStage:1] 2023-09-07 12:59:43,443 CassandraDaemon.java:244 - Exception in thread Thread[MigrationStage:1,5,main]
java.util.concurrent.CancellationException: null
        at java.util.concurrent.FutureTask.report(FutureTask.java:121)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:438)
        at org.apache.cassandra.db.lifecycle.LogTransaction.waitForDeletions(LogTransaction.java:430)
        at org.apache.cassandra.db.lifecycle.LifecycleTransaction.waitForDeletions(LifecycleTransaction.java:623)
        at org.apache.cassandra.db.ColumnFamilyStore.invalidate(ColumnFamilyStore.java:587)
        at org.apache.cassandra.db.ColumnFamilyStore.invalidate(ColumnFamilyStore.java:557)
        at org.apache.cassandra.db.Keyspace.unloadCf(Keyspace.java:378)
        at org.apache.cassandra.db.Keyspace.dropCf(Keyspace.java:371)
        at org.apache.cassandra.config.Schema.dropTable(Schema.java:715)
        at org.apache.cassandra.schema.SchemaKeyspace.lambda$mergeSchema$18(SchemaKeyspace.java:1426)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1426)
        at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1413)
        at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1390)
        at org.apache.cassandra.service.MigrationManager$1.runMayThrow(MigrationManager.java:464)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
        at java.lang.Thread.run(Thread.java:750)
ERROR [Native-Transport-Requests-6] 2023-09-07 12:59:43,444 QueryMessage.java:129 - Unexpected error during query
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:442)
        at org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:429)
        at org.apache.cassandra.service.MigrationManager.announceKeyspaceDrop(MigrationManager.java:363)
        at org.apache.cassandra.cql3.statements.DropKeyspaceStatement.announceMigration(DropKeyspaceStatement.java:63)
        at org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:123)
        at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:243)
        at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:274)
        at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:259)
        at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116)
        at org.apache.cassandra.transport.Message$Dispatcher.processRequest(Message.java:688)
        at org.apache.cassandra.transport.Message$Dispatcher.lambda$channelRead0$0(Message.java:594)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:113)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:438)
        ... 14 common frames omitted
Caused by: java.util.concurrent.CancellationException: null
        at java.util.concurrent.FutureTask.report(FutureTask.java:121)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:438)
        at org.apache.cassandra.db.lifecycle.LogTransaction.waitForDeletions(LogTransaction.java:430)
        at org.apache.cassandra.db.lifecycle.LifecycleTransaction.waitForDeletions(LifecycleTransaction.java:623)
        at org.apache.cassandra.db.ColumnFamilyStore.invalidate(ColumnFamilyStore.java:587)
        at org.apache.cassandra.db.ColumnFamilyStore.invalidate(ColumnFamilyStore.java:557)
        at org.apache.cassandra.db.Keyspace.unloadCf(Keyspace.java:378)
        at org.apache.cassandra.db.Keyspace.dropCf(Keyspace.java:371)
        at org.apache.cassandra.config.Schema.dropTable(Schema.java:715)
        at org.apache.cassandra.schema.SchemaKeyspace.lambda$mergeSchema$18(SchemaKeyspace.java:1426)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1426)
        at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1413)
        at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1390)
        at org.apache.cassandra.service.MigrationManager$1.runMayThrow(MigrationManager.java:464)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
        ... 1 common frames omitted
 {code}
The CancellationException might be safely handled. I haven't observed data integrity problem, but I don't know whether it will cause any other effects.

I executed a long list of commands before shutting down the node. It mainly contains cqlsh related commands. This error message might not be reproduced deterministically since it's more likely related to interleaving between the (1) shutdown process and (2) threads working on requests.

I have attached the log file for the complete error message.