av9ash/GitBugs huggingface dataset:
Bug: Shutdown message doesn't have generation check causing normal node considered shutdown by other nodes in cluster
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Oct/24 22:18
Resolved: 25/Nov/24 10:54

Recently we run into one issue that when we do rolling restart of a cluster. We found one node is UN in it's own gossip state but all other nodes in the same cluster are considering it as a DN node.

Gossip info from the node that is being considered as down by other nodes:

 
{code:java}
/1.1.1.1
  generation:1729812724
  heartbeat:20676
  STATUS:26:NORMAL,-1215648874011476782
  LOAD:20620:8.030878944E10
  SCHEMA:41:e3b3d9f6-e2e2-307b-959e-b493bd1f7bef
  DC:13:dc4
  RACK:15:dc4-0
  RELEASE_VERSION:6:4.1.3
  INTERNAL_IP:11:1.1.1.1
  RPC_ADDRESS:5:1.1.1.1
  NET_VERSION:2:12
  HOST_ID:3:65a87fef-e7f8-41f4-8cdf-9d8ea4f5e4f0
  RPC_READY:161:true
  INTERNAL_ADDRESS_AND_PORT:9:1.1.1.1:27378
  NATIVE_ADDRESS_AND_PORT:4:1.1.1.1:27379
  STATUS_WITH_PORT:25:NORMAL,-1215648874011476782
  SSTABLE_VERSIONS:7:big-nb
  TOKENS:24:<hidden> {code}
Gossip state from other nodes for this node:

 
{code:java}
/1.1.1.1
  generation:1729812724
  heartbeat:2147483647
  STATUS:332020:shutdown,true
  LOAD:30:8.032911052E10
  SCHEMA:19:e3b3d9f6-e2e2-307b-959e-b493bd1f7bef
  DC:13:dc4
  RACK:15:dc4-0
  RELEASE_VERSION:6:4.1.3
  INTERNAL_IP:11:1.1.1.1
  RPC_ADDRESS:5:1.1.1.1
  NET_VERSION:2:12
  HOST_ID:3:65a87fef-e7f8-41f4-8cdf-9d8ea4f5e4f0
  RPC_READY:332021:false
  INTERNAL_ADDRESS_AND_PORT:9:1.1.1.1:27378
  NATIVE_ADDRESS_AND_PORT:4:1.1.1.1:27379
  STATUS_WITH_PORT:332020:shutdown,true
  SSTABLE_VERSIONS:7:big-nb
  TOKENS:24:<hidden> {code}
The share the same generation but the other nodes are considering the node is shutdown.

 

 

After closer look into the problem, I think here's what happened.

When the node get restarted,

1. it first got gracefully shutdown, it will broadcast the GOSSIP_SHUTDOWN message to the rest of the cluster.

2. When it get back up, it will try to update it's generation and gossip with other nodes.

 

If one node get the new generation for this 1.1.1.1 node first, then it receive the GOSSIP_SHUTDOWN message from step 1 (Assuming we have a very large delay in network between 1.1.1.1 node and the bad receiver node). We will run into above situation.

 

I think GOSSIP_SHUTDOWN message should have the generation information and GossipShutdownVerbHandler should only bump the heartbeat for the local state if generation is same. If local generation is higher, we should ignore the shutdown message.
Bug: Directions to disable key cache in cassandra.yaml break startup
Status: Triage Needed
Priority: Normal
Created: 22/Nov/24 23:29


{noformat}
# Default value is empty to make it "auto" (min(5% of Heap (in MiB), 100MiB)). Set to 0 to disable key cache.
#
# This is only relevant to SSTable formats that use key cache, e.g. BIG.
# Min unit: MiB
key_cache_size:
{noformat}

but when you follow the instructions:


{noformat}
Caused by: org.yaml.snakeyaml.constructor.ConstructorException: Cannot create property=key_cache_size for JavaBean=org.apache.cassandra.config.Config@7f608e21
 in 'reader', line 26, column 1:
    cluster_name: 'rokt-density-1732 ...
    ^
Can't construct a java object for scalar tag:yaml.org,2002:int; exception=null
 in 'reader', line 476, column 17:
    key_cache_size: 0
{noformat}
Bug: buffer sstable writes
Status: Triage Needed
Priority: Normal
Created: 22/Nov/24 21:49

Watching the filesystem, we do this when writing an SSTable out:

{noformat}
root@cassandra2:~# xfsslower -p $(cassandra-pid) 0
Tracing XFS operations
TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
20:13:26 CompactionExec 33477  W 16359   5607255     0.04 nb-1499-big-Data.db
20:13:26 CompactionExec 33477  W 1       5607271     0.00 nb-1499-big-Data.db
20:13:26 CompactionExec 33477  W 1       5607271     0.00 nb-1499-big-Data.db
20:13:26 CompactionExec 33477  W 1       5607271     0.00 nb-1499-big-Data.db
20:13:26 CompactionExec 33477  W 1       5607271     0.00 nb-1499-big-Data.db
{noformat}

This is silly.  We should be internally buffering our writes.  It's probably not a ton of overhead, but if we're going to move to writing sstables with direct IO, we have to do it anyways.
Bug: Update shaded-dtest build to include some additional exclusions and relocations for sidecar/analytics usage
Status: Triage Needed
Priority: Normal
Created: 22/Nov/24 19:35

The Analytics and Sidecar projects consume the shaded dtest jars as part of their in-jvm dtest-based tests. We have made some minor modifications to the shading in order to accommodate this use case and would like to contribute those changes back to the main Cassandra project so we don’t have to maintain our own copies of the build scripts.
Bug: Enhance nodetool compactionhistory to report compaction type and strategy
Status: Open
Priority: Normal
Created: 14/Nov/24 20:22

It would be useful for compactionhistory to have:

a) updated documentation for nodetool [compactionhistory |https://cassandra.apache.org/doc/stable/cassandra/tools/nodetool/compactionhistory.html] describing the columns *rows_merged* and {*}compaction_properties{*}. This StackOverflow article provides a good descriptions of rows_merged:  [What does rows_merged mean in compactionhistory|https://stackoverflow.com/questions/27566558/what-does-rows-merged-mean-in-compactionhistory]

b) the compaction strategy used: STCS, LCS, TWCS, UCS and sub-properties. For example, with level, the from-level and to-level could be reported.

c) also useful would be to make the bytes in and out human readable by default, since there is already an -f option for machine readable output.
Bug: nodetool reloadseeds does not reliably reload the seeds
Status: Needs Committer
Priority: Normal
Created: 04/Nov/24 13:21

During re-deploying lots of Cassandra nodes I've observed that some nodes does not reliably reload the seeds when {{nodetool reloadseeds}} command was issued.

After the seeds list was changed in the config:
{code}
$ grep seeds /etc/cassandra/cassandra.yaml
 - seeds: 10.90.44.82
$ nodetool getseeds
Current list of seed node IPs, excluding the current node's IP: /10.90.40.86:7000 /10.90.44.86:7000
$ nodetool reloadseeds
Updated seed node IP list, excluding the current node's IP: /10.90.40.86:7000 /10.90.44.86:7000
{code}

At this instance the following line was logged to debug.log:
{code}
DEBUG [RMI TCP Connection(103568)-127.0.0.1] 2024-11-04 14:04:27,638 YamlConfigurationLoader.java:124 - Loading settings from file:/etc/cassandra/cassandra.yaml
{code}

However, getting the old list:
{code}
$ nodetool getseeds
Current list of seed node IPs, excluding the current node's IP: /10.90.40.86:7000 /10.90.44.86:7000
{code}

These nodes read the seed list only after Cassandra was restarted:

{code}
$ sudo systemctl restart cassandra.service
$ nodetool getseeds
Seed node list does not contain any remote node IPs
{code}

Note: this was observed on a seed node.

Observed on Cassandra 4.1.7.
Bug: Investigate how to ban usages of "var" in production code.
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Oct/24 14:16
Resolved: 31/Oct/24 15:42

In this thread (1) we reached the consensus of banning usages of "var" in the production code but allowing it in tests. We should investigate how to handle this robustly, e.g. by a checkstyle rule. The investigation how to do this should be conducted.

Upon delivery, that probably means that the existing usages of "var" in the production code will need to be fixed and converted to have it all on types again because such rule would fail the compilation of the production code otherwise.

(1) https://lists.apache.org/thread/jfgj8lh1llp9zrodmfqyq22kvtlodyk6
Bug: Flaky PstmtPersistenceTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Nov/24 09:10
Resolved: 15/Nov/24 12:31

PstmtPersistenceTest was found to be flaky in CASSANDRA-19986. It was multiplexed there 500 times * branches but it seems it can still fail

https://app.circleci.com/pipelines/github/instaclustr/cassandra/4921/workflows/90e7e7c8-e897-45b1-991d-953340dd1d5c
Bug: Nodetool rebuild should have option to truncate range table
Status: Triage Needed
Priority: Normal
Created: 14/Nov/24 23:10

When rebuilding a new cluster, we sometimes replace a node from the new ring, and the new node was bootstrapped the data from nodes in the new cluster. It's obvious that other nodes in the new cluster doesn't have any data yet, but the new node is then marking the ranges available. Later the nodetool rebuild just finished immediately.

It would be great to have an option to truncate this available ranges table in this nodetool so it'll just retry the rebuild.

[PR|https://github.com/apache/cassandra/pull/3691/files]
Bug: nodetool proxyhistograms errors out
Status: Triage Needed
Priority: Normal
Created: 14/Nov/24 20:46

I have a test cluster running 5.0.2 which has been running for several hours. The cluster is up and running and has accepted 300MM writes.

{noformat}
ubuntu@cassandra0:~$ nt status
Datacenter: us-west-2
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load        Tokens  Owns (effective)  Host ID                               Rack
UN  10.0.2.4    499.57 GiB  4       100.0%            ac8ef372-d2a9-4d34-9c05-98f2c61d6a5d  us-west-2c
UN  10.0.1.96   501.59 GiB  4       100.0%            82762ee1-3d72-4a22-8b7c-b6da20d5f836  us-west-2b
UN  10.0.0.100  502.24 GiB  4       100.0%            94bacb9f-f568-4091-85fb-b0a67ece6d13  us-west-2a
{noformat}

I bumped into this when trying to get an idea of the node's latency:

{noformat}
ubuntu@cassandra0:~$ nt proxyhistograms
Exception in thread "main" java.lang.IllegalArgumentException: Server is not initialized yet, cannot run nodetool.
	at org.apache.cassandra.tools.NodeTool.err(NodeTool.java:319)
	at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:282)
	at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:85)
{noformat}

I haven't been able to reproduce this locally yet.
Bug: Accord's ConfigService lock is held over large areas which cause deadlocks and performance issues
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 07/Nov/24 23:21
Resolved: 12/Nov/24 20:02

Accord has a AbstractConfigurationService that locks on “this” and does this over large ares, even when the lock is not needed.  This has been found to cause issues as messaging gets blocked on this causing timeouts.

To better improve things, we should lower the locking to what is needed and avoid locking while triggering callbacks
Bug: Expression#isSatisfiedBy() support for post-filtering IN clauses in SAI boolean queries
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Oct/24 16:57
Resolved: 12/Nov/24 16:54

We currently don’t support IN clauses in SAI AND queries, simply because we’ve never implemented {{Expression#isSatisfiedBy()}} for them, and they cannot be safely added to the post-index filter. (See CASSANDRA-19018 for more background on why.) Concerns around how big the {{IN}} clause can be aside, we should be able to add this relatively easily. The most interesting question is probably how we want to test, but we can start with some simple additions to the {{IndexQuerySupport}} and {{IndexingTypeSupport}} frameworks.
Bug: AbstractCommitLogService#lastSyncedAt initialized with currentTimeMillis() but later compared and updated with System.nanoTime() 
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 12/Nov/24 13:12
Resolved: 12/Nov/24 15:16

Within org.apache.cassandra.db.commitlog.AbstractCommitLogService

we use System.currentMillis()-based time to initialize lastSyncedAt field:
{code:java}
protected volatile long lastSyncedAt = currentTimeMillis();{code}
but later we use clock.now() = System.nanoTime() to compare and update the field value:
{code:java}
long pollStarted = clock.now();
boolean flushToDisk = lastSyncedAt + syncIntervalNanos <= pollStarted || state != NORMAL || syncRequested; {code}

As of now we are lucky that in the current JDK implementation  System.nanoTime()  is much bigger than currentTimeMillis() but it is not guaranteed.
Bug: Implement hot reloading of SSLContext for JMX SSL
Status: Triage Needed
Priority: Normal
Created: 08/Nov/24 02:58

As part of https://issues.apache.org/jira/browse/CASSANDRA-18508 we enabled configuring SSL for JMX via the `encryption_options` in the cassandra.yaml. However, that still didn't allow for hot reloading of the SSLContext given that the JMX SSL initialization is quite static in nature.

This ticket is to implement hot reloading of the SSLContext for JMX SSL to further enhance the system.
Bug: Assertion error in tracing while tracing is not enabled on SELECT query
Status: Triage Needed
Priority: Normal
Created: 08/Nov/24 17:33

Some values aren’t parseable by TimeSerializer#toString, even though they were allowed by the type itself, which leads to Tracer throwing.  
In general though, toString on this object should have never been called.

{code}
CREATE TABLE IF NOT EXISTS distributed_test_keyspace.table0 (pk0 uuid,pk1 float,pk2 inet,pk3 time,pk4 timestamp,pk5 smallint,ck0 double,ck1 timestamp,ck2 tinyint,ck3 inet,ck4 bigint,ck5 tinyint,ck6 decimal,ck7 ascii,ck8 boolean,static0 int static,static1 float static,static2 text static,static3 double static,static4 bigint static,static5 tinyint static,regular0 uuid,regular1 smallint,regular2 float,regular3 int,regular4 boolean,regular5 smallint,regular6 ascii,regular7 tinyint,regular8 bigint, PRIMARY KEY ((pk0,pk1,pk2,pk3,pk4,pk5), ck0, ck1, ck2, ck3, ck4, ck5, ck6, ck7, ck8)) WITH CLUSTERING ORDER BY (ck0 DESC,ck1 ASC,ck2 ASC,ck3 DESC,ck4 ASC,ck5 ASC,ck6 DESC,ck7 ASC,ck8 DESC);

SELECT * FROM distributed_test_keyspace.table0 WHERE pk0 = ? AND pk1 = ? AND pk2 = ? AND pk3 = ? AND pk4 = ? AND pk5 = ? AND ck0 > ? AND regular4 < ? AND static2 < ? ALLOW FILTERING;', bindings=0fbbf0b9-1fca-0dee-4470-c1937beb8f34,(float)0.87520605,/154.108.207.6,8947843475553632690L,Wed Sep 13 20:00:59 CEST 206158350,(short)-745,0.11730296006035457,true,"ꋈ??ꍖ?㬰궇鉵??⹍?ὕ貏???ꤸ"


java.lang.AssertionError: 103563
	at org.apache.cassandra.serializers.TimeSerializer.toString(TimeSerializer.java:106)
	at org.apache.cassandra.serializers.TimeSerializer.toString(TimeSerializer.java:27)
	at org.apache.cassandra.db.marshal.AbstractType.getString(AbstractType.java:156)
	at org.apache.cassandra.db.marshal.AbstractCompositeType.getString(AbstractCompositeType.java:204)
	at org.apache.cassandra.db.marshal.AbstractType.getString(AbstractType.java:161)
	at org.apache.cassandra.dht.AbstractBounds.format(AbstractBounds.java:130)
	at org.apache.cassandra.dht.AbstractBounds.getString(AbstractBounds.java:123)
	at org.apache.cassandra.db.PartitionRangeReadCommand.queryStorage(PartitionRangeReadCommand.java:345)
	at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:468)
	at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:2473)
	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2872)
	at org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163)
{code}
Bug: Case-insensitivity in IAuthenticator impls and AuthUtil.handleLogin()
Status: Triage Needed
Priority: Normal
Created: 07/Nov/24 00:30

Hi - I'm working on a proof-of-concept plugin for using a 3rd party IDP to support client-node and node-node authentication (in Cassandra 5). I noticed something a little confusing in AuthUtil.handleLogin() and am not sure if it's intentional, or an oversight (which I'm happy to try to correct). Would like to get some guidance first.

The tl;dr question is: Are implementations of SaslNegotiator.getAuthenticatedUser() supposed to return either a quoted user/role name, or a bare user name forced to lowercase (to match with case-insensitive rolenames created via CREATE ROLE), or should that forcing be happening somewhere in the path of AuthUtil.handleLogin()? (Here: [https://github.com/apache/cassandra/blob/7b33e91852aee7af2e995273237d0f3828cd9e4b/src/java/org/apache/cassandra/transport/messages/AuthUtil.java#L73|https://github.com/apache/cassandra/blob/7b33e91852aee7af2e995273237d0f3828cd9e4b/src/java/org/apache/cassandra/transport/messages/AuthUtil.java#L73),]).

Current behavior is that nowhere in AuthUtil.handleLogin() is a bare user name returned by a SaslNegotiator forced to lowercase for case-insensitive matching. In my case, the IDP considers user names to be case-insensitive but it may return authenticated user names in mixed case (e.g. "Susie"). In that case, the SaslNegotiator considers authentication successful, but the core Cassandra code fails the login attempt because (for example) "Susie" != "susie".

Is this expected behavior, or should bare user names be normalized to all lowercase somewhere in the AuthUtil.handleLogin() implementation to perform a case-insensitive lookup for associated roles?

Thanks - Joel.
Bug: Get Harry working on top of Accord and fix various issues found by TopologyMixupTestBase
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Nov/24 21:20
Resolved: 06/Nov/24 19:01

TopologyMixupTestBase has been useful at finding a lot of unexpected issues, and adding Harry on top of Accord at this layer should help validate Accord correctness while also testing stability.

In running these tests several bugs were found

1) vtable showing what txn are blocking the queried table would throw error when txn isn’t known, which is valid (report historic transaction…)
2) AccordCommandStore submitted sync requests in a blocking manner, but did this on a CommandStore… this lead to a 5 minute deadlock
3) MajorityDepsFetcher would have a deadlock as it triggers waiting notifications while holding the lock, and the waiting callers then access more locks, such as the config service lock
4) when restarting and learning about removed nodes, AccordService is not setup yet, so need to pass this through to avoid startup issues
5) When accord asks TCM for the epoch history, there were no retries which would cause stability issues during startup
6) when learning about min epochs needed for startup, purge all starting epochs that are empty as it isn’t needed and only adds costs for startup
7) when nodes leave the cluster we did not start durability sync (this isn’t working, but thats a different issue… durability sync requires ALL which isn’t possible)
8) TCM’s getLogEntries method hit an edge case with snapshots where it assumed the API was inclusive, but its exclusive; this caused a gap in epochs
9) JVM Dtest now supports startup timeouts, this is to avoid issues where startup will take infinity (due to bugs) causing CI to throw away the logs.
10) fixed a race condition bug in Harry where the TokenPlacementModel could see a partial row causing NPEs down the line
11) Fixed a bug in Harry where Accord timeouts would not retry as they don’t have the expected message
Bug: User types and vectors as clustering keys in descending order are not supported but ascending are
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Nov/24 12:42
Resolved: 06/Nov/24 13:52

This works:

{code}
cassandra@cqlsh:ks> CREATE TABLE tbl (pk int, ck frozen<foo>, PRIMARY KEY(pk, ck)) WITH CLUSTERING ORDER BY (ck ASC);
cassandra@cqlsh:ks> INSERT INTO ks.tbl (pk, ck) VALUES ( 1, {a: 1}) 
{code}

But this does not:

{code}
cassandra@cqlsh:ks> CREATE TABLE tbl (pk int, ck frozen<foo>, PRIMARY KEY(pk, ck)) WITH CLUSTERING ORDER BY (ck DESC);
cassandra@cqlsh:ks> INSERT INTO ks.tbl (pk, ck) VALUES ( 1, {a: 1})  ;
InvalidRequest: Error from server: code=2200 [Invalid query] message="Invalid user type literal for ck of type frozen<foo>"
{code}
Bug: When snapshotting the recreate CQL does not count UDTs from reverse clustering columns
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Oct/24 16:26
Resolved: 06/Nov/24 09:08

When we snapshot a table we create a list of CQL statements to recreate the table, this tries to include the UDT creation as well, but fails to find all UDTs!

The bug is from org.apache.cassandra.schema.TableMetadata#getReferencedUserTypes

{code}
public Set<ByteBuffer> getReferencedUserTypes()
{
    Set<ByteBuffer> types = new LinkedHashSet<>();
    columns().forEach(c -> addUserTypes(c.type, types));
    return types;
}
{code}

c.type maybe ReverseType, so checking for UDT fails and the UDT does not get included.

The following would show this

{code}
CREATE TYPE foo (a int);
CREATE TABLE tbl (pk int, ck frozen<foo>, PRIMARY KEY(pk, ck)) WITH CLUSTERING ORDER BY (ck DESC).
{code}

Additionally, the order that the UDTs found are listed are not in correct order, they are in random hash order…. This becomes a problem when the UDTs depend on each other such as the following example

{code}
CREATE TYPE a (a int);
CREATE TYPE b (b a);
{code}

In this example you must create “a” BEFORE “b”, but with hash ordering that isn’t always true.

Utilities exist in test/unit/org/apache/cassandra/utils/CassandraGenerators.java to handle this case properly
Bug: [Analytics] Extends DataTransferApi and add TwoPhaseImportCoordinator for coordinated write
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/Oct/24 20:42
Resolved: 05/Nov/24 22:23

This patch contains the changes for coordinated write. The CloudStorageDataTransferApi is extended to include the coordinated write specific APIs. It also introduces a new TwoPhaseImportCoordinator to achieve the two phase import across multiple clusters.
Bug: Timeout in config is not respected
Status: Resolved
Priority: Normal
Resolution: Not A Problem
Created: 04/Nov/24 02:00
Resolved: 05/Nov/24 00:07

Come across a bad data model where the request_timeout and read_request_timeout had been increased. On upgrading to 4.1 I noticed the query to the bad partitions (ie. large number of rows and tombstones and large amount of data on disk) would fail with timeout.

I have setup test cluster to reproduce on 5.0.2. I write1Kb rows to the same partition. The client and server side are both set to 15 minute timeouts. Setting page-size to 1 million (so fetching the page takes awhile). I get the following error on the client:

 

!image-2024-11-04-11-43-55-089.png!

And a read timeout shows in the metrics and max latency is 12.1 seconds.
Bug: Consider exposing the ability to select event loops for cqlsh
Status: Triage Needed
Priority: Normal
Created: 04/Nov/24 21:16

The Python driver currently includes support for [multiple event loops|https://docs.datastax.com/en/developer/python-driver/3.29/installation/index.html#supported-event-loops].  Users can specify a specific event loop by defining the "connection_class" param on the Cluster object; if this isn't done a default selection process will be performed.

 

cqlsh doesn't expose the ability to define this property on Cluster objects; this prevents users from leveraging other event loops without explicitly modifying cqlsh code.  This matters in cases where customers see problems with a specific event loop selected by the default selection process (something we have [seen in the wild|https://datastax-oss.atlassian.net/browse/PYTHON-1400]).  It also might help out Python 3.12 users who wish to specify a specific event loop (since the default of asyncore is no longer available to them).
Bug: Nodetool cms reconfigure reports success when the operation fails due to streaming issues
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/Oct/24 15:07
Resolved: 01/Nov/24 18:42

If streaming fails during CMS reconfiguration, the operation enters a {{blocked}} state. The operation will still be pending and resumable by an operator & the KPI that indicates the CMS is pending/due reconfiguration will correctly indicate the state. However, the nodetool command itself exits with a success status which is especially problematic in tests. 
This is easy to repro in an in-jvm dtest by attempting to a CMS reconfiguration without enabling the {{NETWORK}} feature on the cluster.