av9ash/GitBugs huggingface dataset:
Bug: Map and possibly remove Deprecated code
Status: In Progress
Priority: Normal
Created: 18/Oct/23 18:38

This is follow-up of CASSANDRA-18912. We should scan the project and document what was deprecated and when (based on "since" added to Deprecated annotation).

Once this is done, we need to identify what should be removed and what should stay.

It is questionable if this ticket will result in some actual code change. I think we should firstly just map what was deprecated to have a bigger picture where we are. Then we can discuss what to actually remove.
Bug: BLOG - Apache Cassandra 5.0 Features: Unified Compaction Strategy
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Oct/23 18:48
Resolved: 28/Oct/23 03:17

This ticket is to capture the work associated with publishing the blog "Apache Cassandra 5.0 Features: Unified Compaction Strategy"

This blog can be published as soon as possible, but if it cannot be published within a week of the noted publish date *(October 24)*, please contact me, suggest changes, or correct the date when possible in the pull request for the appropriate time that the blog will go live (on both the blog.adoc and the blog post's file).
Bug: BLOG - Apache Cassandra 5.0 Features: Dynamic Data Masking
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Oct/23 23:09
Resolved: 23/Oct/23 17:58

This ticket is to capture the work associated with publishing the blog "Apache Cassandra 5.0 Features: Dynamic Data Masking"

This blog can be published as soon as possible, but if it cannot be published within a week of the noted publish date *(October 11)*, please contact me, suggest changes, or correct the date when possible in the pull request for the appropriate time that the blog will go live (on both the blog.adoc and the blog post's file).
Bug: Two accord transactions have the exact same transaction id
Status: Resolved
Priority: Normal
Resolution: Invalid
Created: 18/Oct/23 15:53
Resolved: 18/Oct/23 16:17

When testing solutions for CASSANDRA-18798 I noticed that two independent transactions running at the same time in two parallel threads ended up having the exact same transaction id:


{code}
    public void testListAddition() throws Exception
    {
        SHARED_CLUSTER.schemaChange("CREATE TABLE " + currentTable + " (k int PRIMARY KEY, l list<int>)");
        SHARED_CLUSTER.forEach(node -> node.runOnInstance(() -> AccordService.instance().setCacheSize(0)));

        CountDownLatch latch = CountDownLatch.newCountDownLatch(1);

        Vector<Integer> completionOrder = new Vector<>();
        try
        {
            for (int i=0; i<100; i++)
            {

                ForkJoinTask<?> add1 = ForkJoinPool.commonPool().submit(() -> {
                    latch.awaitThrowUncheckedOnInterrupt();
                    SHARED_CLUSTER.get(1).executeInternal("BEGIN TRANSACTION " +
                            "UPDATE " + currentTable + " SET l = l + [1] WHERE k = 1; " +
                            "COMMIT TRANSACTION");
                    completionOrder.add(1);
                });

                ForkJoinTask<?> add2 = ForkJoinPool.commonPool().submit(() -> {
                    try {
                        Thread.sleep(0);
{code}

Adding some logging in TxnWrite.java reveals the two threads ave identical executeAt and unix timestamps:

{noformat}
lastmicros 0
DEBUG [node2_Messaging-EventLoop-3-4] node2 2023-10-18 18:26:08,954 AccordVerbHandler.java:54 - Receiving Apply{kind:Minimal, txnId:[10,1697642767659000,10,1], deps:[distributed_test_keyspace:[(-Inf,-1], (-1,9223372036854775805], (9223372036854775805,+Inf]]]:{}, {}, executeAt:[10,1697642767659000,10,1], writes:TxnWrites{executeAt:[10,1697642767659000,10,1], keys:[distributed_test_keyspace:DecoratedKey(-4069959284402364209, 00000001)], write:TxnWrite{}}, result:accord.api.Result$1@253c102e} from /127.0.0.1:7012
raw 0  (NO_LAST_EXECUTED_HLC=-9223372036854775808
lastExecutedTimestamp [0,0,0,0]
lastmicros 1697642767659000
raw -9223372036854775808  (NO_LAST_EXECUTED_HLC=-9223372036854775808
lastExecutedTimestamp [10,1697642767659000,10,1]
DEBUG [node2_CommandStore[1]:1] node2 2023-10-18 18:26:09,023 AccordMessageSink.java:167 - Replying ACCORD_APPLY_RSP ApplyApplied to /127.0.0.1:7012
lastmicros 0
raw 0  (NO_LAST_EXECUTED_HLC=-9223372036854775808
lastExecutedTimestamp [0,0,0,0]
lastmicros 1697642767659000
raw -9223372036854775808  (NO_LAST_EXECUTED_HLC=-9223372036854775808
lastExecutedTimestamp [10,1697642767659000,10,1]
timestamp 1697642767659000    executeAt[10,1697642767659000,10,1]
timestamp 1697642767659000    executeAt[10,1697642767659000,10,1]
{noformat}


Increasing the Thread.sleep() to9 or 10 helps so that the transactions have different IDs.
Bug: Repair vtable caches consume excessive memory
Status: Review In Progress
Priority: Normal
Created: 03/Oct/23 20:30

Currently, the repair vtables (system_views.{repairs,repair_sessions,repair_jobs,repair_participates,repair_validations}) are backed by caches in ActiveRepairService that are bounded by the number of elements in them, controlled by Config.repair_state_size and Config.repair_state_expires.

The individual cached elements are mutable, and can grow to retain a significant amount of heap as the instance uptime increases and more repairs are run. In a heap dump for a real cluster, I found these caches occupying ~1GB of heap total between ActiveRepairService.repairs and ActiveRepairService.participates. Individual cached elements were reaching 100KB in size, so configuring the caches by number of elements introduces a significant amount of potential variance in the actual heap usage of these caches.

We should measure these caches by the heap they retain, not by the number of elements. Users should not be expected to check heap dumps to calibrate the number of elements they configure the caches to consume - specifying a memory total is much more user-friendly.
Bug: NPE in getHostId when fresh starting up the 3.11.16 cluster
Status: Open
Priority: Normal
Created: 02/Oct/23 15:59

When I start up cassandra 3.11.16 cluster (Fresh Start, 3 nodes), I noticed the following NPE in the system log (Seed node).
{code:java}
INFO  [HANDSHAKE-/192.168.77.3] 2023-10-02 12:56:43,180 OutboundTcpConnection.java:561 - Handshaking version with /192.168.77.3
ERROR [GossipStage:1] 2023-10-02 12:56:43,185 CassandraDaemon.java:244 - Exception in thread Thread[GossipStage:1,5,main]
java.lang.NullPointerException: null
        at org.apache.cassandra.gms.Gossiper.getHostId(Gossiper.java:998)
        at org.apache.cassandra.gms.Gossiper.getHostId(Gossiper.java:993)
        at org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2632)
        at org.apache.cassandra.service.StorageService.onChange(StorageService.java:2203)
        at org.apache.cassandra.service.StorageService.onJoin(StorageService.java:3020)
        at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1246)
        at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1358)
        at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:69)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
        at java.lang.Thread.run(Thread.java:750) {code}
h1. Reproduce

It's non-deterministic and manifest for 0.15% of cases (6 out of 4000 run).
h1. Root Cause
{code:java}
public UUID getHostId(InetAddress endpoint, Map<InetAddress, EndpointState> epStates)
{
    return UUID.fromString(epStates.get(endpoint).getApplicationState(ApplicationState.HOST_ID).value);
} {code}
It happens at a {*}fresh start{*}, so my assumption is that StorageService is trying to get the epState of a specific endpoint before the state is created. But I am not sure how it's caused exactly by thread interleaving. I assume this problem also exists in 4.x version since the logic for StorageService.handleStateNormal is almost the same.

Consequences: This NPE won't prevent the cluster from starting up/executing commands. It's probably benign.

I have attached the logs of my cluster. (system.log contains logs from the seed node, and persistent.tar.gz contains logs from all three nodes and the configuration).
Bug: Align values in rows in CQLSH right for numbers, left for text
Status: Open
Priority: Low
Created: 04/Dec/23 11:29

*Updated* Jan 17 2024 after dev discussion

Change CQLSH to left-align text while continue to right-align numbers.  This will match how Postgres shell and Excel treat alignment of text and number.

-------------

*Original*

We need to make this

[https://github.com/apache/cassandra/blob/trunk/pylib/cqlshlib/cqlshmain.py#L1101]

configurable so values in columns are either all on left or on right side of the column (basically change col.rjust to col.ljust).

By default, it would be like it is now but there would be configuration property in cqlsh for that as well as a corresponding CQLSH command (optional), something like
{code:java}
ALIGNMENT LEFT|RIGHT
{code}
cc [~bschoeni]
Bug: NettyStreamingConnectionFactory lacks handling of outcomes other than success
Status: Triage Needed
Priority: Low
Created: 21/Dec/23 10:50

This is not a critical issue, but the errors thrown when we try to connect with incompatible messaging version (for example from an sstable loader) are odd. We get weird looking class cast exceptions like this:

{format}
Session peer /127.0.0.1:7012 Failed because of an unknown exception
java.lang.ClassCastException: class org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot be cast to class org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success (org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible and org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in unnamed module of loader 'app')
	org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
	org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)

{format}

We should probably handle that somehow and error out with a reasonable message, which at least would prove that we consciously refuse to connect.
Bug: Gossiper::hasMajorVersion3Nodes returns true when a cluster is upgrading patch version without Cassandra 3 nodes.
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 03/Nov/23 21:01
Resolved: 10/Jan/24 16:27

When working on https://issues.apache.org/jira/browse/CASSANDRA-18968 we found that {{Gossiper::hasMajorVersion3Nodes}} will return true when the cluster is undergoing an upgrade from a patch version even if the cluster has no Cassandra 3 nodes in it.

This can be reproduced by running this Gossiper test:
{code:java}
    @Test
    public void testHasVersion3NodesShouldReturnFalseWhenNoVersion3NodesDetectedAndCassandra4UpgradeInProgress() throws Exception
    {
        Gossiper.instance.start(0);
        Gossiper.instance.expireUpgradeFromVersion();

        VersionedValue.VersionedValueFactory factory = new VersionedValue.VersionedValueFactory(null);
        EndpointState es = new EndpointState((HeartBeatState) null);
        es.addApplicationState(ApplicationState.RELEASE_VERSION, factory.releaseVersion(CURRENT_VERSION.toString()));
        Gossiper.instance.endpointStateMap.put(InetAddressAndPort.getByName("127.0.0.1"), es);
        Gossiper.instance.liveEndpoints.add(InetAddressAndPort.getByName("127.0.0.1"));

        es = new EndpointState((HeartBeatState) null);
        String previousPatchVersion = String.valueOf(CURRENT_VERSION.major) + '.' + (CURRENT_VERSION.minor) + '.' + (CURRENT_VERSION.patch - 1);
        es.addApplicationState(ApplicationState.RELEASE_VERSION, factory.releaseVersion(previousPatchVersion));
        Gossiper.instance.endpointStateMap.put(InetAddressAndPort.getByName("127.0.0.2"), es);
        Gossiper.instance.liveEndpoints.add(InetAddressAndPort.getByName("127.0.0.2"));
        assertFalse(Gossiper.instance.hasMajorVersion3Nodes());
    }
{code}
This seems to be because of [https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/gms/Gossiper.java#L2360], where an upgrade in progress is possible but we are not upgrading from a lower family version (i.e from 4.1.1 to 4.1.2).

From the comment in this function, it seems instead of the existing check, we would want to iterate over all known endpoints in gossip and return true if any of them do not have a version (similar to [https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/gms/Gossiper.java#L227-L236) |https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/gms/Gossiper.java#L227-L236).]
Bug: Refactor cqlshmain global constants
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 14/Dec/23 22:36
Resolved: 20/Feb/24 12:42

Refactor global constants
 * move globals CASSANDRA_CQL_HTML and cqlruleset
 * move module level defaults (DEFAULT_) into Shell class
 * move setup_cqlruleset into cqlshhandling

Remove unused exceptions 
 * FunctionNotFound(Exception)
 * AggregateNotFound(Exception)
Bug: Nodetool 'tablestats' formatting uses inconsistent significant digits
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 09/Nov/23 14:08
Resolved: 01/Dec/23 10:08

Nodetool reports milliseconds (ms) with anywhere from 3 to 15 significant digits.  Ratios use five or sixteen decimal places.  Averages use 1 or 13 decimal places.
 * milliseconds should use 3 decimal places 
 * ratios should use 3 decimal places (tenths of a percent)
 * averages should use 1 or 2

For readability, it would be helpful if large integers had comma separators.  I.e., space used: as 1,463,210,998,523 and/or in GiB/MiB/KiB.  It's unclear if the exact disk size is somehow useful, as it may change minute-by-minute, if not, rounding would be best, or displaying both   Space used (live): 1,463,210,998,523 (1,463GiB)

Total number of tables: 83
----------------
Keyspace : X
    Read Count: 1007337271
    Read Latency: 8.485891803649942 ms
    Write Count: 67550181
    Write Latency: 0.02556443163342523 ms
    Pending Flushes: 0
        Table: Y
        SSTable count: 7183
        Old SSTable count: 0
        SSTables in each level: [0, 9, 92, 754, 6328, 0, 0, 0, 0]
        Space used (live): 1463210998523
        Space used (total): 1463210998523
        Space used by snapshots (total): 0
        Off heap memory used (total): 607419608
        SSTable Compression Ratio: 0.3146620992793412
        Number of partitions (estimate): 24784137
        Memtable cell count: 106067
        Memtable data size: 248539982
        Memtable off heap memory used: 0
        Memtable switch count: 256
        Local read count: 865440924
        Local read latency: 6.857 ms
        Local write count: 13881409
        Local write latency: 0.037 ms
        Pending flushes: 0
        Percent repaired: 0.0
        Bytes repaired: 0.000KiB
        Bytes unrepaired: 4315.386GiB
        Bytes pending repair: 0.000KiB
        Bloom filter false positives: 11027855
        Bloom filter false ratio: 0.01099
        Bloom filter space used: 33590024
        Bloom filter off heap memory used: 33532560
        Index summary off heap memory used: 8174024
        Compression metadata off heap memory used: 565713024
        Compacted partition minimum bytes: 36
        Compacted partition maximum bytes: 17797419593
        Compacted partition mean bytes: 189740
        Average live cells per slice (last five minutes): 1443.2146104466253
        Maximum live cells per slice (last five minutes): 105778
        Average tombstones per slice (last five minutes): 1.0
        Maximum tombstones per slice (last five minutes): 1
        Dropped Mutations: 0
        Droppable tombstone ratio: 0.00000
Bug: Add cqlsh autocompletion for the vector data type
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 20/Oct/23 14:40
Resolved: 25/Oct/23 11:18

Add cqlsh autocompletion for the vector data type, so we have completions such as:
{code:java}
cqlsh> CREATE TABLE t (v
ascii     boolean   decimal   float     int       set       time      tinyint   varint
bigint    counter   double    frozen    list      smallint  timestamp uuid      vector
blob      date      duration  inet      map       text      timeuuid  varchar

cqlsh> CREATE TABLE t (v vector<
ascii     boolean   decimal   float     int       set       time      tinyint   varint
bigint    counter   double    frozen    list      smallint  timestamp uuid      vector
blob      date      duration  inet      map       text      timeuuid  varchar

cqlsh> CREATE TABLE t (v vector<float,
              <wholenumber>

cqlsh> INSERT INTO t(k, v) VALUES ( 0,
                                 <value for v (vector<float, 3>)>
{code}
Bug: Add jsonl support to sstabledump
Status: Resolved
Priority: Low
Resolution: Won't Fix
Created: 10/Oct/23 23:30
Resolved: 12/Oct/23 03:15

Add a new option to sstabledump to allow the generation of jsonl format.

The -l option for sstabledump does not produce valid jsonl.

[https://jsonlines.org/validator/]

Valid jsonl removes whitespace, including newlines, to make the format compact.  

By adding an option to generate valid jsonl, jsonl can be generated that is smaller.

Smaller output is better for execution time, storage, and transport. 

jsonl also allows the splitting of a file for processing at a known good line for processing without parsing the file, as each row is a line.
Bug: nodetool removenode error “Conflicting replica added”
Status: Resolved
Priority: Urgent
Resolution: Fixed
Created: 12/Jan/21 16:50
Resolved: 03/Mar/21 07:16

When testing elassandra on C* 4.0, integration tests with ccm systematically failed on removing a node with the following error “Conflicting replica added” . [This integration test |https://github.com/strapdata/elassandra/blob/v6.8.4-strapdata/integ-test/test-cleanup-repair.sh#L289] was ok with Elassandra based on Cassandra 3.11, and there is no changes in that test. Moreover, it seems there is no cassandra-test (dtest) for removing a node (there is only one removenode test for transient replication). The topology_test.py remove a node from the CCM cluster, but it does not call nodetool removenode.

I wonder if we have a non-tested regression here in C 4.0 ?
{noformat}
++ ccm node1 nodetool status
++ awk ‘/127.0.0.3/ \{ print $7 }’
+ HOST_ID3=6d2e858f-dacc-4c7c-a626-14b45f6b3b94
+ ccm node3 stop
+ ccm node1 nodetool removenode 6d2e858f-dacc-4c7c-a626-14b45f6b3b94
Traceback (most recent call last):
  File “/usr/local/bin/ccm”, line 4, in <module>
    __import__(‘pkg_resources’).run_script(‘ccm==3.1.6’, ‘ccm’)
  File “/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py”, line 742, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File “/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py”, line 1674, in run_script
    exec(script_code, namespace, namespace)
  File “/Library/Python/2.7/site-packages/ccm-3.1.6-py2.7.egg/EGG-INFO/scripts/ccm”, line 112, in <module>  File “build/bdist.macosx-10.14-intel/egg/ccmlib/cmds/node_cmds.py”, line 233, in run
  File “build/bdist.macosx-10.14-intel/egg/ccmlib/node.py”, line 848, in nodetool
  File “build/bdist.macosx-10.14-intel/egg/ccmlib/node.py”, line 2131, in handle_external_tool_process
ccmlib.node.ToolError: Subprocess [‘nodetool’, ‘-h’, ‘localhost’, ‘-p’, ‘7100’, ‘removenode’, ‘6d2e858f-dacc-4c7c-a626-14b45f6b3b94’] exited with non-zero status; exit status: 1;
stdout: nodetool: Conflicting replica added (expected unique ranges): Full(/127.0.0.1:7000,(4949329179655327935,6135417578204142297]); existing: Full(/127.0.0.1:7000,(4949329179655327935,6135417578204142297])
See ‘nodetool help’ or ‘nodetool help <command>’.++ finish
++ echo ‘ERROR occurs, test failed’
ERROR occurs, test failed
++ ‘[’ ‘!’ -z ‘’ ‘]’
++ exit 1
{noformat}
Bug: secure ssl connection from client to server
Status: Resolved
Priority: Urgent
Resolution: Invalid
Created: 05/Mar/21 05:20
Resolved: 05/Mar/21 14:35

Hello, Soham here.

I am trying to make a secure ssl connection from client AWS instance to server AWS Instance of cassandra using Nodejs.

I have generated self-signed certificates and configured the keystores in cassandra.yaml 

cqlsh connection work with it but over nodejs it's not working.

when clientt instance tries to connect with the server where Cassandra is configured via nodejs, it gives an error like wrong ssl version number, and also want to do connection with CA-signed certificates, so anyone please guide me over here for these.

 

Thanks.
Bug: nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.
Status: Resolved
Priority: Urgent
Resolution: Invalid
Created: 19/Feb/21 13:09
Resolved: 19/Feb/21 15:31

Hi After Installing cassandra 3.11.4 on Ec2 Instance. Iam trying to start cassandra Database It is starring without any errors. when Iam trying to check nodetool status or cqlsh getting below error

 

nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.. Iam bombarded with this over past three days please check and give sugestion to resolve the issue
Bug: Cassandra Logon Triggers
Status: Resolved
Priority: Urgent
Resolution: Invalid
Created: 18/Feb/21 17:30
Resolved: 18/Feb/21 17:45

Hi Team,

 

We have one requirement to create a “logon trigger” for Cassandra DB that will launch a Linux OS shell command when a user connects.

 

Please help us if you have any related information.
Bug: Cassandra JVM Memory Pools -Par Survivor Space UsedPoolMemory
Status: Resolved
Priority: Urgent
Resolution: Invalid
Created: 08/Feb/21 12:45
Resolved: 08/Feb/21 15:45

I have a 9 node cluster of cassandra, with 2gb of data on each node in our production environment. We are continuously facing below warnings from one of our monitoring tool, but there is no suspicious error from system.log file. Also there is sufficient memory available for Heap/server.

 
 warn - Cassandra JVM Memory Pools -Par Survivor Space UsedPoolMemory
 Datasource: Cassandra JVM Memory Pools -Par Survivor Space
 Datapoint: UsedPoolMemory
 Level: warn
 Start: 2021-02-08 10:11:17 GMT
 Duration: 1h 0m
 Value: 100.0
 ClearValue: 1.3631
 Reason: UsedPoolMemory is not >= 90: the current value is 1.3631

The java memory pool Par Survivor Space is now using 100.0 % of the maximum.

 We see the below message in debug.log file.

2021-02-08 08:43:51,310 INFO  [ReadStage-1] NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB

2021-02-08 08:47:28,344 DEBUG [COMMIT-LOG-ALLOCATOR] AbstractCommitLogSegmentManager.java:109 - No segments in reserve; creating a fresh one

 Please help us to fix this issue ASAP.

 Thanks!