av9ash/GitBugs huggingface dataset:
Bug: Forbid analyzed SAI indexes on primary key columns
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Aug/23 12:09
Resolved: 24/Aug/23 09:24

Queries using SAI indexes don't find any results when the index is on a primary key column, the indexing uses analysis, and the queried value is different to the exact value of the column. For example:
{code:java}
CREATE TABLE t(k int, c text, PRIMARY KEY (k, c));
CREATE INDEX ON t(c) USING 'sai' WITH OPTIONS = { 'case_sensitive' : false };
INSERT INTO t(k, c) VALUES (1, 'A');
SELECT * FROM t WHERE c = 'a'; -- no results found!!!
{code}
This happens because the {{ClusteringIndexFilter}} for the query doesn't take analysis into account. Thus, when that filter is applied by [{{QueryController#doesNotSelect(PrimaryKey)}}|https://github.com/apache/cassandra/blob/655a2455ac29395b0a303e6ad7fc4d458b18932d/src/java/org/apache/cassandra/index/sai/plan/QueryController.java#L194-L210] it rejects the results that have been correctly found by the index.

An initial approach to solve this problem could be making {{ClusteringIndexFilter}} aware of the index analysis options. However, this would be problematic for paging. The first page of the query contains a restriction in the clustering that requires analysis. But subsequent queries will contain the last seen clustering, and we don’t want analysis in that case.

Another approach would be not adding a {{ClusteringIndexFilter}} to the query restrictions when it contains this type of restriction on columns. However, this approach would create a weird situation where adding an index might make {{ALLOW FILTERING}} necessary in queries that wouldn’t need it without the index. This is the opposite of the natural way of things, where more indexes mean less AF needed. For example:
{code:java}
CREATE TABLE t(k int, c1 text, c2 int, PRIMARY KEY (k, c1, c2));
CREATE INDEX idx ON t(c1) USING 'sai' WITH OPTIONS = { 'case_sensitive' : false };
SELECT * FROM t WHERE k = 0 AND c1 = 'a' AND c2 = 0 ALLOW FILTERING;
{code}
The query would need AF because it has been translated into an index query without a clustering filter, and {{c2}} is not indexed.

I think there is an ambiguity in the query, and it's not clear if it should use the secondary index filter and use analysis, or it should be a primary index query and not use analysis. Although we can default to one or another interpretation, both can serve different use cases. We will probably need some new CQL syntax to allow users to specify whether they want to use the secondary index or not.

We can work on those CQL improvements during the second phase of SAI. In the meantime, I think we should simply forbid the creation of analyzed indexes on primary key columns.
Bug: Test failure: org.apache.cassandra.cql3.ViewTest.testIgnoreUpdate-compression.jdk11 (from org.apache.cassandra.cql3.ViewTest-compression.jdk11)
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Jul/23 21:51
Resolved: 15/Aug/23 14:42

Seen here - [https://ci-cassandra.apache.org/job/Cassandra-trunk/1650/testReport/junit/org.apache.cassandra.cql3/ViewTest/testIgnoreUpdate_compression_jdk11/]
h3.  
{code:java}
Error Message
expected:<1> but was:<2>

Stacktrace
junit.framework.AssertionFailedError: expected:<1> but was:<2> at org.apache.cassandra.cql3.ViewTest.testIgnoreUpdate(ViewTest.java:328) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.jboss.byteman.contrib.bmunit.BMUnitRunner$6.evaluate(BMUnitRunner.java:263) at org.jboss.byteman.contrib.bmunit.BMUnitRunner$1.evaluate(BMUnitRunner.java:97)
{code}
Bug: Restore accidentally deleted documentation for DDM
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Aug/23 17:23
Resolved: 21/Aug/23 14:21

CASSANDRA-18228 seems to have deleted [the section of documentation about dynamic data masking (CEP-20)|https://github.com/apache/cassandra/blob/63ab8e09286e4b2076695f62e08a7f39cb6860d1/doc/modules/cassandra/pages/cql/dynamic_data_masking.adoc]. The code examples and the (broken) index entry are still present. This ticket should add it back.
Bug: Test failure: dtest.jmx_test.TestJMX.test_compactionstats
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Jul/23 22:12
Resolved: 21/Aug/23 21:35

Seen here:

[https://ci-cassandra.apache.org/job/Cassandra-trunk/1646/testReport/dtest.jmx_test/TestJMX/test_compactionstats/]
h3.  
{code:java}
Error Message
AttributeError: 'NoneType' object has no attribute 'named'

Stacktrace
self = <jmx_test.TestJMX object at 0x7fe78cfc2940> def test_compactionstats(self): """ @jira_ticket CASSANDRA-10504 @jira_ticket CASSANDRA-10427 Test that jmx MBean used by nodetool compactionstats properly updates the progress of a compaction """ cluster = self.cluster cluster.populate(1) node = cluster.nodelist()[0] cluster.start() # Run a quick stress command to create the keyspace and table node.stress(['write', 'n=1', 'no-warmup']) # Disable compaction on the table node.nodetool('disableautocompaction keyspace1 standard1') node.nodetool('setcompactionthroughput 1') node.stress(['write', 'n=150K', 'no-warmup']) node.flush() # Run a major compaction. This will be the compaction whose # progress we track. node.nodetool_process('compact') # We need to sleep here to give compaction time to start # Why not do something smarter? Because if the bug regresses, # we can't rely on jmx to tell us that compaction started. time.sleep(5) compaction_manager = make_mbean('db', type='CompactionManager') with JolokiaAgent(node) as jmx: progress_string = jmx.read_attribute(compaction_manager, 'CompactionSummary')[0] # Pause in between reads # to allow compaction to move forward time.sleep(2) updated_progress_string = jmx.read_attribute(compaction_manager, 'CompactionSummary')[0] var = 'Compaction@{uuid}(keyspace1, standard1, {progress}/{total})bytes' if self.cluster.version() >= LooseVersion('4.0'): # CASSANDRA-15954 var = 'Compaction({taskUuid}, {progress} / {total} bytes)@{uuid}(keyspace1, standard1)' > progress = int(parse.search(var, progress_string).named['progress']) E AttributeError: 'NoneType' object has no attribute 'named' jmx_test.py:218: AttributeError
{code}
Bug: Remove unnecessary Netty dependencies after upgrade to 4.1.96
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 07/Aug/23 09:03
Resolved: 17/Aug/23 09:34

We introduced a lot of unnecessary dependencies in CASSANDRA-17992 which should not be there. This ticket is about their removal.

The original patch also got some dependencies wrong. They are 4.1.94 but we are on 4.1.96.
Bug: Deal with commented assertion in UFTest#testSchemaChange
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Aug/23 18:44
Resolved: 16/Aug/23 15:32

[~maedhroz] reported that there is an assertion left commented out in UFTest#testSchemaChange in CASSANDRA-18252.

It should be either removed or fixed if needed. 

[https://github.com/apache/cassandra/pull/2154/files#diff-d22803d878bbcebc2bfb5e73ec39cea75ae0800f3cfbeb81d0bce358ab2b3c00R118-R126]
Bug: tablestats should show speculative retries
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Aug/23 15:24
Resolved: 18/Aug/23 10:23

NodeProbe makes [provisions|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/tools/NodeProbe.java#L1921] for it, but it is never used later.
Bug: Test Failure: HandshakeTest.testOutboundConnectionfFallbackDuringUpgrades ClassCastException: Established ->  Disconnected
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Jul/23 17:51
Resolved: 03/Aug/23 16:13

The following has been witnesses a number of times in repeated runs, in different jdks and different test configurations.
{noformat}
java.lang.ClassCastException: class org.apache.cassandra.net.OutboundConnection$Established cannot be cast to class org.apache.cassandra.net.OutboundConnection$Disconnected (org.apache.cassandra.net.OutboundConnection$Established and org.apache.cassandra.net.OutboundConnection$Disconnected are in unnamed module of loader 'app')
	at org.apache.cassandra.net.OutboundConnection$State.disconnected(OutboundConnection.java:201)
	at org.apache.cassandra.net.OutboundConnection$1Initiate.initiate(OutboundConnection.java:1248)
	at org.apache.cassandra.net.OutboundConnection.initiate(OutboundConnection.java:1254)
	at org.apache.cassandra.net.HandshakeTest.initiateOutbound(HandshakeTest.java:365)
	at org.apache.cassandra.net.HandshakeTest.testOutboundFallbackOnSSLHandshakeFailure(HandshakeTest.java:380)
	at org.apache.cassandra.net.HandshakeTest.testOutboundConnectionfFallbackDuringUpgrades(HandshakeTest.java:255)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{noformat}

ref: https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/217/workflows/ce7392fa-756a-4392-b3a5-206bb2940553/jobs/15965/tests
Bug: Fix Requires for Java for RPM packages
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Aug/23 18:12
Resolved: 15/Aug/23 12:10

We have to change Requires in redhat/cassandra.spec to

java-1.8.0 for 3.0 and 3.11
(java-1.8.0 or java-11) for 4.0. and 4.1
(java-11 or java-17) for 5.0 and trunk

noboolean would contain only java-1.8.0 and java-11 respectively where applicable.
Bug: bcprov-jdk15on-1.70.jar vulnerability: CVE-2023-33201
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Aug/23 10:58
Resolved: 08/Aug/23 14:58

https://nvd.nist.gov/vuln/detail/CVE-2023-33201

{quote}
Bouncy Castle For Java before 1.74 is affected by an LDAP injection vulnerability. The vulnerability only affects applications that use an LDAP CertStore from Bouncy Castle to validate X.509 certificates. During the certificate validation process, Bouncy Castle inserts the certificate's Subject Name into an LDAP search filter without any escaping, which leads to an LDAP injection vulnerability.
{quote}
Bug: Add jvmargs for Netbeans and Eclipse for JDK11 and 17
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Jul/23 23:37
Resolved: 28/Jul/23 19:54

Add JVM args for Netbeans and Eclipse for JDK11 and 17, similar to what we do for IntellijIdea.

Otherwise, people who use NetBeans and Eclipse should be adding them manually. 

A good reference on how to do it would be generate-idea-files.

We would need to make changes to generate-eclipse-files for Eclipse.

I am not a NetBeans user but it seems we update directly the files, no ant target.

CC [~Bereng] and [~mck] as being one of the few users using those two IDEs
Bug: Improve vector value validation errors
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Jul/23 12:08
Resolved: 10/Jul/23 10:40

Some of the error messages shown when trying to bind a vector column with a wrong value are a bit obscure. For example, if we try to write a value with the wrong number of dimensions:
{code}
createTable("CREATE TABLE %s (pk int primary key, value vector<int, 2>)");
execute("INSERT INTO %s (pk, value) VALUES (0, ?)", vector(1));
{code}
 The error is:
{code}
java.lang.IndexOutOfBoundsException: Attempted to read 4, but the size is 0
{code}
I understand that we cannot provide a lot of information about a non-understandable bind value, since we don't even know whether it's a vector or something entirely different. But we could try to improve it a bit and throw a {{MarshalException}} that will be translated to a {{InvalidRequestException}}, and provide generic messages similar to those used by collections:
{code}
InvalidRequestException: Not enough bytes to read a vector<int, 2>
{code}
Bug: Some unit tests create a heap dump each time
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/Aug/23 11:32
Resolved: 11/Aug/23 15:42

I noticed this by running out of space while working on CASSANDRA-18706.  Something must be calling HeaupUtils.maybeCreateHeapDump because the build/test directory is littered with pidxxxx-epochxxxxxxxx.hprof files despite the tests not crashing.  I checked a couple other random tests, RandomSchemaTest also generates a dump but KeywordSplitTest does not.

This behavior does not occur on 4.1.
Bug: CorruptedSSTablesCompactionsTest is flaky
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Jul/23 14:37
Resolved: 26/Jul/23 11:10

See e.g. [this run|https://app.circleci.com/pipelines/github/blambov/cassandra/510/workflows/fd484f76-b0f0-48c9-8672-d73bdc36b8ec/jobs/13575/tests].

The test was looked at for CASSANDRA-15879, but it is still failing from time to time. One of the failures appears to be introduced by CASSANDRA-14227 and the others by CASSANDRA-18134. The failures are genuine problems with handling corruption, not just test issues.

The {{CorruptSSTableException}} paths in {{SSTableIdentityIterator}} should likely also catch {{AssertionError}} and {{IllegalArgumentException}}, and most probably the tombstone verification should be done on the read path.
Bug: Fix flaky test: org.apache.cassandra.index.sai.disk.SelectiveIntersectionTest.tracingIsCorrectlyReported
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 07/Sep/23 15:35
Resolved: 12/Sep/23 11:15

https://ci-cassandra.apache.org/job/Cassandra-5.0/27/testReport/org.apache.cassandra.index.sai.disk/SelectiveIntersectionTest/tracingIsCorrectlyReported_cdc_jdk17/

{quote}
Error Message

Failed to bind port 34357 on 127.0.0.1.

Stacktrace

java.lang.IllegalStateException: Failed to bind port 34357 on 127.0.0.1.
	at org.apache.cassandra.transport.Server.start(Server.java:132)
	at org.apache.cassandra.cql3.CQLTester.startServer(CQLTester.java:649)
	at org.apache.cassandra.cql3.CQLTester.requireNetwork(CQLTester.java:606)
	at org.apache.cassandra.cql3.CQLTester.requireNetwork(CQLTester.java:591)
	at org.apache.cassandra.index.sai.disk.SelectiveIntersectionTest.setup(SelectiveIntersectionTest.java:52)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use
{quote}
Bug: Fix dtests: replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 31/Aug/23 16:39
Resolved: 06/Sep/23 13:54

https://ci-cassandra.apache.org/job/Cassandra-5.0/18/testReport/dtest-large.replace_address_test/TestReplaceAddress/test_restart_failed_replace/

This and other similar tests recently started failing:

{noformat}
Unexpected error found in node logs (see stdout for full details). Errors: [[replacement] 'ERROR [Stream-Deserializer-/127.0.0.1:7000-91782e47] 2023-08-29 23:05:51,677 StreamSession.java:700 - [Stream #990152d0-46c0-11ee-9290-158c46e94542] Socket closed before session completion, peer 127.0.0.1:7000 is probably down.\njava.nio.channels.ClosedChannelException: null\n\tat org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)\n\tat org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:178)\n\tat org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:49)\n\tat org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)']
{noformat}
Bug: Remove dependency on commons-codec
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Aug/23 20:50
Resolved: 05/Sep/23 18:12

The dependency doesn't seem to have any direct usage in our codebase.
The only reference to how it might be used was found in CASSANDRA-12790. Considering we also deprecate in 5.0 and remove metrics-reporter-config as part of CASSANDRA-18743, I do not see a reason not to remove this one too. 

We should probably take the same approach as CASSANDRA-18743 - deprecate in 5.0 and remove in 5.1. ML thread is to be opened too.
Bug: SAI result retriever is filtering too many rows
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Aug/23 12:51
Resolved: 07/Sep/23 10:59

Performance tests on SAI have shown that the number of rows being filtered for wide row partitions is more than expected. For a 10k row/partition test - limit 10, the following has been observed.

!image-2023-08-08-13-49-19-576.png|width=952,height=368!

This is not the expected outcome of row-awareness and needs investigating. The number of rows read should, roughly, match the number of partitions read.
Bug: BLOG - Cassandra Contributor July Meeting Replay
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Sep/23 19:57
Resolved: 07/Sep/23 11:33

This ticket is to capture the work associated with publishing the September 2023 blog "Cassandra Contributor July Meeting Replay"

This blog can be published as soon as possible, but if it cannot be published within a week of the noted publish date *(September 6)*, please contact me, suggest changes, or correct the date when possible in the pull request for the appropriate time that the blog will go live (on both the blog.adoc and the blog post's file).
Bug: Cleanup behaviour during node decommission caused missing replica
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 06/Sep/23 08:21
Resolved: 06/Sep/23 08:39

Node decommission triggers data transfer to other nodes. While this transfer is in progress,
receiving nodes temporarily hold token ranges in a pending state. However, the cleanup process currently doesn't consider these pending ranges when calculating token ownership.
As a consequence, data that is already stored in sstables gets inadvertently cleaned up.

STR:
 * Create two node cluster
 * Create keyspace with RF=1
 * Insert sample data (assert data is available when querying both nodes)
 * Start decommission process of node 1
 * Start running cleanup in a loop on node 2 until decommission on node 1 finishes
 * Verify of all rows are in the cluster - it will fail as the previous step removed some of the rows

It seems that the cleanup process does not take into account the pending ranges, it uses only the local ranges - https://github.com/apache/cassandra/blob/caad2f24f95b494d05c6b5d86a8d25fbee58d7c2/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L466.

There are two solutions to the problem.

One would be to change the cleanup process in a way that it start taking pending ranges into account. Even thought it might sound tempting at first it will require involving changes and a lot of testing effort.

Alternatively we could interrupt/prevent the cleanup process from running when any pending range on a node is detected. That sounds like a reasonable alternative to the problem and something that is relatively easy to implement.
Bug: Expose per partition on-disk usage through new DataFrame that utilizes the Index.db SSTable file components.
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Jul/23 19:06
Resolved: 05/Sep/23 18:05

This addition to Cassandra Analytics provides a new PartitionSizeTableProvider that generates a DataFrame returning the compressed and uncompressed partition sizes for all partitions in a table by utilizing the Index.db SSTable file component.
Bug: RPM install does not work with jdk17
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Sep/23 17:38
Resolved: 04/Sep/23 15:25

The cassandra.in.sh file was updated as part of CASSANDRA-18255 but the corresponding redhat/cassandra.in.sh was not.

This means when cassandra starts on a rpm install using jdk17 the jvm11-server.options file is still used. And so it fails.

Initial patch that works: https://github.com/apache/cassandra/compare/cassandra-5.0...thelastpickle:cassandra:mck/redhat-jdk17-fix/5.0 

Maybe it is possible to find a superior solution, like wrapping this file instead of duplicating it.
Bug: Re-organize ordering of configuration properties in cassandra.yaml
Status: Triage Needed
Priority: Normal
Created: 04/Sep/23 14:45

I was setting up a cluster from scratch and I was configuring cassandra.yaml, obviously. What I do not like about the current content of cassandra.yaml is that some of its sections are seemingly scattered across that file and if feels like out of place / order.

For example, if one wants to configure encryption it is on line 1458 and 1516 respectively. Snitch is on line 1402. The whole file has 2100 lines. So in order to configure a snitch, which is very common scenario when setting up a node, if I do not know that the field I am going to edit is called "endpoint_snitch" so I can just jump to it, I need to scroll to the three quarters of the file to edit it.

I think that the most commonly edited sections of the configuration file should be on the very top or somewhere around it so as I am editing that file from scratch I just go section by section and I do not jump around, I just edit that or I skip it and I configure what is next - ordered by the probability that a section is going to be configured differently from a default.

I also noticed that these properties are missing in cassandra.yaml, we can add it there upon refactorisation of this.

memtable conf
dynamic_snitch
Bug: Missing license info and headers identified in 5.0-alpha1 candidate
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Aug/23 11:23
Resolved: 31/Aug/23 20:48

LICENSE.txt is missing
python-smhasher
OrderedDict
MagnetoDB
java-driver

LICENSE.txt has mistakes around
Chronicle-Bytes
Guava
PATRICIA Trie

pylib docker files needed license headers

ref: https://lists.apache.org/thread/3db8n220qdnpx3jd40pwbtlf14w1b6m7
Bug: Repair hangs on Cassandra 4.0.11
Status: Resolved
Priority: Normal
Resolution: Invalid
Created: 31/Aug/23 13:12
Resolved: 31/Aug/23 13:57

When we run a full repair on Cassandra 4.0.11, it hangs and doesn't evolve. What we noticed was this message:

_{*}"{*}WARN [Messaging-OUT-/214.5.143.5:7001->/214.5.143.4:7001-LARGE_MESSAGES] 2023-08-18 07:02:54,862 OutboundConnection.java:488 - /214.5. 143.5:7001->/214.5.143.4:7001-LARGE_MESSAGES-[no-channel] *dropping message of type VALIDATION_RSP due to error*_
_{*}java.nio.channels.ClosedChannelException{*}: null"_ 

in one of the nodes, in the validate phase the merkle tree.

Besides that, I found some connection reset , but we do not know if there is a relation it.

_18 07:02:54,860 OutboundConnection.java:1056 - /214.5.143.5:7001->/214.5.143.4:7001-LARGE_MESSAGES-94900b4d channel closed by provider
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer_

 

I have uploaded logs from all nodes
Bug: Add missing features to 5.0 new features page
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Aug/23 10:05
Resolved: 30/Aug/23 14:26

The [page|https://cassandra.apache.org/doc/trunk/cassandra/new/index.html] listing the 5.0 new features is not mentioning multiple new features like trie sstables, unified compaction strategy, dynamic data masking, etc.