av9ash/GitBugs huggingface dataset:
Bug: Incorrect output in "nodetool status -r"
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 18/Nov/20 12:36
Resolved: 03/Feb/21 14:43

nodetool status -r not working well on C* 4,
 Version:
{code:java}
[root@foo001 ~]# nodetool version
ReleaseVersion: 4.0-beta3
{code}
Without resolving:
{code:java}
[root@foo001 ~]# nodetool status
Datacenter: V4CH
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address Load    Tokens  Owns(effective) Host ID                            Rack
UN  1.2.3.4 363.68 KiB  128     ?         92ae4c39-edb3-4e67-8623-b49fd8301b66 RAC1
UN  1.2.3.5 109.71 KiB  128     ?         d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2 RAC1
{code}
With resolving:
{code:java}
[root@foo001 ~]# nodetool status -r
Datacenter: V4CH
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address          Load  Tokens  Owns (effective)  Host ID  Rack
?N  foo001.tab.com   ?     128     ?                          RAC1
?N  foo002.tab.com   ?     128     ?                          RAC1
{code}

I only changed here IPs and hostnames.
Bug: Expose data dirs in ColumnFamilyStoreMBean 
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 11/Dec/20 15:14
Resolved: 05/Mar/21 10:08

As of now, I am not currently aware of any way how to get the information where a CF stores its data. While this might look like a detail, it is important for backup and restore purposes. Lets consider this workflow:


1) There is a keyspace "abc" with table "def", on disk, it will look like /my/data/abc/def-123445/...

2) I take a backup, all SSTables are put somewhere under path /backups/abc/def-12345/....

3) I delete this table by CQL, data ends up in "dropped"

4) I create this table again, but now it will generate other ID - like /my/data/abc/def-6789/...

5) I want to restore /backups/abc/def-123445/... but right now there are two structures - 
{code:java}
├── data
│   ├── abc
│   │   ├── def-12345...
│   │   │   ├── backups
│   │   │   └── snapshots
│   │   │       └── dropped-1607699318139-ghi
│   │   │           ├── manifest.json
│   │   │           ├── na-1-big-CompressionInfo.db
│   │   │           ├── na-1-big-Data.db
│   │   │           ├── na-1-big-Digest.crc32
│   │   │           ├── na-1-big-Filter.db
│   │   │           ├── na-1-big-Index.db
│   │   │           ├── na-1-big-Statistics.db
│   │   │           ├── na-1-big-Summary.db
│   │   │           ├── na-1-big-TOC.txt
│   │   │           └── schema.cql
│   │   └── def-6789...
│   │       ├── backups
│   │       ├── na-1-big-CompressionInfo.db
│   │       ├── na-1-big-Data.db
│   │       ├── na-1-big-Digest.crc32
│   │       ├── na-1-big-Filter.db
│   │       ├── na-1-big-Index.db
│   │       ├── na-1-big-Statistics.db
│   │       ├── na-1-big-Summary.db
│   │       └── na-1-big-TOC.txt
{code}

The question now is, what directory I should restore this to? Sure, into the "active" one, but I can not possibly know which one it is, because one of them is not used anymore and I do not want to do something very smelly like listing directories on disk and checking which one does not contain "dropped" directory ... Yes, one might use importing of SSTables - that is introduced in Cassandra 4, but for Cassandra 3, one can either copy it over or do hardlinks and refresh.

The second scenario is like this:

There is just one "active" table, no structure with "dropped" dir exists, but its id (that part after table name) differs. If I want to copy files over and refresh, I need to resolve this discrepancy and copy SSTables into a directory ending on id which differs from id from backup.

I was trying to get this information from CFSMB but that information is not exposed.

Is there any way how to retrieve via JMX where a table actually stores its data?

I have put this together: https://github.com/apache/cassandra/pull/850/files
Bug: JMH Reports in ci-cassandra.apache.org
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 29/Dec/20 13:35
Resolved: 08/Feb/21 10:40

Enable the  jenkins [jmh reporting plugin|https://plugins.jenkins.io/jmh-report/] and add a jenkins job to run the `ant microbench` target.

Patches
 - cassandra-builds: https://github.com/apache/cassandra-builds/compare/trunk...thelastpickle:mck/jenkins-microbench
 - cassandra: https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/trunk_jmh_report

Example results are:
 - CI JMH report: https://ci-cassandra.apache.org/job/Cassandra-devbranch-microbench/jdk=jdk_11_latest,label=cassandra/1/jmh-run-report/  (screenshot attached)

The value of these reports is limited, with different hardware jenkins agents and two executors on each agent. But visibility from the reports shows what is possible, how agents are different (and behave under saturation), and most importantly when jmh classes are broken (there are no unit tests on these benchmark tests).
Bug: Jenkins doesn't run high resource python dtests with novonode so missing some tests
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 21/Oct/20 20:30
Resolved: 28/Jan/21 11:21

In CASSANDRA-16220 it was found that a test is failing on trunk, and its because of the address to address with port changes; looking closer into it it was found that Jenkins ignores the novnode case which causes this test to get skipped.

We should enable novnode as well to match the rest of the pipeline, but also to handle some of the missing tests not currently run.
Bug: Removed unused static variable in TableParams
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 14/Dec/20 20:48
Resolved: 15/Dec/20 12:56

TableParams.DEFAULT is initialised with {{TableParams.builder().build(), }}but never used. It's mostly harmless, I encountered this while working with CQLSSTableWriter, and it triggered an error.

It was used only in CFMetaData, which was deleted in https://github.com/apache/cassandra/commit/af3fe39dcabd9ef77a00309ce6741268423206df
Bug: nodetool cleanup not working
Status: Open
Priority: Low
Created: 07/Dec/20 01:26

Hi,

 

After setting up the 3 clusters, I want to free up the disk on my first cluster since 

the previous still there.
This is the nodetool status before running the nodetool cleanup
!image-2020-12-07-09-23-02-002.png!

When I run the  nodetool cleanup 

!image-2020-12-07-09-23-33-788.png!

After I run the nodetool cleanup . I check if the node free up the spaces. This is the result

!image-2020-12-07-09-24-54-453.png!

It's seems that the nodetool cleanup not working well

cassandra version and java version

!image-2020-12-07-09-26-28-702.png!

Thanks.
Bug: when running python dtest and using JAVA_TOOL_OPTIONS to inject cassandra configurations, tests fail as they don't expect this flag in stderr
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 20/Oct/20 23:07
Resolved: 21/Oct/20 20:07

We have many flags that are set as system properties, and rather than changing the tests to support the different versions, we try to run the tests with different flag settings; this causes some tests to fail as stderr will contain something like the following

JAVA_TOOL_OPTIONS: -Dcassandra.consistent.simultaneousmoves.allow=true

We should update our tests to ignore this stderr message.
Bug: Add missing tests suites to CircleCI
Status: Resolved
Priority: Urgent
Resolution: Fixed
Created: 01/Nov/22 18:25
Resolved: 11/Feb/23 20:56

Large Python DTests (with/without vnodes), cqlshlib not tested in all branches and with all jdks; Java distributed tests not running with J8/J11 4.0+

A few bugs identified down the road have been fixed. More in the comments
Bug: Avoid schema mismatch problems on memtable API misconfiguration
Status: Resolved
Priority: Urgent
Resolution: Fixed
Created: 14/Nov/22 10:51
Resolved: 16/Nov/22 08:43

The memtable API requires a definition of the memtable configuration to exist in {{{}cassandra.yaml{}}}. If that is not the case when a schema change is received from another node, this can cause the node to reject the schema change and behave incorrectly.

To fix this, a node should only fail to construct a memtable when it is validating {{CREATE}} or {{ALTER}} statements, but should fall back to default when it receives invalid configuration selections in the schema.
Bug: Major Performance degradation of  Casandara 4.0.7   against Casandra 3.11.14
Status: Resolved
Priority: Urgent
Resolution: Invalid
Created: 08/Dec/22 16:28
Resolved: 21/Dec/22 07:35

Our application uses Casandra 3.11.x  and has lot of security vulnerabilities  which are addressed in 4.0.x.  So  we have upgraded the Casandra to 4.0.7 and  our performance tests have shown aorund 20% degradation  compare to 3.11.x    

We are now able to reproduce the same performance degradation using the standalone queries.   Here are the steps.

1. Expand Cassandra 3.11.14 tarball and 4.0.7 tarball to different folders
2. Import the attached data from the snapshot (mailboxes_snapshot.zip) into each Cassandra instance, see schema.cql for CQL for creating the required table and indexes before import
3. With CQLSH run the following query several times with TRACING ON and PAGING OFF against both versions of Cassandra:  select * from mailbox.mailboxes where mbx_id= 6c57da2e-7ddd-4984-be62-105415e6b48a;
4. Compare results

IWe ran the target query 30 times. Here's the average times to run the query:
3.11.14 - 19400.77
4.0.7 - 34906.03
Bug: Few nodes in cassandra 3.11 cluster is unable to connect to v4 protocol version
Status: Resolved
Priority: Urgent
Resolution: Invalid
Created: 10/Nov/22 16:32
Resolved: 10/Nov/22 16:36

We have a Apache Cassandra cluster with 36 nodes running on k8s, cqlsh client was installed in a separate namespace. cqlsh is downgrading version from v4 to v3 on some of the nodes but we have a case where all the nodes need to connect to v4, this is a requirement to set UNSET for avoiding null values to be converted to tombstones (only v4 is supported). 
{code:java}
cqlsh 10.1.2.3 --ssl --protocol-version=3
Connected to production at 10.1.2.3:9042.
[cqlsh 5.0.1 | Cassandra 3.11.5 | CQL spec 3.4.4 | Native protocol v3]
Use HELP for help. 
cqlsh>

cqlsh 10.1.2.3 --ssl --protocol-version=4
Connection error: ('Unable to connect to any servers', {'10.1.2.3': DriverException('ProtocolError returned from server while using explicitly set client protocol_version 4',)})  {code}
Bug: Use G1GC as default
Status: Resolved
Priority: High
Resolution: Fixed
Created: 09/Nov/22 22:47
Resolved: 20/Jan/23 12:19

G1GC is well battle tested now, and the recommended configuration for most users. CMS can work well on smaller heaps but requires more tuning, initially and over time. G1GC just works. CMS was deprecated in JDK 9.

Patch at https://github.com/apache/cassandra/compare/trunk...thelastpickle:cassandra:mck/7486/trunk
Bug: Support manual secondary index selection at the CQL level
Status: Open
Priority: Normal
Created: 13/Dec/22 14:30

It seems that  CQL do not have the ability of INDEX HINT , such as when we have more than one secondary index for some data table,And if the query hit the indexes, the index with more estimate rows will be returned. But if we want the query to be executed under our willing , we can use a hint like ,hint specified index or ignore the index.
At first I want to open a jira that to add the feature of hint for CQL ,But I think that may be  a gigantic task with no clear goal.
Besides I think there may need a DISSCUSS for the specific grammatical form before starting the work.
Bug: Add quick start development guide and docker ant targets
Status: Changes Suggested
Priority: Normal
Created: 10/Nov/22 21:59

I propose adding a new {{QUICKSTART.md}} guide that provides instructions to beginner contributors on how to:
a) Setup a development environment and IDE from scratch on WSL/Ubuntu/Debian environments
b) Run a simple unit test from the IDE
c) Create a simple patch that prints "Hello World" during Cassandra initialization
d) Test the patch on a local docker instance

A new {{docker/}} directory is added along with the guide with the following utilities:
a) Dockerfile: unoptimized vanilla docker image manifest suitable only for local dev testing.
b) build.sh: build a cassandra artifact and cassandra-test image.
c) start.sh: start local cassandra-test container.
d) stop.sh: stop local-cassandra test container.

A couple of follow-up improvements that come to mind:
 - Add ant targets to build/start/stop cassandra-test container.
 - Add OSX instructions.
 - More IDE setup instructions (ie. Eclipse/VScode).
 - A more involved Hello World example, perhaps creating a dummy VirtualTable with unit/dtests.
 - Update Dockerfile to point to build/ instead of creating an artifact.

A DEV mailing list message was sent to gather community feedback on this proposal: https://lists.apache.org/thread/gmb0q5yb2bzbs0cjw7mlcz1vwk2l23g2
Bug: Upgrade Netty on 5.0
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Oct/22 01:19
Resolved: 03/Aug/23 22:16

I haven't been able to identify from the Netty docs which was the lowest version where JDK17 was added but we are about 40 versions behind in netty 4 so I suspect we better update. 

-We need to consider there was an issue with class cast exceptions when building with JDK17 with newer versions of netty (the newest available in March 2022). For the record, we didn't see those when running CI on JDK8 and JDK11. We also need to carefully revise the changes between the netty versions. -->- CASSANDRA-18180

Upgrading will cover also a fix in netty that was discussed in [this|https://the-asf.slack.com/archives/CK23JSY2K/p1665567660202989] ASF Slack thread. 

CC [~benedict] , [~aleksey]
Bug: CEP-15: Multi-Partition Transaction CQL Support (Alpha -> v1)
Status: Open
Priority: Normal
Created: 09/Dec/22 19:49

With CASSANDRA-17719 complete, basic/alpha CQL support for Accord is now available in the [cep-15-accord|https://github.com/apache/cassandra/commits/cep-15-accord] branch. There are, however, a number of features and enhancements that we would still like to add before we make it available to users. Here is a summary of those items, broadly categorized:
h3. *High Priority*

*Error Messages and UX*
 - Fix the element/field names in result metadata to match the user query
 - Better error messages
 -- Append/subtracting to/from frozen list
 -- attempts to use reference in WHERE clause
 -- LET using a reserved word

*Testing*
 - Figure out the original intent of placeholder tests in {{AccordIntegrationTest}} (ex. {{{}acceptInvalidationTest(){}}})

*Features*
 - Support for returning the result of an {{{}IF{}}}/condition
 - Constant terms on the RHS of LET
 - Support for having a reference on both LHS and RHS of {{IF}} predicates
 - Support for {{{}CONTAINS{}}}, {{{}CONTAINS KEY{}}}, and {{IN}} (important for CAS parity)
 - Mixed conditional and unconditional updates
 - Arithmetic operations on references in {{{}INSERT{}}}/{{{}UPDATE{}}} (ex. INSERT INTO ks.tbl1 (k, c, v) VALUES (0, 0, a.v + 1)
 - Support TTLs specified in txn updates.
 - Allow partition reads in SELECT (i.e. only limit to one row on the right-hand side of LET statements)
- Aliased selectors in LET statements

*Tooling*
 - Remove {{reateepochunsafe}} references in {{JMXGetterCheckTest}}

 *Documentation*
 - JavaDoc for all new statement classes

h3. *Nice to Have*

*Domain Object Cleanup*
 - Given {{Selector}} is serializable, attempt to reuse it in {{TxnReference}} rather than hard coding data conversion cases
 - Separate current current {{LET}} statement from {{{}SelectStatement{}}}, likely as a new {{LetStatement}} that names a {{SelectStatement}}
 - Explore unifying {{Operation}} and {{ReferenceOperation}}
 - Include non-Row items of {{FilteredPartition}} in {{TxnData#estimatedSizeOnHeap()}}
 - Explore possibility of making {{Bound}} serializable and using it directly in {{TxnCondition}}
 - Change internal representation of {{TxnDataName}} to avoid {{String}} -> {{ByteBuffer}} conversions
 - Have {{{}TxnCondition{}}}/{{{}TxnUpdate{}}} support {{slice()}} to full take advantage of partial state replication
 - Update serialization logic to reflect CASSANDRA-18099

*Features*
 - Full JSON support
 - Nested UDT support

 *Documentation*
 - CQL language documentation and bump the CQL language specification version
 - JavaDoc for all new statement classes
 - Integrate {{demo.txt}} into broader documentation somewhere or remove
Bug: WEBSITE - Improve docs on allocate_tokens_for_local_replication_factor
Status: Open
Priority: Normal
Created: 23/Oct/22 10:05

A user asked on Stack Overflow [how {{allocate_tokens_for_local_replication_factor}} works|https://stackoverflow.com/questions/74169734/] and that they were not able to find information on it.

We have an opportunity to improve our docs so I've logged this ticket to get it done.
Bug: Centralize all snapshot operations to SnapshotManager and cache snapshots
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Dec/22 15:55
Resolved: 11/Dec/24 02:08

Everytime {{nodetool listsnapshots}} is called, all data directories are scanned to find snapshots, what is inefficient.

For example, fetching the {{org.apache.cassandra.metrics:type=ColumnFamily,name=SnapshotsSize}} metric can take half a second (CASSANDRA-13338).

This improvement will also allow snapshots to be efficiently queried via virtual tables (CASSANDRA-18102).

In order to do this, we should:
a) load all snapshots from disk during initialization
b) keep a collection of snapshots on {{SnapshotManager}}
c) update the snapshots collection anytime a new snapshot is taken or cleared
d) detect when a snapshot is manually removed from disk.
Bug: jackson-databind vulnerability CVE-2022-42003 CVE-2022-42004
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Oct/22 14:55
Resolved: 18/Oct/22 15:40

As the summary says, jackson-databind 2.13.2.2 which was upgraded for a vulnerability in CASSANDRA-17556 is vulnerable again.
Bug: Test Failure: UnavailableSocketErrors: Local socket binding errors showing up intermittently in tests
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Oct/22 19:36
Resolved: 22/Aug/23 10:59

{code}
ccmlib.common.UnavailableSocketError: Inet address 127.0.0.2:7000 is not available: [Errno 98] Address already in use; a cluster may already be running or you may need to add the loopback alias
{code}

This keeps popping up in a variety of tests:

[https://ci-cassandra.apache.org/job/Cassandra-4.1/181/testReport/dtest-offheap.paging_test/TestPagingData/test_paging_with_filtering_on_partition_key_on_static_columns/]

[https://ci-cassandra.apache.org/job/Cassandra-4.1/181/testReport/dtest-offheap.seed_test/TestGossiper/test_startup_after_ring_delay/]

[https://ci-cassandra.apache.org/job/Cassandra-4.1/181/testReport/dtest-offheap.replication_test/TestSnitchConfigurationUpdate/test_rf_collapse_property_file_snitch/]

This stems from ccm in common.py:
{code:java}
itf = ('127.0.0.2', 7000)

    def assert_socket_available(itf):
        info = socket.getaddrinfo(itf[0], itf[1], socket.AF_UNSPEC, socket.SOCK_STREAM)
        if not info:
            raise UnavailableSocketError("Failed to get address info for [%s]:%s" % itf)
    
        (family, socktype, proto, canonname, sockaddr) = info[0]
        s = socket.socket(family, socktype)
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    
        try:
>           s.bind(sockaddr)
E           OSError: [Errno 98] Address already in use {code}
Bug: Broken link in the documentation
Status: Triage Needed
Priority: Normal
Created: 30/Nov/22 13:08

At the page Cassandra
[Operating|https://cassandra.apache.org/doc/latest/cassandra/operating/index.html], the link to Snitches on the left menu-pane is broken for at least latest and 4.0 versions. As [~brandonwilliams] pointed out, the page which should be linked still exists at [https://cassandra.apache.org/doc/latest/cassandra/architecture/snitch.html].
Bug: Counter leader may respond success to coordinator even if counter replica response handler failed
Status: Triage Needed
Priority: Normal
Created: 07/Nov/22 14:06

In {{CounterMutationVerbHandler}}, the counter leader will respond success to coordinator in a {{callback}} which is used to create {{AbstractWriteResponseHandler}} for contacting counter replicas.

The {{AbstractWriteResponseHandler}} will trigger the {{callback}} in two cases:
 # when the handler meets consistency level requirement, it's correct to respond success to coordinator.
 # when the handler failed ({{blockFor + failures > replicas}}), it should respond failure instead of success to coordinator.


Another issue is that {{AbstractWriteResponseHandler#signal}} can be called multiple times if there are multiple failure responses or if there are more responses than CL requirement in {{DatacenterSyncWriteResponseHandler}}, so `callback` may be called multiple times, eg. multiple counter responses to coordinator, multiple view metrics updates.
Bug: Single slow node dramatically reduces cluster logged batch write throughput regardless of CL
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Dec/22 14:50
Resolved: 15/Oct/24 11:51

We issue writes to Cassandra as logged batches(RF=3, Consistency levels=TWO, QUORUM, or LOCAL_QUORUM)

 

On clusters of any size - a single extremely slow node causes a ~90% loss of cluster-wide throughput using batched writes.  We can replicate this in the lab via CPU or disk throttling.  I observe this in 3.11, 4.0, and 4.1.

 

It appears the mechanism in play is:

Those logged batches are immediately written to two replica nodes and the actual mutations aren't processed until those two nodes acknowledge the batch statements.  Those replica nodes are selected randomly from all nodes in the local data center currently up in gossip.  If a single node is slow, but still thought to be up in gossip, this eventually causes every other node to have all of its MutationStages to be waiting while the slow replica accepts batch writes.

 

The code in play appears to be:

See

[https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/locator/ReplicaPlans.java#L245].

  In the method filterBatchlogEndpoints() there is a

Collections.shuffle() to order the endpoints and a

FailureDetector.isEndpointAlive() to test if the endpoint is acceptable.

 

This behavior causes Cassandra to move from a multi-node fault tolerant system toa collection of single points of failure.

 

We try to take administrator actions to kill off the extremely slow nodes, but it would be great to have some notion of "what node is a bad choice" when writing log batches to replica nodes.