av9ash/GitBugs huggingface dataset:
Bug: Node disconnection during cassandra 4.0 upgrade from cassandra 3.11
Status: Triage Needed
Priority: Normal
Created: 16/Nov/22 09:17

We are running Cassandra 3.11.11. We are upgrading to 4.0.5.

The nodes use 11044 for its storage port.

 

Our upgrade process is the usual
 * Boot cassandra 4.0.5 using 3.11.11 data disk
 * Run upgradesstables

 

However, during the upgrade, randomly a node is unable to connect to other nodes in the cluster. This happens very intermittently and gets fixed on restart.

 

On further diagnosis, we found that the problematic node uses 7000 from some communication instead of the configured port

 
{noformat}
 InboundConnectionInitiator.java:127 - Listening on address: (node-1.dev/x.x.x.x:11044), nic: eth0, encryption: optionally encrypted(openssl)
OutboundConnection.java:1150 - node-1.dev/x.x.x.x:7000(/x.x.x.x:50424)->/y.y.y.y:11044-URGENT_MESSAGES-3c193918 successfully connected, version = 12, framing = LZ4, encryption = encryptedfactory=openssl;protocol=TLSv1.2;cipher=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384){noformat}
Notice the x.x.x.x:7000 in log line even though x.x.x.x is starting on 11044.

This gets fixed on restart.

 

The logs on reboot
{noformat}
 InboundConnectionInitiator.java:127 - Listening on address: (/x.x.x.x:11044), nic: eth0, encryption: optionally encrypted(openssl)
InboundConnectionInitiator.java:464 - /y.y.y.y:11044(/y.y.y.y:40656)->/x.x.x.x:11044-URGENT_MESSAGES-cade4755 messaging connection established, version = 12, framing = CRC, encryption = encrypted(factory=openssl;protocol=TLSv1.2;cipher=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384)
OutboundConnection.java:1150 - /x.x.x.x:11044(/x.x.x.x:53316)->/y.y.y.y:11044-URGENT_MESSAGES-92d99f23 successfully connected, version = 12, framing = LZ4, encryption = encrypted(factory=openssl;protocol=TLSv1.2;cipher=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384)
 {noformat}
 

Notice the Outbound connection log line has x.x.x.x:11044 this time.

 

This issue is very random.

 

Looks to be a bug. Is there a fix for this? Are we missing some steps during the upgrade?

 

Some relevant sections of cassandra.yaml on both the cassandra 3.x and 4.x

 
{noformat}
storage_port: 11044
ssl_storage_port: 11044
server_encryption_options:
    internode_encryption: all
    keystore: ---------
    keystore_password: -------
    truststore: ---------
    truststore_password: ---------
    protocol: TLSv1.2
    algorithm: PKIX
    store_type: PKCS12
    cipher_suites:
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    require_client_auth: true {noformat}
Bug: Cqlsh Cython failure on 4.0
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Nov/22 06:33
Resolved: 11/Nov/22 10:03

Cython seems to be failing as [~e.dimitrova]  found out [here.|https://ci-cassandra.apache.org/job/Cassandra-4.0-cqlsh-tests/456/cython=yes,jdk=jdk_1.8_latest,label=cassandra/console]
Bug: Protect against TableId mutability when system tables are created
Status: Triage Needed
Priority: Normal
Created: 11/Nov/22 18:44

In some situations, it may be possible to have non-deterministic CFIDs for system tables. This can happen if system tables were manually created prior to an upgrade (2.1->3.0, for example).

If non-deterministic CFIDs exist for system tables in system_schema.tables, then the cluster can operate correctly, as long as all members use the same CFIDs. This is because CFIDs are serialized as part of ReadCommands, which need to agree between coordinator and replica.

If a non-deterministic ID exists in system_schema.tables, then the instance upgrades to a Cassandra version including CASSANDRA-17044, the upgraded instance will move from the non-deterministic CFIDs to deterministic, causing "Unknown CF" exceptions for queries that span mixed-mode coordinators and replicas. In the case of AuthKeyspace, this can prevent users from authenticating and break all coordinated traffic to the upgraded node.

When system tables are set up, such as StorageService.doAuthSetup, the in-memory deterministic ID from AuthKeyspace.metadata() is used: [https://github.com/apache/cassandra/blob/cbafc28c5aec422e3b953eaa2f03a31d554312da/src/java/org/apache/cassandra/service/StorageService.java#L1242]

Then SchemaTransformations.updateSystemKeyspace will use the deterministic CFID calculated in-memory: [https://github.com/apache/cassandra/blob/cbafc28c5aec422e3b953eaa2f03a31d554312da/src/java/org/apache/cassandra/schema/SchemaTransformations.java#L193]

 

For clusters running without any manual intervention, there is no problem. But this does point to an opportunity to add more protection to our schema evolution code paths, to prevent CFID mutability from happening in the future.
Bug: nodetool resetlocalschema does not sync schema
Status: Open
Priority: Normal
Created: 19/Oct/22 03:44

Steps to reproduce:
{code:java}
$ ccm create -n 2 -v 3.11.13 -s resettest
$ ccm node1 cqlsh -e "CREATE KEYSPACE test WITH replication = {'class':'NetworkTopologyStrategy', 'datacenter1': 2}"
$ ccm node1 cqlsh -e "CREATE TABLE test.messages(id int primary key, message text)"
$ ccm node1 nodetool describecluster

Cluster Information:
        Name: resettest
        Snitch: org.apache.cassandra.locator.SimpleSnitch
        DynamicEndPointSnitch: enabled
        Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
        Schema versions:
                5172094b-dee2-3967-b5f0-dfd81d572a74: [127.0.0.1, 127.0.0.2]

$ ccm node2 nodetool resetlocalschema
$ ccm node2 cqlsh -e 'describe full schema'

$ find ~/.ccm/resettest/node2/data0/system_schema/ -maxdepth 2 -type f
$ ccm node2 nodetool describecluster

Cluster Information:
        Name: resettest
        Snitch: org.apache.cassandra.locator.SimpleSnitch
        DynamicEndPointSnitch: enabled
        Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
        Schema versions:
                5172094b-dee2-3967-b5f0-dfd81d572a74: [127.0.0.1]

                59adb24e-f3cd-3e02-97f0-5b395827453f: [127.0.0.2]

$ ccm node2 stop
$ ccm node2 start
$ ccm node2 nodetool describecluster

Cluster Information:
        Name: jpmctest
        Snitch: org.apache.cassandra.locator.SimpleSnitch
        DynamicEndPointSnitch: enabled
        Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
        Schema versions:
                5172094b-dee2-3967-b5f0-dfd81d572a74: [127.0.0.1, 127.0.0.2]

{code}
Its not resyncing the schema until after restarting the node.
Bug: Tune parallelism for circleci jobs
Status: Triage Needed
Priority: Normal
Created: 02/Nov/22 19:12

We should tune the parallel parameters for our circleci config to be more optimal. From the email / slack conversations on the topic:

{code}
> def java_parallelism(src_dir, kind, num_file_in_worker, include = lambda a, b: True):
>     d = os.path.join(src_dir, 'test', kind)
>     num_files = 0
>     for root, dirs, files in os.walk(d):
>         for f in files:
>             if f.endswith('Test.java') and include(os.path.join(root, f), f):
>                 num_files += 1
>     return math.floor(num_files / num_file_in_worker)
> 
> def fix_parallelism(args, contents):
>     jobs = contents['jobs']
> 
>     unit_parallelism                = java_parallelism(args.src, 'unit', 20)
>     jvm_dtest_parallelism           = java_parallelism(args.src, 'distributed', 4, lambda full, name: 'upgrade' not in full)
>     jvm_dtest_upgrade_parallelism   = java_parallelism(args.src, 'distributed', 2, lambda full, name: 'upgrade' in full)
{code}

bq. `TL;DR - I find all test files we are going to run, and based off a pre-defined variable that says “idea” number of files per worker, I then calculate how many workers we need.  So unit tests are num_files / 20 ~= 35 workers.  Can I be “smarter” by knowing which files have higher cost?  Sure… but the “perfect” and the “average” are too similar that it wasn’t worth it...`

Quoting [~dcapwell]
Bug: Add flag to generate circle config w/out requiring click approval to run
Status: Triage Needed
Priority: Normal
Created: 02/Nov/22 19:43

Currently the config yaml generated by .circleci requires approval (i.e. human interaction) to run. We should add a flag that generates a config yaml that automatically runs on push.

Example of what this looks like can be found here:
[Branch w/circle config generated by David's script|https://github.com/josh-mckenzie/cassandra/commit/97be3d18784c75730e244b11506728c0927df65e]
[CI pipeline running w/out approval in circle|https://app.circleci.com/pipelines/github/josh-mckenzie/cassandra/328/workflows/0b3bad99-7e90-43e9-835f-3cc70ec799a5]
Bug: BLOG - Update author for Higher Availability, Backpressure blog posts
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Nov/22 12:51
Resolved: 04/Nov/22 21:43

Update author for the following blog posts:

* [Improving Apache Cassandra’s Front Door and Backpressure|https://cassandra.apache.org/_/blog/Improving-Apache-Cassandras-Front-Door-and-Backpressure.html]
* [Even Higher Availability with 5x Faster Streaming in Cassandra 4.0|https://cassandra.apache.org/_/blog/Even-Higher-Availability-with-5x-Faster-Streaming-in-Cassandra-4.html]
Bug: WEBSITE - Cleanup sub-pages of What's New section, fix version numbers
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 04/Nov/22 04:37
Resolved: 04/Nov/22 12:15

The [What's New|https://cassandra.apache.org/doc/latest/cassandra/new/index.html] section of the website needs a bit of TLC:
 * replace version number on landing page ({{new/index.adoc}})
 * cleanup sub-pages in {{/doc/latest/cassandra/new/}}

This work needs to be done for both {{trunk}} and {{cassandra-4.1}} branches.

 !c18015-01-new_index.png|width=300! 
 !c18015-02-subpages.png|width=300!
Bug: WEBSITE - November 2022 blog "Apache Cassandra Changelog #20"
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Nov/22 17:18
Resolved: 04/Nov/22 03:43

This ticket is to capture the work associated with publishing the November 2022 blog "Apache Cassandra Changelog #20"

If this blog cannot be published by the November 3, 2022 publish date, please contact me, suggest changes, or correct the date when possible in the pull request for the appropriate time that the blog will go live (on both the blog page card and the post).
Bug: Parameterize and generate .circleci/config-2_1.yml rather than manually maintain
Status: Triage Needed
Priority: Normal
Created: 02/Nov/22 19:53

Reference email thread [DISCUSS] Potential circleci config and workflow changes, specifically David's message [here|https://lists.apache.org/thread/059c4s6v8q9rl5lmhkj8xhfmwdj2fy2d]

To quote him:
bq. CI is a combinatorial problem, we need to run all jobs for all JDKs, vnode on/off, cdc on/off, compression on/of, etc…. But this is currently controlled and fleshed out by humans who want to add new jobs. We should move away from maintaining .circleci/config-2_1.yml and instead auto-generate it. Simple example of this problem is jdk11 support… we run a subset of tests on jdk11 and say its supported… will jdk17 have the same issue? Will it be even less tests? Why does the burden lie on everyone to “do the right thing” when all they want is a simple job?

Derek Chen-Becker has also started a more granular email thread with a potential design alternative along these lines available [here|https://lists.apache.org/thread/mvql1p5y2j7so18427zcg4zxc9vzl7l3].

Based on the output of those discussions, we should reflect the conclusion here and track that work.
Bug: WEBSITE - Add a dedicated Events page
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Oct/22 12:45
Resolved: 28/Oct/22 23:52

This is a proposed "Events" section on the Cassandra website, listing (approved) events as small cards with a picture, a short description and a link to the external event website.

The motivation behind this idea, which was discussed briefly on the [cassandra-website|https://issues.apache.org/jira/browse/CASSANDRA-website] Slack channel as well, is twofold: on one hand, it is desirable to provide this information to website visitors, which helps building the community through dedicated events; on the other hand, the Cassandra blog is preferrably not 'clogged' with too many event announcements so that it can retain its primary "technical" role.

I tried to create a page that blends in with the rest of the website experience, but while doing so I had to make some choices. In particular, I propose an icon for the section (a circus tent) that I hope conveys a "playful but not silly" message; moreover, I added the "Events" page to the "Community" navbar menu to avoid adding another menu, even though all other "Community" menu items are from the community page.
Also I am aware that the buttons alignment on the cards is imperfect. I blame my (very limited) css knowledge, this is of course something to improve if ever this draft gets accepted.

 

The PR (which probably should be taken as a WIP, for the aforementioned button alignment issues if nothing else) is here: https://github.com/apache/cassandra-website/pull/185
Bug: WEBSITE - Fix links on dev/patches page, other minor fixes
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Oct/22 13:43
Resolved: 28/Oct/22 10:04

The [https://cassandra.apache.org/_/development/patches.html] page has several places where references to other pages are not actual links, but are formatted as fixed-width text. For example:

Code must follow the {{code_style}} convention

instead of

Code must follow the [code style|https://cassandra.apache.org/_/development/code_style.html] convention

These should be fixed to be actual links
Bug: BLOG - Cassandra Days Asia - Hanoi, Jakarta, Singapore
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Oct/22 06:36
Resolved: 26/Oct/22 14:23

Blog about Cassandra Days in Hanoi, Jakarta + Singapore.
 * [https://www.datastax.com/events/cassandra-day-hanoi]
 * [https://www.datastax.com/events/cassandra-day-jakarta]
 * [https://www.datastax.com/events/cassandra-day-singapore]

  !cdays-asia-20221108-15.png|width=300!
Bug: CVEs in Cassandra 4.0.5
Status: Resolved
Priority: Normal
Resolution: Invalid
Created: 26/Oct/22 04:14
Resolved: 26/Oct/22 05:02

Hi Team,

We request you to fix below CVE's in Cassandra 4.0.5 

 

[Sonatype-2021-0234, sonatype-2019-0992, CVE-2021-37136, sonatype-2021-0789|https://git.soma.salesforce.com/pages/Infrastructure-Security/ast.github.io/sonatype-2021-0234.html]
Bug: BLOG - Cassandra Day Santa Clara/Bellevue/Houston + Wakanda Forever
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Oct/22 02:25
Resolved: 22/Oct/22 09:37

Promote the November 10 Cassandra Day in Santa Clara/Bellevue/Houston – [https://www.datastax.com/november10].

 !cday-20221110-wakanda_forever.png|width=300!
Bug: Bad format for JMX metrics page in documentation 
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Oct/22 12:34
Resolved: 20/Oct/22 10:09

Hi,

The page metrics in documentation is pretty hard to read :

[https://cassandra.apache.org/doc/4.1/cassandra/operating/metrics.html]

 

Here it is the PR that correct that : [https://github.com/apache/cassandra/pull/1909]
Bug: Dropwizard metrics output in Prometheus format
Status: Triage Needed
Priority: Normal
Created: 18/Oct/22 00:25

in Apache Cassandra :

we already have Dropwizard metrics module pre-integrated. I wrote about it on my GitHub : https://github.com/sarma1807/Prometheus-Grafana-Cassandra/tree/main/01_Cassandra_metrics-reporter

we should have a feature where Cassandra Dropwizard module can directly write these metrics into Prometheus format.

then we can feed those metrics directly to Prometheus for metrics collection.

we do not have to expose this as a webservice - instead of writing just metrics in plain text ... it should write in Prometheus format into a file on each server.
Bug: Add compaction type output result for nodetool compactionhistory
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 21/Nov/22 05:21
Resolved: 17/Jan/23 08:31

If we want to see whether we have made a compaction and what kind of compaction we have done for this node, we may go to see the compaction_history system table for some deftails or use nodetool compactionhistory command , But I found that the table do not specify the compaction type so does the compactionhistory command too, like index build , compaction type, clean or scrub for this node. So I think may be it is need to add a type of compaction column to specify the compaction tpe for system.compaction_history and so we can get the type of compaction through system.compaction_history table or nodetool compactionhistory to see whether we have made a major compact for this node  .:)
Bug: WEBSITE - Update What's New Page for 4.1 and trunk
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 19/Oct/22 19:15
Resolved: 04/Nov/22 13:18

[https://cassandra.apache.org/doc/4.1/cassandra/new/index.html]

The page still says 4.0 and lists 4.0 new developments.

We need to point to 4.1 and list latest developments blog posts, docs; probably we can also link NEWS.txt and CHANGES.txt
Bug: Do not spam the logs with MigrationCoordinator not able to pull schemas
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 06/Dec/22 12:53
Resolved: 20/Dec/22 10:21

When a node is joining a cluster, there is this output upon startup:

{code}
cassandra_node_6  | INFO  [GossipStage:1] 2022-12-06 12:48:07,187 Gossiper.java:1413 - Node /172.19.0.5:7000 is now part of the cluster
cassandra_node_6  | WARN MigrationCoordinator.java:650 - Can't send schema pull request: node /172.19.0.5:7000 is down.
cassandra_node_6  | WARN MigrationCoordinator.java:650 - Can't send schema pull request: node /172.19.0.5:7000 is down.
cassandra_node_6  | WARN MigrationCoordinator.java:650 - Can't send schema pull request: node /172.19.0.5:7000 is down.
cassandra_node_6  | WARN MigrationCoordinator.java:650 - Can't send schema pull request: node /172.19.0.5:7000 is down.
cassandra_node_6  | WARN MigrationCoordinator.java:650 - Can't send schema pull request: node /172.19.0.5:7000 is down.
cassandra_node_6  | WARN MigrationCoordinator.java:650 - Can't send schema pull request: node /172.19.0.5:7000 is down.
cassandra_node_6  | WARN MigrationCoordinator.java:650 - Can't send schema pull request: node /172.19.0.5:7000 is down.
{code}

This is there for a lot of already existing nodes. You got the idea. This log is misleading, it indeed can not pull requests because "node is down" but it is not down, it just thinks it is because Gossiper has not had a chance to receive any gossip about these nodes _yet_.

I put there more logs and it writes this:

{code}
 MigrationCoordinator.java:655 - Can't send schema pull request: node /172.19.0.5:7000 is down: NORMAL, isAlive: false
{code}

When I do this:

{code}
        if (!gossiper.hasEndpointState(endpoint))
            return;

        if (!gossiper.isAlive(endpoint))
        {
            EndpointState endpointStateForEndpoint = gossiper.getEndpointStateForEndpoint(endpoint);
            String status = Gossiper.getGossipStatus(endpointStateForEndpoint);
            logger.warn("Can't send schema pull request: node {} is down: {}, isAlive: {}", endpoint, status, endpointStateForEndpoint.isAlive());
            callback.onFailure(endpoint, RequestFailureReason.UNKNOWN);
            return;
        }
{code}

So it is in NORMAL but it is not alive yet which is quite strange.

The fix is to still return prematurely but we would not skip the logging on WARN only in case isAlive is false and status is _not_NORMAL. We would however still log on TRACE at least.

(1) https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/schema/MigrationCoordinator.java#L648-L653
Bug: Nodetool Compact set the compaction type incorrectly
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 16/Nov/22 16:01
Resolved: 07/Dec/22 11:27

When using nodetool compactionstats to see what does the c*'s compactions are doing ,the output has got a column named "compaction type", but It seem that major compaction and minor compaction 's type are all name Compaction, after read the code I found that may be the the MAJOR_COMPACTION OperationType is not setted into AbstractCompactionTask  
at this method : CompactionStrategyManager -> getMaximalTasks .When we peform a major compact without any arguments we will got this execute path : 

{code:java}
// Some comments here
Compact.java : probe.forceKeyspaceCompaction(splitOutput, keyspace, tableNames);
--->
 ColumnFamilyStore.java :  cfStore.forceMajorCompaction(splitOutput);
---> 
CompactionManager.java : submitMaximal(cfStore, gcBefore, splitOutput, OperationType.MAJOR_COMPACTION); 
{code} 
Unfortunately OperationType.MAJOR_COMPACTION is not rightly setted.


see the picture on the right I perform a major compact , and on the left the compactionstats show the type is only Compaction ;
I think it is import for us to know wether the task is a major or a minor .
Bug: Remove test-memory target
Status: Open
Priority: Low
Created: 03/Oct/22 21:02

I noticed the {{test-memory}} target which seems to have been added in CASSANDRA-15388. Unfortunately, since then _java-allocation-instrumenter_ which was added as a dependency was not really maintained by the authors. I think this is the official repo - [https://github.com/google/allocation-instrumenter] and last commit was to support Java 8 in 2020. The test itself was also not added to CI.

Confirmed with [~dcapwell] in Slack

CC [~bdeggleston] and [~benedict]
Bug: Correct Cassandra Website Testing page
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 01/Nov/22 23:10
Resolved: 11/Nov/22 09:09

1) {{test/burn}} - to run them, first build the uber jar with {{{}ant burn-test-jar{}}}, and then to run the tests call {{ant test-burn}} or {{ant burn-testsome}}

This is incorrect. We do not need the uber jar in order to run the tests. Also that target exists only on newer versions where we see in build.xml 
{code:java}
 <!-- Build a self-contained jar for e.g. remote execution; not currently used for running burn tests with this build script -->{code}
2) 
 
{code:java}
ant testsome -Dtest.name=<TestClassName> -Dtest.method=<testMethodName>{code}
should be changed to
{code:java}
ant testsome -Dtest.name=<TestClassName> -Dtest.methods=<testMethodName>{code}
Bug: Hints not stored after node goes down for the second time
Status: Resolved
Priority: Urgent
Resolution: Fixed
Created: 26/Mar/24 17:30
Resolved: 05/Apr/24 09:53

I have a scenario where a node goes down, hints are recorded on the second node and replayed, as expected. If the first node goes down for a second time and time span between the first time it stopped and the second time it stopped is more than the max_hint_window then the hint is not recorded, no hint file is created, and the mutation never arrives at the node after it comes up again.

I have debugged this and it appears to due to the way hint window is persisted after https://issues.apache.org/jira/browse/CASSANDRA-14309

The code here: [https://github.com/apache/cassandra/blame/cassandra-4.1/src/java/org/apache/cassandra/service/StorageProxy.java#L2402] uses the time stored in the HintsBuffer.earliestHintByHost map.  This map is based on the UUID of the host, but this does not seem to be cleared when the node is back up, and I think this is what is causing the problem.

 

This is in cassandra 4.1.5