av9ash/GitBugs huggingface dataset:
Bug: Resolve the oldest hints just from descriptors and current writer if available
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Apr/24 11:28
Resolved: 02/May/24 21:54

We should not resolve hints from buffers as these are just very short-lived / transient data structures and resolving "the oldest hints" from buffers too just does not make sense when we should just look into descriptors and current writer's descriptor at most.
Bug: flaky IntersectFilteringQueryTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/May/24 09:13
Resolved: 02/May/24 18:48

I start to see flaky IntersectFilteringQueryTest both in 5.0 and trunk.

 
{code:java}
      org.apache.cassandra.distributed.test.guardrails.IntersectFilteringQueryTest shouldNotWarnOrFailOnIndexQuery
        com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency QUORUM (2 responses were required but only 1 replica responded, 1 failed)
                at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:180)
                at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:30)
                at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:35)
                at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:293)
                at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:58)
                at org.apache.cassandra.distributed.test.guardrails.GuardrailTester.executeViaDriver(GuardrailTester.java:92)
                at org.apache.cassandra.distributed.test.guardrails.IntersectFilteringQueryTest.shouldNotWarnOrFailOnIndexQuery(IntersectFilteringQueryTest.java:115)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
                at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        Caused by: com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure during read query at consistency QUORUM (2 responses were required but only 1 replica responded, 1 failed)
                at com.datastax.driver.core.exceptions.ReadFailureException.copy(ReadFailureException.java:192)
                at com.datastax.driver.core.Responses$Error.asException(Responses.java:181)
 {code}
 

It most probably needs a little bit more love and run it through multiplexer maybe?
Bug: Add java version to prepare_release.sh human check step
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/May/24 14:58
Resolved: 02/May/24 17:31

With the many versions of java one may need to float these days, it would be nice for prepare_release.sh to show the version you are using when it asks the "Is this what you want?" question showing the latest commit, that way there is a chance to correct things before it tags.
Bug: Test failure: org.apache.cassandra.db.ImportTest flakiness
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Apr/24 18:30
Resolved: 02/May/24 09:24

As discovered on CASSANDRA-19401, the tests in this class are flaky, at least the following:
 * testImportCorruptWithoutValidationWithCopying
 * testImportInvalidateCache
 * testImportCorruptWithCopying
 * testImportCacheEnabledWithoutSrcDir

[https://app.circleci.com/pipelines/github/instaclustr/cassandra/4199/workflows/a70b41d8-f848-4114-9349-9a01ac082281/jobs/223621/tests]
Bug: Concurrent equivalent schema updates lead to unresolved disagreement
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Apr/24 01:11
Resolved: 30/Apr/24 22:04

As part of CASSANDRA-17819 a check for empty schema changes was added to the updateSchema. This only looks at the _logical_ schema difference of the schemas, but the changes made to the system_schema keyspace are the ones that actually are involved in the digest.

If two nodes issue the same CREATE statement the difference from the keyspace.diff would be empty but the timestamps on the mutations would be different, leading to a pseudo schema disagreement which will never resolve until resetlocalschema or nodes being bounced.

Only impacts 4.1

test and fix : https://github.com/clohfink/cassandra/commit/ba915f839089006ac6d08494ef19dc010bcd6411
Bug: Upgrade deprecated asyncore to asyncio
Status: Open
Priority: Normal
Created: 01/May/24 03:23

The asyncore module is deprecated and has been removed in Python 3.12. To support 3.12, we need to upgrade to use asyncio.

DEBUG    test_cqlsh:run_cqlsh.py:194 read '    from cqlshlib.cqlshmain import main\r\n  File "pylib/cqlshlib/cqlshmain.py", line 37, in <module>\r\n    from cassandra.cluster import Cluster\r\n  File "lib/cassandra-driver-internal-only-3.29.0.zip/cassandra-driver-3.29.0/cassandra/cluster.py", line 173, in <module>\r\ncassandra.DependencyException: Unable to load a default connection class\r\nThe following exceptions were observed: \r\n - The C extension needed to use libev was not found.  This probably means that you didn\'t have the required build dependencies when installing the driver.  See [http://datastax.github.io/python-driver/installation.html#c-extensions] for instructions on installing build dependencies and building the C extension.\r\n - Unable to import asyncore module.  Note that this module has been removed in Python 3.12 so when using the driver with this version (or anything newer) you will need to use one of the other event loop implementations.\r\n' from subproc
Bug: Queries are not visible to the "system_views.queries" virtual table at the coordinator level
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 19/Apr/24 20:07
Resolved: 01/May/24 19:50

There appears to be a hole in the implementation of CASSANDRA-15241 where {{DebuggableTasks}} at the coordinator are not preserved through the creation of {{FutureTasks}} in {{TaskFactory}}. This means that {{QueriesTable}} can't see them when is asks {{SharedExecutorPool}} for running tasks. It should be possible to fix this in {{TaskFactory}} by making sure to propagate any {{RunnableDebuggableTask}} we encounter. We already do this in {{toExecute()}}, but it also needs to happen in the relevant {{toSubmit()}} method(s).
Bug: Accord: NPE in RangeDeps.forEach
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/May/24 15:53
Resolved: 01/May/24 16:12

{code}
java.lang.NullPointerException
	accord.primitives.RangeDeps.visitTxnIdxsForRangeIndex(RangeDeps.java:249)
	accord.utils.CheckpointIntervalArray.forEach(CheckpointIntervalArray.java:219)
	accord.utils.CheckpointIntervalArray.forEach(CheckpointIntervalArray.java:127)
	accord.utils.CheckpointIntervalArray.forEach(CheckpointIntervalArray.java:97)
	accord.primitives.RangeDeps.forEach(RangeDeps.java:178)
	accord.primitives.RangeDeps.forEach(RangeDeps.java:202)
	accord.primitives.RangeDeps.forEach(RangeDeps.java:281)
	accord.primitives.RangeDeps.forEach(RangeDeps.java:273)
	org.apache.cassandra.service.accord.AccordSafeCommandStore.registerHistoricalTransactions(AccordSafeCommandStore.java:190)
{code}
Bug: SystemKeyspace CFS flushing blocked by unrelated keyspace flushing/compaction
Status: Open
Priority: Normal
Created: 29/Apr/24 16:38

There is a single post flush thread and that thread processes tasks in order and one of those tasks can be a memtable flush for an unrelated keyspace/cfs, and that memtable flush can be blocked by slow IntervalTree building and racing with compactors to try and build an interval tree.

Unless there is a requirement for ordering we probably want to loosen this to the actual ordering requirement so that problems in one keyspace can’t effect another.

SystemKeyspace and Gossip in particular cause lots of weird problems like nodes marking each other down because Gossip can’t process nodes being removed (blocking flush each time in SystemKeyspace.removeNode)

A very simple fix here might be to queue the post flush task at the same time as the flush in a per CFS queue, and then submit the task only once the flush is completed.

If flushes complete out of order the queue will still ensure their completions are processed in order.
Bug: Avoid ConcurrentModificationException when removing metrics from CassandraMetricsRegistry
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Apr/24 16:23
Resolved: 30/Apr/24 16:53

Multiple threads can modify the lists in {{CassandraMetricsRegistry}} {{ALIASES}} map values. We have seen this CME when removing metrics, but could possibly affect other paths using those lists

Adding a small delay (10ms) when map:ing to {{getMetricName}} in {{CassandraMetricsRegistry.remove(name)}} makes this reproduce when running {{TransientRangeMovementTest.testLeave}}

Caused by CASSANDRA-14572
Bug: get/set guardrails configuration via CQL
Status: Needs Committer
Priority: Normal
Created: 11/Apr/24 09:54

In CASSANDRA-19552, there is a patch which modifies guardrails via nodetool and via reflection.

I think that it will be better if we do that via CQL.

So far, I have this:

{code}
cqlsh> select * from system_views.guardrails_threshold ;

 name                          | fail  | warn
-------------------------------+-------+-------
               collection_size |     0 |     0
             column_value_size |    -1 |    -1
             columns_per_table |    -1 |    -1
                fields_per_udt |    -1 |    -1
   in_select_cartesian_product |    -1 |    -1
          items_per_collection |    -1 |    -1
                     keyspaces |    -1 |    -1
  materialized_views_per_table |    -1 |    -1
    maximum_replication_factor |    -1 |    -1
             maximum_timestamp |     0 |     0
    minimum_replication_factor |    -1 |    -1
             minimum_timestamp |     0 |     0
                     page_size |    -1 |    -1
      partition_keys_in_select |    -1 |    -1
                partition_size | 40960 | 20480
          partition_tombstones |    -1 |    -1
 sai_sstable_indexes_per_query |    -1 |    32
   secondary_indexes_per_table |    -1 |    -1
                        tables |    -1 |    -1
             vector_dimensions |    -1 |    -1
{code}

{code}
cqlsh> update system_views.guardrails_threshold SET warn = 16, fail = 20 where name = 'sai_sstable_indexes_per_query';
{code}

{code}
cqlsh> select * from system_views.guardrails_threshold where name = 'sai_sstable_indexes_per_query';

 name                          | fail | warn
-------------------------------+------+------
 sai_sstable_indexes_per_query |   20 |   16
{code}

{code}
INFO  [Native-Transport-Requests-1] 2024-04-11 11:51:24,483 GuardrailsOptions.java:998 - Updated sai_sstable_indexes_per_query_warn_threshold from 32 to 16
INFO  [Native-Transport-Requests-1] 2024-04-11 11:51:24,483 GuardrailsOptions.java:998 - Updated sai_sstable_indexes_per_query_fail_threshold from -1 to 20
{code}

{code}
cqlsh> update system_views.guardrails_threshold SET warn = 10, fail = 5 where name = 'sai_sstable_indexes_per_query';
InvalidRequest: Error from server: code=2200 [Invalid query] message="The warn threshold 10 for sai_sstable_indexes_per_query_warn_threshold should be lower than the fail threshold 5"
{code}

We can easily extend this to EnableFlag guardrails for which we would dedicate a separate table (system_views.guadrails_flag).

Other guardrail types are being investigated.
Bug: RepairDigestTrackingTest#testLocalDataAndRemoteRequestConcurrency timing out on QUORUM read
Status: Triage Needed
Priority: Normal
Created: 29/Apr/24 15:50

This test doesn't seem to have any trouble passing locally in trunk, but on cep-15-accord, it reliably times out. From a couple minutes of debugging in {{ReadCallback#onResponse()}}, I think the proximate cause looks like a {{QUORUM}} read w/ 3 nodes getting back 2 responses, but both are digest responses. It seems like one should actually have data. In any case, this manifests as a timeout, because even though we have the right number of responses, we don't signal before the timeout. Also, this still fails even w/ speculative retries disabled (i.e. set to NEVER) in the test table.
Bug: Standalone jenkinsfile first round bug fixes
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Apr/24 13:59
Resolved: 26/Apr/24 09:41

A few follow up improvements and bug fixes for the standalone jenkinsfile.

- add at top a list of test failures in ci_summary.html
- -docker scripts always try to login (as base images need to be pulled too)-
- move simulator-dtests to large containers (they need 8g just heap)
- in ubuntu2004_test.docker make sure /home/cassandra exists and has correct perms (from marcuse)
- persist the jenkinsfile parameters from run to run (important for the post-commit jobs to keep their non-default branch and profile values) (was CASSANDRA-19536)
- increase jvm-dtest splits from 8 to 12
- when on ci-cassandra, replace use of copyArtifacts in Jenkinsfile generateTestReports() with manual wget of test files, allowing the summary phase to be run on any agent (copyArtifact would take >4hrs otherwise) (was INFRA-25694)
- copy ci_summary.html and results_details.tar.xz to nightlies
Bug: JSON encoded timestamp value does not always match non-JSON encoded value
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Apr/24 10:53
Resolved: 22/Apr/24 23:32

Description:

"SELECT JSON ..." and "toJson(...)" on Cassandra 4.1.4 produces different date than "SELECT ..."  for some timestamp type values.

 

Steps to reproduce:
{code:java}
$ sudo docker pull cassandra:4.1.4
$ sudo docker create --name cass cassandra:4.1.4
$ sudo docker start cass
$ # wait for the Cassandra instance becomes ready
$ sudo docker exec -ti cass cqlsh
Connected to Test Cluster at 127.0.0.1:9042
[cqlsh 6.1.0 | Cassandra 4.1.4 | CQL spec 3.4.6 | Native protocol v5]
Use HELP for help.
cqlsh> create keyspace test WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
cqlsh> use test;
cqlsh:test> create table tbl (id int, ts timestamp, primary key (id));
cqlsh:test> insert into tbl (id, ts) values (1, -13767019200000);
cqlsh:test> select tounixtimestamp(ts), ts, tojson(ts) from tbl where id=1;
 system.tounixtimestamp(ts) | ts                              | system.tojson(ts)
----------------------------+---------------------------------+----------------------------
            -13767019200000 | 1533-09-28 12:00:00.000000+0000 | "1533-09-18 12:00:00.000Z"
(1 rows)
cqlsh:test> select json * from tbl where id=1;
 [json]
---------------------------------------------
 {"id": 1, "ts": "1533-09-18 12:00:00.000Z"}
(1 rows)
{code}
 

Expected behaviour:

The "select ts", "select tojson(ts)" and "select json *" should all produce the same date.

 

Actual behaviour:

The "select ts" produced the "1533-09-28" date but the "select tojson(ts)" and "select json *" produced the "1533-09-18" date.
Bug: DOC - Link tickets, pages in FAQ page
Status: Open
Priority: Normal
Created: 24/Apr/24 06:29

The [FAQ|https://cassandra.apache.org/doc/latest/cassandra/overview/faq/index.html] page references tickets and pages which are not hyperlinked. For example:

{quote} See {{256}} and {{43}} for more gory details.{quote}

should instead be:

{quote} See [CASSANDRA-256|https://issues.apache.org/jira/browse/CASSANDRA-256] and [CASSANDRA-43|https://issues.apache.org/jira/browse/CASSANDRA-43] for more gory details.{quote}

A second example is:

{quote}When a new nodes joins a cluster, it will automatically contact the other nodes in the cluster and copy the right data to itself. See {{topology-changes}}.{quote}

should instead be:

{quote}When a new nodes joins a cluster, it will automatically contact the other nodes in the cluster and copy the right data to itself. See [topology changes|https://cassandra.apache.org/doc/latest/cassandra/managing/operating/topo_changes.html].{quote}

NOTE: This issue was raised in [DBA Stack Exchange|https://dba.stackexchange.com/questions/338938/] by user [Akshay Hiremath|https://dba.stackexchange.com/users/219347/].
Bug: syntax formatting on CQL doc is garbled
Status: Open
Priority: Normal
Created: 24/Apr/24 00:39

It looks like the build process for the 4.1 docs isn't correctly processed.  Screenshot attached.

https://cassandra.apache.org/doc/4.1/cassandra/cql/cql_singlefile.html#alterTableStmt

 !image-2024-04-23-17-37-54-438.png!
Bug: [Analytics] Consume new Sidecar client API to stream SSTables
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Apr/24 17:00
Resolved: 23/Apr/24 19:52

A new client API was recently introduced in Sidecar to stream SSTables. Cassandra Analytics needs to start consuming the new API in order to take advantage of the fixes when streaming SSTables from a Cassandra installation with more than one data directory.
Bug: [Analytics] Support bulk write via S3
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Apr/24 18:30
Resolved: 22/Apr/24 22:46

I would like to propose a new write option in Cassandra Analytics to bulk write SSTables via S3, in addition to the previously-implemented "direct upload to all sidecars" (now known as the "Direct" transport). 
The new write option, now being implemented, is the "S3_COMPAT" transport, which allows the job to upload the generated SSTables to an S3-compatible storage system, and then inform the Cassandra Sidecar that those files are available for download & commit.

Additionally, a plug-in system was added to allow communications between custom transport hooks and the job, so the custom hook can provide updated credentials and out-of-band status updates on S3-related issues.
Bug: blocking on memtable allocations blocks compactions from completing
Status: Triage Needed
Priority: Normal
Created: 19/Apr/24 17:25

{{updateCompactionHistory}} writes to compaction_history in a blocking manner.  If we've also run out of memtable memory, this call will block and compaction falls behind.  

It might make sense to have this be non blocking.  We'd risk having a missing entry in nodetool getcompactionhistory, but I think that's preferable to having a dead node.
Bug: sstableupgrade is very slow
Status: Open
Priority: Normal
Created: 18/Apr/24 06:46

We are in the process of migrating cassandra from 3.11.x to 4.1.4 and upgrading the sstables using sstableupgrade from Cassandra V4.1.4, from `me-` to `nb-` Format

Unfortunately, the process is very very slow (less than 0.5 MB/s).

Some observations:
- The process is only slow on (fast) SSDs, but not on ram disks.
- The sstables consist of many partitions (this may be unrelated)
- The upgrade process is fast, if we use `automatic_sstable_upgrade` instead of the sstableupgradetool.
- We give enough RAM (export MAX_HEAP_SIZE=8g)

On profiling, we found out, that sstableupgrade is burning most CPU time on {{posix_fadvise}} (see flamegraph_sstableupgrade.png ).

My naive interpretation of the whole {{maybeReopenEarly}} to {{posix_fadvise}} chain is, that the process just informs the linux kernel, that the written data should not be cached. If we comment out the call to {{NativeLibrary.trySkipCache}}, the conversion is running at expected 10MB/s (see flamegraph_ok.png )
Bug: Test failure: upgrade_tests.py::upgrade_through_versions_test::test_bootstrap_multidc failing counter checks
Status: Open
Priority: Normal
Created: 03/Apr/24 20:50

This appears to be a new failure in {{{}test_bootstrap_multidc{}}}, around {{upgrade_scenario}} verifying counter updates. I can't reproduce locally, so perhaps there is a counter race condition lurking somewhere...
{noformat}
assert 249 == 248 +249 -248

self = <upgrade_tests.upgrade_through_versions_test.TestUpgrade_indev_4_0_x_To_indev_trunk object at 0x7fbae9b96550>

def test_bootstrap_multidc(self):
# try and add a new node
# multi dc, 2 nodes in each dc
self.prepare()
cluster = self.cluster

if cluster.version() >= '5.0':
cluster.set_configuration_options({'user_defined_functions_threads_enabled': 'true',
'scripted_user_defined_functions_enabled': 'false'})
elif cluster.version() >= '3.0':
cluster.set_configuration_options({'enable_user_defined_functions': 'true',
'enable_scripted_user_defined_functions': 'true'})
elif cluster.version() >= '2.2':
cluster.set_configuration_options({'enable_user_defined_functions': 'true'})

cluster.populate([2, 2])
[node.start(use_jna=True, wait_for_binary_proto=True) for node in self.cluster.nodelist()]
self._multidc_schema_create()
> self.upgrade_scenario(populate=False, create_schema=False, after_upgrade_call=(self._bootstrap_new_node_multidc,))

upgrade_tests/upgrade_through_versions_test.py:865: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
upgrade_tests/upgrade_through_versions_test.py:506: in upgrade_scenario
call()
upgrade_tests/upgrade_through_versions_test.py:841: in _bootstrap_new_node_multidc
self._check_counters()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <upgrade_tests.upgrade_through_versions_test.TestUpgrade_indev_4_0_x_To_indev_trunk object at 0x7fbae9b96550>

def _check_counters(self):
logger.debug("Checking counter values...")
session = self.patient_cql_connection(self.node2, protocol_version=self.protocol_version)
session.execute("use upgrade;")

for key1 in list(self.expected_counts.keys()):
for key2 in list(self.expected_counts[key1].keys()):
expected_value = self.expected_counts[key1][key2]

query = SimpleStatement("SELECT c from countertable where k1='{key1}' and k2={key2};".format(key1=key1, key2=key2),
consistency_level=ConsistencyLevel.ONE)
results = session.execute(query)

if results is not None:
actual_value = results[0][0]
else:
# counter wasn't found
actual_value = None

> assert actual_value == expected_value
E assert 249 == 248
E +249
E -248

upgrade_tests/upgrade_through_versions_test.py:789: AssertionError
{noformat}
Bug: Implement support for virtual types
Status: Open
Priority: Normal
Created: 16/Apr/24 08:55

We can not use user types in virtual tables. While it might look strange at first to ask for the support of this, currently we can not do something like this:
{code:java}
cqlsh> select * from system_guardrails.thresholds ;

 name                          | value
-------------------------------+----------------------
               collection_size |   {warn: 0, fail: 0}
             column_value_size | {warn: -1, fail: -1}
             columns_per_table | {warn: -1, fail: -1}
...
{code}
{code:java}
VIRTUAL TABLE system_guardrails.thresholds (
    name text PRIMARY KEY,
    value settings
) WITH comment = 'Guardrails configuration table for thresholds';
{code}
because "settings" is a UDT
{code:java}
cqlsh> DESCRIBE type system_guardrails.settings ;

CREATE TYPE system_guardrails.settings (
    warn bigint,
    fail bigint
);
{code}
While this is not absolutely necessary to implement and it might be worked around by a simple tuple, it feels sad that user experience suffers. Supporting custom types for vtables is indeed possible so we should just do that.

There is additional work needed to do this, because virtual types are not supported in python-driver, I had to do this:

[https://github.com/smiklosovic/python-driver/commit/14b236bb471bc4a7e251a9f6b694de7885b339de]

python-driver reads system_schema.types in order to show the correct output in cqlsh, as it has not recognized virtual types, it was always displaying the results like
{code:java}
settings(warn=-1, fail -1)
{code}
because it could not evaluate it differently.

With the patch for python-driver, it will fetch it from system_virtual_schema.types, where it will be like:
{code:java}
cqlsh> select * from system_virtual_schema.types ;

 keyspace_name     | type_name | field_names      | field_types
-------------------+-----------+------------------+----------------------
 system_guardrails |  settings | ['warn', 'fail'] | ['bigint', 'bigint']

(1 rows)
{code}