av9ash/GitBugs huggingface dataset:
Bug: NPE in materialized_views_test.TestMaterializedViews test_complex_mv_select_statements
Status: Triage Needed
Priority: Normal
Created: 29/Jan/24 15:23

I hit this recently:

https://app.circleci.com/pipelines/github/instaclustr/cassandra/3811/workflows/0e802f47-b17a-427f-8fac-e179d65bb974/jobs/184973/parallel-runs/20?filterBy=FAILED

{code}
ERROR materialized_views_test.py::TestMaterializedViews::test_complex_mv_select_statements - Failed: Unexpected error found in node logs (see stdout for full details). Errors: [[node1] 'ERROR [ViewBuildExecutor:1] 2024-01-26 16:51:12,197 JVMStabilityInspector.java:70 - Exception in thread Thread[ViewBuildExecutor:1,5,ViewBuildExecutor]\njava.lang.NullPointerException: Cannot invoke "org.apache.cassandra.schema.TableMetadata.partitionKeyColumns()" because "this.viewMetadata" is null\n\tat org.apache.cassandra.db.view.ViewUpdateGenerator.<init>(ViewUpdateGenerator.java:99)\n\tat org.apache.cassandra.db.view.TableViews.generateViewUpdates(TableViews.java:228)\n\tat org.apache.cassandra.db.view.ViewBuilderTask.buildKey(ViewBuilderTask.java:114)\n\tat org.apache.cassandra.db.view.ViewBuilderTask.call(ViewBuilderTask.java:154)\n\tat org.apache.cassandra.db.compaction.CompactionManager.lambda$submitViewBuilder$11(CompactionManager.java:2008)\n\tat org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)\n\tat org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)', [node1] 'ERROR [GlobalLogFollower] 2024-01-26 16:51:12,200 LocalLog.java:478 - Could not process the entry\njava.lang.NullPointerException: Cannot invoke "org.apache.cassandra.schema.TableMetadata.partitionKeyColumns()" because "this.viewMetadata" is null\n\tat org.apache.cassandra.db.view.ViewUpdateGenerator.<init>(ViewUpdateGenerator.java:99)\n\tat org.apache.cassandra.db.view.TableViews.generateViewUpdates(TableViews.java:228)\n\tat org.apache.cassandra.db.view.ViewBuilderTask.buildKey(ViewBuilderTask.java:114)\n\tat org.apache.cassandra.db.view.ViewBuilderTask.call(ViewBuilderTask.java:154)\n\tat org.apache.cassandra.db.compaction.CompactionManager.lambda$submitViewBuilder$11(CompactionManager.java:2008)\n\tat org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)\n\tat org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)']
{code}
Bug: Handle MIN_TOKEN placement correctly
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Jan/24 09:04
Resolved: 29/Jan/24 15:02

During concurrent range movements, it is possible to end up in a situation when from the perspective of the node that owns the {{MIN}} token, there are two independent ranges {{(MIN, A]}} and {{{}(Z, MIN]{}}}. And from the perspective of the joining node, it does not know that these are two _separate_ ranges, and thinks they are just one wraparound range, so it tries to just add some placements there.
Bug: Allow setting TTL for snapshots created by Analytics bulk reader
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Jan/24 19:01
Resolved: 27/Jan/24 01:03

Analytics user can add an existing snapshot's name or create a new snapshot through reader options, from which data is bulk read from. Incase of creating new snapshot, we want to allow users to set TTL option and have a default value for the TTL. This is to make sure, in case of job failures, the snapshots are cleared, to release space.
Bug: Bring ALTER ... DROP COMPACT STORAGE out of experimental
Status: Open
Priority: Normal
Created: 24/Jan/24 16:13

{{ALTER ... DROP COMPACT STORAGE}} statements have not been extensively tested and suffer from several issues like:
 * As COMPACT tables did not have primary key liveness there empty rows
inserted AFTER the ALTER will be returned whereas the one inserted before
the ALTER will not.
 * Also due to the lack of primary key liveness the amount of SSTables being
read will increase resulting in slower queries (CASSANDRA-16675)
 * After DROP COMPACT it becomes possible to ALTER the table in a way that
makes all the row disappears
 * There is a loss of functionality around null clustering when dropping
compact storage (CASSANDRA-16069)

To avoid running into those issues, CASSANDRA-16733 introduced a new flag that allows operators to disable those statements on their clusters and pointed out that ALTER ... DROP COMPACT STORAGE is not production-ready.

see [https://www.mail-archive.com/dev@cassandra.apache.org/msg16789.html]

This ticket should facilitate outlining and implementing a plan how to make ALTER ... DROP COMPACT STORAGE production-ready and revert the patch from CASSANDRA-16217.
Bug: Accord: Reduce overhead of NotifyWaitingOn with a WaitingToExecute SaveStatus
Status: Triage Needed
Priority: Normal
Created: 24/Jan/24 11:32

Long execution graphs can do a lot of duplicated work invoking {{NotifyWaitingOn}} repeatedly on a transaction that is already waiting on a dependent transaction to execute. This can easily be avoided by introducing a {{SaveStatus}} that indicates the transaction is actively managing its dependencies for execution.
Bug: Accord: Introduce long-lived callbacks for progress to reduce overhead of repeated FetchData calls
Status: Triage Needed
Priority: Normal
Created: 24/Jan/24 11:31

We currently poll actively on replicas waiting to hear news of a transaction their execution depends upon. We should instead register a long-lived callback at most once per peer, and periodically batch-wise confirm callbacks for our transactions are are still registered. We can simultaneously make our callback management much less costly, by having a global callback manager that just tracks TxnId->Replica.
Bug: Accord: Permit fast path decisions across epoch changes
Status: Triage Needed
Priority: Normal
Created: 24/Jan/24 11:25

Epoch changes that do not modify the shard's membership should be permitted to proceed unimpeded. We can achieve this by special-casing epoch-only timestamp clashes by only forcing an epoch bump if the membership of the shard has changed. If it hasn't, we vote for the proposed timestamp unmodified.
Bug: Accord: Deps.isEqualOrFuller is incorrect
Status: Triage Needed
Priority: Normal
Created: 24/Jan/24 11:11

Deps may be considered equal or fuller if a Deps only has the same TxnId and Keys, when in fact some TxnId may cover different keys. However, any Deps associated with a given Commit Ballot that has been sliced correctly would satisfy this property safely with only the above checks.
Bug: Accord: Improve and Document CoordinateShardDurable semantics
Status: Triage Needed
Priority: Normal
Created: 24/Jan/24 11:06

Firstly, CoordinateShardDurable should retry in future epochs if necessary. In principle this isn't a problem; the next CoordinateShardDurable should pick up where this one left-off. But we should consider the logic very carefully, and anyway not leave dangling waits.

We should also carefully consider the special-case where replicas are bootstrapping in the future and we are coordinating the shard durability. This replica should safely participate in the sync point, waiting for only the transactions it requires to be replicated to it. So this should also function as expected, but this should be tested and documented carefully.
Bug: Accord: Ensure no storage timestamp clashes across Accord bootstrap
Status: Triage Needed
Priority: Normal
Created: 23/Jan/24 15:59

At present bootstrap does not propagate the local metadata associated with the shard that is being bootstrapped. However, due to the many-to-one relation between Accord timestamps and sstable/C* timestamps it is possible for transactions with the same sstable timestamp to occur either side of a bootstrap for a single key. We can resolve this by either
 # Propagating the timestamp state from Accord system tables alongside bootstrap
 # Making the relationship between timestamps 1:1, by
 ** assigning each replica in the cluster a range of timestamps to allocate for Accord transactions
 ** permit timestamps larger than 8 bytes
 # Prevent timestamp clashes across a SyncPoint
Bug: Accord: Safe guards against erroneously marking replicas stale
Status: Triage Needed
Priority: Normal
Created: 23/Jan/24 15:58

Replicas are marked stale using inferences made when data cannot be obtained from peers. However, if a replica is marked stale erroneously it could cause an outage; any systemic issue in this system could be severe. We should harden this system against potential errors, and offer the option of only marking replicas stale that have been marked as stale by an operator (and otherwise reporting an error to the user).
Bug: Mutation.PartitionUpdateCollector#add asserts Partitioner pointers are equal but this may fail during CommitLogReplay for LocalPartitioner
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Jan/24 17:45
Resolved: 19/Jan/24 12:34

This was found on the cep-15-accord branch.  We have a system table all_commands_for_key and it seems that the LocalPartitioner pointer gets changed, causing CommitLogReplyer to fail

{code}
java.lang.AssertionError: Update to key DecoratedKey(11:01c93b4893674349ae:2b74bbca-eae8-3217-a675-115cf5642ef3\:3030312e302e393134, 00040000000b00000901c93b4893674349ae000021000000102b74bbcaeae83217a675115cf5642ef3000000093030312e302e39313400) with partitioner org.apache.cassandra.dht.LocalPartitioner@1fc4de9e (class org.apache.cassandra.dht.LocalPartitioner) had an update ([system_accord.all_commands_for_key]...) with a different partitioner! org.apache.cassandra.dht.LocalPartitioner@c7d8be7 (class org.apache.cassandra.dht.LocalPartitioner)
	at org.apache.cassandra.utils.Throwables.unchecked(Throwables.java:308)
	at org.apache.cassandra.utils.Throwables.cleaned(Throwables.java:327)
	at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:544)
	at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleMutation(CommitLogReplayer.java:521)
	at org.apache.cassandra.db.commitlog.CommitLogReader.readMutation(CommitLogReader.java:478)
	at org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:397)
	at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244)
	at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)
	at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:195)
	at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:225)
	at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:206)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:332)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:726)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:876)
{code}
Bug: Release Cassandra 4.1.4
Status: Resolved
Priority: Normal
Resolution: Invalid
Created: 10/Jan/24 14:24
Resolved: 10/Jan/24 15:37

I wanted to raise a question but as this is not possible I raised a task.

Can we have the release of Cassandra 4.1.4 be done? I see there is no open issue targeted for this release and I think it can be done. 

We need this release done in order to migrate to version 4.1.4 and have the fix for https://issues.apache.org/jira/browse/CASSANDRA-18878

We cannot move the version 4 of Cassandra as is still a Beta and is not allowed by our management to use Beta Versions while 4.1.3 still have this Vulnerability Issue inside it. So we are stuck => either we have a new Cassandra 4.1.4 released or you Release Cassandra 5.0 but make it official. 

Thanks, 

Nicolae
Bug: [Analytics] Add integration tests using in-jvm-dtest to cover blocklisted instances
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Jan/24 17:29
Resolved: 17/Jan/24 22:06

Scope:
- Add default mechanism to load blocklist from configuration
- Relevant in-jvm-dtest based integration tests to validate the CL validations for different CL combinations
Bug: [Analytics] Fix bulk writer consistency level validations for blocked instances
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/Jan/24 02:50
Resolved: 10/Jan/24 23:04

The analytics library bulk writer spark job performs consistency level validations around write operations.

Currently, these validations incorrectly double count the number of blocked instances for these validations resulting in valid scenarios failing CL checks.
Bug: Fix version in run-external.sh
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Jan/24 06:03
Resolved: 17/Jan/24 19:57

_run-external.sh_ has _0.0.1_ version for the {_}harry-integration-external-0.0.1-SNAPSHOT.jar{_}; however, pom.xml has 0.0.2. So in this ticket, fix the version in _run-external.sh_
Bug: Ensure SAI works w/ Transient Replication/Witness Replicas
Status: Triage Needed
Priority: Normal
Created: 11/Jan/24 19:48

Transient replication (see CASSANDRA-14404) and SAI should work together without specific integration. However, some exploratory testing is probably warranted, and there may be some details around the selection of SSTable indexes at transient replicas that may be worth inspecting. As of CASSANDRA-19018, SAI has a mechanism to separate repaired and urepaired SSTable indexes during its local queries. While repaired data should be removed from transient replicas, it still might be a good idea to avoid looking at repaired SSTables altogether if we can (and especially if the removal of repaired data lags at all, creating a window where we look at it unnecessarily).
Bug: Test failure: org.apache.cassandra.fuzz.ring.ClusterMetadataUpgradeHarryTest#simpleUpgradeTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Jan/24 07:12
Resolved: 11/Jan/24 11:26

This fails consistently in trunk. It seems to be introduced since Harry is in-tree.
{code:java}
java.lang.NullPointerException: Cannot invoke "java.util.List.stream()" because the return value of "java.util.Map.get(Object)" is null
	at org.apache.cassandra.distributed.shared.Versions.getLatest(Versions.java:127)
	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.upgradesTo(UpgradeTestBase.java:219)
	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.upgradesToCurrentFrom(UpgradeTestBase.java:204)
	at org.apache.cassandra.fuzz.ring.ClusterMetadataUpgradeHarryTest.simpleUpgradeTest(ClusterMetadataUpgradeHarryTest.java:77)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) {code}
Bug: [Analytics] Update Spark version to 3.3 and release to maven central
Status: Triage Needed
Priority: Normal
Created: 09/Jan/24 23:38

I was told this library will replace the datastax one, and be more performant along the way. In order to use this to solve a goal, I need a release of Spark 3.3 on Scala 2.13. Right now, it seems on 3.2 https://github.com/search?q=repo%3Aapache%2Fcassandra-analytics+spark+language%3AGradle&type=code&l=Gradle

Also, the README should include how to get this library as I can't find anything like it on Maven Central. Perhaps it isn't published, yet?

Once done, I can try replacing the datastax code in https://github.com/openzipkin/zipkin-dependencies
Bug: [Analytics] Speed up integration tests
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Jan/24 17:36
Resolved: 10/Jan/24 13:45

We can speed up the integration tests by reusing the in-jvm dtest cluster in a test class when possible. The proposal has the following pros and cons.

Pros: 
# Ability to iterate faster by running tests faster
# Reduce the resources needed to run tests
# Reduce flakiness due to resource allocation issues
# Remove necessity to spin up a new process for bulk writes
# Adds the ability to use parameterized tests (for example different CLs for the same test)

Cons:
# Potentially having to add new test classes
# No longer be able to use the {{CassandraIntegrationTest}} annotation
# Can only test a single Cassandra version at a time

In terms of resources, the proposed approach brings huge improvements by reducing the current runtime of about 1 hour down to 20 minutes. Additionally, we reduce the resource requirements for the integration tests, and add the ability to iterate faster. Reduces flakiness which helps surface real issues in the tests.
Bug: (Accord) NPE while trying to serialize FoundKnownMap as value is null half the time but unexpected while serializing
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Jan/24 20:01
Resolved: 09/Jan/24 23:55

{code}
java.lang.NullPointerException
	org.apache.cassandra.service.accord.serializers.CommandSerializers$3.serializedSize(CommandSerializers.java:277)
	org.apache.cassandra.service.accord.serializers.CommandSerializers$3.serializedSize(CommandSerializers.java:253)
	org.apache.cassandra.service.accord.serializers.CheckStatusSerializers$1.serializedSize(CheckStatusSerializers.java:75)
	org.apache.cassandra.service.accord.serializers.CheckStatusSerializers$1.serializedSize(CheckStatusSerializers.java:54)
	org.apache.cassandra.service.accord.serializers.CheckStatusSerializers$2.serializedSize(CheckStatusSerializers.java:115)
{code}

{code}
java.lang.NullPointerException
	org.apache.cassandra.service.accord.serializers.CommandSerializers$3.serialize(CommandSerializers.java:257)
	org.apache.cassandra.service.accord.serializers.CommandSerializers$3.serialize(CommandSerializers.java:253)
	org.apache.cassandra.service.accord.serializers.CheckStatusSerializers$1.serialize(CheckStatusSerializers.java:58)
	org.apache.cassandra.service.accord.serializers.CheckStatusSerializers$1.serialize(CheckStatusSerializers.java:54)
	org.apache.cassandra.service.accord.serializers.CheckStatusSerializers$2.serialize(CheckStatusSerializers.java:91)
{code}
Bug: After upgrading from 3.11.16 to 4.1.3 the CREATE MATERIALIZED VIEW generated by DESCRIBE KEYSPACE is invalid.
Status: Open
Priority: Normal
Created: 08/Jan/24 14:46

After upgrading from 3.11.16 to 4.1.3 the CREATE MATERIALIZED VIEW generated by DESCRIBE KEYSPACE is invalid.
Way back when In a 3.11.x cluster we did
{code:java}
CREATE TABLE myks.devices (
    id text,
    serial text,
    PRIMARY KEY (id)
);
CREATE MATERIALIZED VIEW myks.devices_by_serial AS
    SELECT *
    FROM myks.devices
    WHERE serial IS NOT NULL
    PRIMARY KEY (serial, id)
    WITH CLUSTERING ORDER BY (id ASC);{code}
 


This was perfectley ok back then as KEY's was assumed NOT NULL
so going trough various upgrades our last upgrade from 3.11.16 to 4.1.3 it is no longer possible to use the schema generated by DESCRIBE KEYSPACE myks as this outpts 

 
{code:java}
CREATE MATERIALIZED VIEW myks.devices_by_serial AS
    SELECT *
    FROM myks.devices
    WHERE serial IS NOT NULL
    PRIMARY KEY (serial, id)
    WITH CLUSTERING ORDER BY (id ASC);{code}

but this is no longer valid as it is missing 'AND id IS NOT NULL' to be valid cql. The expected output would be 
{code:java}
CREATE MATERIALIZED VIEW myks.devices_by_serial AS
    SELECT *
    FROM myks.devices
    WHERE serial IS NOT NULL AND id IS NOT NULL
    PRIMARY KEY (serial, id)
    WITH CLUSTERING ORDER BY (id ASC);{code}

There is obviousley no ALTER command that will fix this, but i would have expected the upgraded DESCRIBE to have made valid cql.
Bug: IndexStatusManager needs to prioritize SUCCESS over UNKNOWN states to maximize availability
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 15/Feb/24 02:55
Resolved: 24/Jul/24 20:43

{{IndexStatusManager}} is responsible for knowing what SAI indexes are queryable across the ring, endpoint by endpoint. There are two statuses that SAI treats as queryable, but it should not treat them equally. {{BUILD_SUCCEEDED}} means the index is definitely available and should be able to serve queries without issue. {{UNKNOWN}} indicates that the status of the index hasn’t propagated yet to this coordinator. It may be just fine, or it may not be. If it isn’t a query will not return incorrect results, but it will fail. If there are enough {{BUILD_SUCCEEDED}} replicas, we should ignore {{UNKNOWN}} replicas and maximize availability. If the UNKNOWN replica is going to become {{BUILD_SUCCEEDED}} shortly, it will happily start taking requests at that point and spread the load. If not, we’ll avoid futile attempts to query it too early.
Bug: Fix NEWS.txt Compact Storage section
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 23/Jan/24 19:30
Resolved: 29/Jan/24 23:39

In CASSANDRA-16733 we added a note that Compact Storage will no longer be supported in 5.0. The idea was that drop_compact_storage would be pulled out of the experimental version. 
This did not happen, and compact storage is still around. 
I think this will not be handled at least until 6.0 (major breaking changes) and it is good to be corrected. More and more people are upgrading to 4.0+ and they are confused.
Bug: Hygiene updates for warnings and pytests
Status: Resolved
Priority: Low
Resolution: Fixed
Created: 28/Feb/24 20:55
Resolved: 24/May/24 09:23

 
 * -Update 'Warning' message to write to stderr-
 * -Replace TimeoutError Exception with builtin (since Python 3.3)-
 * -Remove re.pattern_type (removed since Python 3.7)-
 * Fix mutable arg [] in test/run_cqlsh.py read_until()
 * Remove redirect of stderr to stdout in pytest fixture with tty=false; Deprecation warnings can otherwise fail unit tests when stdout & stderr output is combined.
 * Fix several pycodestyle issues
Bug: Default nodetool tablestats to Human-Readable Output
Status: Changes Suggested
Priority: Low
Created: 28/Jan/24 16:27

*Current Behavior*

The current implementation of nodetool tablestats in Apache Cassandra outputs statistics in a format that is not immediately human-readable. This output primarily includes raw byte counts, which require additional calculation or conversion to be easily understood by users. This can be inefficient and time-consuming, especially for users who frequently monitor these statistics for performance tuning or maintenance purposes.

*Proposed Change*

We propose that nodetool tablestats should, by default, provide its output in a human-readable format. This change would involve converting byte counts into more understandable units (KiB, MiB, GiB). The tool could still retain the option to display raw data for those who need it, perhaps through a flag such as --no-human-readable or --raw.

*Considerations*

The change should maintain backward compatibility, ensuring that scripts or tools relying on the current output format can continue to function correctly.
We should provide adequate documentation and examples of both the new default output and how to access the raw data format, if needed.

*Alignment*

Discussion in the dev mailing list: [https://lists.apache.org/thread/mlp715kxho5b6f1ql9omlzmmnh4qfby9] 

*Related work*

Previous work in the series:
 # https://issues.apache.org/jira/browse/CASSANDRA-19015 
 # https://issues.apache.org/jira/browse/CASSANDRA-19104