av9ash/GitBugs huggingface dataset:
Bug: CEP-15: (C*) Enhance in-memory FileSystem to work with mmap and support tests to add custom logic
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Apr/23 22:10
Resolved: 08/May/23 23:11

The Simulator currently uses JimFS for its FileSystem, but this lacks any way to add custom logic when file operations are performed; it also lacks mmap support which requires all disk access logic to provide non-mmap solutions as well.

As part of the Simulator work, testing disk corruption will require resolving both these issues, so will need a new FileSystem to rule them all…

This ticket is to define the new FileSystem and add the integration to Simulator, JVM-Dtest, and Unit tests, but does not directly add the fault injections that Simulator will be doing, that will be follow up work.

Goals:
* FileSystem that works for unit, jvm-dtest, and simulator tests
* FileSystem that allows tests to intercept file operations
Bug: When decommissioning should set Severity to limit traffic
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Apr/23 18:43
Resolved: 25/Apr/23 16:27

When we are decommissioning we first set LEAVING, then LEFT, then disable networking; timeouts start to follow at this last stage. LEFT nodes should not be seen as part of the ring, but that may not be seen before the network is disabled.  To better mitigate timeouts we should set severity as part of decom during the LEAVING phase; by setting severity reads should deprioritize traffic to this node.

Remote DC writes do not leverage proximity or severity and instead use random for its select, writes may still timeout even though we know the node is leaving, and severity is set… to work in this model we should update remote DC writes to deprioritize nodes with severity set
Bug: Avoid unnecessary deserialization of terminal arguments when executing CQL functions
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Jun/23 10:47
Resolved: 12/Jun/23 12:38

CQL functions unnecessarily deserialize their terminal arguments on every call.

For example, the function call in {{SELECT mask_inner(column, 1, 0) FROM t}} deserializes {{1}} and {{2}} for each returned row. As another example, the selector in {{SELECT column * 2 FROM t}} also deserializes {{2}} for each returned row.

The goal of this ticket is to improve the function execution code to avoid those deserializations.
Bug: Refactor cloud snitches to get rid of duplicate code
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/Apr/23 03:34
Resolved: 14/Jul/23 10:40

Nowadays we have got about four public cloud platform snitchs ： EC2 snitch for aws, google cloud snitch for google cloud, alibaba cloud snitch for alibaba cloud and multi region snitch for ec2. And the common place for the first three is that we just need to query the zone center to get the ec2 / ecs id , so I think we can refactor the code , and  if some new public cloud platform want to add one more snitch for himself, there is no need to pull a pr for him and  configure some options in yaml is enough .
Besides it would be even better that we may reuse the multic region snitch for ec2 for other public cloud platform.
Bug: nodetool describecluster command is not showing correct Down count.  
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/May/23 05:27
Resolved: 05/Jun/23 14:07

There are some nodes down in the cluster of Cassandra Version 4.x

# nodetool describecluster command output shows these ips as unreachable.

UNREACHABLE: [<ip1>, <ip2>, <ip3>]

Stats for all nodes:
        Live: 3
        Joining: 0
        Moving: 0
        Leaving: 0
        Unreachable: 3

But under data center , count of down pod is always shown as 0.

Data Centers: 
    dc1 #Nodes: 3 #Down: 0
    dc2 #Nodes: 3 #Down: 0

 

Steps to reproduce:
 # Setup two Data centers dc1,dc2, each datacenter was having 3 nodes - dc1:3,dc2:3
 # mark down any 3 nodes of two data centers.
 # Run nodetool describecluster command from the live node and check the Unreachable count , which is 3 and Down Count is 0 , both are not matched.

 

Expected Output: Unreachable and Down count should have the same value.

Data Centers:
        dc1 #Nodes: 3 #Down: 1

        dc2 #Nodes: 3 #Down: 2
Bug: Deduplicate the MixedModeAvailability upgrade jvm-dtests
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/May/23 17:30
Resolved: 13/Jun/23 08:18

There's V30 and V3X duplicates of each subclass test to MixedModeAvailabilityTestBase.

The V3X tests will always also be run in the V30.

Suggestion is to dedup and remove the version from the names.
Bug: Incremental repairs fail on mixed IPv4/v6 addresses serializing SyncRequest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Apr/23 21:10
Resolved: 25/Apr/23 18:19

The {{SyncRequest}} message assumes the initiator/src/dst are all the IP address
type as an optimization when calculating serialized size. It needs to handle mixed address families.

{code}
/1.2.3.4:7000->/[::1]:7000-SMALL_MESSAGES-f83ce0bc dropping message of type SYNC_REQ due to error"
{code}

{Exception}
{code}
org.apache.cassandra.net.InvalidSerializedSizeException: Invalid serialized size; expected 158, actual size at least 170, for verb SYNC_REQ
        at org.apache.cassandra.net.OutboundConnection$EventLoopDelivery.doRun(OutboundConnection.java:816)
        at org.apache.cassandra.net.OutboundConnection$Delivery.run(OutboundConnection.java:687)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
{code}
Bug: CEP-15: (C*) Add notion of CommandsForRanges and make this durable in C*
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/May/23 16:50
Resolved: 25/May/23 20:27

To add support for range transactions in C* we need to make sure

1) their state is durable and can be recovered on restart
2) have some way to find all CommandsForKey that are contained in the range transaction range
3) have some way to find all CommandsForRange that intersect this range

To do this, I propose the following

1) Create a new commands_for_range table that stores: (store, range) -> list<accord_timestamp_tuple> — this is byId, not sure if repair needs byExecuteId as well
2) For C*, store a in-memory mapping of Range -> List<TxnId>, and on-boot repopulate this cache.  This then can be used to construct the CommandsForRange needed by the transaction. This makes an assumption that many ranges will not exist, at least for the time being.
3) Change commands_for_keys to use LocalPartitioner, and order the table by (store, key)
4) When C* sees a range transaction, find all keys that are contained by the range by running the logical query "SELECT key FROM commands_for_keys WHERE key BETWEEN range.start AND range.end". Implementation has to make sure to handle many keys (may need to partition the range to increase parallel access, and may need to page through the table to see all keys (aka multiple ReadCommands)).  Once all keys are found, then must load into the CommandsForKeys cache

For #4, https://github.com/apache/cassandra-accord/pull/27 maybe able to optimize the logic to lazy load only what is actually needed rather than load the whole world
Bug: CIDR filtering authorizer for Cassandra
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Jun/23 11:38
Resolved: 29/Jul/23 03:11

Introducing a new authorizer, to allow or disallow users accesses based on client's IP or CIDR group. Please see the confluence page for requirements and detailed design of this feature [https://cwiki.apache.org/confluence/display/CASSANDRA/CIDR+filtering+authorizer].

Will post PR link and dev discussion soon.
Bug: Add guardrail for partition size
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/May/23 15:27
Resolved: 25/May/23 16:53

Add a guardrail for max partition size, for example:
{code:java}
partition_size_warn_threshold: 50MiB
partition_size_fail_threshold: 100MiB
{code}
Most probably this guardrail would only be checked when writing a new sstable to disk (fush/compact). Triggering the guardrail on sstable write would emit a log message and a diagnostic event, but it wouldn't reject the write.
Bug: Alter Type does not validate changes like Create Type does
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Jun/23 16:33
Resolved: 23/Aug/23 14:05

Create Type attempts to block undesired field types, but this validation is not cared over to Alter Type Add Field; which allows you to add unexpected/desired types

{code}
Assertions.assertThatThrownBy(() -> createType("CREATE TYPE %s (f counter)")).hasRootCauseMessage("A user type cannot contain counters");
String type = createType(KEYSPACE, "CREATE TYPE %s (a int)");
schemaChange(String.format("ALTER TYPE %s.%s ADD f counter", KEYSPACE, type));
UserType udt = Keyspace.open(KEYSPACE).getMetadata().types.get(UTF8Type.instance.decompose(type)).get();
logger.warn("UDT: {}", udt);
{code}

{code}
UDT: org.apache.cassandra.db.marshal.UserType(cql_test_keyspace,747970655f3031,61:org.apache.cassandra.db.marshal.Int32Type,66:org.apache.cassandra.db.marshal.CounterColumnType)
{code}
Bug: Deadlock updating sstable metadata if disk boundaries need reloading
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Apr/23 17:54
Resolved: 21/Apr/23 17:58

{{CompactionStrategyManager.handleNotification}} holds the read lock while processing notifications. When handling metadata changed notifications, an extra call is made to maybeReloadDiskBoundaries which tries to grab the write lock and deadlocks the thread.

Partial stacktrace
{code}
        at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)
        - parking to wait for  <0x00000005cc000078> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock
        at org.apache.cassandra.db.compaction.CompactionStrategyManager.maybeReloadDiskBoundaries(CompactionStrategyManager.java:495)
        at org.apache.cassandra.db.compaction.CompactionStrategyManager.getCompactionStrategyFor(CompactionStrategyManager.java:343)
        at org.apache.cassandra.db.compaction.CompactionStrategyManager.handleMetadataChangedNotification(CompactionStrategyManager.java:796)
        at org.apache.cassandra.db.compaction.CompactionStrategyManager.handleNotification(CompactionStrategyManager.java:838)
        at org.apache.cassandra.db.lifecycle.Tracker.notifySSTableMetadataChanged(Tracker.java:482)
        at org.apache.cassandra.db.compaction.CompactionStrategyManager.handleNotification(CompactionStrategyManager.java:838)
{code}

Deadlocking with the read lock held blocks the SlabpoolCleaner while notifying ColumnFamilyStore so memtables are prevented from being flushed and recycled, causing any thread applying a mutation to the database (at least GossipStage and MutationStage) to be considered down by peers and/or back up with pending requests.

All the cases investigated were during single sstable upleveling by {{org.apache.cassandra.db.compaction.SingleSSTableLCSTask}} added in CASSANDRA-12526.

Other less critical work was also affected, JMX calls to get estimated remaining compaction tasks, the index summary manager redistributing summaries, the StatusLogger trying to log dropped messages, and the ValidationManager.

Workaround is to reboot the affected host.

The fix is to just remove the redundant disk boundary reload check on that path.
Bug: Add vector similarity functions
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Jun/23 17:46
Resolved: 28/Jul/23 14:53

Add the CQL vector similarity functions described by [CEP-30|https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-30%3A+Approximate+Nearest+Neighbor%28ANN%29+Vector+Search+via+Storage-Attached+Indexes]:
* similarity_cosine
* similarity_euclidean
* similarity_dot_product

The three functions take two vectors of floats of the same dimensions and return a float indicating the similarity score for the vectors.
Bug: NPE when deserializing malformed collections from client
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/May/23 16:32
Resolved: 10/May/23 00:13

When deserializing collections sent from the client, if an element in the collection is incorrectly serialized, Collections.getValue can return null if the length of the element is negative.  Currently this isn't detected and serialization continues, calling validate and throwing an NPE in serializers that don't handle null value buffers.

Detect the malformed input and throw a better MarshalException so it will be converted to an InvalidRequestException for the client.
Bug: Slow builds due to checkstyle
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Jun/23 07:20
Resolved: 14/Jun/23 08:20

Builds are terribly slow atm due to checkstyle. But there's an option to cache results and only run on the latest changed files to avoid running over the same files over and over again.

This brings build times from the current 1m30s to the old 5s.
Bug: Docker images can no longer be built due to virtualenv from pip
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Apr/23 14:09
Resolved: 05/May/23 10:18

{noformat}
 => [linux/amd64 35/56] WORKDIR /home/cassandra                                                                                                                                                              0.1s
 => [linux/amd64 36/56] RUN echo 'export ANT_HOME=/usr/share/ant' >> /home/cassandra/.bashrc &&     echo 'export JAVA8_HOME=/usr/lib/jvm/java-8-openjdk-$(dpkg --print-architecture)' >> /home/cassandra/.b  0.2s
 => ERROR [linux/amd64 37/56] RUN virtualenv --python=python2.7 env2.7                                                                                                                                       0.5s
------
 > [linux/amd64 37/56] RUN virtualenv --python=python2.7 env2.7:
#100 0.424 RuntimeError: failed to find interpreter for Builtin discover of python_spec='python2.7'
------
ubuntu2004_j11.docker:128
--------------------
 126 |     # included in the base image, the compiled objects are not updated by pip at run time, which can
 127 |     # cause errors if the tests rely on new driver functionality or bug fixes.
 128 | >>> RUN virtualenv --python=python2.7 env2.7
 129 |     RUN chmod +x env2.7/bin/activate
 130 |     RUN /bin/bash -c "export CASS_DRIVER_NO_CYTHON=1 CASS_DRIVER_NO_EXTENSIONS=1 && source ~/env2.7/bin/activate && pip2 install --upgrade pip && pip2 install -r /opt/requirements.txt && pip2 freeze --user"
--------------------
error: failed to solve: rpc error: code = Unknown desc = process "/bin/sh -c virtualenv --python=python2.7 env2.7" did not complete successfully: exit code: 1
{noformat}
Bug: Fix org.apache.cassandra.transport.MessagePayloadTest-.jdk17
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 07/Apr/23 20:36
Resolved: 10/Apr/23 15:27

[https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2182/workflows/e3dc630b-b7a4-4f5b-8f29-489bf43ad90f]
{code:java}
java.lang.RuntimeException: java.lang.NoSuchFieldException: modifiers at org.apache.cassandra.transport.MessagePayloadTest.resetCqlQueryHandlerField(MessagePayloadTest.java:98) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Caused by: java.lang.NoSuchFieldException: modifiers at java.base/java.lang.Class.getDeclaredField(Class.java:2610) at org.apache.cassandra.transport.MessagePayloadTest.resetCqlQueryHandlerField(MessagePayloadTest.java:88){code}
Bug: Implement/override equals and hashCode methods in the ServerEncryptionOptions class
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Apr/23 22:49
Resolved: 25/Jul/23 21:15

We have {{equals and hashCode}}  methods in [EncryptionOptions|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/config/EncryptionOptions.java#L551] object but not (overridden/extended) in [ServerEncryptionOptions.|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/config/EncryptionOptions.java#L600]

Code is using the EncryptionOptions as the key in the [ConcurrentHashMap|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/security/SSLFactory.java#L84] in the SSLFactory.java.  Hence technically we must have a equals/hashCode override in the ServerEncryptionOptions to account for fields that matter additionally (e.g. outbound_keystore/password). 

We discussed this over the [cassandra-dev slack channel|https://the-asf.slack.com/archives/CK23JSY2K/p1680118849081399] and it seems agreeable to make this change.
Bug: CircleCI: Add separate approval step for oa utests on separate workflows
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Jun/23 12:15
Resolved: 16/Jun/23 12:10

CASSANDRA-14227 added new {{j*_utests_oa}} jobs to run unit tests with the new, not-default "oa" sstable format. Those tests depend on the {{start_j*_unit_tests}} approval step on the separate workflows.

However, the separate workflow is meant to allow running jobs individually, and having a common approval step for both {{j*_unit_tests}} and {{j*_utests_oa}} prevents us from doing so. That's the case for example of the development of CASSANDRA-18504, where I want to run a single unit test job without caring about the specialization, but the common approval step forces me to run both jobs, duplicating the costs. An example run can be seen [here|https://app.circleci.com/pipelines/github/adelapena/cassandra/2948/workflows/67e5bfcd-c8df-4558-a889-c8828e8dd310].

I think that {{j*_utests_oa}} should have its own separate approval step, the same way that {{{}j*_utest_cdc{}}}, {{{}j*_utest_fqltool{}}}, {{j*_utest_compression}} or {{j*_utest_system_keyspace_directory}} have their own approval step.
Bug: snappy-java vulnerability: CVE-2023-34455, CVE-2023-34454, CVE-2023-34453
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Jun/23 13:37
Resolved: 26/Jun/23 11:15

Failing owasp:

[https://nvd.nist.gov/vuln/detail/CVE-2023-34455]
{quote}Due to use of an unchecked chunk length, an unrecoverable fatal error can occur in versions prior to 1.1.10.1.
{quote}

[https://nvd.nist.gov/vuln/detail/CVE-2023-34454]
{quote}Due to unchecked multiplications, an integer overflow may occur in versions prior to 1.1.10.1, causing an unrecoverable fatal error. 
{quote}

[https://nvd.nist.gov/vuln/detail/CVE-2023-34453]
{quote}Due to unchecked multiplications, an integer overflow may occur in versions prior to 1.1.10.1, causing a fatal error.
{quote}
Bug: BulkLoader withCipherSuites option is ignored
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Jun/23 16:50
Resolved: 01/Aug/23 18:37

The {{withCipherSuites}} option of {{BulkLoader}} is being ignored. It seems that since CASSANDRA-16362 the {{BulkLoader.buildSSLOptions}} method no longer applies the cipher suite options provided by {{clientEncryptionOptions}}.
Bug: Remove unnecessary shuffling of GossipDigests in Gossiper#makeRandomGossipDigest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/May/23 00:50
Resolved: 30/May/23 21:05

In going through trying to understand the Gossiper code I come across:
{code:java}
    /**
     * The gossip digest is built based on randomization
     * rather than just looping through the collection of live endpoints.
     *
     * @param gDigests list of Gossip Digests.
     */
    private void makeRandomGossipDigest(List<GossipDigest> gDigests) {code}
But I couldn't see what purpose randomization had. In fact in 3.11 it will call:
{code:java}
 doSort(gDigestList); {code}
On the receiving end, negating the purpose of the randomization.

 

In discussion with [~stefan.miklosovic] he found this ticket CASSANDRA-14174

So it seems to me this randomization may have been to allow for limited sizes of SYN messages. But this feature doesn't exist and as such by randomizing it is:
 * creating more garbage
 * using more CPU (sure its mostly trival; see next point)
 * more time spent on unnecessary functionality on the *single threaded* gossip stage.
 * complicating the code and making it more difficult to understand

In fact there is a bug in the implementation:
{code:java}
        int generation = 0;
        int maxVersion = 0;        // local epstate will be part of endpointStateMap
        List<InetAddress> endpoints = new ArrayList<InetAddress>(endpointStateMap.keySet());
        Collections.shuffle(endpoints, random);
        for (InetAddress endpoint : endpoints)
        {
            epState = endpointStateMap.get(endpoint);
            if (epState != null)
            {
                generation = epState.getHeartBeatState().getGeneration();
                maxVersion = getMaxEndpointStateVersion(epState);
            }
            gDigests.add(new GossipDigest(endpoint, generation, maxVersion));
        } {code}
If epState is null and we already had a non-null epState, then the next digest will use the generation and maxVersion of the previous iterated epState.

 

Here is change to remove this randomization and fix the above bug, [https://github.com/apache/cassandra/pull/2357/commits/1ba422ab5de35f7057c7621ec3607dcbca19768c]
Bug: remove dh_python use from debian packaging
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 31/May/23 22:04
Resolved: 01/Jun/23 20:05

It looks like dh_python2 has been removed from debian, but it also looks like we don't need it:

{noformat}
E: dh_python2 dh_python2:408: no package to act on (python-foo or one with ${python:Depends} in Depends)
{noformat}
Bug: Add duration to sstablemetadata utility
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Jun/23 14:45
Resolved: 08/Jul/23 11:54

The new additions will output metadata information for:
 * Duration

For Time Window Compaction (TWC), the min and max timestamps together with duration describe the bounds of the time window in the table.
{quote}{{Total partitions: 2430}}
{{Total rows: 8000}}

{{Total column set: 100000}}

{{...}}

{{Min Timestamp: 06/28/2023 15:15:04 (1688067443651650)}}
{{Max Timestamp: 06/28/2023 15:15:58 (1688067500268865)}}
{{Duration Days: 0 Hours: 0 Minutes: 0 Seconds: 53}}
{quote}
The online docs in sstablemetadata.adoc will need to be updated as well.
Bug: Generate.sh -s param to skip autodetection of tests
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/May/23 05:47
Resolved: 12/Jun/23 08:07

When using generate.sh auto detection of modified tests always kicks in. That can be a problem during dev when you want to test a given set of tests without getting all the others in the way. Also when you want to run the script without having to checkout the extra branches auto detection needs.