av9ash/GitBugs huggingface dataset:
Bug: CEP-21 Ensure that SchemaTransformation impls correctly set TableMetadata epoch
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Apr/23 09:42
Resolved: 24/Jul/23 12:17

Many existing {{SchemaTransformation}} implementations (not to mention as yet unimplemented ones) have the potential to modify {{TableMetadata}} and it is brittle to require each of them to take care of updating the metadata epoch. We should have {{ClusterMetadata.Transformer}} or the {{AlterSchema}} transformation itself handle this automatically.
Bug: CEP-21 During startup request replay from CMS asynchronously
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Apr/23 09:42
Resolved: 24/Jul/23 12:16

During the startup sequence, nodes first replay any locally persisted metadata changes then request any newer, unseen updates from the CMS. This second part should be both async and optional, meaning a CMS failure or partition shouldn't prevent nodes from starting up.
Bug: CREATE INDEX Modifications for Initial Release of SAI
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Jun/23 22:29
Resolved: 18/Jul/23 19:36

After a lengthy discussion on the dev list, the community seems to have arrived at the following list of TODOs before we release SAI in 5.0:

1.) CREATE INDEX should be expanded to support {{USING … WITH OPTIONS…}}

Essentially, we should be able to do something like {{CREATE INDEX ON tbl(v) USING ’sai’ WITH OPTIONS = ...}} and {{CREATE INDEX ON tbl(v) USING ‘cassandra’}} as a more specific/complete way to emulate the current behavior of {{CREATE INDEX}}.

2.) Allow operators to configure, in the YAML, a.) whether an index implementation must be specified w/ USING and {{CREATE INDEX}} and b.) what the default implementation will be, if {{USING}} isn’t required.

3.) The defaults we ship w/ will avoid breaking existing {{CREATE INDEX}} usage. (i.e. A default is allowed, and that default will remain ‘cassandra’, or the legacy 2i)

With all this in place, users should be able create SAI indexes w/ the simplest possible syntax, no defaults will change, and operators will have the ability to change defaults to favor SAI whenever they like.
Bug: Add basic text tokenisation and analysis
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Apr/23 14:19
Resolved: 12/Jul/23 21:02

[CASSANDRA-16092|https://issues.apache.org/jira/browse/CASSANDRA-16092] removed support for any text analysis or tokenisation. 

SAI currently supports the following analyzers:
* normalize - text normalization using NFC normalization
* case_sensitive - allow control over the case sensitivity of an index
* ascii - allow ascii folding of text
Bug: Checksum per-SSTable and per-column SAI components after streaming
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/May/23 16:55
Resolved: 11/Jul/23 19:19

The SAI code currently does not checksum validate per-column index data files at any point. It does checksum validate per-sstable components after a full rebuild and it checksum validates the per-column metadata on opening.

We should checksum validate all index components on startup, full rebuild and streaming.

EDIT: The changes we ultimately made here were:

- Fix checksum calculation in {{IncrementalChecksumSequentialWriter}}
- Checksum per-SSTable and per-column components after streaming
- Avoid validating indexes when full rebuild is requested
Bug: Optimize Initial Concurrency Selection for Range Read Algorithm During SAI Queries
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/May/23 09:26
Resolved: 11/Jul/23 19:48

The range read algorithm relies on the Index API’s notion of estimated result rows to decide how many replicas to contact in parallel during its first round of requests. The more results expected from a replica for a token range, the fewer replicas the range read will initially try to contact. Like SASI, SAI floors that estimate to a huge negative number to make sure it’s selected over other indexes, and this floors the concurrency factor to 1. The actual formula looks like this:

{code:java}
// resultsPerRange, from SAI, is a giant negative number
concurrencyFactor = Math.max(1, Math.min(ranges.rangeCount(), (int) Math.ceil(command.limits().count() / resultsPerRange)));
{code}

Although that concurrency factor is updated as actual results stream in, only sending a single range request to a single replica in every case for SAI is not ideal. For example, assume I have a 3 node cluster and a keyspace at RF=1, with 10 rows spread across the 3 nodes, without vnodes. Issuing a query that matches all 10 rows with a LIMIT of 10 will make 2 or 3 serial range requests from the coordinator, one to each of the 3 nodes.

This can be fixed by allowing indexes to bypass the initial concurrency calculation allowing SAI queries to contact the entire ring in a single round of queries, or at worst the minimum number of rounds as bounded by the existing statutory maximum ranges per round.
Bug: [Analytics] Add Caching of Node Settings to Improve Efficiency
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Jun/23 20:49
Resolved: 11/Jul/23 16:59

Currently, `CassandraClusterInfo` does not reuse results of a call to `allNodeSettingsBlocking` from `Sidecar`.
We should improve efficiency by caching these results in order to prevent multiple expensive cluster-wide calls.
Bug: Add support for multiple condition branches and results in Accord transaction
Status: In Progress
Priority: Normal
Created: 19/Apr/23 08:58

I'd like to propose adding support for multiple branches and result sets for Accord transactions. It could look like this:

{code:sql}
BEGIN TRANSACTION
  LET a = ...
  LET b = ...
  LET c = ...
  IF condition THEN
    SELECT 1, a.value
    UPDATE ...
  ELSE IF condition2 THEN
    SELECT 2, b.value
    UPDATE ...
  ELSE
    SELECT 3, c.value
  END IF
COMMIT TRANSACTION
{code}

The existing syntax would remain valid, when a single SELECT is defined in which case the conditional SELECTs would not be valid. 

SELECTs would be validated to return columns of the same type. They would be able to return literals as well.

This would be make the result of the transaction more intuitive as the client would know explicitly if the updates where applied or not.

The following syntax would be considered as invalid:

1. SELECTs defined both on the top level and in branches
{code:sql}
BEGIN TRANSACTION
  LET a = ...
  LET b = ...
  SELECT ...  <--------------------------------
  IF condition THEN
    SELECT 'one', a.value  <--------------------------------
    UPDATE ...
  ELSE IF condition2 THEN
    SELECT 'two', b.value  <--------------------------------
    UPDATE ...
  ELSE
    SELECT 'three', NULL  <--------------------------------
  END IF
COMMIT TRANSACTION
{code}

2. SELECT defined after update - select must be before the update to make it clear that the data is read before modification

{code:sql}
BEGIN TRANSACTION
  LET a = ...
  LET b = ...
  IF condition THEN
    UPDATE ...
    SELECT 'one', a.value  <--------------------------------
  ELSE IF condition2 THEN
    UPDATE ...
    SELECT 'two', b.value
  ELSE
    SELECT 'three', NULL
  END IF
COMMIT TRANSACTION
{code}

3. Missing SELECT in a conditional branch - if SELECT is defined in a any conditional branch, it has to be defined in all branches:

{code:sql}
BEGIN TRANSACTION
  LET a = ...
  LET b = ...
  IF condition THEN
    SELECT 'one', a.value
    UPDATE ...
  ELSE IF condition2 THEN
    SELECT 'two', b.value
    UPDATE ...
  ELSE  <--------------------------------
    UPDATE ... 
  END IF
COMMIT TRANSACTION
{code}

{code:sql}
BEGIN TRANSACTION
  LET a = ...
  LET b = ...
  IF condition THEN
    SELECT 'one', a.value
    UPDATE ...
  END IF <-------------------------------- (else branch is implicit and it has no SELECT)
COMMIT TRANSACTION
{code}

Selections in conditional branches will support only returning references - no support for full query is planned in this ticket.
Bug: Repair vtable uses number of records rather than memory limits, which can cause more memory allocated than expected
Status: In Progress
Priority: Normal
Created: 14/Jun/23 20:24

The config “repair_state_size” is used by 2 caches: the coordinator side of repair, and the participate side of repair.  At this point in time we use number of records rather than memory side, and the default is 100k… In one case it was found that there were 95k records of participate history causing 550mb of allocation!

To resolve this we should:

1) migrate to memory size
2) have the 2 caches have 2 different configs (maybe you want more history for one than the other?); this also makes things explicit and doesn’t have any confusion in terms of cost (atm you can have 200k records for repair with a 100k limit)
Bug: Add error detection in BTI indexes
Status: Triage Needed
Priority: Normal
Created: 18/May/23 09:47

Primary indexes should have a CRC code to enable detection of bit rot. 

While the chances of getting data corruption due to the index are very low, it is possible to not find data that has been marked as persisted, which can cause problems with consistency.

See also the discussion in CASSANDRA-18398.
Bug: As DataOutputBuffer approaches the MAX_BUFFER_SIZE limit there should be a warning.
Status: Triage Needed
Priority: Normal
Created: 29/Jun/23 09:06

As DataOuptutBuffer fills the available buffer space it allocates more by doubling the buffer size until it reaches 64Mb (assuming DOB_DOUBLING_THRESHOLD_MB has not been set).  After which it will increase the size by 1.5 until it reaches 0x7FFF_FFF8 bytes (MAX_BUFFER_SIZE) and then the reallocation will fail.  

0x5555_5550 x 1.5 = 0x7FFF_FFF8

Once the buffer is 0x5555_5550 bytes or larger in size the next allocation will return 0x7FFF_FFF8 bytes and the one after that will fail.

I propose adding a configurable warning that applies when the allocated buffer first approaches 0x5555_5550, by defining a WARNING_THRESHOLD  so that the first time the buffer allocation crosses 0x5555_5550 - WARNING_THRESHOLD an alert is sent.

The alert should be instrumented such that automated monitoring systems can report the issue before it becomes emergent.
Bug: Save state about completed streams and display in nodetool netstats
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 27/Jun/23 20:39
Resolved: 28/Jun/23 16:05

We monitor Bootstraps in progress by polling StreamManagerMBean.getCurrentStreams(). While this gives us good signal for how a bootstrap is progressing, we'd also like to check that all streams have completed successfully when a node has finished bootstrapping.

This patch adds a separate map in StreamManager to track completed streams, and reports those stream summaries in StreamManagerMBean.getCompletedStreams() and .getAllStreams()

It also adds an option to nodetool netstats to display completed streams alongside the current streams.

https://github.com/apache/cassandra/pull/2453
Bug: Missed version note for nodetool setmaxhintwindow
Status: Open
Priority: Normal
Created: 27/Jun/23 23:26

In the [documentation about hints for Cassandra 4.1|https://cassandra.apache.org/doc/latest/cassandra/operating/hints.html], there is a note for nodetool getmaxhintwindow: "New in Cassandra 4.0." I think the same note should be for nodetool setmaxhintwindow.

I can create a merge request myself, but don't know where
Bug: AUTH requests use too much resources
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/May/23 07:17
Resolved: 27/Jun/23 23:20

Hello. I see unexpected CPU usage in a rare situation that may be worth digging into.
We have C* 4.0.9 on Debian running on Java 11.0.18.
It's a small cluster of 3 nodes on commodity hardware (6 cores CPU, 32 Gb RAM, 2 x 512 Gb SSD NVME).
This ring has about 35 clients using Datastax Java Driver for Apache Cassandra.
In the driver connection settings, we use the following:
CONNECTION_POOL_LOCAL_SIZE = 400
CONNECTION_POOL_REMOTE_SIZE = 100
 
And for some reason, from time to time, it causes hundreds of AUTH requests per second that leads to an enormous CPU usage.
And yes, it's easy not to use these settings in the driver, leaving defaults that don't produce such an amount of AUTHs. But isn't it weird that ~150 AUTH rps consume ~1200% CPU?
Please see attached graphs.

I have the following in the settings:

authenticator: PasswordAuthenticator

authorizer: CassandraAuthorizer

roles_validity_in_ms: 600000

permissions_validity_in_ms: 600000

credentials_validity_in_ms: 600000

Please let me know if I can provide any other necessary information.
Thanks for your work. Cassandra is amazing :)
Bug: BLOG - Clarify wording on 3.x EOL post
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 26/Jun/23 23:27
Resolved: 27/Jun/23 17:54

CASSANDRA-18531 generated some confusion so attempting to clean up the wording. 

https://cassandra.apache.org/_/blog/Apache-Cassandra-3.0.x-and-3.11.x-End-of-Life-Announcement.html
Bug: Handle special column name when upgrading from 2.2.19 to 3.0.29/3.11.15
Status: Resolved
Priority: Normal
Resolution: Not A Problem
Created: 26/Jun/23 21:27
Resolved: 26/Jun/23 22:07

When I was migrating from 2.2.19 to 3.0.29/3.11.15, I noticed some of my columns could not be accessed because of the syntax problem. I had a column named "Is" in the 2.x. However, after the upgrade process, when I try to perform the same select command, it throws an exception since "Is" will be interpreted separately. 

I observed this after my cluster was fully upgraded. I cannot downgrade it back to 2.2.19. I cannot rename this column directly either.
h2. Steps to reproduce

Execute the following commands in Cassandra-2.2.19 (single node)
{code:java}
CREATE KEYSPACE  ks WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 };
CREATE TABLE IF NOT EXISTS ks.tb (c1 INT,c2 INT,c3 TEXT,rJr INT,c4 TEXT,c5 TEXT,ds INT, Is INT,
PRIMARY KEY (c2, c5, c3, rJr, Is, c1, ds, c4 ));

SELECT c2, ds, Is, c4 FROM ks.tb WHERE c2 = 1566089765 AND c5 = 'teststring1' AND c3 = 'teststring2' AND rJr = -1012111692; 

 c2 | ds | is | c4
----+----+----+----


(0 rows){code}
Then perform an upgrade to Cassandra-3.0.29/Cassandra-3.11.15
{code:java}
// Do the same select
SELECT c2, ds, Is, c4 FROM ks.tb WHERE c2 = 1566089765 AND c5 = 'teststring1' AND c3 = 'teststring2' AND rJr = -1012111692; 

SyntaxException: line 1:15 no viable alternative at input 'Is' (SELECT c2, ds, [Is]...){code}
The column still exists in the table, but I cannot access it.

I am wondering whether I can directly rename it in the new version or get some prompts telling me to rename is before the upgrade process. "Is" is also not marked as a reserved word in the [CQL Keywords doc|https://docs.datastax.com/en/cql-oss/3.3/cql/cql_reference/keywords_r.html]).

Any help is appreciated!
Bug: StorageService throws inconsistent exceptions for unknown hosts
Status: Open
Priority: Normal
Created: 23/Jun/23 02:30

When hosts are unknown, StorageService throws two types of exceptions. The following methods throw RuntimeException:

 
{code:java}
 public String getNativeaddress(InetAddressAndPort endpoint, boolean withPort){
  ...
  try{                
    InetAddressAndPort address =   InetAddressAndPort.getByName(Gossiper.instance.getEndpointStateForEndpoint(endpoint).getApplicationState(ApplicationState.NATIVE_ADDRESS_AND_PORT).value);                
    return address.getHostAddress(withPort);            
}catch (UnknownHostException e) {
      throw new RuntimeException(e);            
 }
}{code}
{code:java}
public void onChange(InetAddressAndPort endpoint, ApplicationState state, VersionedValue value){
...
try { 
  InetAddressAndPort address = InetAddressAndPort.getByName(value.value);   SystemKeyspace.updatePeerNativeAddress(endpoint, address); 
} catch (UnknownHostException e) 
{ 
    throw new RuntimeException(e); 
}
...
}
{code}
A method throws IllegalArgumentException
{code:java}
 public void rebuild(String sourceDc, String keyspace, String tokens, String specificSources, boolean excludeLocalDatacenterNodes){
  ...
  try                        {                            InetAddressAndPort endpoint = InetAddressAndPort.getByName(stringHost);                            if (FBUtilities.getBroadcastAddressAndPort().equals(endpoint))                            {                                throw new IllegalArgumentException("This host was specified as a source for rebuilding. Sources for a rebuild can only be other nodes in the cluster.");                            }                            sources.add(endpoint);                        }                        catch (UnknownHostException ex)                        {                            throw new IllegalArgumentException("Unknown host specified " + stringHost, ex);                        }
} {code}
A method throws no exceptions:
{code:java}
private void handleStateBootreplacing(InetAddressAndPort newNode, String[] pieces){
 ...
try {
      oldNode = InetAddressAndPort.getByName(pieces[1]);        
} catch (Exception e) {            
   logger.error("Node {} tried to replace malformed endpoint {}.", newNode, pieces[1], e);            
   return;        
}
...
} {code}
A method does not rethrow exceptions and throws 
{color:#000000}UnknownHostException{color}
{code:java}
 public List<String> getTokens(String endpoint) throws UnknownHostException    {        return getTokens(InetAddressAndPort.getByName(endpoint));    } {code}
{color:#000000}The treatments look random. {color}
Bug: The exceptions related to queryNames are inconsistently handled in NodeProbe.java
Status: Open
Priority: Normal
Created: 23/Jun/23 04:36

The call of queryNames can cause MalformedObjectNameException and IOException, but the exceptions are inconsistently handled. 

The following method rethrows RuntimeException:
{code:java}
 private static Multimap<String, String> getJmxThreadPools(MBeanServerConnection mbeanServerConn)
{
 try        
{            
   Multimap<String, String> threadPools = HashMultimap.create();
   Set<ObjectName> threadPoolObjectNames = mbeanServerConn.queryNames(                    new ObjectName("org.apache.cassandra.metrics:type=ThreadPools,*"),                    null);
      ...
  }catch (MalformedObjectNameException e) {
          throw new RuntimeException("Bad query to JMX server: ", e);        
  }catch (IOException e)        
{         throw new RuntimeException("Error getting threadpool names from JMX", e);        } 
   {code}
A method swallows the thrown exceptions:
{code:java}
 public ColumnFamilyStoreMBean getCfsProxy(String ks, String cf)
{
  ...
try{
     Set<ObjectName> beans = mbeanServerConn.queryNames(                    new ObjectName("org.apache.cassandra.db:type=*" + type +",keyspace=" + ks + ",columnfamily=" + cf), null);
     ...
}catch (MalformedObjectNameException mone)        { 
     System.err.println("ColumnFamilyStore for " + ks + "/" + cf + " not found.");            System.exit(1);        
}catch (IOException e){
     System.err.println("ColumnFamilyStore for " + ks + "/" + cf + " not found: " + e);            System.exit(1);        }

   {code}
A method does not handle it at all:
{code:java}
private List<Entry<String, ColumnFamilyStoreMBean>> getCFSMBeans(MBeanServerConnection mbeanServerConn, String type)            throws MalformedObjectNameException, IOException
{
   ...
    ObjectName query = new ObjectName("org.apache.cassandra.db:type=" + type +",*");        Set<ObjectName> cfObjects = mbeanServerConn.queryNames(query, null);
  ...
}
    {code}
Bug: Testing required to test upgrade and downgrade completeness
Status: Triage Needed
Priority: Normal
Created: 23/Jun/23 04:56

In the paper "Understanding and Detecting Software Upgrade Failures in Distributed Systems" the authors identify several hidden issues with Cassandra upgrade capabilities.  Indeed these were submitted as bugs and fixed.  The authors also point out that a failure during upgrade (or downgrade) is a critical failure often taking out a whole cluster.

This issue is to find/create a testing library similar to the one described at [2] that is integrated into the Cassandra build system at appropriate points.  Those points to be determined as development of the solution progresses.

 

[1] [https://dl.acm.org/doi/10.1145/3477132.3483577]

[2] https://sysartifacts.github.io/sosp2021/summaries/duptester
Bug: Cassandra Analytics - Upgrade to use JUnit 5
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Jun/23 16:38
Resolved: 22/Jun/23 16:24

For future work, we’d like to use some features only available in JUnit 5 (TestTemplates being the immediate need).
Upgrade cassandra-analytics dependencies to JUnit 5 and fix any test issues.
Bug: snappy-java vulnerability: CVE-2023-34453
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Jun/23 13:40
Resolved: 22/Jun/23 16:45

Failing owasp:

[https://nvd.nist.gov/vuln/detail/CVE-2023-34453]

bq. Due to unchecked multiplications, an integer overflow may occur in versions prior to 1.1.10.1, causing a fatal error.
Bug: nodetool garbagecollect does not use all available compaction executors
Status: Triage Needed
Priority: Normal
Created: 22/Jun/23 11:38

Lets have a node with 8 cores and lets do "nodetool setconcurrentcompactors 4"

When I am doing "nodetool garbagecollect", there is a possibility to specify number of "jobs" via -j flag. If I set it to "2", max two threads will be compacting, if I set it to 6, it will be in practice capped to 4 as that is my "concurrentcompactors" setting.

So far good.

However, when I set jobs to 4 and I execute garbagecollecting on two tables, tb1 and tb2 like this:
{code:java}
nodetool garbagecollect -j 4 -- keyspace1 tb1 tb2
{code}
What it does is that it will start to gc first table, 4 tables at max AND THEN it will start to gc the second table.

In other words, if tb1 has 10 tables to gc and I have 4 jobs at max, it will gc them, but if one looks into compactionstats, she sees that as gc-ing progresses, there might be e.g. just 2 tables left to gc so in theory there is a slot for two additional sstables to gc as well but this will not happen. It will wait until the first table is gc-ed and then it will start to gc the second one with 4 threads.

This might be improved so as soon as there is a free job thread to gc, next sstable would be scheduled to be gc-ed even it is from a different cql table.
Bug: Cassandra Analytics - Adding support for TTL & Timestamps for bulk writes
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Jun/23 20:27
Resolved: 21/Jun/23 21:12

In this patch we are adding support for Spark Bulk Writer writes with TTL & Timestamp. TTL/ Timestamp can be constant where all rows in the RDD are written with same TTL/Timestamp or per row based where each row has a corresponding TTL/Timestamp present in a separate column in the RDD.
Bug: Add jobs for resource-intensive Python upgrade tests
Status: Open
Priority: Normal
Created: 05/Jun/23 19:06

As it was noticed in CASSANDRA-18499, Python upgrade tests should include resource-intensive ones. Some are run in CircleCI but skipped in Jenkins (see CASSANDRA-18499 for reference).

We should add  --skip-resource-intensive-tests to the current Python upgrade tests jobs and create new jobs dedicated to the resource-intensive upgrade tests — the way we do it for the regular Python Dtests(for reference - large DTests).