av9ash/GitBugs huggingface dataset:
Bug: COMPACT STORAGE SSTables created before 3.0 are not correctly skipped by timestamp due to missing primary key liveness info
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Oct/20 21:48
Resolved: 07/Jan/21 23:19

This was discovered while tracking down a spike in the number of  SSTables per read for a COMPACT STORAGE table after a 2.1 -> 3.0 upgrade. Before 3.0, there is no direct analog of 3.0's primary key liveness info. When we upgrade 2.1 COMPACT STORAGE SSTables to the mf format, we simply don't write row timestamps, even if the original mutations were INSERTs. On read, when we look at SSTables in order from newest to oldest max timestamp, we expect to have this primary key liveness information to determine whether we can skip older SSTables after finding completely populated rows.

ex. I have three SSTables in a COMPACT STORAGE table with max timestamps 1000, 2000, and 3000. There are many rows in a particular partition, making filtering on the min and max clustering effectively a no-op. All data is inserted, and there are no partial updates. A fully specified row with timestamp 2500 exists in the SSTable with a max timestamp of 3000. With a proper row timestamp in hand, we can easily ignore the SSTables w/ max timestamps of 1000 and 2000. Without it, we read 3 SSTables instead of 1, which likely means a significant performance regression. 

The following test illustrates this difference in behavior between 2.1 and 3.0:
https://github.com/maedhroz/cassandra/commit/84ce9242bedd735ca79d4f06007d127de6a82800

A solution here might be as simple as having {{SinglePartitionReadCommand#canRemoveRow()}} only inspect primary key liveness information for non-compact/CQL tables. Tombstones seem to be handled at a level above that anyway. (One potential problem with that is whether or not the distinction will continue to exist in 4.0, and dropping compact storage from a table doesn't magically make pk liveness information appear.)
Bug: Fix flaky test mixedModeReadRepairUpdate - org.apache.cassandra.distributed.upgrade.MixedModeReadRepairTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Oct/20 18:23
Resolved: 11/Nov/20 20:49

[https://app.circleci.com/pipelines/github/dcapwell/cassandra/745/workflows/1c7e589e-b5af-4a56-b40a-43da424602c7/jobs/4233]
 
{code}
java.lang.RuntimeException: java.lang.OutOfMemoryError: Metaspace
	at org.apache.cassandra.distributed.impl.Instance.lambda$startup$11(Instance.java:514)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Metaspace
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at org.apache.cassandra.distributed.shared.InstanceClassLoader.loadClassInternal(InstanceClassLoader.java:101)
	at org.apache.cassandra.distributed.shared.InstanceClassLoader.loadClass(InstanceClassLoader.java:87)
	at org.apache.cassandra.net.OutboundConnections$UnusedConnectionMonitor.<init>(OutboundConnections.java:265)
	at org.apache.cassandra.net.OutboundConnections.scheduleUnusedConnectionMonitoring(OutboundConnections.java:312)
	at org.apache.cassandra.net.MessagingService.<init>(MessagingService.java:262)
	at org.apache.cassandra.net.MessagingService$MSHandle.<clinit>(MessagingService.java:226)
	at org.apache.cassandra.net.MessagingService.instance(MessagingService.java:231)
	at org.apache.cassandra.distributed.impl.Instance.registerMockMessaging(Instance.java:253)
	at org.apache.cassandra.distributed.impl.Instance.lambda$startup$11(Instance.java:466)
	at org.apache.cassandra.distributed.impl.Instance$$Lambda$16500/806573706.run(Unknown Source)
{code}
Bug: remove "replace" from cql reserved words
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 28/Oct/20 18:08
Resolved: 11/Feb/21 20:12

CASSANDRA-7395 (2.2) added “replace” as a reserved word into CQL which causes client facing outages upgrading from 2.1 to 3.x.  The reserved word is not needed as a reserved word as parsing works just fine removing it from reserved.

To help upgrade users to the 3.x and 4.x lines, we should remove “replace” from the reserved word list.
Bug: jvm dtest is strict on properties which causes upgrade tests to fail
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Nov/20 18:18
Resolved: 10/Nov/20 18:51

{code}
ConfigurationException: Invalid yaml. Please remove properties [cdc_raw_directory, diagnostic_events_enabled] from your cassandra.yaml
{code}

This was done in CASSANDRA-16152 to make sure we don’t have typo’s in configs, but this makes upgrades harder as InstanceConfig is used and this isn’t version aware.
Bug: Improve Logging Downstream of DataOutputBuffer Overflows
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Oct/20 15:37
Resolved: 30/Oct/20 16:56

There are a number of codepaths that buffer intermediate representations of a partition in memory in a {{DataOutputBuffer}}. Compactions and reads (involving multiple SSTables) can both produce partitions whose serialized size is larger than {{MAX_ARRAY_SIZE}}, in which case a {{RuntimeException}} and really not much else makes it to the logs. We should be able to improve this so that at least the offending keyspace, table, and partition are logged.
Bug: Log Warning Rather than Verbose Trace when Preview Repair Validation Conflicts with Incremental Repair
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Oct/20 17:17
Resolved: 20/Oct/20 23:04

When a preview repair on repaired data identifies which SSTables to validate, it might come across an SSTable that's still pending for an in-progress incremental repair session. It makes sense that we immediately fail the preview repair in that case, but the resulting error and verbose stack trace in the logs is a bit too severe a reaction. We should downgrade this to a simple warning message.
Bug: Eliminate gen-doc template warning and unused (problematic) import
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/Oct/20 11:54
Resolved: 07/Apr/21 16:04

1)

When running 'sphinx-build -b html -d build/doctrees source build/html'
on Linux we get the following warning:
{noformat}
     [exec] generating indices... genindex
     [exec] WARNING: Now base template defindex.html is deprecated.
     [exec] writing additional pages... index search{noformat}
On FreeBSD this causes gen-doc to fail:
{noformat}
     [exec] writing additional pages...  indexfailed
     [exec]
     [exec] Theme error:
     [exec] An error happened in rendering the page index.
     [exec] Reason: UndefinedError("'warn' is undefined")
     [exec] *** Error code 2{noformat}
2)

The patch to doc/source/_util/cql.py removes the unused iteritems import, preventing errors on versions of pygments 2.6.0+:
{noformat}
     [exec] Running Sphinx v3.2.1
     [exec]
     [exec] Exception occurred:
     [exec]   File "/path/to/apache-cassandra-4.0-beta2-src/doc/source/_util/cql.py", line 29, in <module>
     [exec]     from pygments.util import iteritems
     [exec] ImportError: cannot import name 'iteritems' from 'pygments.util' (/usr/local/lib/python3.7/site-packages/pygments/util.py){noformat}
The patch has been tested in the following environments:
 * FreeBSD 12.1-RELEASE-p7
 Python 3.7.9
 Sphinx 3.2.1
 pygments 2.7.1
 * Ubuntu 18.04.1
 Python 3.7.5
 Sphinx 1.6.7
 pygments 2.2.0
 * Ubuntu 18.04.1
 Python 2.7.17
 Sphinx 1.6.7
 pygments 2.5.2
Bug: Remove bad advice on concurrent compactors from cassandra.yaml
Status: Triage Needed
Priority: Normal
Created: 08/Dec/20 00:41

Since CASSANDRA-7551, we gave the following advice for setting {{concurrent_compactors}}:

{code}
# If your data directories are backed by SSD, you should increase this
# to the number of cores.
{code}

However in practice there are a number of problems with this.  While it's true that one can increase {{concurrent_compactors}} to improve efficiency of compactions on machines with more cpu cores, the context switching with random IO and GC associated with bringing compaction data into the heap will work against the additional parallelism.

This has caused problems for those who have taken this advice literally.

I propose that we adjust this language to give a limit on number of {{concurrent_compactors}} for this setting both in the 3.x line and in trunk so that new users do not stumble when reviewing whether to change defaults.

See also CASSANDRA-7139 for a discussion on considerations.

I see two short-term options to avoid new user pain:

1. Change the language to say something like this:

{quote}
When using SSD based storage, you can increase the number of {{concurrent_compactors}}.  However be aware that using too many concurrent compactors can have a detrimental effect such as GC pressure, more context switching among compactors and realtime operations, and more random IO pulling data for different compactions.  It's best to test and measure with your workload and hardware.
{quote}

2. Do some significant testing of compaction efficient and read/write latency/throughput targets to see where the tipping point is - considering some constants around memory and heap size and configuration to keep it simple.
Bug: Cannot run tests on Java 11 with coverage analysis
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Dec/20 06:11
Resolved: 28/Jan/21 22:44

When running tests with coverage analysis on Java 11, we get the problem as follows:

{noformat}
[junit-timeout] FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed
[junit-timeout] Exception in thread "main" java.lang.reflect.InvocationTargetException
[junit-timeout] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit-timeout] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit-timeout] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit-timeout] 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[junit-timeout] 	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
[junit-timeout] 	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
[junit-timeout] Caused by: java.lang.RuntimeException: Class java/util/UUID could not be instrumented.
[junit-timeout] 	at org.jacoco.agent.rt.internal_b0d6a23.core.runtime.ModifiedSystemClassRuntime.createFor(ModifiedSystemClassRuntime.java:139)
[junit-timeout] 	at org.jacoco.agent.rt.internal_b0d6a23.core.runtime.ModifiedSystemClassRuntime.createFor(ModifiedSystemClassRuntime.java:100)
[junit-timeout] 	at org.jacoco.agent.rt.internal_b0d6a23.PreMain.createRuntime(PreMain.java:55)
[junit-timeout] 	at org.jacoco.agent.rt.internal_b0d6a23.PreMain.premain(PreMain.java:47)
[junit-timeout] 	... 6 more
[junit-timeout] Caused by: java.lang.NoSuchFieldException: $jacocoAccess
[junit-timeout] 	at java.base/java.lang.Class.getField(Class.java:1999)
[junit-timeout] 	at org.jacoco.agent.rt.internal_b0d6a23.core.runtime.ModifiedSystemClassRuntime.createFor(ModifiedSystemClassRuntime.java:137)
[junit-timeout] 	... 9 more
[junit-timeout] *** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
{noformat}

It is caused by too old Jacoco which does seem to work well with Java 11. Upgrading Jacoco to the newest version 0.8.6 fixes the problem
Bug: Reading dense table yields invalid results in case of row scan queries
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 22/Oct/20 20:34
Resolved: 02/Nov/20 10:43

{{ThriftIntegrationTest}} is broken in the way that it does not actually test reads before and after flushing, because it does not do flush at all (see https://github.com/apache/cassandra/blob/cassandra-3.11/test/unit/org/apache/cassandra/cql3/validation/ThriftIntegrationTest.java#L939). After fixing that method so that it really flushes memtables to disk, we can see inconsistency in reads from dense table - the results returned from memtable differs from the results returned from sstable (the later are wrong, cell values are skipped unexpectedly).

{noformat}
java.lang.AssertionError: Invalid value for row 0 column 0 (value of type ascii), expected <value1> but got <>
{noformat}

In principle this problems is about skipping column values when doing row scan queries with explicitly selected columns (not wildcard), when the columns belong to a super column. This happens only when reading from sstables, it does not happen when reading from memtables.
Bug: upgraded nodes encountered "Unknown column kind" RuntimeException during upgrade C* from 3.0 to 3.2
Status: Resolved
Priority: Normal
Resolution: Won't Fix
Created: 20/Nov/20 06:24
Resolved: 31/Aug/21 19:03

We tried to upgrade a 3.0.21 C* cluster to 3.2 and run into the following runtime exception. 

 

Error message: 

 
{code:java}
INFO  04:56:43 Initializing system_schema.dropped_columns
ERROR 04:56:43 Exception in thread Thread[SSTableBatchOpen:2,5,main]
java.lang.RuntimeException: Unknown column kind during deserialization
  at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331) ~[main/:na]
  at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:477) ~[main/:na]
  at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:366) ~[main/:na]
  at org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:525) ~[main/:na]
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_252]
  at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_252]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_252]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_252]
  at java.lang.Thread.run(Thread.java:748) [na:1.8.0_252]
{code}
 

Steps to reproduce: 
 # Start up a cluster of version 3.0 with 3 nodes and one seed
 # Run these commands to create keyspaces and tables:

{code:java}
CREATE KEYSPACE ks WITH replication = { 'class':'SimpleStrategy', 'replication_factor':1}  AND DURABLE_WRITES = true";{code}
 
{code:java}
CREATE KEYSPACE IF NOT EXISTS foo WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'}{code}
 
{code:java}
CREATE TABLE foo.bar (key text PRIMARY KEY, val int);{code}
 
{code:java}
CREATE TABLE testTable (pk int PRIMARY KEY, val text){code}
 # Drop one of the keyspaces

{code:java}
DROP KEYSPACE foo;{code}
 # stopp all the nodes, and started the C* nodes running C* compiled from 3.2 
 # After upgrading, we noticed some of the nodes encountered the RuntimeException (Unknown column kind during deserialization)
Bug: Cassandra 2.2 doesn't satisfy Java 1.7 language level
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Nov/20 17:57
Resolved: 11/Dec/20 15:34

I think that 2.2 needs to satisfy Java 1.7 language level. However, there are a few places where newer syntax is used:
 * [MessagingService|https://github.com/apache/cassandra/blob/cassandra-2.2/src/java/org/apache/cassandra/net/MessagingService.java#L779]
 * [ExecutorUtils|https://github.com/apache/cassandra/blob/cassandra-2.2/src/java/org/apache/cassandra/utils/ExecutorUtils.java#L36-L47]
 * [Throwables|https://github.com/apache/cassandra/blob/cassandra-2.2/src/java/org/apache/cassandra/utils/Throwables.java#L36]
 * [YamlConfigurationLoader|https://github.com/apache/cassandra/blob/cassandra-2.2/src/java/org/apache/cassandra/config/YamlConfigurationLoader.java#L142]
 * [YamlConfigurationLoaderTest|https://github.com/apache/cassandra/blob/cassandra-2.2/test/unit/org/apache/cassandra/config/YamlConfigurationLoaderTest.java#L38]

If I'm right and we really need 1.7 language level we should fix those, which seems quite straightforward.
Bug: Improve job metadata queries exception handling in cassandra-diff
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 14/Oct/20 20:08
Resolved: 11/May/21 22:33

The job metadata tracks the progress of the diff job. Sometimes, a job can fail due to the progress update query failures. 
The progress update queries can be categorized into 2 groups, critical and trivial one. 
When a query failed to update a trivial status (e.g. ProgressTracker), we would mostly hope to continue the job and just log the failure. 
When a query failed to update a critical status (e.g. JobLifeCycle), we can apply the client-side retry strategy (e.g. exponential backoff) in addition to the retry policy.
Bug: upgradesstables/scrub support for 2.x sstables
Status: Resolved
Priority: Normal
Resolution: Invalid
Created: 20/Nov/20 19:41
Resolved: 19/Mar/21 14:21

This support is important for 2.0/3.0 cluster or 3.0/4.0 cluster that can/might contain 2.x sstables.

4.0 supports versions down to {{ma}}, and 3.0 supports versions down to {{jb}} (3.0 and 2.0.1 correspondingly). So by the time you're on 4.0, you should have no 2.0 sstables either way, and this needs to be fixed for 3.11

 

*NOTE: I just marked the ticket with both 3.11 and 4.0 rc versions even if the bug is presented only in 3.11 but to make it appear as a blocker on the 4.0 board!!*
Bug: 4.0 Quality: Python Upgrade dtest Audit
Status: Resolved
Priority: Normal
Resolution: Done
Created: 09/Dec/20 21:03
Resolved: 29/Jan/21 20:24

This is a subtask of CASSANDRA-15588 focusing on python dtest upgrade tests.

This ticket is to document and describe the existing coverage so we have a broad view of what is currently covered.  This should be used to come up with new test cases which should be managed in separate tickets.
Bug: Bootstrapping nodes can become unreplaceable
Status: Resolved
Priority: Normal
Resolution: Not A Problem
Created: 16/Nov/20 18:50
Resolved: 24/Jun/21 15:37

Bootstrapping nodes (including replacements) don't gossip their tokens until after they've received their schema. If the node experiences a failure before gossiping tokens, it causes the same problem CASSNDRA-15335 fixed for normal nodes - they cannot be replaced without gossiped tokens. Including tokens in the initial gossip round prevents nodes from getting into this state, although we still have to wait for the schema when the node is configured to allocate tokens for a keyspace.
Bug: Implement repair quality test scenarios
Status: Patch Available
Priority: Normal
Created: 05/Nov/20 08:53

Implement the following test scenarios in a new test suite for repair integration testing with significant load:

Generate/restore a workload of ~100GB per node. Medusa should be considered to create the initial backup which could then be restored from an S3 bucket to speed up node population.
 Data should on purpose require repair and be generated accordingly.

Perform repairs for a 3 nodes cluster with 4 cores each and 16GB-32GB RAM (m5d.xlarge instances would be the most cost efficient type).
 Repaired keyspaces will use RF=3 or RF=2 in some cases (the latter is for subranges with different sets of replicas).
||Mode||Version||Settings||Checks||
|Full repair|trunk|Sequential + All token ranges|"No anticompaction (repairedAt==0)
 Out of sync ranges > 0
 Subsequent run must show no out of sync range"|
|Full repair|trunk|Parallel + Primary range|"No anticompaction (repairedAt==0)
 Out of sync ranges > 0
 Subsequent run must show no out of sync range"|
|Full repair|trunk|Force terminate repair shortly after it was triggered|Repair threads must be cleaned up|
|Subrange repair|trunk|Sequential + single token range|"No anticompaction (repairedAt==0)
 Out of sync ranges > 0
 Subsequent run must show no out of sync range"|
|Subrange repair|trunk|Parallel + 10 token ranges which have the same replicas|"No anticompaction (repairedAt == 0)
 Out of sync ranges > 0
 Subsequent run must show no out of sync range
A single repair session will handle all subranges at once"|
|Subrange repair|trunk|Parallel + 10 token ranges which have different replicas|"No anticompaction (repairedAt==0)
 Out of sync ranges > 0
 Subsequent run must show no out of sync range
More than one repair session is triggered to process all subranges"|
|Incremental repair|trunk|"Parallel (mandatory)
 No compaction during repair"|"Anticompaction status (repairedAt != 0) on all SSTables
 No pending repair on SSTables after completion (could require to wait a bit as this will happen asynchronously)
 Out of sync ranges > 0 + Subsequent run must show no out of sync range"|
|Incremental repair|trunk|"Parallel (mandatory)
 Major compaction triggered during repair"|"Anticompaction status (repairedAt != 0) on all SSTables
 No pending repair on SSTables after completion (could require to wait a bit as this will happen asynchronously)
 Out of sync ranges > 0 + Subsequent run must show no out of sync range"|
|Incremental repair|trunk|Force terminate repair shortly after it was triggered.|Repair threads must be cleaned up|
Bug: DROP COMPACT STORAGE should invalidate prepared statements still using CompactTableMetadata
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Dec/20 21:45
Resolved: 18/Dec/20 17:07

When we drop compact storage from a table, existing prepared statements continue to refer to an instance of {{CompactTableMetadata}}, rather than being invalidated so they can be assigned a new {{TableMetadata}} instance. There are perhaps some brute force ways to fix this, like bouncing the node, but that obviously sub-optimal.

One idea is to have {{TableMetadata#changeAffectsPreparedStatements()}} return true when we go from having to not having the DENSE flag. It should be pretty easy to validate a fix with a small addition to {{CompactTableTest}}.
Bug: Fix flaky testTrackMaxDeletionTime
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Oct/20 17:48
Resolved: 26/Mar/21 20:31

* 
testTrackMaxDeletionTime - org.apache.cassandra.io.sstable.SSTableMetadataTest
 
junit.framework.AssertionFailedError: expected:<1.6038784E9> but was:<1.60387827E9>
	at org.apache.cassandra.io.sstable.SSTableMetadataTest.testTrackMaxDeletionTime(SSTableMetadataTest.java:102)

https://app.circleci.com/pipelines/github/bereng/cassandra/160/workflows/6cb80410-b398-477c-b4c9-cc7369785869/jobs/1317