av9ash/GitBugs huggingface dataset:
Bug: jvm-dtests crash on java 11
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Jul/20 01:04
Resolved: 06/Aug/20 04:23

There is a race condition bug with CMS and class unloading which cause the JVM to crash.  Since jvm-dtests rely on class loaders and unloading, this causes sporadic JVM crashes that look like the following in CI logs

{code}
junit.framework.AssertionFailedError: Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit.
	at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.util.Vector.forEach(Vector.java:1387)
	at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.util.Vector.forEach(Vector.java:1387)
	at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.Thread.run(Thread.java:834)
{code}

Here is the JVM bug https://bugs.openjdk.java.net/browse/JDK-8252955
Bug: Test failure: transient_replication_ring_test.py::TestTransientReplicationRing::test_move_forwards_and_cleanup
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 19/Aug/20 21:49
Resolved: 18/Sep/23 15:31

Failing here, also locally:

[https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/312/workflows/da4ce69c-e778-467e-b9f3-27ab166a8321/jobs/1945]
Bug: Add Index Group Interface for Storage Attached Index
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/Sep/20 13:28
Resolved: 28/Feb/22 19:41

[Index group|https://github.com/datastax/cassandra/blob/storage_attached_index/src/java/org/apache/cassandra/index/Index.java#L634] interface allows:
* indexes on the same table to receive centralized lifecycle events called secondary index groups. Sharing of data between multiple column indexes on the same table allows SAI disk usage to realise significant space savings over other index implementations.
* index-group to analyze user query and provide a query plan that leverages all available indexes within the group.
Bug: CEP-7 Storage Attached Indexes (Phase 1)
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Aug/20 13:29
Resolved: 26/Jul/23 18:50

[CEP|https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage+Attached+Index] - A new index implementation, called Storage
 Attached Index(SAI), based on the advancement made by SASI.
 * disk usage by sharing of common data between multiple column indexes on the same table and better compression of on-disk structures.

 * numeric range query performance with modified KDTree and collection type support.

 * compaction performance and stability for larger data set.
Bug: nodetool drain fails in jvm dtest
Status: Open
Priority: Normal
Created: 22/Jul/20 05:09

Drain fails since MessagingService doesn’t expect to be shutdown twice.

{code}
Caused by: java.lang.AssertionError
	at org.apache.cassandra.net.MessagingService.shutdown(MessagingService.java:766)
	at org.apache.cassandra.distributed.impl.Instance.lambda$parallelRun$28(Instance.java:805)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	... 3 more
{code}
Bug: 4.0 quality testing: Materialized View
Status: Open
Priority: Normal
Created: 03/Jul/20 09:02

The main purpose of this ticket to get a better understanding about 4.0 MV status as a guideline for future improvements. I don't think it should block 4.0 release since it's already marked as experimental.

Main areas to test:
 * Write perf: We expect to see [10% write throughput drop per MV added|https://www.datastax.com/blog/2016/05/materialized-view-performance-cassandra-3x].
** Attached C40_MV.png  is alpha-4, 5-node, rf3 MV write tests: with 1 mv, throughput dropped 50%
 * Read perf: identical to normal table
 * Bootstrap/Decommision: no write-path required since CASSANDRA-13065
 * Repair: write path required
 * Chaos monkey: take down coordinator/base-replica/view-replica during read/write/token-movement and verify data consistency (may need a tool)
 * Hint Replay: able to throttle if table has MV - CASSANDRA-13810
 * Schema race: create/drop - CASSANDRA-15845/CASSANDRA-15918
Bug: Perform garbage collection on specific partitions or range of partitions
Status: Open
Priority: Normal
Created: 11/Aug/20 06:57

Some of our data is quite seasonal and variable based on partitions and certain partitions tend to contain significantly more tombstones than others. We currently run nodetool garbagecollect on tables when this becomes an issue.

However, if the time to complete and resources required to only garbage collect particular partitions on specific tables could improve the performance of this activity it would be useful to only target the specific partitions we need.
Bug: Query SSTable Indexes lazily in token sorted runs for LCS, TWCS or RangeAwaredCompaction
Status: Open
Priority: Normal
Created: 11/Aug/20 07:49

Currently SASI searches all SSTable indexes that may include the query partition key and indexed term, but this will cause large IO overhead with range index query (ie. age > 18) when sstable count is huge.

Proposed improvement: query sstable indexes in token-sorted-runs lazily. When the data in the first few token ranges is sufficient for limit, SASI can reduce the overhead of searching sstable indexes for the remaining ranges.
Bug: Remove Legacy ColumnFamily Aliases from TableMetrics
Status: Open
Priority: Normal
Created: 31/Aug/20 21:05

In 3.0, the primary type of the metric names created in {{TableMetrics}} switched from “ColumnFamily” to “Table”, but “ColumnFamily” remained for backwards compatibility. This clutters TableMetrics and should be on our radar for removal post-4.0, given “ColumnFamily” will have existed in two major versions after the switch tp “Table”. As a secondary goal, we might want to remove the metrics deprecated in CASSANDRA-15909.
Bug: Batch schema statements to create multiple SASI and MV at once to reduce disk IO
Status: Open
Priority: Normal
Created: 27/Aug/20 06:54

Currently, operator has to create SASI/MV one by one on the same table and every index/view build will need to read all data on disk.

In order to speed up multiple SASI/MV creation, I propose to add a new batch schema statement to create multiple SASI/MV at once, so that C* only needs to read on-disk data once.
Bug: Upgrade to lz4-java 1.8
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 13/Jul/20 23:58
Resolved: 03/Feb/22 22:54

After upgrading lz4-java I saw that CentOS 6 was no longer supported via native and filed a ticket https://github.com/lz4/lz4-java/issues/163 on the lz4-java project.  This ticket is now closed and targeted for the next lz4 release (assume to be 1.7.2); once this release comes out we should upgrade.
Bug: StorageService exposes ensureTraceKeyspace that is used by jvm-dtest, but it should be removed and use MigrationManager.evolveSystemKeyspace instead
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Aug/20 22:50
Resolved: 31/Oct/22 17:46

Jvm dtest uses StorageService.ensureTraceKeyspace to register the tracing keyspace, but doesn’t register the other distributed key spaces.  This method should not be used by dtest and instead should use MigrationManager.evolveSystemKeyspace as it registers all keyspaces
Bug: Reduce thread contention in CommitLogSegment and HintsBuffer by rewriting CAS loops to atomic adds
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Aug/20 10:00
Resolved: 01/Aug/21 13:56

Follow up to CASSANDRA-15922

Both CommitLogSegment and HintsBuffer use AtomicIntegers for the current offset when allocating. Like in CASSANDRA\-15922 the loops on {{.compareAndSet(..)}} can be replaced with atomic adds using the {{. getAndAdd(..)}} method.

In highly contended environments the CAS failures can be high, starving writes in a running Cassandra node. On the same cluster CASSANDRA\-15922 was found, after CASSANDRA\-15922's fix was deployed, there was still problems around commit log flushing and hints. No flamegraph was collected that demonstrated the thread contention as clearly as was found in CASSANDRA\-15922, but the performance fix proposed here hopefully is obvious enough.
Bug: Wrong warning about data volumes capacity
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Sep/20 16:28
Resolved: 06/Aug/21 18:21

I see the following warning trying to run *nodetool upgradesstables*
{noformat}
WARN  16:09:24 Only 34988 MB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
{noformat}
This warning is wrong because the wrong storage device capacity gets tested.

All my cassandra data paths are subdirectories of */data* mount point which has enough of space:
{noformat}
$ df -h /data
Filesystem      Size  Used Avail Mounted on
.................     1.2T 200G  1T  /data
{noformat}
However what Warning reports is a OS mount which has nothing to do with Cassandra configuration:
{noformat}
df -h /
Filesystem      Size  Used Avail Use% Mounted on
............        40G  5.7G   35G  15% /
{noformat}

I see this error running Cassandra 3.0.22
Bug: 4.0 Quality: Read Repair Test Audit
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Jul/20 12:18
Resolved: 19/Oct/20 14:47

This is a subtask of CASSANDRA-15579 focusing on read repair.

[This document|https://docs.google.com/document/d/1-gldHcdLSMRbDhhI8ahs_tPeAZsjurjXr38xABVjWHE/edit?usp=sharing] lists and describes the existing functional tests for read repair, so we can have a broad view of what is currently covered. We can comment on this document and add ideas for new cases/tests, so it can gradually evolve to a more or less detailed test plan.
Bug: jvm dtest execute APIs do not support collections
Status: Open
Priority: Normal
Created: 22/Jul/20 05:10

If you use a collection type they will be transferred to the instance and we call org.apache.cassandra.utils.ByteBufferUtil#objectToBytes to convert to ByteBuffers; this doesn’t support collections.  If you try to work around this by converting before sending, it will fail since that method doesn’t support ByteBuffer as input
Bug: SASIIndexTest is flaky in cassandra-3.11
Status: Triage Needed
Priority: Normal
Created: 23/Jul/20 17:33

The following {{SASIIndexTestSASIIndexTest}} tests seem flaky in 3.11:
 * testIndexMemtableSwitching
 * testPagination
 * testIndexRedistribution
 * testTruncate

The test failures can be seen in [ci-cassandra|https://ci-cassandra.apache.org/job/Cassandra-3.11/76/testReport/org.apache.cassandra.index.sasi/SASIIndexTest/], and I have also reproduced in our [internal test multiplexer|https://jenkins-cassandra.datastax.lan/job/parameterized-testall/666/]. They don't seem to happen in trunk.
Bug: Incorrect parsing of the timestamp with less than 3 digits in the milliseconds
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Jul/20 06:26
Resolved: 13/Aug/20 21:20

(tested on 4.0-beta1, but should be in all versions)

Right now, Cassandra incorrectly handles timestamps with less than 3 digits in the milliseconds part.  Timestamps (valid from the Java point of view (see below output in Scala) are either rejected (if we have 1 digit only), or incorrectly parsed when 2 digits are specified:

{noformat}
cqlsh> create table test.tm (id int primary key, tm timestamp);
cqlsh> insert into test.tm(id, tm) values (2, '2020-07-24T10:00:01.2Z');
InvalidRequest: Error from server: code=2200 [Invalid query] message="Unable to coerce '2020-07-24T10:00:01.2Z' to a formatted date (long)"
cqlsh> insert into test.tm(id, tm) values (1, '2020-07-24T10:00:01.12Z');
cqlsh> select * from test.tm;

 id | tm
----+---------------------------------
  1 | 2020-07-24 10:00:01.012000+0000

(1 rows)
{noformat}

Checking with Instant:

{noformat}
scala> java.time.Instant.parse("2020-07-24T10:00:01.12Z")
res0: java.time.Instant = 2020-07-24T10:00:01.120Z
scala> java.time.Instant.parse("2020-07-24T10:00:01.2Z")
res1: java.time.Instant = 2020-07-24T10:00:01.200Z
{noformat}

Imho it should be fixed (Cc: [~aholmber])
Bug: Invalid serialized size for responses
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Sep/20 16:14
Resolved: 29/Oct/20 18:29

Was running a benchmark at LOCAL_ONE and eventually saw the below exception; this is related to CASSANDRA-16097 as it was found during the same test.

{code}
org.apache.cassandra.net.InvalidSerializedSizeException: Invalid serialized size; expected 14, actual size at least 13, for verb HINT_RSP
	at org.apache.cassandra.net.OutboundConnection$EventLoopDelivery.doRun(OutboundConnection.java:816)
	at org.apache.cassandra.net.OutboundConnection$Delivery.run(OutboundConnection.java:687)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)
{code}

{code}
org.apache.cassandra.net.InvalidSerializedSizeException: Invalid serialized size; expected 14, actual size at least 13, for verb MUTATION_RSP
	at org.apache.cassandra.net.OutboundConnection$EventLoopDelivery.doRun(OutboundConnection.java:816)
	at org.apache.cassandra.net.OutboundConnection$Delivery.run(OutboundConnection.java:687)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)
{code}
Bug: MockSchema's SSTableReader creation leaks FileHandle and Channel instances
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Sep/20 20:56
Resolved: 10/Sep/20 19:46

{{MockSchema}} creates {{SSTableReader}} instances for testing, but when it does, it doesn’t seem to ever close the {{FileHandle}} and {{Channel}} instances from which copies are made for the actual readers. ({{FileHandle}} itself also internally copies the channel on creation.) This can trigger leak detection, although perhaps not reliably, from tests like {{AntiCompactionTest}}. A couple well-placed {{try-with-resources}} blocks should help us avoid this (and shouldn't risk closing anything too early, since the close methods for handles and channels seem only to do reference bookkeeping anyway).


Example:

{noformat}
[junit-timeout] ERROR 16:35:47,747 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@487c0fdb) to class org.apache.cassandra.io.util.FileHandle$Cleanup@2072030898:/var/folders/4d/zfjs7m7s6x5_l93k33r5k6680000gn/T/mocksegmentedfile0tmp was not released before the reference was garbage collected
[junit-timeout] ERROR 16:35:47,747 Allocate trace org.apache.cassandra.utils.concurrent.Ref$State@487c0fdb:
[junit-timeout] Thread[main,5,main]
[junit-timeout] 	at java.lang.Thread.getStackTrace(Thread.java:1559)
[junit-timeout] 	at org.apache.cassandra.utils.concurrent.Ref$Debug.<init>(Ref.java:249)
[junit-timeout] 	at org.apache.cassandra.utils.concurrent.Ref$State.<init>(Ref.java:179)
[junit-timeout] 	at org.apache.cassandra.utils.concurrent.Ref.<init>(Ref.java:101)
[junit-timeout] 	at org.apache.cassandra.utils.concurrent.SharedCloseableImpl.<init>(SharedCloseableImpl.java:30)
[junit-timeout] 	at org.apache.cassandra.io.util.FileHandle.<init>(FileHandle.java:74)
[junit-timeout] 	at org.apache.cassandra.io.util.FileHandle.<init>(FileHandle.java:50)
[junit-timeout] 	at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:389)
[junit-timeout] 	at org.apache.cassandra.schema.MockSchema.sstable(MockSchema.java:124)
[junit-timeout] 	at org.apache.cassandra.schema.MockSchema.sstable(MockSchema.java:83)
{noformat}
Bug: Upgrade to snakeyaml >= 1.26 version for CVE-2017-18640 fix
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Sep/20 05:02
Resolved: 09/Oct/20 20:03

There have been critical level CVE (CVE-2017-18640) discovered in snakeyaml version earlier to 1.26. This has been patched into snakeyaml version 1.26.

Reference: [https://nvd.nist.gov/vuln/detail/CVE-2017-18640]

This card is expected to upgrade the snakeyaml version to 1.26.
Bug: NPE thrown while updating speculative execution time if table is removed during task execution
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Jul/20 20:06
Resolved: 16/Sep/20 20:44

CASSANDRA-14338 fixed the scheduling the speculation retry threshold calculation, but if the task happens to be scheduled while a table is being dropped, it triggers an NPE. 


ERROR 2020-07-14T11:34:55,762 [OptionalTasks:1] org.apache.cassandra.service.CassandraDaemon:446 - Exception in thread Thread[OptionalTasks:1,5,main]
java.lang.NullPointerException: null
       at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:444) ~[cassandra-4.0.0.jar:4.0.0]
       at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:346) ~[cassandra-4.0.0.jar:4.0.0]
       at org.apache.cassandra.db.Keyspace.open(Keyspace.java:139) ~[cassandra-4.0.0.jar:4.0.0]
       at org.apache.cassandra.db.Keyspace.open(Keyspace.java:116) ~[cassandra-4.0.0.jar:4.0.0]
       at org.apache.cassandra.db.Keyspace$1.apply(Keyspace.java:102) ~[cassandra-4.0.0.jar:4.0.0]
       at org.apache.cassandra.db.Keyspace$1.apply(Keyspace.java:99) ~[cassandra-4.0.0.jar:4.0.0]
       at com.google.common.collect.Iterables$5.lambda$forEach$0(Iterables.java:704) ~[guava-27.0-jre.jar:?]
       at com.google.common.collect.IndexedImmutableSet.forEach(IndexedImmutableSet.java:45) ~[guava-27.0-jre.jar:?]
       at com.google.common.collect.Iterables$5.forEach(Iterables.java:704) ~[guava-27.0-jre.jar:?]
       at org.apache.cassandra.service.CassandraDaemon.lambda$setup$2(CassandraDaemon.java:412) ~[cassandra-4.0.0.jar:4.0.0]
       at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) [cassandra-4.0.0.jar:4.0.0]
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
       at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
       at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.37.Final.jar:4.1.37.Final]
       at java.lang.Thread.run(Thread.java:834) [?:?]
Bug: Add a new jmxtool which can dump what JMX objects exist and diff
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Aug/20 16:16
Resolved: 16/Sep/20 00:01

In order to help validate metric upgrade we first need to know what is new, what was removed, and what was changed.  To help with this, we should add a new jmxtool which can dump the objects from JMX and diff them.

Once we have this, we can also add a gold list of expected metrics and add tests to validate these metrics don’t change.
Bug: Multiple failures in cqlsh_tests.test_cqlsh.TestCqlsh
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Jul/20 16:36
Resolved: 15/Jul/20 21:49

At least the following tests in cqlsh_tests.test_cqlsh.TestCqlsh need to be investigated:

{{test_describe_types}}
 {{test_describe_functions}}
 {{test_pycodestyle_compliance}}

For a recent example of this, see [https://app.circleci.com/pipelines/github/maedhroz/cassandra/51/workflows/4fd639f6-6523-4520-961e-5b5c384a13b3/jobs/281]. These three tests fail locally and consistently against {{cassandra-3.0}}, but not on {{trunk}}.

 {{test_pycodestyle_compliance}} seems to fail only on 3.0 and 3.11.
Bug: USING_G1 is incorrectly set in cassandra-env.sh if G1 is explicitly disabled
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Jul/20 02:43
Resolved: 08/Jul/20 16:24

{code}
echo $JVM_OPTS | grep -q UseG1GC
USING_G1=$?
{code}
This code will set {{USING_G1}} to {{0}} if G1 is explicitly enabled ({{+UseG1GC}}) *or* explicitly disabled ({{-UseG1GC}}), as found on CASSANDRA-15839.