av9ash/GitBugs huggingface dataset:
Bug: Migrate thresholds for compaction to guardrails
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Dec/21 17:28
Resolved: 01/Jun/23 10:09

Migrate the existing thresholds for compaction:
{code:java}
compaction_large_partition_warning_threshold_mb: 100
compaction_tombstone_warning_threshold: 100000
{code}
To a new set of guardrails under the {{guardrails}} section, for example:
{code:java}
partition_size_warn_threshold: 50MiB
partition_size_fail_threshold: 100MiB 
partition_tombstones_fail_threshold: 50000
partition_tombstones_fail_threshold: 100000
{code}
Note that since these guardrails would be attached to a background process there wouldn't be abort/fail thresholds because we don't want to stop compaction.

At the moment these thresholds are enabled by default, while guardrails are globally disabled by default, so we should consider whether we want to globally enable guardrails, or change these thresholds to disabled by default, or don't migrate the thresholds.
Bug: Provide summary of failed SessionInfo's in StreamResultFuture
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Dec/21 19:46
Resolved: 24/Mar/23 23:10

Currently, we warn about the presence of one or more failed sessions existing in the final state and then an operator/user traces back through the logs to find any failed streams for troubleshooting.
{code:java}
private synchronized void maybeComplete()
{
    if (finishedAllSessions())
    {
        StreamState finalState = getCurrentState();
        if (finalState.hasFailedSession())
        {
            logger.warn("[Stream #{}] Stream failed", planId);
            tryFailure(new StreamException(finalState, "Stream failed"));
        }
        else
        {
            logger.info("[Stream #{}] All sessions completed", planId);
            trySuccess(finalState);
        }
    }
} {code}
It would be helpful to log out a summary of the SessionInfo for each failed session since that should be accessible via the StreamState.

 

This can be especially helpful for longer streaming operations like bootstrap where the failure could have been a long time back and all recent streams leading up to the warning actually are successful.
Bug: Add Math functions 
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Dec/21 17:20
Resolved: 22/Nov/22 17:28

We should add native Maths functions for the most common operations:
* {{abs\(x)}} returns the absolute value of the x
* {{exp\(x)}} returns the value of e (the base of natural logarithms) raised to the power of x
* {{log\(x)}} returns the natural logarithm (base e) of x
* {{log10\(x)}} returns the base-10 logarithm of x
* {{round\(x)}} returns the closest integer to x

+Additional information for newcomers:+

The new functions should be put in a new class {{MathFcts}} similar to {{TimeFcts}}.
The {{MathsFcts.all()}} method should be called in {{SystemKeyspace.functions}}
The unit tests for the functions should be put in a {{MathFctsTest}} similar to {{TimeFctsTest}}
Bug: testsome target doesn't work with wildcards
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Oct/21 08:10
Resolved: 22/Jun/22 05:16

Running {{ant test -Dtest.name=PasswordObfuscator*Test}} runs the test correctly. But {{ant testsome -Dtest.name=PasswordObfuscator*Test}} will make it fail like

{noformat}
[junit-timeout] Testsuite: org.apache.cassandra.cql3.PasswordObfuscatorTest
[junit-timeout] Testsuite: org.apache.cassandra.cql3.PasswordObfuscatorTest Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.038 sec
[junit-timeout] 
[junit-timeout] Testsuite: PasswordObfuscator*Test
[junit-timeout] Testsuite: PasswordObfuscator*Test Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec
[junit-timeout] 
[junit-timeout] Null Test: 	Caused an ERROR
[junit-timeout] PasswordObfuscator*Test
[junit-timeout] java.lang.ClassNotFoundException: PasswordObfuscator*Test
[junit-timeout] 	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
[junit-timeout] 	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
[junit-timeout] 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
[junit-timeout] 	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
[junit-timeout] 	at java.lang.Class.forName0(Native Method)
[junit-timeout] 	at java.lang.Class.forName(Class.java:348)
[junit-timeout] 
[junit-timeout] 
[junit-timeout] Test PasswordObfuscator*Test FAILED
{noformat}

We should fix testsome as this is a useful feature for 'families' of tests such as {{ViewComplex*Test}}
Bug: Remove commitlog_sync_batch_window_in_ms
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Nov/21 20:17
Resolved: 09/Aug/23 20:10

commitlog_sync_batch_window_in_ms is deprecated in 4.0 and has to be removed in 5.0

This will require also some work around the in-jvm upgrade tests.

Currently they do not work if we remove a parameter in a later version.

Side note: They also don't work if we change the type of the parameter in Config class in a newer Cassandra version. If you plan to set it in the tests themselves, it won't work. If you change the type but your don't set the parameter, no issue so there might be some workaround solution instead of working on the dtest framework itself.... TBD
Bug: Fix test TestHintedHandoff#test_hintedhandoff_window
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Nov/21 01:02
Resolved: 08/Jun/22 09:13

This test fails from time to time (see butler https://butler.cassandra.apache.org/#/ci/upstream/workflow/Cassandra-trunk/failure/hintedhandoff_test/TestHintedHandoff/test_hintedhandoff_window)

Example: https://app.circleci.com/pipelines/github/dcapwell/cassandra/1101/workflows/7afb1c7e-8330-4ac6-963d-d7864282f2f3/jobs/7877

{code}
        # Ensure second and third datasets are not present
        for x in range(100, 300):
>           query_c1c2(session, x, ConsistencyLevel.ONE, tolerate_missing=True, must_be_missing=True)

hintedhandoff_test.py:264: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tools/data.py:44: in query_c1c2
    assertions.assert_length_equal(rows, 0)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

object_with_length = [Row(c1='value1', c2='value2')], expected_length = 0

    def assert_length_equal(object_with_length, expected_length):
        """
        Assert an object has a specific length.
        @param object_with_length The object whose length will be checked
        @param expected_length The expected length of the object
    
        Examples:
        assert_length_equal(res, nb_counter)
        """
        assert len(object_with_length) == expected_length, \
            "Expected {} to have length {}, but instead is of length {}"\
>           .format(object_with_length, expected_length, len(object_with_length))
E       AssertionError: Expected [Row(c1='value1', c2='value2')] to have length 0, but instead is of length 1

tools/assertions.py:269: AssertionError
{code}
Bug: Guardrail for column size
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Nov/21 18:23
Resolved: 16/Aug/22 16:37

Add a guardrail for limiting the max size of column values, for example:
{code}
# Failure threshold to prevent writing large column values.
# Defaults to -1 to disable.
column_value_size_failure_threshold_in_kb: -1
{code}
Bug: CEP-10: Simulator Java11 Support
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Nov/21 16:22
Resolved: 13/Dec/22 23:05

Java11 introduces new protection domains for package private methods, so that classes loaded with different class loaders may not access each others’ package private methods, regardless of the package they are declared within. There are differences within the JDK also, and there may be byte weaving targets that need updating (ThreadLocalRandom was one that has been handled)
Bug: Fix failing test: dtest.bootstrap_test.TestBootstrap.test_bootstrap_binary_disabled
Status: Resolved
Priority: Normal
Resolution: Not A Problem
Created: 28/Oct/21 17:32
Resolved: 28/Aug/23 15:41

JDK8 and JDK11

[https://ci-cassandra.apache.org/job/Cassandra-trunk/801/testReport/junit/dtest.bootstrap_test/TestBootstrap/test_bootstrap_binary_disabled/]

 
{code:java}
bootstrap_test.py:1014: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../venv/lib/python3.8/site-packages/ccmlib/node.py:1005: in nodetool
    return handle_external_tool_process(p, ['nodetool', '-h', 'localhost', '-p', str(self.jmx_port)] + shlex.split(cmd))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

process = <subprocess.Popen object at 0x7f6c82e80580>
cmd_args = ['nodetool', '-h', 'localhost', '-p', '7300', 'bootstrap', ...]

    def handle_external_tool_process(process, cmd_args):
        out, err = process.communicate()
        if (out is not None) and isinstance(out, bytes):
            out = out.decode()
        if (err is not None) and isinstance(err, bytes):
            err = err.decode()
        rc = process.returncode
    
        if rc != 0:
>           raise ToolError(cmd_args, rc, out, err)
E           ccmlib.node.ToolError: Subprocess ['nodetool', '-h', 'localhost', '-p', '7300', 'bootstrap', 'resume'] exited with non-zero status; exit status: 1; 
E           stderr: nodetool: Failed to connect to 'localhost:7300' - ConnectException: 'Connection refused (Connection refused)'.

../venv/lib/python3.8/site-packages/ccmlib/node.py:2305: ToolError
{code}
Bug: When streaming sees a ClosedChannelException this triggers the disk failure policy
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Nov/21 23:38
Resolved: 04/Feb/22 19:53

Found in CASSANDRA-17085.

https://app.circleci.com/pipelines/github/dcapwell/cassandra/1069/workflows/26b7b83a-686f-4516-a56a-0709d428d4f2/jobs/7264
https://app.circleci.com/pipelines/github/dcapwell/cassandra/1069/workflows/26b7b83a-686f-4516-a56a-0709d428d4f2/jobs/7256

{code}
ERROR [Stream-Deserializer-/127.0.0.1:7000-f2eb1a15] 2021-11-02 21:35:40,983 DefaultFSErrorHandler.java:104 - Exiting forcefully due to file system exception on startup, disk failure policy "stop"
org.apache.cassandra.io.FSWriteError: java.nio.channels.ClosedChannelException
	at org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:227)
	at org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.writeComponent(BigTableZeroCopyWriter.java:206)
	at org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:125)
	at org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
	at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:51)
	at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:37)
	at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
	at org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:62)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException: null
	at org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:136)
	at org.apache.cassandra.net.AsyncStreamingInputPlus.consume(AsyncStreamingInputPlus.java:155)
	at org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:217)
	... 9 common frames omitted
{code}


When bootstrap fails and streaming is closed, this triggers the disk failure policy which causes the JVM to halt by default (if this happens outside of bootstrap, then we stop transports and keep the JVM up).

org.apache.cassandra.streaming.StreamDeserializingTask attempts to handle this by ignoring this exception, but the call to org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize
 Does try/catch and inspects exception; triggering this condition.
Bug: Add += and -= syntax to ALTER TABLE WITH
Status: In Progress
Priority: Normal
Created: 03/Dec/21 17:33

We have 2 types of table properties in CQL: simple ones (comment, gc_grace_seconds, ...), which are basically of type text (even if, we historically allow constants of any type) and complex ones (compaction, caching...), which are basically map<text, text>.

The only syntax that {{ALTER TABLE WITH}} supports is of setting a property, and so for the later kind, we basically always have to provide the new full map for the option. To take an example, say you have a table with compaction and want to enable compaction logging you will have to provide the all map.

In order to avoid that problem we can add support for the {{+=}} (and {{-=}} ideally) syntax that already exists for maps otherwise, and support:
{code}
ALTER TABLE t WITH compaction+= { 'log_all' : 'true' }
{code}

+Additional info for newcomers:+

In order to implement this change you will need to change the Parser.g ANTLR file located in the src/antlr directory and the java classes corresponding to the different alter statements located in the {{org.apache.cassandra.cql3.statements.schema}} package. 
The unit test for the CQL logic are located under {{org.apache.cassandra.cql3.validation}}
Bug: Not able to override default transport port in cassandra 4.0
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Dec/21 13:58
Resolved: 20/Mar/22 20:22

Bulk Loader is not honouring "-p" option given on the command line of the sstableloader.

The command line used is:

~/apache-cassandra-4.0.1/bin/sstableloader -d 10.14.20.148 -cph 1 -idct 0 -p 9942 -ssp 7011 -sp 7010 --verbose ~/cassandra4_experiment/nishant/employee/

 

but the call is still going to /10.14.20.148:9042. We are just passing the host information to the Cluster.Builder in NativeSSTableLoaderClient.java:
Cluster.Builder builder = Cluster.builder().addContactPointsWithPorts(hosts).allowBetaProtocolVersion();

Looks like default port is getting picked inside com.datastax.driver.core.cluster.
Bug: Guardrail for number of partition keys on IN queries
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Dec/21 12:35
Resolved: 23/Feb/22 12:51

Add a guardrail for limiting the number of partitions restricted with an {{IN}} clause in a {{SELECT}} query, for example:
{code:java}
# Guardrail to warn or abort when querying with an IN restriction selecting
# more partition keys than threshold.
# The two thresholds default to -1 to disable. 
partition_keys_in_select:
    warn_threshold: -1
    abort_threshold: -1
{code}

+Additional information for newcomers:+

*  Add the configuration for the new guardrail on the number of partitions on IN queries in the guardrails section of cassandra.yaml.
*  Add a getPartitionKeysInSelect method in GuardrailsConfig returning a Threshold.Config object
*  Implement that method in GuardrailsOptions, which is the default yaml-based implementation of GuardrailsConfig
*  Add a Threshold guardrail named partitionKeysInSelect in Guardrails, using the previously created config
*  Define JMX-friendly getters and setters for the previously created config in GuardrailsMBean
*  Implement the JMX-friendly getters and setters in Guardrails
*  Now that we have the guardrail ready, it’s time to use it. We should search for a place to invoke the Guardrails.partitionKeysInSelect#guard method with the number of keys specified in select query. The SelectStatement#getSliceCommands methods look like good candidates for this.
*  Finally, add some tests for the new guardrail. Given that the new guardrail is a Threshold, our new test should probably extend ThresholdTester.
Bug: More detailed latency metrics
Status: Open
Priority: Normal
Created: 29/Nov/21 23:57

There is a disconnect with latency clients experience and the latency reported by Cassandra. For example read latency only measures the latency of the StorageProxy::readRows call.

None of the time spent sitting in the Native Transport queue is measured. Neither is any of the time for writing the response back to the channel.

Dispatcher processRequest keeps track of when it first starts processing the request but best I can tell this is only used in tracking for timeouts.

It would be useful for tracking down cause of high client latency if there were more detailed Cassandra metrics around it.

I have attached a patch that adds latency tracking higher in the call stack. Starting timer from before it is put into the Native Transport Request executor. The patch gives 3 different metrics per Request type:

delay - measures time from when it is submitted to NTR pool until it call processRequest

process - time spent in the Dispatcher processRequest call

total - time from when first submitted to NTR pool until the response has been flushed

This patch may not be cleanest or best way of doing this but hopefully gives an idea of what I think would be useful addition that will help operators diagonse latency issues.
Bug: Add a Virtual table to expose Streaming metrics
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 24/Dec/21 14:48
Resolved: 28/Feb/22 09:53

The streaming metrics for the different nodes should be exposed through a virtual table.

+Additional information for newcomers:+

A new class should be created for representing the new Virtual Table in {{org.apache.cassandra.db.virtual}}. You can look at {{ThreadPoolsTable}} for an example of virtual table implementation and to {{TableMetricTables#buildMetadata}} for how {{Timer}} metrics can be exposed. The new table will need to be registered in SystemViewsKeyspace.
For the unit tests the easiest way is to add them at {{StreamingMetrics}}.
Bug: TestMaterializedViews::test_drop_with_stopped_build is flaky
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 16/Dec/21 16:25
Resolved: 24/Feb/22 10:49

The Python dtest [{{materialized_views_test.py::TestMaterializedViews::test_drop_with_stopped_build}}|https://github.com/apache/cassandra-dtest/blob/bca1834dc7075ced5ebaeda15afcf691bcd30ab1/materialized_views_test.py#L1195-L1227] seems to be flaky.

It was discovered during CASSANDRA-17195 with [this CI run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1207/workflows/9f8f6e04-426e-4454-b65f-d6cb84c8cbdf/jobs/10937].

The failure can be reproduced running the test repeatedly for {{{}trunk{}}}, as it can be seen [here|https://app.circleci.com/pipelines/github/adelapena/cassandra/1215/workflows/895ba8d3-925e-456a-8fe6-12ff3e60b450] with this config:
{code:java}
.circleci/generate.sh -m \
  -e REPEATED_DTEST_COUNT=100 \
  -e REPEATED_DTEST_NAME=materialized_views_test.py::TestMaterializedViews::test_drop_with_stopped_build
{code}
Bug: Add new metrics to track the number of requests performed by GROUP BY and Aggregation queries 
Status: Open
Priority: Normal
Created: 19/Dec/21 15:39

When a user perform a GROUP BY query or an aggregate query (e.g. {{SELECT count\(\*) FROM my_table}}) internally C* will send multiple internal requests to avoid running out of memory. The page size used for those internal queries is the same as the external page size.

Having a some visibility on the number of internal requests happening for a group by or an aggregate query is important as it might help administrators to debug performance issues.

We should add some separate metrics for GROUP BY queries and Aggregate queries

+Additional information for newcomers:+

* A new metric class called {{AggregationMetrics}} should be created with an {{Histogram}} called {{internalPagesPerGroupByQuerie}} and another called {{internalPagesPerAggregateQuerie}} (see {{BatchMetrics}} for an example
* High level query paging are managed by {{AggregationQueryPager}}. The number of queries performed should be incremented within {{fetchSubPage}} and the metrics should be updated on close. 
*  To test that the numbers are reliable, you need to create a new Unit Test {{AggregationMetricsTest}}. To have some example of how to test group by queries with paging, you can look into  {{SelectGroupByTest.testGroupByWithPaging()}} to check how to clear the histograms between test you can look into {{BatchMetricsTests.clearHistogram()}}
Bug: Remove use of deprecated java.util.concurrent in tests
Status: Open
Priority: Normal
Created: 08/Nov/21 21:26

{noformat}
      1  Illegal import - java.util.concurrent.ArrayBlockingQueue. [IllegalImport]
     11  Illegal import - java.util.concurrent.CompletableFuture. [IllegalImport]
     39  Illegal import - java.util.concurrent.CountDownLatch. [IllegalImport]
     48  Illegal import - java.util.concurrent.Executors. [IllegalImport]
      3  Illegal import - java.util.concurrent.LinkedBlockingQueue. [IllegalImport]
      5  Illegal import - java.util.concurrent.Semaphore. [IllegalImport]
{noformat}
Bug: Artificial Latency Injection
Status: Patch Available
Priority: Normal
Created: 06/Oct/21 13:02

*Motivation*
Cassandra aims to enable user’s applications to continue to function in the presence of major incidents. To support this clusters must support high-availability topologies, i.e. at present at least 3 DCs at remote distances from each other. Operators wanting to migrate clusters initially created in topologies that do not support this capability, or to move from LOCAL_X to (GLOBAL) X consistency levels, have no mechanism for determining the viability of the strategy for their applications. This is particularly critical for applications that use LWTs, for which the degradation may be unpredictable due to contention effects.

*Goals*
Support cluster-wide latency injection for some subset of messages 
User Impact

*User Impact*
Users will be able to specify the set of Verbs that should be affected, and also which queries within their client-side application. 

Clients will be supported by the introduction of new {{UNSAFE_DELAY_X}} consistency levels that mark queries as suitable to experience additional artificial latency. 

Latency will be controlled by JMX properties specifying whether only {{UNSAFE_DELAY_X}} operations are affected, alongside which verbs and the length of the delay.
Bug: Avoid sstable corruption when upgrading from 4.0-rc1 to 4.0.0
Status: Review In Progress
Priority: Normal
Created: 14/Oct/21 07:19

4.0-rc2 introduced a new sstable version ({{-nb-}}), adding a field in {{StatsMetadata}}. When zero copy streaming a file from 4.0-rc2+ to 4.0-rc1, the rc1 node will write it as an {{-nb-}} file, mutate the metadata to clear repairedAt/level information from StatsMetadata and rewrite the file, keeping the same file name. Problem is that since rc1 doesn't know about the new field in StatsMetadata, it will write the file without the new StatsMetadata field, but with an {{-nb-}} filename. Then, when upgrading this node to rc2+ we will try to read it as a {{-nb-}} file, but will get {{EOFException}} since it doesn't contain the new field.

We should make sure that we never do zero-copy streaming to nodes that don't know about the file version.

The corrupt sstables can be fixed by simply renaming all components of it to {{-na-}} from {{-nb-}}
Bug: Remove use of deprecated Futures in tests
Status: Open
Priority: Normal
Created: 08/Nov/21 21:21

To pass checkstyle:

{noformat}
      Illegal import - com.google.common.util.concurrent.AbstractFuture. [IllegalImport]
     Illegal import - com.google.common.util.concurrent.Futures. [IllegalImport]
     Illegal import - com.google.common.util.concurrent.ListenableFuture. [IllegalImport]
     Illegal import - com.google.common.util.concurrent.ListenableFutureTask. [IllegalImport]
     Illegal import - io.netty.util.concurrent.Future. [IllegalImport]
     Illegal import - io.netty.util.concurrent.Promise. [IllegalImport]
{noformat}