av9ash/GitBugs huggingface dataset:
Bug: Test failure: org.apache.cassandra.distributed.test.ReadRepairEmptyRangeTombstonesTest-_jdk11
Status: Resolved
Priority: Normal
Resolution: Won't Fix
Created: 30/Nov/23 09:35
Resolved: 08/Dec/23 10:13

{code}
java.util.concurrent.TimeoutException 

java.lang.RuntimeException: java.util.concurrent.TimeoutException 
at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:79) 
at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:537) 
at org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:1098) 
at org.apache.cassandra.distributed.test.ReadRepairEmptyRangeTombstonesTest.teardownCluster(ReadRepairEmptyRangeTombstonesTest.java:108) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
Caused by: java.util.concurrent.TimeoutException 
at org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:253) 
at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:529) 
Suppressed: java.util.concurrent.TimeoutException 
{code}
Bug: Don't NPE when initializing CFSs for local system keyspaces with UCS
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Dec/23 19:00
Resolved: 07/Dec/23 17:17

When UnifiedCompactionStrategy is used as the default, NPEs are thrown when
flushing the system keyspace tables early during startup. The system keyspace is
initialised before the cluster metadata, but UCS currently tries to access the
current epoch when initialising the shard manager, to determine whether the
local ranges are out of date. This isn't necessary for the system keyspaces as
they use LocalStrategy and cover the whole token space.
Bug: Test failure: org.apache.cassandra.distributed.test.log.FetchLogFromPeersTest.catchupCoordinatorAheadPlacementsReadTest-_jdk11
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Nov/23 10:08
Resolved: 07/Dec/23 17:16

CircleCI failure: https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/256/workflows/c4fda8f1-a8d6-4523-be83-5e30b9de39fe/jobs/20464/tests

Also failing on 17: Circleci Failure: https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/256/workflows/c4fda8f1-a8d6-4523-be83-5e30b9de39fe/jobs/20500/tests

{code}
junit.framework.AssertionFailedError
	at org.apache.cassandra.distributed.test.log.FetchLogFromPeersTest.catchupCoordinatorAheadPlacementsReadTest(FetchLogFromPeersTest.java:217)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}
Bug: Test Failure: dtest-novnode.cqlsh_tests.test_cqlsh_copy.TestCqlshCopy.test_round_trip_with_sub_second_precision
Status: Open
Priority: Normal
Created: 05/Dec/23 22:44

{quote}
h3. https://ci-cassandra.apache.org/job/Cassandra-trunk/1785/testReport/dtest-novnode.cqlsh_tests.test_cqlsh_copy/TestCqlshCopy/test_round_trip_with_sub_second_precision/
Error Message

AssertionError: assert [['1', '1943-06-19 11:21:01.000+0000'], ['2', '1943-06-19 11:21:01.123+0000'], ['3', '1943-06-19 11:21:01.124+0000']] == [['3', '1943-06-19 11:21:01.124+0000']] At index 0 diff: ['1', '1943-06-19 11:21:01.000+0000'] != ['3', '1943-06-19 11:21:01.124+0000'] Left contains 2 more items, first extra item: ['2', '1943-06-19 11:21:01.123+0000'] Full diff: [ + ['1', + '1943-06-19 11:21:01.000+0000'], + ['2', + '1943-06-19 11:21:01.123+0000'], ['3', '1943-06-19 11:21:01.124+0000'], ]
h3. Stacktrace

self = <cqlsh_tests.test_cqlsh_copy.TestCqlshCopy object at 0x7ff2be7c4110> @since('3.4') def test_round_trip_with_sub_second_precision(self): """ Test that we can import and export timestamp values with millisecond precision: - create a csv file and import it - export the data and check the values are as expected @jira_ticket CASSANDRA-10428 """ self.prepare() self.session.execute("create TABLE testsubsecond(id int PRIMARY KEY, subid timestamp)") tempfile1 = self.get_temp_file() tempfile2 = self.get_temp_file() with open(tempfile1.name, 'w') as csvfile: writer = csv.writer(csvfile) writer.writerow([1, '1943-06-19 11:21:01+0000']) writer.writerow([2, '1943-06-19 11:21:01.123+0000']) writer.writerow([3, '1943-06-19 11:21:01.123456+0000']) logger.debug('Importing from csv file: {}'.format(tempfile1.name)) self.run_cqlsh(cmds="COPY ks.testsubsecond FROM '{}'".format(tempfile1.name)) logger.debug('Exporting to csv file: {}'.format(tempfile2.name)) self.run_cqlsh(cmds="COPY ks.testsubsecond TO '{}'".format(tempfile2.name)) csv_results = sorted(list(csv_rows(tempfile2.name))) > assert [['1', '1943-06-19 11:21:01.000+0000'], ['2', '1943-06-19 11:21:01.123+0000'], ['3', '1943-06-19 11:21:01.124+0000']] == csv_results E AssertionError: assert [['1', '1943-06-19 11:21:01.000+0000'], ['2', '1943-06-19 11:21:01.123+0000'], ['3', '1943-06-19 11:21:01.124+0000']] == [['3', '1943-06-19 11:21:01.124+0000']] E At index 0 diff: ['1', '1943-06-19 11:21:01.000+0000'] != ['3', '1943-06-19 11:21:01.124+0000'] E Left contains 2 more items, first extra item: ['2', '1943-06-19 11:21:01.123+0000'] E Full diff: E [ E + ['1', E + '1943-06-19 11:21:01.000+0000'], E + ['2', E + '1943-06-19 11:21:01.123+0000'], E ['3', E '1943-06-19 11:21:01.124+0000'], E ] cqlsh_tests/test_cqlsh_copy.py:2005: AssertionError
{quote}
Bug: BLOG - Apache Cassandra 5.0 Features: Vector Search
Status: Triage Needed
Priority: Normal
Created: 06/Dec/23 21:16

This ticket it to add a blog post to the site. The change includes:
 * Adding the post : Apache Cassandra 5.0 Features: Vector Search
 * updating the blog index page with the new post
 * A small text change the Apache Catalyst program page.
Bug: Test Failure: dtest.sslnodetonode_test.TestNodeToNodeSSLEncryption.test_ca_mismatch
Status: Open
Priority: Normal
Created: 05/Dec/23 22:49

[https://ci-cassandra.apache.org/job/Cassandra-trunk/1782/testReport/dtest.sslnodetonode_test/TestNodeToNodeSSLEncryption/test_ca_mismatch/]
h3.  
{code:java}
Error Message
assert False

Stacktrace
self = <sslnodetonode_test.TestNodeToNodeSSLEncryption object at 0x7fca5921d050> def test_ca_mismatch(self): """CA mismatch should cause nodes to fail to connect""" credNode1 = sslkeygen.generate_credentials("127.0.0.1") credNode2 = sslkeygen.generate_credentials("127.0.0.2") # mismatching CA! self.setup_nodes(credNode1, credNode2) self.fixture_dtest_setup.allow_log_errors = True self.cluster.start(no_wait=True) found = self._grep_msg(self.node1, _LOG_ERR_HANDSHAKE) self.cluster.stop() > assert found E assert False sslnodetonode_test.py:115: AssertionError
{code}
Bug: Test Failure: org.apache.cassandra.distributed.test.topology.DecommissionAvoidWriteTimeoutsTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 25/Nov/23 07:34
Resolved: 06/Dec/23 09:47

Seen in j17_jvm_dtests  CASSANDRA-19034
https://app.circleci.com/pipelines/github/michaelsembwever/cassandra/257/workflows/ddcb5f4e-e2c5-430b-b922-f7064a244971/jobs/20667/tests
{noformat}
java.lang.IllegalStateException: Live member count did not converge across all instances
	at org.apache.cassandra.distributed.impl.AbstractCluster$ChangeMonitor.waitForCompletion(AbstractCluster.java:939)
	at org.apache.cassandra.distributed.impl.AbstractCluster.startup(AbstractCluster.java:1055)
	at org.apache.cassandra.distributed.shared.AbstractBuilder.start(AbstractBuilder.java:166)
	at org.apache.cassandra.distributed.test.topology.DecommissionAvoidTimeouts.test(DecommissionAvoidTimeouts.java:84)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{noformat}
Bug: Add an offline cluster metadata tool for disaster recovery scenarios
Status: Triage Needed
Priority: Normal
Created: 04/Dec/23 13:11

We currently have an offline tool to make a node CMS


We should add a tool (or update the existing one) to
* completely forget a node id
* add an instance at a given token
* force move a nodeid to JOINED
* dump the cluster metadata in a human readable format
Bug: Test Failure: org.apache.cassandra.audit.BinAuditLoggerTest.testSelectRoundTripQuery-trie_jdk17_x86_64
Status: Open
Priority: Normal
Created: 05/Dec/23 22:46

h3.  

https://ci-cassandra.apache.org/job/Cassandra-trunk/1792/testReport/org.apache.cassandra.audit/BinAuditLoggerTest/testSelectRoundTripQuery_trie_jdk17_x86_64/
{code:java}
Stacktrace
junit.framework.AssertionFailedError at org.apache.cassandra.audit.BinAuditLoggerTest.testSelectRoundTripQuery(BinAuditLoggerTest.java:80) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}
Bug: Consider CONTAINS KEY in SAI unindexed index contexts
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Nov/23 15:10
Resolved: 23/Nov/23 12:04

As reported by CASSANDRA-19034, {{SelectTest#testContainsKeyAndContainsWithIndexOnMapValue}} fails when the default config uses SAI:
{code:java}
[junit-timeout] Testcase: testContainsKeyAndContainsWithIndexOnMapValue(org.apache.cassandra.cql3.validation.operations.SelectTest)-_jdk11:	FAILED
[junit-timeout] Got less rows than expected. Expected 1 but got 0
[junit-timeout] junit.framework.AssertionFailedError: Got less rows than expected. Expected 1 but got 0
[junit-timeout] 	at org.apache.cassandra.cql3.CQLTester.assertRows(CQLTester.java:1849)
[junit-timeout] 	at org.apache.cassandra.cql3.validation.operations.SelectTest.lambda$testContainsKeyAndContainsWithIndexOnMapValue$9(SelectTest.java:625)
[junit-timeout] 	at org.apache.cassandra.cql3.CQLTester.beforeAndAfterFlush(CQLTester.java:2238)
[junit-timeout] 	at org.apache.cassandra.cql3.validation.operations.SelectTest.testContainsKeyAndContainsWithIndexOnMapValue(SelectTest.java:618)
[junit-timeout] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit-timeout] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit-timeout] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit-timeout] 
[junit-timeout] 
{code}
This can also be reproduced with this simple test:
{code:java}
createTable("CREATE TABLE %s (k int PRIMARY KEY, v int, m map<int,int>)");
createIndex("CREATE INDEX ON %s(v) USING 'SAI'");
execute("INSERT INTO %s (k, v, m) VALUES (?, ?, ?)", 0, 1, map(2, 3));
assertRows(execute("SELECT k, v, m FROM %s WHERE v = 1 AND m CONTAINS KEY 2 ALLOW FILTERING"),
           row(0, 1, map(2, 3)));
{code}
I think the problem is that the index context created by the query planner for an unindexed column always has a {{VALUES}} index target type, [here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/index/sai/plan/QueryController.java#L149]. This is problematic when there is a {{CONTAINS KEY}} expression that isn't backed by an index, since we'll be comparing the searched map key to scanned map values.
Bug: Ensure that empty SAI column indexes do not fail on validation after full-SSTable streaming
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 09/Nov/23 20:23
Resolved: 22/Nov/23 15:27

I actually discovered this in some exploratory testing for CASSANDRA-19007...

{noformat}
@Test
public void testRepairWithEmptyColumnIndex() throws IOException
{
   try (Cluster cluster = init(Cluster.build(2).withConfig(config -> config.with(GOSSIP).with(NETWORK)).start()))
   {
       cluster.schemaChange(withKeyspace("CREATE TABLE %s.t (k int PRIMARY KEY, a int, b int)"));
       cluster.schemaChange(withKeyspace("CREATE INDEX ON %s.t(a) USING 'sai'"));
       cluster.schemaChange(withKeyspace("CREATE INDEX ON %s.t(b) USING 'sai'"));

       // insert a split row
       cluster.get(1).executeInternal(withKeyspace("INSERT INTO %s.t(k, a) VALUES (0, 1)"));
       cluster.get(2).executeInternal(withKeyspace("INSERT INTO %s.t(k, b) VALUES (0, 2)"));
       cluster.get(1).flush(KEYSPACE);
       cluster.get(2).flush(KEYSPACE);
       cluster.get(1).nodetoolResult("repair", KEYSPACE).asserts().success(); // fails w/ no data for the "a" index on node 2!
   }
}
{noformat}

{noformat}
java.lang.RuntimeException: Repair job has failed with the error message: Repair command #1 failed with error Repair session 55a34e20-7f24-11ee-b345-d158c708a34e for range [(-1,9223372036854775805], (9223372036854775805,-1]] failed with error Stream failed: 
Session peer /127.0.0.2:7012 Failed because of an unknown exception
java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /private/var/folders/4d/zfjs7m7s6x5_l93k33r5k6680000gn/T/dtests2185335212676803469/node1/data0/distributed_test_keyspace/t-3459c6aa21753083afcc5b0c9f77c3f7/distributed_test_keyspace-t-nc-4-big-SAI+aa+t_a_idx+Meta.db
	org.apache.cassandra.index.sai.disk.v1.V1OnDiskFormat.rethrowIOException(V1OnDiskFormat.java:241)
	org.apache.cassandra.index.sai.disk.v1.V1OnDiskFormat.validatePerColumnIndexComponents(V1OnDiskFormat.java:230)
java.nio.file.NoSuchFileException: /private/var/folders/4d/zfjs7m7s6x5_l93k33r5k6680000gn/T/dtests2185335212676803469/node1/data0/distributed_test_keyspace/t-3459c6aa21753083afcc5b0c9f77c3f7/distributed_test_keyspace-t-nc-4-big-SAI+aa+t_a_idx+Meta.db
	java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
{noformat}

This is an untested side-effect of a change in CASSANDRA-18670. The fix should be fairly simple, essentially making sure we don't assume the column index components are present during validation when the index completion marker is present.
Bug: Fix nodetool enable/disablebinary to correctly set rpc
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 17/Oct/23 22:50
Resolved: 29/Nov/23 11:34

 
{code:java}
        if ((nativeFlag != null && Boolean.parseBoolean(nativeFlag)) || (nativeFlag == null && DatabaseDescriptor.startNativeTransport()))
        {
            startNativeTransport();
            StorageService.instance.setRpcReady(true);
        } {code}
The startup code here only sets RpcReady if native transport is enabled. If you call 
{code:java}
nodetool enablebinary{code}
then this flag doesn't get set.

But with the change from CASSANDRA-13043 it requires RpcReady set to true in order to get a leader for the counter update.

Not sure what the correct fix is here, seems to only really use this flag for counters. So thinking perhaps the fix is to just move this outside the if condition.
Bug: Repeated tests (CircleCI) errors on moving logs
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Nov/23 16:08
Resolved: 01/Dec/23 18:47

https://app.circleci.com/pipelines/github/jacek-lewandowski/cassandra/1114/workflows/a1d02a59-31ee-48e2-9426-a4eca52ed3ff/jobs/50107

{noformat}
BUILD SUCCESSFUL
Total time: 5 seconds
+ dest=/tmp/results/repeated_utests/stdout/passes/01
+ mkdir -p /tmp/results/repeated_utests/stdout/passes/01
+ mv stdout.txt /tmp/results/repeated_utests/stdout/passes/01/org.apache.cassandra.db.commitlog.CommitLogSegmentBackpressureTest.txt
+ source=build/test/output/
+ dest=/tmp/results/repeated_utests/output/passes/01
+ mkdir -p /tmp/results/repeated_utests/output/passes/01
+ [[ -d build/test/output/ ]]
++ ls build/test/output/
+ [[ -n TEST-org.apache.cassandra.db.commitlog.CommitLogSegmentBackpressureTest.xml ]]
+ mv build/test/output//TEST-org.apache.cassandra.db.commitlog.CommitLogSegmentBackpressureTest.xml /tmp/results/repeated_utests/output/passes/01/
+ source=build/test/logs/
+ dest=/tmp/results/repeated_utests/logs/passes/01
+ mkdir -p /tmp/results/repeated_utests/logs/passes/01
+ [[ -d build/test/logs/ ]]
++ ls build/test/logs/
+ [[ -n _jdk11 ]]
+ mv build/test/logs//_jdk11 /tmp/results/repeated_utests/logs/passes/01/
mv: cannot move 'build/test/logs//_jdk11' to '/tmp/results/repeated_utests/logs/passes/01/_jdk11': File exists

Exited with code exit status 1
{noformat}
Bug: Test Failure: org.apache.cassandra.distributed.test.log.FetchLogFromPeersTest.testSchema-_jdk11
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Nov/23 18:37
Resolved: 05/Dec/23 07:53

In j11_jvm_dtests_vnode, ref CASSANDRA-19034

https://app.circleci.com/pipelines/github/mike-tr-adamson/cassandra/402/workflows/92aacb84-fd3a-48e0-9fb2-d1e2fe6fc71a/jobs/35324/tests

{noformat}
org.apache.cassandra.distributed.shared.ShutdownException: Uncaught exceptions were thrown during test
	at org.apache.cassandra.distributed.impl.AbstractCluster.checkAndResetUncaughtExceptions(AbstractCluster.java:1124)
	at org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:1110)
	at org.apache.cassandra.distributed.test.log.FetchLogFromPeersTest.testSchema(FetchLogFromPeersTest.java:75)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	Suppressed: java.lang.IllegalStateException: Could not find range for token 356242581507269238 in PlacementForRange: {}
		at org.apache.cassandra.tcm.ownership.PlacementForRange.forToken(PlacementForRange.java:119)
		at org.apache.cassandra.db.ReadCommandVerbHandler.getLocalReplica(ReadCommandVerbHandler.java:218)
		at org.apache.cassandra.db.ReadCommandVerbHandler.checkTokenOwnership(ReadCommandVerbHandler.java:159)
		at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:57)
		at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:102)
		at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:122)
		at org.apache.cassandra.distributed.impl.Instance.lambda$receiveMessageRunnable$6(Instance.java:538)
		at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
		at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.base/java.lang.Thread.run(Thread.java:829)
{noformat}
Bug: It should be possible to read all supported versions of sstables regardless of the compatibility mode
Status: Triage Needed
Priority: Normal
Created: 09/Nov/23 10:19

Currently when the storage compatibility mode is set to 4, it set the current write version of the ssatable to the latest compatible with 4 (N). However, it additionally make new sstables (O) recognized as invalid, which is wrong. We should be able to read all versions of supported sstables regardless of the compatibility mode. 

We should also ensure testing the scenario when the node tries to stream the new format sstable (node with compat mode = upgrading) to a node with compat mode = 4 - the test should demonstrate that such streaming is refused
Bug: Add replication to rejection and progress barrier pings
Status: Triage Needed
Priority: Normal
Created: 04/Dec/23 13:19

In some cases, we can reduce a number of round-trips in p2p communication. For example, progress barrier can not only attempt to collect majorities, but also send the latest epoch. In majority of cases, node will have to blockingly apply the epoch and progress barrier will succeed right away, resulting into quicker convergence, and we will only have to fill the gaps in case there are any.


During rejection, we also can preventively send the epoch we are at, since the remote processor will attempt to catch up before it makes a retry decision.


Both cases can significantly reduce a number of round trips for simulator, but in prod clusters will probably only be a “nice to have” optimisation.
Bug: Cap epoch during Paxos contention on retries
Status: Triage Needed
Priority: Normal
Created: 04/Dec/23 13:18

Currently, we are performing catch up to the “latest existing” epoch on CMS rejections (both Paxos and non-paxos). But in some of the cases, we actually _do_ know the epoch that is highest in the log, and there is a chance there is already a debounce task that attempts to catch up to that epoch. This means that we could:


a) return Epoch alongside with [applied] value from DistributedLogKeyspace#tryCommitOne, since it is guaranteed to be the highest known epoch that can be read serially
b) return latest Epoch to the remote processor while returning rejection results, if we know we are at the latest epoch
Bug: Allow (at least in principle, not in practice) different Partitioners per ks
Status: Triage Needed
Priority: Normal
Created: 04/Dec/23 13:16

We could remove partitioner as a single, top-level concept in {{ClusterMetadata}} and allow it to be set on a per-keyspace basis. In the first instance, we could enforce a constraint that every non-local keyspace uses the same partitioner but over time we may want to relax that.
Bug: Allow for easy and safe ways to mutate metadata for operators
Status: Triage Needed
Priority: Normal
Created: 04/Dec/23 13:16

Currently, we have two scenarios when the cluster can be put in a (very) bad state by a trivial mistake:
1. When forcing a snapshot, operator may accidentally force a lower _epoch_. If a lower epoch is forced, we will have a log that contains two epochs that are in different periods:
{epoch=23, period=11, kind=FORCE_SNAPSHOT}
{epoch=23, period=8, kind=CUSTOM}
2. When forcing a snapshot, operator may accidentally force a lower _period_, which seems to be even more dangerous. In this case, we will create a new period that immediately follows the last period in the log, but the cluster metadata that is forced in this epoch will have the period that was forced. All subsequent attempts to commit in such cluster will be unsuccesful, since the highest epoch of the period in question will not match the one that is subsequent to the metadata epoch.


Both cases could be fixed by always forcing the epoch that is subsequent to the one that is being forced. It is debatable, if we should disallow commits with +1000 epochs that are seemingly safe altogether, since they seem to be a safeguard in cases when there are ongoing changes in the cluster. But if we could make a clusterwide “pause” to guarantee we remain on the same epoch, probably we can do even that.

Additionally, we should consider switching from using IF NOT EXISTS for new period creation to some form of UPDATE that would detect the problem we have here. That said, I am not fully certain if we will have NULLs in the conditional for non-existing partitions.