av9ash/GitBugs huggingface dataset:
Bug: Creating a SASI index after creating an SAI index breaks secondary index queries
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 18/Oct/23 19:26
Resolved: 05/Jan/24 10:25

In the 5.0 branch, HEAD is e45c1092f91edd63591f562b2120ea6a5fd3edd5, I was able to break secondary indexes by doing the following:

{code}
cqlsh -e "create KEYSPACE test WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};"
cqlsh -e "create table test.blah (id int primary key, val text);"
cqlsh -e "create INDEX on test.blah (val) using 'sai';"

bin/nodetool flush

❯ cqlsh -e "SELECT * from test.blah WHERE val = 'something';"

 id | val
----+-----

(0 rows)

cqlsh -e "create custom INDEX on test.blah (val) using 'org.apache.cassandra.index.sasi.SASIIndex';"

Warnings :
SASI indexes are experimental and are not recommended for production use.

cqlsh -e "SELECT * from test.blah WHERE val = 'something';"
<stdin>:1:ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read] message="Operation failed - received 0 responses and 1 failures: UNKNOWN from localhost/127.0.0.1:7000" info={'consistency': 'ONE', 'required_responses': 1, 'received_responses': 0, 'failures': 1, 'error_code_map': {'127.0.0.1': '0x0000'}}
{code}

Server throws an exception:
{code}
ERROR [ReadStage-2] 2023-10-18 12:09:42,391 JVMStabilityInspector.java:70 - Exception in thread Thread[ReadStage-2,10,SharedPool]
java.lang.RuntimeException: java.lang.ClassCastException: class org.apache.cassandra.index.sai.StorageAttachedIndex cannot be cast to class org.apache.cassandra.index.sasi.SASIIndex (org.apache.cassandra.index.sai.StorageAttachedIndex and org.apache.cassandra.index.sasi.SASIIndex are in unnamed module of loader 'app')
	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2585)
	at org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163)
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassCastException: class org.apache.cassandra.index.sai.StorageAttachedIndex cannot be cast to class org.apache.cassandra.index.sasi.SASIIndex (org.apache.cassandra.index.sai.StorageAttachedIndex and org.apache.cassandra.index.sasi.SASIIndex are in unnamed module of loader 'app')
	at org.apache.cassandra.index.sasi.plan.QueryController.getIndex(QueryController.java:96)
	at org.apache.cassandra.index.sasi.plan.Operation.analyzeGroup(Operation.java:282)
	at org.apache.cassandra.index.sasi.plan.Operation$Builder.complete(Operation.java:433)
	at org.apache.cassandra.index.sasi.plan.SASIIndexSearcher.analyze(SASIIndexSearcher.java:65)
	at org.apache.cassandra.index.sasi.plan.SASIIndexSearcher.search(SASIIndexSearcher.java:77)
	at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:425)
	at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:2184)
	at org.apache.cassandra.service
{code}

Dropping the SASI index restores correct behavior.
Bug: Revert unnecessary read lock acquisition when reading ring version in TokenMetadata introduced in CASSANDRA-16286
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Nov/23 23:09
Resolved: 09/Jan/24 21:11

CASSANDRA-16286 achieved its goal of making sure that concurrent increments to the ring version would independently increment the version (i.e. not "merge" multiple invalidations into single versions), but it also unnecessarily replaced the volatile read on {{ringVersion}} w/ making {{readVersion}} non-volatile and acquiring the read lock on the fair {{ReadWriteLock}} in {{TokenMetadata}}. This can result in unnecessary queueing w/ high CPU usage/read volume. For example, you might see this on a 4.0 cluster...

{noformat}
"Native-Transport-Requests-99" #271 daemon prio=5 os_prio=0 cpu=5822566.56ms elapsed=19477779.40s tid=0x00007fcc96c31b00 nid=0xb7bd waiting on condition  [0x00007fcb7f144000]
  java.lang.Thread.State: WAITING (parking)
       at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)
       - parking to wait for  <0x00000005c0ab92a0> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
       at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.16/AbstractQueuedSynchronizer.java:885)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(java.base@11.0.16/AbstractQueuedSynchronizer.java:1009)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@11.0.16/AbstractQueuedSynchronizer.java:1324)
       at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@11.0.16/ReentrantReadWriteLock.java:738)
       at org.apache.cassandra.locator.TokenMetadata.getRingVersion(TokenMetadata.java:1389)
       at org.apache.cassandra.locator.AbstractReplicationStrategy.getCachedReplicas(AbstractReplicationStrategy.java:82)
       at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalReplicas(AbstractReplicationStrategy.java:116)
       at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalReplicasForToken(AbstractReplicationStrategy.java:109)
       at org.apache.cassandra.locator.ReplicaLayout.forTokenWriteLiveAndDown(ReplicaLayout.java:209)
       at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:328)
       at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1426)
       at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:937)
{noformat}

Reverting to a volatile read makes this no longer possible, but keeps the fix from CASSANDRA-16286 intact.
Bug: Add support of vector type to COPY command
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Nov/23 17:06
Resolved: 22/Jan/24 14:41

Currently it's not possible to import rows with vector literals via {{COPY}} command.

STR:
 * Create a table
{code:sql}
CREATE TABLE testcopyfrom (id text PRIMARY KEY, embedding_vector VECTOR<FLOAT, 6>
{code}

 * Prepare csv file with sample data, for instance:
{code:sql}
1,"[0.1, 0.2, 0.3, 0.4, 0.5, 0.6]"
2,"[-0.1, -0.2, -0.3, -0.4, -0.5, -0.6]" {code}

 * in cqlsh run
{code:sql}
COPY ks.testcopyfrom FROM data.csv
{code}

It will result in getting:
{code:sql}
TypeError: Received an argument of invalid type for column "embedding_vector". Expected: <class 'cassandra.cqltypes.VectorType(6)'>, Got: <class 'str'>; (required argument is not a float){code}
Bug: logback-core-1.2.12.jar vulnerability: CVE-2023-6378
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/Dec/23 11:58
Resolved: 06/Dec/23 12:43

https://nvd.nist.gov/vuln/detail/CVE-2023-6378

{quote}
A serialization vulnerability in logback receiver component part of logback version 1.4.11 allows an attacker to mount a Denial-Of-Service attack by sending poisoned data. 
{quote}
Bug: Check whether the startup warnings for unknown modules represent a legit problem or cosmetic issue
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Nov/23 17:46
Resolved: 22/Jan/24 16:30

During the 5.0 alpha 2 release [vote|https://lists.apache.org/thread/lt3x0obr5cpbcydf5490pj6b2q0mz5zr], [~paulo] raised the following concerns:
{code:java}
Launched a tarball-based 5.0-alpha2 container on top of
"eclipse-temurin:17-jre-focal" and the server starts up fine, can run
nodetool and cqlsh.
I got these seemingly harmless JDK17 warnings during startup and when
running nodetool (no warnings on JDK11):
WARNING: Unknown module: jdk.attach specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-exports
WARNING: Unknown module: jdk.compiler specified to --add-opens
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by
org.apache.cassandra.security.ThreadAwareSecurityManager
(file:/opt/cassandra/lib/apache-cassandra-5.0-alpha2-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of
org.apache.cassandra.security.ThreadAwareSecurityManager
WARNING: System::setSecurityManager will be removed in a future release
Anybody knows if these warnings are legit/expected ? We can create
follow-up tickets if needed.
$ java --version
openjdk 17.0.9 2023-10-17
OpenJDK Runtime Environment Temurin-17.0.9+9 (build 17.0.9+9)
OpenJDK 64-Bit Server VM Temurin-17.0.9+9 (build 17.0.9+9, mixed mode,
sharing)
{code}
{code:java}
Clarification: - When running nodetool only the "Unknown module" warnings show up. All warnings show up during startup.{code}
We need to verify whether this presents a real problem in the features where those modules are expected to be used, or if it is a false alarm.
Bug: local consistencies may get timeout if blocking read repair is sending the read repair mutation to other DC 
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Nov/23 22:59
Resolved: 22/Feb/24 23:14

For a two DCs cluster setup. When a new node is being added to DC1, for blocking read repair triggered by local_quorum in DC1, it will require to send read repair mutation to an extra node(1)(2). The selector for read repair may select *ANY* node that has not been contacted before(3) instead of selecting the DC1 nodes. If a node from DC2 is selected, this will cause 100% timeout because of the bug described below:

When we initialized the latch(4) for blocking read repair, the shouldBlockOn function will only return true for local nodes(5), the blockFor value will be reduced if a local node doesn't require repair(6). The blockFor is same as the number of read repair mutation sent out. But when the coordinator node receives the response from the target nodes, the latch only count down for nodes in same DC(7). The latch will wait till timeout and the read request will timeout.

This can be reproduced if you have a constant load on a 3 + 3 cluster when adding a node. If you have someway to trigger blocking read repair(maybe by adding load using stress tool). If you use local_quorum consistency with a constant read after write load in the same DC that you are adding node. You will see read timeout issue from time to time because of the bug described above

 

I think for read repair when selecting the extra node to do repair, we should prefer local nodes than the nodes from other region. Also, we need to fix the latch part so even if we send mutation to the nodes in other DC, we don't get a timeout.

(1)[https://github.com/apache/cassandra/blob/cassandra-4.0.11/src/java/org/apache/cassandra/locator/ReplicaPlans.java#L455]

(2)[https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/ConsistencyLevel.java#L183]

(3)[https://github.com/apache/cassandra/blob/cassandra-4.0.11/src/java/org/apache/cassandra/locator/ReplicaPlans.java#L458]

(4)[https://github.com/apache/cassandra/blob/cassandra-4.0.11/src/java/org/apache/cassandra/service/reads/repair/BlockingPartitionRepair.java#L96]

(5)[https://github.com/apache/cassandra/blob/cassandra-4.0.11/src/java/org/apache/cassandra/service/reads/repair/BlockingPartitionRepair.java#L71]

(6)[https://github.com/apache/cassandra/blob/cassandra-4.0.11/src/java/org/apache/cassandra/service/reads/repair/BlockingPartitionRepair.java#L88]

(7)[https://github.com/apache/cassandra/blob/cassandra-4.0.11/src/java/org/apache/cassandra/service/reads/repair/BlockingPartitionRepair.java#L113]
Bug: logback-core-1.2.12.jar vulnerability: CVE-2023-6481
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Dec/23 11:49
Resolved: 08/Dec/23 14:57

https://nvd.nist.gov/vuln/detail/CVE-2023-6481

{quote}
A serialization vulnerability in logback receiver component part of logback version 1.4.13, 1.3.13 and 1.2.12 allows an attacker to mount a Denial-Of-Service attack by sending poisoned data. 
{quote}
Bug: nodetool assassinate may cause thread serialization for that node
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Dec/23 17:58
Resolved: 11/Jan/24 11:19

When assassinate an ip address that is not in the gossip map, a "corrupted" entry will be inserted into the gossip map. [(1)|https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/gms/Gossiper.java#L810]

For example, if we do "nodetool assassinate 10.1.1.1"

we will get an entry like below by running "nodetool gossipinfo":

 
{code:java}
/10.1.1.1
  generation:1702006511
  heartbeat:9999
  STATUS:209516:LEFT,-8393921141401589197,1702265651923
  STATUS_WITH_PORT:209515:LEFT,-8393921141401589197,1702265651923
  TOKENS: not present {code}
 

This entry in endpointStateMap will cause issue for [isUpgradingFromVersionLowerThan|https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/gms/Gossiper.java#L2284] function. Because the [upgradeFromVersionSupplier|https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/gms/Gossiper.java#L191] supplier will always set the [allHostsHaveKnownVersion|https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/gms/Gossiper.java#L216] flag to false so no memoized value will be returned. The "get" function will always require a lock from this [line|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/utils/ExpiringMemoizingSupplier.java#L66].

If application is using "fetchAll", the native-transport-requests thread will hit this [line|https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/db/filter/ColumnFilter.java#L574]. This means all the native-transport-requests thread is serialized, also, the lock is shared by GossipStage threads. It means if a node in a cluster with the corrupted gossip map is restart, the node will run into this problem.

To fix the issue,
 # Why we want to add a dummy entry for nodetool assassinate if the endpoint is not in the map anymore. Should we do nothing or throw exception if the node is not in the gossip map anymore?
 # Before checking if a version is null, we should make sure the node is not a dead node. A decommissioned node, a left node should not be considered part of the cluster anymore when calculating "upgradeInProgressPossible"
Bug: Fix cqlshlib cython test
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 08/Dec/23 11:45
Resolved: 14/Dec/23 14:51

This is failing on [trunk|https://app.circleci.com/pipelines/github/driftx/cassandra/1405/workflows/ff7b5121-b6df-4290-8d63-bcc6d6385724/jobs/65554] and [5.0|https://app.circleci.com/pipelines/github/driftx/cassandra/1406/workflows/f7f2b35c-1233-43d7-a942-218f1712add4/jobs/65649]:

{quote}
PermissionError: [Errno 13] Permission denied: '/.ccm'
{quote}
Bug: Aggregation (SUM) in a transaction block fails
Status: Open
Priority: Normal
Created: 26/Oct/23 20:28

*System configuration and information:*

Single node Cassandra with Accord transactions enabled running on docker

Built from commit: [a7cd114435704b988c81f47ef53d0bfd6441f38b|https://github.com/apache/cassandra/commit/a7cd114435704b988c81f47ef53d0bfd6441f38b]

CQLSH: [cqlsh 6.2.0 | Cassandra 5.0-alpha2-SNAPSHOT | CQL spec 3.4.7 | Native protocol v5]

 

*Steps to reproduce in CQLSH:*
{code:java}
CREATE KEYSPACE accord WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'}  AND durable_writes = true;{code}
{code:java}
CREATE TABLE accord.accounts (
    partition text,
    account_id int,
    balance int,
    PRIMARY KEY (partition, account_id)
) WITH CLUSTERING ORDER BY (account_id ASC)
{code}
{code:java}
BEGIN TRANSACTION
    INSERT INTO accord.accounts (partition, account_id, balance) VALUES ('default', 0, 100);
    INSERT INTO accord.accounts (partition, account_id, balance) VALUES ('default', 1, 100);
COMMIT TRANSACTION;{code}
bug happens after executing the following statement:

I expect the result to be a single row consisting of the sum of all the balances (100 + 100 = 200).
{code:java}
BEGIN TRANSACTION
    SELECT SUM(balance)
    FROM accord.accounts
    WHERE
      partition = 'default'
      AND account_id IN (0, 1);
COMMIT TRANSACTION;{code}
 

Bug / Error:
======================================================================

The return result includes all the rows with no sum.
{code:java}
system.sum(balance)
----------------
            100
            100{code}
 

Note that executing the same statement without the transaction returns the expected result:
{code:java}
system.sum(balance)
----------------
            200{code}
{color:#172b4d}This was found while testing Accord transactions with [~henrik.ingo] and team.{color}
Bug: Filter replicas for counter mutation leader to be JOINED replicas to decouple from RPC_READY state
Status: Needs Committer
Priority: Normal
Created: 27/Nov/23 16:14

In CASSANDRA-18935, what I did was that if transport is turned off while node is up, it will set RPC_READY to "false". Before 18935 was in, if you turned off transport while node was up and transport was on, RPC_READY was not changed to false but it stayed to be "true" for ever.

Clearly, this was wrong, so I fixed that, but when I did that, the code which decides who will be the leader of counter mutation was broken, because that node filtered out all nodes for which RPC_READY is false, which never happened before, because nobody has set it to "false" on turned off transport.

So, we decided to revert this regression in 5.0-rc1 and I was waiting for CEP-21 to be in because I saw it is rewritten there with TODO to filter JOINED nodes. 

Hence, we can decouple this logic and fix RPC_READY status finally.
Bug: Python dtest TestWriteFailures.test_paxos is failing with Paxos V2
Status: Open
Priority: Normal
Created: 01/Dec/23 15:20

With configuration changed to engage Paxos V2 with repaired state purging, the dtest fails with:
{code}
test_paxos
write_failures_test.TestWriteFailures

self = <write_failures_test.TestWriteFailures object at 0x7f396bb68910>

    def test_paxos(self):
        """
        A light transaction receives a WriteFailure
        """
>       exc = self._perform_cql_statement("INSERT INTO mytable (key, value) VALUES ('key1', 'Value 1') IF NOT EXISTS")

write_failures_test.py:202: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
write_failures_test.py:88: in _perform_cql_statement
    session.execute(statement)
../env3.7/src/cassandra-driver/cassandra/cluster.py:2618: in execute
    return self.execute_async(query, parameters, trace, custom_payload, timeout, execution_profile, paging_state, host, execute_as).result()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <ResponseFuture: query='<BoundStatement query="INSERT INTO mytable (key, value) VALUES ('key1', 'Value 1') IF NOT EXIS...ncy': 'SERIAL', 'required_responses': 2, 'received_responses': 1, 'write_type': 'CAS'} coordinator_host=127.0.0.1:9042>

    def result(self):
        """
        Return the final result or raise an Exception if errors were
        encountered.  If the final result or error has not been set
        yet, this method will block until it is set, or the timeout
        set for the request expires.
    
        Timeout is specified in the Session request execution functions.
        If the timeout is exceeded, an :exc:`cassandra.OperationTimedOut` will be raised.
        This is a client-side timeout. For more information
        about server-side coordinator timeouts, see :class:`.policies.RetryPolicy`.
    
        Example usage::
    
            >>> future = session.execute_async("SELECT * FROM mycf")
            >>> # do other stuff...
    
            >>> try:
            ...     rows = future.result()
            ...     for row in rows:
            ...         ... # process results
            ... except Exception:
            ...     log.exception("Operation failed:")
    
        """
        self._event.wait()
        if self._final_result is not _NOT_SET:
            return ResultSet(self, self._final_result)
        else:
>           raise self._final_exception
E           cassandra.WriteTimeout: Error from server: code=1100 [Coordinator node timed out waiting for replica nodes' responses] message="CAS operation timed out: received 1 of 2 required responses after 0 contention retries" info={'consistency': 'SERIAL', 'required_responses': 2, 'received_responses': 1, 'write_type': 'CAS'}

../env3.7/src/cassandra-driver/cassandra/cluster.py:4894: WriteTimeout
{code}
Bug: CMS: Additional nodes are not added to CMS
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Dec/23 10:40
Resolved: 14/Feb/24 07:39

When creating a 3 node cluster on a local machine using 3 loopback addresses

127.0.0.1
127.0.0.2
127.0.0.3

The nodes are created correctly and the first node is assigned as a CMS node as shown:

{{bin/nodetool p 7199 status{-}{-}}}
{{{}Datacenter: datacenter1{}}}{{{}======================={}}}{{{}Status=Up/Down{}}}{{{}|/ State=Normal/Leaving/Joining/Moving{}}}{{{}-  Address    Load       Tokens  Owns (effective)  Host ID                               Rack{}}}

{{UN  127.0.0.3  75.55 KiB  16      76.0%             6d194555-f6eb-41d0-c000-000000000003  rack1}}

{{UN  127.0.0.2  67.97 KiB  16      59.3%             6d194555-f6eb-41d0-c000-000000000002  rack1}}

{{UN  127.0.0.1  81 KiB     16      64.7%             6d194555-f6eb-41d0-c000-000000000001  rack1}}

{{bin/nodetool -p 7199 describecms}}
{{Cluster Metadata Service:}}
{{Members: /127.0.0.1:7000}}
{{Is Member: true}}
{{Service State: LOCAL}}
{{Is Migrating: false}}
{{Epoch: 14}}
{{Local Pending Count: 0}}
{{Commits Paused: false}}
{{{}Replication factor: ReplicationParams{class=org.apache.cassandra.locator.MetaStrategy, datacenter1=1{}}}}



{{However after doing a reconfigurecms to create a replication factor of 3, it seems that there is still only one member of cms.}}



{{bin/nodetool -p 7199 reconfigurecms datacenter1:3}}
{{bin/nodetool -p 7199 describecms}}
{{Cluster Metadata Service:}}
{{Members: /127.0.0.1:7000}}
{{Is Member: true}}
{{Service State: LOCAL}}

Is Migrating: false

Epoch: 16

Local Pending Count: 0

Commits Paused: false

Replication factor: ReplicationParams\{class=org.apache.cassandra.locator.MetaStrategy, datacenter1=3}

Is this correct, should all 3 nodes be shown in the Members section ?
Bug: RepairJobTest fails with paxos_variant: v2
Status: Open
Priority: Normal
Created: 21/Nov/23 12:30

{{testEndToEndNoDifferences}} and {{testValidationFailure}} time out. The paxos repair stage never completes --- it's does not appear to be properly wired up (e.g. {{outboundSink}} returns before processing the PAXOS2-specific cases).

This happens for all three options of {{paxos_state_purging}} and both with and without {{storage_compatibility_mode: NONE}}.
Bug: Revisit use of sealed period lookup tables
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 11/Dec/23 13:57
Resolved: 09/Feb/24 14:52

Metadata snapshots are stored locally in the {{system.metadata_snapshots}} table, which is keyed by epoch. Snapshots are retrieved from this table for three purposes:
* to replay locally during startup
* to provide log state for a peer requesting catchup
* to create point-in-time ClusterMetadata, for disaster recovery

In the majority of cases, we always want to replay from the most recent snapshot so we can usually select the appropriate snapshot by simply scanning the snapshots table in reverse, which allows us to considerably simplify the process of looking up the desired snapshot. We will continue to persist historical snapshots, at least for now, so that we are able to select arbitrary snapshots should we want to reconstruct metadata state for arbitrary epochs.