Project: Cassandra (App Definition and Development / Database)
File: SSTable_API.md

/ SSTable format is an implementation of the `SSTableFormat` interface. It is responsible for creating readers, writers, scrubbers, verifiers, and other components for processing the sstables. An SSTable format implementation comes with a factory class implementing the `SSTableFormat.Factory` interface. The factory is required to provide a unique name of the implementation and a method for creating the format instance. SSTable format factories are discovered using mechanism. The loaded format implementations can be used to read the existing sstables. The write format is chosen based on the configuration. If it is not specified, `BigFormat` implementation is assumed. Optional SSTable formats configuration can be supplied in the cassandra.yaml file under the `sstable` key. ```yaml sstable: selected_format: name of the default SSTableFormat implementation format: format1 name: param1: format specific parameter 1 param2: format specific parameter 2 format2 name: param1: format specific parameter 1 param2: format specific parameter 2 ``` Each implementation must have a unique name that is used to unanonimously identify the format. The name must be consistently returned by `name()` methods in the `SSTableFormat` and `SSTableFormat.Factory` implementations. It must include only lowercase ASCII letters. Parameters specified under the key named after the format name are passed to the factory method of the corresponding implementation. All of those parameters are optional and depend on the implementation. The assumed default configuration - which is equivalent to empty configuration: ```yaml sstable: selected_format: big ``` Example configuration which uses `bti` as the default: ```yaml sstable: selected_format: bti format: big: param1: value1 param2: value2 bti: param1: value1 param2: value2 ``` Each sstable consists of a set of components - required and optional. A component constitutes an identifier required to obtain the exact file with an sstable descriptor. Components are grouped by type. A type may define either a singleton component (for example, stats component) or a non-singleton component (for example, secondary index component). A set of generic types of components that are thought of as common to all the sstable implementations is defined in the class. They include singleton types like `DATA`, `COMPRESSION_INFO`, `STATS`, `FILTER`, `DIGEST`, `CRC`, and `TOC`, which comes with predefined singleton component instances, as well as non-singleton types like `SECONDARY_INDEX` and `CUSTOM`. Apart from the generic components, each sstable format implementation may describe its specific component types. For example, the big table format describes additionally `PRIMARY_INDEX` and `SUMMARY` singleton types and the corresponding singleton components (see ). Custom types can be created with one of the `Component.Type.create(name, repr, streamable, formatClass)`, `Component.Type.createSingleton(name, repr, streamable, formatClass)` methods. Each created type is registered in a global types' registry. Types registry is hierarchical which means that an sstable implementation may use types defined for its format class and for all parent format classes (for example, the types defined for the `BigFormat` class extend the set of types defined for the `SSTableFormat` interface). For example, types defined for `BigFormat`: ```java public static class Types extends SSTableFormat.Components.Types { public static final Component.Type PRIMARYINDEX = Component.Type.createSingleton("PRIMARYINDEX", "Index.db", true, BigFormat.class); public static final Component.Type SUMMARY = Component.Type.createSingleton("SUMMARY", "Summary.db", true,

} ``` Singleton components are immediately associated with the singleton types and retrieved with the `<type>.getSingleton()` method: ```java public static class Components extends AbstractSSTableFormat.Components { public final static Component PRIMARYINDEX = Types.PRIMARYINDEX.getSingleton(); public final static Component SUMMARY = Types.SUMMARY.getSingleton(); } ``` Non-singleton components are created explicitly as follows: ```java Component idx1 = Types.SECONDARYINDEX.createComponent("SIidx1.db"); ``` We strongly suggest the main format class to extend because it includes the expected implementation of a couple of methods that should not be reimplemented differently. Cassandra either initializes the sstable format class as a singleton by calling its constructor or obtains the instance by accessing a static field called `instance` in the class. As a part of the initialization, Cassandra calls the `setup` method and provides the configuration parameters. Right after initialization, Cassandra calls the `allComponents` method to confirm all the components defined for the format are initialized and usable. SSTable format defines a couple of collections of components. You should declare those collections as constant and immutable sets. An sstable reader () is responsible for reading the data from an sstable. It is created by a simple builder () or a _loading builder_ (). The builders are supplied by a reader factory (). The constructor of a particular `SSTableReader` implementation should accept two parameters - one is the format-specific simple builder and the other one is an sstable owner (usually a `ColumnFamilyStore` instance, but it can be null either). The constructor should be simple and not do anything but assign internal fields with values from the builder. A simple builder does not perform any logic except basic validation - it only stores the provided values the reader constructor can access. A new reader implementation should include a public static simple builder inner class that extends the `SSTableReader.Builder` generic reader builder (or `SSTableReaderWithFilter.Builder`, see ). In contrast to the simple builder, a loading builder can perform additional operations like more complex validation, opening resources, loading caches, indexes, filters, etc. It internally creates a simple builder and eventually instantiates a reader. Note that if the builder carries some closeable resources to the reader, they should be returned by the `setupInstance` method. You will find some `cloneXXX` methods to implement - remember to create a reader clone in a lambda passed to the `runWithLock()` method. It is convenient to implement the `unbuildTo` method, which takes a simple builder and initializes it so that the builder can produce the same reader. The method should also take the `sharedCopy` boolean argument denoting whether it should copy the fields referencing closeable resources to the builder directly or as (shared) copies. The convention also requires copying the resources only if they are unset in the builder (the field is null). The method should call the `super.unbuildTo` method as a first step so that all the fields managed by the parent class are copied and in the actual implementation only the fields specific to this format have to be assigned. For example, the implementation of that method in a reader for the big table format is as follows: ```java protected final Builder unbuildTo(Builder builder, boolean sharedCopy) { Builder b = super.unbuildTo(builder, sharedCopy); if

== null) b.setIndexFile(sharedCopy ? sharedCopyOrNull(ifile) : ifile); if (builder.getIndexSummary() == null) b.setIndexSummary(sharedCopy ? sharedCopyOrNull(indexSummary) : indexSummary); b.setKeyCache(keyCache); return b; } ``` If the sstable includes a filter, the reader class should extend the abstract reader (and its simple builder should extend the `SSTableReaderWithFilter.SSTableReaderWithFilterBuilder` builder). The reader with filter provides the `isPresentInFilter` method for extending implementation. It also implements other filter-specific methods the system relies on if the implemented reader extends that class. Note that if the implemented reader extends the `SSTableReaderWithFilter` class, it should include the `FILTER` component in the appropriate component sets. The reader with filter implementation comes with additional - read more about custom metrics support . Some sstable format implementations, such as big table format, may use index summaries. If a reader uses _index summaries_ it should implement the interface. The support for index summaries comes with additional - read more about custom metrics support . If an sstable format implementation uses row key cache, it should implement the interface. In particular, it should store a `KeyCache` instance and return it with the `getKeyCache()` method. The interface has the default implementations of several methods the system relies on if the reader implements the `KeyCacheSupport` interface. The interface comes with additional - read more about custom metrics support . A custom sstable format implementation may provide additional metrics on a table, keyspace, and global level. Those metrics are accessible via JMX. The `SSTableFormat` implementation exposes the additional metrics by implementing the `SSTableFormat.getFormatSpecificMetricsProviders` method. The method should return a singleton object implementing the interface. Currently, there is only support for custom gauges, but it can be extended when needed. Each custom metric (gauge) is an implementation of the abstract class. Although the class expects the implementation to provide a gauge for each level of aggregation, there is a helper class - which does that automatically with a supplied reduction lambda. There is class which is a partial implementation of the `MetricsProviders` interface and leverages `SimpleGaugeProvider` in the offered methods. Example - additional metrics for sstables supporting index summaries (see for a full example): ```java private final GaugeProvider<Long> indexSummaryOffHeapMemoryUsed = newGaugeProvider("IndexSummaryOffHeapMemoryUsed", 0L, r -> r.getIndexSummary().getOffHeapSize(), Long::sum); ``` An sstable writer () is responsible for writing the data to sstable files. It is created by a builder (). The builder is supplied by a writer factory (). There are not many methods to be implemented in the writer. The most notable one is `append` which should write the provided partition to disk. However, there is a generic default implementation which handles things like writing to the data file using the default serializers, generic support for a partition index, notifications, metadata collection, and building a filter. The writer triggers fine-grained events when data are added and those methods can be overridden in the subclasses to apply specific behaviours (for example, `onPartitionStart`, `onRow`, `onStaticRow`, etc.). Eventually it calls an abstract `createRowIndexEntry` method which should be implemented in the subclass. A custom sstable format should also come with its own verifier and scrubber implementing and interfaces correspondingly. A generic partial implementation is provided in and .