Scylla Open Source Release 4.2
Tags: Scylladb, Database, Dynamodb, NoSQL, Cassandra

by Tzach Livyatan

The Scylla team is pleased to announce the release of Scylla Open Source 4.2, a production-ready release of our open source NoSQL database.

Scylla is an open source, NoSQL database with superior performance and consistently low latencies. Find the Scylla Open Source 4.2 repository for your Linux distribution here. Scylla 4.2 Docker is also available.

Scylla 4.2 includes performance and stability improvements, more operations to the Alternator API (our Amazon DynamoDB-compatible interface) and bug fixes. The release also improves query performance by adding a binary search to SSTable promoted indexes (see below).

Please note that only the last two minor releases of the Scylla Open Source project are supported. Starting today, only Scylla Open Source 4.2 and Scylla 4.1 are supported, and Scylla 4.0 is retired.

Related Links

New features in Scylla 4.2

Performance: Enable binary search in SSTable promoted index

The “promoted index” is a clustering key index within a given partition stored as part of the SSTable index file structure. The primary purpose is to improve the efficiency of column range lookup in large, with many rows, partitions.

Before Scylla 4.2, lookups in the promoted index are done by linearly scanning the index (the lookup is O(N)). However, for large partitions, this approach is costly and inefficient. It consumes a lot of CPU and I/O and does not deliver efficient performance.

Starting from this release, the reader performs a binary search on the MC SSTable promoted index (the search time is now O(log N)).

For example, testing performed with large partitions consisted of 10,000,000 rows(a value size of 2000, partition size of 20 GB and an index size of 7 MB) demonstrates that the new release is 12x faster, CPU utilization is 10x times smaller, disk I/O utilization is 20x less.

The following charts present the differences in search time and disk utilization for two cases:

Middle: Queried row is in the middle of a (large) partition

Head: Queried row is at the head, as the first element of a (large) partition

As expected, the binary search shines in the middle case and has similar results in the (unlikely) case of searching for the first element.

More details on the results here #4007

Alternator

New Alternator’s SSL options (Alternator Client to Node Encryption on Transit)

Till this release, Alternator SSL configuration was set in the section server_encryption_options of Scylla configuration file. The name was misleading, as the section was also used for the unrelated node-to-node encryption on transit.

Starting from this release, a new section alternator_encryption_options is used to define Alternator SSL.

The old section is supported for backward compatibility, but it is highly recommended to create the new section in scylla.yaml with the relevant parameters, for example

alternator_encryption_options:

certificate: conf/scylla.crt

keyfile: conf/scylla.key

priority_string: use default

#7204

docker : add an option to start Alternator with HTTPS. When using Scylla Docker, you can now use --alternator-https-port in addition to the existing --alternator-port . #6583

: add an option to start Alternator with HTTPS. When using Scylla Docker, you can now use in addition to the existing . #6583 Implement FilterExpression

Adding FilterExpression — a newer syntax for filtering results of Query and Scan requests #5038

Adding FilterExpression — a newer syntax for filtering results of Query and Scan requests #5038 Example usage in query API:

aws dynamodb query \

--table-name Thread \

--key-condition-expression "ForumName = :fn and Subject = :sub" \

--filter-expression "#v >= :num" \

--expression-attribute-names '{"#v": "Views"}' \

--expression-attribute-values file://values.json

Source: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html

Source: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html Alternator : support additional filtering operators #5028

: support additional filtering operators #5028 All operators are now supported. Previously, only the “EQ” operator was implemented.

Either “OR” or “AND” of conditions (previously only “AND”).

Correctly returning Count and ScannedCount for post-filter and pre-filter item counts, respectively.

Additional Features

SSTable Reshard and Reshape in upload to directory

Reshard and Reshape are two transformation on SSTables:

Reshard and Reshape are two transformation on SSTables: Reshard — Splitting a SSTable, that is owned by more than one shard (core), into SSTables that are owned by a single shard, for example, when restoring data from a different server, importing sstSSTables from Apache Cassandra, or changing the number of cores in a machine (upscale)

Reshape — Rewrite a set of SSTable to to satisfy a compaction strategy’s criteria., for example, restoring data from an old backup, before the strategy update.

Until this release, Scylla first moved files from upload to to data directory, and then reshaped and reshard the data. In some cases, this results in a too high number of files in the data directory.

Starting from Scylla 4.2, Scylla first reshard and reshape the SSTables in the upload dir, and only then move them to data directory. This ensures SSTables already match the number of cores and compaction strategy when moved to the data directory, resulting in a smaller number of overall compactions.

SSTtable Reshard and Reshape also happens on boot (if needed) prior to making the node online — making sure the node doesn’t have a lot of backlog work to do in addition to serving traffic.

More on this in commits

https://github.com/scylladb/scylla/commit/94634d9945064d7be41b21ac2af0ae36cef37873

https://github.com/scylladb/scylla/commit/7351db7cab7bbf907172940d0bbf8b90afde90ba

Setup : swapfile setup is now part of scylla_setup .

Scylla setup already warns users when the swap is not set. Now it prompts for the swap file directory #6539

: swapfile setup is now part of . Scylla setup already warns users when the swap is not set. Now it prompts for the swap file directory #6539 Setup: default root disk size is increased from 10G to 30GB. scylla-machine-image #48

Experimental features in Scylla 4.2

Change Data Capture (CDC) . While functionally complete, we are still testing CDC to validate it is production-ready towards GA in a following Scylla 4.x release. No API updates are expected. Refer to the Scylla Open Source 4.1 Release Notes for more information.

. While functionally complete, we are still testing CDC to validate it is production-ready towards GA in a following Scylla 4.x release. No API updates are expected. Refer to the Scylla Open Source 4.1 Release Notes for more information. There’s a new nodetool command #6498

CDC is explicitly disabled on counters #6553

Preimage is no longer created for missing rows #7119

Postimage is no longer created for deleted rows #7120

CDC tables were renamed #6577

Monitoring:

For Metrics updates from 4.1 to 4.2, see here.

Scylla 4.2 dashboard isare available in the latest Scylla Monitoring Stack release 3.2.4 or later.

Other bugs fix and updates in the release