Project: Scylla (App Definition and Development / Database)
File: protocols.md

Scylla is a distributed system, with multiple Scylla nodes communicating with each other and with client nodes making requests. All these different nodes use many different protocols and TCP ports (currently, all use TCP) to communicate with each other. Some of these protocols have an external specification - for example Scylla supports Cassandra's CQL protocol and Amazon's DynamoDB protocol for client requests. Other protocols, both client-facing and internal (between Scylla nodes), are Scylla-defined. The goal of this document is to survey all these different protocols - what goes over which protocol, which port each protocol uses (and how this can be configured), how to disable or enable some of the protocols, and what part of the Scylla source code handles which protocol. Unfortunately, the long evolution of some of these protocols resulted in some of them getting confusing names, and often different names in Scylla's code, documentation and configuration. This document aims to clear up some of this confusion. The Wireshark tool can be used to inspect and understand the messages sent over many of the following protocols. This includes even Scylla's internal inter-node protocol - for instructions and examples check out the blog post: https://www.scylladb.com/2020/05/21/dissecting-scylla-packets-with-wireshark/ All inter-node communiction - communication between the Scylla nodes themselves - happen over one protocol and one TCP port - by default port 7000 (unencrypted) or 7001 (encrypted). These are the same TCP ports used by Cassandra for this purpose, but Scylla's internal protocol is completely different from Cassandra, and incompatible with it (one cannot mix Scylla and Cassandra nodes in a single cluster). Scylla's internal protocol is built on top of Seastar's "RPC" messaging mechanism, itself built on top of TCP. This protocol includes messages of different types ("verbs") - which are all listed in the source file `message/messaging_service.cc`. Here is a non-exhaustive list of things that Scylla does with these messages: "Gossip" between the nodes to maintain knowledge of the cluster topology: Which nodes belong to the cluster, their state and token ownership. Read and write (mutation) messages sent from a node receiving a client's request (a so-called coordinator) to a replica which holds relevant data. In particular, mutations to special system tables are used to maintain cluster-wide agreement on schemas (table definitions). Streaming data to new nodes and repairing data between nodes. Messages used to implement LWT (lightweight transactions). Although all these different types of messages use the same RPC protocol and the same destination port on the receiver, we don't want to multiplex all of these messages over a single TCP connection. If we do, this would allow messages of one type to severely delay messages of other types (head-of-line blocking). So Scylla opens several sockets to the same destination port and uses a different socket for different message types. The function `dogetrpcclientidx` determines which types of messages get bunched together over one socket. As of this writing, the different messages are split into four sockets. As everything in Scylla, the messaging service (providing this internal communication) is a sharded service, so each shard keeps its own sockets to each remote node. The remote node is also

We cannot know for sure which shard on the remote node will handle the messages, but we make an effort (which is not a guarantee!) that if the two nodes have the same number of shards, messages from shard N in the source node arrive to shard N in the destination node. Port 7000 is the default port for Scylla's internal communication. This choice can be overridden by the `storage_port` configuration option. This awkward name, `storage_port`, was kept for backward compatibility with Cassandra's YAML configuration file. In very early Cassandra versions, this port was used only for "storage" messages (read and write), and other messages such as gossip were sent to a different port. So today we are still stuck with this outdated name of this configuration option. Configuring `storageport` or `sslstorage_port` to 0 disables listening on the respective unencrypted/encrypted inter-node communication port. There is also a `listen_address` configuration option to set the IP address (and therefore network interface) on which Scylla should listen for the internal protocol. This address defaults to `localhost`, but in any setup except a one-node test, should be overridden. TODO: there is also `listen_interface` option... Which wins? What's the default? TODO: mention SSL, how it is configured, and `sslstorageport` (default 7001). The CQL binary protocol is Cassandra's and Scylla's main client-facing protocol. Scylla supports several Scylla-only extensions to this protocol, described in . By default, Scylla listens to the CQL protocol on port 9042, which can be configured via the `nativetransportport` configuration option. If set to 0, the OS will choose the port randomly (which also applies to every CQL port discussed later). To explicitly disable listening on a CQL port one should set its value to `~` , or `null`. Scylla also supports the CQL protocol via TLS/SSL encryption which is disabled by default and can be enabled via the `nativetransportport_ssl` configuration option. Traditional choice for the port for secure connections is 9142, but if `clientencryptionoptions` are specified and `nativetransportportssl` is not, then `nativetransport_port` will handle encrypted connections only. The same thing happens when `nativetransportport` and `nativetransportport_ssl` are set to the same value. The rules governing port assignment/encryption are summed up in the table below: ``` np := nativetransportport is set nps := nativetransportport_ssl is set ceo := clientencryptionoptions are enabled eq := nativetransportportssl == nativetransport_port +--+--+--+--+ | np | nps | ceo | eq | +--+--+--+--+ | 0 | 0 | 0 | * | => listen on nativetransportport, unencrypted | 0 | 0 | 1 | * | => listen on nativetransportport, encrypted | 0 | 1 | 0 | * | => don't listen | 0 | 1 | 1 | * | => listen on nativetransportport_ssl, encrypted | 1 | 0 | 0 | * | => listen on nativetransportport, unencrypted | 1 | 0 | 1 | * | => listen on nativetransportport, encrypted | 1 | 1 | 0 | * | => listen on nativetransportport, unencrypted | 1 | 1 | 1 | 0 | => listen on nativetransportport, unencrypted + nativetransportport_ssl, encrypted | 1 | 1 | 1 | 1 | => listen on nativetransportport(_ssl - same thing), encrypted +--+--+--+--+ ``` To allow "advanced shard-awareness" Scylla can accept CQL connections on additional port(s): `nativeshardawaretransportport` (by default 19042) and `nativeshardawaretransportport_ssl` (encrypted, disabled by default just like `nativetransportport_ssl`). The typical choice for `nativeshardawaretransportport_ssl` is

Both shard-aware ports work almost identically as their non-shard-aware counterparts, with only one difference: client connections arriving on "shard-aware" ports are routed to specific shards, determined by the client-side (local) port numbers. This feature is enabled by default and can be disabled by setting `enableshardaware_drivers: false`. The CQL protocol support can be disabled altogether by setting the `startnativetransport` option to `false`. These option names were chosen for backward-compatibility with Cassandra configuration files: they refer to CQL as the "native transport", to contrast with the older Thrift protocol (described below) which wasn't native to Cassandra. There is also a `rpc_address` configuration option to set the IP address (and therefore network interface) on which Scylla should listen for the CQL protocol. This address defaults to `localhost`, but in any setup except a one-node test, should be overridden. Note that the same option `rpc_address` applies to both CQL and Thrift protocols. TODO: there is also `rpc_interface` option... Which wins? What's the default? The Apache Thrift protocol was early Cassandra's client protocol, until it was superseded in Cassandra 1.2 with the binary CQL protocol. Thrift was still nominally supported by both Cassandra and Scylla for many years, but was recently dropped in Cassandra (version 4.0) and is likely to be dropped by Scylla in the future as well, so it is not recommended for new applications. By default, Scylla does not enable the Thrift server. In order to use it, it must be explicitly enabled by setting the `start_rpc` configuration option to true. When Thrift is enabled, by default scylla listens to the Thrift protocol on port 9160, which can be configured via the `rpc_port` configuration option. Again, this confusing name was used for backward-compatibility with Cassandra's configuration files. Cassandra used the term "rpc" because Apache Thrift is a remote procedure call (RPC) framework. In Scylla, this name is especially confusing, because as mentioned above, Scylla's internal communication protocol is based on Seastar's RPC, which has nothing to do with the "`rpc_port`" described here. There is also a `rpc_address` configuration option to set the IP address (and therefore network interface) on which Scylla should listen for the Thrift protocol. This address defaults to `localhost`, but in any setup except a one-node test, should be overridden. Note that the same option `rpc_address` applies to both CQL and Thrift protocols. TODO: there is also `rpc_interface` option... Which wins? What's the default? TODO: is there an SSL version of Thrift? Scylla also supports Amazon's DynamoDB API. The DynamoDB API is a JSON over HTTP (unencrypted) or HTTPS (encrypted) protocol. Support for this protocol is not turned on by default, and must be turned on manually by setting the `alternatorport` and/or `alternatorhttps_port` configuration options. "Alternator" is the codename of Scylla's DynamoDB API support, and is documented in more detail in . The standard ports that DynamoDB uses are the standard HTTP and HTTPS ports (80 and 443, respectively), but in tests we usually use the unprivileged port numbers 8000 and 8043 instead. There is also an `alternator_address` configuration option to set the IP address (and therefore network interface) on which Scylla should listen for the DynamoDB protocol. This address defaults to

When the HTTPS-based protocol is enabled, the server also needs to know the certificate and key files to use. The default locations of these files are `/etc/scylla/scylla.crt` and `/etc/scylla/scylla.key` respectively, but can be overridden by specifying in `alternatorencryptionoptions` the `keyfile` and `certificate` options. For example, `--alternator-encryption-options keyfile="..."`. Scylla also has partial and experimental support for the Redis API. Because this support is experimental, it is not turned on by default, and must be turned on manually by setting the `redis_port` and/or `redissslport` configuration option. The traditional port used for Redis is 6379. Regular Redis does not support SSL, so there is no traditional choice of port for it. The same `rpc_address` configuration option used by the CQL and Thrift protocols to set the IP address (and therefore network interface) on which Scylla should listen also applies to the Redis protocol. See for more information about Scylla's support for the Redis protocol. Scylla provides an HTTP-based protocol to fetch performance and activity metrics from Scylla, which is described in detail in . Scylla listens by default on port 9180 for metric requests. This port number can be configured with the `prometheus_port` configuration option, named after the Prometheus protocol - and the Prometheus server which is usually used to collect these metrics. There is also a `prometheus_address` configuration option to set the IP address (and therefore network interface) on which Scylla should listen for the metrics protocol. This address defaults to 0.0.0.0. The CQL client protocol mentioned above is useful mostly for data and table requests, but does not offer commands for administrative operations such as repair, compact, adding or removing nodes, and so on. Cassandra uses nodetool and JMX for these (see below), but Scylla's native approach is a RESTful API (HTTP requests). Scylla listens for this REST API by default on port 10000, which can be configured with the `api_port` configuration option. The REST API has no notion of authentication of authorization, and allows anyone connecting to it to perform destructive operations. Therefore, it only listens for connection on the localhost (127.0.0.1) interface. This default can be overridden by the `api_address` option - but shouldn't. The available REST API commands are listed in JSON files in the `api/api-doc/` directory of the Scylla source. A user can explore this API interactively by pointing a browser to http://localhost:10000/ui/. There is an ongoing, but incomplete, effort to replace this REST API by a newer, "v2", API. See . When complete, this v2 API will be available on the same port. You can explore the little it offers now in the aforementioned UI by replacing the URL in the box with "http://localhost:10000/v2". The "nodetool" management command connects to Scylla using Java's JMX protocol, not the REST API described above. This protocol was kept for backward compatibility with Cassandra and its unmodified "nodetool" command. To implement the JMX protocol, we have a separate project, , which runs a Java program which accepts the JMX requests supported by Cassandra, and translates them to requests to our own REST API. These REST API requests are sent to Scylla's REST API port over the loopback (localhost) interface. The port on which scylla-jmx listens is by default port 7199. This port, and the listen address, can be overridden with the `-jp` and `-ja` options (respectively) of the `scylla-jmx` script.