Project: FoundationDB (App Definition and Development / Database)
File: dynamic-knobs.md

This document is largely adapted from original design documents by Markus Pilman and Trevor Clinkenbeard. FoundationDB parameters control the behavior of the database, including whether certain features are available and the value of internal constants. Parameters will be referred to as knobs for the remainder of this document. Currently, these knobs are configured through arguments passed to `fdbserver` processes, often controlled by `fdbmonitor`. This has a number of problems: Updating knobs involves updating `foundationdb.conf` files on each host in a cluster. This has a lot of overhead and typically requires external tooling for large scale changes. All knob changes require a process restart. We can't easily track the history of knob changes. The dynamic knobs project creates a strictly serializable quorum-based configuration database stored on the coordinators. Each `fdbserver` process specifies a configuration path and applies knob overrides from the configuration database for its specified classes. The configuration database explicitly does not support the following: A high load. The update rate, while not specified, should be relatively low. A large amount of data. The database is meant to be relatively small (under one megabyte). Data is not sharded and every coordinator stores a complete copy. Concurrent writes. At most one write can succeed at a time, and clients must retry their failed writes. Each `fdbserver` process can now include a `--config_path` argument specifying its configuration path. A configuration path is a hierarchical list of configuration classes specifying which knob overrides the `fdbserver` process should apply from the configuration database. For example: ```bash $ fdbserver --config_path classA/classB/classC ... ``` Knob overrides follow descending priority: Manually specified command line knobs. Individual configuration class overrides. Subdirectories override parent directories. For example, if the configuration path is `az-1/storage/gp3`, the `gp3` configuration takes priority over the `storage` configuration, which takes priority over the `az-1` configuration. Global configuration knobs. Default knob values. For example, imagine an `fdbserver` process run as follows: ```bash $ fdbserver --datadir /mnt/fdb/storage/4500 --logdir /var/log/foundationdb --publicaddress auto:4500 --configpath az-1/storage/gp3 --knobdisableasserts false ``` And the configuration database contains: | ConfigClass | KnobName | KnobValue | |-||--| | az-2 | pagecache4k | 8e9 | | storage | mintraceseverity | 20 | | az-1 | compaction_interval | 280 | | storage | compaction_interval | 350 | | az-1 | disable_asserts | true | | \<global\> | maxmetricsize | 5000 | | gp3 | maxmetricsize | 1000 | The final configuration for the process will be: | KnobName | KnobValue | Explanation | ||-|-| | pagecache4k | \<default\> | The configuration database knob override for `az-2` is ignored, so the compiled default is used | | mintraceseverity | 20 | Because the `storage` configuration class is part of the processs configuration path, the corresponding knob override is applied from the configuration database | | compaction_interval | 350 | The `storage` knob override takes precedence over the `az-1` knob override | | disable_asserts | false | This knob is manually overridden, so all other overrides are ignored | | maxmetricsize | 1000 | Knob overrides for specific configuration classes take precedence over global knob overrides, so the global override is ignored | Clients can write to the configuration database using

Configuration database transactions are differentiated from regular transactions through specification of the `USECONFIGDATABASE` database option. In configuration transactions, the client uses the tuple layer to interact with the configuration database. Keys are tuples of size two, where the first item is the configuration class being written, and the second item is the knob name. The value should be specified as a string. It will be converted to the appropriate type based on the declared type of the knob being set. Below is a sample Python script to write to the configuration database. ```python import fdb fdb.api_version(730) @fdb.transactional def setknob(tr, knobname, knobvalue, configclass, description): tr['\xff\xff/description'] = description tr[fdb.tuple.pack((configclass, knobname,))] = knob_value @fdb.transactional def setmultipleknobs(tr): tr['\xff\xff/description'] = 'description' tr[fdb.tuple.pack((None, 'mintraceseverity',))] = '10' tr[fdb.tuple.pack(('az-1', 'mintraceseverity',))] = '20' db = fdb.open() db.options.setuseconfig_database() setknob(db, 'mintrace_severity', '10', None, 'description') setknob(db, 'mintrace_severity', '20', 'az-1', 'description') ``` Users may also utilize `fdbcli` to set and update knobs dynamically. Usage is as follows ``` setknob <knobname> <knobvalue> [config_class] getknob <knobname> [configclass] clearknob <knobname> [configclass] ``` Where `knobname` is an existing knob, `knobvalue` is the desired value to set the knob and `config_class` is the optional configuration class. Furthermore, `setknob` may be combined within a `begin\commit` to update multiple knobs atomically. If using this option, a description must follow `commit` otherwise a prompt will be shown asking for a description. The description must be non-empty. An example follows. ``` begin setknob mintraceseverity 30 setknob tracingudplistener_addr 192.168.0.1 commit "fdbcli change" ``` Users may only combine knob configuration changes with other knob configuration changes in the same transaction. For example, the following is not permitted and will raise an error. ``` begin set foo bar setknob maxmetricsize 1000 commit "change" ``` Specifically, `set, clear, get, getrange, clearrange` cannot be combined in any transaction with a `setknob` or `getknob`. If using an individual `setknob` or `clearknob` without being inside a `begin\commit` block, then `fdbcli` will prompt for a description as well. Knobs have implicit types attached to them when defined. For example, the knob `tracingudplistener_addr` is set to `"127.0.0.1"` as so the type is string. If a user invokes `setknob` on this knob with an incorrect value that is not a string, the transaction will fail. The configuration database includes both client and server changes and is enabled by default. Thus, to disable the configuration database, changes must be made to both. The configuration database can be disabled by specifying the ``fdbserver`` command line option ``--no-config-db``. Note that this option must be specified for every ``fdbserver`` process. The only client change from the configuration database is as part of the change coordinators command. The change coordinators command is not considered successful until the configuration database is readable on the new coordinators. This will cause the change coordinators command to hang if run against a database with dynamic knobs disabled. To disable the client side configuration database liveness check, specify the ``--no-config-db`` flag when changing coordinators. For example: ``` fdbcli> coordinators auto --no-config-db ``` The current state of the configuration database is output as part of `status json`. The configuration path for each process can be determined from the ``command_line`` key associated with each

Sample from ``status json``: ``` "configuration_database" : { "commits" : [ { "description" : "set some knobs", "timestamp" : 1659570000, "version" : 1 }, { "description" : "make some other changes", "timestamp" : 1659570000, "version" : 2 } ], "lastcompactedversion" : 0, "mostrecentversion" : 2, "mutations" : [ { "config_class" : "<global>", "knobname" : "mintrace_severity", "knob_value" : "int:5", "type" : "set", "version" : 1 }, { "config_class" : "<global>", "knobname" : "compactioninterval", "knob_value" : "double:30.000000", "type" : "set", "version" : 1 }, { "config_class" : "az-1", "knobname" : "compactioninterval", "knob_value" : "double:60.000000", "type" : "set", "version" : 1 }, { "config_class" : "<global>", "knobname" : "compactioninterval", "type" : "clear", "version" : 2 }, { "config_class" : "<global>", "knobname" : "updatenode_timeout", "knob_value" : "double:4.000000", "type" : "set", "version" : 2 } ], "snapshot" : { "<global>" : { "mintraceseverity" : "int:5", "updatenodetimeout" : "double:4.000000" }, "az-1" : { "compaction_interval" : "double:60.000000" } } } ``` After compaction, ``status json`` would show: ``` "configuration_database" : { "commits" : [ ], "lastcompactedversion" : 2, "mostrecentversion" : 2, "mutations" : [ ], "snapshot" : { "<global>" : { "mintraceseverity" : "int:5", "updatenodetimeout" : "double:4.000000" }, "az-1" : { "compaction_interval" : "double:60.000000" } } } ``` The configuration database is implemented as a replicated state machine living on the coordinators. This allows configuration database transactions to continue to function in the event of a catastrophic loss of the transaction subsystem. To commit a transaction, clients run the two phase Paxos protocol. First, the client asks for a live version from a quorum of coordinators. When a coordinator receives a request for its live version, it increments its local live version by one and returns it to the client. Then, the client submits its writes at the live version it received in the previous step. A coordinator will accept the commit if it is still on the same live version. If a majority of coordinators accept the commit, it is considered committed. Each coordinator runs a ``ConfigNode`` which serves as a replica storing one full copy of the configuration database. Coordinators never communicate with other coordinators while processing configuration database transactions. Instead, the client runs the transaction and determines when it has quorum agreement. Coordinators serve the following ``ConfigTransactionInterface`` to allow clients to read from and write to the configuration database. | Request | Request fields | Reply fields | Explanation | ||-|--|| | GetGeneration | (coordinatorsHash) | (generation) or (coordinators_changed error) | Get a new read

This read version is used for all future requests in the transaction | | Get | (configuration class, knob name, coordinatorsHash, generation) | (knob value or empty) or (coordinatorschanged error) or (transactiontoo_old error) | Returns the current value stored at the specified configuration class and knob name, or empty if no value exists | | GetConfigClasses | (coordinatorsHash, generation) | (configuration classes) or (coordinatorschanged error) or (transactiontoo_old error) | Returns a list of all configuration classes stored in the configuration database | | GetKnobs | (configuration class, coordinatorsHash, generation) | (knob names) or (coordinatorschanged error) or (transactiontoo_old error) | Returns a list of all knob names stored for the provided configuration class | | Commit | (mutation list, coordinatorsHash, generation) | ack or (coordinatorschanged error) or (commitunknownresult error) or (notcommitted error) | Commit mutations set by the transaction | Coordinators also serve the following ``ConfigFollowerInterface`` to provide access to (and modification of) their current state. Most interaction through this interface is done by the cluster controller through its ``IConfigConsumer`` implementation living on the ``ConfigBroadcaster``. | Request | Request fields | Reply fields | Explanation | |--|-|--|| | GetChanges | (lastSeenVersion, mostRecentVersion) | (mutation list, version) or (versionalreadycompacted error) or (process_behind error) | Request changes since the last seen version, receive a new most recent version, as well as recent mutations | | GetSnapshotAndChanges | (mostRecentVersion) | (snapshot, snapshotVersion, changes) | Request the full configuration database, in the form of a base snapshot and changes to apply on top of the snapshot | | Compact | (version) | ack | Compact mutations up to the provided version | | Rollforward | (rollbackTo, lastKnownCommitted, target, changes, specialZeroQuorum) | ack or (versionalreadycompacted error) or (transactiontooold error) | Rollback/rollforward mutations on a node to catch it up with the majority | | GetCommittedVersion | () | (registered, lastCompacted, lastLive, lastCommitted) | Request version information from a ``ConfigNode`` | | Lock | (coordinatorsHash) | ack | Lock a ``ConfigNode`` to prevent it from serving requests during a coordinator change | The cluster controller runs a singleton ``ConfigBroadcaster`` which is responsible for periodically polling the ``ConfigNode``s for updates, then broadcasting these updates to workers through the ``ConfigBroadcastInterface``. When workers join the cluster, they register themselves and their ``ConfigBroadcastInterface`` with the broadcaster. The broadcaster then pushes new updates to registered workers. The ``ConfigBroadcastInterface`` is also used by ``ConfigNode``s to register with the ``ConfigBroadcaster``. ``ConfigNode``s need to register with the broadcaster because the broadcaster decides when the ``ConfigNode`` may begin serving requests, based on global information about status of other ``ConfigNode``s. For example, if a system with three ``ConfigNode``s suffers a fault where one ``ConfigNode`` loses data, the faulty ``ConfigNode`` should not be allowed to begin serving requests again until it has been rolled forward and is up to date with the latest state of the configuration database. | Request | Request fields | Reply fields | Explanation | |||-|| | Snapshot | (snapshot, version, restartDelay) | ack | A snapshot of the configuration database sent by the broadcaster to workers | | Changes | (changes, mostRecentVersion, restartDelay) | ack | A list of changes up to and including mostRecentVersion, sent by the broadcaster to workers | | Registered | () | (registered, lastSeenVersion) | Sent by the broadcaster to new ``ConfigNode``s to determine their registration status | | Ready | (snapshot, snapshotVersion, liveVersion, coordinatorsHash) | ack | Sent by the broadcaster to new ``ConfigNode``s to allow them to start serving requests | Each worker runs a ``LocalConfiguration`` instance which receives and applies knob updates from the

The local configuration maintains a durable ``KeyValueStoreMemory`` containing the following: The latest known configuration version The most recently used configuration path All knob overrides corresponding to the configuration path at the latest known version Once a worker starts, it will: Apply manually set knobs Read its local configuration file If the stored configuration path does not match the configuration path specified on the command line, delete the local configuration file Otherwise, apply knob updates from the local configuration file. Manually specified knobs will not be overridden Register with the broadcaster to receive new updates for its configuration classes Persist these updates when received and restart if necessary All knobs are classified as either atomic or non-atomic. Atomic knobs require a process restart when changed, while non-atomic knobs do not. ``ConfigNode``s store individual mutations in order to be able to update other, out of date ``ConfigNode``s without needing to send a full snapshot. Each configuration database commit also contains additional metadata such as a timestamp and a text description of the changes being made. To keep the size of the configuration database manageable, a compaction process runs periodically (defaulting to every five minutes) which compacts individual mutations into a simplified snapshot of key-value pairs. Compaction is controlled by the ``ConfigBroadcaster``, using information it peridiodically requests from ``ConfigNode``s. Compaction will only compact up to the minimum known version across all ``ConfigNode``s. This means that if one ``ConfigNode`` is permanently partitioned from the ``ConfigBroadcaster`` or from clients, no compaction will ever take place. It is necessary to be able to roll ``ConfigNode``s backward and forward with respect to their committed versions due to the nature of quorum logic and unreliable networks. Consider a case where a client commit gets persisted durably on one out of three ``ConfigNode``s (assume commit messages to the other two nodes are lost). Since the value is not committed on a majority of ``ConfigNode``s, it cannot be considered committed. But it is also incorrect to have the value persist on one out of three nodes as future commits are made. In this case, the most common result is that the ``ConfigNode`` will be rolled back when the next commit from a different client is made, and then rolled forward to contain the data from the commit. ``PaxosConfigConsumer`` contains logic to recognize ``ConfigNode`` minorities and update them to match the quorum. Since the configuration database lives on the coordinators and the [coordinators can be changed](https://apple.github.io/foundationdb/configuration.html#configuration-changing-coordination-servers), it is necessary to copy the configuration database from the old to the new coordinators during such an event. A coordinator change performs the following steps in regards to the configuration database: Write ``\xff/coordinatorsKey`` with the new coordinators string. The key ``\xff/previousCoordinators`` contains the current (old) set of coordinators. Lock the old ``ConfigNode``s so they can no longer serve client requests. Start a recovery, causing a new cluster controller (and therefore ``ConfigBroadcaster``) to be selected. Read ``\xff/previousCoordinators`` on the ``ConfigBroadcaster`` and, if present, read an up-to-date snapshot of the configuration database on the old coordinators. Determine if each registering ``ConfigNode`` needs an up-to-date snapshot of the configuration database sent to it, based on its reported version and the snapshot version of the database received from the old coordinators. Some new coordinators which were also coordinators in the previous configuration may not need a snapshot. Send ready requests to new ``ConfigNode``s, including an up-to-date snapshot if necessary. This allows the new coordinators to begin serving configuration database requests from clients. The ``ConfigDatabaseUnitTests`` class unit test a number of different configuration database dimensions. The ``ConfigIncrement`` workload tests contention between clients attempting to write to the configuration database,