ClickHouse Keeper (clickhouse-keeper)

Other common parameters are inherited from the ClickHouse server config (`listen_host`, `logger`, and so on).

#### Internal coordination settings[​](#internal-coordination-settings "Direct link to Internal coordination settings")

Internal coordination settings are located in the `<keeper_server>.<coordination_settings>` section and have the following parameters:

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Parameter Description Default|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `operation_timeout_ms` Timeout for a single client operation (ms) `10000`| `min_session_timeout_ms` Min timeout for client session (ms) `10000`| `session_timeout_ms` Max timeout for client session (ms) `100000`| `dead_session_check_period_ms` How often ClickHouse Keeper checks for dead sessions and removes them (ms) `500`| `heart_beat_interval_ms` How often a ClickHouse Keeper leader will send heartbeats to followers (ms) `500`| `election_timeout_lower_bound_ms` If the follower doesn't receive a heartbeat from the leader in this interval, then it can initiate leader election. Must be less than or equal to `election_timeout_upper_bound_ms`. Ideally they shouldn't be equal. `1000`| `election_timeout_upper_bound_ms` If the follower doesn't receive a heartbeat from the leader in this interval, then it must initiate leader election. `2000`| `rotate_log_storage_interval` How many log records to store in a single file. `100000`| `reserved_log_items` How many coordination log records to store before compaction. `100000`| `snapshot_distance` How often ClickHouse Keeper will create new snapshots (in the number of records in logs). `100000`| `snapshots_to_keep` How many snapshots to keep. `3`| `stale_log_gap` Threshold when leader considers follower as stale and sends the snapshot to it instead of logs. `10000`| `fresh_log_gap` When node became fresh. `200`| `max_requests_batch_size` Max size of batch in requests count before it will be sent to RAFT. `100`| `force_sync` Call `fsync` on each write to coordination log. `true`| `quorum_reads` Execute read requests as writes through whole RAFT consensus with similar speed. `false`| `raft_logs_level` Text logging level about coordination (trace, debug, and so on). `system default`| `auto_forwarding` Allow to forward write requests from followers to the leader. `true`| `shutdown_timeout` Wait to finish internal connections and shutdown (ms). `5000`| `startup_timeout` If the server doesn't connect to other quorum participants in the specified timeout it will terminate (ms). `30000`| `async_replication` Enable async replication. All write and read guarantees are preserved while better performance is achieved. Settings is disabled by default to not break backwards compatibility `false`| `latest_logs_cache_size_threshold` Maximum total size of in-memory cache of latest log entries `1GiB`| `commit_logs_cache_size_threshold` Maximum total size of in-memory cache of log entries needed next for commit `500MiB`| `disk_move_retries_wait_ms` How long to wait between retries after a failure which happened while a file was being moved between disks `1000`| `disk_move_retries_during_init` The amount of retries after a failure which happened while a file was being moved between disks during initialization `100`| `experimental_use_rocksdb` Use rocksdb as backend storage `0` | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

Quorum configuration is located in the `<keeper_server>.<raft_configuration>` section and contain servers description.

The only parameter for the whole quorum is `secure`, which enables encrypted connection for communication between quorum participants. The parameter can be set `true` if SSL connection is required for internal communication between nodes, or left unspecified otherwise.

The main parameters for each `<server>` are:

- `id` — Server identifier in a quorum.
- `hostname` — Hostname where this server is placed.
- `port` — Port where this server listens for connections.
- `can_become_leader` — Set to `false` to set up the server as a `learner`. If omitted, the value is `true`.

Note

In the case of a change in the topology of your ClickHouse Keeper cluster (e.g., replacing a server), make sure to keep the mapping of `server_id` to `hostname` consistent and avoid shuffling or reusing an existing `server_id` for different servers (e.g., it can happen if your rely on automation scripts to deploy ClickHouse Keeper)

If the host of a Keeper instance can change, we recommend defining and using a hostname instead of raw IP addresses. Changing hostname is equal to removing and adding the server back which in some cases can be impossible to do (e.g. not enough Keeper instances for quorum).

Note

`async_replication` is disabled by default to avoid breaking backwards compatibility. If you have all your Keeper instances in a cluster running a version supporting `async_replication` (v23.9+), we recommend enabling it because it can improve performance without any downsides.

Examples of configuration for quorum with three nodes can be found in [integration tests](https://github.com/ClickHouse/ClickHouse/tree/master/tests/integration) with `test_keeper_` prefix. Example configuration for server #1:

```
<keeper_server>
    <tcp_port>2181</tcp_port>
    <server_id>1</server_id>
    <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
    <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>

    <coordination_settings>
        <operation_timeout_ms>10000</operation_timeout_ms>
        <session_timeout_ms>30000</session_timeout_ms>
        <raft_logs_level>trace</raft_logs_level>
    </coordination_settings>

    <raft_configuration>
        <server>
            <id>1</id>
            <hostname>zoo1</hostname>
            <port>9234</port>
        </server>
        <server>
            <id>2</id>
            <hostname>zoo2</hostname>
            <port>9234</port>
        </server>
        <server>
            <id>3</id>
            <hostname>zoo3</hostname>
            <port>9234</port>
        </server>
    </raft_configuration>
</keeper_server>
```

### How to run[​](#how-to-run "Direct link to How to run")

ClickHouse Keeper is bundled into the ClickHouse server package, just add configuration of `<keeper_server>` to your `/etc/your_path_to_config/clickhouse-server/config.xml` and start ClickHouse server as always. If you want to run standalone ClickHouse Keeper you can start it in a similar way with:

```
clickhouse-keeper --config /etc/your_path_to_config/config.xml
```

If you don't have the symlink (`clickhouse-keeper`) you can create it or specify `keeper` as an argument to `clickhouse`: