ClickHouse Keeper (clickhouse-keeper)

## Using disks with Keeper[​](#using-disks-with-keeper "Direct link to Using disks with Keeper")

Keeper supports a subset of [external disks](/docs/operations/storing-data) for storing snapshots, log files, and the state file.

Supported types of disks are:

- s3\_plain
- s3
- local

Following is an example of disk definitions contained inside a config.

```
<clickhouse>
    <storage_configuration>
        <disks>
            <log_local>
                <type>local</type>
                <path>/var/lib/clickhouse/coordination/logs/</path>
            </log_local>
            <log_s3_plain>
                <type>s3_plain</type>
                <endpoint>https://some_s3_endpoint/logs/</endpoint>
                <access_key_id>ACCESS_KEY</access_key_id>
                <secret_access_key>SECRET_KEY</secret_access_key>
            </log_s3_plain>
            <snapshot_local>
                <type>local</type>
                <path>/var/lib/clickhouse/coordination/snapshots/</path>
            </snapshot_local>
            <snapshot_s3_plain>
                <type>s3_plain</type>
                <endpoint>https://some_s3_endpoint/snapshots/</endpoint>
                <access_key_id>ACCESS_KEY</access_key_id>
                <secret_access_key>SECRET_KEY</secret_access_key>
            </snapshot_s3_plain>
            <state_s3_plain>
                <type>s3_plain</type>
                <endpoint>https://some_s3_endpoint/state/</endpoint>
                <access_key_id>ACCESS_KEY</access_key_id>
                <secret_access_key>SECRET_KEY</secret_access_key>
            </state_s3_plain>
        </disks>
    </storage_configuration>
</clickhouse>
```

To use a disk for logs `keeper_server.log_storage_disk` config should be set to the name of disk.
To use a disk for snapshots `keeper_server.snapshot_storage_disk` config should be set to the name of disk.
Additionally, `keeper_server.latest_log_storage_disk` can be used for the latest logs and `keeper_server.latest_snapshot_storage_disk` for the latest snapshots.
In that case, Keeper will automatically move files to correct disks when new logs or snapshots are created.
To use a disk for state file, `keeper_server.state_storage_disk` config should be set to the name of disk.

Moving files between disks is safe and there is no risk of losing data if Keeper stops in the middle of transfer.
Until the file is completely moved to the new disk, it's not deleted from the old one.

Keeper with `keeper_server.coordination_settings.force_sync` set to `true` (`true` by default) can't satisfy some guarantees for all types of disks.
Right now, only disks of type `local` support persistent sync.
If `force_sync` is used, `log_storage_disk` should be a `local` disk if `latest_log_storage_disk` isn't used.
If `latest_log_storage_disk` is used, it should always be a `local` disk.
If `force_sync` is disabled, disks of all types can be used in any setup.

A possible storage setup for a Keeper instance could look like following:

```
<clickhouse>
    <keeper_server>
        <log_storage_disk>log_s3_plain</log_storage_disk>
        <latest_log_storage_disk>log_local</latest_log_storage_disk>

        <snapshot_storage_disk>snapshot_s3_plain</snapshot_storage_disk>
        <latest_snapshot_storage_disk>snapshot_local</latest_snapshot_storage_disk>
    </keeper_server>
</clickhouse>
```

This instance will store all but the latest logs on disk `log_s3_plain`, while the latest log will be on the `log_local` disk.
Same logic applies for snapshots, all but the latest snapshots will be stored on `snapshot_s3_plain`, while the latest snapshot will be on the `snapshot_local` disk.

### Changing disk setup[​](#changing-disk-setup "Direct link to Changing disk setup")

References

Before applying a new disk setup, manually back up all Keeper logs and snapshots.

If a tiered disk setup is defined (using separate disks for the latest files), Keeper will try to automatically move files to the correct disks on startup.
The same guarantee is applied as before; until the file is completely moved to the new disk, it's not deleted from the old one, so multiple restarts
can be safely done.

If it's necessary to move files to a completely new disk (or move from a 2-disk setup to a single disk setup), it's possible to use multiple definitions of `keeper_server.old_snapshot_storage_disk` and `keeper_server.old_log_storage_disk`.

The following config shows how we can move from the previous 2-disk setup to a completely new single-disk setup:

```
<clickhouse>
    <keeper_server>
        <old_log_storage_disk>log_local</old_log_storage_disk>
        <old_log_storage_disk>log_s3_plain</old_log_storage_disk>
        <log_storage_disk>log_local2</log_storage_disk>

        <old_snapshot_storage_disk>snapshot_s3_plain</old_snapshot_storage_disk>
        <old_snapshot_storage_disk>snapshot_local</old_snapshot_storage_disk>
        <snapshot_storage_disk>snapshot_local2</snapshot_storage_disk>
    </keeper_server>
</clickhouse>
```

On startup, all the log files will be moved from `log_local` and `log_s3_plain` to the `log_local2` disk.
Also, all the snapshot files will be moved from `snapshot_local` and `snapshot_s3_plain` to the `snapshot_local2` disk.

## Configuring logs cache[​](#configuring-logs-cache "Direct link to Configuring logs cache")

To minimize the amount of data read from disk, Keeper caches log entries in memory.
If requests are large, log entries will take too much memory so the amount of cached logs is capped.
The limit is controlled with these two configs:

- `latest_logs_cache_size_threshold` - total size of latest logs stored in cache
- `commit_logs_cache_size_threshold` - total size of subsequent logs that need to be committed next

If the default values are too big, you can reduce the memory usage by reducing these two configs.

Note

You can use `pfev` command to check amount of logs read from each cache and from a file.
You can also use metrics from Prometheus endpoint to track the current size of both caches.

## Prometheus[​](#prometheus "Direct link to Prometheus")

Keeper can expose metrics data for scraping from [Prometheus](https://prometheus.io).

Settings:

- `endpoint` – HTTP endpoint for scraping metrics by the Prometheus server. Start from '/'.
- `port` – Port for `endpoint`.
- `metrics` – Flag that sets to expose metrics from the [system.metrics](/docs/operations/system-tables/metrics) table.
- `events` – Flag that sets to expose metrics from the [system.events](/docs/operations/system-tables/events) table.
- `asynchronous_metrics` – Flag that sets to expose current metrics values from the [system.asynchronous\_metrics](/docs/operations/system-tables/asynchronous_metrics) table.

**Example**

```
<clickhouse>
    <listen_host>0.0.0.0</listen_host>
    <http_port>8123</http_port>
    <tcp_port>9000</tcp_port>
    <!-- highlight-start -->
    <prometheus>
        <endpoint>/metrics</endpoint>
        <port>9363</port>
        <metrics>true</metrics>
        <events>true</events>
        <asynchronous_metrics>true</asynchronous_metrics>
    </prometheus>
    <!-- highlight-end -->
</clickhouse>
```

Check (replace `127.0.0.1` with the IP addr or hostname of your ClickHouse server):

```
curl 127.0.0.1:9363/metrics
```

See also the ClickHouse Cloud [Prometheus integration](/docs/integrations/prometheus).

## ClickHouse Keeper user guide[​](#clickhouse-keeper-user-guide "Direct link to ClickHouse Keeper user guide")

This guide provides simple and minimal settings to configure ClickHouse Keeper with an example on how to test distributed operations. This example is performed using 3 nodes on Linux.

### 1. Configure nodes with Keeper settings[​](#1-configure-nodes-with-keeper-settings "Direct link to 1. Configure nodes with Keeper settings")

1. Install 3 ClickHouse instances on 3 hosts (`chnode1`, `chnode2`, `chnode3`). (View the [Quick Start](/docs/install) for details on installing ClickHouse.)
2. On each node, add the following entry to allow external communication through the network interface.