ClickHouse Keeper (clickhouse-keeper)

```
clickhouse-keeper-converter \
  --zookeeper-logs-dir /var/lib/zookeeper/version-2 \
  --zookeeper-snapshots-dir /var/lib/zookeeper/version-2 \
  --output-dir /path/to/clickhouse/keeper/snapshots
```

6. Copy the snapshot to all ClickHouse Keeper nodes. The snapshot must be present on every node before any node starts — if a node starts without a snapshot, it may elect itself leader with empty state.
7. Update your ClickHouse configuration to point at the new Keeper cluster.
8. Start ClickHouse Keeper on all nodes, then restart ClickHouse.
9. Compare metrics against your pre-migration baseline to verify consistency.
10. Resume background tasks and restart data ingestion.

#### Consolidating multiple ZooKeeper clusters[​](#consolidating-multiple-zookeeper-clusters "Direct link to Consolidating multiple ZooKeeper clusters")

If you run multiple ZooKeeper clusters — for example, one per shard group — you can consolidate them into a single ClickHouse Keeper cluster. The official `clickhouse-keeper-converter` tool only supports one-to-one conversions (one ZooKeeper cluster to one Keeper snapshot), so consolidation requires modifying the converter source code to merge multiple snapshots:

1. Run `clickhouse-keeper-converter` separately on each ZooKeeper cluster, writing each output to a distinct directory.
2. Deserialize the snapshot files sequentially. When merging, recalculate `numChildren` values to avoid node ID conflicts between namespaces from different source clusters.
3. Write the merged output to the target ClickHouse Keeper snapshot directory.

#### Handling encryption and ACLs[​](#handling-encryption-and-acls "Direct link to Handling encryption and ACLs")

ClickHouse Keeper supports the same ACL schemes as ZooKeeper (`world`, `auth`, `digest`). How you handle ACLs during conversion depends on your ZooKeeper setup:

- **Fully encrypted or fully unencrypted**: Convert directly. The converter preserves existing ACL information.
- **Partially encrypted**: Before converting, grant a super administrator account and clear ACLs with `setAcl -R` on the affected paths. Convert, then re-enable encryption in ClickHouse Keeper if needed.

#### Verifying the migration[​](#verifying-the-migration "Direct link to Verifying the migration")

After starting ClickHouse Keeper and restarting ClickHouse, compare your key metrics against the pre-migration baseline to confirm the migration succeeded.

When consolidating multiple ZooKeeper clusters, distinguish between:

- **Common paths**: Paths present across multiple source clusters with identical data — these should be deduplicated in the merged output.
- **Differentiated paths**: Paths that exist only under specific clusters (e.g., under `/clickhouse/tables` for each shard group) — these must be preserved from the correct source.

Avoid traversing large ZooKeeper trees directly for comparison. Print all converted paths to a file during conversion instead.

#### Post-migration tuning[​](#post-migration-tuning "Direct link to Post-migration tuning")

After migration, consider tuning these settings for larger clusters or higher throughput:

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Setting Default Recommended Notes|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `max_requests_batch_size` `100` `10000` Increase for clusters with high part counts or many shards|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `force_sync` `true` `false` Asynchronous log writes improve throughput|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | `compress_logs` `false` `true` Compresses Raft log files to reduce disk I/O|  |  |  |  | | --- | --- | --- | --- | | `compress_snapshots_with_zstd_format` `true` — Already enabled by default; compresses snapshots with zstd | | | | | | | | | | | | | | | | | | | |

These settings are configured under `coordination_settings` in your [Keeper configuration](#internal-coordination-settings).

### Recovering after losing quorum[​](#recovering-after-losing-quorum "Direct link to Recovering after losing quorum")

Because ClickHouse Keeper uses Raft it can tolerate a certain amount of node crashes depending on the cluster size.   
E.g. for a 3-node cluster, it will continue working correctly if only 1 node crashes.

Cluster configuration can be dynamically configured, but there are some limitations. Reconfiguration relies on Raft also
so to add/remove a node from the cluster you need to have a quorum. If you lose too many nodes in your cluster at the same time without any chance
of starting them again, Raft will stop working and not allow you to reconfigure your cluster using the conventional way.

Nevertheless, ClickHouse Keeper has a recovery mode which allows you to forcefully reconfigure your cluster with only 1 node.
This should be done only as your last resort if you can't start your nodes again, or start a new instance on the same endpoint.

Important things to note before continuing:

- Make sure that the failed nodes can't connect to the cluster again.
- Don't start any of the new nodes until it's specified in the steps.

After making sure that the above things are true, you need to do the following:

1. Pick a single Keeper node to be your new leader. Be aware that the data of that node will be used for the entire cluster, so we recommend using a node with the most up-to-date state.
2. Before doing anything else, make a backup of the `log_storage_path` and `snapshot_storage_path` folders of the picked node.
3. Reconfigure the cluster on all of the nodes you want to use.
4. Send the four letter command `rcvr` to the node you picked which will move the node to the recovery mode OR stop Keeper instance on the picked node and start it again with the `--force-recovery` argument.
5. One by one, start Keeper instances on the new nodes making sure that `mntr` returns `follower` for the `zk_server_state` before starting the next one.
6. While in the recovery mode, the leader node will return error message for `mntr` command until it achieves quorum with the new nodes and refuse any requests from the client and the followers.
7. After quorum is achieved, the leader node will return to the normal mode of operation, accepting all the requests using Raft-verify with `mntr` which should return `leader` for the `zk_server_state`.

## Using disks with Keeper[​](#using-disks-with-keeper "Direct link to Using disks with Keeper")

Keeper supports a subset of [external disks](/docs/operations/storing-data) for storing snapshots, log files, and the state file.

Supported types of disks are:

- s3\_plain
- s3
- local

Following is an example of disk definitions contained inside a config.