ClickHouse Keeper (clickhouse-keeper)

```
first_log_idx   1
first_log_term  1
last_log_idx    101
last_log_term   1
last_committed_log_idx  100
leader_committed_log_idx    101
target_committed_log_idx    101
last_snapshot_idx   50
```

- `rqld`: Request to become new leader. Return `Sent leadership request to leader.` if request sent or `Failed to send leadership request to leader.` if request not sent. If the node is already leader the outcome is the same as when the request is sent.

```
Sent leadership request to leader.
```

- `ftfl`: Lists all feature flags and whether they're enabled for the Keeper instance.

```
filtered_list   1
multi_read  1
check_not_exists    0
```

- `ydld`: Request to yield leadership and become follower. If the server receiving the request is leader, it will pause write operations first, wait until the successor (current leader can never be successor) finishes the catch-up of the latest log, and then resign. The successor will be chosen automatically. Return `Sent yield leadership request to leader.` if request sent or `Failed to send yield leadership request to leader.` if request not sent. If the node is already a follower the outcome is the same as when the request is sent.

```
Sent yield leadership request to leader.
```

- `pfev`: Returns the values for all collected events. For each event it returns event name, event value, and event's description.

```
FileOpen        62      Number of files opened.
Seek    4       Number of times the 'lseek' function was called.
ReadBufferFromFileDescriptorRead        126     Number of reads (read/pread) from a file descriptor. Does not include sockets.
ReadBufferFromFileDescriptorReadFailed  0       Number of times the read (read/pread) from a file descriptor have failed.
ReadBufferFromFileDescriptorReadBytes   178846  Number of bytes read from file descriptors. If the file is compressed, this will show the compressed data size.
WriteBufferFromFileDescriptorWrite      7       Number of writes (write/pwrite) to a file descriptor. Does not include sockets.
WriteBufferFromFileDescriptorWriteFailed        0       Number of times the write (write/pwrite) to a file descriptor have failed.
WriteBufferFromFileDescriptorWriteBytes 153     Number of bytes written to file descriptors. If the file is compressed, this will show compressed data size.
FileSync        2       Number of times the F_FULLFSYNC/fsync/fdatasync function was called for files.
DirectorySync   0       Number of times the F_FULLFSYNC/fsync/fdatasync function was called for directories.
FileSyncElapsedMicroseconds     12756   Total time spent waiting for F_FULLFSYNC/fsync/fdatasync syscall for files.
DirectorySyncElapsedMicroseconds        0       Total time spent waiting for F_FULLFSYNC/fsync/fdatasync syscall for directories.
ReadCompressedBytes     0       Number of bytes (the number of bytes before decompression) read from compressed sources (files, network).
CompressedReadBufferBlocks      0       Number of compressed blocks (the blocks of data that are compressed independent of each other) read from compressed sources (files, network).
CompressedReadBufferBytes       0       Number of uncompressed bytes (the number of bytes after decompression) read from compressed sources (files, network).
AIOWrite        0       Number of writes with Linux or FreeBSD AIO interface
AIOWriteBytes   0       Number of bytes written with Linux or FreeBSD AIO interface
...
```

### HTTP control[​](#http-control "Direct link to HTTP control")

ClickHouse Keeper provides an HTTP interface to check if a replica is ready to receive traffic. It may be used in cloud environments, such as [Kubernetes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes).

Example of configuration that enables `/ready` endpoint:

```
<clickhouse>
    <keeper_server>
        <http_control>
            <port>9182</port>
            <readiness>
                <endpoint>/ready</endpoint>
            </readiness>
        </http_control>
    </keeper_server>
</clickhouse>
```

### Feature flags[​](#feature-flags "Direct link to Feature flags")

Keeper is fully compatible with ZooKeeper and its clients, but it also introduces some unique features and request types that can be used by ClickHouse client.
Because those features can introduce backward incompatible change, most of them are disabled by default and can be enabled using `keeper_server.feature_flags` config.
All features can be disabled explicitly.
If you want to enable a new feature for your Keeper cluster, we recommend you to first update all the Keeper instances in the cluster to a version that supports the feature and then enable the feature itself.

Example of feature flag config that disables `multi_read` and enables `check_not_exists`:

```
<clickhouse>
    <keeper_server>
        <feature_flags>
            <multi_read>0</multi_read>
            <check_not_exists>1</check_not_exists>
        </feature_flags>
    </keeper_server>
</clickhouse>
```

The following features are available:

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Feature Description Default|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `multi_read` Support for read multi request `1`| `filtered_list` Support for list request which filters results by the type of node (ephemeral or persistent) `1`| `check_not_exists` Support for `CheckNotExists` request, which asserts that node doesn't exist `1`| `create_if_not_exists` Support for `CreateIfNotExists` request, which will try to create a node if it doesn't exist. If it exists, no changes are applied and `ZOK` is returned `1`| `remove_recursive` Support for `RemoveRecursive` request, which removes the node along with its subtree `1` | | | | | | | | | | | | | | | | | |

Note

Some of the feature flags are enabled by default from version 25.7.
The recommended way of upgrading Keeper to 25.7+ is to first upgrade to version 24.9+.

### Migration from ZooKeeper[​](#migration-from-zookeeper "Direct link to Migration from ZooKeeper")

Seamless migration from ZooKeeper to ClickHouse Keeper isn't possible. You have to stop your ZooKeeper cluster, convert data, and start ClickHouse Keeper. The `clickhouse-keeper-converter` tool converts ZooKeeper logs and snapshots to a ClickHouse Keeper snapshot. It requires ZooKeeper 3.4 or later.

#### Pre-migration preparation[​](#pre-migration-preparation "Direct link to Pre-migration preparation")

Migration requires stopping data ingestion. Plan a maintenance window before you begin.

Before stopping ZooKeeper, halt ClickHouse background tasks that modify coordination metadata. For example:

```
SYSTEM STOP MERGES;
```

Record comparison metrics before migration so you can verify consistency afterward.

#### Migration steps[​](#migration-steps "Direct link to Migration steps")

1. Stop data ingestion into all ClickHouse nodes.
2. Stop all background tasks on all ClickHouse nodes (see [above](#pre-migration-preparation)).
3. Stop all ZooKeeper nodes.
4. Optional, but recommended: find the ZooKeeper leader node, start and stop it again. This forces ZooKeeper to write a consistent snapshot to disk before conversion.
5. Run `clickhouse-keeper-converter` on the leader node. If you have the full ClickHouse binary installed, use the `keeper-converter` subcommand instead (`clickhouse keeper-converter`). If neither is available, [download the binary](/docs/getting-started/quick-start/oss#install-clickhouse).

```
clickhouse-keeper-converter \
  --zookeeper-logs-dir /var/lib/zookeeper/version-2 \
  --zookeeper-snapshots-dir /var/lib/zookeeper/version-2 \
  --output-dir /path/to/clickhouse/keeper/snapshots
```