ClickHouse Keeper (clickhouse-keeper)

Example:

```
:) get /keeper/config
server.1=zoo1:9234;participant;1
server.2=zoo2:9234;participant;1
server.3=zoo3:9234;participant;1
```

You can use `reconfig` command to add new servers, remove existing ones, and change existing servers'
priorities, here are examples (using `clickhouse-keeper-client`):

```
# Add two new servers
reconfig add "server.5=localhost:123,server.6=localhost:234;learner"
# Remove two other servers
reconfig remove "3,4"
# Change existing server priority to 8
reconfig add "server.5=localhost:5123;participant;8"
```

And here are examples for `kazoo`:

```
# Add two new servers, remove two other servers
reconfig(joining="server.5=localhost:123,server.6=localhost:234;learner", leaving="3,4")

# Change existing server priority to 8
reconfig(joining="server.5=localhost:5123;participant;8", leaving=None)
```

Servers in `joining` should be in server format described above. Server entries should be delimited by commas.
While adding new servers, you can omit `server_priority` (default value is 1) and `server_type` (default value
is `participant`).

If you want to change existing server priority, add it to `joining` with target priority.
Server host, port, and type must be equal to existing server configuration.

Servers are added and removed in order of appearance in `joining` and `leaving`.
All updates from `joining` are processed before updates from `leaving`.

There are some caveats in Keeper reconfiguration implementation:

- Only incremental reconfiguration is supported. Requests with non-empty `new_members` are declined.

  ClickHouse Keeper implementation relies on NuRaft API to change membership dynamically. NuRaft has a way to
  add a single server or remove a single server, one at a time. This means each change to configuration
  (each part of `joining`, each part of `leaving`) must be decided on separately. Thus there is no bulk
  reconfiguration available as it would be misleading for end users.

  Changing server type (participant/learner) isn't possible either as it's not supported by NuRaft, and
  the only way would be to remove and add server, which again would be misleading.
- You can't use the returned `znodestat` value.
- The `from_version` field isn't used. All requests with set `from_version` are declined.
  This is due to the fact `/keeper/config` is a virtual node, which means it isn't stored in
  persistent storage, but rather generated on-the-fly with the specified node config for every request.
  This decision was made as to not duplicate data as NuRaft already stores this config.
- Unlike ZooKeeper, there is no way to wait on cluster reconfiguration by submitting a `sync` command.
  New config will be *eventually* applied but with no time guarantees.
- `reconfig` command may fail for various reasons. You can check cluster's state and see whether the update
  was applied.

## Converting a single-node keeper into a cluster[​](#converting-a-single-node-keeper-into-a-cluster "Direct link to Converting a single-node keeper into a cluster")

Sometimes it's necessary to extend experimental keeper node into a cluster. Here's a scheme of how to do it step-by-step for 3 nodes cluster:

- **IMPORTANT**: new nodes must be added in batches less than the current quorum, otherwise they will elect a leader among them. In this example one by one.
- The existing keeper node must have `keeper_server.enable_reconfiguration` configuration parameter turned on.
- Start a second node with the full new configuration of keeper cluster.
- After it's started, add it to the node 1 using [`reconfig`](#reconfiguration).
- Now, start a third node and add it using [`reconfig`](#reconfiguration).
- Update the `clickhouse-server` configuration by adding new keeper node there and restart it to apply the changes.
- Update the raft configuration of the node 1 and, optionally, restart it.

To get confident with the process, here's a [sandbox repository](https://github.com/ClickHouse/keeper-extend-cluster).

## Unsupported features[​](#unsupported-features "Direct link to Unsupported features")

While ClickHouse Keeper aims to be fully compatible with ZooKeeper, there are some features that aren't yet implemented (although development is ongoing):

- [`create`](https://zookeeper.apache.org/doc/r3.9.1/apidocs/zookeeper-server/org/apache/zookeeper/ZooKeeper.html#create(java.lang.String,byte%5B%5D,java.util.List,org.apache.zookeeper.CreateMode,org.apache.zookeeper.data.Stat)) doesn't support returning `Stat` object
- [`create`](https://zookeeper.apache.org/doc/r3.9.1/apidocs/zookeeper-server/org/apache/zookeeper/ZooKeeper.html#create(java.lang.String,byte%5B%5D,java.util.List,org.apache.zookeeper.CreateMode,org.apache.zookeeper.data.Stat)) doesn't support [TTL](https://zookeeper.apache.org/doc/r3.9.1/apidocs/zookeeper-server/org/apache/zookeeper/CreateMode.html#PERSISTENT_WITH_TTL)
- [`addWatch`](https://zookeeper.apache.org/doc/r3.9.1/apidocs/zookeeper-server/org/apache/zookeeper/ZooKeeper.html#addWatch(java.lang.String,org.apache.zookeeper.Watcher,org.apache.zookeeper.AddWatchMode)) doesn't work with [`PERSISTENT`](https://zookeeper.apache.org/doc/r3.9.1/apidocs/zookeeper-server/org/apache/zookeeper/AddWatchMode.html#PERSISTENT) watches
- [`removeWatch`](https://zookeeper.apache.org/doc/r3.9.1/apidocs/zookeeper-server/org/apache/zookeeper/ZooKeeper.html#removeWatches(java.lang.String,org.apache.zookeeper.Watcher,org.apache.zookeeper.Watcher.WatcherType,boolean)) and [`removeAllWatches`](https://zookeeper.apache.org/doc/r3.9.1/apidocs/zookeeper-server/org/apache/zookeeper/ZooKeeper.html#removeAllWatches(java.lang.String,org.apache.zookeeper.Watcher.WatcherType,boolean)) aren't supported
- `setWatches` isn't supported
- Creating [`CONTAINER`](https://zookeeper.apache.org/doc/r3.5.1-alpha/api/org/apache/zookeeper/CreateMode.html) type znodes isn't supported
- [`SASL authentication`](https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zookeeper+and+SASL) isn't supported

- [Implementation details](#implementation-details)- [Configuration](#configuration)- [How to run](#how-to-run)- [Four letter word commands](#four-letter-word-commands)- [HTTP control](#http-control)- [Feature flags](#feature-flags)- [Migration from ZooKeeper](#migration-from-zookeeper)- [Recovering after losing quorum](#recovering-after-losing-quorum)- [Using disks with Keeper](#using-disks-with-keeper)
                  - [Changing disk setup](#changing-disk-setup)- [Configuring logs cache](#configuring-logs-cache)- [Prometheus](#prometheus)- [ClickHouse Keeper user guide](#clickhouse-keeper-user-guide)
                        - [1. Configure nodes with Keeper settings](#1-configure-nodes-with-keeper-settings)- [2. Configure a cluster in ClickHouse](#2--configure-a-cluster-in-clickhouse)- [3. Create and test distributed table](#3-create-and-test-distributed-table)- [Summary](#summary)- [Configuring ClickHouse Keeper with unique paths](#configuring-clickhouse-keeper-with-unique-paths)
                          - [Description](#description)- [Example environment](#example-environment)- [Procedures to set up tables to use `{uuid}`](#procedures-to-set-up-tables-to-use-uuid)- [Testing](#testing)- [Alternatives](#alternatives)- [Troubleshooting](#troubleshooting)- [ClickHouse Keeper dynamic reconfiguration](#reconfiguration)
                            - [Description](#description-1)- [Converting a single-node keeper into a cluster](#converting-a-single-node-keeper-into-a-cluster)- [Unsupported features](#unsupported-features)

Was this page helpful?