Replication + Scaling

Set up of ClickHouse Keeper is explained in the next step of the tutorial.

```
<zookeeper>
    <node>
        <host>clickhouse-keeper-01</host>
        <port>9181</port>
    </node>
    <node>
        <host>clickhouse-keeper-02</host>
        <port>9181</port>
    </node>
    <node>
        <host>clickhouse-keeper-03</host>
        <port>9181</port>
    </node>
</zookeeper>
```

Note

Although it is possible to run ClickHouse Keeper on the same server as ClickHouse Server,
in production environments we strongly recommend that ClickHouse Keeper runs on dedicated hosts.

#### Macros configuration[​](#macros-config-explanation "Direct link to Macros configuration")

Additionally, the `<macros>` section is used to define parameter substitutions for
replicated tables. These are listed in `system.macros` and allow using substitutions
like `{shard}` and `{replica}` in queries.

```
<macros>
   <shard>01</shard>
   <replica>01</replica>
</macros>
```

### User configuration[​](#cluster-configuration "Direct link to User configuration")

Now modify each empty configuration file `users.xml` located at
`fs/volumes/clickhouse-{}/etc/clickhouse-server/users.d` with the following:

```
<?xml version="1.0"?>
<clickhouse replace="true">
    <profiles>
        <default>
            <max_memory_usage>10000000000</max_memory_usage>
            <use_uncompressed_cache>0</use_uncompressed_cache>
            <load_balancing>in_order</load_balancing>
            <log_queries>1</log_queries>
        </default>
    </profiles>
    <users>
        <default>
            <access_management>1</access_management>
            <profile>default</profile>
            <networks>
                <ip>::/0</ip>
            </networks>
            <quota>default</quota>
            <access_management>1</access_management>
            <named_collection_control>1</named_collection_control>
            <show_named_collections>1</show_named_collections>
            <show_named_collections_secrets>1</show_named_collections_secrets>
        </default>
    </users>
    <quotas>
        <default>
            <interval>
                <duration>3600</duration>
                <queries>0</queries>
                <errors>0</errors>
                <result_rows>0</result_rows>
                <read_rows>0</read_rows>
                <execution_time>0</execution_time>
            </interval>
        </default>
    </quotas>
</clickhouse>
```

In this example, the default user is configured without a password for simplicity.
In practice, this is discouraged.

Note

In this example, each `users.xml` file is identical for all nodes in the cluster.

Configure ClickHouse Keeper

## Configure ClickHouse Keeper[​](#configure-clickhouse-keeper-nodes "Direct link to Configure ClickHouse Keeper")

Next you will configure ClickHouse Keeper, which is used for coordination.

### Keeper setup[​](#configuration-explanation "Direct link to Keeper setup")

In order for replication to work, a ClickHouse keeper cluster needs to be set up and
configured. ClickHouse Keeper provides the coordination system for data replication,
acting as a stand in replacement for Zookeeper, which could also be used.
ClickHouse Keeper is, however, recommended, as it provides better guarantees and
reliability and uses fewer resources than ZooKeeper. For high availability and to
keep quorum, it is recommended to run at least three ClickHouse Keeper nodes.

Note

ClickHouse Keeper can run on any node of the cluster alongside ClickHouse, although
it is recommended to have it run on a dedicated node which allows scaling and
managing the ClickHouse Keeper cluster independently of the database cluster.

Create the `keeper_config.xml` files for each ClickHouse Keeper node
using the following command from the root of the example folder:

```
for i in {01..03}; do
  touch fs/volumes/clickhouse-keeper-${i}/etc/clickhouse-keeper/keeper_config.xml
done
```

Modify the empty configuration files which were created in each
node directory `fs/volumes/clickhouse-keeper-{}/etc/clickhouse-keeper`. The
highlighted lines below need to be changed to be specific to each node:

```
<clickhouse replace="true">
    <logger>
        <level>information</level>
        <log>/var/log/clickhouse-keeper/clickhouse-keeper.log</log>
        <errorlog>/var/log/clickhouse-keeper/clickhouse-keeper.err.log</errorlog>
        <size>1000M</size>
        <count>3</count>
    </logger>
    <listen_host>0.0.0.0</listen_host>
    <keeper_server>
        <tcp_port>9181</tcp_port>
        <!--highlight-next-line-->
        <server_id>1</server_id>
        <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
        <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>
        <coordination_settings>
            <operation_timeout_ms>10000</operation_timeout_ms>
            <session_timeout_ms>30000</session_timeout_ms>
            <raft_logs_level>information</raft_logs_level>
        </coordination_settings>
        <raft_configuration>
            <server>
                <id>1</id>
                <hostname>clickhouse-keeper-01</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>2</id>
                <hostname>clickhouse-keeper-02</hostname>
                <port>9234</port>
            </server>
            <server>
                <id>3</id>
                <hostname>clickhouse-keeper-03</hostname>
                <port>9234</port>
            </server>
        </raft_configuration>
    </keeper_server>
</clickhouse>
```

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Directory File|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | `fs/volumes/clickhouse-keeper-01/etc/clickhouse-keeper` [`keeper_config.xml`](https://github.com/ClickHouse/examples/blob/main/docker-compose-recipes/recipes/cluster_2S_2R/fs/volumes/clickhouse-keeper-01/etc/clickhouse-keeper/keeper_config.xml)| `fs/volumes/clickhouse-keeper-02/etc/clickhouse-keeper` [`keeper_config.xml`](https://github.com/ClickHouse/examples/blob/main/docker-compose-recipes/recipes/cluster_2S_2R/fs/volumes/clickhouse-keeper-02/etc/clickhouse-keeper/keeper_config.xml)| `fs/volumes/clickhouse-keeper-03/etc/clickhouse-keeper` [`keeper_config.xml`](https://github.com/ClickHouse/examples/blob/main/docker-compose-recipes/recipes/cluster_2S_2R/fs/volumes/clickhouse-keeper-03/etc/clickhouse-keeper/keeper_config.xml) | | | | | | | |

Each configuration file will contain the following unique configuration (shown below).
The `server_id` used should be unique for that particular ClickHouse Keeper node
in the cluster and match the server `<id>` defined in the `<raft_configuration>` section.
`tcp_port` is the port used by *clients* of ClickHouse Keeper.

```
<tcp_port>9181</tcp_port>
<server_id>{id}</server_id>
```

The following section is used to configure the servers that participate in the
quorum for the [raft consensus algorithm](https://en.wikipedia.org/wiki/Raft_(algorithm)):

```
<raft_configuration>
    <server>
        <id>1</id>
        <hostname>clickhouse-keeper-01</hostname>
        <!-- TCP port used for communication between ClickHouse Keeper nodes -->
        <!--highlight-next-line-->
        <port>9234</port>
    </server>
    <server>
        <id>2</id>
        <hostname>clickhouse-keeper-02</hostname>
        <port>9234</port>
    </server>
    <server>
        <id>3</id>
        <hostname>clickhouse-keeper-03</hostname>
        <port>9234</port>
    </server>
</raft_configuration>
```

ClickHouse Cloud simplifies management

[ClickHouse Cloud](/docs/cloud/overview)
removes the operational burden associated with managing shards and replicas. The
platform automatically handles high availability, replication, and scaling decisions.
Compute and storage are separate and scale based on demand without requiring manual
configuration or ongoing maintenance.

[Read more](/docs/manage/scaling)

Test the setup

## Test the setup[​](#test-the-setup "Direct link to Test the setup")