Replication + Scaling

The line

```
<clickhouse replace="true">
```

Ensures that the configuration sections defined in the `config.d` and `users.d`
directories override the default configuration sections defined in the default
`config.xml` and `users.xml` files.

Configure ClickHouse nodes

## Configure ClickHouse nodes[​](#configure-clickhouse-servers "Direct link to Configure ClickHouse nodes")

### Server setup[​](#server-setup "Direct link to Server setup")

Now modify each empty configuration file `config.xml` located at
`fs/volumes/clickhouse-{}/etc/clickhouse-server/config.d`. The lines which are
highlighted below need to be changed to be specific to each node:

```
<clickhouse replace="true">
    <logger>
        <level>debug</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>3</count>
    </logger>
    <!--highlight-next-line-->
    <display_name>cluster_2S_2R node 1</display_name>
    <listen_host>0.0.0.0</listen_host>
    <http_port>8123</http_port>
    <tcp_port>9000</tcp_port>
    <user_directories>
        <users_xml>
            <path>users.xml</path>
        </users_xml>
        <local_directory>
            <path>/var/lib/clickhouse/access/</path>
        </local_directory>
    </user_directories>
    <distributed_ddl>
        <path>/clickhouse/task_queue/ddl</path>
    </distributed_ddl>
    <remote_servers>
        <cluster_2S_2R>
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>clickhouse-01</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>clickhouse-03</host>
                    <port>9000</port>
                </replica>
            </shard>
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>clickhouse-02</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>clickhouse-04</host>
                    <port>9000</port>
                </replica>
            </shard>
        </cluster_2S_2R>
    </remote_servers>
    <zookeeper>
        <node>
            <host>clickhouse-keeper-01</host>
            <port>9181</port>
        </node>
        <node>
            <host>clickhouse-keeper-02</host>
            <port>9181</port>
        </node>
        <node>
            <host>clickhouse-keeper-03</host>
            <port>9181</port>
        </node>
    </zookeeper>
    <!--highlight-start-->
    <macros>
        <shard>01</shard>
        <replica>01</replica>
    </macros>
    <!--highlight-end-->
</clickhouse>
```

|  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Directory File|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | `fs/volumes/clickhouse-01/etc/clickhouse-server/config.d` [`config.xml`](https://github.com/ClickHouse/examples/blob/main/docker-compose-recipes/recipes/cluster_2S_2R/fs/volumes/clickhouse-01/etc/clickhouse-server/config.d/config.xml)| `fs/volumes/clickhouse-02/etc/clickhouse-server/config.d` [`config.xml`](https://github.com/ClickHouse/examples/blob/main/docker-compose-recipes/recipes/cluster_2S_2R/fs/volumes/clickhouse-02/etc/clickhouse-server/config.d/config.xml)| `fs/volumes/clickhouse-03/etc/clickhouse-server/config.d` [`config.xml`](https://github.com/ClickHouse/examples/blob/main/docker-compose-recipes/recipes/cluster_2S_2R/fs/volumes/clickhouse-03/etc/clickhouse-server/config.d/config.xml)| `fs/volumes/clickhouse-04/etc/clickhouse-server/config.d` [`config.xml`](https://github.com/ClickHouse/examples/blob/main/docker-compose-recipes/recipes/cluster_2S_2R/fs/volumes/clickhouse-04/etc/clickhouse-server/config.d/config.xml) | | | | | | | | | |

Each section of the above configuration file is explained in more detail below.

#### Networking and logging[​](#networking "Direct link to Networking and logging")

External communication to the network interface is enabled by activating the listen
host setting. This ensures that the ClickHouse server host is reachable by other
hosts:

```
<listen_host>0.0.0.0</listen_host>
```

The port for the HTTP API is set to `8123`:

```
<http_port>8123</http_port>
```

The TCP port for interaction by ClickHouse's native protocol between clickhouse-client
and other native ClickHouse tools, and clickhouse-server and other clickhouse-servers
is set to `9000`:

```
<tcp_port>9000</tcp_port>
```

Logging configuration is defined in the `<logger>` block. This example configuration gives
you a debug log that will roll over at 1000M three times:

```
<logger>
   <level>debug</level>
   <log>/var/log/clickhouse-server/clickhouse-server.log</log>
   <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
   <size>1000M</size>
   <count>3</count>
</logger>
```

For more information on logging configuration, see the comments included in the
default ClickHouse [configuration file](https://github.com/ClickHouse/ClickHouse/blob/master/programs/server/config.xml).

#### Cluster configuration[​](#cluster-config "Direct link to Cluster configuration")

Configuration for the cluster is set up in the `<remote_servers>` block.
Here the cluster name `cluster_2S_2R` is defined.

The `<cluster_2S_2R></cluster_2S_2R>` block defines the layout of the cluster,
using the `<shard></shard>` and `<replica></replica>` settings, and acts as a
template for distributed DDL queries, which are queries that execute across the
cluster using the `ON CLUSTER` clause. By default, distributed DDL queries
are allowed, but can also be turned off with setting `allow_distributed_ddl_queries`.

`internal_replication` is set to true so that data is written to just one of the replicas.

```
<remote_servers>
   <!-- cluster name (should not contain dots) -->
  <cluster_2S_2R>
      <!-- <allow_distributed_ddl_queries>false</allow_distributed_ddl_queries> -->
      <shard>
          <!-- Optional. Whether to write data to just one of the replicas. Default: false (write data to all replicas). -->
          <internal_replication>true</internal_replication>
          <replica>
              <host>clickhouse-01</host>
              <port>9000</port>
          </replica>
          <replica>
              <host>clickhouse-03</host>
              <port>9000</port>
          </replica>
      </shard>
      <shard>
          <internal_replication>true</internal_replication>
          <replica>
              <host>clickhouse-02</host>
              <port>9000</port>
          </replica>
          <replica>
              <host>clickhouse-04</host>
              <port>9000</port>
          </replica>
      </shard>
  </cluster_2S_2R>
</remote_servers>
```

The `<cluster_2S_2R></cluster_2S_2R>` section defines the layout of the cluster,
and acts as a template for distributed DDL queries, which are queries that execute
across the cluster using the `ON CLUSTER` clause.

#### Keeper configuration[​](#keeper-config-explanation "Direct link to Keeper configuration")

The `<ZooKeeper>` section tells ClickHouse where ClickHouse Keeper (or ZooKeeper) is running.
As we're using a ClickHouse Keeper cluster, each `<node>` of the cluster needs to be specified,
along with its hostname and port number using the `<host>` and `<port>` tags respectively.

Set up of ClickHouse Keeper is explained in the next step of the tutorial.

```
<zookeeper>
    <node>
        <host>clickhouse-keeper-01</host>
        <port>9181</port>
    </node>
    <node>
        <host>clickhouse-keeper-02</host>
        <port>9181</port>
    </node>
    <node>
        <host>clickhouse-keeper-03</host>
        <port>9181</port>
    </node>
</zookeeper>
```

Note

Although it is possible to run ClickHouse Keeper on the same server as ClickHouse Server,
in production environments we strongly recommend that ClickHouse Keeper runs on dedicated hosts.