Configuring automatic TLS provisioning via ACME

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/guides/sre/tls/configuring-tls-acme-client.md)

Experimental feature. [Learn more.](/docs/beta-and-experimental-features#experimental-features)

Not supported in ClickHouse Cloud

Note

This page isn't applicable to [ClickHouse Cloud](https://clickhouse.com/cloud). The procedure documented here is automated in ClickHouse Cloud services.

This guide provides describes how to configure ClickHouse to use [ACME](https://en.wikipedia.org/wiki/Automatic_Certificate_Management_Environment) (described by [RFC8555](https://www.rfc-editor.org/rfc/rfc8555)) protocol.
With ACME support, ClickHouse can obtain and renew certificates from providers like [Let's Encrypt](https://letsencrypt.org/) or [ZeroSSL](https://zerossl.com/) automatically.
TLS encryption protects data in transit between clients and ClickHouse servers, preventing eavesdropping on sensitive queries and results.

## Overview[​](#overview "Direct link to Overview")

ACME protocol defines automatic certificate update process with services like [Let's Encrypt](https://letsencrypt.org/) or [ZeroSSL](https://zerossl.com/). In short, ClickHouse as certificate requester needs to confirm domain ownership via predefined challenge types in order to get a certificate.

To enable ACME, configure HTTP and HTTPS ports along with the `acme` block:

```
<http_port>80</http_port>
<https_port>443</https_port>

<acme>
    <email>[email protected]</email>
    <terms_of_service_agreed>true</terms_of_service_agreed>
    <domains>
        <domain>example.com</domain>
    </domains>
</acme>
```

The HTTP port serves ACME `HTTP-01` challenge (more on challenge types [here](https://letsencrypt.org/docs/challenge-types/)) requests during domain validation. Once validation completes and a certificate is issued, the HTTPS port serves encrypted traffic using the obtained certificate.

The HTTP port doesn't need to be 80 on the server itself; it may be remapped using `nftables` or similar tools. Check your ACME provider's documentation for accepted ports for `HTTP-01` challenges.

In the `acme` block, we're defining `email` for account creation, and accepting ACME service terms of service.
After that, the only thing we need is a list of domains.

### Current limitations[​](#current-limitations "Direct link to Current limitations")

- Only `HTTP-01` challenge type is supported.
- Only `RSA 2048` keys are supported.
- Rate limiting isn't handled.

## Configuration parameters[​](#configuration-parameters "Direct link to Configuration parameters")

Configuration options available in `acme` section:

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Parameter Default value Description|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `zookeeper_path` `/clickhouse/acme` ZooKeeper path used to store ACME account data, certificates, and coordination state between ClickHouse nodes.|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `directory_url` `https://acme-v02.api.letsencrypt.org/directory` ACME directory endpoint used for certificate issuance. Defaults to the Let’s Encrypt production server.|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `email` Email address used to create and manage the ACME account. ACME providers may use it for expiration notices and important updates.|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `terms_of_service_agreed` `false` Indicates whether the ACME provider’s Terms of Service are accepted. Must be set to `true` to enable ACME.| `domains` List of domain names for which TLS certificates should be issued. Each domain is specified as a `<domain>` entry.| `refresh_certificates_before` `2592000` (one month, in seconds) Time before certificate expiration when ClickHouse will attempt to renew the certificate.|  |  |  | | --- | --- | --- | | `refresh_certificates_task_interval` `3600` (one hour, in seconds) Interval at which ClickHouse checks whether certificates need renewal. | | | | | | | | | | | | | | | | | | | | | | | |

Note that configuration uses Let's Encrypt production directory by default. To avoid hitting request quota due to probable misconfiguration, it is recommended to test certificate issuing process with [staging directory](https://letsencrypt.org/docs/staging-environment/) first.

# Administration

## Initial deployment[​](#initial-deployment "Direct link to Initial deployment")

When enabling the ACME client on a cluster with multiple replicas, additional care is required during the initial certificate issuance.

The first replica that starts with ACME enabled will immediately attempt to create an ACME order and perform HTTP-01 challenge validation. If only a subset of replicas is serving traffic at that moment, the challenge is likely to fail, as other replicas won't be able to respond to validation requests.

If possible, it is recommended to temporarily route traffic to a single replica (for example, by adjusting DNS records) and allow it to complete the initial certificate issuance. Once the certificate is successfully issued and stored in Keeper, ACME can be enabled on the remaining replicas. They will automatically reuse the existing certificate and participate in future renewals.

If routing traffic to a single replica isn't feasible, an alternative approach is to manually upload the existing certificate and private key into Keeper before enabling the ACME client. This avoids the initial validation step and allows all replicas to start with a valid certificate already present.

After the initial certificate has been issued or imported, certificate renewal doesn't require special handling, as all replicas will already be running the ACME client and sharing state through Keeper.

## Keeper data structure[​](#keeper-data-structure "Direct link to Keeper data structure")

```
/clickhouse/acme
└── <acme-directory-host>
    ├── account_private_key          # ACME account private key (PEM)
    ├── challenges                   # Active HTTP-01 challenge state
    └── domains
        └── <domain-name>
            ├── certificate          # Issued TLS certificate (PEM)
            └── private_key          # Domain private key (PEM)
```

## Migrating from other ACME clients[​](#migrating-from-other-acme-clients "Direct link to Migrating from other ACME clients")

It is possible to migrate current TLS certificate and key to Keeper for easier migration.
At the moment, server supports only `RSA 2048` keys.

Assuming we're migrating from `certbot` and using `/etc/letsencrypt/live` directory, one may use the following set of commands:

```
DOMAIN=example.com
CERT_DIR=/etc/letsencrypt/live/$DOMAIN
ZK_BASE=/clickhouse/acme/acme-v02.api.letsencrypt.org/domains/$DOMAIN

clickhouse keeper-client -q "create '/clickhouse' ''"
clickhouse keeper-client -q "create '/clickhouse/acme' ''"
clickhouse keeper-client -q "create '/clickhouse/acme/acme-v02.api.letsencrypt.org' ''"
clickhouse keeper-client -q "create '/clickhouse/acme/acme-v02.api.letsencrypt.org/domains' ''"
clickhouse keeper-client -q "create '$ZK_BASE' ''"

clickhouse keeper-client -q "create '$ZK_BASE/certificate' \"$(cat $CERT_DIR/fullchain.pem)\""
clickhouse keeper-client -q "set '$ZK_BASE/certificate' \"$(cat $CERT_DIR/fullchain.pem)\""

clickhouse keeper-client -q "create '$ZK_BASE/private_key' \"$(cat $CERT_DIR/privkey.pem)\""
clickhouse keeper-client -q "set '$ZK_BASE/private_key' \"$(cat $CERT_DIR/privkey.pem)\""
```