ClickHouse Kafka Connect Sink

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-ingestion/kafka/kafka-clickhouse-connect-sink.md)

Note

If you need any help, please [file an issue in the repository](https://github.com/ClickHouse/clickhouse-kafka-connect/issues) or raise a question in [ClickHouse public Slack](https://clickhouse.com/slack).

**ClickHouse Kafka Connect Sink** is the Kafka connector delivering data from a Kafka topic to a ClickHouse table.

### License[​](#license "Direct link to License")

The Kafka Connector Sink is distributed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0)

### Requirements for the environment[​](#requirements-for-the-environment "Direct link to Requirements for the environment")

The [Kafka Connect](https://docs.confluent.io/platform/current/connect/index.html) framework v2.7 or later should be installed in the environment.

### Version compatibility matrix[​](#version-compatibility-matrix "Direct link to Version compatibility matrix")

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ClickHouse Kafka Connect version ClickHouse version Kafka Connect Confluent platform|  |  |  |  | | --- | --- | --- | --- | | 1.0.0 > 23.3 > 2.7 > 6.1 | | | | | | | |

### Main features[​](#main-features "Direct link to Main features")

- Shipped with out-of-the-box exactly-once semantics. It's powered by a new ClickHouse core feature named [KeeperMap](https://github.com/ClickHouse/ClickHouse/pull/39976) (used as a state store by the connector) and allows for minimalistic architecture.
- Support for 3rd-party state stores: Currently defaults to In-memory but can use KeeperMap (Redis to be added soon).
- Core integration: Built, maintained, and supported by ClickHouse.
- Tested continuously against [ClickHouse Cloud](https://clickhouse.com/cloud).
- Data inserts with a declared schema and schemaless.
- Support for all data types of ClickHouse.

### Installation instructions[​](#installation-instructions "Direct link to Installation instructions")

#### Gather your connection details[​](#gather-your-connection-details "Direct link to Gather your connection details")

To connect to ClickHouse with HTTP(S) you need this information:

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Parameter(s) Description|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | `HOST` and `PORT` Typically, the port is 8443 when using TLS or 8123 when not using TLS.|  |  |  |  | | --- | --- | --- | --- | | `DATABASE NAME` Out of the box, there is a database named `default`, use the name of the database that you want to connect to.| `USERNAME` and `PASSWORD` Out of the box, the username is `default`. Use the username appropriate for your use case. | | | | | | | |

The details for your ClickHouse Cloud service are available in the ClickHouse Cloud console.
Select a service and click **Connect**:

Choose **HTTPS**. Connection details are displayed in an example `curl` command.

If you're using self-managed ClickHouse, the connection details are set by your ClickHouse administrator.

#### General installation instructions[​](#general-installation-instructions "Direct link to General installation instructions")

The connector is distributed as a single JAR file containing all the class files necessary to run the plugin.

To install the plugin, follow these steps:

- Download a zip archive containing the Connector JAR file from the [Releases](https://github.com/ClickHouse/clickhouse-kafka-connect/releases) page of ClickHouse Kafka Connect Sink repository.
- Extract the ZIP file content and copy it to the desired location.
- Add a path with the plugin director to [plugin.path](https://kafka.apache.org/documentation/#connectconfigs_plugin.path) configuration in your Connect properties file to allow Confluent Platform to find the plugin.
- Provide a topic name, ClickHouse instance hostname, and password in config.

```
connector.class=com.clickhouse.kafka.connect.ClickHouseSinkConnector
tasks.max=1
topics=<topic_name>
ssl=true
jdbcConnectionProperties=?sslmode=STRICT
security.protocol=SSL
hostname=<hostname>
database=<database_name>
password=<password>
ssl.truststore.location=/tmp/kafka.client.truststore.jks
port=8443
value.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
exactlyOnce=true
username=default
schemas.enable=false
```

- Restart the Confluent Platform.
- If you use Confluent Platform, log into Confluent Control Center UI to verify the ClickHouse Sink is available in the list of available connectors.

### Configuration options[​](#configuration-options "Direct link to Configuration options")

To connect the ClickHouse Sink to the ClickHouse server, you need to provide:

- connection details: hostname (**required**) and port (optional)
- user credentials: password (**required**) and username (optional)
- connector class: `com.clickhouse.kafka.connect.ClickHouseSinkConnector` (**required**)
- topics or topics.regex: the Kafka topics to poll - topic names must match table names (**required**)
- key and value converters: set based on the type of data on your topic. Required if not already defined in worker config.

The full table of configuration options: