Integrating Kafka with ClickHouse

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/integrations/data-ingestion/kafka/index.md)

[Apache Kafka](https://kafka.apache.org/) is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. ClickHouse provides multiple options to **read from** and **write to** Kafka and other Kafka API-compatible brokers (e.g., Redpanda, Amazon MSK).

## Available options[​](#available-options "Direct link to Available options")

Choosing the right option for your use case depends on multiple factors, including your ClickHouse deployment type, data flow direction and operational requirements.

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Option Deployment type Fully managed Kafka to ClickHouse ClickHouse to Kafka|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [ClickPipes for Kafka](/docs/integrations/clickpipes/kafka) [Cloud](/docs/cloud/get-started), [BYOC](/docs/cloud/reference/byoc/overview) (coming soon!) ✅ ✅ |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [Kafka Connect Sink](/docs/integrations/kafka/clickhouse-kafka-connect-sink) [Cloud](/docs/cloud/get-started), [BYOC](/docs/cloud/reference/byoc/overview), [Self-hosted](/docs/intro) ✅ |  |  |  |  |  | | --- | --- | --- | --- | --- | | [Kafka table engine](/docs/integrations/kafka/kafka-table-engine) [Cloud](/docs/cloud/get-started), [BYOC](/docs/cloud/reference/byoc/overview), [Self-hosted](/docs/intro) ✅ ✅ | | | | | | | | | | | | | | | | | | | |

For a more detailed comparison between these options, see [Choosing an option](#choosing-an-option).

### ClickPipes for Kafka[​](#clickpipes-for-kafka "Direct link to ClickPipes for Kafka")

[ClickPipes](/docs/integrations/clickpipes) is a managed integration platform that makes ingesting data from a diverse set of sources as simple as clicking a few buttons. Because it is fully managed and purpose-built for production workloads, ClickPipes significantly lowers infrastructure and operational costs, removing the need for external data streaming and ETL tools.

Tip

This is the recommended option if you're a ClickHouse Cloud user. ClickPipes is **fully managed** and purpose-built to deliver the **best performance** in Cloud environments.

#### Main features[​](#clickpipes-for-kafka-main-features "Direct link to Main features")

- Optimized for ClickHouse Cloud, delivering blazing-fast performance
- Horizontal and vertical scalability for high-throughput workloads
- Built-in fault tolerance with configurable replicas and automatic retries
- Deployment and management via ClickHouse Cloud UI, [Open API](/docs/cloud/manage/api/api-overview), or [Terraform](https://registry.terraform.io/providers/ClickHouse/clickhouse/3.3.3-alpha2/docs/resources/clickpipe)
- Enterprise-grade security with support for cloud-native authorization (IAM) and private connectivity (PrivateLink)
- Supports a wide range of [data sources](/docs/integrations/clickpipes/kafka/reference), including Confluent Cloud, Amazon MSK, Redpanda Cloud, and Azure Event Hubs
- Supports most common serialization formats (JSON, Avro, Protobuf coming soon!)

#### Getting started[​](#clickpipes-for-kafka-getting-started "Direct link to Getting started")

To get started using ClickPipes for Kafka, see the [reference documentation](/docs/integrations/clickpipes/kafka/reference) or navigate to the `Data Sources` tab in the ClickHouse Cloud UI.

### Kafka Connect Sink[​](#kafka-connect-sink "Direct link to Kafka Connect Sink")

Kafka Connect is an open-source framework that works as a centralized data hub for simple data integration between Kafka and other data systems. The [ClickHouse Kafka Connect Sink](https://github.com/ClickHouse/clickhouse-kafka-connect) connector provides a scalable and highly-configurable option to read data from Apache Kafka and other Kafka API-compatible brokers.

Tip

This is the recommended option if you prefer **high configurability** or are already a Kafka Connect user.

#### Main features[​](#kafka-connect-sink-main-features "Direct link to Main features")

- Can be configured to support exactly-once semantics
- Supports most common serialization formats (JSON, Avro, Protobuf)
- Tested continuously against ClickHouse Cloud

#### Getting started[​](#kafka-connect-sink-getting-started "Direct link to Getting started")

To get started using the ClickHouse Kafka Connect Sink, see the [reference documentation](/docs/integrations/kafka/clickhouse-kafka-connect-sink).

### Kafka table engine[​](#kafka-table-engine "Direct link to Kafka table engine")

The [Kafka table engine](/docs/integrations/kafka/kafka-table-engine) can be used to read data from and write data to Apache Kafka and other Kafka API-compatible brokers. This option is bundled with open-source ClickHouse and is available across all deployment types.

Tip

This is the recommended option if you're self-hosting ClickHouse and need a **low entry barrier** option, or if you need to **write** data to Kafka.

#### Main features[​](#kafka-table-engine-main-features "Direct link to Main features")

- Can be used for [reading](/docs/integrations/kafka/kafka-table-engine#kafka-to-clickhouse) and [writing](/docs/integrations/kafka/kafka-table-engine#clickhouse-to-kafka) data
- Bundled with open-source ClickHouse
- Supports most common serialization formats (JSON, Avro, Protobuf)

#### Getting started[​](#kafka-table-engine-getting-started "Direct link to Getting started")

To get started using the Kafka table engine, see the [reference documentation](/docs/integrations/kafka/kafka-table-engine).

### Choosing an option[​](#choosing-an-option "Direct link to Choosing an option")

|  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Product Strengths Weaknesses|  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **ClickPipes for Kafka** • Scalable architecture for high throughput and low latency • Built-in monitoring and schema management • Private networking connections (via PrivateLink) • Supports SSL/TLS authentication and IAM authorization • Supports programmatic configuration (Terraform, API endpoints) • Doesn't support pushing data to Kafka • At-least-once semantics| **Kafka Connect Sink** • Exactly-once semantics • Allows granular control over data transformation, batching and error handling • Can be deployed in private networks • Allows real-time replication from databases not yet supported in ClickPipes via Debezium • Doesn't support pushing data to Kafka • Operationally complex to set up and maintain • Requires Kafka and Kafka Connect expertise| **Kafka table engine** • Supports [pushing data to Kafka](/docs/integrations/kafka/kafka-table-engine#clickhouse-to-kafka) • Operationally simple to set up • At-least-once semantics • Limited horizontal scaling for consumers. Can't be scaled independently from the ClickHouse server • Limited error handling and debugging options • Requires Kafka expertise | | | | | | | | | | | |

### Other options[​](#other-options "Direct link to Other options")