Using the Kafka table engine

Multiple ClickHouse instances can all be configured to read from a topic using the same consumer group id - specified during the Kafka table engine creation. Therefore, each instance will read from one or more partitions, inserting segments to their local target table. The target tables can, in turn, be configured to use a ReplicatedMergeTree to handle duplication of the data. This approach allows Kafka reads to be scaled with the ClickHouse cluster, provided there are sufficient Kafka partitions.

#### Tuning performance[​](#tuning-performance "Direct link to Tuning performance")

Consider the following when looking to increase Kafka Engine table throughput performance:

- The performance will vary depending on the message size, format, and target table types. 100k rows/sec on a single table engine should be considered obtainable. By default, messages are read in blocks, controlled by the parameter kafka\_max\_block\_size. By default, this is set to the [max\_insert\_block\_size](/docs/operations/settings/settings#max_insert_block_size), defaulting to 1,048,576. Unless messages are extremely large, this should nearly always be increased. Values between 500k to 1M aren't uncommon. Test and evaluate the effect on throughput performance.
- The number of consumers for a table engine can be increased using kafka\_num\_consumers. However, by default, inserts will be linearized in a single thread unless kafka\_thread\_per\_consumer is changed from the default value of 1. Set this to 1 to ensure flushes are performed in parallel. Note that creating a Kafka engine table with N consumers (and kafka\_thread\_per\_consumer=1) is logically equivalent to creating N Kafka engines, each with a materialized view and kafka\_thread\_per\_consumer=0.
- Increasing consumers isn't a free operation. Each consumer maintains its own buffers and threads, increasing the overhead on the server. Be conscious of the overhead of consumers and scale linearly across your cluster first and if possible.
- If the throughput of Kafka messages is variable and delays are acceptable, consider increasing the stream\_flush\_interval\_ms to ensure larger blocks are flushed.
- [background\_message\_broker\_schedule\_pool\_size](/docs/operations/server-configuration-parameters/settings#background_message_broker_schedule_pool_size) sets the number of threads performing background tasks. These threads are used for Kafka streaming. This setting is applied at the ClickHouse server start and can't be changed in a user session, defaulting to 16. If you see timeouts in the logs, it may be appropriate to increase this.
- For communication with Kafka, the librdkafka library is used, which itself creates threads. Large numbers of Kafka tables, or consumers, can thus result in large numbers of context switches. Either distribute this load across the cluster, only replicating the target tables if possible, or consider using a table engine to read from multiple topics - a list of values is supported. Multiple materialized views can be read from a single table, each filtering to the data from a specific topic.

Any settings changes should be tested. We recommend monitoring Kafka consumer lags to ensure you're properly scaled.

#### Additional settings[​](#additional-settings "Direct link to Additional settings")

Aside from the settings discussed above, the following may be of interest:

- [Kafka\_max\_wait\_ms](/docs/operations/settings/settings#kafka_max_wait_ms) - The wait time in milliseconds for reading messages from Kafka before retry. Set at a user profile level and defaults to 5000.

[All settings](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) from the underlying librdkafka can also be placed in the ClickHouse configuration files inside a *kafka* element - setting names should be XML elements with periods replaced with underscores e.g.

```
<clickhouse>
   <kafka>
       <enable_ssl_certificate_verification>false</enable_ssl_certificate_verification>
   </kafka>
</clickhouse>
```

These are expert settings and we'd suggest you refer to the Kafka documentation for an in-depth explanation.

- [Kafka to ClickHouse](#kafka-to-clickhouse)- [ClickHouse to Kafka](#clickhouse-to-kafka)- [Clusters and performance](#clusters-and-performance)

Was this page helpful?