ClickHouse Kafka Connect Sink

#### Kafka Producer/Consumer Metrics[​](#kafka-producer-consumer-metrics "Direct link to Kafka Producer/Consumer Metrics")

The connector exposes standard Kafka producer and consumer metrics that provide insights into data flow, throughput, and performance.

**Topic-Level Metrics:**

- `records-sent-total`: Total number of records sent to the topic
- `bytes-sent-total`: Total bytes sent to the topic
- `record-send-rate`: Average rate of records sent per second
- `byte-rate`: Average bytes sent per second
- `compression-rate`: Compression ratio achieved

**Partition-Level Metrics:**

- `records-sent-total`: Total records sent to the partition
- `bytes-sent-total`: Total bytes sent to the partition
- `records-lag`: Current lag in the partition
- `records-lead`: Current lead in the partition
- `replica-fetch-lag`: Lag information for replicas

**Node-Level Connection Metrics:**

- `connection-creation-total`: Total connections created to the Kafka node
- `connection-close-total`: Total connections closed
- `request-total`: Total requests sent to the node
- `response-total`: Total responses received from the node
- `request-rate`: Average request rate per second
- `response-rate`: Average response rate per second

These metrics help monitor:

- **Throughput**: Track data ingestion rates
- **Lag**: Identify bottlenecks and processing delays
- **Compression**: Measure data compression efficiency
- **Connection Health**: Monitor network connectivity and stability

#### Kafka Connect Framework Metrics[​](#kafka-connect-framework-metrics "Direct link to Kafka Connect Framework Metrics")

The connector integrates with the Kafka Connect framework and exposes metrics for task lifecycle and error tracking.

**Task Status Metrics:**

- `task-count`: Total number of tasks in the connector
- `running-task-count`: Number of tasks currently running
- `paused-task-count`: Number of tasks currently paused
- `failed-task-count`: Number of tasks that have failed
- `destroyed-task-count`: Number of destroyed tasks
- `unassigned-task-count`: Number of unassigned tasks

Task status values include: `running`, `paused`, `failed`, `destroyed`, `unassigned`

**Error Metrics:**

- `deadletterqueue-produce-failures`: Number of failed DLQ writes
- `deadletterqueue-produce-requests`: Total DLQ write attempts
- `last-error-timestamp`: Timestamp of the last error
- `records-skip-total`: Total number of records skipped due to errors
- `records-retry-total`: Total number of records that were retried
- `errors-total`: Total number of errors encountered

**Performance Metrics:**

- `offset-commit-failures`: Number of failed offset commits
- `offset-commit-avg-time-ms`: Average time for offset commits
- `offset-commit-max-time-ms`: Maximum time for offset commits
- `put-batch-avg-time-ms`: Average time to process a batch
- `put-batch-max-time-ms`: Maximum time to process a batch
- `source-record-poll-total`: Total records polled

#### Monitoring Best Practices[​](#monitoring-best-practices "Direct link to Monitoring Best Practices")

1. **Monitor Consumer Lag**: Track `records-lag` per partition to identify processing bottlenecks
2. **Track Error Rates**: Watch `errors-total` and `records-skip-total` to detect data quality issues
3. **Observe Task Health**: Monitor task status metrics to ensure tasks are running properly
4. **Measure Throughput**: Use `records-send-rate` and `byte-rate` to track ingestion performance
5. **Monitor Connection Health**: Check node-level connection metrics for network issues
6. **Track Compression Efficiency**: Use `compression-rate` to optimize data transfer

For detailed JMX metric definitions and Prometheus integration, see the [jmx-export-connector.yml](https://github.com/ClickHouse/clickhouse-kafka-connect/blob/main/jmx-export-connector.yml) configuration file.

### Limitations[​](#limitations "Direct link to Limitations")

- Deletes aren't supported.
- Batch size is inherited from the Kafka Consumer properties.
- When using KeeperMap for exactly-once and the offset is changed or re-wound, you need to delete the content from KeeperMap for that specific topic. (See troubleshooting guide below for more details)

### Performance tuning and throughput optimization[​](#tuning-performance "Direct link to Performance tuning and throughput optimization")

This section covers performance tuning strategies for the ClickHouse Kafka Connect Sink. Performance tuning is essential when dealing with high-throughput use cases or when you need to optimize resource utilization and minimize lag.

#### When is performance tuning needed?[​](#when-is-performance-tuning-needed "Direct link to When is performance tuning needed?")

Performance tuning is typically required in the following scenarios:

- **High-throughput workloads**: When processing millions of events per second from Kafka topics
- **Consumer lag**: When your connector can't keep up with the rate of data production, causing increasing lag
- **Resource constraints**: When you need to optimize CPU, memory, or network usage
- **Multiple topics**: When consuming from multiple high-volume topics simultaneously
- **Small message sizes**: When dealing with many small messages that would benefit from server-side batching

Performance tuning is **NOT typically needed** when:

- You're processing low to moderate volumes (< 10,000 messages/second)
- Consumer lag is stable and acceptable for your use case
- Default connector settings already meet your throughput requirements
- Your ClickHouse cluster can easily handle the incoming load

#### Understanding the data flow[​](#understanding-the-data-flow "Direct link to Understanding the data flow")

Before tuning, it's important to understand how data flows through the connector:

1. **Kafka Connect Framework** fetches messages from Kafka topics in the background
2. **Connector polls** for messages from the framework's internal buffer
3. **Connector batches** messages based on poll size
4. **ClickHouse receives** the batched insert via HTTP/S
5. **ClickHouse processes** the insert (synchronously or asynchronously)

Performance can be optimized at each of these stages.

#### Kafka Connect batch size tuning[​](#connect-fetch-vs-connector-poll "Direct link to Kafka Connect batch size tuning")

The first level of optimization is controlling how much data the connector receives per batch from Kafka.

##### Fetch settings[​](#fetch-settings "Direct link to Fetch settings")

Kafka Connect (the framework) fetches messages from Kafka topics in the background, independent of the connector:

- **`fetch.min.bytes`**: Minimum amount of data before the framework passes values to the connector (default: 1 byte)
- **`fetch.max.bytes`**: Maximum amount of data to fetch in a single request (default: 52428800 / 50 MB)
- **`fetch.max.wait.ms`**: Maximum time to wait before returning data if `fetch.min.bytes` isn't met (default: 500 ms)

Note

On Confluent Cloud, adjustment of these settings requires opening a support case through Confluent Cloud.

##### Poll settings[​](#poll-settings "Direct link to Poll settings")

The connector polls for messages from the framework's buffer:

- **`max.poll.records`**: Maximum number of records returned in a single poll (default: 500)
- **`max.partition.fetch.bytes`**: Maximum amount of data per partition (default: 1048576 / 1 MB)

Note

On Confluent Cloud, adjustment of these settings requires opening a support case through Confluent Cloud.

##### Recommended settings for high throughput[​](#recommended-batch-settings "Direct link to Recommended settings for high throughput")

For optimal performance with ClickHouse, aim for larger batches:

```
# Increase the number of records per poll
consumer.max.poll.records=5000

# Increase the partition fetch size (5 MB)
consumer.max.partition.fetch.bytes=5242880

# Optional: Increase minimum fetch size to wait for more data (1 MB)
consumer.fetch.min.bytes=1048576

# Optional: Reduce wait time if latency is critical
consumer.fetch.max.wait.ms=300
```