Monitoring Kafka Metrics with ClickStack

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/ingesting-data/integration-examples/kafka-metrics.md)

TL;DR

Monitor Apache Kafka performance metrics in ClickStack using the OTel JMX Metric Gatherer. Includes a demo dataset and pre-built dashboard.

## Integration with an existing Kafka deployment[​](#existing-kafka "Direct link to Integration with an existing Kafka deployment")

Monitor your existing Kafka deployment by running the OpenTelemetry JMX Metric Gatherer container to collect metrics and send them to ClickStack via OTLP.

If you want to test this integration first without modifying your existing setup, skip to the [demo dataset section](#demo-dataset).

##### Prerequisites[​](#prerequisites "Direct link to Prerequisites")

- ClickStack instance running
- Existing Kafka installation (version 2.0 or newer) with JMX enabled
- Network access between ClickStack and Kafka (JMX port 9999, Kafka port 9092)
- OpenTelemetry JMX Metric Gatherer JAR (download instructions below)

Get ClickStack API key

#### Get ClickStack API key[​](#get-api-key "Direct link to Get ClickStack API key")

The JMX Metric Gatherer sends data to ClickStack's OTLP endpoint, which requires authentication.

1. Open HyperDX at your ClickStack URL (e.g., <http://localhost:8080>)
2. Create an account or log in if needed
3. Navigate to **Team Settings → API Keys**
4. Copy your **Ingestion API Key**

5. Set it as an environment variable:

```
export CLICKSTACK_API_KEY=your-api-key-here
```

Download the OpenTelemetry JMX Metric Gatherer

#### Download the OpenTelemetry JMX Metric Gatherer[​](#download-jmx "Direct link to Download the OpenTelemetry JMX Metric Gatherer")

Download the JMX Metric Gatherer JAR:

```
curl -L -o opentelemetry-jmx-metrics.jar \
  https://github.com/open-telemetry/opentelemetry-java-contrib/releases/download/v1.32.0/opentelemetry-jmx-metrics.jar
```

Verify Kafka JMX is enabled

#### Verify Kafka JMX is enabled[​](#verify-jmx "Direct link to Verify Kafka JMX is enabled")

Ensure JMX is enabled on your Kafka brokers. For Docker deployments:

```
services:
  kafka:
    image: confluentinc/cp-kafka:latest
    environment:
      JMX_PORT: 9999
      KAFKA_JMX_HOSTNAME: kafka
      # ... other Kafka configuration
    ports:
      - "9092:9092"
      - "9999:9999"
```

For non-Docker deployments, set these in your Kafka startup:

```
export JMX_PORT=9999
```

Verify JMX is accessible:

```
netstat -an | grep 9999
```

Deploy JMX Metric Gatherer with Docker Compose

#### Deploy JMX Metric Gatherer with Docker Compose[​](#deploy-jmx "Direct link to Deploy JMX Metric Gatherer with Docker Compose")

This example shows a complete setup with Kafka, the JMX Metric Gatherer, and ClickStack. Adjust service names and endpoints to match your existing deployment:

```
services:
  clickstack:
    image: clickhouse/clickstack-all-in-one:latest
    ports:
      - "8080:8080"
      - "4317:4317"
      - "4318:4318"
    networks:
      - monitoring

  kafka:
    image: confluentinc/cp-kafka:latest
    hostname: kafka
    container_name: kafka
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka:9092'
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093'
      KAFKA_LISTENERS: 'PLAINTEXT://kafka:9092,CONTROLLER://kafka:29093'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
      JMX_PORT: 9999
      KAFKA_JMX_HOSTNAME: kafka
      KAFKA_JMX_OPTS: '-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka -Dcom.sun.management.jmxremote.rmi.port=9999'
    ports:
      - "9092:9092"
      - "9999:9999"
    networks:
      - monitoring

  kafka-jmx-exporter:
    image: eclipse-temurin:11-jre
    depends_on:
      - kafka
      - clickstack
    environment:
      - CLICKSTACK_API_KEY=${CLICKSTACK_API_KEY}
    volumes:
      - ./opentelemetry-jmx-metrics.jar:/app/opentelemetry-jmx-metrics.jar
    command: >
      sh -c "java
      -Dotel.jmx.service.url=service:jmx:rmi:///jndi/rmi://kafka:9999/jmxrmi
      -Dotel.jmx.target.system=kafka
      -Dotel.metrics.exporter=otlp
      -Dotel.exporter.otlp.protocol=http/protobuf
      -Dotel.exporter.otlp.endpoint=http://clickstack:4318
      -Dotel.exporter.otlp.headers=authorization=\${CLICKSTACK_API_KEY}
      -Dotel.resource.attributes=service.name=kafka,kafka.broker.id=broker-0
      -Dotel.jmx.interval.milliseconds=10000
      -jar /app/opentelemetry-jmx-metrics.jar"
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge
```

**Key configuration parameters:**

- `service:jmx:rmi:///jndi/rmi://kafka:9999/jmxrmi` - JMX connection URL (use your Kafka hostname)
- `otel.jmx.target.system=kafka` - Enables Kafka-specific metrics
- `http://clickstack:4318` - OTLP HTTP endpoint (use your ClickStack hostname)
- `authorization=\${CLICKSTACK_API_KEY}` - API key for authentication (required)
- `service.name=kafka,kafka.broker.id=broker-0` - Resource attributes for filtering
- `10000` - Collection interval in milliseconds (10 seconds)

Verify metrics in HyperDX

#### Verify metrics in HyperDX[​](#verify-metrics "Direct link to Verify metrics in HyperDX")

Log into HyperDX and confirm metrics are flowing:

1. Navigate to the Chart Explorer
2. Search for `kafka.message.count` or `kafka.partition.count`
3. Metrics should appear at 10-second intervals

**Key metrics to verify:**

- `kafka.message.count` - Total messages processed
- `kafka.partition.count` - Total partitions
- `kafka.partition.under_replicated` - Should be 0 in a healthy cluster
- `kafka.network.io` - Network throughput
- `kafka.request.time.*` - Request latency percentiles

To generate activity and populate more metrics:

```
# Create a test topic
docker exec kafka bash -c "unset JMX_PORT && kafka-topics --create --topic test-topic --bootstrap-server kafka:9092 --partitions 3 --replication-factor 1"

# Send test messages
echo -e "Message 1\nMessage 2\nMessage 3" | docker exec -i kafka bash -c "unset JMX_PORT && kafka-console-producer --topic test-topic --bootstrap-server kafka:9092"
```

Note

When running Kafka client commands (kafka-topics, kafka-console-producer, etc.) from within the Kafka container, prefix with `unset JMX_PORT &&` to prevent JMX port conflicts.

## Demo dataset[​](#demo-dataset "Direct link to Demo dataset")

For users who want to test the Kafka Metrics integration before configuring their production systems, we provide a pre-generated dataset with realistic Kafka metrics patterns.

Download the sample metrics dataset

#### Download the sample metrics dataset[​](#download-sample "Direct link to Download the sample metrics dataset")

Download the pre-generated metrics files (29 hours of Kafka metrics with realistic patterns):

```
# Download gauge metrics (partition counts, queue sizes, latencies, consumer lag)
curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/kafka/kafka-metrics-gauge.csv

# Download sum metrics (message rates, byte rates, request counts)
curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/kafka/kafka-metrics-sum.csv
```

The dataset includes realistic patterns for a single-broker e-commerce Kafka cluster: