Monitoring Kafka Metrics with ClickStack

Download the sample metrics dataset

#### Download the sample metrics dataset[​](#download-sample "Direct link to Download the sample metrics dataset")

Download the pre-generated metrics files (29 hours of Kafka metrics with realistic patterns):

```
# Download gauge metrics (partition counts, queue sizes, latencies, consumer lag)
curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/kafka/kafka-metrics-gauge.csv

# Download sum metrics (message rates, byte rates, request counts)
curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/kafka/kafka-metrics-sum.csv
```

The dataset includes realistic patterns for a single-broker e-commerce Kafka cluster:

- **06:00-08:00: Morning surge** - Sharp traffic ramp from overnight baseline
- **10:00-10:15: Flash sale** - Dramatic spike to 3.5x normal traffic
- **11:30: Deployment event** - 12x consumer lag spike with under-replicated partitions
- **14:00-15:30: Peak shopping** - Sustained high traffic at 2.8x baseline
- **17:00-17:30: After-work surge** - Secondary traffic peak
- **18:45: Consumer rebalance** - 6x lag spike during rebalancing
- **20:00-22:00: Evening drop** - Steep decline to overnight levels

Start ClickStack

#### Start ClickStack[​](#start-clickstack "Direct link to Start ClickStack")

Start a ClickStack instance:

```
docker run -d --name clickstack-demo \
  -p 8080:8080 -p 4317:4317 -p 4318:4318 \
  clickhouse/clickstack-all-in-one:latest
```

Load metrics into ClickStack

#### Load metrics into ClickStack[​](#load-metrics "Direct link to Load metrics into ClickStack")

Load the metrics directly into ClickHouse:

```
# Load gauge metrics (partition counts, queue sizes, latencies, consumer lag)
cat kafka-metrics-gauge.csv | docker exec -i clickstack-demo \
  clickhouse-client --query "INSERT INTO otel_metrics_gauge FORMAT CSVWithNames"

# Load sum metrics (message rates, byte rates, request counts)
cat kafka-metrics-sum.csv | docker exec -i clickstack-demo \
  clickhouse-client --query "INSERT INTO otel_metrics_sum FORMAT CSVWithNames"
```

Verify metrics in HyperDX

#### Verify metrics in HyperDX[​](#verify-demo-metrics "Direct link to Verify metrics in HyperDX")

Once loaded, the quickest way to see your metrics is through the pre-built dashboard.

Proceed to the [Dashboards and visualization](#dashboards) section to import the dashboard and view all Kafka metrics at once.

Timezone Display

HyperDX displays timestamps in your browser's local timezone. The demo data spans **2025-11-05 16:00:00 - 2025-11-06 16:00:00 (UTC)**. Set your time range to **2025-11-04 16:00:00 - 2025-11-07 16:00:00** to ensure you see the demo metrics regardless of your location. Once you see the metrics, you can narrow the range to a 24-hour period for clearer visualizations.

## Dashboards and visualization[​](#dashboards "Direct link to Dashboards and visualization")

To help you get started monitoring Kafka with ClickStack, we provide essential visualizations for Kafka metrics.

Download the dashboard configuration

#### [Download](/docs/examples/kafka-metrics-dashboard.json) the dashboard configuration[​](#download "Direct link to download")

Import the pre-built dashboard

#### Import the pre-built dashboard[​](#import-dashboard "Direct link to Import the pre-built dashboard")

1. Open HyperDX and navigate to the Dashboards section
2. Click **Import Dashboard** in the upper right corner under the ellipses

3. Upload the `kafka-metrics-dashboard.json` file and click **Finish Import**

View the dashboard

#### View the dashboard[​](#created-dashboard "Direct link to View the dashboard")

The dashboard will be created with all visualizations pre-configured:

Note

For the demo dataset, set the time range to **2025-11-05 16:00:00 - 2025-11-06 16:00:00 (UTC)** (adjust based on your local timezone). The imported dashboard won't have a time range specified by default.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### No metrics appearing in HyperDX[​](#no-metrics "Direct link to No metrics appearing in HyperDX")

**Verify API key is set and passed to the container:**

```
# Check environment variable
echo $CLICKSTACK_API_KEY

# Verify it's in the container
docker exec <jmx-exporter-container> env | grep CLICKSTACK_API_KEY
```

If missing, set it and restart:

```
export CLICKSTACK_API_KEY=your-api-key-here
docker compose up -d kafka-jmx-exporter
```

**Check if metrics are reaching ClickHouse:**

```
docker exec <clickstack-container> clickhouse-client --query "
SELECT DISTINCT MetricName 
FROM otel_metrics_sum 
WHERE ServiceName = 'kafka' 
LIMIT 10
"
```

If you don't see any results, check the JMX exporter logs:

```
docker compose logs kafka-jmx-exporter | grep -i "error\|connection" | tail -10
```

**Generate Kafka activity to populate metrics:**

```
# Create a test topic
docker exec kafka bash -c "unset JMX_PORT && kafka-topics --create --topic test-topic --bootstrap-server kafka:9092 --partitions 3 --replication-factor 1"

# Send test messages
echo -e "Message 1\nMessage 2\nMessage 3" | docker exec -i kafka bash -c "unset JMX_PORT && kafka-console-producer --topic test-topic --bootstrap-server kafka:9092"
```

### Authentication errors[​](#auth-errors "Direct link to Authentication errors")

If you see `Authorization failed` or `401 Unauthorized`:

1. Verify the API key in HyperDX UI (Settings → API Keys → Ingestion API Key)
2. Re-export and restart:

```
export CLICKSTACK_API_KEY=your-correct-api-key
docker compose down
docker compose up -d
```

### Port conflicts with Kafka client commands[​](#port-conflicts "Direct link to Port conflicts with Kafka client commands")

When running Kafka commands from within the Kafka container, you may see:

```
Error: Port already in use: 9999
```

Prefix commands with `unset JMX_PORT &&`:

```
docker exec kafka bash -c "unset JMX_PORT && kafka-topics --list --bootstrap-server kafka:9092"
```

### Network connectivity issues[​](#network-issues "Direct link to Network connectivity issues")

If the JMX exporter logs show `Connection refused`:

Verify all containers are on the same Docker network:

```
docker compose ps
docker network inspect <network-name>
```

Test connectivity:

```
# From JMX exporter to ClickStack
docker exec <jmx-exporter-container> sh -c "timeout 2 bash -c 'cat < /dev/null > /dev/tcp/clickstack/4318' && echo 'Connected' || echo 'Failed'"
```

## Next steps[​](#next-steps "Direct link to Next steps")

- Set up [alerts](/docs/use-cases/observability/clickstack/alerts) for critical metrics (under-replicated partitions, consumer lag growth, request latency spikes)
- Create additional dashboards for specific use cases (per-topic throughput, consumer group monitoring)
- Monitor multiple Kafka brokers by adding additional JMX Metric Gatherer instances with unique `kafka.broker.id` resource attributes

## Going to production[​](#going-to-production "Direct link to Going to production")

This guide sends metrics directly from the JMX Metric Gatherer to ClickStack's OTLP endpoint, which works well for testing and small deployments.

For production environments, deploy your own OpenTelemetry Collector as an agent to receive metrics from the JMX Exporter and forward them to ClickStack. This provides batching, resilience, and centralized configuration management.

See [Ingesting with OpenTelemetry](/docs/use-cases/observability/clickstack/ingesting-data/opentelemetry) for production deployment patterns and collector configuration examples.

- [Integration with an existing Kafka deployment](#existing-kafka)- [Demo dataset](#demo-dataset)- [Dashboards and visualization](#dashboards)- [Troubleshooting](#troubleshooting)
        - [No metrics appearing in HyperDX](#no-metrics)- [Authentication errors](#auth-errors)- [Port conflicts with Kafka client commands](#port-conflicts)- [Network connectivity issues](#network-issues)- [Next steps](#next-steps)- [Going to production](#going-to-production)

Was this page helpful?