Monitoring Kafka Logs with ClickStack

- type: move
        from: attributes.message
        to: body

      - type: add
        field: attributes.source
        value: "kafka-demo"

      - type: add
        field: resource["service.name"]
        value: "kafka-demo"

service:
  pipelines:
    logs/kafka-demo:
      receivers: [filelog/kafka]
      processors:
        - memory_limiter
        - transform
        - batch
      exporters:
        - clickhouse
EOF
```

Run ClickStack with demo configuration

#### Run ClickStack with demo configuration[​](#run-demo "Direct link to Run ClickStack with demo configuration")

Run ClickStack with the demo logs and configuration:

```
docker run --name clickstack-demo \
  -p 8080:8080 -p 4317:4317 -p 4318:4318 \
  -e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
  -v "$(pwd)/kafka-logs-demo.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
  -v "$(pwd)/server.log:/tmp/kafka-demo/server.log:ro" \
  clickhouse/clickstack-all-in-one:latest
```

## Verify logs in HyperDX[​](#verify-demo-logs "Direct link to Verify logs in HyperDX")

Once ClickStack is running:

1. Open [HyperDX](http://localhost:8080/) and log in to your account (you may need to create an account first)
2. Navigate to the Search view and set the source to `Logs`
3. Set the time range to include **2026-03-09 00:00:00 - 2026-03-10 00:00:00 (UTC)**

## Dashboards and visualization[​](#dashboards "Direct link to Dashboards and visualization")

Download the dashboard configuration

#### [Download](/docs/examples/kafka-logs-dashboard.json) the dashboard configuration[​](#download "Direct link to download")

Import pre-built dashboard

#### Import pre-built dashboard[​](#import-dashboard "Direct link to Import pre-built dashboard")

1. Open HyperDX and navigate to the Dashboards section.
2. Click "Import Dashboard" in the upper right corner under the ellipses.

3. Upload the kafka-logs-dashboard.json file and click finish import.

The dashboard will be created with all visualizations pre-configured

#### The dashboard will be created with all visualizations pre-configured[​](#created-dashboard "Direct link to The dashboard will be created with all visualizations pre-configured")

For the demo dataset, set the time range to include **2026-03-09 00:00:00 - 2026-03-10 00:00:00 (UTC)**.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

**Verify the effective config includes your filelog receiver:**

```
docker exec <container> cat /etc/otel/supervisor-data/effective.yaml | grep -A 10 filelog
```

**Check for collector errors:**

```
docker exec <container> cat /etc/otel/supervisor-data/agent.log
```

**Verify Kafka log format matches the expected pattern:**

```
tail -1 /var/log/kafka/server.log
```

If your Kafka installation uses a custom Log4j pattern, adjust the `regex_parser` regex accordingly.

## Next steps[​](#next-steps "Direct link to Next steps")

- Set up [alerts](/docs/use-cases/observability/clickstack/alerts) for critical events (broker failures, replication errors, consumer group issues)
- Combine with [Kafka Metrics](/docs/use-cases/observability/clickstack/integrations/kafka-metrics) for comprehensive Kafka monitoring
- Create additional [dashboards](/docs/use-cases/observability/clickstack/dashboards) for specific use cases (controller events, partition reassignment)

## Going to production[​](#going-to-production "Direct link to Going to production")

This guide extends ClickStack's built-in OpenTelemetry Collector for quick setup. For production deployments, we recommend running your own OTel Collector and sending data to ClickStack's OTLP endpoint. See [Sending OpenTelemetry data](/docs/use-cases/observability/clickstack/ingesting-data/opentelemetry) for production configuration.

- [Integration with existing Kafka](#existing-kafka)
  - [Prerequisites](#prerequisites)- [Demo dataset](#demo-dataset)- [Verify logs in HyperDX](#verify-demo-logs)- [Dashboards and visualization](#dashboards)- [Troubleshooting](#troubleshooting)- [Next steps](#next-steps)- [Going to production](#going-to-production)

Was this page helpful?