Monitoring Kafka Logs with ClickStack

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/ingesting-data/integration-examples/kafka-logs.md)

TL;DR

Collect and visualize Kafka broker logs (Log4j format) in ClickStack using the OTel `filelog` receiver. Includes a demo dataset and pre-built dashboard.

## Integration with existing Kafka[​](#existing-kafka "Direct link to Integration with existing Kafka")

This section covers configuring your existing Kafka installation to send broker logs to ClickStack by modifying the ClickStack OTel collector configuration.
If you would like to test the Kafka logs integration before configuring your own existing setup, you can test with our preconfigured setup and sample data in the ["Demo dataset"](/docs/use-cases/observability/clickstack/integrations/kafka-logs#demo-dataset) section.

### Prerequisites[​](#prerequisites "Direct link to Prerequisites")

- ClickStack instance running
- Existing Kafka installation (version 2.0 or newer)
- Access to Kafka log files (`server.log`, `controller.log`, etc.)

Verify Kafka logging configuration

#### Verify Kafka logging configuration[​](#verify-kafka "Direct link to Verify Kafka logging configuration")

Kafka uses Log4j and writes logs to the directory specified by the `kafka.logs.dir` system property or the `LOG_DIR` environment variable. Check your log file location:

```
# Default locations
ls $KAFKA_HOME/logs/      # Standard Apache Kafka (defaults to <install-dir>/logs/)
ls /var/log/kafka/        # RPM/DEB package installations
```

Key Kafka log files:

- **`server.log`**: General broker logs (startup, connections, replication, errors)
- **`controller.log`**: Controller-specific events (leader election, partition reassignment)
- **`state-change.log`**: Partition and replica state transitions

Kafka's default Log4j pattern produces lines like:

```
[2026-03-09 14:23:45,123] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
```

Note

For Docker-based Kafka deployments (e.g., `confluentinc/cp-kafka`), the default Log4j configuration only includes a console appender — there is no file appender, so logs are written to stdout only. To use the `filelog` receiver, you'll need to redirect logs to a file, either by adding a file appender to `log4j.properties` or by piping stdout (e.g., `| tee /var/log/kafka/server.log`).

Create a custom OTel collector configuration for Kafka

#### Create a custom OTel collector configuration for Kafka[​](#custom-otel "Direct link to Create a custom OTel collector configuration for Kafka")

ClickStack allows you to extend the base OpenTelemetry Collector configuration by mounting a custom configuration file and setting an environment variable. The custom configuration is merged with the base configuration managed by HyperDX via OpAMP.

Create a file named `kafka-logs-monitoring.yaml` with the following configuration:

```
receivers:
  filelog/kafka:
    include:
      - /var/log/kafka/server.log
      - /var/log/kafka/controller.log  # optional, only exists if log4j is configured with separate file appenders
      - /var/log/kafka/state-change.log  # optional, same as above
    start_at: beginning
    multiline:
      line_start_pattern: '^\[\d{4}-\d{2}-\d{2}'
    operators:
      - type: regex_parser
        regex: '^\[(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\] (?P<severity>\w+) (?P<message>.*)'
        parse_from: body
        parse_to: attributes
        timestamp:
          parse_from: attributes.timestamp
          layout: '%Y-%m-%d %H:%M:%S,%L'
        severity:
          parse_from: attributes.severity

      - type: move
        from: attributes.message
        to: body

      - type: add
        field: attributes.source
        value: "kafka"

      - type: add
        field: resource["service.name"]
        value: "kafka-production"

service:
  pipelines:
    logs/kafka:
      receivers: [filelog/kafka]
      processors:
        - memory_limiter
        - transform
        - batch
      exporters:
        - clickhouse
```

Note

- You only define new receivers and pipelines in the custom config. The processors (`memory_limiter`, `transform`, `batch`) and exporters (`clickhouse`) are already defined in the base ClickStack configuration — you just reference them by name.
- The `multiline` configuration ensures stack traces are captured as a single log entry.
- This configuration uses `start_at: beginning` to read all existing logs when the collector starts. For production deployments, change to `start_at: end` to avoid re-ingesting logs on collector restarts.

Configure ClickStack to load custom configuration

#### Configure ClickStack to load custom configuration[​](#load-custom "Direct link to Configure ClickStack to load custom configuration")

To enable custom collector configuration in your existing ClickStack deployment, you must:

1. Mount the custom config file at `/etc/otelcol-contrib/custom.config.yaml`
2. Set the environment variable `CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml`
3. Mount your Kafka log directory so the collector can read them

- Docker Compose- Docker Run (All-in-One Image)

Update your ClickStack deployment configuration:

```
services:
  clickstack:
    # ... existing configuration ...
    environment:
      - CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml
      # ... other environment variables ...
    volumes:
      - ./kafka-logs-monitoring.yaml:/etc/otelcol-contrib/custom.config.yaml:ro
      - /var/log/kafka:/var/log/kafka:ro
      # ... other volumes ...
```

If you're using the all-in-one image with docker, run:

```
docker run --name clickstack \
  -p 8080:8080 -p 4317:4317 -p 4318:4318 \
  -e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
  -v "$(pwd)/kafka-logs-monitoring.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
  -v /var/log/kafka:/var/log/kafka:ro \
  clickhouse/clickstack-all-in-one:latest
```

Note

Ensure the ClickStack collector has appropriate permissions to read the Kafka log files. In production, use read-only mounts (`:ro`) and follow the principle of least privilege.

Verify Logs in HyperDX

#### Verify Logs in HyperDX[​](#verifying-logs "Direct link to Verify Logs in HyperDX")

Once configured, log into HyperDX and verify that logs are flowing:

## Demo dataset[​](#demo-dataset "Direct link to Demo dataset")

Test the Kafka logs integration with a pre-generated sample dataset before configuring your production systems.

Download the sample dataset

#### Download the sample dataset[​](#download-sample "Direct link to Download the sample dataset")

Download the sample log file:

```
curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/kafka/server.log
```

Create test collector configuration

#### Create test collector configuration[​](#test-config "Direct link to Create test collector configuration")

Create a file named `kafka-logs-demo.yaml` with the following configuration:

```
cat > kafka-logs-demo.yaml << 'EOF'
receivers:
  filelog/kafka:
    include:
      - /tmp/kafka-demo/server.log
    start_at: beginning
    multiline:
      line_start_pattern: '^\[\d{4}-\d{2}-\d{2}'
    operators:
      - type: regex_parser
        regex: '^\[(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\] (?P<severity>\w+) (?P<message>.*)'
        parse_from: body
        parse_to: attributes
        timestamp:
          parse_from: attributes.timestamp
          layout: '%Y-%m-%d %H:%M:%S,%L'
        severity:
          parse_from: attributes.severity

      - type: move
        from: attributes.message
        to: body

      - type: add
        field: attributes.source
        value: "kafka-demo"

      - type: add
        field: resource["service.name"]
        value: "kafka-demo"

service:
  pipelines:
    logs/kafka-demo:
      receivers: [filelog/kafka]
      processors:
        - memory_limiter
        - transform
        - batch
      exporters:
        - clickhouse
EOF
```

Run ClickStack with demo configuration