Monitoring Systemd Logs with ClickStack

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/ingesting-data/integration-examples/systemd.md)

TL;DR

Collect and visualize systemd journal logs in ClickStack using the OpenTelemetry Collector's journald receiver. Includes a demo dataset and pre-built dashboard.

## Integration with existing systems[​](#existing-systems "Direct link to Integration with existing systems")

Monitor your existing Linux system's journald logs by running the OpenTelemetry Collector with the journald receiver to collect system logs and send them to ClickStack via OTLP.

If you want to test this integration first without modifying your existing setup, skip to the [demo dataset section](#demo-dataset).

##### Prerequisites[​](#prerequisites "Direct link to Prerequisites")

- ClickStack instance running
- Linux system with systemd (Ubuntu 16.04+, CentOS 7+, Debian 8+)
- Docker or Docker Compose installed on the monitored system

Get ClickStack API key

#### Get ClickStack API key[​](#get-api-key "Direct link to Get ClickStack API key")

The OpenTelemetry Collector sends data to ClickStack's OTLP endpoint, which requires authentication.

1. Open HyperDX at your ClickStack URL (e.g., <http://localhost:8080>)
2. Create an account or log in if needed
3. Navigate to **Team Settings → API Keys**
4. Copy your **Ingestion API Key**

5. Set it as an environment variable:

```
export CLICKSTACK_API_KEY=your-api-key-here
```

Verify systemd journal is running

#### Verify systemd journal is running[​](#verify-systemd "Direct link to Verify systemd journal is running")

Ensure your system is using systemd and has journal logs:

```
# Check systemd version
systemctl --version

# View recent journal entries
journalctl -n 20

# Check journal disk usage
journalctl --disk-usage
```

If journal storage is in memory only, enable persistent storage:

```
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
```

Create OpenTelemetry Collector configuration

#### Create OpenTelemetry Collector configuration[​](#create-otel-config "Direct link to Create OpenTelemetry Collector configuration")

Create a configuration file for the OpenTelemetry Collector:

```
cat > otel-config.yaml << 'EOF'
receivers:
  journald:
    directory: /var/log/journal
    priority: info
    units:
      - sshd
      - nginx
      - docker
      - containerd
      - systemd

processors:
  batch:
    timeout: 10s
    send_batch_size: 10000
  
  resource:
    attributes:
      - key: service.name
        value: systemd-logs
        action: insert
      - key: host.name
        from_attribute: _HOSTNAME
        action: upsert
  
  attributes:
    actions:
      - key: unit
        from_attribute: _SYSTEMD_UNIT
        action: upsert
      - key: priority
        from_attribute: PRIORITY
        action: upsert

exporters:
  otlphttp:
    endpoint: ${CLICKSTACK_ENDPOINT}
    headers:
      authorization: ${CLICKSTACK_API_KEY}

service:
  pipelines:
    logs:
      receivers: [journald]
      processors: [resource, attributes, batch]
      exporters: [otlphttp]
EOF
```

Deploy with Docker Compose

#### Deploy with Docker Compose[​](#deploy-docker-compose "Direct link to Deploy with Docker Compose")

Note

The `journald` receiver requires the `journalctl` binary to read journal files. The official `otel/opentelemetry-collector-contrib` image doesn't include `journalctl` by default.

For containerized deployments, you can either install the collector directly on the host or build a custom image with systemd utilities. See the [troubleshooting section](#journalctl-not-found) for details.

This example shows deploying the OTel Collector alongside ClickStack:

```
services:
  clickstack:
    image: clickhouse/clickstack-all-in-one:latest
    ports:
      - "8080:8080"
      - "4317:4317"
      - "4318:4318"
    networks:
      - monitoring
  
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.115.1
    depends_on:
      - clickstack
    environment:
      - CLICKSTACK_API_KEY=${CLICKSTACK_API_KEY}
      - CLICKSTACK_ENDPOINT=http://clickstack:4318
    volumes:
      - ./otel-config.yaml:/etc/otelcol/config.yaml:ro
      - /var/log/journal:/var/log/journal:ro
      - /run/log/journal:/run/log/journal:ro
      - /etc/machine-id:/etc/machine-id:ro
    command: ["--config=/etc/otelcol/config.yaml"]
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge
```

Start the services:

```
docker compose up -d
```

Verify logs in HyperDX

#### Verify logs in HyperDX[​](#verifying-logs "Direct link to Verify logs in HyperDX")

Once configured, log into HyperDX and verify logs are flowing:

1. Navigate to the Search view
2. Set source to Logs
3. Filter by `service.name:systemd-logs`
4. You should see structured log entries with fields like `unit`, `priority`, `MESSAGE`, `_HOSTNAME`

## Demo dataset[​](#demo-dataset "Direct link to Demo dataset")

For users who want to test the systemd logs integration before configuring their production systems, we provide a sample dataset of pre-generated systemd logs with realistic patterns.

Download the sample dataset

#### Download the sample dataset[​](#download-sample "Direct link to Download the sample dataset")

Download the sample log file:

```
curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/systemd/systemd-demo.log
```

Create demo collector configuration

#### Create demo collector configuration[​](#demo-config "Direct link to Create demo collector configuration")

Create a configuration file for the demo:

```
cat > systemd-demo.yaml << 'EOF'
receivers:
  filelog:
    include:
      - /tmp/systemd-demo/systemd-demo.log
    start_at: beginning
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\S+) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
        parse_from: body
        parse_to: attributes
      - type: time_parser
        parse_from: attributes.timestamp
        layout: '%Y-%m-%dT%H:%M:%S%z'
      - type: add
        field: attributes.source
        value: "systemd-demo"

service:
  pipelines:
    logs/systemd-demo:
      receivers: [filelog]
      processors:
        - memory_limiter
        - transform
        - batch
      exporters:
        - clickhouse
EOF
```

Run ClickStack with demo data

#### Run ClickStack with demo data[​](#run-demo "Direct link to Run ClickStack with demo data")

Start ClickStack with the demo logs:

```
docker run -d --name clickstack-demo \
  -p 8080:8080 -p 4317:4317 -p 4318:4318 \
  -e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
  -v "$(pwd)/systemd-demo.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
  -v "$(pwd)/systemd-demo.log:/tmp/systemd-demo/systemd-demo.log:ro" \
  clickhouse/clickstack-all-in-one:latest
```

Note

The demo uses the `filelog` receiver with text logs instead of `journald` to avoid requiring `journalctl` in the container.

Verify logs in HyperDX

#### Verify logs in HyperDX[​](#verify-demo-logs "Direct link to Verify logs in HyperDX")

Once ClickStack is running:

1. Open [HyperDX](http://localhost:8080/) and log in to your account
2. Navigate to the Search view and set the source to `Logs`
3. Set the time range to **2025-11-14 00:00:00 - 2025-11-17 00:00:00**

Timezone display

HyperDX displays timestamps in your browser's local timezone. The demo data spans **2025-11-15 00:00:00 - 2025-11-16 00:00:00 (UTC)**. The wide time range ensures you'll see the demo logs regardless of your location.

## Dashboards and visualization[​](#dashboards "Direct link to Dashboards and visualization")

To help you get started monitoring systemd logs with ClickStack, we provide essential visualizations for systemd journal data.

Download the dashboard configuration

#### [Download](/docs/examples/systemd-logs-dashboard.json) the dashboard configuration[​](#download "Direct link to download")

Import the pre-built dashboard

#### Import the pre-built dashboard[​](#import-dashboard "Direct link to Import the pre-built dashboard")