Monitoring Systemd Logs with ClickStack

Timezone display

HyperDX displays timestamps in your browser's local timezone. The demo data spans **2025-11-15 00:00:00 - 2025-11-16 00:00:00 (UTC)**. The wide time range ensures you'll see the demo logs regardless of your location.

## Dashboards and visualization[​](#dashboards "Direct link to Dashboards and visualization")

To help you get started monitoring systemd logs with ClickStack, we provide essential visualizations for systemd journal data.

Download the dashboard configuration

#### [Download](/docs/examples/systemd-logs-dashboard.json) the dashboard configuration[​](#download "Direct link to download")

Import the pre-built dashboard

#### Import the pre-built dashboard[​](#import-dashboard "Direct link to Import the pre-built dashboard")

1. Open HyperDX and navigate to the Dashboards section
2. Click **Import Dashboard** in the upper right corner under the ellipses

3. Upload the `systemd-logs-dashboard.json` file and click **Finish Import**

View the dashboard

#### View the dashboard[​](#created-dashboard "Direct link to View the dashboard")

The dashboard includes visualizations for:

- Log volume over time
- Top systemd units by log count
- SSH authentication events
- Service failures
- Error rates

Note

For the demo dataset, set the time range to **2025-11-15 00:00:00 - 2025-11-16 00:00:00 (UTC)** (adjust based on your local timezone).

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### No logs appearing in HyperDX[​](#no-logs "Direct link to No logs appearing in HyperDX")

Check if logs are reaching ClickHouse:

```
docker exec clickstack clickhouse-client --query "
SELECT COUNT(*) as log_count
FROM otel_logs
WHERE ServiceName = 'systemd-logs'
"
```

If no results, check the collector logs:

```
docker logs otel-collector | grep -i "error\|journald" | tail -20
```

### journalctl not found error[​](#journalctl-not-found "Direct link to journalctl not found error")

If you see `exec: "journalctl": executable file not found in $PATH`:

The `otel/opentelemetry-collector-contrib` image doesn't include `journalctl`. You can either:

1. **Install the collector on the host**:

```
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.115.0/otelcol-contrib_0.115.0_linux_amd64.tar.gz
tar -xzf otelcol-contrib_0.115.0_linux_amd64.tar.gz
sudo mv otelcol-contrib /usr/local/bin/
otelcol-contrib --config=otel-config.yaml
```

2. **Use the text export approach** (like the demo) with the `filelog` receiver reading journald exports

## Next steps[​](#next-steps "Direct link to Next steps")

- Set up [alerts](/docs/use-cases/observability/clickstack/alerts) for critical system events (service failures, authentication failures, OOM kills)
- Create additional [dashboards](/docs/use-cases/observability/clickstack/dashboards) for specific use cases (SSH security monitoring, service health)
- Filter by specific systemd units to reduce noise and focus on services that matter

## Going to production[​](#going-to-production "Direct link to Going to production")

This guide uses a separate OpenTelemetry Collector to read systemd logs and send them to ClickStack's OTLP endpoint, which is the recommended production pattern.

For production environments with multiple hosts, consider:

- Deploying the collector as a DaemonSet in Kubernetes
- Running the collector as a systemd service on each host
- Using the OpenTelemetry Operator for automated deployment

See [Ingesting with OpenTelemetry](/docs/use-cases/observability/clickstack/ingesting-data/opentelemetry) for production deployment patterns.

- [Integration with existing systems](#existing-systems)- [Demo dataset](#demo-dataset)- [Dashboards and visualization](#dashboards)- [Troubleshooting](#troubleshooting)
        - [No logs appearing in HyperDX](#no-logs)- [journalctl not found error](#journalctl-not-found)- [Next steps](#next-steps)- [Going to production](#going-to-production)

Was this page helpful?