Monitoring Host Logs with ClickStack

1. Navigate to the search view
2. Set source to Logs
3. Filter by `source:host-logs` to see host-specific logs
4. You should see structured log entries with fields like `unit`, `hostname`, `pid`, `message`, etc.

## Demo dataset[​](#demo-dataset "Direct link to Demo dataset")

For users who want to test the host logs integration before configuring their production systems, we provide a sample dataset of pre-generated system logs with realistic patterns.

Download the sample dataset

#### Download the sample dataset[​](#download-sample "Direct link to Download the sample dataset")

Download the sample log file:

```
curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/host-logs/journal.log
```

The dataset includes:

- System boot sequence
- SSH login activity (successful and failed attempts)
- Security incident (brute force attack with fail2ban response)
- Scheduled maintenance (cron jobs, anacron)
- Service restarts (rsyslog)
- Kernel messages and firewall activity
- Mix of normal operations and notable events

Create test collector configuration

#### Create test collector configuration[​](#test-config "Direct link to Create test collector configuration")

Create a file named `host-logs-demo.yaml` with the following configuration:

```
cat > host-logs-demo.yaml << 'EOF'
receivers:
  filelog/journal:
    include:
      - /tmp/host-demo/journal.log
    start_at: beginning
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\S+) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
        parse_from: body
        parse_to: attributes
      
      - type: time_parser
        parse_from: attributes.timestamp
        layout: '%Y-%m-%dT%H:%M:%S%z'
      
      - type: add
        field: attributes.source
        value: "host-demo"
      
      - type: add
        field: resource["service.name"]
        value: "host-demo"

service:
  pipelines:
    logs/host-demo:
      receivers: [filelog/journal]
      processors:
        - memory_limiter
        - transform
        - batch
      exporters:
        - clickhouse
EOF
```

Run ClickStack with demo configuration

#### Run ClickStack with demo configuration[​](#run-demo "Direct link to Run ClickStack with demo configuration")

Run ClickStack with the demo logs and configuration:

```
docker run --name clickstack-demo \
  -p 8080:8080 -p 4317:4317 -p 4318:4318 \
  -e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
  -v "$(pwd)/host-logs-demo.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
  -v "$(pwd)/journal.log:/tmp/host-demo/journal.log:ro" \
  clickhouse/clickstack-all-in-one:latest
```

Note

**This mounts the log file directly into the container. This is done for testing purposes with static demo data.**

Verify logs in HyperDX

#### Verify logs in HyperDX[​](#verify-demo-logs "Direct link to Verify logs in HyperDX")

Once ClickStack is running:

1. Open [HyperDX](http://localhost:8080/) and log in to your account (you may need to create an account first)
2. Navigate to the Search view and set the source to `Logs`
3. Set the time range to **2025-11-10 00:00:00 - 2025-11-13 00:00:00**

Timezone Display

HyperDX displays timestamps in your browser's local timezone. The demo data spans **2025-11-11 00:00:00 - 2025-11-12 00:00:00 (UTC)**. The wide time range ensures you'll see the demo logs regardless of your location. Once you see the logs, you can narrow the range to a 24-hour period for clearer visualizations.

## Dashboards and visualization[​](#dashboards "Direct link to Dashboards and visualization")

To help you get started monitoring host logs with ClickStack, we provide essential visualizations for system logs.

Download the dashboard configuration

#### [Download](/docs/examples/host-logs-dashboard.json) the dashboard configuration[​](#download "Direct link to download")

Import the pre-built dashboard

#### Import the pre-built dashboard[​](#import-dashboard "Direct link to Import the pre-built dashboard")

1. Open HyperDX and navigate to the Dashboards section
2. Click **Import Dashboard** in the upper right corner under the ellipses

3. Upload the `host-logs-dashboard.json` file and click **Finish Import**

View the dashboard

#### View the dashboard[​](#created-dashboard "Direct link to View the dashboard")

The dashboard will be created with all visualizations pre-configured:

Key visualizations include:

- Log volume over time by severity
- Top systemd units generating logs
- SSH login activity (successful vs failed)
- Firewall activity (blocked vs allowed)
- Security events (failed logins, bans, blocks)
- Service restart activity

Note

For the demo dataset, set the time range to **2025-11-11 00:00:00 - 2025-11-12 00:00:00 (UTC)** (adjust based on your local timezone). The imported dashboard won't have a time range specified by default.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### Custom config not loading[​](#troubleshooting-not-loading "Direct link to Custom config not loading")

Verify the environment variable is set:

```
docker exec <container-name> printenv CUSTOM_OTELCOL_CONFIG_FILE
```

Check the custom config file is mounted and readable:

```
docker exec <container-name> cat /etc/otelcol-contrib/custom.config.yaml | head -10
```

### No logs appearing in HyperDX[​](#no-logs "Direct link to No logs appearing in HyperDX")

**Verify syslog files exist and are being written:**

```
# Check if syslog exists
ls -la /var/log/syslog /var/log/messages

# Verify logs are being written
tail -f /var/log/syslog
```

**Check the collector can read the logs:**

```
docker exec <container> cat /var/log/syslog | head -20
```

**Check the effective config includes your filelog receiver:**

```
docker exec <container> cat /etc/otel/supervisor-data/effective.yaml | grep -A 10 filelog
```

**Check for errors in the collector logs:**

```
docker exec <container> cat /etc/otel/supervisor-data/agent.log | grep -i "filelog\|syslog"
```

**If using the demo dataset, verify the log file is accessible:**

```
docker exec <container> cat /tmp/host-demo/journal.log | wc -l
```

### Logs not parsing correctly[​](#logs-not-parsing "Direct link to Logs not parsing correctly")

**Verify your syslog format matches the configuration you chose:**

For Modern Linux (Ubuntu 24.04+):

```
# Should show ISO8601 format: 2025-11-17T20:55:44.826796+00:00
tail -5 /var/log/syslog
```

For Legacy Linux or macOS:

```
# Should show traditional format: Nov 17 14:16:16
tail -5 /var/log/syslog
# or
tail -5 /var/log/system.log
```

If your format doesn't match, select the appropriate configuration tab in the [Create custom OTel collector configuration](#custom-otel) section.

## Next steps[​](#next-steps "Direct link to Next steps")

- Set up [alerts](/docs/use-cases/observability/clickstack/alerts) for critical system events (service failures, authentication failures, disk warnings)
- Filter by specific units to monitor particular services
- Correlate host logs with application logs for comprehensive troubleshooting
- Create custom dashboards for security monitoring (SSH attempts, sudo usage, firewall blocks)

## Going to production[​](#going-to-production "Direct link to Going to production")

This guide extends ClickStack's built-in OpenTelemetry Collector for quick setup. For production deployments, we recommend running your own OTel Collector and sending data to ClickStack's OTLP endpoint. See [Sending OpenTelemetry data](/docs/use-cases/observability/clickstack/ingesting-data/opentelemetry) for production configuration.

- [Integration with existing hosts](#existing-hosts)- [Demo dataset](#demo-dataset)- [Dashboards and visualization](#dashboards)- [Troubleshooting](#troubleshooting)
        - [Custom config not loading](#troubleshooting-not-loading)- [No logs appearing in HyperDX](#no-logs)- [Logs not parsing correctly](#logs-not-parsing)- [Next steps](#next-steps)- [Going to production](#going-to-production)

Was this page helpful?