Monitoring EC2 Host Logs with ClickStack

EC2 Metadata Enrichment

The `resourcedetection` processor automatically adds these attributes to every log:

- `cloud.provider`: "aws"
- `cloud.platform`: "aws\_ec2"
- `cloud.region`: AWS region (e.g., "us-east-1")
- `cloud.availability_zone`: AZ (e.g., "us-east-1a")
- `cloud.account.id`: AWS account ID
- `host.id`: EC2 instance ID (e.g., "i-1234567890abcdef0")
- `host.type`: Instance type (e.g., "t3.medium")
- `host.name`: Instance hostname

Set ClickStack API key

#### Set ClickStack API key[​](#set-api-key "Direct link to Set ClickStack API key")

Export your ClickStack API key as an environment variable:

```
export CLICKSTACK_API_KEY="your-api-key-here"
```

To make this persistent across reboots, add it to your shell profile:

```
echo 'export CLICKSTACK_API_KEY="your-api-key-here"' >> ~/.bashrc
source ~/.bashrc
```

Run the collector

#### Run the collector[​](#run-collector "Direct link to Run the collector")

Start the OpenTelemetry Collector:

```
CLICKSTACK_API_KEY="your-api-key-here" /usr/local/bin/otelcol-contrib --config /etc/otelcol-contrib/config.yaml
```

For production use

Configure the collector to run as a systemd service so it starts automatically on boot and restarts on failure. See the [OpenTelemetry Collector documentation](https://opentelemetry.io/docs/collector/deployment/) for details.

Verifying Logs in HyperDX

#### Verifying Logs in HyperDX[​](#verifying-logs "Direct link to Verifying Logs in HyperDX")

Once the collector is running, log into HyperDX and verify logs are flowing with EC2 metadata:

1. Navigate to the search view
2. Set source to `Logs`
3. Filter by `source:ec2-host-logs`
4. Click on a log entry to expand it
5. Verify you see EC2 metadata in the resource attributes:
   - `cloud.provider`
   - `cloud.region`
   - `host.id` (instance ID)
   - `host.type` (instance type)
   - `cloud.availability_zone`

## Demo dataset[​](#demo-dataset "Direct link to Demo dataset")

For users who want to test the EC2 host logs integration before configuring their production instances, we provide a sample dataset with simulated EC2 metadata.

Download the sample dataset

#### Download the sample dataset[​](#download-sample "Direct link to Download the sample dataset")

Download the sample log file:

```
curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/host-logs/journal.log
```

The dataset includes:

- System boot sequence
- SSH login activity (successful and failed attempts)
- Security incident (brute force attack with fail2ban response)
- Scheduled maintenance (cron jobs, anacron)
- Service restarts (rsyslog)
- Kernel messages and firewall activity
- Mix of normal operations and notable events

Create test collector configuration

#### Create test collector configuration[​](#test-config "Direct link to Create test collector configuration")

Create a file named `ec2-host-logs-demo.yaml` with the following configuration:

```
cat > ec2-host-logs-demo.yaml << 'EOF'
receivers:
  filelog/journal:
    include:
      - /tmp/host-demo/journal.log
    start_at: beginning
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\S+) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
        parse_from: body
        parse_to: attributes
      
      - type: time_parser
        parse_from: attributes.timestamp
        layout: '%Y-%m-%dT%H:%M:%S%z'
      
      - type: add
        field: attributes.source
        value: "ec2-demo"

processors:
  # Simulate EC2 metadata for demo (no real EC2 instance required)
  resource:
    attributes:
      - key: service.name
        value: "ec2-demo"
        action: insert
      - key: cloud.provider
        value: "aws"
        action: insert
      - key: cloud.platform
        value: "aws_ec2"
        action: insert
      - key: cloud.region
        value: "us-east-1"
        action: insert
      - key: cloud.availability_zone
        value: "us-east-1a"
        action: insert
      - key: host.id
        value: "i-0abc123def456789"
        action: insert
      - key: host.type
        value: "t3.medium"
        action: insert
      - key: host.name
        value: "prod-web-01"
        action: insert

service:
  pipelines:
    logs/ec2-demo:
      receivers: [filelog/journal]
      processors:
        - resource
        - memory_limiter
        - transform
        - batch
      exporters:
        - clickhouse
EOF
```

Note

For demo purposes, we manually add EC2 metadata using the `resource` processor. In production with real EC2 instances, use the `resourcedetection` processor which automatically queries the EC2 metadata API.

Run ClickStack with demo configuration

#### Run ClickStack with demo configuration[​](#run-demo "Direct link to Run ClickStack with demo configuration")

Run ClickStack with the demo logs and configuration:

```
docker run --name clickstack-demo \
  -p 8080:8080 -p 4317:4317 -p 4318:4318 \
  -e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
  -v "$(pwd)/ec2-host-logs-demo.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
  -v "$(pwd)/journal.log:/tmp/host-demo/journal.log:ro" \
  docker.hyperdx.io/hyperdx/hyperdx-all-in-one:latest
```

Verify logs in HyperDX

#### Verify logs in HyperDX[​](#verify-demo-logs "Direct link to Verify logs in HyperDX")

Once the collector is running:

1. Open [HyperDX](http://localhost:8080/) and log in to your account (you may need to create an account first)
2. Navigate to the search view and set the source to `Logs`
3. Set the time range to **2025-11-10 00:00:00 - 2025-11-13 00:00:00**
4. Filter by `source:ec2-demo`
5. Expand a log entry to view EC2 metadata in the resource attributes

Timezone display

HyperDX displays timestamps in your browser's local timezone. The demo data spans **2025-11-11 00:00:00 - 2025-11-12 00:00:00 (UTC)**. The wide time range ensures you'll see the demo logs regardless of your location. Once you see the logs, you can narrow the range to a 24-hour period for clearer visualizations.

You should see logs with simulated EC2 context including:

- Instance ID: `i-0abc123def456789`
- Region: `us-east-1`
- Availability Zone: `us-east-1a`
- Instance Type: `t3.medium`

## Dashboards and visualization[​](#dashboards "Direct link to Dashboards and visualization")

To help you get started monitoring EC2 host logs with ClickStack, we provide essential visualizations with cloud context.

Download the dashboard configuration

#### [Download](/docs/examples/host-logs-dashboard.json) the dashboard configuration[​](#download "Direct link to download")

Import the pre-built dashboard

#### Import the pre-built dashboard[​](#import-dashboard "Direct link to Import the pre-built dashboard")

1. Open HyperDX and navigate to the Dashboards section
2. Click **Import Dashboard** in the upper right corner under the ellipses

3. Upload the `host-logs-dashboard.json` file and click **Finish Import**

View the dashboard

#### View the dashboard[​](#created-dashboard "Direct link to View the dashboard")

The dashboard will be created with all visualizations pre-configured:

You can filter dashboard visualizations by EC2 context:

- `cloud.region:us-east-1` - Show logs from specific region
- `host.type:t3.medium` - Filter by instance type
- `host.id:i-0abc123def456` - Logs from specific instance

Note

For the demo dataset, set the time range to **2025-11-11 00:00:00 - 2025-11-12 00:00:00 (UTC)** (adjust based on your local timezone). The imported dashboard won't have a time range specified by default.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### EC2 metadata not appearing in logs[​](#no-metadata "Direct link to EC2 metadata not appearing in logs")

**Verify the EC2 metadata service is accessible:**

```
# Get metadata token
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Test metadata endpoint
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id
```

If this fails, verify: