Monitoring EC2 Host Logs with ClickStack

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/ingesting-data/integration-examples/host-logs/ec2-host-logs.md)

TL;DR

Collect and visualize EC2 system logs in ClickStack using the OpenTelemetry Collector with automatic EC2 metadata enrichment (instance ID, region, AZ, instance type). Includes a demo dataset and pre-built dashboard.

## Integration with existing EC2 instance[​](#existing-ec2 "Direct link to Integration with existing EC2 instance")

This section covers installing OpenTelemetry Collector on your EC2 instances to collect system logs and send them to ClickStack with automatic EC2 metadata enrichment. This distributed architecture is production-ready and scales to multiple instances.

Running ClickStack on the same EC2 instance?

If ClickStack is running on the same EC2 instance whose logs you want to monitor, you can use the all-in-one approach similar to the [Generic Host Logs guide](/docs/use-cases/observability/clickstack/integrations/host-logs). Mount `/var/log` into the ClickStack container and add the `resourcedetection` processor to your custom config to automatically capture EC2 metadata. This guide focuses on the more common distributed architecture for production deployments.

If you would like to test the EC2 host logs integration before configuring your production instance, you can test with our preconfigured setup and sample data in the ["Demo dataset"](/docs/use-cases/observability/clickstack/integrations/host-logs/ec2#demo-dataset) section.

##### Prerequisites[​](#prerequisites "Direct link to Prerequisites")

- ClickStack instance running (can be on-premises, cloud, or local)
- EC2 instance running (Ubuntu, Amazon Linux, or other Linux distribution)
- Network connectivity from EC2 instance to ClickStack's OTLP endpoint (port 4318 for HTTP or 4317 for gRPC)
- EC2 instance metadata service accessible (enabled by default)

Verify EC2 metadata is accessible

#### Verify EC2 metadata is accessible[​](#verify-metadata "Direct link to Verify EC2 metadata is accessible")

From your EC2 instance, verify the metadata service is accessible:

```
# Get metadata token (IMDSv2)
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Verify instance metadata
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/region
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-type
```

You should see your instance ID, region, and instance type. If these commands fail, verify:

- The instance metadata service is enabled
- IMDSv2 isn't blocked by security groups or network ACLs
- You're running these commands from the EC2 instance itself

Note

EC2 metadata is available at `http://169.254.169.254` from within the instance. The OpenTelemetry `resourcedetection` processor uses this endpoint to automatically enrich logs with cloud context.

Verify syslog files exist

#### Verify syslog files exist[​](#verify-syslog "Direct link to Verify syslog files exist")

Verify that your EC2 instance is writing syslog files:

```
# Ubuntu instances
ls -la /var/log/syslog

# Amazon Linux / RHEL instances
ls -la /var/log/messages

# View recent entries
tail -20 /var/log/syslog
# or
tail -20 /var/log/messages
```

Install OpenTelemetry Collector

#### Install OpenTelemetry Collector[​](#install-collector "Direct link to Install OpenTelemetry Collector")

Install the OpenTelemetry Collector Contrib distribution on your EC2 instance:

```
# Download the latest release
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.114.0/otelcol-contrib_0.114.0_linux_amd64.tar.gz

# Extract and install
tar -xvf otelcol-contrib_0.114.0_linux_amd64.tar.gz
sudo mv otelcol-contrib /usr/local/bin/

# Verify installation
otelcol-contrib --version
```

Create collector configuration

#### Create collector configuration[​](#create-config "Direct link to Create collector configuration")

Create a configuration file for the OpenTelemetry Collector at `/etc/otelcol-contrib/config.yaml`:

```
sudo mkdir -p /etc/otelcol-contrib
```

Choose the configuration based on your Linux distribution:

- Modern Linux (Ubuntu 24.04+)- Legacy Linux (Amazon Linux 2, RHEL, older Ubuntu)

```
sudo tee /etc/otelcol-contrib/config.yaml > /dev/null << 'EOF'
receivers:
  filelog/syslog:
    include:
      - /var/log/syslog
      - /var/log/**/*.log
    start_at: end
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\S+) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
        parse_from: body
        parse_to: attributes
      
      - type: time_parser
        parse_from: attributes.timestamp
        layout_type: gotime
        layout: '2006-01-02T15:04:05.999999-07:00'
      
      - type: add
        field: attributes.source
        value: "ec2-host-logs"

processors:
  resourcedetection:
    detectors: [ec2, system]
    timeout: 5s
    override: false
    ec2:
      tags:
        - ^Name
        - ^Environment
        - ^Team
  
  batch:
    timeout: 10s
    send_batch_size: 10000

exporters:
  otlphttp:
    endpoint: "http://YOUR_CLICKSTACK_HOST:4318"
    headers:
      authorization: "${env:CLICKSTACK_API_KEY}"

service:
  pipelines:
    logs:
      receivers: [filelog/syslog]
      processors: [resourcedetection, batch]
      exporters: [otlphttp]
EOF
```

```
sudo tee /etc/otelcol-contrib/config.yaml > /dev/null << 'EOF'
receivers:
  filelog/syslog:
    include:
      - /var/log/messages
      - /var/log/**/*.log
    start_at: end
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\w+ \d+ \d{2}:\d{2}:\d{2}) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
        parse_from: body
        parse_to: attributes
      
      - type: time_parser
        parse_from: attributes.timestamp
        layout: '%b %d %H:%M:%S'
      
      - type: add
        field: attributes.source
        value: "ec2-host-logs"

processors:
  resourcedetection:
    detectors: [ec2, system]
    timeout: 5s
    override: false
    ec2:
      tags:
        - ^Name
        - ^Environment
        - ^Team
  
  batch:
    timeout: 10s
    send_batch_size: 10000

exporters:
  otlphttp:
    endpoint: "http://YOUR_CLICKSTACK_HOST:4318"
    headers:
      authorization: "${env:CLICKSTACK_API_KEY}"

service:
  pipelines:
    logs:
      receivers: [filelog/syslog]
      processors: [resourcedetection, batch]
      exporters: [otlphttp]
EOF
```

  

**Replace the following in the configuration:**

- `YOUR_CLICKSTACK_HOST`: The hostname or IP address where ClickStack is running
- For local testing, you can use an SSH tunnel (see the [Troubleshooting section](#troubleshooting))

This configuration:

- Reads system log files from standard locations (`/var/log/syslog` for Ubuntu, `/var/log/messages` for Amazon Linux/RHEL)
- Parses syslog format to extract structured fields (timestamp, hostname, unit/service, PID, message)
- **Automatically detects and adds EC2 metadata** using the `resourcedetection` processor
- Optionally includes EC2 tags (Name, Environment, Team) if present
- Sends logs to ClickStack via OTLP HTTP

EC2 Metadata Enrichment

The `resourcedetection` processor automatically adds these attributes to every log:

- `cloud.provider`: "aws"
- `cloud.platform`: "aws\_ec2"
- `cloud.region`: AWS region (e.g., "us-east-1")
- `cloud.availability_zone`: AZ (e.g., "us-east-1a")
- `cloud.account.id`: AWS account ID
- `host.id`: EC2 instance ID (e.g., "i-1234567890abcdef0")
- `host.type`: Instance type (e.g., "t3.medium")
- `host.name`: Instance hostname

Set ClickStack API key

#### Set ClickStack API key[​](#set-api-key "Direct link to Set ClickStack API key")

Export your ClickStack API key as an environment variable:

```
export CLICKSTACK_API_KEY="your-api-key-here"
```

To make this persistent across reboots, add it to your shell profile: