Monitoring EC2 Host Logs with ClickStack

Note

For the demo dataset, set the time range to **2025-11-11 00:00:00 - 2025-11-12 00:00:00 (UTC)** (adjust based on your local timezone). The imported dashboard won't have a time range specified by default.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### EC2 metadata not appearing in logs[​](#no-metadata "Direct link to EC2 metadata not appearing in logs")

**Verify the EC2 metadata service is accessible:**

```
# Get metadata token
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Test metadata endpoint
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id
```

If this fails, verify:

- The instance metadata service is enabled
- IMDSv2 isn't blocked by security groups
- You're running the collector on the EC2 instance itself

**Check collector logs for metadata errors:**

```
# If running as systemd service
sudo journalctl -u otelcol-contrib -f | grep -i "ec2\|metadata\|resourcedetection"

# If running in foreground, check stdout
```

### No logs appearing in HyperDX[​](#no-logs "Direct link to No logs appearing in HyperDX")

**Verify syslog files exist and are being written:**

```
ls -la /var/log/syslog /var/log/messages
tail -f /var/log/syslog
```

**Check collector can read the log files:**

```
cat /var/log/syslog | head -20
```

**Verify network connectivity to ClickStack:**

```
# Test OTLP endpoint
curl -v http://YOUR_CLICKSTACK_HOST:4318/v1/logs

# Should get a response (even if error, means endpoint is reachable)
```

**Check collector logs for errors:**

```
# If running in foreground
# Look for error messages in stdout

# If running as systemd service
sudo journalctl -u otelcol-contrib -f | grep -i "error\|failed"
```

### Logs parsing incorrectly[​](#logs-not-parsing "Direct link to Logs parsing incorrectly")

**Verify your syslog format:**

For Ubuntu 24.04+:

```
# Should show ISO8601 format: 2025-11-17T20:55:44.826796+00:00
tail -5 /var/log/syslog
```

For Amazon Linux 2 / Ubuntu 20.04:

```
# Should show traditional format: Nov 17 14:16:16
tail -5 /var/log/messages
```

If your format doesn't match, use the appropriate configuration tab in the [Create collector configuration](#create-config) section based on your distribution.

### Collector not starting as systemd service[​](#systemd-issues "Direct link to Collector not starting as systemd service")

**Check service status:**

```
sudo systemctl status otelcol-contrib
```

**View detailed logs:**

```
sudo journalctl -u otelcol-contrib -n 50
```

**Common issues:**

- API key not set correctly in environment
- Config file syntax errors
- Permission issues reading log files

## Next steps[​](#next-steps "Direct link to Next steps")

- Set up [alerts](/docs/use-cases/observability/clickstack/alerts) for critical system events (service failures, authentication failures, disk warnings)
- Filter by EC2 metadata attributes (region, instance type, instance ID) to monitor specific resources
- Correlate EC2 host logs with application logs for comprehensive troubleshooting
- Create custom dashboards for security monitoring (SSH attempts, sudo usage, firewall blocks)

## Going to production[​](#going-to-production "Direct link to Going to production")

This guide installs the OpenTelemetry Collector directly on EC2 instances, which is the recommended production pattern for host-level monitoring. For managing collectors across many instances, consider using configuration management tools (Ansible, Chef, Puppet) or the OpenTelemetry Operator in Kubernetes environments. See [Sending OpenTelemetry data](/docs/use-cases/observability/clickstack/ingesting-data/opentelemetry) for production configuration.

- [Integration with existing EC2 instance](#existing-ec2)- [Demo dataset](#demo-dataset)- [Dashboards and visualization](#dashboards)- [Troubleshooting](#troubleshooting)
        - [EC2 metadata not appearing in logs](#no-metadata)- [No logs appearing in HyperDX](#no-logs)- [Logs parsing incorrectly](#logs-not-parsing)- [Collector not starting as systemd service](#systemd-issues)- [Next steps](#next-steps)- [Going to production](#going-to-production)

Was this page helpful?