Monitoring AWS CloudWatch Logs with ClickStack

Look for these key attributes in the logs:

- `ResourceAttributes['aws.region']`: Your AWS region (e.g., "us-east-1")
- `ResourceAttributes['cloudwatch.log.group.name']`: The CloudWatch log group name
- `ResourceAttributes['cloudwatch.log.stream']`: The log stream name
- `Body`: The actual log message content

## Demo dataset[​](#demo-dataset "Direct link to Demo dataset")

For users who want to test the CloudWatch logs integration before configuring their production AWS environment, we provide a sample dataset with pre-generated logs showing realistic patterns from multiple AWS services.

Download the sample dataset

#### Download the sample dataset[​](#download-sample "Direct link to Download the sample dataset")

```
curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/aws/cloudwatch/cloudwatch-logs.jsonl
```

The dataset includes 24 hours of CloudWatch logs from multiple services:

- **Lambda functions**: Payment processing, order management, authentication
- **ECS services**: API gateway with rate limiting and timeouts
- **Background jobs**: Batch processing with retry patterns

Start ClickStack

#### Start ClickStack[​](#start-clickstack "Direct link to Start ClickStack")

If you don't already have ClickStack running:

```
docker run -d --name clickstack \
  -p 8080:8080 -p 4317:4317 -p 4318:4318 \
  clickhouse/clickstack-all-in-one:latest
```

Wait a few moments for ClickStack to fully start up.

Import the demo dataset

#### Import the demo dataset[​](#import-demo-data "Direct link to Import the demo dataset")

```
docker exec -i clickstack clickhouse-client --query="
  INSERT INTO default.otel_logs FORMAT JSONEachRow
" < cloudwatch-logs.jsonl
```

This imports the logs directly into ClickStack's logs table.

Verify the demo data

#### Verify the demo data[​](#verify-demo-logs "Direct link to Verify the demo data")

Once imported:

1. Open HyperDX at <http://localhost:8080> and log in (create an account if needed)
2. Navigate to the **Logs** view
3. Set the time range to **2025-12-07 00:00:00 - 2025-12-08 00:00:00 (UTC)**
4. Search for `cloudwatch-demo` or filter by `LogAttributes['source'] = 'cloudwatch-demo'`

You should see logs from multiple CloudWatch log groups.

Timezone Display

HyperDX displays timestamps in your browser's local timezone. The demo data spans **2025-12-07 00:00:00 - 2025-12-08 00:00:00 (UTC)**. Set your time range to **2025-12-06 00:00:00 - 2025-12-09 00:00:00** to ensure you see the demo logs regardless of your location. Once you see the logs, you can narrow the range to a 24-hour period for clearer visualizations.

## Dashboards and visualization[​](#dashboards "Direct link to Dashboards and visualization")

To help you monitor CloudWatch logs with ClickStack, we provide a pre-built dashboard with essential visualizations.

Download the dashboard configuration

#### [Download](/docs/examples/cloudwatch-logs-dashboard.json) the dashboard configuration[​](#download "Direct link to download")

Import the dashboard

#### Import the dashboard[​](#import-dashboard "Direct link to Import the dashboard")

1. Open HyperDX and navigate to the Dashboards section
2. Click **Import Dashboard** in the upper right corner under the ellipses

3. Upload the `cloudwatch-logs-dashboard.json` file and click **Finish Import**

View the dashboard

#### View the dashboard[​](#created-dashboard "Direct link to View the dashboard")

The dashboard will be created with all visualizations pre-configured:

Note

For the demo dataset, set the time range to **2025-12-07 00:00:00 - 2025-12-08 00:00:00 (UTC)** (adjust based on your local timezone). The imported dashboard won't have a time range specified by default.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### No logs appearing in HyperDX[​](#no-logs "Direct link to No logs appearing in HyperDX")

**Verify AWS credentials are configured:**

```
aws sts get-caller-identity
```

If this fails, your credentials are invalid or expired.

**Check IAM permissions:**
Ensure your AWS credentials have the required `logs:DescribeLogGroups` and `logs:FilterLogEvents` permissions.

**Check collector logs for errors:**

```
# If using Docker directly, logs appear in stdout
# If using Docker Compose:
docker compose logs otel-collector
```

Common errors:

- `The security token included in the request is invalid`: Credentials are invalid or expired. For temporary credentials (SSO), ensure `AWS_SESSION_TOKEN` is set.
- `operation error CloudWatch Logs: FilterLogEvents, AccessDeniedException`: IAM permissions are insufficient
- `failed to refresh cached credentials, no EC2 IMDS role found`: AWS credentials environment variables aren't set
- `connection refused`: ClickStack endpoint is unreachable

**Verify CloudWatch log groups exist and have recent logs:**

```
# List your log groups
aws logs describe-log-groups --region us-east-1

# Check if a specific log group has recent logs (last hour)
aws logs filter-log-events \
  --log-group-name /aws/lambda/my-function \
  --region us-east-1 \
  --start-time $(date -u -v-1H +%s)000 \
  --max-items 5
```

### Only seeing old logs or missing recent logs[​](#missing-recent "Direct link to Only seeing old logs or missing recent logs")

**The CloudWatch receiver starts from "now" by default:**

When the collector first starts, it creates a checkpoint at the current time and only fetches logs after that point. Historical logs aren't retrieved.

**To collect recent historical logs:**

Stop and remove the collector's checkpoint, then restart:

```
# Stop the collector
docker stop <container-id>

# Restart fresh (checkpoints are stored in container, so removing it resets)
docker run --rm ...
```

The receiver will create a new checkpoint and fetch logs from the current time forward.

### Invalid security token / credentials expired[​](#expired-credentials "Direct link to Invalid security token / credentials expired")

If using temporary credentials (AWS SSO, assumed role), they expire after a period of time.

**Re-export fresh credentials:**

```
# For SSO users:
aws sso login --profile YOUR_PROFILE_NAME
eval $(aws configure export-credentials --profile YOUR_PROFILE_NAME --format env)

# For IAM users:
export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"

# Restart the collector
docker restart <container-id>
```

### High latency or missing recent logs[​](#latency "Direct link to High latency or missing recent logs")

**Reduce poll interval:**
The default `poll_interval` is 1 minute. For near-real-time logs, reduce it:

```
logs:
  poll_interval: 30s  # Poll every 30 seconds
```

**Note:** Lower poll intervals increase AWS API calls and may incur higher CloudWatch API costs.

### Collector using too much memory[​](#memory "Direct link to Collector using too much memory")

**Reduce batch size or increase timeout:**

```
processors:
  batch:
    timeout: 5s
    send_batch_size: 100
```

**Limit autodiscovery:**

```
groups:
  autodiscover:
    limit: 50  # Reduce from 100 to 50
```

## Next steps[​](#next-steps "Direct link to Next steps")

- Set up [alerts](/docs/use-cases/observability/clickstack/alerts) for critical events (connection failures, error spikes)
- Reduce CloudWatch costs by adjusting retention periods or archiving to S3, now that you have logs in ClickStack
- Filter noisy log groups by removing them from the collector configuration to reduce ingestion volume

## Going to production[​](#going-to-production "Direct link to Going to production")

This guide demonstrates running the OpenTelemetry Collector locally with Docker Compose for testing. For production deployments, run the collector on infrastructure with AWS access (EC2 with IAM roles, EKS with IRSA, or ECS with task roles) to eliminate the need for managing access keys. Deploy collectors in the same AWS region as your CloudWatch log groups to reduce latency and costs.

See [Ingesting with OpenTelemetry](/docs/use-cases/observability/clickstack/ingesting-data/opentelemetry) for production deployment patterns and collector configuration examples.