Monitoring AWS CloudWatch Logs with ClickStack

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/ingesting-data/integration-examples/cloudwatch.md)

TL;DR

Forward AWS CloudWatch logs to ClickStack using the OpenTelemetry Collector's CloudWatch receiver. Supports named log groups and autodiscovery. Includes a demo dataset and pre-built dashboard.

## Overview[​](#overview "Direct link to Overview")

AWS CloudWatch is a monitoring service for AWS resources and applications. While CloudWatch provides log aggregation, forwarding logs to ClickStack allows you to:

- Analyze logs alongside metrics and traces in a unified platform
- Query logs using ClickHouse's SQL interface
- Reduce costs by archiving or reducing CloudWatch retention

This guide shows you how to forward CloudWatch logs to ClickStack using the OpenTelemetry Collector.

## Integration with existing CloudWatch log groups[​](#existing-cloudwatch "Direct link to Integration with existing CloudWatch log groups")

This section covers configuring the OpenTelemetry Collector to pull logs from your existing CloudWatch log groups and forward them to ClickStack.

If you would like to test the integration before configuring your production setup, you can test with our demo dataset in the [demo dataset section](#demo-dataset).

### Prerequisites[​](#prerequisites "Direct link to Prerequisites")

- ClickStack instance running
- AWS account with CloudWatch log groups
- AWS credentials with appropriate IAM permissions

Note

Unlike file-based log integrations (nginx, Redis), CloudWatch requires running a separate OpenTelemetry Collector that polls the CloudWatch API. This collector can't run inside ClickStack's all-in-one image as it needs AWS credentials and API access.

Get ClickStack API key

#### Get ClickStack API key[​](#get-api-key "Direct link to Get ClickStack API key")

The OpenTelemetry Collector sends data to ClickStack's OTLP endpoint, which requires authentication.

1. Open HyperDX at your ClickStack URL (e.g., <http://localhost:8080>)
2. Create an account or log in if needed
3. Navigate to **Team Settings → API Keys**
4. Copy your **Ingestion API Key**

Save this as an environment variable:

```
export CLICKSTACK_API_KEY="your-api-key-here"
```

Configure AWS credentials

#### Configure AWS credentials[​](#configure-aws "Direct link to Configure AWS credentials")

Export your AWS credentials as environment variables. The method depends on your authentication type:

**For AWS SSO users (recommended for most organizations):**

```
# Login to SSO
aws sso login --profile YOUR_PROFILE_NAME

# Export credentials to environment variables
eval $(aws configure export-credentials --profile YOUR_PROFILE_NAME --format env)

# Verify credentials work
aws sts get-caller-identity
```

Replace `YOUR_PROFILE_NAME` with your AWS SSO profile name (e.g., `AccountAdministrators-123456789`).

**For IAM users with long-term credentials:**

```
export AWS_ACCESS_KEY_ID="your-access-key-id"
export AWS_SECRET_ACCESS_KEY="your-secret-access-key"
export AWS_REGION="us-east-1"

# Verify credentials work
aws sts get-caller-identity
```

**Required IAM permissions:**

The AWS account associated with these credentials needs the following IAM policy to read CloudWatch logs:

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CloudWatchLogsRead",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:FilterLogEvents"
      ],
      "Resource": "arn:aws:logs:*:YOUR_ACCOUNT_ID:log-group:*"
    }
  ]
}
```

Replace `YOUR_ACCOUNT_ID` with your AWS account ID.

Configure the CloudWatch receiver

#### Configure the CloudWatch receiver[​](#configure-receiver "Direct link to Configure the CloudWatch receiver")

Create an `otel-collector-config.yaml` file with the CloudWatch receiver configuration.

**Example 1: Named log groups (recommended)**

This configuration collects logs from specific named log groups:

```
receivers:
  awscloudwatch:
    region: us-east-1
    logs:
      poll_interval: 1m
      max_events_per_request: 100
      groups:
        named:
          /aws/lambda/my-function:
          /aws/ecs/my-service:
          /aws/eks/my-cluster/cluster:

processors:
  batch:
    timeout: 10s

exporters:
  otlphttp:
    endpoint: http://localhost:4318
    headers:
      authorization: ${CLICKSTACK_API_KEY}

service:
  pipelines:
    logs:
      receivers: [awscloudwatch]
      processors: [batch]
      exporters: [otlphttp]
```

**Example 2: Autodiscover log groups with prefix**

This configuration autodiscovers and collects logs from up to 100 log groups starting with the prefix `/aws/lambda`:

```
receivers:
  awscloudwatch:
    region: us-east-1
    logs:
      poll_interval: 1m
      max_events_per_request: 100
      groups:
        autodiscover:
          limit: 100
          prefix: /aws/lambda

processors:
  batch:
    timeout: 10s

exporters:
  otlphttp:
    endpoint: http://localhost:4318
    headers:
      authorization: ${CLICKSTACK_API_KEY}

service:
  pipelines:
    logs:
      receivers: [awscloudwatch]
      processors: [batch]
      exporters: [otlphttp]
```

**Configuration parameters:**

- `region`: AWS region where your log groups are located
- `poll_interval`: How often to check for new logs (e.g., `1m`, `5m`)
- `max_events_per_request`: Maximum number of log events to fetch per request
- `groups.autodiscover.limit`: Maximum number of log groups to discover
- `groups.autodiscover.prefix`: Filter log groups by prefix
- `groups.named`: Explicitly list log group names to collect

For more configuration options, see the [CloudWatch receiver documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/awscloudwatchreceiver).

**Replace the following:**

- `${CLICKSTACK_API_KEY}` → Uses the environment variable you set earlier
- `http://localhost:4318` → Your ClickStack endpoint (use your ClickStack host if running remotely)
- `us-east-1` → Your AWS region
- Log group names/prefixes → Your actual CloudWatch log groups

Note

The CloudWatch receiver only fetches logs from recent time windows (based on `poll_interval`). When first started, it begins from the current time. Historical logs aren't retrieved by default.

Start the collector

#### Start the collector[​](#start-collector "Direct link to Start the collector")

Create a `docker-compose.yaml` file:

```
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-config.yaml
    environment:
      - AWS_ACCESS_KEY_ID
      - AWS_SECRET_ACCESS_KEY
      - AWS_SESSION_TOKEN
      - AWS_REGION
      - CLICKSTACK_API_KEY
    restart: unless-stopped
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

Then start the collector:

```
docker compose up -d
```

View collector logs:

```
docker compose logs -f otel-collector
```

Verify logs in HyperDX

#### Verify logs in HyperDX[​](#verify-logs "Direct link to Verify logs in HyperDX")

Once the collector is running:

1. Open HyperDX at <http://localhost:8080> (or your ClickStack URL)
2. Navigate to the **Logs** view
3. Wait 1-2 minutes for logs to appear (based on your poll interval)
4. Search for logs from your CloudWatch log groups

Look for these key attributes in the logs:

- `ResourceAttributes['aws.region']`: Your AWS region (e.g., "us-east-1")
- `ResourceAttributes['cloudwatch.log.group.name']`: The CloudWatch log group name
- `ResourceAttributes['cloudwatch.log.stream']`: The log stream name
- `Body`: The actual log message content

## Demo dataset[​](#demo-dataset "Direct link to Demo dataset")

For users who want to test the CloudWatch logs integration before configuring their production AWS environment, we provide a sample dataset with pre-generated logs showing realistic patterns from multiple AWS services.

Download the sample dataset

#### Download the sample dataset[​](#download-sample "Direct link to Download the sample dataset")