Monitoring Cloudflare Logs with ClickStack

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/ingesting-data/integration-examples/cloudflare.md)

TL;DR

This guide shows you how to ingest Cloudflare logs into ClickStack using ClickPipes. Cloudflare Logpush writes logs to S3, and ClickPipes continuously ingests new files into ClickHouse. Unlike most ClickStack integration guides that use the OpenTelemetry Collector, this guide uses [ClickPipes](/docs/integrations/clickpipes) to pull data directly from S3.

A demo dataset is available if you want to explore the dashboards before configuring production ingestion.

## Overview[​](#overview "Direct link to Overview")

Cloudflare [Logpush](https://developers.cloudflare.com/logs/about/) exports HTTP request logs to destinations like Amazon S3. Forwarding these logs to ClickStack allows you to:

- Analyze edge traffic, cache performance, and security events alongside your other observability data
- Query logs using ClickHouse SQL
- Retain logs beyond Cloudflare's default retention

This guide uses [ClickPipes](/docs/integrations/clickpipes) to continuously ingest Cloudflare log files from S3 into ClickHouse. S3 acts as a durable buffer between Cloudflare and ClickHouse, providing exactly-once semantics and replay capability.

Alternative: Direct HTTP ingestion

Cloudflare Logpush also supports pushing logs to [HTTP endpoints](https://developers.cloudflare.com/logs/get-started/enable-destinations/http/) directly. Since Cloudflare exports logs as newline-delimited JSON (NDJSON) and ClickHouse natively accepts this format via `JSONEachRow`, you can point Logpush directly at your ClickHouse Cloud HTTP interface using the following endpoint URL format:

```
https://YOUR_CLICKHOUSE_HOST:8443/?query=INSERT+INTO+cloudflare_http_logs+FORMAT+JSONEachRow&header_Authorization=Basic+BASE64_CREDENTIALS
```

Replace `YOUR_CLICKHOUSE_HOST` with your ClickHouse Cloud hostname and `BASE64_CREDENTIALS` with your Base64-encoded credentials (`echo -n 'default:YOUR_PASSWORD' | base64`).

This is simpler to set up (no S3, SQS, or IAM configuration needed), but Cloudflare Logpush [cannot backfill historical data](https://developers.cloudflare.com/logs/logpush/) if delivery fails — so if ClickHouse is unavailable during a push, those logs are permanently lost.

## Integration with existing Cloudflare Logpush[​](#existing-cloudflare "Direct link to Integration with existing Cloudflare Logpush")

This section assumes you have Cloudflare Logpush configured to export logs to S3. If not, follow [Cloudflare's AWS S3 setup guide](https://developers.cloudflare.com/logs/get-started/enable-destinations/aws-s3/) first.

### Prerequisites[​](#prerequisites "Direct link to Prerequisites")

- **ClickHouse Cloud service** running (ClickPipes is a Cloud-only feature — not available in ClickStack OSS)
- Cloudflare Logpush actively writing logs to an S3 bucket
- S3 bucket name and region where Cloudflare writes logs

Configure S3 authentication

#### Configure S3 authentication[​](#configure-auth "Direct link to Configure S3 authentication")

ClickPipes needs permission to read from your S3 bucket. Follow the [Accessing S3 data securely](/docs/cloud/data-sources/secure-s3) guide to configure either IAM role-based access or credentials-based access.

For full details on ClickPipes S3 authentication and permissions, see the [S3 ClickPipes reference documentation](/docs/integrations/clickpipes/object-storage/s3/overview#access-control).

Create ClickPipes job

#### Create ClickPipes job[​](#create-clickpipes "Direct link to Create ClickPipes job")

1. ClickHouse Cloud Console → **Data Sources** → **Create ClickPipe**
2. **Source**: Amazon S3

**Connection:**

- **S3 file path**: Your Cloudflare logs bucket path with a wildcard to match files. If you enabled daily subfolders in Logpush, use `**` to match across subdirectories:
  - No subfolders: `https://your-bucket.s3.us-east-1.amazonaws.com/logs/*`
  - Daily subfolders: `https://your-bucket.s3.us-east-1.amazonaws.com/logs/**/*`
- **Authentication**: Select your authentication method and provide the credentials or IAM Role ARN

**Ingestion settings:**

Click **Incoming data**, then configure:

- Toggle on **Continuous ingestion**
- **Ordering**: Lexicographical order

Cloudflare Logpush writes files with date-based naming (e.g., `20250127/...`), which is naturally lexicographical. ClickPipes polls for new files every 30 seconds and ingests any file with a name greater than the last processed file.

**Schema mapping:**

Click **Parse information**. ClickPipes samples your log files and auto-detects the schema. Review the mapped columns and adjust types as needed. Define a **Sorting key** for the destination table — for Cloudflare logs, a good choice is `(EdgeStartTimestamp, ClientCountry, EdgeResponseStatus)`.

Click **Complete Setup**.

Note

When first created, ClickPipes performs an initial load of **all existing files** in the specified path before switching to continuous polling. If your bucket contains a large backlog of Cloudflare logs, this initial load may take some time.

Configure HyperDX data source

#### Configure HyperDX data source[​](#configure-source "Direct link to Configure HyperDX data source")

ClickPipes ingests Cloudflare logs into a flat table with Cloudflare's native field names. To view these logs in HyperDX, configure a custom data source that maps Cloudflare columns to HyperDX's log view.

1. Open HyperDX → **Team Settings** → **Sources**

2. Click **Add source** and configure the following settings. Click **Configure Optional Fields** to access all fields:

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Setting Value|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **Name** `Cloudflare Logs`| **Source Data Type** Log|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **Database** `default`| **Table** `cloudflare_http_logs`| **Timestamp Column** `toDateTime(EdgeStartTimestamp / 1000000000)`| **Default Select** `EdgeStartTimestamp, ClientRequestMethod, ClientRequestURI, EdgeResponseStatus, ClientCountry`| **Service Name Expression** `'cloudflare'`| **Log Level Expression** `multiIf(EdgeResponseStatus >= 500, 'ERROR', EdgeResponseStatus >= 400, 'WARN', 'INFO')`| **Body Expression** `concat(ClientRequestMethod, ' ', ClientRequestURI, ' ', toString(EdgeResponseStatus))`| **Log Attributes Expression** `map('http.method', ClientRequestMethod, 'http.status_code', toString(EdgeResponseStatus), 'http.url', ClientRequestURI, 'client.country', ClientCountry, 'client.ip', ClientIP, 'cache.status', CacheCacheStatus, 'bot.score', toString(BotScore), 'cloudflare.ray_id', RayID, 'cloudflare.colo', EdgeColoCode)`| **Resource Attributes Expression** `map('cloudflare.zone', ClientRequestHost)`| **Implicit Column Expression** `concat(ClientRequestMethod, ' ', ClientRequestURI)` | | | | | | | | | | | | | | | | | | | | | | | | | |

3. Click **Save Source**

This maps Cloudflare's native columns directly to HyperDX's log viewer without any data transformation or duplication. The **Body** displays a request summary like `GET /api/v1/users 200`, and all Cloudflare fields are available as searchable attributes.

Verify data in HyperDX

#### Verify data in HyperDX[​](#verify-hyperdx "Direct link to Verify data in HyperDX")

Navigate to the **Search** view and select the **Cloudflare Logs** source. Set the time range to cover your data. You should see log entries with: