ClickHouse/clickhouse-docs-embeddings huggingface dataset:
AWS Glue catalog

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/data_lake/reference/aws_glue_catalog.md)

Beta feature. [Learn more.](/docs/beta-and-experimental-features#beta-features)

ClickHouse supports integration with multiple catalogs (Unity, Glue, Polaris,
etc.). In this guide, we will walk you through the steps to query your data in
S3 buckets using ClickHouse and the Glue Data Catalog.

Note

Glue supports many different table formats, but this integration only supports Iceberg tables.

## Configuring Glue in AWS[​](#configuring "Direct link to Configuring Glue in AWS")

To connect to the glue catalog, you will need:

- AWS region of your catalog
- Access Key ID/Access Secret Key (v25.12+) or AWS Role ARN (v26.2+)

For AWS Role auth, AWS Role Session Name is an additional optional field.

Note

You will need to enable it using `SET allow_database_glue_catalog = 1;`

## Creating a connection between Glue data catalog and ClickHouse[​](#connecting "Direct link to Creating a connection between Glue data catalog and ClickHouse")

With your AWS Glue Catalog configured, you can authenticate and establish a
connection between ClickHouse and AWS Glue Catalog.

### AWS Access Key/Secret[​](#aws-access-key-secret "Direct link to AWS Access Key/Secret")

```
CREATE DATABASE glue
ENGINE = DataLakeCatalog
SETTINGS 
    catalog_type = 'glue', 
    region = 'us-west-2', 
    aws_access_key_id = '<access-key>', 
    aws_secret_access_key = '<secret-key>'
```

### AWS Role[​](#aws-role "Direct link to AWS Role")

```
CREATE DATABASE glue
ENGINE = DataLakeCatalog
SETTINGS
    catalog_type = 'glue',
    region = 'us-west-2',
    aws_role_arn = 'arn:aws:iam::1111111111:role/glue_role',
    aws_role_session_name = 'clickhouse-glue-session'
```

## Query the Glue data catalog using ClickHouse[​](#query-glue-catalog "Direct link to Query the Glue data catalog using ClickHouse")

Now that the connection is in place, you can start querying Glue:

```
USE glue;
SHOW TABLES;
```

```
   ┌─name───────────────────────────────────┐
1. │ iceberg-benchmark.hitsiceberg          │
2. │ iceberg-benchmark.hitsparquet          │
3. │ iceberg_benchmark.hitsdailypartitioned │
4. │ iceberg_benchmark.time_travel          │
   └────────────────────────────────────────┘
```

You can see above that some tables above aren't Iceberg tables, for instance
`iceberg-benchmark.hitsparquet`. You won't be able to query these as only Iceberg
is currently supported.

To query a table:

```
SELECT count(*) FROM `iceberg-benchmark.hitsiceberg`;
```

Note

Backticks are required because ClickHouse doesn't support more than one namespace.

To inspect the table DDL, run the following query:

```
SHOW CREATE TABLE `iceberg-benchmark.hitsiceberg`;
```
Support matrix

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/data_lake/support-matrix.md)

This page provides comprehensive support matrices for ClickHouse's data lake integrations. It covers the features available for each open table format, the catalogs ClickHouse can connect to, and the capabilities supported by each catalog.

## Open table format support[​](#format-support "Direct link to Open table format support")

ClickHouse integrates with four open table formats: [Apache Iceberg](/docs/engines/table-engines/integrations/iceberg), [Delta Lake](/docs/engines/table-engines/integrations/deltalake), [Apache Hudi](/docs/engines/table-engines/integrations/hudi), and [Apache Paimon](/docs/sql-reference/table-functions/paimon). Select a format below to view its support matrix.

**Legend:** ✅ Supported | ⚠️ Partial / Experimental | ❌ Not supported

- Apache Iceberg- Delta Lake- Apache Hudi- Apache Paimon
Support matrix

--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Expire snapshots ❌ Not supported|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Remove orphan files ❌ Not supported|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Writing partitions ✅ Supported.|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Altering partitions ❌ The changing of the partitioning scheme from ClickHouse is not supported. ClickHouse can write to iceberg tables which have an evolved partitioning.|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **Metadata** |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Branching and tagging ❌ Iceberg branch/tag references not supported|  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Metadata file resolution ✅ Support for metadata resolution through catalogs, simple directory listing, using 'version-hint' and specific path. Configurable via `iceberg_metadata_file_path` and `iceberg_metadata_table_uuid`. See [Metadata file resolution](/docs/engines/table-engines/integrations/iceberg#metadata-file-resolution).| Data caching ✅ Same mechanism as S3/Azure/HDFS storage engines. See [Data cache](/docs/engines/table-engines/integrations/iceberg#data-cache).| Metadata caching ✅ Manifest and metadata files cached in memory. Enabled by default via `use_iceberg_metadata_files_cache`. See [Metadata cache](/docs/engines/table-engines/integrations/iceberg#metadata-cache). | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
Helm

### v1.x documentation[​](#v1x-documentation "Direct link to v1.x documentation")

- [Helm (v1.x)](/docs/use-cases/observability/clickstack/deployment/helm-v1) - v1.x deployment guide
- [Configuration (v1.x)](/docs/use-cases/observability/clickstack/deployment/helm-configuration-v1) - v1.x configuration
- [Deployment options (v1.x)](/docs/use-cases/observability/clickstack/deployment/helm-deployment-options-v1) - v1.x deployment options
- [Cloud deployments (v1.x)](/docs/use-cases/observability/clickstack/deployment/helm-cloud-v1) - v1.x cloud configurations

### Additional resources[​](#additional-resources "Direct link to Additional resources")

- [ClickStack getting started guide](/docs/use-cases/observability/clickstack/getting-started) - Introduction to ClickStack
- [ClickStack Helm charts repository](https://github.com/ClickHouse/ClickStack-helm-charts) - Chart source code and values reference
- [Kubernetes documentation](https://kubernetes.io/docs/) - Kubernetes reference
- [Helm documentation](https://helm.sh/docs/) - Helm reference

- [Suitable for](#suitable-for)- [Deployment steps](#deployment-steps)
    - [Prerequisites](#prerequisites)- [Add the ClickStack Helm repository](#add-the-clickstack-helm-repository)- [Install the operators](#install-the-operators)- [Install ClickStack](#installing-clickstack)- [Verify the installation](#verify-the-installation)- [Forward ports](#forward-ports)- [Navigate to the UI](#navigate-to-the-ui)- [Customizing values (optional)](#customizing-values)- [Using secrets (optional)](#using-secrets)- [Using ClickHouse Cloud](#using-clickhouse-cloud)- [Production notes](#production-notes)- [Task configuration](#task-configuration)- [Upgrading the chart](#upgrading-the-chart)- [Uninstalling ClickStack](#uninstalling-clickstack)- [Troubleshooting](#troubleshooting)
                - [Checking logs](#checking-logs)- [Debugging a failed install](#debugging-a-failed-install)- [Verifying deployment](#verifying-deployment)- [JSON type support](#json-type-support)- [Related documentation](#related-documentation)
                    - [Deployment guides](#deployment-guides)- [v1.x documentation](#v1x-documentation)- [Additional resources](#additional-resources)

Was this page helpful?
Helm cloud deployments

## Next steps[​](#next-steps "Direct link to Next steps")

- [Configuration guide](/docs/use-cases/observability/clickstack/deployment/helm-configuration) - API keys, secrets, and ingress
- [Deployment options](/docs/use-cases/observability/clickstack/deployment/helm-deployment-options) - External systems configuration
- [Upgrade guide](/docs/use-cases/observability/clickstack/deployment/helm-upgrade) - Migrating from v1.x to v2.x
- [Additional manifests](/docs/use-cases/observability/clickstack/deployment/helm-additional-manifests) - Custom Kubernetes objects
- [Main Helm guide](/docs/use-cases/observability/clickstack/deployment/helm) - Basic installation
- [Cloud deployments (v1.x)](/docs/use-cases/observability/clickstack/deployment/helm-cloud-v1) - v1.x cloud configurations

- [Google Kubernetes Engine (GKE)](#google-kubernetes-engine-gke)
  - [LoadBalancer DNS resolution issue](#loadbalancer-dns-resolution-issue)- [GKE example values](#gke-example-values)- [Amazon EKS](#amazon-eks)- [Azure AKS](#azure-aks)- [Production cloud deployment checklist](#production-cloud-deployment-checklist)- [Production best practices](#production-best-practices)
          - [Resource management](#resource-management)- [High availability](#high-availability)- [Persistent storage](#persistent-storage)- [Browser compatibility notes](#browser-compatibility-notes)- [Next steps](#next-steps)

Was this page helpful?
Helm configuration

## Next steps[​](#next-steps "Direct link to Next steps")

- [Deployment options](/docs/use-cases/observability/clickstack/deployment/helm-deployment-options) - External systems and minimal deployments
- [Cloud deployments](/docs/use-cases/observability/clickstack/deployment/helm-cloud) - GKE, EKS, and AKS configurations
- [Upgrade guide](/docs/use-cases/observability/clickstack/deployment/helm-upgrade) - Migrating from v1.x to v2.x
- [Additional manifests](/docs/use-cases/observability/clickstack/deployment/helm-additional-manifests) - Custom Kubernetes objects
- [Main Helm guide](/docs/use-cases/observability/clickstack/deployment/helm) - Basic installation
- [Configuration (v1.x)](/docs/use-cases/observability/clickstack/deployment/helm-configuration-v1) - v1.x configuration guide

- [Values organization](#values-organization)- [API key setup](#api-key-setup)
    - [Method 1: Update via Helm upgrade with values file](#api-key-values-file)- [Method 2: Update via Helm upgrade with --set flag](#api-key-set-flag)- [Restart pods to apply changes](#restart-pods)- [Secret management](#secret-management)
      - [Default secret values](#default-secret-values)- [Using an external secret](#using-external-secret)- [Ingress setup](#ingress-setup)
        - [General ingress configuration](#general-ingress-configuration)- [Enabling TLS (HTTPS)](#enabling-tls)- [Example ingress configuration](#example-ingress-configuration)- [Common ingress pitfalls](#common-ingress-pitfalls)- [OTEL collector ingress](#otel-collector-ingress)- [OTEL collector configuration](#otel-collector-configuration)- [MongoDB configuration](#mongodb-configuration)- [ClickHouse configuration](#clickhouse-configuration)- [Troubleshooting ingress](#troubleshooting-ingress)- [Customizing values](#customizing-values)- [Next steps](#next-steps)

Was this page helpful?
Helm (v1.x)

### v2.x documentation[​](#v2x-documentation "Direct link to v2.x documentation")

- [Helm (v2.x)](/docs/use-cases/observability/clickstack/deployment/helm) - v2.x deployment guide
- [Upgrade guide](/docs/use-cases/observability/clickstack/deployment/helm-upgrade) - Migrating from v1.x to v2.x

### Additional resources[​](#additional-resources "Direct link to Additional resources")

- [ClickStack getting started guide](/docs/use-cases/observability/clickstack/getting-started) - Introduction to ClickStack
- [ClickStack Helm charts repository](https://github.com/ClickHouse/ClickStack-helm-charts) - Chart source code and values reference
- [Kubernetes documentation](https://kubernetes.io/docs/) - Kubernetes reference
- [Helm documentation](https://helm.sh/docs/) - Helm reference

- [Suitable for](#suitable-for)- [Deployment steps](#deployment-steps)
    - [Prerequisites](#prerequisites)- [Add the ClickStack Helm repository](#add-the-clickstack-helm-repository)- [Installing ClickStack](#installing-clickstack)- [Verify the installation](#verify-the-installation)- [Forward ports](#forward-ports)- [Navigate to the UI](#navigate-to-the-ui)- [Customizing values (optional)](#customizing-values)- [Using secrets (optional)](#using-secrets)- [Using ClickHouse Cloud](#using-clickhouse-cloud)- [Production notes](#production-notes)- [Task configuration](#task-configuration)- [Upgrading the chart](#upgrading-the-chart)- [Uninstalling ClickStack](#uninstalling-clickstack)- [Troubleshooting](#troubleshooting)
                - [Checking logs](#checking-logs)- [Debugging a failed install](#debugging-a-failed-install)- [Verifying deployment](#verifying-deployment)- [JSON type support](#json-type-support)- [Related Documentation](#related-documentation)
                    - [v1.x deployment guides](#deployment-guides)- [v2.x documentation](#v2x-documentation)- [Additional resources](#additional-resources)

Was this page helpful?
Event patterns with ClickStack

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/event_patterns.md)

Event patterns in ClickStack allow you to quickly make sense of large volumes of logs or traces by automatically clustering similar messages together, so instead of digging through millions of individual events, you only need to review a small number of meaningful groups.

This makes it much easier to spot which errors or warnings are new, which are recurring, and which are driving sudden spikes in log volume. Because the patterns are generated dynamically, you don't need to define regular expressions or maintain parsing rules - ClickStack adapts to your events automatically, regardless of format.

Beyond incident response, this high-level view also helps you identify noisy log sources that can be trimmed to reduce cost, discover the different types of logs a service produces, and more quickly answer whether the system is already emitting the signals you care about.

## Accessing event patterns[​](#accessing-event-patterns "Direct link to Accessing event patterns")

Event patterns are available directly through the **Search** panel in ClickStack.

From the top-left **Analysis Mode** selector, choose **Event patterns** to switch from the standard results table to a clustered view of similar events.

This provides an alternative to the default **Results Table** which allows you to scroll through every individual log or trace.

## Recommendations[​](#recommendations "Direct link to Recommendations")

Event patterns are most effective when applied to **narrowed subsets** of your data. For example, filtering down to a single service before enabling event patterns will usually surface more relevant and interesting messages than applying patterns across thousands of services at once.

They're also particularly powerful for summarizing error messages, where repeated errors with varying IDs or payloads are grouped into concise clusters.

For a live example, see how event patterns are used in the [Remote Demo Dataset](/docs/use-cases/observability/clickstack/getting-started/remote-demo-data#identify-error-patterns).

- [Accessing event patterns](#accessing-event-patterns)- [Recommendations](#recommendations)

Was this page helpful?
ClickStack OpenTelemetry collector

#### Create new data sources[​](#create-new-data-sources "Direct link to Create new data sources")

Create new data sources in HyperDX pointing to the JSON tables.

#### Migrating existing data (optional)[​](#migrating-existing-data "Direct link to Migrating existing data (optional)")

To move old data into the new JSON tables:

```
INSERT INTO otel_logs SELECT * FROM otel_logs_map;
INSERT INTO otel_metrics SELECT * FROM otel_metrics_map;
```

Note

Recommended only for datasets smaller than ~10 billion rows. Data previously stored with the Map type didn't preserve type precision (all values were strings). As a result, this old data will appear as strings in the new schema until it ages out, requiring some casting on the frontend. Type for new data will be preserved with the JSON type.

- [Collector roles](#collector-roles)- [Deploying the collector](#configuring-the-collector)- [Securing the collector](#securing-the-collector)- [Processing - filtering, transforming, and enriching](#processing-filtering-transforming-enriching)- [Optimizing inserts](#optimizing-inserts)- [Scaling](#scaling)- [Estimating resources](#estimating-resources)- [JSON support](#json-support)

Was this page helpful?
Monitoring PostgreSQL Metrics with ClickStack

1. Open HyperDX and navigate to the Dashboards section
2. Click **Import Dashboard** in the upper right corner under the ellipses

3. Upload the `postgres-metrics-dashboard.json` file and click **Finish Import**

View the dashboard

#### View the dashboard[​](#created-dashboard "Direct link to View the dashboard")

The dashboard will be created with all visualizations pre-configured:

Note

For the demo dataset, set the time range to **2025-11-10 00:00:00 - 2025-11-11 00:00:00 (UTC)** (adjust based on your local timezone). The imported dashboard won't have a time range specified by default.

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

### Custom config not loading[​](#troubleshooting-not-loading "Direct link to Custom config not loading")

Verify the environment variable is set:

```
docker exec <container-name> printenv CUSTOM_OTELCOL_CONFIG_FILE
```

Check the custom config file is mounted:

```
docker exec <container-name> cat /etc/otelcol-contrib/custom.config.yaml
```

### No metrics appearing in HyperDX[​](#no-metrics "Direct link to No metrics appearing in HyperDX")

Verify PostgreSQL is accessible:

```
docker exec <clickstack-container> psql -h postgres-host -U otel_monitor -d postgres -c "SELECT 1"
```

Check OTel collector logs:

```
docker exec <container> cat /etc/otel/supervisor-data/agent.log | grep -i postgres
```

### Authentication errors[​](#auth-errors "Direct link to Authentication errors")

Verify password is set correctly:

```
docker exec <clickstack-container> printenv POSTGRES_PASSWORD
```

Test credentials directly:

```
psql -h postgres-host -U otel_monitor -d postgres -c "SELECT version();"
```

## Next steps[​](#next-steps "Direct link to Next steps")

- Set up [alerts](/docs/use-cases/observability/clickstack/alerts) for critical thresholds (connection limits, high rollback rates, low cache hit ratios)
- Enable query-level monitoring with `pg_stat_statements` extension
- Monitor multiple PostgreSQL instances by duplicating the receiver configuration with different endpoints and service names

## Going to production[​](#going-to-production "Direct link to Going to production")

This guide extends ClickStack's built-in OpenTelemetry Collector for quick setup. For production deployments, we recommend running your own OTel Collector and sending data to ClickStack's OTLP endpoint. See [Sending OpenTelemetry data](/docs/use-cases/observability/clickstack/ingesting-data/opentelemetry) for production configuration.

- [Integration with existing PostgreSQL](#existing-postgres)- [Demo dataset](#demo-dataset)- [Dashboards and visualization](#dashboards)- [Troubleshooting](#troubleshooting)
        - [Custom config not loading](#troubleshooting-not-loading)- [No metrics appearing in HyperDX](#no-metrics)- [Authentication errors](#auth-errors)- [Next steps](#next-steps)- [Going to production](#going-to-production)

Was this page helpful?
Service maps

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/service-maps.md)

Beta feature. [Learn more.](/docs/beta-and-experimental-features#beta-features)

Service maps visualize how your services interact. ClickStack builds the graph by matching client spans (outgoing requests) with server spans (incoming requests) within the same trace, reconstructing the request path between services.

Click **Service Map** in the left navigation panel to open the full graph. Services appear once you're [ingesting trace data](/docs/use-cases/observability/clickstack/ingesting-data) with OpenTelemetry.

## Exploring the service map[​](#exploring-the-service-map "Direct link to Exploring the service map")

Each node represents a service, identified by the `service.name` resource attribute. Edges (dashed lines) connect services where a client span in one matches a server span in another. Node size reflects relative traffic volume, and red nodes indicate services with errors in the selected time range.

The toolbar above the map lets you filter and adjust the view.

**Source selector** — filter the map to a specific trace source (e.g. "ClickPy Traces").

**Sampling slider** — adjust the sampling rate to balance performance and accuracy. Lower rates load faster on high-volume clusters.

**Date range picker** — set the time window for the trace data used to build the map.

Use the **+/-** buttons in the bottom-left corner of the map or scroll to zoom in and out.

## Trace-level service maps[​](#trace-level-service-maps "Direct link to Trace-level service maps")

When you inspect an individual trace, a focused service map shows how that specific request moved between services. This lets you see the topology for a single request without leaving the trace waterfall.

[](/docs/assets/medias/service-map-demo-4687d70fb61479c0b21c341895282187.mp4)

- [Exploring the service map](#exploring-the-service-map)- [Trace-level service maps](#trace-level-service-maps)

Was this page helpful?
Session replay

- **Live example** — visit [clickpy.clickhouse.com](https://clickpy.clickhouse.com), interact with the app, then view your session replay at [play-clickstack.clickhouse.com](https://play-clickstack.clickhouse.com) under the **ClickPy Sessions** source. For details on how ClickPy was instrumented, see the blog post [Instrumenting your NextJS application with OpenTelemetry and ClickStack](https://clickhouse.com/blog/instrumenting-your-app-with-otel-clickstack).
- **Local demo** — the [Session replay demo](/docs/use-cases/observability/clickstack/example-datasets/session-replay-demo) walks through instrumenting a demo application step by step, including running ClickStack locally and viewing your replays.

## Learn more[​](#learn-more "Direct link to Learn more")

- [Session replay demo](/docs/use-cases/observability/clickstack/example-datasets/session-replay-demo) — interactive local demo application with step-by-step instructions
- [Browser SDK Reference](/docs/use-cases/observability/clickstack/sdks/browser) — full SDK options, source maps, custom actions, and advanced configuration
- [Search](/docs/use-cases/observability/clickstack/search) — search syntax for filtering sessions and events
- [Dashboards](/docs/use-cases/observability/clickstack/dashboards) — build visualizations and dashboards from session and trace data
- [Alerts](/docs/use-cases/observability/clickstack/alerts) — set up alerts on errors, latency, and other signals
- [ClickStack Architecture](/docs/use-cases/observability/clickstack/architecture) — how ClickHouse, HyperDX, and the OTel collector fit together

- [Instrumenting your application](#instrumentation)- [Viewing session replays](#viewing-replays)
    - [From session to trace](#session-to-trace)- [From trace to session](#trace-to-session)- [How session data is stored](#data-storage)- [Try it out](#try-it-out)- [Learn more](#learn-more)

Was this page helpful?