ClickHouse/clickhouse-docs-embeddings huggingface dataset:
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/ingesting-data/overview.md)

All data is ingested into ClickStack Open Source or Managed ClickStack via an **OpenTelemetry (OTel) collector**. This acts as the primary entry point for logs, metrics, traces, and session data.

The key difference between the architectures lies in where the components are hosted. In both cases, you run an OpenTelemetry Collector to receive telemetry data from your applications. With ClickStack Open Source, you also manage and host both ClickHouse and the ClickStack UI (HyperDX) yourself. With Managed ClickStack, ClickHouse and the HyperDX UI are hosted and managed in ClickHouse Cloud, with integrated authentication and operational management handled for you.

- Managed ClickStack- Open Source ClickStack

In both deployment models, the collector exposes two OTLP endpoints:

- **HTTP** - port `4318`
- **gRPC** - port `4317`

You can send data to these endpoints either directly from [language SDKs](/docs/use-cases/observability/clickstack/sdks) or OTel-compatible data collection agents e.g. other OTel collectors collecting infrastructure metrics and logs.

More specifically:

- [**Language SDKs**](/docs/use-cases/observability/clickstack/sdks) are responsible for collecting telemetry from within your application - most notably **traces** and **logs** - and exporting this data to the OpenTelemetry collector, via the OTLP endpoint, which handles ingestion into ClickHouse. For more details on the language SDKs available with ClickStack see [SDKs](/docs/use-cases/observability/clickstack/sdks).
- **Data collection agents** are agents deployed at the edge — on servers, Kubernetes nodes, or alongside applications. They collect infrastructure telemetry (e.g. logs, metrics) or receive events directly from applications instrumented with SDKs. In this case, the agent runs on the same host as the application, often as a sidecar or DaemonSet. These agents forward data to the central ClickStack OTel collector, which acts as a [gateway](/docs/use-cases/observability/clickstack/ingesting-data/otel-collector#collector-roles), typically deployed once per cluster, data center, or region. The [gateway](/docs/use-cases/observability/clickstack/ingesting-data/otel-collector#collector-roles) receives OTLP events from agents or applications and handles ingestion into ClickHouse. See [OTel collector](/docs/use-cases/observability/clickstack/ingesting-data/otel-collector) for more details. These agents can be other instances of the OTel collector or alternative technologies such as [Fluentd](https://www.fluentd.org/) or [Vector](https://vector.dev/).

OpenTelemetry compatibility

While ClickStack offers its own language SDKs and a custom OpenTelemetry, with enhanced telemetry and features, you can also use their existing OpenTelemetry SDKs and agents seamlessly.

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/integration-partners/index.md)

ClickStack provides the core building blocks of modern observability, bringing together data collection, ingestion, storage, and visualization into a unified, high-performance stack. Our integration partners extend these capabilities with complementary tools that add specialized functionality beyond what ClickStack delivers out of the box. Together, ClickStack and its partners enable teams to build a complete observability platform tailored to their needs, without sacrificing performance, flexibility, or scale.

|  |  |  |  |
| --- | --- | --- | --- |
| Section Description|  |  | | --- | --- | | [Bindplane](/docs/use-cases/observability/clickstack/integration-partners/bindplane) Bindplane is an OpenTelemetry-native telemetry pipeline that provides centralized management for OpenTelemetry Collectors. | | | |

Was this page helpful?
title_wip

## Advanced configuration[​](#advanced-configuration "Direct link to Advanced configuration")

### Fan-out routing[​](#fan-out-routing "Direct link to Fan-out routing")

Bindplane supports fan-out routing, allowing you to send the same telemetry streams to multiple destinations simultaneously. You can:

- Send logs, metrics, and traces to ClickStack for long-term storage and analysis
- Route the same data to other observability platforms for real-time alerting
- Forward specific telemetry to SIEM platforms for security analysis

This is configured by adding multiple destinations to your Bindplane configuration.

### Compression and performance[​](#compression "Direct link to Compression and performance")

For high-volume scenarios, configure compression in your ClickStack destination:

- **HTTP**: Supports gzip, deflate, snappy, zstd, or none (default: gzip)
- **gRPC**: Supports gzip, snappy, zstd, or none (default: gzip)

Compression reduces bandwidth usage when sending telemetry to ClickStack, especially important at scale.

## Next steps[​](#next-steps "Direct link to Next steps")

Now that you have telemetry flowing from Bindplane to ClickStack, you can:

- **Build dashboards**: Create visualizations in ClickStack (HyperDX) for logs, metrics, and traces
- **Set up alerts**: Configure alerts in ClickStack for critical conditions
- **Scale your deployment**: Add more collectors and sources as your observability needs grow
- **Optimize your pipeline**: Use Bindplane's pipeline intelligence to identify optimization opportunities

## Read more[​](#read-more "Direct link to Read more")

- [ClickStack Integration in the Bindplane Documentation](https://docs.bindplane.com/integrations/destinations/clickstack)

- [What is Bindplane?](#what-is-bindplane)- [Why Bindplane + ClickStack?](#why-bindplane-clickstack)- [Prerequisites](#prerequisites)- [Integrate ClickStack with Bindplane](#integrate-bindplane-clickstack)- [Advanced configuration](#advanced-configuration)
          - [Fan-out routing](#fan-out-routing)- [Compression and performance](#compression)- [Next steps](#next-steps)- [Read more](#read-more)

Was this page helpful?
Monitoring AWS CloudWatch Logs with ClickStack

## Going to production[​](#going-to-production "Direct link to Going to production")

This guide demonstrates running the OpenTelemetry Collector locally with Docker Compose for testing. For production deployments, run the collector on infrastructure with AWS access (EC2 with IAM roles, EKS with IRSA, or ECS with task roles) to eliminate the need for managing access keys. Deploy collectors in the same AWS region as your CloudWatch log groups to reduce latency and costs.

See [Ingesting with OpenTelemetry](/docs/use-cases/observability/clickstack/ingesting-data/opentelemetry) for production deployment patterns and collector configuration examples.

- [Overview](#overview)- [Integration with existing CloudWatch log groups](#existing-cloudwatch)
    - [Prerequisites](#prerequisites)- [Demo dataset](#demo-dataset)- [Dashboards and visualization](#dashboards)- [Troubleshooting](#troubleshooting)
          - [No logs appearing in HyperDX](#no-logs)- [Only seeing old logs or missing recent logs](#missing-recent)- [Invalid security token / credentials expired](#expired-credentials)- [High latency or missing recent logs](#latency)- [Collector using too much memory](#memory)- [Next steps](#next-steps)- [Going to production](#going-to-production)

Was this page helpful?
Monitoring MySQL Logs with ClickStack

### Slow query logs not appearing[​](#no-slow-queries "Direct link to Slow query logs not appearing")

Verify slow query log is enabled in MySQL:

```
SHOW VARIABLES LIKE 'slow_query_log';
SHOW VARIABLES LIKE 'long_query_time';
```

Check if MySQL is writing slow queries:

```
tail -f /var/log/mysql/mysql-slow.log
```

Generate a test slow query:

```
SELECT SLEEP(2);
```

### Logs not parsing correctly[​](#logs-not-parsing "Direct link to Logs not parsing correctly")

Verify your MySQL log format matches the expected format. The regex patterns in this guide are designed for MySQL 5.7+ and 8.0+ default formats.

Check a few lines from your error log:

```
head -5 /var/log/mysql/error.log
```

Expected format:

```
2025-11-14T10:23:45.123456+00:00 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.35) starting as process 1
```

If your format differs significantly, adjust the regex patterns in the configuration.

## Next steps[​](#next-steps "Direct link to Next steps")

- Set up [alerts](/docs/use-cases/observability/clickstack/alerts) for critical events (connection failures, slow queries exceeding thresholds, error spikes)
- Create custom dashboards for slow query analysis by query pattern
- Tune `long_query_time` based on observed query performance patterns

## Going to production[​](#going-to-production "Direct link to Going to production")

This guide extends ClickStack's built-in OpenTelemetry Collector for quick setup. For production deployments, we recommend running your own OTel Collector and sending data to ClickStack's OTLP endpoint. See [Sending OpenTelemetry data](/docs/use-cases/observability/clickstack/ingesting-data/opentelemetry) for production configuration.

- [Integration with existing MySQL](#existing-mysql)- [Demo dataset](#demo-dataset)- [Dashboards and visualization](#dashboards)- [Troubleshooting](#troubleshooting)
        - [Custom config not loading](#troubleshooting-not-loading)- [No logs appearing in HyperDX](#no-logs)- [Slow query logs not appearing](#no-slow-queries)- [Logs not parsing correctly](#logs-not-parsing)- [Next steps](#next-steps)- [Going to production](#going-to-production)

Was this page helpful?
Monitoring Node.js Traces with ClickStack

```
echo $CLICKSTACK_API_KEY
# Should output your API key

echo $OTEL_EXPORTER_OTLP_ENDPOINT
# Should output http://localhost:4318 or your ClickStack host
```

**Verify network connectivity:**

```
curl -v http://localhost:4318/v1/traces
```

Should connect successfully to the OTLP endpoint.

**Check application logs:**
Look for OpenTelemetry initialization messages when your app starts. The HyperDX SDK should output confirmation that it's initialized.

## Next steps[​](#next-steps "Direct link to Next steps")

- Set up [alerts](/docs/use-cases/observability/clickstack/alerts) for critical metrics (error rates, latency thresholds)
- Create additional dashboards for specific use cases (API monitoring, security events)

## Going to production[​](#going-to-production "Direct link to Going to production")

This guide uses the HyperDX SDK which sends traces directly to ClickStack's OTLP endpoint. This works well for development, testing, and small-to-medium production deployments.
For larger production environments or if you need additional control over telemetry data, consider deploying your own OpenTelemetry Collector as an agent.
See [Ingesting with OpenTelemetry](/docs/use-cases/observability/clickstack/ingesting-data/opentelemetry) for production deployment patterns and collector configuration examples.

- [Integration with existing Node.js application](#existing-nodejs)- [Demo dataset](#demo-dataset)- [Dashboards and visualization](#dashboards)- [Troubleshooting](#troubleshooting)
        - [Demo traces not appearing via curl](#demo-traces-not-appearing)- [No traces appearing in HyperDX](#no-traces)- [Next steps](#next-steps)- [Going to production](#going-to-production)

Was this page helpful?
ClickStack - Materialized Views

- MV has high cardinality due to many dimension columns.
- Solution: Reduce dimension columns to most commonly used ones.

**Issue 3: multiple MVs with high row counts**

- The System is running `EXPLAIN` on each MV.
- Solution: Remove MVs that are rarely used or always skipped.

### Configuration errors[​](#config-errors "Direct link to Configuration errors")

**Error: "At least one aggregated column is required"**

- Add at least one aggregated column to the MV configuration.

**Error: "Source column is required for non-count aggregations"**

- Specify which column to aggregate (only count can omit source column).

**Error: "Invalid granularity format"**

- Use one of the preset granularities from the dropdown.
- Format must be a valid SQL interval (e.g., `1 hour`, not `1 h`).

- [Introduction](#introduction)- [What are incremental materialized views](#what-are-incremental-materialized-views)- [Selecting visualizations for acceleration](#selecting-visualizatons-for-acceleration)
      - [Identify high-impact visualizations](#identify-high-impact-visualizations)- [Balance benefit against insert-time cost](#balance-benefit-against-insert-time-cost)- [Consolidate visualizations into shared views](#consolidate-visualizations-into-shared-views)- [Creating a materialized view](#creating-a-materialized-view)
        - [Example materialized view](#example-materialized-view)- [Using materialized view usage in ClickStack](#materialized-view-usage-in-clickstack)
          - [Registering a Materialized for usage](#registering-a-view)- [Verifying acceleration in dashboards and visualizations](#verifying-acceleration-in-dashboards-and-visualizations)- [How materialized views are selected for visualizations](#how-views-are-selected)- [Alerts](#alerts)- [Backfilling a materialized view](#backfilling-a-materialized-view)
            - [Backfilling approaches](#backfilling-approaches)- [Recommendations](#recommendations)
              - [Granularity selection and alignment](#granularity-selection-and-alignment)- [Limit and consolidate materialized views](#limit-and-consolidate-materialized-views)- [Choose dimensions carefully](#choose-dimensions-carefully)- [Naming conventions for aggregation columns](#naming-conventions-for-aggregation-columns)- [Quantiles and sketch selection](#quantiles-and-sketch-selection)- [Validate effectiveness continuously](#validate-effectiveness-continously)- [Advanced configurations](#advanced-configurations)- [Limitations](#limitations)
                - [Common incompatibility reasons](#common-incompatibility-reasons)- [Design and operational constraints](#design-and-operational-constraints)- [Troubleshooting](#troubleshooting)
                  - [Materialized view not being used](#materialied-view-not-being-used)- [Slow materialized view queries](#slow-mv-queries)- [Configuration errors](#config-errors)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/migration/index.md)

This section provides comprehensive guides for migrating from various observability solutions to ClickStack. Each guide includes detailed instructions for transitioning your data, agents, and workflows while maintaining operational continuity.

|  |  |  |  |
| --- | --- | --- | --- |
| Technology Description|  |  | | --- | --- | | [Elastic Stack](/docs/use-cases/observability/clickstack/migration/elastic) Complete guide for migrating from Elastic Stack to ClickStack, covering data migration, agent transition, and search capabilities | | | |

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/migration/elastic/index.md)

This guide provides a comprehensive approach to migrating from Elastic Stack to ClickStack. We focus on a parallel operation strategy that minimizes risk while leveraging ClickHouse's strengths in observability workloads.

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Section Description|  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [Introduction](/docs/use-cases/observability/clickstack/migration/elastic/intro) Overview of the migration process and key considerations|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [Concepts](/docs/use-cases/observability/clickstack/migration/elastic/concepts) Understanding equivalent concepts between Elastic and ClickStack|  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [Types](/docs/use-cases/observability/clickstack/migration/elastic/types) Mapping Elasticsearch types to ClickHouse equivalents|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | [Search](/docs/use-cases/observability/clickstack/migration/elastic/search) Comparing search capabilities and query syntax|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | [Migrating Data](/docs/use-cases/observability/clickstack/migration/elastic/migrating-data) Strategies for data migration and parallel operation|  |  |  |  | | --- | --- | --- | --- | | [Migrating Agents](/docs/use-cases/observability/clickstack/migration/elastic/migrating-agents) Transitioning from Elastic agents to OpenTelemetry|  |  | | --- | --- | | [Migrating SDKs](/docs/use-cases/observability/clickstack/migration/elastic/migrating-sdks) Replacing Elastic APM agents with OpenTelemetry SDKs | | | | | | | | | | | | | | | |

Was this page helpful?
Migrating to ClickStack from Elastic

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/migration/elastic/intro.md)

## Migrating to ClickStack from Elastic[​](#migrating-to-clickstack-from-elastic "Direct link to Migrating to ClickStack from Elastic")

This guide is intended for users migrating from the Elastic Stack — specifically those using Kibana to monitor logs, traces, and metrics collected via Elastic Agent and stored in Elasticsearch. It outlines equivalent concepts and data types in ClickStack, explains how to translate Kibana Lucene-based queries to HyperDX's syntax, and provides guidance on migrating both data and agents for a smooth transition.

Before beginning a migration, it's important to understand the tradeoffs between ClickStack and the Elastic Stack.

You should consider moving to ClickStack if:

- You're ingesting large volumes of observability data and find Elastic cost-prohibitive due to inefficient compression and poor resource utilization. ClickStack can reduce storage and compute costs significantly — offering at least 10x compression on raw data.
- You experience poor search performance at scale or face ingestion bottlenecks.
- You want to correlate observability signals with business data using SQL, unifying observability and analytics workflows.
- You're committed to OpenTelemetry and want to avoid vendor lock-in.
- You want to take advantage of the separation of storage and compute in ClickHouse Cloud, enabling virtually unlimited scale — paying only for ingestion compute and object storage during idle periods.

However, ClickStack may not be suitable if:

- You use observability data primarily for security use cases and need a SIEM-focused product.
- Universal profiling is a critical part of your workflow.
- You require a business intelligence (BI) dashboarding platform. ClickStack intentionally has opinionated visual workflows for SREs and developers and isn't designed as a Business Intelligence (BI) tool. For equivalent capabilities, we recommend using [Grafana with the ClickHouse plugin](/docs/integrations/grafana) or [Superset](/docs/integrations/superset).

- [Migrating to ClickStack from Elastic](#migrating-to-clickstack-from-elastic)

Was this page helpful?
Searching in ClickStack and Elastic

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/migration/elastic/search.md)

## Search in ClickStack and Elastic[​](#search-in-clickstack-and-elastic "Direct link to Search in ClickStack and Elastic")

ClickHouse is a SQL-native engine, designed from the ground up for high-performance analytical workloads. In contrast, Elasticsearch provides a SQL-like interface, transpiling SQL into the underlying Elasticsearch query DSL — meaning it isn't a first-class citizen, and [feature parity](https://www.elastic.co/docs/explore-analyze/query-filter/languages/sql-limitations) is limited.

ClickHouse not only supports full SQL but extends it with a range of observability-focused functions, such as [`argMax`](/docs/sql-reference/aggregate-functions/reference/argmax), [`histogram`](/docs/sql-reference/aggregate-functions/parametric-functions#histogram), and [`quantileTiming`](/docs/sql-reference/aggregate-functions/reference/quantiletiming), that simplify querying structured logs, metrics, and traces.

For simple log and trace exploration, the ClickStack UI (HyperDX) provides a [Lucene-style syntax](/docs/use-cases/observability/clickstack/search) for intuitive, text-based filtering for field-value queries, ranges, wildcards, and more. This is comparable to the [Lucene syntax](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-syntax) in Elasticsearch and elements of the [Kibana Query Language](https://www.elastic.co/docs/reference/query-languages/kql).

The search interface supports this familiar syntax but translates it behind the scenes into efficient SQL `WHERE` clauses, making the experience familiar for Kibana users while still allowing users to leverage the power of SQL when needed. This allows you to exploit the full range of [string search functions](/docs/sql-reference/functions/string-search-functions), [similarity functions](/docs/sql-reference/functions/string-functions#stringJaccardIndex) and [date time functions](/docs/sql-reference/functions/date-time-functions) in ClickHouse.

Below, we compare the Lucene query languages of ClickStack and Elasticsearch.

## ClickStack search syntax vs Elasticsearch query string[​](#clickstack-vs-elasticsearch-query-string "Direct link to ClickStack search syntax vs Elasticsearch query string")

Both ClickStack and Elasticsearch provide flexible query languages to enable intuitive log and trace filtering. While Elasticsearch's query string is tightly integrated with its DSL and indexing engine, ClickStack supports a Lucene-inspired syntax that translates to ClickHouse SQL under the hood. The table below outlines how common search patterns behave across both systems, highlighting similarities in syntax and differences in backend execution.
Searching in ClickStack and Elastic

## Exists/missing differences[​](#empty-value-differences "Direct link to Exists/missing differences")

Unlike Elasticsearch, where a field can be entirely omitted from an event and therefore truly "not exist," ClickHouse requires all columns in a table schema to exist. If a field isn't provided in an insert event:

- For [`Nullable`](/docs/sql-reference/data-types/nullable) fields, it will be set to `NULL`.
- For non-nullable fields (the default), it will be populated with a default value (often an empty string, 0, or equivalent).

In ClickStack, we use the latter as [`Nullable`](/docs/sql-reference/data-types/nullable) is [not recommended](/docs/optimize/avoid-nullable-columns).

This behavior means that checking whether a field "exists”" in the Elasticsearch sense isn't directly supported.

Instead, you can use `field:*` or `field != ''` to check for the presence of a non-empty value. It is thus not possible to distinguish between truly missing and explicitly empty fields.

In practice, this difference rarely causes issues for observability use cases, but it's important to keep in mind when translating queries between systems.

- [Search in ClickStack and Elastic](#search-in-clickstack-and-elastic)- [ClickStack search syntax vs Elasticsearch query string](#clickstack-vs-elasticsearch-query-string)- [Exists/missing differences](#empty-value-differences)

Was this page helpful?
Mapping types

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/migration/elastic/types.md)

Elasticsearch and ClickHouse support a wide variety of data types, but their underlying storage and query models are fundamentally different. This section maps commonly used Elasticsearch field types to their ClickHouse equivalents, where available, and provides context to help guide migrations. Where no equivalent exists, alternatives or notes are provided in the comments.