ClickHouse/clickhouse-docs-embeddings huggingface dataset:
Mapping types

### Notes[​](#notes "Direct link to Notes")

- **Arrays**: In Elasticsearch, all fields support arrays natively. In ClickHouse, arrays must be explicitly defined (e.g., `Array(String)`), with the advantage specific positions can be accessed and queried e.g. `an_array[1]`.
- **Multi-fields**: Elasticsearch allows indexing the [same field multiple ways](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/multi-fields#_multi_fields_with_multiple_analyzers) (e.g., both `text` and `keyword`). In ClickHouse, this pattern must be modeled using separate columns or views.
- **Map and JSON Types** - In ClickHouse, the [`Map`](/docs/sql-reference/data-types/map) type is commonly used to model dynamic key-value structures such as `resourceAttributes` and `logAttributes`. This type enables flexible schema-less ingestion by allowing arbitrary keys to be added at runtime — similar in spirit to JSON objects in Elasticsearch. However, there are important limitations to consider:

  - **Uniform value types**: ClickHouse [`Map`](/docs/sql-reference/data-types/map) columns must have a consistent value type (e.g., `Map(String, String)`). Mixed-type values aren't supported without coercion.
  - **Performance cost**: accessing any key in a [`Map`](/docs/sql-reference/data-types/map) requires loading the entire map into memory, which can be suboptimal for performance.
  - **No subcolumns**: unlike JSON, keys in a [`Map`](/docs/sql-reference/data-types/map) aren't represented as true subcolumns, which limits ClickHouse’s ability to index, compress, and query efficiently.

  Because of these limitations, ClickStack is migrating away from [`Map`](/docs/sql-reference/data-types/map) in favor of ClickHouse's enhanced [`JSON`](/docs/sql-reference/data-types/newjson) type. The [`JSON`](/docs/sql-reference/data-types/newjson) type addresses many of the shortcomings of `Map`:

  - **True columnar storage**: each JSON path is stored as a subcolumn, allowing efficient compression, filtering, and vectorized query execution.
  - **Mixed-type support**: different data types (e.g., integers, strings, arrays) can coexist under the same path without coercion or type unification.
  - **File system scalability**: internal limits on dynamic keys (`max_dynamic_paths`) and types (`max_dynamic_types`) prevent an explosion of column files on disk, even with high cardinality key sets.
  - **Dense storage**: nulls and missing values are stored sparsely to avoid unnecessary overhead.

    The [`JSON`](/docs/sql-reference/data-types/newjson) type is especially well-suited for observability workloads, offering the flexibility of schemaless ingestion with the performance and scalability of native ClickHouse types — making it an ideal replacement for [`Map`](/docs/sql-reference/data-types/map) in dynamic attribute fields.

    For further details on the JSON type we recommend the [JSON guide](https://clickhouse.com/docs/integrations/data-formats/json/overview) and ["How we built a new powerful JSON data type for ClickHouse"](https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse).

- [Notes](#notes)

Was this page helpful?
ClickStack - The ClickHouse Observability Stack

Managed ClickStack provides several key benefits:

- **Automatic scaling of compute independent of storage**
- **Low-cost and effectively unlimited retention** backed by object storage
- **Independent read and write isolation** using ClickHouse Cloud Warehouses
- **Integrated authentication and access control**
- **Automated backups**
- **Security and compliance features**
- **Seamless upgrades with no operational downtime**

This deployment model allows teams to focus entirely on observability workflows and instrumentation, without the overhead of operating ClickHouse or the ClickStack UI themselves.

For users deploying ClickStack in production, Managed ClickStack is the recommended option. See the [Getting started guide](/docs/use-cases/observability/clickstack/getting-started/managed) for instructions on deploying ClickStack with ClickHouse Cloud.

Open Source ClickStack consists of three core components:

1. **ClickStack UI (HyperDX)**  
   A user-friendly interface built for observability. It supports both Lucene-style and SQL queries, interactive dashboards, alerting, trace exploration, and more—all optimized for ClickHouse as the backend.
2. **OpenTelemetry collector**  
   A custom-built collector configured with an opinionated schema optimized for ClickHouse ingestion. It receives logs, metrics, and traces via OpenTelemetry protocols and writes them directly to ClickHouse using efficient batched inserts.
3. **ClickHouse**  
   The high-performance analytical database that serves as the central data store for wide events. ClickHouse powers fast search, filtering, and aggregation at scale, leveraging its columnar engine and native support for JSON.

In addition to these three components, ClickStack uses a **MongoDB instance** to store application state such as dashboards, user accounts, and configuration settings.

A full architectural diagram and deployment details can be found in the [Architecture section](/docs/use-cases/observability/clickstack/architecture).

For users interesting in deploying Open Source ClickStack to production, we recommend reading the ["Production"](/docs/use-cases/observability/clickstack/production) guide.

- [Features](#features)- [Components](#components)- [Principles](#clickstack-principles)
      - [Easy to set up in minutes](#clickstack-easy-to-setup)- [User-friendly and purpose-built](#user-friendly-purpose-built)- [End-to-end observability](#end-to-end-observability)- [Built for ClickHouse](#built-for-clickhouse)- [OpenTelemetry-native](#open-telemetry-native)- [Open source and fully customizable](#open-source-and-customizable)- [Architectural overview](#architectural-overview)

Was this page helpful?
Sample Datasets

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/example-datasets/index.md)

This section provides various sample datasets and examples to help you get started with ClickStack. These examples demonstrate different ways to work with observability data in ClickStack, from local development to production scenarios.

|  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset Description|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | [Sample Data](/docs/use-cases/observability/clickstack/getting-started/sample-data) Load a sample dataset containing logs, traces and metrics from our demo environment|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | [Local Data](/docs/use-cases/observability/clickstack/getting-started/local-data) Collect local system metrics and logs sending them to ClickStack for analysis|  |  |  |  | | --- | --- | --- | --- | | [Remote Demo Data](/docs/use-cases/observability/clickstack/getting-started/remote-demo-data) Connect to our remote demo cluster and explore an issue|  |  | | --- | --- | | [Session Replay Demo](/docs/use-cases/observability/clickstack/example-datasets/session-replay-demo) Instrument a demo web application for session replay and view your interactions in ClickStack | | | | | | | | | |

Was this page helpful?
title_wip

```
# collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 'localhost:4317'
      http:
        endpoint: 'localhost:4318'

exporters:
  logging:
    verbosity: detailed
  otlphttp:
    endpoint: "<YOU_OTEL_COLLECTOR_HTTP_ENDPOINT>"
    headers:
      authorization: <YOUR_INGESTION_API_KEY>
    compression: gzip

service:
  telemetry:
    logs:
      level: "debug"
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, decouple]
      exporters: [otlphttp, logging]
    metrics:
      receivers: [otlp]
      processors: [batch, decouple]
      exporters: [otlphttp, logging]
    logs:
      receivers: [otlp]
      processors: [batch, decouple]
      exporters: [otlphttp, logging]
```

- [Installing the OpenTelemetry Lambda layers](#installing-the-otel-lambda-layers)
  - [Adding the language-specific auto-instrumentation layer](#adding-language-specific-auto-instrumentation)- [Installing the OpenTelemetry collector Lambda layer](#installing-the-otel-collector-layer)- [Checking the installation](#checking-the-installation)- [Troubleshooting](#troubleshoting)
      - [Custom instrumentation not sending](#custom-instrumentation-not-sending)- [Enabling SDK debug logs](#enabling-sdk-debug-logs)- [Enabling collector debug logs](#enabling-collector-debug-logs)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/ingesting-data/sdks/deno.md)

This guide Integrates the following:

- **Logs**

Note

Currently only supports OpenTelemetry Logging. For tracing support, [see the following guide](https://dev.to/grunet/leveraging-opentelemetry-in-deno-45bj#a-minimal-interesting-example).

## Logging[​](#logging "Direct link to Logging")

Logging is supported by exporting a custom logger for the `std/log` module.

**Example usage:**

```
import * as log from 'https://deno.land/[email protected]/log/mod.ts';
import { OpenTelemetryHandler } from 'npm:@hyperdx/deno';

log.setup({
  handlers: {
    otel: new OpenTelemetryHandler('DEBUG'),
  },

  loggers: {
    'my-otel-logger': {
      level: 'DEBUG',
      handlers: ['otel'],
    },
  },
});

log.getLogger('my-otel-logger').info('Hello from Deno!');
```

### Run the application[​](#run-the-application "Direct link to Run the application")

- Managed ClickStack- ClickStack Open Source

```
OTEL_EXPORTER_OTLP_ENDPOINT=http://your-otel-collector:4318 \
OTEL_SERVICE_NAME="<NAME_OF_YOUR_APP_OR_SERVICE>" \
deno run --allow-net --allow-env --allow-read --allow-sys --allow-run app.ts
```

```
OTEL_EXPORTER_OTLP_HEADERS="authorization=<YOUR_INGESTION_API_KEY>" \
OTEL_EXPORTER_OTLP_ENDPOINT=http://your-otel-collector:4318 \
OTEL_SERVICE_NAME="<NAME_OF_YOUR_APP_OR_SERVICE>" \
deno run --allow-net --allow-env --allow-read --allow-sys --allow-run app.ts
```

- [Logging](#logging)
  - [Run the application](#run-the-application)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/ingesting-data/sdks/elixir.md)

|  |  |  |
| --- | --- | --- |
| ✅ Logs ✖️ Metrics ✖️ Traces | | |

*🚧 OpenTelemetry metrics & tracing instrumentation coming soon!*

## Getting started[​](#getting-started "Direct link to Getting started")

### Install ClickStack logger backend package[​](#install-hyperdx-logger-backend-package "Direct link to Install ClickStack logger backend package")

The package can be installed by adding `hyperdx` to your list of dependencies in
`mix.exs`:

```
def deps do
  [
    {:hyperdx, "~> 0.1.6"}
  ]
end
```

### Configure logger[​](#configure-logger "Direct link to Configure logger")

Add the following to your `config.exs` file:

```
# config/releases.exs

config :logger,
  level: :info,
  backends: [:console, {Hyperdx.Backend, :hyperdx}]
```

### Configure environment variables[​](#configure-environment-variables "Direct link to Configure environment variables")

Afterwards you'll need to configure the following environment variables in your
shell to ship telemetry to ClickStack via the OpenTelemetry collector:

- Managed ClickStack- ClickStack Open Source

```
OTEL_SERVICE_NAME='<NAME_OF_YOUR_APP_OR_SERVICE>'
```

```
export HYPERDX_API_KEY='<YOUR_INGESTION_API_KEY>' \
OTEL_SERVICE_NAME='<NAME_OF_YOUR_APP_OR_SERVICE>'
```

*The `OTEL_SERVICE_NAME` environment variable is used to identify your service
in the HyperDX app, it can be any name you want.*

- [Getting started](#getting-started)
  - [Install ClickStack logger backend package](#install-hyperdx-logger-backend-package)- [Configure logger](#configure-logger)- [Configure environment variables](#configure-environment-variables)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/ingesting-data/sdks/java.md)

ClickStack uses the OpenTelemetry standard for collecting telemetry data (logs and
traces). Traces are auto-generated with automatic instrumentation, so manual
instrumentation isn't required to get value out of tracing.

**This guide Integrates:**

|  |  |  |
| --- | --- | --- |
| ✅ Logs ✅ Metrics ✅ Traces | | |

## Getting started[​](#getting-started "Direct link to Getting started")

Note

At present, the integration is compatible exclusively with **Java 8+**

### Download OpenTelemetry Java agent[​](#download-opentelemetry-java-agent "Direct link to Download OpenTelemetry Java agent")

Download [`opentelemetry-javaagent.jar`](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar)
and place the JAR in your preferred directory. The JAR file contains the agent
and instrumentation libraries. You can also use the following command to
download the agent:

```
curl -L -O https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
```

### Configure environment variables[​](#configure-environment-variables "Direct link to Configure environment variables")

Afterwards you'll need to configure the following environment variables in your shell to ship telemetry to ClickStack via the OpenTelemetry collector:

- Managed ClickStack- ClickStack Open Source

```
export JAVA_TOOL_OPTIONS="-javaagent:PATH/TO/opentelemetry-javaagent.jar" \
OTEL_EXPORTER_OTLP_ENDPOINT=https://your-otel-collector:4318 \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_LOGS_EXPORTER=otlp \
OTEL_SERVICE_NAME='<NAME_OF_YOUR_APP_OR_SERVICE>'
```

```
export JAVA_TOOL_OPTIONS="-javaagent:PATH/TO/opentelemetry-javaagent.jar" \
OTEL_EXPORTER_OTLP_ENDPOINT=https://your-otel-collector:4318 \
OTEL_EXPORTER_OTLP_HEADERS='authorization=<YOUR_INGESTION_API_KEY>' \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_LOGS_EXPORTER=otlp \
OTEL_SERVICE_NAME='<NAME_OF_YOUR_APP_OR_SERVICE>'
```

*The `OTEL_SERVICE_NAME` environment variable is used to identify your service in the HyperDX app, it can be any name you want.*

The `OTEL_EXPORTER_OTLP_HEADERS` environment variable contains the API Key available via HyperDX app in `Team Settings → API Keys`.

### Run the application with OpenTelemetry Java agent[​](#run-the-application-with-otel-java-agent "Direct link to Run the application with OpenTelemetry Java agent")

```
java -jar target/<APPLICATION_JAR_FILE>
```

  

Read more about Java OpenTelemetry instrumentation here: <https://opentelemetry.io/docs/instrumentation/java/>

- [Getting started](#getting-started)
  - [Download OpenTelemetry Java agent](#download-opentelemetry-java-agent)- [Configure environment variables](#configure-environment-variables)- [Run the application with OpenTelemetry Java agent](#run-the-application-with-otel-java-agent)

Was this page helpful?
Self-managed monitoring

## Community monitoring solutions[​](#community-monitoring "Direct link to Community monitoring solutions")

The ClickHouse community has developed comprehensive monitoring solutions that integrate with popular observability stacks. [ClickHouse Monitoring](https://github.com/duyet/clickhouse-monitoring) provides a complete monitoring setup with pre-built dashboards. This open source project offers a quick-start approach for teams looking to implement ClickHouse monitoring with established best practices and proven dashboard configurations.

Note

Like other direct database monitoring approaches, this solution queries ClickHouse system tables directly, which prevents instances from idling and impacts cost optimization.

- [Prometheus-based integration architecture](#prometheus)- [Integration examples](#examples)
    - [Grafana Cloud monitoring](#grafana)- [Datadog monitoring](#datadog)- [ClickStack](#clickstack)- [ClickStack deployment options](#clickstack-deployment)- [Direct Grafana plugin integration](#direct-grafana)- [Direct Datadog Integration](#direct-datadog)- [Using system tables directly](#system-tables)- [Community monitoring solutions](#community-monitoring)

Was this page helpful?
Designing a schema for observability

## Measuring table size & compression[​](#measuring-table-size--compression "Direct link to Measuring table size & compression")

One of the principal reasons ClickHouse is used for Observability is compression.

As well as dramatically reducing storage costs, less data on disk means less I/O and faster queries and inserts. The reduction in IO will out weight the overhead of any compression algorithm with respect to CPU. Improving the compression of the data should therefore be the first focus when working on ensuring ClickHouse queries are fast.

Details on measuring compression can be found [here](/docs/data-compression/compression-in-clickhouse).

- [Extracting structure with SQL](#extracting-structure-with-sql)
  - [Materialized columns](#materialized-columns)- [Materialized views](#materialized-views)
    - [Beware types](#beware-types)- [Choosing a primary (ordering) key](#choosing-a-primary-ordering-key)- [Using maps](#using-maps)- [Using aliases](#using-aliases)- [Optimizing types](#optimizing-types)- [Using codecs](#using-codecs)- [Using dictionaries](#using-dictionaries)
                - [Insert time vs query time](#insert-time-vs-query-time)- [Using IP dictionaries](#using-ip-dictionaries)- [Using regex dictionaries (user agent parsing)](#using-regex-dictionaries-user-agent-parsing)- [Further reading](#further-reading)- [Accelerating queries](#accelerating-queries)
                  - [Using Materialized views (incremental) for aggregations](#using-materialized-views-incremental-for-aggregations)- [Using Materialized views (incremental) for fast lookups](#using-materialized-views-incremental--for-fast-lookups)- [Using projections](#using-projections)- [Secondary/data skipping indices](#secondarydata-skipping-indices)- [Text index for full text search](#text-index-for-full-text-search)- [Extracting from maps](#extracting-from-maps)- [Measuring table size & compression](#measuring-table-size--compression)

Was this page helpful?
title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/time-series/index.md)

Welcome to our time-series use case guide. In this guide you'll learn how you can set up and use ClickHouse for time-series data.

Time-series data is everywhere in modern analytics. From system metrics and application logs to business events and sensor readings, data points collected over time help us understand trends, patterns, and anomalies in our systems and processes.

ClickHouse excels at handling time-series data, offering powerful features for both storage and analysis. Whether you're building a simple monitoring dashboard or processing petabytes of sensor data in real-time, ClickHouse provides the tools and performance you need.

This guide walks you through everything you need to know about working with time-series data in ClickHouse, from basic concepts to advanced optimization techniques. You'll learn how to:

- [Choose the right date and time data types for your use case](/docs/use-cases/time-series/date-time-data-types)
- [Perform common time-series operations and aggregations](/docs/use-cases/time-series/basic-operations)
- [Apply specialized analysis functions for time-based data](/docs/use-cases/time-series/analysis-functions)
- [Optimize storage efficiency for temporal data](/docs/use-cases/time-series/storage-efficiency)
- [Tune query performance for time-series workloads](/docs/use-cases/time-series/query-performance)

Whether you're new to time-series analysis or looking to optimize an existing implementation, this guide will help you make the most of ClickHouse's time-series capabilities.

Was this page helpful?
title_wip

For example, we can use [`toDate`](/docs/sql-reference/functions/type-conversion-functions#toDate) to convert a `DateTime` value to the `Date` type:

```
SELECT
    now() AS current_time,
    toTypeName(current_time),
    toDate(current_time) AS date_only,
    toTypeName(date_only)
FORMAT Vertical;
```

```
Row 1:
──────
current_time:             2025-03-12 12:32:54
toTypeName(current_time): DateTime
date_only:                2025-03-12
toTypeName(date_only):    Date
```

We can use [`toDateTime64`](/docs/sql-reference/functions/type-conversion-functions#toDateTime64) to convert `DateTime` to `DateTime64`:

```
SELECT
    now() AS current_time,
    toTypeName(current_time),
    toDateTime64(current_time, 3) AS date_only,
    toTypeName(date_only)
FORMAT Vertical;
```

```
Row 1:
──────
current_time:             2025-03-12 12:35:01
toTypeName(current_time): DateTime
date_only:                2025-03-12 12:35:01.000
toTypeName(date_only):    DateTime64(3)
```

And we can use [`toDateTime`](/docs/sql-reference/functions/type-conversion-functions#toDateTime) to go from `Date` or `DateTime64` back to `DateTime`:

```
SELECT
    now64() AS current_time,
    toTypeName(current_time),
    toDateTime(current_time) AS date_time1,
    toTypeName(date_time1),
    today() AS current_date,
    toTypeName(current_date),
    toDateTime(current_date) AS date_time2,
    toTypeName(date_time2)
FORMAT Vertical;
```

```
Row 1:
──────
current_time:             2025-03-12 12:41:00.598
toTypeName(current_time): DateTime64(3)
date_time1:               2025-03-12 12:41:00
toTypeName(date_time1):   DateTime
current_date:             2025-03-12
toTypeName(current_date): Date
date_time2:               2025-03-12 00:00:00
toTypeName(date_time2):   DateTime
```

- [Time and Time64 types](#time-series-time-types)- [Timezones](#time-series-timezones)- [Date and time functions](#time-series-date-time-functions)

Was this page helpful?
2020 Changelog

#### Build/Testing/Packaging Improvement[​](#buildtestingpackaging-improvement-2 "Direct link to Build/Testing/Packaging Improvement")

- Update timezones info to 2020e. [#18531](https://github.com/ClickHouse/ClickHouse/pull/18531) ([alesapin](https://github.com/alesapin)).

### ClickHouse release v20.11.6.6-stable, 2020-12-24[​](#clickhouse-release-v201166-stable-2020-12-24 "Direct link to ClickHouse release v20.11.6.6-stable, 2020-12-24")

#### Bug Fix[​](#bug-fix-4 "Direct link to Bug Fix")