ClickStack OpenTelemetry collector

However, if you require high delivery guarantees or the ability to replay data (potentially to multiple sources), Kafka can be a useful architectural addition.

In this case, OTel agents can be configured to send data to Kafka via the [Kafka exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/kafkaexporter/README.md). Gateway instances, in turn, consume messages using the [Kafka receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kafkareceiver/README.md). We recommend the Confluent and OTel documentation for further details.

OTel collector configuration

The ClickStack OpenTelemetry collector distribution can be configured with Kafka using [custom collector configuration](#extending-collector-config).

## Estimating resources[​](#estimating-resources "Direct link to Estimating resources")

Resource requirements for the OTel collector will depend on the event throughput, the size of messages and amount of processing performed. The OpenTelemetry project maintains [benchmarks users](https://opentelemetry.io/docs/collector/benchmarks/) can use to estimate resource requirements.

[In our experience](https://clickhouse.com/blog/building-a-logging-platform-with-clickhouse-and-saving-millions-over-datadog#architectural-overview), a ClickStack gateway instance with 3 cores and 12GB of RAM can handle around 60k events per second. This assumes a minimal processing pipeline responsible for renaming fields and no regular expressions.

For agent instances responsible for shipping events to a gateway, and only setting the timestamp on the event, we recommend users size based on the anticipated logs per second. The following represent approximate numbers you can use as a starting point:

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Logging rate Resources to collector agent|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | 1k/second 0.2CPU, 0.2GiB|  |  |  |  | | --- | --- | --- | --- | | 5k/second 0.5 CPU, 0.5GiB|  |  | | --- | --- | | 10k/second 1 CPU, 1GiB | | | | | | | |

## JSON support[​](#json-support "Direct link to JSON support")

Beta feature. [Learn more.](/docs/beta-and-experimental-features#beta-features)

ClickStack has beta support for the [JSON type](/docs/interfaces/formats/JSON) from version `2.0.4`.

Beta Feature

JSON type support in **ClickStack** is a **beta feature**. While the JSON type itself is production-ready in ClickHouse 25.3+, its integration within ClickStack is still under active development and may have limitations, change in the future, or contain bugs

### Benefits of the JSON type[​](#benefits-json-type "Direct link to Benefits of the JSON type")

The JSON type offers the following benefits to ClickStack users:

- **Type preservation** - Numbers stay numbers, booleans stay booleans—no more flattening everything into strings. This means fewer casts, simpler queries, and more accurate aggregations.
- **Path-level columns** - Each JSON path becomes its own sub-column, reducing I/O. Queries only read the fields they need, unlocking major performance gains over the old Map type which required the entire column to be read in order to query a specific field.
- **Deep nesting just works** - Naturally handle complex, deeply nested structures without manual flattening (as required by the Map type) and subsequent awkward JSONExtract functions.
- **Dynamic, evolving schemas** - Perfect for observability data where teams add new tags and attributes over time. JSON handles these changes automatically, without schema migrations.
- **Faster queries, lower memory** - Typical aggregations over attributes like `LogAttributes` see 5-10x less data read and dramatic speedups, cutting both query time and peak memory usage.
- **Simple management** - No need to pre-materialize columns for performance. Each field becomes its own sub-column, delivering the same speed as native ClickHouse columns.

### Enabling JSON support[​](#enabling-json-support "Direct link to Enabling JSON support")

- Managed ClickStack- Open Source ClickStack

To enable JSON support in Managed ClickStack, please contact our support team prior to configuring the collector below. **The feature must also be enabled in the ClickStack UI (HyperDX) in ClickHouse Cloud.**

To enable this support for your collector, set the environment variable `OTEL_AGENT_FEATURE_GATE_ARG='--feature-gates=clickhouse.json'`. This ensures the schemas are created in ClickHouse using the JSON type.

For example:

```
docker run -e OTEL_AGENT_FEATURE_GATE_ARG='--feature-gates=clickhouse.json' -e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} -e CLICKHOUSE_USER=default -e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} -p 8080:8080 -p 4317:4317 -p 4318:4318 clickhouse/clickstack-otel-collector:latest
```

To enable this support for the collector, set the environment variable `OTEL_AGENT_FEATURE_GATE_ARG='--feature-gates=clickhouse.json'` on any deployment that includes the collector. This ensures the schemas are created in ClickHouse using the JSON type.

HyperDX support

In order to query the JSON type, support must also be enabled in the HyperDX application layer via the environment variable `BETA_CH_OTEL_JSON_SCHEMA_ENABLED=true`.

For example:

```
docker run -e OTEL_AGENT_FEATURE_GATE_ARG='--feature-gates=clickhouse.json' -e OPAMP_SERVER_URL=${OPAMP_SERVER_URL} -e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} -e CLICKHOUSE_USER=default -e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} -p 8080:8080 -p 4317:4317 -p 4318:4318 clickhouse/clickstack-otel-collector:latest
```

### Migrating from map-based schemas to the JSON type[​](#migrating-from-map-based-schemas-to-json "Direct link to Migrating from map-based schemas to the JSON type")

Backwards compatibility

The [JSON type](/docs/interfaces/formats/JSON) is **not backwards compatible** with existing map-based schemas. Enabling this feature will create new tables using the `JSON` type and requires manual data migration.

To migrate from the Map-based schemas, follow these steps:

Stop the OTel collector

#### Stop the OTel collector[​](#stop-the-collector "Direct link to Stop the OTel collector")

Rename existing tables and update sources

#### Rename existing tables and update sources[​](#rename-existing-tables-sources "Direct link to Rename existing tables and update sources")

Rename existing tables and update data sources in HyperDX.

For example:

```
RENAME TABLE otel_logs TO otel_logs_map;
RENAME TABLE otel_metrics TO otel_metrics_map;
```

Deploy the collector

#### Deploy the collector[​](#deploy-the-collector "Direct link to Deploy the collector")

Deploy the collector with `OTEL_AGENT_FEATURE_GATE_ARG` set.

Restart the HyperDX container with JSON schema support

#### Restart the HyperDX container with JSON schema support[​](#restart-the-hyperdx-container "Direct link to Restart the HyperDX container with JSON schema support")

```
export BETA_CH_OTEL_JSON_SCHEMA_ENABLED=true
```

Create new data sources

#### Create new data sources[​](#create-new-data-sources "Direct link to Create new data sources")

Create new data sources in HyperDX pointing to the JSON tables.

#### Migrating existing data (optional)[​](#migrating-existing-data "Direct link to Migrating existing data (optional)")

To move old data into the new JSON tables:

```
INSERT INTO otel_logs SELECT * FROM otel_logs_map;
INSERT INTO otel_metrics SELECT * FROM otel_metrics_map;
```

Note

Recommended only for datasets smaller than ~10 billion rows. Data previously stored with the Map type didn't preserve type precision (all values were strings). As a result, this old data will appear as strings in the new schema until it ages out, requiring some casting on the frontend. Type for new data will be preserved with the JSON type.