[Edit this page](https://github.com/ClickHouse/ClickHouse/tree/master/docs/en/operations/opentelemetry.md)

[OpenTelemetry](https://opentelemetry.io/) is an open standard for collecting traces and metrics from the distributed application. ClickHouse has some support for OpenTelemetry.

## Supplying trace context to ClickHouse[​](#supplying-trace-context-to-clickhouse "Direct link to Supplying trace context to ClickHouse")

ClickHouse accepts trace context HTTP headers, as described by the [W3C recommendation](https://www.w3.org/TR/trace-context/). It also accepts trace context over a native protocol that is used for communication between ClickHouse servers or between the client and server. For manual testing, trace context headers conforming to the Trace Context recommendation can be supplied to `clickhouse-client` using `--opentelemetry-traceparent` and `--opentelemetry-tracestate` flags.

If no parent trace context is supplied or the provided trace context does not comply with W3C standard above, ClickHouse can start a new trace, with probability controlled by the [opentelemetry\_start\_trace\_probability](/docs/operations/settings/settings#opentelemetry_start_trace_probability) setting.

## Propagating the trace context[​](#propagating-the-trace-context "Direct link to Propagating the trace context")

The trace context is propagated to downstream services in the following cases:

- Queries to remote ClickHouse servers, such as when using [Distributed](/docs/engines/table-engines/special/distributed) table engine.
- [url](/docs/sql-reference/table-functions/url) table function. Trace context information is sent in HTTP headers.

## Tracing ClickHouse Keeper Requests[​](#tracing-clickhouse-keeper-requests "Direct link to Tracing ClickHouse Keeper Requests")

ClickHouse supports OpenTelemetry tracing for [ClickHouse Keeper](/docs/guides/sre/keeper/clickhouse-keeper) requests (ZooKeeper-compatible coordination service). This feature provides detailed visibility into the lifecycle of Keeper operations, from client request submission through server-side processing.

### Enabling Keeper Tracing[​](#enabling-keeper-tracing "Direct link to Enabling Keeper Tracing")

To enable tracing for Keeper requests, configure the following settings in your ZooKeeper/Keeper client configuration:

```
<clickhouse>
    <zookeeper>
        <node>
            <host>keeper1</host>
            <port>9181</port>
        </node>
        <!-- Enable OpenTelemetry tracing context propagation -->
        <pass_opentelemetry_tracing_context>true</pass_opentelemetry_tracing_context>
    </zookeeper>
</clickhouse>
```

### Keeper Span Types[​](#keeper-span-types "Direct link to Keeper Span Types")

When tracing is enabled, ClickHouse creates spans for both client-side and server-side Keeper operations:

**Client-side spans:**

- `zookeeper.create` — Create a new node
- `zookeeper.get` — Get node data
- `zookeeper.set` — Set node data
- `zookeeper.remove` — Remove a node
- `zookeeper.list` — List child nodes
- `zookeeper.exists` — Check if a node exists
- `zookeeper.multi` — Execute multiple operations atomically
- `zookeeper.client.requests_queue` — Time spent queueing requests before sending

**Server-side spans (Keeper):**

- `keeper.receive_request` — Receiving and parsing the request from the client
- `keeper.dispatcher.requests_queue` — Request queuing in the dispatcher
- `keeper.write.pre_commit` — Preprocessing write requests before Raft commit
- `keeper.write.commit` — Processing write requests after Raft commit
- `keeper.read.wait_for_write` — Read requests waiting for dependent writes
- `keeper.read.process` — Processing read requests
- `keeper.dispatcher.responses_queue` — Response queuing in the dispatcher
- `keeper.send_response` — Sending the response to the client

### Sampling and Performance[​](#sampling-and-performance "Direct link to Sampling and Performance")

To manage tracing overhead, Keeper implements dynamic sampling. Sampling rate automatically adjusts between 1/10,000 and 1/10 based on request size. All requests (sampled and unsampled) have their durations recorded to histogram metrics for performance monitoring.

## Tracing the ClickHouse Itself[​](#tracing-the-clickhouse-itself "Direct link to Tracing the ClickHouse Itself")

ClickHouse creates `trace spans` for each query and some of the query execution stages, such as query planning or distributed queries.

To be useful, the tracing information has to be exported to a monitoring system that supports OpenTelemetry, such as [Jaeger](https://jaegertracing.io/) or [Prometheus](https://prometheus.io/). ClickHouse avoids a dependency on a particular monitoring system, instead only providing the tracing data through a system table. OpenTelemetry trace span information [required by the standard](https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/overview.md#span) is stored in the [system.opentelemetry\_span\_log](/docs/operations/system-tables/opentelemetry_span_log) table.

The table must be enabled in the server configuration, see the `opentelemetry_span_log` element in the default config file `config.xml`. It is enabled by default.

The tags or attributes are saved as two parallel arrays, containing the keys and values. Use [ARRAY JOIN](/docs/sql-reference/statements/select/array-join) to work with them.

## Log-query-settings[​](#log-query-settings "Direct link to Log-query-settings")

Setting [log\_query\_settings](/docs/operations/settings/settings) allows log changes to query settings during query execution. When enabled, any modifications made to query settings will be recorded in the OpenTelemetry span log. This feature is particularly useful in production environments for tracking configuration changes that may affect query performance.

## Integration with monitoring systems[​](#integration-with-monitoring-systems "Direct link to Integration with monitoring systems")

At the moment, there is no ready tool that can export the tracing data from ClickHouse to a monitoring system.

For testing, it is possible to setup the export using a materialized view with the [URL](/docs/engines/table-engines/special/url) engine over the [system.opentelemetry\_span\_log](/docs/operations/system-tables/opentelemetry_span_log) table, which would push the arriving log data to an HTTP endpoint of a trace collector. For example, to push the minimal span data to a Zipkin instance running at `http://localhost:9411`, in Zipkin v2 JSON format:

```
CREATE MATERIALIZED VIEW default.zipkin_spans
ENGINE = URL('http://127.0.0.1:9411/api/v2/spans', 'JSONEachRow')
SETTINGS output_format_json_named_tuples_as_objects = 1,
    output_format_json_array_of_rows = 1 AS
SELECT
    lower(hex(trace_id)) AS traceId,
    CASE WHEN parent_span_id = 0 THEN '' ELSE lower(hex(parent_span_id)) END AS parentId,
    lower(hex(span_id)) AS id,
    operation_name AS name,
    start_time_us AS timestamp,
    finish_time_us - start_time_us AS duration,
    cast(tuple('clickhouse'), 'Tuple(serviceName text)') AS localEndpoint,
    cast(tuple(
        attribute.values[indexOf(attribute.names, 'db.statement')]),
        'Tuple("db.statement" text)') AS tags
FROM system.opentelemetry_span_log
```

In case of any errors, the part of the log data for which the error has occurred will be silently lost. Check the server log for error messages if the data does not arrive.

## Related content[​](#related-content "Direct link to Related content")

- Blog: [Building an Observability Solution with ClickHouse - Part 2 - Traces](https://clickhouse.com/blog/storing-traces-and-spans-open-telemetry-in-clickhouse)