Integrating OpenTelemetry for data collection

However, ClickHouse can handle inserting data very quickly - millions of rows per second on moderate hardware. Back pressure from ClickHouse is **rare**. Often, leveraging a Kafka queue means more architectural complexity and cost. If you can embrace the principle that logs don't need the same delivery guarantees as bank transactions and other mission-critical data, we recommend avoiding the complexity of Kafka.

However, if you require high delivery guarantees or the ability to replay data (potentially to multiple sources), Kafka can be a useful architectural addition.

In this case, OTel agents can be configured to send data to Kafka via the [Kafka exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/kafkaexporter/README.md). Gateway instances, in turn, consume messages using the [Kafka receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kafkareceiver/README.md). We recommend the Confluent and OTel documentation for further details.

### Estimating resources[​](#estimating-resources "Direct link to Estimating resources")

Resource requirements for the OTel collector will depend on the event throughput, the size of messages and amount of processing performed. The OpenTelemetry project maintains [benchmarks users](https://opentelemetry.io/docs/collector/benchmarks/) can use to estimate resource requirements.

[In our experience](https://clickhouse.com/blog/building-a-logging-platform-with-clickhouse-and-saving-millions-over-datadog#architectural-overview), a gateway instance with 3 cores and 12GB of RAM can handle around 60k events per second. This assumes a minimal processing pipeline responsible for renaming fields and no regular expressions.

For agent instances responsible for shipping events to a gateway, and only setting the timestamp on the event, we recommend users size based on the anticipated logs per second. The following represent approximate numbers you can use as a starting point:

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Logging rate Resources to collector agent|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | 1k/second 0.2CPU, 0.2GiB|  |  |  |  | | --- | --- | --- | --- | | 5k/second 0.5 CPU, 0.5GiB|  |  | | --- | --- | | 10k/second 1 CPU, 1GiB | | | | | | | |

- [ClickHouse relevant components](#clickhouse-relevant-components)- [Distributions](#distributions)- [Ingesting data with OTel](#ingesting-data-with-otel)
      - [Collector deployment roles](#collector-deployment-roles)- [Collecting logs](#collecting-logs)- [Structured vs unstructured](#structured-vs-unstructured)
        - [Example](#example)- [Collecting Kubernetes logs](#collecting-kubernetes-logs)- [Collecting traces](#collecting-traces)
            - [Example](#example-1)- [Processing - filtering, transforming and enriching](#processing---filtering-transforming-and-enriching)
              - [Example](#example-2)- [Exporting to ClickHouse](#exporting-to-clickhouse)- [Out of the box schema](#out-of-the-box-schema)- [Optimizing inserts](#optimizing-inserts)
                    - [Batching](#batching)- [Use asynchronous inserts](#use-asynchronous-inserts)- [Deployment architectures](#deployment-architectures)
                      - [Agents only](#agents-only)- [Scaling with gateways](#scaling-with-gateways)- [Adding Kafka](#adding-kafka)- [Estimating resources](#estimating-resources)

Was this page helpful?