## 4.3.1.8.5 Traces
The Telemetry module supports you in collecting all relevant trace data in a Kyma cluster, enriches them and ships them to a backend for further analysis. Kyma modules like Istio or Serverless contribute traces transparently. You can choose among multiple vendors for OTLP-based backends .
## Overview
Observability tools aim to show the big picture, no matter if you're monitoring just a few or many components. In a cloud-native microservice architecture, a user request often flows through dozens of different microservices. Logging and monitoring tools help to track the request's path. However, they treat each component or microservice in isolation. This individual treatment results in operational issues.
Distributed tracing charts out the transactions in cloud-native systems, helping you to understand the application behavior and relations between the frontend actions and backend implementation.
The following diagram shows how distributed tracing helps to track the request path:
<!-- image -->
The Telemetry module provides a trace gateway for the shipment of traces of any container running in the Kyma runtime.
You can configure the trace gateway with external systems using runtime configuration with a dedicated Kubernetes API (CRD ) named TracePipeline .
The Trace feature is optional. If you don't want to use it, simply don't set up a TracePipeline .
## Prerequisites
For the recording of a distributed trace, every involved component must propagate at least the trace context. For details, see Trace Context .
- In Kyma, all modules involved in users' requests support the W3C Trace Context protocol. The involved Kyma modules are, for example, Istio, Serverless, and Eventing.
- Your application also must propagate the W3C Trace Context for any user-related activity. This can be achieved easily using the Open Telemetry SDKs available for all common programming languages. If your application follows that guidance and is part of the Istio Service Mesh, it's already outlined with dedicated span data in the trace data collected by the Kyma telemetry setup.
- Furthermore, your application must enrich a trace with additional span data and send these data to the cluster-central telemetry services. You can achieve this with Open Telemetry SDKs .
## Architecture
In the Kyma cluster, the Telemetry module provides a central deployment of an OTel Collector acting as a gateway. The gateway exposes endpoints to which all Kyma modules and users' applications should send the trace data.
<!-- image -->
1. An end-to-end request is triggered and populated across the distributed application. Every involved component propagates the trace context using the W3C Trace Context protocol.
2. After contributing a new span to the trace, the involved components send the related span data to the trace gateway using the telemetry-otlp-traces service. The communication happens based on the OpenTelemetry Protocol (OTLP) either using GRPC or HTTP.
3. Istio sends the related span data to the trace gateway as well.
4. The trace gateway discovers metadata that's typical for sources running on Kubernetes, like Pod identifiers, and then enriches the span data with that metadata.
5. Telemetry Manager configures the gateway according to the TracePipeline resource, including the target backend for the trace gateway. Also, it observes the trace flow to the backend and reports problems in the TracePipeline status.
6. The trace gateway sends the data to the observability system that's specified in your TracePipeline resource - either within the Kyma cluster, or, if authentication is set up, to an external observability backend.
7. You can analyze the trace data with your preferred backend system.
## Telemetry Manager
The TracePipeline resource is watched by Telemetry Manager, which is responsible for generating the custom parts of the OTel Collector configuration.
<!-- image -->
1. Telemetry Manager watches all TracePipeline resources and related Secrets.
2. Furthermore, Telemetry Manager takes care of the full lifecycle of the OTel Collector Deployment itself. Only if you defined a TracePipeline , the collector is deployed.
3. Whenever the configuration changes, it validates the configuration and generates a new configuration for OTel Collector, where a ConfigMap for the configuration is generated.
4. Referenced Secrets are copied into one Secret that is mounted to the OTel Collector as well.
## Trace Gateway
In a Kyma cluster, the trace gateway is the central component to which all components can send their individual spans. The gateway collects, enriches, and dispatches the data to the configured backend. For more information, see Telemetry Gateways [page 1953].
## Setting up a TracePipeline
In the following steps, you can see how to construct and deploy a typical TracePipeline . Learn more about the available parameters and attributes .