## 5. Deploy the Pipeline
To activate the TracePipeline , apply the tracepipeline.yaml resource file in your cluster:
```
kubectl apply -f tracepipeline.yaml
```
## Result
You activated a TracePipeline and traces start streaming to your backend.
To check that the pipeline is running, wait until the status conditions of the TracePipeline in your cluster have status True :
```
kubectl get tracepipeline NAME      CONFIGURATION GENERATED   GATEWAY HEALTHY   FLOW HEALTHY backend   True                      True              True
```
## Kyma Modules With Tracing Capabilities
Kyma bundles several modules that can be involved in user flows. Applications involved in a distributed trace must propagate the trace context to keep the trace complete. Optionally, they can enrich the trace with custom spans, which requires reporting them to the backend.
## Istio
The Istio Module [page 1778] is crucial in distributed tracing because it provides the ingress gateway . Typically, this is where external requests enter the cluster scope and are enriched with trace context if it hasn't happened earlier. Furthermore, every component that's part of the Istio Service Mesh runs an Istio proxy, which propagates the context properly but also creates span data.
If Istio tracing is activated and taking care of trace propagation in your application, you get a complete picture of a trace, because every component automatically contributes span data. Also, Istio tracing is pre-configured to be based on the vendor-neutral W3C Trace Context protocol.
The Istio module is configured with an extension provider called kyma-traces . T o activate the provider on the global mesh level using the Istio Telemetry API , place a resource to the istio-system namespace. The following code samples help setting up the Istio tracing feature:
- The following example configures all Istio proxies with the kyma-traces extension provider, which, by default, reports span data to the trace gateway of the Telemetry module.
- Sampling rate: By default, the sampling rate is configured to 1%. That means that only 1 trace out of 100 traces is reported to the trace gateway, and all others are dropped. The sampling decision itself is propagated as part of the trace context so that either all involved components are reporting the span data of a trace, or none.
```
apiVersion: telemetry.istio.io/v1 kind: Telemetry metadata: name: tracing-default namespace: istio-system spec: tracing: - providers: - name: "kyma-traces"
```
<!-- image -->
If you increase the sampling rate, you send more data your tracing backend and cause much higher network utilization in the cluster. To reduce costs and performance impacts in a production setup, a very low percentage of around 5% is recommended.
To configure an 'always-on' sampling, set the sampling rate to 100%:
```
apiVersion: telemetry.istio.io/v1 kind: Telemetry metadata: name: tracing-default namespace: istio-system spec: tracing: - providers: - name: "kyma-traces" randomSamplingPercentage: 100.00
```
- Namespaces or workloads: If you need specific settings for individual namespaces or workloads, place additional Telemetry resources. If you don't want to report spans at all for a specific workload, activate the disableSpanReporting flag with the selector expression.
- Trace context without spans: To enable the propagation of the W3C Trace Context only, without reporting any spans (so the actual tracing feature is disabled), you must enable the kyma-traces provider with a sampling rate of 0. With this configuration, you get the relevant trace context into the access logs without any active trace reporting.
```
apiVersion: telemetry.istio.io/v1 kind: Telemetry metadata: name: tracing-default namespace: my-namespace spec: selector: matchLabels: kubernetes.io/name: "my-app" tracing: - providers: - name: "kyma-traces" randomSamplingPercentage: 100.00
```
```
apiVersion: telemetry.istio.io/v1 kind: Telemetry metadata: name: tracing-default namespace: istio-system spec: tracing: - providers: - name: "kyma-traces" randomSamplingPercentage: 0
```
## Eventing
The Eventing module uses the CloudEvents protocol (which natively supports the W3C Trace Context propagation). Because of that, it propagates trace context properly. However, it doesn't enrich a trace with more advanced span data.
## Serverless
By default, all engines for the Serverless module integrate the Open Telemetry SDK . Thus, the used middlewares are configured to automatically propagate the trace context for chained calls. Because the Telemetry endpoints are configured by default, Serverless also reports custom spans for incoming and outgoing requests. You can customize Function traces to add more spans as part of your Serverless source code.
## Operations
A TracePipeline runs several OTel Collector instances in your cluster. This Deployment serves OTLP endpoints and ships received data to the configured backend.
The Telemetry module ensures that the OTel Collector instances are operational and healthy at any time, for example, with buffering and retries. However, there may be situations when the instances drop traces, or cannot handle the trace load.
To detect and fix such situations, check the pipeline status and check out Troubleshooting [page 1990].
If you have set up pipeline health monitoring, check the alerts and reports in an integrated backend like SAP Cloud Logging. For details, see Setting up a MetricPipeline [page 1996], step Monitor Pipeline Health , as well as Integrate with SAP Cloud Logging [page 2011].
##  Caution
It's not recommended to access the metrics endpoint of the used OTel Collector instances directly, because the exposed metrics are no official API of the Kyma Telemetry module. Breaking changes can happen if the underlying OTel Collector version introduces such.
Instead, use the pipeline status .
## Limitations
- Throughput : Assuming an average span with 40 attributes with 64 characters, the maximum throughput is 4200 span/sec ~= 15.000.000 spans/hour. If this limit is exceeded, spans are refused. To increase the maximum throughput, manually scale out the gateway by increasing the number of replicas for the trace gateway. For details, see Module Configuration and Status [page 1952].
- Unavailability of Output : For up to 5 minutes, a retry for data is attempted when the destination is unavailable. After that, data is dropped.
- No Guaranteed Delivery : The used buffers are volatile. If the OTel Collector instance crashes, trace data can be lost.
- Multiple TracePipeline Support : The maximum amount of TracePipeline resources is 5.
- System Span Filtering : System-related spans reported by Istio are filtered out without the opt-out option, for example:
- Any communication of applications to the Telemetry gateways
- Any communication from the gateways to backends
## Troubleshooting
## No Spans Arrive at the Backend
Symptom : In the TracePipeline status, the TelemetryFlowHealthy condition has status AllDataDropped .
Cause : Incorrect backend endpoint configuration (such as using the wrong authentication credentials), or the backend is unreachable.
## Solution :
1. Check the telemetry-trace-gateway Pods for error logs by calling kubectl logs -n kyma-system {POD\_NAME} .
2. Check if the backend is up and reachable.
3. Fix the errors.
## Not All Spans Arrive at the Backend
## Symptom :
- The backend is reachable and the connection is properly configured, but some spans are refused.
- In the TracePipeline status, the TelemetryFlowHealthy condition has status SomeDataDropped .
Cause : It can happen due to a variety of reasons - for example, the backend is limiting the ingestion rate.
## Solution :
1. Check the telemetry-trace-gateway Pods for error logs by calling kubectl logs -n kyma-system {POD\_NAME} . Also, check your observability backend to investigate potential causes.
2. If the backend is limiting the rate by refusing spans, try the options desribed in Gateway Buffer Filling Up.
3. Otherwise, take the actions appropriate to the cause indicated in the logs.
## Custom Spans Don't Arrive at the Backend, but Istio Spans Do
Cause : Y our SDK version is incompatible with the OTel Collector version.
## Solution
1.
2. Investigate whether it is compatible with the OTel Collector version.
3. If required, upgrade to a supported SDK version.
## Trace Backend Shows Fewer Traces than Expected
Cause : By default, only 1% of the requests are sent to the trace backend for trace recording.
## Solution :
To see more traces in the trace backend, increase the percentage of requests by changing the default settings.
If you just want to see traces for one particular request, you can manually force sampling:
1. Create a values.yaml file. The following example sets the value to 60, which means 60% of the requests are sent to the tracing backend.
2. To override the default percentage, change the value for the randomSamplingPercentage attribute.
3. Deploy the values.yaml to your existing Kyma installation.
```
apiVersion: telemetry.istio.io/v1 kind: Telemetry metadata: name: kyma-traces namespace: istio-system spec: tracing: - providers: - name: "kyma-traces"
```
```
randomSamplingPercentage: 60
```
## Gateway Buffer Filling Up
Symptom : In the TracePipeline status, the TelemetryFlowHealthy condition has status BufferFillingUp .
Cause : The backend export rate is too low compared to the gateway ingestion rate.
## Solution :
- Option 1: Increase the maximum backend ingestion rate - for example, by scaling out the SAP Cloud Logging instances.
- Option 2: Reduce the emitted spans in your applications.
## Gateway Throttling
## Symptom :
- In the TracePipeline status, the TelemetryFlowHealthy condition has status GatewayThrottling .
- Also, your application might have error logs indicating a refusal for sending traces to the gateway.
Cause : Gateway cannot receive spans at the given rate.
Solution : Manually scale out the gateway by increasing the number of replicas for the trace gateway. See Module Configuration and Status [page 1952].