## 11. Deploy the Pipeline
To activate the MetricPipeline , apply the metricpipeline.yaml resource file in your cluster:
```
kubectl apply -f metricpipeline.yaml
```
## Result
You activated a MetricPipeline and metrics start streaming to your backend.
To check that the pipeline is running, wait until the status conditions of the MetricPipeline in your cluster have status True :
<!-- image -->
## Operations
A MetricPipeline runs several OTel Collector instances in your cluster. This Deployment serves OTLP endpoints and ships received data to the configured backend.
The Telemetry module ensures that the OTel Collector instances are operational and healthy at any time, for example, with buffering and retries. However, there may be situations when the instances drop metrics, or cannot handle the metric load.
To detect and fix such situations, check the pipeline status and check out Troubleshooting [page 2008].
If you have set up pipeline health monitoring, check the alerts and reports in an integrated backend like SAP Cloud Logging. For details, see Setting up a MetricPipeline [page 1996], step Monitor Pipeline Health , as well as Integrate with SAP Cloud Logging [page 2011].
##  Caution
It's not recommended to access the metrics endpoint of the used OTel Collector instances directly, because the exposed metrics are no official API of the Kyma Telemetry module. Breaking changes can happen if the underlying OTel Collector version introduces such.
Instead, use the pipeline status .
## Limitations
- Throughput : Assuming an average metric with 20 metric data points and 10 labels, the default metric gateway setup has a maximum throughput of 34K metric data points/sec. If more data is sent to the gateway, it is refused. To increase the maximum throughput, manually scale out the gateway by increasing the number of replicas for the metric gateway. See Module Configuration and Status [page 1952]. The metric agent setup has a maximum throughput of 14K metric data points/sec per instance. If more data must be ingested, it is refused. If a metric data endpoint emits more than 50.000 metric data points per scrape loop, the metric agent refuses all the data.
- Load Balancing With Istio : T o ensure availability, the metric gateway runs with multiple instances. If you want to increase the maximum throughput, use manual scaling and enter a higher number of instances. By design, the connections to the gateway are long-living connections (because OTLP is based on gRPC and HTTP/2). For optimal scaling of the gateway, the clients or applications must balance the connections across the available instances, which is automatically achieved if you use an Istio sidecar. If your application has no Istio sidecar, the data is always sent to one instance of the gateway.
- Unavailability of Output : For up to 5 minutes, a retry for data is attempted when the destination is unavailable. After that, data is dropped.
- No Guaranteed Delivery : The used buffers are volatile. If the gateway or agent instances crash, metric data can be lost.
- Multiple MetricPipeline Support : The maximum amount of MetricPipeline resources is 5.
## Troubleshooting
## No Metrics Arrive at the Backend
## Symptom :
- No metrics arrive at the backend.
- In the MetricPipeline status, the TelemetryFlowHealthy condition has status AllDataDropped .
Cause : Incorrect backend endpoint configuration (such as using the wrong authentication credentials) or the backend is unreachable.
## Solution :
1. Check the telemetry-metric-gateway Pods for error logs by calling kubectl logs -n kymasystem {POD\_NAME} .
2. Check if the backend is up and reachable.
3. Fix the errors.
## Not All Metrics Arrive at the Backend
## Symptom :
- The backend is reachable and the connection is properly configured, but some metrics are refused.
- In the MetricPipeline status, the TelemetryFlowHealthy condition has status SomeDataDropped .
Cause : It can happen due to a variety of reasons - for example, the backend is limiting the ingestion rate.
## Solution :
1. Check the telemetry-metric-gateway Pods for error logs by calling kubectl logs -n kymasystem {POD\_NAME} . Also, check your observability backend to investigate potential causes.
2. If backend is limiting the rate by refusing metrics, try the options described in Gateway Buffer Filling Up.
3. Otherwise, take the actions appropriate to the cause indicated in the logs.
## Only Istio Metrics Arrive at the Backend
Symptom : Custom metrics don't arrive at the backend, but Istio metrics do.
Cause : Y our SDK version is incompatible with the OTel Collector version.
## Solution :
1. Check which SDK version you are using for instrumentation.
2. Investigate whether it is compatible with the OTel Collector version.
3. If required, upgrade to a supported SDK version.
## Log Entry: Failed to Scrape Prometheus Endpoint
Symptom : Custom metrics don't arrive at the destination. The OTel Collector produces log entries saying 'Failed to scrape Prometheus endpoint', such as the following example:
## 
```
Output Code 2023-08-29T09:53:07.123Z warn internal/transaction.go:111 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus/app-pods", "data_type": "metrics", "scrape_timestamp": 1693302787120, "target_labels": "{__name__=\"up\", instance=\"10.42.0.18:8080\", job=\"app-pods\"}"}
```
Cause 1 : The workload is not configured to use 'STRICT' mTLS mode. For details, see Activate PrometheusBased Metrics.
Solution 1 : Y ou can either set up 'STRICT' mTLS mode or HTTP scraping:
- Configure the workload using 'STRICT' mTLS mode (for example, by applying a corresponding PeerAuthentication).
- Set up scraping through HTTP by applying the prometheus.io/scheme=http annotation.
Cause 2 : The Service definition enabling the scrape with Prometheus annotations does not reveal the application protocol to use in the port definition. For details, see Activate Prometheus-Based Metrics.
Solution 2 : Define the application protocol in the Service port definition by either prefixing the port name with the protocol, like in http-metrics or define the appProtocol attribute.
Cause 3 : A deny-all NetworkPolicy was created in the workload namespace, which prevents that the agent can scrape metrics from annotated workloads.
Solution 3 : Create a separate NetworkPolicy to explicitly let the agent scrape your workload using the telemetry.kyma-project.io/metric-scrape label. For example, see the following NetworkPolicy configuration:
```
yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-traffic-from-agent spec: podSelector: matchLabels: app.kubernetes.io/name: "annotated-workload" # <your workload here> ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kyma-system podSelector: matchLabels: telemetry.kyma-project.io/metric-scrape: "true" policyTypes: - Ingress
```
## Gateway Buffer Filling Up
Symptom : In the MetricPipeline status, the TelemetryFlowHealthy condition has status BufferFillingUp .
Cause : The backend export rate is too low compared to the gateway ingestion rate.
## Solution :
- Option 1: Increase maximum backend ingestion rate. For example, by scaling out the SAP Cloud Logging instances.
- Option 2: Reduce emitted metrics by re-configuring the MetricPipeline (for example, by disabling certain inputs or applying namespace filters).
- Option 3: Reduce emitted metrics in your applications.
## Gateway Throttling
Symptom : In the MetricPipeline status, the TelemetryFlowHealthy condition has status GatewayThrottling .
Cause : Gateway cannot receive metrics at the given rate.
Solution : Manually scale out the gateway by increasing the number of replicas for the metric gateway. See Module Configuration and Status [page 1952].