## 4.3.1.8.6 Metrics
The goal of the Telemetry module is to support you in collecting all relevant metrics of a workload in a Kyma cluster and ship them to a backend for further analysis. Kyma modules like Istio Module [page 1778] or Serverless contribute metrics instantly, and the Telemetry module enriches the data. You can choose among multiple vendors for OTLP-based backends .
## Overview
Observability is all about exposing the internals of the components belonging to a distributed application and making that data analysable at a central place. While application logs and traces usually provide requestoriented data, metrics are aggregated statistics exposed by a component to reflect the internal state. Typical statistics like the amount of processed requests, or the amount of registered users, can be very useful to monitor the current state and also the health of a component. Also, you can define proactive and reactive alerts if metrics are about to reach thresholds, or if they already passed thresholds.
The Telemetry module provides a metric gateway and, optionally, an agent for the collection and shipment of metrics of any container running in the Kyma runtime.
You can configure the metric gateway with external systems using runtime configuration with a dedicated Kubernetes API (CRD ) named MetricPipeline . The Metric feature is optional. If you don't want to use it, simply don't set up a MetricPipeline .
## Prerequisites
- Before you can collect metrics data from a component, it must expose (or instrument) the metrics. Typically, it instruments specific metrics for the used language runtime (like Node.js) and custom
- metrics specific to the business logic. Also, the exposure can be in different formats, like the pull-based Prometheus format or the push-based OTLP format .
- If you want to use Prometheus-based metrics, you must have instrumented your application using a library like the Prometheus client library , with a port in your workload exposed serving as a Prometheus metrics endpoint.
- If you want to scrape the metric endpoint with Istio, your Service port definition must define the app protocol. For details, see Setting up a MetricPipeline [page 1996], step Activate Prometheus-Based Metrics .
- For the instrumentation, you typically use an SDK, namely the Prometheus client libraries or the Open Telemetry SDKs . Both libraries provide extensions to activate language-specific auto-instrumentation like for Node.js, and an API to implement custom instrumentation.
## Architecture
In the Telemetry module, a central in-cluster Deployment of an OTel Collector acts as a gateway. The gateway exposes endpoints for the OpenTelemetry Protocol (OTLP) for gRPC and HTTP-based communication using the dedicated telemetry-otlp-metrics service, to which all Kyma modules and users' applications send the metrics data.
Optionally, the Telemetry module provides a DaemonSet of an OTel Collector acting as an agent. This agent can pull metrics of a workload and the Istio sidecar in the Prometheus pull-based format and can provide runtime-specific metrics for the workload.
<!-- image -->
1. An application (exposing metrics in OTLP) sends metrics to the central metric gateway service.
2. An application (exposing metrics in Prometheus protocol) activates the agent to scrape the metrics with an annotation-based configuration.
3. Additionally, you can activate the agent to pull metrics of each Istio sidecar.
4. The agent supports collecting metrics from the Kubelet and Kubernetes APIServer.
5. The agent converts and sends all collected metric data to the gateway in OTLP.
6. The gateway discovers the metadata and enriches all received data with typical metadata of the source by communicating with the Kubernetes APIServer. Furthermore, it filters data according to the pipeline configuration.
7. Telemetry Manager configures the agent and gateway according to the MetricPipeline resource specification, including the target backend for the metric gateway. Also, it observes the metrics flow to the backend and reports problems in the MetricPipeline status.
8. The metric gateway sends the data to the observability system that's specified in your MetricPipeline resource - either within the Kyma cluster, or, if authentication is set up, to an external observability backend.
9. You can analyze the metric data with your preferred backend system.
## Telemetry Manager
The MetricPipeline resource is watched by Telemetry Manager, which is responsible for generating the custom parts of the OTel Collector configuration.
<!-- image -->
1. Telemetry Manager watches all MetricPipeline resources and related Secrets.
2. Furthermore, Telemetry Manager takes care of the full lifecycle of the gateway Deployment and the agent DaemonSet. Only if you defined a MetricPipeline , the gateway and agent are deployed.
3. Whenever the user configuration changes, Telemetry Manager validates it and generates a single configuration for the gateway and agent.
4. Referenced Secrets are copied into one Secret that is mounted to the gateway as well.
## Metric Gateway
In a Kyma cluster, the metric gateway is the central component to which all applications can send their individual metrics. The gateway collects, enriches, and dispatches the data to the configured backend. For more information, see Telemetry Gateways [page 1953].
## Metric Agent
If a MetricPipeline configures a feature in the input section, an additional DaemonSet is deployed acting as an agent. The agent is also based on an OTel Collector and encompasses the collection and conversion of Prometheus-based metrics. Hereby, the workload puts a prometheus.io/scrape annotation on the specification of the Pod or service, and the agent collects it. The agent sends all data in OTLP to the central gateway.
## Setting up a MetricPipeline
In the following steps, you can see how to construct and deploy a typical MetricPipeline . Learn more about the available parameters and attributes .