## 4.3.1.8.3 Application Logs
With application logs, you can debug an application and derive the internal state of an application. The Telemetry module supports observing your applications with logs of the correct severity level and context.
## Overview
The Telemetry module provides a log gateway for push-based collection of logs using OTLP and, optionally, an agent for the collection of logs of any container printing logs to the stdout/stderr channel running in the
Kyma runtime. Kyma modules like Istio contribute access logs. The Telemetry module enriches the data and ships them to your chosen backend (see Vendors who natively support OpenTelemetry ).
You can configure the log gateway and agent with external systems using runtime configuration with a dedicated Kubernetes API (CRD ) named LogPipeline . The Log feature is optional. If you don't want to use it, simply don't set up a LogPipeline .
## Prerequisites
- Before you can collect logs from a component, it must emit the logs. Typically, it uses a logger framework for the used language runtime (like Node.js) and prints them to the stdout or stderr channel (Kubernetes: How nodes handle container logs ). Alternatively, you can use the OTel SDK to use the push-based OTLP format .
- If you want to emit the logs to the stdout/stderr channel, use structured logs in a JSON format with a logger library like log4J. With that, the log agent can parse your log and enrich all JSON attributes as log attributes, and a backend can use that.
- If you prefer the push-based alternative with OTLP, also use a logger library like log4J. However, you additionally instrument that logger and bridge it to the OTel SDK. For details, see OpenTelemetry: New First-Party Application Logs .
## Architecture
In the Telemetry module, a central in-cluster Deployment of an OTel Collector acts as a gateway. The gateway exposes endpoints for the OpenTelemetry Protocol (OTLP) for GRPC and HTTP-based communication using the dedicated telemetry-otlp-logs service, to which your applications send the logs data.
You can choose whether you also want an agent, based on a DaemonSet of an OTel Collector. This agent can tail logs of a container from the underlying container runtime.
<!-- image -->
1. Application containers print JSON logs to the stdout/stderr channel and are stored by the Kubernetes container runtime under the var/log directory and its subdirectories at the related Node. Istio is configured to write access logs to stdout as well.
2. If you choose to use the agent, an OTel Collector runs as a DaemonSet (one instance per Node), detects any new log files in the folder, and tails and parses them.
3. An application (exposing logs in OTLP) sends logs to the central log gateway service. Istio is configured to push access logs with OTLP as well.
4. The gateway and agent discover the metadata and enrich all received data with metadata of the source by communicating with the Kubernetes APIServer. Furthermore, they filter data according to the pipeline configuration.
5. Telemetry Manager configures the agent and gateway according to the LogPipeline resource specification, including the target backend. Also, it observes the logs flow to the backend and reports problems in the LogPipeline status.
6. The log agent and gateway send the data to the observability system that's specified in your LogPipeline resource - either within the Kyma cluster, or, if authentication is set up, to an external observability backend.
7. You can analyze the logs data with your preferred backend.
## Telemetry Manager
The LogPipeline resource is watched by Telemetry Manager, which is responsible for generating the custom parts of the OTel Collector configuration.
<!-- image -->
1. Telemetry Manager watches all LogPipeline resources and related Secrets.
2. Furthermore, Telemetry Manager takes care of the full lifecycle of the gateway Deployment and the agent DaemonSet. Only if you defined a LogPipeline , the gateway and agent are deployed.
3. Whenever the user configuration changes, Telemetry Manager validates it and generates a single configuration for the gateway and agent.
4. Referenced Secrets are copied into one Secret that is mounted to the gateway as well.
## Log Gateway
In a Kyma cluster, the log gateway is the central component to which all components can send their individual logs. The gateway collects, enriches, and dispatches the data to the configured backend. For more information, see Telemetry Gateways [page 1953].
## Log Agent
If you configure a feature in the input section of your LogPipeline , an additional DaemonSet is deployed acting as an agent. The agent is based on an OTel Collector and encompasses the collection and conversion of logs from the container runtime. Hereby, the workload container just prints the structured log to the stdout/stderr channel. The agent picks them up, parses and enriches them, and sends all data in OTLP to the configured backend.
## Setting up a LogPipeline
In the following steps, you can see how to construct and deploy a typical LogPipeline . Learn more about the available parameters and attributes .