## 5. Deploy the Pipeline
To activate the LogPipeline , apply the logpipeline.yaml resource file in your cluster:
```
kubectl apply -f logpipeline.yaml
```
## Result
You activated a LogPipeline and logs start streaming to your backend.
To check that the pipeline is running, wait until the status conditions of the LogPipeline in your cluster have status True :
## 
```
Output Code kubectl get logpipeline NAME      CONFIGURATION GENERATED   AGENT HEALTHY   FLOW HEALTHY backend   True                      True            True
```
## Log Record Processing
After a log record has been read, it is preprocessed by configured plugins, like the kubernetes filter. Thus, when a record is ready to be processed by the sections defined in the LogPipeline definition, it has several attributes available for processing and shipment.
<!-- image -->
Learn more about the flow of the log record through the general pipeline and the available log attributes in the following stages:
## Container Log Message
The following example assumes that there's a container myContainer of Pod myPod , running in namespace myNamespace , logging to stdout with the following log message in the JSON format:
```
 Output Code { "level": "warn", "message": "This is the actual message", "tenant": "myTenant", "traceID": "123" }
```
## Tail Input
The tail input plugin reads the log message from a log file managed by the container runtime. The input plugin brings a dedicated filesystem buffer for the pipeline. The file name contains the namespace, Pod, and container information that will be available later as part of the tag . The tag is prefixed with the pipeline name. The resulting log record available in an internal Fluent Bit representation looks similar to the following example:
```
 Output Code { "time": "2022-05-23T15:04:52.193317532Z", "stream": "stdout", "_p": "F", "log": "{\"level\": \"warn\",\"message\": \"This is the actual message\",\"tenant\": \"myTenant\",\"traceID\": \"123\"} }
```
The attributes have the following meaning:
- time : The timestamp generated by the container runtime at the moment the log was written to the log file.
- stream : The stream to which the application wrote the log, either stdout or stderr .
- \_p : Indicates if the log message is partial (P) or final (F). Optional, dependent on container runtime. Because a CRI multiline parser is applied for the tailing phase, all multilines on the container runtime level are aggregated already and no partial entries must be left.
- log : The raw and unparsed log message.
## Kubernetes Filter (Metadata)
After the tail input, the Kubernetes filter is applied. The container information from the log file name (available in the tag) is interpreted and used for a Kubernetes API Server request to resolve more metadata of the container. All the resolved metadata enrich the existing record as a new attribute kubernetes :
```
 Output Code { "kubernetes": { "pod_name": "myPod-74db47d99-ppnsw", "namespace_name": "myNamespace", "pod_id": "88dbd1ef-d977-4636-804d-ef220454be1c", "host": "myHost1", "container_name": "myContainer", "docker_id": "5649c36fcc1e956fc95e3145441f427d05d6e514fa439f4e4f1ccee80fb2c037", "container_hash": "myImage@sha256:1f8d852989c16345d0e81a7bb49da231ade6b99d51b95c56702d04c417549b 26", "container_image": "myImage:myImageTag", "labels": { "app": "myApp", "sidecar.istio.io/inject": "true", ... } } }
```
## Kubernetes Filter (JSON Parser)
After the enrichment of the log record with the Kubernetes-relevant metadata, the Kubernetes filter also tries to parse the record as a JSON document. If that is successful, all the parsed root attributes of the parsed document are added as new individual root attributes of the log.
The record before applying the JSON parser:
```
 Output Code { "time": "2022-05-23T15:04:52.193317532Z", "stream": "stdout", "_p": "F", "log": "{\"level\": \"warn\",\"message\": \"This is the actual message\",\"tenant\": \"myTenant\",\"traceID\": \"123\"}", "kubernetes": {...} }
```
The record after applying the JSON parser:
```
 Output Code { "time": "2022-05-23T15:04:52.193317532Z", "stream": "stdout", "_p": "F",
```
```
"log": "{\"level\": \"warn\",\"message\": \"This is the actual message\",\"tenant\": \"myTenant\",\"traceID\": \"123\"}", "kubernetes": {...}, "level": "warn", "message": "This is the actual message", "tenant": "myTenant", "traceID": "123" }
```
## Further Enrichment
Additionally, the agent enriches every log record with the cluster\_identifier attribute by setting the APIServer URL of the underlying Kubernetes cluster:
```
{ "cluster_identifier": "<APIServer URL>" ... }
```
For LogPipeline resources that use an HTTP output, the following attributes are enriched for optimized integration with SAP Cloud Logging:
```
{ "@timestamp": "<value of attribute 'time'>", "date": "<agent time in iso8601>", "kubernetes": { "app_name": "<value of Pod label 'app.kubernetes.io/name' or 'app'>" ... }, ... }
```
The enriched timestamp attributes have the following meaning:
- time : The time when the container runtime captured the log on stdout/stderr , which is very close to the time when the log originated in the application.
- date : The time when the log agent processed the log, which is later than the value in time .
- @timestamp : Contains the same value as time , optimized for SAP Cloud Logging integration.
SAP Cloud Logging uses a dedicated attribute called @timestamp to represent the time of a log record. When processing a log, SAP Cloud Logging first checks whether the record contains a date field with a valid value in either Unix time (integer format) or ISO 8601 format. If the date field is missing or contains an invalid value, SAP Cloud Logging generates the @timestamp attribute based on the time the log record was received. This generated timestamp is usually later than the original log time and is not helpful in most scenarios.
Fluent Bit's HTTP output plugin also uses a date field. This field represents the time when Fluent Bit observed the log and is formatted in ISO 8601 with millisecond precision. The field's value may slightly differ from the original log time because while the original log timestamp may have nanosecond precision, the Fluent Bit date field is limited to millisecond precision.
Fluent Bit HTTP output uses an additional filter to improve log time precision. The filter allows copying the original time attribute to the @timestamp field.
## Operations
The Telemetry module ensures that the log agent instances are operational and healthy at any time, for example, with buffering and retries. However, there may be situations when the instances drop logs, or cannot handle the log load.
To detect and fix such situations, check the pipeline status and check out Troubleshooting [page 1979].
If you have set up pipeline health monitoring, check the alerts and reports in an integrated backend like SAP Cloud Logging. For details, see Setting up a MetricPipeline [page 1996], step Monitor Pipeline Health , as well as Integrate with SAP Cloud Logging [page 2011].
##  Caution
It's not recommended to access the metrics endpoint of the used FluentBit instances directly, because the exposed metrics are no official API of the Kyma Telemetry module. Breaking changes can happen if the underlying FluentBit version introduces such.
Instead, use the pipeline status .
## Limitations
- Reserved Log Attributes : The log attribute named kubernetes is a special attribute that's enriched by the kubernetes filter. When you use that attribute as part of your structured log payload, the metadata enriched by the filter are overwritten by the payload data. Filters that rely on the original metadata might no longer work as expected.
- Buffer Limits : Fluent Bit buffers up to 1 GB of logs if a configured output cannot receive logs. The oldest logs are dropped when the limit is reached or after 300 retries.
- Throughput : Each Fluent Bit Pod (each running on a dedicated Node) can process up to 10 MB/s of logs for a single LogPipeline . With multiple pipelines, the throughput per pipeline is reduced. The used logging backend or performance characteristics of the output plugin might limit the throughput earlier.
- Multiple LogPipeline Support : The maximum amount of LogPipeline resources is 5.
## Troubleshooting
## No Logs Arrive at the Backend
## Symptom :
- No logs arrive at the backend.
- In the LogPipeline status, the TelemetryFlowHealthy condition has status AllDataDropped .
Cause : Incorrect backend endpoint configuration (for example, using the wrong authentication credentials) or the backend being unreachable.
## Solution :
- Check the telemetry-fluent-bit Pods for error logs by calling kubectl logs -n kyma-system {POD\_NAME} .
- Check if the backend is up and reachable.
## Not All Logs Arrive at the Backend
Symptom : The backend is reachable and the connection is properly configured, but some logs are refused. In the LogPipeline status, the TelemetryFlowHealthy condition has status SomeDataDropped .
Cause : It can happen due to a variety of reasons. For example, a possible reason may be that the backend is limiting the ingestion rate, or the backend is refusing logs because they are too large.
## Solution :
1. Check the telemetry-fluent-bit Pods for error logs by calling kubectl logs -n kyma-system {POD\_NAME} . Also, check your observability backend to investigate potential causes.
2. If backend is limiting the rate by refusing logs, try the options described in Agent Buffer Filling Up .
3. Otherwise, take the actions appropriate to the cause indicated in the logs.
## Agent Buffer Filling Up
Symptom : In the LogPipeline status, the TelemetryFlowHealthy condition has status BufferFillingUp .
Cause : The backend export rate is too low compared to the log collection rate.
## Solution :
- Option 1: Increase maximum backend ingestion rate. For example, by scaling out the SAP Cloud Logging instances.
- Option 2: Reduce emitted logs by re-configuring the LogPipeline (for example, by applying namespace or container filters).
- Option 3: Reduce emitted logs in your applications (for example, by changing severity level).