Repository: gardener/gardener
Pull request #11861: [GEP-34] Opentelemetry Operator and Collectors
Author: login: nickytd
Labels: name: kind/enhancement
description: Enhancement, improvement, extension
name: size/L
description: Denotes a PR that changes 100-499 lines, ignoring generated files.
name: area/monitoring
description: Monitoring (including availability monitoring and alerting) related
name: area/logging
description: Logging related
name: cla: yes
description: Indicates the PR's author has signed the cla-assistant.io CLA.
name: lgtm
description: Indicates that a PR is ready to be merged.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: area/ipcei
description: IPCEI (Important Project of Common European Interest)
Created: 2025-04-12T07:08:06Z | Merged: 2025-05-19T10:43:06Z

**How to categorize this PR?**
/area logging
/area monitoring
/kind enhancement

**What this PR does / why we need it**:
This PR introduces GEP-34 bringing OpentTelemetry support in Gardener, part of Observability 2.0 initiative.

**Release note**:

```other developer
GEP-34 Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
```



Diff:
diff --git a/docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md b/docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
new file mode 100644
index 00000000000..53b95f597e9
--- /dev/null
+++ b/docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
@@ -0,0 +1,162 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+- "@Kostov6"
+- "@ScheererJ"
+- "@vpnachev"
+- "@rfranzke"
+---
+
+# GEP-34: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry Operator and deploys OpenTelemetry Collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](./19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](./19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry Operator and Collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry Operator to manage Collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry Collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the Collectors receivers and wire the Collector exporters to the current logging backend.
+- **Replace valitail system service with OpenTelemetry Collectors on shoot nodes:** Replace current valitail system service on shoot nodes with an OpenTelemetry Collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md)).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry Operator and OpenTelemetry Collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry Collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry Operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](./19-migrating-observability-stack-to-operators.md).
+- **Configuration:** The OpenTelemetry Operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This means that during the `Seed` reconciliation flow the OpenTelemetry Operator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated OpenTelemetry Collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry Collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry Operator.
+- **Configuration:** During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    extensions:
+      bearertokenauth:
+        filename: {{ .pathAuthToken }}
+
+    exporters:
+      loki:
+        endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        auth:
+          authenticator: bearertokenauth
+        tls:
+          ca_file: {{ .pathCACert }}
+
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      extensions: [bearertokenauth]
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from [*GEP-19*](./19-migrating-observability-stack-to-operators.md).
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry Collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
+  - **Event Logger Replacement:** In a later stage the current `event logger` shall be replaced with the [`k8s-event` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver) in the Collector shoot instance.
+
+![Figure 2](assets/gep-34-shoot-observability-architecture.png)
+
+Example Systemd unit for the OpenTelemetry Collector in the shoot node:
+
+```ini
+[Unit]
+Description=opentelemetry-collector daemon
+Documentation=https://github.com/open-telemetry/opentelemetry-collector
+[Install]
+WantedBy=multi-user.target
+[Service]
+CPUAccounting=yes
+MemoryAccounting=yes
+CPUQuota=3%
+CPUQuotaPeriodSec=1000ms
+MemoryMin=29M
+MemoryHigh=400M
+MemoryMax=800M
+MemorySwapMax=0
+Restart=always
+RestartSec=5
+EnvironmentFile=/etc/environment
+ExecStart=/opt/bin/opentelemetry-collector --config=/var/lib/opentelemetry-collector/config/config
+```
+
+Example configuration for the OpenTelemetry Collector in the shoot node can be found [here](assets/gep-34-shoot-collector-config.yaml). For an overview of how it works, you can view the official [docs](https://opentelemetry.io/docs/collector/configuration/).
+
+A visualisation of the example config (Image generated with [otelbin](https://github.com/dash0hq/otelbin)):
+![Figure 3](assets/gep-34-collector-image.png)
+
+---
+
+## Alternatives
+
+### Direct Collector Deployment Without an Operator
+
+- **Description:** Deploy OpenTelemetry Collectors in each shoot reconcile directly without using an operator.
+- **Pros:** Simplifies initial setup, avoids operator overhead.
+- **Cons:** Lacks scalability and management features (e.g., automatic upgrades, CR-based configuration), which are critical in Gardener’s multi-tenant environment.
+- **Reason Rejected:** The operator aligns with [*GEP-19*](./19-migrating-observability-stack-to-operators.md)’s precedent of using operators for observability components, ensuring consistency and scalability.
+
+---
+
+This GEP establishes a critical milestone in Gardener’s journey toward [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md). By deploying the OpenTelemetry Operator and OpenTelemetry Collectors, Gardener gains a standardized, extensible observability framework that integrates with its existing stack while paving the way for future enhancements like tracing and OpenTelemetry native backends.
diff --git a/docs/proposals/assets/gep-34-collector-image.png b/docs/proposals/assets/gep-34-collector-image.png
new file mode 100644
index 00000000000..f23c21c338f
Binary files /dev/null and b/docs/proposals/assets/gep-34-collector-image.png differ
diff --git a/docs/proposals/assets/gep-34-otel-operator-architecture.png b/docs/proposals/assets/gep-34-otel-operator-architecture.png
new file mode 100644
index 00000000000..2e9a85d54df
Binary files /dev/null and b/docs/proposals/assets/gep-34-otel-operator-architecture.png differ
diff --git a/docs/proposals/assets/gep-34-shoot-collector-config.yaml b/docs/proposals/assets/gep-34-shoot-collector-config.yaml
new file mode 100644
index 00000000000..616dbfea18f
--- /dev/null
+++ b/docs/proposals/assets/gep-34-shoot-collector-config.yaml
@@ -0,0 +1,129 @@
+extensions:
+  file_storage:
+    directory: /var/log/otelcol
+    create_directory: true
+
+receivers:
+  journald/journal:
+    start_at: beginning
+    storage: file_storage
+    operators:
+      - type: move
+        from: body.SYSLOG_IDENTIFIER
+        to: resource.unit
+      - type: move
+        from: body._HOSTNAME
+        to: resource.nodename
+      - type: retain
+        fields:
+          - body.MESSAGE
+
+  filelog/pods:
+    include: [{{ listOfKubeSystemManagedWorkloads }}]
+    storage: file_storage
+    include_file_path: true
+    operators:
+      - type: container
+        format: containerd
+        add_metadata_from_filepath: true
+
+processors:
+  batch:
+    timeout: 10s
+
+  resourcedetection/system:
+    detectors: ["system"]
+    system:
+      hostname_sources: ["os"]
+
+  filter/drop_localhost_journal:
+    logs:
+      exclude:
+        match_type: strict
+        resource_attributes:
+          - key: _HOSTNAME
+            value: localhost
+
+  filter/keep_units_journal:
+    logs:
+      include:
+        match_type: strict
+        resource_attributes:
+          - key: SYSLOG_IDENTIFIER
+            value: kernel
+          - key: _SYSTEMD_UNIT
+            value: kubelet.service
+          - key: _SYSTEMD_UNIT
+            value: containerd.service
+          - key: _SYSTEMD_UNIT
+            value: gardener-node-agent.service
+
+  filter/drop_units_combine:
+    logs:
+      exclude:
+        match_type: strict
+        resource_attributes:
+          - key: SYSLOG_IDENTIFIER
+            value: kernel
+          - key: _SYSTEMD_UNIT
+            value: kubelet.service
+          - key: _SYSTEMD_UNIT
+            value: containerd.service
+          - key: _SYSTEMD_UNIT
+            value: gardener-node-agent.service
+
+  resource/journal:
+    attributes:
+      - action: insert
+        key: origin
+        value: systemd-journal
+      - key: loki.resource.labels
+        value: unit, nodename, origin
+        action: insert
+      - key: loki.format
+        value: logfmt
+        action: insert
+
+  resource/pod_labels:
+    attributes:
+      - key: origin
+        value: "shoot-system"
+        action: insert
+      - key: namespace_name
+        value: "kube-system"
+        action: insert
+      - key: pod_name
+        from_attribute: k8s.pod.name
+        action: insert
+      - key: container_name
+        from_attribute: k8s.container.name
+        action: insert
+      - key: loki.resource.labels
+        value: pod_name, container_name, origin, namespace_name, nodename, host.name
+        action: insert
+      - key: loki.format
+        value: logfmt
+        action: insert
+
+exporters:
+  # Logs are exported to the collector in the shoot namespace.
+  otlp:
+    endpoint: {{ CollectorInShootNamespaceIngress }}
+    tls:
+      cert_file: {{ Certificate }}
+      key_file: {{ KeyFile }}
+service:
+  extensions: [file_storage]
+  pipelines:
+    logs/journal:
+      receivers: [journald/journal]
+      processors: [filter/drop_localhost_journal, filter/keep_units_journal, resource/journal, batch]
+      exporters: [otlp]
+    logs/combine_journal:
+      receivers: [journald/journal]
+      processors: [filter/drop_localhost_journal, filter/drop_units_combine, resource/journal, batch]
+      exporters: [otlp]
+    logs/pods:
+      receivers: [filelog/pods]
+      processors: [resourcedetection/system, resource/pod_labels, batch]
+      exporters: [otlp]
diff --git a/docs/proposals/assets/gep-34-shoot-observability-architecture.png b/docs/proposals/assets/gep-34-shoot-observability-architecture.png
new file mode 100644
index 00000000000..bddacff6607
Binary files /dev/null and b/docs/proposals/assets/gep-34-shoot-observability-architecture.png differ


Reviews:
user: login: Kostov6
body: Thanks for the GEP! I was wondering if the migration will unavoidably lead to some loss of logs. If yes do you have some sort of a rough estimation how much logs will be lost?
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T07:48:57Z
user: login: ScheererJ
body: Thanks for providing a glimpse into what is planned for observability 2.0.

Please take a minute to clarify the scope of the GEP. Title, summary and motivation made me think that this has far more reach than it actually intends to implement according to the proposal section.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-16T14:29:22Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T07:07:56Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T07:09:18Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T07:11:01Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T07:12:55Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T07:13:17Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T07:18:57Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T07:19:39Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T07:23:18Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T07:24:44Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T07:25:14Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T07:31:06Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T07:36:03Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T07:36:51Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T07:42:18Z
user: login: vpnachev
body: Thanks!
Overall looks good to me
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T08:33:18Z
user: login: rrhubenov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-23T07:18:12Z
user: login: rrhubenov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-23T15:03:17Z
user: login: rrhubenov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-23T15:16:04Z
user: login: rfranzke
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-24T07:14:47Z
user: login: vpnachev
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-24T15:56:40Z
user: login: rfranzke
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-25T07:13:46Z
user: login: ScheererJ
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-30T14:12:10Z
user: login: ScheererJ
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-30T14:14:24Z
user: login: ScheererJ
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-30T14:30:55Z
user: login: rrhubenov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-12T06:37:53Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T06:41:51Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T06:42:06Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T06:42:36Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T09:12:04Z
user: login: rfranzke
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-13T12:44:29Z
user: login: rfranzke
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-13T12:44:43Z
user: login: rrhubenov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-13T13:14:02Z
user: login: nickytd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T15:00:21Z
user: login: rrhubenov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-14T11:02:15Z
user: login: rrhubenov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-14T11:21:27Z
user: login: ScheererJ
body: Thanks for addressing the feedback. I read through the document again. It is almost finished from my point of view, but there are a few nits to be resolved.

The only real omission is the addition of the reviewers. Please take care.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-14T13:04:46Z
user: login: vpnachev
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-14T15:44:12Z
user: login: rfranzke
body: /lgtm
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-19T07:29:34Z

Review comments:
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Please use the default header. The existing one looks weird compared to the other GEPs.

```suggestion
---
title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
gep-number: 34
creation-date: 2025-04-12
status: implementable
authors:
- "@nickytd"
- "@rrhubenov"
reviewers:
- 
---

# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
```
created_at: 2025-04-16T13:09:49Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: ```suggestion
Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
```
created_at: 2025-04-16T13:12:31Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: You use various variations of similar terms/casing, e.g. OpenTelemetry Operator, OpenTelemetry operator, otel-collector, OpenTelemetry Collector, OpenTelemetry collector, OTel Collectors.
Could you please stick to less variations and be consistent with the casing? Unless of course those mean very different things, which is not obvious for me as a reader.
created_at: 2025-04-16T13:19:34Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Maybe I misread the goals, but I fail to understand what the boundary conditions of this proposal are. What do you want to achieve? Should all observability data go through open telemetry collectors? Do we only care about the data coming from one shoot cluster? Do we handle both signals from the seed (control plane) as well as the shoot (data plane)? Are all signals in scope, i.e. do we want to adapt logging and metrics or only one of them?
Please clarify.
created_at: 2025-04-16T13:33:27Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Is tracing at all in focus or is this GEP just a precondition to adopt tracing more easily at a later point in time?
created_at: 2025-04-16T13:34:35Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Please add a link.
created_at: 2025-04-16T13:35:38Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Please add a link.
created_at: 2025-04-16T13:35:54Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: ```suggestion
- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This means that during the `Seed` reconciliation flow the OpenTelemetryOperator will be deployed into the `garden` namespace.
```
created_at: 2025-04-16T13:39:28Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: ```suggestion
- **Configuration**: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
```
created_at: 2025-04-16T13:41:53Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Does it have to be an `Ingress` resource or is it also possible to use istio directly, either via istio APIs or Gateway API? The network flow through two reverse proxies is not perfect for `Ingress` flows.
created_at: 2025-04-16T13:43:46Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Indentation seems to be incorrect. Please fix.
created_at: 2025-04-16T13:44:50Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: How does the file include work? Is this a potential security vulnerability waiting to happen as any local file could be exfiltrated?
created_at: 2025-04-16T13:47:29Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Please add a link.
created_at: 2025-04-16T14:06:10Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: What about integration with prometheus?
created_at: 2025-04-16T14:07:51Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: How will this replacement work? Will the same systemd unit be used, i.e. will an update simply switch from one executable to the new one or shall both units work in parallel? Can you please share more details?
created_at: 2025-04-16T14:12:09Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,133 @@
+extensions:
+  file_storage:
+    directory: /var/log/otelcol
+    create_directory: true
+
+receivers:
+  journald/journal:
+    start_at: beginning
+    storage: file_storage
+    operators:
+      - type: move
+        from: body.SYSLOG_IDENTIFIER
+        to: resource.unit
+      - type: move
+        from: body._HOSTNAME
+        to: resource.nodename
+      - type: retain
+        fields:
+          - body.MESSAGE
+
+  filelog/pods:
+    include: [{{ listOfKubeSystemManagedWorkloads }}]
+    storage: file_storage
+    include_file_path: true
+    operators:
+      - type: container
+        format: containerd
+        add_metadata_from_filepath: true
+
+processors:
+  batch:
+    timeout: 10s
+
+  resourcedetection/system:
+    detectors: ["system"]
+    system:
+      hostname_sources: ["os"]
+
+  filter/drop_localhost_journal:
+    logs:
+      exclude:
+        match_type: strict
+        resource_attributes:
+          - key: _HOSTNAME
+            value: localhost
+
+  filter/keep_units_journal:
+    logs:
+      include:
+        match_type: strict
+        resource_attributes:
+          - key: SYSLOG_IDENTIFIER
+            value: kernel
+          - key: _SYSTEMD_UNIT
+            value: kubelet.service
+          - key: _SYSTEMD_UNIT
+            value: docker.service
path: docs/proposals/assets/gep-34-shoot-collector-config.yaml
user: login: ScheererJ
body: Probably not necessary anymore.
created_at: 2025-04-16T14:14:32Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
+  - **Event Logger Replacement:** In a later stage the current `event logger` shall be replaced with the `k8s-event` receiver in  the otel-collector shoot instance
+
+![Figure 2](assets/gep-34-shoot-observability-architecture.png)
+
+Example Systemd unit for the OpenTelemetry Collector in the shoot node:
+
+```ini
+[Unit]
+Description=opentelemetry-collector daemon
+Documentation=https://github.com/open-telemetry/opentelemetry-collector
+[Install]
+WantedBy=multi-user.target
+[Service]
+CPUAccounting=yes
+MemoryAccounting=yes
+CPUQuota=3%
+CPUQuotaPeriodSec=1000ms
+MemoryMin=29M
+MemoryHigh=400M
+MemoryMax=800M
+MemorySwapMax=0
+Restart=always
+RestartSec=5
+EnvironmentFile=/etc/environment
+ExecStartPre=/bin/sh -c "systemctl set-environment HOSTNAME=$(hostname | tr [:upper:] [:lower:])"
+ExecStart=/opt/bin/opentelemetry-collector --config=/var/lib/opentelemetry-collector/config/config
+```
+
+Example configuration for the OpenTelemetry Collector in the shoot node can be found [here](assets/gep-34-shoot-collector-config.yaml). For an overview of how it works, you can view the official [docs](https://opentelemetry.io/docs/collector/configuration/).
+
+A visualisation of the example config (Image generated with [otelbin](https://github.com/dash0hq/otelbin)):
+![Figure 3](assets/gep-34-collector-image.png)
+
+---
+
+## Alternatives
+
+### Direct Collector Deployment Without an Operator
+
+- **Description:** Deploy OpenTelemetry collectors in each shoot reconcile directly without using an operator.
+- **Pros:** Simplifies initial setup, avoids operator overhead.
+- **Cons:** Lacks scalability and management features (e.g., automatic upgrades, CR-based configuration), which are critical in Gardener’s multi-tenant environment.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Could you please explain in more detail the topic automatic upgrades in this context?

From my point of view, regardless if we use an operator or not there would be some version reference for the operator or the collector. This would need to be updated manually, e.g. via a renovate PR or a corresponding version upgrade. Once that change would make it into a gardener release the upgrades would be automatic for landscape operators/shoot owners.

What is the difference of both approaches in this regard? I do not see any.
created_at: 2025-04-16T14:20:04Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
+  - **Event Logger Replacement:** In a later stage the current `event logger` shall be replaced with the `k8s-event` receiver in  the otel-collector shoot instance
+
+![Figure 2](assets/gep-34-shoot-observability-architecture.png)
+
+Example Systemd unit for the OpenTelemetry Collector in the shoot node:
+
+```ini
+[Unit]
+Description=opentelemetry-collector daemon
+Documentation=https://github.com/open-telemetry/opentelemetry-collector
+[Install]
+WantedBy=multi-user.target
+[Service]
+CPUAccounting=yes
+MemoryAccounting=yes
+CPUQuota=3%
+CPUQuotaPeriodSec=1000ms
+MemoryMin=29M
+MemoryHigh=400M
+MemoryMax=800M
+MemorySwapMax=0
+Restart=always
+RestartSec=5
+EnvironmentFile=/etc/environment
+ExecStartPre=/bin/sh -c "systemctl set-environment HOSTNAME=$(hostname | tr [:upper:] [:lower:])"
+ExecStart=/opt/bin/opentelemetry-collector --config=/var/lib/opentelemetry-collector/config/config
+```
+
+Example configuration for the OpenTelemetry Collector in the shoot node can be found [here](assets/gep-34-shoot-collector-config.yaml). For an overview of how it works, you can view the official [docs](https://opentelemetry.io/docs/collector/configuration/).
+
+A visualisation of the example config (Image generated with [otelbin](https://github.com/dash0hq/otelbin)):
+![Figure 3](assets/gep-34-collector-image.png)
+
+---
+
+## Alternatives
+
+### Direct Collector Deployment Without an Operator
+
+- **Description:** Deploy OpenTelemetry collectors in each shoot reconcile directly without using an operator.
+- **Pros:** Simplifies initial setup, avoids operator overhead.
+- **Cons:** Lacks scalability and management features (e.g., automatic upgrades, CR-based configuration), which are critical in Gardener’s multi-tenant environment.
+- **Reason Rejected:** The operator aligns with *GEP-19*’s precedent of using operators for observability components, ensuring consistency and scalability.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Frankly, I do not see the precedent of GEP-19 (link missing) as an important aspect.

An operator can make it easier to manage the resources, as was the case with GEP-19, but we do not use operators for all of our components.
created_at: 2025-04-16T14:22:21Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
+  - **Event Logger Replacement:** In a later stage the current `event logger` shall be replaced with the `k8s-event` receiver in  the otel-collector shoot instance
+
+![Figure 2](assets/gep-34-shoot-observability-architecture.png)
+
+Example Systemd unit for the OpenTelemetry Collector in the shoot node:
+
+```ini
+[Unit]
+Description=opentelemetry-collector daemon
+Documentation=https://github.com/open-telemetry/opentelemetry-collector
+[Install]
+WantedBy=multi-user.target
+[Service]
+CPUAccounting=yes
+MemoryAccounting=yes
+CPUQuota=3%
+CPUQuotaPeriodSec=1000ms
+MemoryMin=29M
+MemoryHigh=400M
+MemoryMax=800M
+MemorySwapMax=0
+Restart=always
+RestartSec=5
+EnvironmentFile=/etc/environment
+ExecStartPre=/bin/sh -c "systemctl set-environment HOSTNAME=$(hostname | tr [:upper:] [:lower:])"
+ExecStart=/opt/bin/opentelemetry-collector --config=/var/lib/opentelemetry-collector/config/config
+```
+
+Example configuration for the OpenTelemetry Collector in the shoot node can be found [here](assets/gep-34-shoot-collector-config.yaml). For an overview of how it works, you can view the official [docs](https://opentelemetry.io/docs/collector/configuration/).
+
+A visualisation of the example config (Image generated with [otelbin](https://github.com/dash0hq/otelbin)):
+![Figure 3](assets/gep-34-collector-image.png)
+
+---
+
+## Alternatives
+
+### Direct Collector Deployment Without an Operator
+
+- **Description:** Deploy OpenTelemetry collectors in each shoot reconcile directly without using an operator.
+- **Pros:** Simplifies initial setup, avoids operator overhead.
+- **Cons:** Lacks scalability and management features (e.g., automatic upgrades, CR-based configuration), which are critical in Gardener’s multi-tenant environment.
+- **Reason Rejected:** The operator aligns with *GEP-19*’s precedent of using operators for observability components, ensuring consistency and scalability.
+
+---
+
+This GEP establishes a critical milestone in Gardener’s journey toward *Observability 2.0*. By deploying the OpenTelemetry operator and collectors, Gardener gains a standardized, extensible observability framework that integrates with its existing stack while paving the way for future enhancements like tracing and OTLP-native backends.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: While this GEP certainly is a stepping stone towards Observability 2.0, I would appreciate it if you would mention more explicitly what this is all about. In this GEP there is almost no mention of any signals but logging. This is fine, but then it should also be clear from reading the summary/motivation that this only handles logging.

Please adapt the document accordingly.
created_at: 2025-04-16T14:27:00Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: The evolution of the topic depends on the concrete scenarios. For metrics for example I plan to bring a prposal for the  hackaton in June hack-the-garden-25 event https://gardener-cloud.slack.com/archives/C0531FVMZFU to use the otel-collector instance on the shoot nodes to also scrape the exportes and push the metrics in the otel-collector in the control-plane namespace and from there to the prometheus 3 instances. The prometheus 3 supports standard otel ingegstion. https://prometheus.io/blog/2024/03/14/commitment-to-opentelemetry/

Short terms goals: is to transition the logs. Shoot metrics are potential candidate that we shall explore next also with the collaboratin with the monitoring team. For this early stage no other changes for metrics are planned.

Long term goal: is to potentially transition all observbaility signals but that will be motivated from the potential scenarios. That is already explained.
created_at: 2025-04-19T07:07:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: thanks for the proposal, will adjust the casing.
created_at: 2025-04-19T07:09:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: Sure but what link?
created_at: 2025-04-19T07:11:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: That has been clarified already. This GEP does not address distributed traces in gardener per se. But is a precondition as you correctly pointed for some time in the futre. 
created_at: 2025-04-19T07:12:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: What link?
created_at: 2025-04-19T07:13:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: I was thinking about that as well, if we can bring Gateway API into our landscapes that definelty is a viable transion step from the ingress. So we can start with the ingress because that is what we have today and transition to Gateway API when that become available. I would not bring a dependency to an istio specific API.
created_at: 2025-04-19T07:18:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: Thanks will update the indentation.
created_at: 2025-04-19T07:19:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: There is no difference than any other configuration or component that the gardener controllers manage.
created_at: 2025-04-19T07:23:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: Yes we are already thinking about potential scenarios. At this stage it is too early to make a concrete proposal. It was explained in other comment.
created_at: 2025-04-19T07:24:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: Link to what?
created_at: 2025-04-19T07:25:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: Current `valitail` service will be replaced with `otel-collector` systemd service. There is no need to have 2 systemd services running in parallel. An implementation is already in progress and we are working out the details. Will share those if still needed. 
created_at: 2025-04-19T07:31:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
+  - **Event Logger Replacement:** In a later stage the current `event logger` shall be replaced with the `k8s-event` receiver in  the otel-collector shoot instance
+
+![Figure 2](assets/gep-34-shoot-observability-architecture.png)
+
+Example Systemd unit for the OpenTelemetry Collector in the shoot node:
+
+```ini
+[Unit]
+Description=opentelemetry-collector daemon
+Documentation=https://github.com/open-telemetry/opentelemetry-collector
+[Install]
+WantedBy=multi-user.target
+[Service]
+CPUAccounting=yes
+MemoryAccounting=yes
+CPUQuota=3%
+CPUQuotaPeriodSec=1000ms
+MemoryMin=29M
+MemoryHigh=400M
+MemoryMax=800M
+MemorySwapMax=0
+Restart=always
+RestartSec=5
+EnvironmentFile=/etc/environment
+ExecStartPre=/bin/sh -c "systemctl set-environment HOSTNAME=$(hostname | tr [:upper:] [:lower:])"
+ExecStart=/opt/bin/opentelemetry-collector --config=/var/lib/opentelemetry-collector/config/config
+```
+
+Example configuration for the OpenTelemetry Collector in the shoot node can be found [here](assets/gep-34-shoot-collector-config.yaml). For an overview of how it works, you can view the official [docs](https://opentelemetry.io/docs/collector/configuration/).
+
+A visualisation of the example config (Image generated with [otelbin](https://github.com/dash0hq/otelbin)):
+![Figure 3](assets/gep-34-collector-image.png)
+
+---
+
+## Alternatives
+
+### Direct Collector Deployment Without an Operator
+
+- **Description:** Deploy OpenTelemetry collectors in each shoot reconcile directly without using an operator.
+- **Pros:** Simplifies initial setup, avoids operator overhead.
+- **Cons:** Lacks scalability and management features (e.g., automatic upgrades, CR-based configuration), which are critical in Gardener’s multi-tenant environment.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: You are correct. But here I refer to a decision that has been made in past to decouple the lifecycle management of the external (to gardener) components to community supoprted operators the same as `fluent-bit` or `prometheus`. 
created_at: 2025-04-19T07:36:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
+  - **Event Logger Replacement:** In a later stage the current `event logger` shall be replaced with the `k8s-event` receiver in  the otel-collector shoot instance
+
+![Figure 2](assets/gep-34-shoot-observability-architecture.png)
+
+Example Systemd unit for the OpenTelemetry Collector in the shoot node:
+
+```ini
+[Unit]
+Description=opentelemetry-collector daemon
+Documentation=https://github.com/open-telemetry/opentelemetry-collector
+[Install]
+WantedBy=multi-user.target
+[Service]
+CPUAccounting=yes
+MemoryAccounting=yes
+CPUQuota=3%
+CPUQuotaPeriodSec=1000ms
+MemoryMin=29M
+MemoryHigh=400M
+MemoryMax=800M
+MemorySwapMax=0
+Restart=always
+RestartSec=5
+EnvironmentFile=/etc/environment
+ExecStartPre=/bin/sh -c "systemctl set-environment HOSTNAME=$(hostname | tr [:upper:] [:lower:])"
+ExecStart=/opt/bin/opentelemetry-collector --config=/var/lib/opentelemetry-collector/config/config
+```
+
+Example configuration for the OpenTelemetry Collector in the shoot node can be found [here](assets/gep-34-shoot-collector-config.yaml). For an overview of how it works, you can view the official [docs](https://opentelemetry.io/docs/collector/configuration/).
+
+A visualisation of the example config (Image generated with [otelbin](https://github.com/dash0hq/otelbin)):
+![Figure 3](assets/gep-34-collector-image.png)
+
+---
+
+## Alternatives
+
+### Direct Collector Deployment Without an Operator
+
+- **Description:** Deploy OpenTelemetry collectors in each shoot reconcile directly without using an operator.
+- **Pros:** Simplifies initial setup, avoids operator overhead.
+- **Cons:** Lacks scalability and management features (e.g., automatic upgrades, CR-based configuration), which are critical in Gardener’s multi-tenant environment.
+- **Reason Rejected:** The operator aligns with *GEP-19*’s precedent of using operators for observability components, ensuring consistency and scalability.
+
+---
+
+This GEP establishes a critical milestone in Gardener’s journey toward *Observability 2.0*. By deploying the OpenTelemetry operator and collectors, Gardener gains a standardized, extensible observability framework that integrates with its existing stack while paving the way for future enhancements like tracing and OTLP-native backends.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: Thanks, the summary is updated as requested.
created_at: 2025-04-19T07:36:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
+  - **Event Logger Replacement:** In a later stage the current `event logger` shall be replaced with the `k8s-event` receiver in  the otel-collector shoot instance
+
+![Figure 2](assets/gep-34-shoot-observability-architecture.png)
+
+Example Systemd unit for the OpenTelemetry Collector in the shoot node:
+
+```ini
+[Unit]
+Description=opentelemetry-collector daemon
+Documentation=https://github.com/open-telemetry/opentelemetry-collector
+[Install]
+WantedBy=multi-user.target
+[Service]
+CPUAccounting=yes
+MemoryAccounting=yes
+CPUQuota=3%
+CPUQuotaPeriodSec=1000ms
+MemoryMin=29M
+MemoryHigh=400M
+MemoryMax=800M
+MemorySwapMax=0
+Restart=always
+RestartSec=5
+EnvironmentFile=/etc/environment
+ExecStartPre=/bin/sh -c "systemctl set-environment HOSTNAME=$(hostname | tr [:upper:] [:lower:])"
+ExecStart=/opt/bin/opentelemetry-collector --config=/var/lib/opentelemetry-collector/config/config
+```
+
+Example configuration for the OpenTelemetry Collector in the shoot node can be found [here](assets/gep-34-shoot-collector-config.yaml). For an overview of how it works, you can view the official [docs](https://opentelemetry.io/docs/collector/configuration/).
+
+A visualisation of the example config (Image generated with [otelbin](https://github.com/dash0hq/otelbin)):
+![Figure 3](assets/gep-34-collector-image.png)
+
+---
+
+## Alternatives
+
+### Direct Collector Deployment Without an Operator
+
+- **Description:** Deploy OpenTelemetry collectors in each shoot reconcile directly without using an operator.
+- **Pros:** Simplifies initial setup, avoids operator overhead.
+- **Cons:** Lacks scalability and management features (e.g., automatic upgrades, CR-based configuration), which are critical in Gardener’s multi-tenant environment.
+- **Reason Rejected:** The operator aligns with *GEP-19*’s precedent of using operators for observability components, ensuring consistency and scalability.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: I do find it important though, we do use operators to manage the lifecycle of non-gardener components. A reason is to remove these specifics from the gardener code base switching to community employed practices and more importantly to bring the declaritive aspects managing those componets and their specifics.
created_at: 2025-04-19T07:42:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: vpnachev
body: I guess https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md ?
created_at: 2025-04-22T08:10:43Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the Observability 2.0 vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
+  - **Event Logger Replacement:** In a later stage the current `event logger` shall be replaced with the `k8s-event` receiver in  the otel-collector shoot instance
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: vpnachev
body: A link to the `k8s-event` receiver would be helpful here, is it https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver?
created_at: 2025-04-22T08:20:34Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the Observability 2.0 vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
+  - **Event Logger Replacement:** In a later stage the current `event logger` shall be replaced with the `k8s-event` receiver in  the otel-collector shoot instance
+
+![Figure 2](assets/gep-34-shoot-observability-architecture.png)
+
+Example Systemd unit for the OpenTelemetry Collector in the shoot node:
+
+```ini
+[Unit]
+Description=opentelemetry-collector daemon
+Documentation=https://github.com/open-telemetry/opentelemetry-collector
+[Install]
+WantedBy=multi-user.target
+[Service]
+CPUAccounting=yes
+MemoryAccounting=yes
+CPUQuota=3%
+CPUQuotaPeriodSec=1000ms
+MemoryMin=29M
+MemoryHigh=400M
+MemoryMax=800M
+MemorySwapMax=0
+Restart=always
+RestartSec=5
+EnvironmentFile=/etc/environment
+ExecStartPre=/bin/sh -c "systemctl set-environment HOSTNAME=$(hostname | tr [:upper:] [:lower:])"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: vpnachev
body: Dealing with the hostname will need to be done with some care, it is handled differently in the different infrastructures, some refs:
- https://github.com/gardener/gardener-extension-provider-openstack/blob/bbe65c6463add13a7f1bdbfd034e9970576ff2e5/pkg/webhook/controlplane/ensurer.go#L193
- https://github.com/gardener/gardener-extension-provider-gcp/blob/24467275e0bf2e6f8bc6713b5ba26339464094d0/pkg/webhook/controlplane/ensurer.go#L360
- https://github.com/gardener/gardener-extension-provider-aws/blob/5f900092991436b748ac8fc805adb3440d93ac54/pkg/webhook/controlplane/ensurer.go#L389
created_at: 2025-04-22T08:24:29Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: vpnachev
body: Do you plan to change how the agents on the node connect and authenticate with the endpoint in the seeds?
Just for completeness, it would be nice to be mentioned if the current solution with the `kube-rbac-proxy` will be preserved or replaced with another one.
created_at: 2025-04-22T08:32:30Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: rrhubenov
body: During initialisation, the collector will replace the template with the contents of the file.

We looked into this more carefully and tried to determine whether this would be a security vulnerability. We came to the conclusion that since this file is hosted on the node itself, anyone who has access to it and can do whatever they want with it.
If an unwanted actor gets access to either file(the collector config or the file containing the token) then that would mean that they had already previously gotten access to the whole machine. The current `valitail` configuration does almost the same thing. Only difference is that `valitail` receives the path to the file with the auth token and reads it, rather than it being embedded in the config file itself.
There is a way for this to be done the same as it currently is in valitail, if that approach is determined to be more appropriate.

We can discuss this in more detail if you have concerns that we haven't been able to address.
created_at: 2025-04-23T07:18:12Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: rrhubenov
body: Taking a step back and assessing the scenario, in the future implementation vali (or whatever log aggregator we'll use) won't be publicly accessible since the collector will theoretically act as the single point of access for logs (maybe all signals). 

That would mean that we can remove the ingress in front of Vali and remove the authentication as well.
The only authentication will be from the shoot nodes to the collector.
created_at: 2025-04-23T15:03:17Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the Observability 2.0 vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
+  - **Event Logger Replacement:** In a later stage the current `event logger` shall be replaced with the `k8s-event` receiver in  the otel-collector shoot instance
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: rrhubenov
body: Yes, thank you
created_at: 2025-04-23T15:16:04Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: rfranzke
body: I guess we should add a goal to also deploy it in the garden runtime cluster?
created_at: 2025-04-24T07:09:50Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: rfranzke
body: Same for virtual garden cluster?
created_at: 2025-04-24T07:10:06Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md)).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This means that during the `Seed` reconciliation flow the OpenTelemetryOperator will be deployed into the `garden` namespace.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: rfranzke
body: Which botanist workflow? Can you share a link? 
created_at: 2025-04-24T07:11:41Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md)).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This means that during the `Seed` reconciliation flow the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- **Configuration:** During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+        endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+          # Authenticate against Vali using a Bearer Token
+          Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: rfranzke
body: This doesn't look good - we probably don't want to put sensitive data like an auth token into an unencrypted resource.
created_at: 2025-04-24T07:13:03Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md)).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This means that during the `Seed` reconciliation flow the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- **Configuration:** During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+        endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+          # Authenticate against Vali using a Bearer Token
+          Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: vpnachev
body: Technically, it should be the same as the `gardener-node-agent` or `kubelet` credentials, no?
created_at: 2025-04-24T15:56:40Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md)).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This means that during the `Seed` reconciliation flow the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- **Configuration:** During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+        endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+          # Authenticate against Vali using a Bearer Token
+          Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: rfranzke
body: Valitail has has its own credentials/token as of today
created_at: 2025-04-25T07:13:45Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Probably just https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md unless you have something else in mind.
`Observability 2.0` by itself is not very helpful for people not in the details. Therefore, pointing to the overall story with a link helps a lot.
created_at: 2025-04-30T14:12:10Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: We usually link between GEPs in case they refer to each other. Hence, please add a link to GEP-19.
created_at: 2025-04-30T14:14:24Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md)).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This means that during the `Seed` reconciliation flow the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- **Configuration:** During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+        endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+          # Authenticate against Vali using a Bearer Token
+          Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: If I understand https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/bearertokenauthextension correctly it should be possible to use `auth.authenticator` and `extensions.bearertokenauth.filename`, which would allow the custom resource to stay clear of any secret, but still use the collector.
@nickytd / @rrhubenov WDYT?
created_at: 2025-04-30T14:30:51Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md)).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This means that during the `Seed` reconciliation flow the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- **Configuration:** During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+        endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+          # Authenticate against Vali using a Bearer Token
+          Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: rrhubenov
body: I'd like to give additional context so that we're on the same page when discussing this config. Sorry if I don't say anything new.

The configuration that we're discussing is the configuration of the `Collector` that will be deployed to the `Shoot Control Plane` via an `OpenTelemetryCollector` Custom Resource. The operator will handle the deployment of the Collector.

The `${file: <file_name>}` placeholder will load the contents of the given filename (from the {{ .pathAuthToken}} placeholder that is filled during yaml rendering in the shoot creation flow) in the configuration during the runtime of the collector. This means that the exposed part in the custom resource is the path to the file containing the token and not the token itself. The collector will expect that it has read permissions on that file and it will load it into its runtime when parsing the configuration file we're discussing. I am not sure whether having a path to the token file in the resource would be considered a vulnerability.

I'm not sure whether the comparison with the `valitail` configuration is appropriate in this context since `valitail` is running on the shoot nodes rather than the control plane. As far as I've understood in the current mechanism of `valitail` authentication, the `gardener-node-agent` syncs a secret with a file inside the node. This file contains the `valitail` auth token. The valitail config references the file and as I've outlined in the previous paragraph, it should be the same as the config we're discussing? (excluding the secret <-> file syncing from the `gardener-node-agent`)

I think this topic is interesting and can be evolved into a more general discussion of 'how would we want the node components to authenticate with the control-plane?'. If we're concerned with exposing tokens/secrets in the shoot nodes, I'd say the issue is not contained only to this discussion around the proposed collector.

With all that being said:
- Yes, the `bearertokenauthextension` that has been linked can absolutely be used. I'd even say it would be cleaner from an implementation standpoint but I'm not sure whether it handles the concern we're discussing. Thank you for proposing it, I'll most likely migrate to it rather than specifying the header itself if we continue with this approach.
- Maybe we'd like to talk about the other config that is for the collector that will be In the shoot nodes (the one that will actually replace the `valitail` component)? 
created_at: 2025-05-12T06:37:53Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: On a later phase that can achieved using the documented mechanisms for deploying components on runtime cluster as well. This is out of scope for the first iteration.
created_at: 2025-05-12T06:41:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: same 
created_at: 2025-05-12T06:42:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md)).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This means that during the `Seed` reconciliation flow the OpenTelemetryOperator will be deployed into the `garden` namespace.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: I am not aware of such link or docs.
created_at: 2025-05-12T06:42:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the Observability 2.0 vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
+  - **Event Logger Replacement:** In a later stage the current `event logger` shall be replaced with the `k8s-event` receiver in  the otel-collector shoot instance
+
+![Figure 2](assets/gep-34-shoot-observability-architecture.png)
+
+Example Systemd unit for the OpenTelemetry Collector in the shoot node:
+
+```ini
+[Unit]
+Description=opentelemetry-collector daemon
+Documentation=https://github.com/open-telemetry/opentelemetry-collector
+[Install]
+WantedBy=multi-user.target
+[Service]
+CPUAccounting=yes
+MemoryAccounting=yes
+CPUQuota=3%
+CPUQuotaPeriodSec=1000ms
+MemoryMin=29M
+MemoryHigh=400M
+MemoryMax=800M
+MemorySwapMax=0
+Restart=always
+RestartSec=5
+EnvironmentFile=/etc/environment
+ExecStartPre=/bin/sh -c "systemctl set-environment HOSTNAME=$(hostname | tr [:upper:] [:lower:])"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: Thanks for the proposal. It makes a lot of sense. Unifying the hostname is important in regard to showing the same observability signal field/attribute in logs. That is event more important to be unified with the observability logs from the kubelet service or any other node specific service that emits logs or metrics. Synergies are mandatory in this case. That being said we will check implementation details but the way kubelets fetch the hostnames shall be utilized also by the Collector as well, ensuring consistent field/attribute values. (That is not the case in current implementation with `valitail`)
created_at: 2025-05-13T09:12:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md)).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This means that during the `Seed` reconciliation flow the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- **Configuration:** During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+        endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+          # Authenticate against Vali using a Bearer Token
+          Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: rfranzke
body: > This means that the exposed part in the custom resource is the path to the file containing the token and not the token itself. The collector will expect that it has read permissions on that file and it will load it into its runtime when parsing the configuration file we're discussing. I am not sure whether having a path to the token file in the resource would be considered a vulnerability.

Okay, then it is fine. This isn't obvious from this line. I am not aware that you can send a header like this (`Authorization: Bearer <path-to-token-file>`). Can you please adapt this YAML example here so that it becomes more clear how this works (i.e., that only the path to the token file is referenced in the resource, not the token itself)?
created_at: 2025-05-13T12:44:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: rfranzke
body: @nickytd ?
created_at: 2025-05-13T12:44:43Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md)).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This means that during the `Seed` reconciliation flow the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- **Configuration:** During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+        endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+          # Authenticate against Vali using a Bearer Token
+          Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: rrhubenov
body: Of course. I agree that it's confusing so I'll be adapting this to the `bearertokenauthextension` that Johannes proposed. I'd first like to test it out so that I'm sure that it works as expected.
created_at: 2025-05-13T13:14:02Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,156 @@
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+**title:** Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+**gep-number:** [34]
+**creation-date:** 2025-04-12
+**status:** implementable
+**authors:**
+
+- "@nickytd"
+- "@rrhubenov"
+
+**reviewers:**
+
+- ""
+
+---
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in *GEP-19*, still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: nickytd
body: We discussed this point in length. Here is the summary:
- Initial implementation in `gep-34` preserves the current authentication (service account) and authorization (rbac) mechanism implemented in the `valitail`. There will be no semantical change to what is employed as of today. Goal is to focus on enabling the e2e scenarios described in `gep-34`, while keeping the changes and the scope under control.

However the goal in the next iteration is to move towards mTLS, especially if [Gateway API](https://gateway-api.sigs.k8s.io/api-types/gateway/) kicks in and provides support for [TLSRoutes](https://gateway-api.sigs.k8s.io/reference/spec/#tlsroute) and backend TLS termination with [BackednTLSPolicy](https://gateway-api.sigs.k8s.io/api-types/backendtlspolicy/) That initiative is rather separate and solves the general question of introducing mTLS configurations for various cluster service consumptions. So yes we want to evolve the current auth & autz mechanism and potentially move towards mTLS after we implemented the `gep-34` proposals, Moving towards GatewayAPI will also open the opportunity for getting rid of the nginx ingress controller especially if we solve the prometheus federation constraint. So it is interesting problem to solve per se with a significant scope. But with `gep-34` we are getting closer and we are laying the foundations for evolving further in the right direction. 
created_at: 2025-05-13T15:00:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md)).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This means that during the `Seed` reconciliation flow the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- **Configuration:** During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+        endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+          # Authenticate against Vali using a Bearer Token
+          Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: rrhubenov
body: I can confirm that it works as expected. Will update the GEP
created_at: 2025-05-14T11:02:15Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the Observability 2.0 vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
+  - **Event Logger Replacement:** In a later stage the current `event logger` shall be replaced with the `k8s-event` receiver in  the otel-collector shoot instance
+
+![Figure 2](assets/gep-34-shoot-observability-architecture.png)
+
+Example Systemd unit for the OpenTelemetry Collector in the shoot node:
+
+```ini
+[Unit]
+Description=opentelemetry-collector daemon
+Documentation=https://github.com/open-telemetry/opentelemetry-collector
+[Install]
+WantedBy=multi-user.target
+[Service]
+CPUAccounting=yes
+MemoryAccounting=yes
+CPUQuota=3%
+CPUQuotaPeriodSec=1000ms
+MemoryMin=29M
+MemoryHigh=400M
+MemoryMax=800M
+MemorySwapMax=0
+Restart=always
+RestartSec=5
+EnvironmentFile=/etc/environment
+ExecStartPre=/bin/sh -c "systemctl set-environment HOSTNAME=$(hostname | tr [:upper:] [:lower:])"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: rrhubenov
body: Since this is an implementation detail, we can discuss this at more length when the corresponding PRs are opened.
WDYT?
created_at: 2025-05-14T11:21:27Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,160 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry Operator and deploys OpenTelemetry Collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Please use relative links when possible, i.e.

```suggestion
This proposal introduces the OpenTelemetry Operator and deploys OpenTelemetry Collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](./19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
```
created_at: 2025-05-14T12:38:57Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,160 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry Operator and deploys OpenTelemetry Collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry Operator and Collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Please use relative links when possible, i.e.

```suggestion
Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](./19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry Operator and Collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
```
created_at: 2025-05-14T12:39:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,160 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry Operator and deploys OpenTelemetry Collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry Operator and Collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry Operator to manage Collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry Collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the Collectors receivers and wire the Collector exporters to the current logging backend.
+- **Replace valitail system service with OpenTelemetry Collectors on shoot nodes:** Replace current valitail system service on shoot nodes with an OpenTelemetry Collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md)).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry Operator and OpenTelemetry Collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry Collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry Operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Please use relative links when possible, i.e.

```suggestion
- **Details:** Deploy the OpenTelemetry Operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](./19-migrating-observability-stack-to-operators.md).
```
created_at: 2025-05-14T12:39:42Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,160 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry Operator and deploys OpenTelemetry Collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry Operator and Collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry Operator to manage Collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry Collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the Collectors receivers and wire the Collector exporters to the current logging backend.
+- **Replace valitail system service with OpenTelemetry Collectors on shoot nodes:** Replace current valitail system service on shoot nodes with an OpenTelemetry Collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md)).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry Operator and OpenTelemetry Collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry Collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry Operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
+- **Configuration:** The OpenTelemetry Operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This means that during the `Seed` reconciliation flow the OpenTelemetry Operator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated OpenTelemetry Collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry Collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry Operator.
+- **Configuration:** During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    extensions:
+      bearertokenauth:
+        filename: {{ .pathAuthToken }}
+
+    exporters:
+      loki:
+        endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        auth:
+          authenticator: bearertokenauth
+        tls:
+          ca_file: {{ .pathCACert }}
+
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      extensions: [bearertokenauth]
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Please use relative links when possible, i.e.

```suggestion
  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from [*GEP-19*](./19-migrating-observability-stack-to-operators.md).
```
created_at: 2025-05-14T12:40:04Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,160 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry Operator and deploys OpenTelemetry Collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry Operator and Collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry Operator to manage Collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry Collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the Collectors receivers and wire the Collector exporters to the current logging backend.
+- **Replace valitail system service with OpenTelemetry Collectors on shoot nodes:** Replace current valitail system service on shoot nodes with an OpenTelemetry Collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md)).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry Operator and OpenTelemetry Collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry Collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry Operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
+- **Configuration:** The OpenTelemetry Operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This means that during the `Seed` reconciliation flow the OpenTelemetry Operator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated OpenTelemetry Collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry Collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry Operator.
+- **Configuration:** During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    extensions:
+      bearertokenauth:
+        filename: {{ .pathAuthToken }}
+
+    exporters:
+      loki:
+        endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        auth:
+          authenticator: bearertokenauth
+        tls:
+          ca_file: {{ .pathCACert }}
+
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      extensions: [bearertokenauth]
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry Collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
+  - **Event Logger Replacement:** In a later stage the current `event logger` shall be replaced with the [`k8s-event` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver) in the Collector shoot instance.
+
+![Figure 2](assets/gep-34-shoot-observability-architecture.png)
+
+Example Systemd unit for the OpenTelemetry Collector in the shoot node:
+
+```ini
+[Unit]
+Description=opentelemetry-collector daemon
+Documentation=https://github.com/open-telemetry/opentelemetry-collector
+[Install]
+WantedBy=multi-user.target
+[Service]
+CPUAccounting=yes
+MemoryAccounting=yes
+CPUQuota=3%
+CPUQuotaPeriodSec=1000ms
+MemoryMin=29M
+MemoryHigh=400M
+MemoryMax=800M
+MemorySwapMax=0
+Restart=always
+RestartSec=5
+EnvironmentFile=/etc/environment
+ExecStartPre=/bin/sh -c "systemctl set-environment HOSTNAME=$(hostname | tr [:upper:] [:lower:])"
+ExecStart=/opt/bin/opentelemetry-collector --config=/var/lib/opentelemetry-collector/config/config
+```
+
+Example configuration for the OpenTelemetry Collector in the shoot node can be found [here](assets/gep-34-shoot-collector-config.yaml). For an overview of how it works, you can view the official [docs](https://opentelemetry.io/docs/collector/configuration/).
+
+A visualisation of the example config (Image generated with [otelbin](https://github.com/dash0hq/otelbin)):
+![Figure 3](assets/gep-34-collector-image.png)
+
+---
+
+## Alternatives
+
+### Direct Collector Deployment Without an Operator
+
+- **Description:** Deploy OpenTelemetry Collectors in each shoot reconcile directly without using an operator.
+- **Pros:** Simplifies initial setup, avoids operator overhead.
+- **Cons:** Lacks scalability and management features (e.g., automatic upgrades, CR-based configuration), which are critical in Gardener’s multi-tenant environment.
+- **Reason Rejected:** The operator aligns with [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md)’s precedent of using operators for observability components, ensuring consistency and scalability.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Please use relative links when possible, i.e.

```suggestion
- **Reason Rejected:** The operator aligns with [*GEP-19*](./19-migrating-observability-stack-to-operators.md)’s precedent of using operators for observability components, ensuring consistency and scalability.
```
created_at: 2025-05-14T12:40:40Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,160 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: You might want to add some reviewers here. It seems at least four people looked at the GEP.
created_at: 2025-05-14T12:52:33Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,160 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: ```suggestion
# GEP-34: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
```
created_at: 2025-05-14T12:53:09Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,160 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry Operator and deploys OpenTelemetry Collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry Operator and Collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry Operator to manage Collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry Collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the Collectors receivers and wire the Collector exporters to the current logging backend.
+- **Replace valitail system service with OpenTelemetry Collectors on shoot nodes:** Replace current valitail system service on shoot nodes with an OpenTelemetry Collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md)).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry Operator and OpenTelemetry Collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the [*Observability 2.0*](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry Collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry Operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
+- **Configuration:** The OpenTelemetry Operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This means that during the `Seed` reconciliation flow the OpenTelemetry Operator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated OpenTelemetry Collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry Collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry Operator.
+- **Configuration:** During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    extensions:
+      bearertokenauth:
+        filename: {{ .pathAuthToken }}
+
+    exporters:
+      loki:
+        endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        auth:
+          authenticator: bearertokenauth
+        tls:
+          ca_file: {{ .pathCACert }}
+
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      extensions: [bearertokenauth]
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md).
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry Collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
+  - **Event Logger Replacement:** In a later stage the current `event logger` shall be replaced with the [`k8s-event` receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver) in the Collector shoot instance.
+
+![Figure 2](assets/gep-34-shoot-observability-architecture.png)
+
+Example Systemd unit for the OpenTelemetry Collector in the shoot node:
+
+```ini
+[Unit]
+Description=opentelemetry-collector daemon
+Documentation=https://github.com/open-telemetry/opentelemetry-collector
+[Install]
+WantedBy=multi-user.target
+[Service]
+CPUAccounting=yes
+MemoryAccounting=yes
+CPUQuota=3%
+CPUQuotaPeriodSec=1000ms
+MemoryMin=29M
+MemoryHigh=400M
+MemoryMax=800M
+MemorySwapMax=0
+Restart=always
+RestartSec=5
+EnvironmentFile=/etc/environment
+ExecStartPre=/bin/sh -c "systemctl set-environment HOSTNAME=$(hostname | tr [:upper:] [:lower:])"
+ExecStart=/opt/bin/opentelemetry-collector --config=/var/lib/opentelemetry-collector/config/config
+```
+
+Example configuration for the OpenTelemetry Collector in the shoot node can be found [here](assets/gep-34-shoot-collector-config.yaml). For an overview of how it works, you can view the official [docs](https://opentelemetry.io/docs/collector/configuration/).
+
+A visualisation of the example config (Image generated with [otelbin](https://github.com/dash0hq/otelbin)):
+![Figure 3](assets/gep-34-collector-image.png)
+
+---
+
+## Alternatives
+
+### Direct Collector Deployment Without an Operator
+
+- **Description:** Deploy OpenTelemetry Collectors in each shoot reconcile directly without using an operator.
+- **Pros:** Simplifies initial setup, avoids operator overhead.
+- **Cons:** Lacks scalability and management features (e.g., automatic upgrades, CR-based configuration), which are critical in Gardener’s multi-tenant environment.
+- **Reason Rejected:** The operator aligns with [*GEP-19*](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md)’s precedent of using operators for observability components, ensuring consistency and scalability.
+
+---
+
+This GEP establishes a critical milestone in Gardener’s journey toward *Observability 2.0*. By deploying the OpenTelemetry Operator and OpenTelemetry Collectors, Gardener gains a standardized, extensible observability framework that integrates with its existing stack while paving the way for future enhancements like tracing and OpenTelemetry native backends.
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: ScheererJ
body: Please add another link to the Observability 2.0 document.
created_at: 2025-05-14T13:03:05Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,153 @@
+---
+title: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+gep-number: 34
+creation-date: 2025-04-12
+status: implementable
+authors:
+- "@nickytd"
+- "@rrhubenov"
+reviewers:
+-
+---
+
+# GEP-[34]: Introducing OpenTelemetry Operator and Collectors in Shoot Control Planes
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Alternatives](#alternatives)
+
+---
+
+## Summary
+
+This proposal introduces the OpenTelemetry operator and deploys OpenTelemetry collectors within the shoot control plane namespaces of Gardener-managed clusters. Building on the foundation laid by [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md) (migration to Prometheus and Fluent-bit operators) and the [Observability 2.0](https://github.com/gardener/logging/blob/master/docs/observability-2.0/Observability%202.0.md) vision, this GEP advances Gardener's observability stack by adopting OpenTelemetry standards. The initiative aims to layout the foundation for building a more cohesive and interoperable observability framework. This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the Observability 2.0 vision, with future enhancements planned to include metrics and optionally traces.
+
+---
+
+## Motivation
+
+Gardener's current observability stack, while improved by the operator-based approach in [GEP-19](https://github.com/gardener/gardener/blob/master/docs/proposals/19-migrating-observability-stack-to-operators.md), still relies on vendor specific format and protocols for collecting, processing and storing observability signals. This brings challenges in extending both components and scenarios used to process these signals and enforces lock-in integration scenarios with external consumers. By introducing the OpenTelemetry operator and collectors, this proposal addresses these challenges, aligning Gardener with industry trends and preparing it for future observability enhancements, such as unified visualization and tracing support.
+
+### Goals
+
+- **Deploy the OpenTelemetry Operator on Seed Clusters:** Install and configure the OpenTelemetry operator to manage otel-collector instances across shoot control plane namespaces.
+- **Deploy Gardener OpenTelemetry Collector Distribution in Shoot Control Planes:** Provision a dedicated Gardener OpenTelemetry collector distribution instance via the OpenTelemetry Operator in each shoot control plane namespace to handle observability data.
+- **Transport Logs signals through the OpenTelemetry Collector Distribution:** Configure fluent-bits to send logs to the OTel Collectors receivers and wire the otel collector exporters to the current logging backend.
+- **Replace valitail system service with otel-collectors on shoot nodes:** Replace current valitail system service on shoot nodes with otel-collector, keeping parity in the collected type of logs.
+
+### Non-Goals
+
+- **Immediate Replacement of Existing Tools:** This proposal does not aim to immediately decommission tools like Vali, Fluent-bit, or Prometheus; the migration will be phased.
+- **Unified Visualization Implementation:** Developing a single UI for logs, metrics, and traces is out of scope, as no mature open-source solution currently exists (per *Observability 2.0*).
+- **Full Tracing Adoption:** While tracing support is enabled, widespread instrumentation of Gardener components for distributed traces is a future step.
+
+---
+
+## Proposal
+
+This section outlines the implementation plan to introduce the OpenTelemetry operator and collectors into Gardener's shoot control plane namespaces and in shoot nodes as a systemd service, aligning with Step 2 of the *Observability 2.0* roadmap.
+
+### 1. Deploy the OpenTelemetry Operator on Seed Clusters
+
+- **Objective:** Establish centralized management of OpenTelemetry collectors across all shoot control planes.
+- **Details:** Deploy the OpenTelemetry operator (e.g., from the [OpenTelemetry Operator for Kubernetes](https://github.com/open-telemetry/opentelemetry-operator)) in the `garden` namespace of each seed cluster using `ManagedResources`, similar to the Prometheus operator deployment in *GEP-19*.
+- **Configuration:** The operator will be deployed following the `botanist` workflow in the same way as the current `prometheus` and `fluent-bit` operators. This meaning that during the `Seed` reconciliation flow, the OpenTelemetryOperator will be deployed into the `garden` namespace.
+
+### 2. Create OpenTelemetry Collector Instances per Shoot
+
+- **Objective:** Provide each shoot control plane with a dedicated collector to process its observability data.
+- **Details:** For each shoot, deploy an OpenTelemetry collector instance in its control plane namespace (e.g., `shoot--project--name`). The collector will run as a `Deployment` managed by the OpenTelemetry operator.
+- Configuration: During the `Shoot` reconciliation flow, an [`OpenTelemetryCollector` Custom Resource](https://github.com/open-telemetry/opentelemetry-operator/blob/main/apis/v1beta1/opentelemetrycollector_types.go) will be created in the `Shoot` namespace. This Custom Resource will trigger the Operator to create an OpenTelemetry Collector Deployment & Ingress inside the namespace.
+- **Example OpenTelemetryCollector CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  labels:
+    resources.gardener.cloud/managed-by: gardener
+  name: opentelemetry-collector
+spec:
+  mode: deployment
+  config:
+    exporters:
+      loki:
+      endpoint: https://v-local--local.ingress.local.seed.local.gardener.cloud/vali/api/v1/push
+        headers:
+        # Authenticate against Vali using a Bearer Token
+        Authorization: "Bearer ${file:{{ .pathAuthToken }}}"
+    receivers:
+      otlp:
+        protocols:
+          http:
+            endpoint: 0.0.0.0:4317
+    service:
+      pipelines:
+        logs:
+          exporters:
+          - loki
+          receivers:
+          - otlp
+```
+
+![Figure 1](assets/gep-34-otel-operator-architecture.png)
+
+### 3. Integrate with Existing Components
+
+- **Objective:** Ensure a smooth transition by interfacing with the current observability stack.
+- **Details:**
+  - **Fluent-bit Integration:** Configure Fluent-bit to forward logs to the collector’s `loki` receiver, maintaining compatibility with the Vali-based setup from *GEP-19*.
+  - **Vali Integration:** Use the `loki` exporter to output the shoot logs to the existing Vali backend.
+  - **Valitail Replacement:** Replace the current `valitail` system service on shoot nodes with the OpenTelemetry collector, ensuring it collects the same types of logs as before. This will be done by deploying a systemd service for the collector on each shoot node, in the same way as the current `valitail` service.
+  - **Event Logger Replacement:** In a later stage the current `event logger` shall be replaced with the `k8s-event` receiver in  the otel-collector shoot instance
+
+![Figure 2](assets/gep-34-shoot-observability-architecture.png)
+
+Example Systemd unit for the OpenTelemetry Collector in the shoot node:
+
+```ini
+[Unit]
+Description=opentelemetry-collector daemon
+Documentation=https://github.com/open-telemetry/opentelemetry-collector
+[Install]
+WantedBy=multi-user.target
+[Service]
+CPUAccounting=yes
+MemoryAccounting=yes
+CPUQuota=3%
+CPUQuotaPeriodSec=1000ms
+MemoryMin=29M
+MemoryHigh=400M
+MemoryMax=800M
+MemorySwapMax=0
+Restart=always
+RestartSec=5
+EnvironmentFile=/etc/environment
+ExecStartPre=/bin/sh -c "systemctl set-environment HOSTNAME=$(hostname | tr [:upper:] [:lower:])"
path: docs/proposals/34-observability2.0-opentelemtry-operator-and-collectors.md
user: login: vpnachev
body: Agree. You can even remove the `ExecStartPre` line from the proposal.
created_at: 2025-05-14T15:44:12Z
author_association: MEMBER

PR comments:
user: login: ScheererJ
created_at: 2025-04-16T13:05:40Z
body: /assign
author_association: MEMBER
user: login: nickytd
created_at: 2025-04-19T06:42:38Z
body: > Thanks for the GEP! I was wondering if the migration will unavoidably lead to some loss of logs. If yes do you have some sort of a rough estimation how much logs will be lost?

Although not directly related to this proposal, the question is still relevant as we plan migration for the backend in some near future. `victoria-logs` have planned a data migration tool https://docs.victoriametrics.com/victorialogs/roadmap, which may facilitate the transition in our setup, or we can do something similar. Will decide where we are close to this phase. 
author_association: CONTRIBUTOR
user: login: nickytd
created_at: 2025-04-19T06:45:51Z
body: > Thanks for providing a glimpse into what is planned for observability 2.0.
> 
> Please take a minute to clarify the scope of the GEP. Title, summary and motivation made me think that this has far more reach than it actually intends to implement according to the proposal section.

Following clarification will be added to address the given feedback.
```
This GEP takes the first step by introducing OpenTelemetry for logging, aligning with the Observability 2.0 vision, with future enhancements planned to include metrics and traces.
```
author_association: CONTRIBUTOR
user: login: nickytd
created_at: 2025-04-19T07:46:21Z
body: @ScheererJ thanks for the thorough read and the suggestions. Will fix the indentation and together with @rrhubenov we will provide more details around the `valitail` service replacement based on the current PoC work and implementation. 
author_association: CONTRIBUTOR
user: login: rrhubenov
created_at: 2025-05-14T11:22:37Z
body: @ScheererJ @rfranzke Appreciate the feedback. Hopefully we were able to address it all. Can we proceed with merging this GEP?
author_association: MEMBER
user: login: ScheererJ
created_at: 2025-05-15T07:20:41Z
body: /lgtm
author_association: MEMBER
user: login: gardener-prow[bot]
created_at: 2025-05-15T07:20:47Z
body: LGTM label has been added.  <details>Git tree hash: eefc5d328586714138e3cdc16d89d8da31727094</details>
author_association: CONTRIBUTOR
user: login: rrhubenov
created_at: 2025-05-15T08:16:41Z
body: Sorry, had to remove the `ExecStartPre` from the systemd unit
author_association: MEMBER
user: login: vpnachev
created_at: 2025-05-15T08:19:56Z
body: /lgtm
author_association: MEMBER
user: login: gardener-prow[bot]
created_at: 2025-05-15T08:20:02Z
body: LGTM label has been added.  <details>Git tree hash: e21975a3a4dca7b642026f33e283be123783eac1</details>
author_association: CONTRIBUTOR
user: login: vpnachev
created_at: 2025-05-19T08:13:46Z
body: Having in mind this is just an enhancement proposal, I guess it makes no sense to waste resources trying to get the test checks pass. 

Command like should let prow merge the PR as it is now, I would appreciate if an admin runs it.

> /override pull-gardener-integration pull-gardener-unit pull-gardener-e2e-kind-upgrade pull-gardener-e2e-kind-ha-single-zone-upgrade pull-gardener-e2e-kind-operator-seed pull-gardener-e2e-kind-ha-multi-zone-upgrade
author_association: MEMBER
user: login: rfranzke
created_at: 2025-05-19T09:14:25Z
body: /approve
/override pull-gardener-integration pull-gardener-unit pull-gardener-e2e-kind-upgrade pull-gardener-e2e-kind-ha-single-zone-upgrade pull-gardener-e2e-kind-operator-seed pull-gardener-e2e-kind-ha-multi-zone-upgrade
author_association: MEMBER
user: login: gardener-prow[bot]
created_at: 2025-05-19T09:14:31Z
body: @rfranzke: Overrode contexts on behalf of rfranzke: pull-gardener-e2e-kind-ha-multi-zone-upgrade, pull-gardener-e2e-kind-ha-single-zone-upgrade, pull-gardener-e2e-kind-operator-seed, pull-gardener-e2e-kind-upgrade, pull-gardener-integration, pull-gardener-unit

<details>

In response to [this](https://github.com/gardener/gardener/pull/11861#issuecomment-2890258717):

>/approve
>/override pull-gardener-integration pull-gardener-unit pull-gardener-e2e-kind-upgrade pull-gardener-e2e-kind-ha-single-zone-upgrade pull-gardener-e2e-kind-operator-seed pull-gardener-e2e-kind-ha-multi-zone-upgrade


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: CONTRIBUTOR
user: login: gardener-prow[bot]
created_at: 2025-05-19T09:14:32Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/gardener/gardener/pull/11861#issuecomment-2890258717" title="Approved">rfranzke</a>*

The full list of commands accepted by this bot can be found [here](https://prow.gardener.cloud/command-help?repo=gardener%2Fgardener).

The pull request process is described [here](https://gardener.cloud/docs/contribute/#pull-request-checklist)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/gardener/gardener/blob/master/OWNERS)~~ [rfranzke]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR