Repository: apache/airflow

Issue:
Reduce irrelevant ERROR level logs from KubernertesPodOperator on pod runtime failure
### Apache Airflow version

Other Airflow 2 version (please specify below)

### What happened

Using KPO that fails on runtime turning on `log_events_on_failure`, using a trivial example,

```python
KubernetesPodOperator(
  task_id=...,
  image=...,
  cmds=["/bin/bash", "-c", "-x"],
  arguments=["eccho"],  # typo
  log_events_on_failure=True,
)
```

returns various lines of logs

```
[2023-12-05, 02:53:08 UTC] {before.py:40} INFO - Starting call to 'airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager.fetch_container_logs.<locals>.consume_logs', this is the 1st time calling it.
[2023-12-05, 02:53:08 UTC] {pod_manager.py:381} INFO - + eccho
[2023-12-05, 02:53:08 UTC] {pod_manager.py:381} INFO - bash: eccho: command not found
[2023-12-05, 02:53:10 UTC] {pod.py:720} ERROR - Pod Event: Scheduled - Successfully assigned airflow/redacted_pod_name
[2023-12-05, 02:53:10 UTC] {pod.py:720} ERROR - Pod Event: Pulling - Pulling image "redacted:latest"
[2023-12-05, 02:53:10 UTC] {pod.py:720} ERROR - Pod Event: Pulled - Successfully pulled image "redacted:latest" in 142.533921ms (142.546182ms including waiting)
[2023-12-05, 02:53:10 UTC] {pod.py:720} ERROR - Pod Event: Created - Created container base
[2023-12-05, 02:53:10 UTC] {pod.py:720} ERROR - Pod Event: Started - Started container base
[2023-12-05, 02:53:10 UTC] {pod.py:726} INFO - Deleting pod: redacted_pod_name
[2023-12-05, 02:53:10 UTC] {taskinstance.py:1824} ERROR - Task failed with exception
Traceback (most recent call last):
remote_pod: {'api_version': 'v1',
 'kind': 'Pod',
 'metadata': {'annotations': None,
  ... # VERY LONG JSON
}
...
```

that are irrelevant for the reason of failure (i.e., the runtime container exited nonzero with $STDERR)

### What you think should happen instead

- Use reasonable log level for each pod event type & reason 
  - Normal->`logging.DEBUG`
  - (Warning, Failed)->`logging.ERROR`
- Print pod runtime stderr to `logging.ERROR`

### How to reproduce

See above.

### Operating System

Debian Bookworm

### Versions of Apache Airflow Providers

```
apache-airflow==2.6.2
apache-airflow-providers-amazon==8.2.0
apache-airflow-providers-cncf-kubernetes==7.1.0
apache-airflow-providers-common-sql==1.5.1
apache-airflow-providers-ftp==3.4.1
apache-airflow-providers-http==4.4.1
apache-airflow-providers-imap==3.2.1
apache-airflow-providers-postgres==5.5.1
apache-airflow-providers-sqlite==3.4.1
```

### Deployment

Docker-Compose

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Related closed issue with a vague scope https://github.com/apache/airflow/issues/10861

---

The logging all events as ERROR still exists in main.

https://github.com/apache/airflow/blob/ace97c0f0b64bd003bafa50e02cdfc10a8303936/airflow/providers/cncf/kubernetes/operators/pod.py#L780-L785

---

Understandably, Kubernetes does not isolate stdout from stderr in its container log API, which is why KPO was implemented to container logs to INFO. 

The options I can think of is make this function aware of the container status (beyond just running)

https://github.com/apache/airflow/blob/ace97c0f0b64bd003bafa50e02cdfc10a8303936/airflow/providers/cncf/kubernetes/utils/pod_manager.py#L398-L473
 
or give the user an ability to write stderr (or anything) to `/dev/termination-log` for KPO (same can be done on @task.kubernetes with the task logger)

https://kubernetes.io/docs/tasks/debug/debug-application/determine-reason-pod-failure/

This seems last part seems out of the scope. I can make another ticket if interested.
Hi, I am an [MLH fellowship](https://fellowship.mlh.io/) intern at **Airflow**. I would like to work on this issue. 
Assigned you :)
Hi @ketozhang 

I'm currently trying to reproduce the problem. I've set up [these test dags](https://github.com/sudiptob2/airflow-kpo-test/tree/main/dev/dags). As per the settings, I should receive pod events in the log for `test_fail_dag`. Despite setting `log_events_on_failure=True`, I haven't received any pod events. There may be a misconfiguration in my dag files.
I'm working with Airflow version `2.8.1`, although this shouldn't be the reason why pod events aren't being logged.

Any thoughts on how to reproduce the issue?
Test run log: [Here](https://github.com/sudiptob2/airflow-kpo-test/blob/71d0a9461f9d402e5ae76d5114a63ae1d425c130/data/dag_id%3Dtest_fail_dag/run_id%3Dmanual__2024-02-15T04%3A52%3A23.659234%2B00%3A00/task_id%3Dtest_fail_task/attempt%3D1.log)
Ref: https://github.com/sudiptob2/airflow-kpo-test
A pod event failure happens on the Kubernetes side when a Pod fails. An example of a failure is when the pod requested resource is much higher than what's available in the cluster.

In your test, your pod succeeded to launch, successfully ran the Python code despite Python raising an Exception. This is not a Pod failure since it did what it's suppose to do (run the Python code).

You can try various realistic scenarios like requesting a large number of CPU and RAM.

Perhaps others here can help point you to any existing testing cases that demonstrate a Pod failure either with real or mocked scenarios.
Thanks 🚀  I was also thinking about that but wasn't sure. 
I attempted to update DAGs in various ways and tried to generate a Pod failure event. However, I have not been able to generate a log for the Pod failure event. 

DAGs I am using: [dev/dags](https://github.com/sudiptob2/airflow-kpo-test/tree/main/dev/dags)
Test run log for [dev/dags/test_fail_dag.py](https://github.com/sudiptob2/airflow-kpo-test/blob/main/dev/dags/test_fail_dag.py):   [LOGFILE](https://github.com/sudiptob2/airflow-kpo-test/blob/53576ff2f3cf3a30781448a945a5e1dbf24808a1/data/dag_id%3Dtest_fail_dag/run_id%3Dmanual__2024-02-15T17%3A25%3A48.698626%2B00%3A00/task_id%3Dtest_fail_task/attempt%3D1.log)
Ref: https://github.com/sudiptob2/airflow-kpo-test

Any feedback from experienced **user/maintainer** would be helpful.
You seem to be getting a different error more so about Kubernetes API permission issue which isn't related to the Pod.

Let's try to use reduce your DAG file into minimal viable example:

```python
dag = DAG('test_fail_dag', schedule_interval='@once', catchup=False)

task = KubernetesPodOperator(
    task_id='test_fail_task',
    dag=dag,
    cmds=["/bin/bash", "-c", "-x"],
    arguments=["eccho"],  # purposeful typo,
    container_resources={"requests": {"memory": "1000Gi", "cpu": "1000"}},
    log_events_on_failure=True,
)
```

Assuming your cluster isn't that large, this should give you in the logs something like `Pod Event: FailedScheduling - ...`
@ketozhang Thanks! will try this shortly
If you wouldn't mind. Can you do one run with and another without `container_resources` set?

I forgot the original issue I outlined here was we are getting Pod Event failures when we are not supposed to. With `container_resources`, you should expected a Pod Event failure. Without `container_resources`, you should expect no Pod Event failure — I did see one which is why I raised this ticket.
Yes sure, Just to mention, I already did try [this](https://github.com/sudiptob2/airflow-kpo-test/blob/main/dev/dags/test_fail_dag_2.py) Dag which is similar to the one you suggested. But did not get Pod event. 

Testing your suggested DAG now.
Hi @ketozhang 
I ran 2 dags according to your suggestion. But could not get the Pod Event log.
1. [minimal_test_fail_dag_without_cr.py](https://github.com/sudiptob2/airflow-kpo-test/blob/main/dev/dags/minimal_test_fail_dag_without_cr.py) : Running without `container_resource` set, See [Logfile](https://github.com/sudiptob2/airflow-kpo-test/blob/abb61c5d67e4961a41584b9fa89973d8e5b51fbd/data/dag_id%3Dminimal_test_fail_task_without_cr/run_id%3Dscheduled__2022-01-01T00%3A00%3A00%2B00%3A00/task_id%3Dminimal_test_fail_task_without_cr/attempt%3D1.log)
2. [minimal_test_fail.py](https://github.com/sudiptob2/airflow-kpo-test/blob/main/dev/dags/minimal_test_fail_dag.py): Run with container resources set, See [Logfile](https://github.com/sudiptob2/airflow-kpo-test/blob/abb61c5d67e4961a41584b9fa89973d8e5b51fbd/data/dag_id%3Dminimal_test_fail_dag/run_id%3Dscheduled__2022-01-01T00%3A00%3A00%2B00%3A00/task_id%3Dminimal_test_fail_task/attempt%3D1.log).
It looks like you're not getting any event failures because of some Kubernetes permission issue. In your logs, you're getting a 403 error:

> HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"events is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"events\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"kind":"events"},"code":403}

I'm not familiar with the Helm chart you're using, ` airflow-k8s-maxcotec`. I'd look into your chart's documentation or try a different helm chart.
Thanks for the hint ✅ 
@ketozhang Thank you so much. There were problem in RBAC, I updated it and now I can get Pod events

```
[2024-02-15, 23:26:26 UTC] {pod.py:980} ERROR - (403)
[2024-02-15, 23:26:26 UTC] {taskinstance.py:2698} ERROR - Task failed with exception
[2024-02-15, 23:26:26 UTC] {standard_task_runner.py:107} ERROR - Failed to execute job 2 for task minimal_test_fail_task_without_cr (Pod minimal-test-fail-task-without-cr-rp9u952i returned a failure.
[2024-02-16, 00:45:26 UTC] {pod.py:732} ERROR - Pod Event: Scheduled - Successfully assigned default/minimal-test-fail-task-without-cr-kfnmnrcd to kind-worker
[2024-02-16, 00:45:26 UTC] {pod.py:732} ERROR - Pod Event: Pulling - Pulling image "alpine:latest"
[2024-02-16, 00:45:26 UTC] {pod.py:732} ERROR - Pod Event: Pulled - Successfully pulled image "alpine:latest" in 566ms (566ms including waiting)
[2024-02-16, 00:45:26 UTC] {pod.py:732} ERROR - Pod Event: Created - Created container base
[2024-02-16, 00:45:26 UTC] {pod.py:732} ERROR - Pod Event: Started - Started container base
[2024-02-16, 00:45:26 UTC] {taskinstance.py:2698} ERROR - Task failed with exception
[2024-02-16, 00:45:26 UTC] {standard_task_runner.py:107} ERROR - Failed to execute job 2 for task minimal_test_fail_task_without_cr (Pod minimal-test-fail-task-without-cr-kfnmnrcd returned a failure.
```

Resolving patch:
diff --git a/airflow/providers/cncf/kubernetes/operators/pod.py b/airflow/providers/cncf/kubernetes/operators/pod.py
--- a/airflow/providers/cncf/kubernetes/operators/pod.py
+++ b/airflow/providers/cncf/kubernetes/operators/pod.py
@@ -27,6 +27,7 @@
 import warnings
 from collections.abc import Container
 from contextlib import AbstractContextManager
+from enum import Enum
 from functools import cached_property
 from typing import TYPE_CHECKING, Any, Callable, Iterable, Sequence
 
@@ -95,6 +96,13 @@
 KUBE_CONFIG_ENV_VAR = "KUBECONFIG"
 
 
+class PodEventType(Enum):
+    """Type of Events emitted by kubernetes pod."""
+
+    WARNING = "Warning"
+    NORMAL = "Normal"
+
+
 class PodReattachFailure(AirflowException):
     """When we expect to be able to find a pod but cannot."""
 
@@ -548,8 +556,7 @@ def await_pod_start(self, pod: k8s.V1Pod):
             )
         except PodLaunchFailedException:
             if self.log_events_on_failure:
-                for event in self.pod_manager.read_pod_events(pod).items:
-                    self.log.error("Pod Event: %s - %s", event.reason, event.message)
+                self._read_pod_events(pod, reraise=False)
             raise
 
     def extract_xcom(self, pod: k8s.V1Pod):
@@ -855,7 +862,10 @@ def _read_pod_events(self, pod, *, reraise=True):
         """Will fetch and emit events from pod."""
         with _optionally_suppress(reraise=reraise):
             for event in self.pod_manager.read_pod_events(pod).items:
-                self.log.error("Pod Event: %s - %s", event.reason, event.message)
+                if event.type == PodEventType.NORMAL.value:
+                    self.log.info("Pod Event: %s - %s", event.reason, event.message)
+                else:
+                    self.log.error("Pod Event: %s - %s", event.reason, event.message)
 
     def is_istio_enabled(self, pod: V1Pod) -> bool:
         """Check if istio is enabled for the namespace of the pod by inspecting the namespace labels."""


Test patch:
diff --git a/tests/providers/cncf/kubernetes/operators/test_pod.py b/tests/providers/cncf/kubernetes/operators/test_pod.py
--- a/tests/providers/cncf/kubernetes/operators/test_pod.py
+++ b/tests/providers/cncf/kubernetes/operators/test_pod.py
@@ -24,14 +24,18 @@
 
 import pendulum
 import pytest
-from kubernetes.client import ApiClient, V1PodSecurityContext, V1PodStatus, models as k8s
+from kubernetes.client import ApiClient, V1Pod, V1PodSecurityContext, V1PodStatus, models as k8s
 from urllib3 import HTTPResponse
 
 from airflow.exceptions import AirflowException, AirflowSkipException, TaskDeferred
 from airflow.models import DAG, DagModel, DagRun, TaskInstance
 from airflow.models.xcom import XCom
 from airflow.providers.cncf.kubernetes import pod_generator
-from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator, _optionally_suppress
+from airflow.providers.cncf.kubernetes.operators.pod import (
+    KubernetesPodOperator,
+    PodEventType,
+    _optionally_suppress,
+)
 from airflow.providers.cncf.kubernetes.secret import Secret
 from airflow.providers.cncf.kubernetes.triggers.pod import KubernetesPodTrigger
 from airflow.providers.cncf.kubernetes.utils.pod_manager import (
@@ -2154,3 +2158,39 @@ def test_async_skip_kpo_wait_termination_with_timeout_event(mock_manager, mocked
 
     # assert that the cleanup is called
     post_complete_action.assert_called_once()
+
+
+@patch("airflow.providers.cncf.kubernetes.operators.pod.KubernetesPodOperator.pod_manager")
+@patch("airflow.providers.cncf.kubernetes.operators.pod.KubernetesPodOperator.log")
+def test_read_pod_events(mock_log, mock_pod_manager):
+    # Create a mock pod
+    pod = V1Pod()
+
+    # Create mock events
+    mock_event_normal = MagicMock()
+    mock_event_normal.type = PodEventType.NORMAL.value
+    mock_event_normal.reason = "test-reason-normal"
+    mock_event_normal.message = "test-message-normal"
+
+    mock_event_error = MagicMock()
+    mock_event_error.type = PodEventType.WARNING.value
+    mock_event_error.reason = "test-reason-error"
+    mock_event_error.message = "test-message-error"
+
+    mock_pod_manager.read_pod_events.return_value.items = [mock_event_normal, mock_event_error]
+
+    operator = KubernetesPodOperator(task_id="test-task")
+    operator._read_pod_events(pod)
+
+    # Assert that event with type `Normal` is logged as info.
+    mock_log.info.assert_called_once_with(
+        "Pod Event: %s - %s",
+        mock_event_normal.reason,
+        mock_event_normal.message,
+    )
+    # Assert that event with type `Warning` is logged as error.
+    mock_log.error.assert_called_once_with(
+        "Pod Event: %s - %s",
+        mock_event_error.reason,
+        mock_event_error.message,
+    )