Repository: apache/airflow

Issue:
KubernetesPodOperator shutting down istio sidecar but not deleting pod upon failure
### Apache Airflow version

2.7.1

### What happened

I start a simple pod (one container and that's it) with the KubernetesPodOperator from the cncf.kubernetes provider.

The task in my container fails (exit code non-zero).
The istio sidecar is indeed terminated but the pod itself remain in an `Error` status even if the `on_finish_action` parameter is set at `"delete_pod"` and the pod is never terminated.

It is expected that the pod is fully deleted, istio or not.

I found that there is a difference in treatment [upon deleting the pod](https://github.com/apache/airflow/blob/5b85442fdc19947e125dcb0591bd59a53626a27b/airflow/providers/cncf/kubernetes/operators/pod.py#L824) so it might be some specific case I'm not aware of.

I'll be happy to help either on documentation or fixing this small issue but I would need confirmation on what is the expected behavior.

### What you think should happen instead

I think the `on_finish_action="delete_pod"` should terminate the pod, not let it hanging the `Error` state with both containers stopped.

### How to reproduce

here is the simplest dag on how to reproduce on my end, note that istio is not visible here since managed cluster-wide.

```python3
from airflow import DAG
from airflow.utils.dates import days_ago
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
    KubernetesPodOperator,
)


with DAG(
    dag_id="issue_reproduction",
    schedule_interval=None,
    start_date=days_ago(0),
    is_paused_upon_creation=False,
    max_active_runs=1,
) as dag:
    pod = KubernetesPodOperator(
        task_id="issue_reproduction",
        image="ubuntu",
        cmds=["bash", "-cx"],
        arguments=["exit 1"],
        dag=dag,
        security_context={
            "runAsNonRoot": True,
            "runAsUser": 65534,
            "seccompProfile": {"type": "RuntimeDefault"},
        },
    )
```

### Operating System

local=macos, deployments=k8s

### Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes==7.5.0
(Didn't see any fix that should change anything between 7.5.0 and 7.6.0 and the code I pointed to has not changed)

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

k8s version: v1.26.3

### Anything else

The issue should be 100% reproductible provided I didn't miss any specifics.

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

cc @Owen-CH-Leung 
Thanks. I'll take a look today
Ok I can reproduce the same issue. I thought I can rely on kubelet for the removal and obviously I was wrong. Will issue PR to fix this soon


Resolving patch:
diff --git a/airflow/providers/cncf/kubernetes/operators/pod.py b/airflow/providers/cncf/kubernetes/operators/pod.py
--- a/airflow/providers/cncf/kubernetes/operators/pod.py
+++ b/airflow/providers/cncf/kubernetes/operators/pod.py
@@ -804,7 +804,6 @@ def kill_istio_sidecar(self, pod: V1Pod) -> None:
             raise Exception("Error while deleting istio-proxy sidecar: %s", output_str)
 
     def process_pod_deletion(self, pod: k8s.V1Pod, *, reraise=True):
-        istio_enabled = self.is_istio_enabled(pod)
         with _optionally_suppress(reraise=reraise):
             if pod is not None:
                 should_delete_pod = (
@@ -818,12 +817,9 @@ def process_pod_deletion(self, pod: k8s.V1Pod, *, reraise=True):
                         and container_is_succeeded(pod, self.base_container_name)
                     )
                 )
-                if should_delete_pod and not istio_enabled:
+                if should_delete_pod:
                     self.log.info("Deleting pod: %s", pod.metadata.name)
                     self.pod_manager.delete_pod(pod)
-                elif should_delete_pod and istio_enabled:
-                    self.log.info("Deleting istio-proxy sidecar inside %s: ", pod.metadata.name)
-                    self.kill_istio_sidecar(pod)
                 else:
                     self.log.info("Skipping deleting pod: %s", pod.metadata.name)
 


Test patch:
diff --git a/tests/providers/cncf/kubernetes/operators/test_pod.py b/tests/providers/cncf/kubernetes/operators/test_pod.py
--- a/tests/providers/cncf/kubernetes/operators/test_pod.py
+++ b/tests/providers/cncf/kubernetes/operators/test_pod.py
@@ -624,7 +624,7 @@ def test_termination_message_policy_default_value_correctly_set(self):
             ({"on_finish_action": "delete_succeeded_pod"}, True, False),
         ],
     )
-    @patch(f"{KPO_MODULE}.KubernetesPodOperator.kill_istio_sidecar")
+    @patch(f"{POD_MANAGER_CLASS}.delete_pod")
     @patch(f"{KPO_MODULE}.KubernetesPodOperator.is_istio_enabled")
     @patch(f"{POD_MANAGER_CLASS}.await_pod_completion")
     @patch(f"{KPO_MODULE}.KubernetesPodOperator.find_pod")
@@ -633,7 +633,7 @@ def test_pod_with_istio_delete_after_await_container_error(
         find_pod_mock,
         await_pod_completion_mock,
         is_istio_enabled_mock,
-        kill_istio_sidecar_mock,
+        delete_pod_mock,
         task_kwargs,
         base_container_fail,
         expect_to_delete_pod,
@@ -662,13 +662,13 @@ def test_pod_with_istio_delete_after_await_container_error(
         cont_status_2.state.running = True
         cont_status_2.state.terminated = False
 
-        await_pod_completion_mock.return_value.spec.containers = [sidecar]
+        await_pod_completion_mock.return_value.spec.containers = [sidecar, cont_status_1, cont_status_2]
         await_pod_completion_mock.return_value.status.phase = "Running"
         await_pod_completion_mock.return_value.status.container_statuses = [cont_status_1, cont_status_2]
         await_pod_completion_mock.return_value.metadata.name = "pod-with-istio-sidecar"
         await_pod_completion_mock.return_value.metadata.namespace = "default"
 
-        find_pod_mock.return_value.spec.containers = [sidecar]
+        find_pod_mock.return_value.spec.containers = [sidecar, cont_status_1, cont_status_2]
         find_pod_mock.return_value.status.phase = "Running"
         find_pod_mock.return_value.status.container_statuses = [cont_status_1, cont_status_2]
         find_pod_mock.return_value.metadata.name = "pod-with-istio-sidecar"
@@ -678,6 +678,7 @@ def test_pod_with_istio_delete_after_await_container_error(
 
         context = create_context(k)
         context["ti"].xcom_push = MagicMock()
+
         if base_container_fail:
             self.await_pod_mock.side_effect = AirflowException("fake failure")
             with pytest.raises(AirflowException, match="my-failure"):
@@ -685,15 +686,11 @@ def test_pod_with_istio_delete_after_await_container_error(
         else:
             k.execute(context=context)
 
-        assert is_istio_enabled_mock(find_pod_mock.return_value)
-        if task_kwargs["on_finish_action"] == "delete_pod":
-            kill_istio_sidecar_mock.assert_called_with(await_pod_completion_mock.return_value)
-        elif expect_to_delete_pod and base_container_fail:
-            kill_istio_sidecar_mock.assert_called_with(find_pod_mock.return_value)
-        elif expect_to_delete_pod and not base_container_fail:
-            kill_istio_sidecar_mock.assert_called_with(await_pod_completion_mock.return_value)
+        if expect_to_delete_pod:
+            assert k.is_istio_enabled(find_pod_mock.return_value)
+            delete_pod_mock.assert_called_with(await_pod_completion_mock.return_value)
         else:
-            kill_istio_sidecar_mock.assert_not_called()
+            delete_pod_mock.assert_not_called()
 
     @pytest.mark.parametrize(
         "task_kwargs, should_be_deleted",