Repository: apache/airflow

Issue:
Kubernetes logging errors - attempting to adopt taskinstance which was not specified by database
### Apache Airflow version

2.4.3

### What happened

Using following config
```
executor = CeleryKubernetesExecutor
delete_worker_pods = False
```

1. Start a few dags running in kubernetes, wait for them to complete.
2. Restart Scheduler.
3. Logs are flooded with hundreds of errors like` ERROR - attempting to adopt taskinstance which was not specified by database: TaskInstanceKey(dag_id='xxx', task_id='yyy', run_id='zzz', try_number=1, map_index=-1)`

This is problematic because:
* Our installation has thousands of dags and pods so this becomes very noisy and the adoption-process adds excessive startup-time to the scheduler, up to a minute some times.
* It's hiding actual errors with resetting orphaned tasks, something that also happens for inexplicable reasons on scheduler restart with following log: `Reset the following 6 orphaned TaskInstances`. Making such much harder to debug. The cause of them can not be easily correlated with those that were not specified by database.


The cause of these logs are the Kubernetes executor on startup loads all pods (`try_adopt_task_instances`), it then cross references them with all `RUNNING` TaskInstances loaded via `scheduler_job.adopt_or_reset_orphaned_tasks`.
For all pods where a running TI can not be found, it logs the error above - But for TIs that were already completed this is not an error, and the pods should not have to be loaded at all.

I have an idea of adding some code in the kubernetes_executor that patches in something like a `completion-acknowleged`-label whenever a pod is completed (unless `delete_worker_pods` is set). Then on startup, all pods having this label can be excluded. Is this a good idea or do you see other potential solutions? 
Another potential solution is to inside `try_adopt_task_instances` only fetch the exact pod-id specified in each task-instance, instead of listing all to later cross-reference them.

### What you think should happen instead

_No response_

### How to reproduce

_No response_

### Operating System

Ubuntu 22.04

### Versions of Apache Airflow Providers

_No response_

### Deployment

Other Docker-based deployment

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
cc: @dstandish @ephraimbuddy WDYT? is the suggestion good? @hterik - feel free to propose a fix in the meantime, I think it will be easier to understand the bug/fix when we see the code proposal and maybe then we can think if better solution is possible?
related to this #27983

Resolving patch:
diff --git a/airflow/executors/kubernetes_executor.py b/airflow/executors/kubernetes_executor.py
--- a/airflow/executors/kubernetes_executor.py
+++ b/airflow/executors/kubernetes_executor.py
@@ -496,13 +496,14 @@ def clear_not_launched_queued_tasks(self, session=None) -> None:
         """
         Clear tasks that were not yet launched, but were previously queued.
 
-        Tasks can end up in a "Queued" state through either the executor being
-        abruptly shut down (leaving a non-empty task_queue on this executor)
-        or when a rescheduled/deferred operator comes back up for execution
-        (with the same try_number) before the pod of its previous incarnation
-        has been fully removed (we think).
+        Tasks can end up in a "Queued" state through when a rescheduled/deferred
+        operator comes back up for execution (with the same try_number) before the
+        pod of its previous incarnation has been fully removed (we think).
 
-        This method checks each of those tasks to see if the corresponding pod
+        It's also possible when an executor abruptly shuts down (leaving a non-empty
+        task_queue on that executor), but that scenario is handled via normal adoption.
+
+        This method checks each of our queued tasks to see if the corresponding pod
         is around, and if not, and there's no matching entry in our own
         task_queue, marks it for re-execution.
         """
@@ -779,7 +780,18 @@ def try_adopt_task_instances(self, tis: Sequence[TaskInstance]) -> Sequence[Task
         kube_client: client.CoreV1Api = self.kube_client
         for scheduler_job_id in scheduler_job_ids:
             scheduler_job_id = pod_generator.make_safe_label_value(str(scheduler_job_id))
-            query_kwargs = {"label_selector": f"airflow-worker={scheduler_job_id}"}
+            # We will look for any pods owned by the no-longer-running scheduler,
+            # but will exclude only successful pods, as those TIs will have a terminal state
+            # and not be up for adoption!
+            # Those workers that failed, however, are okay to adopt here as their TI will
+            # still be in queued.
+            query_kwargs = {
+                "field_selector": "status.phase!=Succeeded",
+                "label_selector": (
+                    "kubernetes_executor=True,"
+                    f"airflow-worker={scheduler_job_id},{POD_EXECUTOR_DONE_KEY}!=True"
+                ),
+            }
             pod_list = self._list_pods(query_kwargs)
             for pod in pod_list:
                 self.adopt_launched_task(kube_client, pod, pod_ids)


Test patch:
diff --git a/tests/executors/test_kubernetes_executor.py b/tests/executors/test_kubernetes_executor.py
--- a/tests/executors/test_kubernetes_executor.py
+++ b/tests/executors/test_kubernetes_executor.py
@@ -654,7 +654,9 @@ def test_try_adopt_task_instances(self, mock_adopt_completed_pods, mock_adopt_la
         # First adoption
         reset_tis = executor.try_adopt_task_instances([mock_ti])
         mock_kube_client.list_namespaced_pod.assert_called_once_with(
-            namespace="default", label_selector="airflow-worker=1"
+            namespace="default",
+            field_selector="status.phase!=Succeeded",
+            label_selector="kubernetes_executor=True,airflow-worker=1,airflow_executor_done!=True",
         )
         mock_adopt_launched_task.assert_called_once_with(mock_kube_client, pod, {ti_key: mock_ti})
         mock_adopt_completed_pods.assert_called_once()
@@ -672,7 +674,9 @@ def test_try_adopt_task_instances(self, mock_adopt_completed_pods, mock_adopt_la
 
         reset_tis = executor.try_adopt_task_instances([mock_ti])
         mock_kube_client.list_namespaced_pod.assert_called_once_with(
-            namespace="default", label_selector="airflow-worker=10"
+            namespace="default",
+            field_selector="status.phase!=Succeeded",
+            label_selector="kubernetes_executor=True,airflow-worker=10,airflow_executor_done!=True",
         )
         mock_adopt_launched_task.assert_called_once()  # Won't check args this time around as they get mutated
         mock_adopt_completed_pods.assert_called_once()
@@ -695,8 +699,16 @@ def test_try_adopt_task_instances_multiple_scheduler_ids(self, mock_adopt_comple
         assert mock_kube_client.list_namespaced_pod.call_count == 2
         mock_kube_client.list_namespaced_pod.assert_has_calls(
             [
-                mock.call(namespace="default", label_selector="airflow-worker=10"),
-                mock.call(namespace="default", label_selector="airflow-worker=40"),
+                mock.call(
+                    namespace="default",
+                    field_selector="status.phase!=Succeeded",
+                    label_selector="kubernetes_executor=True,airflow-worker=10,airflow_executor_done!=True",
+                ),
+                mock.call(
+                    namespace="default",
+                    field_selector="status.phase!=Succeeded",
+                    label_selector="kubernetes_executor=True,airflow-worker=40,airflow_executor_done!=True",
+                ),
             ],
             any_order=True,
         )