Issue 1:
Apache Airflow version

2.3.0

What happened

Here i provide steps to reproduce this.

Goal of this: to describe how to reproduce the "Failed to Adopt pod" error condition.

The DAG->step Described Below should be of type KubernetesPodOperator

NOTE: under normal operation,

(where the MAIN_AIRFLOW_POD is never recycled by k8s, we will never see this edge-case)

(it is only when the workerPod is still running, but the MAIN_AIRFLOW_POD is suddenly restarted/stopped)

(that we would see orphan->workerPods)

1] Implement a contrived-DAG, with a single step -> which is long-running (e.g. 6 minutes)

2] Deploy your airflow-2.1.4 / airfow-2.3.0 together with the contrived-DAG

3] Run your contrived-DAG.

4] in the middle of running the single-step, check via "kubectl" that your Kubernetes->workerPod has been created / running

5] while workerPod still running, do "kubectl delete pod <OF_MAIN_AIRFLOW_POD>". This will mean that the workerPod becomes an orphan.

6] the workerPod still continues to run through to completion. after which the K8S->status of the pod will be Completed, however the pod doesn't shut down itself.

7] "kubectl" start up a new <MAIN_AIRFLOW_POD> so the web-ui is running again.

8] MAIN_AIRFLOW_POD->webUi - Run your contrived-DAG again

9] while the contrived-DAG is starting/tryingToStart etc, you will see in the logs printed out "Failed to adopt pod" -> with 422 error code.

The step-9 with the error message, you will find two appearances of this error msg in the airflow-2.1.4, airflow-2.3.0 source-code.

The step-7 may also - general logging from the MAIN_APP - may also output the "Failed to adopt pod" error message also.

What you think should happen instead

On previous versions of airflow e.g. 1.10.x, the orphan-workerPods would be adopted by the 2nd run-time of the airflowMainApp and either used to continue the same DAG and/or cleared away when complete.

This is not happening with the newer airflow 2.1.4 / 2.3.0 (presumably because the code changed), and upon the 2nd run-time of the airflowMainApp - it would seem to try to adopt-workerPod but fails at that point ("Failed to adopt pod" in the logs and hence it cannot clear away orphan pods).

Given this is an edge-case only, (i.e. we would not expect k8s to be recycling the main airflowApp/pod anyway), it doesn't seem totally urgent bug. However, the only reason for me raising this issue with yourselves is that given any k8s->namespace, in particular in PROD, over time (e.g. 1 month?) the namespace will slowly be being filled up with orphanPods and somebody would need to manually log-in to delete old pods.

How to reproduce

Here i provide steps to reproduce this.

Goal of this: to describe how to reproduce the "Failed to Adopt pod" error condition.

The DAG->step Described Below should be of type KubernetesPodOperator

NOTE: under normal operation,

(where the MAIN_AIRFLOW_POD is never recycled by k8s, we will never see this edge-case)

(it is only when the workerPod is still running, but the MAIN_AIRFLOW_POD is suddenly restarted/stopped)

(that we would see orphan->workerPods)

1] Implement a contrived-DAG, with a single step -> which is long-running (e.g. 6 minutes)

2] Deploy your airflow-2.1.4 / airfow-2.3.0 together with the contrived-DAG

3] Run your contrived-DAG.

4] in the middle of running the single-step, check via "kubectl" that your Kubernetes->workerPod has been created / running

5] while workerPod still running, do "kubectl delete pod <OF_MAIN_AIRFLOW_POD>". This will mean that the workerPod becomes an orphan.

6] the workerPod still continues to run through to completion. after which the K8S->status of the pod will be Completed, however the pod doesn't shut down itself.

7] "kubectl" start up a new <MAIN_AIRFLOW_POD> so the web-ui is running again.

8] MAIN_AIRFLOW_POD->webUi - Run your contrived-DAG again

9] while the contrived-DAG is starting/tryingToStart etc, you will see in the logs printed out "Failed to adopt pod" -> with 422 error code.

The step-9 with the error message, you will find two appearances of this error msg in the airflow-2.1.4, airflow-2.3.0 source-code.

The step-7 may also - general logging from the MAIN_APP - may also output the "Failed to adopt pod" error message also.

Operating System

kubernetes

Versions of Apache Airflow Providers

No response

Deployment

Other 3rd-party Helm chart

Deployment details

nothing special.

it (CI/CD pipeline) builds the app. using requirements.txt to pull-in all the required python dependencies (including there is a dependency for the airflow-2.1.4 / 2.3.0)

it (CI/CD pipeline) packages the app as an ECR image & then deploy directly to k8s namespace.

Anything else

this is 100% reproducible each & every time.

i have tested this multiple times.

also - i tested this on the old airflow-1.10.x a couple of times to verify that the bug did not exist previously

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Issue 2:
Description

Add dependencies with "on-build" feature of Docker build.

Use case / motivation

Some dependencies can be added easily using on-build feature of Docker build.