Issue 1:
Apache Airflow version

2.5.0

What happened

I have a dynamic mapping task that is supposed to launch over 100 KubernetesPodOperator tasks. I have assigned 2.0 CPUs per task. When running the DAG, 16 tasks are in 'running state', however only 3 truly run, the remainder 13 fail with Pod took longer than 120 seconds to start. The remainder of the tasks are either queued or scheduled, and when there are less than 16 active tasks, they run and more or less fail with the same error.

Here is a snapshot of

kubectl -n airflow get all
NAME READY STATUS RESTARTS AGE
pod/airflow-postgresql-0 1/1 Running 5 2d
pod/airflow-scheduler-6dd68b485c-w8bhp 3/3 Running 19 2d
pod/airflow-statsd-586dbdcc6b-h4mnr 1/1 Running 5 2d
pod/airflow-triggerer-95565b95d-phts7 2/2 Running 14 2d
pod/airflow-webserver-599bb95bcd-7dtpk 1/1 Running 5 2d
pod/my-task-17dd038ca4d04164ba90f9c7f9a7fbb6 0/2 Pending 0 49s
pod/my-task-20aba86c65544ea384343f8fb4415d3a 0/2 Pending 0 53s
pod/my-task-3c5b4444a7d242459907ff3be7b7d6f6 0/2 Pending 0 44s
pod/my-task-5c8af5edb0904711b6a76a2edf1d1067 0/2 Pending 0 60s
pod/my-task-6001d3567f96400bb0ae559f22d3d2db 0/2 Pending 0 43s
pod/my-task-6dfb1945f3ff4ac4a06c7e6c6a85099c 0/2 Pending 0 81s
pod/my-task-71ad2fb48fb64f449014bba45bee980f 0/2 ContainerCreating 0 52s
pod/my-task-774216cb5f9344ffb35deac826d71639 0/2 Pending 0 68s
pod/my-task-814266d425254130868c3a5ebc8dce49 0/2 Pending 0 67s
pod/my-task-a11588d878b54944b4c069f49231ac36 0/2 Pending 0 77s
pod/my-task-b16c843fa038441ea31b90363ed86aa0 0/2 Pending 0 49s
pod/my-task-b85e2ed3417a4a62940661f418c900e5 0/2 Pending 0 60s
pod/my-task-d1de2a771a104a2592956a713f785300 0/2 Pending 0 73s
pod/my-task-dbeba55a80074c08bbdf023b3f0b885c 0/2 Completed 0 10m
pod/my-task-f83ad2805d314be3a7307b7216a54e53 2/2 Running 0 10m
pod/pipeline-my-task-0bc9e094afee4527b5b764e32f590282 0/1 Init:0/1 0 1s
pod/pipeline-my-task-1d51c5d3776e4dd8a89461e8a76faba1 1/1 Running 0 62s
pod/pipeline-my-task-24b1326a71d149fb9f62c101647468ee 1/1 Running 0 62s
pod/pipeline-my-task-29b132b7b0ce4832a5e30a821c6405bf 1/1 Running 0 10m
pod/pipeline-my-task-29fb55604eec457fa21d13d85c7889b5 1/1 Running 0 10m
pod/pipeline-my-task-2a337f1cc28b4315945cec8a961b1111 1/1 Running 0 69s
pod/pipeline-my-task-35d5c97570474082bc9b04189c433be7 1/1 Running 0 57s
pod/pipeline-my-task-569de133975d4dbb96becb2a04c0dac3 1/1 Running 0 78s
pod/pipeline-my-task-96a9681ace4441deba4faeef602f6e5b 1/1 Running 0 78s
pod/pipeline-my-task-9dcb9578720643eca5fa918a0a295f87 1/1 Running 0 87s
pod/pipeline-my-task-a643741d29ea4f4baa06e0ea20bc1a57 1/1 Running 0 10m
pod/pipeline-my-task-b04532a9f35a48a09cb1d46c9d9470dd 1/1 Running 0 57s
pod/pipeline-my-task-c9b7bb4ee07749be98083a11a512e1f4 1/1 Running 0 90s
pod/pipeline-my-task-d9c5ce9bf5ce499583cdf0ea3f58b7f0 1/1 Running 0 82s
pod/pipeline-my-task-dd5a5a45374f487fbc34c904e71b93b5 1/1 Running 0 59s
pod/pipeline-my-task-ea8c39d657824a1db505b00e8673b06a 1/1 Running 0 69s
pod/pipeline-my-task-fb5c71d274034f5392aebe0f4b395d98 1/1 Running 0 65s

What you think should happen instead

Only 3 tasks should be running.

The remainder tasks should be scheduled or queued.

How to reproduce

import json
import textwrap

import pendulum

from airflow.decorators import dag, task
from airflow.models.param import Param
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
KubernetesPodOperator,
Secret,
)
from kubernetes.client import models as k8s

@dag(
schedule=None,
start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
catchup=False,
tags=["example"],
)
def pipeline():

container_resources = k8s.V1ResourceRequirements(
limits={
"memory": "512Mi",
"cpu": 2.0,
},
requests={
"memory": "512Mi",
"cpu": 2.0,
},
)

volumes = [
k8s.V1Volume(
name="pvc-airflow",
persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(
claim_name="pvc-airflow"
),
)
]

volume_mounts = [
k8s.V1VolumeMount(mount_path="/airflow", name="pvc-airflow", sub_path=None)
]

@task
def make_list():
return [{"a": "a"}] * 100

my_task = KubernetesPodOperator.partial(
name="my_task",
task_id="my_task",
image="ubuntu:20.04",
namespace="airflow",
container_resources=container_resources,
volumes=volumes,
volume_mounts=volume_mounts,
in_cluster=True,
do_xcom_push=True,
get_logs=True,
cmds=[
"/bin/bash",
"-c",
"""
sleep 600
"""
],
).expand(env_vars=make_list())

Operating System

Ubuntu 20.04.5 LTS

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

I am running this locally using the helm chart on Kind.

My machine is 4 CPU (x2), with 16 GB RAM.

Anything else

I have confirmed that the failing tasks are not starting due to timeouts from waiting for resources too long.

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Issue 2:
The tests that are run in CI should use breeze and parallellism implemented in Python.

NOTE: We should pay an attention to provide better error feedback when docker containers could not be started - see #23523