Issue 1:
Apache Airflow version

2.2.1

What happened

When I try to backfill using airflow dags backfill ..., the 'Run' shows running but the tasks are stuck in scheduled state forever.

Code for DAG to reproduce the problem

import time
from datetime import timedelta

import pendulum
from airflow import DAG
from airflow.decorators import task
from airflow.models.dag import dag
from airflow.operators.bash import BashOperator
from airflow.operators.dummy import DummyOperator
from airflow.operators.python import PythonOperator

default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email': ['airflow@example.com'],
'email_on_failure': True,
'email_on_retry': False,
'retries': 2,
'retry_delay': timedelta(minutes=5),
}

def get_execution_date(**kwargs):
ds = kwargs['ds']
print(ds)

with DAG(
'test_dag_2',
default_args=default_args,
description='Testing dag',
start_date=pendulum.datetime(2022, 4, 2, tz='UTC'),
schedule_interval="@hourly", max_active_runs=5, concurrency=10,
) as dag:
t1 = BashOperator(
task_id='task_1',
depends_on_past=False,
bash_command='sleep 5'
)

t2 = PythonOperator(
task_id='get_execution_date',
python_callable=get_execution_date
)

t1 >> t2

Airflow backfill CLI

$ airflow dags backfill --subdir /opt/airflow/dags/repo --reset-dagruns -s "2022-03-01 01:00:00" -e "2022-03-02 01:00:00" test_dag_2

Result

Discussion

The 'Run' shows running but the tasks are stuck in scheduled state forever.

Here are some variants what I have checked. (but still have this issue.)

max_active_runs, concurrency

-x option for backfill command

--subdir option for backfill command

--reset-dagruns for backfill command

schedule_interval to None and @once

Restart the Airflow scheduler pods.

Upgrade Airflow version to 2.3.0.

Related issue

#13542

What you think should happen instead

The backfill jobs should successfully done from the past.

How to reproduce

No response

Operating System

CentOS Linux release 7.9.2009 (Core)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon 3.3.0

apache-airflow-providers-celery 2.1.0

apache-airflow-providers-cncf-kubernetes 3.0.2

apache-airflow-providers-docker 2.6.0

apache-airflow-providers-elasticsearch 3.0.3

apache-airflow-providers-ftp 2.1.2

apache-airflow-providers-grpc 2.0.4

apache-airflow-providers-hashicorp 2.2.0

apache-airflow-providers-http 2.0.2

apache-airflow-providers-imap 2.2.3

apache-airflow-providers-postgres 4.1.0

apache-airflow-providers-redis 2.0.4

apache-airflow-providers-sendgrid 2.0.4

apache-airflow-providers-sftp 2.6.0

apache-airflow-providers-slack 4.2.3

apache-airflow-providers-sqlite 2.1.3

apache-airflow-providers-ssh 2.4.3

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

Here are some logs from scheduler pod, found kubernetes_executor try to delete worker pod.

[2022-05-13 02:48:08,417] {kubernetes_executor.py:147} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a had an event of type MODIFIED
[2022-05-13 02:48:08,417] {kubernetes_executor.py:206} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a Succeeded
[2022-05-13 02:48:08,423] {kubernetes_executor.py:147} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a had an event of type DELETED
[2022-05-13 02:48:08,423] {kubernetes_executor.py:206} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a Succeeded
[2022-05-13 02:48:08,497] {backfill_job.py:397} INFO - [backfill progress] | finished run 0 of 25 | tasks waiting: 5 | succeeded: 0 | running: 5| failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 5
[2022-05-13 02:48:13,390] {kubernetes_executor.py:375} INFO - Attempting to finish pod; pod_id: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a; state: None; annotations: {'dag_id': 'test_dag_2', 'task_id': 'task_1', 'execution_date': None, 'run_id': 'backfill__2022-03-03T02:00:00+00:00', 'try_number': '1'}
[2022-05-13 02:48:13,391] {kubernetes_executor.py:375} INFO - Attempting to finish pod; pod_id: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a; state: None; annotations: {'dag_id': 'test_dag_2', 'task_id': 'task_1', 'execution_date': None, 'run_id': 'backfill__2022-03-03T02:00:00+00:00', 'try_number': '1'}
[2022-05-13 02:48:13,392] {kubernetes_executor.py:576} INFO - Changing state of (TaskInstanceKey(dag_id='test_dag_2', task_id='task_1', run_id='backfill__2022-03-03T02:00:00+00:00', try_number=1), None, 'testdag2task1.34ae944f04ee4a18a65f559163ca0d1a', 'jutopia-chlee-test-chlee-backfill-test', '3559062271') to None
[2022-05-13 02:48:13,396] {kubernetes_executor.py:661} INFO - Deleted pod: TaskInstanceKey(dag_id='test_dag_2', task_id='task_1', run_id='backfill__2022-03-03T02:00:00+00:00', try_number=1) in namespace jutopia-chlee-test-chlee-backfill-test

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Issue 2:
Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

I am using airlfow 2.3.2 on EKS 1.22, the airflow service on EKS was launched by making minor modifications in official helm chart regarding replicas and resources. It is observed that the memory utilization by scheduler pod keeps on increasing as age of scheduler pod increases. Also this is observed when all dags are paused and nothing is running on airlfow. Different versions(config changes) of official helm chat was used to spin different airflow services in the EKS cluster where this issue was observed.

What you think should happen instead

This shouldn't have happened, the memory should have remain almost same. The increase in memory made the replicas to increase over few days as replication was setup based on memory utilisation even though there was no changes in the env itself and it was serving any traffic.

How to reproduce

Should be able to reproduce by using official airflow helm-chart of 2.3.2 on AWS EKS 1.22

Operating System

PRETTY_NAME=“Debian GNU/Linux 11 (bullseye)” NAME=“Debian GNU/Linux” VERSION_ID=“11” VERSION=“11 (bullseye)” VERSION_CODENAME=bullseye ID=debian

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct