Issue 1:
Apache Airflow version

2.3.2 (latest released)

What happened

After the update to 2.3.0, the ability to remove dags from the ui, if they had tags, disappeared.

With update 2.3.2, the problem should have been solved, but the delete now takes a very long time.

When I try to remove the DAG from the interface, I get a 504 Gateway Timeout error, since the request to the database takes a long time.

This is repeated with any DAG where there are many runs.

What you think should happen instead

The DAG execution history should be deleted.

How to reproduce

Run a dag and generate more than 1000 runs.

Remove dag via web ui.

Operating System

Ubuntu 18.04.4 LTS (Bionic Beaver)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==3.4.0

apache-airflow-providers-celery==2.1.4

apache-airflow-providers-cncf-kubernetes==4.0.2

apache-airflow-providers-docker==2.7.0

apache-airflow-providers-elasticsearch==3.0.3

apache-airflow-providers-ftp==2.1.2

apache-airflow-providers-google==7.0.0

apache-airflow-providers-grpc==2.0.4

apache-airflow-providers-hashicorp==2.2.0

apache-airflow-providers-http==2.1.2

apache-airflow-providers-imap==2.2.3

apache-airflow-providers-jdbc==2.1.3

apache-airflow-providers-microsoft-azure==3.9.0

apache-airflow-providers-microsoft-mssql==2.1.3

apache-airflow-providers-mongo==2.3.3

apache-airflow-providers-mysql==2.2.3

apache-airflow-providers-odbc==2.0.4

apache-airflow-providers-oracle==2.2.3

apache-airflow-providers-postgres==4.1.0

apache-airflow-providers-redis==2.0.4

apache-airflow-providers-samba==3.0.0

apache-airflow-providers-sendgrid==2.0.4

apache-airflow-providers-sftp==2.6.0

apache-airflow-providers-slack==4.2.3

apache-airflow-providers-sqlite==2.1.3

apache-airflow-providers-ssh==2.4.4

apache-airflow-providers-trino==2.3.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

Kubernetes v1.18.15

Anything else

I tried to run the analysis of the SQL request to delete the dag manually and got the following result:

DELETE FROM dag_run WHERE dag_run.dag_id IN ('test') RETURNING dag_run.id

Delete on dag_run (cost=1440.76..32883.24 rows=27398 width=6) (actual time=2169082.358..2169985.242 rows=26750 loops=1)
-> Bitmap Heap Scan on dag_run (cost=1440.76..32883.24 rows=27398 width=6) (actual time=7.132..319.677 rows=26750 loops=1)
Recheck Cond: ((dag_id)::text = 'test'::text)
Heap Blocks: exact=6157
-> Bitmap Index Scan on dag_id_state (cost=0.00..1433.91 rows=27398 width=0) (actual time=5.963..5.963 rows=26750 loops=1)
Index Cond: ((dag_id)::text = 'test'::text)
Planning Time: 0.134 ms
Trigger for constraint task_instance_dag_run_fkey on dag_run: time=1694.339 calls=26750
Trigger for constraint task_reschedule_dr_fkey on dag_run: time=408.320 calls=26750
Trigger for constraint task_reschedule_ti_fkey on task_instance: time=2103.582 calls=53500
Trigger for constraint task_map_task_instance_fkey on task_instance: time=1802.304 calls=53500
Trigger for constraint xcom_task_instance_fkey on task_instance: time=2264561.198 calls=53500
Trigger for constraint rtif_ti_fkey on task_instance: time=5526.311 calls=53500
Trigger for constraint task_fail_ti_fkey on task_instance: time=3081.837 calls=53500
Execution Time: 4449310.647 ms

It runs for a very long time, it is unrealistic to increase the worker timeout to such a value.

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Issue 2:
Apache Airflow version

2.1.2

What happened

We had restarted Airflow scheduler. After restart, we suddenly saw all our DAG's were missing from the UI. When we go to old tasks from task_instaces page & look at the old task it says this error.

Something bad has happened.

Please consider letting us know by creating a bug report using GitHub.

Python version: 3.8.11

Airflow version: 2.1.2

Node: airflow-web-77b556776d-wrmt9

Traceback (most recent call last):

File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app

response = self.full_dispatch_request()

File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request

rv = self.handle_user_exception(e)

File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception

reraise(exc_type, exc_value, tb)

File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise

raise value

File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request

rv = self.dispatch_request()

File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request

return self.view_functionsrule.endpoint

File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/www/auth.py", line 34, in decorated

return func(*args, **kwargs)

File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/www/decorators.py", line 97, in view_func

return f(*args, **kwargs)

File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/www/decorators.py", line 60, in wrapper

return f(*args, **kwargs)

File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper

return func(*args, session=session, **kwargs)

File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/www/views.py", line 2156, in graph

dag = current_app.dag_bag.get_dag(dag_id)

File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper

return func(*args, session=session, **kwargs)

File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/models/dagbag.py", line 186, in get_dag

self._add_dag_from_db(dag_id=dag_id, session=session)

File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/models/dagbag.py", line 258, in _add_dag_from_db

raise SerializedDagNotFound(f"DAG '{dag_id}' not found in serialized_dag table")

airflow.exceptions.SerializedDagNotFound: DAG 'DAP_dbricks_user_creation' not found in serialized_dag table

What have we noticed is is_active column for all DAG's in public.dag table is set to 'false' and public.serialized_dag table is empty.

We tried to restore these 2 tables & after restoration in 1-2 mins inserted records is getting deleted from serialized_dag & it is getting updated back to false in dag table.

We also don't find .py files regarding DAG's in /opt/bitnami/airflow/dags/git_airflow-dags/ path.

What you expected to happen

No response

How to reproduce

No response

Operating System

Ubuntu (AKS)

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct