Repository: apache/airflow

Issue:
Deleted DAG raises SerializedDagNotFound exception when accessing webserver
**Apache Airflow version**: 2.1.2


**Environment**:

- **Docker image**: official apache/airflow:2.1.2 extended with pip installed packages
- **Executor**: CeleryExecutor
- **Database**: PostgreSQL (engine version: 12.5)

**What happened**:

I've encountered `SerializedDagNotFound: DAG 'my_dag_id' not found in serialized_dag table` after deleting DAG via web UI. Exception is raised each time when UI is accessed. The exception itself does not impact UI, but still an event is sent to Sentry each time I interact with it.

**What you expected to happen**:

After DAG deletion I've expected that all records of it apart from logs would be deleted, but it's DAG run was still showing up in Webserver UI (even though, I couldn't find any records of DAG in the metadb, apart from dag_tag table still containing records related to deleted DAG (Which may be a reason for a new issue), but manual deletion of those records had no impact. 

After I've deleted DAG Run via Web UI, the exception is no longer raised.

**How to reproduce it**:

To this moment, I've only encountered this with one DAG, which was used for debugging purposes. It consists of multiple PythonOperators and it uses the same boilerplate code I use for dynamic DAG generation (return DAG object from `create_dag` function, add it to globals), but in it I've replaced dynamic generation logic with just `dag = create_dag(*my_args, **my_kwargs)`. I think this may have been caused by DAG run still running, when DAG was deleted, but cannot support this theory with actual information. 

<details>
  <summary>Traceback</summary> 

```python
SerializedDagNotFound: DAG 'my_dag_id' not found in serialized_dag table
  File "flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "flask/_compat.py", line 39, in reraise
    raise value
  File "flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "airflow/www/auth.py", line 34, in decorated
    return func(*args, **kwargs)
  File "airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "airflow/www/views.py", line 1679, in blocked
    dag = current_app.dag_bag.get_dag(dag_id)
  File "airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "airflow/models/dagbag.py", line 186, in get_dag
    self._add_dag_from_db(dag_id=dag_id, session=session)
  File "airflow/models/dagbag.py", line 258, in _add_dag_from_db
    raise SerializedDagNotFound(f"DAG '{dag_id}' not found in serialized_dag table")
```
</details>



Discussion:

Resolving patch:
diff --git a/airflow/www/views.py b/airflow/www/views.py
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -93,7 +93,7 @@
     set_dag_run_state_to_success,
 )
 from airflow.configuration import AIRFLOW_CONFIG, conf
-from airflow.exceptions import AirflowException
+from airflow.exceptions import AirflowException, SerializedDagNotFound
 from airflow.executors.executor_loader import ExecutorLoader
 from airflow.jobs.base_job import BaseJob
 from airflow.jobs.scheduler_job import SchedulerJob
@@ -1761,7 +1761,10 @@ def blocked(self, session=None):
         payload = []
         for dag_id, active_dag_runs in dags:
             max_active_runs = 0
-            dag = current_app.dag_bag.get_dag(dag_id)
+            try:
+                dag = current_app.dag_bag.get_dag(dag_id)
+            except SerializedDagNotFound:
+                dag = None
             if dag:
                 # TODO: Make max_active_runs a column so we can query for it directly
                 max_active_runs = dag.max_active_runs


Test patch:
diff --git a/tests/www/views/test_views_blocked.py b/tests/www/views/test_views_blocked.py
new file mode 100644
--- /dev/null
+++ b/tests/www/views/test_views_blocked.py
@@ -0,0 +1,84 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import pytest
+
+from airflow.models import DagModel
+from airflow.models.dagbag import DagBag
+from airflow.models.serialized_dag import SerializedDagModel
+from airflow.operators.dummy import DummyOperator
+from airflow.operators.subdag import SubDagOperator
+from airflow.utils import timezone
+from airflow.utils.session import create_session
+from airflow.utils.state import State
+from tests.test_utils.db import clear_db_runs
+
+
+@pytest.fixture()
+def running_subdag(admin_client, dag_maker):
+    with dag_maker(dag_id="running_dag.subdag") as subdag:
+        DummyOperator(task_id="dummy")
+
+    with dag_maker(dag_id="running_dag") as dag:
+        SubDagOperator(task_id="subdag", subdag=subdag)
+
+    dag_bag = DagBag(include_examples=False, include_smart_sensor=False)
+    dag_bag.bag_dag(dag, root_dag=dag)
+
+    with create_session() as session:
+        # This writes both DAGs to DagModel, but only serialize the parent DAG.
+        dag_bag.sync_to_db(session=session)
+
+        # Simulate triggering the SubDagOperator to run the subdag.
+        subdag.create_dagrun(
+            run_id="blocked_run_example_bash_operator",
+            state=State.RUNNING,
+            execution_date=timezone.datetime(2016, 1, 1),
+            start_date=timezone.datetime(2016, 1, 1),
+            session=session,
+        )
+
+        # Now delete the parent DAG but leave the subdag.
+        session.query(DagModel).filter(DagModel.dag_id == dag.dag_id).delete()
+        session.query(SerializedDagModel).filter(SerializedDagModel.dag_id == dag.dag_id).delete()
+
+    yield subdag
+
+    with create_session() as session:
+        session.query(DagModel).filter(DagModel.dag_id == subdag.dag_id).delete()
+    clear_db_runs()
+
+
+def test_blocked_subdag_success(admin_client, running_subdag):
+    """Test the /blocked endpoint works when a DAG is deleted.
+
+    When a DAG is bagged, it is written to both DagModel and SerializedDagModel,
+    but its subdags are only written to DagModel (without serialization). Thus,
+    ``DagBag.get_dag(subdag_id)`` would raise ``SerializedDagNotFound`` if the
+    subdag was not previously bagged in the dagbag (perhaps due to its root DAG
+    being deleted). ``DagBag.get_dag()`` calls should catch the exception and
+    properly handle this situation.
+    """
+    resp = admin_client.post("/blocked", data={"dag_ids": [running_subdag.dag_id]})
+    assert resp.status_code == 200
+    assert resp.json == [
+        {
+            "dag_id": running_subdag.dag_id,
+            "active_dag_run": 1,
+            "max_active_runs": 0,  # Default value for an unserialized DAG.
+        },
+    ]