Repository: apache/airflow

Issue:
DAGs deleted from zips aren't deactivated
### Apache Airflow version

2.5.3

### What happened

When a DAG is removed from a zip in the DAGs directory, but the zip file remains, it is not marked correctly as inactive. It is still visible in the UI, and attempting to open the DAG results in an `DAG "mydag" seems to be missing from DagBag.` error in the UI.

The DAG is removed from the SerializedDag table, resulting in the scheduler repeatedly erroring with `[2023-04-12T12:43:51.165+0000] {scheduler_job.py:1063} ERROR - DAG 'mydag' not found in serialized_dag table`.

I have done some minor investigating and it appears that [this piece of code](https://github.com/apache/airflow/blob/2.5.3/airflow/dag_processing/manager.py#L748-L772) may be the cause.

`dag_filelocs` provides the path to a specific python file within a zip, so `SerializedDagModel.remove_deleted_dags` is able to remove the missing DAG.

However, `self._file_paths` only contains the top-level zip name, so `DagModel.deactivate_deleted_dags` will only deactivate DAGs where the zip they are contained in is deleted, regardless of whether the DAG is still inside the zip.

I can see there are [other methods that handle DAG deactivation](https://github.com/apache/airflow/blob/2.5.3/airflow/models/dag.py#L2945-L2968) and I'm not sure how these all interact but this does seem to cause this specific issue.

### What you think should happen instead

DAGS that are no longer in the DagBag are marked as inactive

### How to reproduce

Running airflow locally with docker-compose:
- Create a zipfile with 2 DAG py files in in ./dags
- Wait for the DAGs to be parsed by the scheduler and appear in the UI
- Overwrite the existing DAG zip, with a new zip containing only 1 of the original DAG py files
- Wait for scheduler loop to parse the new zip
- Attempt to open the removed DAG in the UI, you will see an error



### Operating System

Debian GNU/Linux 11 (bullseye)

### Versions of Apache Airflow Providers

_No response_

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

_No response_

### Anything else

If I replace the docker image in the docker compose with an image built from this Dockerfile:

```
FROM apache/airflow:2.5.3
RUN sed -i '772s/self._file_paths/dag_filelocs/' /home/airflow/.local/lib/python3.7/site-packages/airflow/dag_processing/manager.py
RUN sed -i '3351s/correct_maybe_zipped(dag_model.fileloc)/dag_model.fileloc/' /home/airflow/.local/lib/python3.7/site-packages/airflow/models/dag.py
```

The DAG is deactivated as expected

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.


Resolving patch:
diff --git a/airflow/dag_processing/manager.py b/airflow/dag_processing/manager.py
--- a/airflow/dag_processing/manager.py
+++ b/airflow/dag_processing/manager.py
@@ -791,7 +791,7 @@ def _refresh_dag_dir(self):
                 alive_dag_filelocs=dag_filelocs,
                 processor_subdir=self.get_dag_directory(),
             )
-            DagModel.deactivate_deleted_dags(self._file_paths)
+            DagModel.deactivate_deleted_dags(dag_filelocs)
 
             from airflow.models.dagcode import DagCode
 
diff --git a/airflow/models/dag.py b/airflow/models/dag.py
--- a/airflow/models/dag.py
+++ b/airflow/models/dag.py
@@ -99,7 +99,6 @@
 from airflow.utils.dag_cycle_tester import check_cycle
 from airflow.utils.dates import cron_presets, date_range as utils_date_range
 from airflow.utils.decorators import fixup_decorator_warning_stack
-from airflow.utils.file import correct_maybe_zipped
 from airflow.utils.helpers import at_most_one, exactly_one, validate_key
 from airflow.utils.log.logging_mixin import LoggingMixin
 from airflow.utils.session import NEW_SESSION, provide_session
@@ -3375,9 +3374,8 @@ def deactivate_deleted_dags(cls, alive_dag_filelocs: list[str], session=NEW_SESS
 
         dag_models = session.query(cls).all()
         for dag_model in dag_models:
-            if dag_model.fileloc is not None:
-                if correct_maybe_zipped(dag_model.fileloc) not in alive_dag_filelocs:
-                    dag_model.is_active = False
+            if dag_model.fileloc is not None and dag_model.fileloc not in alive_dag_filelocs:
+                dag_model.is_active = False
             else:
                 continue
 


Test patch:
diff --git a/tests/dag_processing/test_job_runner.py b/tests/dag_processing/test_job_runner.py
--- a/tests/dag_processing/test_job_runner.py
+++ b/tests/dag_processing/test_job_runner.py
@@ -964,6 +964,41 @@ def test_refresh_dags_dir_doesnt_delete_zipped_dags(self, tmpdir):
         assert SerializedDagModel.has_dag("test_zip_dag")
         # assert code not deleted
         assert DagCode.has_dag(dag.fileloc)
+        # assert dag still active
+        assert dag.get_is_active()
+
+    def test_refresh_dags_dir_deactivates_deleted_zipped_dags(self, tmpdir):
+        """Test DagProcessorJobRunner._refresh_dag_dir method"""
+        manager = DagProcessorJobRunner(
+            job=Job(),
+            processor=DagFileProcessorManager(
+                dag_directory=TEST_DAG_FOLDER,
+                max_runs=1,
+                processor_timeout=timedelta(days=365),
+                signal_conn=MagicMock(),
+                dag_ids=[],
+                pickle_dags=False,
+                async_mode=True,
+            ),
+        )
+        dagbag = DagBag(dag_folder=tmpdir, include_examples=False)
+        zipped_dag_path = os.path.join(TEST_DAGS_FOLDER, "test_zip.zip")
+        dagbag.process_file(zipped_dag_path)
+        dag = dagbag.get_dag("test_zip_dag")
+        dag.sync_to_db()
+        SerializedDagModel.write_dag(dag)
+        manager.processor.last_dag_dir_refresh_time = timezone.utcnow() - timedelta(minutes=10)
+
+        # Mock might_contain_dag to mimic deleting the python file from the zip
+        with mock.patch("airflow.dag_processing.manager.might_contain_dag", return_value=False):
+            manager.processor._refresh_dag_dir()
+
+        # Assert dag removed from SDM
+        assert not SerializedDagModel.has_dag("test_zip_dag")
+        # assert code deleted
+        assert not DagCode.has_dag(dag.fileloc)
+        # assert dag deactivated
+        assert not dag.get_is_active()
 
     @conf_vars(
         {