Repository: apache/airflow

Issue:
DAGs are able to see historical dataset events when created new / disabled - enabled / deleted - recreated.
### Apache Airflow version

2.9.0b2

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

* If a new dataset triggreed DAG is created for an already existing dataset. (Dataset has already existing dataset events)
    * DAG catch up false.
        * DAG see all dataset events from very first event for dataset.
    * DAG catch up true.
        * DAG see all dataset events from very first event for dataset.
* If a dataset triggreed DAG is disabled and then enabled back. (In b/w disable and enable of the DAG dataset events)
    * DAG catch up false.
        * DAG see all dataset events received during the timeframe when DAG was disabled.
    * DAG catch up true.
        * DAG see all dataset events received during the timeframe when DAG was disabled.
* If a dataset triggreed DAG is deleted and then recreated back keeping dag_id same. (In b/w delete and recreate of the DAG dataset events)
    * DAG catch up false.
        * DAG see all dataset events received during the timeframe when DAG was not enabled.
    * DAG catch up true.
        * DAG see all dataset events received during the timeframe when DAG was deleted.

### What you think should happen instead?

* If a new dataset triggreed DAG is created for an already existing dataset. (Dataset has already existing dataset events)
    * DAG catch up false.
        * Expected behaviour: DAG should see all the dataset events from the time when DAG was enabled.
        * Actual Behaviour: DAG see all dataset events from very first event for dataset.
    * DAG catch up true.
        * Expected behaviour: DAG should see all the dataset events from the time when DAG was created.
        * Actual Behaviour: DAG see all dataset events from very first event for dataset.
* If a dataset triggreed DAG is disabled and then enabled back. (In b/w disable and enable of the DAG dataset events)
    * DAG catch up false.
        * Expected behaviour: DAG should see all the dataset events from the time when DAG was enabled back.
        * Actual Behaviour: DAG see all dataset events received during the timeframe when DAG was disabled.
    * DAG catch up true.
        * Expected behaviour: DAG see all dataset events received during the timeframe when DAG was disabled.
        * Actual Behaviour: DAG see all dataset events received during the timeframe when DAG was disabled.
* If a dataset triggreed DAG is deleted and then recreated back keeping dag_id same. (In b/w delete and recreate of the DAG dataset events)
    * DAG catch up false.
        * Expected behaviour: DAG should see all the dataset events from the time when DAG was enabled and not all the events received during the timeframe when DAG was deleted or recreated and not enabled.
        * Actual Behaviour: DAG see all dataset events received during the timeframe when DAG was not enabled.
    * DAG catch up true.
        * Expected behaviour: DAG should see all the dataset events from the time when DAG was recreated and not all the events received during the timeframe when DAG was deleted.
        * Actual Behaviour: DAG see all dataset events received during the timeframe when DAG was deleted.

### How to reproduce

* Create a new DAG which has dataset event producer. Enable both the producer.
* Run producer dag multiple times.
* Create a new consumer DAG which also get triggered from same dataset with catchup false.
* Enable the new DAG.
* You will see new DAG will run as soon as enabled and see very first event in the dataset.
* In the next run whenever that was scheduled DAG will get all the events other than the first event it has already consumed.

```from __future__ import annotations

import pendulum

from airflow.datasets import Dataset
from airflow.models.dag import DAG
from airflow.operators.bash import BashOperator

# [START dataset_def]
poc_dag1_dataset = Dataset("test-cluster/test-schema/test-table")
poc_dag2_dataset = Dataset("test-cluster/test-schema/test-table2")
poc_dataset_consumes_check = Dataset("test-cluster/test-schema/test-table-check")
poc_dataset_consumes_check2 = Dataset("test-cluster/test-schema/test-table-check2")


def _write2_post_execute(context, _):
    context["dataset_events"]["test-cluster/test-schema/test-table"].extra = {"x_tk": 1}


with DAG(
    dag_id="poc_dataset_produces_1",
    catchup=False,
    start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
    schedule="@daily",
    tags=["produces", "dataset-scheduled"],
) as dag1:
    # [START task_outlet]
    BashOperator(
        outlets=[poc_dag1_dataset], 
        task_id="producing_task_1", 
        bash_command="sleep 5",
        post_execute=_write2_post_execute,
    )
    # [END task_outlet]
```

After this add  
```
# [START dag_dep]
with DAG(
    dag_id="poc_dataset_consumes_1",
    catchup=False,
    start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
    schedule=[poc_dag1_dataset],
    tags=["consumes", "dataset-scheduled"],
) as dag3:
    # [END dag_dep]
    BashOperator(
        task_id="consuming_1",
        bash_command='echo "ti_key={{ triggering_dataset_events }}"',
    )
```

### Operating System

Debian GNU/Linux 12 (bookworm)

### Versions of Apache Airflow Providers

_No response_

### Deployment

Docker-Compose

### Deployment details

Plain vanilla airflow.

### Anything else?

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

I agree with the `catchup=False` expectations. Not sure about the `True` ones. When doing timed scheduling (i.e. timetables), we use `start_date` to decide how far back to catch up, and I think we should use the same mechanism, not based on when a DAG is (re)created. You could argue this is not smart enough (and I would agree), but now is not the time to change this.
On the implementation front—the `catchup=True` case is not too difficult to implement, but `catchup=False` presents problems. When scheduling against time, we can simply create a DAG run in the future because we know when that’ll happen, and that future run also acts as the cutoff point in subsequent iterations for how long the scheduler should look back. With event-based scheduling, however, we don’t know when the next run will happen, and need another way to tell the scheduler when to stop looking into the past.

We could add a new field on `DagModel` for this (say a datetime field `last_unpause`). Alternatively… maybe we can cheat instead and create one run from the most recent past events? Once that one run exists, the scheduler would know not to look past it in later iterations. Are there other solutions? I would love some ideas.
`DagModel.ast_unpause` sounds like a reasonable choice. But I thought DDRQs are only added to active DAGs? If that's the case, why would the DAG with `catchup=False` looks for past events 🤔 Or maybe we need to check whether the DAG is pausing when creating DDRQ?
> DDRQs are only added to active DAGs

Are they? From what I can tell all DAGs are referenced. The code creating DDRQs uses `consuming_dags`, which is based on the through table `DagScheduleDatasetReference`, which does not exclude inactive DAGs.

Maybe the right fix is to only create entries in `DagScheduleDatasetReference` for active DagModel rows, or add additional logic in `consuming_dags` to only include active DAGs.
@uranusjr I agree, adding `catchup` in the mix of dataset will introduce a lot of complexity.  For now we can avoid that. But we need to tackle this issue at both publishing side and consumption side.

**Publishing Side:**
- As soon an a DAG is deleted remove entry from dag_schedule_dataset_reference so that entries are not added in the dataset_dag_run_queue table for any of the dataset events. This is mainly to avoid any deleted DAG coming back and getting all the events since it last run.

**Consumption Side:**
- For any new DAG only give events from the time when it is active and no past events. For this filter of dataset event query need to be updated https://github.com/apache/airflow/blob/main/airflow/jobs/scheduler_job_runner.py#L1264 for dag which doesn't have previous runs. 

Above one will fix broader issue of all historical events being available to newly created DAG for an historical DATASET present in system from quite some time and events being available to DAG which was deleted and came back.
@Lee-W I have checked when DAGs are deleted/ disabled its entry from `dag_schedule_dataset_reference` is not deleted which lead to addition of entry in `dataset_dag_run_queue` when an event is received. On event consumer side no check is in place it just check and provide all the events from the last run.

For more details see https://github.com/apache/airflow/blob/main/airflow/jobs/scheduler_job_runner.py#L1260-L1274.

@uranusjr do you think its a reasonable fix for the consumption path
` dataset_event_filters.append(DatasetEvent.timestamp > DagScheduleDatasetReference.created_at)`

```
                dataset_event_filters = [
                    DagScheduleDatasetReference.dag_id == dag.dag_id,
                    DatasetEvent.timestamp <= exec_date,
                ]
                if previous_dag_run:
                    dataset_event_filters.append(DatasetEvent.timestamp > DagScheduleDatasetReference.execution_date)
                else:
                    dataset_event_filters.append(DatasetEvent.timestamp > DagScheduleDatasetReference.created_at)

                dataset_events = session.scalars(
                    select(DatasetEvent)
                    .join(
                        DagScheduleDatasetReference,
                        DatasetEvent.dataset_id == DagScheduleDatasetReference.dataset_id,
                    )
                    .where(*dataset_event_filters)
                ).all()
```

```
> * As soon an a DAG is deleted remove entry from dag_schedule_dataset_reference so that entries are not added in the dataset_dag_run_queue table for any of the dataset events.

I don’t think we need to do that. Instead, the query to `dag_schedule_dataset_reference` can check if a DAG has been paused or deleted (actually deactivated; if the row is actually _removed_ the database automatically propogates the deletion via foreign key). This way we only need to handle the comsuming side.

Resolving patch:
diff --git a/airflow/datasets/manager.py b/airflow/datasets/manager.py
--- a/airflow/datasets/manager.py
+++ b/airflow/datasets/manager.py
@@ -25,13 +25,14 @@
 from airflow.configuration import conf
 from airflow.datasets import Dataset
 from airflow.listeners.listener import get_listener_manager
-from airflow.models.dataset import DatasetDagRunQueue, DatasetEvent, DatasetModel
+from airflow.models.dataset import DagScheduleDatasetReference, DatasetDagRunQueue, DatasetEvent, DatasetModel
 from airflow.stats import Stats
 from airflow.utils.log.logging_mixin import LoggingMixin
 
 if TYPE_CHECKING:
     from sqlalchemy.orm.session import Session
 
+    from airflow.models.dag import DagModel
     from airflow.models.taskinstance import TaskInstance
 
 
@@ -73,7 +74,7 @@ def register_dataset_change(
         dataset_model = session.scalar(
             select(DatasetModel)
             .where(DatasetModel.uri == dataset.uri)
-            .options(joinedload(DatasetModel.consuming_dags))
+            .options(joinedload(DatasetModel.consuming_dags).joinedload(DagScheduleDatasetReference.dag))
         )
         if not dataset_model:
             self.log.warning("DatasetModel %s not found", dataset)
@@ -99,8 +100,7 @@ def register_dataset_change(
         self.notify_dataset_changed(dataset=dataset)
 
         Stats.incr("dataset.updates")
-        if dataset_model.consuming_dags:
-            self._queue_dagruns(dataset_model, session)
+        self._queue_dagruns(dataset_model, session)
         session.flush()
         return dataset_event
 
@@ -127,27 +127,35 @@ def _queue_dagruns(self, dataset: DatasetModel, session: Session) -> None:
         return self._slow_path_queue_dagruns(dataset, session)
 
     def _slow_path_queue_dagruns(self, dataset: DatasetModel, session: Session) -> None:
-        consuming_dag_ids = [x.dag_id for x in dataset.consuming_dags]
-        self.log.debug("consuming dag ids %s", consuming_dag_ids)
-
-        # Don't error whole transaction when a single RunQueue item conflicts.
-        # https://docs.sqlalchemy.org/en/14/orm/session_transaction.html#using-savepoint
-        for dag_id in consuming_dag_ids:
-            item = DatasetDagRunQueue(target_dag_id=dag_id, dataset_id=dataset.id)
+        def _queue_dagrun_if_needed(dag: DagModel) -> str | None:
+            if not dag.is_active or dag.is_paused:
+                return None
+            item = DatasetDagRunQueue(target_dag_id=dag.dag_id, dataset_id=dataset.id)
+            # Don't error whole transaction when a single RunQueue item conflicts.
+            # https://docs.sqlalchemy.org/en/14/orm/session_transaction.html#using-savepoint
             try:
                 with session.begin_nested():
                     session.merge(item)
             except exc.IntegrityError:
                 self.log.debug("Skipping record %s", item, exc_info=True)
+            return dag.dag_id
+
+        queued_results = (_queue_dagrun_if_needed(ref.dag) for ref in dataset.consuming_dags)
+        if queued_dag_ids := [r for r in queued_results if r is not None]:
+            self.log.debug("consuming dag ids %s", queued_dag_ids)
 
     def _postgres_queue_dagruns(self, dataset: DatasetModel, session: Session) -> None:
         from sqlalchemy.dialects.postgresql import insert
 
+        values = [
+            {"target_dag_id": dag.dag_id}
+            for dag in (r.dag for r in dataset.consuming_dags)
+            if dag.is_active and not dag.is_paused
+        ]
+        if not values:
+            return
         stmt = insert(DatasetDagRunQueue).values(dataset_id=dataset.id).on_conflict_do_nothing()
-        session.execute(
-            stmt,
-            [{"target_dag_id": target_dag.dag_id} for target_dag in dataset.consuming_dags],
-        )
+        session.execute(stmt, values)
 
 
 def resolve_dataset_manager() -> DatasetManager:
diff --git a/airflow/models/dataset.py b/airflow/models/dataset.py
--- a/airflow/models/dataset.py
+++ b/airflow/models/dataset.py
@@ -112,6 +112,8 @@ class DagScheduleDatasetReference(Base):
     updated_at = Column(UtcDateTime, default=timezone.utcnow, onupdate=timezone.utcnow, nullable=False)
 
     dataset = relationship("DatasetModel", back_populates="consuming_dags")
+    dag = relationship("DagModel")
+
     queue_records = relationship(
         "DatasetDagRunQueue",
         primaryjoin="""and_(


Test patch:
diff --git a/tests/datasets/test_manager.py b/tests/datasets/test_manager.py
--- a/tests/datasets/test_manager.py
+++ b/tests/datasets/test_manager.py
@@ -70,8 +70,8 @@ def test_register_dataset_change(self, session, dag_maker, mock_task_instance):
         dsem = DatasetManager()
 
         ds = Dataset(uri="test_dataset_uri")
-        dag1 = DagModel(dag_id="dag1")
-        dag2 = DagModel(dag_id="dag2")
+        dag1 = DagModel(dag_id="dag1", is_active=True)
+        dag2 = DagModel(dag_id="dag2", is_active=True)
         session.add_all([dag1, dag2])
 
         dsm = DatasetModel(uri="test_dataset_uri")
diff --git a/tests/jobs/test_scheduler_job.py b/tests/jobs/test_scheduler_job.py
--- a/tests/jobs/test_scheduler_job.py
+++ b/tests/jobs/test_scheduler_job.py
@@ -30,7 +30,7 @@
 import psutil
 import pytest
 import time_machine
-from sqlalchemy import func
+from sqlalchemy import func, select, update
 
 import airflow.example_dags
 from airflow import settings
@@ -39,6 +39,7 @@
 from airflow.callbacks.pipe_callback_sink import PipeCallbackSink
 from airflow.dag_processing.manager import DagFileProcessorAgent
 from airflow.datasets import Dataset
+from airflow.datasets.manager import DatasetManager
 from airflow.exceptions import AirflowException
 from airflow.executors.base_executor import BaseExecutor
 from airflow.executors.executor_constants import MOCK_EXECUTOR
@@ -3784,6 +3785,55 @@ def dict_from_obj(obj):
 
         assert dag3.get_last_dagrun().creating_job_id == scheduler_job.id
 
+    @pytest.mark.need_serialized_dag
+    @pytest.mark.parametrize(
+        "disable, enable",
+        [
+            pytest.param({"is_active": False}, {"is_active": True}, id="active"),
+            pytest.param({"is_paused": True}, {"is_paused": False}, id="paused"),
+        ],
+    )
+    def test_no_create_dag_runs_when_dag_disabled(self, session, dag_maker, disable, enable):
+        ds = Dataset("ds")
+        with dag_maker(dag_id="consumer", schedule=[ds], session=session):
+            pass
+        with dag_maker(dag_id="producer", schedule="@daily", session=session):
+            BashOperator(task_id="task", bash_command="echo 1", outlets=ds)
+        dsm = DatasetManager()
+
+        ds_id = session.scalars(select(DatasetModel.id).filter_by(uri=ds.uri)).one()
+
+        dse_q = select(DatasetEvent).where(DatasetEvent.dataset_id == ds_id).order_by(DatasetEvent.timestamp)
+        ddrq_q = select(DatasetDagRunQueue).where(
+            DatasetDagRunQueue.dataset_id == ds_id, DatasetDagRunQueue.target_dag_id == "consumer"
+        )
+
+        # Simulate the consumer DAG being disabled.
+        session.execute(update(DagModel).where(DagModel.dag_id == "consumer").values(**disable))
+
+        # A DDRQ is not scheduled although an event is emitted.
+        dr1: DagRun = dag_maker.create_dagrun(run_type=DagRunType.SCHEDULED)
+        dsm.register_dataset_change(
+            task_instance=dr1.get_task_instance("task", session=session),
+            dataset=ds,
+            session=session,
+        )
+        assert session.scalars(dse_q).one().source_run_id == dr1.run_id
+        assert session.scalars(ddrq_q).one_or_none() is None
+
+        # Simulate the consumer DAG being enabled.
+        session.execute(update(DagModel).where(DagModel.dag_id == "consumer").values(**enable))
+
+        # A DDRQ should be scheduled for the new event, but not the previous one.
+        dr2: DagRun = dag_maker.create_dagrun_after(dr1, run_type=DagRunType.SCHEDULED)
+        dsm.register_dataset_change(
+            task_instance=dr2.get_task_instance("task", session=session),
+            dataset=ds,
+            session=session,
+        )
+        assert [e.source_run_id for e in session.scalars(dse_q)] == [dr1.run_id, dr2.run_id]
+        assert session.scalars(ddrq_q).one().target_dag_id == "consumer"
+
     @time_machine.travel(DEFAULT_DATE + datetime.timedelta(days=1, seconds=9), tick=False)
     @mock.patch("airflow.jobs.scheduler_job_runner.Stats.timing")
     def test_start_dagruns(self, stats_timing, dag_maker):