Repository: apache/airflow

Issue:
Manual DAG triggers with Logical Date in the Past trigger a second run when schedule is timedelta
### Apache Airflow version

2.7.0

### What happened

Relates to investigation of #30327

When you define a DAG with a `schedule=datetime.timedelta(days=1)` and `catchup=False` and you manually trigger a run in the past (logical date in the past, e.g. a few days ago) then upon completion of the run another scheduled run is triggered - irrespective of a previous scheduled run was made just before (not a day ago)

### What you think should happen instead

A manual run should not trigger another run as the scheduling in general should follow the delta time definition of the DAG.

### How to reproduce

Use the following example DAG code and enable scheduling:
```
with DAG(
    dag_id="after_workday_delta_regression",
    start_date=pendulum.datetime(2023, 8, 1, tz="UTC"),
    catchup=False,
    schedule=datetime.timedelta(days=1),
    params={"test": 123}
):
    @task
    def test(ti: TaskInstance=None):
        print(ti.execution_date)
    test()
```

Wait a moment and see that one schedule is run automatically.
Then trigger a manual DAG run and set the execution date/logical date a few days into the past.
After execution you see a third run of the DAG just at the time of completion of the manual trigger as execution date. But the first execution was just a few moments before, not respecting the desired delta of 1 day.

### Operating System

Ubuntu 20.04 / Breeze Dev setup in Py 3.8 Container

### Versions of Apache Airflow Providers

not relevant

### Deployment

Other

### Deployment details

Started and tested with latest main branch and breeze.

### Anything else

I tried to find the root cause but was not able to locate it. Suspect it is rooted in scheduler_job_runner.py.
I'd supply a fix but was not able to fund the root cause. Expert knowledge is needed probably.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
I reproduced it locally and can confirm it. I am willing to work on a PR to fix this.

After debugging scheduler it seems that the problem is with `calculate_dagrun_date_fields` method of `DagModel`, which is used to plan the next scheduled execution. This method is invoked after completing either manual or scheduled run. It select `next_dagrun_data_interval` based on the date interval of the DAG run that was just completed.
The issue here is that this DAG run might be in the past (as in this scenario) in which case it will incorrectly decide that there should be an extra run scheduled in the past (right after this one).

Resolving patch:
diff --git a/airflow/jobs/scheduler_job_runner.py b/airflow/jobs/scheduler_job_runner.py
--- a/airflow/jobs/scheduler_job_runner.py
+++ b/airflow/jobs/scheduler_job_runner.py
@@ -1192,7 +1192,11 @@ def _create_dag_runs(self, dag_models: Collection[DagModel], session: Session) -
                 )
                 active_runs_of_dags[dag.dag_id] += 1
             if self._should_update_dag_next_dagruns(
-                dag, dag_model, active_runs_of_dags[dag.dag_id], session=session
+                dag,
+                dag_model,
+                last_dag_run=None,
+                total_active_runs=active_runs_of_dags[dag.dag_id],
+                session=session,
             ):
                 dag_model.calculate_dagrun_date_fields(dag, data_interval)
         # TODO[HA]: Should we do a session.flush() so we don't have to keep lots of state/object in
@@ -1300,9 +1304,22 @@ def _create_dag_runs_dataset_triggered(
                 )
 
     def _should_update_dag_next_dagruns(
-        self, dag: DAG, dag_model: DagModel, total_active_runs: int | None = None, *, session: Session
+        self,
+        dag: DAG,
+        dag_model: DagModel,
+        *,
+        last_dag_run: DagRun | None = None,
+        total_active_runs: int | None = None,
+        session: Session,
     ) -> bool:
         """Check if the dag's next_dagruns_create_after should be updated."""
+        # If last_dag_run is defined, the update was triggered by a scheduling decision in this DAG run.
+        # In such case, schedule next only if last_dag_run is finished and was an automated run.
+        if last_dag_run and not (
+            last_dag_run.state in State.finished_dr_states
+            and last_dag_run.run_type in [DagRunType.SCHEDULED, DagRunType.BACKFILL_JOB]
+        ):
+            return False
         # If the DAG never schedules skip save runtime
         if not dag.timetable.can_be_scheduled:
             return False
@@ -1437,8 +1454,8 @@ def _schedule_dag_run(
                 session.merge(task_instance)
             session.flush()
             self.log.info("Run %s of %s has timed-out", dag_run.run_id, dag_run.dag_id)
-            # Work out if we should allow creating a new DagRun now?
-            if self._should_update_dag_next_dagruns(dag, dag_model, session=session):
+
+            if self._should_update_dag_next_dagruns(dag, dag_model, last_dag_run=dag_run, session=session):
                 dag_model.calculate_dagrun_date_fields(dag, dag.get_run_data_interval(dag_run))
 
             callback_to_execute = DagCallbackRequest(
@@ -1465,11 +1482,9 @@ def _schedule_dag_run(
             return callback
         # TODO[HA]: Rename update_state -> schedule_dag_run, ?? something else?
         schedulable_tis, callback_to_run = dag_run.update_state(session=session, execute_callbacks=False)
-        # Check if DAG not scheduled then skip interval calculation to same scheduler runtime
-        if dag_run.state in State.finished_dr_states:
-            # Work out if we should allow creating a new DagRun now?
-            if self._should_update_dag_next_dagruns(dag, dag_model, session=session):
-                dag_model.calculate_dagrun_date_fields(dag, dag.get_run_data_interval(dag_run))
+
+        if self._should_update_dag_next_dagruns(dag, dag_model, last_dag_run=dag_run, session=session):
+            dag_model.calculate_dagrun_date_fields(dag, dag.get_run_data_interval(dag_run))
         # This will do one query per dag run. We "could" build up a complex
         # query to update all the TIs across all the execution dates and dag
         # IDs in a single query, but it turns out that can be _very very slow_


Test patch:
diff --git a/tests/jobs/test_scheduler_job.py b/tests/jobs/test_scheduler_job.py
--- a/tests/jobs/test_scheduler_job.py
+++ b/tests/jobs/test_scheduler_job.py
@@ -1756,10 +1756,7 @@ def test_runs_are_created_after_max_active_runs_was_reached(self, dag_maker, ses
             # Need to use something that doesn't immediately get marked as success by the scheduler
             BashOperator(task_id="task", bash_command="true")
 
-        dag_run = dag_maker.create_dagrun(
-            state=State.RUNNING,
-            session=session,
-        )
+        dag_run = dag_maker.create_dagrun(state=State.RUNNING, session=session, run_type=DagRunType.SCHEDULED)
 
         # Reach max_active_runs
         for _ in range(3):
@@ -3458,7 +3455,50 @@ def test_should_update_dag_next_dagruns(self, schedule, number_running, excepted
         self.job_runner = SchedulerJobRunner(job=scheduler_job)
 
         assert excepted is self.job_runner._should_update_dag_next_dagruns(
-            dag, dag_model, number_running, session=session
+            dag, dag_model, total_active_runs=number_running, session=session
+        )
+
+    @pytest.mark.parametrize(
+        "run_type, should_update",
+        [
+            (DagRunType.MANUAL, False),
+            (DagRunType.SCHEDULED, True),
+            (DagRunType.BACKFILL_JOB, True),
+            (DagRunType.DATASET_TRIGGERED, False),
+        ],
+        ids=[
+            DagRunType.MANUAL.name,
+            DagRunType.SCHEDULED.name,
+            DagRunType.BACKFILL_JOB.name,
+            DagRunType.DATASET_TRIGGERED.name,
+        ],
+    )
+    def test_should_update_dag_next_dagruns_after_run_type(self, run_type, should_update, session, dag_maker):
+        """Test that whether next dagrun is updated depends on run type"""
+        with dag_maker(
+            dag_id="test_should_update_dag_next_dagruns_after_run_type",
+            schedule="*/1 * * * *",
+            max_active_runs=10,
+        ) as dag:
+            EmptyOperator(task_id="dummy")
+
+        dag_model = dag_maker.dag_model
+
+        run = dag_maker.create_dagrun(
+            run_id="run",
+            run_type=run_type,
+            execution_date=DEFAULT_DATE,
+            start_date=timezone.utcnow(),
+            state=State.SUCCESS,
+            session=session,
+        )
+
+        session.flush()
+        scheduler_job = Job(executor=self.null_exec)
+        self.job_runner = SchedulerJobRunner(job=scheduler_job)
+
+        assert should_update is self.job_runner._should_update_dag_next_dagruns(
+            dag, dag_model, last_dag_run=run, total_active_runs=0, session=session
         )
 
     def test_create_dag_runs(self, dag_maker):