Repository: apache/airflow

Issue:
Airflow scheduler crashed with TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
### Apache Airflow version

2.1.4

### Operating System

Ubuntu 20.04.3 LTS

### Versions of Apache Airflow Providers

_No response_

### Deployment

Other Docker-based deployment

### Deployment details

_No response_

### What happened

Airflow scheduler crashed with following exception
```
[2021-11-23 22:41:16,528] {scheduler_job.py:662} INFO - Starting the scheduler
[2021-11-23 22:41:16,528] {scheduler_job.py:667} INFO - Processing each file at most -1 times
[2021-11-23 22:41:16,639] {manager.py:254} INFO - Launched DagFileProcessorManager with pid: 19
[2021-11-23 22:41:16,641] {scheduler_job.py:1217} INFO - Resetting orphaned tasks for active dag runs
[2021-11-23 22:41:16,644] {settings.py:51} INFO - Configured default timezone Timezone('Etc/GMT-7')
[2021-11-23 22:41:19,016] {scheduler_job.py:711} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/airflow/jobs/scheduler_job.py", line 695, in _execute
    self._run_scheduler_loop()
  File "/usr/local/lib/python3.8/dist-packages/airflow/jobs/scheduler_job.py", line 788, in _run_scheduler_loop
    num_queued_tis = self._do_scheduling(session)
  File "/usr/local/lib/python3.8/dist-packages/airflow/jobs/scheduler_job.py", line 901, in _do_scheduling
    callback_to_run = self._schedule_dag_run(dag_run, session)
  File "/usr/local/lib/python3.8/dist-packages/airflow/jobs/scheduler_job.py", line 1143, in _schedule_dag_run
    schedulable_tis, callback_to_run = dag_run.update_state(session=session, execute_callbacks=False)
  File "/usr/local/lib/python3.8/dist-packages/airflow/utils/session.py", line 67, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/airflow/models/dagrun.py", line 438, in update_state
    info = self.task_instance_scheduling_decisions(session)
  File "/usr/local/lib/python3.8/dist-packages/airflow/utils/session.py", line 67, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/airflow/models/dagrun.py", line 539, in task_instance_scheduling_decisions
    schedulable_tis, changed_tis = self._get_ready_tis(scheduleable_tasks, finished_tasks, session)
  File "/usr/local/lib/python3.8/dist-packages/airflow/models/dagrun.py", line 565, in _get_ready_tis
    if st.are_dependencies_met(
  File "/usr/local/lib/python3.8/dist-packages/airflow/utils/session.py", line 67, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 890, in are_dependencies_met
    for dep_status in self.get_failed_dep_statuses(dep_context=dep_context, session=session):
  File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 911, in get_failed_dep_statuses
    for dep_status in dep.get_dep_statuses(self, session, dep_context):
  File "/usr/local/lib/python3.8/dist-packages/airflow/ti_deps/deps/base_ti_dep.py", line 101, in get_dep_statuses
    yield from self._get_dep_statuses(ti, session, dep_context)
  File "/usr/local/lib/python3.8/dist-packages/airflow/ti_deps/deps/ready_to_reschedule.py", line 66, in _get_dep_statuses
    if now >= next_reschedule_date:
TypeError: '>=' not supported between instances of 'datetime.datetime' and 'NoneType'
[2021-11-23 22:41:20,020] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 19
```

### What you expected to happen

_No response_

### How to reproduce

Define a `BaseSensorOperator` task with large `poke_interval` with `reschedule` mode

```
BaseSensorOperator(
    task_id='task',
    poke_interval=863998946,
    mode='reschedule',
    dag=dag
)
```

### Anything else

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

As per https://dev.mysql.com/doc/refman/5.7/en/datetime.html, 
> The TIMESTAMP data type is used for values that contain both date and time parts. TIMESTAMP has a range of '1970-01-01 00:00:01' UTC to '2038-01-19 03:14:07' UTC.
> Invalid DATE, DATETIME, or TIMESTAMP values are converted to the “zero” value of the appropriate type ('0000-00-00' or '0000-00-00 00:00:00'), if the SQL mode permits this conversion

reschedule_date was set to `0000-00-00 00:00:00.000000` since it's exceeded the upper limit.
It's a big problem if the scheduler crash due to problem with a specific DAG. This means that a single DAG causes problems for all the other DAGs.

I'm also interested in why would you want to set interval for more than 9999 days? While crashed scheduler is a big problem the use case of getting there seems theoretical to me so I wouldn't say that this is require an urgent fix.
@eladkal 
I generated the dag from code but the interval was mistakenly calculated in milliseconds rather than seconds. 
So the value was unintended? In that case I guess we could just invent an arbitrary cap to `poke_interval` and just make the DAG error out on parse. It does not need to be that precise either.
Ugh, MySQLs date "handling" makes me sad.
> Ugh, MySQLs date "handling" makes me sad.

Very much so.

Resolving patch:
diff --git a/airflow/sensors/base.py b/airflow/sensors/base.py
--- a/airflow/sensors/base.py
+++ b/airflow/sensors/base.py
@@ -17,12 +17,14 @@
 # under the License.
 
 import datetime
+import functools
 import hashlib
 import os
 import time
 from datetime import timedelta
 from typing import Any, Callable, Dict, Iterable
 
+from airflow import settings
 from airflow.configuration import conf
 from airflow.exceptions import (
     AirflowException,
@@ -41,6 +43,16 @@
 # See  https://github.com/apache/airflow/issues/16035
 from airflow.utils.decorators import apply_defaults
 
+# As documented in https://dev.mysql.com/doc/refman/5.7/en/datetime.html.
+_MYSQL_TIMESTAMP_MAX = datetime.datetime(2038, 1, 19, 3, 14, 7, tzinfo=timezone.utc)
+
+
+@functools.lru_cache(maxsize=None)
+def _is_metadatabase_mysql() -> bool:
+    if settings.engine is None:
+        raise AirflowException("Must initialize ORM first")
+    return settings.engine.url.get_backend_name() == "mysql"
+
 
 class BaseSensorOperator(BaseOperator, SkipMixin):
     """
@@ -125,6 +137,17 @@ def _validate_input_values(self) -> None:
                 f"The mode must be one of {self.valid_modes},'{self.dag.dag_id if self.has_dag() else ''}.{self.task_id}'; received '{self.mode}'."
             )
 
+        # Sanity check for poke_interval isn't immediately over MySQL's TIMESTAMP limit.
+        # This check is only rudimentary to catch trivial user errors, e.g. mistakenly
+        # set the value to milliseconds instead of seconds. There's another check when
+        # we actually try to reschedule to ensure database sanity.
+        if self.reschedule and _is_metadatabase_mysql():
+            if timezone.utcnow() + datetime.timedelta(seconds=self.poke_interval) > _MYSQL_TIMESTAMP_MAX:
+                raise AirflowException(
+                    f"Cannot set poke_interval to {self.poke_interval} seconds in reschedule "
+                    f"mode since it will take reschedule time over MySQL's TIMESTAMP limit."
+                )
+
     def poke(self, context: Dict) -> bool:
         """
         Function that the sensors defined while deriving this class should
@@ -233,9 +256,13 @@ def run_duration() -> float:
                 else:
                     raise AirflowSensorTimeout(f"Snap. Time is OUT. DAG id: {log_dag_id}")
             if self.reschedule:
-                reschedule_date = timezone.utcnow() + timedelta(
-                    seconds=self._get_next_poke_interval(started_at, run_duration, try_number)
-                )
+                next_poke_interval = self._get_next_poke_interval(started_at, run_duration, try_number)
+                reschedule_date = timezone.utcnow() + timedelta(seconds=next_poke_interval)
+                if _is_metadatabase_mysql() and reschedule_date > _MYSQL_TIMESTAMP_MAX:
+                    raise AirflowSensorTimeout(
+                        f"Cannot reschedule DAG {log_dag_id} to {reschedule_date.isoformat()} "
+                        f"since it is over MySQL's TIMESTAMP storage limit."
+                    )
                 raise AirflowRescheduleException(reschedule_date)
             else:
                 time.sleep(self._get_next_poke_interval(started_at, run_duration, try_number))


Test patch:
diff --git a/tests/sensors/test_base.py b/tests/sensors/test_base.py
--- a/tests/sensors/test_base.py
+++ b/tests/sensors/test_base.py
@@ -498,6 +498,28 @@ def run_duration():
             assert interval2 >= sensor.poke_interval
             assert interval2 > interval1
 
+    @pytest.mark.backend("mysql")
+    def test_reschedule_poke_interval_too_long_on_mysql(self, make_sensor):
+        with pytest.raises(AirflowException) as ctx:
+            make_sensor(poke_interval=863998946, mode="reschedule", return_value="irrelevant")
+        assert str(ctx.value) == (
+            "Cannot set poke_interval to 863998946 seconds in reschedule mode "
+            "since it will take reschedule time over MySQL's TIMESTAMP limit."
+        )
+
+    @pytest.mark.backend("mysql")
+    def test_reschedule_date_too_late_on_mysql(self, make_sensor):
+        sensor, _ = make_sensor(poke_interval=60 * 60 * 24, mode="reschedule", return_value=False)
+
+        # A few hours until TIMESTAMP's limit, the next poke will take us over.
+        with freeze_time(datetime(2038, 1, 19, tzinfo=timezone.utc)):
+            with pytest.raises(AirflowSensorTimeout) as ctx:
+                self._run(sensor)
+        assert str(ctx.value) == (
+            "Cannot reschedule DAG unit_test_dag to 2038-01-20T00:00:00+00:00 "
+            "since it is over MySQL's TIMESTAMP storage limit."
+        )
+
     def test_reschedule_and_retry_timeout(self, make_sensor):
         """
         Test mode="reschedule", retries and timeout configurations interact correctly.