Repository: apache/airflow

Issue:
Sensors in reschedule mode are not rescheduled
**Apache Airflow version**:
2.0.0dev

**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):

**Environment**:
```
./breeze --python=3.8 --backend=postgres --db-reset restart
```

**What happened**:

Sensors in reschedule mode are not rescheduled by scheduler.

**What you expected to happen**:

Sensors in both poke and reschedule mode should work.

**How to reproduce it**:

```
from airflow import DAG
from airflow.sensors.base_sensor_operator import BaseSensorOperator
from airflow.utils.dates import days_ago

class DummySensor(BaseSensorOperator):
    def poke(self, context):
        return False

with DAG(
    "other_dag",
    start_date=days_ago(1),
    schedule_interval="*/5 * * * *",
    catchup=False
) as dag3:
    DummySensor(
        task_id='wait-task',
        poke_interval=60 * 5,
        mode='reschedule'
    )
```

Then:
```
root@053f6ca34e24: /opt/airflow# airflow dags unpause other_dag
Dag: other_dag, paused: False
root@053f6ca34e24: /opt/airflow# airflow scheduler
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2020-12-03 14:18:58,404] {scheduler_job.py:1247} INFO - Starting the scheduler
[2020-12-03 14:18:58,404] {scheduler_job.py:1252} INFO - Processing each file at most -1 times
[2020-12-03 14:18:58,571] {dag_processing.py:250} INFO - Launched DagFileProcessorManager with pid: 63835
[2020-12-03 14:18:58,576] {scheduler_job.py:1757} INFO - Resetting orphaned tasks for active dag runs
[2020-12-03 14:18:58,660] {settings.py:52} INFO - Configured default timezone Timezone('UTC')
[2020-12-03 14:18:58,916] {scheduler_job.py:944} INFO - 1 tasks up for execution:
	<TaskInstance: other_dag.wait-task 2020-12-03 14:10:00+00:00 [scheduled]>
[2020-12-03 14:18:58,920] {scheduler_job.py:973} INFO - Figuring out tasks to run in Pool(name=default_pool) with 128 open slots and 1 task instances ready to be queued
[2020-12-03 14:18:58,921] {scheduler_job.py:1001} INFO - DAG other_dag has 0/16 running and queued tasks
[2020-12-03 14:18:58,921] {scheduler_job.py:1066} INFO - Setting the following tasks to queued state:
	<TaskInstance: other_dag.wait-task 2020-12-03 14:10:00+00:00 [scheduled]>
[2020-12-03 14:18:58,925] {scheduler_job.py:1108} INFO - Sending TaskInstanceKey(dag_id='other_dag', task_id='wait-task', execution_date=datetime.datetime(2020, 12, 3, 14, 10, tzinfo=Timezone('UTC')), try_number=1) to executor with priority 1 and queue default
[2020-12-03 14:18:58,926] {base_executor.py:79} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'other_dag', 'wait-task', '2020-12-03T14:10:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/files/dags/the_old_issue.py']
[2020-12-03 14:18:58,935] {local_executor.py:80} INFO - QueuedLocalWorker running ['airflow', 'tasks', 'run', 'other_dag', 'wait-task', '2020-12-03T14:10:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/files/dags/the_old_issue.py']
[2020-12-03 14:18:59,063] {dagbag.py:440} INFO - Filling up the DagBag from /files/dags/the_old_issue.py
Running <TaskInstance: other_dag.wait-task 2020-12-03T14:10:00+00:00 [queued]> on host 053f6ca34e24
[2020-12-03 14:19:00,022] {scheduler_job.py:944} INFO - 1 tasks up for execution:
	<TaskInstance: other_dag.wait-task 2020-12-03 14:10:00+00:00 [scheduled]>
[2020-12-03 14:19:00,029] {scheduler_job.py:973} INFO - Figuring out tasks to run in Pool(name=default_pool) with 128 open slots and 1 task instances ready to be queued
[2020-12-03 14:19:00,029] {scheduler_job.py:1001} INFO - DAG other_dag has 0/16 running and queued tasks
[2020-12-03 14:19:00,029] {scheduler_job.py:1066} INFO - Setting the following tasks to queued state:
	<TaskInstance: other_dag.wait-task 2020-12-03 14:10:00+00:00 [scheduled]>
[2020-12-03 14:19:00,033] {scheduler_job.py:1108} INFO - Sending TaskInstanceKey(dag_id='other_dag', task_id='wait-task', execution_date=datetime.datetime(2020, 12, 3, 14, 10, tzinfo=Timezone('UTC')), try_number=1) to executor with priority 1 and queue default
[2020-12-03 14:19:00,033] {base_executor.py:82} ERROR - could not queue task TaskInstanceKey(dag_id='other_dag', task_id='wait-task', execution_date=datetime.datetime(2020, 12, 3, 14, 10, tzinfo=Timezone('UTC')), try_number=1)
[2020-12-03 14:19:00,038] {scheduler_job.py:1199} INFO - Executor reports execution of other_dag.wait-task execution_date=2020-12-03 14:10:00+00:00 exited with status success for try_number 1
[2020-12-03 14:19:00,045] {scheduler_job.py:1235} ERROR - Executor reports task instance <TaskInstance: other_dag.wait-task 2020-12-03 14:10:00+00:00 [queued]> finished (success) although the task says its queued. (Info: None) Was the task killed externally?
[2020-12-03 14:19:01,173] {dagrun.py:429} ERROR - Marking run <DagRun other_dag @ 2020-12-03 14:10:00+00:00: scheduled__2020-12-03T14:10:00+00:00, externally triggered: False> failed
```


**Anything else we need to know**:

Discovered when working on #10790 

Thank @nathadfield for helping discover this issue!



Discussion:
On of the problems is that scheduler schedule a task two times:
```
[2020-12-03 15:23:03,332] {scheduler_job.py:1066} INFO - Setting the following tasks to queued state:
	<TaskInstance: other_dag.wait-task 2020-12-03 15:22:00+00:00 [scheduled]>
[2020-12-03 15:23:03,336] {scheduler_job.py:1108} INFO - Sending TaskInstanceKey(dag_id='other_dag', task_id='wait-task', execution_date=datetime.datetime(2020, 12, 3, 15, 22, tzinfo=Timezone('UTC')), try_number=1) to executor with priority 1 and queue default
[2020-12-03 15:23:03,337] {base_executor.py:78} DEBUG - TI: TaskInstanceKey(dag_id='other_dag', task_id='wait-task', execution_date=datetime.datetime(2020, 12, 3, 15, 22, tzinfo=Timezone('UTC')), try_number=1) QUEUED: OrderedDict(), RUNNING: {TaskInstanceKey(dag_id='other_dag', task_id='wait-task', execution_date=datetime.datetime(2020, 12, 3, 15, 22, tzinfo=Timezone('UTC')), try_number=1)}
[2020-12-03 15:23:03,338] {base_executor.py:83} ERROR - Could not queue task TaskInstanceKey(dag_id='other_dag', task_id='wait-task', execution_date=datetime.datetime(2020, 12, 3, 15, 22, tzinfo=Timezone('UTC')), try_number=1)
```
I can confirm that this issue does not exists in 1.10 branch and at ff1a2aa. Sensor is rescheduled as it should be.
I've worked out the problem -- it's caused by the switch to Serialized DAGs only in HA scheduler.

I have a hacky-one line fix. and am working on a better fix now.

Resolving patch:
diff --git a/airflow/models/baseoperator.py b/airflow/models/baseoperator.py
--- a/airflow/models/baseoperator.py
+++ b/airflow/models/baseoperator.py
@@ -657,19 +657,19 @@ def dag_id(self) -> str:
         else:
             return 'adhoc_' + self.owner
 
-    @property
-    def deps(self) -> Set[BaseTIDep]:
-        """
-        Returns the set of dependencies for the operator. These differ from execution
-        context dependencies in that they are specific to tasks and can be
-        extended/overridden by subclasses.
-        """
-        return {
+    deps: Iterable[BaseTIDep] = frozenset(
+        {
             NotInRetryPeriodDep(),
             PrevDagrunDep(),
             TriggerRuleDep(),
             NotPreviouslySkippedDep(),
         }
+    )
+    """
+    Returns the set of dependencies for the operator. These differ from execution
+    context dependencies in that they are specific to tasks and can be
+    extended/overridden by subclasses.
+    """
 
     def prepare_for_execution(self) -> "BaseOperator":
         """
diff --git a/airflow/sensors/base_sensor_operator.py b/airflow/sensors/base_sensor_operator.py
--- a/airflow/sensors/base_sensor_operator.py
+++ b/airflow/sensors/base_sensor_operator.py
@@ -279,7 +279,6 @@ def reschedule(self):
         """Define mode rescheduled sensors."""
         return self.mode == 'reschedule'
 
-    # pylint: disable=no-member
     @property
     def deps(self):
         """
@@ -287,8 +286,8 @@ def deps(self):
         checks if a sensor task instance can be rescheduled.
         """
         if self.reschedule:
-            return BaseOperator.deps.fget(self) | {ReadyToRescheduleDep()}
-        return BaseOperator.deps.fget(self)
+            return super().deps | {ReadyToRescheduleDep()}
+        return super().deps
 
 
 def poke_mode_only(cls):
diff --git a/airflow/serialization/serialized_objects.py b/airflow/serialization/serialized_objects.py
--- a/airflow/serialization/serialized_objects.py
+++ b/airflow/serialization/serialized_objects.py
@@ -20,7 +20,7 @@
 import enum
 import logging
 from inspect import Parameter, signature
-from typing import Any, Dict, Iterable, Optional, Set, Union
+from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Optional, Set, Union
 
 import cattr
 import pendulum
@@ -58,6 +58,10 @@
     HAS_KUBERNETES = False
 
 
+if TYPE_CHECKING:
+    from airflow.ti_deps.deps.base_ti_dep import BaseTIDep
+
+
 log = logging.getLogger(__name__)
 FAILED = 'serialization_failed'
 
@@ -361,7 +365,7 @@ def task_type(self, task_type: str):
         self._task_type = task_type
 
     @classmethod
-    def serialize_operator(cls, op: BaseOperator) -> dict:
+    def serialize_operator(cls, op: BaseOperator) -> Dict[str, Any]:
         """Serializes operator into a JSON object."""
         serialize_op = cls.serialize_to_json(op, cls._decorated_fields)
         serialize_op['_task_type'] = op.__class__.__name__
@@ -375,6 +379,22 @@ def serialize_operator(cls, op: BaseOperator) -> dict:
                 op.operator_extra_links
             )
 
+        if op.deps is not BaseOperator.deps:
+            # Are the deps different to BaseOperator, if so serialize the class names!
+            # For Airflow 2.0 expediency we _only_ allow built in Dep classes.
+            # Fix this for 2.0.x or 2.1
+            deps = []
+            for dep in op.deps:
+                klass = type(dep)
+                module_name = klass.__module__
+                if not module_name.startswith("airflow.ti_deps.deps."):
+                    raise ValueError(
+                        f"Cannot serialize task with `deps` from non-core module {module_name!r}"
+                    )
+
+                deps.append(f'{module_name}.{klass.__name__}')
+            serialize_op['deps'] = deps
+
         # Store all template_fields as they are if there are JSON Serializable
         # If not, store them as strings
         if op.template_fields:
@@ -438,6 +458,9 @@ def deserialize_operator(cls, encoded_op: Dict[str, Any]) -> BaseOperator:
 
                 v = list(op_predefined_extra_links.values())
                 k = "operator_extra_links"
+
+            elif k == "deps":
+                v = cls._deserialize_deps(v)
             elif k in cls._decorated_fields or k not in op.get_serialized_fields():
                 v = cls._deserialize(v)
             # else use v as it is
@@ -467,6 +490,20 @@ def _is_excluded(cls, var: Any, attrname: str, op: BaseOperator):
                 return True
         return super()._is_excluded(var, attrname, op)
 
+    @classmethod
+    def _deserialize_deps(cls, deps: List[str]) -> Set["BaseTIDep"]:
+        instances = set()
+        for qualname in set(deps):
+            if not qualname.startswith("airflow.ti_deps.deps."):
+                log.error("Dep class %r not registered", qualname)
+                continue
+
+            try:
+                instances.add(import_string(qualname)())
+            except ImportError:
+                log.warning("Error importing dep %r", qualname, exc_info=True)
+        return instances
+
     @classmethod
     def _deserialize_operator_extra_links(cls, encoded_op_links: list) -> Dict[str, BaseOperatorLink]:
         """


Test patch:
diff --git a/tests/serialization/test_dag_serialization.py b/tests/serialization/test_dag_serialization.py
--- a/tests/serialization/test_dag_serialization.py
+++ b/tests/serialization/test_dag_serialization.py
@@ -944,6 +944,32 @@ def check_task_group(node):
 
         check_task_group(serialized_dag.task_group)
 
+    @parameterized.expand(
+        [
+            ("poke", False),
+            ("reschedule", True),
+        ]
+    )
+    def test_serialize_sensor(self, mode, expect_custom_deps):
+        from airflow.sensors.base_sensor_operator import BaseSensorOperator
+
+        class DummySensor(BaseSensorOperator):
+            def poke(self, context):
+                return False
+
+        op = DummySensor(task_id='dummy', mode=mode, poke_interval=23)
+
+        blob = SerializedBaseOperator.serialize_operator(op)
+
+        if expect_custom_deps:
+            assert "deps" in blob
+        else:
+            assert "deps" not in blob
+
+        serialized_op = SerializedBaseOperator.deserialize_operator(blob)
+
+        assert op.deps == serialized_op.deps
+
 
 def test_kubernetes_optional():
     """Serialisation / deserialisation continues to work without kubernetes installed"""