Repository: dask/distributed

Issue:
What are the usecases for `Scheduler.handle_missing_data`?
## Introduction

The scheduler offers a handler [`handle_missing_data`](https://github.com/dask/distributed/blob/ba39915ff754c1683c744f99b6643f9832db0375/distributed/scheduler.py#L4654-L4686) that is called in
one place on the worker, during the response handling of [`gather_dep`](https://github.com/dask/distributed/blob/ba39915ff754c1683c744f99b6643f9832db0375/distributed/worker.py#L3352-L3362)

This handling has been known to cause stability and deadlocking issues. In
recent refactoring attempts the `Worker.find_missing` functionality has been
flagged as a source of complexity and I'm wondering if we should remove this
entire system in favor of a much simpler mechanism.

Looking at the scheduler side, reading code and doc string, there are a couple
of different use cases where this handler is supposed to be called. It always
assumes that the signal originates from a worker, called `worker`, which reports
that another worker `errant_worker` is not in possession of a key `key`.

The doc string explains that this can occur in two situations
1. `worker` is operating on stale data, i.e.
   `worker.TaskState.who_has`/`Worker.has_what` includes stale information, such
   that after requesting `Worker.get_data` on `errant_worker` it returns a
   response without data for `key`.
2. `errant_worker` is indeed dead which is likely inferred by `worker` because
   it encountered a network error.

If the scheduler already knows about the death of `errant_worker`, this request is
ignored.

Otherwise, the scheduler accepts this signal as the ultimate truth and removes
all state indicating that `key` was on `errant_worker`. There is some transition
logic in place to heal and/or raise the problem as is appropriate.

The circumstances under which a worker actually calls this handler is a bit
harder to describe. I am much more interested in whether or not this handler on
scheduler side makes sense.

---

Investigating use case 1.) - The worker is operating on stale data. This can
have two reasons

1.a) The data is indeed stale. Something happened that caused us to request
`key` to be forgotten on `errant_worker` but we didn't tell `worker`.

I believe the only valid reason for this is AMM requesting `worker` to forget
the data. All other instances where we would properly release keys due to
transitions, `worker` would eventually be asked to release the key as well.

I would argue that this case is not something a worker should escalate to the
scheduler. If anything, the worker should ask the scheduler for the most recent,
accurate information and try again. Particularly with the changes proposed in
https://github.com/dask/distributed/pull/6435 where the worker indeed removes no
longer accurate entries, this use case feels obsolete

1.b) The scheduler has faulty information

I am struggling to come up with a situation where the scheduler is indeed
operating on false information since workers are never allowed to forget data
without the scheduler telling them so. It is possible for workers to have more
data than the scheduler knows (e.g. because messages where not, yet, delivered)
but it's hard to come up with a scenario where the worker lost data.
This could obviously be possible if a worker would loose its data due to
external factors, e.g. disk is lost or a user deliberately removes keys from
`Worker.data`. I would argue in this situation `errant_worker` should notice
that an external party is messing with its state and it should escalate to the
scheduler, iff we even want to deal with this edge case.


2.) `errant_worker` is indeed dead. The scheduler would eventually notice
because `Scheduler.stream_comms[errant_worker]` would be aborted (at the latest
after `distributed.comm.timeouts.tcp` seconds)
The only reason why `worker` should ping the scheduler on this event is if we
want to accelerate recovery. From a consistency perspective, this should not be
required


## Conclusion

I am starting to believe that `worker` should simply remove `errant_worker` from
it's internal bookkeeping and rely on the scheduler to eventually clean up. We
would accept that a `worker.TaskState` in state `fetch` is allowed to have an
empty `who_has` and we'd rely on the scheduler to clean this up eventually.

The only functionality we'd need on `Worker` is a mechanism to eventually
refresh the worker's `who_has`/`has_what` information. 

This could be a system that is *not* connected to the state machine directly
that ensures that all, or subsets of, data locality information
(who_has/has_what) is updated _eventually_. This generic system would also
benefit `busy` tasks.

I believe this reduced coupling would significantly reduce complexity. I'm
wondering if I missed a valid use case.


cc @gjoseph92 @crusaderky 


Discussion:
I think there are two distinct use cases:
1. the sender worker is alive and well, but responds to gather_dep that it does not have a particular key.
I agree with you that missing-data should not be necessary in this case, as I cannot think of any way where a `release-worker-data` message won't reach the scheduler soon enough
2. the sender worker is dead, and the recipient worker just happened to notice before the scheduler.
I also agree that `missing-data` is inappropriate here. However I wonder if the recipient should just send a `missing-worker` event instead, which accelerates the last rites on the scheduler side? To clarify: this is a matter of performance, not consistency.
xref https://github.com/dask/distributed/pull/6348

Resolving patch:
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -3023,7 +3023,6 @@ def __init__(
             "task-erred": self.handle_task_erred,
             "release-worker-data": self.release_worker_data,
             "add-keys": self.add_keys,
-            "missing-data": self.handle_missing_data,
             "long-running": self.handle_long_running,
             "reschedule": self.reschedule,
             "keep-alive": lambda *args, **kwargs: None,
@@ -4667,40 +4666,6 @@ def handle_task_erred(self, key: str, stimulus_id: str, **msg) -> None:
         self._transitions(recommendations, client_msgs, worker_msgs, stimulus_id)
         self.send_all(client_msgs, worker_msgs)
 
-    def handle_missing_data(
-        self, key: str, worker: str, errant_worker: str, stimulus_id: str
-    ) -> None:
-        """Signal that `errant_worker` does not hold `key`.
-
-        This may either indicate that `errant_worker` is dead or that we may be working
-        with stale data and need to remove `key` from the workers `has_what`. If no
-        replica of a task is available anymore, the task is transitioned back to
-        released and rescheduled, if possible.
-
-        Parameters
-        ----------
-        key : str
-            Task key that could not be found
-        worker : str
-            Address of the worker informing the scheduler
-        errant_worker : str
-            Address of the worker supposed to hold a replica
-        """
-        logger.debug(f"handle missing data {key=} {worker=} {errant_worker=}")
-        self.log_event(errant_worker, {"action": "missing-data", "key": key})
-
-        ts = self.tasks.get(key)
-        ws = self.workers.get(errant_worker)
-        if not ts or not ws or ws not in ts.who_has:
-            return
-
-        self.remove_replica(ts, ws)
-        if ts.state == "memory" and not ts.who_has:
-            if ts.run_spec:
-                self.transitions({key: "released"}, stimulus_id)
-            else:
-                self.transitions({key: "forgotten"}, stimulus_id)
-
     def release_worker_data(self, key: str, worker: str, stimulus_id: str) -> None:
         ts = self.tasks.get(key)
         ws = self.workers.get(worker)
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -127,7 +127,6 @@
     InvalidTaskState,
     InvalidTransition,
     LongRunningMsg,
-    MissingDataMsg,
     RecommendationsConflict,
     Recs,
     RecsInstrs,
@@ -3387,13 +3386,6 @@ def done_event():
                     self.has_what[worker].discard(ts.key)
                     self.data_needed_per_worker[worker].discard(ts)
                     self.log.append((d, "missing-dep", stimulus_id, time()))
-                    instructions.append(
-                        MissingDataMsg(
-                            key=d,
-                            errant_worker=worker,
-                            stimulus_id=stimulus_id,
-                        )
-                    )
                     recommendations[ts] = "fetch"
 
             if refresh_who_has:
diff --git a/distributed/worker_state_machine.py b/distributed/worker_state_machine.py
--- a/distributed/worker_state_machine.py
+++ b/distributed/worker_state_machine.py
@@ -389,15 +389,6 @@ class ReleaseWorkerDataMsg(SendMessageToScheduler):
     key: str
 
 
-@dataclass
-class MissingDataMsg(SendMessageToScheduler):
-    op = "missing-data"
-
-    __slots__ = ("key", "errant_worker")
-    key: str
-    errant_worker: str
-
-
 # Not to be confused with RescheduleEvent below or the distributed.Reschedule Exception
 @dataclass
 class RescheduleMsg(SendMessageToScheduler):


Test patch:
diff --git a/distributed/tests/test_cancelled_state.py b/distributed/tests/test_cancelled_state.py
--- a/distributed/tests/test_cancelled_state.py
+++ b/distributed/tests/test_cancelled_state.py
@@ -101,19 +101,6 @@ def f(ev):
     )
 
 
-@gen_cluster(client=True)
-async def test_worker_find_missing(c, s, a, b):
-    fut = c.submit(inc, 1, workers=[a.address])
-    await fut
-    # We do not want to use proper API since it would ensure that the cluster is
-    # informed properly
-    del a.data[fut.key]
-    del a.tasks[fut.key]
-
-    # Actually no worker has the data; the scheduler is supposed to reschedule
-    assert await c.submit(inc, fut, workers=[b.address]) == 3
-
-
 @gen_cluster(client=True)
 async def test_worker_stream_died_during_comm(c, s, a, b):
     write_queue = asyncio.Queue()
diff --git a/distributed/tests/test_worker.py b/distributed/tests/test_worker.py
--- a/distributed/tests/test_worker.py
+++ b/distributed/tests/test_worker.py
@@ -3129,6 +3129,7 @@ async def test_gather_dep_cancelled_rescheduled(c, s):
     for in-flight state. The response parser, however, did not distinguish
     resulting in unwanted missing-data signals to the scheduler, causing
     potential rescheduling or data leaks.
+    (Note: missing-data was removed in #6445).
 
     If a cancelled key is rescheduled for fetching while gather_dep waits
     internally for get_data, the response parser would misclassify this key and
@@ -3177,6 +3178,8 @@ async def test_gather_dep_do_not_handle_response_of_not_requested_tasks(c, s, a)
     for in-flight state. The response parser, however, did not distinguish
     resulting in unwanted missing-data signals to the scheduler, causing
     potential rescheduling or data leaks.
+    (Note: missing-data was removed in #6445).
+
     This test may become obsolete if the implementation changes significantly.
     """
     async with BlockedGatherDep(s.address) as b: