Repository: dask/distributed

Issue:
Redesign worker exponential backoff on busy-gather
When a worker receives a gather_dep request from another worker, but it is too busy with other transfers already, it will reply ``{"status": "busy"}``. When this happens, there are three possible use cases:

1. there are other workers that hold replicas of the same task, and the requesting worker knows about them
2. there are other workers that hold replicas of the same task, but the requesting worker doesn't know. The worker needs to ask the scheduler through ``query_who_has``.
3. the busy worker holds the only replica of that task

worker.py implements an exponential backoff system, implemented in 2018 and never revisited (#2092) which, in theory, prevents it from
- spamming the busy worker with consecutive gather requests
- spamming the scheduler with``query_who_has``

It is implemented as follows (pseudocode):
```python
class Worker:
    repetitively_busy = 0
    data_needed = Heap()  # tasks in fetch status
    
    async def handle_scheduler(self):
        while True:
            await next_message_from_scheduler
            self.ensure_communicating()

    def ensure_communicating(self):
        while self.data_needed:
            ts = self.data_needed.pop()

            ws_local = [ws for ws in ts.who_has if same_host(ws, self)]
            ws_remote = [ws for ws in ts.who_has if not same_host(ws, self)]
            ws = random.choice(ws_local or ws_remote)

            self.transition_fetch_flight(ts)
            loop.add_callback(self.gather_dep, ws, ts)  # fire and forget task

    async def gather_dep(self, ws, ts):
        response = await get_data_from_worker(ws, ts)
        if response["status"] == "busy":
            self.transition_flight_fetch(ts)  # Add task back to self.data_needed
            # Exponential backoff to avoid hammering scheduler/worker
            self.repetitively_busy += 1
            await asyncio.sleep(0.100 * 1.5**self.repetitively_busy)
            await self.query_who_has(ts)
        else:
            self.repetitively_busy = 0
            # Omissis: deal with successful response and other failure use cases
        self.ensure_communicating()
```

If any other unrelated gather_dep completes or any message arrives from the scheduler, ensure_communicating will try fetching the task again. On a busy worker, this will happen almost instantly.
As there is no way to flag *which* worker was busy, if the busy worker was either the only local worker or the only worker on the whole cluster holding the data, it will be targeted again with get_data_from_worker.
This means that, almost instantly, the busy worker will reply the same thing and repetitively_busy will bump up a notch - *unless* a gather_dep from another worker recently terminated and the other worker was not busy.
It also means that there will be multiple identical gather_dep calls running in parallel, all waiting on the sleep.
query_who_has will kick off potentially several seconds later, potentially when it's no longer needed, e.g. because another iteration of ensure_communicating successfully fetched the task from another worker.

On the flip side, the worker may experience a burst of heavy chatter with the scheduler (but not other workers), followed by silence. For example, a bunch of tasks are submitted in a short timeframe, with just enough time between each so that they land in different messages, and now we need to wait for their execution to complete. This may cause repetitively_busy to rocket up to a very high number and sleep for - potentially - centuries before anything kicks off ensure_communicating again. **DEADLOCK!**

Finally, this interacts with ``select_keys_for_gather``, which coagulates individually small fetches into the same message. If there are multiple tasks in fetch status pending and they're fairly large (``target_message_size`` = 50MB), then a single ensure_communicating will spawn multiple contemporaneous gather_dep requests to the same worker, up to ``total_out_connections``. They may all return busy, which in turn will increase ``repetitively_busy`` by several notches in a single go. Reducing ``target_message_size`` may result in a substantial slowdown, not because of the ``select_keys_for_gather`` system itself, but because ``repetitively_busy`` is reaching a higher number in a single iteration of ensure_communicating.

Besides its own shortcomings, this design is a blocker to #5896. The refactor removes the periodic kick to ensure_communicating from handle_scheduler, and instead kicks it off from ``transition_*_fetch``. This means that the above will enter an infinite loop where the requesting worker starts accumulating more and more gather_dep tasks, and the busy worker is constantly hammered with get_data_from_worker. The sleep time quickly rises into the centuries.
A naive solution would be to move the sleep before ``self.transition_flight_fetch(ts)``. However, this would mean losing an unplanned, but highly desirable feature of the current design, which is if there are multiple workers holding replicas of the same task and the requesting worker is currently chatty, then another random worker will be tried straight away without waiting for the sleep.

# Proposed design
- Get rid of repetitively_busy.
- Add to distributed.worker.WorkerState a ``busy: bool`` flag.
- Worker changes as follows:

```python
class Worker:
    data_needed = Heap()  # tasks in fetch status
    
    async def handle_scheduler(self):
        while True:
            await next_message_from_scheduler
            self.ensure_communicating()  # To be removed in #5896

    def ensure_communicating(self):
        busy_pushback = []
        while self.data_needed:
            ts = self.data_needed.pop()

            who_has = [ws for ws in ts.who_has if not ws.busy]
            if not who_has:
                busy_pushback.append(ts)
                continue

            ws_local = [ws for ws in who_has if same_host(ws, self)]
            ws_remote = [ws for ws in who_has if not same_host(ws, self)]
            ws = random.choice(ws_local or ws_remote)

            self.transition_fetch_flight(ts)
            loop.add_callback(self.gather_dep, ws, ts)  # fire and forget task

        for ts in busy_pushback:
            self.data_needed.push(ts)

    async def gather_dep(self, ws, ts):
        response = await get_data_from_worker(ws, ts)
        if response["status"] == "busy":
            ws.busy = True

            def reset_busy():
                ws.busy = False
                self.ensure_communicating()

            call_later(0.15, reset_busy)  # Hardcoded, not incremental

            self.transition_flight_fetch(ts)  # Add task back to self.data_needed
            self.ensure_communicating()
            if ts.status != "fetch":
                return  # fetching from another worker

            if all(ws2.busy for ws2 in ts.who_has):
                await self.query_who_has(ts)
                self.ensure_communicating()
```
The new algorithm works as follows:
1. try fetching the task from a random, non-busy local worker
2. if busy and there are non-busy local workers, goto 1
3. if busy and all local workers are busy, try fetching the task from a random, non-busy remote worker
4. if busy and there are non-busy remote workers, goto 3
5. if at any time while trying remote workers a local worker becomes non-busy, goto 1
6. if all local and remote workers are busy, immediately query_who_has the scheduler to discover more workers
7. if the scheduler responds saying that previously unknown workers hold a replica, goto 1
8. if all else fails, sleep 150ms and then goto 1

#### Note
If at least one local worker holds a replica, the old algorithm waits indefinitely for a local worker to become non-busy and completely ignores remote workers, thus avoiding the cost of network transfer but incurring in a delay. The new algorithm instead immediately falls back to remote workers.


Discussion:
Quick historical note, one of the objectives of the busy signal was to avoid swarming a single worker, yes.  A possibly larger objective, and on that's not immediately obvious from the implementation, is that we should spread data out using a tree of workers.  The who_has check with the scheduler and the random worker selection help to ensure this behavior in aggregate.  

I don't see any concern with the new design with regards to this constraint, I just wanted to make sure that it was in your mind as you were working on this.
Correction: ``target_message_size`` and ``select_keys_for_gather`` are not influential, because you can't have more than one active transfer per worker thanks to ``in_flight_workers``. You can, however, have multiple gather_deps sleeping per worker, since the sleep is after the removal from in_flight_workers.
The problem about waiting potentially indefinitely on a local worker was raised with a proposed fix in https://github.com/dask/distributed/pull/5206
It scratches a slightly different itch but also proposes a busy_worker state on `Worker`

Resolving patch:
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -221,7 +221,10 @@ class Worker(ServerNode):
     * **tasks**: ``{key: TaskState}``
         The tasks currently executing on this worker (and any dependencies of those tasks)
     * **data_needed**: UniqueTaskHeap
-        The tasks which still require data in order to execute, prioritized as a heap
+        The tasks which still require data in order to execute and are in memory on at
+        least another worker, prioritized as a heap
+    * **data_needed_per_worker**: ``{worker: UniqueTaskHeap}``
+        Same as data_needed, split by worker
     * **ready**: [keys]
         Keys that are ready to run.  Stored in a LIFO stack
     * **constrained**: [keys]
@@ -236,14 +239,16 @@ class Worker(ServerNode):
         long-running clients.
     * **has_what**: ``{worker: {deps}}``
         The data that we care about that we think a worker has
-    * **pending_data_per_worker**: ``{worker: UniqueTaskHeap}``
-        The data on each worker that we still want, prioritized as a heap
     * **in_flight_tasks**: ``int``
         A count of the number of tasks that are coming to us in current
         peer-to-peer connections
     * **in_flight_workers**: ``{worker: {task}}``
         The workers from which we are currently gathering data and the
-        dependencies we expect from those connections
+        dependencies we expect from those connections. Workers in this dict won't be
+        asked for additional dependencies until the current query returns.
+    * **busy_workers**: ``{worker}``
+        Workers that recently returned a busy status. Workers in this set won't be
+        asked for additional dependencies for some time.
     * **comm_bytes**: ``int``
         The total number of bytes in flight
     * **threads**: ``{key: int}``
@@ -338,11 +343,12 @@ class Worker(ServerNode):
     tasks: dict[str, TaskState]
     waiting_for_data_count: int
     has_what: defaultdict[str, set[str]]  # {worker address: {ts.key, ...}
-    pending_data_per_worker: defaultdict[str, UniqueTaskHeap]
+    data_needed: UniqueTaskHeap
+    data_needed_per_worker: defaultdict[str, UniqueTaskHeap]
     nanny: Nanny | None
     _lock: threading.Lock
-    data_needed: UniqueTaskHeap
     in_flight_workers: dict[str, set[str]]  # {worker address: {ts.key, ...}}
+    busy_workers: set[str]
     total_out_connections: int
     total_in_connections: int
     comm_threshold_bytes: int
@@ -374,7 +380,6 @@ class Worker(ServerNode):
     incoming_count: int
     outgoing_count: int
     outgoing_current_count: int
-    repetitively_busy: int
     bandwidth: float
     latency: float
     profile_cycle_interval: float
@@ -483,13 +488,13 @@ def __init__(
         self.tasks = {}
         self.waiting_for_data_count = 0
         self.has_what = defaultdict(set)
-        self.pending_data_per_worker = defaultdict(UniqueTaskHeap)
+        self.data_needed = UniqueTaskHeap()
+        self.data_needed_per_worker = defaultdict(UniqueTaskHeap)
         self.nanny = nanny
         self._lock = threading.Lock()
 
-        self.data_needed = UniqueTaskHeap()
-
         self.in_flight_workers = {}
+        self.busy_workers = set()
         self.total_out_connections = dask.config.get(
             "distributed.worker.connections.outgoing"
         )
@@ -583,7 +588,6 @@ def __init__(
         self.outgoing_transfer_log = deque(maxlen=100000)
         self.outgoing_count = 0
         self.outgoing_current_count = 0
-        self.repetitively_busy = 0
         self.bandwidth = parse_bytes(dask.config.get("distributed.scheduler.bandwidth"))
         self.bandwidth_workers = defaultdict(
             lambda: (0, 0)
@@ -1018,13 +1022,14 @@ def _to_dict(self, *, exclude: Container[str] = ()) -> dict:
             "ready": self.ready,
             "constrained": self.constrained,
             "data_needed": list(self.data_needed),
-            "pending_data_per_worker": {
-                w: list(v) for w, v in self.pending_data_per_worker.items()
+            "data_needed_per_worker": {
+                w: list(v) for w, v in self.data_needed_per_worker.items()
             },
             "long_running": self.long_running,
             "executing_count": self.executing_count,
             "in_flight_tasks": self.in_flight_tasks,
             "in_flight_workers": self.in_flight_workers,
+            "busy_workers": self.busy_workers,
             "log": self.log,
             "stimulus_log": self.stimulus_log,
             "tasks": self.tasks,
@@ -1926,6 +1931,8 @@ def transition_missing_fetch(
         ts.state = "fetch"
         ts.done = False
         self.data_needed.push(ts)
+        for w in ts.who_has:
+            self.data_needed_per_worker[w].push(ts)
         return {}, []
 
     def transition_missing_released(
@@ -1961,11 +1968,11 @@ def transition_released_fetch(
         if self.validate:
             assert ts.state == "released"
             assert ts.priority is not None
-        for w in ts.who_has:
-            self.pending_data_per_worker[w].push(ts)
         ts.state = "fetch"
         ts.done = False
         self.data_needed.push(ts)
+        for w in ts.who_has:
+            self.data_needed_per_worker[w].push(ts)
         return {}, []
 
     def transition_generic_released(
@@ -2401,7 +2408,7 @@ def transition_flight_fetch(self, ts: TaskState, *, stimulus_id: str) -> RecsIns
         else:
             self.data_needed.push(ts)
             for w in ts.who_has:
-                self.pending_data_per_worker[w].push(ts)
+                self.data_needed_per_worker[w].push(ts)
         return recommendations, []
 
     def transition_flight_error(
@@ -2709,17 +2716,18 @@ def stimulus_story(
 
     def ensure_communicating(self) -> None:
         stimulus_id = f"ensure-communicating-{time()}"
-        skipped_worker_in_flight = []
+        skipped_worker_in_flight_or_busy = []
 
         while self.data_needed and (
             len(self.in_flight_workers) < self.total_out_connections
             or self.comm_nbytes < self.comm_threshold_bytes
         ):
             logger.debug(
-                "Ensure communicating. Pending: %d. Connections: %d/%d",
+                "Ensure communicating. Pending: %d. Connections: %d/%d. Busy: %d",
                 len(self.data_needed),
                 len(self.in_flight_workers),
                 self.total_out_connections,
+                len(self.busy_workers),
             )
 
             ts = self.data_needed.pop()
@@ -2729,20 +2737,20 @@ def ensure_communicating(self) -> None:
 
             if self.validate:
                 assert ts.who_has
+                assert self.address not in ts.who_has
 
-            workers = [w for w in ts.who_has if w not in self.in_flight_workers]
+            workers = [
+                w
+                for w in ts.who_has
+                if w not in self.in_flight_workers and w not in self.busy_workers
+            ]
             if not workers:
-                assert ts.priority is not None
-                skipped_worker_in_flight.append(ts)
+                skipped_worker_in_flight_or_busy.append(ts)
                 continue
 
             host = get_address_host(self.address)
             local = [w for w in workers if get_address_host(w) == host]
-            if local:
-                worker = random.choice(local)
-            else:
-                worker = random.choice(list(workers))
-            assert worker != self.address
+            worker = random.choice(local or workers)
 
             to_gather, total_nbytes = self.select_keys_for_gather(worker, ts.key)
 
@@ -2765,8 +2773,8 @@ def ensure_communicating(self) -> None:
                 stimulus_id=stimulus_id,
             )
 
-        for el in skipped_worker_in_flight:
-            self.data_needed.push(el)
+        for ts in skipped_worker_in_flight_or_busy:
+            self.data_needed.push(ts)
 
     def _get_task_finished_msg(self, ts: TaskState) -> TaskFinishedMsg:
         if ts.key not in self.data and ts.key not in self.actors:
@@ -2854,14 +2862,16 @@ def select_keys_for_gather(self, worker, dep):
         deps = {dep}
 
         total_bytes = self.tasks[dep].get_nbytes()
-        L = self.pending_data_per_worker[worker]
+        tasks = self.data_needed_per_worker[worker]
 
-        while L:
-            ts = L.pop()
+        while tasks:
+            ts = tasks.peek()
             if ts.state != "fetch":
+                tasks.pop()
                 continue
             if total_bytes + ts.get_nbytes() > self.target_message_size:
                 break
+            tasks.pop()
             deps.add(ts.key)
             total_bytes += ts.get_nbytes()
 
@@ -3051,7 +3061,7 @@ async def gather_dep(
         except OSError:
             logger.exception("Worker stream died during communication: %s", worker)
             has_what = self.has_what.pop(worker)
-            self.pending_data_per_worker.pop(worker)
+            self.data_needed_per_worker.pop(worker)
             self.log.append(
                 ("receive-dep-failed", worker, has_what, stimulus_id, time())
             )
@@ -3083,6 +3093,12 @@ async def gather_dep(
                 self.log.append(
                     ("busy-gather", worker, to_gather_keys, stimulus_id, time())
                 )
+                # Avoid hammering the worker. If there are multiple replicas
+                # available, immediately try fetching from a different worker.
+                self.busy_workers.add(worker)
+                self.io_loop.call_later(0.15, self._readd_busy_worker, worker)
+
+            refresh_who_has = set()
 
             for d in self.in_flight_workers.pop(worker):
                 ts = self.tasks[d]
@@ -3096,6 +3112,8 @@ async def gather_dep(
                     recommendations[ts] = ("memory", data[d])
                 elif busy:
                     recommendations[ts] = "fetch"
+                    if not ts.who_has - self.busy_workers:
+                        refresh_who_has.add(ts.key)
                 elif ts not in recommendations:
                     ts.who_has.discard(worker)
                     self.has_what[worker].discard(ts.key)
@@ -3110,17 +3128,21 @@ async def gather_dep(
             del data, response
             self.transitions(recommendations, stimulus_id=stimulus_id)
 
-            if not busy:
-                self.repetitively_busy = 0
-            else:
-                # Exponential backoff to avoid hammering scheduler/worker
-                self.repetitively_busy += 1
-                await asyncio.sleep(0.100 * 1.5**self.repetitively_busy)
-
-                await self.query_who_has(*to_gather_keys)
+            if refresh_who_has:
+                # All workers that hold known replicas of our tasks are busy.
+                # Try querying the scheduler for unknown ones.
+                who_has = await retry_operation(
+                    self.scheduler.who_has, keys=refresh_who_has
+                )
+                self.update_who_has(who_has)
 
             self.ensure_communicating()
 
+    @log_errors
+    def _readd_busy_worker(self, worker: str) -> None:
+        self.busy_workers.remove(worker)
+        self.ensure_communicating()
+
     @log_errors
     async def find_missing(self) -> None:
         if not self._missing_dep_flight:
@@ -3150,12 +3172,6 @@ async def find_missing(self) -> None:
             ].callback_time = self.periodic_callbacks["heartbeat"].callback_time
             self.ensure_communicating()
 
-    @log_errors
-    async def query_who_has(self, *deps: str) -> dict[str, Collection[str]]:
-        who_has = await retry_operation(self.scheduler.who_has, keys=deps)
-        self.update_who_has(who_has)
-        return who_has
-
     def update_who_has(self, who_has: dict[str, Collection[str]]) -> None:
         try:
             for dep, workers in who_has.items():
@@ -3176,7 +3192,7 @@ def update_who_has(self, who_has: dict[str, Collection[str]]) -> None:
 
                     for worker in workers:
                         self.has_what[worker].add(dep)
-                        self.pending_data_per_worker[worker].push(dep_ts)
+                        self.data_needed_per_worker[worker].push(dep_ts)
         except Exception as e:  # pragma: no cover
             logger.exception(e)
             if LOG_PDB:
@@ -3928,7 +3944,7 @@ def validate_task_fetch(self, ts):
 
         for w in ts.who_has:
             assert ts.key in self.has_what[w]
-            assert ts in self.pending_data_per_worker[w]
+            assert ts in self.data_needed_per_worker[w]
 
     def validate_task_missing(self, ts):
         assert ts.key not in self.data


Test patch:
diff --git a/distributed/tests/test_worker.py b/distributed/tests/test_worker.py
--- a/distributed/tests/test_worker.py
+++ b/distributed/tests/test_worker.py
@@ -1813,59 +1813,85 @@ async def test_gather_dep_one_worker_always_busy(c, s, a, b):
     # the data cannot be fetched
     # In the past it was important that there is more than one key on the
     # worker. This should be kept to avoid any edge case specific to one
-    f = c.submit(inc, 1, workers=[a.address])
-    g = c.submit(
-        inc,
-        2,
-        workers=[a.address],
-    )
+    f = c.submit(inc, 1, key="f", workers=[a.address])
+    g = c.submit(inc, 2, key="g", workers=[a.address])
+    await wait([f, g])
+    assert set(a.tasks) == {"f", "g"}
 
-    await f
-    await g
     # We will block A for any outgoing communication. This simulates an
     # overloaded worker which will always return "busy" for get_data requests,
     # effectively blocking H indefinitely
     a.outgoing_current_count = 10000000
-    assert f.key in a.tasks
-    assert g.key in a.tasks
-    # Ensure there are actually two distinct tasks and not some pure=True
-    # caching
-    assert f.key != g.key
-    h = c.submit(add, f, g, workers=[b.address])
 
-    fut = asyncio.wait_for(h, 0.1)
+    h = c.submit(add, f, g, key="h", workers=[b.address])
 
     while h.key not in b.tasks:
         await asyncio.sleep(0.01)
 
-    ts_h = b.tasks[h.key]
-    ts_f = b.tasks[f.key]
-    ts_g = b.tasks[g.key]
+    assert b.tasks[h.key].state == "waiting"
+    assert b.tasks[f.key].state in ("flight", "fetch")
+    assert b.tasks[g.key].state in ("flight", "fetch")
 
     with pytest.raises(asyncio.TimeoutError):
-        assert ts_h.state == "waiting"
-        assert ts_f.state in ["flight", "fetch"]
-        assert ts_g.state in ["flight", "fetch"]
-        await fut
-
-    # Ensure B wasn't lazy but tried at least once
-    assert b.repetitively_busy
-
-    x = await Worker(s.address, name="x")
-    # We "scatter" the data to another worker which is able to serve this data.
-    # In reality this could be another worker which fetched this dependency and
-    # got through to A or another worker executed the task using work stealing
-    # or any other. To avoid cross effects, we'll just put the data onto the
-    # worker ourselves
-    x.update_data(data={key: a.data[key] for key in [f.key, g.key]})
-
-    assert await h == 5
-
-    # Since we put the data onto the worker ourselves, the gather_dep might
-    # still be mid execution and we'll get a dangling task. Let it finish
-    # naturally
-    while any(["Worker.gather_dep" in str(t) for t in asyncio.all_tasks()]):
-        await asyncio.sleep(0.05)
+        await h.result(timeout=0.5)
+
+    story = b.story("busy-gather")
+    # 1 busy response straight away, followed by 1 retry every 150ms for 500ms.
+    # The requests for b and g are clustered together in single messages.
+    assert 3 <= len(story) <= 7
+
+    async with Worker(s.address, name="x") as x:
+        # We "scatter" the data to another worker which is able to serve this data.
+        # In reality this could be another worker which fetched this dependency and got
+        # through to A or another worker executed the task using work stealing or any
+        # other. To avoid cross effects, we'll just put the data onto the worker
+        # ourselves. Note that doing so means that B won't know about the existence of
+        # the extra replicas until it takes the initiative to invoke scheduler.who_has.
+        x.update_data({"f": 2, "g": 3})
+        assert await h == 5
+
+
+@pytest.mark.skipif(not LINUX, reason="Need 127.0.0.2 to mean localhost")
+@gen_cluster(
+    client=True,
+    nthreads=[("127.0.0.1", 1)] * 2 + [("127.0.0.2", 2)] * 10,  # type: ignore
+)
+async def test_gather_dep_local_workers_first(c, s, a, lw, *rws):
+    f = (
+        await c.scatter(
+            {"f": 1}, workers=[w.address for w in (lw,) + rws], broadcast=True
+        )
+    )["f"]
+    g = c.submit(inc, f, key="g", workers=[a.address])
+    assert await g == 2
+    assert_worker_story(a.story("f"), [("receive-dep", lw.address, {"f"})])
+
+
+@pytest.mark.skipif(not LINUX, reason="Need 127.0.0.2 to mean localhost")
+@gen_cluster(
+    client=True,
+    nthreads=[("127.0.0.2", 1)] + [("127.0.0.1", 1)] * 10,  # type: ignore
+)
+async def test_gather_dep_from_remote_workers_if_all_local_workers_are_busy(
+    c, s, rw, a, *lws
+):
+    f = (
+        await c.scatter(
+            {"f": 1}, workers=[w.address for w in (rw,) + lws], broadcast=True
+        )
+    )["f"]
+    for w in lws:
+        w.outgoing_current_count = 10000000
+
+    g = c.submit(inc, f, key="g", workers=[a.address])
+    assert await g == 2
+
+    # Tried fetching from each local worker exactly once before falling back to the
+    # remote worker
+    assert sorted(ev[1] for ev in a.story("busy-gather")) == sorted(
+        w.address for w in lws
+    )
+    assert_worker_story(a.story("receive-dep"), [("receive-dep", rw.address, {"f"})])
 
 
 @gen_cluster(client=True, nthreads=[("127.0.0.1", 0)])
@@ -3434,6 +3460,7 @@ async def test_Worker__to_dict(c, s, a):
         "executing_count",
         "in_flight_tasks",
         "in_flight_workers",
+        "busy_workers",
         "log",
         "stimulus_log",
         "tasks",
@@ -3442,7 +3469,7 @@ async def test_Worker__to_dict(c, s, a):
         "incoming_transfer_log",
         "outgoing_transfer_log",
         "data_needed",
-        "pending_data_per_worker",
+        "data_needed_per_worker",
         # attributes of WorkerMemoryManager
         "data",
         "max_spill",