Repository: dask/distributed

Issue:
Repeatedly use the same worker on first task
Apparently in a quiet cluster we end up sending new tasks to the same worker repeatedly.  So probably the following test would fail:

```python
@gen_cluster(client=True)
def test_round_robin(c, s, a, b):
    await c.submit(inc, 1)
    await c.submit(inc, 2)
    await c.submit(inc, 3)
    assert a.transition_log and b.transition_log
```

This outcome is determined here:

https://github.com/dask/distributed/blob/bf9ddab3fd67c33b0d87da4f18bcdc7e47262393/distributed/scheduler.py#L2122-L2136

It looks like when there are no dependencies and we have only a few workers we currently choose the worker with minimum occupancy.  In a quiet cluster all workers have zero occupancy, so probably we're getting whatever Python uses to break a tie in this setting.

In the case where the occupancy is zero we might do something like a round-robin (this is done just below in the case where we have greater than 20 workers) among the worker pool for as long as that worker has zero occupancy.

Reported in conversation by @crusaderky 


Discussion:
Tinkering with this. The test indeed fails ;) but perhaps the reason is not what you were trying to trigger? I'm seeing `transition_log` is not an attribute of `distributed.worker.Worker`. Searching through the repo I only see instances of `s.transition_log` scattered in a bunch of tests.

Is your `assert` trying to test that both workers `a` and `b` have some evidence of usage? I might be barking up the wrong tree.
> Is your assert trying to test that both workers a and b have some evidence of usage?

Yes, this was exactly my intent.  The issue was authored in haste, my apologies for the messiness here.  Looking again at worker.py it looks like we might look at `Worker.log` instead.  
Thanks for the clarification!

Resolving patch:
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -2141,11 +2141,24 @@ def decide_worker(self, ts: TaskState) -> WorkerState:
         else:
             worker_pool = self._idle or self._workers
             worker_pool_dv = cast(dict, worker_pool)
+            wp_vals = worker_pool.values()
             n_workers: Py_ssize_t = len(worker_pool_dv)
             if n_workers < 20:  # smart but linear in small case
-                ws = min(worker_pool.values(), key=operator.attrgetter("occupancy"))
+                ws = min(wp_vals, key=operator.attrgetter("occupancy"))
+                if ws._occupancy == 0:
+                    # special case to use round-robin; linear search
+                    # for next worker with zero occupancy (or just
+                    # land back where we started).
+                    wp_i: WorkerState
+                    start: Py_ssize_t = self._n_tasks % n_workers
+                    i: Py_ssize_t
+                    for i in range(n_workers):
+                        wp_i = wp_vals[(i + start) % n_workers]
+                        if wp_i._occupancy == 0:
+                            ws = wp_i
+                            break
             else:  # dumb but fast in large case
-                ws = worker_pool.values()[self._n_tasks % n_workers]
+                ws = wp_vals[self._n_tasks % n_workers]
 
         if self._validate:
             assert ws is None or isinstance(ws, WorkerState), (


Test patch:
diff --git a/distributed/tests/test_client_executor.py b/distributed/tests/test_client_executor.py
--- a/distributed/tests/test_client_executor.py
+++ b/distributed/tests/test_client_executor.py
@@ -210,26 +210,24 @@ def test_unsupported_arguments(client, s, a, b):
 def test_retries(client):
     args = [ZeroDivisionError("one"), ZeroDivisionError("two"), 42]
 
-    with client.get_executor(retries=3, pure=False) as e:
+    with client.get_executor(retries=5, pure=False) as e:
         future = e.submit(varying(args))
         assert future.result() == 42
 
-    with client.get_executor(retries=2) as e:
+    with client.get_executor(retries=4) as e:
         future = e.submit(varying(args))
         result = future.result()
         assert result == 42
 
-    with client.get_executor(retries=1) as e:
+    with client.get_executor(retries=2) as e:
         future = e.submit(varying(args))
-        with pytest.raises(ZeroDivisionError) as exc_info:
+        with pytest.raises(ZeroDivisionError, match="two"):
             res = future.result()
-        exc_info.match("two")
 
     with client.get_executor(retries=0) as e:
         future = e.submit(varying(args))
-        with pytest.raises(ZeroDivisionError) as exc_info:
+        with pytest.raises(ZeroDivisionError, match="one"):
             res = future.result()
-        exc_info.match("one")
 
 
 def test_shutdown(loop):
diff --git a/distributed/tests/test_scheduler.py b/distributed/tests/test_scheduler.py
--- a/distributed/tests/test_scheduler.py
+++ b/distributed/tests/test_scheduler.py
@@ -2234,3 +2234,11 @@ async def test_get_worker_monitor_info(s, a, b):
         assert all(res[w.address]["range_query"][m] is not None for m in ms)
         assert res[w.address]["count"] is not None
         assert res[w.address]["last_time"] is not None
+
+
+@gen_cluster(client=True)
+async def test_quiet_cluster_round_robin(c, s, a, b):
+    await c.submit(inc, 1)
+    await c.submit(inc, 2)
+    await c.submit(inc, 3)
+    assert a.log and b.log