Repository: dask/distributed

Issue:
Task stream dashboard plot broken on `main`? 
While following up on https://github.com/dask/distributed/issues/7663#issuecomment-1474055091, I noticed that the task stream plot on the dashboard is sporadically broken when using the `main` branch of `distributed` (it appears okay when using the latest release). 

With `bokeh=2.4.3` and `dask` + `distributed` `main` I ran the following little snippet locally

```python
import time

from dask import delayed
from dask.distributed import Client, wait, performance_report

@delayed
def inc(x):
    time.sleep(0.1)
    return x + 1

@delayed
def double(x):
    time.sleep(0.1)
    return 2 * x

@delayed
def add(x, y):
    time.sleep(0.1)
    return x + y

if __name__ == "__main__":
    with Client(n_workers=4, threads_per_worker=2, memory_limit="4 GiB") as client:
        print(f"{client.dashboard_link = }")
        with performance_report():
            data = list(range(500))
            output = []
            for x in data:
                a = inc(x)
                b = double(x)
                c = add(a, b)
                output.append(c)

            total = delayed(sum)(output)
            total = total.persist()
            wait(total)
            del total
```

While that script is running, I navigate to the dashboard link in my browser (have used Firefox, Safari, and Chrome). Frequently, maybe 60% of the time?, the task stream is broken and looks like this

![Screenshot 2023-03-17 at 2 45 36 PM](https://user-images.githubusercontent.com/11656932/226018932-56f74b84-0170-462b-8238-75e17d704304.png)

Sometimes it's not broken, but becomes broken if I refresh the dashboard page enough. 

Are others able to reproduce this same behavior?

Manually going through commits in the `distributed` git history since the last release, it looks like https://github.com/dask/distributed/pull/7586 is where things appear to start breaking. But looking through that PR I don't see anything that looks obviously related. 


Discussion:
cc @ncclementi, who is trying this out locally to see if she can reproduce, and @crusaderky, who might have thoughts on the https://github.com/dask/distributed/pull/7586 connection 
Confirming here, I was able to reproduce it consistently, at least the the 3 times I tried
![Screen Shot 2023-03-17 at 4 17 02 PM](https://user-images.githubusercontent.com/7526622/226026965-d7a219eb-d89b-475e-bbce-6d016be5dd7c.png)

Thanks for trying it out @ncclementi. @j-bennet mentioned offline she was also able to reproduce on her machine 

Resolving patch:
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -2059,7 +2059,7 @@ async def gather_dep(
         logger.debug("Request %d keys from %s", len(to_gather), worker)
 
         try:
-            with context_meter.meter("network") as m:
+            with context_meter.meter("network", func=time) as m:
                 response = await get_data_from_worker(
                     rpc=self.rpc, keys=to_gather, worker=worker, who=self.address
                 )


Test patch:
diff --git a/distributed/tests/test_worker.py b/distributed/tests/test_worker.py
--- a/distributed/tests/test_worker.py
+++ b/distributed/tests/test_worker.py
@@ -3755,3 +3755,25 @@ async def test_offload_getdata(c, s, a, b):
     x = c.submit(EnsureOffloaded, threading.get_ident(), key="x", workers=[a.address])
     y = c.submit(lambda x: None, x, key="y", workers=[b.address])
     await y
+
+
+@gen_cluster(client=True)
+async def test_startstops(c, s, a, b):
+    t0 = time()
+    x = c.submit(inc, 1, key="x", workers=[a.address])
+    y = c.submit(inc, x, key="y", workers=[b.address])
+    await wait(y)
+    t1 = time()
+    ss = b.state.tasks["y"].startstops
+    assert len(ss) == 2
+    assert ss[0]["action"] == "transfer"
+    assert ss[0]["source"] == a.address
+    assert ss[1]["action"] == "compute"
+    assert (
+        t0 + b.scheduler_delay
+        < ss[0]["start"]
+        < ss[0]["stop"]
+        < ss[1]["start"]
+        < ss[1]["stop"]
+        < t1 + b.scheduler_delay
+    )