Repository: dask/distributed

Issue:
Exposing workers' SystemMonitors to scheduler
<!-- Please do a quick search of existing issues to make sure that this has not been asked before. -->

I am working on [GPU] worker resource tracking, as discussed in #4494 - some suggestions that came up were:

- Showing the user the average/peak usage of a worker
- Plotting a time series of the worker(s) metrics, similar to the dashboard's System tab

In working on these suggestions in #4614 and #4627, I've noticed some limitations centered around the fact that the scheduler can only access worker resource metrics through a `WorkerState` object; this means that it only has access to *present* worker metrics, as past metrics are thrown out.

It is still possible to make a dashboard component with time series / aggregated charts of worker resources, but this requires the metrics from the `WorkerState` to be collected within the component itself, which has some issues:

- Reloading the component causes all this data to be thrown out, meaning all plots and charts will be reset.
- Because the performance report loads components *after* an operation is finished, these components will only be populated with worker data from after the profiling period.

The workers have their own `SystemMonitors` similar to the scheduler, which should be logging most of the metrics we're interested in,  though I don't think there is any way to access them from the scheduler object - can we make it so they are accessible?

```
>>> scheduler.worker_monitors
SortedDict({
    'tcp://127.0.0.1:34215': <SystemMonitor: cpu: 6 memory: 565 MB fds: 209>,
    'tcp://127.0.0.1:37983': <SystemMonitor: cpu: 6 memory: 565 MB fds: 209>, 
    'tcp://127.0.0.1:38587': <SystemMonitor: cpu: 6 memory: 565 MB fds: 209>, 
    'tcp://127.0.0.1:39987': <SystemMonitor: cpu: 6 memory: 565 MB fds: 209>, 
    'tcp://127.0.0.1:41055': <SystemMonitor: cpu: 6 memory: 565 MB fds: 209>, 
    'tcp://127.0.0.1:41683': <SystemMonitor: cpu: 6 memory: 565 MB fds: 209>, 
    'tcp://127.0.0.1:44779': <SystemMonitor: cpu: 6 memory: 565 MB fds: 209>, 
    'tcp://127.0.0.1:45823': <SystemMonitor: cpu: 6 memory: 565 MB fds: 209>
    })
```

This could allow for more advanced charting / plotting of worker resources.


Discussion:
Thanks for the detailed write-up @charlesbluca!  I'm not sure what is best here.  Some of my thoughts/concerns:

1) The nanny is not always around -- some users will run the dask worker with the `no-nanny` option.  But maybe that's ok, they won't be able to get this info unless the nanny is actively monitoring resource consumption
2) Reloading the component in the live dashboard is ok.  I believe this happens for the dashboard components as well like `/individual-memory-by-key`.   However, for the performance report we could, query all the nannies when the time comes to pull all the data -- there's probably a better way but simply `client.run(lambda dask_worker: dask_worker.monitor.gpu/cpu/memory`
3) Instead of the nannies, we can store metrics from the worker state in a configurable object like https://github.com/dask/distributed/blob/3b8b97e31ed10598531fb458b53db16683c92c47/distributed/scheduler.py#L3275-L3276

this would be stored during the [heartbeat_worker](https://github.com/dask/distributed/blob/3b8b97e31ed10598531fb458b53db16683c92c47/distributed/scheduler.py#L3604) call

@jacobtomlinson do you have any additional thoughts ? 
> they won't be able to get this info unless the nanny is actively monitoring resource consumption

I was under the impression that workers and nannies, both being subclasses of `ServerNode` and `Server`, should each have their own independent monitors - is it the nanny's monitor we're interested for keeping track of worker resource consumption?

> However, for the performance report we could, query all the nannies when the time comes to pull all the data

This sounds like a good solution that I'll try out. I guess the idea of making the worker monitors accessible to the scheduler is that it would make the process of generating these plots on the performance report / dashboard essentially congruous to how it's being done in #4561, but that's not really a big deal.

> we can store metrics from the worker state in a configurable object

I guess my concern is if this is redundant; if the `WorkerState` is already getting its metrics from some existing monitor, wouldn't storing this data in a configurable object within the scheduler be essentially recreating that original monitor?
> query all the nannies when the time comes to pull all the data

I wonder if we should make this an RPC call to retrieve metrics?
> I wonder if we should make this an RPC call to retrieve metrics?

That sounds like a good idea - a worker handler function could return the output of `worker.monitor.quantities` or optionally `worker.monitor.range_query(...)`, which could then maybe be gathered by something like `scheduler.worker_monitors(count=None)` - does that sound doable?

Resolving patch:
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -6500,6 +6500,18 @@ def get_events(self, comm=None, topic=None):
         else:
             return valmap(tuple, self.events)
 
+    async def get_worker_monitor_info(self, recent=False, starts=None):
+        parent: SchedulerState = cast(SchedulerState, self)
+        if starts is None:
+            starts = {}
+        results = await asyncio.gather(
+            *(
+                self.rpc(w).get_monitor_info(recent=recent, start=starts.get(w, 0))
+                for w in parent._workers_dv
+            )
+        )
+        return dict(zip(parent._workers_dv, results))
+
     ###########
     # Cleanup #
     ###########
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -681,6 +681,7 @@ def __init__(
             "actor_execute": self.actor_execute,
             "actor_attribute": self.actor_attribute,
             "plugin-add": self.plugin_add,
+            "get_monitor_info": self.get_monitor_info,
         }
 
         stream_handlers = {
@@ -1069,6 +1070,17 @@ async def gather(self, comm=None, who_has=None):
             self.update_data(data=result, report=False)
             return {"status": "OK"}
 
+    def get_monitor_info(self, comm=None, recent=False, start=0):
+        return dict(
+            range_query=(
+                self.monitor.recent()
+                if recent
+                else self.monitor.range_query(start=start)
+            ),
+            count=self.monitor.count,
+            last_time=self.monitor.last_time,
+        )
+
     #############
     # Lifecycle #
     #############


Test patch:
diff --git a/distributed/tests/test_scheduler.py b/distributed/tests/test_scheduler.py
--- a/distributed/tests/test_scheduler.py
+++ b/distributed/tests/test_scheduler.py
@@ -18,7 +18,7 @@
 from distributed import Client, Nanny, Worker, fire_and_forget, wait
 from distributed.client import wait
 from distributed.comm import Comm
-from distributed.compatibility import MACOS
+from distributed.compatibility import MACOS, WINDOWS
 from distributed.core import ConnectionPool, Status, connect, rpc
 from distributed.metrics import time
 from distributed.protocol.pickle import dumps
@@ -2201,3 +2201,15 @@ async def test_configurable_events_log_length(c, s, a, b):
     assert s.events["test"][0][1] == "dummy message 2"
     assert s.events["test"][1][1] == "dummy message 3"
     assert s.events["test"][2][1] == "dummy message 4"
+
+
+@gen_cluster()
+async def test_get_worker_monitor_info(s, a, b):
+    res = await s.get_worker_monitor_info()
+    ms = ["cpu", "time", "read_bytes", "write_bytes"]
+    if not WINDOWS:
+        ms += ["num_fds"]
+    for w in (a, b):
+        assert all(res[w.address]["range_query"][m] is not None for m in ms)
+        assert res[w.address]["count"] is not None
+        assert res[w.address]["last_time"] is not None