Repository: dask/distributed

Issue:
Observed worker network bandwidth chart
We have a couple of charts for inter-worker bandwidth that use our own metrics.  I don't trust these metrics that much.  We also keep per-worker metrics.  These are currently reported in the /workers tab with read/write at the end of the table.  I think that these are recorded with psutil and are sent in the worker heartbeats.  We might want to create another real-time bar-chart like occupancy/cpu/memory but for read/write traffic.  

This is slightly more complex because there are a couple of different values to show for each worker.  As options I could imagine ...

1.  Summing these up
2. Overplotting both with colors that make it clear which one is ahead
3. Plotting two skinnier rectangles per worker


Discussion:
I would actually love to see the measured bandwidths as a timeseries plot since they vary quite a bit over time, but I'm not sure how feasible this is
It's pretty feasible.  We would copy the /system charts.

Personally, I'd love both :)

On Mon, Jul 19, 2021 at 1:18 AM Florian Jetter ***@***.***>
wrote:

> I would actually love to see the measured bandwidths as a timeseries plot
> since they vary quite a bit over time, but I'm not sure how feasible this is
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/5090#issuecomment-882346862>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTEZMJJGTOFICRMENVTTYPNWZANCNFSM5ASI6J3Q>
> .
>

For context we already have this data rendered in the workers chart here:

![image](https://user-images.githubusercontent.com/306380/126406711-7fd4641e-60c4-44e8-9d8b-80a351c4f8d2.png)

(see the last two columns, the top row is the sum)

it would be good to have this same information rendered much in the same way we do for cpu/occupancy/nbytes charts for realtime status

![image](https://user-images.githubusercontent.com/306380/126406762-c1eab711-462d-4599-98a4-c8bd5f2394ee.png)

As well as a timeseries plot, much like how we do the /system charts

![image](https://user-images.githubusercontent.com/306380/126406823-fdecaf24-3534-4fe4-b9d9-c069bea3c9cb.png)


Personally, for the realtime chart I would just copy the occupancy/cpu/... charts, swap out CPU stats for network io stats (consulting the /workers chart to see where those are coming from) and then maybe that's it.

For the timeseries chart I would probably just plot the total bandwidth, rather than one value for every worker.  If we wanted to be clever we could have both a left and right axis for total and average bandwidth respectively.
After looking into this a bit more I have some questions/ comments. 
For the first plot (Nbytes looking one)
- Is the idea to have read and write per worker overlapped with different colors and some alpha? 
- If we are plotting memory, what is the connection with the cpu/occupancy/etc plots, don't we want a plot more like the Bytes stored per worker but that has read bytes and write bytes with network io stats? 

For example, if I have 2 workers I would have to horizontal bars that each one shows read_bytes and write_bytes(where this data comes from `ws.metrics.read_bytes` and `ws.metrics.write_bytes`)

For the times series chart, I'm not sure if I'm understanding this correctly: 
We want a time series per worker that has the total bandwidth and this is would be the sum of `ws.metrics.read_bytes` and `ws.metrics.write_bytes`, is this correct? 
> Is the idea to have read and write per worker overlapped with different colors and some alpha?

Naty and I discussed this briefly we decided to go with multi-bar plots like the following

![image](https://user-images.githubusercontent.com/306380/126552352-0d75f207-bd74-4a67-9490-82fdc7b42384.png)

Although I think in our situation we should skip the whitespace.  It would be good if these bars corresponded with the other horizontal bar charts so that people can compare CPU use against bandwidth.
I'm going to reopen this so that we continue to track the timeseries option (which I think would be valuable)
Whoops, thanks for re-opening 

Resolving patch:
diff --git a/distributed/dashboard/components/scheduler.py b/distributed/dashboard/components/scheduler.py
--- a/distributed/dashboard/components/scheduler.py
+++ b/distributed/dashboard/components/scheduler.py
@@ -737,9 +737,12 @@ def __init__(self, scheduler, **kwargs):
                     "y_write": [],
                     "x_read": [],
                     "x_write": [],
+                    "x_read_disk": [],
+                    "x_write_disk": [],
                 }
             )
-            self.root = figure(
+
+            self.bandwidth = figure(
                 title="Worker Network Bandwidth",
                 tools="",
                 id="bk-worker-net-bandwidth",
@@ -748,7 +751,7 @@ def __init__(self, scheduler, **kwargs):
             )
 
             # read_bytes
-            self.root.hbar(
+            self.bandwidth.hbar(
                 y="y_read",
                 right="x_read",
                 line_color=None,
@@ -760,7 +763,7 @@ def __init__(self, scheduler, **kwargs):
             )
 
             # write_bytes
-            self.root.hbar(
+            self.bandwidth.hbar(
                 y="y_write",
                 right="x_write",
                 line_color=None,
@@ -771,15 +774,55 @@ def __init__(self, scheduler, **kwargs):
                 source=self.source,
             )
 
-            self.root.axis[0].ticker = BasicTicker(**TICKS_1024)
-            self.root.xaxis[0].formatter = NumeralTickFormatter(format="0.0 b")
-            self.root.xaxis.major_label_orientation = XLABEL_ORIENTATION
-            self.root.xaxis.minor_tick_line_alpha = 0
-            self.root.x_range = Range1d(start=0)
-            self.root.yaxis.visible = False
-            self.root.ygrid.visible = False
-            self.root.toolbar_location = None
-            self.root.yaxis.visible = False
+            self.bandwidth.axis[0].ticker = BasicTicker(**TICKS_1024)
+            self.bandwidth.xaxis[0].formatter = NumeralTickFormatter(format="0.0 b")
+            self.bandwidth.xaxis.major_label_orientation = XLABEL_ORIENTATION
+            self.bandwidth.xaxis.minor_tick_line_alpha = 0
+            self.bandwidth.x_range = Range1d(start=0)
+            self.bandwidth.yaxis.visible = False
+            self.bandwidth.ygrid.visible = False
+            self.bandwidth.toolbar_location = None
+
+            self.disk = figure(
+                title="Workers Disk",
+                tools="",
+                id="bk-workers-disk",
+                name="worker_disk",
+                **kwargs,
+            )
+
+            # read_bytes_disk
+            self.disk.hbar(
+                y="y_read",
+                right="x_read_disk",
+                line_color=None,
+                left=0,
+                height=0.5,
+                fill_color="red",
+                legend_label="read",
+                source=self.source,
+            )
+
+            # write_bytes_disk
+            self.disk.hbar(
+                y="y_write",
+                right="x_write_disk",
+                line_color=None,
+                left=0,
+                height=0.5,
+                fill_color="blue",
+                legend_label="write",
+                source=self.source,
+            )
+
+            self.disk.axis[0].ticker = BasicTicker(**TICKS_1024)
+            self.disk.xaxis[0].formatter = NumeralTickFormatter(format="0.0 b")
+            self.disk.xaxis.major_label_orientation = XLABEL_ORIENTATION
+            self.disk.xaxis.minor_tick_line_alpha = 0
+            self.disk.x_range = Range1d(start=0)
+            self.disk.yaxis.visible = False
+            self.disk.ygrid.visible = False
+            self.disk.toolbar_location = None
 
     @without_property_validation
     def update(self):
@@ -792,28 +835,235 @@ def update(self):
 
             x_read = []
             x_write = []
+            x_read_disk = []
+            x_write_disk = []
 
             for ws in workers:
                 x_read.append(ws.metrics["read_bytes"])
                 x_write.append(ws.metrics["write_bytes"])
+                x_read_disk.append(ws.metrics["read_bytes_disk"])
+                x_write_disk.append(ws.metrics["write_bytes_disk"])
 
-            self.root.x_range.end = max(
-                max(x_read),
-                max(x_write),
-                100_000_000,
-                0.95 * self.root.x_range.end,
-            )
+            if self.scheduler.workers:
+                self.bandwidth.x_range.end = max(
+                    max(x_read),
+                    max(x_write),
+                    100_000_000,
+                    0.95 * self.bandwidth.x_range.end,
+                )
+
+                self.disk.x_range.end = max(
+                    max(x_read_disk),
+                    max(x_write_disk),
+                    100_000_000,
+                    0.95 * self.disk.x_range.end,
+                )
+            else:
+                self.bandwidth.x_range.end = 100_000_000
+                self.disk.x_range.end = 100_000_000
 
             result = {
                 "y_read": y_read,
                 "y_write": y_write,
                 "x_read": x_read,
                 "x_write": x_write,
+                "x_read_disk": x_read_disk,
+                "x_write_disk": x_write_disk,
             }
 
             update(self.source, result)
 
 
+class SystemTimeseries(DashboardComponent):
+    """Timeseries for worker network bandwidth, cpu, memory and disk.
+
+    bandwidth: plots the average of read_bytes and write_bytes for the workers
+    as a function of time.
+    cpu: plots the average of cpu for the workers as a function of time.
+    memory: plots the average of memory for the workers as a function of time.
+    disk: plots the average of read_bytes_disk and write_bytes_disk for the workers
+    as a function of time.
+
+    The metrics plotted come from the aggregation of
+    from ws.metrics["val"] for ws in scheduler.workers.values() divided by nuber of workers.
+    """
+
+    def __init__(self, scheduler, **kwargs):
+        with log_errors():
+            self.scheduler = scheduler
+            self.source = ColumnDataSource(
+                {
+                    "time": [],
+                    "read_bytes": [],
+                    "write_bytes": [],
+                    "cpu": [],
+                    "memory": [],
+                    "read_bytes_disk": [],
+                    "write_bytes_disk": [],
+                }
+            )
+
+            update(self.source, self.get_data())
+
+            x_range = DataRange1d(follow="end", follow_interval=20000, range_padding=0)
+            tools = "reset, xpan, xwheel_zoom"
+
+            self.bandwidth = figure(
+                title="Workers Network Bandwidth",
+                x_axis_type="datetime",
+                tools=tools,
+                x_range=x_range,
+                id="bk-worker-network-bandwidth-ts",
+                name="worker_network_bandwidth-timeseries",
+                **kwargs,
+            )
+
+            self.bandwidth.line(
+                source=self.source,
+                x="time",
+                y="read_bytes",
+                color="red",
+                legend_label="read (mean)",
+            )
+            self.bandwidth.line(
+                source=self.source,
+                x="time",
+                y="write_bytes",
+                color="blue",
+                legend_label="write (mean)",
+            )
+
+            self.bandwidth.legend.location = "top_left"
+            self.bandwidth.yaxis.axis_label = "bytes / second"
+            self.bandwidth.yaxis[0].formatter = NumeralTickFormatter(format="0.0b")
+            self.bandwidth.y_range.start = 0
+            self.bandwidth.yaxis.minor_tick_line_alpha = 0
+            self.bandwidth.xgrid.visible = False
+
+            self.cpu = figure(
+                title="Workers CPU",
+                x_axis_type="datetime",
+                tools=tools,
+                x_range=x_range,
+                id="bk-worker-cpu-ts",
+                name="worker_cpu-timeseries",
+                **kwargs,
+            )
+
+            self.cpu.line(
+                source=self.source,
+                x="time",
+                y="cpu",
+            )
+            self.cpu.yaxis.axis_label = "Utilization"
+            self.cpu.y_range.start = 0
+            self.cpu.yaxis.minor_tick_line_alpha = 0
+            self.cpu.xgrid.visible = False
+
+            self.memory = figure(
+                title="Workers Memory",
+                x_axis_type="datetime",
+                tools=tools,
+                x_range=x_range,
+                id="bk-worker-memory-ts",
+                name="worker_memory-timeseries",
+                **kwargs,
+            )
+
+            self.memory.line(
+                source=self.source,
+                x="time",
+                y="memory",
+            )
+            self.memory.yaxis.axis_label = "Bytes"
+            self.memory.yaxis[0].formatter = NumeralTickFormatter(format="0.0b")
+            self.memory.y_range.start = 0
+            self.memory.yaxis.minor_tick_line_alpha = 0
+            self.memory.xgrid.visible = False
+
+            self.disk = figure(
+                title="Workers Disk",
+                x_axis_type="datetime",
+                tools=tools,
+                x_range=x_range,
+                id="bk-worker-disk-ts",
+                name="worker_disk-timeseries",
+                **kwargs,
+            )
+
+            self.disk.line(
+                source=self.source,
+                x="time",
+                y="read_bytes_disk",
+                color="red",
+                legend_label="read (mean)",
+            )
+            self.disk.line(
+                source=self.source,
+                x="time",
+                y="write_bytes_disk",
+                color="blue",
+                legend_label="write (mean)",
+            )
+
+            self.disk.legend.location = "top_left"
+            self.disk.yaxis.axis_label = "bytes / second"
+            self.disk.yaxis[0].formatter = NumeralTickFormatter(format="0.0b")
+            self.disk.y_range.start = 0
+            self.disk.yaxis.minor_tick_line_alpha = 0
+            self.disk.xgrid.visible = False
+
+    def get_data(self):
+        workers = self.scheduler.workers.values()
+
+        read_bytes = 0
+        write_bytes = 0
+        cpu = 0
+        memory = 0
+        read_bytes_disk = 0
+        write_bytes_disk = 0
+        time = 0
+        for ws in workers:
+            read_bytes += ws.metrics["read_bytes"]
+            write_bytes += ws.metrics["write_bytes"]
+            cpu += ws.metrics["cpu"]
+            memory += ws.metrics["memory"]
+            read_bytes_disk += ws.metrics["read_bytes_disk"]
+            write_bytes_disk += ws.metrics["write_bytes_disk"]
+            time += ws.metrics["time"]
+
+        result = {
+            # use `or` to avoid ZeroDivision when no workers
+            "time": [time / (len(workers) or 1) * 1000],
+            "read_bytes": [read_bytes / (len(workers) or 1)],
+            "write_bytes": [write_bytes / (len(workers) or 1)],
+            "cpu": [cpu / (len(workers) or 1)],
+            "memory": [memory / (len(workers) or 1)],
+            "read_bytes_disk": [read_bytes_disk / (len(workers) or 1)],
+            "write_bytes_disk": [write_bytes_disk / (len(workers) or 1)],
+        }
+        return result
+
+    @without_property_validation
+    def update(self):
+        with log_errors():
+            self.source.stream(self.get_data(), 1000)
+
+            if self.scheduler.workers:
+                y_end_cpu = sum(
+                    ws.nthreads or 1 for ws in self.scheduler.workers.values()
+                ) / len(self.scheduler.workers.values())
+                y_end_mem = sum(
+                    ws.memory_limit for ws in self.scheduler.workers.values()
+                ) / len(self.scheduler.workers.values())
+            else:
+                y_end_cpu = 1
+                y_end_mem = 100_000_000
+
+            self.cpu.y_range.end = y_end_cpu * 100
+            self.memory.y_range.end = y_end_mem
+
+
 class ComputePerKey(DashboardComponent):
     """Bar chart showing time spend in action by key prefix"""
 
diff --git a/distributed/dashboard/scheduler.py b/distributed/dashboard/scheduler.py
--- a/distributed/dashboard/scheduler.py
+++ b/distributed/dashboard/scheduler.py
@@ -15,6 +15,7 @@
     MemoryByKey,
     Occupancy,
     SystemMonitor,
+    SystemTimeseries,
     TaskGraph,
     TaskGroupGraph,
     TaskProgress,
@@ -69,7 +70,22 @@
     "/individual-bandwidth-types": individual_doc(BandwidthTypes, 500),
     "/individual-bandwidth-workers": individual_doc(BandwidthWorkers, 500),
     "/individual-workers-network-bandwidth": individual_doc(
-        WorkerNetworkBandwidth, 500
+        WorkerNetworkBandwidth, 500, fig_attr="bandwidth"
+    ),
+    "/individual-workers-disk": individual_doc(
+        WorkerNetworkBandwidth, 500, fig_attr="disk"
+    ),
+    "/individual-workers-network-bandwidth-timeseries": individual_doc(
+        SystemTimeseries, 500, fig_attr="bandwidth"
+    ),
+    "/individual-workers-cpu-timeseries": individual_doc(
+        SystemTimeseries, 500, fig_attr="cpu"
+    ),
+    "/individual-workers-memory-timeseries": individual_doc(
+        SystemTimeseries, 500, fig_attr="memory"
+    ),
+    "/individual-workers-disk-timeseries": individual_doc(
+        SystemTimeseries, 500, fig_attr="disk"
     ),
     "/individual-memory-by-key": individual_doc(MemoryByKey, 500),
     "/individual-compute-time-per-key": individual_doc(ComputePerKey, 500),
diff --git a/distributed/system_monitor.py b/distributed/system_monitor.py
--- a/distributed/system_monitor.py
+++ b/distributed/system_monitor.py
@@ -35,6 +35,19 @@ def __init__(self, n=10000):
             self._last_io_counters = ioc
             self._collect_net_io_counters = True
 
+        try:
+            disk_ioc = psutil.disk_io_counters()
+        except Exception:
+            self._collect_disk_io_counters = False
+        else:
+            self.last_time_disk = time()
+            self.read_bytes_disk = deque(maxlen=n)
+            self.write_bytes_disk = deque(maxlen=n)
+            self.quantities["read_bytes_disk"] = self.read_bytes_disk
+            self.quantities["write_bytes_disk"] = self.write_bytes_disk
+            self._last_disk_io_counters = disk_ioc
+            self._collect_disk_io_counters = True
+
         if not WINDOWS:
             self.num_fds = deque(maxlen=n)
             self.quantities["num_fds"] = self.num_fds
@@ -86,6 +99,27 @@ def update(self):
                 result["read_bytes"] = read_bytes
                 result["write_bytes"] = write_bytes
 
+        if self._collect_disk_io_counters:
+            try:
+                disk_ioc = psutil.disk_io_counters()
+            except Exception:
+                pass
+            else:
+                last_disk = self._last_disk_io_counters
+                duration_disk = now - self.last_time_disk
+                read_bytes_disk = (disk_ioc.read_bytes - last_disk.read_bytes) / (
+                    duration_disk or 0.5
+                )
+                write_bytes_disk = (disk_ioc.write_bytes - last_disk.write_bytes) / (
+                    duration_disk or 0.5
+                )
+                self.last_time_disk = now
+                self._last_disk_io_counters = disk_ioc
+                self.read_bytes_disk.append(read_bytes_disk)
+                self.write_bytes_disk.append(write_bytes_disk)
+                result["read_bytes_disk"] = read_bytes_disk
+                result["write_bytes_disk"] = write_bytes_disk
+
         if not WINDOWS:
             num_fds = self.proc.num_fds()
             self.num_fds.append(num_fds)


Test patch:
diff --git a/distributed/dashboard/tests/test_scheduler_bokeh.py b/distributed/dashboard/tests/test_scheduler_bokeh.py
--- a/distributed/dashboard/tests/test_scheduler_bokeh.py
+++ b/distributed/dashboard/tests/test_scheduler_bokeh.py
@@ -31,6 +31,7 @@
     StealingEvents,
     StealingTimeSeries,
     SystemMonitor,
+    SystemTimeseries,
     TaskGraph,
     TaskGroupGraph,
     TaskProgress,
@@ -506,6 +507,78 @@ async def test_WorkerNetworkBandwidth_metrics(c, s, a, b):
     for idx, ws in enumerate(s.workers.values()):
         assert ws.metrics["read_bytes"] == nb.source.data["x_read"][idx]
         assert ws.metrics["write_bytes"] == nb.source.data["x_write"][idx]
+        assert ws.metrics["read_bytes_disk"] == nb.source.data["x_read_disk"][idx]
+        assert ws.metrics["write_bytes_disk"] == nb.source.data["x_write_disk"][idx]
+
+
+@gen_cluster(client=True)
+async def test_SystemTimeseries(c, s, a, b):
+    # Disable system monitor periodic callback to allow us to manually control
+    # when it is called below
+    a.periodic_callbacks["monitor"].stop()
+    b.periodic_callbacks["monitor"].stop()
+
+    # Update worker system monitors and send updated metrics to the scheduler
+    a.monitor.update()
+    b.monitor.update()
+    await asyncio.gather(a.heartbeat(), b.heartbeat())
+
+    systs = SystemTimeseries(s)
+    workers = s.workers.values()
+
+    assert all(len(v) == 1 for v in systs.source.data.values())
+    assert systs.source.data["read_bytes"][0] == sum(
+        [ws.metrics["read_bytes"] for ws in workers]
+    ) / len(workers)
+    assert systs.source.data["write_bytes"][0] == sum(
+        [ws.metrics["write_bytes"] for ws in workers]
+    ) / len(workers)
+    assert systs.source.data["cpu"][0] == sum(
+        [ws.metrics["cpu"] for ws in workers]
+    ) / len(workers)
+    assert systs.source.data["memory"][0] == sum(
+        [ws.metrics["memory"] for ws in workers]
+    ) / len(workers)
+    assert systs.source.data["read_bytes_disk"][0] == sum(
+        [ws.metrics["read_bytes_disk"] for ws in workers]
+    ) / len(workers)
+    assert systs.source.data["write_bytes_disk"][0] == sum(
+        [ws.metrics["write_bytes_disk"] for ws in workers]
+    ) / len(workers)
+    assert (
+        systs.source.data["time"][0]
+        == sum([ws.metrics["time"] for ws in workers]) / len(workers) * 1000
+    )
+
+    # Update worker system monitors and send updated metrics to the scheduler
+    a.monitor.update()
+    b.monitor.update()
+    await asyncio.gather(a.heartbeat(), b.heartbeat())
+    systs.update()
+
+    assert all(len(v) == 2 for v in systs.source.data.values())
+    assert systs.source.data["read_bytes"][1] == sum(
+        [ws.metrics["read_bytes"] for ws in workers]
+    ) / len(workers)
+    assert systs.source.data["write_bytes"][1] == sum(
+        [ws.metrics["write_bytes"] for ws in workers]
+    ) / len(workers)
+    assert systs.source.data["cpu"][1] == sum(
+        [ws.metrics["cpu"] for ws in workers]
+    ) / len(workers)
+    assert systs.source.data["memory"][1] == sum(
+        [ws.metrics["memory"] for ws in workers]
+    ) / len(workers)
+    assert systs.source.data["read_bytes_disk"][1] == sum(
+        [ws.metrics["read_bytes_disk"] for ws in workers]
+    ) / len(workers)
+    assert systs.source.data["write_bytes_disk"][1] == sum(
+        [ws.metrics["write_bytes_disk"] for ws in workers]
+    ) / len(workers)
+    assert (
+        systs.source.data["time"][1]
+        == sum([ws.metrics["time"] for ws in workers]) / len(workers) * 1000
+    )
 
 
 @gen_cluster(client=True)