Repository: dask/distributed

Issue:
Observed worker network bandwidth chart
We have a couple of charts for inter-worker bandwidth that use our own metrics.  I don't trust these metrics that much.  We also keep per-worker metrics.  These are currently reported in the /workers tab with read/write at the end of the table.  I think that these are recorded with psutil and are sent in the worker heartbeats.  We might want to create another real-time bar-chart like occupancy/cpu/memory but for read/write traffic.  

This is slightly more complex because there are a couple of different values to show for each worker.  As options I could imagine ...

1.  Summing these up
2. Overplotting both with colors that make it clear which one is ahead
3. Plotting two skinnier rectangles per worker


Discussion:
I would actually love to see the measured bandwidths as a timeseries plot since they vary quite a bit over time, but I'm not sure how feasible this is
It's pretty feasible.  We would copy the /system charts.

Personally, I'd love both :)

On Mon, Jul 19, 2021 at 1:18 AM Florian Jetter ***@***.***>
wrote:

> I would actually love to see the measured bandwidths as a timeseries plot
> since they vary quite a bit over time, but I'm not sure how feasible this is
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/5090#issuecomment-882346862>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTEZMJJGTOFICRMENVTTYPNWZANCNFSM5ASI6J3Q>
> .
>

For context we already have this data rendered in the workers chart here:

![image](https://user-images.githubusercontent.com/306380/126406711-7fd4641e-60c4-44e8-9d8b-80a351c4f8d2.png)

(see the last two columns, the top row is the sum)

it would be good to have this same information rendered much in the same way we do for cpu/occupancy/nbytes charts for realtime status

![image](https://user-images.githubusercontent.com/306380/126406762-c1eab711-462d-4599-98a4-c8bd5f2394ee.png)

As well as a timeseries plot, much like how we do the /system charts

![image](https://user-images.githubusercontent.com/306380/126406823-fdecaf24-3534-4fe4-b9d9-c069bea3c9cb.png)


Personally, for the realtime chart I would just copy the occupancy/cpu/... charts, swap out CPU stats for network io stats (consulting the /workers chart to see where those are coming from) and then maybe that's it.

For the timeseries chart I would probably just plot the total bandwidth, rather than one value for every worker.  If we wanted to be clever we could have both a left and right axis for total and average bandwidth respectively.
After looking into this a bit more I have some questions/ comments. 
For the first plot (Nbytes looking one)
- Is the idea to have read and write per worker overlapped with different colors and some alpha? 
- If we are plotting memory, what is the connection with the cpu/occupancy/etc plots, don't we want a plot more like the Bytes stored per worker but that has read bytes and write bytes with network io stats? 

For example, if I have 2 workers I would have to horizontal bars that each one shows read_bytes and write_bytes(where this data comes from `ws.metrics.read_bytes` and `ws.metrics.write_bytes`)

For the times series chart, I'm not sure if I'm understanding this correctly: 
We want a time series per worker that has the total bandwidth and this is would be the sum of `ws.metrics.read_bytes` and `ws.metrics.write_bytes`, is this correct? 
> Is the idea to have read and write per worker overlapped with different colors and some alpha?

Naty and I discussed this briefly we decided to go with multi-bar plots like the following

![image](https://user-images.githubusercontent.com/306380/126552352-0d75f207-bd74-4a67-9490-82fdc7b42384.png)

Although I think in our situation we should skip the whitespace.  It would be good if these bars corresponded with the other horizontal bar charts so that people can compare CPU use against bandwidth.

Resolving patch:
diff --git a/distributed/dashboard/components/scheduler.py b/distributed/dashboard/components/scheduler.py
--- a/distributed/dashboard/components/scheduler.py
+++ b/distributed/dashboard/components/scheduler.py
@@ -147,7 +147,7 @@ def __init__(self, scheduler, **kwargs):
     @without_property_validation
     def update(self):
         with log_errors():
-            workers = list(self.scheduler.workers.values())
+            workers = self.scheduler.workers.values()
 
             y = list(range(len(workers)))
             occupancy = [ws.occupancy for ws in workers]
@@ -735,6 +735,98 @@ def name(address):
             update(self.source, result)
 
 
+class WorkerNetworkBandwidth(DashboardComponent):
+    """Worker network bandwidth chart
+
+    Plots horizontal bars with the read_bytes and write_bytes worker state
+    """
+
+    def __init__(self, scheduler, **kwargs):
+        with log_errors():
+            self.scheduler = scheduler
+            self.source = ColumnDataSource(
+                {
+                    "y_read": [],
+                    "y_write": [],
+                    "x_read": [],
+                    "x_write": [],
+                }
+            )
+            self.root = figure(
+                title="Worker Network Bandwidth",
+                tools="",
+                id="bk-worker-net-bandwidth",
+                name="worker_network_bandwidth",
+                **kwargs,
+            )
+
+            # read_bytes
+            self.root.hbar(
+                y="y_read",
+                right="x_read",
+                line_color=None,
+                left=0,
+                height=0.5,
+                fill_color="red",
+                legend_label="read",
+                source=self.source,
+            )
+
+            # write_bytes
+            self.root.hbar(
+                y="y_write",
+                right="x_write",
+                line_color=None,
+                left=0,
+                height=0.5,
+                fill_color="blue",
+                legend_label="write",
+                source=self.source,
+            )
+
+            self.root.axis[0].ticker = BasicTicker(**TICKS_1024)
+            self.root.xaxis[0].formatter = NumeralTickFormatter(format="0.0 b")
+            self.root.xaxis.major_label_orientation = XLABEL_ORIENTATION
+            self.root.xaxis.minor_tick_line_alpha = 0
+            self.root.x_range = Range1d(start=0)
+            self.root.yaxis.visible = False
+            self.root.ygrid.visible = False
+            self.root.toolbar_location = None
+            self.root.yaxis.visible = False
+
+    @without_property_validation
+    def update(self):
+        with log_errors():
+            workers = self.scheduler.workers.values()
+
+            h = 0.1
+            y_read = [i + 0.75 + i * h for i in range(len(workers))]
+            y_write = [i + 0.25 + i * h for i in range(len(workers))]
+
+            x_read = []
+            x_write = []
+
+            for ws in workers:
+                x_read.append(ws.metrics["read_bytes"])
+                x_write.append(ws.metrics["write_bytes"])
+
+            self.root.x_range.end = max(
+                max(x_read),
+                max(x_write),
+                100_000_000,
+                0.95 * self.root.x_range.end,
+            )
+
+            result = {
+                "y_read": y_read,
+                "y_write": y_write,
+                "x_read": x_read,
+                "x_write": x_write,
+            }
+
+            update(self.source, result)
+
+
 class ComputePerKey(DashboardComponent):
     """Bar chart showing time spend in action by key prefix"""
 
@@ -1167,7 +1259,7 @@ def __init__(self, scheduler, width=600, **kwargs):
     @without_property_validation
     def update(self):
         with log_errors():
-            workers = list(self.scheduler.workers.values())
+            workers = self.scheduler.workers.values()
             now = time()
             if not any(ws.processing for ws in workers) and now < self.last + 1:
                 return
diff --git a/distributed/dashboard/components/shared.py b/distributed/dashboard/components/shared.py
--- a/distributed/dashboard/components/shared.py
+++ b/distributed/dashboard/components/shared.py
@@ -532,8 +532,20 @@ def __init__(self, worker, height=150, last_count=None, **kwargs):
             tools=tools,
             **kwargs,
         )
-        self.bandwidth.line(source=self.source, x="time", y="read_bytes", color="red")
-        self.bandwidth.line(source=self.source, x="time", y="write_bytes", color="blue")
+        self.bandwidth.line(
+            source=self.source,
+            x="time",
+            y="read_bytes",
+            color="red",
+            legend_label="read",
+        )
+        self.bandwidth.line(
+            source=self.source,
+            x="time",
+            y="write_bytes",
+            color="blue",
+            legend_label="write",
+        )
         self.bandwidth.yaxis.axis_label = "Bytes / second"
 
         # self.cpu.yaxis[0].formatter = NumeralTickFormatter(format='0%')
diff --git a/distributed/dashboard/scheduler.py b/distributed/dashboard/scheduler.py
--- a/distributed/dashboard/scheduler.py
+++ b/distributed/dashboard/scheduler.py
@@ -24,6 +24,7 @@
     TaskGroupGraph,
     TaskProgress,
     TaskStream,
+    WorkerNetworkBandwidth,
     WorkersMemory,
     WorkerTable,
     events_doc,
@@ -72,6 +73,9 @@
     "/individual-workers": individual_doc(WorkerTable, 500),
     "/individual-bandwidth-types": individual_doc(BandwidthTypes, 500),
     "/individual-bandwidth-workers": individual_doc(BandwidthWorkers, 500),
+    "/individual-workers-network-bandwidth": individual_doc(
+        WorkerNetworkBandwidth, 500
+    ),
     "/individual-memory-by-key": individual_doc(MemoryByKey, 500),
     "/individual-compute-time-per-key": individual_doc(ComputePerKey, 500),
     "/individual-aggregate-time-per-action": individual_doc(AggregateAction, 500),


Test patch:
diff --git a/distributed/dashboard/tests/test_scheduler_bokeh.py b/distributed/dashboard/tests/test_scheduler_bokeh.py
--- a/distributed/dashboard/tests/test_scheduler_bokeh.py
+++ b/distributed/dashboard/tests/test_scheduler_bokeh.py
@@ -35,6 +35,7 @@
     TaskGroupGraph,
     TaskProgress,
     TaskStream,
+    WorkerNetworkBandwidth,
     WorkersMemory,
     WorkersMemoryHistogram,
     WorkerTable,
@@ -476,6 +477,37 @@ async def test_WorkerTable_with_memory_limit_as_0(c, s, a, b):
     assert wt.source.data["memory_percent"][0] == ""
 
 
+@gen_cluster(client=True)
+async def test_WorkerNetworkBandwidth(c, s, a, b):
+    nb = WorkerNetworkBandwidth(s)
+    nb.update()
+
+    assert all(len(v) == 2 for v in nb.source.data.values())
+
+    assert nb.source.data["y_read"] == [0.75, 1.85]
+    assert nb.source.data["y_write"] == [0.25, 1.35]
+
+
+@gen_cluster(client=True)
+async def test_WorkerNetworkBandwidth_metrics(c, s, a, b):
+    # Disable system monitor periodic callback to allow us to manually control
+    # when it is called below
+    a.periodic_callbacks["monitor"].stop()
+    b.periodic_callbacks["monitor"].stop()
+
+    # Update worker system monitors and send updated metrics to the scheduler
+    a.monitor.update()
+    b.monitor.update()
+    await asyncio.gather(a.heartbeat(), b.heartbeat())
+
+    nb = WorkerNetworkBandwidth(s)
+    nb.update()
+
+    for idx, ws in enumerate(s.workers.values()):
+        assert ws.metrics["read_bytes"] == nb.source.data["x_read"][idx]
+        assert ws.metrics["write_bytes"] == nb.source.data["x_write"][idx]
+
+
 @gen_cluster(client=True)
 async def test_TaskGraph(c, s, a, b):
     gp = TaskGraph(s)