Repository: dask/distributed

Issue:
Add bar chart for worker occupancy
We currently have bar charts for worker memory and the number of tasks processing.  A similar chart to number of tasks processing would be worker occupancy, which records the sum of the amount of compute time of all tasks currently residing on the worker.  This would be exactly like the number of tasks processing chart, but use `ws.occupancy` rather than `len(ws.processing)`

I would have used this today when trying to diagnose work stealing issues.  

The relevant code for the number of processing tasks is here.  It's made slightly complex because the code for this chart is currently merged with the memory chart.  It might make sense to break these apart.

https://github.com/dask/distributed/blob/9d4f0bf2fc804f955a869febd3b51423c4382908/distributed/dashboard/components/scheduler.py#L1017-L1139


Discussion:
Good idea -- a worker occupancy plot would be useful. cc'ing @charlesbluca as you may find this interesting
It turns out we already have an occupancy bar chart for each worker

https://github.com/dask/distributed/blob/7d0f0102ddfec1195257c82889ff695d32d61fbc/distributed/dashboard/components/scheduler.py#L95-L96

which is used in our work stealing dashboard

https://github.com/dask/distributed/blob/7d0f0102ddfec1195257c82889ff695d32d61fbc/distributed/dashboard/components/scheduler.py#L2189

Creating a standalone occupancy plot would involve adding a new dashboard endpoint here

https://github.com/dask/distributed/blob/d5fc324bdef22f19d77c3d36e63a3778ceaac0b0/distributed/dashboard/scheduler.py#L70-L104

which uses the existing `Occupancy` dashboard component

cc @ncclementi
Ah, grand

On Tue, Jun 22, 2021 at 2:21 PM James Bourbeau ***@***.***>
wrote:

> It turns out we already have an occupancy bar chart for each worker
>
>
> https://github.com/dask/distributed/blob/7d0f0102ddfec1195257c82889ff695d32d61fbc/distributed/dashboard/components/scheduler.py#L95-L96
>
> which is used in our work stealing dashboard
>
>
> https://github.com/dask/distributed/blob/7d0f0102ddfec1195257c82889ff695d32d61fbc/distributed/dashboard/components/scheduler.py#L2189
>
> Creating a standalone occupancy plot would involve adding a new dashboard
> endpoint here
>
>
> https://github.com/dask/distributed/blob/d5fc324bdef22f19d77c3d36e63a3778ceaac0b0/distributed/dashboard/scheduler.py#L70-L104
>
> which uses the existing Occupancy dashboard component
>
> cc @ncclementi <https://github.com/ncclementi>
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/4870#issuecomment-866266078>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTG5F4SZ6QYIGHXORXTTUDPDLANCNFSM456YRPNQ>
> .
>

@jrbourbeau 
The occupancy chart you are pointing to has the `width` and `x` of the bars varying on time. Is this what we want, or we want a different representation? 

Since occupancy is in units of time

https://github.com/dask/distributed/blob/6d15b013ebe59f07e6f563cf73a4e8a7268d260c/distributed/scheduler.py#L442-L446

I would expect the units of the bars (i.e. the x-axis) to be time and for that to be periodically updated. Is that what you mean by varying on time, or do you mean something else? 
I think that this chart should probably look like the charts on the status
page, with number of processing tasks and the amount of memory used.  Both
are horizontally aligned bar-charts that update sub-second-ish.  We
probably want all three of these plots to be somewhat consistent.

On Tue, Jun 22, 2021 at 4:50 PM James Bourbeau ***@***.***>
wrote:

> Since occupancy is in units of time
>
>
> https://github.com/dask/distributed/blob/6d15b013ebe59f07e6f563cf73a4e8a7268d260c/distributed/scheduler.py#L442-L446
>
> I would expect the units of the bars (i.e. the x-axis) to be time and for
> that to be periodically updated. Is that what you mean by varying on time,
> or do you mean something else?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/4870#issuecomment-866360707>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTD2GXKVMKJM3G2LYKDTUEATNANCNFSM456YRPNQ>
> .
>

Yeah, I agree those charts should be consistent. @ncclementi if the existing `Occupancy` chart isn't consistent with the other plots @mrocklin mentioned, it'd be good to update it so it is 
It looks like the occupancy chart is currently in line with what we would want:

![image](https://user-images.githubusercontent.com/20627856/123009038-fb001900-d389-11eb-94ac-d7dc572f38a1.png)

Is there anything we can add here to make this more useful, or is a standalone page in the vein of `/individual-*` good here?
> Is there anything we can add here to make this more useful, or is a standalone page in the vein of /individual-* good here?

That's a good question. For now I think an `/individual-occupancy` plot would be a good start. I think @ncclementi is currently working on adding this

Resolving patch:
diff --git a/distributed/dashboard/scheduler.py b/distributed/dashboard/scheduler.py
--- a/distributed/dashboard/scheduler.py
+++ b/distributed/dashboard/scheduler.py
@@ -19,6 +19,7 @@
     MemoryByKey,
     NBytes,
     NBytesCluster,
+    Occupancy,
     SystemMonitor,
     TaskGraph,
     TaskProgress,
@@ -90,6 +91,7 @@ def connect(application, http_server, scheduler, prefix=""):
     "/individual-nprocessing": individual_doc(
         CurrentLoad, 100, fig_attr="processing_figure"
     ),
+    "/individual-occupancy": individual_doc(Occupancy, 100),
     "/individual-workers": individual_doc(WorkerTable, 500),
     "/individual-bandwidth-types": individual_doc(BandwidthTypes, 500),
     "/individual-bandwidth-workers": individual_doc(BandwidthWorkers, 500),


Test patch: