Repository: dask/distributed

Issue:
nbytes flicker prevention causes high scheduler CPU
Unfortunately, I think the slowdown we were concerned about with https://github.com/dask/distributed/pull/4997#discussion_r661831758 turns out to be significant. Scheduler CPU, when totally idle with no workers, is now around 20%

1. Run `dask-scheduler`
2. Go directly to the system plot (http://localhost:8787/system)
3. Now click on the status plot (http://localhost:8787/status)
4. Now go back to the system plot; CPU is much higher

Before #4997 (c6a12d843f91efcefd3bd020ed0eb5165a1ab808)
![before](https://user-images.githubusercontent.com/3309802/124841000-b30bf500-df38-11eb-9edc-782547d481f2.gif)
#4997 (8b79bd95d0c5a681f6d0543eb20d645dc1182ec1)
![after](https://user-images.githubusercontent.com/3309802/124841005-b43d2200-df38-11eb-8a5c-e6fb77915608.gif)

Here's a py-spy profile: https://www.speedscope.app/#profileURL=https%3A%2F%2Fgistcdn.githack.com%2Fgjoseph92%2F689bfa3543a47708d0b0e3d1e236b150%2Fraw%2Fad939f0cb54d5f592a110fa34aa22b51e13b9a9e%2Fidle-main.json. Make sure to go to `MainThread` with the thread selector, and check out the left heavy view. We're spending most of the time on
https://github.com/dask/distributed/blob/8b79bd95d0c5a681f6d0543eb20d645dc1182ec1/distributed/dashboard/components/scheduler.py#L509 and https://github.com/dask/distributed/blob/8b79bd95d0c5a681f6d0543eb20d645dc1182ec1/distributed/dashboard/components/scheduler.py#L345 constructing the `DataRange1d`.

FWIW I did a `git bisect` to be extra sure that it was actually this change, and indeed it was :(

cc @jrbourbeau @ncclementi 



Discussion:
Mmmh, we new it was slower but we didn't think it would affect that much (you can see the discussion in the PR [here](https://github.com/dask/distributed/pull/4997#discussion_r661831758)). Unfortunately, that was the only thing that worked to stop the flickering, at least that I was aware. We might need to go back to the flickering and try to find a different solution. 
@jrbourbeau What do you think? 
Thanks for flagging this @gjoseph92. In light of the release tomorrow (xref https://github.com/dask/community/issues/169), I think reverting makes sense if we don't come up with a different solution before then. See https://github.com/dask/distributed/pull/5041 for a PR which reverts the changes in https://github.com/dask/distributed/pull/4997
Is it because `DataRange1d` is building out an array of values? If so, wonder if something like `range` that is more lazy would avoid that issue
From the profile Gabe posted, creating a `DataRange1d` instance results in functions inside bokeh like `_invalidate_all_models`, `_recompute_all_models`, etc. being run which appear to take a long time (relative to other code which is run in the dashboard). My guess is this is also where the increased CPU load is coming from. 

<img width="1552" alt="Screen Shot 2021-07-08 at 4 17 13 PM" src="https://user-images.githubusercontent.com/11656932/124992052-17e25080-e008-11eb-9fc4-13ce431f3919.png">

> Is it because DataRange1d is building out an array of values? If so, wonder if something like range that is more lazy would avoid that issue

I might be misinterrepting your comment, but locally I tried the naive approach of replacing one of the `DataRange1d` instances with with `self.root.x_range = range(0, x_end)`, but got an error from bokeh

```python
TypeError: Object of type range is not JSON serializable
```

That's probably why we were explicitly setting `self.root.x_range.start` and `self.root.x_range.end` before
cc @bryevdv (in case you have suggestions here)
If you mean to use dumb (non-autoranging) ranges, then this
```
self.root.x_range = range(0, x_end)
```
Should be 
```
self.root.x_range = Range1d(0, x_end)
```
The `range` function is Python's builtin range generator, not one of Bokeh's models.

Otherwise, offhand I would say `_invalidate_all_models` will occur when models in the document are added, deleted, etc. It's not really possible to speculate without knowing what `func` does. The general guidance to set up as much as possible up front and make the smallest changes is advised, e.g. as an example: if the choice is between adding every glyph up front and toggling their visibility as needed later (cheap, lightweight), or actually deleting and adding entire glyph objects later (expensive, heavyweight), choose the first option if possible. 

Resolving patch:
diff --git a/distributed/dashboard/components/scheduler.py b/distributed/dashboard/components/scheduler.py
--- a/distributed/dashboard/components/scheduler.py
+++ b/distributed/dashboard/components/scheduler.py
@@ -282,6 +282,7 @@ def __init__(self, scheduler, width=600, **kwargs):
             self.root.xaxis[0].formatter = NumeralTickFormatter(format="0.0 b")
             self.root.xaxis.major_label_orientation = XLABEL_ORIENTATION
             self.root.xaxis.minor_tick_line_alpha = 0
+            self.root.x_range = Range1d(start=0)
             self.root.yaxis.visible = False
             self.root.ygrid.visible = False
 
@@ -342,7 +343,7 @@ def update(self):
             }
 
             x_end = max(limit, meminfo.process + meminfo.managed_spilled)
-            self.root.x_range = DataRange1d(start=0, end=x_end, range_padding=0)
+            self.root.x_range.end = x_end
 
             title = f"Bytes stored: {format_bytes(meminfo.process)}"
             if meminfo.managed_spilled:
@@ -399,6 +400,7 @@ def __init__(self, scheduler, width=600, **kwargs):
             self.root.xaxis[0].formatter = NumeralTickFormatter(format="0.0 b")
             self.root.xaxis.major_label_orientation = XLABEL_ORIENTATION
             self.root.xaxis.minor_tick_line_alpha = 0
+            self.root.x_range = Range1d(start=0)
             self.root.yaxis.visible = False
             self.root.ygrid.visible = False
 
@@ -506,7 +508,7 @@ def quadlist(i) -> list:
                 k: [vi for vi, w in zip(v, width) if w] for k, v in result.items()
             }
 
-            self.root.x_range = DataRange1d(start=0, end=max_limit, range_padding=0)
+            self.root.x_range.end = max_limit
             update(self.source, result)
 
 


Test patch: