Repository: dask/distributed

Issue:
Dashboard views that may crash your browser should be gated
If I have a dask task graph with 200k tasks then clicking on the "tasks" tab of dask dasboard will lead to doom. Specifically it will just hang my browser. This is particularly disastrous if you're using dask-jupyterlab because then you break your jupyterlab session. 

I'm wondering if we can implement a gate that, for a given threshold (say 50k for arguments sake), delays the loading of the plot and presents the user a message saying "this view has more than 50k items, are you sure you want to display" or something like that.


Discussion:
Sounds good to me.

Or WebGL?

I think that we currently limit this to 100k in the tasks tab.  We could also reduce this limit (I think that it's configurable?)  I don't think that we do any kind of limit on the graph tab, which fails earlier at a few 10s of thousands.
For these plots we could also think about alternative visualizations that did a bit of aggregation.  (although that's probably a separate issue)
WebGL will alleviate this to a certain extent but it's not a total fix as the data still needs to arrive. 

Limiting seems less good as how else would you see the complete history.

That said, we could add a slider that defaults to nk history items, but then the user could increase that history.

I like the slider idea and is easy for tasks, but i think graph will need to be gated.
Yeah, gating graph makes sense regardless.  The graph loses a lot of value anyway with that many points.
I think I crashed my whole scheduler and lost all progress by trying to open the Tasks page with 20k tasks in the graph. The scheduler container had a memory limit of 1400Mi.
We are seeing the same behavior with a graph containing 180k nodes. Once such a large task is complete, the scheduler keeps all the timing metrics in memory in case the UI attempts to display anything (which is the point at which the browser will freeze, the scheduler might crash due to mem limits, etc.).

Until there is some sort of fix for the UI, is it possible to programatically tell the scheduler to go ahead and drop successfully processed nodes from memory? After we complete large jobs such as these, we do not require them to hang around in the scheduler any longer. Tips or pointers on achieving this would be appreciated.


It's much much easier to just fix the UI.  If anyone wants to dive into the scheduler code here is an example place to start: 

https://github.com/dask/distributed/blob/eda27bee472090f5370121d7d35c201d2facd79e/distributed/dashboard/components/scheduler.py#L1111

One fix might be to modify the `update` method to pass if there are over a certain number of tasks.
There is still an issue though. When the scheduler gets very large graphs,it is holding on to the timing results of all those nodes. I have seen instances where a scheduler running in a containeer with a large amount of reserved memory will eventually blow the limit and get killed when new (potentially large) graphs get submitted. Normally, I believe it will drop older nodes in order to make room for the new (I am guessing, based on watching its behavior). But in some cases, it just fails. 

If a user application has no need to keep the timing info for all those processed nodes, and there have been no exceptions, would it make sense to have an API to clear them out of the scheduler? Or am I misunderstanding the intended usage here?
It sounds like you're describing a different issue.  I recommend raising a
separate issue.

On Fri, Mar 27, 2020, 7:31 PM Dave Berton <notifications@github.com> wrote:

> There is still an issue though. When the scheduler gets very large
> graphs,it is holding on to the timing results of all those nodes. I have
> seen instances where a scheduler running in a containeer with a large
> amount of reserved memory will eventually blow the limit and get killed
> when new (potentially large) graphs get submitted. Normally, I believe it
> will drop older nodes in order to make room for the new (I am guessing,
> based on watching its behavior). But in some cases, it just fails.
>
> If a user application has no need to keep the timing info for all those
> processed nodes, and there have been no exceptions, would it make sense to
> have an API to clear them out of the scheduler? Or am I misunderstanding
> the intended usage here?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/2979#issuecomment-605382747>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTDIGU2NBK4EHTAOMKLRJVOPRANCNFSM4IOORWIA>
> .
>


Resolving patch:
diff --git a/distributed/dashboard/components/scheduler.py b/distributed/dashboard/components/scheduler.py
--- a/distributed/dashboard/components/scheduler.py
+++ b/distributed/dashboard/components/scheduler.py
@@ -34,6 +34,7 @@
 from bokeh.transform import factor_cmap, linear_cmap
 from bokeh.io import curdoc
 import dask
+from dask import config
 from dask.utils import format_bytes, key_split
 from tlz import pipe
 from tlz.curried import map, concat, groupby
@@ -1171,6 +1172,7 @@ def __init__(self, scheduler, **kwargs):
         tap = TapTool(callback=OpenURL(url="info/task/@key.html"), renderers=[rect])
         rect.nonselection_glyph = None
         self.root.add_tools(hover, tap)
+        self.max_items = config.get("distributed.dashboard.graph-max-items", 5000)
 
     @without_property_validation
     def update(self):
@@ -1206,6 +1208,10 @@ def add_new_nodes_edges(self, new, new_edges, update=False):
             y = self.layout.y
 
             tasks = self.scheduler.tasks
+            if len(tasks) > self.max_items:
+                # graph to big - no update, reset for next time
+                self.invisible_count = len(tasks)
+                return
             for key in new:
                 try:
                     task = tasks[key]


Test patch:
diff --git a/distributed/dashboard/tests/test_scheduler_bokeh.py b/distributed/dashboard/tests/test_scheduler_bokeh.py
--- a/distributed/dashboard/tests/test_scheduler_bokeh.py
+++ b/distributed/dashboard/tests/test_scheduler_bokeh.py
@@ -36,6 +36,7 @@
     ProfileServer,
     MemoryByKey,
 )
+from distributed.utils_test import async_wait_for
 
 from distributed.dashboard import scheduler
 
@@ -501,6 +502,35 @@ def test_TaskGraph_clear(c, s, a, b):
         assert time() < start + 5
 
 
+@gen_cluster(
+    client=True, config={"distributed.dashboard.graph-max-items": 2,},
+)
+def test_TaskGraph_limit(c, s, a, b):
+    gp = TaskGraph(s)
+
+    def func(x):
+        return x
+
+    f1 = c.submit(func, 1)
+    yield wait(f1)
+    gp.update()
+    assert len(gp.node_source.data["x"]) == 1
+    f2 = c.submit(func, 2)
+    yield wait(f2)
+    gp.update()
+    assert len(gp.node_source.data["x"]) == 2
+    f3 = c.submit(func, 3)
+    yield wait(f3)
+    gp.update()
+    assert len(gp.node_source.data["x"]) == 2
+    del f1
+    del f2
+    del f3
+    _ = c.submit(func, 1)
+
+    async_wait_for(lambda: len(gp.node_source.data["x"]) == 1, timeout=1)
+
+
 @gen_cluster(client=True, timeout=30)
 def test_TaskGraph_complex(c, s, a, b):
     da = pytest.importorskip("dask.array")