Repository: dask/distributed

Issue:
Bokeh Dashboard - Task Stream stops working after an extended period of time
This has happened to me a few times.

I have a 30 worker cluster, it's been up for a while doing lots of different things. At some point, the task stream plot just gives up and doesn't render anything.  You can see in the screenshot below that the other plots are ticking along nicely.

![screenshot from 2019-02-01 18-28-38](https://user-images.githubusercontent.com/1796208/52156739-b2782180-264f-11e9-9bf7-683a7302a314.png)

The tasks tab (which takes a long time to load presumably does to the fact that i've done a lot of jobs over time) looks like this:

![screenshot from 2019-02-01 18-32-45](https://user-images.githubusercontent.com/1796208/52156757-db98b200-264f-11e9-819d-186067066a65.png)

There is the following error in the console, but I suspect it's unrelated and is just a bokeh artifact from me loading the page.

![screenshot from 2019-02-01 18-34-25](https://user-images.githubusercontent.com/1796208/52156809-1b5f9980-2650-11e9-880d-303c821a79be.png)






Discussion:
Anything in the scheduler logs?

On Fri, Feb 1, 2019, 4:35 PM Sarah Bird <notifications@github.com wrote:

> This has happened to me a few times.
>
> I have a 30 worker cluster, it's been up for a while doing lots of
> different things. At some point, the task stream plot just gives up and
> doesn't render anything. You can see in the screenshot below that the other
> plots are ticking along nicely.
>
> [image: screenshot from 2019-02-01 18-28-38]
> <https://user-images.githubusercontent.com/1796208/52156739-b2782180-264f-11e9-9bf7-683a7302a314.png>
>
> The tasks tab (which takes a long time to load presumably does to the fact
> that i've done a lot of jobs over time) looks like this:
>
> [image: screenshot from 2019-02-01 18-32-45]
> <https://user-images.githubusercontent.com/1796208/52156757-db98b200-264f-11e9-819d-186067066a65.png>
>
> There is the following error in the console, but I suspect it's unrelated
> and is just a bokeh artifact from me loading the page.
>
> [image: screenshot from 2019-02-01 18-34-25]
> <https://user-images.githubusercontent.com/1796208/52156809-1b5f9980-2650-11e9-880d-303c821a79be.png>
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/2501>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszEG-zy2A7fVWoLd10fjZ0tna9LMOks5vJN1DgaJpZM4afT06>
> .
>

I can't see. My cluster was started with dask-yarn.
It's not a big deal. If you leave me notes of things to setup / configure when I start my next cluster I can post back when/if it happens again with more details. Of course my current rather intense workload will be dying off in a bit so it may be a while before i reproduce.

I'm on EMR. Latest versions of everything (cluster was built a couple of days ago).
There may be a way to get logs from your cluster object.  I would look for
a logs method

On Fri, Feb 1, 2019, 8:12 PM Sarah Bird <notifications@github.com wrote:

> It's not a big deal. If you leave me notes of things to setup / configure
> when I start my next cluster I can post back when/if it happens again with
> more details. Of course my current rather intense workload will be dying
> off in a bit so it may be a while before i reproduce.
>
> I'm on EMR. Latest versions of everything (cluster was built a couple of
> days ago).
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/2501#issuecomment-459933662>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszL422AMhone5F4zMUyu5h6V8JFZFks5vJRBEgaJpZM4afT06>
> .
>

If there is it isn't documented and I can't see it in the code either. Any thoughts @jcrist?
There is no logs method. You can see live logs via the YARN web ui (see https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html and https://jcrist.github.io/skein/web-ui.html). To get logs from stopped applications, run `yarn logs -applicationId <your application id>`.
OKAY cool. It's happened again and I've made my way to skein web ui and through that to hadoop ui.  Poking around i'm seeing logs but nothing looks relevant or like a bokeh error / info. Any pointers on where to look?
I would expect some log to come from distributed.bokeh.  It would probably
be pretty obvious though.  If nothing is happening then I'm not sure.

On Sat, Feb 2, 2019 at 8:37 PM Sarah Bird <notifications@github.com> wrote:

> OKAY cool. It's happened again and I've made my way to skein web ui and
> through that to hadoop ui. Poking around i'm seeing logs but nothing looks
> relevant or like a bokeh error / info. Any pointers on where to look?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/2501#issuecomment-460022530>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszMfV1z0Zl5eoSmbPtZhFav0cFtXPks5vJmd1gaJpZM4afT06>
> .
>

I am encountering the same issue with a `LocalCluster` on a remote server. Where do I find the logs? I have checked these at `localhost:xxxx/info/main/logs.html` and they show nothing special:

```
distributed.scheduler - INFO - Clear task state

distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:34458

distributed.scheduler - INFO - bokeh at: 127.0.0.1:6657

distributed.scheduler - INFO - Register tcp://127.0.0.1:40071

distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:40071

distributed.scheduler - INFO - Register tcp://127.0.0.1:42353

distributed.scheduler - INFO - Register tcp://127.0.0.1:33325

distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:42353

distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:33325

distributed.scheduler - INFO - Register tcp://127.0.0.1:37151

distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:37151

distributed.scheduler - INFO - Register tcp://127.0.0.1:35367

distributed.scheduler - INFO - Register tcp://127.0.0.1:35153

distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:35367

distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:35153

distributed.scheduler - INFO - Receive client connection: Client-1a043cb8-40fc-11e9-8be5-246e96ad1c18
```

If you're starting a LocalCluster then try the silence_logs= keyword

On Thu, Mar 7, 2019 at 11:12 AM Julius Busecke <notifications@github.com>
wrote:

> I am encountering the same issue with a LocalCluster on a remote server.
> Where do I find the logs? I have checked these at
> localhost:xxxx/info/main/logs.html and they show nothing special:
>
> distributed.scheduler - INFO - Clear task state
>
> distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:34458
>
> distributed.scheduler - INFO - bokeh at: 127.0.0.1:6657
>
> distributed.scheduler - INFO - Register tcp://127.0.0.1:40071
>
> distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:40071
>
> distributed.scheduler - INFO - Register tcp://127.0.0.1:42353
>
> distributed.scheduler - INFO - Register tcp://127.0.0.1:33325
>
> distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:42353
>
> distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:33325
>
> distributed.scheduler - INFO - Register tcp://127.0.0.1:37151
>
> distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:37151
>
> distributed.scheduler - INFO - Register tcp://127.0.0.1:35367
>
> distributed.scheduler - INFO - Register tcp://127.0.0.1:35153
>
> distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:35367
>
> distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:35153
>
> distributed.scheduler - INFO - Receive client connection: Client-1a043cb8-40fc-11e9-8be5-246e96ad1c18
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/2501#issuecomment-470612699>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszJNHhyW8BdLKiH5XbqJBr6DvdV6oks5vUUiLgaJpZM4afT06>
> .
>

Was this solved?  It happens to me as well and I'm also not seeing anything in the logs that points to a cause.
Nope. Still happens to me all the time.
Actually, here is an error in my scheduler readout that might be connected to the issue:

distributed.utils - ERROR - 'start'
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/distributed/utils.py", line 713, in log_errors
    yield
  File "/usr/local/lib/python3.7/dist-packages/distributed/bokeh/components.py", line 542, in update
    ts = metadata["keys"][self.key]
KeyError: 'start'
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7f8a22ba8d68>>, <Future finished exception=KeyError('start')>)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/usr/local/lib/python3.7/dist-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.send(value)
  File "/home/dan/.local/lib/python3.7/site-packages/bokeh/server/session.py", line 70, in _needs_document_lock_wrapper
    result = yield yield_for_all_futures(func(self, *args, **kwargs))
  File "/home/dan/.local/lib/python3.7/site-packages/bokeh/server/session.py", line 191, in with_document_locked
    return func(*args, **kwargs)
  File "/home/dan/.local/lib/python3.7/site-packages/bokeh/document/document.py", line 1126, in wrapper
    return doc._with_self_as_curdoc(invoke)
  File "/home/dan/.local/lib/python3.7/site-packages/bokeh/document/document.py", line 1112, in _with_self_as_curdoc
    return f()
  File "/home/dan/.local/lib/python3.7/site-packages/bokeh/document/document.py", line 1125, in invoke
    return f(*args, **kwargs)
  File "/home/dan/.local/lib/python3.7/site-packages/bokeh/document/document.py", line 915, in remove_then_invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/distributed/bokeh/components.py", line 564, in <lambda>
    self.doc().add_next_tick_callback(lambda: self.update(prof, metadata))
  File "/home/dan/.local/lib/python3.7/site-packages/bokeh/core/property/validation.py", line 97, in func
    return input_function(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/distributed/bokeh/components.py", line 542, in update
    ts = metadata["keys"][self.key]
KeyError: 'start'
Hrm, that traceback seems to be related to the profile plot, not the task stream plot.  It still looks like it's worth tracking down, but doesn't seem related to this issue.


If anyone is able to provide a minimal reproducible example here that would be welcome.  Ideally this would be a small set of steps to reproduce the issue.
I was unable to reproduce the issue with a couple attempts at creating a minimal reproducible example.  But, it happens regularly with my real code.  Is there a way to view and/or enable verbose logs from bokeh and the code that creates the task stream?  
@simaster123 you can set the log level in dask config file:
http://docs.dask.org/en/latest/configuration.html#yaml-files
Thanks, @quasiben.  I uncommented the logging section in the distributed.yaml file and changed bokeh to info (I also tried debug), but I'm not seeing any additional logging information.  Am I doing something wrong?

  logging:
     #distributed: info
     #distributed.client: warning
     bokeh: info
     # http://stackoverflow.com/questions/21234772/python-tornado-disable-logging-to-stderr
     #tornado: critical
     #tornado.application: error

I set up an example where it happens quite often, you can run it on Pangeo's binder:
http://binder.pangeo.io/v2/gh/davidbrochart/pangeo-streamflow/master
And execute the `notebooks/dask_stream_bug.ipynb` notebook. I'm not sure you can see the issue locally, but you can try by setting `is_pangeo_data=False`:
https://github.com/davidbrochart/pangeo-streamflow

Resolving patch:
diff --git a/distributed/dashboard/scheduler.py b/distributed/dashboard/scheduler.py
--- a/distributed/dashboard/scheduler.py
+++ b/distributed/dashboard/scheduler.py
@@ -35,6 +35,7 @@
 from bokeh.themes import Theme
 from bokeh.transform import factor_cmap
 from bokeh.io import curdoc
+import dask
 from toolz import pipe, merge
 from tornado import escape
 
@@ -1417,7 +1418,9 @@ def tasks_doc(scheduler, extra, doc):
     with log_errors():
         ts = TaskStream(
             scheduler,
-            n_rectangles=100000,
+            n_rectangles=dask.config.get(
+                "distributed.scheduler.dashboard.tasks.task-stream-length"
+            ),
             clear_interval="60s",
             sizing_mode="stretch_both",
         )
@@ -1447,7 +1450,9 @@ def status_doc(scheduler, extra, doc):
     with log_errors():
         task_stream = TaskStream(
             scheduler,
-            n_rectangles=1000,
+            n_rectangles=dask.config.get(
+                "distributed.scheduler.dashboard.status.task-stream-length"
+            ),
             clear_interval="10s",
             sizing_mode="stretch_both",
         )
diff --git a/distributed/diagnostics/task_stream.py b/distributed/diagnostics/task_stream.py
--- a/distributed/diagnostics/task_stream.py
+++ b/distributed/diagnostics/task_stream.py
@@ -3,6 +3,7 @@
 from collections import deque
 import logging
 
+import dask
 from .progress_stream import color_of
 from .plugin import SchedulerPlugin
 from ..utils import key_split, format_time, parse_timedelta
@@ -13,7 +14,16 @@
 
 
 class TaskStreamPlugin(SchedulerPlugin):
-    def __init__(self, scheduler, maxlen=100000):
+    def __init__(self, scheduler, maxlen=None):
+        if maxlen is None:
+            maxlen = max(
+                dask.config.get(
+                    "distributed.scheduler.dashboard.status.task-stream-length"
+                ),
+                dask.config.get(
+                    "distributed.scheduler.dashboard.tasks.task-stream-length"
+                ),
+            )
         self.buffer = deque(maxlen=maxlen)
         self.scheduler = scheduler
         scheduler.add_plugin(self)
@@ -74,8 +84,8 @@ def rectangles(self, istart, istop=None, workers=None, start_boundary=0):
         msgs = []
         diff = self.index - len(self.buffer)
         if istop is None:
-            istop = len(self.buffer)
-        for i in range((istart or 0) - diff, istop - diff if istop else istop):
+            istop = self.index
+        for i in range(max(0, (istart or 0) - diff), istop - diff if istop else istop):
             msg = self.buffer[i]
             msgs.append(msg)
 


Test patch:
diff --git a/distributed/dashboard/tests/test_scheduler_bokeh.py b/distributed/dashboard/tests/test_scheduler_bokeh.py
--- a/distributed/dashboard/tests/test_scheduler_bokeh.py
+++ b/distributed/dashboard/tests/test_scheduler_bokeh.py
@@ -627,3 +627,31 @@ def test_proxy_to_workers(c, s, a, b):
             assert b"pip install jupyter-server-proxy" in response_proxy.body
         assert response_direct.code == 200
         assert b"Crossfilter" in response_direct.body
+
+
+@gen_cluster(
+    client=True,
+    scheduler_kwargs={"services": {("dashboard", 0): BokehScheduler}},
+    config={
+        "distributed.scheduler.dashboard.tasks.task-stream-length": 10,
+        "distributed.scheduler.dashboard.status.task-stream-length": 10,
+    },
+)
+async def test_lots_of_tasks(c, s, a, b):
+    import toolz
+
+    ts = TaskStream(s)
+    ts.update()
+    futures = c.map(toolz.identity, range(100))
+    await wait(futures)
+
+    tsp = [p for p in s.plugins if "taskstream" in type(p).__name__.lower()][0]
+    assert len(tsp.buffer) == 10
+    ts.update()
+    assert len(ts.source.data["start"]) == 10
+    assert "identity" in str(ts.source.data)
+
+    futures = c.map(lambda x: x, range(100), pure=False)
+    await wait(futures)
+    ts.update()
+    assert "lambda" in str(ts.source.data)