Repository: dask/distributed

Issue:
Dashboards fail with 500 status code when using `bokeh<3.3.0`
When using the latest `main` with `bokeh<3.3.0`, the dashboards fail with a 500 status code.

Scheduler traceback:
```
2023-11-30 18:00:07,300 - tornado.application - ERROR - Uncaught exception GET /status (192.168.178.45)
HTTPServerRequest(protocol='http', host='192.168.178.45:8787', method='GET', uri='/status', version='HTTP/1.1', remote_ip='192.168.178.45')
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/dask-distributed/lib/python3.11/site-packages/tornado/web.py", line 1786, in _execute
    result = await result
             ^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/dask-distributed/lib/python3.11/site-packages/bokeh/server/views/doc_handler.py", line 57, in get
    resources=self.application.resources(),
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hendrikmakait/projects/dask/distributed/distributed/dashboard/core.py", line 37, in resources
    return super().resources(absolute_url)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/dask-distributed/lib/python3.11/site-packages/bokeh/server/tornado.py", line 621, in resources
    return Resources(mode="server", root_url=root_url, path_versioner=StaticHandler.append_version)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/dask-distributed/lib/python3.11/site-packages/bokeh/resources.py", line 377, in __init__
    if root_url and not root_url.endswith("/"):
                        ^^^^^^^^^^^^^^^^^
AttributeError: 'bool' object has no attribute 'endswith'
```

git bisect blames #8347


Discussion:
#8347 hasn't been released, yet. @jacobtomlinson do you have time to look at this?
@hendrikmakait which version of bokeh do you have? I've just tested distributed `main` against bokeh `3.3.0`, `3.2.2`, `3.1.1` and `2.4.2` (the min dep) and I can't reproduce.
That's weird, I've been testing `3.2.0`, `3.2.2`, and `3.1.2` and had this pop up in different environments.
I'll wipe my development environment including all directories and see if this issue still pops up.

Ah I figured out the difference. The problem happens if I create a `LocalCluster`.

```python
# I get the 500
from dask.distributed import Client
client = Client()
print(client.dashboard_link)
```

But not if I start a scheduler via the CLI.

```console
$ # I don't get the 500
$ dask scheduler
2023-11-30 17:22:59,964 - distributed.scheduler - INFO - -----------------------------------------------
2023-11-30 17:23:01,379 - distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
2023-11-30 17:23:01,430 - distributed.scheduler - INFO - State start
2023-11-30 17:23:01,437 - distributed.scheduler - INFO - -----------------------------------------------
2023-11-30 17:23:01,438 - distributed.scheduler - INFO -   Scheduler at:  tcp://10.51.100.253:8786
2023-11-30 17:23:01,438 - distributed.scheduler - INFO -   dashboard at:  http://10.51.100.253:8787/status
```
> Ah I figured out the difference. The problem happens if I create a `LocalCluster`.
> 
> ```python
> from dask.distributed import Client
> client = Client()
> print(client.dashboard_link)
> ```

My bad, I should've specified how I instantiate the scheduler. I use `dask scheduler` and the error pops up when I actually _visit_ the dashboard.
> But not if I start a scheduler via the CLI.

Huh.
Maybe my dev environment is a bit confused too. Let me make a new env.
I've created a new environment in a sandbox directory, `pip` installed `distributed@main` and `bokeh=3.2`, the error is still around:

```
(sandbox) ➜  sandbox dask scheduler 
2023-11-30 18:31:10,304 - distributed.scheduler - INFO - -----------------------------------------------
2023-11-30 18:31:10,719 - distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
2023-11-30 18:31:10,744 - distributed.scheduler - INFO - State start
2023-11-30 18:31:10,747 - distributed.scheduler - INFO - -----------------------------------------------
2023-11-30 18:31:10,748 - distributed.scheduler - INFO -   Scheduler at: tcp://192.168.178.45:8786
2023-11-30 18:31:10,748 - distributed.scheduler - INFO -   dashboard at:  http://192.168.178.45:8787/status
2023-11-30 18:31:10,748 - distributed.scheduler - INFO - Registering Worker plugin shuffle
2023-11-30 18:31:21,812 - tornado.application - ERROR - Uncaught exception GET /status (192.168.178.45)
HTTPServerRequest(protocol='http', host='192.168.178.45:8787', method='GET', uri='/status', version='HTTP/1.1', remote_ip='192.168.178.45')
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/sandbox/lib/python3.11/site-packages/tornado/web.py", line 1786, in _execute
    result = await result
             ^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/sandbox/lib/python3.11/site-packages/bokeh/server/views/doc_handler.py", line 57, in get
    resources=self.application.resources(),
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/sandbox/lib/python3.11/site-packages/distributed/dashboard/core.py", line 37, in resources
    return super().resources(absolute_url)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/sandbox/lib/python3.11/site-packages/bokeh/server/tornado.py", line 621, in resources
    return Resources(mode="server", root_url=root_url, path_versioner=StaticHandler.append_version)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/sandbox/lib/python3.11/site-packages/bokeh/resources.py", line 377, in __init__
    if root_url and not root_url.endswith("/"):
                        ^^^^^^^^^^^^^^^^^
AttributeError: 'bool' object has no attribute 'endswith'
2023-11-30 18:31:21,814 - distributed.core - INFO - Event loop was unresponsive in Scheduler for 8.05s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
```
Leave it with me, I'm putting a fix together now.
Jumping through the source code in VS Code leads to this definition of `BokehTornado.resources` which does not expect a `bool` as input input. However, #8347 defaults to `True`.

```python3
    def resources(self, absolute_url: str | None = None) -> Resources:
        ''' Provide a :class:`~bokeh.resources.Resources` that specifies where
        Bokeh application sessions should load BokehJS resources from.

        Args:
            absolute_url (bool):
                An absolute URL prefix to use for locating resources. If None,
                relative URLs are used (default: None)

        '''
        mode = settings.resources(default="server")
        if mode == "server" or mode == "server-dev":
            root_url = urljoin(absolute_url or self._absolute_url or "", self._prefix)
            return Resources(mode="server", root_url=root_url, path_versioner=StaticHandler.append_version)
        return Resources(mode=mode)
```
Yeah the patch suggested by @bryevdv was to change the default to `True` because we can't explicitly set this anywhere.

However looking at the source of `2.4.2` we still need this to be `None`. Given we are working around a breaking change in `3.3.0` I wonder if we need to check the bokeh version and only do this in newer versions of `bokeh`.
> Given we are working around a breaking change in `3.3.0` I wonder if we need to check the bokeh version and only do this in newer versions of `bokeh`.

Sounds reasonable to me

Resolving patch:
diff --git a/distributed/dashboard/core.py b/distributed/dashboard/core.py
--- a/distributed/dashboard/core.py
+++ b/distributed/dashboard/core.py
@@ -6,8 +6,8 @@
 from bokeh.application import Application
 from bokeh.application.handlers.function import FunctionHandler
 from bokeh.resources import Resources
-from bokeh.server.server import BokehTornado
 from bokeh.server.util import create_hosts_allowlist
+from packaging.version import parse as parse_version
 
 import dask
 
@@ -32,9 +32,14 @@
     from bokeh.models import TabPanel  # noqa: F401
 
 
-class DaskBokehTornado(BokehTornado):
-    def resources(self, absolute_url: str | bool | None = True) -> Resources:
-        return super().resources(absolute_url)
+if BOKEH_VERSION < parse_version("3.3.0"):
+    from bokeh.server.server import BokehTornado as DaskBokehTornado
+else:
+    from bokeh.server.server import BokehTornado
+
+    class DaskBokehTornado(BokehTornado):  # type: ignore[no-redef]
+        def resources(self, absolute_url: str | bool | None = True) -> Resources:
+            return super().resources(absolute_url)
 
 
 def BokehApplication(applications, server, prefix="/", template_variables=None):


Test patch: