Repository: dask/distributed

Issue:
dask-scheduler --jupyter fails to start
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**Describe the issue**:
I have been using `Scheduler(jupyter=True)` for a while without issue; I just tried the CLI version and it seems to fail.


**Minimal Complete Verifiable Example**:

```
(.venv)  ➜  distributed git:(main) dask-scheduler --jupyter
/Users/brett/distributed/distributed/cli/dask_scheduler.py:140: FutureWarning: dask-scheduler is deprecated and will be removed in a future release; use `dask scheduler` instead
  warnings.warn(
2023-02-21 18:29:56,114 - distributed.scheduler - INFO - -----------------------------------------------
2023-02-21 18:29:56,130 - distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
usage: dask-scheduler [-h] [--debug] [--show-config] [--show-config-json] [--generate-config] [-y]
                      [--allow-root] [--no-browser] [--autoreload] [--script] [--no-script]
                      [--log-level ServerApp.log_level] [--config ServerApp.config_file]
                      [--ip ServerApp.ip] [--port ServerApp.port]
                      [--port-retries ServerApp.port_retries] [--sock ServerApp.sock]
                      [--sock-mode ServerApp.sock_mode] [--transport KernelManager.transport]
                      [--keyfile ServerApp.keyfile] [--certfile ServerApp.certfile]
                      [--client-ca ServerApp.client_ca] [--notebook-dir ServerApp.root_dir]
                      [--preferred-dir ServerApp.preferred_dir] [--browser ServerApp.browser]
                      [--pylab ServerApp.pylab] [--gateway-url GatewayClient.url]
                      [extra_args ...]
dask-scheduler: error: argument --jupyter: expected one argument
```

**Anything else we need to know?**:
Adding `sys.argv.remove("--jupyter")` right before `Scheduler()` gets called [here](https://github.com/dask/distributed/blob/main/distributed/cli/dask_scheduler.py#L209) resolves the issue
```
(.venv)  ➜  distributed git:(main) ✗ dask-scheduler --jupyter
/Users/brett/distributed/distributed/cli/dask_scheduler.py:140: FutureWarning: dask-scheduler is deprecated and will be removed in a future release; use `dask scheduler` instead
  warnings.warn(
2023-02-21 18:32:27,072 - distributed.scheduler - INFO - -----------------------------------------------
2023-02-21 18:32:27,081 - distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
[I 2023-02-21 18:32:27.752 ServerApp] jupyterlab | extension was successfully linked.
[I 2023-02-21 18:32:27.754 ServerApp] nbclassic | extension was successfully linked.
[I 2023-02-21 18:32:27.861 ServerApp] notebook_shim | extension was successfully linked.
[W 2023-02-21 18:32:27.880 ServerApp] All authentication is disabled.  Anyone who can connect to this server will be able to run code.
[I 2023-02-21 18:32:27.881 ServerApp] notebook_shim | extension was successfully loaded.
[I 2023-02-21 18:32:27.881 LabApp] JupyterLab extension loaded from /Users/brett/distributed/.venv/lib/python3.10/site-packages/jupyterlab
[I 2023-02-21 18:32:27.881 LabApp] JupyterLab application directory is /Users/brett/distributed/.venv/share/jupyter/lab
[I 2023-02-21 18:32:27.883 ServerApp] jupyterlab | extension was successfully loaded.
```
So it seems like something down in the Jupyter initialization stack is getting confused? I also tried on an old Docker image from back when this feature was first released and it seems like it didn't work then either, so I don't think it's anything to do with recent updates in `jupyter-server` etc.

**Environment**:

- Dask version: current main `9e8876dba90e4fac5101a1a3dba0f658b279a243`
- Python version: 3.10


Discussion:

Resolving patch:
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -3510,6 +3510,7 @@ def __init__(
             )
             j.initialize(
                 new_httpserver=False,
+                argv=[],
             )
             self._jupyter_server_application = j
             self.http_application.add_application(j.web_app)


Test patch:
diff --git a/distributed/tests/test_jupyter.py b/distributed/tests/test_jupyter.py
--- a/distributed/tests/test_jupyter.py
+++ b/distributed/tests/test_jupyter.py
@@ -2,12 +2,19 @@
 
 import pytest
 
+pytest.importorskip("requests")
+
+import requests
+
+from distributed import Client
+
 pytest.importorskip("jupyter_server")
 
 from tornado.httpclient import AsyncHTTPClient
 
 from distributed import Scheduler
-from distributed.utils_test import gen_test
+from distributed.utils import open_port
+from distributed.utils_test import gen_test, popen
 
 
 @gen_test()
@@ -18,3 +25,22 @@ async def test_jupyter_server():
             f"http://localhost:{s.http_server.port}/jupyter/api/status"
         )
         assert response.code == 200
+
+
+@pytest.mark.slow
+def test_jupyter_cli(loop):
+    port = open_port()
+    with popen(
+        [
+            "dask",
+            "scheduler",
+            "--jupyter",
+            "--no-dashboard",
+            "--host",
+            f"127.0.0.1:{port}",
+        ],
+        capture_output=True,
+    ):
+        with Client(f"127.0.0.1:{port}", loop=loop):
+            response = requests.get("http://127.0.0.1:8787/jupyter/api/status")
+            assert response.status_code == 200