Repository: dask/distributed

Issue:
SSHCluster can't turn dashboard off
Trying to turn off the dashboard results in a traceback at launch for SSHCluster
```python
from distributed import SSHCluster
cluster = SSHCluster(['localhost']*4, scheduler_options={'dashboard_address':None})
```
<details>
  <summary>Traceback - Click to expand!</summary>
  
```
distributed.deploy.ssh - INFO - distributed.scheduler - INFO - -----------------------------------------------
distributed.deploy.ssh - INFO - distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
distributed.deploy.ssh - INFO - Traceback (most recent call last):
distributed.deploy.ssh - INFO - File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/node.py", line 232, in start_http_server
distributed.deploy.ssh - INFO - self.http_server.listen(**http_address)
distributed.deploy.ssh - INFO - File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/tornado/tcpserver.py", line 151, in listen
distributed.deploy.ssh - INFO - sockets = bind_sockets(port, address=address)
distributed.deploy.ssh - INFO - File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/tornado/netutil.py", line 117, in bind_sockets
distributed.deploy.ssh - INFO - socket.getaddrinfo(address, port, family, socket.SOCK_STREAM, 0, flags),
distributed.deploy.ssh - INFO - File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/socket.py", line 918, in getaddrinfo
distributed.deploy.ssh - INFO - for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
distributed.deploy.ssh - INFO - socket.gaierror: [Errno 8] nodename nor servname provided, or not known
Traceback (most recent call last):
  File "/Users/shannon/Library/Application Support/JetBrains/PyCharm2020.1/scratches/scratch_1.py", line 19, in <module>
    cluster = SSHCluster(['localhost']*4, scheduler_options={'dashboard_address':None})
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/deploy/ssh.py", line 336, in SSHCluster
    return SpecCluster(workers, scheduler, name="SSHCluster", **kwargs)
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/deploy/spec.py", line 256, in __init__
    self.sync(self._start)
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/deploy/cluster.py", line 163, in sync
    return sync(self.loop, func, *args, **kwargs)
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/utils.py", line 347, in sync
    raise exc.with_traceback(tb)
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/utils.py", line 331, in f
    result[0] = yield future
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/deploy/spec.py", line 284, in _start
    self.scheduler = await self.scheduler
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/deploy/spec.py", line 50, in _
    await self.start()
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/deploy/ssh.py", line 203, in start
    raise Exception("Worker failed to start")
Exception: Worker failed to start
distributed.deploy.spec - WARNING - Cluster closed without starting up
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/utils.py", line 198, in ignoring
    yield
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/deploy/spec.py", line 611, in close_clusters
    cluster.close(timeout=10)
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/deploy/cluster.py", line 84, in close
    return self.sync(self._close, callback_timeout=timeout)
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/deploy/cluster.py", line 163, in sync
    return sync(self.loop, func, *args, **kwargs)
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/utils.py", line 347, in sync
    raise exc.with_traceback(tb)
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/utils.py", line 331, in f
    result[0] = yield future
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/deploy/spec.py", line 400, in _close
    await super()._close()
  File "/Users/shannon/opt/miniconda3/envs/pywakes/lib/python3.8/site-packages/distributed/deploy/cluster.py", line 73, in _close
    await self._watch_worker_status_comm.close()
AttributeError: 'SpecCluster' object has no attribute '_watch_worker_status_comm'
```

</details>

**Environment**:

- Dask version: 2.17.2
- Python version: 3.8.3
- Operating System: macOS
- Install method (conda, pip, source): conda



Discussion:
Thank you for the excellent issue @Keou0007 

I have a possible fix in https://github.com/dask/distributed/pull/3907 but this upsets some downstream work in dask-cuda.  I suggest that we wait a little while to see how they feel about it.
No worries. I usually turn off the dashboard because I don't use it, but it doesn't really hurt anything to have it running, so no impact on my work.

Resolving patch:
diff --git a/distributed/deploy/ssh.py b/distributed/deploy/ssh.py
--- a/distributed/deploy/ssh.py
+++ b/distributed/deploy/ssh.py
@@ -2,15 +2,13 @@
 import sys
 import warnings
 import weakref
+from json import dumps
 from typing import List, Union
 
 import dask
 import dask.config
 
 from ..core import Status
-from ..scheduler import Scheduler as _Scheduler
-from ..utils import cli_keywords
-from ..worker import Worker as _Worker
 from .spec import ProcessInterface, SpecCluster
 
 logger = logging.getLogger(__name__)
@@ -55,8 +53,8 @@ class Worker(Process):
         The address of the scheduler
     address: str
         The hostname where we should run this worker
-    worker_module: str
-        The python module to run to start the worker.
+    worker_class: str
+        The python class to use to create the worker.
     connect_options: dict
         kwargs to be passed to asyncssh connections
     remote_python: str
@@ -72,16 +70,23 @@ def __init__(
         address: str,
         connect_options: dict,
         kwargs: dict,
-        worker_module="distributed.cli.dask_worker",
+        worker_module="deprecated",
+        worker_class="distributed.Nanny",
         remote_python=None,
         loop=None,
         name=None,
     ):
         super().__init__()
 
+        if worker_module != "deprecated":
+            raise ValueError(
+                "worker_module has been deprecated in favor of worker_class. "
+                "Please specify a Python class rather than a CLI module."
+            )
+
         self.address = address
         self.scheduler = scheduler
-        self.worker_module = worker_module
+        self.worker_class = worker_class
         self.connect_options = connect_options
         self.kwargs = kwargs
         self.name = name
@@ -116,12 +121,20 @@ async def start(self):
                 set_env,
                 self.remote_python,
                 "-m",
-                self.worker_module,
+                "distributed.cli.dask_spec",
                 self.scheduler,
-                "--name",
-                str(self.name),
+                "--spec",
+                "'%s'"
+                % dumps(
+                    {
+                        "cls": self.worker_class,
+                        "opts": {
+                            **self.kwargs,
+                            "name": self.name,
+                        },
+                    }
+                ),
             ]
-            + cli_keywords(self.kwargs, cls=_Worker, cmd=self.worker_module)
         )
 
         self.proc = await self.connection.create_process(cmd)
@@ -197,9 +210,10 @@ async def start(self):
                 set_env,
                 self.remote_python,
                 "-m",
-                "distributed.cli.dask_scheduler",
+                "distributed.cli.dask_spec",
+                "--spec",
+                "'%s'" % dumps({"cls": "distributed.Scheduler", "opts": self.kwargs}),
             ]
-            + cli_keywords(self.kwargs, cls=_Scheduler)
         )
         self.proc = await self.connection.create_process(cmd)
 
@@ -240,7 +254,8 @@ def SSHCluster(
     connect_options: Union[List[dict], dict] = {},
     worker_options: dict = {},
     scheduler_options: dict = {},
-    worker_module: str = "distributed.cli.dask_worker",
+    worker_module: str = "deprecated",
+    worker_class: str = "distributed.Nanny",
     remote_python: Union[str, List[str]] = None,
     **kwargs,
 ):
@@ -278,8 +293,8 @@ def SSHCluster(
         Keywords to pass on to workers.
     scheduler_options : dict, optional
         Keywords to pass on to scheduler.
-    worker_module : str, optional
-        Python module to call to start the worker.
+    worker_class: str
+        The python class to use to create the worker(s).
     remote_python : str or list of str, optional
         Path to Python on remote nodes.
 
@@ -294,15 +309,15 @@ def SSHCluster(
     ... )
     >>> client = Client(cluster)
 
-    An example using a different worker module, in particular the
-    ``dask-cuda-worker`` command from the ``dask-cuda`` project.
+    An example using a different worker class, in particular the
+    ``CUDAWorker`` from the ``dask-cuda`` project:
 
     >>> from dask.distributed import Client, SSHCluster
     >>> cluster = SSHCluster(
     ...     ["localhost", "hostwithgpus", "anothergpuhost"],
     ...     connect_options={"known_hosts": None},
     ...     scheduler_options={"port": 0, "dashboard_address": ":8797"},
-    ...     worker_module="dask_cuda.cli.dask_cuda_worker")
+    ...     worker_class="dask_cuda.CUDAWorker")
     >>> client = Client(cluster)
 
     See Also
@@ -311,6 +326,12 @@ def SSHCluster(
     dask.distributed.Worker
     asyncssh.connect
     """
+    if worker_module != "deprecated":
+        raise ValueError(
+            "worker_module has been deprecated in favor of worker_class. "
+            "Please specify a Python class rather than a CLI module."
+        )
+
     if set(kwargs) & old_cluster_kwargs:
         from .old_ssh import SSHCluster as OldSSHCluster
 
@@ -360,7 +381,7 @@ def SSHCluster(
                 if isinstance(connect_options, dict)
                 else connect_options[i + 1],
                 "kwargs": worker_options,
-                "worker_module": worker_module,
+                "worker_class": worker_class,
                 "remote_python": remote_python[i + 1]
                 if isinstance(remote_python, list)
                 else remote_python,


Test patch:
diff --git a/distributed/deploy/tests/test_ssh.py b/distributed/deploy/tests/test_ssh.py
--- a/distributed/deploy/tests/test_ssh.py
+++ b/distributed/deploy/tests/test_ssh.py
@@ -51,7 +51,6 @@ async def test_keywords():
         connect_options=dict(known_hosts=None),
         asynchronous=True,
         worker_options={
-            "nprocs": 2,  # nprocs checks custom arguments with cli_keywords
             "nthreads": 2,
             "memory_limit": "2 GiB",
             "death_timeout": "5s",