Repository: dask/distributed

Issue:
Support for multiple protocols or heterogenous protocols
I have a use case where the `client<=>scheduler` communication needs to be proxied, so we use the [websocket protocol](
https://github.com/dask/distributed/blob/main/distributed/comm/ws.py).

However, doing so means that `scheduler<=>worker` and `worker<=>worker` communication must also use websockets as we assume consistency. 

This results in performance issues on systems with high-performance networking like Infiniband or NVLink. We usually use [UCX](https://github.com/dask/distributed/blob/main/distributed/comm/ucx.py) for `worker<=>worker` communication to get the best out of the hardware.

I wanted to start a discussion around a solution for this. The two things that spring to mind are:
- Add support for multiple protocols simultaneously.
- Allow for one protocol for `client<=>scheduler` and a different one for `worker<=>worker`.


Discussion:
It seems to me like supporting different protocols for `client<=>scheduler` and `worker<=>worker` would be a lot easier. I'm not familiar with internals of websocket, but it looks like we would need a UCX-based implementation (and any other potential future communication frameworks that are supported) of it to make it work. This new layer would very likely incur in additional overhead that wouldn't really add any extra benefit for `worker<=>worker` scenario, or at least I can't think of any. Given we're talking about high-performance communications, I would any day rather avoid introducing additional overhead.
This is already done I think.  Here is a test

```python
@gen_test()
async def test_multiple_listeners():
    with captured_logger(logging.getLogger("distributed.scheduler")) as log:
        async with Scheduler(dashboard_address=":0", protocol=["inproc", "tcp"]) as s:
            async with Worker(s.listeners[0].contact_address) as a:
                async with Worker(s.listeners[1].contact_address) as b:
                    assert a.address.startswith("inproc")
                    assert a.scheduler.address.startswith("inproc")
                    assert b.address.startswith("tcp")
                    assert b.scheduler.address.startswith("tcp")

                    async with Client(s.address, asynchronous=True) as c:
                        futures = c.map(inc, range(20))
                        await wait(futures)

                        # Force inter-worker communication both ways
                        await c.submit(sum, futures, workers=[a.address])
                        await c.submit(len, futures, workers=[b.address])

    log = log.getvalue()
    assert re.search(r"Scheduler at:\s*tcp://", log)
    assert re.search(r"Scheduler at:\s*inproc://", log)
```
Oooh awesome, I didn't know protocol takes a list. I think there would be a need to specify which ports the different protocols listen on and set this from the `dask-scheduler` CLI. But that's a pretty superficial change if the core work has already been done.

Resolving patch:
diff --git a/distributed/cli/dask_scheduler.py b/distributed/cli/dask_scheduler.py
--- a/distributed/cli/dask_scheduler.py
+++ b/distributed/cli/dask_scheduler.py
@@ -27,7 +27,7 @@
 
 @click.command(context_settings=dict(ignore_unknown_options=True))
 @click.option("--host", type=str, default="", help="URI, IP or hostname of this server")
-@click.option("--port", type=int, default=None, help="Serving port")
+@click.option("--port", type=str, default=None, help="Serving port")
 @click.option(
     "--interface",
     type=str,
@@ -130,6 +130,8 @@
 def main(
     host,
     port,
+    protocol,
+    interface,
     bokeh_port,
     show,
     dashboard,
@@ -162,8 +164,29 @@ def main(
         )
         dashboard = bokeh
 
+    if interface and "," in interface:
+        interface = interface.split(",")
+
+    if protocol and "," in protocol:
+        protocol = protocol.split(",")
+
+    if port:
+        if "," in port:
+            port = [int(p) for p in port.split(",")]
+        else:
+            port = int(port)
+
     if port is None and (not host or not re.search(r":\d", host)):
-        port = 8786
+        if isinstance(protocol, list):
+            port = [8786] + [0] * (len(protocol) - 1)
+        else:
+            port = 8786
+
+    if isinstance(protocol, list) or isinstance(port, list):
+        if (not isinstance(protocol, list) or not isinstance(port, list)) or len(
+            port
+        ) != len(protocol):
+            raise ValueError("--protocol and --port must both be lists of equal length")
 
     sec = {
         k: v
@@ -202,6 +225,8 @@ async def run():
             security=sec,
             host=host,
             port=port,
+            protocol=protocol,
+            interface=interface,
             dashboard=dashboard,
             dashboard_address=dashboard_address,
             http_prefix=dashboard_prefix,


Test patch:
diff --git a/distributed/cli/tests/test_dask_scheduler.py b/distributed/cli/tests/test_dask_scheduler.py
--- a/distributed/cli/tests/test_dask_scheduler.py
+++ b/distributed/cli/tests/test_dask_scheduler.py
@@ -131,6 +131,22 @@ def test_dashboard_non_standard_ports(loop):
         requests.get(f"http://localhost:{port2}/status/")
 
 
+def test_multiple_protocols(loop):
+    port1 = open_port()
+    port2 = open_port()
+    with popen(
+        [
+            "dask-scheduler",
+            "--protocol=tcp,ws",
+            f"--port={port1},{port2}",
+        ]
+    ) as _:
+        with Client(f"tcp://127.0.0.1:{port1}", loop=loop):
+            pass
+        with Client(f"ws://127.0.0.1:{port2}", loop=loop):
+            pass
+
+
 @pytest.mark.skipif(not LINUX, reason="Need 127.0.0.2 to mean localhost")
 def test_dashboard_allowlist(loop):
     pytest.importorskip("bokeh")