Repository: dask/distributed

Issue:
Worker-worker connections fail with "missing port number in address `<closed>`"
**Describe the issue**: We sometimes see a ValueError with the message `"missing port number in address '<closed>'"` with this stacktrace:

Worker-worker connections fail with "missing port number in address `<closed>`"


```
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/distributed/worker.py", line 2035, in gather_dep
    response = await get_data_from_worker(
  File "/usr/local/lib/python3.9/site-packages/distributed/worker.py", line 2868, in get_data_from_worker
    comm = await rpc.connect(worker)
  File "/usr/local/lib/python3.9/site-packages/distributed/core.py", line 1606, in connect
    return await connect_attempt
  File "/usr/local/lib/python3.9/site-packages/distributed/core.py", line 1527, in _connect
    comm = await connect(
  File "/usr/local/lib/python3.9/site-packages/distributed/comm/core.py", line 367, in connect
    **comm.handshake_info(),
  File "/usr/local/lib/python3.9/site-packages/distributed/comm/core.py", line 158, in handshake_info
    if self.same_host:
  File "/usr/local/lib/python3.9/site-packages/distributed/comm/core.py", line 126, in same_host
    local_ipaddr = get_address_host(resolve_address(self.local_address))
  File "/usr/local/lib/python3.9/site-packages/distributed/comm/addressing.py", line 181, in resolve_address
    return unparse_address(scheme, backend.resolve_address(loc))
  File "/usr/local/lib/python3.9/site-packages/distributed/comm/tcp.py", line 699, in resolve_address
    host, port = parse_host_port(loc)
  File "/usr/local/lib/python3.9/site-packages/distributed/comm/addressing.py", line 95, in parse_host_port
    port = _default()
  File "/usr/local/lib/python3.9/site-packages/distributed/comm/addressing.py", line 73, in _default
    raise ValueError(f"missing port number in address {address!r}")
```

I was not able to produce a minimal verifiable example, but I think I have a good idea what is going on:

 * `distributed.comm.core.connect` initiates the underlying connection (TLS over TCP in this case); this calls into another function:
 * `distributed.comm.tcp.BaseTCPConnector.connect`. This method creates the underlying connection just fine, but it is closed immediately (by the server-side of the connection; note that this is not directly another distributed worker, since there is a service mesh of proxies in between). With this closed connection, this method runs this line: `local_address = self.prefix + get_stream_address(stream)`, which effectively runs `local_address = "tcp://<closed>"` (!), but does not raise an exception (yet)
 * control returns to `distributed.comm.core.connect` which now runs `comm.handshake_info` and produces the stacktrace above when it tried to parse the address with the string "\<closed\>".

This produces a `ValueError`, which -- unlike most other errors with the connection -- is not retried  in `distributed.comm.core.connect`, since it only retries on some exception classes like `OSError`, but not on `ValueError`.

Note that some of the code was introduced recently, in particular the "same_host" function. With earlier versions of distributed, such cases of closed connections would raise an `OSError`, which would be retried.

I think a possible fix is to change the implementation of `get_stream_address` to always raise a `CommClosedError` rather than returning "\<closed\>", so the connection attempt is retried via the usual retry code.

**Environment**:

- Dask version: 2023.6.0
- Python version: observed for several, at least 3.9 / 3.11
- Operating System: Debian 11
- Install method (conda, pip, source): observed for both conda and pip



Discussion:
> I think a possible fix is to change the implementation of get_stream_address to always raise a CommClosedError rather than returning "<closed>", so the connection attempt is retried via the usual retry code.

Sounds reasonable. Are you interested in opening a PR for this?

Resolving patch:
diff --git a/distributed/comm/tcp.py b/distributed/comm/tcp.py
--- a/distributed/comm/tcp.py
+++ b/distributed/comm/tcp.py
@@ -119,14 +119,13 @@ def get_stream_address(comm):
     """
     Get a stream's local address.
     """
+    # raise OSError in case the comm is closed, s.t.
+    # retry code can handle it appropriately; see also
+    # https://github.com/dask/distributed/issues/7953
     if comm.closed():
-        return "<closed>"
+        raise CommClosedError()
 
-    try:
-        return unparse_host_port(*comm.socket.getsockname()[:2])
-    except OSError:
-        # Probably EBADF
-        return "<closed>"
+    return unparse_host_port(*comm.socket.getsockname()[:2])
 
 
 def convert_stream_closed_error(obj, exc):


Test patch:
diff --git a/distributed/comm/tests/test_comms.py b/distributed/comm/tests/test_comms.py
--- a/distributed/comm/tests/test_comms.py
+++ b/distributed/comm/tests/test_comms.py
@@ -25,6 +25,7 @@
     unparse_host_port,
 )
 from distributed.comm.registry import backends, get_backend
+from distributed.comm.tcp import get_stream_address
 from distributed.compatibility import asyncio_run
 from distributed.config import get_loop_factory
 from distributed.metrics import time
@@ -1342,6 +1343,15 @@ async def test_tcp_adresses(tcp):
     await check_addresses(a, b)
 
 
+@gen_test()
+async def test_get_stream_address_raises_if_closed():
+    a, b = await get_tcp_comm_pair()
+    a.abort()
+    with pytest.raises(OSError):
+        get_stream_address(a)
+    b.abort()
+
+
 @gen_test()
 async def test_tls_adresses(tcp):
     a, b = await get_tls_comm_pair()