Repository: dask/distributed

Issue:
Wait on single connection or try multiple connections
When we connect to a remote connection we currently wait on for our full timeout, often ten seconds, something like this:

```python
comm = await connect(address, timeout="10s")
```

However, @jacobtomlinson and I just ran into a situation with Kubernetes where the address that we were connecting to was created at just about the same time, so when we first tried to connect we were sent somewhere that would never receive the connection, but if we try again a second later, things are fine.

```python
comm = await connect(address, timeout="10s")  # this hangs for 10s
```
```python
for i in range(10):  # this connects after 1s
    with ignoring(TimeoutError):
        comm = await comm(address, timeout="1s")
```

This seems to work because, presumably, after the first connection fails and we try reconnecting the network now routes us to the correct location.

In general this second approach seems more robust to networks that might be fiddled with on-the-fly, which is presumably more common in cloud and Kubernetes situations.  However, it also means that we need to become better about cleaning up missed connections.

cc @jcrist @jacobtomlinson and @mmccarty 

The actual code for this is here: https://github.com/dask/distributed/blob/549660e07c0c70fdb17e07c6a18ca438933bd8ba/distributed/comm/core.py#L205-L228


Discussion:

Resolving patch:
diff --git a/distributed/comm/core.py b/distributed/comm/core.py
--- a/distributed/comm/core.py
+++ b/distributed/comm/core.py
@@ -7,7 +7,7 @@
 from tornado import gen
 
 from ..metrics import time
-from ..utils import parse_timedelta
+from ..utils import parse_timedelta, ignoring
 from . import registry
 from .addressing import parse_address
 
@@ -188,6 +188,7 @@ async def connect(addr, timeout=None, deserialize=True, connection_args=None):
     scheme, loc = parse_address(addr)
     backend = registry.get_backend(scheme)
     connector = backend.get_connector()
+    comm = None
 
     start = time()
     deadline = start + timeout
@@ -205,14 +206,19 @@ def _raise(error):
     # This starts a thread
     while True:
         try:
-            future = connector.connect(
-                loc, deserialize=deserialize, **(connection_args or {})
-            )
-            comm = await gen.with_timeout(
-                timedelta(seconds=deadline - time()),
-                future,
-                quiet_exceptions=EnvironmentError,
-            )
+            while deadline - time() > 0:
+                future = connector.connect(
+                    loc, deserialize=deserialize, **(connection_args or {})
+                )
+                with ignoring(gen.TimeoutError):
+                    comm = await gen.with_timeout(
+                        timedelta(seconds=min(deadline - time(), 1)),
+                        future,
+                        quiet_exceptions=EnvironmentError,
+                    )
+                    break
+            if not comm:
+                _raise(error)
         except FatalCommClosedError:
             raise
         except EnvironmentError as e:
@@ -222,8 +228,6 @@ def _raise(error):
                 logger.debug("sleeping on connect")
             else:
                 _raise(error)
-        except gen.TimeoutError:
-            _raise(error)
         else:
             break
 


Test patch: