Repository: dask/distributed

Issue:
Incompatible equality test of addresses in Distributed 2021.3.0
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.


Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**:

After updating from 2021.2.0 to 2021.3.0, our regular tests started to observe this error. 
Any pointers as of what might be a good way to mitigate the issue?

````
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/yfeng1/source/nbodykit/build/testenv/lib/python3.8/site-packages/nbodykit/tests/test_distributed.py:32: in test_save
    cat.save(tmpfile, dataset='data')
/home/yfeng1/source/nbodykit/build/testenv/lib/python3.8/site-packages/nbodykit/base/catalog.py:692: in save
    source.store(target, regions=region, lock=False, compute=True)
/home/yfeng1/.conda/envs/test38/lib/python3.8/site-packages/dask/array/core.py:1540: in store
    r = store([self], [target], **kwargs)
/home/yfeng1/.conda/envs/test38/lib/python3.8/site-packages/dask/array/core.py:1041: in store
    result.compute(**kwargs)
/home/yfeng1/.conda/envs/test38/lib/python3.8/site-packages/dask/base.py:283: in compute
    (result,) = compute(self, traverse=False, **kwargs)
/home/yfeng1/.conda/envs/test38/lib/python3.8/site-packages/dask/base.py:553: in compute
    schedule = get_scheduler(
/home/yfeng1/.conda/envs/test38/lib/python3.8/site-packages/dask/base.py:1151: in get_scheduler
    return get_scheduler(scheduler=config.get("scheduler", None))
/home/yfeng1/.conda/envs/test38/lib/python3.8/site-packages/dask/base.py:1141: in get_scheduler
    return get_client().get
/home/yfeng1/.conda/envs/test38/lib/python3.8/site-packages/distributed/worker.py:3200: in get_client
    return worker._get_client(timeout=timeout)
/home/yfeng1/.conda/envs/test38/lib/python3.8/site-packages/distributed/worker.py:3076: in _get_client
    or client._start_arg == self.scheduler.address
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = LocalCluster(d66622f8, 'inproc://192.168.86.31/163961/9', workers=1, threads=1, memory=16.59 GB)
other = 'inproc://192.168.86.31/163961/1'

    def __eq__(self, other):
>       return self.name == other.name
E       AttributeError: 'str' object has no attribute 'name'

/home/yfeng1/.conda/envs/test38/lib/python3.8/site-packages/distributed/deploy/cluster.py:461: AttributeError
```

**What you expected to happen**:

It is not clear how we could triggered malformed input. We are simply creating a LocalCluster in this case. 
Might be a regression on distributed's side. 

**Minimal Complete Verifiable Example**:

```python
# Put your MCVE code here
```

**Anything else we need to know?**:

**Environment**:

- Dask version:  2021.3.0
- Python version: 3.8
- Operating System: Linux
- Install method (conda, pip, source):
Conda



Discussion:
Thanks for raising an issue @rainwoodman. It's not immediately clear to me what in the latest release could be causing this. The particular `AttributeError` which is being triggered is happening in some code which was added in 2021.01.1. Do you have a code snippet you could post that I could run? 
Thanks for looking into it.

I actually have had difficulty reproducing this on my local machine. It used to occur randomly on our github workflow, but has become a persistent error recently:

https://pipelines.actions.githubusercontent.com/4WG3RDC5LrvSQSfVuzaeJ6lLFmXDn3iE4xIDQGf4ZMGoiw0ciN/_apis/pipelines/1/runs/206/signedlogcontent/3?urlExpires=2021-05-08T09%3A25%3A08.0077385Z&urlSigningMethod=HMACV1&urlSignature=NWLiHD2e9K0Xp9e6ZAskBUQpF5SX9bi5rds55o4zYKU%3D

I'll try to contain this error more closely and report back.
@jrbourbeau I think I found the problem and filed an illustrative PR. 

The root cause for triggering the elsebranch of the try-catch is likely a race on the configuration files. The problem only occurs when multiple processes (MPI ranks in my context) each try to start its own LocalCluster. It is not always reproducible for my test case.

It's not very clear to me how to best construct a unit test case for this branch, or if it is already covered by a test and we just need to upgrade it to the case where _start_arg is a LocalCluster object. -- would you take it from here?



Resolving patch:
diff --git a/distributed/deploy/cluster.py b/distributed/deploy/cluster.py
--- a/distributed/deploy/cluster.py
+++ b/distributed/deploy/cluster.py
@@ -422,7 +422,9 @@ async def __aexit__(self, typ, value, traceback):
         await self.close()
 
     @property
-    def scheduler_address(self):
+    def scheduler_address(self) -> str:
+        if not self.scheduler_comm:
+            return "<Not Connected>"
         return self.scheduler_comm.address
 
     @property
@@ -458,7 +460,7 @@ def observed(self):
         return {d["name"] for d in self.scheduler_info["workers"].values()}
 
     def __eq__(self, other):
-        return self.name == other.name
+        return type(other) == type(self) and self.name == other.name
 
     def __hash__(self):
         return id(self)
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -3259,10 +3259,21 @@ def _get_client(self, timeout=None):
         except ValueError:  # no clients found, need to make a new one
             pass
         else:
+            # must be lazy import otherwise cyclic import
+            from distributed.deploy.cluster import Cluster
+
             if (
                 client.scheduler
                 and client.scheduler.address == self.scheduler.address
-                or client._start_arg == self.scheduler.address
+                # The below conditions should only happen in case a second
+                # cluster is alive, e.g. if a submitted task spawned its onwn
+                # LocalCluster, see gh4565
+                or (
+                    isinstance(client._start_arg, str)
+                    and client._start_arg == self.scheduler.address
+                    or isinstance(client._start_arg, Cluster)
+                    and client._start_arg.scheduler_address == self.scheduler.address
+                )
             ):
                 self._client = client
 


Test patch:
diff --git a/distributed/deploy/tests/test_cluster.py b/distributed/deploy/tests/test_cluster.py
new file mode 100644
--- /dev/null
+++ b/distributed/deploy/tests/test_cluster.py
@@ -0,0 +1,28 @@
+import pytest
+
+from distributed.deploy.cluster import Cluster
+from distributed.utils_test import cleanup  # noqa: F401
+
+
+@pytest.mark.asyncio
+async def test_eq(cleanup):
+    clusterA = Cluster(asynchronous=True, name="A")
+    clusterA2 = Cluster(asynchronous=True, name="A2")
+    clusterB = Cluster(asynchronous=True, name="B")
+    assert clusterA != "A"
+    assert not (clusterA == "A")
+    assert clusterA == clusterA
+    assert not (clusterA != clusterA)
+    assert clusterA != clusterA2
+    assert not (clusterA == clusterA2)
+    assert clusterA != clusterB
+    assert not (clusterA == clusterB)
+
+
+@pytest.mark.asyncio
+async def test_repr(cleanup):
+    cluster = Cluster(asynchronous=True, name="A")
+    assert cluster.scheduler_address == "<Not Connected>"
+    res = repr(cluster)
+    expected = "Cluster(A, '<Not Connected>', workers=0, threads=0, memory=0 B)"
+    assert res == expected
diff --git a/distributed/tests/test_client.py b/distributed/tests/test_client.py
--- a/distributed/tests/test_client.py
+++ b/distributed/tests/test_client.py
@@ -6602,3 +6602,32 @@ async def test_workers_collection_restriction(c, s, a, b):
     future = c.compute(da.arange(10), workers=a.address)
     await future
     assert a.data and not b.data
+
+
+@gen_cluster(client=True, nthreads=[("127.0.0.1", 0)])
+async def test_get_client_functions_spawn_clusters(c, s, a):
+    # see gh4565
+
+    scheduler_addr = c.scheduler.address
+
+    def f(x):
+        ref = None
+        with LocalCluster(
+            n_workers=1,
+            processes=False,
+            dashboard_address=False,
+            worker_dashboard_address=False,
+        ) as cluster2:
+            with Client(cluster2) as c1:
+                c2 = get_client()
+
+                c1_scheduler = c1.scheduler.address
+                c2_scheduler = c2.scheduler.address
+                assert c1_scheduler != c2_scheduler
+                assert c2_scheduler == scheduler_addr
+
+    await c.gather(c.map(f, range(2)))
+    await a.close()
+
+    c_default = default_client()
+    assert c is c_default
diff --git a/distributed/tests/test_worker.py b/distributed/tests/test_worker.py
--- a/distributed/tests/test_worker.py
+++ b/distributed/tests/test_worker.py
@@ -1905,6 +1905,7 @@ async def test_worker_client_uses_default_no_close(c, s, a):
     If a default client is available in the process, the worker will pick this
     one and will not close it if it is closed
     """
+    assert not Worker._initialized_clients
     assert default_client() is c
     existing_client = c.id
 
@@ -1957,7 +1958,7 @@ def get_worker_client_id():
             default_client()
 
 
-@gen_cluster(timeout=3600)
+@gen_cluster()
 async def test_worker_client_closes_if_created_on_worker_last_worker_alive(s, a, b):
     async with Client(s.address, set_as_default=False, asynchronous=True) as c:
 
diff --git a/distributed/utils_test.py b/distributed/utils_test.py
--- a/distributed/utils_test.py
+++ b/distributed/utils_test.py
@@ -1479,6 +1479,7 @@ def check_instances():
     Worker._instances.clear()
     Scheduler._instances.clear()
     SpecCluster._instances.clear()
+    Worker._initialized_clients.clear()
     # assert all(n.status == "closed" for n in Nanny._instances), {
     #     n: n.status for n in Nanny._instances
     # }
@@ -1502,6 +1503,12 @@ def check_instances():
                 w.loop.add_callback(w.close)
     Worker._instances.clear()
 
+    start = time()
+    while any(c.status != "closed" for c in Worker._initialized_clients):
+        sleep(0.1)
+        assert time() < start + 10
+    Worker._initialized_clients.clear()
+
     for i in range(5):
         if all(c.closed() for c in Comm._instances):
             break