Repository: dask/distributed

Issue:
client.restart() crashes client
dask version: 2.4.0
python version: 3.7.3 (packaged by conda-forge)

Steps to reproduce:
```python
import dask
from dask.distributed import Client
client = Client()  
# open dashboard http://localhost:8787 and everything looks fine

x = client.submit(lambda x: x+1, 10)
x.result()  # works

client.restart()
x = client.submit(lambda x: x+1, 10)
# dashboard still looks fine

x.result()  # might work or freeze, keep repeating until it freezes
# dashboard is empty (no workers and no bytes stored)
```

Traceback:

```bash
^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/.../python3.7/site-packages/distributed/client.py", line 219, in result
    result = self.client.sync(self._result, callback_timeout=timeout, raiseit=False)
  File "/home/.../python3.7/site-packages/distributed/client.py", line 754, in sync
    self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
  File "/home/.../python3.7/site-packages/distributed/utils.py", line 334, in sync
    e.wait(10)
  File "/home/.../python3.7/threading.py", line 552, in wait
    signaled = self._cond.wait(timeout)
  File "/home/.../python3.7/threading.py", line 300, in wait
    gotit = waiter.acquire(True, timeout)
KeyboardInterrupt
```


Discussion:
I tested this with the versions you listed and was unable to reproduce.  what version of distributed are you using ?  And what platform ?
Hmmm it looks like the task becomes lost after a certain amount of time. Running

```python
from time import sleep
from distributed import Client

if __name__ == "__main__":
    with Client() as client:

        x = client.submit(lambda x: x + 1, 10)
        x.result()
        print(x)

        client.restart()

        x = client.submit(lambda x: x + 1, 10)
        for _ in range(10):
            print(x)
            sleep(2)
```

locally with the current distributed `master` branch outputs:

```
<Future: status: finished, type: int, key: <lambda>-83b6f133bbdaf59a76c04ca19de925ba>
<Future: status: pending, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
<Future: status: finished, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
<Future: status: finished, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
<Future: status: finished, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
<Future: status: finished, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
<Future: status: finished, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
<Future: status: finished, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
<Future: status: finished, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
<Future: status: lost, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
<Future: status: lost, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
```

@quasiben whenever you get a moment, could you try running the above example to see if you get a similar output? 
Folks here may be interested in looking at
`client.cluster.scheduler.story(keyname)`.  I'm not sure that it will have
much information, but it might help.

On Wed, Sep 18, 2019 at 3:06 PM James Bourbeau <notifications@github.com>
wrote:

> Hmmm it looks like the task becomes lost after a certain amount of time.
> Running
>
> from time import sleepfrom distributed import Client
> if __name__ == "__main__":
>     with Client() as client:
>
>         x = client.submit(lambda x: x + 1, 10)
>         x.result()
>         print(x)
>
>         client.restart()
>
>         x = client.submit(lambda x: x + 1, 10)
>         for _ in range(10):
>             print(x)
>             sleep(2)
>
> locally with the current distributed master branch outputs:
>
> <Future: status: finished, type: int, key: <lambda>-83b6f133bbdaf59a76c04ca19de925ba>
> <Future: status: pending, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
> <Future: status: finished, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
> <Future: status: finished, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
> <Future: status: finished, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
> <Future: status: finished, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
> <Future: status: finished, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
> <Future: status: finished, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
> <Future: status: finished, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
> <Future: status: lost, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
> <Future: status: lost, type: int, key: <lambda>-cabccca250a18c90efeff00fb32ef496>
>
> @quasiben <https://github.com/quasiben> whenever you get a moment, could
> you try running the above example to see if you get a similar output?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/3062?email_source=notifications&email_token=AACKZTGDXJN3LOYWBFQN66TQKKC57A5CNFSM4IX6GNZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7BJEBQ#issuecomment-532845062>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AACKZTFBL4PRU4NNHNEVZR3QKKC57ANCNFSM4IX6GNZQ>
> .
>

@jrbourbeau, yes I get the same output as you.  

the output from what @mrocklin suggested looks like the following:

```
In [16]: client.cluster.scheduler.story(x.key)
Out[16]:
[('<lambda>-d9e821c4ba91472850200ead3ce00016',
  'released',
  'waiting',
  OrderedDict([('<lambda>-d9e821c4ba91472850200ead3ce00016', 'processing')]),
  1568837796.725915),
 ('<lambda>-d9e821c4ba91472850200ead3ce00016',
  'waiting',
  'processing',
  {},
  1568837796.7259789),
 ('<lambda>-d9e821c4ba91472850200ead3ce00016',
  'processing',
  'memory',
  OrderedDict(),
  1568837796.7314382),
 ('<lambda>-d9e821c4ba91472850200ead3ce00016',
  'memory',
  'released',
  OrderedDict([('<lambda>-d9e821c4ba91472850200ead3ce00016', 'waiting')]),
  1568837807.074343),
 ('<lambda>-d9e821c4ba91472850200ead3ce00016',
  'released',
  'waiting',
  OrderedDict([('<lambda>-d9e821c4ba91472850200ead3ce00016', 'processing')]),
  1568837807.074356),
 ('<lambda>-d9e821c4ba91472850200ead3ce00016',
  'waiting',
  'processing',
  {},
  1568837807.0744011),
 ('<lambda>-d9e821c4ba91472850200ead3ce00016',
  'processing',
  'released',
  OrderedDict([('<lambda>-d9e821c4ba91472850200ead3ce00016', 'waiting')]),
  1568837807.074603),
 ('<lambda>-d9e821c4ba91472850200ead3ce00016',
  'released',
  'no-worker',
  OrderedDict(),
  1568837807.074625)]
```
This is also interesting behavior:

```python
In [1]: import dask
   ...: from dask.distributed import Client
   ...: client = Client()

In [2]: client
Out[2]: <Client: 'tcp://127.0.0.1:64842' processes=4 threads=12, memory=17.18 GB>

In [3]: client.restart()
Out[3]: <Client: 'tcp://127.0.0.1:64842' processes=4 threads=12, memory=17.18 GB>

In [4]: x = client.submit(lambda x: x + 1, 10)

In [5]: client
Out[5]: <Client: 'tcp://127.0.0.1:64842' processes=4 threads=12, memory=17.18 GB>

In [6]: client
Out[6]: <Client: 'tcp://127.0.0.1:64842' processes=0 threads=0, memory=0 B>
```


Perhaps the `client = Client()` instantiation of a cluster is not robust to `restart` 
@quasiben: answering to your question:
distributed version: 2.4.0
platform: linux 4.19.69-1-MANJARO
@mgsnuno I'm not sure why the workers are dying shortly after restart.  If I had to guess , I would look at something not being set properly when `Client` calls `LocalCluster()` .  @mrocklin do you have an idea what might be happening here ?
Looked into it and narrowed it down to:
-  with dask 2.4.0 and distributed 2.4.0 we have the problem
- with dask 2.4.0 and distributed 2.3.0 we don't have the problem

The problem is tied with distributed 2.4.0, so recently introduced. 

Looked into `class Client(Node):` and the recent changes would be #2871 I would say.

This should help to narrow things down I hope.
That's surprising.  That PR was merged before distributed 2.3.0.  

How did you determine that that PR was responsible?  If you have time to try `git bisect` then that might be helpful in determining the issue.
@mrocklin I'll try `git bisect`. 

I didn't mean to state that the PR was the cause of the problem, while doing `git blame` on `Client` saw that commit and it being recent, possible on that timeframe, and thought it to be relevant to mention. Sorry for the confusion.

Resolving patch:
diff --git a/distributed/deploy/spec.py b/distributed/deploy/spec.py
--- a/distributed/deploy/spec.py
+++ b/distributed/deploy/spec.py
@@ -306,7 +306,14 @@ def _update_worker_status(self, op, msg):
             name = self.scheduler_info["workers"][msg]["name"]
 
             def f():
-                if name in self.workers and msg not in self.scheduler_info:
+                if (
+                    name in self.workers
+                    and msg not in self.scheduler_info["workers"]
+                    and not any(
+                        d["name"] == name
+                        for d in self.scheduler_info["workers"].values()
+                    )
+                ):
                     self._futures.add(asyncio.ensure_future(self.workers[name].close()))
                     del self.workers[name]
 


Test patch:
diff --git a/distributed/deploy/tests/test_spec_cluster.py b/distributed/deploy/tests/test_spec_cluster.py
--- a/distributed/deploy/tests/test_spec_cluster.py
+++ b/distributed/deploy/tests/test_spec_cluster.py
@@ -156,6 +156,26 @@ async def test_unexpected_closed_worker(cleanup):
             assert len(cluster.workers) == 2
 
 
+@pytest.mark.slow
+@pytest.mark.asyncio
+async def test_restart(cleanup):
+    # Regression test for https://github.com/dask/distributed/issues/3062
+    worker = {"cls": Nanny, "options": {"nthreads": 1}}
+    with dask.config.set({"distributed.deploy.lost-worker-timeout": "2s"}):
+        async with SpecCluster(
+            asynchronous=True, scheduler=scheduler, worker=worker
+        ) as cluster:
+            async with Client(cluster, asynchronous=True) as client:
+                cluster.scale(2)
+                await cluster
+                assert len(cluster.workers) == 2
+
+                await client.restart()
+                await asyncio.sleep(3)
+
+                assert len(cluster.workers) == 2
+
+
 @pytest.mark.asyncio
 async def test_broken_worker():
     with pytest.raises(Exception) as info: