Repository: dask/distributed

Issue:
Calling "cluster.adapt()" multiple times causes strange behavior and `KilledWorker` errors
**What happened**:

We tried scaling an adaptive cluster twice like this:
```python
cluster.adapt(maximum=16)
cluster.adapt(maximum=8)
```

After submitting tasks to the cluster workers will crash with
```
KilledWorker: ('sample_pi_monte_carlo-c4b060d1-c020-4aa1-b1ea-9f7fadeb881a', <Worker 'tcp://10.56.4.77:40643', name: 7, memory: 0, processing: 94>)
```
and logs on the scheduler show that it scales up the cluster then immediately retires the workers:
```
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Retire worker names (1, 2, 3, 4, 5, 6, 7, 8)
distributed.deploy.adaptive - INFO - Retiring workers [1, 2, 3, 4, 5, 6, 7, 8]
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.56.11.43:33153', name: 12, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.56.11.43:33153
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'tcp://10.56.6.29:35277', name: 6, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, tcp://10.56.6.29:35277
[more register worker / starting worker messages]
...
[more remove worker / removing comms messages]
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.56.6.29:35277', name: 6, memory: 14, processing: 85>
distributed.core - INFO - Removing comms to tcp://10.56.6.29:35277
distributed.scheduler - INFO - Remove worker <Worker 'tcp://10.56.4.77:40643', name: 7, memory: 14, processing: 94>
distributed.core - INFO - Removing comms to tcp://10.56.4.77:40643
```

**What you expected to happen**:

Calling `cluster.adapt()` multiple times should have no effect if the cluster size is within bounds since you're only changing the minimum and maximum number of workers.

**Minimal Complete Verifiable Example**:

```python
import time

import dask
import numpy as np
from dask.distributed import Client
from dask_kubernetes import KubeCluster

cluster = KubeCluster.from_dict(...)
client = Client(cluster)

cluster.adapt(maximum=16)
cluster.adapt(maximum=8)

def sample_pi_monte_carlo(sleep_time):
    x = np.random.uniform(-1, 1)
    y = np.random.uniform(-1, 1)
    time.sleep(sleep_time)
    return (np.sqrt(x**2 + y**2) < 1)

results = client.compute([
    dask.delayed(sample_pi_monte_carlo)(0.1)
    for _ in range(1000)
], sync=True)

print("pi is:", 4 * np.mean(results))
```

**Anything else we need to know?**:

This uses `dask_kubernetes` to implement adaptive scaling. I originally reported this at https://github.com/dask/dask-kubernetes/issues/250 but it might be an upstream issue (their `KubeCluster` class inherits from `SpecCluster` doesn't add any logic to `.adapt()`)

**Environment**:

- Dask version: 2.15.0
- Python version: 3.7
- Operating System: Ubuntu 18
- Install method (conda, pip, source): pip

Let me know if there's anything else I can do to help!


Discussion:
Thanks for the issue @zo7 

I don't have a kubernetes cluster handy, so I ran it locally instead, but unfortunately things passed

```python
In [1]: import time 
   ...:  
   ...: import dask 
   ...: import numpy as np 
   ...: from dask.distributed import Client, LocalCluster 
   ...:  
   ...: cluster = LocalCluster() 
   ...: client = Client(cluster) 
   ...:  
   ...: cluster.adapt(maximum=16) 
   ...: cluster.adapt(maximum=8) 
   ...:  
   ...: def sample_pi_monte_carlo(sleep_time): 
   ...:     x = np.random.uniform(-1, 1) 
   ...:     y = np.random.uniform(-1, 1) 
   ...:     time.sleep(sleep_time) 
   ...:     return (np.sqrt(x**2 + y**2) < 1) 
   ...:  
   ...: results = client.compute([ 
   ...:     dask.delayed(sample_pi_monte_carlo)(0.1) 
   ...:     for _ in range(1000) 
   ...: ], sync=True) 
   ...:  
   ...: print("pi is:", 4 * np.mean(results)) 
   ...:                                                                                                                                     
pi is: 3.168
```

Looking at the error message I don't see anything jump out at me, so unfortunately without diving in more deeply I'm not sure what could be going on here.  

> Let me know if there's anything else I can do to help!

The thing to do here is probably to try to create an even more minimal example that can be run on a developer's laptop.  Once this is easy to reproduce it will probably be a lot easier to resolve.
@jacobtomlinson if you have a k8s cluster handy and some time could you test the above ?
I have seen this before. I'll do some testing and get back to you. 
I'm not seeing the `KilledWorker` error, I do get a result. However I am seeing workers being immediately retired and recreated.

```
distributed.deploy.adaptive - INFO - Retiring workers [14]
distributed.deploy.adaptive - INFO - Retiring workers [14]
distributed.deploy.adaptive - INFO - Retiring workers [32, 33, 34, 35, 36, 37, 30, 31]
distributed.deploy.adaptive - INFO - Retiring workers [38, 39, 40, 41, 42, 43, 44, 45]
distributed.deploy.adaptive - INFO - Retiring workers [46, 47, 48, 49, 50, 51, 52, 53]
distributed.deploy.adaptive - INFO - Retiring workers [64, 65, 66, 67, 68, 54, 55, 56]
distributed.deploy.adaptive - INFO - Retiring workers [68, 69, 70, 71, 72, 73, 74, 75]
distributed.deploy.adaptive - INFO - Retiring workers [75, 76, 77, 78, 79, 80, 81, 82]
distributed.deploy.adaptive - INFO - Retiring workers [82, 83, 84, 85, 86, 87, 88, 89]
distributed.deploy.adaptive - INFO - Retiring workers [61, 89, 90, 91, 92, 93, 95]
distributed.deploy.adaptive - INFO - Retiring workers [96, 97, 98, 99, 95]
distributed.deploy.adaptive - INFO - Retiring workers [99, 100, 101, 102]
distributed.deploy.adaptive - INFO - Retiring workers [102, 103]
distributed.deploy.adaptive - INFO - Retiring workers [62]
distributed.deploy.adaptive - INFO - Retiring workers [62]
distributed.deploy.adaptive - INFO - Retiring workers [94]
distributed.deploy.adaptive - INFO - Retiring workers [94]

pi is: 3.08

distributed.deploy.adaptive - INFO - Retiring workers [38, 57, 58, 59, 60, 63]
distributed.deploy.adaptive - INFO - Retiring workers [38, 57, 58, 59, 60, 63]
distributed.deploy.adaptive - INFO - Retiring workers [29]
distributed.deploy.adaptive - INFO - Retiring workers [29]

```
@jacobtomlinson are you seeing that on Kubernetes?

I mentioned this briefly in my original report, but the `KilledWorker` seems to only happen for certain args for `cluster.adapt(maximum=N)`. When it works, it does seem to get in this cycle of retiring and recreating workers like you're seeing.

(sorry I haven't been more helpful – I haven't been able to create a minimal example that demos this without Kubernetes yet)
My testing was using Kubernetes. 
Actually nvm, I tried running @mrocklin's example with `LocalCluster` again and got it to crash. It seems to be stochastic – sometimes it crashes, sometimes it emits a ton of warnings and completes, sometimes it runs perfectly fine.

If I remove the `cluster.adapt(maximum=16)` line, it runs successfully every time with no warnings or crashes.

### What I did:

Environment:

* macOS Catalina 10.15.4
* Python 3.7.7 (installed via pyenv)

I wrapped @mrocklin's example in `if __name__ == '__main__':` to run it as a script:
```python
import time

import dask
import numpy as np
from dask.distributed import Client, LocalCluster

if __name__ == '__main__':

    cluster = LocalCluster()
    client = Client(cluster)

    cluster.adapt(maximum=16)
    cluster.adapt(maximum=8)

    def sample_pi_monte_carlo(sleep_time):
        x = np.random.uniform(-1, 1)
        y = np.random.uniform(-1, 1)
        time.sleep(sleep_time)
        return (np.sqrt(x**2 + y**2) < 1)

    results = client.compute([
        dask.delayed(sample_pi_monte_carlo)(0.1)
        for _ in range(1000)
    ], sync=True)

    print("pi is:", 4 * np.mean(results))
```

Installed Dask in a fresh virtualenv:
```
$ virtualenv --version
virtualenv 20.0.23 from /Users/flynn/.pyenv/versions/3.7.7/lib/python3.7/site-packages/virtualenv/__init__.py
$ virtualenv -p python3.7 venv
...
$ . venv/bin/activate.fish
(venv) $ python --version
Python 3.7.7
(venv) $ pip install "dask[distributed]" numpy
...
(venv) $ pip freeze
click==7.1.2
cloudpickle==1.4.1
dask==2.19.0
distributed==2.19.0
HeapDict==1.0.1
msgpack==1.0.0
numpy==1.18.5
psutil==5.7.0
PyYAML==5.3.1
sortedcontainers==2.2.2
tblib==1.6.0
toolz==0.10.0
tornado==6.0.4
zict==2.0.0
```

If I remove the `cluster.adapt(maximum=16)` line and run the script, output is consistently normal:
```
(venv) $ python example.py
pi is: 3.132
(venv) $ python example.py
pi is: 3.14
(venv) $ python example.py
pi is: 3.108
```

If I add the line back in and run it, I get a massive amount of logs, and the script sometimes crashes.

Some of the messages I am seeing (many of these are repeated a lot):
```
distributed.utils - ERROR - Timed out trying to connect to 'tcp://127.0.0.1:57631' after 10 s: Timed out trying to connect to 'tcp://127.0.0.1:57631' after 10 s: [Errno 24] Too many open files
Traceback (most recent call last):
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/distributed/comm/core.py", line 234, in connect
    _raise(error)
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/distributed/comm/core.py", line 215, in _raise
    raise IOError(msg)
OSError: Timed out trying to connect to 'tcp://127.0.0.1:57631' after 10 s: [Errno 24] Too many open files

...

distributed.scheduler - ERROR - Workers don't have promised key: ['tcp://127.0.0.1:57694'], sample_pi_monte_carlo-f2716df9-3b62-4f3f-89a3-e599b5fc1243
NoneType: None

...

distributed.client - WARNING - Couldn't gather 436 keys, rescheduling {'sample_pi_monte_carlo-9d1a973b-3109-4da3-8cf7-e8989f474f42': ('tcp://127.0.0.1:57694',), ...

...

handle: <Handle BaseAsyncIOLoop._handle_events(14, 1)>
Traceback (most recent call last):
  File "/Users/flynn/.pyenv/versions/3.7.3/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/tornado/platform/asyncio.py", line 139, in _handle_events
    handler_func(fileobj, events)
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/tornado/netutil.py", line 260, in accept_handler
    connection, address = sock.accept()
  File "/Users/flynn/.pyenv/versions/3.7.3/lib/python3.7/socket.py", line 212, in accept
    fd, addr = self._accept()
OSError: [Errno 24] Too many open files

...

distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - ERROR - Failed to restart worker after its process exited

...

distributed.utils - ERROR - Timed out trying to connect to 'tcp://127.0.0.1:57691' after 10 s: Timed out trying to connect to 'tcp://127.0.0.1:57691' after 10 s: [Errno 24] Too many open files
Traceback (most recent call last):
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/distributed/utils.py", line 656, in log_errors
    yield
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/distributed/deploy/adaptive.py", line 186, in scale_down
    names=workers, remove=True, close_workers=True
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/distributed/core.py", line 790, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/distributed/core.py", line 660, in send_recv
    raise exc.with_traceback(tb)
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/distributed/core.py", line 513, in handle_comm
    result = await result
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/distributed/scheduler.py", line 3513, in retire_workers
    lock=False,
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/distributed/scheduler.py", line 3277, in replicate
    for w, who_has in gathers.items()
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/distributed/utils_comm.py", line 390, in retry_operation
    operation=operation,
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/distributed/core.py", line 858, in send_recv_from_rpc
    comm = await self.pool.connect(self.addr)
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/distributed/core.py", line 1013, in connect
    **self.connection_args,
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/distributed/comm/core.py", line 245, in connect
    _raise(error)
  File "/Users/flynn/code/debug-dask/venv/lib/python3.7/site-packages/distributed/comm/core.py", line 215, in _raise
    raise IOError(msg)
OSError: Timed out trying to connect to 'tcp://127.0.0.1:57691' after 10 s: Timed out trying to connect to 'tcp://127.0.0.1:57691' after 10 s: [Errno 24] Too many open files
```

Can either of you replicate this? Does anything stick out to you or do you notice something I'm doing wrong?
Yeah I am able to replicate this using master for distributed.

Running it manually seems fine, but then if I tell my shell to run it 10 times I get a variety of bad output.

```zsh
$ repeat 10 {python /tmp/test.py}
pi is: 3.108
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x1086d9a90>>, <Task finished coro=<Worker.handle_scheduler() done, defined at /Users/jtomlinson/Projects/dask/distributed/distributed/worker.py:916> exception=CommClosedError('in <closed TCP>: Stream is closed')>)
Traceback (most recent call last):
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/worker.py", line 929, in handle_scheduler
    await self.close(report=False)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/worker.py", line 1126, in close
    await r.close_gracefully()
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 199, in read
    convert_stream_closed_error(self, e)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 123, in convert_stream_closed_error
    raise CommClosedError("in %s: %s" % (obj, exc))
distributed.comm.core.CommClosedError: in <closed TCP>: Stream is closed
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x11daf4610>>, <Task finished coro=<Worker.handle_scheduler() done, defined at /Users/jtomlinson/Projects/dask/distributed/distributed/worker.py:916> exception=CommClosedError('in <closed TCP>: Stream is closed')>)
Traceback (most recent call last):
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/worker.py", line 929, in handle_scheduler
    await self.close(report=False)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/worker.py", line 1126, in close
    await r.close_gracefully()
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 199, in read
    convert_stream_closed_error(self, e)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 123, in convert_stream_closed_error
    raise CommClosedError("in %s: %s" % (obj, exc))
distributed.comm.core.CommClosedError: in <closed TCP>: Stream is closed
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x10a4d8390>>, <Task finished coro=<Worker.handle_scheduler() done, defined at /Users/jtomlinson/Projects/dask/distributed/distributed/worker.py:916> exception=CommClosedError('in <closed TCP>: Stream is closed')>)
Traceback (most recent call last):
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/worker.py", line 929, in handle_scheduler
    await self.close(report=False)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/worker.py", line 1126, in close
    await r.close_gracefully()
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 199, in read
    convert_stream_closed_error(self, e)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 123, in convert_stream_closed_error
    raise CommClosedError("in %s: %s" % (obj, exc))
distributed.comm.core.CommClosedError: in <closed TCP>: Stream is closed
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x114e35c50>>, <Task finished coro=<Worker.handle_scheduler() done, defined at /Users/jtomlinson/Projects/dask/distributed/distributed/worker.py:916> exception=CommClosedError('in <closed TCP>: Stream is closed')>)
Traceback (most recent call last):
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/worker.py", line 929, in handle_scheduler
    await self.close(report=False)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/worker.py", line 1126, in close
    await r.close_gracefully()
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 199, in read
    convert_stream_closed_error(self, e)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 123, in convert_stream_closed_error
    raise CommClosedError("in %s: %s" % (obj, exc))
distributed.comm.core.CommClosedError: in <closed TCP>: Stream is closed
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x10820f650>>, <Task finished coro=<Worker.handle_scheduler() done, defined at /Users/jtomlinson/Projects/dask/distributed/distributed/worker.py:916> exception=CommClosedError('in <closed TCP>: Stream is closed')>)
Traceback (most recent call last):
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/worker.py", line 929, in handle_scheduler
    await self.close(report=False)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/worker.py", line 1126, in close
    await r.close_gracefully()
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 199, in read
    convert_stream_closed_error(self, e)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 123, in convert_stream_closed_error
    raise CommClosedError("in %s: %s" % (obj, exc))
distributed.comm.core.CommClosedError: in <closed TCP>: Stream is closed
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x10ab89b10>>, <Task finished coro=<Worker.handle_scheduler() done, defined at /Users/jtomlinson/Projects/dask/distributed/distributed/worker.py:916> exception=CommClosedError('in <closed TCP>: Stream is closed')>)
Traceback (most recent call last):
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/worker.py", line 929, in handle_scheduler
    await self.close(report=False)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/worker.py", line 1126, in close
    await r.close_gracefully()
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 199, in read
    convert_stream_closed_error(self, e)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 123, in convert_stream_closed_error
    raise CommClosedError("in %s: %s" % (obj, exc))
distributed.comm.core.CommClosedError: in <closed TCP>: Stream is closed
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x1164c7050>>, <Task finished coro=<Worker.handle_scheduler() done, defined at /Users/jtomlinson/Projects/dask/distributed/distributed/worker.py:916> exception=CommClosedError('in <closed TCP>: Stream is closed')>)
Traceback (most recent call last):
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 184, in read
    n_frames = await stream.read_bytes(8)
tornado.iostream.StreamClosedError: Stream is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/worker.py", line 929, in handle_scheduler
    await self.close(report=False)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/worker.py", line 1126, in close
    await r.close_gracefully()
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 861, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 644, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 199, in read
    convert_stream_closed_error(self, e)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/comm/tcp.py", line 123, in convert_stream_closed_error
    raise CommClosedError("in %s: %s" % (obj, exc))
distributed.comm.core.CommClosedError: in <closed TCP>: Stream is closed
pi is: 3.168
pi is: 3.184
pi is: 3.104
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
pi is: 3.112
pi is: 3.136
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.scheduler - ERROR - Couldn't gather keys {'sample_pi_monte_carlo-f17726aa-97ad-46f8-ae50-49283c67ece5': [], 'sample_pi_monte_carlo-f54642c3-7c03-4e50-ad66-0b7e90a8eb13': [], 'sample_pi_monte_carlo-f4469808-c8ba-45ec-a42c-eedb7eb3529f': [], 'sample_pi_monte_carlo-ff2d47a3-2ecc-4d84-b5a4-d03f76af37f6': [], 'sample_pi_monte_carlo-fde265a3-5715-4853-8dba-e428ffd14dd7': [], 'sample_pi_monte_carlo-f42bd846-5962-4a43-a4a9-e4bb69264140': [], 'sample_pi_monte_carlo-f5800b01-a235-473c-b47c-3736ab6195ce': [], 'sample_pi_monte_carlo-f4390cf9-3f90-4d53-b9d6-5cd346fd9fbb': [], 'sample_pi_monte_carlo-faf5e474-d459-4339-9043-a287e3f7d5d0': [], 'sample_pi_monte_carlo-f5559307-f103-4549-827f-4783c8d8598f': [], 'sample_pi_monte_carlo-fd011a1e-b1e4-47e7-8e8c-e777b57d6877': [], 'sample_pi_monte_carlo-d72e1ce9-105c-4cd3-9833-bd6435e13177': [], 'sample_pi_monte_carlo-ea50b281-b45c-4f5f-a299-e26f9c461348': [], 'sample_pi_monte_carlo-fd783abb-a7ac-4700-b3c3-8cc2f071ed71': [], 'sample_pi_monte_carlo-fe8112c9-08d4-4462-9420-172651266f5c': [], 'sample_pi_monte_carlo-f8ae8dc4-259f-4d18-9ee8-a4a02906c933': [], 'sample_pi_monte_carlo-fddb65c7-5893-4bf0-a94c-efc77a037274': [], 'sample_pi_monte_carlo-d89a1593-57dc-426b-aaca-79e3a0b7165e': [], 'sample_pi_monte_carlo-fa14f7a6-b85b-42e4-bb4a-7fad4f4c179b': [], 'sample_pi_monte_carlo-ff17a6d3-1ad7-4df1-b64a-c27d68cb79f9': [], 'sample_pi_monte_carlo-f65424de-1479-4864-b31d-dfac5484d1cc': [], 'sample_pi_monte_carlo-f9257779-309d-47a7-898d-331d866328d5': [], 'sample_pi_monte_carlo-fe4f1d90-aa61-4e40-9059-c802c31b3a69': [], 'sample_pi_monte_carlo-fa3787fa-167a-4ae1-a323-4eaa0af8a2f8': [], 'sample_pi_monte_carlo-f8481ba4-4f16-49ff-82fd-e0520a376e1d': [], 'sample_pi_monte_carlo-f7bbba40-0869-461f-a1cc-12c33fcfd9cf': [], 'sample_pi_monte_carlo-f2a9c347-71f7-4b62-96ac-c5ac59919950': [], 'sample_pi_monte_carlo-fe031db1-8b16-41f7-bd52-a077c65e484e': [], 'sample_pi_monte_carlo-fefde2cd-9181-4ceb-94ae-231167b5f487': [], 'sample_pi_monte_carlo-fdd9cf36-8dae-4891-95f6-357b22ecc853': [], 'sample_pi_monte_carlo-f88b0c55-fa38-45a1-8d10-cdd9501f722f': [], 'sample_pi_monte_carlo-ee9ddaa4-e6db-4ace-99a1-385167873b42': [], 'sample_pi_monte_carlo-f5c7df8a-18f4-4410-9e78-0ca916571f80': [], 'sample_pi_monte_carlo-fe0ecf09-f9cd-4fb5-b10b-b1c175ab028d': [], 'sample_pi_monte_carlo-ff3120af-a9a3-46f2-bee5-c1c81d1a2da7': [], 'sample_pi_monte_carlo-e23340b1-cb38-4f6b-b0f0-098d608b7ea7': [], 'sample_pi_monte_carlo-faca8db9-1f51-4cc1-8da3-ccdad14a8d4e': [], 'sample_pi_monte_carlo-f61aa457-1c54-4fdc-91b7-b5ec09011f1b': [], 'sample_pi_monte_carlo-fda41613-4a73-4765-bc73-79100624cdcd': [], 'sample_pi_monte_carlo-f2f9012c-302e-4208-93c2-9fcd55197219': [], 'sample_pi_monte_carlo-ff3c4f01-fadb-40c7-a713-bd2c1f702de0': [], 'sample_pi_monte_carlo-ff57e378-6860-446a-87c9-3275a360fdd1': []} state: ['processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'processing', 'memory', 'processing', 'processing', 'processing', 'processing', 'processing'] workers: []
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f17726aa-97ad-46f8-ae50-49283c67ece5
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f54642c3-7c03-4e50-ad66-0b7e90a8eb13
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f4469808-c8ba-45ec-a42c-eedb7eb3529f
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-ff2d47a3-2ecc-4d84-b5a4-d03f76af37f6
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-fde265a3-5715-4853-8dba-e428ffd14dd7
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f42bd846-5962-4a43-a4a9-e4bb69264140
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f5800b01-a235-473c-b47c-3736ab6195ce
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f4390cf9-3f90-4d53-b9d6-5cd346fd9fbb
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-faf5e474-d459-4339-9043-a287e3f7d5d0
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f5559307-f103-4549-827f-4783c8d8598f
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-fd011a1e-b1e4-47e7-8e8c-e777b57d6877
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-d72e1ce9-105c-4cd3-9833-bd6435e13177
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-ea50b281-b45c-4f5f-a299-e26f9c461348
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-fd783abb-a7ac-4700-b3c3-8cc2f071ed71
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-fe8112c9-08d4-4462-9420-172651266f5c
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f8ae8dc4-259f-4d18-9ee8-a4a02906c933
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-fddb65c7-5893-4bf0-a94c-efc77a037274
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-d89a1593-57dc-426b-aaca-79e3a0b7165e
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-fa14f7a6-b85b-42e4-bb4a-7fad4f4c179b
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-ff17a6d3-1ad7-4df1-b64a-c27d68cb79f9
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f65424de-1479-4864-b31d-dfac5484d1cc
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f9257779-309d-47a7-898d-331d866328d5
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-fe4f1d90-aa61-4e40-9059-c802c31b3a69
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-fa3787fa-167a-4ae1-a323-4eaa0af8a2f8
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f8481ba4-4f16-49ff-82fd-e0520a376e1d
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f7bbba40-0869-461f-a1cc-12c33fcfd9cf
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f2a9c347-71f7-4b62-96ac-c5ac59919950
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-fe031db1-8b16-41f7-bd52-a077c65e484e
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-fefde2cd-9181-4ceb-94ae-231167b5f487
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-fdd9cf36-8dae-4891-95f6-357b22ecc853
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f88b0c55-fa38-45a1-8d10-cdd9501f722f
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-ee9ddaa4-e6db-4ace-99a1-385167873b42
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f5c7df8a-18f4-4410-9e78-0ca916571f80
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-fe0ecf09-f9cd-4fb5-b10b-b1c175ab028d
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-ff3120af-a9a3-46f2-bee5-c1c81d1a2da7
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-e23340b1-cb38-4f6b-b0f0-098d608b7ea7
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-faca8db9-1f51-4cc1-8da3-ccdad14a8d4e
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f61aa457-1c54-4fdc-91b7-b5ec09011f1b
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-fda41613-4a73-4765-bc73-79100624cdcd
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-f2f9012c-302e-4208-93c2-9fcd55197219
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-ff3c4f01-fadb-40c7-a713-bd2c1f702de0
NoneType: None
distributed.scheduler - ERROR - Workers don't have promised key: [], sample_pi_monte_carlo-ff57e378-6860-446a-87c9-3275a360fdd1
NoneType: None
distributed.client - WARNING - Couldn't gather 42 keys, rescheduling {'sample_pi_monte_carlo-f17726aa-97ad-46f8-ae50-49283c67ece5': (), 'sample_pi_monte_carlo-f54642c3-7c03-4e50-ad66-0b7e90a8eb13': (), 'sample_pi_monte_carlo-f4469808-c8ba-45ec-a42c-eedb7eb3529f': (), 'sample_pi_monte_carlo-ff2d47a3-2ecc-4d84-b5a4-d03f76af37f6': (), 'sample_pi_monte_carlo-fde265a3-5715-4853-8dba-e428ffd14dd7': (), 'sample_pi_monte_carlo-f42bd846-5962-4a43-a4a9-e4bb69264140': (), 'sample_pi_monte_carlo-f5800b01-a235-473c-b47c-3736ab6195ce': (), 'sample_pi_monte_carlo-f4390cf9-3f90-4d53-b9d6-5cd346fd9fbb': (), 'sample_pi_monte_carlo-faf5e474-d459-4339-9043-a287e3f7d5d0': (), 'sample_pi_monte_carlo-f5559307-f103-4549-827f-4783c8d8598f': (), 'sample_pi_monte_carlo-fd011a1e-b1e4-47e7-8e8c-e777b57d6877': (), 'sample_pi_monte_carlo-d72e1ce9-105c-4cd3-9833-bd6435e13177': (), 'sample_pi_monte_carlo-ea50b281-b45c-4f5f-a299-e26f9c461348': (), 'sample_pi_monte_carlo-fd783abb-a7ac-4700-b3c3-8cc2f071ed71': (), 'sample_pi_monte_carlo-fe8112c9-08d4-4462-9420-172651266f5c': (), 'sample_pi_monte_carlo-f8ae8dc4-259f-4d18-9ee8-a4a02906c933': (), 'sample_pi_monte_carlo-fddb65c7-5893-4bf0-a94c-efc77a037274': (), 'sample_pi_monte_carlo-d89a1593-57dc-426b-aaca-79e3a0b7165e': (), 'sample_pi_monte_carlo-fa14f7a6-b85b-42e4-bb4a-7fad4f4c179b': (), 'sample_pi_monte_carlo-ff17a6d3-1ad7-4df1-b64a-c27d68cb79f9': (), 'sample_pi_monte_carlo-f65424de-1479-4864-b31d-dfac5484d1cc': (), 'sample_pi_monte_carlo-f9257779-309d-47a7-898d-331d866328d5': (), 'sample_pi_monte_carlo-fe4f1d90-aa61-4e40-9059-c802c31b3a69': (), 'sample_pi_monte_carlo-fa3787fa-167a-4ae1-a323-4eaa0af8a2f8': (), 'sample_pi_monte_carlo-f8481ba4-4f16-49ff-82fd-e0520a376e1d': (), 'sample_pi_monte_carlo-f7bbba40-0869-461f-a1cc-12c33fcfd9cf': (), 'sample_pi_monte_carlo-f2a9c347-71f7-4b62-96ac-c5ac59919950': (), 'sample_pi_monte_carlo-fe031db1-8b16-41f7-bd52-a077c65e484e': (), 'sample_pi_monte_carlo-fefde2cd-9181-4ceb-94ae-231167b5f487': (), 'sample_pi_monte_carlo-fdd9cf36-8dae-4891-95f6-357b22ecc853': (), 'sample_pi_monte_carlo-f88b0c55-fa38-45a1-8d10-cdd9501f722f': (), 'sample_pi_monte_carlo-ee9ddaa4-e6db-4ace-99a1-385167873b42': (), 'sample_pi_monte_carlo-f5c7df8a-18f4-4410-9e78-0ca916571f80': (), 'sample_pi_monte_carlo-fe0ecf09-f9cd-4fb5-b10b-b1c175ab028d': (), 'sample_pi_monte_carlo-ff3120af-a9a3-46f2-bee5-c1c81d1a2da7': (), 'sample_pi_monte_carlo-e23340b1-cb38-4f6b-b0f0-098d608b7ea7': (), 'sample_pi_monte_carlo-faca8db9-1f51-4cc1-8da3-ccdad14a8d4e': (), 'sample_pi_monte_carlo-f61aa457-1c54-4fdc-91b7-b5ec09011f1b': (), 'sample_pi_monte_carlo-fda41613-4a73-4765-bc73-79100624cdcd': (), 'sample_pi_monte_carlo-f2f9012c-302e-4208-93c2-9fcd55197219': (), 'sample_pi_monte_carlo-ff3c4f01-fadb-40c7-a713-bd2c1f702de0': (), 'sample_pi_monte_carlo-ff57e378-6860-446a-87c9-3275a360fdd1': ()}
pi is: 3.092
pi is: 3.104
distributed.worker - WARNING - Could not find data: {'sample_pi_monte_carlo-8bd484b2-b924-46ea-8d39-01c4614c8f27': ['tcp://127.0.0.1:51025']} on workers: [] (who_has: {'sample_pi_monte_carlo-4c869402-2e34-433f-affd-ae75e7b4c43f': ['tcp://127.0.0.1:51028'], 'sample_pi_monte_carlo-3cd138ee-0028-4767-8412-8f8eaafcc75b': ['tcp://127.0.0.1:51031'], 'sample_pi_monte_carlo-be94f6f9-e8e6-49d3-bb67-be49d1bd3ae1': ['tcp://127.0.0.1:51028'], 'sample_pi_monte_carlo-84dc535d-ccf0-4c15-b597-3ac1e4361b7d': ['tcp://127.0.0.1:51020'], 'sample_pi_monte_carlo-9093fbd0-cd0f-4c22-8c28-a17b68233ec0': ['tcp://127.0.0.1:51034'], 'sample_pi_monte_carlo-653b9580-213b-4d4a-b3b3-0cbdb11461ee': ['tcp://127.0.0.1:51019'], 'sample_pi_monte_carlo-8f0b0286-1c74-4433-a567-49d5b71c0562': ['tcp://127.0.0.1:51019'], 'sample_pi_monte_carlo-85c71499-6dc8-4f3b-948d-f3e892aaa735': ['tcp://127.0.0.1:51020'], 'sample_pi_monte_carlo-72fee335-6b00-41f4-a8f6-9055673e137c': ['tcp://127.0.0.1:51016'], 'sample_pi_monte_carlo-a8dfa892-d395-4376-b016-618fb90f75f4': ['tcp://127.0.0.1:51016'], 'sample_pi_monte_carlo-b81ab7ce-45d9-4df3-a8cf-669d054b73fc': ['tcp://127.0.0.1:51031'], 'sample_pi_monte_carlo-b952d17f-f8e6-4eee-aef5-4ac17d40f356': ['tcp://127.0.0.1:51016'], 'sample_pi_monte_carlo-f7024879-a5f8-444d-bcee-10a8c53334bd': ['tcp://127.0.0.1:51034'], 'sample_pi_monte_carlo-af4e7c59-c484-45b1-a36d-13e5c8c73c76': ['tcp://127.0.0.1:51036'], 'sample_pi_monte_carlo-3eb411a0-f91f-4b13-878c-096e0144170e': ['tcp://127.0.0.1:51028'], 'sample_pi_monte_carlo-b41a4243-9325-4f07-9569-1e7d1851855d': ['tcp://127.0.0.1:51036'], 'sample_pi_monte_carlo-66e084dd-52a9-48a4-a192-432e3b23e364': ['tcp://127.0.0.1:51025'], 'sample_pi_monte_carlo-8bd484b2-b924-46ea-8d39-01c4614c8f27': ['tcp://127.0.0.1:51025'], 'sample_pi_monte_carlo-d8285082-46d2-4661-8a97-311ac79edfe8': ['tcp://127.0.0.1:51034'], 'sample_pi_monte_carlo-5b2702b2-7a51-4e0e-83d8-e4bc0adca5c4': ['tcp://127.0.0.1:51036'], 'sample_pi_monte_carlo-a517e20d-6ca4-4d22-b374-3c2ed645be96': ['tcp://127.0.0.1:51019'], 'sample_pi_monte_carlo-c16cf588-83d4-4cf7-b559-0f6e74ebf975': ['tcp://127.0.0.1:51025'], 'sample_pi_monte_carlo-94784ac8-a9c4-4184-9118-aa409afb8c74': ['tcp://127.0.0.1:51020'], 'sample_pi_monte_carlo-b69094be-2856-4c51-a447-eb048fd439d0': ['tcp://127.0.0.1:51020'], 'sample_pi_monte_carlo-d214656f-938f-4e02-98a4-bdb8accbc8fd': ['tcp://127.0.0.1:51036'], 'sample_pi_monte_carlo-9a6efe7b-33a5-4fa4-a663-f9f9d31951a5': ['tcp://127.0.0.1:51036'], 'sample_pi_monte_carlo-3eb20639-7889-4129-aec6-c8870be175bb': ['tcp://127.0.0.1:51034'], 'sample_pi_monte_carlo-a447b64e-f815-43e5-9075-098f8b174511': ['tcp://127.0.0.1:51036'], 'sample_pi_monte_carlo-ff593391-f5cd-470b-ad67-3e1b4b83b039': ['tcp://127.0.0.1:51028'], 'sample_pi_monte_carlo-5fa2433a-4a60-4e24-a6db-355dacbc49c6': ['tcp://127.0.0.1:51028'], 'sample_pi_monte_carlo-e684734a-9ac7-4758-a980-dc238aa6838c': ['tcp://127.0.0.1:51016'], 'sample_pi_monte_carlo-6c36725a-ae3e-410f-907b-fb2a2d30e026': ['tcp://127.0.0.1:51031'], 'sample_pi_monte_carlo-955d154a-d163-4147-a480-1baa23023a30': ['tcp://127.0.0.1:51034'], 'sample_pi_monte_carlo-7679dcc8-691b-43e4-a20e-3fa72d049b84': ['tcp://127.0.0.1:51025'], 'sample_pi_monte_carlo-af8427e8-2799-432a-b41f-0957df834291': ['tcp://127.0.0.1:51020'], 'sample_pi_monte_carlo-d6bc4341-8f3a-4c82-90a0-914057dbb79b': ['tcp://127.0.0.1:51034'], 'sample_pi_monte_carlo-ba4fb0ad-c025-4e1d-afd9-9aee4709ee56': ['tcp://127.0.0.1:51020'], 'sample_pi_monte_carlo-89226f7b-a43d-476a-a222-d6a56c90c84e': ['tcp://127.0.0.1:51020'], 'sample_pi_monte_carlo-7b070c72-8cd9-40be-998f-a2ceca45686b': ['tcp://127.0.0.1:51020'], 'sample_pi_monte_carlo-9f41f183-d0eb-4b3c-a07c-3101ad21b950': ['tcp://127.0.0.1:51034'], 'sample_pi_monte_carlo-c633bd36-1885-45ed-abef-c299203231e4': ['tcp://127.0.0.1:51031'], 'sample_pi_monte_carlo-6b7981f7-f201-4fc5-9704-124b32253067': ['tcp://127.0.0.1:51036'], 'sample_pi_monte_carlo-b7c65913-0c49-411d-874c-7f4f2bc472a9': ['tcp://127.0.0.1:51019'], 'sample_pi_monte_carlo-aa9d70a6-6b28-4f55-a121-bce27558820e': ['tcp://127.0.0.1:51020'], 'sample_pi_monte_carlo-eae31312-1be4-4a2e-adca-ee721b4cd121': ['tcp://127.0.0.1:51031'], 'sample_pi_monte_carlo-cd772f35-0623-433f-bb09-baabeb5f2052': ['tcp://127.0.0.1:51034'], 'sample_pi_monte_carlo-c4ba69aa-0869-47a2-8c10-a176666e59e8': ['tcp://127.0.0.1:51031'], 'sample_pi_monte_carlo-b46fa8e8-5bec-4f25-8a5b-c8948db87e16': ['tcp://127.0.0.1:51036']})
distributed.worker - WARNING - Could not find data: {'sample_pi_monte_carlo-c12c424e-7120-48b5-811f-1b232bef9a1a': ['tcp://127.0.0.1:51034']} on workers: [] (who_has: {'sample_pi_monte_carlo-5faeb8bd-4dca-4e01-a8bc-867c98e3cb94': ['tcp://127.0.0.1:51019'], 'sample_pi_monte_carlo-7964855f-729a-48bd-adb3-e69cd5f7a6ce': ['tcp://127.0.0.1:51031'], 'sample_pi_monte_carlo-9a5d98da-846e-4389-a679-28f3426c06b2': ['tcp://127.0.0.1:51028'], 'sample_pi_monte_carlo-dca8e394-bd19-4823-9ed0-54b0b7cb476b': ['tcp://127.0.0.1:51034'], 'sample_pi_monte_carlo-9fe0d532-cd72-4854-813c-e9ff47c321ed': ['tcp://127.0.0.1:51016'], 'sample_pi_monte_carlo-a85caa58-fd9b-4f3f-a094-df4d855af7c6': ['tcp://127.0.0.1:51016'], 'sample_pi_monte_carlo-3e5e04e8-d120-42b7-992a-7ed0ee75c2b7': ['tcp://127.0.0.1:51016'], 'sample_pi_monte_carlo-76d73639-61b1-41a9-9da7-077abbd363eb': ['tcp://127.0.0.1:51034'], 'sample_pi_monte_carlo-b7cbc503-39d6-4f5b-8a75-f96c866ff21a': ['tcp://127.0.0.1:51028'], 'sample_pi_monte_carlo-c73bf352-0bf2-4798-ae31-221c07398e53': ['tcp://127.0.0.1:51034'], 'sample_pi_monte_carlo-61217b96-c9cc-4cd8-a470-c08eb47f44e7': ['tcp://127.0.0.1:51034'], 'sample_pi_monte_carlo-56e9583d-afc8-4594-8e34-f9932e502999': ['tcp://127.0.0.1:51020'], 'sample_pi_monte_carlo-565ac4c4-f328-4ff6-90ec-17bd793ced0e': ['tcp://127.0.0.1:51028'], 'sample_pi_monte_carlo-b8f97451-a89a-4083-b8a7-9155d3bb14ed': ['tcp://127.0.0.1:51034'], 'sample_pi_monte_carlo-f96522b6-7237-4e2e-a2f8-69a2d22a6760': ['tcp://127.0.0.1:51031'], 'sample_pi_monte_carlo-94b607c1-542c-4d48-b00c-abb75bc48adb': ['tcp://127.0.0.1:51016'], 'sample_pi_monte_carlo-8048b1c3-bbe9-4f41-b42d-f61e96a8d198': ['tcp://127.0.0.1:51019'], 'sample_pi_monte_carlo-67630445-9022-415e-a75d-da0094b78a6b': ['tcp://127.0.0.1:51028'], 'sample_pi_monte_carlo-a84b78aa-aa14-4613-b3bc-d24a9bf6ee3a': ['tcp://127.0.0.1:51020'], 'sample_pi_monte_carlo-c12c424e-7120-48b5-811f-1b232bef9a1a': ['tcp://127.0.0.1:51034'], 'sample_pi_monte_carlo-d4df0795-95e4-4f0a-8900-e4339714ac0c': ['tcp://127.0.0.1:51025'], 'sample_pi_monte_carlo-42896e1f-1947-48f9-8b0c-e3cb9d9e08c1': ['tcp://127.0.0.1:51016'], 'sample_pi_monte_carlo-b3e14592-76c8-4f57-a385-fc803b3575ce': ['tcp://127.0.0.1:51016'], 'sample_pi_monte_carlo-a959e359-69c4-44fc-a15a-545439d11a72': ['tcp://127.0.0.1:51019'], 'sample_pi_monte_carlo-bfa43c43-aca5-4cfc-8d58-0e4feb7fbdf0': ['tcp://127.0.0.1:51031'], 'sample_pi_monte_carlo-92ea3b68-f541-4ad1-b912-6e0c67b985e4': ['tcp://127.0.0.1:51028'], 'sample_pi_monte_carlo-a43dcda8-cd72-49d4-b823-a32a96e690ee': ['tcp://127.0.0.1:51019'], 'sample_pi_monte_carlo-e46b9e93-a86c-4e20-99bf-a09cb349cf85': ['tcp://127.0.0.1:51016'], 'sample_pi_monte_carlo-b7a28199-22a3-464f-bfa3-581543935cf8': ['tcp://127.0.0.1:51031'], 'sample_pi_monte_carlo-601e03ad-e2e8-4053-9d5e-a9d306fbd43e': ['tcp://127.0.0.1:51034'], 'sample_pi_monte_carlo-88a5cec6-21e2-4e43-b4d1-9f525635156f': ['tcp://127.0.0.1:51019'], 'sample_pi_monte_carlo-f40744ea-1d69-4d95-9568-bcb09077a2c7': ['tcp://127.0.0.1:51020'], 'sample_pi_monte_carlo-63fd48eb-7229-432c-b8e1-9602013bb254': ['tcp://127.0.0.1:51019'], 'sample_pi_monte_carlo-85c3ab75-7fb0-4aca-ac91-0e7bdec64b15': ['tcp://127.0.0.1:51036'], 'sample_pi_monte_carlo-aee06abd-c075-436a-b543-6441c1a47740': ['tcp://127.0.0.1:51025'], 'sample_pi_monte_carlo-66e0cd68-3038-43e4-9c70-fff3909b7323': ['tcp://127.0.0.1:51028'], 'sample_pi_monte_carlo-677419d4-62cb-45aa-b58e-174236ab8c71': ['tcp://127.0.0.1:51016'], 'sample_pi_monte_carlo-c51e7a41-5b4c-4628-b04a-b0c5fd6f3487': ['tcp://127.0.0.1:51036'], 'sample_pi_monte_carlo-7383bf98-dc46-42b3-a59e-f94f33940cde': ['tcp://127.0.0.1:51016'], 'sample_pi_monte_carlo-e11cded8-e8c6-4fce-861f-9d726eb1acff': ['tcp://127.0.0.1:51031'], 'sample_pi_monte_carlo-5b8cc96f-d609-4bf3-8bc5-6fbab4d770fe': ['tcp://127.0.0.1:51019'], 'sample_pi_monte_carlo-b5a590d7-1f38-48c5-b98d-983ff09d8266': ['tcp://127.0.0.1:51019'], 'sample_pi_monte_carlo-ed2a09d9-b19e-49fb-ba6b-c3981c375a20': ['tcp://127.0.0.1:51016'], 'sample_pi_monte_carlo-3e725e2c-5612-4956-957c-0a06ae9432db': ['tcp://127.0.0.1:51036'], 'sample_pi_monte_carlo-9b06abaf-dddb-4672-aac7-f70b109fe0c4': ['tcp://127.0.0.1:51025']})
distributed.scheduler - WARNING - Communication failed during replication: {'status': 'missing-data', 'keys': {'sample_pi_monte_carlo-c12c424e-7120-48b5-811f-1b232bef9a1a': ('tcp://127.0.0.1:51034',)}}
distributed.scheduler - WARNING - Communication failed during replication: {'status': 'missing-data', 'keys': {'sample_pi_monte_carlo-8bd484b2-b924-46ea-8d39-01c4614c8f27': ('tcp://127.0.0.1:51025',)}}
distributed.utils - ERROR - 
Traceback (most recent call last):
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils.py", line 656, in log_errors
    yield
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 3513, in retire_workers
    lock=False,
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 3267, in replicate
    assert count > 0
AssertionError
distributed.core - ERROR - 
Traceback (most recent call last):
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 513, in handle_comm
    resupi is: 3.096
lt = await result
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 3513, in retire_workers
    lock=False,
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 3267, in replicate
    assert count > 0
AssertionError
distributed.utils - ERROR - 
Traceback (most recent call last):
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/utils.py", line 656, in log_errors
    yield
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/deploy/adaptive.py", line 186, in scale_down
    names=workers, remove=True, close_workers=True
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 790, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 660, in send_recv
    raise exc.with_traceback(tb)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 513, in handle_comm
    result = await result
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 3513, in retire_workers
    lock=False,
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 3267, in replicate
    assert count > 0
AssertionError
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x11086fe10>>, <Task finished coro=<AdaptiveCore.adapt() done, defined at /Users/jtomlinson/Projects/dask/distributed/distributed/deploy/adaptive_core.py:170> exception=AssertionError()>)
Traceback (most recent call last):
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/Users/jtomlinson/miniconda3/envs/dask/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/deploy/adaptive_core.py", line 194, in adapt
    await self.scale_down(**recommendations)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/deploy/adaptive.py", line 186, in scale_down
    names=workers, remove=True, close_workers=True
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 790, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 660, in send_recv
    raise exc.with_traceback(tb)
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/core.py", line 513, in handle_comm
    result = await result
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 3513, in retire_workers
    lock=False,
  File "/Users/jtomlinson/Projects/dask/distributed/distributed/scheduler.py", line 3267, in replicate
    assert count > 0
AssertionError
pi is: 3.14
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
```

Resolving patch:
diff --git a/distributed/deploy/adaptive_core.py b/distributed/deploy/adaptive_core.py
--- a/distributed/deploy/adaptive_core.py
+++ b/distributed/deploy/adaptive_core.py
@@ -78,10 +78,13 @@ def __init__(
         self.periodic_callback = None
 
         def f():
-            self.periodic_callback = PeriodicCallback(self.adapt, self.interval * 1000)
-            self.periodic_callback.start()
+            try:
+                self.periodic_callback.start()
+            except AttributeError:
+                pass
 
         if self.interval:
+            self.periodic_callback = PeriodicCallback(self.adapt, self.interval * 1000)
             try:
                 self.loop.add_callback(f)
             except AttributeError:


Test patch:
diff --git a/distributed/deploy/tests/test_adaptive.py b/distributed/deploy/tests/test_adaptive.py
--- a/distributed/deploy/tests/test_adaptive.py
+++ b/distributed/deploy/tests/test_adaptive.py
@@ -430,3 +430,22 @@ def test_adaptive_config():
         assert adapt.maximum == math.inf
         assert adapt.interval == 5
         assert adapt.wait_count == 8
+
+
+@pytest.mark.asyncio
+async def test_update_adaptive(cleanup):
+    async with LocalCluster(
+        0,
+        threads_per_worker=2,
+        memory_limit="3 GB",
+        scheduler_port=0,
+        silence_logs=False,
+        processes=False,
+        dashboard_address=None,
+        asynchronous=True,
+    ) as cluster:
+        first = cluster.adapt(maxmimum=1)
+        second = cluster.adapt(maxmimum=2)
+        await asyncio.sleep(0.2)
+        assert first.periodic_callback is None
+        assert second.periodic_callback.is_running()