Repository: dask/distributed

Issue:
Computation deadlocks due to worker rapidly running out of memory instead of spilling
The below script is pretty reliably triggering deadlocks. I'm sure this can be reduced further but I haven't had time to do so, yet. 

```python
import coiled.v2
from distributed import Client
cluster = coiled.v2.Cluster(
    n_workers=20
)
client = Client(cluster)

from distributed import Client
from dask.datasets import timeseries
ddf = timeseries(
    "2020",
    "2025",
    partition_freq='2w',
)
ddf2 = timeseries(
    "2020",
    "2023",
    partition_freq='2w',
)
def slowident(df):
    import random
    import time
    time.sleep(random.randint(1, 5))
    return df
               
while True:
    client.restart()
    demo1 = ddf.map_partitions(slowident)
    (demo1.x + demo1.y).mean().compute()

    demo2 = ddf.merge(ddf2)
    demo2 = demo2.map_partitions(slowident)
    (demo2.x + demo2.y).mean().compute()
```

We could confirm that version `2022.1.1` is *not* affected by this but it appears that all follow up versions might be affected (haven't tested all of them, can definitely confirm for `2022.4.0`)


Discussion:
Has anyone tried bisecting yet?

On Tue, Apr 12, 2022 at 10:20 AM Florian Jetter ***@***.***>
wrote:

> The below script is pretty reliably triggering deadlocks. I'm sure this
> can be reduced further but I haven't had time to do so, yet.
>
> import coiled.v2from distributed import Clientcluster = coiled.v2.Cluster(
>     n_workers=20
> )client = Client(cluster)
> from distributed import Clientfrom dask.datasets import timeseriesddf = timeseries(
>     "2020",
>     "2025",
>     partition_freq='2w',
> )ddf2 = timeseries(
>     "2020",
>     "2023",
>     partition_freq='2w',
> )def slowident(df):
>     import random
>     import time
>     time.sleep(random.randint(1, 5))
>     return df
>                while True:
>     client.restart()
>     demo1 = ddf.map_partitions(slowident)
>     (demo1.x + demo1.y).mean().compute()
>
>     demo2 = ddf.merge(ddf2)
>     demo2 = demo2.map_partitions(slowident)
>     (demo2.x + demo2.y).mean().compute()
>
> We could confirm that version 2022.1.1 is *not* affected by this but it
> appears that all follow up versions might be affected (haven't tested all
> of them, can definitely confirm for 2022.4.0)
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/6110>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTEFDNKXA5XGISTMBPLVEWIDNANCNFSM5THVRWOQ>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>

> is pretty reliably triggering

Do you have an estimate of what fraction of runs? As @mrocklin says, this script is ideal for a bisect (behind a timeout failure).
I'm currently trying to run this locally without Coiled.  I encourage other folks to hop on as well, but I'll at least try to run with this for the rest of the day when I have free time.
I'm also curious about how often this fails.  Currently running this locally and not running into anything.  I'm wondering how long I should wait before moving up to Coiled.
I'm getting these when running locally, but I suspect that they're due to being unclean on restart

<details>

<summary> Traceback </summary>


```python-traceback
Future exception was never retrieved
future: <Future finished exception=CommClosedError("Exception while trying to call remote method 'restart' before comm was established.")>
Traceback (most recent call last):
  File "/home/mrocklin/workspace/distributed/distributed/comm/tcp.py", line 226, in read
    frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mrocklin/workspace/distributed/distributed/core.py", line 864, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/mrocklin/workspace/distributed/distributed/core.py", line 709, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/mrocklin/workspace/distributed/distributed/comm/tcp.py", line 242, in read
    convert_stream_closed_error(self, e)
  File "/home/mrocklin/workspace/distributed/distributed/comm/tcp.py", line 150, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) rpc.restart local=tcp://127.0.0.1:51626 remote=tcp://127.0.0.1:33741>: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mrocklin/mambaforge/lib/python3.9/site-packages/tornado/gen.py", line 769, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/home/mrocklin/workspace/distributed/distributed/utils.py", line 220, in quiet
    yield task
  File "/home/mrocklin/mambaforge/lib/python3.9/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/home/mrocklin/workspace/distributed/distributed/core.py", line 867, in send_recv_from_rpc
    raise type(e)(
distributed.comm.core.CommClosedError: Exception while trying to call remote method 'restart' before comm was established.
2022-04-12 10:34:22,259 - tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x7f1dc605be80>>, <Task finished name='Task-33593' coro=<Scheduler.restart() done, defined at /home/mrocklin/workspace/distributed/distributed/scheduler.py:6061> exception=CommClosedError("Exception while trying to call remote method 'restart' before comm was established.")>)
Traceback (most recent call last):
  File "/home/mrocklin/workspace/distributed/distributed/comm/tcp.py", line 226, in read
    frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mrocklin/workspace/distributed/distributed/core.py", line 864, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/mrocklin/workspace/distributed/distributed/core.py", line 709, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/mrocklin/workspace/distributed/distributed/comm/tcp.py", line 242, in read
    convert_stream_closed_error(self, e)
  File "/home/mrocklin/workspace/distributed/distributed/comm/tcp.py", line 150, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) rpc.restart local=tcp://127.0.0.1:51462 remote=tcp://127.0.0.1:35655>: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mrocklin/mambaforge/lib/python3.9/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/home/mrocklin/mambaforge/lib/python3.9/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
    future.result()
  File "/home/mrocklin/workspace/distributed/distributed/scheduler.py", line 6116, in restart
    resps = await asyncio.wait_for(resps, timeout)
  File "/home/mrocklin/mambaforge/lib/python3.9/asyncio/tasks.py", line 481, in wait_for
    return fut.result()
  File "/home/mrocklin/workspace/distributed/distributed/utils.py", line 207, in All
    result = await tasks.next()
  File "/home/mrocklin/workspace/distributed/distributed/core.py", line 867, in send_recv_from_rpc
    raise type(e)(
distributed.comm.core.CommClosedError: Exception while trying to call remote method 'restart' before comm was established.
Task exception was never retrieved
future: <Task finished name='Task-33599' coro=<rpc.__getattr__.<locals>.send_recv_from_rpc() done, defined at /home/mrocklin/workspace/distributed/distributed/core.py:855> exception=CommClosedError("Exception while trying to call remote method 'restart' before comm was established.")>
Traceback (most recent call last):
  File "/home/mrocklin/workspace/distributed/distributed/comm/tcp.py", line 226, in read
    frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mrocklin/workspace/distributed/distributed/core.py", line 864, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/mrocklin/workspace/distributed/distributed/core.py", line 709, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/mrocklin/workspace/distributed/distributed/comm/tcp.py", line 242, in read
    convert_stream_closed_error(self, e)
  File "/home/mrocklin/workspace/distributed/distributed/comm/tcp.py", line 150, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) rpc.restart local=tcp://127.0.0.1:50694 remote=tcp://127.0.0.1:35275>: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mrocklin/workspace/distributed/distributed/core.py", line 867, in send_recv_from_rpc
    raise type(e)(
distributed.comm.core.CommClosedError: Exception while trying to call remote method 'restart' before comm was established.
Task exception was never retrieved
future: <Task finished name='Task-33600' coro=<rpc.__getattr__.<locals>.send_recv_from_rpc() done, defined at /home/mrocklin/workspace/distributed/distributed/core.py:855> exception=CommClosedError("Exception while trying to call remote method 'restart' before comm was established.")>
Traceback (most recent call last):
  File "/home/mrocklin/workspace/distributed/distributed/comm/tcp.py", line 226, in read
    frames_nbytes = await stream.read_bytes(fmt_size)
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mrocklin/workspace/distributed/distributed/core.py", line 864, in send_recv_from_rpc
    result = await send_recv(comm=comm, op=key, **kwargs)
  File "/home/mrocklin/workspace/distributed/distributed/core.py", line 709, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/home/mrocklin/workspace/distributed/distributed/comm/tcp.py", line 242, in read
    convert_stream_closed_error(self, e)
  File "/home/mrocklin/workspace/distributed/distributed/comm/tcp.py", line 150, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) rpc.restart local=tcp://127.0.0.1:45222 remote=tcp://127.0.0.1:37231>: Stream is closed

The above exception was the direct cause of the following exception:
```

</traceback>

Shifting to https://github.com/dask/distributed/issues/5951 , which seems easier to iterate on
Shifting back.  I'm now running this at scale on Coiled.  I'm not yet seeing a deadlock.  @fjetter can you say a bit more about how often you ran into this?
We reverted the default environment on coiled so the above will not trigger an issue anymore. With 2022.4.1 it will deadlock within seconds / few minutes
I've been running this on Coiled. It reliably deadlocks within 2-3 iterations.

<details><summary>Script</summary>

```python
import coiled.v2
from distributed import Client

cluster = coiled.v2.Cluster(
    name="florian-deadlock",
    n_workers=20,
    software="florian-deadlock",
    # environ=dict(
    #     DASK_LOGGING__DISTRIBUTED="debug",
    #     DASK_DISTRIBUTED__ADMIN__LOG_LENGTH="1000000"
    # )
)
client = Client(cluster)

from dask.datasets import timeseries

ddf = timeseries(
    "2020",
    "2025",
    partition_freq="2w",
)
ddf2 = timeseries(
    "2020",
    "2023",
    partition_freq="2w",
)


def slowident(df):
    import random
    import time

    time.sleep(random.randint(1, 5))
    return df


i = 1
while True:
    print(f"Iteration {i}")
    client.restart()
    demo1 = ddf.map_partitions(slowident)
    (demo1.x + demo1.y).mean().compute()

    demo2 = ddf.merge(ddf2)
    demo2 = demo2.map_partitions(slowident)
    (demo2.x + demo2.y).mean().compute()
    i += 1
```

```toml
[tool.poetry]
name = "florian-deadlock"
version = "0.1.0"
description = ""
authors = ["Gabe Joseph <gjoseph92@gmail.com>"]

[tool.poetry.dependencies]
python = "=3.9.1"
dask = {extras = ["complete"], version = "^2022.4.0"}
ipython = "^8.2.0"
coiled = "^0.0.73"
scheduler-profilers = {git = "https://github.com/gjoseph92/scheduler-profilers.git", rev = "main"}

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
```

```bash
#!/bin/bash

cat > postbuild.sh <<EOF
#!/bin/bash

pip install --no-deps "git+https://github.com/dask/distributed.git@main"

echo "export DASK_LOGGING__DISTRIBUTED=debug" >> ~/.bashrc
echo "export DASK_DISTRIBUTED__ADMIN__LOG_LENGTH=1000000" >> ~/.bashrc
EOF

poetry export --without-hashes > requirements.txt

coiled env create -n florian-deadlock --pip requirements.txt --post-build postbuild.sh

rm requirements.txt
rm postbuild.sh
```

----

</details>

I've tried distributed from main to get https://github.com/dask/distributed/pull/6112. Still deadlocks.

The overall pattern I'm seeing is at least 1 worker which is effectively dead, but hasn't been removed from the scheduler yet.

Other works seem to still want to fetch data from that worker.

<img width="1379" alt="Screen Shot 2022-04-14 at 11 39 17 AM" src="https://user-images.githubusercontent.com/3309802/163443871-3993e001-4066-4427-9926-1b2f8c2747f5.png">

```python
In [9]: client.run(lambda dask_worker: [ts.who_has for ts in dask_worker.data_needed], workers=['tls://10.6.0.70:34327'])
Out[9]: 
{'tls://10.6.0.70:34327': [{'tls://10.6.12.222:42533'},
  {'tls://10.6.12.222:42533'},
  {'tls://10.6.12.222:42533'}]}
# notice the `who_has` addresses are all the same, and all reference the unresponsive red worker above
```

Here are logs (via Coiled) from one of the unresponsive workers:
```python-traceback
2022-04-14 18:55:24,233 - distributed.worker - ERROR - Worker stream died during communication: tls://10.6.3.191:45383
Traceback (most recent call last):
  File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/iostream.py", line 1592, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
  File "/opt/conda/envs/coiled/lib/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/opt/conda/envs/coiled/lib/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 3019, in gather_dep
    response = await get_data_from_worker(
  File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 4320, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/utils_comm.py", line 381, in retry_operation
    return await retry(
  File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/utils_comm.py", line 366, in retry
    return await coro()
  File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 4300, in _get_data
    response = await send_recv(
  File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/core.py", line 709, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/tcp.py", line 242, in read
    convert_stream_closed_error(self, e)
  File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/tcp.py", line 148, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TLS (closed) Ephemeral Worker->Worker for gather local=tls://10.6.3.169:44316 remote=tls://10.6.3.191:45383>: ConnectionResetError: [Errno 104] Connection reset by peer
2022-04-14 18:56:15,295 - distributed.worker - INFO - Stopping worker at tls://10.6.3.169:43833
2022-04-14 18:56:15,318 - distributed.nanny - INFO - Worker closed
2022-04-14 18:56:16,231 - distributed.worker - INFO -       Start worker at:     tls://10.6.3.169:40133
2022-04-14 18:56:16,231 - distributed.worker - INFO -          Listening to:     tls://10.6.3.169:40133
2022-04-14 18:56:16,231 - distributed.worker - INFO -          dashboard at:           10.6.3.169:39063
2022-04-14 18:56:16,231 - distributed.worker - INFO - Waiting to connect to:   tls://34.221.160.71:8786
2022-04-14 18:56:16,231 - distributed.worker - INFO - -------------------------------------------------
2022-04-14 18:56:16,231 - distributed.worker - INFO -               Threads:                          2
2022-04-14 18:56:16,233 - distributed.worker - INFO -                Memory:                   3.78 GiB
2022-04-14 18:56:16,233 - distributed.worker - INFO -       Local Directory: /dask-worker-space/worker-c7603wej
2022-04-14 18:56:16,234 - distributed.worker - INFO - -------------------------------------------------
2022-04-14 18:56:16,477 - distributed.worker - WARNING - Mismatched versions found
+-------------+-----------------------+-----------------------+---------------------------------------+
| Package     | This Worker           | scheduler             | workers                               |
+-------------+-----------------------+-----------------------+---------------------------------------+
| blosc       | MISSING               | MISSING               | {None, 'MISSING'}                     |
| distributed | 2022.4.0+33.g41ecbca2 | 2022.4.0+33.g41ecbca2 | {'2022.4.0+33.g41ecbca2', '2022.4.0'} |
| python      | 3.9.12.final.0        | 3.9.12.final.0        | {'3.9.12.final.0', '3.9.1.final.0'}   |
+-------------+-----------------------+-----------------------+---------------------------------------+
2022-04-14 18:56:16,478 - distributed.worker - INFO - Starting Worker plugin coiled-aws-env
2022-04-14 18:56:16,480 - distributed.worker - INFO -         Registered to:   tls://34.221.160.71:8786
2022-04-14 18:56:16,482 - distributed.worker - INFO - -------------------------------------------------
2022-04-14 18:56:16,483 - distributed.core - INFO - Starting established connection
```
You'll notice:
* Connection to a different worker fails
* The worker shuts down for some reason. I don't know why. But the nanny restarts it, and it reconnects.

I'm seeing the nanny-restart for all three unresponsive workers in my most recent run. Other workers don't show that pattern. I feel like the reconnection could have something to do with it.

I'm also confused why the workers seem to think they've reconnected to the scheduler, yet the scheduler still sees their `last_seen` time as many minutes ago.

I have cluster dumps, but I've found them less useful so far than just poking at things manually.
I'm playing with this.  I've gotten only about as far as @gjoseph92 has.  I'm seeing the same thing.  I don't see any reason _why_ the worker would have restarted, which is also strange.  (I'm not seeing the same comm issue)

<details> 

<summary> Logs </summary>

```

Apr 14 22:35:49 ip-10-4-8-234 cloud-init[981]: {"msg":"Successfully pulled docker image","id":"7c4d42","username":"mrocklin","account":"mrocklin","level":20,"event_type":"vm_event","elapsed":50,"logging_context":{"user.username":"mrocklin","user.id":5,"user.is_authenticated":true}}COILED SERVER IS: https://cloud.coiled.io
Apr 14 22:35:50 ip-10-4-8-234 cloud-init[981]: Creating tmp_worker_1 ...
Apr 14 22:35:50 ip-10-4-8-234 systemd[1]: var-lib-docker-overlay2-2af07f68a80e4655a49a5f17b672a3a1c3a2cea0cbdec1b5e6efd24ba2de339b\x2dinit-merged.mount: Succeeded.
Apr 14 22:35:51 ip-10-4-8-234 systemd[1]: var-lib-docker-overlay2-2af07f68a80e4655a49a5f17b672a3a1c3a2cea0cbdec1b5e6efd24ba2de339b-merged.mount: Succeeded.
Apr 14 22:35:51 ip-10-4-8-234 containerd[477]: time="2022-04-14T22:35:51.834555695Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/ca0f33aa36f274dd26955208983421f1f1237cd88739058b216fc2a4980c988b pid=1926
Apr 14 22:35:51 ip-10-4-8-234 systemd[1]: run-docker-runtime\x2drunc-moby-ca0f33aa36f274dd26955208983421f1f1237cd88739058b216fc2a4980c988b-runc.AXTQoI.mount: Succeeded.
Apr 14 22:35:52 ip-10-4-8-234 cloud-init[981]: Creating tmp_worker_1 ... done
Apr 14 22:35:53 ip-10-4-8-234 cloud-init[981]: Attaching to tmp_worker_1
Apr 14 22:35:53 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:53,249 - distributed.preloading - INFO - Downloading preload at https://cloud.coiled.io/api/v2/cluster_facing/preload/worker
Apr 14 22:35:53 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:53,424 - distributed.preloading - INFO - Run preload setup function: https://cloud.coiled.io/api/v2/cluster_facing/preload/worker
Apr 14 22:35:53 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:53,433 - distributed.nanny - INFO -         Start Nanny at: 'tls://10.4.8.234:37539'
Apr 14 22:35:53 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:53,986 - distributed.worker - INFO -       Start worker at:     tls://10.4.8.234:34059
Apr 14 22:35:53 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:53,986 - distributed.worker - INFO -          Listening to:     tls://10.4.8.234:34059
Apr 14 22:35:53 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:53,987 - distributed.worker - INFO -          dashboard at:           10.4.8.234:45439
Apr 14 22:35:53 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:53,987 - distributed.worker - INFO - Waiting to connect to:   tls://18.215.64.220:8786
Apr 14 22:35:53 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:53,990 - distributed.worker - INFO - -------------------------------------------------
Apr 14 22:35:53 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:53,990 - distributed.worker - INFO -               Threads:                          2
Apr 14 22:35:53 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:53,990 - distributed.worker - INFO -                Memory:                   3.78 GiB
Apr 14 22:35:53 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:53,991 - distributed.worker - INFO -       Local Directory: /dask-worker-space/worker-hos3u3lc
Apr 14 22:35:53 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:53,991 - distributed.worker - INFO - -------------------------------------------------
Apr 14 22:35:54 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:54,522 - distributed.worker - WARNING - Mismatched versions found
Apr 14 22:35:54 ip-10-4-8-234 cloud-init[981]: +-------------+-----------------------+-----------------------+----------------------------------------------------+
Apr 14 22:35:54 ip-10-4-8-234 cloud-init[981]: | Package     | This Worker           | scheduler             | workers                                            |
Apr 14 22:35:54 ip-10-4-8-234 cloud-init[981]: +-------------+-----------------------+-----------------------+----------------------------------------------------+
Apr 14 22:35:54 ip-10-4-8-234 cloud-init[981]: | distributed | 2022.4.0+38.g0e937b3b | 2022.4.0+38.g0e937b3b | {'2022.4.0+33.g7dffe7d5', '2022.4.0+38.g0e937b3b'} |
Apr 14 22:35:54 ip-10-4-8-234 cloud-init[981]: +-------------+-----------------------+-----------------------+----------------------------------------------------+
Apr 14 22:35:54 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:54,523 - distributed.worker - INFO - Starting Worker plugin coiled-aws-env
Apr 14 22:35:54 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:54,523 - distributed.worker - INFO -         Registered to:   tls://18.215.64.220:8786
Apr 14 22:35:54 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:54,524 - distributed.worker - INFO - -------------------------------------------------
Apr 14 22:35:54 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:35:54,540 - distributed.core - INFO - Starting established connection
Apr 14 22:36:17 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:17,226 - distributed.worker - INFO - Stopping worker at tls://10.4.8.234:34059
Apr 14 22:36:17 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:17,241 - distributed.nanny - INFO - Worker closed
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:18,047 - distributed.worker - INFO -       Start worker at:     tls://10.4.8.234:42651
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:18,047 - distributed.worker - INFO -          Listening to:     tls://10.4.8.234:42651
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:18,047 - distributed.worker - INFO -          dashboard at:           10.4.8.234:46367
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:18,047 - distributed.worker - INFO - Waiting to connect to:   tls://18.215.64.220:8786
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:18,047 - distributed.worker - INFO - -------------------------------------------------
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:18,047 - distributed.worker - INFO -               Threads:                          2
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:18,047 - distributed.worker - INFO -                Memory:                   3.78 GiB
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:18,047 - distributed.worker - INFO -       Local Directory: /dask-worker-space/worker-m62zv9vy
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:18,049 - distributed.worker - INFO - -------------------------------------------------
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:18,364 - distributed.worker - WARNING - Mismatched versions found
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: +-------------+-----------------------+-----------------------+----------------------------------------------------+
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: | Package     | This Worker           | scheduler             | workers                                            |
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: +-------------+-----------------------+-----------------------+----------------------------------------------------+
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: | distributed | 2022.4.0+38.g0e937b3b | 2022.4.0+38.g0e937b3b | {'2022.4.0+33.g7dffe7d5', '2022.4.0+38.g0e937b3b'} |
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: +-------------+-----------------------+-----------------------+----------------------------------------------------+
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:18,364 - distributed.worker - INFO - Starting Worker plugin coiled-aws-env
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:18,365 - distributed.worker - INFO -         Registered to:   tls://18.215.64.220:8786
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:18,366 - distributed.worker - INFO - -------------------------------------------------
Apr 14 22:36:18 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:36:18,383 - distributed.core - INFO - Starting established connection
Apr 14 22:37:54 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:54,192 - distributed.worker - INFO - Stopping worker at tls://10.4.8.234:42651
Apr 14 22:37:54 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:54,215 - distributed.nanny - INFO - Worker closed
Apr 14 22:37:55 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:55,950 - distributed.worker - INFO -       Start worker at:     tls://10.4.8.234:33597
Apr 14 22:37:55 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:55,950 - distributed.worker - INFO -          Listening to:     tls://10.4.8.234:33597
Apr 14 22:37:55 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:55,950 - distributed.worker - INFO -          dashboard at:           10.4.8.234:33789
Apr 14 22:37:55 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:55,950 - distributed.worker - INFO - Waiting to connect to:   tls://18.215.64.220:8786
Apr 14 22:37:55 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:55,951 - distributed.worker - INFO - -------------------------------------------------
Apr 14 22:37:55 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:55,951 - distributed.worker - INFO -               Threads:                          2
Apr 14 22:37:55 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:55,951 - distributed.worker - INFO -                Memory:                   3.78 GiB
Apr 14 22:37:55 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:55,953 - distributed.worker - INFO -       Local Directory: /dask-worker-space/worker-ig_y0ab9
Apr 14 22:37:55 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:55,953 - distributed.worker - INFO - -------------------------------------------------
Apr 14 22:37:56 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:56,601 - distributed.worker - WARNING - Mismatched versions found
Apr 14 22:37:56 ip-10-4-8-234 cloud-init[981]: +-------------+-----------------------+-----------------------+----------------------------------------------------+
Apr 14 22:37:56 ip-10-4-8-234 cloud-init[981]: | Package     | This Worker           | scheduler             | workers                                            |
Apr 14 22:37:56 ip-10-4-8-234 cloud-init[981]: +-------------+-----------------------+-----------------------+----------------------------------------------------+
Apr 14 22:37:56 ip-10-4-8-234 cloud-init[981]: | distributed | 2022.4.0+38.g0e937b3b | 2022.4.0+38.g0e937b3b | {'2022.4.0+33.g7dffe7d5', '2022.4.0+38.g0e937b3b'} |
Apr 14 22:37:56 ip-10-4-8-234 cloud-init[981]: +-------------+-----------------------+-----------------------+----------------------------------------------------+
Apr 14 22:37:56 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:56,602 - distributed.worker - INFO - Starting Worker plugin coiled-aws-env
Apr 14 22:37:56 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:56,603 - distributed.worker - INFO -         Registered to:   tls://18.215.64.220:8786
Apr 14 22:37:56 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:56,604 - distributed.worker - INFO - -------------------------------------------------
Apr 14 22:37:56 ip-10-4-8-234 cloud-init[981]: 2022-04-14 22:37:56,605 - distributed.core - INFO - Starting established connection
```

</details>

I see a few things to inspect:

1.  Try logging way more information to explain why a worker might be closing (good to do anyway)
2. Maybe instrument the worker to check in on its heartbeat messages to the scheduler and start to complain if they're not being listened to
3.  See if maybe the event loop is stuck on something (also with printing to logs)

None of these will find the answer, but I think that they're all increase visibility.  These are my next steps.  I'm also open to other suggestions.
Also, two quick notes, my recent PRs don't seem to help with this.

If I use m5.large instances (my default type) this problem doesn't occur.  My guess is that there is some disk thing happening here.  We're running out of space and so failing hard when trying to touch disk.  I'm going to aim towards that as a cause and just try to increase visibility along the way.
cc @ntabris without SSDs Coiled instances have 10GB drives right now.  Is that right?
> without SSDs Coiled instances have 10GB drives right now. Is that right?

No, prod and staging are still putting 100GB boot (EBS) drives on all instances.
Good to know.  It doesn't seem like we could dump that much data this quickly with this workload.  I did throw up a test to see what happens with Disk stops cooperating.  We're not graceful here.  https://github.com/dask/distributed/pull/6128

I also tried to see what happens when we fail writing, and things were actually ok.
Oh, I'm being dumb.  The closed workers are due to the client.restart command :facepalm: 
Sorry, should have posted that. I was also confused until I actually read the code 😁
So if I set `freq="5s"`, effectively reducing the data by 80% then this issue doesn't arise.  I continue to suspect disk.
Setting `worker_options={"data": dict()}` (which should disable disk) I actually found that I stopped receiving heartbeats from *lots* of workers.

I'm going to switch away from `t3`s to `m5.larges`.  I now suspect that this is just due to Amazon's t-type nodes, which are known to have variable performance.  I wonder if maybe they're getting angry at us for abusing them.  I'll stress out disk to compensate.

Nope.  Still an issue with m5's.  t-type nodes and disk are both in the clear it seems.  I can reliably recreate the situation way more easily now though :slightly_smiling_face: 
Playing with triggering worker failures.  I was able to kill workers in a way such that they didn't come back somehow.

<details>

```
Apr 15 14:17:49 ip-10-4-4-178 cloud-init[1012]: 2022-04-15 14:17:49,992 - distributed.worker - INFO - Starting Worker plugin coiled-aws-env
Apr 15 14:17:49 ip-10-4-4-178 cloud-init[1012]: 2022-04-15 14:17:49,992 - distributed.worker - INFO -         Registered to:    tls://34.201.27.34:8786
Apr 15 14:17:49 ip-10-4-4-178 cloud-init[1012]: 2022-04-15 14:17:49,992 - distributed.worker - INFO - -------------------------------------------------
Apr 15 14:17:50 ip-10-4-4-178 cloud-init[1012]: 2022-04-15 14:17:49,994 - distributed.core - INFO - Starting established connection
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]: 2022-04-15 14:18:29,747 - distributed.worker - ERROR - Worker stream died during communication: tls://10.4.4.203:45141
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]: Traceback (most recent call last):
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/tcp.py", line 236, in read
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:     n = await stream.read_into(chunk)
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]: tornado.iostream.StreamClosedError: Stream is closed
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]: The above exception was the direct cause of the following exception:
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]: Traceback (most recent call last):
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 3021, in gather_dep
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:     response = await get_data_from_worker(
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 4325, in get_data_from_worker
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:     return await retry_operation(_get_data, operation="get_data_from_worker")
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/utils_comm.py", line 381, in retry_operation
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:     return await retry(
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/utils_comm.py", line 366, in retry
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:     return await coro()
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 4305, in _get_data
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:     response = await send_recv(
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/core.py", line 709, in send_recv
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:     response = await comm.read(deserializers=deserializers)
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/tcp.py", line 242, in read
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:     convert_stream_closed_error(self, e)
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/tcp.py", line 150, in convert_stream_closed_error
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]:     raise CommClosedError(f"in {obj}: {exc}") from exc
Apr 15 14:18:29 ip-10-4-4-178 cloud-init[1012]: distributed.comm.core.CommClosedError: in <TLS (closed) Ephemeral Worker->Worker for gather local=tls://10.4.4.178:34448 remote=tls://10.4.4.203:45141>: Stream is closed
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]: 2022-04-15 14:18:45,763 - distributed.worker - ERROR - Worker stream died during communication: tls://10.4.10.64:37185
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]: Traceback (most recent call last):
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/tcp.py", line 236, in read
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:     n = await stream.read_into(chunk)
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]: tornado.iostream.StreamClosedError: Stream is closed
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]: The above exception was the direct cause of the following exception:
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]: Traceback (most recent call last):
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 3021, in gather_dep
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:     response = await get_data_from_worker(
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 4325, in get_data_from_worker
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:     return await retry_operation(_get_data, operation="get_data_from_worker")
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/utils_comm.py", line 381, in retry_operation
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:     return await retry(
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/utils_comm.py", line 366, in retry
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:     return await coro()
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 4305, in _get_data
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:     response = await send_recv(
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/core.py", line 709, in send_recv
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:     response = await comm.read(deserializers=deserializers)
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/tcp.py", line 242, in read
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:     convert_stream_closed_error(self, e)
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/tcp.py", line 150, in convert_stream_closed_error
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]:     raise CommClosedError(f"in {obj}: {exc}") from exc
Apr 15 14:18:45 ip-10-4-4-178 cloud-init[1012]: distributed.comm.core.CommClosedError: in <TLS (closed) Ephemeral Worker->Worker for gather local=tls://10.4.4.178:36410 remote=tls://10.4.10.64:37185>: Stream is closed
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]: 2022-04-15 14:18:47,912 - distributed.worker - ERROR - Worker stream died during communication: tls://10.4.5.79:34281
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]: Traceback (most recent call last):
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/tcp.py", line 226, in read
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:     frames_nbytes = await stream.read_bytes(fmt_size)
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]: tornado.iostream.StreamClosedError: Stream is closed
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]: The above exception was the direct cause of the following exception:
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]: Traceback (most recent call last):
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 3021, in gather_dep
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:     response = await get_data_from_worker(
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 4325, in get_data_from_worker
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:     return await retry_operation(_get_data, operation="get_data_from_worker")
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/utils_comm.py", line 381, in retry_operation
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:     return await retry(
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/utils_comm.py", line 366, in retry
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:     return await coro()
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 4305, in _get_data
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:     response = await send_recv(
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/core.py", line 709, in send_recv
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:     response = await comm.read(deserializers=deserializers)
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/tcp.py", line 242, in read
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:     convert_stream_closed_error(self, e)
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/tcp.py", line 150, in convert_stream_closed_error
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]:     raise CommClosedError(f"in {obj}: {exc}") from exc
Apr 15 14:18:48 ip-10-4-4-178 cloud-init[1012]: distributed.comm.core.CommClosedError: in <TLS (closed) Ephemeral Worker->Worker for gather local=tls://10.4.4.178:56458 remote=tls://10.4.5.79:34281>: Stream is closed
Apr 15 14:18:51 ip-10-4-4-178 cloud-init[1012]: 2022-04-15 14:18:51,740 - distributed.worker_memory - WARNING - Worker exceeded 95% memory budget. Restarting
Apr 15 14:18:51 ip-10-4-4-178 cloud-init[1012]: 2022-04-15 14:18:51,841 - distributed.nanny - INFO - Worker process 30 was killed by signal 15
Apr 15 14:24:11 ip-10-4-4-178 systemd[1]: Starting Daily apt upgrade and clean activities...
Apr 15 14:24:15 ip-10-4-4-178 dbus-daemon[461]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.17' (uid=0 pid=1456 comm="/usr/bin/gdbus call --system --dest org.freedeskto" label="unconfined")
Apr 15 14:24:15 ip-10-4-4-178 systemd[1]: Starting PackageKit Daemon...
Apr 15 14:24:15 ip-10-4-4-178 PackageKit: daemon start
Apr 15 14:24:15 ip-10-4-4-178 dbus-daemon[461]: [system] Successfully activated service 'org.freedesktop.PackageKit'
Apr 15 14:24:15 ip-10-4-4-178 systemd[1]: Started PackageKit Daemon.
Apr 15 14:24:25 ip-10-4-4-178 kernel: [  534.068725] audit: type=1400 audit(1650032665.145:34): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/sbin/tcpdump" pid=2846 comm="apparmor_parser"
Apr 15 14:24:26 ip-10-4-4-178 systemd[1]: apt-daily-upgrade.service: Succeeded.
Apr 15 14:24:26 ip-10-4-4-178 systemd[1]: Finished Daily apt upgrade and clean activities.
```

</details>
After merging in a bunch of things to make the other failures go away I'm back at this issue.  I tried running with `data=dict()` again (no disk) and am still able to recreate the issue far more often.

I tried checking to see if the event loop was stuck on something by sending in a plugin that writes what's going on in a separate thread 

```python
from dask.distributed import WorkerPlugin
from dask.utils import parse_timedelta
import threading
import time
from distributed import profile
import sys

class PrintThread(WorkerPlugin):
    name = "print-event-loop-thread"
    def __init__(self, delay="10s"):
        self.delay = parse_timedelta(delay)
    
    async def setup(self, worker):
        self.worker = worker
        self.thread_id = threading.get_ident()
        self.stop = threading.Event()
        self._thread = threading.Thread(target=self.go, daemon=True)
        self._thread.start()
        
    def go(self):
        while True:
            frame = sys._current_frames()[self.thread_id]
            
            print("-" * 40)
            for line in profile.call_stack(frame):
                print(line, end="")
            
            if self.stop.wait(timeout=self.delay):
                break
            
    def teardown(self, worker):
        self.stop.set()
        self._thread.join()
            
client.register_worker_plugin(PrintThread("5s"))
```

And while we get interesting output in logs, we stop getting that output once the worker hangs.  So the problem isn't in the worker necessarily. I would expect this to run in Python pretty much no matter what.  The problem is deeper I think.
On restart the frozen workers are cleared out of the scheduler (which makes sense, it cleans metadata) but they don't come back.  This means that even the Nanny is borked.  Something is wrong with these instances I think.  Coiled folks, I might want to engage you all on this next week.
Example failed cluster: https://cloud.coiled.io/mrocklin/clusters/7589/details
Example failed worker: play-worker-821298da7e / i-0d288f69dd47e247e
Yup, I just changed the worker plugin to a nanny plugin and we lose track of any ongoing output.  Somehow this process has frozen in such a way that the network connection isn't severed (Dask would react well in this case) but that Python itself is also totally non-responsive.  This is atypical.
Sorry, and not just the worker process (which might have some crazy user code or something (but in this case it's just pandas)) but also the nanny process, which generally doesn't do anything at all.  Something has frozen the entire box somehow.

If I were a cloud provider, I might freeze VMs in this way if they were behaving badly.
I connected to the serial console and it was logging something like this quickly and repeatedly:

```
Apr 16 16:20:52 ip-10-4-7-226 cloud-init[1004]:   File "<string>", line 1, in <module>
--
Apr 16 16:20:52 ip-10-4-7-226 cloud-init[1004]: #011  File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
Apr 16 16:20:52 ip-10-4-7-226 cloud-init[1004]: #011exitcode = _main(fd, parent_sentinel)
Apr 16 16:20:52 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 129, in _main
Apr 16 16:20:52 ip-10-4-7-226 cloud-init[1004]: #011return self._bootstrap(parent_sentinel)
Apr 16 16:20:52 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
Apr 16 16:20:52 ip-10-4-7-226 cloud-init[1004]: #011self.run()
Apr 16 16:20:52 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 108, in run
Apr 16 16:20:53 ip-10-4-7-226 cloud-init[1004]: #011self._target(*self._args, **self._kwargs)
Apr 16 16:20:53 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/process.py", line 175, in _run
Apr 16 16:20:53 ip-10-4-7-226 cloud-init[1004]: #011target(*args, **kwargs)
Apr 16 16:20:53 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/nanny.py", line 907, in _run
Apr 16 16:20:53 ip-10-4-7-226 cloud-init[1004]: #011loop.run_sync(run)
Apr 16 16:20:53 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/ioloop.py", line 524, in run_sync
Apr 16 16:20:53 ip-10-4-7-226 cloud-init[1004]: #011self.start()
Apr 16 16:20:53 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
Apr 16 16:20:53 ip-10-4-7-226 cloud-init[1004]: #011self.asyncio_loop.run_forever()
Apr 16 16:20:53 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
Apr 16 16:20:53 ip-10-4-7-226 cloud-init[1004]: #011self._run_once()
Apr 16 16:20:53 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 1869, in _run_once
Apr 16 16:20:53 ip-10-4-7-226 cloud-init[1004]: #011event_list = self._selector.select(timeout)
Apr 16 16:20:53 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/selectors.py", line 469, in select
Apr 16 16:20:53 ip-10-4-7-226 cloud-init[1004]: #011fd_event_list = self._selector.poll(timeout, max_ev)
```

After a while dask did stop with message below and the VM started the shutdown process after that:

```
Apr 16 16:21:10 ip-10-4-7-226 cloud-init[1004]:   File "<string>", line 1, in <module>
--
Apr 16 16:21:10 ip-10-4-7-226 cloud-init[1004]: #011  File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
Apr 16 16:21:10 ip-10-4-7-226 cloud-init[1004]: #011exitcode = _main(fd, parent_sentinel)
Apr 16 16:21:10 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 129, in _main
Apr 16 16:21:10 ip-10-4-7-226 cloud-init[1004]: #011return self._bootstrap(parent_sentinel)
Apr 16 16:21:10 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
Apr 16 16:21:10 ip-10-4-7-226 cloud-init[1004]: #011self.run()
Apr 16 16:21:10 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 108, in run
Apr 16 16:21:10 ip-10-4-7-226 cloud-init[1004]: #011self._target(*self._args, **self._kwargs)
Apr 16 16:21:10 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/process.py", line 175, in _run
Apr 16 16:21:10 ip-10-4-7-226 cloud-init[1004]: #011target(*args, **kwargs)
Apr 16 16:21:10 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/nanny.py", line 907, in _run
Apr 16 16:21:10 ip-10-4-7-226 cloud-init[1004]: #011loop.run_sync(run)
Apr 16 16:21:10 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/ioloop.py", line 524, in run_sync
Apr 16 16:21:11 ip-10-4-7-226 cloud-init[1004]: #011self.start()
Apr 16 16:21:11 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
Apr 16 16:21:11 ip-10-4-7-226 cloud-init[1004]: #011self.asyncio_loop.run_forever()
Apr 16 16:21:11 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
Apr 16 16:21:11 ip-10-4-7-226 cloud-init[1004]: #011self._run_once()
Apr 16 16:21:11 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 1869, in _run_once
Apr 16 16:21:11 ip-10-4-7-226 cloud-init[1004]: #011event_list = self._selector.select(timeout)
Apr 16 16:21:11 ip-10-4-7-226 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/selectors.py", line 469, in select
Apr 16 16:21:11 ip-10-4-7-226 cloud-init[1004]: #011fd_event_list = self._selector.poll(timeout, max_ev)
Apr 16 16:21:11 ip-10-4-7-226 cloud-init[1004]: 2022-04-16 16:21:06,237 - distributed.nanny - INFO - Closing Nanny at 'tls://10.4.7.226:36233'. Report closure to scheduler: None
```

yeah, so the repeated thing is the plugin.  I'm asking it to report what the main event loop thread is doing every 10s.  My experience with the downloaded logs is that once the worker seems to stall, those logs stop coming.  Is what you're saying that it is still consistently printing every ten seconds?  Or is there possibly a gap of a few minutes?  If it's consistently printing every ten seconds then what is happening that they're no longer showing up in the downloaded logs?
I don't ask because I care about the downloadable logs (although I do) I ask because *something* is happening at that time that is causing the whole worker to appear to freeze.
Also, this can totally wait until next week.  
I was continuing to see that on the serial console, but it also now shows in CloudWatch. If the logs you were pulling from CloudWatch weren't getting updating while this was in progress, one possibility is that the log agent was behind on shipping the logs and once dask shutdown it was able to ship them (we have delay in VM shutdown specifically so it has a chance to ship off the final logs).
I'm not surprised by a delay of 30s or so.  I was surprised to see a delay of a few minutes.  The start of this larger delay corresponded with the bad event.  Do you know what might cause the log agent to lag behind?
Not sure about the delay. If this is fairly reproducible, my inclination would be to (next week) get things setup so we can SSH into the workers and see what we can see there.
Sounds good
I've also tried running this with larger instances (m5.large with 8GiB RAM), but asking Dask to keep the memory use down to 4 GiB, while also turning off pausing.  This forces Dask to get into all sorts of trouble with dumping data, restarting workers, etc, but we never run into the frozen worker issue. 

I am running into a serialization issue intermittently, but my guess is that it's due to my own mismatched environments:

```
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
File /opt/conda/envs/coiled/lib/python3.9/site-packages/dask/dataframe/__init__.py:15, in <module>

ImportError: cannot import name 'read_sql' from partially initialized module 'dask.dataframe.io' (most likely due to a circular import) (/opt/conda/envs/coiled/lib/python3.9/site-packages/dask/dataframe/io/__init__.py)

The above exception was the direct cause of the following exception:
```
I looked briefly at file descriptors and didn't find anything interesting.  Workers had modest number of open file descriptors (~50-60) open when they stopped reporting in (this is pretty close to middle of the range for the computation).
This node went dark for around two minutes.  Here is what is was doing just before that happened (dask disk management), and the OOM killer logs (not dask)

```

Apr 16 22:12:26 ip-10-4-6-179 cloud-init[1004]:   File "<string>", line 1, in <module>
Apr 16 22:12:26 ip-10-4-6-179 cloud-init[1004]: #011  File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
Apr 16 22:12:26 ip-10-4-6-179 cloud-init[1004]: #011exitcode = _main(fd, parent_sentinel)
Apr 16 22:12:26 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 129, in _main
Apr 16 22:12:26 ip-10-4-6-179 cloud-init[1004]: #011return self._bootstrap(parent_sentinel)
Apr 16 22:12:26 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
Apr 16 22:12:26 ip-10-4-6-179 cloud-init[1004]: #011self.run()
Apr 16 22:12:26 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 108, in run
Apr 16 22:12:26 ip-10-4-6-179 cloud-init[1004]: #011self._target(*self._args, **self._kwargs)
Apr 16 22:12:26 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/process.py", line 175, in _run
Apr 16 22:12:26 ip-10-4-6-179 cloud-init[1004]: #011target(*args, **kwargs)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/nanny.py", line 907, in _run
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011loop.run_sync(run)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/ioloop.py", line 524, in run_sync
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011self.start()
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011self.asyncio_loop.run_forever()
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011self._run_once()
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011handle._run()
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/events.py", line 80, in _run
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011self._context.run(self._callback, *self._args)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 1191, in handle_scheduler
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011await self.handle_stream(comm, every_cycle=[self.ensure_communicating])
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/core.py", line 625, in handle_stream
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011handler(**merge(extra, msg))
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 1727, in handle_free_keys
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011self.transitions(recommendations, stimulus_id=stimulus_id)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 2611, in transitions
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011a_recs, a_instructions = self._transition(
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 2523, in _transition
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011recs, instructions = func(ts, *args, stimulus_id=stimulus_id, **kwargs)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 2037, in transition_memory_released
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011recs, instructions = self.transition_generic_released(
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 1978, in transition_generic_released
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011self.release_key(ts.key, stimulus_id=stimulus_id)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 3244, in release_key
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011del self.data[key]
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/spill.py", line 217, in __delitem__
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011super().__delitem__(key)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/zict/buffer.py", line 99, in __delitem__
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011del self.fast[key]
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/zict/lru.py", line 112, in __delitem__
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011del self.d[key]
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: ----------------------------------------
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "<string>", line 1, in <module>
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011  File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011exitcode = _main(fd, parent_sentinel)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 129, in _main
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011return self._bootstrap(parent_sentinel)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011self.run()
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 108, in run
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011self._target(*self._args, **self._kwargs)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/process.py", line 175, in _run
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011target(*args, **kwargs)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/nanny.py", line 907, in _run
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011loop.run_sync(run)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/ioloop.py", line 524, in run_sync
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011self.start()
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011self.asyncio_loop.run_forever()
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011self._run_once()
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011handle._run()
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/events.py", line 80, in _run
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011self._context.run(self._callback, *self._args)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 1636, in get_data
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011compressed = await comm.write(msg, serializers=serializers)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/tcp.py", line 271, in write
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011frames = await to_frames(
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/utils.py", line 70, in to_frames
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011return _to_frames()
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/utils.py", line 53, in _to_frames
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011return list(protocol.dumps(msg, **kwargs))
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/core.py", line 118, in dumps
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011frames[0] = msgpack.dumps(msg, default=_encode_default, use_bin_type=True)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/msgpack/__init__.py", line 35, in packb
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011return Packer(**kwargs).pack(o)
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/core.py", line 109, in _encode_default
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011frames.extend(create_serialized_sub_frames(obj))
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/core.py", line 67, in create_serialized_sub_frames
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]: #011sub_header, sub_frames = serialize_and_split(
Apr 16 22:12:36 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/serialize.py", line 431, in serialize_and_split
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011header, frames = serialize(x, serializers, on_error, context)
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/serialize.py", line 258, in serialize
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011return serialize(
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/serialize.py", line 336, in serialize
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011header, frames = dumps(x, context=context) if wants_context else dumps(x)
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/serialize.py", line 63, in pickle_dumps
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011frames[0] = pickle.dumps(
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/pickle.py", line 40, in dumps
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011result = pickle.dumps(x, **dump_kwargs)
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: ----------------------------------------
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "<string>", line 1, in <module>
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011  File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011exitcode = _main(fd, parent_sentinel)
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 129, in _main
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011return self._bootstrap(parent_sentinel)
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011self.run()
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 108, in run
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011self._target(*self._args, **self._kwargs)
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/process.py", line 175, in _run
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011target(*args, **kwargs)
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/nanny.py", line 907, in _run
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011loop.run_sync(run)
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/ioloop.py", line 524, in run_sync
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011self.start()
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011self.asyncio_loop.run_forever()
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011self._run_once()
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011handle._run()
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/events.py", line 80, in _run
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011self._context.run(self._callback, *self._args)
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker_memory.py", line 171, in memory_monitor
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011await self._maybe_spill(worker, memory)
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker_memory.py", line 250, in _maybe_spill
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011weight = data.evict()
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/spill.py", line 211, in evict
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011_, _, weight = self.fast.evict()
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/zict/lru.py", line 100, in evict
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011cb(k, v)
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/zict/buffer.py", line 62, in fast_to_slow
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011self.slow[key] = value
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/spill.py", line 284, in __setitem__
Apr 16 22:12:37 ip-10-4-6-179 cloud-init[1004]: #011pickled = self.dump(value)  # type: ignore
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262359] systemd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262367] CPU: 0 PID: 1 Comm: systemd Not tainted 5.13.0-1021-aws #23~20.04.2-Ubuntu
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262371] Hardware name: Amazon EC2 m5.large/, BIOS 1.0 10/16/2017
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262373] Call Trace:
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262375]  <TASK>
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262377]  dump_stack+0x7d/0x9c
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262384]  dump_header+0x4f/0x1f6
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262390]  oom_kill_process.cold+0xb/0x10
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262394]  out_of_memory+0x1cf/0x520
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262399]  __alloc_pages_slowpath.constprop.0+0xc86/0xd50
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262406]  __alloc_pages+0x2ce/0x320
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262409]  alloc_pages+0x87/0xe0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262412]  __page_cache_alloc+0x89/0xb0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262417]  pagecache_get_page+0x218/0x4d0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262421]  filemap_fault+0x5bf/0x9b0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262424]  ? do_set_pte+0xc8/0x140
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262430]  ? next_uptodate_page+0x160/0x270
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262434]  ext4_filemap_fault+0x32/0x50
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262439]  __do_fault+0x3e/0xc0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262443]  do_fault+0x1e6/0x400
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262446]  __handle_mm_fault+0x619/0x8e0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262451]  handle_mm_fault+0xda/0x2b0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262455]  do_user_addr_fault+0x1bb/0x650
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262458]  exc_page_fault+0x7d/0x170
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262462]  ? asm_exc_page_fault+0x8/0x30
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262467]  asm_exc_page_fault+0x1e/0x30
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262470] RIP: 0033:0x55c2fd7cf7a0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262477] Code: Unable to access opcode bytes at RIP 0x55c2fd7cf776.
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262478] RSP: 002b:00007ffe3d71d128 EFLAGS: 00010202
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262481] RAX: ffffffffffffffff RBX: 0000000000000000 RCX: 000000000004b120
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262483] RDX: 0000000052b0a1e0 RSI: 000000005d6b36e0 RDI: 000055c2febc2bd0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262485] RBP: 000055c2fea9a990 R08: 00007ffe3d79f080 R09: 000000000000056b
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262486] R10: 00007ffe3d79f090 R11: 00000000000a2424 R12: 000055c2febcefe0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262488] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262491]  </TASK>
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262492] Mem-Info:
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262494] active_anon:1139 inactive_anon:1893734 isolated_anon:0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262494]  active_file:15 inactive_file:222 isolated_file:21
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262494]  unevictable:5751 dirty:0 writeback:0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262494]  slab_reclaimable:7066 slab_unreclaimable:8391
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262494]  mapped:2056 shmem:234 pagetables:4816 bounce:0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262494]  free:25084 free_pcp:541 free_cma:0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262500] Node 0 active_anon:4556kB inactive_anon:7574936kB active_file:60kB inactive_file:888kB unevictable:23004kB isolated(anon):0kB isolated(file):84kB mapped:8224kB dirty:0kB writeback:0kB shmem:936kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 1988608kB writeback_tmp:0kB kernel_stack:3520kB pagetables:19264kB all_unreclaimable? no
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262506] Node 0 DMA free:14336kB min:132kB low:164kB high:196kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262512] lowmem_reserve[]: 0 2945 7627 7627 7627
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262517] Node 0 DMA32 free:44668kB min:26044kB low:32552kB high:39060kB reserved_highatomic:0KB active_anon:0kB inactive_anon:2970348kB active_file:136kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129256kB managed:3029416kB mlocked:0kB bounce:0kB free_pcp:360kB local_pcp:32kB free_cma:0kB
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262523] lowmem_reserve[]: 0 0 4682 4682 4682
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262527] Node 0 Normal free:41332kB min:41404kB low:51752kB high:62100kB reserved_highatomic:0KB active_anon:4556kB inactive_anon:4604588kB active_file:212kB inactive_file:844kB unevictable:23004kB writepending:0kB present:4935680kB managed:4794892kB mlocked:18468kB bounce:0kB free_pcp:1804kB local_pcp:512kB free_cma:0kB
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262533] lowmem_reserve[]: 0 0 0 0 0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262538] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (M) 3*4096kB (M) = 14336kB
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262552] Node 0 DMA32: 17*4kB (UME) 44*8kB (UME) 9*16kB (UME) 12*32kB (UME) 39*64kB (UME) 40*128kB (UE) 21*256kB (UE) 19*512kB (UME) 19*1024kB (UME) 1*2048kB (E) 0*4096kB = 45172kB
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262572] Node 0 Normal: 414*4kB (UME) 584*8kB (UME) 260*16kB (UME) 346*32kB (UME) 131*64kB (UME) 66*128kB (UE) 11*256kB (UME) 2*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 42232kB
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262590] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262593] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262594] 3696 total pagecache pages
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262595] 0 pages in swap cache
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262596] Swap cache stats: add 0, delete 0, find 0/0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262598] Free swap  = 0kB
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262599] Total swap = 0kB
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262600] 2020232 pages RAM
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262601] 0 pages HighMem/MovableOnly
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262602] 60315 pages reserved
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262603] 0 pages hwpoisoned
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262603] Tasks state (memory values in pages):
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262604] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262609] [    165]     0   165     9227      915   102400        0          -250 systemd-journal
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262614] [    198]     0   198     4973      815    69632        0         -1000 systemd-udevd
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262618] [    289]     0   289    70035     4487    98304        0         -1000 multipathd
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262621] [    329]   102   329    22493      741    73728        0             0 systemd-timesyn
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262625] [    402]   100   402     6678      899    77824        0             0 systemd-network
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262628] [    405]   101   405     5967     1647    86016        0             0 systemd-resolve
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262630] [    451]     0   451    60259      992   102400        0             0 accounts-daemon
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262634] [    452]     0   452      637      166    45056        0             0 acpid
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262637] [    453]   997   453   192636     3423   241664        0             0 amazon-cloudwat
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262640] [    459]     0   459     2136      471    53248        0             0 cron
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262643] [    461]   103   461     1895      882    57344        0          -900 dbus-daemon
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262645] [    468]     0   468    20458      538    57344        0             0 irqbalance
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262648] [    469]     0   469     7300     2742    90112        0             0 networkd-dispat
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262651] [    471]     0   471    59106      666    98304        0             0 polkitd
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262654] [    472]   104   472    56125      918    81920        0             0 rsyslogd
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262657] [    474]     0   474   327263     1603   200704        0             0 amazon-ssm-agen
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262659] [    476]     0   476   199934     3882   229376        0          -900 snapd
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262662] [    478]     0   478     4136      766    73728        0             0 systemd-logind
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262665] [    480]     0   480    98692     1154   139264        0             0 udisksd
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262667] [    485]     0   485      950      525    49152        0             0 atd
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262670] [    487]     0   487   317539     3792   270336        0          -999 containerd
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262673] [    504]     0   504     1840      437    53248        0             0 agetty
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262676] [    520]     0   520     1459      387    49152        0             0 agetty
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262678] [    560]     0   560    27028     2719   102400        0             0 unattended-upgr
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262681] [    568]     0   568    78585     1042   110592        0             0 ModemManager
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262684] [    649]     0   649     3045      842    61440        0         -1000 sshd
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262687] [    678]     0   678   382538     8134   483328        0          -500 dockerd
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262690] [    726]     0   726      624      144    40960        0             0 bpfilter_umh
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262693] [    981]     0   981   330315     2734   225280        0             0 ssm-agent-worke
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262696] [    999]     0   999    12403     8406   135168        0             0 cloud-init
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262699] [   1003]     0  1003      654      120    45056        0             0 sh
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262702] [   1004]     0  1004     1811       27    53248        0             0 tee
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262705] [   1005]     0  1005      654      119    40960        0             0 runcmd
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262707] [   1011]     0  1011     2858      700    57344        0             0 sudo
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262710] [   1012]     0  1012     2176      512    49152        0             0 run_coiled
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262713] [   1016]     0  1016     2156      147    49152        0             0 bash
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262716] [   1062]     0  1062     2176      498    53248        0             0 notify_coiled
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262718] [   1063]     0  1063     1889     1605    53248        0             0 docker-compose
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262721] [   1064]     0  1064     1811       16    49152        0             0 tee
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262724] [   1065]     0  1065    73387     8201   339968        0             0 docker-compose
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262726] [   1096]     0  1096   177758      610   110592        0          -998 containerd-shim
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262729] [   1119]     0  1119   168327    11240   253952        0             0 python
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262732] [   1143]     0  1143     3421     1343    65536        0             0 python
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262735] [   3178]     0  3178  2135855  1830505 15339520        0             0 python
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262738] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/5dc36d4084c4ea2a711a290e94e256ebb19cf554c9a253bff3ad5d978b4a32b8,task=python,pid=3178,uid=0
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.262774] Out of memory: Killed process 3178 (python) total-vm:8543420kB, anon-rss:7322012kB, file-rss:0kB, shmem-rss:8kB, UID:0 pgtables:14980kB oom_score_adj:0
Apr 16 22:14:48 ip-10-4-6-179 cloud-init[1004]: 2022-04-16 22:14:48,224 - distributed.core - INFO - Event loop was unresponsive in Nanny for 121.77s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
Apr 16 22:14:48 ip-10-4-6-179 kernel: [ 1388.725552] oom_reaper: reaped process 3178 (python), now anon-rss:0kB, file-rss:0kB, shmem-rss:12kB
Apr 16 22:14:48 ip-10-4-6-179 cloud-init[1004]: 2022-04-16 22:14:48,491 - distributed.nanny - INFO - Worker process 150 was killed by signal 9
Apr 16 22:14:48 ip-10-4-6-179 cloud-init[1004]: 2022-04-16 22:14:48,613 - distributed.nanny - WARNING - Restarting worker
Apr 16 22:14:50 ip-10-4-6-179 cloud-init[1004]: 2022-04-16 22:14:50,589 - distributed.worker - INFO -       Start worker at:     tls://10.4.6.179:38211
Apr 16 22:14:50 ip-10-4-6-179 cloud-init[1004]: 2022-04-16 22:14:50,589 - distributed.worker - INFO -          Listening to:     tls://10.4.6.179:38211
Apr 16 22:14:50 ip-10-4-6-179 cloud-init[1004]: 2022-04-16 22:14:50,589 - distributed.worker - INFO -          dashboard at:           10.4.6.179:33771
Apr 16 22:14:50 ip-10-4-6-179 cloud-init[1004]: 2022-04-16 22:14:50,589 - distributed.worker - INFO - Waiting to connect to:    tls://44.200.14.55:8786
Apr 16 22:14:50 ip-10-4-6-179 cloud-init[1004]: 2022-04-16 22:14:50,589 - distributed.worker - INFO - -------------------------------------------------
Apr 16 22:14:50 ip-10-4-6-179 cloud-init[1004]: 2022-04-16 22:14:50,589 - distributed.worker - INFO -               Threads:                          2
Apr 16 22:14:50 ip-10-4-6-179 cloud-init[1004]: 2022-04-16 22:14:50,597 - distributed.worker - INFO -                Memory:                   7.48 GiB
Apr 16 22:14:50 ip-10-4-6-179 cloud-init[1004]: 2022-04-16 22:14:50,597 - distributed.worker - INFO -       Local Directory: /scratch/dask-worker-space/worker-qek3rqt8
Apr 16 22:14:50 ip-10-4-6-179 cloud-init[1004]: 2022-04-16 22:14:50,597 - distributed.worker - INFO - -------------------------------------------------
Apr 16 22:14:51 ip-10-4-6-179 cloud-init[1004]: 2022-04-16 22:14:51,593 - distributed.worker - WARNING - Mismatched versions found
```

https://cloud.coiled.io/mrocklin/clusters/7596/details

mrocklin-880b2f98-e-worker-cad561d427 - started - Successfully phoned home
instance - i-054e536faaea722df - ready - 


Another

<details>

```
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: ----------------------------------------
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "<string>", line 1, in <module>
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011  File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011exitcode = _main(fd, parent_sentinel)
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 129, in _main
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011return self._bootstrap(parent_sentinel)
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011self.run()
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 108, in run
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011self._target(*self._args, **self._kwargs)
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/process.py", line 175, in _run
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011target(*args, **kwargs)
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/nanny.py", line 907, in _run
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011loop.run_sync(run)
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/ioloop.py", line 524, in run_sync
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011self.start()
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011self.asyncio_loop.run_forever()
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011self._run_once()
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011handle._run()
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/events.py", line 80, in _run
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011self._context.run(self._callback, *self._args)
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker_memory.py", line 171, in memory_monitor
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011await self._maybe_spill(worker, memory)
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker_memory.py", line 250, in _maybe_spill
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011weight = data.evict()
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/spill.py", line 211, in evict
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011_, _, weight = self.fast.evict()
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/zict/lru.py", line 100, in evict
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011cb(k, v)
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/zict/buffer.py", line 62, in fast_to_slow
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011self.slow[key] = value
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/spill.py", line 284, in __setitem__
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011pickled = self.dump(value)  # type: ignore
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/serialize.py", line 630, in serialize_bytelist
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011header, frames = serialize_and_split(x, **kwargs)
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/serialize.py", line 431, in serialize_and_split
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011header, frames = serialize(x, serializers, on_error, context)
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/serialize.py", line 336, in serialize
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011header, frames = dumps(x, context=context) if wants_context else dumps(x)
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/serialize.py", line 63, in pickle_dumps
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011frames[0] = pickle.dumps(
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/pickle.py", line 40, in dumps
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011result = pickle.dumps(x, **dump_kwargs)
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: ----------------------------------------
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "<string>", line 1, in <module>
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011  File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011exitcode = _main(fd, parent_sentinel)
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 129, in _main
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011return self._bootstrap(parent_sentinel)
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
Apr 16 22:12:45 ip-10-4-7-213 cloud-init[1006]: #011self.run()
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 108, in run
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011self._target(*self._args, **self._kwargs)
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/process.py", line 175, in _run
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011target(*args, **kwargs)
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/nanny.py", line 907, in _run
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011loop.run_sync(run)
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/ioloop.py", line 524, in run_sync
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011self.start()
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011self.asyncio_loop.run_forever()
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011self._run_once()
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011handle._run()
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/events.py", line 80, in _run
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011self._context.run(self._callback, *self._args)
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker_memory.py", line 171, in memory_monitor
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011await self._maybe_spill(worker, memory)
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker_memory.py", line 250, in _maybe_spill
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011weight = data.evict()
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/spill.py", line 211, in evict
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011_, _, weight = self.fast.evict()
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/zict/lru.py", line 100, in evict
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011cb(k, v)
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/zict/buffer.py", line 62, in fast_to_slow
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011self.slow[key] = value
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/spill.py", line 284, in __setitem__
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011pickled = self.dump(value)  # type: ignore
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/serialize.py", line 630, in serialize_bytelist
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011header, frames = serialize_and_split(x, **kwargs)
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/serialize.py", line 431, in serialize_and_split
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011header, frames = serialize(x, serializers, on_error, context)
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/serialize.py", line 336, in serialize
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011header, frames = dumps(x, context=context) if wants_context else dumps(x)
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/serialize.py", line 63, in pickle_dumps
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011frames[0] = pickle.dumps(
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/pickle.py", line 40, in dumps
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011result = pickle.dumps(x, **dump_kwargs)
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/pandas/core/internals/blocks.py", line 238, in mgr_locs
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011@property
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: ----------------------------------------
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "<string>", line 1, in <module>
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011  File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011exitcode = _main(fd, parent_sentinel)
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/spawn.py", line 129, in _main
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011return self._bootstrap(parent_sentinel)
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011self.run()
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/multiprocessing/process.py", line 108, in run
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011self._target(*self._args, **self._kwargs)
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/process.py", line 175, in _run
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011target(*args, **kwargs)
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/nanny.py", line 907, in _run
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011loop.run_sync(run)
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/ioloop.py", line 524, in run_sync
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011self.start()
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011self.asyncio_loop.run_forever()
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011self._run_once()
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011handle._run()
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/asyncio/events.py", line 80, in _run
Apr 16 22:12:46 ip-10-4-7-213 cloud-init[1006]: #011self._context.run(self._callback, *self._args)
Apr 16 22:17:28 ip-10-4-7-213 systemd[1]: snapd.service: Watchdog timeout (limit 5min)!
Apr 16 22:17:29 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:13:07,210 - distributed.core - INFO - Event loop was unresponsive in Nanny for 17.77s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
Apr 16 22:17:29 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:17:29,065 - distributed.core - INFO - Event loop was unresponsive in Nanny for 261.86s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
Apr 16 22:17:29 ip-10-4-7-213 CRON[3225]: pam_unix(cron:session): session opened for user root by (uid=0)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: SIGABRT: abort
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: PC=0x5606fe5f447d m=2 sigcode=0
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: goroutine 0 [idle]:
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: runtime.usleep(0x16700000a00, 0xd00000000, 0x1, 0xa0000000008, 0x167e2545a64, 0x0, 0x0, 0x0, 0x39, 0x0, ...)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/runtime/sys_linux_amd64.s:131 +0x3d
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: runtime.sysmon()
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/runtime/proc.go:4296 +0x8f
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: runtime.mstart1()
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/runtime/proc.go:1201 +0xc7
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: runtime.mstart()
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/runtime/proc.go:1167 +0x70
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: goroutine 1 [select (scan), 25 minutes]:
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: main.run(0xc00022cfc0, 0x0, 0x0)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/cmd/snapd/main.go:153 +0x4e5
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: main.main()
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/cmd/snapd/main.go:64 +0x10e
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: goroutine 6 [syscall, 25 minutes]:
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: os/signal.signal_recv(0x0)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/runtime/sigqueue.go:147 +0x9e
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: os/signal.loop()
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/os/signal/signal_unix.go:23 +0x24
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: created by os/signal.init.0
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/os/signal/signal_unix.go:29 +0x43
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: goroutine 35 [select]:
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: main.runWatchdog.func1(0xc00046fa40, 0xc0001049a0)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/cmd/snapd/main.go:93 +0xe5
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: created by main.runWatchdog
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/cmd/snapd/main.go:91 +0x1e2
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: goroutine 34 [IO wait, 25 minutes]:
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: internal/poll.runtime_pollWait(0x7f09ff1b4f08, 0x72, 0x0)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/runtime/netpoll.go:184 +0x57
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: internal/poll.(*pollDesc).wait(0xc0000e2818, 0x72, 0x0, 0x0, 0x5606fefb38b7)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/internal/poll/fd_poll_runtime.go:87 +0x47
Apr 16 22:17:29 ip-10-4-7-213 systemd[1]: snapd.service: Killing process 476 (snapd) with signal SIGABRT.
Apr 16 22:17:29 ip-10-4-7-213 CRON[3228]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Apr 16 22:17:29 ip-10-4-7-213 CRON[3225]: pam_unix(cron:session): session closed for user root
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: internal/poll.(*pollDesc).waitRead(...)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/internal/poll/fd_poll_runtime.go:92
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: internal/poll.(*FD).Accept(0xc0000e2800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/internal/poll/fd_unix.go:384 +0x1fa
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: net.(*netFD).accept(0xc0000e2800, 0x203000, 0xc0001d5cb0, 0x5606fe5c1b73)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/net/fd_unix.go:238 +0x44
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: net.(*UnixListener).accept(0xc000245800, 0x5606ff863000, 0xc0001d5ce8, 0xc0001d5cc0)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/net/unixsock_posix.go:162 +0x34
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: net.(*UnixListener).Accept(0xc000245800, 0xc0002e4088, 0x8, 0xc0001d5cc0, 0x5606fef9a6e5)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/net/unixsock.go:260 +0x49
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: github.com/snapcore/snapd/daemon.(*ucrednetListener).Accept(0xc000245920, 0xc0001d5da8, 0x18, 0xc000293380, 0x5606fe88ac76)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/daemon/ucrednet.go:112 +0x4a
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: net/http.(*Server).Serve(0xc0002ca2a0, 0x5606ff89b8c0, 0xc000245920, 0x0, 0x0)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/net/http/server.go:2896 +0x282
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: github.com/snapcore/snapd/daemon.(*Daemon).Start.func1(0x5606fe597cf5, 0xc0002427e0)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/daemon/daemon.go:377 +0x56
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: gopkg.in/tomb%2ev2.(*Tomb).run(0xc0001049f0, 0xc0003f5160)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2d
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: created by gopkg.in/tomb%2ev2.(*Tomb).Go
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xc9
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: goroutine 16 [select]:
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: github.com/snapcore/snapd/overlord.(*Overlord).Loop.func1(0x5606fe597cf5, 0xc000242b40)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/overlord/overlord.go:405 +0x17a
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: gopkg.in/tomb%2ev2.(*Tomb).run(0xc0000a1f90, 0xc00000ee40)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2d
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: created by gopkg.in/tomb%2ev2.(*Tomb).Go
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xc9
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: goroutine 15 [runnable]:
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: github.com/snapcore/snapd/overlord/standby.(*StandbyOpinions).CanStandby(0xc00046f950, 0x4a817c8000)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/overlord/standby/standby.go:52 +0x2ff
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: github.com/snapcore/snapd/overlord/standby.(*StandbyOpinions).Start.func1(0xc00046f950)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/overlord/standby/standby.go:100 +0xab
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: created by github.com/snapcore/snapd/overlord/standby.(*StandbyOpinions).Start
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/overlord/standby/standby.go:96 +0xaa
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: goroutine 67 [select, 25 minutes]:
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: github.com/snapcore/snapd/overlord/ifacestate/udevmonitor.(*Monitor).Run.func1(0xc0006e4de0, 0xc000680f60)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/overlord/ifacestate/udevmonitor/udevmon.go:147 +0x329
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: gopkg.in/tomb%2ev2.(*Tomb).run(0xc000598d80, 0xc000076900)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2d
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: created by gopkg.in/tomb%2ev2.(*Tomb).Go
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xc9
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: goroutine 36 [IO wait, 25 minutes]:
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: internal/poll.runtime_pollWait(0x7f09ff1b4d68, 0x72, 0x1000)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/runtime/netpoll.go:184 +0x57
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: internal/poll.(*pollDesc).wait(0xc000256118, 0x72, 0x0, 0x10, 0xc000139320)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/internal/poll/fd_poll_runtime.go:87 +0x47
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: internal/poll.(*pollDesc).waitRead(...)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/internal/poll/fd_poll_runtime.go:92
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: internal/poll.(*FD).ReadMsg(0xc000256100, 0xc00040bd20, 0x10, 0x10, 0xc000139320, 0x1000, 0x1000, 0x0, 0x0, 0x0, ...)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/internal/poll/fd_unix.go:243 +0x230
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: net.(*netFD).readMsg(0xc000256100, 0xc00040bd20, 0x10, 0x10, 0xc000139320, 0x1000, 0x1000, 0xc0001dbd50, 0x5606fff5db76, 0x1, ...)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/net/fd_unix.go:214 +0x92
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: net.(*UnixConn).readMsg(0xc00024a1b8, 0xc00040bd20, 0x10, 0x10, 0xc000139320, 0x1000, 0x1000, 0xc00040bbf0, 0xc00040bbec, 0xc00058f680, ...)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/net/unixsock_posix.go:115 +0x93
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: net.(*UnixConn).ReadMsgUnix(0xc00024a1b8, 0xc00040bd20, 0x10, 0x10, 0xc000139320, 0x1000, 0x1000, 0xc00058fa80, 0x20, 0x18, ...)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/net/unixsock.go:143 +0x9a
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: github.com/godbus/dbus.(*oobReader).Read(0xc000139300, 0xc00040bd20, 0x10, 0x10, 0xc0001dbdf0, 0x5606fe59fb8a, 0x20)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/vendor/github.com/godbus/dbus/transport_unix.go:21 +0x8f
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: io.ReadAtLeast(0x5606ff882820, 0xc000139300, 0xc00040bd20, 0x10, 0x10, 0x10, 0xc00049aad0, 0x0, 0x0)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/io/io.go:310 +0x89
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: io.ReadFull(...)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/io/io.go:329
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: github.com/godbus/dbus.(*unixTransport).ReadMessage(0xc00000f360, 0xc000000002, 0x2, 0xc000134f01)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/vendor/github.com/godbus/dbus/transport_unix.go:91 +0x11a
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: github.com/godbus/dbus.(*Conn).inWorker(0xc0003f9e60)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/vendor/github.com/godbus/dbus/conn.go:294 +0x42
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: created by github.com/godbus/dbus.(*Conn).Auth
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/vendor/github.com/godbus/dbus/auth.go:118 +0x623
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: goroutine 38 [IO wait, 25 minutes]:
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: internal/poll.runtime_pollWait(0x7f09ff1b4e38, 0x72, 0x0)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/runtime/netpoll.go:184 +0x57
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: internal/poll.(*pollDesc).wait(0xc0000e2918, 0x72, 0x0, 0x0, 0x5606fefb38b7)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/internal/poll/fd_poll_runtime.go:87 +0x47
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: internal/poll.(*pollDesc).waitRead(...)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/internal/poll/fd_poll_runtime.go:92
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: internal/poll.(*FD).Accept(0xc0000e2900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/internal/poll/fd_unix.go:384 +0x1fa
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: net.(*netFD).accept(0xc0000e2900, 0x7f09fe970cb8, 0x81, 0x6c6c61202020200a)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/net/fd_unix.go:238 +0x44
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: net.(*UnixListener).accept(0xc000245890, 0xc0003c5830, 0x30, 0x30)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/net/unixsock_posix.go:162 +0x34
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: net.(*UnixListener).Accept(0xc000245890, 0xc0003c5830, 0x5606ff863000, 0xc00029b528, 0xc000296700)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/net/unixsock.go:260 +0x49
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: github.com/snapcore/snapd/daemon.(*ucrednetListener).Accept(0xc000245950, 0x5606ff7cbf00, 0xc0003c5830, 0x5606ff703600, 0x5606fff37e60)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/daemon/ucrednet.go:112 +0x4a
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: net/http.(*Server).Serve(0xc0002ca2a0, 0x5606ff89b8c0, 0xc000245950, 0x0, 0x0)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/net/http/server.go:2896 +0x282
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: github.com/snapcore/snapd/daemon.(*Daemon).Start.func1.1(0x6e6f632d68736572, 0x20200a3a6c6f7274)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/daemon/daemon.go:369 +0x4b
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: gopkg.in/tomb%2ev2.(*Tomb).run(0xc0001049f0, 0xc0003f5350)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2d
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: created by gopkg.in/tomb%2ev2.(*Tomb).Go
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xc9
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: goroutine 63 [syscall, 25 minutes]:
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: syscall.Syscall6(0x17, 0xa, 0xc00029f628, 0x0, 0x0, 0x0, 0x0, 0xc0001e7380, 0x200000003, 0xc0001e7380)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/syscall/asm_linux_amd64.s:44 +0x5
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: syscall.Select(0xa, 0xc00029f628, 0x0, 0x0, 0x0, 0x5606fe5c1b73, 0x5606ff862520, 0xc00029f648)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/usr/lib/go-1.13/src/syscall/zsyscall_linux_amd64.go:1320 +0xa2
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: github.com/snapcore/snapd/osutil/udev/netlink.stopperSelectReadable(0x8, 0x9, 0x0, 0xc00022cf58, 0xc000299df8)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/osutil/udev/netlink/rawsockstop.go:57 +0x16c
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: github.com/snapcore/snapd/osutil/udev/netlink.RawSockStopper.func1(0xc000299da0, 0xc00029f780, 0x1)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/osutil/udev/netlink/rawsockstop.go:31 +0x4e
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: github.com/snapcore/snapd/osutil/udev/netlink.(*UEventConn).Monitor.func4(0xc000559a20, 0xc000082f00, 0xc000082f60, 0xc000082fc0, 0xc000562800, 0x5606ff8a4400, 0xc000559a00, 0xc000082e40)
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/osutil/udev/netlink/conn.go:139 +0x46
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: created by github.com/snapcore/snapd/osutil/udev/netlink.(*UEventConn).Monitor
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: #011/build/snapd/parts/snapd-deb/build/osutil/udev/netlink/conn.go:136 +0x247
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: rax    0xfffffffffffffffc
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: rbx    0xa00
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: rcx    0x5606fe5f447d
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: rdx    0x0
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: rdi    0x7f0a011f2dd8
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: rsi    0x0
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: rbp    0x7f0a011f2de8
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: rsp    0x7f0a011f2dd8
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: r8     0x1563
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: r9     0x609
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: r10    0x7ffcad188090
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: r11    0x206
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: r12    0xf1
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: r13    0x0
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: r14    0x5606ff1b6a0c
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: r15    0x0
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: rip    0x5606fe5f447d
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: rflags 0x206
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: cs     0x33
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: fs     0x0
Apr 16 22:17:29 ip-10-4-7-213 snapd[476]: gs     0x0
Apr 16 22:17:29 ip-10-4-7-213 systemd[1]: snapd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 16 22:17:29 ip-10-4-7-213 systemd[1]: snapd.service: Failed with result 'watchdog'.
Apr 16 22:17:29 ip-10-4-7-213 systemd[1]: snapd.service: Scheduled restart job, restart counter is at 1.
Apr 16 22:17:29 ip-10-4-7-213 systemd[1]: Stopped Snap Daemon.
Apr 16 22:17:29 ip-10-4-7-213 systemd[1]: Starting Snap Daemon...
Apr 16 22:19:40 ip-10-4-7-213 snapd[3231]: AppArmor status: apparmor is enabled and all features are available
Apr 16 22:19:49 ip-10-4-7-213 systemd[1]: snapd.service: start operation timed out. Terminating.
Apr 16 22:19:50 ip-10-4-7-213 systemd[1]: snapd.service: Failed with result 'timeout'.
Apr 16 22:19:50 ip-10-4-7-213 systemd[1]: Failed to start Snap Daemon.
Apr 16 22:19:50 ip-10-4-7-213 systemd[1]: snapd.service: Scheduled restart job, restart counter is at 2.
Apr 16 22:19:50 ip-10-4-7-213 systemd[1]: Stopped Snap Daemon.
Apr 16 22:19:50 ip-10-4-7-213 systemd[1]: Starting Snap Daemon...
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:50,089 - distributed.core - INFO - Event loop was unresponsive in Nanny for 139.91s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:50,589 - distributed.worker - ERROR - Worker stream died during communication: tls://10.4.7.67:33281
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]: Traceback (most recent call last):
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:     bytes_read = self.read_from_fd(buf)
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/tornado/iostream.py", line 1592, in read_from_fd
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:     return self.socket.recv_into(buf, len(buf))
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/ssl.py", line 1241, in recv_into
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:     return self.read(nbytes, buffer)
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/ssl.py", line 1099, in read
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:     return self._sslobj.read(len, buffer)
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]: ConnectionResetError: [Errno 104] Connection reset by peer
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]: The above exception was the direct cause of the following exception:
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]: Traceback (most recent call last):
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 3023, in gather_dep
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:     response = await get_data_from_worker(
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 4328, in get_data_from_worker
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:     return await retry_operation(_get_data, operation="get_data_from_worker")
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/utils_comm.py", line 381, in retry_operation
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:     return await retry(
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/utils_comm.py", line 366, in retry
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:     return await coro()
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/worker.py", line 4308, in _get_data
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:     response = await send_recv(
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/core.py", line 709, in send_recv
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:     response = await comm.read(deserializers=deserializers)
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/tcp.py", line 242, in read
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:     convert_stream_closed_error(self, e)
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:   File "/opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/comm/tcp.py", line 148, in convert_stream_closed_error
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]:     raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
Apr 16 22:19:50 ip-10-4-7-213 cloud-init[1006]: distributed.comm.core.CommClosedError: in <TLS (closed) Ephemeral Worker->Worker for gather local=tls://10.4.7.213:38318 remote=tls://10.4.7.67:33281>: ConnectionResetError: [Errno 104] Connection reset by peer
Apr 16 22:19:50 ip-10-4-7-213 snapd[3248]: AppArmor status: apparmor is enabled and all features are available
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.473997] python invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474006] CPU: 0 PID: 3214 Comm: python Not tainted 5.13.0-1021-aws #23~20.04.2-Ubuntu
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474009] Hardware name: Amazon EC2 m5.large/, BIOS 1.0 10/16/2017
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474011] Call Trace:
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474013]  <TASK>
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474016]  dump_stack+0x7d/0x9c
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474023]  dump_header+0x4f/0x1f6
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474029]  oom_kill_process.cold+0xb/0x10
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474032]  out_of_memory+0x1cf/0x520
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474037]  __alloc_pages_slowpath.constprop.0+0xc86/0xd50
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474044]  __alloc_pages+0x2ce/0x320
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474047]  alloc_pages_vma+0x7f/0x210
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474050]  wp_page_copy+0x30d/0x590
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474056]  do_wp_page+0xeb/0x2f0
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474060]  __handle_mm_fault+0x8b5/0x8e0
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474065]  handle_mm_fault+0xda/0x2b0
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474068]  do_user_addr_fault+0x1bb/0x650
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474072]  exc_page_fault+0x7d/0x170
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474076]  ? asm_exc_page_fault+0x8/0x30
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474080]  asm_exc_page_fault+0x1e/0x30
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474084] RIP: 0033:0x7f5fba4c14ce
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474090] Code: Unable to access opcode bytes at RIP 0x7f5fba4c14a4.
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474091] RSP: 002b:00007f5fa3ffaed0 EFLAGS: 00010206
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474094] RAX: 000000000000000a RBX: 0000000000026efe RCX: 0000000000000001
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474096] RDX: 0000000000000000 RSI: 00007f5dff9ed000 RDI: 00007f5f49fd03a0
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474098] RBP: 00007f5f98012b70 R08: 0000000000000001 R09: 00005592e2739da0
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474099] R10: 0000000000000000 R11: 0000000000000206 R12: 00007f5e638b5520
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474101] R13: 0000000000026efe R14: 00007f5f98015bc0 R15: 00000000001377f0
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474104]  </TASK>
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474106] Mem-Info:
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474107] active_anon:236 inactive_anon:1918578 isolated_anon:0
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474107]  active_file:48 inactive_file:198 isolated_file:10
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474107]  unevictable:5751 dirty:0 writeback:0
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474107]  slab_reclaimable:6982 slab_unreclaimable:8368
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474107]  mapped:2096 shmem:234 pagetables:4795 bounce:0
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474107]  free:24937 free_pcp:0 free_cma:0
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474113] Node 0 active_anon:944kB inactive_anon:7674312kB active_file:192kB inactive_file:792kB unevictable:23004kB isolated(anon):0kB isolated(file):40kB mapped:8384kB dirty:0kB writeback:0kB shmem:936kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 2004992kB writeback_tmp:0kB kernel_stack:3440kB pagetables:19180kB all_unreclaimable? no
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474119] Node 0 DMA free:14336kB min:128kB low:160kB high:192kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474125] lowmem_reserve[]: 0 2987 7711 7711 7711
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474130] Node 0 DMA32 free:44608kB min:26128kB low:32660kB high:39192kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3006964kB active_file:0kB inactive_file:188kB unevictable:0kB writepending:0kB present:3129256kB managed:3063720kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474136] lowmem_reserve[]: 0 0 4724 4724 4724
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474140] Node 0 Normal free:40804kB min:41320kB low:51648kB high:61976kB reserved_highatomic:0KB active_anon:940kB inactive_anon:4666580kB active_file:56kB inactive_file:1100kB unevictable:23004kB writepending:0kB present:5023744kB managed:4846612kB mlocked:18468kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474146] lowmem_reserve[]: 0 0 0 0 0
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474151] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (M) 3*4096kB (M) = 14336kB
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474164] Node 0 DMA32: 72*4kB (UME) 105*8kB (UME) 44*16kB (UME) 26*32kB (UME) 41*64kB (UME) 43*128kB (UME) 19*256kB (UME) 10*512kB (UE) 12*1024kB (UME) 6*2048kB (M) 0*4096kB = 45352kB
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474184] Node 0 Normal: 980*4kB (UME) 451*8kB (UME) 134*16kB (UME) 241*32kB (UME) 136*64kB (UME) 56*128kB (UME) 21*256kB (UE) 2*512kB (U) 2*1024kB (UM) 0*2048kB 0*4096kB = 41704kB
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474203] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474206] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474207] 3689 total pagecache pages
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474208] 0 pages in swap cache
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474209] Swap cache stats: add 0, delete 0, find 0/0
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474211] Free swap  = 0kB
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474212] Total swap = 0kB
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474213] 2042248 pages RAM
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474214] 0 pages HighMem/MovableOnly
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474215] 60825 pages reserved
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474215] 0 pages hwpoisoned
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474216] Tasks state (memory values in pages):
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474217] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474222] [    165]     0   165     8767     1034    98304        0          -250 systemd-journal
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474226] [    198]     0   198     4973      747    65536        0         -1000 systemd-udevd
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474230] [    289]     0   289    70035     4487    90112        0         -1000 multipathd
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474233] [    338]   102   338    22493      671    73728        0             0 systemd-timesyn
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474236] [    402]   100   402     6678      966    77824        0             0 systemd-network
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474239] [    405]   101   405     5967     1288    94208        0             0 systemd-resolve
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474242] [    451]     0   451    60259      687    98304        0             0 accounts-daemon
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474245] [    454]     0   454      637      182    49152        0             0 acpid
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474248] [    455]   997   455   192572     3217   245760        0             0 amazon-cloudwat
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474252] [    459]     0   459     2136      486    53248        0             0 cron
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474255] [    460]   103   460     1894      784    57344        0          -900 dbus-daemon
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474258] [    467]     0   467    20458      661    53248        0             0 irqbalance
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474260] [    469]     0   469     7300     2725    90112        0             0 networkd-dispat
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474263] [    471]     0   471    59106      541    98304        0             0 polkitd
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474266] [    472]   104   472    56125      787    81920        0             0 rsyslogd
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474269] [    473]     0   473   327263     1607   200704        0             0 amazon-ssm-agen
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474271] [    477]     0   477     4134      398    73728        0             0 systemd-logind
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474274] [    479]     0   479    98668     1145   131072        0             0 udisksd
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474277] [    482]     0   482      950      488    49152        0             0 atd
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474279] [    483]     0   483   335972     3842   274432        0          -999 containerd
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474282] [    496]     0   496     1840      436    53248        0             0 agetty
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474285] [    503]     0   503     1459      346    45056        0             0 agetty
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474288] [    523]     0   523    78584     1004   118784        0             0 ModemManager
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474291] [    588]     0   588    27028     2741   106496        0             0 unattended-upgr
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474293] [    664]     0   664     3045      796    69632        0         -1000 sshd
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474296] [    682]     0   682   382586     7985   475136        0          -500 dockerd
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474299] [    725]     0   725      624      143    45056        0             0 bpfilter_umh
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474302] [    981]     0   981   330315     2628   217088        0             0 ssm-agent-worke
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474305] [    999]     0   999    12339     8441   131072        0             0 cloud-init
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474307] [   1005]     0  1005      654      119    45056        0             0 sh
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474310] [   1006]     0  1006     1811       18    49152        0             0 tee
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474313] [   1007]     0  1007      654      103    40960        0             0 runcmd
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474315] [   1013]     0  1013     2858      684    53248        0             0 sudo
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474318] [   1014]     0  1014     2176      513    49152        0             0 run_coiled
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474321] [   1016]     0  1016     2156      488    57344        0             0 bash
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474323] [   1063]     0  1063     2176      511    49152        0             0 notify_coiled
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474326] [   1064]     0  1064     1889     1619    57344        0             0 docker-compose
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474329] [   1065]     0  1065     1811       16    49152        0             0 tee
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474331] [   1066]     0  1066    73387     8236   348160        0             0 docker-compose
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474334] [   1096]     0  1096   177694      570   106496        0          -998 containerd-shim
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474337] [   1117]     0  1117   168519    11450   253952        0             0 python
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474340] [   1144]     0  1144     3421     1342    61440        0             0 python
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474343] [   3203]     0  3203  2115038  1856020 15314944        0             0 python
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474346] [   3247]     0  3247     1809      286    53248        0             0 sleep
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474348] [   3248]     0  3248   107265     2389   151552        0          -900 snapd
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474351] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=3f79ed588d91ab6b9cc491714c795aa3e46db1050c49a38fe54efed969b0aeda,mems_allowed=0,global_oom,task_memcg=/docker/3f79ed588d91ab6b9cc491714c795aa3e46db1050c49a38fe54efed969b0aeda,task=python,pid=3203,uid=0
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.474382] Out of memory: Killed process 3203 (python) total-vm:8460152kB, anon-rss:7424072kB, file-rss:0kB, shmem-rss:8kB, UID:0 pgtables:14956kB oom_score_adj:0
Apr 16 22:19:53 ip-10-4-7-213 kernel: [ 1690.870474] oom_reaper: reaped process 3203 (python), now anon-rss:0kB, file-rss:0kB, shmem-rss:12kB
Apr 16 22:19:53 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:53,807 - distributed.nanny - INFO - Worker process 165 was killed by signal 9
Apr 16 22:19:54 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:53,976 - distributed.nanny - WARNING - Restarting worker
Apr 16 22:19:54 ip-10-4-7-213 snapd[3248]: AppArmor status: apparmor is enabled and all features are available
Apr 16 22:19:54 ip-10-4-7-213 snapd[3248]: daemon.go:246: started snapd/2.54.4 (series 16; classic) ubuntu/20.04 (amd64) linux/5.13.0-1021-aws.
Apr 16 22:19:54 ip-10-4-7-213 kernel: [ 1691.685160] loop5: detected capacity change from 0 to 8
Apr 16 22:19:54 ip-10-4-7-213 systemd[1]: tmp-sanity\x2dmountpoint\x2d323821411.mount: Succeeded.
Apr 16 22:19:54 ip-10-4-7-213 snapd[3248]: daemon.go:339: adjusting startup timeout by 55s (pessimistic estimate of 30s plus 5s per snap)
Apr 16 22:19:54 ip-10-4-7-213 systemd[1]: Started Snap Daemon.
Apr 16 22:19:54 ip-10-4-7-213 dbus-daemon[460]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested by ':1.42' (uid=0 pid=3248 comm="/usr/lib/snapd/snapd " label="unconfined")
Apr 16 22:19:54 ip-10-4-7-213 systemd[1]: Starting Time & Date Service...
Apr 16 22:19:54 ip-10-4-7-213 dbus-daemon[460]: [system] Successfully activated service 'org.freedesktop.timedate1'
Apr 16 22:19:54 ip-10-4-7-213 systemd[1]: Started Time & Date Service.
Apr 16 22:19:55 ip-10-4-7-213 snapd[3248]: stateengine.go:149: state ensure error: cannot decode new commands catalog: got unexpected HTTP status code 403 via GET to "https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic"
Apr 16 22:19:56 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:56,057 - distributed.worker - INFO -       Start worker at:     tls://10.4.7.213:34201
Apr 16 22:19:56 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:56,057 - distributed.worker - INFO -          Listening to:     tls://10.4.7.213:34201
Apr 16 22:19:56 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:56,057 - distributed.worker - INFO -          dashboard at:           10.4.7.213:40319
Apr 16 22:19:56 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:56,057 - distributed.worker - INFO - Waiting to connect to:    tls://44.200.14.55:8786
Apr 16 22:19:56 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:56,058 - distributed.worker - INFO - -------------------------------------------------
Apr 16 22:19:56 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:56,058 - distributed.worker - INFO -               Threads:                          2
Apr 16 22:19:56 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:56,058 - distributed.worker - INFO -                Memory:                   7.56 GiB
Apr 16 22:19:56 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:56,058 - distributed.worker - INFO -       Local Directory: /scratch/dask-worker-space/worker-9iulu73y
Apr 16 22:19:56 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:56,058 - distributed.worker - INFO - -------------------------------------------------
Apr 16 22:19:57 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:57,070 - distributed.worker - WARNING - Mismatched versions found
Apr 16 22:19:57 ip-10-4-7-213 cloud-init[1006]: +-------------+----------------------+----------------------+---------------------------------------------------+
Apr 16 22:19:57 ip-10-4-7-213 cloud-init[1006]: | Package     | This Worker          | scheduler            | workers                                           |
Apr 16 22:19:57 ip-10-4-7-213 cloud-init[1006]: +-------------+----------------------+----------------------+---------------------------------------------------+
Apr 16 22:19:57 ip-10-4-7-213 cloud-init[1006]: | dask        | 2022.04.1            | 2022.04.1            | {'2022.04.0', '2022.04.1'}                        |
Apr 16 22:19:57 ip-10-4-7-213 cloud-init[1006]: | distributed | 2022.4.1+7.gf0e9f890 | 2022.4.1+7.gf0e9f890 | {'2022.4.1+7.gf0e9f890', '2022.4.0+33.g7dffe7d5'} |
Apr 16 22:19:57 ip-10-4-7-213 cloud-init[1006]: +-------------+----------------------+----------------------+---------------------------------------------------+
Apr 16 22:19:57 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:57,070 - distributed.worker - INFO - Starting Worker plugin coiled-aws-env
Apr 16 22:19:57 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:57,071 - distributed.worker - INFO - Starting Worker plugin print-event-loop-thread
Apr 16 22:19:57 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:57,077 - distributed.worker - INFO -         Registered to:    tls://44.200.14.55:8786
Apr 16 22:19:57 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:57,077 - distributed.worker - INFO - -------------------------------------------------
Apr 16 22:19:57 ip-10-4-7-213 cloud-init[1006]: 2022-04-16 22:19:57,078 - distributed.core - INFO - Starting established connection
```

</details>

https://cloud.coiled.io/mrocklin/clusters/7596/details
mrocklin-880b2f98-e-worker-2922473b93
instance - i-0729a97d84d19abd0
I tried this as an alternative workflow

```python
import dask.array as da

x = da.random.random((100000, 100000), chunks=(50000, "auto"))

x.rechunk(("auto", 50000)).sum().compute()
```

Lots of disk, lots of transfers.  It ran smoothly.  Maybe because numpy is easier to move around than pandas?
I tried this without the string object dtype column (only numeric / numpy columns) to see if that was the difference between the pandas and numpy case.  No luck.  The problem persisted.
A few observations:
* When a worker stops talking to the scheduler, I'm unable to SSH into it (I can SSH into other workers though). It does indicate something's borked at the VM level. However, I see that the last known memory report for the worker was close to to the limit, so it may just be that the machine is out of memory. In my experience, it's pretty typical to not be able to SSH into a box that's under critical memory pressure.
* Adding `DASK_DISTRIBUTED__SCHEDULER__WORKER_TTL="120s"` "un-sticks" the unresponsive workers as expected (https://github.com/dask/distributed/issues/6148). So things still hang mid-shuffle for 120s, then start moving again.
* However with the TTL, it then deadlocks at the end in a more classic-style deadlock (all remaining workers are responsive and heartbeating, 2 `processing` tasks on the scheduler on 2 different workers, but nothing actually executing on those workers). This is running f0e9f890 FYI, so post-#6123. Haven't investigated this deadlock further yet.
* Of course, the stuck instance is also still running and costing money. On the EC2 monitoring page, you can see the CPU usage flatlines around 66%: <img width="1458" alt="image" src="https://user-images.githubusercontent.com/3309802/164070245-23e91bcf-f78b-4121-94b3-8f0d777cf0aa.png">
    This is a maybe interesting but not-that-important case for Coiled. The scheduler knows a worker should be shut down, but can't communicate with the worker to tell it to stop. Should it have a way to tell Coiled about this, and ask Coiled to clean up the underlying instance? cc @ntabris
I remember with Pangeo we had to get a little bit more conservative on the cloud with our memory-limit boundaries.  At the time it was due to lots of extra Kubernetes pods from different services lying around.  Maybe we want to do something similar here.  Maybe we don't actually have `4GiB`, but something closer to `4 * 0.95 GiB`, for example.

The fact that the dask array workload worked is interesting.  This maybe points to wanting to slim down copies in the protocol a bit (do we use pickle-5 by default for objects without serializer functions today?)
> This is a maybe interesting but not-that-important case for Coiled. The scheduler knows a worker should be shut down, but can't communicate with the worker to tell it to stop. Should it have a way to tell Coiled about this, and ask Coiled to clean up the underlying instance? cc @ntabris

This is interesting.  Actually, rather than asking Coiled to clean up the underlying instance, can we ask it to restart that instance?  (so that we save on the time to get a new instance)
> rather than asking Coiled to clean up the underlying instance, can we ask it to restart that instance? (so that we save on the time to get a new instance)

Hm, this would require some work (not as easy as just restarting) but is definitely something to think about. (Currently we use cloud-init to run dask etc on initial boot, we'd need to make things run again on reboot. We'd also need to figure out how to distinguish whether dask stopped and we want to terminate instance or dask stopped and we want to reboot instance.)
@jacobtomlinson speculative work on a new dask-worker: [dask-agent](https://github.com/jacobtomlinson/dask-agent) I think might be helpful (at least in the future) for this kind of scenario where we want a harder reset of the worker without costly time of re-provisioning

This comment is off topic and we should move to a new issue if others want to engage on dask-agent ideas
I'm not sure if that would provide any value above what the Nanny already does.  The Nanny *should* be doing this, but even the nanny is frozen out in this case.
I think that we haven't yet tried `git bisect` here, but we probably should.  @gjoseph92 do you have time for this?  Alternatively @fjetter someone on your team?  This will be annoying because of the long iteration time with `create_software_environment`, but it'll only be logarithmically long :)  I think that this is probably twenty iterations or so of minute-long disruptions in a workday, spread out by ten minutes each.
We have a winner! `git bisect` (via https://github.com/gjoseph92/coiled-bisect) implicates #5878. My guess, which I'll test next, is that undoing https://github.com/dask/distributed/pull/3424 [here](https://github.com/dask/distributed/pull/5878/files#diff-0c7375c95aaa92986f8ed5460bae4e98720a14fca13556e01fd44ab6b927c24dL3776-L3782) is allowing the worker to get in a memory-critical state so quickly that it becomes unresponsive (and even the nanny can't run anymore to restart it).

cc @crusaderky

Resolving patch:
diff --git a/distributed/worker_memory.py b/distributed/worker_memory.py
--- a/distributed/worker_memory.py
+++ b/distributed/worker_memory.py
@@ -39,6 +39,7 @@
 
 from distributed import system
 from distributed.core import Status
+from distributed.metrics import monotonic
 from distributed.spill import ManualEvictProto, SpillBuffer
 from distributed.utils import log_errors
 from distributed.utils_perf import ThrottledGC
@@ -234,6 +235,8 @@ async def _maybe_spill(self, worker: Worker, memory: int) -> None:
         )
         count = 0
         need = memory - target
+        last_checked_for_pause = last_yielded = monotonic()
+
         while memory > target:
             if not data.fast:
                 logger.warning(
@@ -255,7 +258,6 @@ async def _maybe_spill(self, worker: Worker, memory: int) -> None:
 
             total_spilled += weight
             count += 1
-            await asyncio.sleep(0)
 
             memory = worker.monitor.get_process_memory()
             if total_spilled > need and memory > target:
@@ -265,7 +267,23 @@ async def _maybe_spill(self, worker: Worker, memory: int) -> None:
                 self._throttled_gc.collect()
                 memory = worker.monitor.get_process_memory()
 
-        self._maybe_pause_or_unpause(worker, memory)
+            now = monotonic()
+
+            # Spilling may potentially take multiple seconds; we may pass the pause
+            # threshold in the meantime.
+            if now - last_checked_for_pause > self.memory_monitor_interval:
+                self._maybe_pause_or_unpause(worker, memory)
+                last_checked_for_pause = now
+
+            # Increase spilling aggressiveness when the fast buffer is filled with a lot
+            # of small values. This artificially chokes the rest of the event loop -
+            # namely, the reception of new data from other workers. While this is
+            # somewhat of an ugly hack,  DO NOT tweak this without a thorough cycle of
+            # stress testing. See: https://github.com/dask/distributed/issues/6110.
+            if now - last_yielded > 0.5:
+                await asyncio.sleep(0)
+                last_yielded = monotonic()
+
         if count:
             logger.debug(
                 "Moved %d tasks worth %s to disk",


Test patch:
diff --git a/distributed/tests/test_worker_memory.py b/distributed/tests/test_worker_memory.py
--- a/distributed/tests/test_worker_memory.py
+++ b/distributed/tests/test_worker_memory.py
@@ -2,7 +2,8 @@
 
 import asyncio
 import logging
-from collections import UserDict
+import threading
+from collections import Counter, UserDict
 from time import sleep
 
 import pytest
@@ -12,6 +13,7 @@
 import distributed.system
 from distributed import Client, Event, Nanny, Worker, wait
 from distributed.core import Status
+from distributed.metrics import monotonic
 from distributed.spill import has_zict_210
 from distributed.utils_test import captured_logger, gen_cluster, inc
 from distributed.worker_memory import parse_memory_limit
@@ -636,6 +638,117 @@ def leak():
     assert "memory" in out.lower()
 
 
+@gen_cluster(
+    nthreads=[("", 1)],
+    client=True,
+    worker_kwargs={"memory_limit": "10 GiB"},
+    config={
+        "distributed.worker.memory.target": False,
+        "distributed.worker.memory.spill": 0.7,
+        "distributed.worker.memory.pause": 0.9,
+        "distributed.worker.memory.monitor-interval": "10ms",
+    },
+)
+async def test_pause_while_spilling(c, s, a):
+    N_PAUSE = 3
+    N_TOTAL = 5
+
+    def get_process_memory():
+        if len(a.data) < N_PAUSE:
+            # Don't trigger spilling until after all tasks have completed
+            return 0
+        elif a.data.fast and not a.data.slow:
+            # Trigger spilling
+            return 8 * 2**30
+        else:
+            # Trigger pause, but only after we started spilling
+            return 10 * 2**30
+
+    a.monitor.get_process_memory = get_process_memory
+
+    class SlowSpill:
+        def __init__(self, _):
+            # Can't pickle a Semaphore, so instead of a default value, we create it
+            # here. Don't worry about race conditions; the worker is single-threaded.
+            if not hasattr(type(self), "sem"):
+                type(self).sem = threading.Semaphore(N_PAUSE)
+            # Block if there are N_PAUSE tasks in a.data.fast
+            self.sem.acquire()
+
+        def __reduce__(self):
+            paused = distributed.get_worker().status == Status.paused
+            if not paused:
+                sleep(0.1)
+            self.sem.release()
+            return bool, (paused,)
+
+    futs = c.map(SlowSpill, range(N_TOTAL))
+    while len(a.data.slow) < N_PAUSE + 1:
+        await asyncio.sleep(0.01)
+
+    assert a.status == Status.paused
+    # Worker should have become paused after the first `SlowSpill` was evicted, because
+    # the spill to disk took longer than the memory monitor interval.
+    assert len(a.data.fast) == 0
+    assert len(a.data.slow) == N_PAUSE + 1
+    n_spilled_while_paused = sum(paused is True for paused in a.data.slow.values())
+    assert N_PAUSE <= n_spilled_while_paused <= N_PAUSE + 1
+
+
+@pytest.mark.slow
+@gen_cluster(
+    nthreads=[("", 1)],
+    client=True,
+    worker_kwargs={"memory_limit": "10 GiB"},
+    config={
+        "distributed.worker.memory.target": False,
+        "distributed.worker.memory.spill": 0.6,
+        "distributed.worker.memory.pause": False,
+        "distributed.worker.memory.monitor-interval": "10ms",
+    },
+)
+async def test_release_evloop_while_spilling(c, s, a):
+    N = 100
+
+    def get_process_memory():
+        if len(a.data) < N:
+            # Don't trigger spilling until after all tasks have completed
+            return 0
+        return 10 * 2**30
+
+    a.monitor.get_process_memory = get_process_memory
+
+    class SlowSpill:
+        def __reduce__(self):
+            sleep(0.01)
+            return SlowSpill, ()
+
+    futs = [c.submit(SlowSpill, pure=False) for _ in range(N)]
+    while len(a.data) < N:
+        await asyncio.sleep(0)
+
+    ts = [monotonic()]
+    while a.data.fast:
+        await asyncio.sleep(0)
+        ts.append(monotonic())
+
+    # 100 tasks taking 0.01s to pickle each = 2s to spill everything
+    # (this is because everything is pickled twice:
+    # https://github.com/dask/distributed/issues/1371).
+    # We should regain control of the event loop every 0.5s.
+    c = Counter(round(t1 - t0, 1) for t0, t1 in zip(ts, ts[1:]))
+    # Depending on the implementation of WorkerMemoryMonitor._maybe_spill:
+    # if it calls sleep(0) every 0.5s:
+    #   {0.0: 315, 0.5: 4}
+    # if it calls sleep(0) after spilling each key:
+    #   {0.0: 233}
+    # if it never yields:
+    #   {0.0: 359, 2.0: 1}
+    # Make sure we remain in the first use case.
+    assert 1 < sum(v for k, v in c.items() if 0.5 <= k <= 1.9), dict(c)
+    assert not any(v for k, v in c.items() if k >= 2.0), dict(c)
+
+
 @pytest.mark.parametrize(
     "cls,name,value",
     [