Repository: DataBiosphere/toil

Issue:
toil-worker.service can't recover from docker daemon failure
I seem to have run into a rare dockerd crash caused by a race condition in some SELinux handling code. Obviously that isn't Toil's problem, but the issue is that toil-worker.service won't properly restart without manual intervention:

```
Jan 31 02:09:52 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: Started toil-worker container.
Jan 31 02:09:52 ip-172-31-25-156.us-west-2.compute.internal dockerd[21255]: time="2017-01-31T02:09:52.886397513Z" level=error msg="Handler for POST /v1.22/containers/create returned error: Conflict. The name \"/toil_worker\" is already in use by container e785866fe546375d155faf1892e9606fd90694f35222eb501f18b09e55b0ab60. You have to remove (or rename) that container to be able to reuse that name."
Jan 31 02:09:52 ip-172-31-25-156.us-west-2.compute.internal docker[21458]: docker: Error response from daemon: Conflict. The name "/toil_worker" is already in use by container e785866fe546375d155faf1892e9606fd90694f35222eb501f18b09e55b0ab60. You have to remove (or rename) that container to be able to reuse that name..
Jan 31 02:09:52 ip-172-31-25-156.us-west-2.compute.internal docker[21458]: See '/usr/bin/docker run --help'.
Jan 31 02:09:52 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: toil-worker.service: Main process exited, code=exited, status=125/n/a
Jan 31 02:09:52 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: toil-worker.service: Unit entered failed state.
Jan 31 02:09:52 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: toil-worker.service: Failed with result 'exit-code'.
Jan 31 02:09:53 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: toil-worker.service: Service hold-off time over, scheduling restart.
Jan 31 02:09:53 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: Stopped toil-worker container.
```
Several repetitions later:
```
Jan 31 02:09:53 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: toil-worker.service: Start request repeated too quickly.
Jan 31 02:09:53 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: Failed to start toil-worker container.
```

This causes the cluster to effectively lose one machine, and the autoscaler code (correctly) detects that the machine is still actually running, so it won't replace it with a fresh one.

It seems like the only problem is that the `toil_worker` name is being requested every time. I think modifying the service file to:
- remove and/or stop the `toil_worker` container if it exists
- delay restart attempts by a few seconds, in case there is a short-term problem starting the container (there were ~5 restart attempts in 2 seconds)

should fix things. I'll try to submit a PR tomorrow.
toil-worker.service can't recover from docker daemon failure
I seem to have run into a rare dockerd crash caused by a race condition in some SELinux handling code. Obviously that isn't Toil's problem, but the issue is that toil-worker.service won't properly restart without manual intervention:

```
Jan 31 02:09:52 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: Started toil-worker container.
Jan 31 02:09:52 ip-172-31-25-156.us-west-2.compute.internal dockerd[21255]: time="2017-01-31T02:09:52.886397513Z" level=error msg="Handler for POST /v1.22/containers/create returned error: Conflict. The name \"/toil_worker\" is already in use by container e785866fe546375d155faf1892e9606fd90694f35222eb501f18b09e55b0ab60. You have to remove (or rename) that container to be able to reuse that name."
Jan 31 02:09:52 ip-172-31-25-156.us-west-2.compute.internal docker[21458]: docker: Error response from daemon: Conflict. The name "/toil_worker" is already in use by container e785866fe546375d155faf1892e9606fd90694f35222eb501f18b09e55b0ab60. You have to remove (or rename) that container to be able to reuse that name..
Jan 31 02:09:52 ip-172-31-25-156.us-west-2.compute.internal docker[21458]: See '/usr/bin/docker run --help'.
Jan 31 02:09:52 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: toil-worker.service: Main process exited, code=exited, status=125/n/a
Jan 31 02:09:52 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: toil-worker.service: Unit entered failed state.
Jan 31 02:09:52 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: toil-worker.service: Failed with result 'exit-code'.
Jan 31 02:09:53 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: toil-worker.service: Service hold-off time over, scheduling restart.
Jan 31 02:09:53 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: Stopped toil-worker container.
```
Several repetitions later:
```
Jan 31 02:09:53 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: toil-worker.service: Start request repeated too quickly.
Jan 31 02:09:53 ip-172-31-25-156.us-west-2.compute.internal systemd[1]: Failed to start toil-worker container.
```

This causes the cluster to effectively lose one machine, and the autoscaler code (correctly) detects that the machine is still actually running, so it won't replace it with a fresh one.

It seems like the only problem is that the `toil_worker` name is being requested every time. I think modifying the service file to:
- remove and/or stop the `toil_worker` container if it exists
- delay restart attempts by a few seconds, in case there is a short-term problem starting the container (there were ~5 restart attempts in 2 seconds)

should fix things. I'll try to submit a PR tomorrow.


Discussion:
@joelarmstrong makes sense, thanks for reporting this. I'll forward to your PR!
@joelarmstrong makes sense, thanks for reporting this. I'll forward to your PR!

Resolving patch:
diff --git a/src/toil/provisioners/aws/__init__.py b/src/toil/provisioners/aws/__init__.py
--- a/src/toil/provisioners/aws/__init__.py
+++ b/src/toil/provisioners/aws/__init__.py
@@ -275,6 +275,8 @@ def _get_spot_history(ctx, instance_type):
 
         [Service]
         Restart=on-failure
+        RestartSec=2
+        ExecPre=-/usr/bin/docker rm toil_{role}
         ExecStart=/usr/bin/docker run \
             --entrypoint={entrypoint} \
             --net=host \


Test patch: