Repository: deis/deis

Issue:
Container corruption after reboot in controller VM
After a few days playing with deis, I noticed that it is very easy to corrupt the deis containers (especially registry and logger) to the point that a re-installation is needed; the problem triggers most of the times by just restarting CoreOS without explicitly stopping the deis services before.

I have reproduced multiple times the issues while logging the results with `journalctl -f -u deis-*` (have full log available on demand). Summary below
## Normal case: starting from empty CoreOS VM

Starting from an empty CoreOS VM (such as the vagrant image), pulling, building, installing and starting the services using the top-level Makefile.

After successfully doing all these steps (resuming by hand after "docker pull" network-related errors), the deis services in the VM look like this:

``` sh
$ sudo systemctl list-units --all | grep -y deis
  deis-builder.service                                                                loaded    active   running   deis-builder
  deis-cache.service                                                                  loaded    active   running   deis-cache
  deis-controller.service                                                             loaded    active   running   deis-controller
  deis-database.service                                                               loaded    active   running   deis-database
  deis-logger.service                                                                 loaded    active   running   deis-logger
  deis-registry.service                                                               loaded    active   running   deis-registry
  deis-router.service                                                                 loaded    active   running   deis-router
  seed-docker-images.service                                                          loaded    inactive dead      Seed Docker images used by Deis
```
## Mostly working: restarting after explicitly stopping all deis services

Before restarting the VM, I `make stop` to gracefully stop all the containers. The containers actually end up in "failed" instead of "stopped" state but it mostly works after the reboot.

``` sh
$ sudo systemctl list-units --all | grep -y deis
● deis-builder.service                                                                loaded    failed   failed    deis-builder
● deis-cache.service                                                                  loaded    failed   failed    deis-cache
● deis-controller.service                                                             loaded    failed   failed    deis-controller
● deis-database.service                                                               loaded    failed   failed    deis-database
● deis-logger.service                                                                 loaded    failed   failed    deis-logger
● deis-registry.service                                                               loaded    failed   failed    deis-registry
● deis-router.service                                                                 loaded    failed   failed    deis-router
  seed-docker-images.service                                                          loaded    inactive dead      Seed Docker images used by Deis
```

After the reboot, all deis services are "running" except the controller which is "failed". After restarting the service explicitly with `sudo systemctl start deis-controller.service` everything is running again (it would be nice is the controller restarted on its own).
## Broken: Rebooting without explicitly stopping the deis services

For the third case, I just `shutdown -r now` without stopping any service, I expect that systemd will notify docker and that the container has a way to persist its state without losing data. The reboot is abnormally fast.

On reboot all deis services are in messy state.

``` sh
$ sudo systemctl list-units --all | grep -y deis
● deis-builder.service                                                                loaded    failed     failed           deis-builder
● deis-cache.service                                                                  loaded    failed     failed           deis-cache
● deis-controller.service                                                             loaded    failed     failed           deis-controller
● deis-database.service                                                               loaded    failed     failed           deis-database
● deis-logger.service                                                                 loaded    failed     failed           deis-logger
  deis-registry.service                                                               loaded    activating start-post start deis-registry
● deis-router.service                                                                 loaded    failed     failed           deis-router
  seed-docker-images.service                                                          loaded    inactive   dead             Seed Docker images used by Deis
```

The registry service is busy looping until something listens on port 5000 (it never comes). I stop the registry service with `sudo systemctl stop deis-registry`. When I restart the registry I get the following message in the log:

```
Apr 21 11:19:38 localhost systemd[1]: Starting deis-registry...
Apr 21 11:19:38 localhost bash[3721]: deis-registry-data
Apr 21 11:19:38 localhost sh[3741]: Waiting for listener on 5000/tcp...
Apr 21 11:19:38 localhost sh[3741]: /bin/sh: connect: Connection refused
Apr 21 11:19:38 localhost sh[3741]: /bin/sh: /dev/tcp/10.0.2.15/5000: Connection refused
Apr 21 11:19:38 localhost sh[3740]: 2014/04/21 11:19:38 Error: Conflict, The name deis-registry is already assigned to a87ab93873c0. You have to delete (or rename) that container to be able to assign deis-registry to a container again.
Apr 21 11:19:38 localhost systemd[1]: deis-registry.service: main process exited, code=exited, status=1/FAILURE
```

After removing the container (as the message suggests), I can restart the registry service and it works, the slugrunner is pushed to the local repository as expected.

Trying to restart the other deis services, I find the logger service broken:

```
Apr 21 10:23:12 localhost systemd[1]: Starting deis-logger...
Apr 21 10:23:12 localhost bash[4109]: deis-logger-data
Apr 21 10:23:12 localhost systemd[1]: Started deis-logger.
Apr 21 10:23:12 localhost sh[4126]: 2014/04/21 10:23:12 Error: Conflict, The name deis-logger is already assigned to 554f8c79abb2. You have to delete (or rename) that container to be able to assign deis-logger to a container again.
Apr 21 10:23:12 localhost systemd[1]: deis-logger.service: main process exited, code=exited, status=1/FAILURE
Apr 21 10:23:12 localhost docker[4146]: Error: Cannot destroy container deis-logger: Driver btrfs failed to remove root filesystem 554f8c79abb2437863ed21a7317aa95443dc975d70815d6e1bf54875fd80cf3d: stat 
/var/lib/docker/btrfs/subvolumes/554f8c79abb2437863ed21a7317aa95443dc975d70815d6e1bf54875fd80cf3d: no such file or directory
Apr 21 10:23:12 localhost docker[4146]: 2014/04/21 10:23:12 Error: failed to remove one or more containers
Apr 21 10:23:12 localhost systemd[1]: deis-logger.service: control process exited, code=exited status=1
Apr 21 10:23:12 localhost systemd[1]: Unit deis-logger.service entered failed state.
```

Except that this time, the docker filesystem tree is corrupted, I cannot remove the container and I'm forced to restart provisioning my deis controller from a clean VM. After triggering the problem 3 times I gave up.

If this is working as intended, what strategies or workaround to have a usable deis controller? We cannot always rely on the containers being gracefully shut down (even less with CoreOS restarting automatically on system updates).



Discussion:
@jperville thanks for a very well written bug report.  I can tell you that we've had anecdotal reports of this happening (and I've seen the corruption myself).  The core issue seems related to btrfs subvolumes.  Now that we have some reproduce steps, we'll get right on it.

Agreed. The way we start/stop containers both gracefully and ungracefully needs to be more fleshed out. Right now we always run the main components with a canonical name via `docker run --name deis-<component>`, which is a bit of a pain when trying to restart containers because Docker will complain until you actually remove that named container with `docker rm -f deis-<component>`, hence why you're seeing those errors in systemd. We need a better way to gracefully stop and start these containers.

https://github.com/deis/deis/pull/871 addresses resiliency when restarting Deis' own containers that should address this issue. @jperville, could you try this against current Deis master?

Hi @mboersma, I have tried Deis master with a 2-node cluster, unfortunately I still see services that won't restart after clean shutdown (eg. "sudo shutdown -h now"), this time it was "deis-cache". I installed Vagrant on OVH server to relieve my poor Internet connection (Docker behaves badly when downloading over unreliable/slow links, timeout issues everywhere).

The following [thread](https://groups.google.com/forum/#!topic/coreos-dev/an_vM3N3PE8) is talking about this issue and provides interesting workarounds. It also points to dotcloud/docker#4897 which is still open as of docker 0.10.

As I understand it, when I do "shutdown -r now" in coreos, all deis services will execute their `ExecStop` entry is something like `/usr/bin/docker rm -f deis-${name}`. That command is very fast, probably because it SIGTERM the process inside the container (does it leave a chance to gracefully save the data?). The problem arrives when for some reason the container cannot be `docker rm -f`'d then on reboot the container won't start.

A last question: what are your thoughts on http://phusion.github.io/baseimage-docker/ which advocates running a special `/sbin/init` in the container to handle syslog and take cares of zombie processes (which are adopted by PID 1 and need to be handled when shutting down gracefully)?

I have some more testing to do now, what a pity that my Internet connection is so slow and docker won't use my http proxy to cache downloads.

Still experimenting, doing `vagrant reload` after explicitly stopping the services with `fleetctl --strict-host-key-checking=false stop deis-{builder,cache,controller,database,logger,registry,router}.service` works, however if I don't stop explicitly the services some containers won't restart.

The output of `fleetctl status` on a failed service looks like:

``` sh
julien@deis:~/deis$ fleetctl status deis-router.service
● deis-router.service - deis-router
   Loaded: loaded (/run/systemd/system/deis-router.service; enabled-runtime)
   Active: failed (Result: exit-code) since Sun 2014-05-11 14:02:11 UTC; 54s ago
  Process: 387 ExecStop=/usr/bin/docker rm -f deis-router (code=exited, status=1/FAILURE)
  Process: 348 ExecStart=/usr/bin/docker run --name deis-router -p 80:80 -e PUBLISH=80 -e HOST=${COREOS_PRIVATE_IPV4} deis/router (code=exited, status=1/FAILURE)
  Process: 32691 ExecStartPre=/bin/sh -c docker inspect deis-router >/dev/null && docker rm -f deis-router || true (code=exited, status=0/SUCCESS)
  Process: 32646 ExecStartPre=/bin/sh -c docker history deis/router >/dev/null || docker pull deis/router (code=exited, status=0/SUCCESS)
 Main PID: 348 (code=exited, status=1/FAILURE)

May 11 14:02:11 deis-1 sh[32691]: Error: Cannot destroy container deis-router: Driver btrfs failed to remove root filesystem 12ca9acb2309757d2b6be81f39757b05518470dc91131729effc45cd653fc2bc: stat /var/lib/docker/btrfs/subvolumes/12ca9acb2309757d2b6be81f39757b05518470dc91131729effc45cd653fc2bc: no such file or directory
May 11 14:02:11 deis-1 sh[32691]: 2014/05/11 14:02:11 Error: failed to remove one or more containers
May 11 14:02:11 deis-1 systemd[1]: Started deis-router.
May 11 14:02:11 deis-1 docker[348]: 2014/05/11 14:02:11 Error: Conflict, The name deis-router is already assigned to 12ca9acb2309. You have to delete (or rename) that container to be able to assign deis-router to a container again.
May 11 14:02:11 deis-1 systemd[1]: deis-router.service: main process exited, code=exited, status=1/FAILURE
May 11 14:02:11 deis-1 docker[387]: Error: Cannot destroy container deis-router: Driver btrfs failed to remove root filesystem 12ca9acb2309757d2b6be81f39757b05518470dc91131729effc45cd653fc2bc: stat /var/lib/docker/btrfs/subvolumes/12ca9acb2309757d2b6be81f39757b05518470dc91131729effc45cd653fc2bc: no such file or directory
May 11 14:02:11 deis-1 docker[387]: 2014/05/11 14:02:11 Error: failed to remove one or more containers
May 11 14:02:11 deis-1 systemd[1]: deis-router.service: control process exited, code=exited status=1
May 11 14:02:11 deis-1 systemd[1]: Unit deis-router.service entered failed state.
julien@deis:~/deis$ 
```

This is symptom of dotcloud/docker#4897. To recover and manually remove the broken brtfs volumes, I have to manually:

``` sh
sudo btrfs subvolume create /var/lib/docker/btrfs/subvolumes/b23c0bd501842afb0efb2a6b29f8215823286fc3f5860e4ea73e20eaa944bc9c
sudo btrfs subvolume create /var/lib/docker/btrfs/subvolumes/b23c0bd501842afb0efb2a6b29f8215823286fc3f5860e4ea73e20eaa944bc9c-init
```

And then try to `docker rm -rf deis-${name}` again and then run `make run` again, which is painful.

@jperville You have done pretty deep research on that. And as you said, we can not keep doing such practices which will lead to unhealthy cluster. Because developers are not system admins and it is hard to make them learn to fix it on their development environment. Currently given solution is working for me. Atleast no need to do all things from scratch on new vm. 

``` shell
sudo btrfs subvolume create /var/lib/docker/btrfs/subvolumes/b23c0bd501842afb0efb2a6b29f8215823286fc3f5860e4ea73e20eaa944bc9c
sudo btrfs subvolume create /var/lib/docker/btrfs/subvolumes/b23c0bd501842afb0efb2a6b29f8215823286fc3f5860e4ea73e20eaa944bc9c-init
```


Resolving patch:
diff --git a/controller/scheduler/coreos.py b/controller/scheduler/coreos.py
--- a/controller/scheduler/coreos.py
+++ b/controller/scheduler/coreos.py
@@ -95,24 +95,24 @@ def start(self, name):
 
     def _start_log(self, name, env):
         subprocess.check_call(
-            'fleetctl.sh start {name}-log.service'.format(**locals()),
+            'fleetctl.sh start -no-block {name}-log.service'.format(**locals()),
             shell=True, env=env)
 
     def _start_container(self, name, env):
         return subprocess.check_call(
-            'fleetctl.sh start {name}.service'.format(**locals()),
+            'fleetctl.sh start -no-block {name}.service'.format(**locals()),
             shell=True, env=env)
 
     def _start_announcer(self, name, env):
         return subprocess.check_call(
-            'fleetctl.sh start {name}-announce.service'.format(**locals()),
+            'fleetctl.sh start -no-block {name}-announce.service'.format(**locals()),
             shell=True, env=env)
 
     def _wait_for_announcer(self, name, env):
         status = None
         for _ in range(60):
             status = subprocess.check_output(
-                "fleetctl.sh list-units | grep {name}-announce.service | awk '{{print $4}}'".format(**locals()),
+                "fleetctl.sh list-units | grep {name}-announce.service | awk '{{print $5}}'".format(**locals()),
                 shell=True, env=env).strip('\n')
             if status == 'running':
                 break
@@ -132,17 +132,17 @@ def stop(self, name):
 
     def _stop_container(self, name, env):
         return subprocess.check_call(
-            'fleetctl.sh stop {name}.service'.format(**locals()),
+            'fleetctl.sh stop -block-attempts=600 {name}.service'.format(**locals()),
             shell=True, env=env)
 
     def _stop_announcer(self, name, env):
         return subprocess.check_call(
-            'fleetctl.sh stop {name}-announce.service'.format(**locals()),
+            'fleetctl.sh stop -block-attempts=600 {name}-announce.service'.format(**locals()),
             shell=True, env=env)
 
     def _stop_log(self, name, env):
         return subprocess.check_call(
-            'fleetctl.sh stop {name}-announce.service'.format(**locals()),
+            'fleetctl.sh stop -block-attempts=600 {name}-log.service'.format(**locals()),
             shell=True, env=env)
 
     def destroy(self, name):


Test patch: