Repository: deis/deis

Issue:
fix: Move /var/lib/docker to separate volume, prevents all units from crashing on a single machine
When debugging performance issues with Deis on Azure, we discovered that machines became unresponsive whenever we had high activity (such as when doing a `docker pull` when components start). We actually witnessed systemd timing out and issuing a sigterm to all of its processes(!), effectively emulating a system reboot. This of course causes issues for etcd/fleet, as well as Deis and anything else on the machine. We suspect this is the cause of all of the units on a machine showing as dead.

In attempting to resolve this issue, we tried moving Docker's `/var/lib/docker` directory to a separate volume. The difference is night and day - no more performance issues, and Deis came up fine on the first try.

We should try this on EC2 as well, as we see what appears to be this same issue there (see #2506). We should also generally document this as a best practice.



Discussion:
ping @gabrtv to make sure this is correct and I didn't forget anything

@carmstrong this could be a good idea for another reason: format the volume as ext4 in preparation to OverlayFS

@carmstrong the symptoms you outlined are 100% accurate, though I haven't myself seen how moving `/var/lib/docker` affects things.  Regardless of the fix, I can see justification for moving the volume since `/var/lib/docker` should be resizable on its own.  :+1: 

As @aledbf points out we can mount it as ext4 in preparation for CoreOS switch to overlayfs: https://groups.google.com/forum/#!topic/coreos-dev/NDEOXchAbuU


Resolving patch:
diff --git a/contrib/ec2/gen-json.py b/contrib/ec2/gen-json.py
--- a/contrib/ec2/gen-json.py
+++ b/contrib/ec2/gen-json.py
@@ -1,13 +1,44 @@
 #!/usr/bin/env python
 import json
 import os
+import yaml
 
-template = json.load(open("deis.template.json",'r'))
+CURR_DIR = os.path.dirname(os.path.realpath(__file__))
 
-with open('../coreos/user-data','r') as f:
-  lines = f.readlines()
+# Add EC2-specific units to the shared user-data
+FORMAT_EPHEMERAL = '''
+  [Unit]
+  Description=Formats the ephemeral drive
+  ConditionPathExists=!/etc/docker-volume-formatted
+  [Service]
+  Type=oneshot
+  RemainAfterExit=yes
+  ExecStart=/usr/sbin/wipefs -f /dev/xvdf
+  ExecStart=/usr/sbin/mkfs.btrfs -f /dev/xvdf
+  ExecStart=/bin/touch /etc/docker-volume-formatted
+'''
+DOCKER_MOUNT = '''
+  [Unit]
+  Description=Mount ephemeral to /var/lib/docker
+  Requires=format-ephemeral.service
+  After=format-ephemeral.service
+  Before=docker.service
+  [Mount]
+  What=/dev/xvdf
+  Where=/var/lib/docker
+  Type=btrfs
+'''
 
-template['Resources']['CoreOSServerLaunchConfig']['Properties']['UserData']['Fn::Base64']['Fn::Join'] = [ '', lines ]
+data = yaml.load(file(os.path.join(CURR_DIR, '..', 'coreos', 'user-data'), 'r'))
+data['coreos']['units'].append(dict({'name': 'format-ephemeral.service', 'command': 'start', 'content': FORMAT_EPHEMERAL}))
+data['coreos']['units'].append(dict({'name': 'var-lib-docker.mount', 'command': 'start', 'content': DOCKER_MOUNT}))
+
+header = ["#cloud-config", "---"]
+dump = yaml.dump(data, default_flow_style=False)
+
+template = json.load(open(os.path.join(CURR_DIR, 'deis.template.json'),'r'))
+
+template['Resources']['CoreOSServerLaunchConfig']['Properties']['UserData']['Fn::Base64']['Fn::Join'] = [ "\n", header + dump.split("\n") ]
 template['Parameters']['ClusterSize']['Default'] = str(os.getenv('DEIS_NUM_INSTANCES', 3))
 
 VPC_ID = os.getenv('VPC_ID', None)


Test patch: