Repository: deis/deis

Issue:
ec2: Consider using EBS volume for etcd
When running a 5 node cluster, stopping and starting a node causes the cluster to fail.

Using the cloud formation scripts do the following:
- Boot a 5 node cluster in amazon.
- Ensure etcd is running and has consensus.
- Stop a node (thus wiping out ephemeral storage).
- Start the Node.

Expected: Cluster Unaffected, Node Rejoin
Actual: Cluster-wide failure, Ephemeral disk fails to mount, All nodes crash etcd.

Solution?

Don't use ephemeral storage for etcd.



Discussion:
We use ephemeral storage because it's fast. We previously had issues where etcd's disk writes would be blocked by system IO to other disks, and fleet would consider the host dead because it would effectively leave the cluster. This caused etcd leader thrashing, constant rebalancing, and a slew of other issues.

I was under the impression that stopping a node would indeed wipe ephemeral storage, but a reboot will not. I would think that stopping an instance is something that wouldn't normally happen with day-to-day usage. Is this something you saw in a live cluster, @jwaldrip, or was this something you came across during resiliency testing?

> Actual: Cluster-wide failure, Ephemeral disk fails to mount, All nodes crash etcd.

I'm not sure how one node would take down the whole cluster. Do you have etcd logs from the other cluster members? This sounds like a bug we could open upstream at coreos/etcd.

This is something we saw in a live cluster. It may have to do with the disk trying to mount that triggered the failure. Unfortunately I do not have the logs handy anymore. Also is gp2 storage that much slower than ephemeral?

Ok - this is something we should try to reproduce. 

> Also is gp2 storage that much slower than ephemeral?

Maybe not. Originally we were using the default EBS volume types, which were magnetic, so the difference was significant. We should revisit this and consider using an EBS volume for etcd too.

gp2 would be slower than ephemeral due to network but you can also get Provisioned IOPS (PIOPS) if you wanted to make it a bit more consistent and closer to ephemeral (I have no real numbers at hand). 

One thing to consider is that without PIOPS you get a certain allotment of IOPS (3k) per each half hour interval so depending on churn you could hit a wall (https://aws.amazon.com/blogs/aws/new-ssd-backed-elastic-block-storage/ the under the hood bit has the relevant details).

Maybe consider one of the memory filesystems that drop to disk to allow for the persistance if this becomes a bottleneck. Generally gp2 should be enough for this, just wanted to give some background.


Resolving patch:
diff --git a/contrib/ec2/gen-json.py b/contrib/ec2/gen-json.py
--- a/contrib/ec2/gen-json.py
+++ b/contrib/ec2/gen-json.py
@@ -6,39 +6,6 @@
 CURR_DIR = os.path.dirname(os.path.realpath(__file__))
 
 # Add EC2-specific units to the shared user-data
-FORMAT_EPHEMERAL_VOLUME = '''
-  [Unit]
-  Description=Formats the ephemeral volume
-  ConditionPathExists=!/etc/ephemeral-volume-formatted
-  [Service]
-  Type=oneshot
-  RemainAfterExit=yes
-  ExecStart=/usr/sbin/wipefs -f /dev/xvdb
-  ExecStart=/usr/sbin/mkfs.ext4 -i 4096 -b 4096 /dev/xvdb
-  ExecStart=/bin/touch /etc/ephemeral-volume-formatted
-'''
-MOUNT_EPHEMERAL_VOLUME = '''
-  [Unit]
-  Description=Formats and mounts the ephemeral drive
-  Requires=format-ephemeral-volume.service
-  After=format-ephemeral-volume.service
-  [Mount]
-  What=/dev/xvdb
-  Where=/media/ephemeral
-  Type=ext4
-'''
-PREPARE_ETCD_DATA_DIRECTORY = '''
-  [Unit]
-  Description=Prepares the etcd data directory
-  Requires=media-ephemeral.mount
-  After=media-ephemeral.mount
-  Before=etcd.service
-  [Service]
-  Type=oneshot
-  RemainAfterExit=yes
-  ExecStart=/usr/bin/mkdir -p /media/ephemeral/etcd
-  ExecStart=/usr/bin/chown -R etcd:etcd /media/ephemeral/etcd
-'''
 FORMAT_DOCKER_VOLUME = '''
   [Unit]
   Description=Formats the added EBS volume for Docker
@@ -61,13 +28,45 @@
   Where=/var/lib/docker
   Type=ext4
 '''
+FORMAT_ETCD_VOLUME = '''
+  [Unit]
+  Description=Formats the etcd device
+  ConditionPathExists=!/etc/etcd-volume-formatted
+  [Service]
+  Type=oneshot
+  RemainAfterExit=yes
+  ExecStart=/usr/sbin/wipefs -f /dev/xvdg
+  ExecStart=/usr/sbin/mkfs.ext4 -i 4096 -b 4096 /dev/xvdg
+  ExecStart=/bin/touch /etc/etcd-volume-formatted
+'''
+MOUNT_ETCD_VOLUME = '''
+  [Unit]
+  Description=Formats and mounts the ephemeral drive
+  Requires=format-etcd-volume.service
+  After=format-etcd-volume.service
+  [Mount]
+  What=/dev/xvdg
+  Where=/media/etcd
+  Type=ext4
+'''
+PREPARE_ETCD_DATA_DIRECTORY = '''
+  [Unit]
+  Description=Prepares the etcd data directory
+  Requires=media-etcd.mount
+  After=media-etcd.mount
+  Before=etcd.service
+  [Service]
+  Type=oneshot
+  RemainAfterExit=yes
+  ExecStart=/usr/bin/chown -R etcd:etcd /media/etcd
+'''
 
 new_units = [
-  dict({'name': 'format-ephemeral-volume.service', 'command': 'start', 'content': FORMAT_EPHEMERAL_VOLUME}),
-  dict({'name': 'media-ephemeral.mount', 'command': 'start', 'content': MOUNT_EPHEMERAL_VOLUME}),
-  dict({'name': 'prepare-etcd-data-directory.service', 'command': 'start', 'content': PREPARE_ETCD_DATA_DIRECTORY}),
   dict({'name': 'format-docker-volume.service', 'command': 'start', 'content': FORMAT_DOCKER_VOLUME}),
-  dict({'name': 'var-lib-docker.mount', 'command': 'start', 'content': MOUNT_DOCKER_VOLUME})
+  dict({'name': 'var-lib-docker.mount', 'command': 'start', 'content': MOUNT_DOCKER_VOLUME}),
+  dict({'name': 'format-etcd-volume.service', 'command': 'start', 'content': FORMAT_ETCD_VOLUME}),
+  dict({'name': 'media-etcd.mount', 'command': 'start', 'content': MOUNT_ETCD_VOLUME}),
+  dict({'name': 'prepare-etcd-data-directory.service', 'command': 'start', 'content': PREPARE_ETCD_DATA_DIRECTORY})
 ]
 
 data = yaml.load(file(os.path.join(CURR_DIR, '..', 'coreos', 'user-data'), 'r'))
@@ -76,8 +75,8 @@
 # are started
 data['coreos']['units'] = new_units + data['coreos']['units']
 
-# configure etcd to use the ephemeral drive
-data['coreos']['etcd']['data-dir'] = '/media/ephemeral/etcd'
+# configure etcd to use its EBS volume
+data['coreos']['etcd']['data-dir'] = '/media/etcd'
 
 header = ["#cloud-config", "---"]
 dump = yaml.dump(data, default_flow_style=False)


Test patch: