Repository: deis/deis

Issue:
AWS provisioning process fails in 1.9.0
`gen-json.py` in the AWS provisioning process adds in some AWS-specific user-data and also modifies portions of the user-data for AWS.  In AWS, the CloudFormation templates we use specify launch configuration for CoreOS hosts wherein we mount a few EBS volumes-- among them, we mount one to `/media/etcd`.  One of the corresponding modification to the user-data is to string replace certain instances of `/var/lib/etcd2` with `/media/etcd`

See relevant lines from `gen-json.py` here:

```
# configure etcd to use its EBS volume
dump = dump.replace('-data-dir /var/lib/etcd2', '-data-dir /media/etcd')
dump = dump.replace('--volume=/var/lib/etcd2', '--volume=/media/etcd')
```

https://github.com/deis/deis/blob/master/contrib/aws/gen-json.py#L108-L109

This worked great until v1.9.0 when one of the strings that is ordinarily modified was changed in `user-data.example`.

Here's the relevant commit:

https://github.com/deis/deis/commit/ccb2f9c8ce4dc4d2589bae7ea8f3bc7e87bb315d

What this results in is user-data that's actually used that looks something like this:

```
  - name: etcd.service
    # ...
      [Service]
      # ...
      Environment="ETCD_DATA_DIR=/var/lib/etcd2"
      # ...
      ExecStart=/usr/bin/docker run --net=host --rm \
        --volume=${ETCD_DATA_DIR}:/var/lib/etcd2 \
        # ...
        -data-dir /media/etcd \
        # ...
    # ...
```

As you can see, the wrong directory on the host (should be `/media/etcd`, but is `/var/lib/etcd2`) is mounted into the wrong directory on the container (should be `/media/etcd`, but is `/var/lib/etcd2`).  The containerized etcd2, however, per `-data-dir /media/etcd` is looking for the host's `/media/etcd' to have been mounted into the container's`/media/etcd`... which isn't the case.

This _is_ definitely wrong.  The very odd part about it however, is that whatever issue this actually causes appears to be non-deterministic.  Sometimes this fails spectacularly; sometimes it _somehow_ seems to work (although I'm not sure how).



Discussion:
Good find. We'll need to fix this, and maybe even cut a 1.9.1.

> Sometimes this fails spectacularly; sometimes it somehow seems to work (although I'm not sure how).

Hmm. Smells like a race condition maybe?

> Hmm. Smells like a race condition maybe?

I'd buy that.

Thinking about this, I think that the bug here is probably: https://github.com/deis/deis/blob/101bbba3788043e64545a2c67ead224c4958b048/contrib/aws/gen-json.py#L108

We never want to rewrite the `-data-dir` parameter to etcd. Inside the container, it should _always_ be `/var/lib/etcd`. We only want to change which volume on the host gets bind-mounted into the container.

@aledbf does that make sense?

> We never want to rewrite the -data-dir parameter to etcd. Inside the container, it should always be /var/lib/etcd. We only want to change which volume on the host gets bind-mounted into the container.

That sounds pretty reasonable to me.

Altho, for etcd2, I think the data dir should be `/var/lib/etcd2`, but otherwise, I agree.

> I think the data dir should be /var/lib/etcd2

Yep, that was a typo. :+1: 

Even if the line you referenced comes out, we should still be adding a line like this one:

```
dump = dump.replace('ETCD_DATA_DIR=/var/lib/etcd2', 'ETCD_DATA_DIR=/media/etcd')
```

That will change the directory on the _host_ side of the `--volume` flag to match where the EBS volume is actually mounted on the host.

I actually propose removing the `ETCD_DATA_DIR` envvar entirely - it's not really what we're using it for... for us, it's the host's data directory. I think overloading the use of "data directory" is confusing.

I'm ok with that... it's only used in one place.


Resolving patch:
diff --git a/contrib/aws/gen-json.py b/contrib/aws/gen-json.py
--- a/contrib/aws/gen-json.py
+++ b/contrib/aws/gen-json.py
@@ -105,8 +105,7 @@
 dump = yaml.dump(data, default_flow_style=False)
 
 # configure etcd to use its EBS volume
-dump = dump.replace('-data-dir /var/lib/etcd2', '-data-dir /media/etcd')
-dump = dump.replace('--volume=/var/lib/etcd2', '--volume=/media/etcd')
+dump = dump.replace('ETCD_HOST_DATA_DIR=/var/lib/etcd2', 'ETCD_HOST_DATA_DIR=/media/etcd')
 
 template = json.load(open(os.path.join(CURR_DIR, 'deis.template.json'), 'r'))
 


Test patch: