Repository: deis/deis

Issue:
n-node vagrant broadcasting wrong network bridge to etcd
When deploying a 2-node cluster with Vagrant, the wrong public IP address for the deis components are being published to etcd. For example, here is the network information for the second node:

```
core@deis-2 ~ $ ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.1.42.1 netmask 255.255.0.0 broadcast 0.0.0.0
inet6 fe80::5484:7aff:fefe:9799 prefixlen 64 scopeid 0x20<link>
ether 56:84:7a:fe:97:99 txqueuelen 0 (Ethernet)
RX packets 63078 bytes 3776393 (3.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 59301 bytes 316543797 (301.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255
inet6 fe80::a00:27ff:feac:4e08 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:ac:4e:08 txqueuelen 1000 (Ethernet)
RX packets 1161605 bytes 990614320 (944.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 393652 bytes 24244415 (23.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.8.101 netmask 255.255.255.0 broadcast 172.17.8.255
inet6 fe80::a00:27ff:fe1b:1b97 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:1b:1b:97 txqueuelen 1000 (Ethernet)
RX packets 296211 bytes 44085574 (42.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 180764 bytes 27624609 (26.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 28425 bytes 2540130 (2.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 28425 bytes 2540130 (2.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

veth8a3c: flags=67<UP,BROADCAST,RUNNING> mtu 1500
inet6 fe80::9c69:c4ff:fe99:bc1d prefixlen 64 scopeid 0x20<link>
ether 9e:69:c4:99:bc:1d txqueuelen 1000 (Ethernet)
RX packets 63076 bytes 4659305 (4.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 59307 bytes 316544265 (301.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

core@deis-2 ~ $ netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.0.2.2 0.0.0.0 UG 0 0 0 enp0s3
10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 enp0s3
10.0.2.2 0.0.0.0 255.255.255.255 UH 0 0 0 enp0s3
10.1.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.17.8.0 0.0.0.0 255.255.255.0 U 0 0 0 enp0s8
```

and here is the broadcasted host IP for `deis-controller`:

```
$ etcdctl get /deis/controller/host
10.0.2.15
```

This is because enp0s3 is set as the default gateway, which resolves to 10.0.2.15. Instead, it should be 172.17.8.101, which is enp0s8.



Discussion:
Wondering if this is because of: https://github.com/coreos/coreos-vagrant/blob/master/override-plugin.rb#L83,L92

That always looked odd to me, like it's "last IP wins".

This is actually likely due to our use of `$private_ipv4` in our `user-data` file.

The upstream Vagrant user-data file uses $public_ipv4 for both etcd listeners: https://github.com/coreos/coreos-vagrant/blob/master/user-data.sample

We use $public_ipv4 and $private_ipv4, so that the same user-data file can be used on ec2, rackspace, vagrant. The assumption was that private_ipv4 should respect COREOS_PRIVATE_IPV4 in /etc/environment, which is the same as COREOS_PUBLIC_IPV4: https://github.com/coreos/coreos-vagrant/blob/master/override-plugin.rb#L88-L92

So, that begs the question: just how are `$public_ipv4` and `$private_ipv4` set on Vagrant?

Actually, changing the coreos-vagrant user-data file to use `$private_ipv4` like ours works just fine:

``` console
$ cat user-data
#cloud-config

coreos:
  etcd:
      #generate a new token for each unique cluster from https://discovery.etcd.io/new
      discovery: https://discovery.etcd.io/f89d46586fbc3f7ec3fabc0ecf1a42e4
      addr: $public_ipv4:4001
      peer-addr: $private_ipv4:7001
  units:
    - name: etcd.service
      command: start
    - name: fleet.service
      command: start
      runtime: no
      content: |
        [Unit]
        Description=fleet

        [Service]
        Environment=FLEET_PUBLIC_IP=$public_ipv4
        ExecStart=/usr/bin/fleet
```

``` console
core@core-01 ~ $ journalctl -fu etcd
-- Logs begin at Tue 2014-04-29 22:41:58 UTC. --
Apr 29 22:42:10 core-01 systemd[1]: Starting etcd...
Apr 29 22:42:10 core-01 systemd[1]: Started etcd.
Apr 29 22:42:10 core-01 etcd[3191]: [etcd] Apr 29 22:42:10.169 INFO      | Discovery via https://discovery.etcd.io using prefix /f89d46586fbc3f7ec3fabc0ecf1a42e4.
Apr 29 22:42:10 core-01 etcd[3191]: [etcd] Apr 29 22:42:10.416 INFO      | Discovery _state was empty, so this machine is the initial leader.
Apr 29 22:42:10 core-01 etcd[3191]: [etcd] Apr 29 22:42:10.416 INFO      | 4c54f8ca90964f0e8c538f3b5427d5c9: state changed from 'stopped' to 'follower'.
Apr 29 22:42:10 core-01 etcd[3191]: [etcd] Apr 29 22:42:10.416 INFO      | 4c54f8ca90964f0e8c538f3b5427d5c9: state changed from 'follower' to 'leader'.
Apr 29 22:42:10 core-01 etcd[3191]: [etcd] Apr 29 22:42:10.417 INFO      | 4c54f8ca90964f0e8c538f3b5427d5c9: leader changed from '' to '4c54f8ca90964f0e8c538f3b5427d5c9'.
Apr 29 22:42:10 core-01 etcd[3191]: [etcd] Apr 29 22:42:10.418 INFO      | etcd server [name 4c54f8ca90964f0e8c538f3b5427d5c9, listen on [::]:4001, advertised url http://172.17.8.101:4001]
Apr 29 22:42:10 core-01 etcd[3191]: [etcd] Apr 29 22:42:10.419 INFO      | peer server [name 4c54f8ca90964f0e8c538f3b5427d5c9, listen on [::]:7001, advertised url http://172.17.8.101:7001]
Apr 29 22:42:31 core-01 etcd[3191]: [etcd] Apr 29 22:42:31.229 INFO      | 4c54f8ca90964f0e8c538f3b5427d5c9: peer added: '2222d44fe77843b2974358d99dd4ff0e'
```

Actually, things are hunky-dory for us in this department as well - deis looks just fine on the public_ip/private_ip issue. If the containers are advertising the public route, we can force it to use this as the default route.


Resolving patch:
diff --git a/controller/scheduler/coreos.py b/controller/scheduler/coreos.py
--- a/controller/scheduler/coreos.py
+++ b/controller/scheduler/coreos.py
@@ -223,7 +223,7 @@ def attach(self, name):
 [Service]
 EnvironmentFile=/etc/environment
 ExecStartPre=/bin/sh -c "until /usr/bin/docker port {name} {port} >/dev/null 2>&1; do sleep 2; done; port=$(docker port {name} {port} | cut -d ':' -f2); echo Waiting for $port/tcp...; until netstat -lnt | grep :$port >/dev/null; do sleep 1; done"
-ExecStart=/bin/sh -c "port=$(docker port {name} {port} | cut -d ':' -f2); echo Connected to $COREOS_PUBLIC_IPV4:$port/tcp, publishing to etcd...; while netstat -lnt | grep :$port >/dev/null; do etcdctl set /deis/services/{app}/{name} $COREOS_PUBLIC_IPV4:$port --ttl 60 >/dev/null; sleep 45; done"
+ExecStart=/bin/sh -c "port=$(docker port {name} {port} | cut -d ':' -f2); echo Connected to $COREOS_PRIVATE_IPV4:$port/tcp, publishing to etcd...; while netstat -lnt | grep :$port >/dev/null; do etcdctl set /deis/services/{app}/{name} $COREOS_PRIVATE_IPV4:$port --ttl 60 >/dev/null; sleep 45; done"
 ExecStop=/usr/bin/etcdctl rm --recursive /deis/services/{app}/{name}
 
 [X-Fleet]


Test patch: