Repository: deis/deis

Issue:
deploying example-dockerfile-python fails to start
This looks to be a bug with my branch of django-fsm more than anything, but more testing is needed in this department before we start pointing fingers.

Steps to reproduce:
- deploy CoreOS cluster
- set up account
- push https://github.com/opdemand/example-dockerfile-python to Deis
- run `make logs` to follow the server logs

```
Apr 22 18:46:44 deis sh[5089]: Job feline-airfield_v2.web.1.service scheduled to ab405e9f.../10.0.2.15
Apr 22 18:46:45 deis sh[3532]: 10.0.2.15 - - [22/Apr/2014:18:46:45] "GET /v1/_ping HTTP/1.1" 200 4 "-" "Go 1.1 package http"
Apr 22 18:46:45 deis sh[3532]: 2014-04-22 18:46:45,140 INFO: 10.0.2.15 - - [22/Apr/2014:18:46:45] "GET /v1/_ping HTTP/1.1" 200 4 "-" "Go 1.1 package http"
Apr 22 18:46:45 deis sh[3532]: 10.0.2.15 - - [22/Apr/2014:18:46:45] "GET /v1/repositories/bacongobbler/feline-airfield/tags HTTP/1.1" 200 152 "-" "docker/0.10.0 go/go1.2 git-commit/dc9c28f kernel/3.13.10+ os/linux arch/amd64"
Apr 22 18:46:45 deis sh[3532]: 2014-04-22 18:46:45,145 INFO: 10.0.2.15 - - [22/Apr/2014:18:46:45] "GET /v1/repositories/bacongobbler/feline-airfield/tags HTTP/1.1" 200 152 "-" "docker/0.10.0 go/go1.2 git-commit/dc9c28f kernel/3.13.10+ os/linux arch/amd64"
Apr 22 18:46:45 deis sh[3532]: 10.0.2.15 - - [22/Apr/2014:18:46:45] "GET /v1/images/846441e6f39940f295740798ef928e45846441e6f39940f295740798ef928e45/ancestry HTTP/1.1" 200 612 "-" "docker/0.10.0 go/go1.2 git-commit/dc9c28f kernel/3.13.10+ os/linux arch/amd64"
Apr 22 18:46:45 deis sh[3532]: 2014-04-22 18:46:45,147 INFO: 10.0.2.15 - - [22/Apr/2014:18:46:45] "GET /v1/images/846441e6f39940f295740798ef928e45846441e6f39940f295740798ef928e45/ancestry HTTP/1.1" 200 612 "-" "docker/0.10.0 go/go1.2 git-commit/dc9c28f kernel/3.13.10+ os/linux arch/amd64"
Apr 22 18:46:45 deis sh[3532]: 10.0.2.15 - - [22/Apr/2014:18:46:45] "GET /v1/repositories/bacongobbler/feline-airfield/images HTTP/1.1" 200 608 "-" "docker/0.10.0 go/go1.2 git-commit/dc9c28f kernel/3.13.10+ os/linux arch/amd64"
Apr 22 18:46:45 deis sh[3532]: 2014-04-22 18:46:45,148 INFO: 10.0.2.15 - - [22/Apr/2014:18:46:45] "GET /v1/repositories/bacongobbler/feline-airfield/images HTTP/1.1" 200 608 "-" "docker/0.10.0 go/go1.2 git-commit/dc9c28f kernel/3.13.10+ os/linux arch/amd64"
Apr 22 18:46:45 deis sh[5089]: Job feline-airfield_v2.web.1-announce.service scheduled to ab405e9f.../10.0.2.15
Apr 22 18:47:56 deis sh[5089]: [2014-04-22 12:47:56,851: WARNING/Worker-8] Exception in thread Thread-2:
Apr 22 18:47:56 deis sh[5089]: Traceback (most recent call last):
Apr 22 18:47:56 deis sh[5089]: File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
Apr 22 18:47:56 deis sh[5089]: self.run()
Apr 22 18:47:56 deis sh[5089]: File "/usr/lib/python2.7/threading.py", line 504, in run
Apr 22 18:47:56 deis sh[5089]: self.__target(*self.__args, **self.__kwargs)
Apr 22 18:47:56 deis sh[5089]: File "/usr/local/lib/python2.7/dist-packages/django_fsm/__init__.py", line 267, in _change_state
Apr 22 18:47:56 deis sh[5089]: return fsm_meta.field.change_state(instance, func, *args, **kwargs)
Apr 22 18:47:56 deis sh[5089]: File "/usr/local/lib/python2.7/dist-packages/django_fsm/__init__.py", line 161, in change_state
Apr 22 18:47:56 deis sh[5089]: result = method(instance, *args, **kwargs)
Apr 22 18:47:56 deis sh[5089]: File "/app/api/models.py", line 276, in start
Apr 22 18:47:56 deis sh[5089]: self._scheduler.start(self._job_id)
Apr 22 18:47:56 deis sh[5089]: File "/app/scheduler/coreos.py", line 96, in start
Apr 22 18:47:56 deis sh[5089]: self._wait_for_announcer(name, env)
Apr 22 18:47:56 deis sh[5089]: File "/app/scheduler/coreos.py", line 123, in _wait_for_announcer
Apr 22 18:47:56 deis sh[5089]: raise RuntimeError('Container failed to start')
Apr 22 18:47:56 deis sh[5089]: RuntimeError: Container failed to start
Apr 22 18:47:56 deis sh[5089]: [2014-04-22 12:47:56,860: INFO/MainProcess] Task api.tasks.start_containers[7b56fe0c-e6f4-4925-81bd-332701376555] succeeded in 85.174724498s: None
Apr 22 18:47:56 deis sh[5089]: INFO feline-airfield: Containers scaled web=1
```



Discussion:
I've fixed most of this through https://github.com/deis/slugrunner/commit/dc5f5f6465c9e2704a5ad2ea988b67aa76b78ff9, but now the announce service is not properly receiving the hostname. The [announce template](https://github.com/deis/deis/blob/master/controller/scheduler/coreos.py#L225) is not properly routing the host. For example, this is what I see in the host:

```
$ fleetctl status vulcan-idealist_v2.web.1-announce.service
● vulcan-idealist_v2.web.1-announce.service - vulcan-idealist_v2.web.1 announce
   Loaded: loaded (/run/systemd/system/vulcan-idealist_v2.web.1-announce.service; static)
   Active: active (running) since Thu 2014-04-24 18:18:35 UTC; 10min ago
  Process: 15016 ExecStartPre=/bin/sh -c until /usr/bin/docker port vulcan-idealist_v2.web.1 5000 >/dev/null 2>&1; do sleep 2; done; port=$(docker port vulcan-idealist_v2.web.1 5000 | cut -d ':' -f2); host=$(getent hosts deis | awk {'print $1'}); echo Waiting for $port/tcp...; until netstat -lnt | grep :$port >/dev/null; do sleep 1; done (code=exited, status=0/SUCCESS)
 Main PID: 15736 (sh)
   CGroup: /system.slice/vulcan-idealist_v2.web.1-announce.service
           ├─15736 /bin/sh -c port=$(docker port vulcan-idealist_v2.web.1 5000 | cut -d ':' -f2); host=$(getent hosts deis | awk {'print $1'}); echo Connected to $host:$port/tcp, publishing to etcd...; while netstat -lnt | grep :$port >/dev/null; do etcdctl set /deis/services/vulcan-idealist/vulcan-idealist_v2.web.1 $host:$port --ttl 60 >/dev/null; sleep 45; done
           └─28026 sleep 45

Apr 24 18:18:21 deis-1 systemd[1]: Starting vulcan-idealist_v2.web.1 announce...
Apr 24 18:18:35 deis-1 sh[15016]: Waiting for 49153/tcp...
Apr 24 18:18:35 deis-1 systemd[1]: Started vulcan-idealist_v2.web.1 announce.
Apr 24 18:18:35 deis-1 sh[15736]: Connected to :49153/tcp, publishing to etcd...
```

And when I run `getent hosts`, this is what I see:

```
$ getent hosts
127.0.0.1       localhost
127.0.0.1       localhost
```

So it looks like we lost the `deis` hostname. @gabrtv, any ideas? I'm thinking this was introduced after #799, but I can't confirm.


Resolving patch:
diff --git a/controller/scheduler/coreos.py b/controller/scheduler/coreos.py
--- a/controller/scheduler/coreos.py
+++ b/controller/scheduler/coreos.py
@@ -221,8 +221,9 @@ def attach(self, name):
 BindsTo={name}.service
 
 [Service]
-ExecStartPre=/bin/sh -c "until /usr/bin/docker port {name} {port} >/dev/null 2>&1; do sleep 2; done; port=$(docker port {name} {port} | cut -d ':' -f2); host=$(getent hosts deis | awk {{'print $1'}}); echo Waiting for $port/tcp...; until netstat -lnt | grep :$port >/dev/null; do sleep 1; done"
-ExecStart=/bin/sh -c "port=$(docker port {name} {port} | cut -d ':' -f2); host=$(getent hosts deis | awk {{'print $1'}}); echo Connected to $host:$port/tcp, publishing to etcd...; while netstat -lnt | grep :$port >/dev/null; do etcdctl set /deis/services/{app}/{name} $host:$port --ttl 60 >/dev/null; sleep 45; done"
+EnvironmentFile=/etc/environment
+ExecStartPre=/bin/sh -c "until /usr/bin/docker port {name} {port} >/dev/null 2>&1; do sleep 2; done; port=$(docker port {name} {port} | cut -d ':' -f2); echo Waiting for $port/tcp...; until netstat -lnt | grep :$port >/dev/null; do sleep 1; done"
+ExecStart=/bin/sh -c "port=$(docker port {name} {port} | cut -d ':' -f2); echo Connected to $COREOS_PUBLIC_IPV4:$port/tcp, publishing to etcd...; while netstat -lnt | grep :$port >/dev/null; do etcdctl set /deis/services/{app}/{name} $COREOS_PUBLIC_IPV4:$port --ttl 60 >/dev/null; sleep 45; done"
 ExecStop=/usr/bin/etcdctl rm --recursive /deis/services/{app}/{name}
 
 [X-Fleet]


Test patch: