Repository: deis/deis

Issue:
Integration tests fail on intermittent `deis run` error
This is a manifestation of #1516 in the tests.  Test output:

```
apps:run echo hello

503 SERVICE UNAVAILABLE
could not attach to container

error at command wait
--- FAIL: TestApps (135.71 seconds)
    itutils.go:206: exit status 1
```

Relevant controller logs:

```
2014-09-29 07:49:25 [169] [DEBUG] POST /api/apps/appssample/run
INFO appssample: test runs 'echo hello'
ERROR appssample: appssample_v2.admin.1 (run): could not attach to container
```



Discussion:
@gabrtv I see the same thing using the new builder #1859 (this is the only error that I get using that PR).
https://github.com/deis/deis/pull/1859#issuecomment-57156667

Another failure here: http://ci.deis.io/job/test-nightly-ruby-sinatra/151/console

Finally got a lead on the root-cause bug here, which seems very related to #1968.

After disabling cleanup of `deis run` containers and reproducing the 503 error, here's what I found:

``` console
$ fleetctl list-units
UNIT                MACHINE         ACTIVE  SUB
deis-builder@1.service      c717b778.../10.21.1.139 active  running
deis-cache@1.service        5a2f611d.../10.21.1.140 active  running
deis-controller@1.service   14e7cfb1.../10.21.2.190 active  running
deis-database-data.service  14e7cfb1.../10.21.2.190 active  exited
deis-database@1.service     14e7cfb1.../10.21.2.190 active  running
deis-logger-data.service    14e7cfb1.../10.21.2.190 active  exited
deis-logger@1.service       14e7cfb1.../10.21.2.190 active  running
deis-registry-data.service  14e7cfb1.../10.21.2.190 active  exited
deis-registry@1.service     14e7cfb1.../10.21.2.190 active  running
deis-router@1.service       5a2f611d.../10.21.1.140 active  running
example-ruby_v1.admin.1.service c717b778.../10.21.1.139 failed  failed

$ fleetctl status example-ruby_v1.admin.1.service
● example-ruby_v1.admin.1.service - example-ruby_v1.admin.1 admin command
   Loaded: loaded (/run/fleet/units/example-ruby_v1.admin.1.service; linked-runtime)
   Active: failed (Result: exit-code) since Tue 2014-09-30 15:56:22 UTC; 51s ago
  Process: 4217 ExecStartPre=/bin/sh -c IMAGE=$(etcdctl get /deis/registry/host 2>&1):$(etcdctl get /deis/registry/port 2>&1)/deis/helloworld:v1; docker pull $IMAGE (code=exited, status=1/FAILURE)

Sep 30 15:56:22 ip-10-21-1-139.us-west-2.compute.internal sh[4217]: Pulling repository 10.21.2.190:5000/deis/helloworld
Sep 30 15:56:22 ip-10-21-1-139.us-west-2.compute.internal sh[4217]: 2014/09/30 15:56:22 Error: image deis/helloworld not found
Sep 30 15:56:22 ip-10-21-1-139.us-west-2.compute.internal systemd[1]: example-ruby_v1.admin.1.service: control process exited, code=exited status=1
Sep 30 15:56:22 ip-10-21-1-139.us-west-2.compute.internal systemd[1]: Failed to start example-ruby_v1.admin.1 admin command.
Sep 30 15:56:22 ip-10-21-1-139.us-west-2.compute.internal systemd[1]: Unit example-ruby_v1.admin.1.service entered failed state.
```

The `deis run` _should_ have run against v2 of the app, but instead ran against v1.  That's the bug.

We should still have `deis/helloworld` in the registry so it doesn't fail, but that's a separate issue.


Resolving patch:
diff --git a/controller/scheduler/coreos.py b/controller/scheduler/coreos.py
--- a/controller/scheduler/coreos.py
+++ b/controller/scheduler/coreos.py
@@ -232,19 +232,19 @@ def _destroy_announcer(self, name):
     def _destroy_log(self, name):
         return self._delete_unit(name+'-log')
 
-    def run(self, name, image, command):
+    def run(self, name, image, command):  # noqa
         """Run a one-off command"""
         self._create_container(name, image, command, copy.deepcopy(RUN_TEMPLATE))
 
-        # wait for the container to return something
-        for _ in range(1200):
+        # wait for the container to get scheduled
+        for _ in range(30):
             states = self._get_state(name)
             if states and len(states.get('states', [])) == 1:
                 state = states.get('states')[0]
-                subState = state.get('systemdSubState')
-                if subState == 'exited' or subState == 'failed' or subState == 'dead':
-                    break
+                break
             time.sleep(1)
+        else:
+            raise RuntimeError('container did not report state')
         machineID = state.get('machineID')
 
         # find the machine
@@ -268,28 +268,50 @@ def run(self, name, image, command):
         ssh = paramiko.SSHClient()
         ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
         ssh.connect(primaryIP, username="core", pkey=pkey)
-
-        # get a pty so stdout/stderr look right
+        # share a transport
         tran = ssh.get_transport()
-        chan = tran.open_session()
-        chan.get_pty()
-        out = chan.makefile()
-
-        # exec the command to gather container output
-        chan.exec_command('docker logs {name}'.format(**locals()))
-        rc, output = chan.recv_exit_status(), out.read()
-        if rc != 0:
+
+        def _do_ssh(cmd):
+            chan = tran.open_session()
+            # get a pty so stdout/stderr look right
+            chan.get_pty()
+            out = chan.makefile()
+            chan.exec_command(cmd)
+            rc, output = chan.recv_exit_status(), out.read()
+            return rc, output
+
+        # wait for container to start
+        for _ in range(1200):
+            rc, _ = _do_ssh('docker inspect {name}'.format(**locals()))
+            if rc == 0:
+                break
+            time.sleep(1)
+        else:
+            raise RuntimeError('container failed to start on host')
+
+        # wait for container to complete
+        for _ in range(1200):
+            _rc, _output = _do_ssh('docker inspect {name}'.format(**locals()))
+            if _rc != 0:
+                raise RuntimeError('failed to inspect container')
+            _container = json.loads(_output)
+            finished_at = _container[0]["State"]["FinishedAt"]
+            if not finished_at.startswith('0001'):
+                break
+            time.sleep(1)
+        else:
+            raise RuntimeError('container timed out')
+
+        # gather container output
+        _rc, output = _do_ssh('docker logs {name}'.format(**locals()))
+        if _rc != 0:
             raise RuntimeError('could not attach to container')
 
-        # use another channel to inspect the container
-        chan = tran.open_session()
-        chan.get_pty()
-        out = chan.makefile()
-        chan.exec_command('docker inspect {name}'.format(**locals()))
-        rc, inspect_output = chan.recv_exit_status(), out.read()
-        if rc != 0:
+        # determine container exit code
+        _rc, _output = _do_ssh('docker inspect {name}'.format(**locals()))
+        if _rc != 0:
             raise RuntimeError('could not determine exit code')
-        container = json.loads(inspect_output)
+        container = json.loads(_output)
         rc = container[0]["State"]["ExitCode"]
 
         # cleanup


Test patch: