Repository: deis/deis

Issue:
Need a way to stop one-off stale commands
With `deis info` I get 

```
=== xx-prod Application
{
  "updated": "2015-02-04T10:08:37UTC", 
  "uuid": "b1812xxx1", 
  "created": "2015-01-20T12:46:39UTC", 
  "url": "xx-prod.masterfirefoxos.com", 
  "owner": "xx", 
  "id": "xx-prod", 
  "structure": {
    "cmd": 3, 
    "run": 32
  }
}
```
1. Deis believes that there are 32 "run" processes but "docker ps" in all hosts returns 0 "run" processes
2. Even if there were 32 running processes, we need a way to stop them

`deis scale run=0` returns 

```
400 BAD REQUEST
Container type run does not exist in application
```



Discussion:
I think this is just false info that shouldn't be shown in the client. run processes are the process type when you perform `deis run`, which mean you've run 32 `deis apps:run` commands against it. :)

I didn't explain myself correctly:
It should totally be shown in the client, when they do exist. These don't exist (no docker containers related to the "run" command") and I've no way to reset Deis.

These containers are [killed after they're executed](https://github.com/deis/deis/blob/master/controller/scheduler/fleet.py#L306-L308), so they do not exist in the scheduler. It's just an implementation detail of the platform so we can run multiple one-off commands in parallel. We keep the database entry around such that each one-off job has a unique number, so there's nothing that you need to do on your end to "reset Deis".

what version of deis are you you running, by the way?

OK I get that but in this case why other apps on which I've used `deis run` don't display this line?

```
=== xx-dev Application
{
  "updated": "2015-01-20T14:59:25UTC", 
  "uuid": "93d069d8-xx-4bc8-9ef3-163c1203c502", 
  "created": "2015-01-20T12:45:17UTC", 
  "url": "xx-dev.masterfirefoxos.com", 
  "owner": "xx", 
  "id": "xx-dev", 
  "structure": {
    "cmd": 3
  }
}
```

(I'm running 1.1.1)

More info:
Running other commands in the same app doesn't increase the displayed "run" number, i.e. the output of `deis info` is exactly the same. I guess that Deis believes that there are 32 currently running "run" commands.

My assumption is that since we do not clean up the Container objects with a type of `run` in the database, the [structure is updated](https://github.com/deis/deis/blob/master/controller/api/models.py#L230-L232) with the run values, when in reality these should [be excluded](https://github.com/deis/deis/blob/master/controller/api/models.py#L270).

More info:
Deis probably believes that there are 32 commands running due to #3035.  

if you could ssh into one of the boxes and run `fleetctl list-units`, that would be helpful to show if there is indeed 32 commands running.

These are the units related to the app under investigation, returned by `fleetctl list-units`

```
masterfirefoxos-prod_v48.run.20.service 1ecca637.../10.152.74.15    inactive    dead
masterfirefoxos-prod_v52.run.22.service 6e753008.../10.152.74.225   inactive    dead
masterfirefoxos-prod_v53.run.23.service 96834fc1.../10.152.74.172   inactive    dead
masterfirefoxos-prod_v53.run.26.service 1ecca637.../10.152.74.15    inactive    dead
masterfirefoxos-prod_v53.run.27.service 6e753008.../10.152.74.225   inactive    dead
masterfirefoxos-prod_v53.run.28.service 96834fc1.../10.152.74.172   inactive    dead
masterfirefoxos-prod_v53.run.29.service 96834fc1.../10.152.74.172   inactive    dead
masterfirefoxos-prod_v53.run.30.service 6e753008.../10.152.74.225   inactive    dead
masterfirefoxos-prod_v55.run.33.service 1ecca637.../10.152.74.15    inactive    dead
masterfirefoxos-prod_v58.run.36.service 6e753008.../10.152.74.225   inactive    dead
masterfirefoxos-prod_v58.run.37.service 96834fc1.../10.152.74.172   inactive    dead
masterfirefoxos-prod_v59.run.41.service 1ecca637.../10.152.74.15    inactive    dead
masterfirefoxos-prod_v60.run.42.service 6e753008.../10.152.74.225   inactive    dead
```


Resolving patch:
diff --git a/controller/scheduler/fleet.py b/controller/scheduler/fleet.py
--- a/controller/scheduler/fleet.py
+++ b/controller/scheduler/fleet.py
@@ -221,92 +221,95 @@ def run(self, name, image, entrypoint, command):  # noqa
             time.sleep(1)
         else:
             raise RuntimeError('container did not report state')
-        machineID = state.get('machineID')
-
-        # find the machine
-        machines = self._get_machines()
-        if not machines:
-            raise RuntimeError('no available hosts to run command')
-
-        # find the machine's primaryIP
-        primaryIP = None
-        for m in machines.get('machines', []):
-            if m['id'] == machineID:
-                primaryIP = m['primaryIP']
-        if not primaryIP:
-            raise RuntimeError('could not find host')
-
-        # prepare ssh key
-        file_obj = cStringIO.StringIO(base64.b64decode(self.pkey))
-        pkey = paramiko.RSAKey(file_obj=file_obj)
-
-        # grab output via docker logs over SSH
-        ssh = paramiko.SSHClient()
-        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
-        ssh.connect(primaryIP, username="core", pkey=pkey)
-        # share a transport
-        tran = ssh.get_transport()
-
-        def _do_ssh(cmd):
-            chan = tran.open_session()
-            # get a pty so stdout/stderr look right
-            chan.get_pty()
-            out = chan.makefile()
-            chan.exec_command(cmd)
-            rc, output = chan.recv_exit_status(), out.read()
-            return rc, output
-
-        # wait for container to launch
-        # we loop indefinitely here, as we have no idea how long the docker pull will take
-        while True:
-            rc, _ = _do_ssh('docker inspect {name}'.format(**locals()))
-            if rc == 0:
-                break
-            time.sleep(1)
-        else:
-            raise RuntimeError('failed to create container')
 
-        # wait for container to start
-        for _ in range(2):
-            _rc, _output = _do_ssh('docker inspect {name}'.format(**locals()))
+        try:
+            machineID = state.get('machineID')
+
+            # find the machine
+            machines = self._get_machines()
+            if not machines:
+                raise RuntimeError('no available hosts to run command')
+
+            # find the machine's primaryIP
+            primaryIP = None
+            for m in machines.get('machines', []):
+                if m['id'] == machineID:
+                    primaryIP = m['primaryIP']
+            if not primaryIP:
+                raise RuntimeError('could not find host')
+
+            # prepare ssh key
+            file_obj = cStringIO.StringIO(base64.b64decode(self.pkey))
+            pkey = paramiko.RSAKey(file_obj=file_obj)
+
+            # grab output via docker logs over SSH
+            ssh = paramiko.SSHClient()
+            ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+            ssh.connect(primaryIP, username="core", pkey=pkey)
+            # share a transport
+            tran = ssh.get_transport()
+
+            def _do_ssh(cmd):
+                chan = tran.open_session()
+                # get a pty so stdout/stderr look right
+                chan.get_pty()
+                out = chan.makefile()
+                chan.exec_command(cmd)
+                rc, output = chan.recv_exit_status(), out.read()
+                return rc, output
+
+            # wait for container to launch
+            # we loop indefinitely here, as we have no idea how long the docker pull will take
+            while True:
+                rc, _ = _do_ssh('docker inspect {name}'.format(**locals()))
+                if rc == 0:
+                    break
+                time.sleep(1)
+            else:
+                raise RuntimeError('failed to create container')
+
+            # wait for container to start
+            for _ in range(2):
+                _rc, _output = _do_ssh('docker inspect {name}'.format(**locals()))
+                if _rc != 0:
+                    raise RuntimeError('failed to inspect container')
+                _container = json.loads(_output)
+                started_at = _container[0]["State"]["StartedAt"]
+                if not started_at.startswith('0001'):
+                    break
+                time.sleep(1)
+            else:
+                raise RuntimeError('container failed to start')
+
+            # wait for container to complete
+            for _ in range(1200):
+                _rc, _output = _do_ssh('docker inspect {name}'.format(**locals()))
+                if _rc != 0:
+                    raise RuntimeError('failed to inspect container')
+                _container = json.loads(_output)
+                finished_at = _container[0]["State"]["FinishedAt"]
+                if not finished_at.startswith('0001'):
+                    break
+                time.sleep(1)
+            else:
+                raise RuntimeError('container timed out')
+
+            # gather container output
+            _rc, output = _do_ssh('docker logs {name}'.format(**locals()))
             if _rc != 0:
-                raise RuntimeError('failed to inspect container')
-            _container = json.loads(_output)
-            started_at = _container[0]["State"]["StartedAt"]
-            if not started_at.startswith('0001'):
-                break
-            time.sleep(1)
-        else:
-            raise RuntimeError('container failed to start')
+                raise RuntimeError('could not attach to container')
 
-        # wait for container to complete
-        for _ in range(1200):
+            # determine container exit code
             _rc, _output = _do_ssh('docker inspect {name}'.format(**locals()))
             if _rc != 0:
-                raise RuntimeError('failed to inspect container')
-            _container = json.loads(_output)
-            finished_at = _container[0]["State"]["FinishedAt"]
-            if not finished_at.startswith('0001'):
-                break
-            time.sleep(1)
-        else:
-            raise RuntimeError('container timed out')
-
-        # gather container output
-        _rc, output = _do_ssh('docker logs {name}'.format(**locals()))
-        if _rc != 0:
-            raise RuntimeError('could not attach to container')
-
-        # determine container exit code
-        _rc, _output = _do_ssh('docker inspect {name}'.format(**locals()))
-        if _rc != 0:
-            raise RuntimeError('could not determine exit code')
-        container = json.loads(_output)
-        rc = container[0]["State"]["ExitCode"]
-
-        # cleanup
-        self._destroy_container(name)
-        self._wait_for_destroy(name)
+                raise RuntimeError('could not determine exit code')
+            container = json.loads(_output)
+            rc = container[0]["State"]["ExitCode"]
+
+        finally:
+            # cleanup
+            self._destroy_container(name)
+            self._wait_for_destroy(name)
 
         # return rc and output
         return rc, output


Test patch: