Repository: deis/deis

Issue:
deis run cannot handle large output
I need to get a large generated text file from deis. I do the following

`deis run -- cat foo`

If foo is large enough, in my case is (7468 lines   26438 words  186989 bytes) , then `deis run` hangs and no output gets displayed.

Running `deis logs` afterwards shows the full output of the cat command in the logs.

If I run `deis run -- "cat foo | head -n 20"` everything works OK.



Discussion:
I can reproduce this using Deis master and vagrant, although my numbers are different. I committed the plain text [jargon file](http://www.gutenberg.org/cache/epub/3008/pg3008.txt) to example-go and pushed that for testing.

``` console
$ deis run 'cat jargon_file.txt | head -n 48500 | wc -c'
Running `cat jargon_file.txt | head -n 48500 | wc -c`...
2179187
$ deis run 'cat jargon_file.txt | head -n 48600 | wc -c'
Running `cat jargon_file.txt | head -n 48600 | wc -c`...
2182300
$ deis run 'cat jargon_file.txt | head -n 48500'  # tons of output
...
6101. jarg422h.htm#%3d%20G%20%3d
6102. jarg422h.htm#gorets
6103. jarg422h.htm#gopher
$ deis run 'cat jargon_file.txt | head -n 48600'  # this will hang forever
```

Whenever one of these `deis run` processes hangs, a fleet unit is orphaned:

``` console
$ fleetctl list-units
UNIT                    MACHINE             ACTIVE      SUB
deis-builder.service            498d07d3.../172.17.8.100    active      running
...
deis-store-volume.service       ed25cb20.../172.17.8.102    active      running
yeasty-teamwork_v2.run.1.service    ed25cb20.../172.17.8.102    inactive    dead
yeasty-teamwork_v2.run.2.service    498d07d3.../172.17.8.100    inactive    dead
yeasty-teamwork_v3.run.17.service   ba4aa214.../172.17.8.101    inactive    dead
yeasty-teamwork_v3.run.26.service   ba4aa214.../172.17.8.101    inactive    dead
yeasty-teamwork_v3.run.30.service   ed25cb20.../172.17.8.102    inactive    dead
yeasty-teamwork_v3.run.36.service   498d07d3.../172.17.8.100    inactive    dead
yeasty-teamwork_v3.run.41.service   ba4aa214.../172.17.8.101    inactive    dead
yeasty-teamwork_v3.run.42.service   ed25cb20.../172.17.8.102    inactive    dead
yeasty-teamwork_v3.run.45.service   498d07d3.../172.17.8.100    inactive    dead
yeasty-teamwork_v3.web.1.service    ed25cb20.../172.17.8.102    active      running
yeasty-teamwork_v3.web.2.service    498d07d3.../172.17.8.100    active      running
yeasty-teamwork_v3.web.3.service    ed25cb20.../172.17.8.102    active      running
yeasty-teamwork_v3.web.4.service    ba4aa214.../172.17.8.101    active      running
yeasty-teamwork_v3.web.5.service    498d07d3.../172.17.8.100    active      running
yeasty-teamwork_v3.web.6.service    ba4aa214.../172.17.8.101    active      running
```

A stuck command like this will eventually cause the `gunicorn` web server to send `SIGABRT` to the worker, which should call a [`worker_abort`](http://docs.gunicorn.org/en/latest/settings.html#worker-abort) function if it is defined. That looks like the only reasonable way to ensure we always clean up the fleet unit.

However, `worker_abort` is never called in deis-controller. Although it works as advertised from a sample `gunicorn` app I created. Something in the container environment or in our controller code seems to be short-circuiting the normal signal handling. (I've also tried many variations on `except SystemExit` and `atexit.register`: no joy.)

Update: it is the [signal handling code](https://github.com/deis/deis/blob/master/controller/bin/boot#L76-L83) in deis-controller that interferes with some gunicorn [server hooks](http://docs.gunicorn.org/en/latest/settings.html#worker-abort). Commenting out the signal hander in `boot` allows a `worker_abort` function to be called in this case.

But frankly I'm not sure it's going to be possible to clean up an orphaned fleet unit in this case--the server hook code lives at the WSGI level and really shouldn't be importing the Django world just to call a [cleanup method](https://github.com/deis/deis/blob/master/controller/scheduler/fleet.py#L201).


Resolving patch:
diff --git a/controller/scheduler/fleet.py b/controller/scheduler/fleet.py
--- a/controller/scheduler/fleet.py
+++ b/controller/scheduler/fleet.py
@@ -268,7 +268,8 @@ def _do_ssh(cmd):
                 chan.get_pty()
                 out = chan.makefile()
                 chan.exec_command(cmd)
-                rc, output = chan.recv_exit_status(), out.read()
+                output = out.read()
+                rc = chan.recv_exit_status()
                 return rc, output
 
             # wait for container to launch


Test patch: