Repository: readthedocs/readthedocs.org

Issue:
Keep the command's output when it's killed
When a command is killed because of memory consumption we replace the output with a message

https://github.com/rtfd/readthedocs.org/blob/85d9a1fb236e2c2149358b1cf462159e523e80bb/readthedocs/doc_builder/environments.py#L274-L276

the output can be helpful to users to debug the build. I guess we should only show the error about the memory in the top? Or maybe just concatenated the output with that msg?

Raised in https://github.com/rtfd/readthedocs.org/issues/4946#issuecomment-444232307
Related to https://github.com/rtfd/readthedocs.org/issues/4468


Discussion:
This would be very useful :) right now I need to blindly reduce the size of my example, make another pull request, rerun the build and hope :)
I suppose the reason why we put this pre-defined output is that when the command is killed, we don't get any output in return from docker.
Interesting, I'll try to reproduce that locally and see
So, after some attempts, I was able to reproduce and catch locally, this is what I found

```
296  	            killed_in_output = 'Killed' in '\n'.join(self.output.splitlines()[-15:])
297  	            if self.exit_code == DOCKER_OOM_EXIT_CODE or (self.exit_code == 1 and killed_in_output):
298  	                from celery.contrib import rdb
299  	                rdb.set_trace()
300  ->	                self.output = _('Command killed due to excessive memory '
301  	                                'consumption\n')
302  	        except DockerAPIError:
303  	            self.exit_code = -1
304  	            if self.output is None or not self.output:
305  	                self.output = _('Command exited abnormally')
(Pdb) p self.output
'Collecting Pygments==2.2.0\n  Using cached https://files.pythonhosted.org/packages/02/ee/b6e02dc6529e82b75bb06823ff7d005b141037cb1416b10c6f00fc419dca/Pygments-2.2.0-py2.py3-none-any.whl\nRequirement already up-to-date: setuptools<40 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages\nCollecting docutils==0.13.1\n  Using cached https://files.pythonhosted.org/packages/7c/30/8fb30d820c012a6f701a66618ce065b6d61d08ac0a77e47fc7808dbaee47/docutils-0.13.1-py3-none-any.whl\nCollecting mock==1.0.1\nRequirement already up-to-date: pillow==2.6.1 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages\nRequirement already up-to-date: alabaster!=0.7.5,<0.8,>=0.7 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages\nCollecting commonmark==0.5.4\nRequirement already up-to-date: recommonmark==0.4.0 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages\nCollecting sphinx<1.8\n  Using cached https://files.pythonhosted.org/packages/90/f9/a0babe32c78480994e4f1b93315558f5ed756104054a7029c672a8d77b72/Sphinx-1.7.9-py2.py3-none-any.whl\nRequirement already up-to-date: sphinx-rtd-theme<0.5 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages\nRequirement already up-to-date: readthedocs-sphinx-ext<0.6 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages\nRequirement already up-to-date: imagesize in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from sphinx<1.8)\nRequirement already up-to-date: packaging in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from sphinx<1.8)\nRequirement already up-to-date: babel!=2.0,>=1.3 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from sphinx<1.8)\nRequirement already up-to-date: Jinja2>=2.3 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from sphinx<1.8)\nCollecting requests>=2.0.0 (from sphinx<1.8)\n  Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl\nRequirement already up-to-date: sphinxcontrib-websupport in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from sphinx<1.8)\nRequirement already up-to-date: snowballstemmer>=1.1 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from sphinx<1.8)\nCollecting six>=1.5 (from sphinx<1.8)\n  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl\nRequirement already up-to-date: pyparsing>=2.0.2 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from packaging->sphinx<1.8)\nRequirement already up-to-date: pytz>=0a in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from babel!=2.0,>=1.3->sphinx<1.8)\nRequirement already up-to-date: MarkupSafe>=0.23 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from Jinja2>=2.3->sphinx<1.8)\nCollecting idna<2.9,>=2.5 (from requests>=2.0.0->sphinx<1.8)\n  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl\nRequirement already up-to-date: certifi>=2017.4.17 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from requests>=2.0.0->sphinx<1.8)\nRequirement already up-to-date: urllib3<1.25,>=1.21.1 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from requests>=2.0.0->sphinx<1.8)\nRequirement already up-to-date: chardet<3.1.0,>=3.0.2 in /home/stsewd/rtd/readthedocs.org/user_builds/read-the-docs/envs/latest/lib/python3.6/site-packages (from requests>=2.0.0->sphinx<1.8)\nInstalling collected packages: Pygments, docutils, mock, commonmark, idna, requests, six, sphinx\n  Found existing installation: Pygments 2.3.0\n    Uninstalling Pygments-2.3.0:\n      Successfully uninstalled Pygments-2.3.0\n  Found existing installation: docutils 0.14\n    Uninstalling docutils-0.14:\n      Successfully uninstalled docutils-0.14\n  Found existing installation: mock 2.0.0\n    Uninstalling mock-2.0.0:\n      Successfully uninstalled mock-2.0.0\n  Found existing installation: CommonMark 0.5.5\n    Uninstalling CommonMark-0.5.5:\n      Successfully uninstalled CommonMark-0.5.5\n  Found existing installation: idna 2.7\n    Uninstalling idna-2.7:\n      Successfully uninstalled idna-2.7\nKilled\n'
(Pdb) 

```

We have some output, it may be helpful for debugging, and there is a `Killed` at the end, so I think we can append the message or just rely on the top message to communicate the error.

I'm going to keep investigating if there are some side effects.

Resolving patch:
diff --git a/readthedocs/doc_builder/environments.py b/readthedocs/doc_builder/environments.py
--- a/readthedocs/doc_builder/environments.py
+++ b/readthedocs/doc_builder/environments.py
@@ -3,10 +3,13 @@
 """Documentation Builder Environments."""
 
 from __future__ import (
-    absolute_import, division, print_function, unicode_literals)
+    absolute_import,
+    division,
+    print_function,
+    unicode_literals,
+)
 
 import logging
-import json
 import os
 import re
 import socket
@@ -32,13 +35,28 @@
 from readthedocs.restapi.client import api as api_v2
 
 from .constants import (
-    DOCKER_HOSTNAME_MAX_LEN, DOCKER_IMAGE, DOCKER_LIMITS, DOCKER_OOM_EXIT_CODE,
-    DOCKER_SOCKET, DOCKER_TIMEOUT_EXIT_CODE, DOCKER_VERSION,
-    MKDOCS_TEMPLATE_DIR, SPHINX_TEMPLATE_DIR)
+    DOCKER_HOSTNAME_MAX_LEN,
+    DOCKER_IMAGE,
+    DOCKER_LIMITS,
+    DOCKER_OOM_EXIT_CODE,
+    DOCKER_SOCKET,
+    DOCKER_TIMEOUT_EXIT_CODE,
+    DOCKER_VERSION,
+    MKDOCS_TEMPLATE_DIR,
+    SPHINX_TEMPLATE_DIR,
+)
 from .exceptions import (
-    BuildEnvironmentCreationFailed, BuildEnvironmentError,
-    BuildEnvironmentException, BuildEnvironmentWarning, BuildTimeoutError,
-    ProjectBuildsSkippedError, VersionLockedError, YAMLParseError, MkDocsYAMLParseError)
+    BuildEnvironmentCreationFailed,
+    BuildEnvironmentError,
+    BuildEnvironmentException,
+    BuildEnvironmentWarning,
+    BuildTimeoutError,
+    MkDocsYAMLParseError,
+    ProjectBuildsSkippedError,
+    VersionLockedError,
+    YAMLParseError,
+)
+
 
 log = logging.getLogger(__name__)
 
@@ -295,8 +313,9 @@ def run(self):
             # is in the last 15 lines of the command's output
             killed_in_output = 'Killed' in '\n'.join(self.output.splitlines()[-15:])
             if self.exit_code == DOCKER_OOM_EXIT_CODE or (self.exit_code == 1 and killed_in_output):
-                self.output = _('Command killed due to excessive memory '
-                                'consumption\n')
+                self.output += str(_(
+                    '\n\nCommand killed due to excessive memory consumption\n'
+                ))
         except DockerAPIError:
             self.exit_code = -1
             if self.output is None or not self.output:


Test patch:
diff --git a/readthedocs/rtd_tests/tests/test_doc_building.py b/readthedocs/rtd_tests/tests/test_doc_building.py
--- a/readthedocs/rtd_tests/tests/test_doc_building.py
+++ b/readthedocs/rtd_tests/tests/test_doc_building.py
@@ -6,7 +6,11 @@
 * the Command wrappers encapsulate the bytes and expose unicode
 """
 from __future__ import (
-    absolute_import, division, print_function, unicode_literals)
+    absolute_import,
+    division,
+    print_function,
+    unicode_literals,
+)
 
 import json
 import os
@@ -27,8 +31,11 @@
 from readthedocs.builds.models import Version
 from readthedocs.doc_builder.config import load_yaml_config
 from readthedocs.doc_builder.environments import (
-    BuildCommand, DockerBuildCommand, DockerBuildEnvironment,
-    LocalBuildEnvironment)
+    BuildCommand,
+    DockerBuildCommand,
+    DockerBuildEnvironment,
+    LocalBuildEnvironment,
+)
 from readthedocs.doc_builder.exceptions import BuildEnvironmentError
 from readthedocs.doc_builder.python_environments import Conda, Virtualenv
 from readthedocs.projects.models import Project
@@ -36,6 +43,7 @@
 from readthedocs.rtd_tests.mocks.paths import fake_paths_lookup
 from readthedocs.rtd_tests.tests.test_config_integration import create_load
 
+
 DUMMY_BUILD_ID = 123
 SAMPLE_UNICODE = u'HérÉ îß sömê ünïçó∂é'
 SAMPLE_UTF8_BYTES = SAMPLE_UNICODE.encode('utf-8')
@@ -1127,9 +1135,10 @@ def test_command_oom_kill(self):
         cmd.build_env.get_client.return_value = self.mocks.docker_client
         type(cmd.build_env).container_id = PropertyMock(return_value='foo')
         cmd.run()
-        self.assertEqual(
-            str(cmd.output),
-            u'Command killed due to excessive memory consumption\n')
+        self.assertIn(
+            'Command killed due to excessive memory consumption\n',
+            str(cmd.output)
+        )
 
 
 class TestPythonEnvironment(TestCase):