Repository: tox-dev/tox
Pull request #760 (2018-02-02 23:07:54)

Display exit code with InvocationErrors
This is a PR for @blueyed patch (Issue #290), supplemented with tests. 
For exit codes larger than 128, a hint is given: on unix systems this often means a fatal error.
(Not always, since an application might exit with a large custom error code)

Not showing this hint for windows platform was considered, but discarded to keep code short and
simplify tests.

Before:

> ERROR: InvocationError: '/home/ederag/share/prog/python/tox/.tox/exitcode/bin/python3.6 -c import sys; sys.exit(5)'

After:

> ERROR: InvocationError for command '/home/ederag/share/prog/python/tox/.tox/exitcode/bin/python3.6 -c import sys; sys.exit(5)' (exited with code 5)

or, for an exit code larger than 128:

> ERROR: InvocationError for command '/home/ederag/share/prog/python/tox/.tox/exitcode/bin/python3.6 -c import sys; sys.exit(129)' (exited with code 129)
> Note: On unix systems, an exit code larger than 128 often means a fatal error (e.g. 139=128+11: segmentation fault)

To generate these outputs, a section was added to `tox.ini`:
```
[testenv:exitcode]
basepython = python3.6
description = commands with several exit codes
skip_install = True
commands = python3.6 -c "import sys; sys.exit(5)"
```
and run with `python3 -m tox -e exitcode`.
The modified section is too specific to be included in the PR, isn't it ?


Tests have been added to `test_z_cmdline.py`.
Safely replacing these tests with faster ones (mock) would require someone experienced with this technique.


The documentation might be completed (still have to think about it).
Probably add a section "Understanding InvocationError exit codes"
in [example/general.html](https://tox.readthedocs.io/en/latest/example/general.html)
with links to `pytest` exit codes, to http://www.faqs.org/docs/abs/HTML/exitcodes.html, 
and explaining that no exit code is shown when the process has been aborted (like Qt does).
And for good measure, giving a link to [ignoring-a-command-exit-code](https://tox.readthedocs.io/en/latest/example/basic.html#ignoring-a-command-exit-code).


## Contribution checklist:

- [x] wrote descriptive pull request text
- [x] added/updated test(s)
- [x] updated/extended the documentation
- [x] added relevant [issue keyword](https://help.github.com/articles/closing-issues-using-keywords/)
      in message body
- [x] added news fragment in [changelog folder](https://github.com/tox-dev/tox/tree/master/changelog)
  * fragment name: `<issue number>.<type>.rst` for example (588.bugfix.rst)
  * `<type>` is must be one of `bugfix`, `feature`, `deprecation`,`breaking`, `doc`, `misc`
  * if pr has no issue: consider creating one first or change it to the pr number after creating the pr
  * "sign" fragment with "by @<your username>"
  * please use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files - by @superuser."
  * also see [examples](https://github.com/tox-dev/tox/tree/master/changelog/examples.rst)
- [x] added yourself to `CONTRIBUTORS` (preserving alphabetical order)


codecov[bot]: # [Codecov](https://codecov.io/gh/tox-dev/tox/pull/760?src=pr&el=h1) Report
> Merging [#760](https://codecov.io/gh/tox-dev/tox/pull/760?src=pr&el=desc) into [master](https://codecov.io/gh/tox-dev/tox/commit/a65a1c1c9b0c4cf8f63636266ebacdec5faec459?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/tox-dev/tox/pull/760/graphs/tree.svg?height=150&width=650&token=DYodAwDCZ5&src=pr)](https://codecov.io/gh/tox-dev/tox/pull/760?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #760      +/-   ##
==========================================
- Coverage   94.88%   94.87%   -0.02%     
==========================================
  Files          11       11              
  Lines        2386     2400      +14     
==========================================
+ Hits         2264     2277      +13     
- Misses        122      123       +1
```


| [Impacted Files](https://codecov.io/gh/tox-dev/tox/pull/760?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [tox/\_\_init\_\_.py](https://codecov.io/gh/tox-dev/tox/pull/760/diff?src=pr&el=tree#diff-dG94L19faW5pdF9fLnB5) | `95.45% <100%> (+2.12%)` | :arrow_up: |
| [tox/\_pytestplugin.py](https://codecov.io/gh/tox-dev/tox/pull/760/diff?src=pr&el=tree#diff-dG94L19weXRlc3RwbHVnaW4ucHk=) | `96% <0%> (-0.45%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/tox-dev/tox/pull/760?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/tox-dev/tox/pull/760?src=pr&el=footer). Last update [a65a1c1...309ae93](https://codecov.io/gh/tox-dev/tox/pull/760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 1217c0b5..05ae5566 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -229,7 +229,7 @@ If no contributor strongly objects, the repository can then be
 transferred to the ``tox-dev`` organisation. For details see
 `about repository transfers <https://help.github.com/articles/about-repository-transfers/>`_
 
-Memebrs of the tox organization have write access to all projects.
+Members of the tox organization have write access to all projects.
 We recommend that each plugin has at least three people who have the right to release to PyPI.
 
 Repository owners can rest assured that no ``tox-dev`` administrator will ever make
diff --git a/CONTRIBUTORS b/CONTRIBUTORS
index c2a73e0e..ccf2a0c5 100644
--- a/CONTRIBUTORS
+++ b/CONTRIBUTORS
@@ -16,6 +16,7 @@ Chris Jerdonek
 Chris Rose
 Clark Boylan
 Cyril Roelandt
+Ederag
 Eli Collins
 Eugene Yunak
 Fernando L. Pereira
diff --git a/changelog/290.feature.rst b/changelog/290.feature.rst
new file mode 100644
index 00000000..be93618c
--- /dev/null
+++ b/changelog/290.feature.rst
@@ -0,0 +1,1 @@
+``tox`` displays exit code together with ``InvocationError`` - by @blueyed and @ederag.
diff --git a/doc/example/basic.rst b/doc/example/basic.rst
index 4ebfdcdb..cd3872da 100644
--- a/doc/example/basic.rst
+++ b/doc/example/basic.rst
@@ -289,6 +289,7 @@ Integration with "setup.py test" command
   point it's maybe best to not go for any ``setup.py test`` integration.
 
 
+.. _`ignoring exit code`:
 
 Ignoring a command exit code
 ----------------------------
diff --git a/doc/example/general.rst b/doc/example/general.rst
index a37433d3..684d143c 100644
--- a/doc/example/general.rst
+++ b/doc/example/general.rst
@@ -204,3 +204,49 @@ There is an optimization coded in to not bother re-running the command if
 ``$projectname.egg-info`` is newer than ``setup.py`` or ``setup.cfg``.
 
 .. include:: ../links.rst
+
+
+Understanding ``InvocationError`` exit codes
+--------------------------------------------
+
+When a command (defined by ``commands =`` in ``tox.ini``) fails,
+it has a non-zero exit code,
+and an ``InvocationError`` exception is raised by ``tox``:
+
+.. code-block:: shell
+
+    ERROR: InvocationError for command
+           '<command defined in tox.ini>' (exited with code 1)
+
+If the command starts with ``pytest`` or ``python setup.py test`` for instance,
+then the `pytest exit codes`_ are relevant.
+
+On unix systems, there are some rather `common exit codes`_.
+This is why for exit codes larger than 128, an additional hint is given:
+
+.. code-block:: shell
+
+    ERROR: InvocationError for command
+           '<command defined in tox.ini>' (exited with code 139)
+    Note: On unix systems, an exit code larger than 128 often means a fatal error (e.g. 139=128+11: segmentation fault)
+
+The signal numbers (e.g. 11 for a segmentation fault) can be found in the
+"Standard signals" section of the `signal man page`_.
+Their meaning is described in `POSIX signals`_.
+
+Beware that programs may issue custom exit codes with any value,
+so their documentation should be consulted.
+
+
+Sometimes, no exit code is given at all.
+An example may be found in `pytest-qt issue #170`_,
+where Qt was calling ``abort()`` instead of ``exit()``.
+
+.. seealso:: :ref:`ignoring exit code`.
+
+.. _`pytest exit codes`: https://docs.pytest.org/en/latest/usage.html#possible-exit-codes
+.. _`common exit codes`: http://www.faqs.org/docs/abs/HTML/exitcodes.html
+.. _`abort()``: http://www.unix.org/version2/sample/abort.html
+.. _`pytest-qt issue #170` : https://github.com/pytest-dev/pytest-qt/issues/170
+.. _`signal man page`: http://man7.org/linux/man-pages/man7/signal.7.html
+.. _`POSIX signals`: https://en.wikipedia.org/wiki/Signal_(IPC)#POSIX_signals
diff --git a/tox/__init__.py b/tox/__init__.py
index ba64b356..25b468a1 100644
--- a/tox/__init__.py
+++ b/tox/__init__.py
@@ -34,6 +34,19 @@ class exception:
 
     class InvocationError(Error):
         """ an error while invoking a script. """
+        def __init__(self, command, exitcode=None):
+            super(exception.Error, self).__init__(command, exitcode)
+            self.command = command
+            self.exitcode = exitcode
+
+        def __str__(self):
+            str_ = "%s for command %s" % (self.__class__.__name__, self.command)
+            if self.exitcode:
+                str_ += " (exited with code %d)" % (self.exitcode)
+                if self.exitcode > 128:
+                    str_ += ("\nNote: On unix systems, an exit code larger than 128 "
+                             "often means a fatal error (e.g. 139=128+11: segmentation fault)")
+            return str_
 
     class MissingFile(Error):
         """ an error while invoking a script. """


Test patch:
diff --git a/tests/test_z_cmdline.py b/tests/test_z_cmdline.py
index 3aebee11..c1e09ae7 100644
--- a/tests/test_z_cmdline.py
+++ b/tests/test_z_cmdline.py
@@ -474,7 +474,7 @@ def test_package_install_fails(cmd, initproj):
     })
     result = cmd()
     assert result.ret
-    assert result.outlines[-1].startswith('ERROR:   python: InvocationError: ')
+    assert result.outlines[-1].startswith('ERROR:   python: InvocationError for command ')
 
 
 @pytest.fixture
@@ -904,3 +904,20 @@ def test_tox_quickstart_script():
 def test_tox_cmdline(monkeypatch):
     with pytest.raises(SystemExit):
         tox.cmdline(['caller_script', '--help'])
+
+
+@pytest.mark.parametrize('exitcode', [0, 5, 129])
+def test_exitcode(initproj, cmd, exitcode):
+    tox_ini_content = "[testenv:foo]\ncommands=python -c 'import sys; sys.exit(%d)'" % exitcode
+    initproj("foo", filedefs={'tox.ini': tox_ini_content})
+    result = cmd()
+    if exitcode:
+        needle = "(exited with code %d)" % exitcode
+        assert any(needle in line for line in result.outlines)
+        if exitcode > 128:
+            needle = ("Note: On unix systems, an exit code larger than 128 "
+                      "often means a fatal error (e.g. 139=128+11: segmentation fault)")
+            assert any(needle in line for line in result.outlines)
+    else:
+        needle = "(exited with code"
+        assert all(needle not in line for line in result.outlines)