Repository: pex-tool/pex

Issue:
pex -c incorrectly alters exit codes
Running pytest from a pex always exits 0 if there are test failures, but not if there are command line argument processing errors.

For example, given this `test_fail.py`:

``` python
assert False
```

A pytest pex will exit zero:

``` console
$ pex --version && pex pytest -c py.test -o pytest.pex && ./pytest.pex test_fail.py ; echo $?
pex 1.0.1
=========================================================================== test session starts ============================================================================
platform darwin -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2
rootdir: /Users/rouge8/forks/pex, inifile:
collected 0 items / 1 errors

================================================================================== ERRORS ==================================================================================
______________________________________________________________________ ERROR collecting test_fail.py _______________________________________________________________________
test_fail.py:1: in <module>
    assert False
E   assert False
========================================================================= 1 error in 0.00 seconds ==========================================================================
0
```

But running `py.test` in a virtualenv or similar will exit 1.

On the other hand, giving a bad option to `py.test` will exit non-zero, as expected:

``` console
$ pex --version && pex pytest -c py.test -o pytest.pex && ./pytest.pex --bad-option test_fail.py ; echo $?
pex 1.0.1
usage: pytest.pex [options] [file_or_dir] [file_or_dir] [...]
pytest.pex: error: unrecognized arguments: --bad-option
2
```



Discussion:
I'm able to repro.  This is probably a bug with "-c".  If you do "pex pytest -m pytest -o pytest.pex" then it works:

``` bash
mba=2015.07.29.pex137=; pex pytest -m pytest -o pytest.pex
mba=2015.07.29.pex137=; ./pytest.pex test_fail.py
====================================================== test session starts =======================================================
platform darwin -- Python 2.7.9 -- py-1.4.30 -- pytest-2.7.2
rootdir: /Users/wickman/work/2015.07.29.pex137, inifile: 
collected 1 items 

test_fail.py F

============================================================ FAILURES ============================================================
___________________________________________________________ test_fail ____________________________________________________________

    def test_fail():
>     assert False
E     assert False

test_fail.py:2: AssertionError
==================================================== 1 failed in 0.01 seconds ====================================================
mba=2015.07.29.pex137=; echo $?
1
```

Yup, using `-m pytest` instead of `-c py.test` works. Thanks!

I looked into this a bit this morning:
- `pex -c py.test` finds the console script `pytest:main`, which [returns an integer exit code](https://github.com/pytest-dev/pytest/blob/2.9.2/_pytest/config.py#L29)
- `pex -m pytest` hits [`pytest.py`](https://github.com/pytest-dev/pytest/blob/2.9.2/pytest.py#L14) which raises `SystemExit` with that integer exit code

According to the [setuptools docs](https://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-script-creation):

> The functions you specify are called with no arguments, and their return value is passed to `sys.exit()`, so you can return an errorlevel or message to print to stderr

pex currently just calls the entrypoint function but does nothing with the return value. I would guess returning rather than exiting from the entrypoint is uncommon (`pex.bin.pex:main` calls `sys.exit()` itself), which is why I only saw this with py.test.

I'm not sure if I'll have time to work on a fix immediately, but hopefully this writeup is helpful for anyone who does.


Resolving patch:
diff --git a/pex/pex.py b/pex/pex.py
--- a/pex/pex.py
+++ b/pex/pex.py
@@ -391,7 +391,7 @@ def execute_script(self, script_name):
 
     entry_point = get_entry_point_from_console_script(script_name, dists)
     if entry_point:
-      return self.execute_entry(entry_point)
+      sys.exit(self.execute_entry(entry_point))
 
     dist, script_path, script_content = get_script_from_distributions(script_name, dists)
     if not dist:
@@ -426,7 +426,7 @@ def execute_content(cls, name, content, argv0=None):
   @classmethod
   def execute_entry(cls, entry_point):
     runner = cls.execute_pkg_resources if ':' in entry_point else cls.execute_module
-    runner(entry_point)
+    return runner(entry_point)
 
   @staticmethod
   def execute_module(module_name):
@@ -444,7 +444,7 @@ def execute_pkg_resources(spec):
     else:
       # setuptools < 11.3
       runner = entry.load(require=False)
-    runner()
+    return runner()
 
   def cmdline(self, args=()):
     """The commandline to run this environment.


Test patch:
diff --git a/tests/test_integration.py b/tests/test_integration.py
--- a/tests/test_integration.py
+++ b/tests/test_integration.py
@@ -3,13 +3,15 @@
 
 import os
 import sys
+from textwrap import dedent
 
 import pytest
 from twitter.common.contextutil import environment_as, temporary_dir
 
 from pex.compatibility import WINDOWS
-from pex.testing import run_pex_command, run_simple_pex_test
-from pex.util import named_temporary_file
+from pex.installer import EggInstaller
+from pex.testing import run_pex_command, run_simple_pex_test, temporary_content
+from pex.util import DistributionHelper, named_temporary_file
 
 
 def test_pex_execute():
@@ -88,3 +90,31 @@ def test_pex_python_symlink():
       body = "print('Hello')"
       _, rc = run_simple_pex_test(body, coverage=True)
       assert rc == 0
+
+
+def test_entry_point_exit_code():
+  setup_py = dedent("""
+    from setuptools import setup
+
+    setup(
+      name='my_app',
+      version='0.0.0',
+      zip_safe=True,
+      packages=[''],
+      entry_points={'console_scripts': ['my_app = my_app:do_something']},
+    )
+  """)
+
+  error_msg = 'setuptools expects this to exit non-zero'
+
+  my_app = dedent("""
+    def do_something():
+      return '%s'
+  """ % error_msg)
+
+  with temporary_content({'setup.py': setup_py, 'my_app.py': my_app}) as project_dir:
+    installer = EggInstaller(project_dir)
+    dist = DistributionHelper.distribution_from_path(installer.bdist())
+    so, rc = run_simple_pex_test('', env={'PEX_SCRIPT': 'my_app'}, dists=[dist])
+    assert so.decode('utf-8').strip() == error_msg
+    assert rc == 1