Repository: pytest-dev/pytest
Pull request: Disable output capturing in doctest on macOS
... to avoid losing reference to stdout. Fixes #985.

This patch implements the [change proposed](https://github.com/pytest-dev/pytest/issues/985#issuecomment-357541238), but encapsulates the workaround and documents it with a docstring.

- [x] Create a new changelog file in the `changelog` folder, with a name like `<ISSUE NUMBER>.<TYPE>.rst`. See [changelog/README.rst](/changelog/README.rst) for details.
- [x] Target the `master` branch for bug fixes, documentation updates and trivial changes.
- [x] Target the `features` branch for new features and removals/deprecations.
- [x] Include documentation when adding new features.
- [x] Include new tests or update existing tests when applicable.


Problem statement:
pdb prompt unusable if doctests ran first (macOS and Python 3)
Consider this simple test:

```
">>> 3\n3"

def test_stuff():
    import pdb; pdb.set_trace()
```

Create a virtualenv on Python 3.4.3, install pytest 2.7.2 and py 1.4.30 and run py.test:

```
(pytest) $ py.test
====================================== test session starts =======================================
platform darwin -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2
rootdir: /Users/jaraco/foo, inifile: 
collected 1 items 

test_stuff.py 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
--Return--
> /Users/jaraco/foo/test_stuff.py(4)test_stuff()->None
-> import pdb; pdb.set_trace()
(Pdb) q
F

============================================ FAILURES ============================================
___________________________________________ test_stuff ___________________________________________

    def test_stuff():
>       import pdb; pdb.set_trace()

test_stuff.py:4: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/bdb.py:52: in trace_dispatch
    return self.dispatch_return(frame, arg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pdb.Pdb object at 0x102a3f7f0>, frame = <frame object at 0x102a54848>, arg = None

    def dispatch_return(self, frame, arg):
        if self.stop_here(frame) or frame == self.returnframe:
            # Ignore return events in generator except when stepping.
            if self.stopframe and frame.f_code.co_flags & CO_GENERATOR:
                return self.trace_dispatch
            try:
                self.frame_returning = frame
                self.user_return(frame, arg)
            finally:
                self.frame_returning = None
>           if self.quitting: raise BdbQuit
E           bdb.BdbQuit

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/bdb.py:96: BdbQuit
==================================== 1 failed in 1.95 seconds ====================================
```

Everything's fine - the pdb prompt is displayed and works as expected. However, invoke pytest with `--doctest-modules`, and the story is different.

```
(pytest) $ py.test --doctest-modules
====================================== test session starts =======================================
platform darwin -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2
rootdir: /Users/jaraco/foo, inifile: 
collected 2 items 

test_stuff.py .
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
--Return--
> /Users/jaraco/foo/test_stuff.py(4)test_stuff()->None
-> import pdb; pdb.set_trace()
^D
F

============================================ FAILURES ============================================
___________________________________________ test_stuff ___________________________________________

    def test_stuff():
>       import pdb; pdb.set_trace()

test_stuff.py:4: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/bdb.py:52: in trace_dispatch
    return self.dispatch_return(frame, arg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pdb.Pdb object at 0x102a51be0>, frame = <frame object at 0x102af8248>, arg = None

    def dispatch_return(self, frame, arg):
        if self.stop_here(frame) or frame == self.returnframe:
            # Ignore return events in generator except when stepping.
            if self.stopframe and frame.f_code.co_flags & CO_GENERATOR:
                return self.trace_dispatch
            try:
                self.frame_returning = frame
                self.user_return(frame, arg)
            finally:
                self.frame_returning = None
>           if self.quitting: raise BdbQuit
E           bdb.BdbQuit

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/bdb.py:96: BdbQuit
=============================== 1 failed, 1 passed in 8.34 seconds ===============================
```

Although the code stops in the right place, the pdb prompt does not appear. Keystrokes appear to be ignored. Only sending an EOF (`^d`) allows the test to fail and pytest to exit.

This issue seems to have emerged recently, though I can't imagine what could be causing it. I've tested in a virtualenv, tried upgrading and downgrading pytest, and tried older setuptools versions.

What's the next step to troubleshoot?

Hints from issue discussion:
for testing purposes i'd suggest to write s small script using python pexpect that then can be used to test against different versions an potentially for bisecting the responsible change

do you have a rough set of known good/bad releases

I rolled back to pytest 2.6.x and 2.7.0, but the issue remained. I'll try some older versions. I bet there's something else in my environment causing the issue but I can't imagine what. I'll try even older versions. Good idea.

I can reproduce the issue back to pytest 2.4 or py 1.4.24 on Python 2.7 or Python 3, but only on OS X. I've been unable to reproduce the issue on Ubuntu.

FWIW I can't reproduce the problem on Windows either. :confused: 

Given that I adopted Mac in May 2015 and that the issue appears to only affect Mac, even on Pytest 2.4 which was released in 2013, I believe the issue is not a recent regression, as I suspected, but simply a broken behavior on macOS (formerly OS X).

i see, as such most of us are unable to test/experiment :/

Would someone on the pytest team point me to the code where doctests are invoked and where pdb is invoked? From there, I'll try to devise a minimal test that replicates the failure.

@jaraco could you please try running pytest's own test suite first? The current tests might detect the problem already.

The tests all passed on Python 2.6, but on 3.5, there are [many failures](https://gist.github.com/jaraco/6be7e8fb4d9f8e21b0883e6f56db2985).

I don't see anything in those results that appears to me to relate to doctests or pdb prompts. I do see a lot of skips in the pdb tests, but I can't see which skip message(s) are relevant. Aha, so running just the tests in `test_pdb`, I see the reason so many are skipped: "could not import 'pexpect'".

>  I see the reason so many are skipped: "could not import 'pexpect'".

Those are the ones we want to run, actually. 😁 Are you running using `tox`?

Commit message: Restrict fix to macOS only. Ref #3215.

Patch to review:
diff --git a/_pytest/doctest.py b/_pytest/doctest.py
index bba90e551c5..6a2918b6f5a 100644
--- a/_pytest/doctest.py
+++ b/_pytest/doctest.py
@@ -2,6 +2,8 @@
 from __future__ import absolute_import, division, print_function
 
 import traceback
+import sys
+import platform
 
 import pytest
 from _pytest._code.code import ExceptionInfo, ReprFileLocation, TerminalRepr
@@ -103,8 +105,21 @@ def setup(self):
 
     def runtest(self):
         _check_all_skipped(self.dtest)
+        self._disable_output_capturing()
         self.runner.run(self.dtest)
 
+    def _disable_output_capturing(self):
+        """
+        Disable output capturing. Otherwise, stdout is lost to doctest (#985)
+        """
+        if platform.system() != 'Darwin':
+            return
+        capman = self.config.pluginmanager.getplugin("capturemanager")
+        if capman:
+            out, err = capman.suspend_global_capture(in_=True)
+            sys.stdout.write(out)
+        sys.stdout.write(err)
+
     def repr_failure(self, excinfo):
         import doctest
         if excinfo.errisinstance((doctest.DocTestFailure,
diff --git a/changelog/985.bugfix b/changelog/985.bugfix
new file mode 100644
index 00000000000..0024b9aa2f7
--- /dev/null
+++ b/changelog/985.bugfix
@@ -0,0 +1 @@
+Fixed output capture handling in doctests on macOS.
diff --git a/testing/test_pdb.py b/testing/test_pdb.py
index b286d57a854..8618473bb15 100644
--- a/testing/test_pdb.py
+++ b/testing/test_pdb.py
@@ -267,10 +267,6 @@ def test_2():
         child.read()
         self.flush(child)
 
-    # For some reason the interaction between doctest's and pytest's output
-    # capturing mechanisms are messing up the stdout on mac. (See #985).
-    # Should be solvable, but skipping until we have a chance to investigate.
-    @pytest.mark.xfail("sys.platform == 'darwin'", reason='See issue #985', run=False)
     def test_pdb_interaction_doctest(self, testdir):
         p1 = testdir.makepyfile("""
             import pytest


Code review comments:
diff_hunk: @@ -103,8 +105,21 @@ def setup(self):
 
     def runtest(self):
         _check_all_skipped(self.dtest)
+        self._disable_output_capturing()
         self.runner.run(self.dtest)
 
+    def _disable_output_capturing(self):
path: _pytest/doctest.py
text: @user1:
please include the platform name in the method name since its a platform hack
i propose `_disable_output_capturing_for_darwin`