Repository: pytest-dev/pytest
Pull request: Prevent pytest from printing ConftestImportFailure traceback
Closes: #6956 

@asottile I was not able to satisfy mypy so I added `# type: ignore` to the two lines. Let me know what you think :)

- [X] Include new tests or update existing tests when applicable.
- [X] Allow maintainers to push and squash when merging my commits. Please uncheck this if you prefer to squash the commits yourself.
- [X] Create a new changelog file in the `changelog` folder, with a name like `<ISSUE NUMBER>.<TYPE>.rst`

Problem statement:
ConftestImportFailure: internal traceback when non-root conftest raises error
an internal traceback is exposed when a nested conftest raises an error (doesn't happen for top level conftest failure)

- [x] a detailed description of the bug or suggestion
- [x] output of `pip list` from the virtual environment you are using
- [x] pytest and operating system versions
- [x] minimal example if possible

```console
$ pip freeze --all
attrs==19.3.0
importlib-metadata==1.5.0
more-itertools==8.2.0
packaging==20.3
pip==20.0.2
pluggy==0.13.1
py==1.8.1
pyparsing==2.4.6
pytest==5.4.1
setuptools==45.2.0
six==1.14.0
wcwidth==0.1.8
wheel==0.34.2
zipp==3.1.0
$ tree tests
tests
└── wat
    └── conftest.py

1 directory, 1 file
$ cat tests/wat/conftest.py 
import wat
```

```console
$ pytest tests
============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /tmp/y
collected 0 items / 1 error                                                    

==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
venv/lib/python3.6/site-packages/_pytest/config/__init__.py:495: in _importconftest
    return self._conftestpath2mod[key]
E   KeyError: PosixPath('/tmp/y/tests/wat/conftest.py')

During handling of the above exception, another exception occurred:
venv/lib/python3.6/site-packages/_pytest/config/__init__.py:501: in _importconftest
    mod = conftestpath.pyimport()
venv/lib/python3.6/site-packages/py/_path/local.py:701: in pyimport
    __import__(modname)
venv/lib/python3.6/site-packages/_pytest/assertion/rewrite.py:152: in exec_module
    exec(co, module.__dict__)
tests/wat/conftest.py:1: in <module>
    import wat
E   ModuleNotFoundError: No module named 'wat'

During handling of the above exception, another exception occurred:
venv/lib/python3.6/site-packages/py/_path/common.py:383: in visit
    for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
venv/lib/python3.6/site-packages/py/_path/common.py:424: in gen
    dirs = self.optsort([p for p in entries
venv/lib/python3.6/site-packages/py/_path/common.py:425: in <listcomp>
    if p.check(dir=1) and (rec is None or rec(p))])
venv/lib/python3.6/site-packages/_pytest/nodes.py:506: in _recurse
    ihook = self._gethookproxy(dirpath)
venv/lib/python3.6/site-packages/_pytest/nodes.py:487: in _gethookproxy
    my_conftestmodules = pm._getconftestmodules(fspath)
venv/lib/python3.6/site-packages/_pytest/config/__init__.py:473: in _getconftestmodules
    mod = self._importconftest(conftestpath)
venv/lib/python3.6/site-packages/_pytest/config/__init__.py:509: in _importconftest
    raise ConftestImportFailure(conftestpath, sys.exc_info())
E   _pytest.config.ConftestImportFailure: (local('/tmp/y/tests/wat/conftest.py'), (<class 'ModuleNotFoundError'>, ModuleNotFoundError("No module named 'wat'",), <traceback object at 0x7ff80ffddb08>))
=========================== short test summary info ============================
ERROR  - _pytest.config.ConftestImportFailure: (local('/tmp/y/tests/wat/conft...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.24s ===============================
```

note  that this does not occur with a toplevel conftest (it is caught and reported properly)

```console
$ pytest tests/wat
ImportError while loading conftest '/tmp/y/tests/wat/conftest.py'.
tests/wat/conftest.py:1: in <module>
    import wat
E   ModuleNotFoundError: No module named 'wat'
```

Hints from issue discussion:
I might be misunderstanding, but it looks like you're trying to import the package that the conftest is inside of into the conftest, which would be circular.
it has this behaviour for any exception, the concidence in import name was perhaps not the best choice when I wrote up this bug report (I was being too hasty!)
I would say it is related to #7150 and both can be closed due to merging #7244.
seems to still not quite be right, though improved a bit:

```
$ pytest tests
============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-5.4.1.dev356+g981b09694, py-1.8.1, pluggy-0.13.1
rootdir: /tmp
collected 0 items / 1 error                                                    

==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
venv/lib/python3.8/site-packages/_pytest/config/__init__.py:515: in _importconftest
    mod = conftestpath.pyimport()
venv/lib/python3.8/site-packages/py/_path/local.py:701: in pyimport
    __import__(modname)
venv/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:152: in exec_module
    exec(co, module.__dict__)
tests/x/conftest.py:1: in <module>
    import wat
E   ModuleNotFoundError: No module named 'wat'

The above exception was the direct cause of the following exception:
venv/lib/python3.8/site-packages/py/_path/common.py:383: in visit
    for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
venv/lib/python3.8/site-packages/py/_path/common.py:424: in gen
    dirs = self.optsort([p for p in entries
venv/lib/python3.8/site-packages/py/_path/common.py:425: in <listcomp>
    if p.check(dir=1) and (rec is None or rec(p))])
venv/lib/python3.8/site-packages/_pytest/nodes.py:525: in _recurse
    ihook = self._gethookproxy(dirpath)
venv/lib/python3.8/site-packages/_pytest/nodes.py:506: in _gethookproxy
    my_conftestmodules = pm._getconftestmodules(fspath)
venv/lib/python3.8/site-packages/_pytest/config/__init__.py:485: in _getconftestmodules
    mod = self._importconftest(conftestpath)
venv/lib/python3.8/site-packages/_pytest/config/__init__.py:517: in _importconftest
    raise ConftestImportFailure(conftestpath, sys.exc_info()) from e
E   _pytest.config.ConftestImportFailure: ModuleNotFoundError: No module named 'wat' (from /tmp/tests/x/conftest.py)
=========================== short test summary info ============================
ERROR  - _pytest.config.ConftestImportFailure: ModuleNotFoundError: No module...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.25s ===============================
```
Ah right. I guess it's because we now return the internal traceback if called with `--pdb` and not every time. I can investigate it further if you wish :)
I identified the `_repr_failure_py()` function to be the one to be changed. (https://github.com/pytest-dev/pytest/blob/master/src/_pytest/nodes.py#L333)
Added a test for it and adjust an existing test, which expects `ConftestImportFailure` to be present in the output. PR is coming.

Commit message: Move ConftestImportFailure check to correct position and add typing

Patch to review:
diff --git a/changelog/6956.bugfix.rst b/changelog/6956.bugfix.rst
new file mode 100644
index 00000000000..a88ef94b6d5
--- /dev/null
+++ b/changelog/6956.bugfix.rst
@@ -0,0 +1 @@
+Prevent pytest from printing ConftestImportFailure traceback to stdout.
diff --git a/src/_pytest/nodes.py b/src/_pytest/nodes.py
index 2f5f9bdb801..5aae211cd7e 100644
--- a/src/_pytest/nodes.py
+++ b/src/_pytest/nodes.py
@@ -19,6 +19,7 @@
 from _pytest.compat import cached_property
 from _pytest.compat import TYPE_CHECKING
 from _pytest.config import Config
+from _pytest.config import ConftestImportFailure
 from _pytest.config import PytestPluginManager
 from _pytest.deprecated import NODE_USE_FROM_PARENT
 from _pytest.fixtures import FixtureDef
@@ -331,13 +332,19 @@ def _prunetraceback(self, excinfo):
         pass
 
     def _repr_failure_py(
-        self, excinfo: ExceptionInfo[Union[Failed, FixtureLookupError]], style=None
+        self,
+        excinfo: ExceptionInfo[
+            Union[Failed, FixtureLookupError, ConftestImportFailure]
+        ],
+        style=None,
     ) -> Union[str, ReprExceptionInfo, ExceptionChainRepr, FixtureLookupErrorRepr]:
         if isinstance(excinfo.value, fail.Exception):
             if not excinfo.value.pytrace:
                 return str(excinfo.value)
         if isinstance(excinfo.value, FixtureLookupError):
             return excinfo.value.formatrepr()
+        if isinstance(excinfo.value, ConftestImportFailure):
+            excinfo = ExceptionInfo(excinfo.value.excinfo)  # type: ignore
         if self.config.getoption("fulltrace", False):
             style = "long"
         else:
diff --git a/testing/python/collect.py b/testing/python/collect.py
index 2807cacc90a..cbc798ad8e0 100644
--- a/testing/python/collect.py
+++ b/testing/python/collect.py
@@ -1251,7 +1251,7 @@ def test_syntax_error_with_non_ascii_chars(testdir):
     result.stdout.fnmatch_lines(["*ERROR collecting*", "*SyntaxError*", "*1 error in*"])
 
 
-def test_collecterror_with_fulltrace(testdir):
+def test_collect_error_with_fulltrace(testdir):
     testdir.makepyfile("assert 0")
     result = testdir.runpytest("--fulltrace")
     result.stdout.fnmatch_lines(
@@ -1259,15 +1259,12 @@ def test_collecterror_with_fulltrace(testdir):
             "collected 0 items / 1 error",
             "",
             "*= ERRORS =*",
-            "*_ ERROR collecting test_collecterror_with_fulltrace.py _*",
-            "",
-            "*/_pytest/python.py:*: ",
-            "_ _ _ _ _ _ _ _ *",
+            "*_ ERROR collecting test_collect_error_with_fulltrace.py _*",
             "",
             ">   assert 0",
             "E   assert 0",
             "",
-            "test_collecterror_with_fulltrace.py:1: AssertionError",
+            "test_collect_error_with_fulltrace.py:1: AssertionError",
             "*! Interrupted: 1 error during collection !*",
         ]
     )
diff --git a/testing/test_reports.py b/testing/test_reports.py
index 13f5932156b..64d86e9534b 100644
--- a/testing/test_reports.py
+++ b/testing/test_reports.py
@@ -396,6 +396,18 @@ def check_longrepr(longrepr):
         # for same reasons as previous test, ensure we don't blow up here
         loaded_report.longrepr.toterminal(tw_mock)
 
+    def test_report_prevent_ConftestImportFailure_hiding_exception(self, testdir):
+        sub_dir = testdir.tmpdir.join("ns").ensure_dir()
+        sub_dir.join("conftest").new(ext=".py").write("import unknown")
+
+        result = testdir.runpytest_subprocess(".")
+        result.stdout.fnmatch_lines(
+            ["E   ModuleNotFoundError: No module named 'unknown'"]
+        )
+        result.stdout.no_fnmatch_line(
+            "ERROR  - _pytest.config.ConftestImportFailure: ModuleNotFoundError:*"
+        )
+
 
 class TestHooks:
     """Test that the hooks are working correctly for plugins"""


Code review comments:
diff_hunk: @@ -331,13 +332,19 @@ def _prunetraceback(self, excinfo):
         pass
 
     def _repr_failure_py(
-        self, excinfo: ExceptionInfo[Union[Failed, FixtureLookupError]], style=None
+        self,
+        excinfo: ExceptionInfo[
+            Union[Failed, FixtureLookupError, ConftestImportFailure]
+        ],
+        style=None,
     ) -> Union[str, ReprExceptionInfo, ExceptionChainRepr, FixtureLookupErrorRepr]:
         if isinstance(excinfo.value, fail.Exception):
             if not excinfo.value.pytrace:
                 return str(excinfo.value)
         if isinstance(excinfo.value, FixtureLookupError):
             return excinfo.value.formatrepr()
+        if isinstance(excinfo.value, ConftestImportFailure):
+            excinfo = ExceptionInfo(excinfo.value.excinfo)  # type: ignore
path: src/_pytest/nodes.py
text: @user1:
Please add the error code to the `type: ignore`, e.g. `type: ignore[arg-type]`.

What is the type error here?

@author:
Solved by using `BaseException` for `excinfo`

diff_hunk: @@ -331,13 +332,19 @@ def _prunetraceback(self, excinfo):
         pass
 
     def _repr_failure_py(
-        self, excinfo: ExceptionInfo[Union[Failed, FixtureLookupError]], style=None
+        self,
+        excinfo: ExceptionInfo[
path: src/_pytest/nodes.py
text: @user1:
These types (existing ones) are wrong BTW - the last commit of #7142 fixes that. 

I recommend just changing to `ExceptionInfo[BaseException]` here.

@author:
done

diff_hunk: @@ -331,13 +332,19 @@ def _prunetraceback(self, excinfo):
         pass
 
     def _repr_failure_py(
-        self, excinfo: ExceptionInfo[Union[Failed, FixtureLookupError]], style=None
+        self,
+        excinfo: ExceptionInfo[
+            Union[Failed, FixtureLookupError, ConftestImportFailure]
+        ],
+        style=None,
     ) -> Union[str, ReprExceptionInfo, ExceptionChainRepr, FixtureLookupErrorRepr]:
         if isinstance(excinfo.value, fail.Exception):
             if not excinfo.value.pytrace:
                 return str(excinfo.value)
         if isinstance(excinfo.value, FixtureLookupError):
             return excinfo.value.formatrepr()
+        if isinstance(excinfo.value, ConftestImportFailure):
path: src/_pytest/nodes.py
text: @user1:
This should probably come first, in case the wrapped exception is a `fail.Exception` or a `FixtureLookupError` (if that's possible -- but even if not, probably a good idea to move).

@author:
done

diff_hunk: @@ -396,6 +396,18 @@ def check_longrepr(longrepr):
         # for same reasons as previous test, ensure we don't blow up here
         loaded_report.longrepr.toterminal(tw_mock)
 
+    def test_report_prevent_ConftestImportFailure_hiding_exception(self, testdir):
+        sub_dir = testdir.tmpdir.join("ns").ensure_dir()
+        sub_dir.join("conftest").new(ext=".py").write("import unknown")
+
+        result = testdir.runpytest_subprocess(".")
+        result.stdout.fnmatch_lines(
+            ["E   ModuleNotFoundError: No module named 'unknown'"]
+        )
+        result.stdout.no_fnmatch_line(
path: testing/test_reports.py
text: @user1:
To make this more robust, I'd match just `*ConftestImportFailure*`.

@author:
I needed to include `ERROR  - *ConftestImportFailure*` as otherwise tests including `ConftestImportFailure` in their name would fail.