Repository: HypothesisWorks/hypothesis

Issue:
Distinct failure tracebacks of `ExceptionGroup` are not printed by pytest
I'm getting a notion about 4 distinct errors from the test and nothing else.

---

Tested on python 3.7.5 and 3.10.5. Also on pytest 6.2.4 and 7.1.2

Running the pytest from the terminal. In this case like that
```
python3 -m pytest -vv tests/unit/unifiers/test_frame_scaling.py
```

The output: 
```
platform linux -- Python 3.7.5, pytest-7.1.2, pluggy-1.0.0                                                                                                                                                                                                                                                                    
rootdir: /app/core/tests, configfile: pytest.ini                                                                                                                                                                                                                                                                              
plugins: hypothesis-6.54.0, cov-3.0.0                                                                                                                                                                                                                                                                                         
collected 1 item                                                                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                                                                              
core/tests/unit/unifiers/test_frame_scaling.py F                                                                                                                                                                                                                                                                       [100%] 
                                                                                                                                                                                                                                                                                                                              
================== FAILURES ==================
test_frame_persists_aspect_ratio_after_scale 
                                                                                                                                                                                                                                                                                                                              
    @given(...)                                                                                                                                                                                                                                                                                                               
>   def test_frame_persists_aspect_ratio_after_scale(resolution_in: ImageResolution, resolution_out: ImageResolution):                                                                                                                                                                                                        
E   exceptiongroup.ExceptionGroup: Hypothesis found 4 distinct failures. (4 sub-exceptions)                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                              
core/tests/unit/unifiers/test_frame_scaling.py:7: ExceptionGroup                                                                                                                                                                                                                                                              
generated xml file: /app/reports/report.xml 
=========== short test summary info ============
FAILED core/tests/unit/unifiers/test_frame_scaling.py::test_frame_persists_aspect_ratio_after_scale - exceptiongroup.ExceptionGroup: Hypothesis found 4 distinct failures. (4 sub-exceptions)                                                                                                                                 
=========== 1 failed in 1.28s ===========
```

Desired outcome: unwrapped tracebacks for every sub-exception


Discussion:
This is due to #3308, where tl;dr a new standard feature has been adopted ([`ExceptionGroup`](https://peps.python.org/pep-0654/)), but pytest has to catch up a bit still.

**For now, you can use pytest's `--tb=native` flag to see the tracebacks for each sub-exception.** In the future pytest should show these by default https://github.com/pytest-dev/pytest/issues/9159
Worked like a charm. Thank you!
Thanks for opening, this seems like it'll trip up a few folks so I'll pin it.
I also thought that I'd disabled multi-bug reporting for these cases where it doesn't work 🤔

Needs more tests + a fix on our end too. 

Resolving patch:
diff --git a/hypothesis-python/src/_hypothesis_pytestplugin.py b/hypothesis-python/src/_hypothesis_pytestplugin.py
--- a/hypothesis-python/src/_hypothesis_pytestplugin.py
+++ b/hypothesis-python/src/_hypothesis_pytestplugin.py
@@ -178,13 +178,6 @@ def pytest_configure(config):
                 pass
             core.global_force_seed = seed
 
-        core.pytest_shows_exceptiongroups = (
-            sys.version_info[:2] >= (3, 11)
-            ## See https://github.com/pytest-dev/pytest/issues/9159
-            # or pytest_version >= (7, 2)  # TODO: fill in correct version here
-            or config.getoption("tbstyle", "auto") == "native"
-        )
-
     @pytest.hookimpl(hookwrapper=True)
     def pytest_runtest_call(item):
         __tracebackhide__ = True
@@ -195,6 +188,11 @@ def pytest_runtest_call(item):
         from hypothesis import core
         from hypothesis.internal.detection import is_hypothesis_test
 
+        ## See https://github.com/pytest-dev/pytest/issues/9159
+        # TODO: add `pytest_version >= (7, 2) or` once the issue above is fixed.
+        core.pytest_shows_exceptiongroups = (
+            item.config.getoption("tbstyle", "auto") == "native"
+        )
         core.running_under_pytest = True
 
         if not is_hypothesis_test(item.obj):


Test patch:
diff --git a/hypothesis-python/tests/pytest/test_reporting.py b/hypothesis-python/tests/pytest/test_reporting.py
--- a/hypothesis-python/tests/pytest/test_reporting.py
+++ b/hypothesis-python/tests/pytest/test_reporting.py
@@ -8,6 +8,8 @@
 # v. 2.0. If a copy of the MPL was not distributed with this file, You can
 # obtain one at https://mozilla.org/MPL/2.0/.
 
+import pytest
+
 pytest_plugins = "pytester"
 
 
@@ -33,3 +35,24 @@ def test_runs_reporting_hook(testdir):
     assert "Captured stdout call" not in out
     assert "Falsifying example" in out
     assert result.ret != 0
+
+
+TEST_EXCEPTIONGROUP = """
+from hypothesis import given, strategies as st
+
+@given(x=st.booleans())
+def test_fuzz_sorted(x):
+    raise ValueError if x else TypeError
+"""
+
+
+@pytest.mark.parametrize("tb", ["auto", "long", "short", "native"])
+def test_no_missing_reports(testdir, tb):
+    script = testdir.makepyfile(TEST_EXCEPTIONGROUP)
+    result = testdir.runpytest(script, f"--tb={tb}")
+    out = "\n".join(result.stdout.lines)
+    # If the False case is missing, that means we're not printing exception info.
+    # See https://github.com/HypothesisWorks/hypothesis/issues/3430  With --tb=native,
+    # we should show the full ExceptionGroup with *both* errors.
+    assert "x=False" in out
+    assert "x=True" in out or tb != "native"