Repository: HypothesisWorks/hypothesis

Issue:
Change in behaviour with multiple examples and pytest.skip
There is a change in behaviour between hypothesis 6.54.1 and 6.54.2.

This change is for the better, I think, but just wanted to confirm it was expected, since it doesn't obviously line up with the only changelog entry:
> This patch fixes our workaround for [a pytest bug where the inner exceptions in an ExceptionGroup are not displayed](https://github.com/pytest-dev/pytest/issues/9159) ([issue #3430](https://github.com/HypothesisWorks/hypothesis/issues/3430)).

Change wasn't obvious to me at a skim from https://github.com/HypothesisWorks/hypothesis/compare/hypothesis-python-6.54.1..hypothesis-python-6.54.2 either, but I don't know too much about pytest internals

Consider a hypothesis test that has multiple examples which runs `pytest.skip` (I'm aware that this is perhaps a little unidiomatic):
```
import hypothesis
import hypothesis.strategies as st

import pytest

@hypothesis.given(value=st.text())
@hypothesis.example("hello")
@hypothesis.example("goodbye")
def test_skip(value):
    pytest.skip("TODO: fix this test")
```

With 6.54.1, you get:
```
 exceptiongroup.BaseExceptionGroup: Hypothesis found 2 distinct failures in explicit examples. (2 sub-exceptions)
```
With 6.54.2, you get a skipped test.

In both versions, if you have only a single example, you get a skipped test.

If appropriate, I'd also be happy to add a regression test to the test suite.


Discussion:
...ah, I see what's happening here now, but it's certainly not obvious.

- https://github.com/HypothesisWorks/hypothesis/pull/3308 switched over to using `ExceptionGroup`, and *also* adds a forced `report_multiple_bugs=False` setting if you're using Pytest but not `--tb=native` (because in that case the inner exceptions weren't rendered)
- v6.54.2 *fixes* the detection, so that we actually do stop after discovering the first exception.  When Hypothesis re-raises a `Skipped` exception, Pytest skips the test, but there's no special handling for exception groups.
  - In https://github.com/pytest-dev/pytest/issues/9680#issuecomment-1143277400, I decided on behalf of Pytest that downstream libraries should be responsible for handling marker exceptions
- For *generated* examples, we explicitly stop generating at the first 'skip' exception and immediately reraise.  For consistency, I think that we should do the same for explicit examples, which means reverting to the behaviour of 6.54.1.  Here's a diff:

```diff
@@ -437,7 +437,11 @@ def execute_explicit_examples(state, wrapped_test, arguments, kwargs, original_s
                     err = new

                 yield (fragments_reported, err)
-                if state.settings.report_multiple_bugs and pytest_shows_exceptiongroups:
+                if (
+                    state.settings.report_multiple_bugs
+                    and pytest_shows_exceptiongroups
+                    and not isinstance(err, skip_exceptions_to_reraise())
+                ):
                     continue
                 break
             finally:
@@ -1192,6 +1196,13 @@ def given(
                 # If we're not going to report multiple bugs, we would have
                 # stopped running explicit examples at the first failure.
                 assert len(errors) == 1 or state.settings.report_multiple_bugs
+
+                # If an explicit example raised a 'skip' exception, ensure it's never
+                # wrapped up in an exception group.  Because we break out of the loop
+                # immediately on finding a skip, if present it's always the last error.
+                if isinstance(errors[-1], skip_exceptions_to_reraise()):
+                    del errors[:-1]
+
                 _raise_to_user(errors, state.settings, [], " in explicit examples")

             # If there were any explicit examples, they all ran successfully.
```

If you'd like to add a regression test and make a PR, go ahead!  Otherwise I'll probably get to it next week, after PyBay 🙂 

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -437,7 +437,11 @@ def execute_explicit_examples(state, wrapped_test, arguments, kwargs, original_s
                     err = new
 
                 yield (fragments_reported, err)
-                if state.settings.report_multiple_bugs and pytest_shows_exceptiongroups:
+                if (
+                    state.settings.report_multiple_bugs
+                    and pytest_shows_exceptiongroups
+                    and not isinstance(err, skip_exceptions_to_reraise())
+                ):
                     continue
                 break
             finally:
@@ -1192,6 +1196,14 @@ def wrapped_test(*arguments, **kwargs):
                 # If we're not going to report multiple bugs, we would have
                 # stopped running explicit examples at the first failure.
                 assert len(errors) == 1 or state.settings.report_multiple_bugs
+
+                # If an explicit example raised a 'skip' exception, ensure it's never
+                # wrapped up in an exception group.  Because we break out of the loop
+                # immediately on finding a skip, if present it's always the last error.
+                if isinstance(errors[-1][1], skip_exceptions_to_reraise()):
+                    # Covered by `test_issue_3453_regression`, just in a subprocess.
+                    del errors[:-1]  # pragma: no cover
+
                 _raise_to_user(errors, state.settings, [], " in explicit examples")
 
             # If there were any explicit examples, they all ran successfully.
diff --git a/hypothesis-python/src/hypothesis/extra/codemods.py b/hypothesis-python/src/hypothesis/extra/codemods.py
--- a/hypothesis-python/src/hypothesis/extra/codemods.py
+++ b/hypothesis-python/src/hypothesis/extra/codemods.py
@@ -197,7 +197,10 @@ def leave_Call(self, original_node, updated_node):
         # Get the actual function object so that we can inspect the signature.
         # This does e.g. incur a dependency on Numpy to fix Numpy-dependent code,
         # but having a single source of truth about the signatures is worth it.
-        params = signature(get_fn(*qualnames)).parameters.values()
+        try:
+            params = signature(get_fn(*qualnames)).parameters.values()
+        except ModuleNotFoundError:
+            return updated_node
 
         # st.floats() has a new allow_subnormal kwonly argument not at the end,
         # so we do a bit more of a dance here.
diff --git a/tooling/src/hypothesistooling/__main__.py b/tooling/src/hypothesistooling/__main__.py
--- a/tooling/src/hypothesistooling/__main__.py
+++ b/tooling/src/hypothesistooling/__main__.py
@@ -380,9 +380,9 @@ def run_tox(task, version, *args):
 # When a version is added or removed, manually update the env lists in tox.ini and
 # workflows/main.yml, and the `Programming Language ::` specifiers in setup.py
 PYTHONS = {
-    "3.7": "3.7.13",
-    "3.8": "3.8.13",
-    "3.9": "3.9.13",
+    "3.7": "3.7.14",
+    "3.8": "3.8.14",
+    "3.9": "3.9.14",
     "3.10": "3.10.7",
     "3.11": "3.11-dev",
     "3.12": "3.12-dev",


Test patch:
diff --git a/hypothesis-python/tests/pytest/test_skipping.py b/hypothesis-python/tests/pytest/test_skipping.py
--- a/hypothesis-python/tests/pytest/test_skipping.py
+++ b/hypothesis-python/tests/pytest/test_skipping.py
@@ -39,3 +39,25 @@ def test_no_falsifying_example_if_pytest_skip(testdir):
     )
     out = "\n".join(result.stdout.lines)
     assert "Falsifying example" not in out
+
+
+def test_issue_3453_regression(testdir):
+    """If ``pytest.skip() is called during a test, Hypothesis should not
+    continue running the test and shrink process, nor should it print anything
+    about falsifying examples."""
+    script = testdir.makepyfile(
+        """
+from hypothesis import example, given, strategies as st
+import pytest
+
+@given(value=st.none())
+@example("hello")
+@example("goodbye")
+def test_skip_on_first_skipping_example(value):
+    assert value is not None
+    assert value != "hello"  # queue up a non-skip error which must be discarded
+    pytest.skip()
+"""
+    )
+    result = testdir.runpytest(script, "--tb=native")
+    result.assert_outcomes(skipped=1)