Repository: HypothesisWorks/hypothesis

Issue:
Release 6.49.1 breaks @example in user code
In PINT we have a number of `hypothesis` tests, some of which used to fail and thus have `@example` attached. These were constructed in the usual way, by pasting them in from `hypothesis` output. This was a while ago! But with release 6.49.1, presumably the part about fixing inconsistencies in argument handling between `given` and `example`. 

The release that seems to have caused the problem: https://github.com/HypothesisWorks/hypothesis/compare/hypothesis-python-6.49.0...hypothesis-python-6.49.1

The problems we are seeing: https://github.com/nanograv/PINT/runs/7203801688?check_suite_focus=true

Specifically:
```
________________________ test_d2tf_tf2d_roundtrip[9-1] _________________________
ndp = 9, k = 1
    @given(floats(-2, 2, allow_nan=False))
>   @example(ndp=10, f=1.3322676295501882e-15)
/home/runner/work/PINT/PINT/tests/test_precision.py:732: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
state = <hypothesis.core.StateForActualGivenExecution object at 0x7ff9bc348e80>
wrapped_test = <function accept.<locals>.test_d2tf_tf2d_roundtrip at 0x7ff9d15adbd0>
arguments = (), kwargs = {'k': 1, 'ndp': 9}
original_sig = <Signature (ndp, k, f)>
    def execute_explicit_examples(state, wrapped_test, arguments, kwargs, original_sig):
        posargs = [
            p.name
            for p in original_sig.parameters.values()
            if p.kind is p.POSITIONAL_OR_KEYWORD
        ]
        for example in reversed(getattr(wrapped_test, "hypothesis_explicit_examples", ())):
            if example.args:
                assert not example.kwargs
                if len(example.args) > len(posargs):
                    raise InvalidArgument(
                        "example has too many arguments for test. Expected at most "
                        f"{len(posargs)} but got {len(example.args)}"
                    )
                example_kwargs = dict(zip(posargs[::-1], example.args[::-1]))
            else:
                example_kwargs = dict(example.kwargs)
>           bound = original_sig.bind(*arguments, **example_kwargs, **kwargs)
E           TypeError: inspect.Signature.bind() got multiple values for keyword argument 'ndp'
/home/runner/work/PINT/PINT/.tox/py310-test-cov/lib/python3.10/site-packages/hypothesis/core.py:373: TypeError
```

If our `@example` syntax is somehow obsolete, we can try to fix it. Does `hypothesis` now suggest examples with syntax that works?


Discussion:
The actual code that's triggering this (one of many): https://github.com/nanograv/PINT/blob/6c4deece5074248a2aa26b90088da37d430852d5/tests/test_precision.py#L745

```python
@given(floats(-2, 2, allow_nan=False))
@example(ndp=10, f=1.3322676295501882e-15)
@example(ndp=11, f=4.440892098500627e-16)
@example(ndp=12, f=4.440892098500627e-16)
@pytest.mark.parametrize(
    "ndp, k",
    [
        (8, 1),
        (9, 1),
        pytest.param(10, 100, marks=pytest.mark.xfail(reason="ERFA limitations")),
        pytest.param(11, 1000, marks=pytest.mark.xfail(reason="ERFA limitations")),
        pytest.param(12, 10000, marks=pytest.mark.xfail(reason="ERFA limitations")),
    ],
)
def test_d2tf_tf2d_roundtrip(ndp, k, f):
    assert abs(tf2d_nice(*d2tf_nice(ndp, f)) - f) * 86400 < k * 10 ** (-ndp)
```
The `@example` decorator should reflect only those arguments that `@given` supplies (in this instance, `f`). In the code you shared `@given` specifies `f` but your `@example` specifies both `f` and `ndp`. 

Thus I believe the "got multiple values for keyword argument 'ndp'" error that you are seeing now was also happening before, but instead of raising an error the pytest-supplied value for `ndp` was always being used and only the value for `f` was being supplied by your `@example` decorator. This is something that Hypothesis should have raised on in earlier versions.

For example, when I run the following test using Hypothesis 6.47.3 the test passes (it shouldn't!)

```python
from hypothesis import given, example
from hypothesis.strategies import floats
import pytest

@example(ndp=10, f=1.3322676295501882e-15)
@example(ndp=11, f=4.440892098500627e-16)
@example(ndp=12, f=4.440892098500627e-16)
@given(floats(-2, 2, allow_nan=False))
@pytest.mark.parametrize(
    "ndp, k",
    [
        (8, 1),
        (9, 1),
    ],
)
def test_d2tf_tf2d_roundtrip(ndp, k, f):
    assert ndp != 10  # should cause first example to fail
```

```console
$ pytest test_pint.py
 ================= test session starts =================
platform linux -- Python 3.10.5, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/ry26099/raise_architecture
plugins: hypothesis-6.47.3
collected 2 items                                                                                                                                                                                                                           

src/raise_arch/typing/array_protocols.py .. 
```

So, to the best of my knowledge, what has actually been tested all along is:

```python
@example(f=1.3322676295501882e-15)
@example(f=4.440892098500627e-16)
@example(f=4.440892098500627e-16)
@given(floats(-2, 2, allow_nan=False))
@pytest.mark.parametrize(
    "ndp, k",
    [
        (8, 1),
        (9, 1),
    ],
)
def test_d2tf_tf2d_roundtrip(ndp, k, f):
    ...
```


Is there a way to specify the actual example that causes this problem? It only occurs for that specific value of the `ndp` parameter, how can I specify that?
I realize I could convert many uses of `pytest.mark.parametrize` to hypothesis' pick-from-a-list strategies, but that doesn't allow for `xfail` annotations, which in this case are meant to let us know when the bug is fixed, and to document this limitation of ERFA. 
> but that doesn't allow for xfail annotations

I agree that keeping pytest-parameterized capabilities like `xfail` is a must for your applications. You definitely should not sacrifice that.

I am not sure that there is a clean way to do this in a single test, given that you are mixing `@given` with `@parameterized` and you want to capture regression examples. You might consider making two tests:

```python
def d2tf_tf2d_roundtrip(ndp, k, f):
    # actually performs the roundtrip checks
    ...
    
@given(floats(-2, 2, allow_nan=False))
@pytest.mark.parametrize(
    "ndp, k",
    [
        (8, 1),
        (9, 1),
    ],
)
def test_d2tf_tf2d_roundtrip(ndp, k, f):
    d2tf_tf2d_roundtrip(ndp, k, f)

@pytest.mark.parametrize(
    "ndp, f, k",
    [
        (10, 1.3322676295501882e-15, 1),
        (11, 4.440892098500627e-16, 1),
        (12, 4.440892098500627e-16, 1),
    ],
)
def test_d2tf_tf2d_roundtrip_regression(ndp, k, f):
    d2tf_tf2d_roundtrip(ndp, k, f)
```

Here `d2tf_tf2d_roundtrip` saves you the trouble of duplicating the body of your tests between your hypothesis test and your regression test.

You could also make `@given` and `@example` responsible for producing the same fields -- `ndp_h` and `f` -- but only use `ndp_h` when it is supplied by examples

```python
from hypothesis import given, example
from hypothesis.strategies import none, floats

@given(ndp_h=none(), f=floats(-2, 2, allow_nan=False))
@example(ndp_h=10, f=1.3322676295501882e-15)
@example(ndp_h=11, f=4.440892098500627e-16)
@example(ndp_h=12, f=4.440892098500627e-16)
@pytest.mark.parametrize(
    "ndp, k",
    [
        (8, 1),
        (9, 1),
        pytest.param(10, 100, marks=pytest.mark.xfail(reason="ERFA limitations")),
        pytest.param(11, 1000, marks=pytest.mark.xfail(reason="ERFA limitations")),
        pytest.param(12, 10000, marks=pytest.mark.xfail(reason="ERFA limitations")),
    ],
)
def test_d2tf_tf2d_roundtrip(ndp_h, ndp, k, f):
    if ndp_h is not None:
        ndp = ndp_h
    assert abs(tf2d_nice(*d2tf_nice(ndp, f)) - f) * 86400 < k * 10 ** (-ndp)
```

This saves the trouble of splitting out the tests, but it makes your test less legible and arguably more error-prone. Personally, I would go for the former. Note that this refactoring should only be necessary when your tests used `@example` to specify more/different arguments than `@given`.
I'll also improve the error message for this case to explain why it occurs and how to fix it. 
Thanks for the suggestions. I guess the short answer is, `hypothesis` cannot coexist well with `pytest.mark.parametrize`, so if you're using both you can't use `@example` to specify a case that you're interested in. (Any such cases are broadcast across all the `parametrize` values.) Given that, it may just make sense to use a comment to indicate which `parametrize` values each example is relevant for:

```python
@given(floats(-2, 2, allow_nan=False))
@example(f=1.3322676295501882e-15)  # ndp=10
@example(f=4.440892098500627e-16)  # ndp=11
@example(f=4.440892098500627e-16)  # ndp=12
@pytest.mark.parametrize(
    "ndp, k",
    [
        (8, 1),
        (9, 1),
        pytest.param(10, 100, marks=pytest.mark.xfail(reason="ERFA limitations")),
        pytest.param(11, 1000, marks=pytest.mark.xfail(reason="ERFA limitations")),
        pytest.param(12, 10000, marks=pytest.mark.xfail(reason="ERFA limitations")),
    ],
)
def test_d2tf_tf2d_roundtrip(ndp, k, f):
    assert abs(tf2d_nice(*d2tf_nice(ndp, f)) - f) * 86400 < k * 10 ** (-ndp)
```

I'd be fine with sticking to `hypothesis` to handle the parametrization, but I don't think it's possible to `xfail` only certain cases (and they don't turn into separate tests anyway, so troubleshooting can be more difficult). This Stack Exchange, uh, exchange illustrates that the problem is known: https://stackoverflow.com/questions/56941115/xfail-single-hypothesis-case-using-pytest Paul Ganssle also points out that the interaction of `hypothesis` and `xfail` is not good (papercut): https://ganssle.io/files/talks/2022-03-26_pytexas_2022_xfail.pdf



> I'll also improve the error message for this case to explain why it occurs and how to fix it.

I think in terms of this specific bug, a clearer error message would be a help. Does it make sense to open an issue on the broader questions of coexistence of `hypothesis`, `pytest.mark.parametrize`, and `xfail`?

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -359,29 +359,50 @@ def execute_explicit_examples(state, wrapped_test, arguments, kwargs, original_s
     ]
 
     for example in reversed(getattr(wrapped_test, "hypothesis_explicit_examples", ())):
+        # All of this validation is to check that @example() got "the same" arguments
+        # as @given, i.e. corresponding to the same parameters, even though they might
+        # be any mixture of positional and keyword arguments.
         if example.args:
             assert not example.kwargs
+            if any(
+                p.kind is p.POSITIONAL_ONLY for p in original_sig.parameters.values()
+            ):
+                raise InvalidArgument(
+                    "Cannot pass positional arguments to @example() when decorating "
+                    "a test function which has positional-only parameters."
+                )
             if len(example.args) > len(posargs):
                 raise InvalidArgument(
                     "example has too many arguments for test. Expected at most "
                     f"{len(posargs)} but got {len(example.args)}"
                 )
-            example_kwargs = dict(zip(posargs[::-1], example.args[::-1]))
+            example_kwargs = dict(zip(posargs[-len(example.args) :], example.args))
         else:
             example_kwargs = dict(example.kwargs)
+        given_kws = ", ".join(
+            repr(k) for k in sorted(wrapped_test.hypothesis._given_kwargs)
+        )
+        example_kws = ", ".join(repr(k) for k in sorted(example_kwargs))
+        if given_kws != example_kws:
+            raise InvalidArgument(
+                f"Inconsistent args: @given() got strategies for {given_kws}, "
+                f"but @example() got arguments for {example_kws}"
+            ) from None
 
-        bound = original_sig.bind(*arguments, **example_kwargs, **kwargs)
+        # This is certainly true because the example_kwargs exactly match the params
+        # reserved by @given(), which are then remove from the function signature.
+        assert set(example_kwargs).isdisjoint(kwargs)
+        example_kwargs.update(kwargs)
 
         if Phase.explicit not in state.settings.phases:
             continue
 
-        args, kw = convert_positional_arguments(original_sig, bound.args, bound.kwargs)
         with local_settings(state.settings):
             fragments_reported = []
             try:
                 with with_reporter(fragments_reported.append):
                     state.execute_once(
-                        ArtificialDataForExample(args, kw),
+                        ArtificialDataForExample(arguments, example_kwargs),
                         is_final=True,
                         print_example=True,
                     )
diff --git a/hypothesis-python/src/hypothesis/internal/reflection.py b/hypothesis-python/src/hypothesis/internal/reflection.py
--- a/hypothesis-python/src/hypothesis/internal/reflection.py
+++ b/hypothesis-python/src/hypothesis/internal/reflection.py
@@ -186,10 +186,7 @@ def convert_positional_arguments(function, args, kwargs):
 
     new_args will only be non-empty if function has pos-only args or *args.
     """
-    if isinstance(function, inspect.Signature):
-        sig = function
-    else:
-        sig = inspect.signature(function, follow_wrapped=False)
+    sig = inspect.signature(function, follow_wrapped=False)
     bound = sig.bind(*args, **kwargs)
     new_args = []
     new_kwargs = dict(bound.arguments)


Test patch:
diff --git a/hypothesis-python/tests/cover/test_lookup_py38.py b/hypothesis-python/tests/cover/test_lookup_py38.py
--- a/hypothesis-python/tests/cover/test_lookup_py38.py
+++ b/hypothesis-python/tests/cover/test_lookup_py38.py
@@ -9,12 +9,13 @@
 # obtain one at https://mozilla.org/MPL/2.0/.
 
 import dataclasses
+import re
 import sys
 import typing
 
 import pytest
 
-from hypothesis import given, strategies as st
+from hypothesis import example, given, strategies as st
 from hypothesis.errors import InvalidArgument, Unsatisfiable
 from hypothesis.internal.reflection import (
     convert_positional_arguments,
@@ -194,3 +195,30 @@ def test(x, /, y):
 @given(st.booleans())
 def test_cannot_pass_strategies_for_posonly_args(x, /):
     pass
+
+
+@given(y=st.booleans())
+def has_posonly_args(x, /, y):
+    pass
+
+
+def test_example_argument_validation():
+    example(y=None)(has_posonly_args)(1)  # Basic case is OK
+
+    with pytest.raises(
+        InvalidArgument,
+        match=re.escape(
+            "Cannot pass positional arguments to @example() when decorating "
+            "a test function which has positional-only parameters."
+        ),
+    ):
+        example(None)(has_posonly_args)(1)
+
+    with pytest.raises(
+        InvalidArgument,
+        match=re.escape(
+            "Inconsistent args: @given() got strategies for 'y', "
+            "but @example() got arguments for 'x'"
+        ),
+    ):
+        example(x=None)(has_posonly_args)(1)