Repository: HypothesisWorks/hypothesis
Pull request: add .hypothesis to invalid tests
adds `.hypothesis` to invalid tests.

this fixes #2966

I'm not quite sure where (or how) I should test it, I couldn't find any tests using .hypothesis other than the ones swapping the test function and that won't work here without making the invalid tests far more complicated...

all existing tests seem to pass.

Problem statement:
confusing errors when using hypothesis with pytest-asyncio
given the following invalid test as `foo` does not exist...

```python
import hypothesis
import pytest

@pytest.mark.asyncio
@hypothesis.given(foo=hypothesis.infer)
async def test_foo() -> None:
    pass
```

pytest will error the test with the following...

```python-traceback
item = <Function test_foo>

    def pytest_runtest_setup(item):
        if 'asyncio' in item.keywords:
            # inject an event loop fixture for all async tests
            if 'event_loop' in item.fixturenames:
                item.fixturenames.remove('event_loop')
            item.fixturenames.insert(0, 'event_loop')
        if item.get_closest_marker("asyncio") is not None \
            and not getattr(item.obj, 'hypothesis', False) \
            and getattr(item.obj, 'is_hypothesis_test', False):
>               pytest.fail(
                    'test function `%r` is using Hypothesis, but pytest-asyncio '
                    'only works with Hypothesis 3.64.0 or later.' % item
                )
E               Failed: test function `<Function test_foo>` is using Hypothesis, but pytest-asyncio only works with Hypothesis 3.64.0 or later.
```

without the async mark

```python
import hypothesis
import pytest

@hypothesis.given(foo=hypothesis.infer)
async def test_foo() -> None:
    pass
```

you get the real error back....

```python-traceback
arguments = (), kwargs = {}

    def wrapped_test(*arguments, **kwargs):
>       raise InvalidArgument(message)
E       hypothesis.errors.InvalidArgument: test_foo() got an unexpected keyword argument 'foo', from `foo=infer` in @given
```

I am assuming it fails because the `invalid()` test returned by given has `is_hypothesis_test` set, but does not also set `hypothesis` to `HypothesisHandle`. and the hypotheisis detection code in pytest-asyncio relies on both or neither being present.

```
platform linux -- Python 3.9.4, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /workspace, configfile: pyproject.toml
plugins: asyncio-0.14.0, hypothesis-6.12.0
```

Hints from issue discussion:
Yep, looks like you're exactly right.  Want to open a PR to fix it?  If not, we'll be happy to fix

https://github.com/HypothesisWorks/hypothesis/blob/06aa7b45e57ac90e3c66e24b0b8157d70af1aa08/hypothesis-python/src/hypothesis/core.py#L238-L243
I am happy to make a PR, but what would the preferred fix be? Deleting the `is_hypothesis_test` property or adding the `hypothesis` property?
Adding the `.hypothesis` property would be best!

Commit message: test: adding tests for .hypothesis on invalid tests

Patch to review:
diff --git a/AUTHORS.rst b/AUTHORS.rst
index cede38481b..0fa24c42bf 100644
--- a/AUTHORS.rst
+++ b/AUTHORS.rst
@@ -36,6 +36,7 @@ their individual contributions.
 * `Cristi Cobzarenco <https://github.com/cristicbz>`_ (cristi@reinfer.io)
 * `Damon Francisco <https://github.com/dtfrancisco>`_ (damontfrancisco@yahoo.com)
 * `Daniel J. West <https://github.com/danieljwest>`_
+* `Daniel Knell <https://github.com/danielknell>`_ (contact@danielknell.co.uk)
 * `David Bonner <https://github.com/rascalking>`_ (dbonner@gmail.com)
 * `David Chudzicki <https://github.com/dchudz>`_ (dchudz@gmail.com)
 * `David Mascharka <https://github.com/davidmascharka>`_
diff --git a/hypothesis-python/RELEASE.rst b/hypothesis-python/RELEASE.rst
new file mode 100644
index 0000000000..e1d42afc9a
--- /dev/null
+++ b/hypothesis-python/RELEASE.rst
@@ -0,0 +1,6 @@
+RELEASE_TYPE: patch
+
+This patch adds a `.hypothesis` property to invalid test functions, bringing
+them inline with valid tests and fixing a bug where `pytest-asyncio` would
+swallow the real error message and mistakenly raise a version incompatibility
+error.
\ No newline at end of file
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
index 287a0260b2..b11b420cc6 100644
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -226,7 +226,7 @@ def __repr__(self):
         return f"WithRunner({self.mapped_strategy!r}, runner={self.runner!r})"
 
 
-def is_invalid_test(name, original_argspec, given_arguments, given_kwargs):
+def is_invalid_test(test, original_argspec, given_arguments, given_kwargs):
     """Check the arguments to ``@given`` for basic usage constraints.
 
     Most errors are not raised immediately; instead we return a dummy test
@@ -240,6 +240,7 @@ def wrapped_test(*arguments, **kwargs):
             raise InvalidArgument(message)
 
         wrapped_test.is_hypothesis_test = True
+        wrapped_test.hypothesis = HypothesisHandle(test, wrapped_test, given_kwargs)
         return wrapped_test
 
     if not (given_arguments or given_kwargs):
@@ -258,7 +259,7 @@ def wrapped_test(*arguments, **kwargs):
         return invalid(
             "Too many positional arguments for %s() were passed to @given "
             "- expected at most %d arguments, but got %d %r"
-            % (name, len(original_argspec.args), len(args), args)
+            % (test.__name__, len(original_argspec.args), len(args), args)
         )
 
     if infer in given_arguments:
@@ -278,7 +279,7 @@ def wrapped_test(*arguments, **kwargs):
         arg = extra_kwargs[0]
         return invalid(
             "%s() got an unexpected keyword argument %r, from `%s=%r` in @given"
-            % (name, arg, arg, given_kwargs[arg])
+            % (test.__name__, arg, arg, given_kwargs[arg])
         )
     if original_argspec.defaults or original_argspec.kwonlydefaults:
         return invalid("Cannot apply @given to a function with defaults.")
@@ -959,7 +960,7 @@ def run_test_as_given(test):
         original_argspec = getfullargspec(test)
 
         check_invalid = is_invalid_test(
-            test.__name__, original_argspec, given_arguments, given_kwargs
+            test, original_argspec, given_arguments, given_kwargs
         )
 
         # If the argument check found problems, return a dummy test function
diff --git a/hypothesis-python/tests/cover/test_fuzz_one_input.py b/hypothesis-python/tests/cover/test_fuzz_one_input.py
index 7fc85391ab..14508643c0 100644
--- a/hypothesis-python/tests/cover/test_fuzz_one_input.py
+++ b/hypothesis-python/tests/cover/test_fuzz_one_input.py
@@ -21,8 +21,11 @@
 
 from hypothesis import Phase, given, settings, strategies as st
 from hypothesis.database import InMemoryExampleDatabase
+from hypothesis.errors import InvalidArgument
 from hypothesis.internal.conjecture.shrinker import sort_key
 
+from tests.common.utils import raises
+
 
 @pytest.mark.parametrize(
     "buffer_type",
@@ -155,3 +158,15 @@ def test(s):
     (saved_examples,) = db.data.values()
     assert seen == buffers
     assert len(saved_examples) == db_size
+
+
+def test_fuzzing_invalid_test_raises_error():
+    # Invalid: @given with too many positional arguments
+    @given(st.integers(), st.integers())
+    def invalid_test(s):
+        pass
+
+    with raises(InvalidArgument) as e:
+        invalid_test.hypothesis.fuzz_one_input(b"abc")
+
+    assert "Too many positional arguments" in e.value.args[0]
diff --git a/hypothesis-python/tests/nocover/test_modify_inner_test.py b/hypothesis-python/tests/nocover/test_modify_inner_test.py
index 8ca571cf3d..780b138e63 100644
--- a/hypothesis-python/tests/nocover/test_modify_inner_test.py
+++ b/hypothesis-python/tests/nocover/test_modify_inner_test.py
@@ -18,6 +18,9 @@
 import pytest
 
 from hypothesis import given, strategies as st
+from hypothesis.errors import InvalidArgument
+
+from tests.common.utils import raises
 
 
 def always_passes(*args, **kwargs):
@@ -62,3 +65,39 @@ def test_can_replace_when_parametrized(x, y):
 
 
 test_can_replace_when_parametrized.hypothesis.inner_test = always_passes
+
+
+def test_can_replace_when_original_is_invalid():
+    # Invalid: @given with too many positional arguments
+    @given(st.integers(), st.integers())
+    def invalid_test(x):
+        assert False
+
+    invalid_test.hypothesis.inner_test = always_passes
+
+    # Even after replacing the inner test, calling the wrapper should still
+    # fail.
+    with raises(InvalidArgument) as e:
+        invalid_test()
+
+    assert "Too many positional arguments" in e.value.args[0]
+
+
+def test_inner_is_original_even_when_invalid():
+    def invalid_test(x):
+        assert False
+
+    original = invalid_test
+
+    # Invalid: @given with no arguments
+    invalid_test = given()(invalid_test)
+
+    assert invalid_test.hypothesis.inner_test == original
+
+    # Verify that the test is actually invalid
+    with raises(InvalidArgument) as e:
+        invalid_test()
+
+    assert "given must be called with at least one argument" in e.value.args[0]
+
+    assert invalid_test.hypothesis.inner_test == original


Code review comments:
diff_hunk: @@ -155,3 +158,15 @@ def test(s):
     (saved_examples,) = db.data.values()
     assert seen == buffers
     assert len(saved_examples) == db_size
+
+
+def test_fuzzing_invalid_test_raises_error():
+    # Invalid: @given with too many positional arguments
+    @given(st.integers(), st.integers())
+    def invalid_test(s):
+        pass
+
+    with raises(InvalidArgument) as e:
path: hypothesis-python/tests/cover/test_fuzz_one_input.py
text: @user1:
```suggestion
    with pytest.raises(InvalidArgument, match="Too many positional arguments"):
```

See [`pytest.raises` docs](https://docs.pytest.org/en/latest/reference/reference.html#pytest-raises); I think we can use this in each case for somewhat clearer tests.

diff_hunk: @@ -240,6 +240,7 @@ def wrapped_test(*arguments, **kwargs):
             raise InvalidArgument(message)
 
         wrapped_test.is_hypothesis_test = True
+        wrapped_test.hypothesis = HypothesisHandle(test, wrapped_test, given_kwargs)
path: hypothesis-python/src/hypothesis/core.py
text: @user1:
I think this would be a lot easier to read with keyword arguments.

@author:
should I change the main call inside given too? at the moment it's consistent with that.

@author:
```TypeError: __init__() got an unexpected keyword argument '_get_fuzz_target'```

when I try and call with kwargs.

diff_hunk: @@ -155,3 +158,15 @@ def test(s):
     (saved_examples,) = db.data.values()
     assert seen == buffers
     assert len(saved_examples) == db_size
+
+
+def test_fuzzing_invalid_test_raises_error():
+    # Invalid: @given with too many positional arguments
+    @given(st.integers(), st.integers())
+    def invalid_test(s):
+        pass
+
+    with raises(InvalidArgument) as e:
+        invalid_test.hypothesis.fuzz_one_input(b"abc")
path: hypothesis-python/tests/cover/test_fuzz_one_input.py
text: @user1:
```suggestion
        invalid_test.hypothesis.fuzz_one_input
```

Because we specifically want to test that this happens during setup.