Repository: HypothesisWorks/hypothesis

Issue:
pytest function scoped fixtures should run once per example, not once per test
Because of the way Hypothesis just exposes a single test function, pytest just runs its fixtures once for that function.

This is a long running problem with Hypothesis's interaction with pytest and there is [an open bug about it on the pytest side too](https://github.com/pytest-dev/pytest/issues/916).

Fixing this is tricky. The [fixtures-integration](https://github.com/HypothesisWorks/hypothesis-python/tree/fixtures-integration) branch has a partial prototype which makes _some_ progress towards making it work, but it's very far from complete and has a number of issues which are likely to need substantial work to fix.

I [am currently seeking funding to work on this ticket](http://hypothesis.works/articles/pytest-integration-sponsorship/) and it is unlikely to make much progress without said funding.



Discussion:
Related to #59
Note that the [`pytest-subtesthack`](https://pypi.org/project/pytest-subtesthack/) package by @untitaker provides a decent workaround for this case.
Actually... it might be possible to incorporate a version of `pytest-subtesthack` into our pytest plugin, that activates this logic *only* for Hypothesis tests.  @untitaker, do you think this could work?
Maybe but the version would look very differently, assuming you want this to be the new default behavior of "given"
> Actually... it might be possible to incorporate a version of pytest-subtesthack into our pytest plugin, that activates this logic only for Hypothesis tests

I think we need to be quite careful about this. There are a bunch of problems with `pytest-subtesthack` (or were last I looked) and it's a fairly major change in behaviour. If we're going to support pytest function scoped fixtures I think we need to do it properly.
Fair enough!  I didn't expect it to be that easy, but thought it was worth mentioning.
I think @ossronny can judge better than me how bad the hack really is. I'm not really sure about adding this to hypothesis but I am not sure if it will be fixed in pytest somehow either
After using subtesthack for a while I am now of the opinion that it's a good thing for the user to explicitly think about which fixtures really need setup/teardown, or if they are even testing the right thing if they have fixtures like that.

If `given` were to implicitly setup/teardown fixtures it would result in less test runs than possible with a smaller, more unit-y test.

I am not the kind of guy who has a strong opinion on which tests are supposed to test what (unit tests vs integration tests... most of the time I am just glad to have any tests at all), but I think hypothesis doesn't really work that well with I/O heavy tests. I'm constantly finding old integration tests of mine that use hypothesis and run way slower than necessary and test less of the input space than possible.
> After using subtesthack for a while I am now of the opinion that it's a good thing for the user to explicitly think about which fixtures really need setup/teardown, or if they are even testing the right thing if they have fixtures like that.

My time doing triage on the pytest issue tracker has left me feeling the same way, as well as less optimistic that there's a nice way to hook into fixtures internals.  (even if there was, it would be horribly pytest-version-dependent)

@DRMacIver, how would you feel about outright deprecating use of function-scoped fixtures with `@given`?  I think we can detect that, and it's a much cleaner solution than the current silent subtle maybe-broken state they're left in...
Not sure what exact changes you are proposing? My comment was in advocacy for the status quo, where the default behavior is as-is and people can opt into more test isolation with subtesthack.

On October 10, 2019 9:21:45 AM GMT+02:00, Zac Hatfield-Dodds <notifications@github.com> wrote:
>> After using subtesthack for a while I am now of the opinion that it's
>a good thing for the user to explicitly think about which fixtures
>really need setup/teardown, or if they are even testing the right thing
>if they have fixtures like that.
>
>My time doing triage on the pytest issue tracker has left me feeling
>the same way, as well as less optimistic that there's a nice way to
>hook into fixtures internals.  (even if there was, it would be horribly
>pytest-version-dependent)
>
>@DRMacIver, how would you feel about outright deprecating use of
>function-scoped fixtures with `@given`?  I think we can detect that,
>and it's a much cleaner solution than the current silent subtle
>maybe-broken state they're left in...
>
>-- 
>You are receiving this because you were mentioned.
>Reply to this email directly or view it on GitHub:
>https://github.com/HypothesisWorks/hypothesis/issues/377#issuecomment-540429507

Allow use of fixtures with >function scope; emit warnings for function-scoped fixtures.

IMO this is a strict improvement over the current situation where we silently reuse them and surprise users who actually did need those values scoped to a single test invocation.
> @DRMacIver, how would you feel about outright deprecating use of function-scoped fixtures with `@given`? I think we can detect that, and it's a much cleaner solution than the current silent subtle maybe-broken state they're left in...

I think this is a good idea. Ideally I'd still like to transparently support it in future, but we're not likely to make any progress on this in the foreseeable future, and adding a deprecation warning now gives us a certain leeway in future - we can either make it an error, or we can support it "properly", but until we do one or the other of these it will have significantly fewer gotchas for users if we warn them.
How do I disable this warning? Am I supposed to refactor my tests to nested fns?

```
def test_foo(fixture1, fixture2):

    @given(a=st.text())
    def inner(a):
        assert a or not a

    inner()
```

That would be less of a gotcha for sure and at the same time we can get rid of some of the argument faking code in hypothesis.
> How do I disable this warning? Am I supposed to refactor my tests to nested fns?

It's certainly an option. I've been idly thinking it would be sensible to have a decorator to make it easier to run tests directly (basically just the equivalent of `@given` plus immediately calling the function) but TBH the main way to disable this warning is "don't do that then". There are very few circumstances under which the combination of `given` and a function scoped fixture is the right thing to do.

Another option which would get the correct semantics but I'm not sure how easy it is to do is to refactor your function scoped fixtures into functions you can call explicitly within the test body.
> at the same time we can get rid of some of the argument faking code in hypothesis.

This wouldn't be true FWIW. We'd still need the argument handling code for less fine grained fixtures and for e.g. handling `self` on methods. There may be other use cases too, I don't know. It's specifically only function-scoped fixtures only that we'd want to warn on.
Forwarding, sure, but not going through the effort of making `inspect` return the correct results. Right?
> Forwarding, sure, but not going through the effort of making inspect return the correct results. Right?

No we would still need that in order to get proper support for module scoped fixtures. All pytest fixtures get passed through function arguments regardless of their scope.
Got it, I was thinking of discouraging decorating a test function altogether (and always going for nested functions), since a lot of fixtures are function-scoped anyway

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/extra/pytestplugin.py b/hypothesis-python/src/hypothesis/extra/pytestplugin.py
--- a/hypothesis-python/src/hypothesis/extra/pytestplugin.py
+++ b/hypothesis-python/src/hypothesis/extra/pytestplugin.py
@@ -14,10 +14,12 @@
 # END HEADER
 
 from distutils.version import LooseVersion
+from inspect import signature
 
 import pytest
 
 from hypothesis import Verbosity, core, settings
+from hypothesis._settings import note_deprecation
 from hypothesis.errors import InvalidArgument
 from hypothesis.internal.detection import is_hypothesis_test
 from hypothesis.reporting import default as default_reporter, with_reporter
@@ -144,6 +146,25 @@ def pytest_runtest_call(item):
                     raise InvalidArgument(message % (name,))
             yield
         else:
+            # Warn about function-scoped fixtures, excluding autouse fixtures because
+            # the advice is probably not actionable and the status quo seems OK...
+            # See https://github.com/HypothesisWorks/hypothesis/issues/377 for detail.
+            argnames = None
+            for fx_defs in item._request._fixturemanager.getfixtureinfo(
+                node=item, func=item.function, cls=None
+            ).name2fixturedefs.values():
+                if argnames is None:
+                    argnames = frozenset(signature(item.function).parameters)
+                for fx in fx_defs:
+                    if fx.scope == "function" and fx.argname in argnames:
+                        note_deprecation(
+                            "%s uses the %r fixture, but function-scoped fixtures "
+                            "should not be used with @given(...) tests, because "
+                            "fixtures are not reset between generated examples!"
+                            % (item.nodeid, fx.argname),
+                            since="RELEASEDAY",
+                        )
+
             if item.get_closest_marker("parametrize") is not None:
                 # Give every parametrized test invocation a unique database key
                 key = item.nodeid.encode("utf-8")


Test patch:
diff --git a/hypothesis-python/tests/cover/test_mock.py b/hypothesis-python/tests/cover/test_mock.py
--- a/hypothesis-python/tests/cover/test_mock.py
+++ b/hypothesis-python/tests/cover/test_mock.py
@@ -19,7 +19,7 @@
 import math
 from unittest import mock
 
-from _pytest.capture import CaptureFixture
+from _pytest.config import Config
 
 import hypothesis.strategies as st
 from hypothesis import given
@@ -34,15 +34,15 @@ def test_can_mock_inside_given_without_fixture(atan, thing):
 
 @mock.patch("math.atan")
 @given(thing=st.text())
-def test_can_mock_outside_given_with_fixture(atan, capsys, thing):
+def test_can_mock_outside_given_with_fixture(atan, pytestconfig, thing):
     assert isinstance(atan, mock.MagicMock)
     assert isinstance(math.atan, mock.MagicMock)
-    assert isinstance(capsys, CaptureFixture)
+    assert isinstance(pytestconfig, Config)
 
 
 @given(thing=st.text())
-def test_can_mock_within_test_with_fixture(capsys, thing):
-    assert isinstance(capsys, CaptureFixture)
+def test_can_mock_within_test_with_fixture(pytestconfig, thing):
+    assert isinstance(pytestconfig, Config)
     assert not isinstance(math.atan, mock.MagicMock)
     with mock.patch("math.atan") as atan:
         assert isinstance(atan, mock.MagicMock)
diff --git a/hypothesis-python/tests/pytest/test_fixtures.py b/hypothesis-python/tests/pytest/test_fixtures.py
--- a/hypothesis-python/tests/pytest/test_fixtures.py
+++ b/hypothesis-python/tests/pytest/test_fixtures.py
@@ -21,18 +21,20 @@
 from hypothesis.strategies import integers
 from tests.common.utils import fails
 
+pytest_plugins = "pytester"
 
-@pytest.fixture
+
+@pytest.fixture(scope="session")
 def infinity():
     return float("inf")
 
 
-@pytest.fixture
+@pytest.fixture(scope="module")
 def mock_fixture():
     return Mock()
 
 
-@pytest.fixture
+@pytest.fixture(scope="module")
 def spec_fixture():
     class Foo:
         def __init__(self):
@@ -77,3 +79,26 @@ def test_can_inject_mock_via_fixture(mock_fixture, xs):
 def test_can_inject_autospecced_mock_via_fixture(spec_fixture, xs):
     spec_fixture.bar.return_value = float("inf")
     assert xs <= spec_fixture.bar()
+
+
+TESTSUITE = """
+import pytest
+from hypothesis import given, strategies as st
+
+@pytest.fixture(scope="function", autouse=True)
+def autofix(request):
+    pass
+
+@given(x=st.integers())
+def test_requests_function_scoped_fixture(capsys, x):
+    pass
+
+@given(x=st.integers())
+def test_autouse_function_scoped_fixture(x):
+    pass
+"""
+
+
+def test_given_plus_function_scoped_non_autouse_fixtures_are_deprecated(testdir):
+    script = testdir.makepyfile(TESTSUITE)
+    testdir.runpytest(script, "-Werror").assert_outcomes(passed=1, failed=1)