Repository: HypothesisWorks/hypothesis

Issue:
Turn pytest function-scoped fixtures into a health check error (instead of a hard error)
As I mentioned in https://github.com/HypothesisWorks/hypothesis/issues/2709#issuecomment-753732671, I'm thinking about adding a health check for function-scoped fixtures, so that rare intentional uses can continue to opt-out when the warning becomes an error (e.g. #2731)

(Context: Using pytest function-scoped fixtures with `given` is almost always a bad idea, because the fixtures only run once for the whole function, and are not reset between individual examples.)

Implementing this is slightly annoying, because our pytest plugin doesn't have convenient access to the actual settings object that will be used by the underlying test. Currently my plan is to have the plugin continue to generate a warning/error message, but instead of calling `note_deprecation`, it will merely stash the message in a `_hypothesis_internal...` property of the test. Then I'll add a check in `given`/`wrapped_test` (near the existing `not_a_test_method` check) that will pull out the stashed message and pass it to `fail_health_check`, which will raise an error or silently do nothing as appropriate.

The main questions I have right now are:

- Should we actually preserve the opt-out?
  - If we do ever get proper per-example fixture support, it will be a breaking change for anybody relying on the opt-out, which is awkward (and presumably one of the motivations behind the error).
  - On the other hand, this doesn't seem likely to happen in the foreseeable future.
  - Aside from compatibility, it could be argued that having a convenient opt-out would lead to more harm than good, if people use it to silence a pesky error without understanding the actual problem.
  - I'm inclined not to worry about this too much.

- Is a health check the right mechanism?
  - Adding a pytest-specific health check is arguably a layering violation, though in practice I think it's fine.
  - In https://github.com/HypothesisWorks/hypothesis/issues/2731#issuecomment-753561077, @Zac-HD proposed a `@pytest.mark.hypothesis_function_scoped_is_fine` annotation to achieve the same result in a different way.
  - Using the existing health-check mechanism seems better to me, since it's a good fit for the semantics of health checks (severe enough to be an error by default, but there are legitimate reasons to opt out).

- Is it OK to roll this change into our annual major version bump, assuming I have a suitable patch in good time?


Discussion:
I agree that this should be a health check rather than an unconditional error, and that allowing opt-out is OK since there is a legitimate interest in doing so.  I'm not at all attached to using a pytest mark, that's just what I thought of off-hand and I prefer the healthcheck idea.

I'd actually rather ship this *before* our major version bump, as soon as convenient for you - for now the setting allows you to silence the *deprecation warning*, and then we upgrade that to a health check error in 6.0 as scheduled.  It's also fine if someone does eventually contribute or fund fixture integration; we just remove the healthcheck error and deprecate that enum entry.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/_settings.py b/hypothesis-python/src/hypothesis/_settings.py
--- a/hypothesis-python/src/hypothesis/_settings.py
+++ b/hypothesis-python/src/hypothesis/_settings.py
@@ -478,6 +478,24 @@ def all(cls) -> List["HealthCheck"]:
     """Checks if :func:`@given <hypothesis.given>` has been applied to a
     method defined by :class:`python:unittest.TestCase` (i.e. not a test)."""
 
+    function_scoped_fixture = 9
+    """Check if :func:`@given <hypothesis.given>` has been applied to a test
+    with a pytest function-scoped fixture. Function-scoped fixtures run once
+    for the whole function, not once per example, and this is usually not what
+    you want.
+
+    Because of this limitation, tests that need to need to set up or reset
+    state for every example need to do so manually within the test itself,
+    typically using an appropriate context manager.
+
+    Suppress this health check only in the rare case that you are using a
+    function-scoped fixture that does not need to be reset between individual
+    examples, but for some reason you cannot use a wider fixture scope
+    (e.g. session scope, module scope, class scope).
+
+    This check requires the :ref:`Hypothesis pytest plugin<pytest-plugin>`,
+    which is enabled by default when running Hypothesis inside pytest."""
+
 
 @unique
 class Verbosity(IntEnum):
diff --git a/hypothesis-python/src/hypothesis/extra/pytestplugin.py b/hypothesis-python/src/hypothesis/extra/pytestplugin.py
--- a/hypothesis-python/src/hypothesis/extra/pytestplugin.py
+++ b/hypothesis-python/src/hypothesis/extra/pytestplugin.py
@@ -17,7 +17,7 @@
 
 import pytest
 
-from hypothesis import Verbosity, core, settings
+from hypothesis import HealthCheck, Verbosity, core, settings
 from hypothesis._settings import note_deprecation
 from hypothesis.errors import InvalidArgument
 from hypothesis.internal.detection import is_hypothesis_test
@@ -146,32 +146,46 @@ def pytest_runtest_call(item):
                     raise InvalidArgument(message % (name,))
             yield
         else:
-            # Warn about function-scoped fixtures, excluding autouse fixtures because
-            # the advice is probably not actionable and the status quo seems OK...
-            # See https://github.com/HypothesisWorks/hypothesis/issues/377 for detail.
-            msg = (
-                "%s uses the %r fixture, which is reset between function calls but not "
-                "between test cases generated by `@given(...)`.  You can change it to "
-                "a module- or session-scoped fixture if it is safe to reuse; if not "
-                "we recommend using a context manager inside your test function.  See "
-                "https://docs.pytest.org/en/latest/fixture.html#sharing-test-data "
-                "for details on fixture scope."
-            )
-            argnames = None
-            for fx_defs in item._request._fixturemanager.getfixtureinfo(
-                node=item, func=item.function, cls=None
-            ).name2fixturedefs.values():
-                if argnames is None:
-                    argnames = frozenset(signature(item.function).parameters)
-                for fx in fx_defs:
-                    if fx.argname in argnames:
-                        active_fx = item._request._get_active_fixturedef(fx.argname)
-                        if active_fx.scope == "function":
-                            note_deprecation(
-                                msg % (item.nodeid, fx.argname),
-                                since="2020-02-29",
-                                has_codemod=False,
-                            )
+            # Retrieve the settings for this test from the test object, which
+            # is normally a Hypothesis wrapped_test wrapper. If this doesn't
+            # work, the test object is probably something weird
+            # (e.g a stateful test wrapper), so we skip the function-scoped
+            # fixture check.
+            settings = getattr(item.obj, "_hypothesis_internal_use_settings", None)
+
+            # Check for suspicious use of function-scoped fixtures, but only
+            # if the corresponding health check is not suppressed.
+            if (
+                settings is not None
+                and HealthCheck.function_scoped_fixture
+                not in settings.suppress_health_check
+            ):
+                # Warn about function-scoped fixtures, excluding autouse fixtures because
+                # the advice is probably not actionable and the status quo seems OK...
+                # See https://github.com/HypothesisWorks/hypothesis/issues/377 for detail.
+                msg = (
+                    "%s uses the %r fixture, which is reset between function calls but not "
+                    "between test cases generated by `@given(...)`.  You can change it to "
+                    "a module- or session-scoped fixture if it is safe to reuse; if not "
+                    "we recommend using a context manager inside your test function.  See "
+                    "https://docs.pytest.org/en/latest/fixture.html#sharing-test-data "
+                    "for details on fixture scope."
+                )
+                argnames = None
+                for fx_defs in item._request._fixturemanager.getfixtureinfo(
+                    node=item, func=item.function, cls=None
+                ).name2fixturedefs.values():
+                    if argnames is None:
+                        argnames = frozenset(signature(item.function).parameters)
+                    for fx in fx_defs:
+                        if fx.argname in argnames:
+                            active_fx = item._request._get_active_fixturedef(fx.argname)
+                            if active_fx.scope == "function":
+                                note_deprecation(
+                                    msg % (item.nodeid, fx.argname),
+                                    since="2020-02-29",
+                                    has_codemod=False,
+                                )
 
             if item.get_closest_marker("parametrize") is not None:
                 # Give every parametrized test invocation a unique database key


Test patch:
diff --git a/hypothesis-python/tests/pytest/test_fixtures.py b/hypothesis-python/tests/pytest/test_fixtures.py
--- a/hypothesis-python/tests/pytest/test_fixtures.py
+++ b/hypothesis-python/tests/pytest/test_fixtures.py
@@ -99,6 +99,11 @@ def test_requests_function_scoped_fixture_percent_parametrized(capsys, x, percen
     # See https://github.com/HypothesisWorks/hypothesis/issues/2469
     pass
 
+class TestClass:
+    @given(x=st.integers())
+    def test_requests_method_scoped_fixture(capsys, x):
+        pass
+
 @given(x=st.integers())
 def test_autouse_function_scoped_fixture(x):
     pass
@@ -107,8 +112,54 @@ def test_autouse_function_scoped_fixture(x):
 
 def test_given_plus_function_scoped_non_autouse_fixtures_are_deprecated(testdir):
     script = testdir.makepyfile(TESTSUITE)
-    testdir.runpytest(script).assert_outcomes(passed=4)
-    testdir.runpytest(script, "-Werror").assert_outcomes(passed=1, failed=3)
+    testdir.runpytest(script).assert_outcomes(passed=5)
+    testdir.runpytest(script, "-Werror").assert_outcomes(passed=1, failed=4)
+
+
+CONFTEST_SUPPRESS = """
+from hypothesis import HealthCheck, settings
+
+settings.register_profile(
+    "suppress",
+    suppress_health_check=[HealthCheck.function_scoped_fixture],
+)
+"""
+
+
+def test_suppress_fixture_health_check_via_profile(testdir):
+    script = testdir.makepyfile(TESTSUITE)
+    testdir.makeconftest(CONFTEST_SUPPRESS)
+
+    testdir.runpytest(script, "-Werror").assert_outcomes(passed=1, failed=4)
+    testdir.runpytest(
+        script, "-Werror", "--hypothesis-profile=suppress"
+    ).assert_outcomes(passed=5)
+
+
+TESTSCRIPT_SUPPRESS_FIXTURE = """
+import pytest
+from hypothesis import HealthCheck, given, settings, strategies as st
+
+@given(x=st.integers())
+def test_fails_health_check(capsys, x):
+    pass
+
+@settings(suppress_health_check=[HealthCheck.function_scoped_fixture])
+@given(x=st.integers())
+def test_suppresses_health_check(capsys, x):
+    pass
+
+@given(x=st.integers())
+@settings(suppress_health_check=[HealthCheck.function_scoped_fixture])
+def test_suppresses_health_check_2(capsys, x):
+    pass
+"""
+
+
+def test_suppress_health_check_function_scoped_fixture(testdir):
+    script = testdir.makepyfile(TESTSCRIPT_SUPPRESS_FIXTURE)
+    testdir.runpytest(script).assert_outcomes(passed=3)
+    testdir.runpytest(script, "-Werror").assert_outcomes(passed=2, failed=1)
 
 
 TESTSCRIPT_OVERRIDE_FIXTURE = """