Repository: HypothesisWorks/hypothesis

Issue:
tests/cover/test_lookup.py::test_resolving_mutually_recursive_types{,_with_defaults} fail randomly when running the test suite w/ xdist
Well, you're not going to like this but I'm getting quasi-random test failures from the following two tests when running the large subset of the test suite in parallel:

```
FAILED tests/cover/test_lookup.py::test_resolving_mutually_recursive_types - RecursionError: maximum recursion depth exceeded
FAILED tests/cover/test_lookup.py::test_resolving_mutually_recursive_types_with_defaults - RecursionError: maximum recursion depth exceeded
```

This seems to be very flaky. It has been happening to us with PyPy3 before but I've assumed it's PyPy-specific. With 6.76.0, it suddenly started happening with CPython 3.11 as well. Curiously enough, I can reproduce it reliably when running a very specific set of tests:

```
python -m pytest -n 12 tests/cover tests/pytest tests/quality
```

but if I change tested files even a little, it stops happening.

Full log with tracebacks: [dev-python:hypothesis-6.76.0:20230606-074652.log](https://github.com/HypothesisWorks/hypothesis/files/11662376/dev-python.hypothesis-6.76.0.20230606-074652.log)

I realize this is going to be very hard to reproduce, and I'll be glad to try anything you may suggest while I still can.


Discussion:
I'm not able to reproduce — but that is hardly surprising, from your description.

I would be curious if it may be related to borderline recursion limits, if perhaps the tests are executed at different stack depths depending on the specific pytest-xdist workload. One way to rule this out would be to add f.x.

```
sys.setrecursionlimit(5000)
```
 near the top of `tests/cover/test_lookup.py`.
@jobh, indeed that seems to help. Could you add it or should I make a pull request?
(that said, I'm not 100% sure if it just doesn't change some other random factor but I'm happy enough that the issue disappears for now)
@mgorny I'm not sure if this is the right fix, but it certainly helps with narrowing down the problem.

On my local runs, a limit of 300 is sufficient (already a lot), and the default is 1000. I wonder what makes it grow so much.
Hmm, we do try to handle recursion-depth problems internally, but maybe need to repeat this trick somewhere else to handle strategy resolution?

https://github.com/HypothesisWorks/hypothesis/blob/fc566bf820a4b418a6cf54a15defbe9b44aaca5d/hypothesis-python/src/hypothesis/internal/conjecture/engine.py#L183-L197
@Zac-HD That was my concern, that this problem was in strategy resolution due to the new recursion guard not doing its job. But it seems that it's "just" the new test having deeper effective nesting than any of the old. I.e., it could happen without recursion, at roughly 30-deep strategies (but >100-deep to fail consistently).

The error happens in strategy validation, starting here:
https://github.com/HypothesisWorks/hypothesis/blob/fc566bf820a4b418a6cf54a15defbe9b44aaca5d/hypothesis-python/src/hypothesis/core.py#L619

...but I don't know if bumping the limit here will just move the problem elsewhere (draw time)?

To answer your previous question @mgorny , I don't have time to follow up further now I'm afraid.
... except possibly to add a skip to the two failing tests as a short term measure
> ... except possibly to add a skip to the two failing tests as a short term measure

We're already skipping them on our end via `--deselect`, so there's no need to do that.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -76,6 +76,7 @@
 )
 from hypothesis.internal.conjecture.data import ConjectureData, Status
 from hypothesis.internal.conjecture.engine import BUFFER_SIZE, ConjectureRunner
+from hypothesis.internal.conjecture.junkdrawer import ensure_free_stackframes
 from hypothesis.internal.conjecture.shrinker import sort_key
 from hypothesis.internal.entropy import deterministic_PRNG
 from hypothesis.internal.escalation import (
@@ -614,9 +615,10 @@ def process_arguments_to_given(wrapped_test, arguments, kwargs, given_kwargs, pa
 
     arguments = tuple(arguments)
 
-    for k, s in given_kwargs.items():
-        check_strategy(s, name=k)
-        s.validate()
+    with ensure_free_stackframes():
+        for k, s in given_kwargs.items():
+            check_strategy(s, name=k)
+            s.validate()
 
     stuff = Stuff(selfy=selfy, args=arguments, kwargs=kwargs, given_kwargs=given_kwargs)
 
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
@@ -9,7 +9,6 @@
 # obtain one at https://mozilla.org/MPL/2.0/.
 
 import math
-import sys
 import time
 from collections import defaultdict
 from contextlib import contextmanager
@@ -37,7 +36,7 @@
     PreviouslyUnseenBehaviour,
     TreeRecordingObserver,
 )
-from hypothesis.internal.conjecture.junkdrawer import clamp, stack_depth_of_caller
+from hypothesis.internal.conjecture.junkdrawer import clamp, ensure_free_stackframes
 from hypothesis.internal.conjecture.pareto import NO_SCORE, ParetoFront, ParetoOptimiser
 from hypothesis.internal.conjecture.shrinker import Shrinker, sort_key
 from hypothesis.internal.healthcheck import fail_health_check
@@ -133,9 +132,6 @@ def __init__(
         # executed test case.
         self.__data_cache = LRUReusedCache(CACHE_SIZE)
 
-        # We ensure that the test has this much stack space remaining, no matter
-        # the size of the stack when called, to de-flake RecursionErrors (#2494).
-        self.__recursion_limit = sys.getrecursionlimit()
         self.__pending_call_explanation = None
 
     def explain_next_call_as(self, explanation):
@@ -169,32 +165,22 @@ def __stoppable_test_function(self, data):
         """Run ``self._test_function``, but convert a ``StopTest`` exception
         into a normal return and avoid raising Flaky for RecursionErrors.
         """
-        depth = stack_depth_of_caller()
-        # Because we add to the recursion limit, to be good citizens we also add
-        # a check for unbounded recursion.  The default limit is 1000, so this can
-        # only ever trigger if something really strange is happening and it's hard
-        # to imagine an intentionally-deeply-recursive use of this code.
-        assert depth <= 1000, (
-            "Hypothesis would usually add %d to the stack depth of %d here, "
-            "but we are already much deeper than expected.  Aborting now, to "
-            "avoid extending the stack limit in an infinite loop..."
-            % (self.__recursion_limit, depth)
-        )
-        try:
-            sys.setrecursionlimit(depth + self.__recursion_limit)
-            self._test_function(data)
-        except StopTest as e:
-            if e.testcounter == data.testcounter:
-                # This StopTest has successfully stopped its test, and can now
-                # be discarded.
-                pass
-            else:
-                # This StopTest was raised by a different ConjectureData. We
-                # need to re-raise it so that it will eventually reach the
-                # correct engine.
-                raise
-        finally:
-            sys.setrecursionlimit(self.__recursion_limit)
+        # We ensure that the test has this much stack space remaining, no
+        # matter the size of the stack when called, to de-flake RecursionErrors
+        # (#2494, #3671).
+        with ensure_free_stackframes():
+            try:
+                self._test_function(data)
+            except StopTest as e:
+                if e.testcounter == data.testcounter:
+                    # This StopTest has successfully stopped its test, and can now
+                    # be discarded.
+                    pass
+                else:
+                    # This StopTest was raised by a different ConjectureData. We
+                    # need to re-raise it so that it will eventually reach the
+                    # correct engine.
+                    raise
 
     def test_function(self, data):
         if self.__pending_call_explanation is not None:
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/junkdrawer.py b/hypothesis-python/src/hypothesis/internal/conjecture/junkdrawer.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/junkdrawer.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/junkdrawer.py
@@ -14,6 +14,7 @@
 
 import array
 import sys
+import warnings
 from random import Random
 from typing import (
     Callable,
@@ -30,6 +31,8 @@
     overload,
 )
 
+from hypothesis.errors import HypothesisWarning
+
 ARRAY_CODES = ["B", "H", "I", "L", "Q", "O"]
 
 
@@ -271,6 +274,42 @@ def stack_depth_of_caller() -> int:
     return size
 
 
+class ensure_free_stackframes:
+    """Context manager that ensures there are at least N free stackframes. The
+    value of N is chosen to be the recursion limit at import time, or at least
+    500.
+    """
+
+    initial_maxdepth: int = sys.getrecursionlimit()
+
+    def __enter__(self):
+        cur_depth = stack_depth_of_caller()
+        self.old_maxdepth = sys.getrecursionlimit()
+        self.new_maxdepth = cur_depth + max(self.initial_maxdepth, 500)
+        # Because we add to the recursion limit, to be good citizens we
+        # also add a check for unbounded recursion.  The default limit
+        # is 1000, so this can only ever trigger if something really
+        # strange is happening and it's hard to imagine an
+        # intentionally-deeply-recursive use of this code.
+        assert cur_depth <= self.initial_maxdepth, (
+            "Hypothesis would usually add %d to the stack depth of %d here, "
+            "but we are already much deeper than expected.  Aborting now, to "
+            "avoid extending the stack limit in an infinite loop..."
+            % (self.new_maxdepth - self.old_maxdepth, self.old_maxdepth)
+        )
+        sys.setrecursionlimit(self.new_maxdepth)
+
+    def __exit__(self, *args, **kwargs):
+        if self.new_maxdepth == sys.getrecursionlimit():
+            sys.setrecursionlimit(self.old_maxdepth)
+        else:  # pragma: no cover
+            warnings.warn(
+                "The recursion limit will not be reset, since it was changed "
+                "from another thread or during execution of a test.",
+                HypothesisWarning,
+            )
+
+
 def find_integer(f: Callable[[int], bool]) -> int:
     """Finds a (hopefully large) integer such that f(n) is True and f(n + 1) is
     False.


Test patch:
diff --git a/hypothesis-python/tests/cover/test_lookup.py b/hypothesis-python/tests/cover/test_lookup.py
--- a/hypothesis-python/tests/cover/test_lookup.py
+++ b/hypothesis-python/tests/cover/test_lookup.py
@@ -29,6 +29,7 @@
 from hypothesis import HealthCheck, assume, given, settings, strategies as st
 from hypothesis.errors import InvalidArgument, ResolutionFailed, SmallSearchSpaceWarning
 from hypothesis.internal.compat import PYPY, get_origin, get_type_hints
+from hypothesis.internal.conjecture.junkdrawer import stack_depth_of_caller
 from hypothesis.internal.reflection import get_pretty_function_description
 from hypothesis.strategies import from_type
 from hypothesis.strategies._internal import types
@@ -602,10 +603,6 @@ def __repr__(self):
         return f"B({self.nxt})"
 
 
-@pytest.mark.skipif(
-    PYPY and sys.version_info[:2] < (3, 9),
-    reason="mysterious failure on pypy/python<3.9",
-)
 @given(nxt=st.from_type(A))
 def test_resolving_mutually_recursive_types(nxt):
     i = 0
@@ -615,6 +612,22 @@ def test_resolving_mutually_recursive_types(nxt):
         i += 1
 
 
+def test_resolving_mutually_recursive_types_with_limited_stack():
+    orig_recursionlimit = sys.getrecursionlimit()
+    current_stack_depth = stack_depth_of_caller()
+    sys.setrecursionlimit(current_stack_depth + 100)
+    try:
+
+        @given(nxt=st.from_type(A))
+        def test(nxt):
+            pass
+
+        test()
+    finally:
+        assert sys.getrecursionlimit() == current_stack_depth + 100
+        sys.setrecursionlimit(orig_recursionlimit)
+
+
 class A_with_default:
     def __init__(self, nxt: typing.Optional["B_with_default"] = None):
         self.nxt = nxt
diff --git a/hypothesis-python/tests/nocover/test_recursive.py b/hypothesis-python/tests/nocover/test_recursive.py
--- a/hypothesis-python/tests/nocover/test_recursive.py
+++ b/hypothesis-python/tests/nocover/test_recursive.py
@@ -8,7 +8,9 @@
 # v. 2.0. If a copy of the MPL was not distributed with this file, You can
 # obtain one at https://mozilla.org/MPL/2.0/.
 
+import sys
 import threading
+import warnings
 
 from hypothesis import HealthCheck, given, settings, strategies as st
 
@@ -140,12 +142,18 @@ def test_drawing_from_recursive_strategy_is_thread_safe():
     @given(data=st.data())
     def test(data):
         try:
-            data.draw(shared_strategy)
+            # We may get a warning here about not resetting recursionlimit,
+            # since it was changed during execution; ignore it.
+            with warnings.catch_warnings():
+                warnings.simplefilter("ignore")
+                data.draw(shared_strategy)
         except Exception as exc:
             errors.append(exc)
 
     threads = []
 
+    original_recursionlimit = sys.getrecursionlimit()
+
     for _ in range(4):
         threads.append(threading.Thread(target=test))
 
@@ -155,6 +163,10 @@ def test(data):
     for thread in threads:
         thread.join()
 
+    # Cleanup: reset the recursion limit that was (probably) not reset
+    # automatically in the threaded test.
+    sys.setrecursionlimit(original_recursionlimit)
+
     assert not errors