Repository: HypothesisWorks/hypothesis

Issue:
Stack overflows are often reported as flaky.
As mentioned [here](https://stackoverflow.com/a/31770016/3654346), deterministic tests that can hit python's maximum recursion depth may be reported as flaky by Hypothesis, because it is attempted at different initial stack depths.

A simple example that Hypothesis claims is flaky:
```
from hypothesis import given, strategies

def slow(i):
    return 0 if i <= 0 else slow(i - 1) + 1

@given(strategies.integers(min_value=0))
def test_slow(i) -> None:
    assert slow(i) == i

test_slow()
```

Something of a question, rather than a bug report: would we consider detecting RecursionError and giving it some kind of special treatment?



Discussion:
Sure, I'd be happy to consider that as a feature.

I'm not sure what the best special handling might be for it though. The easiest thing to do might be always to just immediately re-raise `RecursionError` as soon as it's thrown rather than attempt to shrink and report it, but that's not necessarily great user experience either. We could potentially do something with fiddling around with getting and setting the recursion limit but that sounds a bit yikes. Any ideas?
Yeah, I considered the recursion limit fiddling too and agree it sounds scary. I guess I'd say that although re-raising RecursionError doesn't sound like the optimal behavior, it feels incrementally better?
When I saw this organically, I spent a fair amount of time trying to figure out whether my code was actually nondeterministic, so just suppressing that seems pretty helpful.

> Yeah, I considered the recursion limit fiddling too and agree it sounds scary.

I wonder if there's something relatively unintrusive we can do here, like setting the recursionlimit so that inside the test function we are always effectively at the top level. Something like setting it to the default recursion limit plus the current stack depth?

> When I saw this organically, I spent a fair amount of time trying to figure out whether my code was actually nondeterministic, so just suppressing that seems pretty helpful.

One light touch option might be to improve the error message for flakiness for `RecursionError`? 

Hmm. Potentially another problem here is that the way we differentiate exceptions is by type and line number. We might need to stop doing that for `RecursionError` because if the stack depth changes at all it might raise `RecursionError` on a different line number, which will cause Hypothesis to think it's a different exception.
Interesting. Indeed, perhaps the tools we need to reset the stack are there, and would work?:
```
cur_depth = len(inspect.stack()[0])
sys.setrecursionlimit(sys.getrecursionlimit() + cur_depth)
```
... and inside a context manager or something.
Sounds good to me - it's a terrible hack to fix a rare UX problem, but that's most of our internals :joy: 

I'd want to add a check on `__exit__` that the recursion limit is wherever we left it, and fail loudly if we're undoing a change made by user code... hopefully this will never actually trigger, but it'll save someone a lot of debugging time if it does.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
@@ -14,6 +14,7 @@
 # END HEADER
 
 import math
+import sys
 import time
 from collections import defaultdict
 from contextlib import contextmanager
@@ -40,7 +41,7 @@
     PreviouslyUnseenBehaviour,
     TreeRecordingObserver,
 )
-from hypothesis.internal.conjecture.junkdrawer import clamp
+from hypothesis.internal.conjecture.junkdrawer import clamp, stack_depth_of_caller
 from hypothesis.internal.conjecture.pareto import NO_SCORE, ParetoFront, ParetoOptimiser
 from hypothesis.internal.conjecture.shrinker import Shrinker, sort_key
 from hypothesis.internal.healthcheck import fail_health_check
@@ -133,6 +134,10 @@ def __init__(self, test_function, settings=None, random=None, database_key=None)
         # executed test case.
         self.__data_cache = LRUReusedCache(CACHE_SIZE)
 
+        # We ensure that the test has this much stack space remaining, no matter
+        # the size of the stack when called, to de-flake RecursionErrors (#2494).
+        self.__recursion_limit = sys.getrecursionlimit()
+
     @contextmanager
     def _log_phase_statistics(self, phase):
         self.stats_per_test_case.clear()
@@ -156,9 +161,21 @@ def __tree_is_exhausted(self):
 
     def __stoppable_test_function(self, data):
         """Run ``self._test_function``, but convert a ``StopTest`` exception
-        into a normal return.
+        into a normal return and avoid raising Flaky for RecursionErrors.
         """
+        depth = stack_depth_of_caller()
+        # Because we add to the recursion limit, to be good citizens we also add
+        # a check for unbounded recursion.  The default limit is 1000, so this can
+        # only ever trigger if something really strange is happening and it's hard
+        # to imagine an intentionally-deeply-recursive use of this code.
+        assert depth <= 1000, (
+            "Hypothesis would usually add %d to the stack depth of %d here, "
+            "but we are already much deeper than expected.  Aborting now, to "
+            "avoid extending the stack limit in an infinite loop..."
+            % (self.__recursion_limit, depth)
+        )
         try:
+            sys.setrecursionlimit(depth + self.__recursion_limit)
             self._test_function(data)
         except StopTest as e:
             if e.testcounter == data.testcounter:
@@ -170,6 +187,8 @@ def __stoppable_test_function(self, data):
                 # need to re-raise it so that it will eventually reach the
                 # correct engine.
                 raise
+        finally:
+            sys.setrecursionlimit(self.__recursion_limit)
 
     def test_function(self, data):
         assert isinstance(data.observer, TreeRecordingObserver)
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/junkdrawer.py b/hypothesis-python/src/hypothesis/internal/conjecture/junkdrawer.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/junkdrawer.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/junkdrawer.py
@@ -18,6 +18,7 @@
 anything that lives here, please move it."""
 
 import array
+import sys
 
 
 def array_or_list(code, contents):
@@ -216,3 +217,19 @@ def swap(ls, i, j):
     if i == j:
         return
     ls[i], ls[j] = ls[j], ls[i]
+
+
+def stack_depth_of_caller():
+    """Get stack size for caller's frame.
+
+    From https://stackoverflow.com/a/47956089/9297601 , this is a simple
+    but much faster alternative to `len(inspect.stack(0))`.  We use it
+    with get/set recursionlimit to make stack overflows non-flaky; see
+    https://github.com/HypothesisWorks/hypothesis/issues/2494 for details.
+    """
+    frame = sys._getframe(2)
+    size = 1
+    while frame:
+        frame = frame.f_back
+        size += 1
+    return size


Test patch:
diff --git a/hypothesis-python/tests/conjecture/test_engine.py b/hypothesis-python/tests/conjecture/test_engine.py
--- a/hypothesis-python/tests/conjecture/test_engine.py
+++ b/hypothesis-python/tests/conjecture/test_engine.py
@@ -155,6 +155,24 @@ def tf(data):
     assert count == [MIN_TEST_CALLS + 1]
 
 
+def recur(i, data):
+    try:
+        if i >= 1:
+            recur(i - 1, data)
+    except RecursionError:
+        data.mark_interesting()
+
+
+def test_recursion_error_is_not_flaky():
+    def tf(data):
+        i = data.draw_bits(16)
+        recur(i, data)
+
+    runner = ConjectureRunner(tf)
+    runner.run()
+    assert runner.exit_reason == ExitReason.finished
+
+
 def test_variadic_draw():
     def draw_list(data):
         result = []
diff --git a/hypothesis-python/tests/conjecture/test_junkdrawer.py b/hypothesis-python/tests/conjecture/test_junkdrawer.py
--- a/hypothesis-python/tests/conjecture/test_junkdrawer.py
+++ b/hypothesis-python/tests/conjecture/test_junkdrawer.py
@@ -13,6 +13,8 @@
 #
 # END HEADER
 
+import inspect
+
 import pytest
 
 from hypothesis import example, given, strategies as st
@@ -22,6 +24,7 @@
     binary_search,
     clamp,
     replace_all,
+    stack_depth_of_caller,
 )
 
 
@@ -147,3 +150,13 @@ def test_int_list_extend():
 def test_binary_search(n):
     i = binary_search(0, 100, lambda i: i <= n)
     assert i == n
+
+
+def recur(i):
+    assert len(inspect.stack(0)) == stack_depth_of_caller()
+    if i >= 1:
+        recur(i - 1)
+
+
+def test_stack_size_detection():
+    recur(100)