Repository: HypothesisWorks/hypothesis

Issue:
Random failures with hypothesis.errors.StopTest exception raised
I have a rather simple test case:
https://github.com/tlsfuzzer/tlslite-ng/blob/b6dd1c2bd3b3dc05bf1d794344ad4c2e4a0e20d9/unit_tests/test_tlslite_utils_cryptomath.py#L175-L184
```python
    @given(integers(min_value=0, max_value=1<<16384))
    @example(0)
    @example(255)
    @example(256)
    @example((1<<1024)-1)
    @example((1<<521)-1)
    @example(1<<8192)
    @example((1<<8192)-1)
    def test_numBits(self, number):
        self.assertEqual(numBits(number), self.num_bits(number))
```

It's randomly failing:
```
ERROR: test_numBits (unit_tests.test_tlslite_utils_cryptomath.TestNumBits.test_numBits)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/tlslite-ng/tlslite-ng/unit_tests/test_tlslite_utils_cryptomath.py", line 176, in test_numBits
    @example(0)
               ^
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/core.py", line 1602, in wrapped_test
    raise the_error_hypothesis_found
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/core.py", line 1565, in wrapped_test
    state.run_engine()
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/core.py", line 1097, in run_engine
    runner.run()
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/internal/conjecture/engine.py", line 496, in run
    self._run()
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/internal/conjecture/engine.py", line 905, in _run
    self.generate_new_examples()
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/internal/conjecture/engine.py", line 699, in generate_new_examples
    prefix = self.generate_novel_prefix()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/internal/conjecture/engine.py", line 352, in generate_novel_prefix
    return self.tree.generate_novel_prefix(self.random)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/internal/conjecture/datatree.py", line 672, in generate_novel_prefix
    (v, buf) = self._draw(
               ^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/internal/conjecture/datatree.py", line 770, in _draw
    value = draw_func(**kwargs, forced=forced)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/internal/conjecture/data.py", line 1565, in draw_integer
    value = self.provider.draw_integer(**kwargs, forced=forced)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/internal/conjecture/data.py", line 1087, in draw_integer
    return self._draw_bounded_integer(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/internal/conjecture/data.py", line 1312, in _draw_bounded_integer
    probe = self._cd.draw_bits(
            ^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/internal/conjecture/data.py", line 1863, in draw_bits
    self.__check_capacity(n_bytes)
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/internal/conjecture/data.py", line 1905, in __check_capacity
    self.mark_overrun()
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/internal/conjecture/data.py", line 1930, in mark_overrun
    self.conclude_test(Status.OVERRUN)
  File "/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/hypothesis/internal/conjecture/data.py", line 1917, in conclude_test
    raise StopTest(self.testcounter)
hypothesis.errors.StopTest: 3178

----------------------------------------------------------------------
Ran 1686 tests in 18.615s

FAILED (errors=1, skipped=32, expected failures=2)
You can add @seed(63542525054868526981564917001292471200) to this test to reproduce this failure.
```

This is with hypothesis-6.98.2 on python 3.11.7 (main, Dec  5 2023, 08:47:01) [GCC 11.4.0]

But I've seen it failing on Pythons from 3.8 up.

It's failing once about every 20-30 executions.


Discussion:
Thanks for the report! I have no doubt this regressed in https://github.com/HypothesisWorks/hypothesis/pull/3818. I'll take a look later today.
Hmm. This is an overrun during `generate_novel_prefix`, due to the possibility that drawing a *single* node during novel generation takes more than `BUFFER_SIZE` in extreme cases. In the case above, this is a combination of unlucky successive probes + large max integer bounds (1500 bytes per probe).

One solution is retrying these overruns in `generate_novel_prefix` via a try/catch of `StopTest`.  I'm nervous this will end in infinite loops or severe slowdowns in extreme cases, but I can't think of something that would repeatedly trigger this behavior in `generate_novel_prefix` while avoiding `HealthCheck.too_large`, so maybe this would be ok? (thinking aloud: what about a case where every novel prefix is > BUFFER_SIZE, so we hit this infinite loop before reaching enough overruns to raise the health check?)
(yeah, yeah, I know some of those words.jpg)

Unfortunately I'm not able to help with this. What I can say, that I have some [other non-trivial](https://github.com/tlsfuzzer/python-ecdsa/blob/0d5a38ca8ada5f2fec67904236f563f562b402d4/src/ecdsa/test_malformed_sigs.py#L286) usages of hypothesis, ones that are complex enough that sometimes they time-out with old versions; so they may be good to verify sanity of complex test generation.
@tybug I still have regular failures because of this. Any chance of this getting fixed soon?
I also run into this recently. While I cannot offer any asistance in fixing it, I have a workaround: using `flaky` and marking the test as being flaky, re-running it if it fails with `StopTest` exception. So far re-running the test once is usually enough for me to not trigger this bug.
with such level of workarounding, I'd rather just mark it as skipped, and point to this bug
Sorry, I know this is a pretty frustrating bug to run into. I'll look into this within the week.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/datatree.py b/hypothesis-python/src/hypothesis/internal/conjecture/datatree.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/datatree.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/datatree.py
@@ -755,7 +755,14 @@ def append_buf(buf):
                     attempts = 0
                     while True:
                         if attempts <= 10:
-                            (v, buf) = self._draw(ir_type, kwargs, random=random)
+                            try:
+                                (v, buf) = self._draw(ir_type, kwargs, random=random)
+                            except StopTest:  # pragma: no cover
+                                # it is possible that drawing from a fresh data can
+                                # overrun BUFFER_SIZE, due to eg unlucky rejection sampling
+                                # of integer probes. Retry these cases.
+                                attempts += 1
+                                continue
                         else:
                             (v, buf) = self._draw_from_cache(
                                 ir_type, kwargs, key=id(current_node), random=random
@@ -781,9 +788,13 @@ def append_buf(buf):
                 attempts = 0
                 while True:
                     if attempts <= 10:
-                        (v, buf) = self._draw(
-                            branch.ir_type, branch.kwargs, random=random
-                        )
+                        try:
+                            (v, buf) = self._draw(
+                                branch.ir_type, branch.kwargs, random=random
+                            )
+                        except StopTest:  # pragma: no cover
+                            attempts += 1
+                            continue
                     else:
                         (v, buf) = self._draw_from_cache(
                             branch.ir_type, branch.kwargs, key=id(branch), random=random


Test patch:
diff --git a/hypothesis-python/tests/conjecture/test_engine.py b/hypothesis-python/tests/conjecture/test_engine.py
--- a/hypothesis-python/tests/conjecture/test_engine.py
+++ b/hypothesis-python/tests/conjecture/test_engine.py
@@ -1626,3 +1626,38 @@ def test(cd):
 
     runner.cached_test_function_ir([node] if forced_first else [forced_node])
     assert runner.call_count == 1
+
+
+def test_simulate_to_evicted_data(monkeypatch):
+    # test that we do not rely on the false invariant that correctly simulating
+    # a data to a result means we have that result in the cache, due to e.g.
+    # cache evictions (but also potentially other trickery).
+    monkeypatch.setattr(engine_module, "CACHE_SIZE", 1)
+
+    node_0 = IRNode(
+        ir_type="integer",
+        value=0,
+        kwargs={
+            "min_value": None,
+            "max_value": None,
+            "weights": None,
+            "shrink_towards": 0,
+        },
+        was_forced=False,
+    )
+    node_1 = node_0.copy(with_value=1)
+
+    def test(data):
+        data.draw_integer()
+
+    runner = ConjectureRunner(test)
+    runner.cached_test_function_ir([node_0])
+    # cache size is 1 so this evicts node_0
+    runner.cached_test_function_ir([node_1])
+    assert runner.call_count == 2
+
+    # we dont throw PreviouslyUnseenBehavior when simulating, but the result
+    # was evicted to the cache so we will still call through to the test function.
+    runner.tree.simulate_test_function(ConjectureData.for_ir_tree([node_0]))
+    runner.cached_test_function_ir([node_0])
+    assert runner.call_count == 3
diff --git a/hypothesis-python/tests/nocover/test_regressions.py b/hypothesis-python/tests/nocover/test_regressions.py
--- a/hypothesis-python/tests/nocover/test_regressions.py
+++ b/hypothesis-python/tests/nocover/test_regressions.py
@@ -46,7 +46,6 @@ def test_performance_issue_2027(x, y):
     pass
 
 
-# if this test ever fails with a flaky error, #3926 has regressed.
 @given(
     st.lists(
         st.floats(allow_infinity=False),
@@ -55,3 +54,10 @@ def test_performance_issue_2027(x, y):
 )
 def test_unique_floats_with_nan_is_not_flaky_3926(ls):
     pass
+
+
+# this will take a while to find the regression, but will eventually trigger it.
+# min_value=0 is critical to trigger the probing behavior which exhausts our buffer.
+@given(st.integers(min_value=0, max_value=1 << 25_000))
+def test_overrun_during_datatree_simulation_3874(n):
+    pass