Repository: HypothesisWorks/hypothesis

Issue:
Slow reduction for a buffer-building stateful test in v5.19.3
I am testing an incremental hashing function by comparing its results to a reference one-shot call in a stateful testing invariant. This lead to the following "bad shrinking" report with Hypothesis 5.19.3

```
WARNING: Hypothesis has spent more than five minutes working to shrink a failing example, and stopped because it is making very slow progress.  When you re-run your tests, shrinking will resume and may take this long before aborting again.
PLEASE REPORT THIS if you can provide a reproducing example, so that we can improve shrinking performance for everyone.
Falsifying example:
state = IncrementalUmash()
state.initialize(seed=1, which=0)
state.add_binary(data=b'')
state.add_binary(data=b'')
state.add_binary(data=b'')
state.add_binary(data=b'')
state.add_binary(data=b'')
state.add_binary(data=b'\x00')
state.add_binary(data=b'\x00')
state.add_binary(data=b'\x00')
state.add_binary(data=b'\x00')
state.add_binary(data=b'\x00\x00')
state.add_binary(data=b'\x00\x00')
state.add_binary(data=b'\x00\x00\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x01\x01\x01\x01\x01\x01')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x01\x01')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x00\x00\x00\x01')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x01\x01\x01\x01\x01\x01')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x00\x00\x01\x01')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x00\x00\x01\x01\x01\x01\x01\x01')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x00\x01\x01\x01\x01')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x01\x01\x01')
state.add_binary(data=b'\x00\x00\x00\x00\x01')
state.add_binary(data=b'\x00\x00\x00\x00\x01')
state.add_binary(data=b'\x00\x00\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00\x00\x00\x00')
state.add_binary(data=b'\x00\x00')
state.add_binary(data=b'\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00')
state.add_binary(data=b'\x00\x00\x00')
state.add_binary(data=b'\x00')
state.add_binary(data=b'\x00')
state.add_binary(data=b'\x00')
state.add_binary(data=b'\x00')
state.add_binary(data=b'\x01')
state.add_binary(data=b'\x01\x01\x01')
state.teardown()
Traceback (most recent call last):
  File "/home/pkhuong/backtrace/backtrace-backend/src/crdb/regressions/python/test_umash.py", line 230, in finish_matches
    )
AssertionError: assert ...
```

I reduced the erroneous system under test to the following Python code

```
class IncrementalUmash(RuleBasedStateMachine):
    def __init__(self):
        super().__init__()
        self.buf = b""
        self.state = None
        self.which = None
        self.seed = None

    @invariant()
    def finish_matches_full_hash(self):
        if self.state is None:
            return

        assert self.which == 0  # Unrelated failure that's easy to slide into
        assert self.seed == 0 or len(buf) <= 256  # The initial failing issue

    @initialize(
        which=st.integers(min_value=0, max_value=1),
        seed=st.integers(min_value=0, max_value=2 ** 64 - 1),
    )
    def initialize(self, which, seed):
        self.state = "Initialised"
        self.which = which
        self.seed = seed

    @precondition(lambda self: self.state is not None)
    @rule(data=st.binary())
    def add_binary(self, data):
        self.buf += data


TestIncrementalUmash = IncrementalUmash.TestCase
```

The failing example is "minimal" in that it sets seed to 1 and passes 257 bytes. However, it does so in a convoluted manner, and it's not obviously clear how to help reduction do something useful with this monoid-esque API.


Discussion:
I think there must be something missing from this example? `self.state` is never set anywhere.
As more general advice, this is probably a better fit for standard `@given()` tests than a state machine - compare e.g. [this test for incremental compression](https://github.com/Zac-HD/stdlib-property-tests/blob/a750727809fa2dc32206f3618c63414ed99e1775/tests/test_compression.py#L137-L148).
> As more general advice, this is probably a better fit for standard @given() tests than a state machine - compare e.g. this test for incremental compression.

Yeah, though it's kinda weird that this is the case here. Theoretically, stateful shrinking should be no different than given-based shrinking for this case - it's only bundles that will make it hard.
Also swarm testing!
Removed a little too much external noise while pasting the reduced tester. Fixed now.
I think the fact that it's stateful testing is a red herring and this is a general problem with the shrinker on examples that have to be quite large. The following similar test also takes ages to shrink:

```python
import hypothesis.strategies as st
from hypothesis import given, settings, target

@settings(max_examples=10**5)
@given(st.lists(st.binary()))
def test(ls):
    size = sum(map(len, ls))
    target(float(size))
    assert size <= 256
```
I think the best we might be able to do in this case is improve the stopping heuristics for the shrinker and not warn when we hit because of the new heuristic. The basic problem we run into here is that it's not that hard for us to find the optimal example, but it takes a *really* long time for us to verify that we can't shrink it further, and this is for intrinsic reasons: The final example is very large (by shrunk standards) and there are just a lot of possible transformations to it that don't work as a result.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
@@ -145,6 +145,13 @@ def __init__(
         # We ensure that the test has this much stack space remaining, no matter
         # the size of the stack when called, to de-flake RecursionErrors (#2494).
         self.__recursion_limit = sys.getrecursionlimit()
+        self.__pending_call_explanation = None
+
+    def explain_next_call_as(self, explanation):
+        self.__pending_call_explanation = explanation
+
+    def clear_call_explanation(self):
+        self.__pending_call_explanation = None
 
     @contextmanager
     def _log_phase_statistics(self, phase):
@@ -199,6 +206,10 @@ def __stoppable_test_function(self, data):
             sys.setrecursionlimit(self.__recursion_limit)
 
     def test_function(self, data):
+        if self.__pending_call_explanation is not None:
+            self.debug(self.__pending_call_explanation)
+            self.__pending_call_explanation = None
+
         assert isinstance(data.observer, TreeRecordingObserver)
         self.call_count += 1
 
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/shrinker.py b/hypothesis-python/src/hypothesis/internal/conjecture/shrinker.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/shrinker.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/shrinker.py
@@ -289,7 +289,13 @@ def __init__(self, engine, initial, predicate, allow_transition):
         self.update_shrink_target(initial)
         self.shrinks = 0
 
+        # We terminate shrinks that seem to have reached their logical
+        # conclusion: If we've called the underlying test function at
+        # least self.max_stall times since the last time we shrunk,
+        # it's time to stop shrinking.
+        self.max_stall = 200
         self.initial_calls = self.engine.call_count
+        self.calls_at_last_shrink = self.initial_calls
 
         self.passes_by_name = {}
         self.passes = []
@@ -308,12 +314,6 @@ def accept(f):
 
         return accept
 
-    def explain_next_call_as(self, explanation):
-        self.__pending_shrink_explanation = explanation
-
-    def clear_call_explanation(self):
-        self.__pending_shrink_explanation = None
-
     def add_new_pass(self, run):
         """Creates a shrink pass corresponding to calling ``run(self)``"""
 
@@ -388,13 +388,11 @@ def cached_test_function(self, buffer):
         too short to be a valid test case) or a ConjectureData object
         with status >= INVALID that would result from running this buffer."""
 
-        if self.__pending_shrink_explanation is not None:
-            self.debug(self.__pending_shrink_explanation)
-            self.__pending_shrink_explanation = None
-
         buffer = bytes(buffer)
         result = self.engine.cached_test_function(buffer)
         self.incorporate_test_data(result)
+        if self.calls - self.calls_at_last_shrink >= self.max_stall:
+            raise StopShrinking()
         return result
 
     def debug(self, msg):
@@ -425,6 +423,8 @@ def shrink(self):
 
         try:
             self.greedy_shrink()
+        except StopShrinking:
+            pass
         finally:
             if self.engine.report_debug_info:
 
@@ -534,6 +534,13 @@ def fixate_shrink_passes(self, passes):
             # try again once all of the passes have been run.
             can_discard = self.remove_discarded()
 
+            calls_at_loop_start = self.calls
+
+            # We keep track of how many calls can be made by a single step
+            # without making progress and use this to test how much to pad
+            # out self.max_stall by as we go along.
+            max_calls_per_failing_step = 1
+
             for sp in passes:
                 if can_discard:
                     can_discard = self.remove_discarded()
@@ -546,6 +553,20 @@ def fixate_shrink_passes(self, passes):
                 failures = 0
                 max_failures = 20
                 while failures < max_failures:
+                    # We don't allow more than max_stall consecutive failures
+                    # to shrink, but this means that if we're unlucky and the
+                    # shrink passes are in a bad order where only the ones at
+                    # the end are useful, if we're not careful this heuristic
+                    # might stop us before we've tried everything. In order to
+                    # avoid that happening, we make sure that there's always
+                    # plenty of breathing room to make it through a single
+                    # iteration of the fixate_shrink_passes loop.
+                    self.max_stall = max(
+                        self.max_stall,
+                        2 * max_calls_per_failing_step
+                        + (self.calls - calls_at_loop_start),
+                    )
+
                     prev = self.shrink_target
                     initial_calls = self.calls
                     # It's better for us to run shrink passes in a deterministic
@@ -572,6 +593,9 @@ def fixate_shrink_passes(self, passes):
                         if prev is not self.shrink_target:
                             failures = 0
                         else:
+                            max_calls_per_failing_step = max(
+                                max_calls_per_failing_step, self.calls - initial_calls
+                            )
                             failures += 1
 
                 # We reorder the shrink passes so that on our next run through
@@ -796,6 +820,16 @@ def update_shrink_target(self, new_target):
         assert isinstance(new_target, ConjectureResult)
         if self.shrink_target is not None:
             self.shrinks += 1
+            # If we are just taking a long time to shrink we don't want to
+            # trigger this heuristic, so whenever we shrink successfully
+            # we give ourselves a bit of breathing room to make sure we
+            # would find a shrink that took that long to find the next time.
+            # The case where we're taking a long time but making steady
+            # progress is handled by `finish_shrinking_deadline` in engine.py
+            self.max_stall = max(
+                self.max_stall, (self.calls - self.calls_at_last_shrink) * 2
+            )
+            self.calls_at_last_shrink = self.calls
         else:
             self.__all_changed_blocks = set()
             self.__last_checked_changed_at = new_target
@@ -1514,7 +1548,7 @@ def step(self, random_order=False):
         initial_shrinks = self.shrinker.shrinks
         initial_calls = self.shrinker.calls
         size = len(self.shrinker.shrink_target.buffer)
-        self.shrinker.explain_next_call_as(self.name)
+        self.shrinker.engine.explain_next_call_as(self.name)
 
         if random_order:
             selection_order = random_selection_order(self.shrinker.random)
@@ -1530,7 +1564,7 @@ def step(self, random_order=False):
             self.calls += self.shrinker.calls - initial_calls
             self.shrinks += self.shrinker.shrinks - initial_shrinks
             self.deletions += size - len(self.shrinker.shrink_target.buffer)
-            self.shrinker.clear_call_explanation()
+            self.shrinker.engine.clear_call_explanation()
         return True
 
     @property
@@ -1554,3 +1588,7 @@ def expand_region(f, a, b):
     b += find_integer(lambda k: f(a, b + k))
     a -= find_integer(lambda k: f(a - k, b))
     return (a, b)
+
+
+class StopShrinking(Exception):
+    pass


Test patch:
diff --git a/hypothesis-python/tests/conjecture/test_shrinker.py b/hypothesis-python/tests/conjecture/test_shrinker.py
--- a/hypothesis-python/tests/conjecture/test_shrinker.py
+++ b/hypothesis-python/tests/conjecture/test_shrinker.py
@@ -13,12 +13,19 @@
 #
 # END HEADER
 
+import time
+
 import pytest
 
 from hypothesis.internal.compat import int_to_bytes
 from hypothesis.internal.conjecture import floats as flt
 from hypothesis.internal.conjecture.engine import ConjectureRunner
-from hypothesis.internal.conjecture.shrinker import Shrinker, ShrinkPass, block_program
+from hypothesis.internal.conjecture.shrinker import (
+    Shrinker,
+    ShrinkPass,
+    StopShrinking,
+    block_program,
+)
 from hypothesis.internal.conjecture.shrinking import Float
 from hypothesis.internal.conjecture.utils import Sampler
 from tests.conjecture.common import SOME_LABEL, run_to_buffer, shrinking_from
@@ -570,3 +577,42 @@ def shrinker(data):
     sp = shrinker.shrink_pass("adaptive_example_deletion")
     assert isinstance(sp, ShrinkPass)
     assert shrinker.shrink_pass(sp) is sp
+
+
+def test_will_terminate_stalled_shrinks():
+    # Suppress the time based slow shrinking check - we only want
+    # the one that checks if we're in a stall where we've shrunk
+    # as far as we're going to get.
+    time.freeze()
+
+    @shrinking_from([255] * 100)
+    def shrinker(data):
+        count = 0
+
+        for _ in range(100):
+            if data.draw_bits(8) != 255:
+                count += 1
+                if count >= 10:
+                    return
+        data.mark_interesting()
+
+    shrinker.shrink()
+    assert shrinker.calls <= 1 + 2 * shrinker.max_stall
+
+
+def test_will_let_fixate_shrink_passes_do_a_full_run_through():
+    @shrinking_from(range(50))
+    def shrinker(data):
+        for i in range(50):
+            if data.draw_bits(8) != i:
+                data.mark_invalid()
+        data.mark_interesting()
+
+    shrinker.max_stall = 5
+
+    passes = [block_program("X" * i) for i in range(1, 11)]
+
+    with pytest.raises(StopShrinking):
+        shrinker.fixate_shrink_passes(passes)
+
+    assert shrinker.shrink_pass(passes[-1]).calls > 0
diff --git a/hypothesis-python/tests/cover/test_debug_information.py b/hypothesis-python/tests/cover/test_debug_information.py
--- a/hypothesis-python/tests/cover/test_debug_information.py
+++ b/hypothesis-python/tests/cover/test_debug_information.py
@@ -33,7 +33,7 @@ def test(i):
 
     value = out.getvalue()
 
-    assert "adaptive_example_deletion" in value
+    assert "minimize_individual_blocks" in value
     assert "calls" in value
     assert "shrinks" in value
 
diff --git a/hypothesis-python/tests/nocover/test_conjecture_engine.py b/hypothesis-python/tests/nocover/test_conjecture_engine.py
--- a/hypothesis-python/tests/nocover/test_conjecture_engine.py
+++ b/hypothesis-python/tests/nocover/test_conjecture_engine.py
@@ -138,6 +138,8 @@ def shrinker(data):
         Shrinker, "run_block_program", counts_calls(Shrinker.run_block_program)
     )
 
+    shrinker.max_stall = 500
+
     shrinker.fixate_shrink_passes([block_program("XX")])
 
     assert shrinker.shrinks == 0