Repository: HypothesisWorks/hypothesis

Issue:
Stuck in a loop forever when shrinking with targeted property-based testing
Hi, thanks for the awesome library! I'm trying to wrap by head around `hypothesis.target` so I tried playing with an artificial example. The test generates floats smaller than a threshold value and it's expected to fail when the float is close enough to the magic value 42.5. Targeting works fine here as in many cases it can find the problematic value when non-targeted generation cannot. 

However, sometimes it gets stuck in a loop forever. Here's an example that gets stuck in my case:

```python
from hypothesis import given, target, settings, Phase, Verbosity, seed
import hypothesis.strategies as some
import math

@given(d=some.floats().filter(lambda x: abs(x) < 1000))
@settings(
    max_examples=1000,
    verbosity=Verbosity.debug,
    phases=[
        Phase.generate,
        Phase.target,
        Phase.shrink,
    ],  # Skip Phase.reuse to not run previous counter-examples
)
@seed(93962505385993024185959759429298090872)
def test_targeting_square_loss(d):
    """
    Contrived example of targeting properties.
    Disabling Phase.target or removing `target(-loss)` will most likely make the test pass.
    """
    # Assume this value triggers a bug
    target_value = 42.5

    should_fail = abs(d - target_value) < 0.5

    if should_fail:
        print("Failing with value {}".format(d))
        raise Exception("Critically close to {}, got {}".format(target_value, d))

    # Target the value
    loss = math.pow((d - target_value), 2.0)
    target(-loss)
```

The verbose printing keeps printing lines such as:

```

---------------------
Shrink pass profiling
---------------------

Shrinking made a total of 0 calls of which 2 shrank. This deleted 0 bytes out of 11.

Useful passes:


Useless passes:


block_program('XXX')
...
```

I'm using Hypothesis 5.8.3 and running the test in pytest. I know the test is contrived, but I'm mostly wondering why it gets stuck.

EDIT: I'm exaggerating that it would be stuck forever, I stopped waiting after a minute or so :)


Discussion:
Thanks for the bug report! I've got a hunch as to what might be going on here - we've got a bit at the end which uses shrinking to try to find minimized versions of each target score, and I suspect it's getting over eager in terms of how much work it's doing. I'll take a look into this today.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/pareto.py b/hypothesis-python/src/hypothesis/internal/conjecture/pareto.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/pareto.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/pareto.py
@@ -298,6 +298,7 @@ def run(self):
         # we run the tests we improve the pareto front because we work on the
         # bits that we haven't covered yet.
         i = len(self.front) - 1
+        prev = None
         while i >= 0:
             assert self.front
             i = min(i, len(self.front) - 1)
@@ -305,20 +306,37 @@ def run(self):
             if target.buffer in seen:
                 i -= 1
                 continue
-
-            # Note that during shrinking we may discover other smaller examples
-            # that will get added to the front. It's OK to slip and and out of
-            # these - in particular we may move to new shrink targets that are
-            # not dominating the current target. This is fine, because they
-            # will be added to the front and processed on later iterations of
-            # this loop.
-            shrunk = self.__engine.shrink(
-                target,
-                lambda data: data.status >= Status.VALID
-                and dominance(data, target)
-                in (DominanceRelation.EQUAL, DominanceRelation.LEFT_DOMINATES),
-            )
-            seen.add(shrunk.buffer)
+            assert target is not prev
+            prev = target
+
+            def shrink_to(data):
+                """Shrink to data that strictly pareto dominates the current
+                best value we've seen, which is the current target of the
+                shrinker.
+
+                Note that during shrinking we may discover other smaller
+                examples that this function will reject and will get added to
+                the front. This is fine, because they will be processed on
+                later iterations of this loop."""
+                if data.status < Status.VALID:
+                    return False
+                current = shrinker.shrink_target
+                dom = dominance(data, current)
+                if dom == DominanceRelation.LEFT_DOMINATES:
+                    # If ``data`` dominates ``current`` then ``current``
+                    # must be dominated in the front - either ``data`` is in
+                    # the front, or it was not added to it because it was
+                    # dominated by something in it.,
+                    try:
+                        self.front.front.remove(current)
+                    except ValueError:
+                        pass
+                    return True
+                return False
+
+            shrinker = self.__engine.new_shrinker(target, shrink_to)
+            shrinker.shrink()
+            seen.add(shrinker.shrink_target.buffer)
 
             # Note that the front may have changed shape arbitrarily when
             # we ran the shrinker. If it didn't change shape then this is


Test patch:
diff --git a/hypothesis-python/tests/cover/test_targeting.py b/hypothesis-python/tests/cover/test_targeting.py
--- a/hypothesis-python/tests/cover/test_targeting.py
+++ b/hypothesis-python/tests/cover/test_targeting.py
@@ -144,3 +144,13 @@ def test(ls):
         return result[0]
 
     assert score(enabled=True) > score(enabled=False)
+
+
+def test_issue_2395_regression():
+    @given(d=st.floats().filter(lambda x: abs(x) < 1000))
+    @settings(max_examples=1000, database=None)
+    @seed(93962505385993024185959759429298090872)
+    def test_targeting_square_loss(d):
+        target(-((d - 42.5) ** 2.0))
+
+    test_targeting_square_loss()