Repository: HypothesisWorks/hypothesis

Issue:
Handle non-equality constraints when performing simultaneous shrinking
Note: This is part of #1093, where I encourage people to work on the shrinker. As such I will not be working on it myself and encourage you to take a look at it if it takes your fancy.

Currently Hypothesis has a special case in its shrinker for duplicate primitive values:

```pycon
>>> b = st.integers(0, 255)
>>> find(st.tuples(b, b), lambda x: x[0] == x[1] > 0)
(1, 1)
```

This works by spotting that the underlying byte representation is the same and shrinking them together. This approach fails as soon as the constraint between the two values is not an equality one:

```pycon
>>> b = st.integers(0, 255)
find(st.tuples(b, b), lambda x: x[0] == x[1] + 1)
>>> (22, 21)
``` 

(the optimal answer for this is of course `(1, 0)`)

This is actually the underlying source of literally our only real world shrink quality issue that is currently open, #359, so fixing this bug gets you extra kudos.

Proposed solution:

We adapt the solution from https://github.com/HypothesisWorks/hypothesis-python/pull/1082 that removes an offset from multiple blocks simultaneously to other subsets of the indices.

The shrinker has a notion of "expensive shrink passes" that get enabled once the normal ones stop being enough to make progress. Once these start running the input size is small and most bytes are zero. In particular, it's OK for them to be quadratic in the size of the input.

It might make sense to write a quadratic cost shrink pass that takes every pair of non-zero blocks (probably only ones that are not shrinking blocks? Changing structural bytes this way is *probably* useless and definitely expensive) and applies a similar technique to that pair.
Possibly non-optimal shrink when generating datetimes()
Here’s a test I wrote for the python-dateutil library:

``` python
#!/usr/bin/env python
# -*- encoding: utf-8 -*-

from hypothesis.extra.datetime import datetimes
from hypothesis import given

from dateutil.parser import parse

@given(datetimes())
def test_round_trip(d):
    """Cast a datetime() to a string, then unpack with dateutil. Check it matches."""
    assert parse(str(d)) == d
```

This test eventually fails. When it first failed, Hypothesis presented the following counterexample:

```
Falsifying example: test_round_trip(d=datetime.datetime(5, 5, 1, 0, 0, tzinfo=<UTC>)) [*]
```

I ran the test a dozen more times, and it continued to fail with the same example (thanks, examples database!). But in fact, any date with the same month/year value, but distinct day value, hits the same failure:

```
datetime.datetime(2, 2, 1, 0, 0, tzinfo=<UTC>)
datetime.datetime(3, 3, 1, 0, 0, tzinfo=<UTC>)
...
datetime.datetime(12, 12, 1, 0, 0, tzinfo=<UTC>)
```

To my eye, the first example is “simpler” than the example it actually printed. Smaller years and months are better, no?

I’m not totally surprised Hypothesis didn’t find the failure immediately – it’s quite an unusual case, and it took multiple runs to find. I’m more surprised that on all the subsequent runs, I got the same failure `[*]`. There isn’t much further it can shrink downwards without going into negative years, but it never seems to even try. In fact, most of the examples it tries on subsequent runs seem to be _more_ complicated, if anything.

Not sure if this is actually bug – but it seems weird to me.

<details>
<summary>

Examples tried on a subsequent run</summary>



```
-------------------------------------------- Captured stdout call ---------------------------------------------
0003-03-01 00:00:00+00:00
0003-03-01 00:00:00+00:00
2000-03-01 00:00:00+00:00
0003-01-01 00:00:00+00:00
2257-03-01 00:00:00+00:00
2514-03-01 00:00:00+00:00
2771-03-01 00:00:00+00:00
3028-03-01 00:00:00+00:00
3285-03-01 00:00:00+00:00
3542-03-01 00:00:00+00:00
3799-03-01 00:00:00+00:00
4056-03-01 00:00:00+00:00
4313-03-01 00:00:00+00:00
4570-03-01 00:00:00+00:00
4827-03-01 00:00:00+00:00
5084-03-01 00:00:00+00:00
5340-03-01 00:00:00+00:00
5596-03-01 00:00:00+00:00
5852-03-01 00:00:00+00:00
6108-03-01 00:00:00+00:00
6364-03-01 00:00:00+00:00
6620-03-01 00:00:00+00:00
6876-03-01 00:00:00+00:00
7132-03-01 00:00:00+00:00
7388-03-01 00:00:00+00:00
7644-03-01 00:00:00+00:00
7900-03-01 00:00:00+00:00
8156-03-01 00:00:00+00:00
8412-03-01 00:00:00+00:00
8668-03-01 00:00:00+00:00
8924-03-01 00:00:00+00:00
9180-03-01 00:00:00+00:00
9436-03-01 00:00:00+00:00
9692-03-01 00:00:00+00:00
9948-03-01 00:00:00+00:00
1795-03-01 00:00:00+00:00
1539-03-01 00:00:00+00:00
1283-03-01 00:00:00+00:00
1027-03-01 00:00:00+00:00
0771-03-01 00:00:00+00:00
0515-03-01 00:00:00+00:00
0259-03-01 00:00:00+00:00
2012-03-01 00:00:00+00:00
2255-03-01 00:00:00+00:00
3804-03-01 00:00:00+00:00
4047-03-01 00:00:00+00:00
0784-03-01 00:00:00+00:00
0272-03-01 00:00:00+00:00
0016-03-01 00:00:00+00:00
0015-03-01 00:00:00+00:00
0011-03-01 00:00:00+00:00
0007-03-01 00:00:00+00:00
0004-03-01 00:00:00+00:00
0003-02-01 00:00:00+00:00
0003-03-01 00:00:00+00:00
------------------------------------------------- Hypothesis --------------------------------------------------
Falsifying example: test_round_trio(d=datetime.datetime(3, 3, 1, 0, 0, tzinfo=<UTC>))
```

</details>



Discussion:

Associated dateutil issue: https://github.com/dateutil/dateutil/issues/293

Hmm. That's an interesting example.

Without debugging this in depth I think what's happening is that the bug is only triggered if the month and year are the same value. For a couple reasons (different offsets and word sizes for the underlying generated integer ranges), Hypothesis's heuristics for simultaneous shrinking don't fire because it can't tell that these values are the same.

I'll leave this open, but it's a tricky problem to solve. Or rather it's an easy problem to solve if I don't mind every shrink hitting the timeout of 60 seconds and a hard problem to solve without that. :-)

The examples tried on a subsequent run thing is weird though. I don't know what's up with that. I'm not sure if it's a bug either, but it's probably not - often intermediate shrinks don't look entirely like shrinks due to the way the data is transformed from the underlying representation, but the final shrinks tend to work well.


Resolving patch:
diff --git a/src/hypothesis/internal/conjecture/engine.py b/src/hypothesis/internal/conjecture/engine.py
--- a/src/hypothesis/internal/conjecture/engine.py
+++ b/src/hypothesis/internal/conjecture/engine.py
@@ -1320,6 +1320,7 @@ def greedy_shrink(self):
             if not run_expensive_shrinks:
                 continue
 
+            self.shrink_offset_pairs()
             self.interval_deletion_with_block_lowering()
             self.pass_to_interval()
             self.reorder_bytes()
@@ -1527,6 +1528,68 @@ def reoffset(o):
 
         minimize_int(offset, reoffset)
 
+    def shrink_offset_pairs(self):
+        """Lower any two blocks offset from each other the same ammount.
+
+        Before this shrink pass, two blocks explicitly offset from each
+        other would not get minimized properly:
+         >>> b = st.integers(0, 255)
+         >>> find(st.tuples(b, b), lambda x: x[0] == x[1] + 1)
+        (149,148)
+
+        This expensive (O(n^2)) pass goes through every pair of non-zero
+        blocks in the current shrink target and sees if the shrink
+        target can be improved by applying an offset to both of them.
+        """
+        self.debug('Shrinking offset pairs.')
+
+        current = [self.shrink_target.buffer[u:v] for u, v in self.blocks]
+
+        def int_from_block(i):
+            return int_from_bytes(current[i])
+
+        def block_len(i):
+            u, v = self.blocks[i]
+            return v - u
+
+        # Try reoffseting every pair
+        def reoffset_pair(pair, o):
+            n = len(self.blocks)
+            # Number of blocks may have changed, need to validate
+            valid_pair = [p for p in pair if p < n and int_from_block(p) > 0]
+
+            if len(valid_pair) < 2:
+                return
+
+            m = min([int_from_block(p) for p in valid_pair])
+
+            new_blocks = [self.shrink_target.buffer[u:v]
+                          for u, v in self.blocks]
+            for i in valid_pair:
+                new_blocks[i] = int_to_bytes(
+                    int_from_block(i) + o - m, block_len(i))
+            buffer = hbytes().join(new_blocks)
+            return self.incorporate_new_buffer(buffer)
+
+        i = 0
+        while i < len(self.blocks):
+            if not self.is_shrinking_block(i) and int_from_block(i) > 0:
+                j = i + 1
+                while j < len(self.shrink_target.blocks):
+                    block_val = int_from_block(j)
+                    i_block_val = int_from_block(i)
+                    if not self.is_shrinking_block(j) \
+                       and block_val > 0 and i_block_val > 0:
+                        offset = min(int_from_block(i),
+                                     int_from_block(j))
+                        # Save current before shrinking
+                        current = [self.shrink_target.buffer[u:v]
+                                   for u, v in self.blocks]
+                        minimize_int(
+                            offset, lambda o: reoffset_pair((i, j), o))
+                    j += 1
+            i += 1
+
     def mark_shrinking(self, blocks):
         """Mark each of these blocks as a shrinking block: That is, lowering
         its value lexicographically may cause less data to be drawn after."""


Test patch:
diff --git a/tests/cover/test_conjecture_engine.py b/tests/cover/test_conjecture_engine.py
--- a/tests/cover/test_conjecture_engine.py
+++ b/tests/cover/test_conjecture_engine.py
@@ -1243,6 +1243,78 @@ def x(data):
     assert x == hbytes([0, 0, 2])
 
 
+def test_shrinking_block_pairs(monkeypatch):
+    monkeypatch.setattr(
+        Shrinker, 'shrink', lambda self: (
+            self.shrink_offset_pairs()
+        )
+    )
+
+    monkeypatch.setattr(
+        ConjectureRunner, 'generate_new_examples',
+        lambda runner: runner.test_function(
+            ConjectureData.for_buffer([12, 10])))
+
+    @run_to_buffer
+    def x(data):
+        m = data.draw_bits(8)
+        n = data.draw_bits(8)
+        if m == n + 2:
+            data.mark_interesting()
+    assert x == hbytes([2, 0])
+
+
+def test_non_minimal_pair_shrink(monkeypatch):
+    monkeypatch.setattr(
+        Shrinker, 'shrink', lambda self: (
+            self.shrink_offset_pairs()
+        )
+    )
+
+    monkeypatch.setattr(
+        ConjectureRunner, 'generate_new_examples',
+        lambda runner: runner.test_function(
+            ConjectureData.for_buffer([12, 10])))
+
+    @run_to_buffer
+    def x(data):
+        m = data.draw_bits(8)
+        if m < 5:
+            data.mark_invalid()
+        if m == 5:
+            data.mark_interesting()
+        n = data.draw_bits(8)
+        if m == n + 2:
+            data.mark_interesting()
+    assert x == hbytes([5])
+
+
+def test_buffer_changes_during_pair_shrink(monkeypatch):
+    monkeypatch.setattr(
+        Shrinker, 'shrink', lambda self: (
+            self.shrink_offset_pairs()
+        )
+    )
+
+    monkeypatch.setattr(
+        ConjectureRunner, 'generate_new_examples',
+        lambda runner: runner.test_function(
+            ConjectureData.for_buffer([12, 10])))
+
+    @run_to_buffer
+    def x(data):
+        m = data.draw_bits(8)
+        if m < 5:
+            data.mark_invalid()
+        if m == 5:
+            data.write(hbytes([1]))
+            data.mark_interesting()
+        n = data.draw_bits(8)
+        if m == n + 2:
+            data.mark_interesting()
+    assert x == hbytes([5, 1])
+
+
 def test_shrinking_blocks_from_common_offset(monkeypatch):
     monkeypatch.setattr(
         Shrinker, 'shrink', lambda self: (