Repository: HypothesisWorks/hypothesis

Issue:
The shrinker cannot sort most objects in lists
Note: This is part of #1093, where I encourage people to work on the shrinker. As such I will not be working on it myself and encourage you to take a look at it if it takes your fancy.

Due to the underlying representation, Hypothesis usually wants to return lists of things in sorted order of increasing complexity (this is strictly speaking not always true, but it is true for "well behaved" element strategies, including anything in hypothesis.strategies).

Sometimes this works currently:

```pycon
>>> find(st.lists(st.integers(), min_size=5, unique=True), lambda x: True)
[0, 1, -1, 2, -2]
```

(simplicity for integers is size then sign, so that's the correct order)

However for more complex objects it doesn't:

```pycon
>>> find(st.lists(st.text(), min_size=5, unique_by=len), lambda x: True)
['0000', '0', '00', '', '000']
```

These *should* be in order of ascending size, so that the correct answer is `['', '0', '00', '000', '0000']`.

The problem is that our current sorting logic is focused entirely on *blocks*. These basically correspond to individual low-level primitives. This does not allow the sorting to have any sort of higher level knowledge of how to move things around, which is required here.

I *believe* that we have all of the correct information under the hood to do this relatively easily. One possible approach here would be:

* Partition all drawn examples by their depth and label (this ensures you get non-overlapping examples that are "roughly similar")
* Where those partitions have more than one element, attempt an insertion sort on them to reorder the byte stream to put them in order.

Some care may be needed when examples are of difference size because swapping them might not be valid even if it looks it from the individual examples because of what bytes come after it in the underlying representation.


Discussion:

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
@@ -1335,6 +1335,7 @@ def greedy_shrink(self):
             if not run_expensive_shrinks:
                 continue
 
+            self.reorder_examples()
             self.shrink_offset_pairs()
             self.interval_deletion_with_block_lowering()
             self.pass_to_interval()
@@ -2340,3 +2341,44 @@ def trial(x, y):
                             )
                     j += 1
             i += 1
+
+    def reorder_examples(self):
+        """This pass allows us to reorder pairs of examples which come from the
+        same strategy (or strategies that happen to pun to the same label by
+        accident, but that shouldn't happen often).
+
+        For example, consider the following:
+
+        .. code-block:: python
+
+            import hypothesis.strategies as st
+            from hypothesis import given
+
+            @given(st.text(), st.text())
+            def test_does_not_exceed_100(x, y):
+                assert x != y
+
+        Without the ability to reorder x and y this could fail either with
+        ``x="", ``y="0"``, or the other way around. With reordering it will
+        reliably fail with ``x=""``, ``y="0"``.
+        """
+        i = 0
+        while i < len(self.shrink_target.examples):
+            j = i + 1
+            while j < len(self.shrink_target.examples):
+                ex1 = self.shrink_target.examples[i]
+                ex2 = self.shrink_target.examples[j]
+                if ex1.label == ex2.label and ex2.start >= ex1.end:
+                    buf = self.shrink_target.buffer
+                    attempt = (
+                        buf[:ex1.start] +
+                        buf[ex2.start:ex2.end] +
+                        buf[ex1.end:ex2.start] +
+                        buf[ex1.start:ex1.end] +
+                        buf[ex2.end:]
+                    )
+                    assert len(attempt) == len(buf)
+                    if attempt < buf:
+                        self.incorporate_new_buffer(attempt)
+                j += 1
+            i += 1


Test patch:
diff --git a/hypothesis-python/tests/cover/test_conjecture_engine.py b/hypothesis-python/tests/cover/test_conjecture_engine.py
--- a/hypothesis-python/tests/cover/test_conjecture_engine.py
+++ b/hypothesis-python/tests/cover/test_conjecture_engine.py
@@ -1372,3 +1372,27 @@ def x(data):
                 if m + n == 200:
                     data.mark_interesting()
     assert len(x) == 1
+
+
+def test_can_reorder_examples(monkeypatch):
+    monkeypatch.setattr(
+        ConjectureRunner, 'generate_new_examples',
+        lambda runner: runner.test_function(
+            ConjectureData.for_buffer([1, 0, 1, 1, 0, 1, 0, 0, 0])))
+
+    monkeypatch.setattr(
+        Shrinker, 'shrink', Shrinker.reorder_examples,
+    )
+
+    @run_to_buffer
+    def x(data):
+        total = 0
+        for _ in range(5):
+            data.start_example(0)
+            if data.draw_bits(8):
+                total += data.draw_bits(9)
+            data.stop_example(0)
+        if total == 2:
+            data.mark_interesting()
+
+    assert list(x) == [0, 0, 0, 1, 0, 1, 1, 0, 1]