Repository: HypothesisWorks/hypothesis

Issue:
Massive performance drop in test_blacklisted_characters
While testing some other changes, I stumbled across the fact that `test_blacklisted_characters` now takes a very long time to run.

This can be seen in https://travis-ci.org/HypothesisWorks/hypothesis/jobs/504749908, and seems to be linked to #1846 in some way.


Discussion:
Sadly I haven't had a chance to track this down more precisely, or to figure out whether it's a real bug or a test bug.
I've determined that the slowdown is in the `assert_no_examples` part of the test.

Looking through the `--hypothesis-verbosity=debug -s` output, it is doing more-or-less what it was doing before: It quickly finds all of the unique output values, and then spends the rest of its exploration budget generating increasingly-elaborate rejection-sampler sequences.

The difference is that instead of finishing in an instant, this process becomes slower and slower (inconsistently) as the test continues to run.
I traced the problem to `generate_novel_prefix` in `DataTree`:

https://github.com/HypothesisWorks/hypothesis/blob/73e41c3514e073a1c1249d5bfd7ab7b46f47c6ec/hypothesis-python/src/hypothesis/internal/conjecture/datatree.py#L256-L284

As the test continues to run, the `while True` loop takes more and more iterations to complete on average. It often requires hundreds or thousands of retries before it can find a novel prefix by chance.

The underlying cause is that this novelty-generator has no way to detect and avoid exhausted parts of the tree. So when novel prefixes are rare, it spins over and over until it gets lucky enough to stumble upon one.
This effect isn't specific to `characters`. I can reproduce it with this:

```python
def test_integers():
    assert_no_examples(st.integers(0, 5), lambda x: False)
```

Range sizes that are `(2 ** n) - 2` are the most severely affected.
> The underlying cause is that this novelty-generator has no way to detect and avoid exhausted parts of the tree. So when novel prefixes are rare, it spins over and over until it gets lucky enough to stumble upon one.

This isn't quite true. Note that it detects exhausted parts of the tree up until the point where the first possible branch occurs. This means that when there is a long forced prefix it finds that without spinning. I had (apparently mistakenly) assumed that that would be enough and that the cases where there were multiple possible branches to take there but we still got high rejection rates would be relatively uncommon.

I think a fix that is *probably* sufficient for this case and is equivalent in result to the current is to always have the first non-forced block chosen uniformly at random from the set of available possibilities. Maybe something more sophisticated is needed in the general case.
`@pytest.mark.skip` was added in 354258d4e2b0298ca6768727c8d41067c3b9ca64 so this isn't killing our CI, but the underlying problem is still there.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/data.py b/hypothesis-python/src/hypothesis/internal/conjecture/data.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/data.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/data.py
@@ -752,6 +752,7 @@ def __init__(self, max_length, draw_bytes, observer=None):
         self.draw_times = []
         self.max_depth = 0
         self.has_discards = False
+        self.consecutive_discard_counts = []
 
         self.__result = None
 
@@ -862,15 +863,34 @@ def start_example(self, label):
         if self.depth > self.max_depth:
             self.max_depth = self.depth
         self.__example_record.start_example(label)
+        self.consecutive_discard_counts.append(0)
 
     def stop_example(self, discard=False):
         if self.frozen:
             return
+        self.consecutive_discard_counts.pop()
         if discard:
             self.has_discards = True
         self.depth -= 1
         assert self.depth >= -1
         self.__example_record.stop_example(discard)
+        if self.consecutive_discard_counts:
+            # We block long sequences of discards. This helps us avoid performance
+            # problems where there is rejection sampling. In particular tests which
+            # have a very small actual state space but use rejection sampling will
+            # play badly with generate_novel_prefix() in DataTree, and will end up
+            # generating very long tests with long runs of the rejection sample.
+            if discard:
+                self.consecutive_discard_counts[-1] += 1
+                # 20 is a fairly arbitrary limit chosen mostly so that all of the
+                # existing tests passed under it. Essentially no reasonable
+                # generation should hit this limit when running in purely random
+                # mode, but unreasonable generation is fairly widespread, and our
+                # manipulation of the bitstream can make it more likely.
+                if self.consecutive_discard_counts[-1] > 20:
+                    self.mark_invalid()
+            else:
+                self.consecutive_discard_counts[-1] = 0
 
     def note_event(self, event):
         self.events.add(event)


Test patch:
diff --git a/hypothesis-python/tests/cover/test_conjecture_test_data.py b/hypothesis-python/tests/cover/test_conjecture_test_data.py
--- a/hypothesis-python/tests/cover/test_conjecture_test_data.py
+++ b/hypothesis-python/tests/cover/test_conjecture_test_data.py
@@ -470,3 +470,16 @@ def test_example_equality():
 
         assert not (ex == "hello")
         assert ex != "hello"
+
+
+def test_discarded_data_is_eventually_terminated():
+
+    data = ConjectureData.for_buffer(hbytes(100))
+
+    with pytest.raises(StopTest):
+        for _ in hrange(100):
+            data.start_example(1)
+            data.draw_bits(1)
+            data.stop_example(discard=True)
+
+    assert data.status == Status.INVALID
diff --git a/hypothesis-python/tests/cover/test_simple_characters.py b/hypothesis-python/tests/cover/test_simple_characters.py
--- a/hypothesis-python/tests/cover/test_simple_characters.py
+++ b/hypothesis-python/tests/cover/test_simple_characters.py
@@ -130,7 +130,6 @@ def test_whitelisted_characters_override():
     assert_no_examples(st, lambda c: c not in good_characters + "0123456789")
 
 
-@pytest.mark.skip  # temporary skip due to 560 second (!) perf regression; see #1864
 def test_blacklisted_characters():
     bad_chars = u"te02тест49st"
     st = characters(