Repository: HypothesisWorks/hypothesis

Issue:
Multi-bug discovery during generation
As a consequence of #836 we can now find and display multiple bugs if these occur during either shrinking or retrieving examples from the database.

Amusingly though, we can't currently find multiple bugs in the phase that is actually designed to find bugs, generation! We should probably fix that.

However it is not sufficient to just run generation to completion, because a reasonable usage pattern is "Run generation until you find a bug and then stop and tell me that bug", which could be done by e.g. setting `max_examples` absurdly high and turning off the time limit. In this case we would not want to keep running indefinitely after we've found a bug.

I think the correct logic here is:

1. Skip generation altogether if we found a bug in example replay
2. Always respect `max_examples` in the sense that we stop running generation after we've generated that many valid or interesting examples.
3. Stop running generation at some point "reasonably soon" after we've found our first bug.

The devil is of course in the details of what "reasonably soon" means. A non-exhaustive list of possibilities:

1. Just do it time based - run for at most a few seconds after the first bug is found.
2. Stop when we've run twice as many examples as it took to find the first bug
3. Stop when we've run twice as many examples as it took to find the *last* bug
4. Do something clever and statistical to estimate the number of bugs we can expect to find and stop when we've found that many
5. Some hybrid of the above.

My preference is probably some hybrid of 1 and 3.


Discussion:
We should indeed fix this!

I'm much less eager to skip generation though - particularly I don't see much use in "run until you find a bug, then stop".  This seems to fall in to a awkward spot where it's mostly a case for AFL, and if Hypothesis can't find the bug you'll just waste a lot of cycles on it - especially if your tests are running approximately serially rather than interleaved.

I agree with the 'correct logic'; not much more to say there.

I *don't* like the idea of reintroducing time into the number of examples run.  If we're not simply running until we hit max_examples - which is after all the expected behaviour in all non-buggy runs! - I favor option three as a simple version of option four.  If you found one or several bugs in `0 .. n` examples but none in `n+1 .. 2n`, that's a reasonable place to stop looking.  (not counting examples run while shrinking and possibly slipping between bugs)
> I'm much less eager to skip generation though - particularly I don't see much use in "run until you find a bug, then stop". This seems to fall in to a awkward spot where it's mostly a case for AFL, and if Hypothesis can't find the bug you'll just waste a lot of cycles on it - especially if your tests are running approximately serially rather than interleaved.

I find myself running with large numbers of examples fairly often. It's not quite "run until you hit a bug", but it's still going to be running for minutes or tens of minutes. When I'm doing this I would definitely find it annoying if Hypothesis found a bug and then spent ages looking for another one.

I agree that something like AFL would be more useful here, but for now Hypothesis scratches a lot of itches that AFL can't (and Hypothesis should get more AFL-like over time, ideally)

> I don't like the idea of reintroducing time into the number of examples run. 

I don't think it's too problematic. The problem with timeout was that it gave users a false sense of confidence because it looked like you were running more examples than you were. In this case that's a non-issue, because Hypothesis is running either the amount of examples you asked for, or it found a bug in which case there was never any guarantee of how many examples were run anyway.
To make the point, we should also update our headline example in the readme:

```python
@given(st.lists(st.floats(allow_nan=False)))
def test_mean(xs):
    mean = sum(xs) / len(xs)
    assert min(xs) <= mean <= max(xs)
```
```
Falsifying example: test_mean(xs=[9.9792015476736e+291, 1.7976931348623157e+308])
Traceback (most recent call last):
    ...
AssertionError: mean=inf

Falsifying example: test_mean(xs=[])
Traceback (most recent call last):
    ...
ZeroDivisionError: division by zero

---------------------------------------------------------------------------
MultipleFailures: Hypothesis found 2 distinct failures.
```

(Note that this is on the current released version!  We already find two distinct bugs, just by shrinking the list 😄)
I'm going to start working on this
I'm not working on this anymore. Can a maintainer grab this?

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
@@ -56,6 +56,7 @@
 MAX_SHRINKS = 500
 CACHE_SIZE = 10000
 MUTATION_POOL_SIZE = 100
+MIN_TEST_CALLS = 10
 
 
 @attr.s
@@ -98,6 +99,9 @@ def __init__(self, test_function, settings=None, random=None, database_key=None)
         self.target_selector = TargetSelector(self.random)
 
         self.interesting_examples = {}
+        # We use call_count because there may be few possible valid_examples.
+        self.first_bug_found_at = None
+        self.last_bug_found_at = None
 
         self.shrunk_examples = set()
 
@@ -182,6 +186,9 @@ def test_function(self, data):
                 existing = self.interesting_examples[key]
             except KeyError:
                 changed = True
+                self.last_bug_found_at = self.call_count
+                if self.first_bug_found_at is None:
+                    self.first_bug_found_at = self.call_count
             else:
                 if sort_key(data.buffer) < sort_key(existing.buffer):
                     self.shrinks += 1
@@ -199,6 +206,10 @@ def test_function(self, data):
                 self.exit_with(ExitReason.max_shrinks)
 
         if not self.interesting_examples:
+            # Note that this logic is reproduced to end the generation phase when
+            # we have interesting examples.  Update that too if you change this!
+            # (The doubled implementation is because here we exit the engine entirely,
+            #  while in the other case below we just want to move on to shrinking.)
             if self.valid_examples >= self.settings.max_examples:
                 self.exit_with(ExitReason.max_examples)
             if self.call_count >= max(
@@ -588,6 +599,11 @@ def exit_with(self, reason):
     def generate_new_examples(self):
         if Phase.generate not in self.settings.phases:
             return
+        if self.interesting_examples:
+            # The example database has failing examples from a previous run,
+            # so we'd rather report that they're still failing ASAP than take
+            # the time to look for additional failures.
+            return
 
         zero_data = self.cached_test_function(hbytes(self.settings.buffer_size))
         if zero_data.status > Status.OVERRUN:
@@ -635,8 +651,35 @@ def generate_new_examples(self):
 
         self.health_check_state = HealthCheckState()
 
+        def should_generate_more():
+            # If we haven't found a bug, keep looking.  We check this before
+            # doing anything else as it's by far the most common case.
+            if not self.interesting_examples:
+                return True
+            # If we've found a bug and won't report more than one, stop looking.
+            elif not self.settings.report_multiple_bugs:
+                return False
+            assert self.first_bug_found_at <= self.last_bug_found_at <= self.call_count
+            # End the generation phase where we would have ended it if no bugs had
+            # been found.  This reproduces the exit logic in `self.test_function`,
+            # but with the important distinction that this clause will move on to
+            # the shrinking phase having found one or more bugs, while the other
+            # will exit having found zero bugs.
+            if (
+                self.valid_examples >= self.settings.max_examples
+                or self.call_count >= max(self.settings.max_examples * 10, 1000)
+            ):  # pragma: no cover
+                return False
+            # Otherwise, keep searching for between ten and 'a heuristic' calls.
+            # We cap 'calls after first bug' so errors are reported reasonably
+            # soon even for tests that are allowed to run for a very long time,
+            # or sooner if the latest half of our test effort has been fruitless.
+            return self.call_count < MIN_TEST_CALLS or self.call_count < min(
+                self.first_bug_found_at + 1000, self.last_bug_found_at * 2
+            )
+
         count = 0
-        while not self.interesting_examples and (
+        while should_generate_more() and (
             count < 10 or self.health_check_state is not None
         ):
             prefix = self.generate_novel_prefix()
@@ -661,7 +704,7 @@ def draw_bytes(data, n):
 
         zero_bound_queue = []
 
-        while not self.interesting_examples:
+        while should_generate_more():
             if zero_bound_queue:
                 # Whenever we generated an example and it hits a bound
                 # which forces zero blocks into it, this creates a weird


Test patch:
diff --git a/hypothesis-python/tests/cover/test_conjecture_engine.py b/hypothesis-python/tests/cover/test_conjecture_engine.py
--- a/hypothesis-python/tests/cover/test_conjecture_engine.py
+++ b/hypothesis-python/tests/cover/test_conjecture_engine.py
@@ -36,6 +36,7 @@
     Status,
 )
 from hypothesis.internal.conjecture.engine import (
+    MIN_TEST_CALLS,
     ConjectureRunner,
     ExitReason,
     TargetSelector,
@@ -191,7 +192,7 @@ def tf(data):
     runner = ConjectureRunner(tf)
     runner.run()
     assert runner.exit_reason == ExitReason.flaky
-    assert count == [2]
+    assert count == [MIN_TEST_CALLS + 1]
 
 
 def test_variadic_draw():
@@ -271,6 +272,32 @@ def f(data):
     assert len(seen) == 1
 
 
+@pytest.mark.parametrize("examples", [1, 5, 20, 50])
+def test_stops_after_max_examples_when_generating_more_bugs(examples):
+    seen = []
+    bad = [False, False]
+
+    def f(data):
+        seen.append(data.draw_bits(32))
+        # Rare, potentially multi-error conditions
+        if seen[-1] > 2 ** 31:
+            bad[0] = True
+            raise ValueError
+        bad[1] = True
+        raise Exception
+
+    runner = ConjectureRunner(
+        f, settings=settings(max_examples=examples, phases=[Phase.generate])
+    )
+    try:
+        runner.run()
+    except Exception:
+        pass
+    # No matter what, whether examples is larger or smalller than MAX_TEST_CALLS,
+    # we stop looking at max_examples.  (and re-run each failure for the traceback)
+    assert len(seen) <= examples + sum(bad)
+
+
 def test_interleaving_engines():
     children = []
 
@@ -304,7 +331,7 @@ def f(data):
         f, settings=settings(database=None, phases=(Phase.reuse, Phase.generate))
     )
     runner.run()
-    assert len(seen) == 1
+    assert len(seen) == MIN_TEST_CALLS
 
 
 def test_erratic_draws():
diff --git a/hypothesis-python/tests/cover/test_flakiness.py b/hypothesis-python/tests/cover/test_flakiness.py
--- a/hypothesis-python/tests/cover/test_flakiness.py
+++ b/hypothesis-python/tests/cover/test_flakiness.py
@@ -21,6 +21,7 @@
 
 from hypothesis import HealthCheck, Verbosity, assume, example, given, reject, settings
 from hypothesis.errors import Flaky, Unsatisfiable, UnsatisfiedAssumption
+from hypothesis.internal.conjecture.engine import MIN_TEST_CALLS
 from hypothesis.strategies import booleans, composite, integers, lists, random_module
 from tests.common.utils import no_shrink
 
@@ -59,6 +60,7 @@ def oops(s):
 def test_does_not_attempt_to_shrink_flaky_errors():
     values = []
 
+    @settings(database=None)
     @given(integers())
     def test(x):
         values.append(x)
@@ -66,7 +68,12 @@ def test(x):
 
     with pytest.raises(Flaky):
         test()
-    assert len(set(values)) == 1
+    # We try a total of ten calls in the generation phase, each usually a
+    # unique value, looking briefly (and unsuccessfully) for another bug.
+    assert 1 < len(set(values)) <= MIN_TEST_CALLS
+    # We don't try any new values while shrinking, just execute the test
+    # twice more (to check for flakiness and to raise the bug to the user).
+    assert set(values) == set(values[:-2])
 
 
 class SatisfyMe(Exception):
diff --git a/hypothesis-python/tests/cover/test_slippage.py b/hypothesis-python/tests/cover/test_slippage.py
--- a/hypothesis-python/tests/cover/test_slippage.py
+++ b/hypothesis-python/tests/cover/test_slippage.py
@@ -20,9 +20,10 @@
 import pytest
 
 import hypothesis.strategies as st
-from hypothesis import given, settings
+from hypothesis import Phase, given, settings
 from hypothesis.database import InMemoryExampleDatabase
 from hypothesis.errors import Flaky, MultipleFailures
+from hypothesis.internal.conjecture.engine import MIN_TEST_CALLS
 from tests.common.utils import capture_out, non_covering_examples
 
 
@@ -239,3 +240,51 @@ def test(i):
     with pytest.raises(MultipleFailures if allow_multi else TypeError):
         test()
     assert seen == {TypeError, ValueError}
+
+
+def test_finds_multiple_failures_in_generation():
+    @settings(phases=[Phase.generate])
+    @given(st.lists(st.floats()))
+    def test(x):
+        mean = sum(x) / len(x)  # ZeroDivisionError
+        if len(x) == 1:
+            assert mean == x[0]  # x[0] is nan
+        assert min(x) <= mean <= max(x)  # x contains nan or +&- inf
+
+    with pytest.raises(MultipleFailures):
+        test()
+
+
+def test_stops_immediately_if_not_report_multiple_bugs():
+    seen = set()
+
+    @settings(phases=[Phase.generate], report_multiple_bugs=False)
+    @given(st.integers())
+    def test(x):
+        seen.add(x)
+        assert False
+
+    with pytest.raises(AssertionError):
+        test()
+    assert len(seen) == 1
+
+
+def test_stops_immediately_on_replay():
+    seen = set()
+
+    @settings(database=InMemoryExampleDatabase())
+    @given(st.integers())
+    def test(x):
+        seen.add(x)
+        assert x
+
+    # On the first run, we look for up to ten examples:
+    with pytest.raises(AssertionError):
+        test()
+    assert 1 < len(seen) <= MIN_TEST_CALLS
+
+    # With failing examples in the database, we stop at one.
+    seen.clear()
+    with pytest.raises(AssertionError):
+        test()
+    assert len(seen) == 1
diff --git a/hypothesis-python/tests/cover/test_testdecorators.py b/hypothesis-python/tests/cover/test_testdecorators.py
--- a/hypothesis-python/tests/cover/test_testdecorators.py
+++ b/hypothesis-python/tests/cover/test_testdecorators.py
@@ -23,6 +23,7 @@
 
 import hypothesis.reporting as reporting
 from hypothesis import HealthCheck, Verbosity, assume, given, note, settings
+from hypothesis.internal.conjecture.engine import MIN_TEST_CALLS
 from hypothesis.strategies import (
     binary,
     booleans,
@@ -417,7 +418,7 @@ def foo(x):
     with raises(AssertionError):
         with capture_out() as out:
             foo()
-    assert failing == [2]
+    assert failing[0] <= MIN_TEST_CALLS + 2
     assert "Falsifying example" in out.getvalue()
     assert "Lo" in out.getvalue()