Repository: HypothesisWorks/hypothesis

Issue:
Better docs for stopping immediately after failure
I'm writing integration tests that I'd like randomized, but don't particular care about being exhaustive, because the tests can take a few seconds to run. So I set `max_examples` to 1 to stop after the first success, and `phases` to `[Phase.generate]` to avoid shrinking after a failure. But I noticed that hypothesis still kept rerunning the tests; bumping the verbosity shows multiple "Trying example" logs, which didn't match the expected behavior I thought hypothesis did.

After scouring the docs, I found the `report_multiple_bugs` setting, and after disabling it, it seems to correctly stop after the first failure. I think my first point of confusion is why `report_multiple_bugs` is True by default, as it's not the default behavior for other frameworks like Haskell's QuickCheck or Javascript's fast-check. But separately, it seems to me that there is a hidden phase here, in addition to shrinking, that retries runs to find bugs, that should be possible to be disabled, just like the shrinking phase


Discussion:
I think it makes sense to stop generating immediately on failure if shrinking is disabled, even if `report_multiple_failures=True`, to better match user expectations.

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/_settings.py b/hypothesis-python/src/hypothesis/_settings.py
--- a/hypothesis-python/src/hypothesis/_settings.py
+++ b/hypothesis-python/src/hypothesis/_settings.py
@@ -430,12 +430,12 @@ def _validate_database(db):
 
 @unique
 class Phase(IntEnum):
-    explicit = 0
-    reuse = 1
-    generate = 2
-    target = 3
-    shrink = 4
-    explain = 5
+    explicit = 0  #: controls whether explicit examples are run.
+    reuse = 1  #: controls whether previous examples will be reused.
+    generate = 2  #: controls whether new examples will be generated.
+    target = 3  #: controls whether examples will be mutated for targeting.
+    shrink = 4  #: controls whether examples will be shrunk.
+    explain = 5  #: controls whether Hypothesis attempts to explain test failures.
 
     def __repr__(self):
         return f"Phase.{self.name}"
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
@@ -577,8 +577,12 @@ def should_generate_more(self):
         # the run.
         if not self.interesting_examples:
             return True
+        # Users who disable shrinking probably want to exit as fast as possible.
         # If we've found a bug and won't report more than one, stop looking.
-        elif not self.settings.report_multiple_bugs:
+        elif (
+            Phase.shrink not in self.settings.phases
+            or not self.settings.report_multiple_bugs
+        ):
             return False
         assert self.first_bug_found_at <= self.last_bug_found_at <= self.call_count
         # Otherwise, keep searching for between ten and 'a heuristic' calls.


Test patch:
diff --git a/hypothesis-python/tests/conjecture/test_engine.py b/hypothesis-python/tests/conjecture/test_engine.py
--- a/hypothesis-python/tests/conjecture/test_engine.py
+++ b/hypothesis-python/tests/conjecture/test_engine.py
@@ -302,7 +302,7 @@ def f(data):
         f, settings=settings(database=None, phases=(Phase.reuse, Phase.generate))
     )
     runner.run()
-    assert len(seen) == MIN_TEST_CALLS
+    assert len(seen) == 1
 
 
 def test_reuse_phase_runs_for_max_examples_if_generation_is_disabled():
diff --git a/hypothesis-python/tests/cover/test_slippage.py b/hypothesis-python/tests/cover/test_slippage.py
--- a/hypothesis-python/tests/cover/test_slippage.py
+++ b/hypothesis-python/tests/cover/test_slippage.py
@@ -246,7 +246,7 @@ def test_finds_multiple_failures_in_generation():
     special = []
     seen = set()
 
-    @settings(phases=[Phase.generate], max_examples=100)
+    @settings(phases=[Phase.generate, Phase.shrink], max_examples=100)
     @given(st.integers(min_value=0))
     def test(x):
         """Constructs a test so the 10th largeish example we've seen is a