Repository: HypothesisWorks/hypothesis

Issue:
`@settings(phases=)` API makes it very easy to accidentally enable `Phase.explain`
Pop quiz, what does this line of code do?

```python
@settings(phases=set(Phase) - {Phase.target})
```

If your answer didn't include “enables explain mode, potentially making test runs and failure-reporting *much* slower”, then perhaps you see the problem.

---

I think there are two issues here:

- 9X% of the time, `phases` is used for *excluding* phases, yet the common idiom for doing so is quite clunky (and often incorrect if your current profile deliberately excludes some phases).
- Having an expensive off-by-default phase makes the above incorrectness problem all but guaranteed, with potentially severe consequences.

---

I therefore tentatively propose two changes:

- Add some kind of `skip_phases` setting, making it much easier to write correct code that excludes phases without accidentally enabling others.
- Don't enable the explain-tracer and explainer on the basis of `Phase.explain` alone; require explicit opt-in via a separate setting.


Discussion:
See https://github.com/HypothesisWorks/hypothesis/pull/2981/commits/eb271787fbb8e86e33ead94cb5182e32790a7a24 for a recent example of this actually happening.

Note that if I artificially add a failing assert to that test, the time taken to actually report failure jumps from ~20 seconds (without explain) to ~3 minutes (with explain).
I think this is an important thing to get right, but I'd much rather take longer to think about it if that means we can change it *once* to a design that we'll be happy with for the long term.  In particular, I'm concerned that adding a `skip_phases` setting would lead to the same unintuitive and ambiguous behaviour that we saw from the `suppress_health_check` and [(now removed) `perform_health_check `](https://hypothesis.readthedocs.io/en/latest/changes.html#v3-56-0) setting.

I'm also not quite convinced that this is hurting projects which aren't Hypothesis or direct extensions - the only downstream uses I've seen for the `phases` setting are `phases={Phase.explain}` to disable all but explicit examples, and `phases=set(Phase) - {Phase.shrink}` to disable shrinking.  In the former case there's obviously no problem; and in the latter case explain-mode is also disabled because it only applies while shrinking a failing example.

The main reason that I made `explain` a phase in the first place was that it interacts with `shrink` in roughly the same way that `target` does with `generate` - the latter builds on the former.  Using the existing semantics for phases means avoiding the mess of handling potentially-conflicting settings for "explain without shrinking".

Performance idea: make the `Tracer` class faster - using `__slots__`, reducing attribute lookups, maybe not-tracing some files.  Unfortunately all my other performance ideas are prone to breaking in subtle ways and therefore *not* a UX improvement.
> I'm also not quite convinced that this is hurting projects which aren't Hypothesis or direct extensions - the only downstream uses I've seen for the `phases` setting are `phases={Phase.explain}` to disable all but explicit examples, and `phases=set(Phase) - {Phase.shrink}` to disable shrinking. In the former case there's obviously no problem; and in the latter case explain-mode is also disabled because it only applies while shrinking a failing example.

Ah, that sounds plausible. And I hadn't noticed that the tracer only gets installed after at least one failure.

(Actually, how does this interact with the continue-after-failure feature? Is it possible to disable shrinking, accidentally enable explaining, and then have the explainer decide to run on just the examples between the first failure and the cutoff?)
> Actually, how does this interact with the continue-after-failure feature? Is it possible to disable shrinking, accidentally enable explaining, and then have the explainer decide to run on just the examples between the first failure and the cutoff?

Yes, that's exactly what would happen.  This is actually pretty useful when shrinking is enabled, because it gives the explainer a larger pool of traces to compare with less correlation than those observed while shrinking a particular failure, but we can disable it for the no-shrinking case.

(Perhaps we should disable it for tests which are failing due to a deadline though; both to avoid masking the real perf issue and because pointing to a point on the control-flow graph is less likely to be helpful)

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -529,6 +529,7 @@ def __init__(
 
         self.files_to_propagate = set()
         self.failed_normally = False
+        self.failed_due_to_deadline = False
 
         self.explain_traces = defaultdict(set)
 
@@ -668,6 +669,8 @@ def _execute_once_for_engine(self, data):
             trace = frozenset()
             if (
                 self.failed_normally
+                and not self.failed_due_to_deadline
+                and Phase.shrink in self.settings.phases
                 and Phase.explain in self.settings.phases
                 and sys.gettrace() is None
                 and not PYPY
@@ -736,6 +739,9 @@ def _execute_once_for_engine(self, data):
                 if trace:  # pragma: no cover
                     # Trace collection is explicitly disabled under coverage.
                     self.explain_traces[interesting_origin].add(trace)
+                if interesting_origin[0] == DeadlineExceeded:
+                    self.failed_due_to_deadline = True
+                    self.explain_traces.clear()
                 data.mark_interesting(interesting_origin)
 
     def run_engine(self):
diff --git a/hypothesis-python/src/hypothesis/internal/scrutineer.py b/hypothesis-python/src/hypothesis/internal/scrutineer.py
--- a/hypothesis-python/src/hypothesis/internal/scrutineer.py
+++ b/hypothesis-python/src/hypothesis/internal/scrutineer.py
@@ -31,11 +31,13 @@ def should_trace_file(fname):
 
 
 class Tracer:
-    """A super-simple line coverage tracer."""
+    """A super-simple branch coverage tracer."""
+
+    __slots__ = ("branches", "_previous_location")
 
     def __init__(self):
         self.branches = set()
-        self.prev = None
+        self._previous_location = None
 
     def trace(self, frame, event, arg):
         if event == "call":
@@ -43,8 +45,9 @@ def trace(self, frame, event, arg):
         elif event == "line":
             fname = frame.f_code.co_filename
             if should_trace_file(fname):
-                self.branches.add((self.prev, (fname, frame.f_lineno)))
-                self.prev = (fname, frame.f_lineno)
+                current_location = (fname, frame.f_lineno)
+                self.branches.add((self._previous_location, current_location))
+                self._previous_location = current_location
 
 
 def get_explaining_locations(traces):
@@ -94,6 +97,7 @@ def get_explaining_locations(traces):
     "Explanation:",
     "    These lines were always and only run by failing examples:",
 )
+HAD_TRACE = "    We didn't try to explain this, because sys.gettrace()="
 
 
 def make_report(explanations, cap_lines_at=5):
@@ -114,8 +118,9 @@ def make_report(explanations, cap_lines_at=5):
 
 def explanatory_lines(traces, settings):
     if Phase.explain in settings.phases and sys.gettrace() and not traces:
-        msg = "    We didn't try to explain this, because sys.gettrace()="
-        return defaultdict(lambda: [EXPLANATION_STUB[0], msg + repr(sys.gettrace())])
+        return defaultdict(
+            lambda: [EXPLANATION_STUB[0], HAD_TRACE + repr(sys.gettrace())]
+        )
     # Return human-readable report lines summarising the traces
     explanations = get_explaining_locations(traces)
     max_lines = 5 if settings.verbosity <= Verbosity.normal else 100


Test patch:
diff --git a/hypothesis-python/tests/cover/test_scrutineer.py b/hypothesis-python/tests/cover/test_scrutineer.py
--- a/hypothesis-python/tests/cover/test_scrutineer.py
+++ b/hypothesis-python/tests/cover/test_scrutineer.py
@@ -19,60 +19,17 @@
 import pytest
 
 from hypothesis.internal.compat import PYPY
-from hypothesis.internal.scrutineer import make_report
+from hypothesis.internal.scrutineer import HAD_TRACE
 
-BUG_MARKER = "# BUG"
-PRELUDE = """
+CODE_SAMPLE = """
 from hypothesis import Phase, given, settings, strategies as st
 
 @settings(phases=tuple(Phase), derandomize=True)
-"""
-
-
-def expected_reports(file_contents, fname):
-    # Takes the source code string with "# BUG" comments, and returns a list of
-    # multi-line report strings which we expect to see in explain-mode output.
-    # The list length is the number of explainable bugs, usually one.
-    explanations = {
-        i: {(fname, i)}
-        for i, line in enumerate(file_contents.splitlines())
-        if line.endswith(BUG_MARKER)
-    }
-    return ["\n".join(r) for k, r in make_report(explanations).items()]
-
-
-TRIVIAL = """
 @given(st.integers())
 def test_reports_branch_in_test(x):
     if x > 10:
         raise AssertionError  # BUG
 """
-MULTIPLE_BUGS = """
-@given(st.integers(), st.integers())
-def test_reports_branch_in_test(x, y):
-    if x > 10:
-        raise (AssertionError if x % 2 else Exception)  # BUG
-"""
-
-
-# We skip tracing for explanations under PyPy, where it has a large performance
-# impact, or if there is already a trace function (e.g. coverage or a debugger)
-@pytest.mark.skipif(PYPY or sys.gettrace(), reason="See comment")
-@pytest.mark.parametrize(
-    "code",
-    [
-        pytest.param(TRIVIAL, id="trivial"),
-        pytest.param(MULTIPLE_BUGS, id="multiple-bugs"),
-    ],
-)
-def test_explanations(code, testdir):
-    code = PRELUDE + code
-    test_file = str(testdir.makepyfile(code))
-    pytest_stdout = str(testdir.runpytest_inprocess(test_file, "--tb=native").stdout)
-    expected = expected_reports(code, fname=test_file)
-    assert len(expected) == code.count(BUG_MARKER)
-    for report in expected:
-        assert report in pytest_stdout
 
 
 @pytest.mark.skipif(PYPY, reason="Tracing is slow under PyPy")
@@ -85,16 +42,9 @@ def test_cannot_explain_message(testdir):
     try:
         if no_tracer:
             sys.settrace(lambda frame, event, arg: None)
-
-        code = PRELUDE + TRIVIAL
-        test_file = testdir.makepyfile(code)
-        result = testdir.runpytest_inprocess(test_file, "--tb=native")
-        result.stdout.re_match_lines(
-            [
-                r"Explanation:",
-                fr"""\s*{re.escape("We didn't try to explain this, because sys.gettrace()=")}.*""",
-            ],
-            consecutive=True,
+        test_file = testdir.makepyfile(CODE_SAMPLE)
+        testdir.runpytest_inprocess(test_file, "--tb=native").stdout.re_match_lines(
+            [r"Explanation:", re.escape(HAD_TRACE)], consecutive=True
         )
     finally:
         if no_tracer:
diff --git a/hypothesis-python/tests/nocover/test_scrutineer.py b/hypothesis-python/tests/nocover/test_scrutineer.py
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/tests/nocover/test_scrutineer.py
@@ -0,0 +1,83 @@
+# This file is part of Hypothesis, which may be found at
+# https://github.com/HypothesisWorks/hypothesis/
+#
+# Most of this work is copyright (C) 2013-2021 David R. MacIver
+# (david@drmaciver.com), but it contains contributions by others. See
+# CONTRIBUTING.rst for a full list of people who may hold copyright, and
+# consult the git log if you need to determine who owns an individual
+# contribution.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public License,
+# v. 2.0. If a copy of the MPL was not distributed with this file, You can
+# obtain one at https://mozilla.org/MPL/2.0/.
+#
+# END HEADER
+
+import sys
+
+import pytest
+
+from hypothesis.internal.compat import PYPY
+from hypothesis.internal.scrutineer import make_report
+
+# We skip tracing for explanations under PyPy, where it has a large performance
+# impact, or if there is already a trace function (e.g. coverage or a debugger)
+pytestmark = pytest.mark.skipif(PYPY or sys.gettrace(), reason="See comment")
+
+BUG_MARKER = "# BUG"
+DEADLINE_PRELUDE = """
+from datetime import timedelta
+from hypothesis.errors import DeadlineExceeded
+"""
+PRELUDE = """
+from hypothesis import Phase, given, settings, strategies as st
+
+@settings(phases=tuple(Phase), derandomize=True)
+"""
+TRIVIAL = """
+@given(st.integers())
+def test_reports_branch_in_test(x):
+    if x > 10:
+        raise AssertionError  # BUG
+"""
+MULTIPLE_BUGS = """
+@given(st.integers(), st.integers())
+def test_reports_branch_in_test(x, y):
+    if x > 10:
+        raise (AssertionError if x % 2 else Exception)  # BUG
+"""
+FRAGMENTS = (
+    pytest.param(TRIVIAL, id="trivial"),
+    pytest.param(MULTIPLE_BUGS, id="multiple-bugs"),
+)
+
+
+def get_reports(file_contents, *, testdir):
+    # Takes the source code string with "# BUG" comments, and returns a list of
+    # multi-line report strings which we expect to see in explain-mode output.
+    # The list length is the number of explainable bugs, usually one.
+    test_file = str(testdir.makepyfile(file_contents))
+    pytest_stdout = str(testdir.runpytest_inprocess(test_file, "--tb=native").stdout)
+
+    explanations = {
+        i: {(test_file, i)}
+        for i, line in enumerate(file_contents.splitlines())
+        if line.endswith(BUG_MARKER)
+    }
+    expected = ["\n".join(r) for k, r in make_report(explanations).items()]
+    return pytest_stdout, expected
+
+
+@pytest.mark.parametrize("code", FRAGMENTS)
+def test_explanations(code, testdir):
+    pytest_stdout, expected = get_reports(PRELUDE + code, testdir=testdir)
+    assert len(expected) == code.count(BUG_MARKER)
+    for report in expected:
+        assert report in pytest_stdout
+
+
+@pytest.mark.parametrize("code", FRAGMENTS)
+def test_no_explanations_if_deadline_exceeded(code, testdir):
+    code = code.replace("AssertionError", "DeadlineExceeded(timedelta(), timedelta())")
+    pytest_stdout, _ = get_reports(DEADLINE_PRELUDE + PRELUDE + code, testdir=testdir)
+    assert "Explanation:" not in pytest_stdout