Repository: HypothesisWorks/hypothesis

Issue:
hypothesis.errors.Flaky raised with tight deadline setting
I'm currently seeing a failure which confuses me a bit (which was probably introduced by things getting slower because of coverage). It roughly looks like this:

```
_________________ test_parse_content_disposition[inline; {}] __________________
caplog = <pytest_catchlog.CompatLogCaptureFixture object at 0x7feb5c413470>
template = 'inline; {}'
stubs = <module 'helpers.stubs' (<_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7febd89cf780>)>
    @pytest.mark.parametrize('template', [
>       '{}',
        'attachment; filename="{}"',
        'inline; {}',
        'attachment; {}="foo"',
        'attachment; filename*=iso-8859-1''{}',
        'attachment; filename*={}',
    ])
    @hypothesis.given(strategies.text(alphabet=[chr(x) for x in range(255)]))
    def test_parse_content_disposition(caplog, template, stubs, s):
tests/unit/browser/webkit/http/test_http_hypothesis.py:30: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.tox/py36/lib/python3.6/site-packages/hypothesis/executors.py:58: in default_new_style_executor
    return function(data)
.tox/py36/lib/python3.6/site-packages/hypothesis/core.py:140: in run
    return test(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
args = ()
kwargs = {'caplog': <pytest_catchlog.CompatLogCaptureFixture object at 0x7feb5c413470>, 's': '\x00', 'stubs': <module 'helpers.stubs' (<_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7febd89cf780>)>, 'template': 'inline; {}'}
text_repr = "caplog=<pytest_catchlog.CompatLogCaptureFixture at 0x7feb5c413470>, template='inline; {}', stubs=<module 'helpers.stubs' (<_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7febd89cf780>)>, s='\\x00'"
    @functools.wraps(test)
    def test_or_flaky(*args, **kwargs):
        text_repr = arg_string(test, args, kwargs)
        raise Flaky(
            (
                'Hypothesis %s(%s) produces unreliable results: Falsified'
                ' on the first call but did not on a subsequent one'
>           ) % (test.__name__, text_repr,))
E       hypothesis.errors.Flaky: Hypothesis test_parse_content_disposition(caplog=<pytest_catchlog.CompatLogCaptureFixture at 0x7feb5c413470>, template='inline; {}', stubs=<module 'helpers.stubs' (<_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7febd89cf780>)>, s='\x00') produces unreliable results: Falsified on the first call but did not on a subsequent one
.tox/py36/lib/python3.6/site-packages/hypothesis/core.py:87: Flaky
------------------------------ Captured log call -------------------------------
[...]
---------------------------------- Hypothesis ----------------------------------
Falsifying example: test_parse_content_disposition(caplog=<pytest_catchlog.CompatLogCaptureFixture at 0x7feb5c413470>, template='inline; {}', stubs=<module 'helpers.stubs' (<_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7febd89cf780>)>, s='\x00')
Failed to reproduce exception. Expected: 
Traceback (most recent call last):
  File "/home/user/qutebrowser.git/.tox/py36/lib/python3.6/site-packages/hypothesis/core.py", line 705, in evaluate_test_data
    self.search_strategy, self.test,
  File "/home/user/qutebrowser.git/.tox/py36/lib/python3.6/site-packages/hypothesis/executors.py", line 58, in default_new_style_executor
    return function(data)
  File "/home/user/qutebrowser.git/.tox/py36/lib/python3.6/site-packages/hypothesis/core.py", line 140, in run
    return test(*args, **kwargs)
  File "/home/user/qutebrowser.git/tests/unit/browser/webkit/http/test_http_hypothesis.py", line 30, in test_parse_content_disposition
    '{}',
  File "/home/user/qutebrowser.git/.tox/py36/lib/python3.6/site-packages/hypothesis/core.py", line 608, in timed_test
    '%.2fms') % (runtime, self.settings.deadline)
hypothesis.errors.DeadlineExceeded: Test took 406.39ms, which exceeds the deadline of 400.00ms
Falsifying example: test_parse_content_disposition(caplog=<pytest_catchlog.CompatLogCaptureFixture at 0x7feb5c413470>, s='\x00', stubs=<module 'helpers.stubs' (<_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7febd89cf780>)>, template='inline; {}')
You can add @seed(302934307671667531413257853548643485645) to this test or run pytest with --hypothesis-seed=302934307671667531413257853548643485645 to reproduce this failure.
```

I don't see how that test could be flaky, it'll always pass with that input. However, I think what happened is that it was just above the deadline once, and just above it when re-running? I'm not sure how, but I feel like hypothesis could give me a better indication of what happened there.


Discussion:
> I don't see how that test could be flaky, it'll always pass with that input. However, I think what happened is that it was just above the deadline once, and just above it when re-running? I'm not sure how, but I feel like hypothesis could give me a better indication of what happened there.

Hmm. Yeah, I was worried about this possibility and was hoping it wouldn't come up quite so soon. Thanks, past-@DRMacIver 😜.

I agree we should handle this better. The easy options as I see it are:

* Lower the deadline slightly for replay - if, say, the last example still takes > 90% of the deadline, raise anyway with a better error.
* (Possibly overlapping with the above) When shrinking a slow example that initially exceeds the deadline by some large margin, temporarily *raise* the deadline so that when we replay it at the end we get something that is comfortably above that margin.
* Change the error message for Flaky when the exception is a DeadlineExceeded to be more informative (nb: Must deal properly with when the deadline is one of multiple failures)

I suspect the correct answer is "all of the above"

Resolving patch:
diff --git a/src/hypothesis/_settings.py b/src/hypothesis/_settings.py
--- a/src/hypothesis/_settings.py
+++ b/src/hypothesis/_settings.py
@@ -633,9 +633,13 @@ def _validate_phases(phases):
     default=not_set,
     description=u"""
 If set, a time in milliseconds (which may be a float to express
-smaller units of time) that a test is not allowed to exceed. Tests which take
-longer than that will be converted into errors. Set this to None to disable
-this behaviour entirely.
+smaller units of time) that each individual example (i.e. each time your test
+function is called, not the whole decorated test) within a test is not
+allowed to exceed. Tests which take longer than that may be converted into
+errors (but will not necessarily be if close to the deadline, to allow some
+variability in test run time).
+
+Set this to None to disable this behaviour entirely.
 
 In future this will default to 200. For now, a
 HypothesisDeprecationWarning will be emitted if you exceed that default
diff --git a/src/hypothesis/core.py b/src/hypothesis/core.py
--- a/src/hypothesis/core.py
+++ b/src/hypothesis/core.py
@@ -578,15 +578,19 @@ def __init__(self, test_runner, search_strategy, test, settings, random):
         self.random = random
         self.__warned_deadline = False
         self.__existing_collector = None
+        self.__test_runtime = None
+        self.__in_final_replay = False
 
         if self.settings.deadline is None:
             self.test = test
         else:
             @proxies(test)
             def timed_test(*args, **kwargs):
+                self.__test_runtime = None
                 start = time.time()
                 result = test(*args, **kwargs)
                 runtime = (time.time() - start) * 1000
+                self.__test_runtime = runtime
                 if self.settings.deadline is not_set:
                     if (
                         not self.__warned_deadline and
@@ -602,11 +606,8 @@ def timed_test(*args, **kwargs):
                             'it to None to disable this check entirely.') % (
                                 runtime, ceil(runtime / 100) * 100,
                         ))
-                elif runtime >= self.settings.deadline:
-                    raise DeadlineExceeded((
-                        'Test took %.2fms, which exceeds the deadline of '
-                        '%.2fms') % (runtime, self.settings.deadline)
-                    )
+                elif runtime >= self.current_deadline:
+                    raise DeadlineExceeded(runtime, self.settings.deadline)
                 return result
             self.test = timed_test
 
@@ -634,6 +635,14 @@ def timed_test(*args, **kwargs):
         else:
             self.collector = None
 
+    @property
+    def current_deadline(self):
+        base = self.settings.deadline
+        if self.__in_final_replay:
+            return base
+        else:
+            return base * 1.25
+
     def should_trace(self, original_filename, frame):  # pragma: no cover
         disp = FileDisposition()
         assert original_filename is not None
@@ -732,10 +741,11 @@ def evaluate_test_data(self, data):
             StopTest,
         ) + exceptions_to_reraise:
             raise
-        except Exception:
+        except Exception as e:
             escalate_hypothesis_internal_error()
-            data.__expected_exception = traceback.format_exc()
-            verbose_report(data.__expected_exception)
+            data.__expected_traceback = traceback.format_exc()
+            data.__expected_exception = e
+            verbose_report(data.__expected_traceback)
 
             error_class, _, tb = sys.exc_info()
 
@@ -822,6 +832,8 @@ def run(self):
 
         flaky = 0
 
+        self.__in_final_replay = True
+
         for falsifying_example in self.falsifying_examples:
             self.__was_flaky = False
             raised_exception = False
@@ -846,10 +858,28 @@ def run(self):
                 report(traceback.format_exc())
 
             if not raised_exception:
-                report(
-                    'Failed to reproduce exception. Expected: \n' +
-                    falsifying_example.__expected_exception,
-                )
+                if (
+                    isinstance(
+                        falsifying_example.__expected_exception,
+                        DeadlineExceeded
+                    ) and self.__test_runtime is not None
+                ):
+                    report((
+                        'Unreliable test timings! On an initial run, this '
+                        'test took %.2fms, which exceeded the deadline of '
+                        '%.2fms, but on a subsequent run it took %.2f ms, '
+                        'which did not. If you expect this sort of '
+                        'variability in your test timings, consider turning '
+                        'deadlines off for this test by setting deadline=None.'
+                    ) % (
+                        falsifying_example.__expected_exception.runtime,
+                        self.settings.deadline, self.__test_runtime
+                    ))
+                else:
+                    report(
+                        'Failed to reproduce exception. Expected: \n' +
+                        falsifying_example.__expected_traceback,
+                    )
 
                 filter_message = (
                     'Unreliable test data: Failed to reproduce a failure '
diff --git a/src/hypothesis/errors.py b/src/hypothesis/errors.py
--- a/src/hypothesis/errors.py
+++ b/src/hypothesis/errors.py
@@ -195,3 +195,10 @@ class MultipleFailures(HypothesisException):
 
 class DeadlineExceeded(HypothesisException):
     """Raised when an individual test body has taken too long to run."""
+
+    def __init__(self, runtime, deadline):
+        super(DeadlineExceeded, self).__init__((
+            'Test took %.2fms, which exceeds the deadline of '
+            '%.2fms') % (runtime, deadline))
+        self.runtime = runtime
+        self.deadline = deadline


Test patch:
diff --git a/tests/cover/test_deadline.py b/tests/cover/test_deadline.py
--- a/tests/cover/test_deadline.py
+++ b/tests/cover/test_deadline.py
@@ -23,7 +23,7 @@
 import pytest
 
 import hypothesis.strategies as st
-from hypothesis import given, settings
+from hypothesis import HealthCheck, given, settings, unlimited
 from hypothesis.errors import Flaky, DeadlineExceeded, \
     HypothesisDeprecationWarning
 from tests.common.utils import capture_out, checks_deprecated_behaviour
@@ -88,3 +88,48 @@ def slow_if_large(i):
         with pytest.raises(DeadlineExceeded):
             slow_if_large()
     assert 'slow_if_large(i=10000)' in o.getvalue()
+
+
+def test_keeps_you_well_above_the_deadline():
+    seen = set()
+    failed_once = [False]
+
+    @settings(deadline=100, timeout=unlimited, suppress_health_check=[
+        HealthCheck.hung_test
+    ])
+    @given(st.integers(0, 2000))
+    def slow(i):
+        # Make sure our initial failure isn't something that immediately goes
+        # flaky.
+        if not failed_once[0]:
+            if i * 0.9 <= 100:
+                return
+            else:
+                failed_once[0] = True
+
+        t = i / 1000
+        if i in seen:
+            time.sleep(0.9 * t)
+        else:
+            seen.add(i)
+            time.sleep(t)
+
+    with pytest.raises(DeadlineExceeded):
+        slow()
+
+
+def test_gives_a_deadline_specific_flaky_error_message():
+    once = [True]
+
+    @settings(deadline=100)
+    @given(st.integers())
+    def slow_once(i):
+        if once[0]:
+            once[0] = False
+            time.sleep(0.2)
+
+    with capture_out() as o:
+        with pytest.raises(Flaky):
+            slow_once()
+    assert 'Unreliable test timing' in o.getvalue()
+    assert 'took 2' in o.getvalue()