Repository: HypothesisWorks/hypothesis

Issue:
Deadline feature interacts poorly with slow data generation in interactive examples
It turns out that currently one of the sources of slowness in Hypothesis for large examples is that our mutation loop is actually quite slow. In particular, replaying an example from a fixed buffer is *much* faster than generating it in the first place.

This has amusing consequences for deadlines. And by amusing I mean sad.

The following script reliably fails with a flaky error from the deadline (the binary search is required because the actual timings will vary from system to system so it's hard to find the deadline needed to trigger this, but I discovered this based on a more complicated real world example):

```python
from hypothesis import given, settings, unlimited, Verbosity
import hypothesis.strategies as st
from hypothesis.errors import DeadlineExceeded


if __name__ == '__main__':
    lo = 1
    hi = 1000

    while lo + 1 < hi:
        mid = (lo + hi) // 2

        @settings(
            deadline=mid, timeout=unlimited, max_examples=10,
            verbosity=Verbosity.quiet, max_shrinks=0
        )
        @given(st.data())
        def test_slow_draw(data):
            for _ in range(1000):
                data.draw(st.binary(min_size=1, max_size=1))

        try:
            test_slow_draw()
            hi = mid
        except DeadlineExceeded:
            lo = mid
```

When I run this locally I get the following failure:

```
Unreliable test timings! On an initial run, this test took 161.08ms, which exceeded the deadline of 125.00ms, but on a subsequent run it took 78.93 ms, which did not. If you expect this sort of variability in your test timings, consider turning deadlines off for this test by setting deadline=None.
Falsifying example: test_slow_draw(data=data(...))
You can add @seed(302934307671667531413257853548643485645) to this test to reproduce this failure.
Traceback (most recent call last):
  File "slow_draw.py", line 24, in <module>
    test_slow_draw()
  File "slow_draw.py", line 15, in test_slow_draw
    deadline=mid, timeout=unlimited, max_examples=10,
  File "/home/david/scratch/v/lib/python3.5/site-packages/hypothesis/core.py", line 1008, in wrapped_test
    state.run()
  File "/home/david/scratch/v/lib/python3.5/site-packages/hypothesis/core.py", line 900, in run
    print_example=True, is_final=True
  File "/home/david/scratch/v/lib/python3.5/site-packages/hypothesis/executors.py", line 58, in default_new_style_executor
    return function(data)
  File "/home/david/scratch/v/lib/python3.5/site-packages/hypothesis/core.py", line 140, in run
    return test(*args, **kwargs)
  File "/home/david/scratch/v/lib/python3.5/site-packages/hypothesis/core.py", line 87, in test_or_flaky
    ) % (test.__name__, text_repr,))
hypothesis.errors.Flaky: Hypothesis test_slow_draw(data=data(...)) produces unreliable results: Falsified on the first call but did not on a subsequent one
```

That's quite a big timing difference! Nearly a factor of two.

I think this admits two solutions.

* As part of the Smarkets work I will be adding a feature for measuring the total amount of time spent in the test *excluding* data generation. I think the deadline should be based on that, 
* I've been thinking that we should change the interface to `ConjectureData` so that it is always initialized with a buffer anyway and remove the closure based API. This would require some revamping to the mutator, but it could probably use that anyway! This would solve the problem by making generation and shrinking the same speed (within the test function that is).

I think both solutions are worth doing on their own rights, and either would work for solving this as a problem.


Discussion:
- Absolutely agree, deadlines and test timing should only measure user code and exclude data generation
- I don't understand Conjecture well enough to have an opinion on the second point
> Absolutely agree, deadlines and test timing should only measure user code and exclude data generation

Nb. (which I forgot when writing this ticket)  this is currently the case unless you use data(). The time measured is only that of the test body, it's just that the test body sometimes contains data generation. 

Resolving patch:
diff --git a/src/hypothesis/core.py b/src/hypothesis/core.py
--- a/src/hypothesis/core.py
+++ b/src/hypothesis/core.py
@@ -23,7 +23,6 @@
 import os
 import sys
 import time
-import functools
 import traceback
 from random import Random
 
@@ -45,7 +44,7 @@
 from hypothesis.reporting import report, verbose_report, current_verbosity
 from hypothesis.statistics import note_engine_for_statistics
 from hypothesis.internal.compat import ceil, str_to_bytes, \
-    get_type_hints, getfullargspec, encoded_filepath
+    benchmark_time, get_type_hints, getfullargspec, encoded_filepath
 from hypothesis.internal.coverage import IN_COVERAGE_TESTS
 from hypothesis.utils.conventions import infer, not_set
 from hypothesis.internal.escalation import is_hypothesis_file, \
@@ -76,18 +75,6 @@ def new_random():
     return random.Random(random.getrandbits(128))
 
 
-def test_is_flaky(test, expected_repr):
-    @functools.wraps(test)
-    def test_or_flaky(*args, **kwargs):
-        text_repr = arg_string(test, args, kwargs)
-        raise Flaky(
-            (
-                'Hypothesis %s(%s) produces unreliable results: Falsified'
-                ' on the first call but did not on a subsequent one'
-            ) % (test.__name__, text_repr,))
-    return test_or_flaky
-
-
 @attr.s()
 class Example(object):
     args = attr.ib()
@@ -113,37 +100,6 @@ def accept(test):
     return accept
 
 
-def reify_and_execute(
-    search_strategy, test,
-    print_example=False,
-    is_final=False, collector=None
-):
-    def run(data):
-        with BuildContext(data, is_final=is_final):
-            import random as rnd_module
-            rnd_module.seed(0)
-            args, kwargs = data.draw(search_strategy)
-
-            if print_example:
-                report(
-                    lambda: 'Falsifying example: %s(%s)' % (
-                        test.__name__, arg_string(test, args, kwargs)))
-            elif current_verbosity() >= Verbosity.verbose:
-                report(
-                    lambda: 'Trying example: %s(%s)' % (
-                        test.__name__, arg_string(test, args, kwargs)))
-            if collector is None:
-                return test(*args, **kwargs)
-            else:  # pragma: no cover
-                try:
-                    collector.start()
-                    return test(*args, **kwargs)
-                finally:
-                    collector.stop()
-
-    return run
-
-
 def seed(seed):
     """seed: Start the test execution from a specific seed.
 
@@ -438,44 +394,14 @@ def __init__(self, test_runner, search_strategy, test, settings, random):
         self.settings = settings
         self.at_least_one_success = False
         self.last_exception = None
-        self.repr_for_last_exception = None
         self.falsifying_examples = ()
         self.__was_flaky = False
         self.random = random
         self.__warned_deadline = False
         self.__existing_collector = None
         self.__test_runtime = None
-        self.__in_final_replay = False
 
-        if self.settings.deadline is None:
-            self.test = test
-        else:
-            @proxies(test)
-            def timed_test(*args, **kwargs):
-                self.__test_runtime = None
-                start = time.time()
-                result = test(*args, **kwargs)
-                runtime = (time.time() - start) * 1000
-                self.__test_runtime = runtime
-                if self.settings.deadline is not_set:
-                    if (
-                        not self.__warned_deadline and
-                        runtime >= 200
-                    ):
-                        self.__warned_deadline = True
-                        note_deprecation((
-                            'Test took %.2fms to run. In future the default '
-                            'deadline setting will be 200ms, which will '
-                            'make this an error. You can set deadline to '
-                            'an explicit value of e.g. %d to turn tests '
-                            'slower than this into an error, or you can set '
-                            'it to None to disable this check entirely.') % (
-                                runtime, ceil(runtime / 100) * 100,
-                        ))
-                elif runtime >= self.current_deadline:
-                    raise DeadlineExceeded(runtime, self.settings.deadline)
-                return result
-            self.test = timed_test
+        self.test = test
 
         self.coverage_data = CoverageData()
         self.files_to_propagate = set()
@@ -501,13 +427,106 @@ def timed_test(*args, **kwargs):
         else:
             self.collector = None
 
-    @property
-    def current_deadline(self):
-        base = self.settings.deadline
-        if self.__in_final_replay:
-            return base
+    def execute(
+        self, data,
+        print_example=False,
+        is_final=False,
+        expected_failure=None, collect=False,
+    ):
+        text_repr = [None]
+        if self.settings.deadline is None:
+            test = self.test
         else:
-            return base * 1.25
+            @proxies(self.test)
+            def test(*args, **kwargs):
+                self.__test_runtime = None
+                initial_draws = len(data.draw_times)
+                start = benchmark_time()
+                result = self.test(*args, **kwargs)
+                finish = benchmark_time()
+                internal_draw_time = sum(data.draw_times[initial_draws:])
+                runtime = (finish - start - internal_draw_time) * 1000
+                self.__test_runtime = runtime
+                if self.settings.deadline is not_set:
+                    if (
+                        not self.__warned_deadline and
+                        runtime >= 200
+                    ):
+                        self.__warned_deadline = True
+                        note_deprecation((
+                            'Test took %.2fms to run. In future the default '
+                            'deadline setting will be 200ms, which will '
+                            'make this an error. You can set deadline to '
+                            'an explicit value of e.g. %d to turn tests '
+                            'slower than this into an error, or you can set '
+                            'it to None to disable this check entirely.') % (
+                                runtime, ceil(runtime / 100) * 100,
+                        ))
+                else:
+                    current_deadline = self.settings.deadline
+                    if not is_final:
+                        current_deadline *= 1.25
+                    if runtime >= current_deadline:
+                        raise DeadlineExceeded(runtime, self.settings.deadline)
+                return result
+
+        def run(data):
+            with self.settings:
+                with BuildContext(data, is_final=is_final):
+                    import random as rnd_module
+                    rnd_module.seed(0)
+                    args, kwargs = data.draw(self.search_strategy)
+                    if expected_failure is not None:
+                        text_repr[0] = arg_string(test, args, kwargs)
+
+                    if print_example:
+                        report(
+                            lambda: 'Falsifying example: %s(%s)' % (
+                                test.__name__, arg_string(test, args, kwargs)))
+                    elif current_verbosity() >= Verbosity.verbose:
+                        report(
+                            lambda: 'Trying example: %s(%s)' % (
+                                test.__name__, arg_string(test, args, kwargs)))
+
+                    if self.collector is None or not collect:
+                        return test(*args, **kwargs)
+                    else:  # pragma: no cover
+                        try:
+                            self.collector.start()
+                            return test(*args, **kwargs)
+                        finally:
+                            self.collector.stop()
+
+        result = self.test_runner(data, run)
+        if expected_failure is not None:
+            exception, traceback = expected_failure
+            if (
+                isinstance(
+                    exception,
+                    DeadlineExceeded
+                ) and self.__test_runtime is not None
+            ):
+                report((
+                    'Unreliable test timings! On an initial run, this '
+                    'test took %.2fms, which exceeded the deadline of '
+                    '%.2fms, but on a subsequent run it took %.2f ms, '
+                    'which did not. If you expect this sort of '
+                    'variability in your test timings, consider turning '
+                    'deadlines off for this test by setting deadline=None.'
+                ) % (
+                    exception.runtime,
+                    self.settings.deadline, self.__test_runtime
+                ))
+            else:
+                report(
+                    'Failed to reproduce exception. Expected: \n' +
+                    traceback,
+                )
+            self.__flaky((
+                'Hypothesis %s(%s) produces unreliable results: Falsified'
+                ' on the first call but did not on a subsequent one'
+            ) % (test.__name__, text_repr[0],))
+        return result
 
     def should_trace(self, original_filename, frame):  # pragma: no cover
         disp = FileDisposition()
@@ -555,9 +574,7 @@ def save_data(covdata):
     def evaluate_test_data(self, data):
         try:
             if self.collector is None:
-                result = self.test_runner(data, reify_and_execute(
-                    self.search_strategy, self.test,
-                ))
+                result = self.execute(data)
             else:  # pragma: no cover
                 # This should always be a no-op, but the coverage tracer has
                 # a bad habit of resurrecting itself.
@@ -565,10 +582,7 @@ def evaluate_test_data(self, data):
                 sys.settrace(None)
                 try:
                     self.collector.data = {}
-                    result = self.test_runner(data, reify_and_execute(
-                        self.search_strategy, self.test,
-                        collector=self.collector
-                    ))
+                    result = self.execute(data, collect=True)
                 finally:
                     sys.settrace(original)
                     covdata = CoverageData()
@@ -686,18 +700,18 @@ def run(self):
 
         flaky = 0
 
-        self.__in_final_replay = True
-
         for falsifying_example in self.falsifying_examples:
             self.__was_flaky = False
+            assert falsifying_example.__expected_exception is not None
             try:
-                with self.settings:
-                    self.test_runner(
-                        ConjectureData.for_buffer(falsifying_example.buffer),
-                        reify_and_execute(
-                            self.search_strategy, self.test,
-                            print_example=True, is_final=True
-                        ))
+                self.execute(
+                    ConjectureData.for_buffer(falsifying_example.buffer),
+                    print_example=True, is_final=True,
+                    expected_failure=(
+                        falsifying_example.__expected_exception,
+                        falsifying_example.__expected_traceback,
+                    )
+                )
             except (UnsatisfiedAssumption, StopTest):
                 report(traceback.format_exc())
                 self.__flaky(
@@ -708,56 +722,6 @@ def run(self):
                 if len(self.falsifying_examples) <= 1:
                     raise
                 report(traceback.format_exc())
-            else:
-                if (
-                    isinstance(
-                        falsifying_example.__expected_exception,
-                        DeadlineExceeded
-                    ) and self.__test_runtime is not None
-                ):
-                    report((
-                        'Unreliable test timings! On an initial run, this '
-                        'test took %.2fms, which exceeded the deadline of '
-                        '%.2fms, but on a subsequent run it took %.2f ms, '
-                        'which did not. If you expect this sort of '
-                        'variability in your test timings, consider turning '
-                        'deadlines off for this test by setting deadline=None.'
-                    ) % (
-                        falsifying_example.__expected_exception.runtime,
-                        self.settings.deadline, self.__test_runtime
-                    ))
-                else:
-                    report(
-                        'Failed to reproduce exception. Expected: \n' +
-                        falsifying_example.__expected_traceback,
-                    )
-
-                filter_message = (
-                    'Unreliable test data: Failed to reproduce a failure '
-                    'and then when it came to recreating the example in '
-                    'order to print the test data with a flaky result '
-                    'the example was filtered out (by e.g. a '
-                    'call to filter in your strategy) when we didn\'t '
-                    'expect it to be.'
-                )
-
-                try:
-                    self.test_runner(
-                        ConjectureData.for_buffer(falsifying_example.buffer),
-                        reify_and_execute(
-                            self.search_strategy,
-                            test_is_flaky(
-                                self.test, self.repr_for_last_exception),
-                            print_example=True, is_final=True
-                        ))
-                except (UnsatisfiedAssumption, StopTest):
-                    self.__flaky(filter_message)
-                except Flaky as e:
-                    if len(self.falsifying_examples) > 1:
-                        self.__flaky(e.args[0])
-                    else:
-                        raise
-
             if self.__was_flaky:
                 flaky += 1
 


Test patch:
diff --git a/tests/cover/test_deadline.py b/tests/cover/test_deadline.py
--- a/tests/cover/test_deadline.py
+++ b/tests/cover/test_deadline.py
@@ -133,3 +133,26 @@ def slow_once(i):
             slow_once()
     assert 'Unreliable test timing' in o.getvalue()
     assert 'took 2' in o.getvalue()
+
+
+@pytest.mark.parametrize('slow_strategy', [False, True])
+@pytest.mark.parametrize('slow_test', [False, True])
+def test_should_only_fail_a_deadline_if_the_test_is_slow(
+    slow_strategy, slow_test
+):
+    s = st.integers()
+    if slow_strategy:
+        s = s.map(lambda x: time.sleep(0.08))
+
+    @settings(deadline=50)
+    @given(st.data())
+    def test(data):
+        data.draw(s)
+        if slow_test:
+            time.sleep(0.1)
+
+    if slow_test:
+        with pytest.raises(DeadlineExceeded):
+            test()
+    else:
+        test()