Repository: HypothesisWorks/hypothesis

Issue:
Unreliable test timings when using data.draw with text strategy
Using `strategies.data` to draw `strategies.text` consistently raises `hypothesis.errors.Flaky` due to `Unreliable test timings` for minimal examples such as `''`

Minimal example:
```python
@given(st.data())
def test_hypothesis_flaky_text(data):
    data.draw(st.text())
    assert 1
```

Produces:
```
============================================== FAILURES ==============================================
_____________________________________ test_hypothesis_flaky_text _____________________________________

    @given(st.data())
>   def test_hypothesis_flaky_text(data):

natural_merge_sort.py:78: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/ben/.local/lib/python3.7/site-packages/hypothesis/core.py:592: in execute
    % (test.__name__, text_repr[0])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <hypothesis.core.StateForActualGivenExecution object at 0x7fbc41de3c88>
message = 'Hypothesis test_hypothesis_flaky_text(data=data(...)) produces unreliable results: Falsified on the first call but did not on a subsequent one'

    def __flaky(self, message):
        if len(self.falsifying_examples) <= 1:
>           raise Flaky(message)
E           hypothesis.errors.Flaky: Hypothesis test_hypothesis_flaky_text(data=data(...)) produces unreliable results: Falsified on the first call but did not on a subsequent one

/home/ben/.local/lib/python3.7/site-packages/hypothesis/core.py:751: Flaky
--------------------------------------------- Hypothesis ---------------------------------------------
Falsifying example: test_hypothesis_flaky_text(data=data(...))
Draw 1: ''
Unreliable test timings! On an initial run, this test took 902.74ms, which exceeded the deadline of 200.00ms, but on a subsequent run it took 0.04 ms, which did not. If you expect this sort of variability in your test timings, consider turning deadlines off for this test by setting deadline=None.
==================================== 1 failed, 1 skipped in 1.00s ====================================
```
Using `strategies.text` directly with `@given` **does not** produce the error. The following passes consistently:
```
@given(st.text())
def test_hypothesis_flaky_text(txt):
    assert 1
```

## System Information
- Python 3.7.3
- Hypothesis 4.36.2
- pytest 5.1.2
- Ubuntu 19.04

Happy to look into a fix if this can be determined to be a real problem and not an isolated issue :+1: 


Discussion:
I think this is a real problem, similar to #911.  I'm not entirely sure why it's happening, but I really appreciate your writeup - thanks!

My guess is that `text()` specifically is doing a lot of work calculating Unicode spans etc. at instantiation, so excluding the draw-time proper isn't really helping.  On the other hand excluding instantiation time from measurement might not help much (and would be intractable in general).  Or perhaps the unreliable timings detection isn't using our draw-aware timer? 

If you want to investigate I'd start by reading through `core.py`, especially the parts touched by #976.  And feel free to ask questions as you go!
> My guess is that text() specifically is doing a lot of work calculating Unicode spans etc. at instantiation, so excluding the draw-time proper isn't really helping. On the other hand excluding instantiation time from measurement might not help much (and would be intractable in general). Or perhaps the unreliable timings detection isn't using our draw-aware timer?

Because of the way strategies are evaluated lazily I think this _shouldn't_ be the case - the load time should be part of the first draw time - but it's possible something is going wrong, or that it's a function of time meaurement on computers being weird.

In general I wonder if we should modify the deadline code to be a little more forgiving of flakiness to cope with this sort of lazy loading of global resources in general - say give the code some small bounded number of deadline-exceeding test runs which it will rerun to see if they're still deadline exceeding and just shrug and ignore them if they're not.
@Zac-HD You're more than welcome. 
That's my initial thinking too; since ithe fact that it's almost instant on the subsequent run suggests that the time is initially spent doing work at instantiation :thinking:  Thanks for the pointers -  I'll take a look at `core.py`.

@DRMacIver I agree that the load time should be part of the first draw time otherwise slow draws due to genuinely complex generation could lead to slow tests that don't trigger the hypothesis deadline. Being more forgiving does sound like a reasonable solution, exactly what the limit of deadline-exceeding runs should be I'm not sure. Would the `Unreliable test timings` error be raised if the deadline wasn't exceeded but the time between runs still varied largely? i.e. 180ms and 0.04ms the first and second runs respectively?
Adding a regression test to the hypothesis test suite has uncovered some more *interesting behaviour* :smile: .

When added as a test to `tests/cover/test_regressions.py`, the example i originally gave would not fail:
```python
# hypothesis/hypothesis-python/tests/cover/test_regressions.py
#... existing tests

@given(st.data())
def test_flaky_text(data):
    data.draw(st.text())
```
```
python -m pytest hypothesis-python/tests/cover/test_regressions.py
======================================= 7 passed in 1.05s =======================================
```
However, moving the test to it's own file *outside* of the hypothesis test suite *does* fail:
```python
# hypothesis/test_flaky_text.py
from hypothesis import given, strategies as st


@given(st.data())
def test_flaky_text(data):
    data.draw(st.text())
```

Furthermore, the same issue appears for `characters`:
```python
# hypothesis/test_flaky_text.py
from hypothesis import given, strategies as st


@given(st.data())
def test_flaky_text(data):
    data.draw(st.characters())
```

**However**, if both the `text` test and `characters` test are in the same test file only the **first** fails:
```python
# hypothesis/test_flaky_text.py
from hypothesis import given, strategies as st

#  fails
@given(st.data())
def test_flaky_text(data):
    data.draw(st.text())

#  passes
@given(st.data())
def test_flaky_text(data):
    data.draw(st.characters())
```
or

```python
# hypothesis/test_flaky_text.py
from hypothesis import given, strategies as st

#  fails
@given(st.data())
def test_flaky_text(data):
    data.draw(st.characters())


#  passes
@given(st.data())
def test_flaky_text(data):
    data.draw(st.text())
```

Setting an increased deadline for the first test only causes *both* to pass:

```python
# hypothesis/test_flaky_text.py
from hypothesis import given, settings, strategies as st

# passes
@settings(deadline=900)
@given(st.data())
def test_flaky_text(data):
    data.draw(st.characters())


# passes
@given(st.data())
def test_flaky_text(data):
    data.draw(st.text())
```

Only the *very first* usage of either `text` or `characters` seems to be subject to the issue; all subsequent uses are ok. Being unable to reproduce when within the existing test suite implies that somewhere during the pytest run the `text` and/or `characters` strategies are being instantiated somehow (even targeting the single test file directly didn't work) perhaps in a `conftest.py` :thinking: 
The slow calls are in [`charmap.py`](https://github.com/HypothesisWorks/hypothesis/blob/master/hypothesis-python/src/hypothesis/internal/charmap.py), within `characters()` but with a couple of caches - we know that calculating all that stuff from unicode chars is slow, so we try to only do it once.  (while not actually a solution, it could probably be made faster by calculating more ahead of time)
Yep, looks like the tests pass when within the test suite because of `charmap.charmap()` begin called during setup: https://github.com/HypothesisWorks/hypothesis/blob/ac24b2f4e98f6ae0bf2860a36f8b2d2b41bcb7cd/hypothesis-python/tests/common/setup.py#L69 .

Adding a call to `charmap.charmap()` at the top of the test file *outside* of the test suite allows the tests to pass:

```python
# hypothesis/test_flaky_text.py
from hypothesis import given, settings, strategies as st
from hypothesis.internal.charmap import charmap

charmap()

# passes
@given(st.data())
def test_flaky_text(data):
    data.draw(st.characters())


# passes
@given(st.data())
def test_flaky_text(data):
    data.draw(st.text())
```

It might be worth creating a deliberately slow on the first invocation dummy strategy to test with. You could do something like:

```python
slow_down_call = True

@composite
def flaky_timing_strategy(draw):
    global slow_down_call
    if slow_down_call:
        time.sleep(1)
        slow_down_call = False
    # dummy draw to make sure we draw some data and don't hit
    # the "this test doesn't actually draw anything" logic
    return draw(integers(0, 10))
```

Using this you can control the logic of whether it's slow independently from the text logic. This doesn't necessarily have the same behaviour as for text, but if it doesn't that is in and of itself interesting information!
Ah yes that's a good idea thank you -  I'll investigate this behaviour and whether it represents the behaviour of text :+1: 
Using a deliberately slow strategy doesn't trigger the error, as the time spent in the draw is correctly factored out in `execute` here": https://github.com/HypothesisWorks/hypothesis/blob/5d2c6557f7400e8f2a9bc8b9ec5c1e93219a6eff/hypothesis-python/src/hypothesis/core.py#L524

**However**, I *think* I've tracked down the issue is to the lazy evaluation of the cached strategy when using `draw` interactively within a test. During the first run, there is always a cache miss in the `@cacheable` decorator (as expected): https://github.com/HypothesisWorks/hypothesis/blob/689b605a307e71064c8bd7738e8fd8eb048c4de7/hypothesis-python/src/hypothesis/_strategies.py#L196 The cached result however, is wrapped in a `LazyStrategy`: https://github.com/HypothesisWorks/hypothesis/blob/689b605a307e71064c8bd7738e8fd8eb048c4de7/hypothesis-python/src/hypothesis/_strategies.py#L221 and has therefore not been evaluated yet. Both the `character` and `text` strategies make use of `OneCharStringStrategy` which calls `charmap.categories` upon initialisation: https://github.com/HypothesisWorks/hypothesis/blob/f7730d8f7561d8427e487ffd5c261c5e519d8969/hypothesis-python/src/hypothesis/searchstrategy/strings.py#L40 This will then call `charmap` as the global `_charmap` isn't initialised yet https://github.com/HypothesisWorks/hypothesis/blob/6c9ef1827f5edcc2a31bdce873e6c967567f1595/hypothesis-python/src/hypothesis/internal/charmap.py#L116. Using `data.draw` means that this initialisation of `charmap` takes place **within the test function** and is therefore included in the test duration.

Using `@given(st.characters())` or `@given(st.text()) directly instead of via `data.draw` doesn't produce the same issue as the initialisation of charmap takes place during the draw phase *before* the test function itself begins and is thus factored out of the test duration as expected.

The only way I can see around this is to either pre-initialise charmap somehow before entering the test function/factor out the time taken to do so or as @DRMacIver suggested being more forgiving with flaky test timings :thinking: 
The thing that's surprising to me is that that the lazy instantiation doesn't happen inside the first `draw` call though. What *should* happen (from memory, I haven't checked the code at the moemnt) is that the `LazyStrategy` instantiates `wrapped_strategy` the first time it is accessed, which should be inside `draw`. Maybe it's being evaluated earlier than that in evaluating a strategy property? e.g. is it possible that our `is_empty` check is outside the excluded period for calculating draw times?
The `wrapped_strategy` is instantiated the first time it's accessed:
https://github.com/HypothesisWorks/hypothesis/blob/2a782cddb5f27c9e7ff7fd64fe07ee3c30e1997f/hypothesis-python/src/hypothesis/searchstrategy/lazy.py#L106

Which does happen during the first `draw` call within the test function. Here's a screenshot from debugging the test run:
![image](https://user-images.githubusercontent.com/14834132/66155195-012eee80-e617-11e9-80c0-12516443e61b.png)
The call stack on the left show's that it's within the test function `test_flaky_draw_characters`. The `is_empty` check is indeed within the `draw` call (the second `draw` stack frame) https://github.com/HypothesisWorks/hypothesis/blob/ce3c734df45fe0f10d8ca9942f23c0aa3fdafc77/hypothesis-python/src/hypothesis/internal/conjecture/data.py#L828 

Because this is within the test function itself I don't think the time for the draw is being excluded.

A bit of good old print statement debugging confirms that the first time the `characters` strategy is being initialised is indeed within the test function:
```python3
# test_regressions.py

@settings(max_examples=1)
@given(st.data())
def test_flaky_draw_characters(data):
    print("test function starts")
    data.draw(st.characters())


# src/hypothesis/internal/conjecture/data.py
class ConjectureData(object)
    def draw(self, strategy, label=None):
        # ....
            print(f"is_empty check for {strategy}")
            if strategy.is_empty:
                self.mark_invalid()
        # ...

# src/hypothesis/internal/charmap.py
def charmap():
    # ...
    global _charmap
    # Best-effort caching in the face of missing files and/or unwritable
    # filesystems is fairly simple: check if loaded, else try loading,
    # else calculate and try writing the cache.
    if _charmap is None:
        print("_charmap initialisation")
        # ...
```

Running the test outputs:
```
collected 1 item

hypothesis-python/tests/cover/test_regressions.py
is_empty check for TupleStrategy((just(()), fixed_dictionaries({'data': data()}).map(lambda args: dict(args, **kwargs))))
is_empty check for just(())
is_empty check for fixed_dictionaries({'data': data()}).map(lambda args: dict(args, **kwargs))
is_empty check for fixed_dictionaries({'data': data()})
is_empty check for FixedKeysDictStrategy(('data',), TupleStrategy((data(),)))
is_empty check for TupleStrategy((data(),))
is_empty check for data()
test function starts
is_empty check for characters()
_charmap initialisation
is_empty check for <hypothesis.searchstrategy.strings.OneCharStringStrategy object at 0x7fd07f296208>
is_empty check for TupleStrategy((just(()), fixed_dictionaries({'data': data()}).map(lambda args: dict(args, **kwargs))))
is_empty check for just(())
is_empty check for fixed_dictionaries({'data': data()}).map(lambda args: dict(args, **kwargs))
is_empty check for fixed_dictionaries({'data': data()})
is_empty check for FixedKeysDictStrategy(('data',), TupleStrategy((data(),)))
is_empty check for TupleStrategy((data(),))
is_empty check for data()
test function starts
is_empty check for characters()
is_empty check for <hypothesis.searchstrategy.strings.OneCharStringStrategy object at 0x7fd07f296208>
is_empty check for TupleStrategy((just(()), fixed_dictionaries({'data': data()}).map(lambda args: dict(args, **kwargs))))
is_empty check for just(())
is_empty check for fixed_dictionaries({'data': data()}).map(lambda args: dict(args, **kwargs))
is_empty check for fixed_dictionaries({'data': data()})
is_empty check for FixedKeysDictStrategy(('data',), TupleStrategy((data(),)))
is_empty check for TupleStrategy((data(),))
is_empty check for data()
test function starts
is_empty check for characters()
is_empty check for <hypothesis.searchstrategy.strings.OneCharStringStrategy object at 0x7fd07f296208>
is_empty check for TupleStrategy((just(()), fixed_dictionaries({'data': data()}).map(lambda args: dict(args, **kwargs))))
is_empty check for just(())
is_empty check for fixed_dictionaries({'data': data()}).map(lambda args: dict(args, **kwargs))
is_empty check for fixed_dictionaries({'data': data()})
is_empty check for FixedKeysDictStrategy(('data',), TupleStrategy((data(),)))
is_empty check for TupleStrategy((data(),))
is_empty check for data()
```
Some more debugging shows that the interactive draw within the test function isn't being added to the excluded draw times. This test 
https://github.com/HypothesisWorks/hypothesis/blob/ce3c734df45fe0f10d8ca9942f23c0aa3fdafc77/hypothesis-python/src/hypothesis/internal/conjecture/data.py#L842-L843 evaluates to `True` when the draw from characters is evaluated within the test function. As such the draw time is not added to `ConjectureData.draw_times` here https://github.com/HypothesisWorks/hypothesis/blob/ce3c734df45fe0f10d8ca9942f23c0aa3fdafc77/hypothesis-python/src/hypothesis/internal/conjecture/data.py#L844-L855 Therefore it is not excluded in `execute` here https://github.com/HypothesisWorks/hypothesis/blob/5d2c6557f7400e8f2a9bc8b9ec5c1e93219a6eff/hypothesis-python/src/hypothesis/core.py#L524

So I think that either the test for `at_top_level` isn't correct for determining whether the draw time should be excluded, or the value of `at_top_level` is being calculated wrong in this case.
> Some more debugging shows that the interactive draw within the test function isn't being added to the excluded draw times. 

Ooh. Good catch. That's definitely a bug.

> So I think that either the test for at_top_level isn't correct for determining whether the draw time should be excluded, or the value of at_top_level is being calculated wrong in this case.

I think it's the former. It's correct that interactive draws are not at the top level, but interactive draws should definitely be excluded from the total time.
Ok excellent, I'll look into how the draw times can be excluded for interactive draws in addition to top level draws :+1: 
I had an epiphany this morning whilst making breakfast and I think there's two bugs here :smile: 

This first is the already discovered fact that the *interactive* draw times are not taken into account when excluding draw times from the test run time (https://github.com/HypothesisWorks/hypothesis/issues/2108#issuecomment-538251172)

The second is that because the draw time is calculated within `ConjectureData.__draw` here:
https://github.com/HypothesisWorks/hypothesis/blob/ce3c734df45fe0f10d8ca9942f23c0aa3fdafc77/hypothesis-python/src/hypothesis/internal/conjecture/data.py#L847
the time spent in it's caller `ConnjectureData.draw` is *not* included https://github.com/HypothesisWorks/hypothesis/blob/ce3c734df45fe0f10d8ca9942f23c0aa3fdafc77/hypothesis-python/src/hypothesis/internal/conjecture/data.py#L834

For most strategies this is not a problem, as the time spent in `ConjectureData.draw` is minimal. However as previously mentioned (https://github.com/HypothesisWorks/hypothesis/issues/2108#issuecomment-538078345) for the characters/text strategies the `is_empty` check https://github.com/HypothesisWorks/hypothesis/blob/ce3c734df45fe0f10d8ca9942f23c0aa3fdafc77/hypothesis-python/src/hypothesis/internal/conjecture/data.py#L828 causes initialisation of `charmap._charmap` which takes a significant amount of time. Since only the internal draw times (those measured by `ConjectureData.__draw`) are factored out of the overall test time here https://github.com/HypothesisWorks/hypothesis/blob/5d2c6557f7400e8f2a9bc8b9ec5c1e93219a6eff/hypothesis-python/src/hypothesis/core.py#L524 the `_charmap` initialisation is included - causing the timeout on the first run.

To validate this hypothesis (pun intended), I put in a quick and dirty hack to start the draw timer in `ConjectureData.draw` instead of `ConjectureData.__draw` and to *always* measure all draw times (not a solution of course, but just to narrow down this issue) and the test passes without the unreliable timings error:
```python
# src/hypothesis/internal/conjecture/data.py

#...
draw_start = None

class ConjectureData:
    # ...
    def draw(self, strategy, label=None):
        #...
        # start timing the draw
        global draw_start
        draw_start = benchmark_time()
        
        # _charmap initialisation will be included in draw time 
        # for characters/text strategy
        if strategy.is_empty:
            self.mark_invalid()

        if self.depth >= MAX_DEPTH:
            self.mark_invalid()

        return self.__draw(strategy, label=label)

    def __draw(self, strategy, label):
        at_top_level = self.depth == 0
        if label is None:
            label = strategy.label
        self.start_example(label=label)

        try:
            if not True: # force all draw times to be measured
                return strategy.do_draw(self)
            else:
                try:
                    strategy.validate()
                    global draw_start
                    start_time = draw_time
                    try:
                        return strategy.do_draw(self)
                    finally:
                        draw_time = benchmark_time() - start_time
                        self.draw_times.append(benchmark_time() - draw_start)
                except BaseException as e:
                    mark_for_escalation(e)
                    raise
        finally:
            self.stop_example()

# src/hypothesis/core.py

class StateForActualGivenExecution(object):
    # ...
    def execute():
    # ...
           @proxies(self.test)
            def test(*args, **kwargs):
                self.__test_runtime = None
                initial_draws = len(data.draw_times)
                start = benchmark_time()
                result = self.test(*args, **kwargs)
                finish = benchmark_time()
                # force all draw times to be taken into account not just 'internal'
                # not a general solution
                internal_draw_time = sum(data.draw_times)
                runtime = datetime.timedelta(
                    seconds=finish - start - internal_draw_time
                )
                self.__test_runtime = runtime
                current_deadline = self.settings.deadline
                if not is_final:
                    current_deadline = (current_deadline // 4) * 5
                if runtime >= current_deadline:
                    raise DeadlineExceeded(runtime, self.settings.deadline)
                return result    
```

So I think the overall solution comes in two main parts:
1. Include the full `ConjectureData.draw` call in the draw timing measurements
2. Include interactive draws when calculating the internal draw time to factor out of the overall test runtime in the proxied `test` method within `StateForActualGivenExecution.execute`

Please do tell me if you think I've misdiagnosed the issue or the proposed (high level) solution isn't correct :smile: 
I've found what looks like a bug related to this, tested in Python 3.6.9. In `hypothesis/internal/charmap.py`:

https://github.com/HypothesisWorks/hypothesis/blob/020df5cfc3d4ee301af799b96a8d3fa254f85ad8/hypothesis-python/src/hypothesis/internal/charmap.py#L60-L65

The cached charmap is opened as a `GzipFile` which is then passed to `json.loads`, which expects a string or byte array, rather than `json.load`, which expects a file pointer. As a result, this generates an exception which is then slightly caught, leading to the generation of a temporary charmap. That involves an expensive loop which iterates from 0 to `sys.maxunicode` (over a million iterations).

This type mismatch occurs on every run, on the first call to something that uses the charmap, regardless of whether a cached file is present or not.


Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/data.py b/hypothesis-python/src/hypothesis/internal/conjecture/data.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/data.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/data.py
@@ -830,16 +830,20 @@ def draw(self, strategy, label=None):
                 % (strategy,)
             )
 
+        at_top_level = self.depth == 0
+        if at_top_level:
+            # We start this timer early, because accessing attributes on a LazyStrategy
+            # can be almost arbitrarily slow.  In cases like characters() and text()
+            # where we cache something expensive, this led to Flaky deadline errors!
+            # See https://github.com/HypothesisWorks/hypothesis/issues/2108
+            start_time = benchmark_time()
+
         if strategy.is_empty:
             self.mark_invalid()
 
         if self.depth >= MAX_DEPTH:
             self.mark_invalid()
 
-        return self.__draw(strategy, label=label)
-
-    def __draw(self, strategy, label):
-        at_top_level = self.depth == 0
         if label is None:
             label = strategy.label
         self.start_example(label=label)
@@ -849,7 +853,6 @@ def __draw(self, strategy, label):
             else:
                 try:
                     strategy.validate()
-                    start_time = benchmark_time()
                     try:
                         return strategy.do_draw(self)
                     finally:


Test patch:
diff --git a/hypothesis-python/tests/cover/test_health_checks.py b/hypothesis-python/tests/cover/test_health_checks.py
--- a/hypothesis-python/tests/cover/test_health_checks.py
+++ b/hypothesis-python/tests/cover/test_health_checks.py
@@ -27,6 +27,7 @@
 from hypothesis.control import assume
 from hypothesis.errors import FailedHealthCheck, InvalidArgument
 from hypothesis.internal.compat import int_from_bytes
+from hypothesis.searchstrategy.lazy import LazyStrategy
 from hypothesis.searchstrategy.strategies import SearchStrategy
 from tests.common.utils import checks_deprecated_behaviour, no_shrink
 
@@ -198,3 +199,24 @@ def test_hung_test_is_deprecated():
 def test_it_is_an_error_to_suppress_non_healthchecks():
     with raises(InvalidArgument):
         settings(suppress_health_check=[1])
+
+
+slow_down_init = True
+
+
+def slow_init_integers(*args, **kwargs):
+    # This mimics st.characters() or st.text(), which perform some
+    # expensive Unicode calculations when the cache is empty.
+    global slow_down_init
+    if slow_down_init:
+        time.sleep(0.5)  # We monkeypatch time, so this is fast
+        slow_down_init = False
+    return st.integers(*args, **kwargs)
+
+
+@given(st.data())
+def test_lazy_slow_initialization_issue_2108_regression(data):
+    # Slow init in strategies wrapped in a LazyStrategy, inside an interactive draw,
+    # should be attributed to drawing from the strategy (not the test function).
+    # Specificially, this used to fail with a DeadlineExceeded error.
+    data.draw(LazyStrategy(slow_init_integers, (), {}))