Repository: HypothesisWorks/hypothesis

Issue:
test_removes_needless_steps fails only under coverage?
```python-traceback
_________________________ test_removes_needless_steps __________________________
Traceback (most recent call last):
  File "/home/travis/build/HypothesisWorks/hypothesis/hypothesis-python/tests/cover/test_stateful.py", line 830, in test_removes_needless_steps
    run_state_machine_as_test(IncorrectDeletion)
  File "/home/travis/build/HypothesisWorks/hypothesis/hypothesis-python/.tox/coverage/lib/python3.6/site-packages/_pytest/python_api.py", line 729, in __exit__
    fail(self.message)
  File "/home/travis/build/HypothesisWorks/hypothesis/hypothesis-python/.tox/coverage/lib/python3.6/site-packages/_pytest/outcomes.py", line 113, in fail
    raise Failed(msg=msg, pytrace=pytrace)
Failed: DID NOT RAISE <class 'AssertionError'>
```

Both [on Pipelines](https://dev.azure.com/HypothesisWorks/Hypothesis/_build/results?buildId=1875&view=logs&jobId=d1a131c4-77ff-5e6b-1472-1f9b988f8070&taskId=9188b80d-c627-5cf4-3df1-73e650170c8f&lineStart=296&lineEnd=297&colStart=1&colEnd=1) and [on Travis](https://travis-ci.org/HypothesisWorks/hypothesis/jobs/513687673#L962).  No idea why, since all the non-coverage runs of the same code passed and the coverage runs passed too before merging :confused: 

I also can't reproduce the problem locally.


Discussion:
~~I was able to see this locally, but only once so far.~~

~~(This was while I was on my #1904 branch, though that's probably coincidence.)~~

I saw it again locally on master.
I haven't been able to reproduce this locally, but it did hit me on #1878 
I've been able to reproduce this several times now, always through `./build.sh check-coverage`.

For me, it seems to be a lot more common than can be explained by ordinary flakiness. Something suspicious is going on.
My guess would be that the culprit is that [we run coverage tests with different settings than the main tests](https://github.com/HypothesisWorks/hypothesis/blob/master/hypothesis-python/tests/common/setup.py#L87), so it's going to be a function of those settings rather than the fact that it's actually running under coverage.

If you want a faster way to reproduce the problem, maybe edit those settings and rerun the test on its own?
I haven't succeeded in reproducing the issue outside of `check-coverage`.

I have found that `check-coverage` fails consistently for me, as long as pytest's `--failed-first` doesn't activate. (To get around this I cancel and re-start `check-coverage`.)

Commenting out `test_flatmap_with_invalid_expand` (#1905) seems to make the failure disappear. This makes no sense on its face, so there seems to be some kind of global-state interference happening somewhere.
If I undo #1906, that also seems to make the failure disappear.

So merging both of those PRs somehow got us into a state where `test_remove_needless_steps` fails consistently under `check-coverage`, despite neither PR being enough to trigger it in isolation.
We've worked around this by just slapping on a `@settings(derandomize=True)`, but it would be nice to understand and fix the root cause at some point.
Indeed.

This has all the hallmarks of an ordinary flaky test, yet under the right circumstances it was failing consistently. If I get a chance, I’d like to figure out what was really happening.
Findings so far:

- The test is indeed generating 10 examples, none of which fail.
- Across multiple runs, the examples generated are *exactly the same*. This is suspicious.
- Across multiple runs, Python's `random.getstate()` is identical at the start of `test_removes_needless_steps`.

My current guess is that one of our tests sets the Python RNG to something deterministic, and then fails to properly restore it. This results in the top-level Python RNG unexpectedly being consistent across multiple runs for the remainder of the test suite.

(This is all assuming that a deterministic top-level Python RNG is unintended.)
Debugging notes:

An important local change I made was to always print the sub-test's output in `test_removes_needless_steps`, regardless of whether it succeeds or fails:

```python
    try:
        with capture_out() as o:
            with pytest.raises(AssertionError):
                run_state_machine_as_test(IncorrectDeletion)
    finally:
        print(o.getvalue())
```

I had to remove `@Settings(derandomize=True)` to undo #1912 and let me keep investigating the bug, and I replaced it with `@Settings(verbosity=Verbosity.debug)` to print details about what was generated.

I removed `-ff` from the coverage line in `tox.ini`.

I was able to print the Python random state inside the test as follows:

```python
import random
print(random.getstate())
```
Related to #1295 then? 
I assume it's something along those lines, though I haven't confirmed it yet.

I'll have to figure out a good way to pinpoint what's responsible for leaking a deterministic RNG state into the rest of the test suite.

(This theory neatly explains why I couldn't reproduce the issue when `-ff` kicked in, or in manual `pytest` even with a reduced `max_examples`.)
The trouble appears to start somewhere in `tests/cover/test_conjecture_engine.py`.
This looks like the culprit:

https://github.com/HypothesisWorks/hypothesis/blob/05623d2cb28ae5f57bab3e917e0cea22bf0b4f15/hypothesis-python/tests/cover/test_conjecture_engine.py#L337-L350
Replacing that with `deterministic_PRNG` didn't seem to make the original failure disappear, so maybe there are multiple culprits?

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -542,8 +542,8 @@ def run(data):
             if not hasattr(data, "can_reproduce_example_from_repr"):
                 data.can_reproduce_example_from_repr = True
             with local_settings(self.settings):
-                with BuildContext(data, is_final=is_final):
-                    with deterministic_PRNG():
+                with deterministic_PRNG():
+                    with BuildContext(data, is_final=is_final):
                         args, kwargs = data.draw(self.search_strategy)
                         if expected_failure is not None:
                             text_repr[0] = arg_string(test, args, kwargs)
@@ -1070,15 +1070,15 @@ def find(
     last_repr = [None]
 
     def template_condition(data):
-        with BuildContext(data):
-            try:
-                data.is_find = True
-                with deterministic_PRNG():
+        with deterministic_PRNG():
+            with BuildContext(data):
+                try:
+                    data.is_find = True
                     result = data.draw(search)
                     data.note(result)
                     success = condition(result)
-            except UnsatisfiedAssumption:
-                data.mark_invalid()
+                except UnsatisfiedAssumption:
+                    data.mark_invalid()
 
         if success:
             successful_examples[0] += 1
@@ -1109,8 +1109,8 @@ def template_condition(data):
         data = ConjectureData.for_buffer(
             list(runner.interesting_examples.values())[0].buffer
         )
-        with BuildContext(data):
-            with deterministic_PRNG():
+        with deterministic_PRNG():
+            with BuildContext(data):
                 return data.draw(search)
     if runner.valid_examples == 0 and (runner.exit_reason != ExitReason.finished):
         raise Unsatisfiable(


Test patch:
diff --git a/hypothesis-python/tests/cover/test_conjecture_engine.py b/hypothesis-python/tests/cover/test_conjecture_engine.py
--- a/hypothesis-python/tests/cover/test_conjecture_engine.py
+++ b/hypothesis-python/tests/cover/test_conjecture_engine.py
@@ -18,7 +18,7 @@
 from __future__ import absolute_import, division, print_function
 
 import re
-from random import Random, seed as seed_random
+from random import Random
 
 import attr
 import pytest
@@ -335,19 +335,19 @@ def x(data):
 
 
 def test_one_dead_branch():
-    seed_random(0)
-    seen = set()
+    with deterministic_PRNG():
+        seen = set()
 
-    @run_to_buffer
-    def x(data):
-        i = data.draw_bytes(1)[0]
-        if i > 0:
-            data.mark_invalid()
-        i = data.draw_bytes(1)[0]
-        if len(seen) < 255:
-            seen.add(i)
-        elif i not in seen:
-            data.mark_interesting()
+        @run_to_buffer
+        def x(data):
+            i = data.draw_bytes(1)[0]
+            if i > 0:
+                data.mark_invalid()
+            i = data.draw_bytes(1)[0]
+            if len(seen) < 255:
+                seen.add(i)
+            elif i not in seen:
+                data.mark_interesting()
 
 
 def test_saves_on_interrupt():
diff --git a/hypothesis-python/tests/cover/test_core.py b/hypothesis-python/tests/cover/test_core.py
--- a/hypothesis-python/tests/cover/test_core.py
+++ b/hypothesis-python/tests/cover/test_core.py
@@ -71,6 +71,7 @@ def test_settings_are_default_in_given(x):
 def test_given_shrinks_pytest_helper_errors():
     final_value = [None]
 
+    @settings(derandomize=True)
     @given(s.integers())
     def inner(x):
         final_value[0] = x
@@ -85,6 +86,7 @@ def inner(x):
 def test_pytest_skip_skips_shrinking():
     values = []
 
+    @settings(derandomize=True)
     @given(s.integers())
     def inner(x):
         values.append(x)
diff --git a/hypothesis-python/tests/cover/test_random_module.py b/hypothesis-python/tests/cover/test_random_module.py
--- a/hypothesis-python/tests/cover/test_random_module.py
+++ b/hypothesis-python/tests/cover/test_random_module.py
@@ -22,9 +22,10 @@
 import pytest
 
 import hypothesis.strategies as st
-from hypothesis import given, register_random, reporting
+from hypothesis import find, given, register_random, reporting
 from hypothesis.errors import InvalidArgument
 from hypothesis.internal import entropy
+from hypothesis.internal.entropy import deterministic_PRNG
 from tests.common.utils import capture_out
 
 
@@ -112,3 +113,31 @@ def test_will_actually_use_the_random_seed(rnd):
     random.seed(rnd.seed)
     assert a == random.randint(0, 100)
     assert b == random.randint(0, 100)
+
+
+def test_given_does_not_pollute_state():
+    with deterministic_PRNG():
+
+        @given(st.random_module())
+        def test(r):
+            pass
+
+        test()
+        state_a = random.getstate()
+
+        test()
+        state_b = random.getstate()
+
+        assert state_a != state_b
+
+
+def test_find_does_not_pollute_state():
+    with deterministic_PRNG():
+
+        find(st.random_module(), lambda r: True)
+        state_a = random.getstate()
+
+        find(st.random_module(), lambda r: True)
+        state_b = random.getstate()
+
+        assert state_a != state_b