Repository: HypothesisWorks/hypothesis

Issue:
Flaky tests caused by mishandling of global PRNG state
I've been running into this problem: https://paste.pound-python.org/show/qchV5aFbMkvs0uoYfArd/

This particular test has a `@flaky` decorator but no `@given` decorator. Despite the decorator, the problem was happening 100% reproducibly for me, and upon debugging, I found that the stdlib random singleton was in a deterministic state. Further, when running the test alone (`-k "half and endpoint"`), the test passed, and no longer had determinism.

This is caused by cross-test pollution. The `@given` decorator results in `random.seed(0)` running, so that the prior test causes the next test to be deterministic even though it wouldn't normally.

I'm not sure of the best fix, but this patch fixes the problem for me, and obviates the `@flaky` decorator too:

```diff
--- b/hypothesis-python/tests/common/debug.py
+++ a/tests/common/debug.py
@@ -73,6 +73,10 @@
         definition, condition=None,
         settings=None, random=None
 ):
+    if random is None:
+      from random import seed
+      seed(0)
+
     settings = Settings(
         settings,
         max_examples=10000,
```
Stateful testing doesn't explicitly seed global random
`@given` seeds the global random number generator to zero to minimize flakiness. Stateful testing should do this too but currently doesn't.


Discussion:
I think the proper solution will avoid causing this reseeding pollution in the first place.  For example, a new context manager that can be used in `core.py`:

```python
@contextlib.context_manager
def deterministic_context(seed=0):
    state = random.getstate()
    random.seed(seed)
    try:
        yield
    finally:
        random.setstate(state)
```

And then ban direct use of `random.seed` to avoid similar problems in future.

Note: this is equivalent to building the `random_module()` strategy into every use of `@given`, which seems like a user-interface win to me.
@Zac-HD At that point, I think you'd be happier with an autouse pytest fixture. It's *much* more foolproof, and I feel sure you won't regret it.

In `tests/conftest.py`:
```python
import pytest
import random

@pytest.fixture(autouse=True)
def deterministic_random():
    random.seed(0)
```

If you want to *assert* that seed isn't used, you can take it a bit farther:
```python
import pytest
import random

@pytest.yield_fixture(autouse=True)
def deterministic_random():
    random.seed(0)
    seed = random.seed
    del random.seed
    yield
    random.seed = seed
```

> I think the proper solution will avoid causing this reseeding pollution in the first place.

To be clear, there's no **re**seeding involved in the problem above. The test does no seeding at all, but other tests in the module do, so it has differing behavior when run alone or along with all tests in the module.
I mean that the other tests should not have the side-effect of changing the `random` state, which can be implemented via the decorator I suggested.
Note: fixing this issue will remove the last reason to use the `random_module` strategy, which should therefore be turned into a no-op and deprecated.

Edit: *if* we integrate that into the internal runner mechanisms, which looks impractical.
@Zac-HD Is there a ticket for that idea? How would I generate a pareto-random number in that world? Right now I can use st.randoms() and random.pareto().
This issue is the ticket!  I've assigned myself to make that clearer 😄 

In that world you can use `st.randoms()` and methods on the instance it gives you (as now), *or* just use the global random - Hypothesis will pick a seed for each example and take care of resetting the global random state after each example.
> Note: fixing this issue will remove the last reason to use the random_module strategy, which should therefore be turned into a no-op and deprecated.

Wait, what? It might remove *our* last use of it, but the `random_module` strategy serves a perfectly valid (if somewhat niche) use case. Why would a change about how we test our code remove that?
Sorry, looks like I need to actually write out my chain of logic here 😊 

- The failure is *caused* by the way that every test with `@given` includes a `random.seed(0)` in `core.py`, but never resets the PRNG to its previous state.  This massively degrades the quality of randomness in any test suite using Hypothesis, which is *very bad*.  VERY VERY BAD.  See e.g. #1174 for an example where this has bitten us 😭.  We should fix the general problem rather than patching the tests to avoid it, *especially* because this impacts users too.

- The `random_module()` strategy is a special magic thing that doesn't give you a value (users: "*what???*"), but instead draws an integer seed and does the whole save/seed/resetting thing for you.  Sadly, this only helps if you (a) know it exists, (b) know you need it for a particular test, and (c) remember to use it.

- So we need to do a thing where we save and reset the PRNG state in `@given` (and IIRC `find`, and for state machines - see #702) to avoid polluting the randomness of the rest of the test suite.  If we do this, the damage is contained to each test where `random_module()` should have been used, but wasn't.  🎉 

Before doing some deeper investigation I had hoped it would be practical to integrate the `random_module()` functionality into internals, but adding seeding to the mix makes it a lot harder.
I agree that we should reset the random module state after given rather than letting `random.seed(0)` pollute the global state, and I'd be happy for suggestions for a better API for `random_module` (I agree it's weird), but the `random_module` strategy is there not to clean up state (which it doesn't - the `random.seed(0)` is called beforehand), but to allow people who are testing code which depends on the global random number generator to vary the seed between examples.
I'll open a pull to reset state in a few days when life calms down a little :smile: 

We agree on the actual purpose of `random_module` being to vary the PRNG state between examples.  However, it *does* reset the state in cleanup... right back the the `random.seed(0)` state that we set up earlier, which causes the original problem! https://github.com/HypothesisWorks/hypothesis/blob/d522674b77a90ca02d87293dedebc731497c5b4e/hypothesis-python/src/hypothesis/strategies.py#L1002-L1009
> However, it does reset the state in cleanup... 

Yes, sorry, that's what I meant by "the `random.seed(0)` is called beforehand". i.e. `random_module` does exactly as much state cleanup as without it effectively (less even! At least the state of the system after can vary a bit with normal `@given`)
As I can see [here](https://github.com/HypothesisWorks/hypothesis-python/blob/master/src/hypothesis/core.py#L288) `@given` uses `get_random_for_wrapped_test`. Can you please clarify what this issue is about?
The idea is to similarly seed the random generator in [`hypothesis.stateful`](https://github.com/HypothesisWorks/hypothesis-python/blob/master/src/hypothesis/stateful.py), probably in `find_breaking_runner`, to ensure that failing tests are reproducible even for code that uses the `random` module internally.

Resolving patch:
diff --git a/hypothesis-python/docs/conf.py b/hypothesis-python/docs/conf.py
--- a/hypothesis-python/docs/conf.py
+++ b/hypothesis-python/docs/conf.py
@@ -81,6 +81,7 @@
 from hypothesis.strategies import *
 # Run deterministically, and don't save examples
 import random
+_random_state = random.getstate()
 random.seed(0)
 doctest_settings = settings(database=None, derandomize=True)
 settings.register_profile('doctests', doctest_settings)
@@ -90,6 +91,10 @@
 warnings.filterwarnings('error', category=DeprecationWarning)
 '''
 
+doctest_global_cleanup = '''
+random.setstate(_random_state)
+'''
+
 # This config value must be a dictionary of external sites, mapping unique
 # short alias names to a base URL and a prefix.
 # See http://sphinx-doc.org/ext/extlinks.html
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -56,6 +56,7 @@
 from hypothesis.internal.compat import ceil, hbytes, qualname, \
     str_to_bytes, benchmark_time, get_type_hints, getfullargspec, \
     int_from_bytes, encoded_filepath, bad_django_TestCase
+from hypothesis.internal.entropy import deterministic_PRNG
 from hypothesis.internal.coverage import IN_COVERAGE_TESTS
 from hypothesis.utils.conventions import infer, not_set
 from hypothesis.internal.escalation import is_hypothesis_file, \
@@ -554,8 +555,8 @@ def run(data):
                 data.can_reproduce_example_from_repr = True
             with self.settings:
                 with BuildContext(data, is_final=is_final):
-                    rnd_module.seed(0)
-                    args, kwargs = data.draw(self.search_strategy)
+                    with deterministic_PRNG():
+                        args, kwargs = data.draw(self.search_strategy)
                     if expected_failure is not None:
                         text_repr[0] = arg_string(test, args, kwargs)
 
@@ -573,11 +574,13 @@ def run(data):
                                 test.__name__, arg_string(test, args, kwargs)))
 
                     if self.collector is None or not collect:
-                        return test(*args, **kwargs)
+                        with deterministic_PRNG():
+                            return test(*args, **kwargs)
                     else:  # pragma: no cover
                         try:
                             self.collector.start()
-                            return test(*args, **kwargs)
+                            with deterministic_PRNG():
+                                return test(*args, **kwargs)
                         finally:
                             self.collector.stop()
 
@@ -1118,9 +1121,10 @@ def template_condition(data):
         with BuildContext(data):
             try:
                 data.is_find = True
-                result = data.draw(search)
-                data.note(result)
-                success = condition(result)
+                with deterministic_PRNG():
+                    result = data.draw(search)
+                    data.note(result)
+                    success = condition(result)
             except UnsatisfiedAssumption:
                 data.mark_invalid()
 
@@ -1157,7 +1161,8 @@ def template_condition(data):
         data = ConjectureData.for_buffer(
             list(runner.interesting_examples.values())[0].buffer)
         with BuildContext(data):
-            return data.draw(search)
+            with deterministic_PRNG():
+                return data.draw(search)
     if runner.valid_examples == 0 and (
         runner.exit_reason != ExitReason.finished
     ):
diff --git a/hypothesis-python/src/hypothesis/internal/entropy.py b/hypothesis-python/src/hypothesis/internal/entropy.py
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/src/hypothesis/internal/entropy.py
@@ -0,0 +1,38 @@
+# coding=utf-8
+#
+# This file is part of Hypothesis, which may be found at
+# https://github.com/HypothesisWorks/hypothesis-python
+#
+# Most of this work is copyright (C) 2013-2018 David R. MacIver
+# (david@drmaciver.com), but it contains contributions by others. See
+# CONTRIBUTING.rst for a full list of people who may hold copyright, and
+# consult the git log if you need to determine who owns an individual
+# contribution.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public License,
+# v. 2.0. If a copy of the MPL was not distributed with this file, You can
+# obtain one at http://mozilla.org/MPL/2.0/.
+#
+# END HEADER
+
+from __future__ import division, print_function, absolute_import
+
+import random
+import contextlib
+
+
+@contextlib.contextmanager
+def deterministic_PRNG():
+    """Context manager that handles random.seed without polluting global state.
+
+    See issue #1255 and PR #1295 for details and motivation - in short,
+    leaving the global pseudo-random number generator (PRNG) seeded is a very
+    bad idea in principle, and breaks all kinds of independence assumptions
+    in practice.
+    """
+    _random_state = random.getstate()
+    random.seed(0)
+    try:
+        yield
+    finally:
+        random.setstate(_random_state)


Test patch:
diff --git a/hypothesis-python/tests/cover/test_regressions.py b/hypothesis-python/tests/cover/test_regressions.py
--- a/hypothesis-python/tests/cover/test_regressions.py
+++ b/hypothesis-python/tests/cover/test_regressions.py
@@ -17,6 +17,8 @@
 
 from __future__ import division, print_function, absolute_import
 
+import random
+
 import pytest
 
 from hypothesis import Verbosity, seed, given, assume, settings
@@ -101,3 +103,23 @@ def test_false_is_false(params):
 
     with pytest.raises(ValueError):
         test_false_is_false()
+
+
+@given(st.integers())
+def random_func(x):
+    random.random()
+
+
+def test_prng_state_unpolluted_by_given_issue_1266():
+    # Checks that @given doesn't leave the global PRNG in a particular
+    # modified state; there may be no effect or random effect but not
+    # a consistent end-state.
+    first = random.getstate()
+    random_func()
+    second = random.getstate()
+    random_func()
+    third = random.getstate()
+    if first == second:
+        assert second == third
+    else:
+        assert second != third
diff --git a/hypothesis-python/tests/cover/test_testdecorators.py b/hypothesis-python/tests/cover/test_testdecorators.py
--- a/hypothesis-python/tests/cover/test_testdecorators.py
+++ b/hypothesis-python/tests/cover/test_testdecorators.py
@@ -22,7 +22,7 @@
 from collections import namedtuple
 
 import hypothesis.reporting as reporting
-from hypothesis import Verbosity, HealthCheck, note, seed, given, assume, \
+from hypothesis import Verbosity, HealthCheck, note, given, assume, \
     reject, settings
 from hypothesis.errors import Unsatisfiable
 from tests.common.utils import fails, raises, fails_with, capture_out
@@ -331,21 +331,6 @@ def test_has_ascii(x):
     assert any(c in ascii_characters for c in x)
 
 
-def test_uses_provided_seed():
-    import random
-    random.seed(0)
-    initial = random.getstate()
-
-    @given(integers())
-    @seed(42)
-    @settings(database=None)
-    def test_foo(x):
-        pass
-
-    test_foo()
-    assert hash(repr(random.getstate())) == hash(repr(initial))
-
-
 def test_can_derandomize():
     values = []
 
diff --git a/hypothesis-python/tests/cover/test_unusual_settings_configs.py b/hypothesis-python/tests/cover/test_unusual_settings_configs.py
--- a/hypothesis-python/tests/cover/test_unusual_settings_configs.py
+++ b/hypothesis-python/tests/cover/test_unusual_settings_configs.py
@@ -29,7 +29,7 @@ def test_single_example(n):
 
 @settings(
     max_examples=1, database=None,
-    suppress_health_check=[HealthCheck.filter_too_much],
+    suppress_health_check=[HealthCheck.filter_too_much, HealthCheck.too_slow],
     verbosity=Verbosity.debug,
 )
 @given(st.integers())