Repository: HypothesisWorks/hypothesis (Python)

Issue:
`random_module()` not working, or am I misunderstanding the documentation?
Hi,

I would like to test a non-deterministic piece of code in a way that'd allow me to reproduce a failure. I thought the `random_module()` strategy would help me do that, but it doesn't seem to work as I expected.

---
#### Example
I would think that the following example would return different values of `a` and `b` for each iteration of the tests. From what I understood, the `random_module()` strategy is supposed to call `random.seed(SEED)` before the test (where `SEED` is actually `rand_mod.seed` in this example).
```python
@given(random_module())
def test_random_module(rand_mod):
    a = random.randint(0, 100)
    b = random.randint(0, 100)
    print(f'a={a} b={b} | Expected seed: {rand_mod.seed} ')
```
But the result looks like this:
```
a=49 b=97 | Expected seed: 16859
a=49 b=97 | Expected seed: 26741
a=49 b=97 | Expected seed: 107
a=49 b=97 | Expected seed: 100
a=49 b=97 | Expected seed: 38947
a=49 b=97 | Expected seed: 14469
a=49 b=97 | Expected seed: 52059
a=49 b=97 | Expected seed: 8055
a=49 b=97 | Expected seed: 180
. . .
```

---
#### Quick Fix
A quick fix seems to be to generate `integers()` and then use that as a seed. I tested it seems it fits the reproducibility criteria. But obviously if I could use the `random_module()` strategy it'd be much more flexible and readable.

```python
@given(integers())
def test_random_module_quick_fix(seed):
    deterministic_random = random.Random(seed)
    with patch('random.randint', deterministic_random.randint):
        a = random.randint(0, 100)
        b = random.randint(0, 100)

        print(f'QF: a={a} b={b} | Expected seed: {seed} ')
```

---
#### Questions
* Is this a bug? Or the intended behavior? 
* If it is the intended behavior, what does the `random_module()` strategy actually do?  
   I couldn't see any difference when using it vs not.

Thanks for the great work on the framework, besides the headache of this afternoon trying to figure out the `random_module()` thing, I found the rest of the framework incredibly simple to use. After a few **minutes** only I was able to significantly shrink down some particularly verbose tests. Great job, that's amazing 😍


Discussion:
#### Quick Fix - UPDATE

I just found a cleaner version of the "quick fix":
```python
@given(randoms())
def test_random_module_quick_fix(hypothesis_random):
    with patch('random.randint', hypothesis_random.randint):
        a = random.randint(0, 100)
        b = random.randint(0, 100)

        print(f'QF: a={a} b={b} | Expected seed: {hypothesis_random.seed} ')
```

But it still doesn't answer the question of whether or not it is normal for the `random_module()` strategy to behave the way it does :)
I don't think you're missing anything! This looks like it's almost certainly a bug in Hypothesis. Just trying to figure out what's going on now. My guess is we broke this with some recent changes to how we handle random state and apparently don't have an adequate test to cover that we actually ever produce different random answers here. 😞 
Huh. Longer standing issue than I thought it was - this was actually introduced in May last year. Good spot! 
Oh wow! Now that was efficient!
I did try to revert to an earlier version, the one before the `numpy` random seeding was introduced, but had no luck. 

Thanks for the fix @DRMacIver



Patch:
diff --git a/hypothesis-python/RELEASE.rst b/hypothesis-python/RELEASE.rst
new file mode 100644
index 0000000000..cea12d5a11
--- /dev/null
+++ b/hypothesis-python/RELEASE.rst
@@ -0,0 +1,5 @@
+RELEASE_TYPE: patch
+
+This release fixes :issue:`1813`, a bug introduced in :ref:`3.59.1 <v3.59.1>`,
+which caused :py:meth:`~hypothesis.strategies.random_module` to no longer affect the body of the test:
+Although Hypothesis would claim to be seeding the random module in fact tests would always run with a seed of zero.
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
index 86b87a387c..6ef2650fe7 100644
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -533,26 +533,24 @@ def run(data):
                 with BuildContext(data, is_final=is_final):
                     with deterministic_PRNG():
                         args, kwargs = data.draw(self.search_strategy)
-                    if expected_failure is not None:
-                        text_repr[0] = arg_string(test, args, kwargs)
+                        if expected_failure is not None:
+                            text_repr[0] = arg_string(test, args, kwargs)
 
-                    if print_example:
-                        example = "%s(%s)" % (
-                            test.__name__,
-                            arg_string(test, args, kwargs),
-                        )
-                        try:
-                            ast.parse(example)
-                        except SyntaxError:
-                            data.can_reproduce_example_from_repr = False
-                        report("Falsifying example: %s" % (example,))
-                    elif current_verbosity() >= Verbosity.verbose:
-                        report(
-                            lambda: "Trying example: %s(%s)"
-                            % (test.__name__, arg_string(test, args, kwargs))
-                        )
-
-                    with deterministic_PRNG():
+                        if print_example:
+                            example = "%s(%s)" % (
+                                test.__name__,
+                                arg_string(test, args, kwargs),
+                            )
+                            try:
+                                ast.parse(example)
+                            except SyntaxError:
+                                data.can_reproduce_example_from_repr = False
+                            report("Falsifying example: %s" % (example,))
+                        elif current_verbosity() >= Verbosity.verbose:
+                            report(
+                                lambda: "Trying example: %s(%s)"
+                                % (test.__name__, arg_string(test, args, kwargs))
+                            )
                         return test(*args, **kwargs)
 
         result = self.test_runner(data, run)


Test patch:
diff --git a/hypothesis-python/tests/cover/test_random_module.py b/hypothesis-python/tests/cover/test_random_module.py
index 97b262ccf0..391bc8b339 100644
--- a/hypothesis-python/tests/cover/test_random_module.py
+++ b/hypothesis-python/tests/cover/test_random_module.py
@@ -103,3 +103,12 @@ def inner(x):
 
     entropy.RANDOMS_TO_MANAGE.remove(r)
     assert r not in entropy.RANDOMS_TO_MANAGE
+
+
+@given(st.random_module())
+def test_will_actually_use_the_random_seed(rnd):
+    a = random.randint(0, 100)
+    b = random.randint(0, 100)
+    random.seed(rnd.seed)
+    assert a == random.randint(0, 100)
+    assert b == random.randint(0, 100)