Repository: HypothesisWorks/hypothesis

Issue:
`random_module()` not working, or am I misunderstanding the documentation?
Hi,

I would like to test a non-deterministic piece of code in a way that'd allow me to reproduce a failure. I thought the `random_module()` strategy would help me do that, but it doesn't seem to work as I expected.

---
#### Example
I would think that the following example would return different values of `a` and `b` for each iteration of the tests. From what I understood, the `random_module()` strategy is supposed to call `random.seed(SEED)` before the test (where `SEED` is actually `rand_mod.seed` in this example).
```python
@given(random_module())
def test_random_module(rand_mod):
    a = random.randint(0, 100)
    b = random.randint(0, 100)
    print(f'a={a} b={b} | Expected seed: {rand_mod.seed} ')
```
But the result looks like this:
```
a=49 b=97 | Expected seed: 16859
a=49 b=97 | Expected seed: 26741
a=49 b=97 | Expected seed: 107
a=49 b=97 | Expected seed: 100
a=49 b=97 | Expected seed: 38947
a=49 b=97 | Expected seed: 14469
a=49 b=97 | Expected seed: 52059
a=49 b=97 | Expected seed: 8055
a=49 b=97 | Expected seed: 180
. . .
```

---
#### Quick Fix
A quick fix seems to be to generate `integers()` and then use that as a seed. I tested it seems it fits the reproducibility criteria. But obviously if I could use the `random_module()` strategy it'd be much more flexible and readable.

```python
@given(integers())
def test_random_module_quick_fix(seed):
    deterministic_random = random.Random(seed)
    with patch('random.randint', deterministic_random.randint):
        a = random.randint(0, 100)
        b = random.randint(0, 100)

        print(f'QF: a={a} b={b} | Expected seed: {seed} ')
```

---
#### Questions
* Is this a bug? Or the intended behavior? 
* If it is the intended behavior, what does the `random_module()` strategy actually do?  
   I couldn't see any difference when using it vs not.

Thanks for the great work on the framework, besides the headache of this afternoon trying to figure out the `random_module()` thing, I found the rest of the framework incredibly simple to use. After a few **minutes** only I was able to significantly shrink down some particularly verbose tests. Great job, that's amazing 😍


Discussion:
#### Quick Fix - UPDATE

I just found a cleaner version of the "quick fix":
```python
@given(randoms())
def test_random_module_quick_fix(hypothesis_random):
    with patch('random.randint', hypothesis_random.randint):
        a = random.randint(0, 100)
        b = random.randint(0, 100)

        print(f'QF: a={a} b={b} | Expected seed: {hypothesis_random.seed} ')
```

But it still doesn't answer the question of whether or not it is normal for the `random_module()` strategy to behave the way it does :)
I don't think you're missing anything! This looks like it's almost certainly a bug in Hypothesis. Just trying to figure out what's going on now. My guess is we broke this with some recent changes to how we handle random state and apparently don't have an adequate test to cover that we actually ever produce different random answers here. 😞 

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/core.py b/hypothesis-python/src/hypothesis/core.py
--- a/hypothesis-python/src/hypothesis/core.py
+++ b/hypothesis-python/src/hypothesis/core.py
@@ -533,26 +533,24 @@ def run(data):
                 with BuildContext(data, is_final=is_final):
                     with deterministic_PRNG():
                         args, kwargs = data.draw(self.search_strategy)
-                    if expected_failure is not None:
-                        text_repr[0] = arg_string(test, args, kwargs)
+                        if expected_failure is not None:
+                            text_repr[0] = arg_string(test, args, kwargs)
 
-                    if print_example:
-                        example = "%s(%s)" % (
-                            test.__name__,
-                            arg_string(test, args, kwargs),
-                        )
-                        try:
-                            ast.parse(example)
-                        except SyntaxError:
-                            data.can_reproduce_example_from_repr = False
-                        report("Falsifying example: %s" % (example,))
-                    elif current_verbosity() >= Verbosity.verbose:
-                        report(
-                            lambda: "Trying example: %s(%s)"
-                            % (test.__name__, arg_string(test, args, kwargs))
-                        )
-
-                    with deterministic_PRNG():
+                        if print_example:
+                            example = "%s(%s)" % (
+                                test.__name__,
+                                arg_string(test, args, kwargs),
+                            )
+                            try:
+                                ast.parse(example)
+                            except SyntaxError:
+                                data.can_reproduce_example_from_repr = False
+                            report("Falsifying example: %s" % (example,))
+                        elif current_verbosity() >= Verbosity.verbose:
+                            report(
+                                lambda: "Trying example: %s(%s)"
+                                % (test.__name__, arg_string(test, args, kwargs))
+                            )
                         return test(*args, **kwargs)
 
         result = self.test_runner(data, run)


Test patch:
diff --git a/hypothesis-python/tests/cover/test_random_module.py b/hypothesis-python/tests/cover/test_random_module.py
--- a/hypothesis-python/tests/cover/test_random_module.py
+++ b/hypothesis-python/tests/cover/test_random_module.py
@@ -103,3 +103,12 @@ def inner(x):
 
     entropy.RANDOMS_TO_MANAGE.remove(r)
     assert r not in entropy.RANDOMS_TO_MANAGE
+
+
+@given(st.random_module())
+def test_will_actually_use_the_random_seed(rnd):
+    a = random.randint(0, 100)
+    b = random.randint(0, 100)
+    random.seed(rnd.seed)
+    assert a == random.randint(0, 100)
+    assert b == random.randint(0, 100)