Repository: HypothesisWorks/hypothesis

Issue:
Allow customization of step runner in run_state_machine_as_test
Hi,

Release 3.84 broke hypothesis-trio (it used to monkey patch `StateMachineRunner.run`, but `StateMachineRunner` is gone now)

This is needed because we must start a trio loop to run the `TrioGenericStateMachine` (given it `execute_step`, `check_invariants` and `teardown` methods are asynchronous)

I think we could clean this hack by having `run_state_machine_as_test` checking if the given state machine contains a custom step runner:

```python
async def run_state_machine_as_test(state_machine_factory, settings=None):
    if settings is None:
        try:
            settings = state_machine_factory.TestCase.settings
            check_type(Settings, settings, "state_machine_factory.TestCase.settings")
        except AttributeError:
            settings = Settings(
                timeout=unlimited,
                deadline=None,
                suppress_health_check=HealthCheck.all(),
            )
    check_type(Settings, settings, "settings")

    @settings
    @given(st.data())
    def run_state_machine(factory, data):
        machine = factory()
        check_type(GenericStateMachine, machine, "state_machine_factory()")
        data.conjecture_data.hypothesis_runner = machine

        n_steps = settings.stateful_step_count
        should_continue = cu.many(
            data.conjecture_data, min_size=1, max_size=n_steps, average_size=n_steps
        )

        print_steps = (
            current_build_context().is_final or current_verbosity() >= Verbosity.debug
        )

        # =================== Changes ================
        def _default_step_runner(data, print_steps, should_continue):
            try:
                if print_steps:
                    machine.print_start()
                machine.check_invariants()

                while should_continue.more():
                    value = data.conjecture_data.draw(machine.steps())
                    if print_steps:
                        machine.print_step(value)
                    machine.execute_step(value)
                    machine.check_invariants()
            finally:
                if print_steps:
                    machine.print_end()
                machine.teardown()

        step_runner = getattr(machine, 'step_runner', _default_step_runner)
        step_runner(data, print_steps, should_continue)
        # =================== End of changes ================

    # Use a machine digest to identify stateful tests in the example database
    run_state_machine.hypothesis.inner_test._hypothesis_internal_add_digest = function_digest(
        state_machine_factory
    )
    # Copy some attributes so @seed and @reproduce_failure "just work"
    run_state_machine._hypothesis_internal_use_seed = getattr(
        state_machine_factory, "_hypothesis_internal_use_seed", None
    )
    run_state_machine._hypothesis_internal_use_reproduce_failure = getattr(
        state_machine_factory, "_hypothesis_internal_use_reproduce_failure", None
    )

    run_state_machine(state_machine_factory)
```

What do you think ? :smiley: 


Discussion:
I think async stateful testing is harder than you think 😨.  [This code doesn't really work...](https://github.com/python-trio/hypothesis-trio/blob/bb769ef57ba905db8223eb2c7e4d0c4392925ead/hypothesis_trio/stateful.py#L100-L114)

```python
try:
    if print_steps:
        self.print_start()
    await self.check_invariants()
    while should_continue.more():
        value = data.draw(self.steps())
        if print_steps:
            self.print_step(value)
        await self.execute_step(value)
        await self.check_invariants()
finally:
    if print_steps:
        self.print_end()
    await self.teardown()
    self._nursery.cancel_scope.cancel()
```

...because the order in which functions execute is really important and Hypothesis can't control or even see it.  Trio [randomises task order on it's event loop](https://github.com/python-trio/trio/blob/master/trio/_core/_run.py#L1424-L1436), which is fine.  Unfortunately it *also* uses an independent `random.Random` instance per `Runner`, so this remains nondeterministic even when [Hypothesis seeds the global `random` module](https://hypothesis.readthedocs.io/en/latest/data.html#hypothesis.strategies.random_module).

(to clarify: Hypothesis can't shrink or even reliably report non-deterministic bugs, because they tend to be so flaky, and that includes everything on top of the non-deterministic scheduler)

So we have two potential solutions, which might end up as two distinct `StateMachine` types:

- Define the `execute_step` and `check_invariants` methods as *sync* functions, which schedule async rule or invariant methods on a class-level event loop and run them to completion.
  Pros: reproducible, clean, compatible with older versions of both Trio and Hypothesis.
  Cons: manually handling the Trio event loop (!?), not very async at all.

- Provide a hook somewhere in Hypothesis to insert an async-to-sync shim in `run_state_machine_as_test`, as we do for direct `@given` tests.  This would also require some changes to Trio so the scheduler can be made deterministic for testing.  Ideally the design would be extensible to injecting delays and maybe other faults as described in python-trio/trio#239 as a future change.
  Pros: Really does interleave tasks from various async rule methods.
  Cons: difficult and unproven approach may not even work.  Still less deterministic.  Invariants may all trigger at the beginning and not be checked after later rules.
I would also remove `TrioGenericStateMachine` in favor of `st.data()` (see #1693), so we only need to fix (or reimplement) `TrioRuleBasedStateMachine`.
I think it would be problematic for Trio to use the global `random` instance (discussed here: https://github.com/python-trio/trio/issues/239#issuecomment-448861272), but it would be easy to provide some way to explicitly seed Trio's random state (e.g. by passing an argument to `trio.run`). Would that be enough? I guess the question would be how to glue things together appropriately. Hypothesis has built-in knowledge of `random`'s seed and `numpy.random`'s seed, so I guess it could have built-in knowledge of trio's seed too, but I'm not sure that's the best approach. pytest-trio [knows when it's handling a hypothesis test](https://github.com/python-trio/pytest-trio/blob/5df0bde47a4a61ca83306830255af8e566c7fd63/pytest_trio/plugin.py#L343-L347), so it could potentially take care of the seeding too. Of course that would only work when using pytest-trio + `@given` on an `async` test, which covers a lot of the cases but I don't know if it covers all of them. In particular, I don't know if it helps with the stateful testing stuff :-)

Alternatively: `random.Random()` pulls entropy from the operating system. Maybe while hypothesis is running, it should monkeypatch that constructor so it can control the seed of the new `Random` object? That wouldn't actually help for Trio right now because we construct a single global `Random` object at import time, but it would be trivial to delay that until the call to `trio.run`.
I think the general problem of "the Trio scheduler is non-deterministic, but we want deterministic or at least reproducible tests" should probably be solved in `trio.testing`!  In particular it affects users who are *not* using Hypothesis, or even `pytest-trio`.

> right now ... we [Trio] construct a single global `Random` object at import time, but it would be trivial to delay that until the call to `trio.run`.

Nope, each call to `trio.run` [creates a new `Random` object](https://github.com/python-trio/trio/blob/3877632bd9b03901134ecae803130c648d582847/trio/_core/_run.py#L674)!  Using a global object which can be monkey-patched by `pytest-trio` and `hypothesis-trio` would significantly improve the reproducibility of scheduler-related heisenbugs.

Long-term, trio may well adopt a deterministic scheduling policy so I'd rather not over-engineer anything right now 😉.  Truly deterministic tests also require that the completion order of concurrent operations (hence tasks available to schedule) be deterministic too.  However I'm happy to leave controlling IO delays and faults for another day!








> Nope, each call to trio.run creates a new Random object!

Oh, we're both right. There's a global Random object (used to give the system clock a random offset), and a per-`run` Random object (used for random scheduling). But system clock is effectively non-deterministic anyway, so I guess that doesn't matter.

> I think the general problem of "the Trio scheduler is non-deterministic, but we want deterministic or at least reproducible tests" should probably be solved in trio.testing! In particular it affects users who are not using Hypothesis, or even pytest-trio.

But if you're not using hypothesis, then you should use the random scheduling, because it's a kind of poor-man's-hypothesis, to make sure your code doesn't depend on scheduling order :-). If the randomness affects your tests then that means you have a bug, and a non-deterministic failure is better than a deterministic pass that hides the bug until it's too late.

Of course when you're using hypothesis that's different: you still want random scheduling order, it's just that you want the randomness to be under hypothesis's control. (And ideally with a custom strategy that picks "weird" schedules to flush out bugs, but we can ignore that for now.)
> If the randomness affects your tests then that means you have a bug, and a non-deterministic failure is better than a deterministic pass that hides the bug until it's too late.

Agreed!  You usually need a pretty good trace or replay capability to understand and fix this kind of bug, but Trio is about as good for that as you can get :smile: 

> Oh, we're both right. [...]

Concretely then, would you be happy with a PR against Trio to make the scheduler use the global `Random`, and then one against `pytest-trio` to monkey-patch that while Hypothesis tests are running?

Note that this would make the scheduler deterministic so that Hypothesis can minimise examples which exhibit observable scheduler-related behaviour, at the cost of hiding the relatively rare scheduling-related bugs that would only be exposed by a different seed.
BTW what we do in hypothesis core which would also work here is to create new `Random` objects explictly seeded by `random.getrandbits()` from the global PRNG. This retains the desired locality of randomness but means that when someone calls `random.seed()` (as Hypothesis does) the system becomes deterministic.
@Zac-HD 
> Concretely then, would you be happy with a PR against Trio to make the scheduler use the global Random, and then one against pytest-trio to monkey-patch that while Hypothesis tests are running?

An argument to `trio.run` might be cleaner than reaching into `trio._core._run` to monkeypatch stuff? But either would probably be fine, really.

@DRMacIver 
> BTW what we do in hypothesis core which would also work here is to create new Random objects explictly seeded by random.getrandbits() from the global PRNG. This retains the desired locality of randomness but means that when someone calls random.seed() (as Hypothesis does) the system becomes deterministic.

This still perturbs the global PRNG though, so if someone has some script where they use `random.seed`, then importing Trio could suddenly change that script's behavior. Possibly my time working with newbie scientists has made me over-paranoid about this :-).
Alright, I think there's two parts to this issue:

First, how the Trio scheduler can be made deterministic for Hypothesis: #1714 and python-trio/pytest-trio#73 between them manage it.  

`hypothesis-trio` will also need to manage the seed (by the same method), since not all users will also be using `pytest-trio`.  If they are it's zero-cost though as registering the same random.Random instance a second time is a no-op.  Deeper control of the scheduler is also possible but beyond the scope of this issue.

Second, I think `hypothesis-trio` will need to have an independent implementation of `run_state_machine_as_test` - aysnc is just too different to sync to share this very short function.  I'd also recommend removing the idea of a generic state machine, as we're planning to in #1693, which should simplify the rule-based option a bit.  You'd then have an async inner function that gets passed to `trio.run` instead of a sync inner function that gets called.

So under my proposal there's not much more to be done in upstream Hypothesis - though I remain happy to help with integrations on the Trio end too 😄 

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/__init__.py b/hypothesis-python/src/hypothesis/__init__.py
--- a/hypothesis-python/src/hypothesis/__init__.py
+++ b/hypothesis-python/src/hypothesis/__init__.py
@@ -27,6 +27,7 @@
 from hypothesis.version import __version_info__, __version__
 from hypothesis.control import assume, note, reject, event
 from hypothesis.core import given, find, example, seed, reproduce_failure, PrintSettings
+from hypothesis.internal.entropy import register_random
 from hypothesis.utils.conventions import infer
 
 
@@ -47,6 +48,7 @@
     "note",
     "event",
     "infer",
+    "register_random",
     "__version__",
     "__version_info__",
 ]
diff --git a/hypothesis-python/src/hypothesis/_strategies.py b/hypothesis-python/src/hypothesis/_strategies.py
--- a/hypothesis-python/src/hypothesis/_strategies.py
+++ b/hypothesis-python/src/hypothesis/_strategies.py
@@ -21,7 +21,6 @@
 import enum
 import math
 import operator
-import random
 import string
 import sys
 from decimal import Context, Decimal, localcontext
@@ -55,6 +54,7 @@
     check_sample,
     integer_range,
 )
+from hypothesis.internal.entropy import get_seeder_and_restorer
 from hypothesis.internal.floats import (
     count_between_floats,
     float_of,
@@ -132,6 +132,7 @@
     numpy = None
 
 if False:
+    import random  # noqa
     from types import ModuleType  # noqa
     from typing import Any, Dict, Union, Sequence, Callable, Pattern  # noqa
     from typing import TypeVar, Tuple, List, Set, FrozenSet, overload  # noqa
@@ -1089,26 +1090,23 @@ def __init__(self, seed):
         self.seed = seed
 
     def __repr__(self):
-        return "random.seed(%r)" % (self.seed,)
+        return "RandomSeeder(%r)" % (self.seed,)
 
 
 class RandomModule(SearchStrategy):
     def do_draw(self, data):
         data.can_reproduce_example_from_repr = False
         seed = data.draw(integers(0, 2 ** 32 - 1))
-        state = random.getstate()
-        random.seed(seed)
-        cleanup(lambda: random.setstate(state))
-        if numpy is not None:  # pragma: no cover
-            npstate = numpy.random.get_state()
-            numpy.random.seed(seed)
-            cleanup(lambda: numpy.random.set_state(npstate))
+        seed_all, restore_all = get_seeder_and_restorer(seed)
+        seed_all()
+        cleanup(restore_all)
         return RandomSeeder(seed)
 
 
 @cacheable
 @defines_strategy
 def random_module():
+    # type: () -> SearchStrategy[RandomSeeder]
     """The Hypothesis engine handles PRNG state for the stdlib and Numpy random
     modules internally, always seeding them to zero and restoring the previous
     state after the test.
diff --git a/hypothesis-python/src/hypothesis/internal/entropy.py b/hypothesis-python/src/hypothesis/internal/entropy.py
--- a/hypothesis-python/src/hypothesis/internal/entropy.py
+++ b/hypothesis-python/src/hypothesis/internal/entropy.py
@@ -20,10 +20,73 @@
 import contextlib
 import random
 
+from hypothesis.errors import InvalidArgument
+from hypothesis.internal.compat import integer_types
+
+RANDOMS_TO_MANAGE = [random]  # type: list
+
 try:
     import numpy.random as npr
 except ImportError:
-    npr = None
+    pass
+else:
+
+    class NumpyRandomWrapper(object):
+        """A shim to remove those darn underscores."""
+
+        seed = npr.seed
+        getstate = npr.get_state
+        setstate = npr.set_state
+
+    RANDOMS_TO_MANAGE.append(NumpyRandomWrapper)
+
+
+def register_random(r):
+    # type: (random.Random) -> None
+    """Register the given Random instance for management by Hypothesis.
+
+    You can pass ``random.Random`` instances (or other objects with seed,
+    getstate, and setstate methods) to ``register_random(r)`` to have their
+    states seeded and restored in the same way as the global PRNGs from the
+    ``random`` and ``numpy.random`` modules.
+
+    All global PRNGs, from e.g. simulation or scheduling frameworks, should
+    be registered to prevent flaky tests.  Hypothesis will ensure that the
+    PRNG state is consistent for all test runs, or reproducibly varied if you
+    choose to use the :func:`~hypothesis.strategies.random_module` strategy.
+    """
+    if not (hasattr(r, "seed") and hasattr(r, "getstate") and hasattr(r, "setstate")):
+        raise InvalidArgument("r=%r does not have all the required methods" % (r,))
+    if r not in RANDOMS_TO_MANAGE:
+        RANDOMS_TO_MANAGE.append(r)
+
+
+def get_seeder_and_restorer(seed=0):
+    """Return a pair of functions which respectively seed all and restore
+    the state of all registered PRNGs.
+
+    This is used by the core engine via `deterministic_PRNG`, and by users
+    via `register_random`.  We support registration of additional random.Random
+    instances (or other objects with seed, getstate, and setstate methods)
+    to force determinism on simulation or scheduling frameworks which avoid
+    using the global random state.  See e.g. #1709.
+    """
+    assert isinstance(seed, integer_types) and 0 <= seed < 2 ** 32
+    states = []  # type: list
+
+    def seed_all():
+        assert not states
+        for r in RANDOMS_TO_MANAGE:
+            states.append(r.getstate())
+            r.seed(seed)
+
+    def restore_all():
+        assert len(states) == len(RANDOMS_TO_MANAGE)
+        for r, state in zip(RANDOMS_TO_MANAGE, states):
+            r.setstate(state)
+        del states[:]
+
+    return seed_all, restore_all
 
 
 @contextlib.contextmanager
@@ -35,15 +98,9 @@ def deterministic_PRNG():
     bad idea in principle, and breaks all kinds of independence assumptions
     in practice.
     """
-    _random_state = random.getstate()
-    random.seed(0)
-    # These branches are covered by tests/numpy/, not tests/cover/
-    if npr is not None:  # pragma: no cover
-        _npr_state = npr.get_state()
-        npr.seed(0)
+    seed_all, restore_all = get_seeder_and_restorer()
+    seed_all()
     try:
         yield
     finally:
-        random.setstate(_random_state)
-        if npr is not None:  # pragma: no cover
-            npr.set_state(_npr_state)
+        restore_all()


Test patch:
diff --git a/hypothesis-python/tests/cover/test_random_module.py b/hypothesis-python/tests/cover/test_random_module.py
--- a/hypothesis-python/tests/cover/test_random_module.py
+++ b/hypothesis-python/tests/cover/test_random_module.py
@@ -22,7 +22,9 @@
 import pytest
 
 import hypothesis.strategies as st
-from hypothesis import given, reporting
+from hypothesis import given, register_random, reporting
+from hypothesis.errors import InvalidArgument
+from hypothesis.internal import entropy
 from tests.common.utils import capture_out
 
 
@@ -36,7 +38,7 @@ def test(r):
                     assert False
 
                 test()
-    assert "random.seed(0)" in out.getvalue()
+    assert "RandomSeeder(0)" in out.getvalue()
 
 
 @given(st.random_module(), st.random_module())
@@ -47,3 +49,57 @@ def test_seed_random_twice(r, r2):
 @given(st.random_module())
 def test_does_not_fail_health_check_if_randomness_is_used(r):
     random.getrandbits(128)
+
+
+def test_cannot_register_non_Random():
+    with pytest.raises(InvalidArgument):
+        register_random("not a Random instance")
+
+
+def test_registering_a_Random_is_idempotent():
+    r = random.Random()
+    register_random(r)
+    register_random(r)
+    assert entropy.RANDOMS_TO_MANAGE.pop() is r
+    assert r not in entropy.RANDOMS_TO_MANAGE
+
+
+def test_manages_registered_Random_instance():
+    r = random.Random()
+    register_random(r)
+    state = r.getstate()
+    result = []
+
+    @given(st.integers())
+    def inner(x):
+        v = r.random()
+        if result:
+            assert v == result[0]
+        else:
+            result.append(v)
+
+    inner()
+    assert state == r.getstate()
+
+    entropy.RANDOMS_TO_MANAGE.remove(r)
+    assert r not in entropy.RANDOMS_TO_MANAGE
+
+
+def test_registered_Random_is_seeded_by_random_module_strategy():
+    r = random.Random()
+    register_random(r)
+    state = r.getstate()
+    results = set()
+    count = [0]
+
+    @given(st.integers())
+    def inner(x):
+        results.add(r.random())
+        count[0] += 1
+
+    inner()
+    assert count[0] > len(results) * 0.9, "too few unique random numbers"
+    assert state == r.getstate()
+
+    entropy.RANDOMS_TO_MANAGE.remove(r)
+    assert r not in entropy.RANDOMS_TO_MANAGE