Repository: HypothesisWorks/hypothesis

Issue:
stateful testing prints undefined variables
### The Problem
When stateful prints a falsifying example sometimes the example contains variables that are not defined. This means that one cannot simply copy/paste the example into a Python shell to reproduce it.

### Example
```python
import hypothesis.strategies as st
from hypothesis.stateful import (
    Bundle, RuleBasedStateMachine, rule, consumes, multiple
)

class Foo(RuleBasedStateMachine):

    bar = Bundle("bar")

    @rule(target=bar, l=st.lists(st.integers(), 2))
    def step_one(self, l):
        return multiple(*l)

    @rule(n=consumes(bar), m=consumes(bar))
    def step_two(self, n, m):
        assert n + m > 10

TestFoo = Foo.TestCase
```

Running the above test (with `pytest`) I get the following traceback:

```
_________________________________________________ TestFoo.runTest __________________________________________________

self = <hypothesis.stateful.Foo.TestCase testMethod=runTest>

    def runTest(self):
>       run_state_machine_as_test(state_machine_class)

.local/miniconda/lib/python3.7/site-packages/hypothesis/stateful.py:243: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.local/miniconda/lib/python3.7/site-packages/hypothesis/stateful.py:141: in run_state_machine_as_test
    run_state_machine(state_machine_factory)
.local/miniconda/lib/python3.7/site-packages/hypothesis/stateful.py:89: in run_state_machine
    @given(st.data())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Foo({'bar': []}), n = 0, m = 0

    @stateful.rule(n=stateful.consumes(bar), m=stateful.consumes(bar))
    def step_two(self, n, m):
>       assert n + m > 10
E       AssertionError: assert (0 + 0) > 10

poc.py:14: AssertionError
---------------------------------------------------- Hypothesis ----------------------------------------------------
Falsifying example: run_state_machine(factory=Foo, data=data(...))
state = Foo()
v1 = state.step_one(l=[0, 0])
state.step_two(m=v2, n=v1)
state.teardown()
```

Note that the falsifying example at the bottom of the traceback refers to a variable `v2`, which was never defined.

I suspect that this has to do with the fact that `rule_1` in the example returns `stateful.multiple`.



### Extra Information
Python version: `3.7.3`
Hypothesis version: `4.36.2`
Make stateful assign to multiple variables when printing steps that return MultipleResults
If a rule returns MultipleResults then when a step using that rule
is printed the return value should be expanded into multiple
variables. Currently this is not the case

Fixes #2139.


Discussion:
AFAICT the problem is that `RuleBasedStateMachine._print_step` does not know about rules that return `MultipleResult` ([this line here](https://github.com/HypothesisWorks/hypothesis/blob/master/hypothesis-python/src/hypothesis/stateful.py#L777)), but then when executing a step new variable names are assigned to each of the results in the `MultipleResult` [here](https://github.com/HypothesisWorks/hypothesis/blob/master/hypothesis-python/src/hypothesis/stateful.py#L784)
One solution would be to have several variables assigned if a rule returns `MultipleResult`, e.g.

```
v1, v2 = state.step_one()  # Returns MultipleResult
```

A disadvantage of this is that if a large number of results are returned then we get an unwieldy selection of variables.

An alternative could be to allow `MultipleResult` to be indexable, so that the following could be produced:

```
v1 = state.step_one()  # Returns MultipleResult
state.step_two(m=v1[1])
```

The disadvantage of this is that the implementation would be more complicated because the state machine would need to keep track of `MultipleResult`s separately (so that it would know which variable name and index to use), whereas now they are treated on the same footing as single results.
> AFAICT the problem is that RuleBasedStateMachine._print_step does not know about rules that return MultipleResult (this line here), but then when executing a step new variable names are assigned to each of the results in the MultipleResult here

Yup, that's definitely it. Thanks for the bug report!

Changing the definition of `upcoming_name` should be enough to fix this I think.
> The disadvantage of this is that the implementation would be more complicated because the state machine would need to keep track of MultipleResults separately (so that it would know which variable name and index to use), whereas now they are treated on the same footing as single results.

This isn't necessarily as much of an issue as you might be imagining. Internally variable names are just strings, so there's no difficulty in principle with naming one `v1[1]` - they don't have to be valid python identifiers.

For maximum readability at the cost of a slightly more fiddly implementation (we do that a lot...) we could even switch between the two. e.g. if three or fewer results are returned, unpack it, if more than that are returned use indexing.

ETA: We could also not worry about it and just unpack them all actually. Generally what will happen (I think) is that the multiple will be hand-written and thus not too long or generated and thus will tend to get smaller during shrinking so should have a feasible number of variables.


Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/stateful.py b/hypothesis-python/src/hypothesis/stateful.py
--- a/hypothesis-python/src/hypothesis/stateful.py
+++ b/hypothesis-python/src/hypothesis/stateful.py
@@ -43,7 +43,7 @@
 from hypothesis.control import current_build_context
 from hypothesis.core import given
 from hypothesis.errors import InvalidArgument, InvalidDefinition
-from hypothesis.internal.compat import quiet_raise, string_types
+from hypothesis.internal.compat import hrange, quiet_raise, string_types
 from hypothesis.internal.reflection import function_digest, nicerepr, proxies, qualname
 from hypothesis.internal.validation import check_type
 from hypothesis.reporting import current_verbosity, report
@@ -117,9 +117,16 @@ def run_state_machine(factory, data):
 
             while should_continue.more():
                 value = data.conjecture_data.draw(machine.steps())
-                if print_steps:
-                    machine.print_step(value)
-                machine.execute_step(value)
+                # Assign 'result' here in case 'execute_step' fails below
+                result = multiple()
+                try:
+                    result = machine.execute_step(value)
+                finally:
+                    if print_steps:
+                        # 'result' is only used if the step has target bundles.
+                        # If it does, and the result is a 'MultipleResult',
+                        # then 'print_step' prints a multi-variable assignment.
+                        machine.print_step(value, result)
                 machine.check_invariants()
         finally:
             if print_steps:
@@ -190,7 +197,10 @@ def steps(self):
         raise NotImplementedError(u"%r.steps()" % (self,))
 
     def execute_step(self, step):
-        """Execute a step that has been previously drawn from self.steps()"""
+        """Execute a step that has been previously drawn from self.steps()
+
+        Returns the result of the step execution.
+        """
         raise NotImplementedError(u"%r.execute_step()" % (self,))
 
     def print_start(self):
@@ -205,10 +215,10 @@ def print_end(self):
         By default does nothing.
         """
 
-    def print_step(self, step):
+    def print_step(self, step, result):
         """Print a step to the current reporter.
 
-        This is called right before a step is executed.
+        This is called right after a step is executed.
         """
         self.step_count = getattr(self, u"step_count", 0) + 1
         report(u"Step #%d: %s" % (self.step_count, nicerepr(step)))
@@ -682,6 +692,11 @@ def __repr__(self):
     def upcoming_name(self):
         return u"v%d" % (self.name_counter,)
 
+    def last_names(self, n):
+        assert self.name_counter > n
+        count = self.name_counter
+        return [u"v%d" % (i,) for i in hrange(count - n, count)]
+
     def new_name(self):
         result = self.upcoming_name()
         self.name_counter += 1
@@ -777,16 +792,27 @@ def print_start(self):
     def print_end(self):
         report(u"state.teardown()")
 
-    def print_step(self, step):
+    def print_step(self, step, result):
         rule, data = step
         data_repr = {}
         for k, v in data.items():
             data_repr[k] = self.__pretty(v)
         self.step_count = getattr(self, u"step_count", 0) + 1
+        # If the step has target bundles, and the result is a MultipleResults
+        # then we want to assign to multiple variables.
+        if isinstance(result, MultipleResults):
+            n_output_vars = len(result.values)
+        else:
+            n_output_vars = 1
+        output_assignment = (
+            u"%s = " % (", ".join(self.last_names(n_output_vars)),)
+            if rule.targets and n_output_vars >= 1
+            else u""
+        )
         report(
             u"%sstate.%s(%s)"
             % (
-                u"%s = " % (self.upcoming_name(),) if rule.targets else u"",
+                output_assignment,
                 rule.function.__name__,
                 u", ".join(u"%s=%s" % kv for kv in data_repr.items()),
             )
@@ -816,6 +842,7 @@ def execute_step(self, step):
                 self._add_result_to_targets(rule.targets, result)
         if self._initialize_rules_to_run:
             self._initialize_rules_to_run.remove(rule)
+        return result
 
     def check_invariants(self):
         for invar in self.invariants():


Test patch:
diff --git a/hypothesis-python/tests/cover/test_stateful.py b/hypothesis-python/tests/cover/test_stateful.py
--- a/hypothesis-python/tests/cover/test_stateful.py
+++ b/hypothesis-python/tests/cover/test_stateful.py
@@ -388,6 +388,60 @@ def do_not_populate(self):
 TestMachineUsingMultiple = MachineUsingMultiple.TestCase
 
 
+def test_multiple_variables_printed():
+    class ProducesMultiple(RuleBasedStateMachine):
+        b = Bundle("b")
+
+        @initialize(target=b)
+        def populate_bundle(self):
+            return multiple(1, 2)
+
+        @rule()
+        def fail_fast(self):
+            assert False
+
+    with capture_out() as o:
+        # The state machine must raise an exception for the
+        # falsifying example to be printed.
+        with raises(AssertionError):
+            run_state_machine_as_test(ProducesMultiple)
+
+    # This is tightly coupled to the output format of the step printing.
+    # The first line is "Falsifying Example:..." the second is creating
+    # the state machine, the third is calling the "initialize" method.
+    assignment_line = o.getvalue().split("\n")[2]
+    # 'populate_bundle()' returns 2 values, so should be
+    # expanded to 2 variables.
+    assert assignment_line == "v1, v2 = state.populate_bundle()"
+
+
+def test_no_variables_printed():
+    class ProducesNoVariables(RuleBasedStateMachine):
+        b = Bundle("b")
+
+        @initialize(target=b)
+        def populate_bundle(self):
+            return multiple()
+
+        @rule()
+        def fail_fast(self):
+            assert False
+
+    with capture_out() as o:
+        # The state machine must raise an exception for the
+        # falsifying example to be printed.
+        with raises(AssertionError):
+            run_state_machine_as_test(ProducesNoVariables)
+
+    # This is tightly coupled to the output format of the step printing.
+    # The first line is "Falsifying Example:..." the second is creating
+    # the state machine, the third is calling the "initialize" method.
+    assignment_line = o.getvalue().split("\n")[2]
+    # 'populate_bundle()' returns 0 values, so there should be no
+    # variable assignment.
+    assert assignment_line == "state.populate_bundle()"
+
+
 def test_consumes_typecheck():
     with pytest.raises(TypeError):
         consumes(integers())