Repository: HypothesisWorks/hypothesis

Issue:
`test_explore_an_arbitrary_language` is flaky, due to generation bug?
[`test_explore_an_arbitrary_language`, in nocover, is very rarely flaky:](https://dev.azure.com/HypothesisWorks/Hypothesis/_build/results?buildId=1962&view=logs&jobId=e771faf7-decc-5078-b020-68d81ccdda55&taskId=f294e442-b720-52ee-9932-30c03a7f96ef&lineStart=465&lineEnd=495&colStart=1&colEnd=1)

```python-traceback
______________________ test_explore_an_arbitrary_language ______________________
[gw1] linux2 -- Python 2.7.14 /home/vsts/work/1/s/hypothesis-python/.tox/py27-full/bin/python
Traceback (most recent call last):
  File "/home/vsts/work/1/s/hypothesis-python/tests/nocover/test_explore_arbitrary_languages.py", line 119, in test_explore_an_arbitrary_language
    suppress_health_check=HealthCheck.all(),
  File "/home/vsts/work/1/s/hypothesis-python/.tox/py27-full/lib/python2.7/site-packages/hypothesis/core.py", line 987, in wrapped_test
    state.run()
  File "/home/vsts/work/1/s/hypothesis-python/.tox/py27-full/lib/python2.7/site-packages/hypothesis/core.py", line 704, in run
    info.__expected_traceback,
  File "/home/vsts/work/1/s/hypothesis-python/.tox/py27-full/lib/python2.7/site-packages/hypothesis/core.py", line 571, in execute
    result = self.test_runner(data, run)
  File "/home/vsts/work/1/s/hypothesis-python/.tox/py27-full/lib/python2.7/site-packages/hypothesis/executors.py", line 56, in default_new_style_executor
    return function(data)
  File "/home/vsts/work/1/s/hypothesis-python/.tox/py27-full/lib/python2.7/site-packages/hypothesis/core.py", line 569, in run
    return test(*args, **kwargs)
  File "/home/vsts/work/1/s/hypothesis-python/tests/nocover/test_explore_arbitrary_languages.py", line 127, in test_explore_an_arbitrary_language
    run_language_test_for(root, data, seed)
  File "/home/vsts/work/1/s/hypothesis-python/tests/nocover/test_explore_arbitrary_languages.py", line 111, in run_language_test_for
    runner.run()
  File "/home/vsts/work/1/s/hypothesis-python/.tox/py27-full/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 398, in run
    self._run()
  File "/home/vsts/work/1/s/hypothesis-python/.tox/py27-full/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 763, in _run
    self.generate_new_examples()
  File "/home/vsts/work/1/s/hypothesis-python/.tox/py27-full/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 744, in generate_new_examples
    origin = self.target_selector.select()
  File "/home/vsts/work/1/s/hypothesis-python/.tox/py27-full/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 1003, in select
    return self.random.choice(self.used_examples)
  File "/home/vsts/.cache/hypothesis-build-runtimes/python-versions/2.7.14/lib/python2.7/random.py", line 277, in choice
    return seq[int(self.random() * len(seq))]  # raises IndexError if seq is empty
IndexError: list index out of range
---------------------------------- Hypothesis ----------------------------------
Falsifying example: test_explore_an_arbitrary_language(data=data(...))
Draw 1: Branch(bits=11, children={})
Draw 2: 187
Branch(bits=11, children={
    0: Terminal(status=Status.INTERESTING, payload=5),
    64: Terminal(status=Status.INTERESTING, payload=2),
    1891: Terminal(status=Status.INTERESTING, payload=5),
    1894: Terminal(status=Status.INTERESTING, payload=7),
    1575: Terminal(status=Status.INTERESTING, payload=2),
    1224: Terminal(status=Status.INTERESTING, payload=3),
    528: Terminal(status=Status.INTERESTING, payload=2),
    439: Terminal(status=Status.INTERESTING, payload=7),
    1753: Terminal(status=Status.INTERESTING, payload=2),
    1658: Write(value='\x05', child=Terminal(status=Status.INTERESTING, payload=5)),
    543: Terminal(status=Status.INTERESTING, payload=7)
})
```

because in `hypothesis.internal.conjecture.engine`, `TargetSelector.select()` is called before `.add()`.  This can happen if `self.test_function` *always* raises an exception until we move on to the mutation stage.  I think that never moving to the mutation stage before finding an input to mutate is sufficient to fix this:

```diff
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
index 033ebbe..d461232 100644
--- a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
@@ -680,7 +680,9 @@ class ConjectureRunner(object):

         count = 0
         while should_generate_more() and (
-            count < 10 or self.health_check_state is not None
+            count < 10
+            or self.health_check_state is not None
+            or len(self.target_selector) == 0
         ):
             prefix = self.generate_novel_prefix()

```


Discussion:
I think it's less that it's *very* rarely flaky and more that this wasn't possible to trigger before the multi bug discover during generation change (though it's probably still quite hard to hit)
@DRMacIver - do you agree that the patch above would be an appropriate fix?  If so I'll open a PR since it was probably my feature that broke it, but if not I'll need some more hints!

Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
--- a/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
+++ b/hypothesis-python/src/hypothesis/internal/conjecture/engine.py
@@ -680,7 +680,13 @@ def should_generate_more():
 
         count = 0
         while should_generate_more() and (
-            count < 10 or self.health_check_state is not None
+            count < 10
+            or self.health_check_state is not None
+            # If we have not found a valid prefix yet, the target selector will
+            # be empty and the mutation stage will fail with a very rare internal
+            # error.  We therefore continue this initial random generation step
+            # until we have found at least one prefix to mutate.
+            or len(self.target_selector) == 0
         ):
             prefix = self.generate_novel_prefix()
 


Test patch: