Repository: HypothesisWorks/hypothesis

Issue:
Use doctest to check examples in documentation are still valid
Because it's bad when the example code doesn't actually run, eg in #320.
docs: streaming() example raises InvalidArgument
The [Infinite Streams](http://hypothesis.readthedocs.io/en/latest/data.html#infinite-streams) section of the data generation docs contains the following example:

``` python
>>> from hypothesis.strategies import streaming, integers
>>> x = streaming(integers()).example()
>>> x
Stream(...)
```

But if I run the first two incantations in a fresh Python 2.7.11 virtualenv with hypothesis 3.1.3, I instead see the following:

``` python
>>> from hypothesis.strategies import streaming, integers
>>> x = streaming(integers()).example()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/searchstrategy/strategies.py", line 115, in example
    database=None
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/core.py", line 648, in find
    runner.run()
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 168, in run
    self._run()
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 287, in _run
    self.new_buffer()
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 61, in new_buffer
    self.test_function(self.last_data)
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 68, in test_function
    self._test_function(data)
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/core.py", line 614, in template_condition
    result = data.draw(search)
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/internal/conjecture/data.py", line 97, in draw
    ) % (strategy,))
hypothesis.errors.InvalidArgument: Cannot use strategy streaming(integers()) within a call to find (presumably because it would be invalid after the call had ended).
```

The [changelog for 3.0.0](http://hypothesis.readthedocs.io/en/latest/changes.html?highlight=streaming#id10) mentions that:

> - choices and streaming strategies may no longer be used with find().

and sure enough, `example()` is implemented in terms of `find()`. Is there another way to interactively experiment with a strategy that the documentation could suggest here?

The following `streaming()` example, from slightly further down the same section, fails differently:

``` python
>>> streaming(int).example()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/searchstrategy/strategies.py", line 115, in example
    database=None
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/core.py", line 648, in find
    runner.run()
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 168, in run
    self._run()
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 287, in _run
    self.new_buffer()
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 61, in new_buffer
    self.test_function(self.last_data)
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 68, in test_function
    self._test_function(data)
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/core.py", line 614, in template_condition
    result = data.draw(search)
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/internal/conjecture/data.py", line 93, in draw
    if self.is_find and not strategy.supports_find:
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/searchstrategy/deferred.py", line 56, in supports_find
    return self.wrapped_strategy.supports_find
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/searchstrategy/deferred.py", line 68, in wrapped_strategy
    **self.__kwargs
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/strategies.py", line 581, in streaming
    check_strategy(elements)
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/strategies.py", line 1033, in check_strategy
    check_type(SearchStrategy, arg)
  File "/tmp/tmp.AuVdBj9Wer/lib/python2.7/site-packages/hypothesis/strategies.py", line 1029, in check_type
    'Expected %s but got %r' % (typ_string, arg,))
hypothesis.errors.InvalidArgument: Expected SearchStrategy but got <type 'int'>
```



Discussion:
I'm keen on this idea FWIW.
Looks like [`sphinx.ext.doctest`](http://www.sphinx-doc.org/en/stable/ext/doctest.html) would be good for this.  You can provide code to insert in every example too - it's designed for imports, but we can also use it to seed the global random module on a per-block basis so that every example is independent.

- [x] initial setup, enabling ext.doctest and the seeding
- [x] change all the `code-block::` directives to `doctest::` so they're picked up
- [ ] fix all the suddenly failing examples\*
- [ ] add to build so it's checked in CI

\* working around windows shell bugs (turns out the escapes ina unicode repr break ansi handline) and flatmap examples not shrinking
    >>> rectangle_lists = integers(min_value=0, max_value=10).flatmap(
    ... lambda n: lists(lists(integers(), min_size=n, max_size=n)))
    >>> find(rectangle_lists, lambda x: len(x) >= 10)
    [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0]]
    # should be [[], [], [], [], [], [], [], [], [], []]
    >>> find(rectangle_lists, lambda t: sum(len(s) for s in t) >= 10)
    [[0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0]]
    # should be [[0], [0], [0], [0], [0], [0], [0], [0], [0], [0]]

We're not shrinking as well as the docs (`data.rst:229`) say we should be.
Thanks for looking into this. ❤️ 

> We're not shrinking as well as the docs (data.rst:229) say we should be.

Interesting.

I guess I can see why that might have happened. In most ways the shrinking of flatmap has got better since that example was written, but there have been some regressions where the shape doesn't play well with the new model.

I think the first one is a genuine regression, but I'm not at all sure how to fix it. I'll open an issue though. I'd argue that the second one is an example where *both* of them are failing (the correct example is [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]] which neither are finding), and that there's not much to argue in favour of either of them.
>  there's not much to argue in favour of either of them.

Wait, no, that's not true. According to the Hypothesis size heuristics the new version *is* better, just not good enough.
Just wondering if this is a bug in the docs or in hypothesis itself? I get this exception with code like this:

    streaming(sampled_from(['a', 'b', 'c', 'd'])).example()
AFAICT the new behaviour is actually intended. BTW, the link to the relevant section of the changelog is invalid, try this [permanent link](https://github.com/HypothesisWorks/hypothesis-python/blob/0d9219836be6dcd801ca0cedb0fbeec080565b16/docs/changes.rst#300---2016-02-17).

So this would be a bug in the documentation. Of course I relied on the documented behaviour and it would be really helpful to have it back.

In my tests I used strategies.streaming to create execution steps for multiple threads. I take from those streams only when the relevant thread is scheduled. Having many threads, this make the tests run much faster as the steps that are never executed (since the thread is blocked anyway) are not even computed/drawn.

However, because of this, I can no longer use the `example()` method, which proved very helpful in building the composite strategy. Therefore I am now using `strategies.lists()` with a fixed min and max size set to the upper bound of steps the thread could execute, generating many steps that are never used.

So far I am switching between `lists` and `streaming` but this is somewhat tedious and on the other hand I run the risk that I commit the `lists` variant by accident.

So, streaming was a really cool feature for that. Can you elaborate why it is infeasible to get a working implementation as mentioned in the changelog? If I can't have it I would at least like to know why it's impossible ;-)

> However, because of this, I can no longer use the example() method, which proved very helpful in building the composite strategy.

If you are combining strategies, you should use `st.composite` and explicitly draw values from sub-strategies - `.example()` has very poor behaviour for minimising example or reproducing saved failures.

The docs should absolutely be fixed though.

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -36,6 +36,7 @@
 
 extensions = [
     'sphinx.ext.autodoc',
+    'sphinx.ext.doctest',
     'sphinx.ext.viewcode',
     'sphinx.ext.intersphinx',
 ]
@@ -69,6 +70,15 @@
 
 autodoc_mock_imports = ['numpy']
 
+doctest_global_setup = '''
+# Some standard imports
+from hypothesis import *
+from hypothesis.strategies import *
+# Ensure that output (including from strategies) is deterministic
+import random
+random.seed(0)
+'''
+
 
 # -- Options for HTML output ----------------------------------------------
 
diff --git a/src/hypothesis/strategies.py b/src/hypothesis/strategies.py
--- a/src/hypothesis/strategies.py
+++ b/src/hypothesis/strategies.py
@@ -965,6 +965,7 @@ def shared(base, key=None):
     base. Any two shared instances with the same key will share the same value,
     otherwise the identity of this strategy will be used. That is:
 
+    >>> s = integers()  # or any other strategy
     >>> x = shared(s)
     >>> y = shared(s)
 
diff --git a/src/hypothesis/version.py b/src/hypothesis/version.py
--- a/src/hypothesis/version.py
+++ b/src/hypothesis/version.py
@@ -17,5 +17,5 @@
 
 from __future__ import division, print_function, absolute_import
 
-__version_info__ = (3, 8, 0)
+__version_info__ = (3, 8, 1)
 __version__ = '.'.join(map(str, __version_info__))


Test patch: