Repository: HypothesisWorks/hypothesis

Issue:
Automating keeping documentation examples up to date
I really like that we check documentation examples are up to date in CI. I think it's a great addition to the build and very much want to keep it.

But as it stands it's basically untenable, because it means that every time I make a change to to something low level in Hypothesis that has an impact on the data generation I have to run the following loop:

1. Run "make documentation"
2. Find the one example that it tells me about by hand
3. Manually copy and paste the test output into that one example

In particular if I do something that changes the distribution of `integers()` I have to run this loop several dozen times. This is both slow (the docs build process is ~10 seconds) and incredibly annoying. I'm doing this for #710 at the moment and I basically never want to do it again ever. This makes me want to not touch Hypothesis internals at all, which is pretty counterproductive. 

So in order to get the good without the annoyance, we need some form of automation.

Options I can think of:

1. Some sort of hacky script that parses doctest output and puts it in place.
2. Use [sphinx-autorun](https://pypi.python.org/pypi/sphinx-autorun) instead of doctest. This has the downside that we won't see the output in diffs, and it will make it much harder to detect documentation breakages, but is probably the most viable option.
3. Delete most of the examples of doing this from the documentation so the number of iterations of the loop is less annoying.

I also propose adding an item to the guide to the tune of "to the greatest extent possible, any checks for which the fixes could be automated should be automated".


Discussion:
FWIW pytest uses [regendoc](https://bitbucket.org/pytest-dev/regendoc/src) for this. I'm not sure how suitable this is for projects other than pytest (and it's pretty bare-bones at the moment, check pytest's [doc makefile](https://github.com/pytest-dev/pytest/blob/master/doc/en/Makefile#L15-L44) for how to use it). It seems to work pretty well for pytest though.

cc @RonnyPfannschmidt
As an additional data-point about the shape of what a solution to this should look like, I've just changed something in #710 on the basis of how bad it made the documentation look. This is a strong argument in favour of making the diffs visible in the changelog.

It also means I have to redo all of the bits I've manually fixed so far. :-(
@The-Compiler Interesting. Does this actually do the doctest stuff already? It looks more like it's for keeping version numbers and stuff up to date.
Hmm, right, I don't think it does doctest-like things, it's more intended for `$ invocations` of a tool, which probably isn't your usecase.

See e.g. https://github.com/pytest-dev/pytest/pull/1780/files for a bigger change - it just reruns all invocations and updates the output.
Yeah, the specific use-case is that we have a lot of console examples that are currently run via doctests, but their output necessarily changes when Hypothesis's implementation does.
Initial investigation of rundoc is that it's a bit on the primitive side. In particular there isn't any sort of global setup like we currently do for doctest. OTOH it is also [very small](https://github.com/hobarrera/sphinx-autorun/tree/master/sphinx_autorun) so we could just maintain a patched fork....
Yeah, the current situation is completely untenable I'm afraid. I've just discovered another problem: Because it's relying on the global random number generator, any change in testing has arbitrary knock on effects for other changes in testing.

I would much rather have out of date docs than deal with this for any changes I'm making to Hypothesis, so I'm going to open a pull request disabling doctests until we can resolve this issue. Sorry.
Helpful suggestion from Twitter: https://twitter.com/meejah/status/881545340360376320

Basically if we split out the doctest examples into their own files and include them in sphinx with a literalinclude, we can then run doctest on those files rather than on the documentation.

It's then *much* easier to update those examples. We'd still have to write a small script to do it, but it should be comparatively straightforward I think.
I have a strong preference for option # 1, to catch examples that we deprecate or outright break.  Nobody wants to read the docs and see an example followed by a massive traceback - even if it's honest, fixing the example is better.

I also prefer to keep the examples inline as they are now for ease of editing, so I volunteer ~~as tribute~~ to write that update script I mentioned in #585 when converting them in the first place.  Medium priority, but I assume everyone else is happy to just leave them disabled until I finish (or give up)?
>  I volunteer as tribute to write that update script I mentioned in #585 when converting them in the first place.

We accept your sacrifice. 😈 

More seriously, if you're willing to do it then I agree that this would be the best option. Thanks.

> Medium priority, but I assume everyone else is happy to just leave them disabled until I finish (or give up)?

Yup, fine by me.

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -79,12 +79,14 @@
 from hypothesis import *
 from hypothesis.strategies import *
 # Run deterministically, and don't save examples
+import random
+random.seed(0)
 doctest_settings = settings(database=None, derandomize=True)
 settings.register_profile('doctests', doctest_settings)
 settings.load_profile('doctests')
 # Never show deprecated behaviour in code examples
 import warnings
-warnings.filterwarnings('error', category=HypothesisDeprecationWarning)
+warnings.filterwarnings('error', category=DeprecationWarning)
 '''
 
 # This config value must be a dictionary of external sites, mapping unique
diff --git a/scripts/fix_doctests.py b/scripts/fix_doctests.py
new file mode 100644
--- /dev/null
+++ b/scripts/fix_doctests.py
@@ -0,0 +1,126 @@
+#!/usr/bin/env python3
+
+# coding=utf-8
+#
+# This file is part of Hypothesis, which may be found at
+# https://github.com/HypothesisWorks/hypothesis-python
+#
+# Most of this work is copyright (C) 2013-2017 David R. MacIver
+# (david@drmaciver.com), but it contains contributions by others. See
+# CONTRIBUTING.rst for a full list of people who may hold copyright, and
+# consult the git log if you need to determine who owns an individual
+# contribution.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public License,
+# v. 2.0. If a copy of the MPL was not distributed with this file, You can
+# obtain one at http://mozilla.org/MPL/2.0/.
+#
+# END HEADER
+
+from __future__ import division, print_function, absolute_import
+
+import os
+import re
+import sys
+from subprocess import PIPE, run
+from collections import defaultdict
+
+import hypothesistooling as tools
+
+
+class FailingExample(object):
+
+    def __init__(self, chunk):
+        """Turn a chunk of text into an object representing the test."""
+        location, *lines = [l + '\n' for l in chunk.split('\n') if l.strip()]
+        self.location = location.strip()
+        pattern = 'File "(.+?)", line (\d+?), in .+'
+        file, line = re.match(pattern, self.location).groups()
+        self.file = os.path.join('docs', file)
+        self.line = int(line) + 1
+        got = lines.index('Got:\n')
+        self.expected_lines = lines[lines.index('Expected:\n') + 1:got]
+        self.got_lines = lines[got + 1:]
+        self.checked_ok = None
+        self.adjust()
+
+    @property
+    def indices(self):
+        return slice(self.line, self.line + len(self.expected_lines))
+
+    def adjust(self):
+        with open(self.file) as f:
+            lines = f.readlines()
+        # The raw line number is the first line of *input*, so adjust to
+        # first line of output by skipping lines which start with a prompt
+        while self.line < len(lines):
+            if lines[self.line].strip()[:4] not in ('>>> ', '... '):
+                break
+            self.line += 1
+        # Sadly the filename and line number for doctests in docstrings is
+        # wrong - see https://github.com/sphinx-doc/sphinx/issues/4223
+        # Luckily, we can just cheat because they're all in one file for now!
+        # (good luck if this changes without an upstream fix...)
+        if lines[self.indices] != self.expected_lines:
+            self.file = 'src/hypothesis/strategies.py'
+            with open(self.file) as f:
+                lines = f.readlines()
+            self.line = 0
+            while self.expected_lines[0] in lines:
+                self.line = lines[self.line:].index(self.expected_lines[0])
+                if lines[self.indices] == self.expected_lines:
+                    break
+        # Finally, set the flag for location quality
+        self.checked_ok = lines[self.indices] == self.expected_lines
+
+    def __repr__(self):
+        return '{}\nExpected: {!r:.60}\nGot:      {!r:.60}'.format(
+            self.location, self.expected, self.got)
+
+
+def get_doctest_output():
+    # Return a dict of filename: list of examples, sorted from last to first
+    # so that replacing them in sequence works
+    command = run(['sphinx-build', '-b', 'doctest', 'docs', 'docs/_build'],
+                  stdout=PIPE, stderr=PIPE, encoding='utf-8')
+    output = [FailingExample(c) for c in command.stdout.split('*' * 70)
+              if c.strip().startswith('File "')]
+    if not all(ex.checked_ok for ex in output):
+        broken = '\n'.join(ex.location for ex in output if not ex.checked_ok)
+        print('Could not find some tests:\n' + broken)
+        sys.exit(1)
+    tests = defaultdict(set)
+    for ex in output:
+        tests[ex.file].add(ex)
+    return {fname: sorted(examples, key=lambda x: x.line, reverse=True)
+            for fname, examples in tests.items()}
+
+
+def main():
+    os.chdir(tools.ROOT)
+    failing = get_doctest_output()
+    if not failing:
+        print('All doctests are OK')
+        sys.exit(0)
+    if tools.has_uncommitted_changes('.'):
+        print('Cannot fix doctests in place with uncommited changes')
+        sys.exit(1)
+
+    for fname, examples in failing.items():
+        with open(fname) as f:
+            lines = f.readlines()
+        for ex in examples:
+            lines[ex.indices] = ex.got_lines
+        with open(fname, 'w') as f:
+            f.writelines(lines)
+
+    still_failing = get_doctest_output()
+    if still_failing:
+        print('Fixes failed: script broken or flaky tests.\n', still_failing)
+        sys.exit(1)
+    print('All failing doctests have been fixed.')
+    sys.exit(0)
+
+
+if __name__ == '__main__':
+    main()
diff --git a/src/hypothesis/strategies.py b/src/hypothesis/strategies.py
--- a/src/hypothesis/strategies.py
+++ b/src/hypothesis/strategies.py
@@ -1671,18 +1671,18 @@ def deferred(definition):
     >>> import hypothesis.strategies as st
     >>> x = st.deferred(lambda: st.booleans() | st.tuples(x, x))
     >>> x.example()
-    (False, (False, True))
+    (((False, (True, True)), (False, True)), (True, True))
     >>> x.example()
-    True
+    (True, True)
 
     Mutual recursion also works fine:
 
     >>> a = st.deferred(lambda: st.booleans() | b)
     >>> b = st.deferred(lambda: st.tuples(a, a))
     >>> a.example()
-    (((True, True), False), True)
+    (True, (True, False))
     >>> b.example()
-    (((( False, ((True, False), (True, True))), True), False), True)
+    (False, True)
 
     """
     from hypothesis.searchstrategy.deferred import DeferredStrategy


Test patch: