Repository: HypothesisWorks/hypothesis

Issue:
`magic()` function to write the entire test, like `hypothesis-auto`
@timothycrosley['s `hypothesis-auto` extension](https://timothycrosley.github.io/hypothesis-auto/) made some waves on social media lately, going a step beyond our existing inference logic for strategies to create your entire test!  As discussed on #2103, we're mutually keen to grow that kind of functionality upstream, so this issue is to track where we're up to and hash out any design issues.

The ideal is to provide **an easy way to start** property-based testing, **and a seamless transition** to our current API once you need it.  Users getting stuck and abandoning PBT altogether would not be a good outcome!

## Proposed API

Minimal changes from the existing downstream API, aimed mostly at making it smaller, easier to teach, and easy to transition off at the end.  By example:

```python
def write_test(func, *, except_=(), use_defaults=False):
    check_argument(callable(func), "Got non-callable func=%r" % (func,))
    if isinstance(except_, tuple):
        for i, e in enumerate(except_):
            check_type(Exception, e, "except_[{}]".format(i))
    else:
        check_type(Exception, e, "except_")
    check_type(bool, use_defaults, "use_defaults")
    ...
```

We could add a `kwargs` argument for a dictionary of strategies, an `oracle` to check that the return value is OK, and perhaps `settings` (more options, but easier transition than `auto_runs_`).

My concern is that over-complicating it encourages people to invest really deeply in the `auto_*` model, and discourages moving to traditional Hypothesis tests.  Perhaps add all arguments, but make it an error to specify more than one of them?  Passing two or three could print source code for an equivalent test, which I'd like to have accessible some other way too...  Not really sure of this part but that's what discussion is for!

## What next?

Timothy has a few other things on the go - including easier `isort` config with `black`, which we'll probably be using soon - but does intend to come back to this soon.  There might be a `@given`-based version downstream first, or we might do it here and add docs+runtime warnings recommending the new upstream version.


Discussion:
Having thought about this some more, I think that creating a function at runtime is the wrong approach - that kind of magic is very hard to learn from, let alone edit and extend.  However, spitting out example source code would answer all my objections: it gives users a solid starting point, educates as well as assists, and there's no difference between the output and any other Hypothesis test.

The module could be called something like `hypothesis.ghostwriter` or `hypothesis.scribe`; I'd want to be clear that the generated code is under CC0 (public domain dedication) but the *generating* code is MPL-2.0 like the rest of Hypothesis.

As a bonus, a test templates give us an easy way to show several common properties and test tactics, from round-trips to simple fuzzing.
nice!
Demo: https://github.com/HypothesisWorks/hypothesis/compare/master...Zac-HD:ghostwriter

The implementation is pretty ugly, but it does spit out tests which usually work and are always a decent starting point for actual development.

CC @timothycrosley FYI - you might have some feedback based on `hypothesis-auto`?  
@HypothesisWorks/hypothesis-python-contributors: a demo, for your viewing pleasure and comments:

```shell
$ hypothesis.ghostwriter fuzz re.compile --except re.error
```

```python
# This test template was produced by the `hypothesis.ghostwriter` module.
import re
from hypothesis import assume, given, strategies as st

# TODO: replace st.nothing() with an appropriate strategy


@given(pattern=st.nothing(), flags=st.just(0))
def test_fuzz_compile(pattern, flags):
    try:
        re.compile(pattern=pattern, flags=flags)
    except re.error:
        assume(False)
```

And if you replace `st.nothing()` with `st.binary()` or `st.text()`, you have a decent test!

---------------

Here's a somewhat longer demo: json parsing is *really* configurable...

```shell
$ hypothesis.ghostwriter roundtrip json.dumps json.loads
```

<details><summary>Ghostwritten json-round-trip test</summary>

```python
# This test template was produced by the `hypothesis.ghostwriter` module.
import json
from hypothesis import assume, given, strategies as st

# TODO: replace st.nothing() with an appropriate strategy


@given(
    allow_nan=st.booleans(),
    check_circular=st.booleans(),
    cls=st.none(),
    default=st.none(),
    ensure_ascii=st.booleans(),
    indent=st.none(),
    obj=st.nothing(),
    separators=st.none(),
    skipkeys=st.booleans(),
    sort_keys=st.booleans(),
    encoding=st.none(),
    object_hook=st.none(),
    object_pairs_hook=st.none(),
    parse_constant=st.none(),
    parse_float=st.none(),
    parse_int=st.none(),
)
def test_roundtrip_dumps_loads(
    allow_nan,
    check_circular,
    cls,
    default,
    ensure_ascii,
    indent,
    obj,
    separators,
    skipkeys,
    sort_keys,
    encoding,
    object_hook,
    object_pairs_hook,
    parse_constant,
    parse_float,
    parse_int,
):
    value0 = json.dumps(
        obj=obj,
        skipkeys=skipkeys,
        ensure_ascii=ensure_ascii,
        check_circular=check_circular,
        allow_nan=allow_nan,
        cls=cls,
        indent=indent,
        separators=separators,
        default=default,
        sort_keys=sort_keys,
    )
    value1 = json.loads(
        s=value0,
        encoding=encoding,
        cls=cls,
        object_hook=object_hook,
        parse_float=parse_float,
        parse_int=parse_int,
        parse_constant=parse_constant,
        object_pairs_hook=object_pairs_hook,
    )
    assert obj == value1
```

(note that output is formatted with Black, if it's available!)

</details>

Admittedly you'll probably want to delete a bunch of these parameters, but *as a template to start from* I think it's better to err on the side of specifying too much - it's much safer to rely on users deleting arguments they don't care about than adding those they should.

--------------

So - your thoughts?  

- Is this worth shipping because: it's useful / because it ~~makes the first hit free~~ lowers perceived barriers to entry / _____
- a bad idea to ship because: it leads new users in the wrong direction / will be a PITA to maintain / doesn't help with any nuanced scenario / _____
- clearly violates the will of the gods because: it works / have you *seen* the implementation / seriously, this is your code on introspection / _____
- The ghostwriters for `fuzz`, `idempotent`, `roundtrip`, and `oracle` are cool, add one for _____ 
-  something else?

Resolving patch:
diff --git a/hypothesis-python/setup.py b/hypothesis-python/setup.py
--- a/hypothesis-python/setup.py
+++ b/hypothesis-python/setup.py
@@ -57,6 +57,8 @@ def local_file(name):
 
 
 extras = {
+    "cli": ["click>=7.0", "black>=19.10b0"],
+    "ghostwriter": ["black>=19.10b0"],
     "pytz": ["pytz>=2014.1"],
     "dateutil": ["python-dateutil>=1.4"],
     "lark": ["lark-parser>=0.6.5"],
@@ -114,7 +116,10 @@ def local_file(name):
         "Topic :: Software Development :: Testing",
         "Typing :: Typed",
     ],
-    entry_points={"pytest11": ["hypothesispytest = hypothesis.extra.pytestplugin"]},
+    entry_points={
+        "pytest11": ["hypothesispytest = hypothesis.extra.pytestplugin"],
+        "console_scripts": ["hypothesis = hypothesis.extra.cli:main"],
+    },
     long_description=open(README).read(),
     long_description_content_type="text/x-rst",
     keywords="python testing fuzzing property-based-testing",
diff --git a/hypothesis-python/src/hypothesis/extra/cli.py b/hypothesis-python/src/hypothesis/extra/cli.py
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/src/hypothesis/extra/cli.py
@@ -0,0 +1,140 @@
+# This file is part of Hypothesis, which may be found at
+# https://github.com/HypothesisWorks/hypothesis/
+#
+# Most of this work is copyright (C) 2013-2020 David R. MacIver
+# (david@drmaciver.com), but it contains contributions by others. See
+# CONTRIBUTING.rst for a full list of people who may hold copyright, and
+# consult the git log if you need to determine who owns an individual
+# contribution.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public License,
+# v. 2.0. If a copy of the MPL was not distributed with this file, You can
+# obtain one at https://mozilla.org/MPL/2.0/.
+#
+# END HEADER
+
+"""
+.. _hypothesis-cli:
+
+----------------
+hypothesis[cli]
+----------------
+
+This module provides Hypothesis' command-line interface, for e.g.
+:doc:`'ghostwriting' tests <ghostwriter>` via the terminal.
+It requires the :pypi:`click` package.
+
+Run :command:`hypothesis --help` in your terminal for details.
+"""
+
+import builtins
+import importlib
+import sys
+from difflib import get_close_matches
+
+from hypothesis.extra import ghostwriter
+
+try:
+    import click
+except ImportError:
+
+    def main():
+        """If `click` is not installed, tell the user to install it then exit."""
+        sys.stderr.write(
+            """
+The Hypothesis command-line interface requires the `click` package,
+which you do not have installed.  Run:
+
+    python -m pip install --upgrade hypothesis[cli]
+
+and try again.
+"""
+        )
+        sys.exit(1)
+
+
+else:
+    # Ensure that Python scripts in the current working directory are importable,
+    # on the principle that Ghostwriter should 'just work' for novice users.  Note
+    # that we append rather than prepend to the module search path, so this will
+    # never shadow the stdlib or installed packages.
+    sys.path.append(".")
+
+    @click.group(context_settings={"help_option_names": ("-h", "--help")})
+    @click.version_option()
+    def main():
+        pass
+
+    def obj_name(s: str) -> object:
+        """This "type" imports whatever object is named by a dotted string."""
+        try:
+            return importlib.import_module(s)
+        except ImportError:
+            pass
+        if "." not in s:
+            modulename, module, funcname = "builtins", builtins, s
+        else:
+            modulename, funcname = s.rsplit(".", 1)
+            try:
+                module = importlib.import_module(modulename)
+            except ImportError:
+                raise click.UsageError(
+                    f"Failed to import the {modulename} module for introspection.  "
+                    "Check spelling and your Python import path, or use the Python API?"
+                )
+        try:
+            return getattr(module, funcname)
+        except AttributeError:
+            public_names = [name for name in vars(module) if not name.startswith("_")]
+            matches = get_close_matches(funcname, public_names)
+            raise click.UsageError(
+                f"Found the {modulename!r} module, but it doesn't have a "
+                f"{funcname!r} attribute."
+                + (f"  Closest matches: {matches!r}" if matches else "")
+            )
+
+    @main.command()  # type: ignore  # Click adds the .command attribute
+    @click.argument("func", type=obj_name, required=True, nargs=-1)
+    @click.option("--idempotent", "writer", flag_value="idempotent")
+    @click.option("--equivalent", "writer", flag_value="equivalent")
+    @click.option("--roundtrip", "writer", flag_value="roundtrip")
+    @click.option(
+        "--style",
+        type=click.Choice(["pytest", "unittest"]),
+        default="pytest",
+        help="pytest-style function, or unittest-style method?",
+    )
+    @click.option(
+        "-e",
+        "--except",
+        "except_",
+        type=obj_name,
+        multiple=True,
+        help="dotted name of exception(s) to ignore",
+    )
+    def write(func, writer, except_, style):  # noqa: D301  # \b disables autowrap
+        """`hypothesis write` writes property-based tests for you!
+
+        Type annotations are helpful but not required for our advanced introspection
+        and templating logic.  Try running the examples below to see how it works:
+
+        \b
+            hypothesis write gzip
+            hypothesis write re.compile --except re.error
+            hypothesis write --style=unittest --idempotent sorted
+            hypothesis write --equivalent ast.literal_eval eval
+            hypothesis write --roundtrip json.dumps json.loads
+        """
+        # NOTE: if you want to call this function from Python, look instead at the
+        # ``hypothesis.extra.ghostwriter`` module.  Click-decorated functions have
+        # a different calling convention, and raise SystemExit instead of returning.
+        if writer is None:
+            writer = "magic"
+        elif writer == "idempotent" and len(func) > 1:
+            raise click.UsageError("Test functions for idempotence one at a time.")
+        elif writer == "roundtrip" and len(func) == 1:
+            writer = "idempotent"
+        elif writer == "equivalent" and len(func) == 1:
+            writer = "fuzz"
+
+        print(getattr(ghostwriter, writer)(*func, except_=except_ or (), style=style))
diff --git a/hypothesis-python/src/hypothesis/extra/ghostwriter.py b/hypothesis-python/src/hypothesis/extra/ghostwriter.py
--- a/hypothesis-python/src/hypothesis/extra/ghostwriter.py
+++ b/hypothesis-python/src/hypothesis/extra/ghostwriter.py
@@ -14,13 +14,37 @@
 # END HEADER
 
 """
-WARNING: this module is under development and should not be used... yet.
-See https://github.com/HypothesisWorks/hypothesis/pull/2344 for progress.
+Writing tests with Hypothesis frees you from the tedium of deciding on and
+writing out specific inputs to test.  Now, the ``hypothesis.extra.ghostwriter``
+module can write your test functions for you too!
+
+The idea is to provide **an easy way to start** property-based testing,
+**and a seamless transition** to more complex test code - because ghostwritten
+tests are source code that you could have written for yourself.
+
+So just pick a function you'd like tested, and feed it to one of the functions
+below or :ref:`our command-line interface <hypothesis-cli>` :command:`hypothesis write -h`!
+They follow imports, use but do not require type annotations, and generally
+do their best to write you a useful test.
+
+.. note::
+
+    The ghostwriter requires Python 3.6+ and :pypi:`black`, but the generated
+    code supports Python 3.5+ and has no dependencies beyond Hypothesis itself.
+
+.. note::
+
+    Legal questions?  While the ghostwriter fragments and logic is under the
+    MPL-2.0 license like the rest of Hypothesis, the *output* from the ghostwriter
+    is made available under the `Creative Commons Zero (CC0)
+    <https://creativecommons.org/share-your-work/public-domain/cc0/>`__
+    public domain dedication, so you can use it without any restrictions.
 """
 
 import builtins
 import enum
 import inspect
+import re
 import sys
 import types
 from collections import OrderedDict
@@ -351,6 +375,26 @@ def _is_probably_ufunc(obj):
     return callable(obj) and all(hasattr(obj, name) for name in has_attributes)
 
 
+# If we have a pair of functions where one name matches the regex and the second
+# is the result of formatting the template with matched groups, our magic()
+# ghostwriter will write a roundtrip test for them.  Additional patterns welcome.
+ROUNDTRIP_PAIRS = (
+    # Defined prefix, shared postfix.  The easy cases.
+    (r"write(.+)", "read{}"),
+    (r"save(.+)", "load{}"),
+    (r"dump(.+)", "load{}"),
+    (r"to(.+)", "from{}"),
+    # Known stem, maybe matching prefixes, maybe matching postfixes.
+    (r"(.*)encode(.*)", "{}decode{}"),
+    # Shared postfix, prefix only on "inverse" function
+    (r"(.+)", "de{}"),
+    (r"(.+)", "un{}"),
+    # a2b_postfix and b2a_postfix.  Not a fan of this pattern, but it's pretty
+    # common in code imitating an C API - see e.g. the stdlib binascii module.
+    (r"(.+)2(.+?)(_.+)?", "{1}2{0}{2}"),
+)
+
+
 def magic(
     *modules_or_functions: Union[Callable, types.ModuleType],
     except_: Except = (),
@@ -366,6 +410,8 @@ def magic(
     After finding the public functions attached to any modules, the ``magic``
     ghostwriter looks for pairs of functions to pass to :func:`~roundtrip`,
     and any others are passed to :func:`~fuzz`.
+
+    For example, try :command:`hypothesis write gzip`!
     """
     except_ = _check_except(except_)
     _check_style(style)
@@ -399,7 +445,20 @@ def magic(
     if len(by_name) < len(functions):
         raise InvalidArgument("Functions to magic() test must have unique names")
 
-    # TODO: identify roundtrip pairs, and write a specific test for each
+    # Look for pairs of functions that roundtrip, based on known naming patterns.
+    for writename, readname in ROUNDTRIP_PAIRS:
+        for name in sorted(by_name):
+            match = re.fullmatch(writename, name)
+            if match:
+                other = readname.format(*match.groups())
+                if other in by_name:
+                    imp, body = _make_roundtrip_body(
+                        (by_name.pop(name), by_name.pop(other)),
+                        except_=except_,
+                        style=style,
+                    )
+                    imports |= imp
+                    parts.append(body)
 
     # For all remaining callables, just write a fuzz-test.  In principle we could
     # guess at equivalence or idempotence; but it doesn't seem accurate enough to
@@ -458,3 +517,133 @@ def test_fuzz_compile(pattern, flags):
         func, test_body=_write_call(func), except_=except_, ghost="fuzz", style=style
     )
     return _make_test(imports, body)
+
+
+def idempotent(func: Callable, *, except_: Except = (), style: str = "pytest") -> str:
+    """Write source code for a property-based test of ``func``.
+
+    The resulting test checks that if you call ``func`` on it's own output,
+    the result does not change.  For example:
+
+    .. code-block:: python
+
+        from typing import Sequence
+        from hypothesis.extra import ghostwriter
+
+        def timsort(seq: Sequence[int]) -> Sequence[int]:
+            return sorted(seq)
+
+        ghostwriter.idempotent(timsort)
+
+    Gives:
+
+    .. code-block:: python
+
+        # This test code was written by the `hypothesis.extra.ghostwriter` module
+        # and is provided under the Creative Commons Zero public domain dedication.
+
+        from hypothesis import given, strategies as st
+
+        @given(seq=st.one_of(st.binary(), st.binary().map(bytearray), st.lists(st.integers())))
+        def test_idempotent_timsort(seq):
+            result = timsort(seq=seq)
+            repeat = timsort(seq=result)
+            assert result == repeat, (result, repeat)
+    """
+    if not callable(func):
+        raise InvalidArgument(f"Got non-callable func={func!r}")
+    except_ = _check_except(except_)
+    _check_style(style)
+
+    test_body = "result = {}\nrepeat = {}\n{}".format(
+        _write_call(func),
+        _write_call(func, "result"),
+        _assert_eq(style, "result", "repeat"),
+    )
+
+    imports, body = _make_test_body(
+        func, test_body=test_body, except_=except_, ghost="idempotent", style=style
+    )
+    return _make_test(imports, body)
+
+
+def _make_roundtrip_body(funcs, except_, style):
+    first_param = next(iter(_get_params(funcs[0])))
+    test_lines = [
+        "value0 = " + _write_call(funcs[0]),
+        *(
+            f"value{i + 1} = " + _write_call(f, f"value{i}")
+            for i, f in enumerate(funcs[1:])
+        ),
+        _assert_eq(style, first_param, f"value{len(funcs) - 1}"),
+    ]
+    return _make_test_body(
+        *funcs,
+        test_body="\n".join(test_lines),
+        except_=except_,
+        ghost="roundtrip",
+        style=style,
+    )
+
+
+def roundtrip(*funcs: Callable, except_: Except = (), style: str = "pytest") -> str:
+    """Write source code for a property-based test of ``funcs``.
+
+    The resulting test checks that if you call the first function, pass the result
+    to the second (and so on), the final result is equal to the first input argument.
+
+    This is a *very* powerful property to test, especially when the config options
+    are varied along with the object to round-trip.  For example, try ghostwriting
+    a test for :func:`python:json.dumps` - would you have thought of all that?
+
+    .. code-block:: shell
+
+        hypothesis write --roundtrip json.dumps json.loads
+    """
+    if not funcs:
+        raise InvalidArgument("Round-trip of zero functions is meaningless.")
+    for i, f in enumerate(funcs):
+        if not callable(f):
+            raise InvalidArgument(f"Got non-callable funcs[{i}]={f!r}")
+    except_ = _check_except(except_)
+    _check_style(style)
+    return _make_test(*_make_roundtrip_body(funcs, except_, style))
+
+
+def equivalent(*funcs: Callable, except_: Except = (), style: str = "pytest") -> str:
+    """Write source code for a property-based test of ``funcs``.
+
+    The resulting test checks that calling each of the functions gives
+    the same result.  This can be used as a classic 'oracle', such as testing
+    a fast sorting algorithm against the :func:`python:sorted` builtin, or
+    for differential testing where none of the compared functions are fully
+    trusted but any difference indicates a bug (e.g. running a function on
+    different numbers of threads, or simply multiple times).
+
+    The functions should have reasonably similar signatures, as only the
+    common parameters will be passed the same arguments - any other parameters
+    will be allowed to vary.
+    """
+    if len(funcs) < 2:
+        raise InvalidArgument("Need at least two functions to compare.")
+    for i, f in enumerate(funcs):
+        if not callable(f):
+            raise InvalidArgument(f"Got non-callable funcs[{i}]={f!r}")
+    except_ = _check_except(except_)
+    _check_style(style)
+
+    var_names = [f"result_{f.__name__}" for f in funcs]
+    if len(set(var_names)) < len(var_names):
+        var_names = [f"result_{i}_{ f.__name__}" for i, f in enumerate(funcs)]
+    test_lines = [
+        vname + " = " + _write_call(f) for vname, f in zip(var_names, funcs)
+    ] + [_assert_eq(style, var_names[0], vname) for vname in var_names[1:]]
+
+    imports, body = _make_test_body(
+        *funcs,
+        test_body="\n".join(test_lines),
+        except_=except_,
+        ghost="equivalent",
+        style=style,
+    )
+    return _make_test(imports, body)


Test patch:
diff --git a/hypothesis-python/tests/ghostwriter/recorded/base64_magic.txt b/hypothesis-python/tests/ghostwriter/recorded/base64_magic.txt
--- a/hypothesis-python/tests/ghostwriter/recorded/base64_magic.txt
+++ b/hypothesis-python/tests/ghostwriter/recorded/base64_magic.txt
@@ -8,101 +8,74 @@ from hypothesis import given, strategies as st
 
 
 @given(
-    b=st.nothing(),
-    foldspaces=st.booleans(),
     adobe=st.booleans(),
-    ignorechars=st.just(b" \t\n\r\x0b"),
-)
-def test_fuzz_a85decode(b, foldspaces, adobe, ignorechars):
-    base64.a85decode(b=b, foldspaces=foldspaces, adobe=adobe, ignorechars=ignorechars)
-
-
-@given(
     b=st.nothing(),
     foldspaces=st.booleans(),
-    wrapcol=st.just(0),
+    ignorechars=st.just(b" \t\n\r\x0b"),
     pad=st.booleans(),
-    adobe=st.booleans(),
+    wrapcol=st.just(0),
 )
-def test_fuzz_a85encode(b, foldspaces, wrapcol, pad, adobe):
-    base64.a85encode(b=b, foldspaces=foldspaces, wrapcol=wrapcol, pad=pad, adobe=adobe)
-
-
-@given(s=st.nothing(), casefold=st.booleans())
-def test_fuzz_b16decode(s, casefold):
-    base64.b16decode(s=s, casefold=casefold)
+def test_roundtrip_a85encode_a85decode(adobe, b, foldspaces, ignorechars, pad, wrapcol):
+    value0 = base64.a85encode(
+        b=b, foldspaces=foldspaces, wrapcol=wrapcol, pad=pad, adobe=adobe
+    )
+    value1 = base64.a85decode(
+        b=value0, foldspaces=foldspaces, adobe=adobe, ignorechars=ignorechars
+    )
+    assert b == value1, (b, value1)
 
 
-@given(s=st.nothing())
-def test_fuzz_b16encode(s):
-    base64.b16encode(s=s)
-
+@given(casefold=st.booleans(), s=st.nothing())
+def test_roundtrip_b16encode_b16decode(casefold, s):
+    value0 = base64.b16encode(s=s)
+    value1 = base64.b16decode(s=value0, casefold=casefold)
+    assert s == value1, (s, value1)
 
-@given(s=st.nothing(), casefold=st.booleans(), map01=st.none())
-def test_fuzz_b32decode(s, casefold, map01):
-    base64.b32decode(s=s, casefold=casefold, map01=map01)
 
-
-@given(s=st.nothing())
-def test_fuzz_b32encode(s):
-    base64.b32encode(s=s)
+@given(casefold=st.booleans(), map01=st.none(), s=st.nothing())
+def test_roundtrip_b32encode_b32decode(casefold, map01, s):
+    value0 = base64.b32encode(s=s)
+    value1 = base64.b32decode(s=value0, casefold=casefold, map01=map01)
+    assert s == value1, (s, value1)
 
 
-@given(s=st.nothing(), altchars=st.none(), validate=st.booleans())
-def test_fuzz_b64decode(s, altchars, validate):
-    base64.b64decode(s=s, altchars=altchars, validate=validate)
-
-
-@given(s=st.nothing(), altchars=st.none())
-def test_fuzz_b64encode(s, altchars):
-    base64.b64encode(s=s, altchars=altchars)
-
-
-@given(b=st.nothing())
-def test_fuzz_b85decode(b):
-    base64.b85decode(b=b)
+@given(altchars=st.none(), s=st.nothing(), validate=st.booleans())
+def test_roundtrip_b64encode_b64decode(altchars, s, validate):
+    value0 = base64.b64encode(s=s, altchars=altchars)
+    value1 = base64.b64decode(s=value0, altchars=altchars, validate=validate)
+    assert s == value1, (s, value1)
 
 
 @given(b=st.nothing(), pad=st.booleans())
-def test_fuzz_b85encode(b, pad):
-    base64.b85encode(b=b, pad=pad)
-
-
-@given(input=st.nothing(), output=st.nothing())
-def test_fuzz_decode(input, output):
-    base64.decode(input=input, output=output)
-
-
-@given(s=st.nothing())
-def test_fuzz_decodebytes(s):
-    base64.decodebytes(s=s)
+def test_roundtrip_b85encode_b85decode(b, pad):
+    value0 = base64.b85encode(b=b, pad=pad)
+    value1 = base64.b85decode(b=value0)
+    assert b == value1, (b, value1)
 
 
 @given(input=st.nothing(), output=st.nothing())
-def test_fuzz_encode(input, output):
-    base64.encode(input=input, output=output)
-
-
-@given(s=st.nothing())
-def test_fuzz_encodebytes(s):
-    base64.encodebytes(s=s)
-
-
-@given(s=st.nothing())
-def test_fuzz_standard_b64decode(s):
-    base64.standard_b64decode(s=s)
+def test_roundtrip_encode_decode(input, output):
+    value0 = base64.encode(input=input, output=output)
+    value1 = base64.decode(input=value0, output=output)
+    assert input == value1, (input, value1)
 
 
 @given(s=st.nothing())
-def test_fuzz_standard_b64encode(s):
-    base64.standard_b64encode(s=s)
+def test_roundtrip_encodebytes_decodebytes(s):
+    value0 = base64.encodebytes(s=s)
+    value1 = base64.decodebytes(s=value0)
+    assert s == value1, (s, value1)
 
 
 @given(s=st.nothing())
-def test_fuzz_urlsafe_b64decode(s):
-    base64.urlsafe_b64decode(s=s)
+def test_roundtrip_standard_b64encode_standard_b64decode(s):
+    value0 = base64.standard_b64encode(s=s)
+    value1 = base64.standard_b64decode(s=value0)
+    assert s == value1, (s, value1)
 
 
 @given(s=st.nothing())
-def test_fuzz_urlsafe_b64encode(s):
-    base64.urlsafe_b64encode(s=s)
+def test_roundtrip_urlsafe_b64encode_urlsafe_b64decode(s):
+    value0 = base64.urlsafe_b64encode(s=s)
+    value1 = base64.urlsafe_b64decode(s=value0)
+    assert s == value1, (s, value1)
diff --git a/hypothesis-python/tests/ghostwriter/recorded/eval_equivalent.txt b/hypothesis-python/tests/ghostwriter/recorded/eval_equivalent.txt
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/tests/ghostwriter/recorded/eval_equivalent.txt
@@ -0,0 +1,16 @@
+# This test code was written by the `hypothesis.extra.ghostwriter` module
+# and is provided under the Creative Commons Zero public domain dedication.
+
+import ast
+from hypothesis import given, strategies as st
+
+# TODO: replace st.nothing() with an appropriate strategy
+
+
+@given(
+    globals=st.none(), locals=st.none(), node_or_string=st.text(), source=st.nothing()
+)
+def test_equivalent_eval_literal_eval(globals, locals, node_or_string, source):
+    result_eval = eval(source, globals, locals)
+    result_literal_eval = ast.literal_eval(node_or_string=node_or_string)
+    assert result_eval == result_literal_eval, (result_eval, result_literal_eval)
diff --git a/hypothesis-python/tests/ghostwriter/recorded/sorted_idempotent.txt b/hypothesis-python/tests/ghostwriter/recorded/sorted_idempotent.txt
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/tests/ghostwriter/recorded/sorted_idempotent.txt
@@ -0,0 +1,15 @@
+# This test code was written by the `hypothesis.extra.ghostwriter` module
+# and is provided under the Creative Commons Zero public domain dedication.
+
+from hypothesis import given, strategies as st
+
+
+@given(
+    iterable=st.one_of(st.iterables(st.integers()), st.iterables(st.text())),
+    key=st.none(),
+    reverse=st.booleans(),
+)
+def test_idempotent_sorted(iterable, key, reverse):
+    result = sorted(iterable, key=key, reverse=reverse)
+    repeat = sorted(result, key=key, reverse=reverse)
+    assert result == repeat, (result, repeat)
diff --git a/hypothesis-python/tests/ghostwriter/recorded/sorted_self_equivalent.txt b/hypothesis-python/tests/ghostwriter/recorded/sorted_self_equivalent.txt
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/tests/ghostwriter/recorded/sorted_self_equivalent.txt
@@ -0,0 +1,17 @@
+# This test code was written by the `hypothesis.extra.ghostwriter` module
+# and is provided under the Creative Commons Zero public domain dedication.
+
+from hypothesis import given, strategies as st
+
+
+@given(
+    iterable=st.one_of(st.iterables(st.integers()), st.iterables(st.text())),
+    key=st.none(),
+    reverse=st.booleans(),
+)
+def test_equivalent_sorted_sorted_sorted(iterable, key, reverse):
+    result_0_sorted = sorted(iterable, key=key, reverse=reverse)
+    result_1_sorted = sorted(iterable, key=key, reverse=reverse)
+    result_2_sorted = sorted(iterable, key=key, reverse=reverse)
+    assert result_0_sorted == result_1_sorted, (result_0_sorted, result_1_sorted)
+    assert result_0_sorted == result_2_sorted, (result_0_sorted, result_2_sorted)
diff --git a/hypothesis-python/tests/ghostwriter/recorded/timsort_idempotent.txt b/hypothesis-python/tests/ghostwriter/recorded/timsort_idempotent.txt
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/tests/ghostwriter/recorded/timsort_idempotent.txt
@@ -0,0 +1,12 @@
+# This test code was written by the `hypothesis.extra.ghostwriter` module
+# and is provided under the Creative Commons Zero public domain dedication.
+
+import test_expected_output
+from hypothesis import given, strategies as st
+
+
+@given(seq=st.one_of(st.binary(), st.binary().map(bytearray), st.lists(st.integers())))
+def test_idempotent_timsort(seq):
+    result = test_expected_output.timsort(seq=seq)
+    repeat = test_expected_output.timsort(seq=result)
+    assert result == repeat, (result, repeat)
diff --git a/hypothesis-python/tests/ghostwriter/test_expected_output.py b/hypothesis-python/tests/ghostwriter/test_expected_output.py
--- a/hypothesis-python/tests/ghostwriter/test_expected_output.py
+++ b/hypothesis-python/tests/ghostwriter/test_expected_output.py
@@ -19,6 +19,7 @@
 To update the recorded outputs, run `pytest --hypothesis-update-outputs ...`.
 """
 
+import ast
 import base64
 import pathlib
 import re
@@ -63,6 +64,10 @@ def a_classmethod(cls, arg: int):
         ),
         ("re_compile_unittest", ghostwriter.fuzz(re.compile, style="unittest")),
         ("base64_magic", ghostwriter.magic(base64)),
+        ("sorted_idempotent", ghostwriter.idempotent(sorted)),
+        ("timsort_idempotent", ghostwriter.idempotent(timsort)),
+        ("eval_equivalent", ghostwriter.equivalent(eval, ast.literal_eval)),
+        ("sorted_self_equivalent", ghostwriter.equivalent(sorted, sorted, sorted)),
     ],
     ids=lambda x: x[0],
 )
diff --git a/hypothesis-python/tests/ghostwriter/test_ghostwriter.py b/hypothesis-python/tests/ghostwriter/test_ghostwriter.py
--- a/hypothesis-python/tests/ghostwriter/test_ghostwriter.py
+++ b/hypothesis-python/tests/ghostwriter/test_ghostwriter.py
@@ -24,7 +24,7 @@
 
 import pytest
 
-from hypothesis.errors import InvalidArgument
+from hypothesis.errors import InvalidArgument, MultipleFailures, Unsatisfiable
 from hypothesis.extra import ghostwriter
 from hypothesis.strategies import from_type, just
 
@@ -142,7 +142,17 @@ def test_no_hashability_filter():
     assert "_can_hash" not in source_code
 
 
-@pytest.mark.parametrize("gw,args", [(ghostwriter.fuzz, ["not callable"])])
+@pytest.mark.parametrize(
+    "gw,args",
+    [
+        (ghostwriter.fuzz, ["not callable"]),
+        (ghostwriter.idempotent, ["not callable"]),
+        (ghostwriter.roundtrip, []),
+        (ghostwriter.roundtrip, ["not callable"]),
+        (ghostwriter.equivalent, [sorted]),
+        (ghostwriter.equivalent, [sorted, "not callable"]),
+    ],
+)
 def test_invalid_func_inputs(gw, args):
     with pytest.raises(InvalidArgument):
         gw(*args)
@@ -176,3 +186,54 @@ def test_exception_deduplication(exceptions, output):
         lambda: None, ghost="", test_body="pass", except_=exceptions, style="pytest"
     )
     assert f"except {output}:" in body
+
+
+def test_run_ghostwriter_roundtrip():
+    # This test covers the whole lifecycle: first, we get the default code.
+    # The first argument is unknown, so we fail to draw from st.nothing()
+    source_code = ghostwriter.roundtrip(json.dumps, json.loads)
+    with pytest.raises(Unsatisfiable):
+        get_test_function(source_code)()
+
+    # Replacing that nothing() with a strategy for JSON allows us to discover
+    # two possible failures: `nan` is not equal to itself, and if dumps is
+    # passed allow_nan=False it is a ValueError to pass a non-finite float.
+    source_code = source_code.replace(
+        "st.nothing()",
+        "st.recursive(st.one_of(st.none(), st.booleans(), st.floats(), st.text()), "
+        "lambda v: st.lists(v, max_size=2) | st.dictionaries(st.text(), v, max_size=2)"
+        ", max_leaves=2)",
+    )
+    try:
+        get_test_function(source_code)()
+    except (AssertionError, ValueError, MultipleFailures):
+        pass
+
+    # Finally, restricting ourselves to finite floats makes the test pass!
+    source_code = source_code.replace(
+        "st.floats()", "st.floats(allow_nan=False, allow_infinity=False)"
+    )
+    get_test_function(source_code)()
+
+
+@varied_excepts
+@pytest.mark.parametrize("func", [sorted, timsort])
+def test_ghostwriter_idempotent(func, ex):
+    source_code = ghostwriter.idempotent(func, except_=ex)
+    test = get_test_function(source_code)
+    if "=st.nothing()" in source_code:
+        with pytest.raises(Unsatisfiable):
+            test()
+    else:
+        test()
+
+
+def test_overlapping_args_use_union_of_strategies():
+    def f(arg: int) -> None:
+        pass
+
+    def g(arg: float) -> None:
+        pass
+
+    source_code = ghostwriter.equivalent(f, g)
+    assert "arg=st.one_of(st.integers(), st.floats())" in source_code
diff --git a/hypothesis-python/tests/ghostwriter/test_ghostwriter_cli.py b/hypothesis-python/tests/ghostwriter/test_ghostwriter_cli.py
new file mode 100644
--- /dev/null
+++ b/hypothesis-python/tests/ghostwriter/test_ghostwriter_cli.py
@@ -0,0 +1,115 @@
+# This file is part of Hypothesis, which may be found at
+# https://github.com/HypothesisWorks/hypothesis/
+#
+# Most of this work is copyright (C) 2013-2020 David R. MacIver
+# (david@drmaciver.com), but it contains contributions by others. See
+# CONTRIBUTING.rst for a full list of people who may hold copyright, and
+# consult the git log if you need to determine who owns an individual
+# contribution.
+#
+# This Source Code Form is subject to the terms of the Mozilla Public License,
+# v. 2.0. If a copy of the MPL was not distributed with this file, You can
+# obtain one at https://mozilla.org/MPL/2.0/.
+#
+# END HEADER
+
+import ast
+import json
+import re
+import subprocess
+
+import pytest
+
+from hypothesis.errors import StopTest
+from hypothesis.extra.ghostwriter import equivalent, fuzz, idempotent, roundtrip
+
+
+@pytest.mark.parametrize(
+    "cli,code",
+    [
+        # Passing one argument falls back to one-argument tests
+        ("--equivalent re.compile", lambda: fuzz(re.compile)),
+        ("--roundtrip sorted", lambda: idempotent(sorted)),
+        # For multiple arguments, they're equivalent to the function call
+        (
+            "--equivalent eval ast.literal_eval",
+            lambda: equivalent(eval, ast.literal_eval),
+        ),
+        (
+            "--roundtrip json.loads json.dumps --except ValueError",
+            lambda: roundtrip(json.loads, json.dumps, except_=ValueError),
+        ),
+        # Imports submodule (importlib.import_module passes; __import__ fails)
+        ("hypothesis.errors.StopTest", lambda: fuzz(StopTest)),
+    ],
+)
+def test_cli_python_equivalence(cli, code):
+    result = subprocess.run(
+        "hypothesis write " + cli,
+        stderr=subprocess.PIPE,
+        stdout=subprocess.PIPE,
+        shell=True,
+        universal_newlines=True,
+    )
+    cli_output = result.stdout.strip()
+    assert not result.stderr
+    code_output = code().strip()
+    assert code_output == cli_output
+    result.check_returncode()
+
+
+@pytest.mark.parametrize(
+    "cli,err_msg",
+    [
+        ("--idempotent sorted sorted", "Test functions for idempotence one at a time."),
+        (
+            "xxxx",
+            "Found the 'builtins' module, but it doesn't have a 'xxxx' attribute.",
+        ),
+        (
+            "re.srch",
+            "Found the 're' module, but it doesn't have a 'srch' attribute.  "
+            "Closest matches: ['search']",
+        ),
+        (
+            "re.fmatch",
+            "Found the 're' module, but it doesn't have a 'fmatch' attribute.  "
+            "Closest matches: ['match', 'fullmatch'",
+            # Python >= 3.7 has 'Match' objects too
+        ),
+    ],
+)
+def test_cli_too_many_functions(cli, err_msg):
+    # Supplying multiple functions to writers that only cope with one
+    result = subprocess.run(
+        "hypothesis write " + cli,
+        stderr=subprocess.PIPE,
+        stdout=subprocess.PIPE,
+        shell=True,
+        universal_newlines=True,
+    )
+    assert result.returncode == 2
+    assert "Error: " + err_msg in result.stderr
+    assert ("Closest matches" in err_msg) == ("Closest matches" in result.stderr)
+
+
+CODE_TO_TEST = """
+from typing import Sequence, List
+
+def sorter(seq: Sequence[int]) -> List[int]:
+    return sorted(seq)
+"""
+
+
+def test_can_import_from_scripts_in_working_dir(tmpdir):
+    (tmpdir / "mycode.py").write(CODE_TO_TEST)
+    result = subprocess.run(
+        "hypothesis write mycode.sorter",
+        stderr=subprocess.PIPE,
+        stdout=subprocess.PIPE,
+        shell=True,
+        universal_newlines=True,
+        cwd=tmpdir,
+    )
+    assert result.returncode == 0
+    assert "Error: " not in result.stderr