Package: xdoctest
Xdoctest - Execute doctests. A Python package for executing tests in
documentation strings!What is adoctest?
It is example code you write in a docstring!
What is adocstring?
Its a string you use as a comment! They get attached to Python functions and
classes as metadata. They are often used to auto-generate documentation.
Why is it cool?
Because you can write tests while you code!Tests are good. Documentation is good. Examples are good.  Doctests have low
boilerplate, you write them in the same file you write your code. It often can
help you write the function. Write down how to construct minimal demo inputs
(it helps to have tools to create these) in your file.  Copy that code into
IPython/Jupyter, and play with your implementation.  Copy your finished code
into the body. Write down how to call the function with the demo inputs. If you
feel inclined, check that the result matches an expected result (while asserts
and checks are nice, a test that just shows how to run the code is better than
no test at all).defan_algorithm(data,config):"""
    Example:
        >>> data = '([()[]])[{}([[]])]'
        >>> config = {'outer': sum, 'inner': ord}
        >>> an_algorithm(data, config)
        1411
    """# I wrote this function by first finding some interesting demodata# then I wrote the body in IPython and copied it back in.# Now I can re-use this test code I wrote in development as a test!# Covered Code is much easier to debug (we have a MWE)!result=config['outer'](map(config['inner'],data))returnresultThe problem? How do you run the code in your doctest?Xdoctest finds and executes your doctests for you.
Just runxdoctest<path-to-my-module>.
It plugs into pytest to make it easy to run on a CI. Install and runpytest--xdoctest.Thexdoctestpackage is a re-write of Python’s builtindoctestmodule. It replaces the old regex-based parser with a new
abstract-syntax-tree based parser (using Python’sastmodule). The
goal is to make doctests easier to write, simpler to configure, and
encourage the pattern of test driven development.Read the docshttps://xdoctest.readthedocs.ioGithubhttps://github.com/Erotemic/xdoctestPypihttps://pypi.org/project/xdoctestPyCon 2020Youtube VideoandGoogle SlidesQuick StartInstallation: from pypiXdoctest is distributed on pypi as a universal wheel and can be pip installed on
Python 3.6+ (Python 2.7 and 3.4 / 3.5 support was removed in Version 1.1.0).
Installations are tested on CPython and PyPy implementations.pip install xdoctestDistributions on pypi are signed with a GPG public key:D297D757. If you
care enough to check the gpg signature (hopefully pip will just do this in the
future), you should also verify this agrees with the contents ofdev/public_gpg_key.Usage: run your doctestsAfter installing, the fastest way to run all doctests in your project
is:python -m xdoctest /path/to/your/pkg-or-module.pyor if your module has been pip-installed / is in the PYTHONPATH runpython -m xdoctest yourmodnameGetting StartedThere are two ways to usexdoctest: viapytestor via the native
interface. The native interface is less opaque and implicit, but its
purpose is to run doctests. The other option is to use the widely usedpytestpackage. This allows you to run both unit tests and doctests
with the same command and has many other advantages.It is recommended to usepytestfor automatic testing (e.g. in your
CI scripts), but for debugging it may be easier to use the native
interface.Check if xdoctest will work on your packageYou can quickly check ifxdoctestwill work on your package
out-of-the box by installing it via pip and runningpython-mxdoctest <pkg> all, where<pkg>is the path to your
python package / module (or its name if it is installed in yourPYTHONPATH).For example with you might test ifxdoctestworks onnetworkxorsklearnas such:python-mxdoctest networkx all/python-mxdoctest sklearn all.Using the pytest interfaceWhenpytestis run,xdoctestis automatically discovered, but is
disabled by default. This is becausexdoctestneeds to replace the builtindoctestplugin.To enable this plugin, runpytestwith--xdoctestor--xdoc.
This can either be specified on the command line or added to youraddoptsoptions in the[pytest]section of yourpytest.iniortox.ini.To run a specific doctest,xdoctestsets uppytestnode names
for these doctests using the following pattern:<path/to/file.py>::<callname>:<num>. For example a doctest for a
function might look like thismymod.py::funcname:0, and a class
method might look like this:mymod.py::ClassName::method:0Using the native interface.In addition to thepytestplugin, xdoctest has a native doctest runner.
You can use thexdoctestcommand line tool that is installed with the
package and point it a module directory or a particular file.You can also make it such that invoking your module as__main__invokes the
xdoctest native runner using the using thexdoctest.doctest_module(path)method, which can be placed in the__main__section of any module as such:if__name__=='__main__':importxdoctestxdoctest.doctest_module(__file__)This sets up the ability to invoke thexdoctestcommand line
interface.python-m<modname> <command>.However, it is typically prefered to just use thexdoctestexecutable and
pass it the path to your file, or the name of an installed module. In this case
it is invoked likexdoctest-m<modname> <command>.Using either of these methods you can natively invoke xdoctest on a module or
package, which exposes the command line interface. Both of these expose the
command line interface, allowing you to pass a command to xdoctest.If<command>isall, then each enabled doctest in the module
is executed:python-m<modname> allIf<command>islist, then the names of each enabled doctest
is listed.If<command>isdump, then all doctests are converted into a format
suitable for unit testing, and dumped to stdout (new in 0.4.0).If<command>is acallname(name of a function or a class and
method), then that specific doctest is executed:python-m<modname> <callname>. Note: you can execute disabled
doctests or functions without any arguments (zero-args) this way.For example if you created a modulemymod.pywith the following
code:deffunc1():"""
    Example:
        >>> assert func1() == 1
    """return1deffunc2(a):"""
    Example:
        >>> assert func2(1) == 2
        >>> assert func2(2) == 3
    """returna+1You couldUse the commandxdoctest-mmymod listto list the names of all functions with doctestsUse the commandxdoctest-mmymod allto run all functions with doctestsUse the commandxdoctest-mmymod func1to run only func1’s doctestUse the commandxdoctest-mmymod func2to run only func2’s doctestPassing--helpto either way of invoking the native runner will result in
something similar to the following that outlines what other options are
available:usage: xdoctest [-h] [--version] [-m MODNAME] [-c COMMAND] [--style {auto,google,freeform}] [--analysis {auto,static,dynamic}] [--durations DURATIONS] [--time]
                [--colored COLORED] [--nocolor] [--offset] [--report {none,cdiff,ndiff,udiff,only_first_failure}] [--options OPTIONS] [--global-exec GLOBAL_EXEC]
                [--verbose VERBOSE] [--quiet] [--silent]
                [arg ...]

Xdoctest 1.0.1 - on Python - 3.9.9 (main, Jun 10 2022, 17:45:11)
[GCC 11.2.0] - discover and run doctests within a python package

positional arguments:
  arg                   Ignored if optional arguments are specified, otherwise: Defaults --modname to arg.pop(0). Defaults --command to arg.pop(0). (default: None)

optional arguments:
  -h, --help            show this help message and exit
  --version             Display version info and quit (default: False)
  -m MODNAME, --modname MODNAME
                        Module name or path. If specified positional modules are ignored (default: None)
  -c COMMAND, --command COMMAND
                        A doctest name or a command (list|all|<callname>). Defaults to all (default: None)
  --style {auto,google,freeform}
                        Choose the style of doctests that will be parsed (default: auto)
  --analysis {auto,static,dynamic}
                        How doctests are collected (default: auto)
  --durations DURATIONS
                        Specify execution times for slowest N tests.N=0 will show times for all tests (default: None)
  --time                Same as if durations=0 (default: False)
  --colored COLORED     Enable or disable ANSI coloration in stdout (default: True)
  --nocolor             Disable ANSI coloration in stdout
  --offset              If True formatted source linenumbers will agree with their location in the source file. Otherwise they will be relative to the doctest itself. (default:
                        False)
  --report {none,cdiff,ndiff,udiff,only_first_failure}
                        Choose another output format for diffs on xdoctest failure (default: udiff)
  --options OPTIONS     Default directive flags for doctests (default: None)
  --global-exec GLOBAL_EXEC
                        Custom Python code to execute before every test (default: None)
  --verbose VERBOSE     Verbosity level. 0 is silent, 1 prints out test names, 2 additionally prints test stdout, 3 additionally prints test source (default: 3)
  --quiet               sets verbosity to 1
  --silent              sets verbosity to 0Zero-args runnerThe native interface has a “zero-args” mode in thexdoctestrunner. This allows you to run functions in your modules
via the command line as long as they take no arguments. The purpose is
to create a quick entry point to functions in your code (becausexdoctestis taking the space in the__main__block).For example, you might create a modulemymod.pywith the following
code:defmyfunc():print('hello world')if__name__=='__main__':importxdoctestxdoctest.doctest_module(__file__)Even thoughmyfunchas no doctest it can still be run using the
commandpython-mmymod myfunc.Note, even though “zero-arg” functions can be run via this interface
they are not run bypython-mmymod all, nor are they listed bypython-mmymod list.However, if you are doing this often, you may be better served byfire.EnhancementsThe main enhancementsxdoctestoffers overdoctestare:All lines in the doctest can now be prefixed with>>>. There is
no need for the developer to differentiate betweenPS1andPS2lines. However, old-style doctests wherePS2lines are
prefixed with...are still valid.Additionally, the multi-line strings don’t require any prefix (but
its ok if they do have either prefix).Tests are executed in blocks, rather than line-by-line, thus
comment-based directives (e.g.# doctest: +SKIP) can now applied
to an entire block (by placing it one the line above), in addition to having
it just apply to a single line (by placing it in-line at the end).Tests without a “want” statement will ignore any stdout / final
evaluated value. This makes it easy to use simple assert statements
to perform checks in code that might write to stdout.If your test has a “want” statement and ends with both a value and
stdout, both are checked, and the test will pass if either matches.Ouptut from multiple sequential print statements can now be checked by
a single “got” statement. (new in 0.4.0).See code indev/_compare/demo_enhancements.pyfor a demo that illustrates
several of these enhancements. This demo shows cases wherexdoctestworks
butdoctestfails. As of version 0.9.1, there are no known syntax backwards
incompatability. Please submit an issue if you can find any backwards
incompatible cases.ExamplesHere is an example demonstrating the new relaxed (and
backwards-compatible) syntax:deffunc():"""
    # Old way
    >>> def func():
    ...     print('The old regex-based parser required specific formatting')
    >>> func()
    The old regex-based parser required specific formatting

    # New way
    >>> def func():
    >>>     print('The new ast-based parser lets you prefix all lines with >>>')
    >>> func()
    The new ast-based parser lets you prefix all lines with >>>
    """deffunc():"""
    # Old way
    >>> print('''
    ... It would be nice if we didnt have to deal with prefixes
    ... in multiline strings.
    ... '''.strip())
    It would be nice if we didnt have to deal with prefixes
    in multiline strings.

    # New way
    >>> print('''
        Multiline can now be written without prefixes.
        Editing them is much more natural.
        '''.strip())
    Multiline can now be written without prefixes.
    Editing them is much more natural.

    # This is ok too
    >>> print('''
    >>> Just prefix everything with >>> and the doctest should work
    >>> '''.strip())
    Just prefix everything with >>> and the doctest should work

    """Xdoctest Parsing StyleThere are currently two main doctest parsing styles:googleandfreeform, as well as a third style:auto, which is a hybrid.The parsing style can be set via the--stylecommand line argument in the
Xdoctest CLI, or via the--xdoctest-styleif using pytest.Setting--style=google(or--xdoctest-style=googlein pytest) enables
google-style parsing.
AGoogle-styledoctest is
expected to exist in  Google “docblock” with anExample:orDoctest:tag. All code in this block is parsed out as a single doctest.Setting--style=freeform(or--xdoctest-style=freeformin pytest) enables
freeform-style parsing.
A freeform style doctest is any contiguous block of lines prefixed by>>>.
This is the original parsing style of the builtin doctest module. Each block is
listed as its own test.By default Xdoctest sets--style=auto(or--xdoctest-style=autoin
pytest) which will pull all google-style blocks out as single doctests, while
still all other>>>prefixed code out as a freeform doctest.Notes On Got/Want TestsThe new got/want tester is very permissive by default; it ignores
differences in whitespace, tries to normalize for python 2/3
Unicode/bytes differences, ANSI formatting, and it uses the old doctest
ELLIPSIS fuzzy matcher by default. If the “got” text matches the “want”
text at any point, the test passes.Currently, this permissiveness is not highly configurable as it was in
the original doctest module. It is an open question as to whether or not
this module should support that level of configuration. If the test
requires a high degree of specificity in the got/want checker, it may
just be better to use anassertstatement.Backwards CompatibilityThere are no known syntax incompatibilities with original doctests. This is
based on running doctests on real life examples inboltons,ubelt,networkx,pytorch, and on a set of extensive testing suite. Please
raise an issue or submit a merge/pull request if you find any incompatibility.Despite full syntax backwards compatibility, there some runtime
incompatibilities by design. Specifically, Xdoctest enables a different set of
default directives, such that the “got”/”want” checker is more permissive.
Thus, a test that fails indoctestbased on a “got”/”want” check, may pass
inxdoctest. For this reason it is recommended that you rely on codedassert-statements for system-critical code. This also makes it much easier
to transform yourxdoctestinto aunittestwhen you realize your
doctests are getting too long.One Last ExampleXDoctest is a good demonstration of itself. After pip installing xdoctest, try
running xdoctest on xdoctest.xdoctestxdoctestIf you would like a slightly less verbose output, tryxdoctestxdoctest--verbose=1# orxdoctestxdoctest--verbose=0You could also consider running xdoctests tests through pytest:pytest$(python-c'import xdoctest, pathlib; print(pathlib.Path(xdoctest.__file__).parent)')--xdoctestIf you would like a slightly more verbose output, trypytest-s--verbose--xdoctest-verbose=3--xdoctest$(python-c'import xdoctest, pathlib; print(pathlib.Path(xdoctest.__file__).parent)')If you ran these commands, the myriad of characters that flew across your
screen are lots more examples of what you can do with doctests.