Package: pytest-phmdoctest
pytest-phmdoctest 1.0.0IntroductionPython syntax highlighted Markdown doctest pytest plugin.Apytestplugin based on thephmdoctestcommand line tool.If you have Python syntax highlighted examples in Markdown
like this Python code...print("Hello World!")plus expected output.Hello World!Thispytestplugin will test them, as is, without edits.
On this file try the command ...pytest-v--phmdoctestREADME.mdpytest console output ...plugins: phmdoctest-1.0.0
collected 1 item

::README.py::test_code_12_output_16 PASSEDThe plugin also tests Python interactive sessions
described bydoctest.
Seetesting examplesbelow.The12in the function nametest_code_12_output_16is the
line number of the first line
of python code.16shows the line number of the expected
terminal output.DescriptionThe plugin is based on the Python toolphmdoctestversion >= 1.4.0
It generates a pytest test file from a Markdown file.Reads these from Markdown fenced code blocks:Python source code and expected terminal output.Python interactive sessions described bydoctest.Simple use case is possible with no Markdown edits.More features selected by adding HTML comment directives
to the Markdown. See Directives inphmdoctest Directives.Run on pytest >= 5.4.3.main branch statusWebsite|Docs|Repos|pytest|Codecov|LicenseIntroduction|Description|Installation|Usage|Testing Python interactive sessions|Generate test files|Generate and collect test files|Help|Configure collection|Hints|Related projectsChanges|ContributionsInstallationIt is advisable to install in a virtual environment.python -m pip install pytest-phmdoctestUsageConsider a project with the following files.
Not all files are shown.README.md
doc/directive2.md
doc/nocode.md
tests/test_example.pypytest-vplugins: phmdoctest-1.0.0
collected 1 item

tests/test_example.py::test_example PASSEDUse--phmdoctestto also collect Markdown files.pytest-v--phmdoctestplugins: phmdoctest-1.0.0
collected 6 items

::README.py::test_code_10_output_17 PASSED
::doc__directive2.py::test_code_25_output_32 PASSED
::doc__directive2.py::test_code_42_output_47 PASSED
::doc__directive2.py::test_code_52_output_56 PASSED
::doc__project.py::test_code_12_output_19 PASSED
tests/test_example.py::test_example PASSEDThe sample project above can be viewed on GitHub attests/sample.The doc__ indicates the Markdown file was collected from the
doc folder.Markdown ".md" files get discovered by pytest.
pytest finds them in the same way it finds Python test files.
For each Markdown file discovered, the plugin generates a pytest
test file.A Markdown file that does not have any Python fenced code block examples
is not tested.To avoid collecting .md files use pytest--ignoreand--ignore-globon the command line or in the addopts part of the pytest ini file.
These commands work on .md files and use Unix shell-style wildcards.Use--phmdoctestin a pytest ini file instead of on the
command line as shown below:# pytest.ini and tox.ini[pytest]addopts=--phmdoctest# pyproject.toml[tool.pytest.ini_options]addopts="--phmdoctest"# setup.cfg  Note: Use is discouraged by pytest docs.[tool:pytest]addopts=--phmdoctestTesting Python interactive sessionsThe plugin also tests Python interactive sessions described bydoctestlike this one:>>>importmath>>>math.floor(9.1)9Use the--phmdoctest-docmodoption to collect both Python code/expected output
and Python interactive sessions.pytest-v--phmdoctest-docmodplugins: phmdoctest-1.0.0
collected 10 items

::README.py::README.session_00001_line_24 PASSED
::README.py::test_code_10_output_17 PASSED
::doc__directive2.py::test_code_25_output_32 PASSED
::doc__directive2.py::test_code_42_output_47 PASSED
::doc__directive2.py::test_code_52_output_56 PASSED
::doc__project.py::doc__project.session_00001_line_31 PASSED
::doc__project.py::doc__project.session_00002_line_46 PASSED
::doc__project.py::doc__project.session_00003_line_55 PASSED
::doc__project.py::test_code_12_output_19 PASSED
tests/test_example.py::test_example PASSEDThe--phmdoctest-docmodoption uses a non-public pytest class DoctestModule.
There is a slight chance a pytest future
major release changes the DoctestModule API.DoctestModule works ok on pytest major releases 5, 6, and 7
as verified by tests in .github/workflows/ci.yml.If--phmdoctest-docmoddoctest collection breaks,
the rest of the plugin--phmdoctestand--phmdoctest-generateoptions still work. The test suite simulates such breaking
changes. See the test_readme and test_docmod.py test functions
that take the monkeypatch fixture.Theline_24in the function namesession_00001_line_24is the
line number in tests/sample/README.md of the first line
of the interactive session.00001is a sequence number to
order the doctests.Here is simulated output captured when--phmdoctest-docmoddoctest collection breaks due to incompatible DoctestModule API.pytest-v--phmdoctest-docmodplugins: phmdoctest-1.0.0
collected 8 items

::README.py::test_code_10_output_17 PASSED
::README.py::test_unable_to_collect_doctests FAILED
::doc__directive2.py::test_code_25_output_32 PASSED
::doc__directive2.py::test_code_42_output_47 PASSED
::doc__directive2.py::test_code_52_output_56 PASSED
::doc__project.py::test_code_12_output_19 PASSED
::doc__project.py::test_unable_to_collect_doctests FAILED
tests/test_example.py::test_example PASSEDThere is one FAILED test called for each Markdown file with
Python interactive sessions.The Python code/expected output examples still run successfully.The pytest test case in tests/test_example.py succeeds.Generate test filesSave generated test files to the file system. Do not collect them.
The plugin does not use the non-public DoctestModule when doing this.pytest-v--phmdoctest-generate.gendirplugins: phmdoctest-1.0.0
collected 1 item

tests/test_example.py::test_example PASSEDNote that only test_example.py was collected.With--phmdoctest-generatethe test files
generated from Markdown do not get collected.Run pytest again on the generated test files.The generated test files become stale with time.Test files should be regenerated after modifying the Markdown.See below to generate and collect test files with
a single pytest command..gendir is cleaned of all *.py
files before writing new test files.
Pre-existing *.py files in the output directory get renamed. If
output_directory inadvertently gets pointed at a Python
source directory, the renamed files can be recovered by renaming them.The FILENAME.py files found in the output directory are renamed
to noFILENAME.sav.If a noFILENAME.sav already exists it is not modified.Files in target_dir with other extensions are not modified.A FILENAME.py pre-existing in target_dir is only renamed
and not deleted..gendir is cleaned of all *.md files as well.
Pre-existing FILENAME.md files in the output directory get renamed
to FILENAME_md.sav.If .gendir was empty, it will now have these *.py files:test_doc__directive2.py
test_doc__project.py
test_README.pyGenerate and collect test filesA single pytest command will generate test files and collect them.
The plugin does not use the non-public DoctestModule when doing this.pytest-v--phmdoctest-generate.gendir..gendir--doctest-modules--ignoresrcplugins: phmdoctest-1.0.0
collected 10 items

tests/test_example.py::test_example PASSED
.gendir/test_README.py::test_README.session_00001_line_24 PASSED
.gendir/test_README.py::test_code_10_output_17 PASSED
.gendir/test_doc__directive2.py::test_code_25_output_32 PASSED
.gendir/test_doc__directive2.py::test_code_42_output_47 PASSED
.gendir/test_doc__directive2.py::test_code_52_output_56 PASSED
.gendir/test_doc__project.py::test_doc__project.session_00001_line_31 PASSED
.gendir/test_doc__project.py::test_doc__project.session_00002_line_46 PASSED
.gendir/test_doc__project.py::test_doc__project.session_00003_line_55 PASSED
.gendir/test_doc__project.py::test_code_12_output_19 PASSEDHow it works:The plugin writes the generated test files during
the pytest collection phase.This happens while pytest is collecting from"."pytest does not collect from.gendiruntil after discovering
and collecting files in".".The leading"."in.gendirprevents pytest from searching
there for test files while searching".".
See norecursedirs default values in Pytest Documentation |
API reference | Configuration Options | norecursedirs.The--doctest-modulesoption tells pytest to look for doctests in
docstrings of *.py files.When doing--doctest-modules, the--ignore srcoption tells
pytest not to collect modules from the src folder. We only
want to collect doctests from .gendir.
This prevents pytest from importing modules
there to look for doctests.These are the ini file settings:# pytest.ini and tox.ini[pytest]addopts=--phmdoctest-generate=.gendir --doctest-modules --ignore src# pyproject.toml[tool.pytest.ini_options]addopts="--phmdoctest-generate=.gendir --doctest-modules --ignore src"# setup.cfg  Note: Use is discouraged by pytest docs.[tool:pytest]addopts=--phmdoctest-generate=.gendir --doctest-modules --ignore srcHere is a demo that runs on a checked out copy of the repository.# With a terminal in the tests/sample directory
# The first line collects 5 items.
# The second line collects 3 items.

pytest -v --phmdoctest-generate=.gendir "." .gendir --ignore README.md --ignore doc/directive2.md --doctest-modules --ignore src
pytest -v --phmdoctest-generate=.gendir "." .gendir  --ignore-glob */*.md --doctest-modules --ignore srcHelppytest--helpcontains aphmdoctest:group in the middle and
an ini-option near the bottom.  The help contains:--phmdoctest--phmdoctest-generate--phmdoctest-docmodphmdoctest-collectConfigure collectionAn optionalphmdoctest-collect =section can be placed in the pytest
ini file. It is a list of lines of the formatglob [options]Consider using the section to pass collection options described below.
The options have the same names and behave like the options
accepted byphmdoctest usage.The Markdown file must match one of the globs.The glob is processed by Path.glob() from the Python standard library pathlib.
Path.glob() offers a "**" recursive pattern that means
"this directory and all subdirectories recursively."The globs are checked from top to bottom. The first glob to match the Markdown
file determines the phmdoctest command line options.If there is no match the file willnotbe collected.A line can have just the glob and no options. The glob is required.The options should look like and have the same spacing as the command line
options passed to the toolphmdoctest usage.Use double quotes as needed in TEXT.The plugin does not support\"escaped double quote.Look for list of options in the next section.If a line that does not parse is needed, the plugin collects
a special test file that contains a failing test case with
an embedded error message.Example# pytest.ini[pytest]addopts=--phmdoctest-docmodphmdoctest-collect=doc/project.md --skip greeting --skip enjoyment**/*.mdThen run this pytest command on the project files from
the Usage section ...pytest-v--ignoretests/test_example.pyoutputplugins: phmdoctest-1.0.0
collected 7 items

::README.py::README.session_00001_line_24 PASSED
::README.py::test_code_10_output_17 PASSED
::doc__directive2.py::test_code_25_output_32 PASSED
::doc__directive2.py::test_code_42_output_47 PASSED
::doc__directive2.py::test_code_52_output_56 PASSED
::doc__project.py::doc__project.session_00001_line_31 PASSED
::doc__project.py::doc__project.session_00002_line_46 PASSEDThe example passes the options--skip greetingand--skip enjoymentwhen testing doc/project.md. The--skip greetingand--skip enjoymentoptions cause 2 examples to be skipped. These are the first and
last examples in the file.The example tests the two Python interactive sessions in the midlle
of the file as the last two items above.The "**/*.md" recursively matches other Markdown files.
There are no otions.The ini file globs above only apply to .md files.
We ignore the Python pytest test file tests/test_example.py
by adding the--ignoreoption.Look for more phmdoctest-collect examples on GitHub in
tests/test_collect_section.py.phmdoctest-collect options-s, --skip TEXTDo not test blocks with substring TEXT. Allowed multiple times.--fail-nocodeMarkdown file with no code blocks left after applying skips
generates a failing test.-u, --setup TEXTRun block with substring TEXT at test module setup time.-d, --teardown TEXTRun block with substring TEXT at test module teardown time.--setup-doctestMake globals created by the--setupPython code block
or setup directive visible to Python interactive session >>> blocks.Caution:The globals are set at pytest Session scope.
The globals are visible to all doctests in the test suite.
This includes doctests collected by the plugin
and doctests collected from other files due to--doctest-modules.NotesFenced code blocks are searched for the substring TEXT.--skip TEXTcan apply to more than one block.Exactly one block can match--setupand--teardown.Equivalent directivesThe HTML directive comments below placed in the Markdown file
can be used instead of specifying options in the
phmdoctest-collect section.
Directives select a single fenced code block.
There are 10 directives inphmdoctest Directives.
Here are the directives equivalent to the collect section options.collect-section          HTML Directive
  option                   comment
------------------------------------------------
-s, --skip TEXT        <!--phmdoctest-skip-->
-u, --setup TEXT       <!--phmdoctest-setup-->
-d, --teardown TEXT    <!--phmdoctest-teardown-->HintsWhen invoking pytest, cwd must be in the subpath of the files to be collected
to avoid this error from pathlib.py in relative_to():
ValueError:<file to be collected>is not in the subpath of<working directory>Note the plugin does not accept single quoted phmdoctest args in the
phmdoctest-collect section. A failing test will be collected.Use underscore in conftest.py for pytest_plugins:pytest_plugins = ["pytest_phmdoctest"]An ImportPathMismatchError indicates two test files have the same name.If using--phmdoctest-generateadd.gendirto .gitignore.pytest -vv output shows the path to the plugin temporary directory.The--reportoption of the phmdoctest command lists all
fenced code blocks in the Markdown file.phmdoctest can generate test files for multiple Markdown files
with one call by specifying a configuration file.phmdoctest offers pytest pytester fixtures (suitable for plugin development)
to generate and run tests for a single Markdown file.Related projectsphmdoctestrundocbyexamplesphinx.ext.doctestsybildoxecegtestpytest-codeblocksMIT LicenseCopyright (c) 2021-2022 Mark TaylorPermission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.