Package: nbval
Py.test plugin for validating Jupyter notebooksThe plugin adds functionality to py.test to recognise and collect Jupyter
notebooks. The intended purpose of the tests is to determine whether execution
of the stored inputs match the stored outputs of the.ipynbfile. Whilst also
ensuring that the notebooks are running without errors.The tests were designed to ensure that Jupyter notebooks (especially those for
reference and documentation), are executing consistently.Each cell is taken as a test, a cell that doesn't reproduce the expected
output will fail.Seedocs/source/index.ipynbfor the full documentation.InstallationAvailable on PyPi:pip install nbvalor install the latest version from cloning the repository and running:pip install .from the main directory. To uninstall:pip uninstall nbvalHow it worksThe extension looks through every cell that contains code in an IPython notebook
and then thepy.testsystem compares the outputs stored in the notebook
with the outputs of the cells when they are executed. Thus, the notebook itself is
used as a testing function.
The output lines when executing the notebook can be sanitized passing an
extra option and file, when calling thepy.testcommand. This file
is a usual configuration file for theConfigParserlibrary.Regarding the execution, roughly, the script initiates an
IPython Kernel with ashelland
aniopubsockets. Theshellis needed to execute the cells in
the notebook (it sends requests to the Kernel) and theiopubprovides
an interface to get the messages from the outputs. The contents
of the messages obtained from the Kernel are organised in dictionaries
with different information, such as time stamps of executions,
cell data types, cell types, the status of the Kernel, username, etc.In general, the functionality of the IPython notebook system is
quite complex, but a detailed explanation of the messages
and how the system works, can be found herehttps://jupyter-client.readthedocs.io/en/latest/messaging.html#messagingExecutionTo execute this plugin, you need to executepy.testwith thenbvalflag
to differentiate the testing from the usual python files:py.test --nbvalYou can also specify--nbval-lax, which runs notebooks and checks for
errors, but only compares the outputs of cells with a#NBVAL_CHECK_OUTPUTmarker comment.py.test --nbval-laxThe commands above will execute all the.ipynbfiles and 'pytest' tests in the current folder.
Specify-p no:pythonif you would like to execute notebooks only. Alternatively, you can execute a specific notebook:py.test --nbval my_notebook.ipynbBy default, each.ipynbfile will be executed using the kernel
specified in its metadata. You can override this behavior by passing
either--nbval-kernel-name mykernelto run all the notebooks usingmykernel, or--current-envto use a kernel in the same environment
in which pytest itself was launched.If the output lines are going to be sanitized, an extra flag,--nbval-sanitize-withtogether with the path to a confguration file with regex expressions, must be passed,
i.e.py.test --nbval my_notebook.ipynb --nbval-sanitize-with path/to/my_sanitize_filewheremy_sanitize_filehas the following structure.[Section1]
regex: [a-z]*
replace: abcd

regex: [1-9]*
replace: 0000

[Section2]
regex: foo
replace: barTheregexoption contains the expression that is going to be matched in the outputs, andreplaceis the string that will replace theregexmatch. Currently, the section
names do not have any meaning or influence in the testing system, it will take
all the sections and replace the corresponding options.CoverageTo use notebooks to generate coverage for imported code, use the pytest-cov plugin.
nbval should automatically detect the relevant options and configure itself with it.Parallel executionnbval is compatible with the pytest-xdist plugin for parallel running of tests. It does
however require the use of the--dist loadscopeflag to ensure that all cells of one
notebook are run on the same kernel.DocumentationThe narrative documentation for nbval can be found athttps://nbval.readthedocs.io.HelpThepy.testsystem help can be obtained withpy.test -h, which will
show all the flags that can be passed to the command, such as the
verbose-voption. Nbval's options can be found under theJupyter Notebook validationsection.AcknowledgementsThis plugin was inspired by Andrea Zonca's py.test plugin for collecting unit
tests in the IPython notebooks (https://github.com/zonca/pytest-ipynb).The original prototype was based on the template inhttps://gist.github.com/timo/2621679and the code of a testing system
for notebookshttps://gist.github.com/minrk/2620735which we
integrated and mixed with thepy.testsystem.We acknowledge financial support fromOpenDreamKit Horizon 2020 European Research Infrastructures project (#676541),http://opendreamkit.orgEPSRC's Centre for Doctoral Training in Next Generation
Computational Modelling,http://ngcm.soton.ac.uk(#EP/L015382/1) and
EPSRC's Doctoral Training Centre in Complex System Simulation
((EP/G03690X/1),The Gordon and Betty Moore Foundation through Grant GBMF #4856, by the
Alfred P. Sloan Foundation and by the Helmsley Trust.Authors2014 - 2017 David Cortes-Ortuno, Oliver Laslett, T. Kluyver, Vidar
Fauske, Maximilian Albert, MinRK, Ondrej Hovorka, Hans Fangohr