Package: nbvalx
nbvalxnbvalxis a collection ofpytestutils built on top ofnbval.nbvalxis currently developed and maintained atUniversità Cattolica del Sacro CuorebyDr. Francesco Ballarin.nbvalxcan bepip installed fromits GitHub repositoryor fromPyPICell magics for conditional running based on tagsAdd a cell with%load_ext nbvalxat the beginning of a notebook to loadnbvalxIPythonextension. The extension is implemented innbvalx/jupyter_magics.py.The extension allows to register a list of allowed tags, and their values%%register_run_if_allowed_tags
bool_tag: True, False
int_tag: 2, 10
string_tag: "value1", "value2"and set the current value of each tag with%%register_run_if_current_tags
bool_tag = False
int_tag = 10
string_tag = "value1"The tags can then be used to conditionally run cells. As an example, if the subsequent two cells were%%run_if bool_tag and int_tag > 5
print("Cell with float and integer comparison")%%run_if string_tag == "value1"
print("Cell with string comparison")the string"Cell with float and integer comparison"would never get printed, while the"Cell with string comparison"would indeed be printed.Withoutnbvalx, the four cells above could have equivalently been written asbool_tag = False
int_tag = 10
string_tag = "value1"
if bool_tag and int_tag > 5:
    print("Cell with float and integer comparison")
if string_tag == "value1":
    print("Cell with string comparison")The plain formulation is certainly less verbose and more compact thannbvalxequivalent one with four cells. However, it is less "testing-friendly", because the values ofbool_tag,int_tagandstring_tagare hardcoded in the notebook and cannot be easily varied. With apytestterminology,nbvalxtags correspond to defining a parametrization of the notebook.A similar statement can be used as an HTML comment in markdown cells<!-- keep_if string_tag == "value1" -->
This is a markdown cell which explains a code cell that will be run only if string_tag == "value1".Markdown cells will be stripped only when running throughpytestwith the--tag-collapseoption, see below.Seetests/notebooks/data/tagsfor a few simple notebooks using tags.Custom pytest hooks for jupyter notebooksThe filenbvalx/pytest_hooks_notebooks.pycontains a few utility functions to be used in pytest configuration file for notebooks tests.
Thepytesthooks which can be customized in this way are:pytest_addoption,pytest_collect_file, andpytest_sessionstart.For clarity, the hooks implemented innbvalx/pytest_hooks_notebooks.pydo not have apytest_prefix, as it will be the user's responsability to pick them up and assign them to the correspondingpytesthook in a customconftest.py, as show intests/notebooks/conftest.py.The hooks change the default behavior ofnbvalin the following ways:the options--nbvaland--nbval-lax, whichnbvalrequires to pass explicitly, are here enabled implicitly;support forMPIrun by providing the--npoption topytest. When runningpytest --np 2,nbvalxwill start aipyparallel.Clusterand run notebooks tests in parallel on 2 cores. In the default case one core is employed, and anipyparallel.Clusteris not started;support for tags, as introduced in the previous section, as governed by two flags:--tag-collapse: if enabled (default), strip all cells for which the%%run_if ...or<!-- keep_if ... -->conditions do not evaluate toTrue. This may be used to prepare notebook files to be read by the end user, as stripping unused cells may improve the readability of the notebook. If not enabled, all cells will be kept.--ipynb-action: eithercollect-notebooks(default) orcreate-notebook. Both actions create several copies of the original notebook that differ by the currently enabled tag. For instance, if the original notebook in the section above is callednotebook.ipynband has two allowed tags, the action will generate a filenotebook[tag1].ipynbwheretag1is assigned as the current value of the tag, and a filenotebook[tag2].ipynbwheretag2is assigned as the current value of the tag. Iftag-collapseis enabled, cells associated to all remaining tags are stripped. Thecreate-notebookaction only generates the tagged notebooks; instead, thecollect-notebooksadditionally also runs them throughpytest;support for collecting cell outputs to log files, which are saved in a work directory provided by the user with the argument--work-dir. This is helpful to debug failures while testing notebooks. Log files are of two formats: a text log, with extension.logwhen running without--npor.log-{rank}when running in parallel; a notebook log, with extension.log.ipynb. If no work directory is specified, the default value isf".ipynb_pytest/np_{np}/collapse_{tag_collapse}". In case the notebook depends on additonal data files (e.g., local python modules), the flag--link-data-in-work-dircan be passed with glob patterns of data files that need to be symbolically linked in the work directory. The option can be passed multiple times in case multiple patterns are desired, and they will be joined with an or condition;the notebook is treated as if it were a demo or tutorial, rather than a collection of unit tests in different cells. For this reason, if a cell fails, the next cells will be skipped;a new# PYTEST_XFAILmarker is introduced to mark cells as expected to fail. The marker must be the first entry of the cell. A similar marker# PYTEST_XFAIL_AND_SKIP_NEXTmarks the cell as expected to fail and interrupts execution of the subsequent cells. Both previous markers have a variant withXFAIL_IN_PARALLELinstead ofXFAIL, that consider the cell to be expected to fail only when the value provided to--npis greater than one;support for running notebooks throughcoveragewithout having to install thepyvtest-covplugin. Use flag--coverage-sourceto set the module name for which coverage testing is requested.Custom pytest hooks for unit testsThe filenbvalx/pytest_unit_tests.pycontains a few utility functions to be used in pytest configuration file for notebooks tests.
Thepytesthooks which can be customized in this way are:pytest_runtest_setup, andpytest_runtest_teardown.For clarity, the hooks implemented innbvalx/pytest_unit_tests.pydo not have apytest_prefix, as it will be the user's responsability to pick them up and assign them to the correspondingpytesthook in a customconftest.py, as show intests/unit/conftest.py.The hooks are typically employed to obtain aMPI-parallel safe execution of python unit tests by calling garbage collection and putting aMPIbarrier after each test.Custom pytest hooks for unit testsThe filenbvalx/tempfile.pycontainsMPIparallel-safe context managers to create temporary files and directories. Similarly to thetempfilemodule in the standard library, the following context managers are provided:nbvalx.tempfile.TemporaryDirectory,nbvalx.tempfile.TemporaryFile.