Package: nbmake
nbmakeWhat?Pytest plugin for testing and releasing notebook documentationWhy?To raise the quality of scientific material through better automationWho is this for?Research/Machine Learning Software Engineers who maintain packages/teaching materials with documentation written in notebooks.✨ New: Automate the generation of notebook-based tutorials withnbwriteFunctionalityExecutes notebooks using pytest and nbclient, allowing parallel notebook testingOptionally writes back to the repo, allowing faster building ofnbsphinxorjupyter bookdocsQuick StartIf you have a notebook that runs interactively using an ipython kernel,
you can try testing it automatically as follows:pipinstallpytestnbmake
pytest--nbmake**/*ipynbConfigure Cell TimeoutsYou can configure the cell timeout with the following pytest flag:pytest--nbmake--nbmake-timeout=3000# allows each cell 3000 seconds to finishAllow Errors For a Whole NotebookThis configuration must be placed in the notebook'stop-level metadata(not cell-level metadata).Your notebook should look like this:{"cells":[...],"metadata":{"kernelspec":{...},"execution":{"allow_errors":true,"timeout":300}}}Allow a Cell to Throw an ExceptionA cell with the following metadata can throw an exception without failing the test:"metadata":{"tags":["raises-exception"]}Ignore a Code CellA cell with the following metadata will not be executed by nbmake{"language":"python","custom":{"metadata":{"tags":["skip-execution"]}}}Override Notebook Kernels when TestingRegardless of the kernel configured in the notebook JSON, you can force nbmake to use a specific kernel when testing:pytest --nbmake --nbmake-kernel=mycustomkernelAdd Missing Jupyter Kernel to Your CI EnvironmentIf you are not using the flag above and are using a kernel name other than the default ‘python3’, you will see an error message when executing your notebooks in a fresh CI environment:Error - No such kernel: 'mycustomkernel'Use ipykernel to install the custom kernel:python-mipykernelinstall--user--namemycustomkernelIf you are using another language such as c++ in your notebooks, you may have a different process for installing your kernel.ParallelisationFor repos containing a large number of notebooks that run slowly, you can run each notebook
in parallel usingpytest-xdist.pipinstallpytest-xdist

pytest--nbmake-n=autoIt is also possible to parallelise at a CI-level using strategies, seeexampleBuild Jupyter Books FasterUsing xdist and the--overwriteflag let you build a large jupyter book repo faster:pytest--nbmake--overwrite-n=autoexamples
jbbuildexamplesFind missing imports in a directory of failing notebooksIt's not always feasible to get notebooks running from top to bottom from the start.You can however, use nbmake to check that there are noModuleNotFoundErrors:pytest\--nbmake\--nbmake-find-import-errors\# Ignore all errors except ModuleNotFoundError--nbmake-timeout=20# Skip past cells longer than 20sMock out variables to simplify testingIf your notebook runs a training process that takes a long time to run, you can use nbmake's
mocking feature to overwrite variables after a cell runs:{"cells":[...,{"cell_type":"code","execution_count":null,"metadata":{"nbmake":{"mock":{// these keys will override global variables after this cell runs"epochs":2,"config":"/test/config.json","args":{"env":"test"}}}},"outputs":[],"source":["epochs = 10\n","..."]},...],...}Run test logic after a cell executesYou can fetch CI secrets and run assertions after any cell by putting scripts in the cell metadata undernbmake.post_cell_execute:{"cells":[{"cell_type":"code","execution_count":null,"metadata":{"nbmake":{"post_cell_execute":["y = 3","z = x+y"]}},"outputs":[],"source":["x = 1\n","y = 2\n","z = 0\n","# this cell has a post_cell_execute that assigns y and z"]},Advice on Usagenbmake is best used in a scenario where you use the ipynb files only for development. Consumption of notebooks is primarily done via a docs site, built through jupyter book, nbsphinx, or some other means. If using one of these tools, you are able to write assertion code in cells which will behidden from readers.Pre-commitTreating notebooks like source files lets you keep your repo minimal. Some tools, such as plotly may drop several megabytes of javascript in your output cells, as a result, stripping out notebooks on pre-commit is advisable:# .pre-commit-config.yaml
repos:
  - repo: https://github.com/kynan/nbstripout
    rev: master
    hooks:
      - id: nbstripoutSeehttps://pre-commit.com/for more...Disable NbmakeImplicitly:pytestExplicitly:pytest -p no:nbmakeSee Also:A more in-depthintro to nbmakerunning on Semaphore CInbmake actionpytestjupyter bookjupyter cacheMyST-NBℹ️ Get help with machine learning infrastructureBeyond testing notebooks, the maintainers of nbmake help software and finance companies scale their machine learning products.Find out more.