Package: elastic-nbtest
nbtestnbtestis Elastic's Search Labs testing and validation tool for Python
notebooks.Installationpipinstallelastic-nbtestUsagenbtestmy-notebook.ipynbanother-notebook.ipynb...How it worksnbtestruns all the code cells in the notebook in order from top to bottom,
and reports two error situations:If any code cells raise an unexpected exceptionIf any code cells that have output saved in the notebook generate a different
output (not counting especially designated sections, as described in the
"Configuration" section below)Something to keep in mind when designing notebooks that are testable is that
for any operations that are asynchronous it is necessary to add code that
blocks until these operations complete, so that the entire notebook can
execute in batch mode without errors.Configurationnbtestlooks for a configuration file named.nbtest.ymlin the same
directory as the target notebook. If the configuration file is found, it is
imported and applied for all the notebooks in that directory.There is currently one supported configuration variable, calledmasks. This
variable can be set to a list of regular expresssions that capture details in
the output of code cells that vary from one run to the next and should be
masked when comparing the previously stored output against output from the
current run.Here is an example.nbtest.ymlfile:masks:-"'name':'[^']+'"-"'cluster_name':'[^']+'"-"'cluster_uuid':'[^']+'"-"'build_flavor':'[^']+'"-'[0-9]+\.[0-9]+\.[0-9]+'-"'build_hash':'[^']+'"-"'build_date':'[^']+'"Handling of thegetpassfunctionThenbtestscript installs an alternative version of thegetpass()function
that looks for requested values in environment variables instead of starting an
interactive prompt.Consider the following example, which is used in many Elastic notebooks:CLOUD_ID=getpass("Elastic Cloud ID:")ELASTIC_API_KEY=getpass("Elastic Api Key:")Thegetpass()function used bynbtesttakes the prompt given as an
argument and converts it to an environment variable name with the following
rules:Spaces are converted to underscoresNon-alphanumeric characters are removedLetters are uppercasedIn the above example, the variables that will be used to source these prompts
areELASTIC_CLOUD_IDandELASTIC_API_KEY.As a convenience,nbtestimports all variables defined in a.envfile if
found. The--env-fileoption can be used to provide an alternative
environment file.Set up and tear down proceduresSometimes it is necessary to perform "set up" and/or "tear down" operations
before and after a notebook runs.nbtestwill look for notebooks with special
names designated as set up or tear down and execute those notebooks to perform
any necessary actions.Set up notebooksnbtestwill look for the following notebooks names and execute any that are
found before running the target notebook:_nbtest_setup.ipynb_nbtest_setup.[notebook-name].ipynbThe first one can be used for general set up logic that applies to all the
notebooks in the directory. TheNBTEST["notebook"]expression can be used
inside this notebook to obtain the name of the notebook under test.The second one should be used for set up actions that are specific to one
notebook.A global setup notebook can also be provided in the--setup-notebookcommand
line argument. This notebook is executed once, before any tests.Tear down notebooksnbtestwill look for the following notebooks names and execute any that are
found after running the target notebook, regardless of the testing having
succeeded or failed:_nbtest_teardown.[notebook-name].ipynb_nbtest_teardown.ipynbThese notebooks are inteded for cleanup that needs to happen after a text, for
example to delete indexes that were created. As in the set up case,NBTEST["notebook"]is set to the notebook that was tested.A global teardown notebook can also be provided in the--teardown-notebookcommand line argument. This notebook is executed once, after all the tests.Mocking and/or Monkey-PatchingFor specific cases in which it is necessary for tests to alter the behavior of
functions or methods,nbtestprovides the--mocksoption. The argument to
this option is a Python module name, exactly as it would be entered in animportstatement. The module will be imported in the context of the notebook
and any defined setup and teardown procedures.The module passed to the--mocksoption can apply any patching techniques as
needed by the tests. The following example patches theElasticsearchclass to
always connect to a locally hosted instance without authentication, regardless
of the connection arguments that are passed:importelasticsearchorig_es_init=elasticsearch.Elasticsearch.__init__defpatched_es_init(self,*args,**kwargs):returnorig_es_init(self,'http://localhost:9200')elasticsearch.Elasticsearch.__init__=patched_es_init