Package: nbscript
Run Jupyter notebooks as scriptsJupyter notebooks are designed almost exclusively for interactive use, but
many people want to use them for heavy-duty computational usage.nbscriptis designed to process notebooks as scripts and provide the
most common script features: clear start and end, arguments (argv),
(stdin) and stdout, and so on.We also take the perspective of batch processing, so also have a wrapper
that allows you to submit notebooks as Slurm scripts (similar sbatch
with a Python as the interpreter).Notebooks are very good for interactive work, but for large
computation interactive just isn't an efficient use of resources.  For
other expensive resources that can't be shared (GPUs for example),
interactive work even for development can be a bit questionable.  A
proper course of action would be to create proper programs to run
separate from notebooks... but sometimes people prefer to stay in
notebooks.Many other modules (see references below) try to allow notebooks to be
run, but we take the viewpoint that the traditional UNIX script
interface is good and notebooks should be made like scripts:nbscript notebook.ipynbshould behave similarly topython notebook.py.  This
also allows us to provide a logical pathway to non-notebook programs.Quick examples of invocationnbscript:nbscript input.ipynb [argv]: runs, prints results as asciidoc to
stdout.  Within the script,from nbscript import argvto get the
argv.nbscript --save input.ipynb: runs, saves toinput.out.ipynbnbscript --save --timestamp input.ipynb: runs, saves toinput.out.TIMESTAMP.ipynbsnotebook:snotebook [slurm opts] input.ipynb: submits to slurm withsbatch, using the--saveoption like you see above.  Slurm
output is ininput.out.ipynb.log.snotebook [slurm opts] --- --timestamp input.ipynb: like above,
but adds--timestampoption like you see above.Usagenbscript is still in development, so not all of this functionality
exists yet.  In general,nbscript notebook.ipynbshould have as
similar an interface aspython notebook.py.Run a notebook from the command line:nbscript nb.ipynb arg1 arg2 ....  Within the notebook, you can
access the arguments byimport nbscript ; nbscript.argv(these are
currently transferred via environment variables).  Note thatargv[0]is the notebook name if it is known, otherwiseNone.By default, only the output of the cells is printed to stdout.
Options may used to save the notebook to a file in any of
nbconvert's supported output formats.You may also run a notebook via IPython extensions:%nbscript nb.ipynb [arg1 arg2 ...].  By default the output isn't
substituted back in, because we couldn't do much with that.
Instead, it is saved to a HTML file with the output and errors.  If
you don't give an output name, the output is timestamped.Currently not implemented, use!nbscriptinstead.nbscriptsets theNBSCRIPT_RUNNINGenvironment variable, and if
this is already set it won't run again.  That way, you can have a
notebook execute itself with the%nbscriptmagic function.Interface within notebooks:import nbscript ; nbscript.argvis theargvin analogy tosys.argv.  (json-encoded in the environment variableNB_ARGV).One would useargparsewithnbscript.argv, in particularlyparser.parse_args(args=nbscript.argv[1:]).Other environment variables:NB_NAMEis the notebook name (note
that there is no way for Jupyter kernels to know the currently
executing notebook name, this seems to be intentional because it's a
protocol layer violation).nbscriptsets the environment variableNBSCRIPT_RUNNINGbefore
it executes a notebook, and if this is already set then it will do
nothing if it tries to execute again (print an error message and
exit).  This is so that an notebook cannbscript-execute itself
without recursive execution.  This behavior is up for debate.Submit a notebook via Slurmsnotebook nb.ipynb arg1 arg2.  This is similar tosbatch script.sh arg1 arg2- it will search for#SBATCHlines and
process them (stopping the search after the first cell that has
any).Similar to the%nbscriptmagic function, there is the%snotebookmagic function.Saving output state:When a notebook is run non-interactively, it would be useful to save
the state so that the output variables can be re-loaded and played
with.  To do that, we should find some way to serialize the state
and re-load it.  It would be nice if nbscript could automate this,
but perhaps that makes things too fragile.Thedillmodule is supposed to be
able to serialize most Python objects (but starts failing at some
complex machine learning pipeline objects).  One can try to
serialize the state at the end of the execution.In the future, we want to add support for automatically serializing
the final state after the notebook is run in parallel to the html
output.  This would allow one to re-load the state to continue
post-processing.  For now, though, we recommend you explicitly save
whatever is important (this is probably more reliable anyway).See alsoThere are many commands to execute notebooks, but most of these do not
see the notebooks as a first-class script, but as an interactive thing
which happens to be run.  that would have run, arguments, stdout, etc.https://github.com/nteract/papermillseems to be one of the most
similar projects to this.  Cells are tagged as containing
parameters, which means that they can be overridden from the command
line. It still takes the view that this is mainly a notebook.nbconvertis the
default way for executing notebooks.  There is no default way to
pass arguments and output formats are designed to look like
notebooks.https://pypi.org/project/runipy/is a pretty basic script similar tonbconvert --executeit seems (deprecated in favor ofnbconvert --execute).Several tools callednbrun, some of which are deprecated in favor
ofnbconvert.https://github.com/takluyver/nbparameterisealso dynamically
replaces values in cells.https://github.com/NERSC/slurm-magicis IPython magic functions for
interacting with Slurm.  It doesn't do anything special about the
notebook format.  The%%sbatchmagic submits a cell as a Slurm
shell script and probably the%srunmagic runs a command line.
This makes a logical companion tonbscriptand is perhaps better
than an interface we might make.and many more...All of these accomplish the same thing but have different (a few) or
no (most) ways of passing parameters.Development status and maintnanceCurrently this is a usable alpha - the main invocations work, but get
too creative and expect problems!  There are tests to verify the
important stuff works, though.Maintainer: Richard Darst, Aalto University.  Feedback and
improvements encouraged.