Package: jptest2
JPTest2JPTest is a unit testing framework for Jupyter Notebooks and aims for fast test writing in less lines of code. It
creates the possibility to score and automatically grade exams with separate notebook (.ipynb) and test (.py) files.Quick StartJPTest relies onJupyter. If you need any other libraries for executing your notebook cells they
can be installed in your environment the usual way.The preferred way to use JPTest is in a virtual environment:python-mvenvvenvsourcevenv/bin/activateUsepipto download and install JPTest. Make sure not to install the first version of JPTest, which is missing the2at the end of the package name, as it is only available for compatibility reasons.pipinstalljptest2Now imagine that your student has to implement a Fibonacci function calledfibonacci, which accepts a single parameternand returns the n-th fibonacci number. Inside the notebook to submit, you prepared a cell with the tagtask-1. A
simple test could look like the following example:fromjptest2import*importasyncio# Create test with name "Task 1" and a maximum score of 1.# Execute *all* cells with tag "task-1" prior to executing the test function.# Please note: Every test function must be async!@JPTest('Task 1',max_score=1,execute=('task-1',))asyncdeftest_task1(nb:Notebook):# Create a reference to the function `fibonacci` inside the notebook.fib_fun_in_nb=nb.ref('fibonacci')# Receive five results from the fibonacci function.result=awaitasyncio.gather(fib_fun_in_nb(1),fib_fun_in_nb(2),fib_fun_in_nb(3),fib_fun_in_nb(4),fib_fun_in_nb(5))# Yield a tuple containing a condition, an award, a comment in case the# condition is false and a comment in case the condition is true.yieldresult==[1,1,2,3,5],1,'fib fun incorrect','fib fun correct'Let us assume the notebook file is callednotebook.ipynband the test file is calledtests.py. Call JPTest with the
following command:python-mjptest2notebook.ipynbtests.pyPlease note the test function is anasyncfunction!Table of ContentsContexts and ProcessesThe Execute ParameterExecute CodeReferencesAnnotations and ParametersFunction InjectionFunction ReplacingFunction TrackingSetup and Teardown MethodsOutput FormatsParallelizationRunning Without TestsLive PreviewOther KernelsContexts and ProcessesJPTest manages different processes. The first process that is started collects all annotations like@JPTestand
stores them together with their respecting test functions. Later, this process will control the startup of notebooks
and take over the evaluation. We refer to this process as the test context.Jupyter uses kernels that are started in a separate process. JPTest supports Python3 kernels and does not share
them between tests, so for each test at least one independent kernel process is started to run the contents of the
notebook. However, as we will see later, it is also possible to start multiple kernels per test. We refer to this set
of processes as the notebook context.JPTest always runs on an in-memory copy of the notebook and does not modify files, but tests and code in the notebook
still have the possibility to do so.The Execute ParameterOne can control in detail what code is executed prior to a test. Therefore, theexecuteparameter accepts different
types, which can also be nested recursively:String.If the parameter is of typestr, the value is considered as code and injected into the notebook.Tuple.If the parameter is of typetuple, the value is considered as tags. If there is one element in the tuple, every cell
with this tag is executed. If there are two elements in the tuple, every cell between the first appearance of the first
tag and the first appearence of the second tag (including) is executed.Function.If the parameter is a function (Callable), it will be executed in the notebook context usingexecute_fun.List.If the parameter is of typelist, every element will be executed in the order of its appearence, following the rules
stated above. Even though it is not actually needed because of list operations in Python, nested statements are possible
with it.Execute CodeThe easiest way to execute code in the notebook is via thecellsproperty. It returns a list of all cells present in
the notebook and allows to filter and execute them one by one.forcellinnb.cells:print(cell.tags)ifcell.type=='code':awaitcell.execute()The functionexecute_cellsrepresents a shortcut to select only code cells by tags prior to executing them in their
order of appearance.# execute all cells with tag `task-1`awaitnb.execute_cells('task-1')# execute all cells from `task-3` to `task-5`awaitnb.execute_cells(from_tag='task-3',to_tag='task-5')It is also possible to inject code into the notebook context.execute_codecreates a new code cell from the given
string, inserts it at the end of the notebook and executes it. Additional indentation of otherwise correct code is
possible.awaitnb.execute_code('''a = 5b = 10''')Please note that there are functionsstoreandstoresto store values in the notebook. Unlike the previous example
this also works with non-primitive types and references.ref1=awaitnb.store(5,'a')ref2=awaitnb.store(6)awaitnb.store(ref1,'copy_of_ref1')ref3,ref4=awaitnb.stores(b=10,c={'tiger':'dangerous'})ReferencesIt is possible to interact with objects and code in the notebook context. The most important class in this regard isNotebookReference. References are returned, for example, by therefandgetfunctions, represent objects in the
notebook context and may be used for interaction in various ways:receiveserializes the referenced object and transfers it from the notebook context to the test context.execute,
on the other hand, executes a statement without processing the result and transferring it to the test context.Callingawaiton aNotebookReferenceobject directly triggers a call toreceive.Access to an object's attributes or items is possible with the usual syntax. Note that the result is not evaluated
immediately and thus errors due to missing attributes or keys are carried over until the actual execution.References may be called like functions. The parameters are either other references, then these are resolved within
the notebook context, or local variables from the test context, then these are transferred into the notebook context
and used for the call. Function calls are also not executed immediately.For references to lists and other sequences there is a functionlento determine the length. The built-inlenfunction, however, cannot be used with theasync/awaitsyntax.my_fun_in_nb=nb.my_funmy_fun_return=my_fun_in_nb()my_dict_in_nb=nb.my_dictval_of_x=my_dict_in_nb['x']# Raises an exception if `my_fun` does not exist# or if it raises an exception itself.print(awaitmy_fun_return.receive())# You can omit the call to `receive`.print(awaitmy_fun_return)# Raises an exception if `my_dict` does not `x` is not a key in `my_dict`.print(awaitval_of_x)Pickle is used to serialize and deserialize objects. Therefore, it is also possible to transfer more complex objects
like Pandas DataFrames or NumPy Arrays.References to a notebook's objects can be used as parameters to call a function within another notebook. JPTest will
transfer the value to the notebook containing the function prior to calling it. This means the used reference has to be
serializable.Annotations and ParametersPreviously you have already seen the annotation@JPTest. It has two additional optional parameters.timeoutspecifies a timeout in secondsper cell. As you may have noticed, the notebook is passed as a parameter to the test
functionaftertheexecuteparameter is executed. You can setprepare_secondtoTrueto get a second notebook
with the same settings as a second parameter.@JPTest('Task 1',max_score=1,execute=('task-1',),prepare_second=True)asyncdeftest_task1(nb1:Notebook,nb2:Notebook):# `nb1` equals `nb2`, but they were created and prepared independently!passFurthermore, there is@JPTestGetif you are only interested in data stored within the notebook. To this annotation
you pass a name, a maximum score, a timeout and an execute command. It further accepts a list of names that are
variables inside the notebook. All of these are transferred to the test context and used as parameters for your test
function. You can not access the notebook using this annotation!@JPTestGet('Task 1',max_score=1,execute=('task-1',),get=['first_var','second_var'])asyncdeftest_task1(first_value,second_value):# `first_value` and `second_value` are the received values.passThe last annotation is@JPTestComparison. It allows using two notebooks prepared in different ways inside the test
function. We use this mainly to compare the student's results with those from a sample solution.defimport_pandas():# noinspection PyUnresolvedReferencesimportpandasaspddefsample_solution():correct_df=pd.read_csv('my_dataset.csv')# Everything passed to `prepare` is executed in both notebooks independently.# Everything passed to `execute_left` is executed in the first notebook.# `execute_right` does the same in the second notebook.# `hold_left` expects a list of variable names to copy to the test context# from the first notebook. `hold_right` does the same to the second notebook.# Every received value is used as a parameter for the test function.@JPTestComparison('Task 1',max_score=1,execute=import_pandas,prepare_left=('task-1',),hold_left='students_df',prepare_right=sample_solution,hold_right='correct_df')asyncdeftest_task1(students_val,correct_val):passFunction InjectionThere are two ways to inject functions:The first methodinject_funtransfers a function to the notebook context and returns a reference. This can be called
as described before or passed as a parameter to another function.deffun(i:int):returni+1injected=awaitnb.inject_fun(fun)result=awaitinjected(5)# `result` equals `6`.You can also send classes to the notebook context. But there is no way to transfer needed superclasses automatically
as well.The second methodexecute_funexecutes a function's body in the notebook context while the header is only used in the
test context. This makes it possible to write syntactically correct code with alle benefits of analysis within an IDE,
although it is later executed in the notebook context.deffun(i:int):k=i+1# `i` has to be available in the notebook context!awaitnb.execute_fun(fun)# `k` is defined globally in the notebook context# after the execution.Function ReplacingFunctions in the notebook context can be replaced with others, for example to skip network requests and return a fixed
response instead to speed them up.awaitnb.execute_code('''from time import sleepdef my_fun():sleep(10)return 1''')defreplacement():return2asyncwithnb.replace_fun('my_fun',replacement):# executes `replacement` in notebook context.result=awaitnb.ref('my_fun')()# prints `2`print(result)Function TrackingFurthermore, it is possible to track function calls. This may be used to check if an implementation uses recursion.
In addition, the parameters and return values used can be extracted.awaitnb.execute_code('''def fib(i):return i if i <= 1 else fib(i-1) + fib(i-2)''')asyncwithnb.track_fun('fib')ascalls:awaitnb.ref('fib')(15).execute()print(len(awaitcalls)>1000)Setup and Teardown MethodsUse@JPSetupand@JPTeardownto annotateasyncfunctions. Setup functions are run prior to all tests and teardown
functions after all tests have completed.@JPSetupasyncdefsetup():print('setup')@JPTeardownasyncdefteardown():print('teardown')Multiple setup or teardown functions are run in parallel.Output FormatsThe default output format is JSON. You can switch it to Markdown using the command line flag--md.ParallelizationSince all notebook kernels are started in different processes, multicore processors can be fully utilized. However,
there are a few things to keep in mind:The test context can become a bottleneck because it uses only one thread. Therefore, the notebooks should work as
independently as possible and the test context should only be used for coordination and evaluation.Keep inter-process communication to a minimum and outsource computationally intensive operations to the notebook
context.Use the parameter--teststo limit the number of concurrently running tests.Running Without TestsIf no test file is given on startup, JPTest will choose a default test set. It executes all cells once in the correct
order, does not score and passes exceptions. This can be used to check notebooks for syntax errors, determine if
libraries are missing within an image or if data sets have not been shipped.Use the command line parameter--quietto suppress any output other than exceptions and stacktraces.Live PreviewThe live preview is activated with the--liveswitch. It monitors changes to the given files and automatically reruns
the evaluation as soon as one of them is modified. This mode is useful for developing tests or for demonstrations.Install the packagejptest2[demo]to receive the optional dependencies. Theclearcommand is used to clear the
terminal window and therefore must be available.Other KernelsWhile the main focus is on Python notebooks, development to support other kernels is possible. Set thekernelparameter to specify which language is used in the notebook. Besides Python3, two databases are supported at the moment.
However, they do not start an actual Jupyter Kernel but send the queries directly to a connected database.SQLiteusingjptest2[sqlite]DuckDBusingjptest2[duckdb]