Package: jupyter-autograde
autogradeautogradeis a toolbox for testingJupyternotebooks. Its features include execution of notebooks (optionally
isolated via docker/podman) with consecutive unit testing of the final notebook state. An audit mode allows for refining
results (e.g. grading plots by hand). Eventually,autogradecan summarize these results in human and machine-readable
formats.setupInstallautogradefromPyPIusingpiplike thispipinstalljupyter-autogradeAlternatively,autogradecan be set up from source code by cloning this repository and installing it
usingpoetrygitclonehttps://github.com/cssh-rwth/autograde.git&&cdautograde
poetryinstallIf you intend to use autograde in a sandboxed environment
ensurerootless dockerorpodmanare available on your system. So far, only rootless mode is supported!UsageOnce installed,autogradecan be invoked via theautogradecommand. If you are using a virtual environment (which
poetry does implicitly) you may have to activate it first. Alternative methods:path/to/python -m autograderunsautogradewith a specific python binary, e.g. the one of your virtual
environment.poetry run autogradeif you've installedautogradefrom sourceTo get an overview over all options available, runautograde[subcommand]--helpTestingautogradecomes with some example files located in thedemo/subdirectory that we will use for now to illustrate the workflow. Runautogradetestdemo/test.pydemo/notebook.ipynb--target/tmp--contextdemo/contextWhat happened? Let's first have a look at the arguments ofautograde:demo/test.pya script with test cases we want to applydemo/notebook.ipynbis the a notebook to be tested (here you may also specify a directory to be recursively searched
for notebooks)The optional flag--targettellsautogradewhere to store results,/tmpin our case, and the current working
directory by default.The optional flag--contextspecifies a directory that is mounted into the sandbox and may contain arbitrary files
or subdirectories. This is useful when the notebook expects some external files to be present such as data sets.The output is a compressed archive that is named something likeresults-Member1Member2Member3-XXXXXXXXXX.zipand which has the following contents:artifacts/: directory with all files that where created or modified by the tested notebook as well as rendered
matplotlib plots.code.py: code extracted from the notebook includingstdout/stderras commentsnotebook.ipynb: an identical copy of the tested notebookrestults.json: test resultsAudit ModeThe interactive audit mode allows for manual refining the result files. This is useful for grading parts that cannot be
tested automatically such as plots or text comments.autogradeauditpath/to/resultsOverviewAuditingReport PreviewGenerate ReportsThereportsub command creates human readable HTML reports from test results:autogradereportpath/to/result(s)The report is added to the results archive inplace.Patch Result ArchivesResults from multiple test runs can be merged via thepatchsub command:autogradepatchpath/to/result(s)/path/to/patch/result(s)Summarize Multiple ResultsIn a typical scenario, test cases are not just applied to one notebook but many at a time. Therefore,autogradecomes
with a summary feature, that aggregates results, shows you a score distribution and has some very basic fraud detection.
To create a summary, simply run:autogradesummarypath/to/resultsTwo new files will appear in the result directory:summary.csv: aggregated resultssummary.html: human readable summary reportSnippetsWork with result archives programmaticallyFix score for a test case in all result archives:frompathlibimportPathfromautograde.backend.local.utilimportfind_archives,traverse_archivesdeffix_test(path:Path,faulty_test_id:str,new_score:float):forarchiveintraverse_archives(find_archives(path),mode='a'):results=archive.results.copy()forfaulty_testinfilter(lambdat:t.id==faulty_test_id,results.unit_test_results):faulty_test.score_max=new_scorearchive.inject_patch(results)fix_test(Path('...'),'...',13.37)Special Test CasesEnsure a student id occurs at most once:fromcollectionsimportCounterfromautogradeimportNotebookTestnbt=NotebookTest('demo notebook test')@nbt.register(target='__TEAM_MEMBERS__',label='check for duplicate student id')deftest_special_variables(team_members):id_counts=Counter(member.student_idformemberinteam_members)duplicates={student_idforstudent_id,countinid_counts.items()ifcount>1}assertnotduplicates,f'multiple members share same id ({duplicates})'