Package: mcpmark
(Multiple) component markingSome tools I use when marking homework with with single or multiple Jupyter
notebook components.The notebooks may have questions for manual marking, and plots for marking.They assume some Canvas](https://www.instructure.com/canvas) conventions of
naming files, and grade output CSV format.The tools consist primarily command line utilities, with some supporting code
in a utility library.QuickstartFor single component submission:COMPONENTS_DIR=components
mcp-check-unpack
mcp-prepare-components
mcp-find-duplicates $COMPONENTS_DIR/*/*.Rmd
mcp-cp-models
mcp-extract-manual
rnbg-allow-raise $COMPONENTS_DIR/*/*.Rmd --show-error
mcp-extract-plots
mcp-grade-nbs
# Review `<component>/marking/autograde.md`.
# Rerun after any edits.
mcp-grade-nbs
mcp-grade-component
mcp-scale-combineFor multiple component submission:COMPONENTS_DIR=components
mcp-check-unpack
mcp-prepare-components
mcp-find-duplicates $COMPONENTS_DIR/*/*.Rmd
mcp-cp-models
# For each component
    COMPONENT=my_component
    rnbg-allow-raise $COMPONENTS_DIR/$COMPONENT/*.Rmd --show-error
    mcp-grade-nbs $COMPONENT
    # Review `$COMPONENT/marking/autograde.md`.
    # Rerun after any edits.
    mcp-grade-nbs $COMPONENT
    mcp-extract-manual $COMPONENT
    mcp-extract-plots $COMPONENT
    mcp-grade-component $COMPONENT
# Finally
mcp-scale-combineGetting set upMake a virtual environment / Conda environment for running the marking code, and set yourself up in that environment:python -m virtualenv ~/envs/marking-env
source ~/envs/marking-env/bin/activateorconda create --name marking-env
conda activate marking-env
conda install pipTo install locally from the repository, you will needflit:pip install flitThen install MCPMark with its dependencies:cd mcpmark  # Directory containing this README
flit install -sTest all is working as expected with:pip install -r test-requirements.txt
pytest mcpmarkA typical marking runMake sure you have activated the environment above with e.g.source ~/envs/marking-envorconda activate marking-envMake a directory for marking, call thishomework1or similar.cd homework1Download submissions (.zipfiles for multiple notebook submission,.ipynbfiles for single notebook submission). Download some directory e.g.submissionsin current directory. There should be one.zipfile per
student in the case of multiple notebook submissions, or one.ipynbfile
per student in case of single submissions.Download Canvas marks CSV file to this (homework1) directory.Editassign_config.yaml--- seedoc/for an example.   Use thecomponentsfield to name and define components.  Each component corresponds
to one notebook, so there will be one component for single notebook
submissions, and multiple component for multiple notebook submissions.In what follows below, a "component name" is the name you have given for
a single notebook assignment in theassign_config.yamlfile.Develop script to identify notebooks by their content - seedocfor an
example, andmcpmark/cli/prepare_components.pyfor code using this script.
This allows Mcpmark to check that a student does have a matching notebook for
each required component.Runmcp-check-unpack.  If any errors arise, check and maybe change the
submission filenames.Runmcp-prepare-components.  This will check that all the students in the
relevant student files have got matching notebook submissions for all
required components.  The error message should tell you what is missing.  If
you discover that the warning is a false positive, and you were not expecting
this student to submit (yet), then fill in their ID in theknown_missinglist of theassign_config.yamlfile, to tell Mcpmark not to check their
submissions.  Then re-runmcp-prepare-components, repeating until you get
no errors.In what follows, you can generally omit the<component_name>argument when
you only have one component.For items below, assume scriptrerunis on the path and has contentswhile true; do $@; donePer notebook / component:Develop tests inmodel/<component_name>/testsdirectory.Test tests withgrade_oknb.py.Copy tests etc into components directory withmcp-cp-modelse.g.mcp-find-duplicates components/my_component/*.Rmdto analyze
duplicates, write summary into some file, sayreport.md.Check notebook execution withmcp-run-notebooks <path_to_notebooks>.
Consider running this with e.g.rerun mcp-run-notebooks components/panderingto continuously test notebooks.Move any irreparable notebooks tobrokendirectory, and mark inmarking/broken.csvfile.mcp-extract-manual <component_name>(component name optional for single
component submissions). Edit notebooks where manual component not found.
Maybe e.g.rerun mcp-extract-manual pandering.Mark generated manual file in<component>/marking/*_report.md.Check manual scoring with something likemcp-manual-scores components/lymphoma/dunleavy_plausible_report.md.  Or you can leave
that until grading the whole component withmcp-grade-component.mcp-extract-plots <component_name>(component name optional for single
component submissions).  Editmarked/plot_nbs.ipynbto add marks.Run auto-grading withmcp-grade-nbs <component_name>(<component_name>) is optional if a single component.Review<component>/marking/autograde.md.Update any manual fixes with#M:notation to add / subtract marks.
These are lines in code cells / chunks, of form#M: <score_increment>-- e.g.#M: -2.5.  They reach the final score viamcp-grade-components.Final run ofmcp-grade-nbsmcp-grade-component <component_name>; (<component_name>) is optional
if a single component.When done:mcp-scale-combineto rescale the component marks to their out-of figure
given inassign_config.yaml, and generate the summary.csvfile.  Do this
even when there is only one component (in order to do the rescaling).mcp-export-marksto convert the output ofncp-rescale-combinesto
a format for import into Canvas.Utilitiesmcputils- various utilities for supporting the scripts.