Package: curricula-grade
Curricula GradeThe other half ofcurricula's core functionality is automated grading.
In order to use automated grading, material writers have to implement tests using thecurricula_gradetoolkit.
Automated grading for individual problems can be combined into composite assignments, complete with problem weights and other metadata.Writing TestsWhile somewhat bulkier than unit-test frameworks, the additional syntax and runtime overhead backing theGradermake it much easier to generate and manage reports for students.
Let's walk through anexample.
Note that we'll be usingcurricula_grade_cppas our example assignment is written in C++, but the framework itself is language-agnostic.fromcurricula_grade.shortcutsimport*fromcurricula_grade.setup.commonimportcheck_file_existsfromcurricula_grade_cpp.setup.common.gppimportgpp_compile_objectfromcurricula.library.filesimportdelete_filefrompathlibimportPathroot=Path(__file__).absolute().parentgrader=Grader()GPP_OPTIONS=("-std=c++14","-Wall")To start off, we include several basic utilities fromcurricula_grade.
We also import two commons functions,check_file_existsandgpp_compile_object.
Lastly, we includedelete_fileandPathfor generalized file system access.Using our imports, we create aGraderinstance to register tasks to.
TherootandGPP_OPTIONSare just constants for reference that we'll use for convenience later.
With the boilerplate out of the way, we start by writing a task that checks if a file we're expecting the student to implement is present in the submission:Checks@grader.register(tags={"sanity"},graded=False)defcheck_submission(submission:Submission,resources:dict)->SetupResult:"""Check submission.cpp is present."""header_path=resources["source"]=submission.problem_path.joinpath("submission.cpp")returncheck_file_exists(header_path)The first thing to notice is thegrader.registerdecorator we're using.
This line indicates to the grader that we want it to run this function, which we'll refer to generally as a runnable, during grading.
Of the several ways we can register a task to a grader, this is the simplest.The arguments passed tograder.registerdefine metadata about the task.
In this particular case, we're only specifying a tag that the task falls under.
We can use tags to specify subsets of tasks to run while grading, which can be useful for sanity checks in an online submission, etc.
Other metadata that can be assigned in the registration call include the task name and description, which default to the function name and docstring.When a task is executed, it looks at the arguments in its runnable's signature and injects the corresponding resources from the resource pool by name.
We ask for the full resourcesdictby includingresourcesas a parameter to the runnable.
By default,resourcescontains:ASubmission, which describes the submission currently being graded.AContext, which includes options passed to the grader, either from the command line or otherwise.A reference to theresourcesmap itself.Anything we put in theresourcesmap manually.
In this particular task, we're putting a new item calledsourceinto the resource map that points to the path to the file we want to grade.
This simply makes it easier to reference later; we can just include asourceparameter in theRunnablesignature.The last important detail in this example is the annotated return type of the runnable,SetupResult.
Because we may find ourselves deserializing grading results down the road (to summarize a batch, generate a formatted report, etc.), we need to be able to determine what kind ofResultsubclass we're expecting without executing the runnable.
If we use a decorator to register the runnable, it must either have an annotated return type, an attributeresult_type, or you must subscript the registrar with the result type.Let's move on to compiling our submission.Compilation@grader.register(passing={"check_submission"},tags={"sanity"},graded=False)defbuild_submission(source:Path,resources:dict)->SetupResult:"""Compile the submission."""result,resources["binary"]=gpp_compile_object(source,destination_path=source.parent.joinpath("binary"),gpp_options=GPP_OPTIONS)returnresultThis segment builds a filesubmission.cppin a target directory usingg++.
As specified by the registrar invocation, this task depends oncheck_submissionpassing.
Ifcheck_submissionreturns aSetupResult(passing=False), this task will never be executed.Our call togpp_compile_objecthere does exactly what you might expect: invokesg++in a subprocess to compile the C++ file.
You can take a look at the source in thecurricula_grade_cpplibrary.
Notice that nothing there is particularly fancy or magical; it's simply boilerplate I've encapsulated for convenience.Next, we'll actually run a test:Tests@grader.register()deftest_simple_addition(binary:ExecutableFile)->CorrectnessResult:"""Check 1 + 1."""runtime=binary.execute("1","1")ifruntime.stdout.strip()!=b"2":returnCorrectnessResult(passing=False,actual=runtime.stdout.strip().decode(),expected="2",error=Error(description="incorrect result"),details=dict(runtime=runtime.dump()))returnCorrectnessResult(passing=True)This is an actual test case, and it affirms any given submission writes the sum of the first two arguments tostdout.
In a proper problem, there will most likely be many more of these.
In the first line of the runnable body, we invoke the compiled binary in a subprocess, passing two arguments:1and1.
Let's take a look at an example submission:#include<iostream>usingnamespacestd;intmain(intargc,char**argv){cout<<stoi(argv[1])+stoi(argv[2])<<endl;return0;}We can determine whether this submission prints what we expect it to by comparing theruntime.stdout.
If it's not2, we return a failing result with some extra details on what happened.
Otherwise, we return a passing result.CleanupFinally, let's add a cleanup task to avoid leaving extraneous files in our submissions.
The only thing we have to do is delete our binary, which we can again ask for via injected arguments:@grader.register[CleanupResult](tags={"sanity"},graded=False)defcleanup_submission(binary:ExecutableFile):"""Delete the binary."""delete_file(binary.path)Notice something new is going on here: instead of using the return annotation, we're providing the result type via bracket notation in the registration decorator.
Why?
If a runnable returns None, the task executor will attempt to instantiate a defaultresult_type.
In this case, it will substitute aCleanupResult.default()for the result automatically.
It would not be semantically correct to use a return annotation on this runnable because it does not return anything, and so a secondary mechanism is provided to specifyresult_type, i.e.grader.register[result_type].SummaryThe entire grading script is as follows:fromcurricula_grade.shortcutsimport*fromcurricula_grade.setup.commonimportcheck_file_existsfromcurricula_grade_cpp.setup.common.gppimportgpp_compile_objectfromcurricula.library.filesimportdelete_filefrompathlibimportPathroot=Path(__file__).absolute().parentgrader=Grader()GPP_OPTIONS=("-std=c++14","-Wall")@grader.register(tags={"sanity"},graded=False)defcheck_submission(submission:Submission,resources:dict)->SetupResult:"""Check submission.cpp is present."""header_path=resources["source"]=submission.problem_path.joinpath("submission.cpp")returncheck_file_exists(header_path)@grader.register(passing={"check_submission"},tags={"sanity"},graded=False)defbuild_submission(source:Path,resources:dict)->SetupResult:"""Compile the submission."""result,resources["binary"]=gpp_compile_object(source,destination_path=source.parent.joinpath("binary"),gpp_options=GPP_OPTIONS)returnresult@grader.register()deftest_simple_addition(binary:ExecutableFile)->CorrectnessResult:"""Check 1 + 1."""runtime=binary.execute("1","1")ifruntime.stdout.strip()!=b"2":returnCorrectnessResult(passing=False,actual=runtime.stdout.strip().decode(),expected="2",error=Error(description="incorrect result"),details=dict(runtime=runtime.dump()))returnCorrectnessResult(passing=True)@grader.register[CleanupResult](tags={"sanity"},graded=False)defcleanup_submission(binary:ExecutableFile):"""Delete the binary."""delete_file(binary.path)Running the GraderIn order to grade an assignment, you will first need to make sure that you've constructed the grading artifact correctly.
It is recommended you usecurricula-compilefor this, but if you insist on doing it manually, the following directory structure is required:grading/
  index.json
  problem1/
    tests.py
    ...
  problem2/
    tests.py
    ...
  ...Once you've created the grading artifact, you can run grading from the command line.
Currently, the options are as follow:usage: curricula grade run [-h] --grading GRADING [--skip] [--report] [--concise] [--progress] [--sample SAMPLE] [--tags TAGS [TAGS ...]] [--tasks TASKS [TASKS ...]] [--summarize] [--thin] [--amend]
                           (-f FILE | -d DIRECTORY)
                           submissions [submissions ...]

positional arguments:
  submissions           run tests on a single target

optional arguments:
  -h, --help            show this help message and exit
  --grading GRADING, -g GRADING
                        the grading artifact
  --skip                skip reports that have already been run
  --report, -r          whether to log test results
  --concise, -c         whether to report concisely
  --progress, -p        show progress in batch
  --sample SAMPLE       randomly sample batch
  --tags TAGS [TAGS ...]
                        only run tasks with the specified tags
  --tasks TASKS [TASKS ...]
                        only run the specified tasks
  --summarize           summarize after running batch
  --thin                shorten output for space
  --amend               amend any existing report at the destination
  -f FILE, --file FILE  output file for single report
  -d DIRECTORY, --directory DIRECTORY
                        where to write reports to if batchedIf you have all of the submissions in a folder by username, i.e.submissions/noahbkimetc., you would do something like this:$curriculagraderun--gradinggrading/--progress--concise--summarizesubmissions/*/-dreports/Once the grader is finished, there will be a*.report.jsonfile inreports/corresponding to each of the submissions in the arguments.
You can then use included tools to format these reports in Markdown, etc.Grader OutputGrading an assignment will yield a serializableAssignmentReport, which is composed ofProblemReportobjects for each problem graded automatically.
This report, formatted as JSON, will look something like this:{"problem":{"short":"test","title":"Test"},"automated":{"partial":false,"results":{"check_submission":{"complete":true,"passing":true,"score":null,"error":null,"messages":[],"task":{"name":"check_submission","description":"Check submission.cpp is present."},"details":{}},"build_submission":{"complete":true,"passing":true,"score":null,"error":null,"messages":[],"task":{"name":"build_submission","description":"Compile the submission."},"details":{"runtime":{"args":["g++","/Users/noahbkim/Repositories/curricula/curricula-grade-cpp/example/submission.cpp","-std=c++14","-Wall","-o","/Users/noahbkim/Repositories/curricula/curricula-grade-cpp/example/binary"],"cwd":null,"stdin":null,"stdout":"","stderr":"","elapsed":1.189917291,"code":0,"timeout":null,"timed_out":false,"raised_exception":false,"exception":null}}},"test_simple_addition":{"complete":true,"passing":true,"score":null,"error":null,"messages":[],"task":{"name":"test_simple_addition","description":"Check 1 + 1."},"details":{},"expected":null,"actual":null},"cleanup_submission":{"complete":true,"passing":true,"score":null,"error":null,"messages":[],"task":{"name":"cleanup_submission","description":"Delete the binary."},"details":{}}}}}