Package: coursera-autograder
This command-line tool is a software development toolkit to help instructional
teams author asynchronous graders for Coursera (typically programming
assignments). Coursera’s asynchronous grading environment is based upondocker. While use of this tool is by no means
required to develop the docker container images, we believe it is helpful in the
endeavour. See below for brief descriptions of this tool’s capabilities.InstallationYou can installcoursera_autograderusing pip! Simply run:python3 -m pip install coursera-autograderUpgrade thecoursera_autograderpackage by running:pip install coursera-autograder --upgradeIf you’ve previously installedcoursera_autograderfrom source, you can uninstall it by
navigating to the directory where it is installed and running:python setup.py develop --uninstallAlternatively, you may use the following commands to install from source:git clone https://github.com/coursera/coursera_autograder.git
cd coursera_autograder
virtualenv -p python3 venv (Optional)
source venv/bin/activate (Optional)
python setup.py develop
pip install -r test_requirements.txt
pip install -e .If you would like to separate your build environments, we recommend installingcoursera_autograderwithin a virtual environment.pipis a python package manager.
If you do not havepipinstalled on your machine, please follow the
installation instructions for your platform found at:https://pip.pypa.io/en/latest/installing.html#install-or-upgrade-pipThe tool includes its own usage information and documentation. Simply run:coursera_autograder -hor:coursera_autograder --helpfor a complete list of features, flags, and documentation.Note: the tool requiresdockerto already be installed on your machine.
Please see the dockerinstallation instructionsfor further
information.Subcommandsgrade localThisgrade localsubcommand loosely replicates the production grading environment on
your local machine. Note that because the GrID system has
adopted a defense-in-depth or layered defensive posture, not all layers of the
production environment can be faithfully replicated locally.grade localruns a local grader
container image on a sample submission on the local file system, provided as part of the command. This command is intended
to help instructional teams verify new versions of their graders correctly
handle problematic submissions.Examples:coursera_autograder grade local $MY_CONTAINER_IMAGE_TAG /path/to/sample/submission/ $ENV_VAR_JSON--dst-dir~/Desktopinvokes the grader passing in the sample submission into the grader.When the grade local is successful, you can verify that feedback.json is produced in dst-dir (in this case it is ~/Desktop).$ENV_VAR_JSON is a json string like’{“partId”: “Zb6wb”}’,coursera_autograder grade local--helpdisplays the full list of
flags and options available.coursera_autograder grade local python_grader ./submission'{"partId":"5ShhY"}'--dst-dir~/DesktopPlease make sure there is only the correct solution file in the submission directory (./submission).In contrast to this local tester, Coursera’s production system will also set these environment variables for internal purposes. In local testing, it is possible to specify these as well with the environment variable JSON, although it’s completely up to the grading Docker you create to use them or not. In typical usage, you would not set or read these variables.filename- The original filename of the file the student has chosen to submit, prior to being renamed on the server automatically. To avoid confusion, note that thegrade localcommand takes a directory path, not a file path; the directory should contain a submission file with the same filename as the “suggested filename” you’ve configured for the assignment as published on Coursera’s UI, and your autograder should also look for the file with the “suggested filename.” Thefilenameenvironment variabledoes notspecify the “suggested filename”. Also, autograders live in production will find the submitted file has already been renamed, so thefilenameenv var does not have much usefulness inside the grader. One use case might be to display a warning to learners if the file they chose to submit does not have the correct file extension prior to being automatically renamed.userId- a unique string Coursera uses to disambiguate learners.uploadAllows an instructional team to upload their containers to Coursera without
using a web browser. It is designed to even work in an unattended fashion (i.e.
from a jenkins job). In order to make the upload command work from a Jenkins
automated build machine, simply copy the~/.courseradirectory from a working
machine, and install it in the jenkins home folder. Beware that the oauth2_cache
file within that directory contains a refresh token for the user who authorized
themselves. This refresh token should be treated as if it were a password and
not shared or otherwise disclosed!To find the course/branch id, item id, and part id:Go to the web authoring interface for your programming assignment. There, the URL will be of the form:/:courseSlug/author/outline/programming/:itemId/Thepart idcan be found in the authoring user interface for each part.To convert thecourseSluginto acourseId, you can take advantage of ourCourse APIputting in the appropriatecourseSlug. For example, given a course slug ofdeveloper-iot, you can query the course id by making the request:https://api.coursera.org/api/onDemandCourses.v1?q=slug&slug=developer-iotThe response will be a JSON object containing anidfield with the value:iRl53_BWEeW4_wr--Yv6AwIf the grader that you’re looking to upload/update is not in theOriginal Versionof the course, you’ll most likely need to get a branch id. To this, you’ll need thecourseIdand theversionName. Make the following request:https://api.coursera.org/api/authoringBranchProperties.v1?q=course&courseId={course_id}&fields=propertiesSearch for your version name (it’ll be in theproperties.namefield), and find the associatedid(it should look something like:authoringBranch~xxxxxxxxxxxxxxxxxxxxxx. (Please note that this form of branch id is only applicable for non-original versions of the course. For the original version, the id will be a twenty-two character long string with no prefix).The uploaded grader can be linked to multiple (itemId, partId) pairs without making duplicate uploads by using the--additional_item_and_partflag.This command can also be used to customize the resources that will be allocated
to your grader when it grades learner submissions. The CPU, memory limit and
timeout are all customizable.--grader-cputakes a value of 1, 2 or 4, representing the number of vCPUs.
the grader will have access to when grading. The default is 1.--grader-memory-limittakes a value between 4096 to 16384, increnment of 1024. representing the
amount of memory in MB the grader will have access to when grading. The
default is 4096 (4GB).Not all combinations of cpu and memory are supported. The supported combinations is listed here:For 1024 (1 vCPU), Memory needs to be between 2048 (2GB) and 8192 (8GB) in increments of 1024 (1GB).For 2048 (2 vCPU), Memory needs to be between 4096 (4GB) and 16384 (16GB) in increments of 1024 (1GB).For 4096 (4 vCPU), Memory needs to be between 8192 (8GB) and 16384 (16GB) in increments of 1024 (1GB)--grader-timeouttakes a value between 300 and 3600, representing the
amount of time the grader is allowed to run before it times out. Note this
value is counted from the moment the grader starts execution and does not
include the time it takes Coursera to schedule the grader. The default value
is 1200.Examples:coursera_autograder upload $PATH_TO_IMAGE_ZIP_FILE $COURSE_OR_BRANCH_ID $ITEM_ID $PART_IDuploads the specified grader container image to Coursera, begins
the post-upload processing, and associates the new grader with the
specified item part in a new draft. Navigate to the course authoring UI
or use thepublishcommand to publish the draft to make it live.coursera_autograder upload $PATH_TO_IMAGE_ZIP_FILE $COURSE_OR_BRANCH_ID $ITEM_ID $PART_ID--additional_item_and_part$ITEM_ID2 $PART_ID2 $ITEM_ID3 $PART_ID3uploads
the specified graded container image to Coursera, begins the post-upload procesing,
and associates the new grader with all the three item_id part_id pairs.
Navigate to the course authoring UI for each item to publish the draft to make it live.coursera_autograder upload--helpdisplays all available options
for theuploadsubcommand.zip-rPythonGrader.zip .(Make sure you are in the directory containing the Dockerfile. This must be the top level directory)coursera_autograder upload ./PythonGrader.zipiRl53_BWEeW4_wr--Yv6AwrLa7F Zb6wbget_resource_limitsAllows an instructional team to view the resource limits (vCPU’s, MiB, timeout) allocated to the grader for a given programming assignment.
It requires the instructor to provide the course id, item id, and part id to identify the specific programming assignment. Instructions on
how to find these values can be found in the previous section for theuploadcommand.Usage:coursera_autograder get_resource_limits $COURSE_OR_BRANCH_ID $ITEM_ID $PART_IDupdate_resource_limitsAllows an instructional team to update the resource limits (vCPU’s, MiB, timeout) allocated to the grader for a given programming assignment.
It requires the instructor to provide the course id, item id, and part id to identify the specific programming assignment. Instructions on
how to find these values can be found in the previous section for theuploadcommand. In addition, the instructor must provide the values
they wish to update by using the parameter flags--grader-cputo update the allocated vCPU’s--grader-memory-limitto update the memory limit--grader-timeoutto update the timeout thresholdIf a certain parameter is not provided, then we will simply use the previously existing value. Note that there are restrictions on which
combinations of CPU’s and memory values are valid. These restrictions can be found in theuploadsection above.Usage:coursera_autograder update_resource_limits $COURSE_OR_BRANCH_ID $ITEM_ID $PART_ID--grader-cpu$CPU--grader-memory-limit$MEMORY--grader-timeout$TIMEOUTconfigureMakes sure that the instructor is able to communicate with the coursera.org API servers with the correct authentication.Usage:coursera_autograder configurecheck-authcoursera_autograder configuredisplay-auth-cachelist_gradersLists all graders uploaded to the given course.Usage:coursera_autograder list_graders $COURSE_IDget_statusGets the upload status of a grader given its executor id. The executor id can be found by using thelist_graderscommand.Usage:coursera_autograder get_status $EXECUTOR_IDBugs / Issues / Feature RequestsPlease reach out to your partner support teams to file an enhancement request or report a bug. While we check in on the issues logged on this repository from time to time, reaching out to your partner support teams will likely provide you with a faster response. We appreciate your support and dedication to improving this tool for all Coursera partners!Supported PlatformsNote: We do not have the bandwidth to officially support this tool on windows.
That said, patches to add / maintain windows support are welcome!Developing / ContributingWe recommend developingcoursera_autograderwithin a pythonvirtualenv.
To get your environment set up properly, do the following:virtualenv venv
source venv/bin/activate
python setup.py develop
pip install -r test_requirements.txtTestsTo run tests, simply run:nosetests, ortox.Code StyleCode should conform to pep8 style requirements. To check, simply run:pep8 coursera_autograder tests