Package: penngrader
PennGraderWelcome to the PennGrader!Here at PennGrader, we believe that learning comes from lots of practice...and from making lots of mistakes.After many years as a student I found myself very frustrated in the following homework timeline: struggle on a homework assignment for weeks, submit something that may or may not be right and then wait a few more weeks to receive any type of feedback, at which point I had forgotten all about the homework. After many years as a TA, I also found myself very frustrated with the common auto-grading tools, the hours and hours of manual grading and the onslaught of re-grade requests that came thereafter.From these frustrations, the PennGrader was born!The PennGrader was built to allow students to get instant feedback and many opportunities for re-submission. After all, programming is about making mistakes and learning from feedback! Moreover, we wanted to allow TAs and Instructors to write their homework in any way they pleased, without having to worry about the structure of a specific auto-grader. The examples below are done using Jupyter Notebooks which is the most common use case, but you can use this for normal Python homework as well.Here is what a student sees in his Homework Notebook. All a student has to do is write her solution and run the auto-grading cell.Through the magic of AWS Lambdas, the student's answer (in this case theaddition_functionobject) is packaged and shipped to the backend where it is checked against the teacher-defined test case. Finally, a score is returned. All students' scores are saved in the backend and are easily accessible to the instructors. If at first they don't succeed, they can learn from their mistakes and try again.  (Yes, if you'd like, can set a maximum number of daily submissions to incentivize students to start early). This "grader" function can easily be used from any Jupyter notebook (even Google Colab), all you have to do is topip install penngrader. See templates below.Ok, ok, you might be saying to yourself: "That looks easy enough, but what about us TAs, we want something that simple too!" Well, look no further. The TAs/Instructors' experience is just as seamless. All TAs will share aTeacher_Backendnotebook, which contains all the test case functions. The logic of how testing is done is simple: whatever Python object gets passed through theanswerfield in thegrade(...)function (see above) will be the input to the test case function (see below). In the above example,addition_functionis passed as the answer to a test case named"test_case_1". Therefore, the TAs will need to write atest_case_1(addition_function)function in theTeacher_Backendnotebook, as follows:As you can see, this function tests thataddition_function(1,2) == 3, if correct it adds 5 points to thestudent_score. The test must then return an integer tuple(student_score, max_score), which is what will be displayed to the student. As you can see this type of test function gives the Instructor complete flexibility on what to test and how much partial credit to give. Remember that the answer that gets passed to the test case could be anything... a function, a class, a DataFrame, a list, a picture... anything! The PennGrader automatically serializes it and all its dependencies and ships to AWS for grading.To create homework for your class you will need a courseSECRET_KEY. We will open to the public soon, however for now contact me atleonardo.murri1995@gmail.comif you are interested in using the PennGrader for your class.PennGrader_Homework_Template.ipynbPennGrader_TeacherBackend.ipynbDownload these two notebooks and launch them via Jupyter. They will show you how to add grading cells in your homework notebook and add write/update test cases via the teacher backend, as well as view student's grades.Behind the scenes...In the following section, I will go into detail about the system implementation. Below is the system design overview we will go into.ClientsThere are two pip installable clients, one for students and one for instructors. You can install these two clients by runningpip install penngraderin your favorite terminal.  When creating a new homework download theHomework_Template.ipynband theTeacherBackend.ipynbnotebooks and follow the instructions. More details are presented below.Student's Client: PennGraderThe student's client will be embedded in the homework release notebook. Its main purpose will be to interface the student's homework with the AWS backend. This client is represented by thePennGraderclass which needs to be initialized by the instructor when writing the homework as follows. Note: the HOMEWORK_ID needs to be filled in before releasing the notebook. The student should only enter his or her's student id.import penngrader.grader
grader = PennGrader(homework_id = HOMEWORK_ID, student_id = STUDENT_ID)The HOMEWORK_ID is the string obtained when creating new homework via the teacher backend, see below.STUDENT_ID is the student defined variable representing their 8-digit PennID. The student will need to run this cell at the beginning of the notebook to initialize the grader. After every question, the Instructor will also need to write a grading cell which the student will run to invoke the grader. A grading cell looks as follows:grader.grade(test_case_id = TEST_CASE_NAME, answer = ANSWER)TEST_CASE_NAME is the string name of the test case function that will grader the given question.ANSWER is the object that needs to be graded.For example, you might have a question where you instruct the student to create a DataFrame calledanswer_df. You will need to writeanswer_dfas the input answer parameter of the grader function as follows:grader.grade(test_case_id = TEST_CASE_NAME, answer = answer_df)That way, when the student runs this cell, the grader will automatically find the test function named TEST_CASE_NAME, serialize theanswer_dfpython variable and ship it to AWS. The cool thing about the PennGrader is that you can pass pretty much anything as the answer.Teacher's Client: PennGraderBackendThe teacher client allows instructors to create and edit the test cases function mentioned earlier, as well as define multiple homework metadata parameters. As shown in the template notebooks linked above, you first need to initialize thePennGraderBackendfor a specific homework as follows:backend = PennGraderBackend(secret_key = SECRET_KEY, homework_number = HOMEWORK_NUMBER)SECRET_KEY is the string variable obtained when creating a course.HOMEWORK_NUMBER identifies which homework number you are planning to write/edit.After running the above cell in a Jupyter Notebook, given a correct SECRET_KEY,  the assigned HOMEWORK_ID string will be printed out. This HOMEWORK_ID needs to be copied into the initialization of the PennGrader student client for release. Just the homework id, without the secret key, will not allow students to see any of the test cases, so make sure the secret key does not get out.After initialization, thebackendcan be used to 1) update metadata 2) edit/write test cases.You can edit the following metadata parmaters by runnin the following code:TOTAL_SCORE = 100
DEADLINE = '2019-12-05 11:59 PM'
MAX_DAILY_TEST_CASE_SUBMISSIONS = 100

backend.update_metadata(DEADLINE, TOTAL_SCORE, MAX_DAILY_TEST_CASE_SUBMISSIONS)TOTAL_SCORErepresents the total number of points the homework is worth and should equal the sum of all test cases weights.DEADLINErepresents the deadline of the homework, with format:YYYY-MM-DD HH:MM A.MAX_DAILY_TEST_CASE_SUBMISSIONSrepresents the number of allowed submissions per test case per day.Writing test cases is also just as simple. In a Jupyter Notebook (see Teacher Backend template above), you just need to write test case functions for each gradable question.  Each test case will be identified by atest_case_idwhich is the name of the test case function. A test case functions needs to follow the following format:def test_case_name(answer_object_name):
	student_score = 0 # Current student score
	max_score = 5 # Number of points this question is worth

	# some kind of grading that adds or subtracts from the student_score #

	return (student_score, max_score)The test case function needs to return an integer tuple representing the student score for their answer and max number of points that can be earned in the given question.After writing all the test cases you need, simply run the following code in a cell and the PennGraderBackend class will automatically extract all user-defined functions in the current notebook and upload them to AWS.backend.update_test_cases()A success message will print once the operation has succeeded. If loading a lot of external libraries this might take a few minutes.LambdasGraderTheGraderlambda gets triggered from an API Gateway URL from the student's PennGrader client. The student's client as defined above will serialize its answer and make a POST request to the lambda with the following body parameters:{'homework_id' : ______, 'student_id' : ________, 'test_case_id' : ________, 'answer' : _______ }The lambda will proceed by downloading the correct serializedtest_case's andlibrariesfrom theHomeworksTestCasesDyanmoDB table. It will then deserialize these objects and extract the correct test case given thetest_case_id. import the correct libraries used by given test case. If the submission is valid the student score will be recorded in the backend.GradesThe grades lamabda interfaces the TeacherBackend and student notebook with the Gradebook. Using this API Gateway trigged lambda the TeacherBackend client can request all grades for a given homework. The payload body used to trigger this lambda will need to include{'homework_id' : ______, 'secret_key' : ________, 'request_type' : ________ }Where thesecret_keyparameter is optional and will allow a properly initialized TeacherBackend to download all grades for the selected homework. Students will also be able to view their scores.HomeworkConfigTheHomeworkConfiglambda interfaces the TeacherBackend with theHomeworkTestCasesandHomeworkMetadataDynamoDB Tables (see below for table schema). This lambda can be triggered in two ways 1) to update metadata and 2) to update test cases. After unparsing and deserializing the triggered payload, the inputted data gets written into the appropriate DynamoDB table.DynamoDB Tables & S3 BucketsAs shown in the above schematic we maintain the majority of the data needed for grading and grade storage on DynamoDB. Below we list the information recorded in each table.Classes DynamoDB TableClassescontains information about all courses currently registered for the PennGrader. The grading protocol is on a per-class basis. Each class that wants to create a course that uses the PennGrader will receiveSECRET_KEY, this secret key will be passed in the TeacherBackend client to allow instructors to edit test cases. The tables contain the following schema:secret_key: Unique UUID used as a secret identifier for a course.course_id: Human-readable identifier representing the course number and semester of the class offered i.e. 'CIS545_Spring_2019'. This ID will be the pre-fix of thehomework_id, which will be used to identify a homework assignment.HomeworksMetadata DynamoDB TableTheHomeworksMetadatais used to maintain updatable information about a specific homework. The information in this table will be editable from the TeacherBackend even after homework release. The tables contain the following schema:homework_id: Unique identifier representing a course + homework number pair.homeowork_idis constructed by taking thecourse_iddefined above and appending the homework number of the assignment in question. For example, given the course id defined above (CIS545_Spring_2019), thehomework_idfor the first homework will be 'CIS545_Spring_2019_HW1'. This homework_id will be passed into the PennGrader Grader class in the student's homework notebook and will be used to correctly find the correct test cases and store the student scores correctly.deadline: String representing the deadline of the homework in local time i.e. 2019-12-05 11:59 PM (format: YYYY-MM-DD HH:MM A)max_daily_submissions: Number representing the total number of submissions allows per test case per day. For example, if this number is set to 5, it means that all students can submit an answer to a specific test case 5 times a day. Resetting at midnight.max_score: The total number of points this homework is worth. This number should be equal to the sum of all test case weights.max_scoreis used to show students how many points they have earned out of the total assignment.HomeworksTestCases DynamoDB TableTheHomeworksTestCasestable contains a serialized encoding of the test cases and libraries imports needed to run a student's answer. The tables contain the following schema:homework_id: Samehomework_idfrom theHomeworksMetadatatable.test_cases: This field contains a dill UTF-8 string serialization of the test cases defined in the teacher backend. The teacher backend extracts all test case functions from the notebook and creates a dictionary ofname->function. This parameter is deserialized when a new grading request is made and the correct test case is extracted and ran.libraries: Similar to thetest_casesfield, the libraries are UTF-8 dill serialized list of tuples that contain all libraries and functions imported in the teacher backend notebook and their appropriate short name. This list is used to import all needed libraries to run a specific test case.Gradebook DynamoDB TableTheGradebooktable contains all grading submissions and student scores. The table contains the following schema:homework_id: Samehomework_idfrom theHomeworksMetadatatable.student_submission_id: String representing a student submission. This string is create by appending the student's PennID to the test case name i.e. '12345678_test_case_1'.max_score: Maximum points that can be earned for this test casestudent_score: Points earned by the student on this test case.Note: Currently only the last submission is recorded, thus the latest student score will overwrite all previous scores.