Deep Grade : A visual approach to grading student programming assignments

As classes get larger, the process of evaluating student performance becomes more time-consuming. While content delivery can be scaled to large classrooms in the form of webcasted lectures or online materials, grading and assessing students requires human grader input to identify areas of misunderstanding and provide individual student feedback. In computer science courses, student programming assignments can sometimes be autograded with delicate unit tests. However, these unit tests are time-consuming to design and are specifically tailored to the assignment; they cannot be extended to other homeworks. Furthermore, unit tests are often low-level in nature and cannot handle more qualitative checks. In particular, introductory computer science programming assignments often involve interactive, open-ended graphics. Unit tests for such graphics-based assignments are consequently particularly difficult to design. More open-ended approaches to grading computer programs for functionality often focus on the student code itself, parsing the student programs to produce abstract syntax trees (ASTs), after which the programs can be compared and evaluated [2, 4]. Yet these representations become impossibly complex when student solutions grow longer than twenty lines or contain diverse variable names. In this project, we present Deep Grade, a deep learning approach to grading graphics-based programming assignments relying on neither the student code nor unit test development. Inspired by the Deep Mind project, a deep Qlearner for Atari games [1], we aim to develop an assistive grading system that requires only raw pixels as input for suggesting grades. We focus on Breakout, a classic brick break game used in Stanford’s introductory CS 1 course [3]. Deep Grade takes video playthroughs of student-coded programs and outputs a grade according to a given rubric; it also annotates sequences of frames corresponding to each reported rubric item. The goal of Deep Grade is to develop a detection system that can be used by human graders as an assistive technology for grading. For example, the system should detect that the bricks are correct aligned and that the paddle stays on-screen in the left frame of Breakout in Figure 1 (whereas the other frame shows incorrectly aligned bricks and a slightly off-screen paddle); it can justify its claims with timestamped output so that human graders can make the final decision. Students can also use this system as a visual medium for understanding their homework grade. Because our system requires only a temporal, raw pixel representation of a student program run, it can be generalized and applied to many other homework assignments. The assignments supported by our system can range from animation assignments like Breakout to computer systems assignments, such as designing an Internet router, where the output can be represented graphically over time (e.g., raw network packet bits transmitted with time delays). This abstract has three components: we define a novel problem at the intersection of computer vision and computer science education, describe a zero-shot learning method by training on a synthetically generated dataset of student code, and present preliminary work on a supervised learning implementation of the Deep Grade system.