Play to Grade: Grading Interactive Coding Games as Classifying Markov Decision Process

1 INTRODUCTION . The rise of online coding education platforms accelerates the trend to democratize high quality computer science education for millions of students each year . Corbett ( 2001 ) suggests that providing feedback to students can have an enormous impact on efficiently and effectively helping students learn . Unfortunately contemporary coding education has a clear limitation . Students are able to get automatic feedback only up until they start writing interactive programs . When a student authors a program that requires user interaction , e.g . where a user interacts with the student ’ s program using a mouse , or by clicking on button it becomes exceedingly difficult to grade automatically . Even for well defined challenges , if the user has any creative discretion , or the problem involves any randomness , the task of automatically assessing the work is daunting . Yet creating more open-ended assignments for students can be particularly motivating and engaging , and also help allow students to practice key skills that will be needed in commercial projects . Generating feedback on interactive programs from humans is more laborious than it might seem . Though the most common student solution to an assignment may be submitted many thousands of times , even for introductory computer science education , the probability distribution of homework submissions follows the very heavy tailed Zipf distribution – the statistical distribution of natural language . This makes grading exceptionally hard for contemporary AI ( Wu et al. , 2019 ) as well as massive crowd sourced human efforts ( Code.org , 2014 ) . While code as text has proved difficult to grade , actually running student code is a promising path forward ( Yan et al. , 2019 ) . We formulate the grading via playing task as equivalent to classifying whether an ungraded student program – a new Markov Decision Process ( MDP ) – belongs to a latent class of correct Markov Decision Processes ( representing correct programming solutions to the assignment ) . Given a discrete set of environments E = { en = ( Sn , A , Rn , Pn ) : n = 1 , 2 , 3 , ... } , we can partition them into E ? and E ′ . E ? is the set of latent MDPs . It includes a handful of reference programs that a teacher has implemented or graded . E ′ is the set of environments specified by student submitted programs . We are building a classifier that determines whether e , a new input decision process is behaviorally identical to the latent decision process . Prior work on providing feedback for code has focused on text-based syntactic analysis and automatically constructing solution space ( Rivers & Koedinger , 2013 ; Ihantola et al. , 2015 ) . Such feedback orients around providing hints and unable to determine an interactive program ’ s correctness . Other intelligent tutoring systems focused on math or other skills that don ’ t require creating interactive programs ( Ruan et al. , 2019 ; 2020 ) . Note that in principle one could analyze the raw code and seek to understand if the code produces a dynamics and reward model that is isomorphic to the dynamics and reward generated by a correct program . However , there are many different ways to express the same correct program and classifying such text might require a large amount of data : as a first approach , we avoid this by instead deploying a policy and observing the resulting program behavior , thereby generating execution traces of the student ’ s implicitly specified MDP that can be used for classification . Main contributions in this paper : • We introduce the reinforcement learning challenge of Play to Grade . • We propose a baseline algorithm where an agent learns to play a game and use features such as total reward and anticipated reward to determine correctness . • Our classifier obtains 93.1 % accuracy on 8359 most frequent programs that cover 50 % of the overall submissions and achieve 89.0 % accuracy on programs that are submitted by less than 5 times . We gained 14-19 % absolute improvement over grading programs via code text . • We will release a dataset of over 700k student submissions to support further research . 2 THE PLAY TO GRADE CHALLENGE . We formulate the challenge with constraints that are often found in the real world . Given an interactive coding assignment , teacher often has a few reference implementations of the assignment . Teachers use them to show students what a correct solution should look like . We also assume that the teacher can prepare a few incorrect implementations that represent their “ best guesses ” of what a wrong program should look like . To formalize this setting , we consider a set of programs , each fully specifies an environment and its dynamics : E = { en = ( Sn , A , Rn , Pn ) : n = 1 , 2 , 3 , ... } . A subset of these environments are reference environments that are accessible during training , we refer to them as E ? , and we also have a set of environments that are specified by student submitted programs E ′ . We can further specify a training set D = { ( τ i , yi ) ; y ∈ { 0 , 1 } } where τ i ∼ π ( e ( i ) ) and e ( i ) ∼ E ? , and a test set Dtest where e ( i ) ∼ E ′ . The overall objective of this challenge is : minL ( θ ) = min θ min π Ee∼E [ Eτ ′∼π ( e ) [ L ( pθ ( φ ( τ ′ , π ) ) , y ) ] ] ( 1 ) We want a policy that can generate trajectory τ that can help a classifier easily distinguish between an input environment that is correctly implemented and one that is not . We also allow a feature mapping function φ that takes the trajectory and estimations from the agent as intput and output features for the classifier . We can imagine a naive classifier that labels any environment that is playable ( defined by being able to obtain rewards ) by our agent as correct . A trivial failure case for this classifier would be that if the agent is badly trained and fails to play successfully in a new environment ( returning zero reward ) , we would not know if zero reward indicates the wrongness of the program or the failure of our agent . Generalization challenge In order to avoid the trivial failure case described above – the game states observed are a result of our agent ’ s failure to play the game , not a result of correctness or wrongness of the program , it is crucial that the agent operates successfully under different correct environments . For any correct environment , E+ = { E ? + , E ′+ } , the goal is for our agent to obtain the high expected reward . π ? = argmax π Ee∼E+ [ Eτ∼π ( e ) [ R ( τ ) ] ] ( 2 ) Additionally , we choose the state space to be the pixel-based screenshot of the game . This assumption imposes the least amount of modification on thousands of games that teaching platforms have created for students over the years . This decision poses a great challenge to our agent . When students create a game , they might choose to express their creativity through a myriad of ways , including but not limited to using exciting background pictures , changing shape or color or moving speed of the game objects , etc . Some of these creative expressions only affect game aesthetics , but other will affect how the game is played ( i.e. , changing the speed of an object ) . The agent needs to succeed in these creative settings so that the classifier will not treat creativity as incorrect . 2.1 BOUNCE GAME SIMULATOR . We pick the coding game Bounce to be the main game for this challenge . Bounce is a block-based educational game created to help students understand conditionals1 . We show actual game scenes in Figure 1 , and the coding area in Figure 2a . 1https : //studio.code.org/s/course3/stage/15/puzzle/10 The choice gives us three advantages . First , the popularity of this game on Code.org gives us an abundance of real student submissions over the years , allowing us to compare algorithms with real data . Second , a block-based program can be easily represented in a structured format , eliminating the need to write a domain-specific parser for student ’ s program . Last , in order to measure real progress of this challenge , we need gold labels for each submission . Block-based programming environment allows us to specify a list of legal and illegal commands under each condition which will provide perfect gold labels . The Bounce exercise does not have a bounded solution space , similar to other exercises developed at Code.org . This means that the student can produce arbitrarily long programs , such as repeating the same command multiple times ( Figure 3 ( b ) ) or changing themes whenever a condition is triggered ( Figure 3 ( a ) ) . These complications can result in very different game dynamics . We created a simulator faithfully executes command under each condition and will return positive reward when “ Score point ” block is activated , and negative reward when “ Socre opponent point ” block is activated . In deployment , such simulator needs not be created because coding platforms have already created simulators to run and render student programs . 2.2 CODE.ORG BOUNCE DATASET . Code.org is an online computer science education platform that teaches beginner programming . They designed a drag-and-drop interface to teach K-12 students basic programming concepts . Our dataset is compiled from 453,211 students . Each time a student runs their code , the submission is saved . In total , there are 711,274 submissions , where 323,516 unique programs were submitted . Evaluation metric In an unbounded solution space , the distribution of student submissions incur a heavy tail , observed by Wu et al . ( 2019 ) . We show that the distribution of submission in dataset conforms to a Zipf distribution . This suggests that we can partition this dataset into two sections , as seen in Figure 2b . Head + Body : the 8359 most frequently submitted programs that covers 50.5 % of the total submissions ( 359,266 ) . This set contains 4,084 correct programs ( 48.9 % ) and 4,275 incorrect programs ( 51.1 % ) . Tail : This set represents any programs that are submitted less than 5 times . There are 315,157 unique programs in this set and 290,953 of them ( 92.3 % ) were only submitted once . We sample 250 correct and 250 incorrect programs uniformly from this set for evaluation . Reference programs Before looking at the student submitted programs , we attempted to solve the assignment ourselves . Through our attempt , we form an understanding of where the student might make a mistake and what different variations of correct programs could look like . Our process can easily be replicated by teachers . We come up with 8 correct reference programs and 10 incorrect reference programs . This can be regarded as our training data . Gold annotations We generate the ground-truth gold annotations by defining legal or illegal commands under each condition . For example , having more than one “ launch new ball ” under “ when run ” is incorrect . Placing “ score opponent point ” under “ when run ” is also incorrect . Abiding by this logic , we put down a list of legal and illegal commands for each condition . We note that , we intentionally chose the bounce program as it was amenable to generating gold annotations due to the API that code.org exposed to students . While our methods apply broadly , this gold annotation system will not scale to other assignments . The full annotation schema is in Appendix A.5 .

Summary: The authors contribute an approach to automatically distinguish between good and bad student assignment submissions by modeling the assignment submissions as MDPs. The authors hypothesize that satisfactory assignments modeled as MDPs will be more alike than they are to unsatisfactory assignments. Therefore this can potentially be used as part of some kind of future automated feedback system. The authors demonstrate this approach on an assignment for students to recreate a simple pong-like environment. They are able to achieve high accuracy over the most common submissions.