Giving Feedback on Interactive Student Programs with Meta-Exploration
Authors: Evan Zheran Liu
Moritz Pascal Stephan
Allen Nie
Christopher J Piech
Emma Brunskill
Chelsea Finn
Venue: NeurIPS 2022 Accept (2022)
Keywords: meta-reinforcement learning
education
exploration

Abstract: Developing interactive software, such as websites or games, is a particularly engaging way to learn computer science. However, teaching and giving feedback on such software is time-consuming — standard approaches require instructors to manually grade student-implemented interactive programs. As a result, online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs, which critically hinders students’ ability to learn. One approach toward automatic grading is to learn an agent that interacts with a student’s program and explores states indicative of errors via reinforcement learning. However, existing work on this approach only provides binary feedback of whether a program is correct or not, while students require finer-grained feedback on the specific errors in their programs to understand their mistakes. In this work, we show that exploring to discover errors can be cast as a meta-exploration problem. This enables us to construct a principled objective for discovering errors and an algorithm for optimizing this objective, which provides fine-grained feedback. We evaluate our approach on a set of over 700K real anonymized student programs from a Code.org interactive assignment. Our approach provides feedback with 94.3% accuracy, improving over existing approaches by 17.7% and coming within 1.5% of human-level accuracy. Project web page: https://ezliu.github.io/dreamgrader.

Giving Feedback on Interactive Student Programs with Meta-Exploration
Evan Zheran Liu", Moritz Stephan", Allen Nie, Chris Piech, Emma Brunskill, Chelsea Finn Stanford University
Abstract
Developing interactive software, such as websites or games, a particularly engaging way to learn computer science. However, teaching and giving feedback such software is time-consuming standard approaches require instructors manually grade mudentt-mmllemented interactive programs. As result, online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs, which critically hinders students' ability to learn. One approach toward automatic grading is to learn an agent that interacts with student's program and explores states indicative of errors via reinforcement learning. However, existing work on this approach only provides binary feedback of whether a program correct or not, while students require finer-grained feedback on the specific errors in their programs to understand their mistakes. In this work, we show that exploring to discover construct errors can be cast meta-exploration problem. This enables principled objective for discovering errors and an algorithm for optimizing this objective, which provides fine-grained feedback. We evaluate approach set of over 700K real anonymized student programs from a Code.org interactive assignment. Our approach provides feedback with 94.3% accuracy, improving over existing approaches by 17.7% and coming within 1. 1.5% of human-level accuracy. Project web page: https: wwte//eezl.//// waw.giithub... io/ dr dreangrader
Introduction
Feedback plays a critical role in high-quality education, but can require significant time and expertise to provide (7]. We focus one area where providing feedback particularly burdensome: modern computer science education, where students are often tasked with devel- oping interactive programs, such as websites games (e.g., Figure 1. While developing such programs is highly engaging (35 and has become ubiquitous in contemporary classrooms (12), these programs can include stochastic or creative elements, so they cannot be graded with traditional unit tests and must instead be manually graded. However, such manual grading is increasingly
Existing work oveer 21]: Coarse binary foedback
Score 0
P
The program is incorfect
This work grainga feedback
PO
ball incorrectly bounces off the goal
Agent plays program bugs
Agent provides feedback
Figure 1: learned Play-to-Grade agent ming assignment. The agent tests what happens when the hall is hit into the goal, and finds that the hall incorrectly bounces out instead of scoring point. Whereas prior work provides coarse feedback of whether the program is correct not, our goal to provide fine- grainer feedback on the specific mistakes a student has made.
Bounce program-
Co-first authors, Correspondence to evanliudcs. wtw.uiccssttaorrrdrooa stanford.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

infeasible with the growing demand for computer science education and rise of massive online learning platforms. For example, one popular platform, Code.org has enrolled over 70M students M As manually grading a submission can take up to 6 minutes, a single assignment creates decades of grading labor. Consequently, platforms like Code.or org cannot yet provide feedback about whether an interactive assignment submission is correct or not, let alone more fine-grained feedback.
To alleviate this enormous grading burden, Nie et al. 30 introduce the Play-to-Grade paradigm for automatically providing feedback by training a reinforcement learning agent to grade a program the same way humans do: by interacting or playing with the program. The idea is for the agent to visit states that reveal errors in the program, and then aggregate this information as feedback. Such an agent trained on set of training programs labeled with feedback (e.g.. provided by an instructor), and the goal is to generalize to new student programs. Figure shows an example learned agent that tests what happens when the ball is hit into the goal, exposing an error where the ball bounces off the goal instead of entering and scoring a point. The state-of-the-ard approach in this paradigm provides accurate coarse feedback of whether the program is completely correct or not [30]. However, to understand their mistakes, students usually require more specific feedback about what errors are in their programs.
Learning an agent to explore and discover the errors in a program to provide such fine-grained feedback is challenging: Most errors cannot be discovered with simple random exploration, and instead require targeted exploration, such as deliberately hitting the ball into the goal. addition, the agent must be able to adapt its exploration to different programs, which each behave differently and may present unexpected obstacles, such as multiple balls. Our key technical insight is that learning to discover errors connects with the meta-exploration problem in meta-reinforcemment learning (meta- RL.. This insight enables us to leverage techniques from the meta-exploration literature to construct and optimize principled objective for producing fine-grained feedback. Specifically, we follow the Play-to-Grade paradigm and assume access to 3,556 training programs, labeled with the errors in the program. Then, we formulate the problem maximizing the mutual information between states vis- ited by our agent and the label. Finally, we use techniques from the DREAM meta-RL algorithm [25) to decompose this objective into shaped rewards that enable learning sophisticated exploration.
Overall, the main contribution of this work is to connect the Play-to- Gradee paradigm with the meta-RL literature, and consequently, to provide an effective method for providing -ra-inri feedback for interactive student programs, which we call DREAMGRADER, Additionally, we release our code to propose automatic feedback as a new meta-RL. testbed that fulfills an unmet need in the community for a benchmark that is simultaneously readily accessible and directly impactful. We evaluate our system on 711,274 anonymized student submissions of the Bounce assignment from Code.org (30]. Trained on 3,556 programs, DREAMGRADER achieves an accuracy of 94.3%, which improves over existing approaches by 17.7% and comes within 1.5% of human-level grading accuracy. In addition, our approach can significantly reduce instructor grading burden: while manually grading all student submissions would require 4 years of work, our system grade the same set of programs 180x faster on a single GPU and can be parallelized for even faster grading over multiple GPUs.
2 Related Works
Educational feedback, We consider the problem of automatically providing feedback, which plays an important role in student learning and motivation [32] Though we specifically focus on feedback, other works also leverage machine learning for other aspects of education, including tracking what students know 46 36), predicting student retention 2 4 41 3, , and building intelligent tutoring
systems 1 Work on automatically providing feedback for computer science assignments focuses on two main approaches: analyzing either (i) the code or (ii) the behavior of a program. Methods that analyze code provide feedback by passing the code through neural network [37 6. 26 5 1]. or by constructing syntax trees [4] c.g., to predict useful next implementation steps 39 33 Analyzing
code works well for shorter programs (e.g., under 50 lines of code) and has even been deployed in
online courses [52]. However, this approach struggles to scale to lengthier or more complex programs. Hence, we instead opt for the second approach of analyzing program behavior, which conveniently does not depend on program length, though it requires the program to be executable.
Arguably, the simplest method of analyzing program behavior is unit testing. Unit testing can provide automatic feedback to some extent when the desired output of each input is known, but this is typically not the case with interactive programs, such as websites or games, Instead, work on
2

---

automated testing can provide feedback by generating corner-case inputs that reveal enrors via imput fuzzing [4], symbolic generation [2I]]. or reinforcement learning exploration objectives [54 However, this line of work assumes that errors are easy to detect when revealed, while detecting a revealed error itself can be challenging 30
Consequently, Nie et al. [30] propose the Play-to-Grade paradigm to both learn an agent to discover states that reveal errors, and a model detect and output the revealed errors. Our work builds upon the Play-to-Grade paradigm, but differs from Nie et al. [0] in the provided feedback. While Nie et al. [3)] only provide coarse binary feedback of whether program correct, we introduce a new principled objective to provide fine tee-grained feedback of what specific errors are present to help students understand their mistakes.
Meta-reinforcement learning. To provide fine-grained feedback, we connect the problem of discovering errors with the meta-exploration problem in meta-RL.. There is rich literature of approaches that learn to explore via meta-RL. [T6
44
[38] 45, 556 17 20 [25]. We specifically
leverage ideas from the DREAM algorithm [25] construct a shaped reward function for learning exploration. Our work has two key differences from prior meta-RL, research. First, we introduce novel factorization of the DREAM objective that better generalizes new programs. Second, and more importantly, we focus on the problem of providing feedback on interactive programs. This differs from a large body of meta-RL work that focuses on interesting, yet synthetic problems.
such as 2D and 3D maze navigation [1O 27 56 (25,, simulated control problems [1] 38 [53] and
alchemy 48). While meta-RL has been applied realistic settings robotics
29
such
application requires costly equipment. In contrast, this work provides a new meta-RI. problem that is both realistic and readily accessible.
3 The Fine-Grained Feedback Problem
We consider the problem of automatically providing feedback on programs. During training, assume access to a set of programs labeled with the errors made in the program (i.e., ground-truth instructor feedback). During testing, the grading system is presented with a new student program and must output feedback of what errors are in the program. To produce this feedback, the grading system is allowed interact with the program.
More formally, we consider a distribution over programs p(u), where each program μ defines a Markov decision process (MDP) μ = (S, A. T, R) with states S, actions A, dynamics T, and rewards R. We assume that the instructor creates a rubric: an ordered list of K potential errors that occur in program. Each program μ is associated with ground truth label y E {0, 1 of which errors are made in the program. The t index yA denotes that the 1h error of the rubric is present in the program J..
During training, the grading system is given set of N labeled training programs {(x0, )) The goal is to learn a feedbuck function I that takes program μ and predicts the label f(u) to maximize the expected grading accuracy Jynade(]) over test programs μ with unobserved labels y:
K
Tgrude (f) = Bymopia)
Σ K
I[f(n) = yel
where is an indicator variable. Effectively, Tgrdde measures the per-rubric item accuracy of predicting the ground-truth label y. To predict the label y, the feedback function may optionally interact with the MDP y defined by the program for any small number of episodes.
Bounce programming assignment. Though the methods we propose in this work generally apply to any interactive programs with instructor- creatte rubrics and we include experiments on another interactive assignment in Appendix D.2] we primarily focus on the Bounce programming assignment from Code.org, a real online assignment that has been completed nearly a million times. As providing feedback for interactive assignments challenging, the assignment currently provides no feedback whatsoever on Code.org, and instead relies on the student to discover their own mistakes by playing their program. This assignment is illustrated where we use the state representation from Nie et al. [30]: each state consists of the (x, -)-oodndinates of the paddle and balls, as well as the (I, y) -)-velocitses of the balls. There are three actions: moving the paddle left or right, keeping the paddle in the current position. In the dynamics of correct program,
Figure I Each student program defines an MDP,
3

---

the ball bounces off the paddle and wall. When the ball hits the goal or floor, it disappears and launches a new ball, which increments the player score and opponent score respectively. However, the student code may define other erroneous dynamies, such as the ball passing through the paddle or bouncing off the goal. The reward is +1 when the player score increments and when the opponent score increments. An episode terminates after 100 steps or if either the player opponent score exceeds 30.
Our experiments use a dataset of 711,274 real anonymized student submissions to this assignment, released by Nie et al. [3)] We use 0.5% of these programs for training, corresponding to N 3,556 and uniformly sample from the remaining programs for testing.
Possible errors in a student program take the form of "when event occurs, there is an incorrect consequence," where the list of all events and con- sequences is listed in Table | For example, Figure iillstrates the error where the event is the ball hitting the goal, and the consequence is that the ball incorrectly bounces off the goal, rather than entering the goal. For simplicity, we primarily consider a representative rubric of K 8 errors, spanning all event and consequence types, listed in Appendix A though we include some experiments on all error types in Appendix D
Table I: Possible event and consequence types of program enrors.
Event
Consequence
Ball hits paddle
Ball bounces / does not bounce Increments / does not increment player score Increments/ / does not increment opponent score Launches / does not launch new hall Moves the paddle
Ball hits wall
Ball hits goal
X
Ball hits floor
Paddle moves
Program starts
Prior approach for program feedback. Accurately determining which emoos are in given program is challenging, because it requires targeted exploration that adapts to variability in the programs. Often, the presence of some errors makes it difficult find other error, such as multiple balls making it difficult to determine which events change the score. Prior work by Nie et al. [30] sidesteps this challenge by only providing coarse feedback of whether a program is correct or not, by determining if a student program differs from a reference solution program. Such coarse feedback is much easier to provide, it often involves only finding the most obvious error, which can frequently be found with relatively untargeted exploration. In the next section, we present a new approach that instead targets exploration toward uncovering specific misconceptions to effectively provide fine- une-grained feedback. We discuss how our approach differs from Nie et al. 30 in greater detail in Appendix [
4 Automatically Providing Fine-Grained Feedback with DREAMGRADER
In this section, we detail our approach, DREAMGRADER, for automatically providing fine-grained feedback to help students understand their mistakes. From high level, DREAMGRADER learns two components that together form the feedback function f:
An exploration policy x that acts on program f to produce trajectory 7 (su. ag. ..).
A feedback classifier g T) that defines a distribution over labels given a trajectory т.
The idea is to explore states that either indicate or rule out errors with the exploration policy, and then summarize the discovered errors with the feedback classifier. To provide feedback on a new program μ. we first roll out the exploration policy π on the program to obtain trajectory т, and then obtain the predicted label arg max, ( r) by applying the feedback classifier. Under this parametrization of the feedback function f. we can rewrite the expected grading accuracy objective in Equation as:
K
JOREAMGRADER (T, Bp)p(a))
Σ
Marg max g0 7)x yx] 9
K
where π (1)) denotes the distribution over trajectories from rolling out the policy x on the program μ..
After this rewriting of the objective, our approach is conceptually straightforward: we learn both the exploration policy and classifier to maximize our rewritten objective. We can easily learn the feedback classifier g by maximizing the probability of the correct label given a trajectory generated by the exploration policy with standard supervised learning (i.e., cross-entropy loss), but learning the exploration policy is more challenging. Note that we could directly optimize our objective in Equation by treating the inside of the expectation as a reward received the end of the episode and use this to learn the exploration policy π with reinforcement learning. However, this reward

---

Error type: Does hitting the ball into the goel increment the score?
information observed about
Score
Score 0
g r) = 0.12
Error observed: Ball does increment the
gieror
2.12
Ball goel again, but no info gained
g
New ball launched
Figure 2: DREAMGRADER provides credit assignment for learning exploration by leveraging the feedback classifier g(y 7). Here, we consider the error "when the ball enters the goal, the player score does not increment." At and @ no information is observed that either rules out or indicates the error. Hence, no exploration reward ri is provided, and the classifier assigns 0.12 probability that the error is present, reflecting the prior that 12% of the training programs have this error. At @ the ball enters the goal, but does not score point and high exploration reward and credit assignment for learning exploration. At @ the ball enters the goal again. However, no additional information is gained, so the classifier does not change, and no reward is given. Overall, this enables learning effective exploration that purposely hits the ball into the goal once.
ball launched. This indicates that the error present, so the classifier updates, which creates
signal makes credit assignment difficult for learning the exploration policy, since it is given at the end of the episode, rather than at the states where the exploration policy discovers errors. Indeed, we empirically find that learning from this reward signal struggles to adequately explore (Section E5.
Hence, our goal is instead to construct reward signal that helps assign eredit for the exploration policy and provides rewards at the states that indicate or rule out errors in the program. To do this, we first propose an alternative objective that is sufficient to maximize the Play-to- Grade objective, but can be decomposed into per-timestep rewards that correctly assign credit (Section [.Ipp Intuitively, these rewards leverage the feedback classifier to provide high reward when an action leads to a state that increases the classifier's 's certainty about whether an error is present. Then, we detail practical design choices for implementing this approach with neural networks (Section (42) and conclude by drawing a connection between learning to find errors and the meta-exploration problem, which motivates our choice of alternative objective and its subsequent decomposition (Section
4.1 Assigning Credit to Learn Exploration
We now obtain reward signal to help assign credit for learning exploration. From a high level, we first propose to maximize mutual information objective that sufficient to maximize our objective JDREaMGRADER (T. in Equation 2 We then rewrite
mutual information objective in terms of per-timestep exploration rewards related to the information gain of the feedback classifier on the true label y when it observes the transition (st, az, rz). This helps assign credit for learning the exploration policy, as the transitions that either indicate or rule out errors in the program are exactly those that have high information gain. Figure illustrates example of the derived exploration rewards.
Objective. Intuitively, we want our exploration policy to visit states that either indicate or rule out errors. We can formalize this intuition by maximizing the mutual information [(T; v) between the trajectories T ~ æ visited by the policy and the feedback label of the program y, which causes the policy to visit states that make the feedback label highly predictable. Importantly, maximizing this objective is sufficient to maximize the expected grading accuracy Joreamgrader (T, g) in Equation 2 Let p(y r) be the true posterior over labels given trajectories ~ sampled from the policy. Maximizing the mutual information / (T; y) produces trajectories that maximize the probability of the label under the true posterior p(y r). Then, if the feedback classifier is learned to match the true posterior g r) = p(y r) while I(T:y) is maximized, the expected grading accuracy JDreamGradDER(T1 (T. is also maximized.
Optimization. We can efficiently maximize our objective I(T;y) by maximizing variational lower bound [5] and decomposing the lower bound into per-timestep reward that helps assign credit single
for learning the exploration policy æ. Below, we derive this for the case of learning only a exploration policy to uncover all errors, though we will later discuss how can factorize this to
5

---

leamn N exploration policies { { that each explore to uncover a single enror type.
((::)) H [g] Hyy r]
(3)
H [y] + Bpmp(e))
[log P(v
[r]
(4)
2 H [y] + Buppp()) (N) [lgg9( r]]
(5)
T-1
[y] + (p)-p(u).7~7(( 10 80)
+
Σ
exp
(6)
1=
where rt = g T(++1) log svv Tix) and
the length of the trajectory r (80, ao: ro,-..,, ST).
The inequality in ( holds for replacing the true posterior p(y T) with any distribution, and (6)
comes from expanding a telescoping series as done by DREAM 25 where Tu (50, ao, To,, ........ St)
denotes the the trajectory up to the 1th state.
This derivation provides shaped reward function r. for learning the exploration policy. Intuitively, this reward captures how much new information the transition (st, at, Tt, $c+1) provides to the feedback classifier g what errors are in the program: The reward is high if observing this transition either indicates an error (c.g., the ball enters the goal, but does not score a point) or rules out an error (e.g., the ball enters the goal and scores point), and is low otherwise.
Additionally, we now have recipe for maximizing the mutual information I(r; y) to learn both the policy π and feedback classifier g. Only the second term in 6 depends on and g. Hence, we can maximize this lower bound on [T; 3) by maximizing the log-likelihood of the label with respect to the feedback classifier Treedhack (g) Bu--p()),r~T(() [log glv
7)], and maximizing the rewards log g3y (T++1)= log y T.a) with respect the policy æ via reinforcement learning.
eex
Ts
Factorizing our objective. So far, our approach learns a single exploration policy that must uncover all error types in the rubric. However, learning such policy can be challenging, especially if uncovering different error types requires visiting very different states. We instead propose to learn a separate exploration policy тı. for each error index of the rubric k = I,..., K. We can accomplish this by observing that maximizing the mutual information T; is also sufficient to maximize the expected grading accuracy. Furthermore, maximizing the mutual information with cach dimension of the label I [(Tx: Uk), where T ~ T for each k. is sufficient to maximize the mutual information with the entire label / ((TI...., TK); y). We therefore can derive exploration rewards for each term I (Txi yk) to learn each policy Ty with rewards ca log g() T log (M T:)). We find that this improves grading accuracy in our experiments (Section 5) and enables parallel training and testing for the K exploration policies.
Tk);y) between K trajectories
4.2 A Practical Implementation
Overall, DREAMGRADER consists of feedback classifier g and K exploration policies (Tx K 1== where the ttth policy Te tries to visit states indica-
Algorithm 1 Training episode for policy TT
1: Sample a training program μ with label
2: Roll out policy to obtain trajectory T ~ (x(
tive of whether the 1tth error type is present in the program. We learn these components by repeatedly
3: Compute rewards with feedback classifier
r log g(y== Ta-1) log gluk 7.))
running training episodes for each policy T& with Algorithm|] T We first sample labeled training pro- 5: Update feedback classifier to max. log ggt.
Update policy to maximize rewards esp with RL
gram and follow the policy on the program (lines 1-
2). Then, we maximize our mutual information objective by updating the policy with our exploration rewards (lines 3-4), and by updating the classifier to maximize the lig-likelihood of the label (line 5).
In practice, we parametrize the exploration policies and feedback classifier as neural networks. Since the exploration rewards r: depend on the past and are non-Markov, we make each exploration policy Te recurrent: At timestep f, the policy Ta (a, (Sp, an, states, actions, and observed rewards for each k. We parametrize each policy as deep dueling double Q-networks [0) 50 45). Conseguently, our policy updates in line 4 consist of placing the trajectory in replay buffer with rewards ri
st)) conditions on all past
exp and sampling from the replay buffer to perform Q-learning updates.
We parametrize each dimension of the feedback classifier g(yk 7) for k = 1,.... K as a separate neural network. We choose not to share parameters between the exploration policies and between

---

the dimensions of the feedback classifier for simplicity. However, we note that significant parameter sharing is likely possible, as exploration policies for two different error types can be extremely similar (e.g., two errors that involve hitting the ball into the goal). Automatically determining which parameters to share could be an interesting direction for future work, as it is not known priori which error types are related to each other. See Appendix for full architecture and model details.
4.3 Play-to-Grade as Meta-Exploration
Our choice to optimize the mutual information objective [(T: y) and decompose this objective into per- per-timestep rewards using techniques from the DREAM meta-RL algorithm stems from the fact that the Play-to-Gras paradigm can be cast as meta-exploration problem. Specifically, meta-RL. aims to learn agents that can quickly learn new tasks by leveraging prior experience related tasks. The standard few-shot meta-RI. setting formalizes this by allowing the agent to train on several MDPs (tasks). At time time, the agent is presented with new MDP and is allowed first explore the new MDP for several episodes (i.e., the few shots) to gather information. Then, it must use the information it gathered solve the MDP and maximize returns on new episodes. Learning to efficiently spend these allowed few exploration episodes to best solve the test MDP is the meta-exploration problem.
Our setting of providing feedback follows the exact same structure as few-shot meta-RL. In our setting, we can view each student program as a new 1-step task of predicting the feedback label, where the reward is the number of dimensions of the label that are correctly predicted. To predict this label, the Play-to- to-Grade paradigm first explores the program for several episodes to discover states indicative of errors, which corresponds to the few shots. The key challenge in this setting is exactly how to best spend those few exploration episodes: i.e., to gather the information needed to predict the label, which is exactly the meta-exploration problem. This bridge between identifying errors in programs and meta-exploration suggests that techniques from each body of literature could mutually benefit each other. Indeed, DREAMGRADE explore other techniques to transfer across the two areas. Additionally, this connection also offers a new testbed for meta- extt-exxloootioon and meta-RL RL research. As discussed in Section 2 while existing meta-RL benchmarks tend to be either readily accessible or impactful and realistic, automatically providing feedback simultaneously provides both, and we release code for meta-RL wrapper of the Bounce programming assignment to spur further research in this direction.
leverages ideas from DREAM and future work could
5 Experiments
In our experiments, we aim to answer five main questions: (1) How does automated feedback grading accuracy compare to human grading accuracy? (2) How does DREAMGRADER compare with Nie et al. [30]. the state-of-the art Play-to- -rade approach? (3) What are the effects of our proposed factorization and derived exploration rewards on DREAMGRADER? (4) How much human labor is saved by automating feedback? (5) Interactive programs can be particularly challenging to grade because test programs can contain behaviors not seen during training - how well do automated feedback systems generalize such unseen behaviors? To answer these questions we consider the dataset of 700K real anonymized Bounce student programs, described in Section 3
Below, we first establish the points of comparison to necessary answer these questions (Section [.J]. Then, we evaluate these approaches to answer the first four questions (Section Finally, we answer question (5) by evaluating DREAM DREAMGRADER on variants of Bounce student programs that modify the ball and paddle speeds, including speeds not seen during training (Section (5.3). Additionally, in Appendix D we test if DREAMGRADES can scale to all error types and an additional interactive assignment called Breakout, which is widely taught in university and highschool classrooms.
5.1 Points of Comparison
We compare with the following four approaches. Unless otherwise noted, we train 3 seeds of each automated approach for 5M steps on N 3,556 training programs, consisting of 0.5% of the dataset.
Human grading. To measure the grading accuracy of humans, we asked for volunteers to grade Bounce programming assignments. We obtained volunteers consisting of computer science under- graduate and PhD students, 7 of whom had previously instructed or been a teaching assistant for
7

---

computer science course. Each volunteer received training on the Bounce programming assignment and then was asked to grade 6 randomly sampled Bounce programs. See Appendix [.1 for details.
Nie et al. [30] extended to provide fine-grained feedback. We extend the original Play-to-Grade approach, which provides binary feedback about whether program is completely correct or not, to provide fine-grained feedback. Specifically, Nie et al. [30] choose a small set of 10 training programs, curated so that each program exhibits single error, and together, they span all errors. Then, for each training program, the approach learns (i) distance function d(s, a) that takes a state-action tuple (s, a). trained to be large for tuples from the buggy training program and small for tuples from a correct reference implementation; and (ii) an exploration policy that learns to visit state-actions where d(s,, a) is large. To provide feedback to a new student program, the approach runs each exploration policy on the program and outputs that the program has an error if any of the distance functions is high for any of the tuples visited by the exploration policies.
We extend this approach to provide fine-grained feedback by following a similar set up. We follow the original procedure to train separate policy and distance function on K 8 curated training programs, where the hthh program exhibits only the kdhh error from the rubric we consider. Then, to provide fine-grained feedback on a new program, we run each policy on the new program and predict that the hthh error is present (i.e., i/ 1) if the 1ith distance function is high on any state-action tuple. We use code released by the authors without significant fine-tuning or modification. We emphasize that this approach only uses 8 curated training programs, as opposed to the N 3,556 randomly sampled programs used by other automated approaches, as this approach is not designed to use more training programs, and furthermore cannot feasibly scale to many more training programs, as it learns a distance function and policy for every training program.
DREAMGRADES (direct max). To study the effect of our derived exploration rewards Tr exp we consider the approach of directly maximizing the DREAMGRADER objective in Equation 2 described at the beginning of Section 4.1 This approach treats the inside of the expectation as end-of-episode returns, and does not provide explicit credit assignment. This approach is equivalent to maximizing
the DREAMGRADER objective with the RL2 meta-RL algorithm [10 47
DREAMGRADER (unfactorized), Finally, to study the effect of our proposed factorization scheme, described at the end of Section [.1] we consider a variant of DREAMGR ADRR where we do not factorize the objective and instead only learn single exploration policy to uncover all errors.
5.2 Main Results
We compare the approaches based on grading accuracy, precision, recall, and Fl scores averaged across the 8 error types in the rubric. Table summarizes the re- sults. Overall, we find that DREAMGRADER achieves the highest grading accuracy of the automated grading approaches, providing feedback with 17.7%
Table 2: Accuracy, precision, recall and F1 of grading systems, averaged across the K = errors of the rubric, with I 1-standard deviation error bars.
Accuracy
Precision
Recall
Fl
Human
95.8 :: 3.9%
95.0 :: 13.2%
91.1 : 10.0%
91.9
8.3%
DREAMGRADER
94,3 + 1.3%
76.7 5.8%
94.3 = 16%
84.6
1.5%
DREAMGRADER (unfactorized)
0.4%
0.5%
0.7%
DREAMGRADER (direct max)
84.8 4: 2.2%
36.3
1.7%
37.8
9.7%
36.6
4: 5.1%
Nie al. 50]
75.5 1: 0.9%
24.9
5.0%
27.7
7.1%
26.1 1.115.5 5.7%
greater accuracy than Nie et al. (0). Furthermore, DREAMGRADER comes within 1.5% of human- achieves this by learning exploration behaviors that probe each possible error event (see Appendix D.3 or httts://eeeti::i gi thub. io io/dreamgrader for visualizations of the learned behaviors).
level grading accuracy. DREAMGRADS
Analysis. To further understand these results, we plot the training curves of the grading accuracy on each error type in the rubric vs. the number of training steps of DREAMGRADER, as well as the final grading accuracies of the other approaches in Figure 3 The performance of variants of DREAMGRADER underscores the importance of our design choices: DREAMGRADER (direct max) achieves significantly lower accuracy than DREAMGRADER across all error types, indicating the importance of our shaped exploration rewards r for learning effective exploration. Additionally, while DREAMGRADES (unfactorized) achieves relatively high average accuracy, it still performs worse than DRFAMGRAN ADER. This illustrates the difficulty of learning single exploration policy to uncover all errors, which is alleviated by our factorization.
8

---

Ball Rounces off Goal
Goal Increments Oppanent Score
Wall Increments Opponent Score
No New Ball after Goal
Avenge Acconaty
87-
use 2000 side uion sic
2000
2013
3500
15m
1990 aie
s001 abl Timestops (1x33
5000
4200
5000
Timestaps (1e3)
Timesteps (1x3)
Timestops (1x3)
Missing Ball does Increment Opponent Score
Incorrect
Paddle Increments Player Score
Ball at
Avennge Accuneyy
45
2600 Timestaps (1x3)
2091
1991
3001
abc
4000
Timestapss (1x22
Timestops (1x3)
Timestops (163)
Human
DREAMGRADER
DREAMGRADER (unfactorized)
DREAMGRADER (direct max)
Ni et (2021)
Figure 3: Average grading accuracy for each error type vs. number of training steps for DREAMGRADER with 1-stddev error bars. We plot the final grading accuracies of the other approaches as horizontal lines.
Additionally, we find that DREAMGRADER achieves human-level grading accuracy on of the error types, but relatively struggles the 3 errors on the left of the top row of Figure 3 which leads to overall lower average grading accuracy. For the errors involving incrementing the opponent score, we qualitatively find that DREAMGRADER most commonly struggles when there are multiple balls, which makes difficult to ascertain which events are causing the opponent score to increment. Human grading was able to circumvent this issue, but humans required playing the program for up to 40 episodes, while we limited DREAMGRADER to a single episode per error type.
Reduction in human grading burden. We found that our grading volunteers required between 1-6 minutes to grade each program, averaging around 3 minutes. At this grading speed, grading all of the 700K Code.org submissions would take close to 4 years of human labor. In contrast, automatic grading with DREAMGRADER requires only second per program on single NVIDIA RTX 2080 GPU, which is 180x faster.
5.3 Generalizing to Unseen Behaviors
key challenge in providing feedback for interactive programs that student programs may include behaviors not seen during training. To evaluate generalization to unseen behaviors, we consider vary- ing the speeds of the ball and paddle. There are five possible settings for the ball and paddle speeds: very slow, slow, normal, fast, and very fast. To test the ability of DREAMGRADED to generalize to un- seen behaviors at test time, we train DREAMGRADER on programs where we hold out the normal ball and paddle speeds, and then test DREAMGRAD we use the same N 3.556 training programs as before, but we uniformly randomize the ball and pad- dle speeds independently to be one of the four non-held out speeds. Then we evaluate on test programs with (1) held out ball and paddle speeds; (2) held out ball speed with random training paddle speed; (3) held out paddle speed with random training ball speed; and (4) training ball and paddle speeds.
on programs with the held out speeds. Specifically,
DREAMGRADER general- izes to unseen ball and paddle speeds test time. Compared to the standard training the previous sec- tion, where all ball and paddles had the "normal" speed, performance drops, as under faster balls, certain behaviors like hitting the ball into the goal cannot be reliably achieved, and increasing the training data does not decrease the performance drop. However, accuracy remains relatively high, and DREAMGRADER performs about the same on
Table 3: DREAMGRADER's 's results under held out ball and paddle speeds. DREAMGRADER generalizes to ball and paddle speeds not seen during training.
Both beld out
Held out ball speed
Held out paddic spoed
No held out speed
Accuracy
89.0
1.9%
89.7 + 1.7%
89.6 + 2.0%
89.3 + 1.8%
Precision
40.0
1.0%
44.1
1.8%
46.2 J. 1.3%
41.6 1. 2.1%
Recall
88.2 2.3%
89.3 + 1.5%
89.0 + 1.7%
85.5 + 2.4%
55.0 1.3%
59.1 1.9%
60.1 1 0.8%
56.0 J. 2.4%

---

test programs regardless of whether the ball and paddle speeds were seen during training or not. This indicates some ability generalize unseen behaviors test time. Table displays the full results.
Conclusion
In this work, we introduced DRFAMGRADER, an automatic grading system for interactive programs that provides fine-grained feedback at near human-level accuracy. The key insight behind our system connecting the problem of automatically discovering errors with the meta-cxploration problem in meta-RL, which yields important benefits for both sides. On the one hand, this connection offers a powerful and previously unexplored toolkit to computer science education, and more generally, to discovering errors in websites or other interactive programs. On the other hand, this connection also opens impactful and readily accessible applications for meta-RL RL research, which has formerly primarily focused synthetic tasks due to the lack of more compelling accessible applications.
While DREAMGRADER nears human-level grading accuracy, we caution against blindly replacing instructor feedback with automated grading systems, such as DREAMGRADDE which can lead to potentially negative educational and societal impacts without conscientious application. For example, practitioners must ensure that automated feedback is equitable and not biased against certain classes of solutions that may be correlated with socioeconomic status. One option to mitigate the risk of potential negative consequences while still reducing instructor labor is to use automated feedback systems to assist instructors similar to prior work [13,, e.g., by querying the instructor on examples where the system has low certainty, presenting videos of exploration behavior from the system for the instructor to provide the final feedback.
Finally, this work takes an important step to reduce teaching burden and improve education, but we also acknowledge DREAMGRADER still has important limitations to overcome. Beyond the remaining small accuracy gap between DREAMGRADER and human grading, DREAMGRADER also requires a substantial amount of training data. While we only use 0.5% of the Bounce dataset for training, it still amounts to 3,556 labeled training programs, and labeling this many programs can be prohibitive for smaller-scale classrooms, though feasible for larger online platforms. We hope that release of Bounce a meta- RL problem can help spur future work to overcome these limitations.
Reproducibility. Our code is publicly available at which includes Bounce as new meta-RL testbed.
com/ ezliu/ http://ww..ccoeeeeee/ararrrrrraa dreamgrader
Acknowledgments and Disclosure of Funding
We thank Kali Stover from the Institutional Review Board (IRB) and Ruth O'Hara and Tallie Wetzel from the Student Data Oversight Committee (SDOC) for reviewing our process of asking humans to grade programs. We thank our human graders: Annie Xie, Sahaana Suri, Cesar Lema, Olivia Lee, Moritz Stephan, Kaien Yang, Maximilian Du, Patricia Strutz, and Govind Chada.
We thank Annie Xie and Sahaamble Suri for thwarting EL's best attempts at procrastination, without whom, this work would not have been completed in time.
EL is supported by a National Science Foundation Graduate Research Fellowship under Grant No. 16561165 CF is Fellow in the CIFAR Learning in Machines and Brains Program. This work was also supported in part by Google, Intel, and a Stanford Human-Centered AI Hoffman Yee grant. Icons in this work were made by FreePik from Flaticon.
References
[1] John R Anderson, C Franklin Boyle, Albert T Corbett, and Matthew W Lewis. Cognitive modeling and intelligent tutoring. Artificial intelligence, 42(1):7-49, 1990.
[2] Karol Arndt, Murtaza Hazara, Ali Ghadirzadeh, and Ville Kyrki. Meta reinforcement learning for sim- to-real domain adaptation. In International Conference on Roboties and Automation (ICRA), pages 2725 2731, 2020.
[3] Lovenoor Aulck, Nishant Velagapudi, Joshua Blumenstock, and Jevin West. Predicting student dropout in higher education, arXiv preprint 4rXiv:16066.06344 2016.
10

---

[4] Girish Balakrishnan and Derrick Coetzee. Predicting student retention in massive open online courses using hidden markov models. Science, 53:57-58, 2013.
[5] David Barber and Felix V Agakov. The IM algorithm: a variational approach to information maximization. In Advances in neural information processing systems, 2003.
[6] Sahil Bhatia and Rishabh Singh. Automated correction for syntax errors in programming assignments using recurrent neural networks. arXiv preprint arXiv: 1603.06/29, 2016.
[7] Code.org. Code.org. httpps//code org/about about 2022.
[8] Ryan ST Baker, Alhert Corbett, and Vincent Aleven. More accurate student modeling through contextual estimation of slip and guess probabilities bayesian knowledge tracing. In International conference on intelligent tutoring systems, pages 406-415, 2008.
[9] Dursun Delen. Predicting student attrition with data mining methods. Journal of College Student Retention: Research, Theory & Practice, 13(1):17-35 35, 2011.
[10] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Picter Abbcel. RL2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint 16 2016.
[11] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. International Conference on Machine Learning (ICML), 2017.
[12] Jeffiey E Froyd, Phillip C Wankat, and Karl A Smith. Five major shifts in 100 years of engineering education. Proceedings of the IEEE, 100(0):1344 1360, 2012.
[13] Elena L Glassman, Jeremy Scott, Rishabh Singh, Philip J Guo, and Robert C Miller. Overcode: Visualizing variation in student solutions programming problems scale. In Conference Human Factors in Computing Syxtems (CHI), number 2, pages 1-35, 2015.
[14] Patrice Godefroid. Michael Y Levin, David A Molnar, et al. Automated whitebox fuzz testing. In NDSS, volume 8, pages 151 156, 2008.
[15] Camilo Gordillo, Joakim Bergdahl, Konrad Tollmar, and Linus Gisslén, Improving playtesting coverage via curiosity driven reinforcement learning agents. arXiv preprint 103.::::133137900) 2021.
[16] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta-reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems (NeurlPS), pages 5302 5311, 2018
[17] Swaminathan Gurumurthy, Sumit Kumar, and Katia Sycara. Mame: Model-agnostic meta-cxploration. arXiv preprint arXiv: /tt1:/99/.040 1.04024, 2019.
[18] Sepp Hochreiter and Jürgen Schmidhuber. Long short- sert-term memory. Neural Computation, ((8):1735-1788) 1997.
[19] Jan Humplik. Alexandre Galashov, Leonard Hasenclever, Pedro A Ortega, Yee whye Teh, and Nicolas Heess, Meta reinforcement learning task inference. arXiv preprint 1905v 2019.
[20] Pierre-Alexanree Kamienny, Matteo Pirotta, Alessandro Lazaric, Thibault Lavril, Nicolas Usunier, and Ludovic Denoyer. Learning adaptive exploration strategies in dynamic enviromments through informed policy regularization. arXiv preprint arXiv:2005.02934 2020.
[21] James C King. Symbolic execution and program testing. Communications of the ACM, 19(7):885-39 -94. 1976.
[22] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.
[23] Evan Zheran Liu, Milad Hashemi, Kevin Swersky, Parthasarathy Ranganathan, and Junwhan Ahn. An imitation learning approach for cache replacement. arXiv preprint arXiv:2006. 200.,,999.6,39 2020.
[24] Evan Zheran Liu, Ramtin Keramati, Sudarshan Seshadri, Kelvin Guu, Panupong Pasupat, Emma Brunskill, and Percy Liang. Learning abstract models for strategic exploration and fast reward transfer. arXiv preprint wwXiv:2007.05896 2020.
[25] Evan Zheran Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Decoupling exploration and exploitation for meta- reinforeinor learning without sacrifices, In International Conference on Machine Learning (ICML) 2021.
11

---

[26] Ali Malik, Mike Wu, Vrinda Vasavada, Jinpeng Song. Madison Coots, John Mitchell, Noah Goodman, and Chris Piech. Generative grading: Near human-level accuracy for automated feedback richly structured problems, arXiv preprint 1rXiv:1905.0900 09916, 2019.
[27] Nikhil Mishra. Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. arXiv preprint arXiv:1707.0314. 2017.
[28] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller. Andreas K. Fidjeland, Georg Ostrovski, et al. Human-leve control through deep reinforcement learning. Nature, 518(7540):529- 533, 2015.
[29] Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta rentforeifor learning. arXiv preprint arXiv:1803.11 11 11347, 2018
[30] Allen Nie, Emma Brunskill, and Chris Piech. Play to grade: Testing coding games as classifying markov decision process. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[31] Hyacinth Nwana. Intelligent tutoring systems: an overview. Artificial Intelligence Review, 4(4):251-277, 1990.
[32] Eleanor O' d'Ruurke, Christy Ballweher, and Zoran Popovií. Hint Hint systems may negatively impact performance in educational games. In Proceedings of the first ACM conference on Learning @ scale conference, pages 51-60, 2014.
[33] Benjamin Paaßen, Barbara Hammer, Thomas William Price, Tiffany Bames, Schastian Gross, and Niels Pinkwart. The continuous hint dactory-providing hints in vast and sparsely populated edit distance spaces. arXiv preprint 17XX::7008005644 2017.
[34] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Veti,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, feigh--effrrancee deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[35] Jay A Pfaffmaan. Manipulating and measuring student engagement in computer-basee instruction. PhD thesis, Vanderbilt University, 2003 2003.
[36] Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl- Dieksiies Deep knowledge tracing. Advances in neural information processing systems, 28, 2015.
[37] Chris Piech. Jonathan Huang, Andy Nguyen, Mike Phulsuksombati, Mehran Sahami, and Leonidas Guibas. Learning program embeddings to propagate feedback on student code. In International conference on machine Learning, pages 1093-1102, 2015.
[38] Kate Rakelly, Aurick Zhou Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off-policy meta--riinoorcement learning via probabilistic context variables. orXiv preprint 1903::9003.0820 2019.
[39] Kelly Rivers and Kenneth R Koedinger. Data-driven hint generation in vast solution spaces: self- improving python programming tutor. International Journal of Artificial Intelligence in Education, 27(1): 37-64, 2017
[40] Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbecl. Promp: Proximal meta- policy search, arXiv preprint arXiv:1810.063: 067884 2018
[41] Daniel A Sass, Felicia Castro- Villarreal, Steve Wilkerson, Norma Guerra, and Jeremy Sullivan. A structural model for predicting student retention. The Review of Higher Education, 42(1):103-135, 2018
[42] Gerrit Schoettler, Ashvin Nair. Juan Aparicio Ojea, Sergey Levine, and Eugen Solowjow. Meta- reinforcement learning for robotic industrial insertion tasks. In International Conference on Intelligent 2020.
Robots and Systems (IROS), pages
[43] Rishabh Singh, Sumit Gulwani, and Armando LezaraL Automated feedback generation for introduc- tory programming assignments. In Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation, pages 15-26, 2013.
[44] Bradly Stadie, Ge Yang. Rein Houthooft, Peter Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and llya Sutskever. The importance of sampling inmeta- reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 9280-9220 9990, 2018.
12

---

[45] Hado van Hasselt, Arthur Guez. and David Silver. Deep reinforcement learning with double Q-learning. In Association for the Advancement of Artificial Intelligence (AAAT), volume 16, pages 2094-2100, 2016.
[46] Michael Villano. Probabilistic student models: Bayesian helief networks and knowledge space theory. In International Conference Intelligent Tutoring Systems, pages 491-498, 1992.
[47] Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint IoX.v 101100563,,, 2016.
[48] Jane X Wang. Michael King, Nicolas Pierre Mickael Porcel, Zeb Kurth-NNlsson, Tina Zhu, Charlie Deck, Peter Choy, Mary Cassin, Malcolm Reynolds, H Francis Song, al. Alchemy: A benchmark and analysis toolkit for mneta-reinorreemen learning agents. In Thirty-fithh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[49] Ke Wang, Benjamin Lin, Bjom Rettig. Paul Pardi, and Rishabh Singh. Data-driven feedback generator for online programing courses. In Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale, pages 257-260, 2017.
[50] Ziyu Wang. Tom Schaul, Matteo Hessel, Hado Van Hasselt, Mare Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learning (ICML), 2016.
[51] Mike Wu, Milan Mosse, Noah Goodman, and Chris Piech. Zero shot learning for code education: Rubric sampling with deep learning inference. In Association for the Advancement of Artificial Intelligence (AAAI), volume 33, pages 782-790, 2019.
[52] Mike Wu, Noah Goodman. Chris Piech, and Chelsea Finn. Prototransformer: A meta-learning approach to providing student feedback. arXiv preprint arXiv:2107. 1403,,,, 2021.
[53] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan C. Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. world: A benchmark and evaluation for multi task and meta reinforcement learning. In Conference on Robot Learning (CoRL.,, 2019.
[54] Yan Zheng, Xiaofci Xie, Ting Su, Lei Ma, Jianye Hao, Zhaopeng Meng, Yang Liu, Ruimin Shen, Yingfeng Chen, and Changjie Fan. Wuji: Automatic online combat game testing using cvolutionary deep reinforcement learning. In Automated Software Engineering (ASE), pages 772-784, 2019.
[55] Wenxuan Zhou, Lerrel Pinto, and Abhinav Gupta. Environment prohing interaction policies. arXiv preprint arXiv: /907. 17400 2019.
[56] Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for adaptive deep RL via meta- meta-ica arXiv preprint arx00:1910008338, 2019.
13


Reviews:
[{"review_id": "q-ahgoby_Ir", "reviewer": "Reviewer_CtwK", "summary": "The paper presents a novel approach called DreamGrader to give feedback on\ninteractive student programs. In contrast to previous approaches which only gave\nbinary feedback (correct/incorrect), they assign more fine-grained feedback from\nan error rubric of 8 possible errors. They tackle giving feedback as a\nmeta-exploration problem where each program defines an MDP, and the task is to\nexplore these efficiently to predict the errors accurately. For this both an\nexploration policy and a feedback classifier is learned.\n\nThe exploration policy is learned by maximizing the mutual information between\nthe exploration trajectories and the feedback labels, similarly to how Dream\nmaximizes the mutual information between the trajectories and the task-relevant\ninformation. Following Dream further, this objective is decomposed according to\nthe timesteps to yield a shaped reward function which captures how much new\ninformation a transition provides to the feedback classifier on the errors.\n", "questions": "Table 1 shows $6 \\times 5 = 30$ possible error cases, of which 8 are included in\nthe rubric. I find that some possibly important cases are missing like \"When the\nball hits the wall, it incorrectly doesn't bounce\". What was the reason for\nconstraining the error rubric to 8 errors? Could the method scale to 30 error\ncases?\n\nIn Figure 3, I find the human baselines surprisingly low in some cases. For\nexample, I don't see how a human grader could miss the \"No Ball at Start\" error,\nas it should be obvious after launching the game. Could you explain?\n\nThis is more of a curiosity question: The method works on programs that are\nMDPs. What can we do when this assumption does not hold?\n", "limitations": "I cannot think of any limitations which are not addressed.\n", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nAs far as I know, the method is a first of its kind, able to give fine-grained\nfeedback on interactive programs that correspond to MDPs automatically. It could\nhave significant impact in computer science education research and applications.\n\n## Weaknesses\n\nI think the main weakness of the paper is that its connection to the Dream\nmeta-RL algorithm is not clear enough. In my understanding, the main\ncontribution of Dream is Decoupling the exploRation and ExploitAtion in Meta-RL\n(if we read the capitalized letters together we get Dream).\nHowever, the present paper does not make use of this concept, because we already\nhave the labels for the programs (those correspond to the task-relevant\ninformation in Dream), so it needs to utilize only the exploration part of Dream\nalong with training the classifier.\n\nMoreover, Section 4.3, which should connect the presented method to Dream does\nnot really describe Dream but meta-RL in general, which is confusing, as\nexploration and exploitation works somewhat differently in Dream.\n\nI think the connection between Dream and the presented method should be much\nclearer, and the authors could maybe consider it naming something different\ninstead of DreamGrader.\n\nEven though the play to grade paper [25] also included just one non-toy task, it\nwould be good if the results were demonstrated on more than one programming\ntask.\n\nI find it somewhat of an exaggeration that changing the speed of the ball and\npaddle are examples of creativity.\n\nSmall typo: line 269 - time time\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QudBvlb4HbA", "reviewer": "Reviewer_DZH4", "summary": "In this paper, the authors present DreamGrader, an automated fine-grained grading system for identifying known errors in interactive programs. DreamGrader consists of an program-interaction agent and an error-classifier. The agent interacts with the program and the classifier identifies the presence of different errors based on the agent interaction. The agent and classifier are learnt from a dataset of programs with error annotation. Towards this end, the authors present and leverage a compelling connection between error discovery and meta-reinforcement learning, leading to a unified object for simultaneously learning both the agent and the classifier. The proposed system outperforms strong baselines and prior work by a significant margin. ", "questions": "## Questions\n\n**Q1.** **Meta-RL benchmark**: Can the authors provide more details regarding the benchmark?\n \n**Q2.** Since the classifier is used to generate the rewards for the exploration agent, how is cold-start problem (i.e. an under trained classifier would provide poor reward signals as well) resolved?\n\n **Q3.** Regarding Figure 3. In plot 1 the performance decreases between time-steps 4000-5000. That is quite curious. Is there any explanation for this behavior?\n\n**Q4.**  Regarding Figure 3: Why are all the human performance line at the same level? Since they return the list of found errors (cf. appendix C.1) shouldn't the plot be different for at least some error-types? \n\n**Q5.** Section 5.3: the result of having equally poor performance on training set speeds as on test set speeds indicates perhaps more data could have resolved the issue. Have the authors tried to have more data for this experiment? Does it change the results?\n\n**Q6.** Do the authors have any explanation for why the extension of [1] performs quite poorly?\n\n## Suggestion:\n\nMaking Figure 3 color-blind friendly would be nice! Further, it would be valuable to have the y-axis annotated on all the graphs. \n\n## References\n\n[1] Play to Grade: Testing Coding Games as Classifying Markov Decision Process, Allen Nie et al., NeurIPS 2021", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n\n1. **Originality**: The key insight of the authors, i.e., to pose the known-error discovery problem as an exploration-in-meta-RL problem, is original and novel. This work opens this research direction for further investigation.\n\n2. **Presentation**: The paper is very well written. I really appreciate the authors for providing a well grounded motivation for the problem at hand (providing fine-grained feedback on interactive program assignments). \n\n3. **Impact**: The proposed system is fairly feasible to deploy, specially in MOOCs, and can lead to impactful improvements in student grading in massive open online courses (MOOCs). \n\n4. **Experiments**: The proposed method is compared against a power extension of prior work ([1]) and surpasses it by a large margin. Further, the results on real-world programs rather than simulated ones, which is always a plus point.\n\n# Minor Weaknesses:\n\n1. The proposed method is verified for only a single assignment. While this is due to the lack of annotated datasets, it does raise questions about the generality of the results. \n\n2. **Meta-RL benchmark**: The paper espouses the use of its student-program dataset as a meta-exploration benchmark, it doesn't show thorough details (for example, proper statistics of the dataset or performance of baselines).  While the authors can be commended for releasing the code, simply releasing the dataset is not sufficient to call it a benchmark, *and* to count it as a contribution.\n\n3. The approach is tested on very simple settings - 8 error-types and 100 time-steps. Interactive programming assignments typically can consist of an order of magnitude more types of error and number of time-steps. The method would become inapplicable if data requirements also grow proportionally.\n\n## References\n\n[1] Play to Grade: Testing Coding Games as Classifying Markov Decision Process, Allen Nie et al., NeurIPS 2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FrCMPds4Xkv", "reviewer": "Reviewer_BUXC", "summary": "This paper proposes a meta exploration reinforcement learning algorithm, dreamgrader, that tries to provide feedback to student-authored interactive programs. The authors conducted experiments on a large real dataset and showed that their approach is effective. ", "questions": "1. If I understood correctly (I may not have), an important benefit of meta learning is generalization to new tasks. A task can be a new student program, which is what this paper studies in its experiments, for sure, but it can be a new error type? What if you leave out an error during training, train on only 7 errors, and see if your learned policy can adapt to the 8th? Like Figure 7 (middle) in the DREAM paper? A discussion or an additional experiment would help a lot here. \n\n2. Following the previous question - the factorized version of the method learns a separate exploration policy and a separate feedback classification model for each error. This means 8 different sets of models. I don't know if this is any different than the original play to grade approach (1 binary label) since you can treat each error label independently. Which means fundamentally I don't see the improvement here over that paper: feedback simply means doing 8 separate binary classification tasks, so can we do the same with the previous approach just repeated 8 times? \n\n3. Following the previous question - why are the errors a ordered list (line 121) but treated independently? Intuitively, I guess some errors could be related, which means there could be some shared layers in these models (e.g., for trajectory representations) before going their separate ways. Is there no sweet spot between one unified model for all errors and 8 separate models?\n\n4. The baseline is basically a strawman (one that cannot leverage more than 8 training data points). It's still good to see the direct max version, though, since that illustrate the benefit of credit assignment (which I don't know is the new thing in this paper compared to DREAM). \n\n5. More experimental details can help: you claim super human performance on some errors, which is understandable, but what is human interreater agreement? How are the ground truth on the 99.5% of the programs not seen during training obtained? Each grader only graded 6 problems and those are averaged in Table 2? Are the results reliable with such a small evaluation set? Any statistical significance?", "limitations": "The authors did discuss potential limitations in Section 6.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The overall idea is good: using meta RL to learn how to balance exploration and exploitation for different errors when grading programs\n- Experiments are conducted on a large, real dataset, not a synthetic one\n\nWeaknesses:\n- Some details of the experiments are unclear. But hopefully this can be resolved through discussions, I'm happy to revisit the rating again depending on what the authors and other reviewers say\n- The connection to meta reinforcement learning needs more discussion (essentially only 2 sentences lines 273-275). More discussions would help", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept