Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors present DreamGrader, an automated fine-grained grading system for identifying known errors in interactive programs. DreamGrader consists of an program-interaction agent and an error-classifier. The agent interacts with the program and the classifier identifies the presence of different errors based on the agent interaction. The agent and classifier are learnt from a dataset of programs with error annotation. Towards this end, the authors present and leverage a compelling connection between error discovery and meta-reinforcement learning, leading to a unified object for simultaneously learning both the agent and the classifier. The proposed system outperforms strong baselines and prior work by a significant margin. 
strengths_and_weaknesses: # Strengths:

1. **Originality**: The key insight of the authors, i.e., to pose the known-error discovery problem as an exploration-in-meta-RL problem, is original and novel. This work opens this research direction for further investigation.

2. **Presentation**: The paper is very well written. I really appreciate the authors for providing a well grounded motivation for the problem at hand (providing fine-grained feedback on interactive program assignments). 

3. **Impact**: The proposed system is fairly feasible to deploy, specially in MOOCs, and can lead to impactful improvements in student grading in massive open online courses (MOOCs). 

4. **Experiments**: The proposed method is compared against a power extension of prior work ([1]) and surpasses it by a large margin. Further, the results on real-world programs rather than simulated ones, which is always a plus point.

# Minor Weaknesses:

1. The proposed method is verified for only a single assignment. While this is due to the lack of annotated datasets, it does raise questions about the generality of the results. 

2. **Meta-RL benchmark**: The paper espouses the use of its student-program dataset as a meta-exploration benchmark, it doesn't show thorough details (for example, proper statistics of the dataset or performance of baselines).  While the authors can be commended for releasing the code, simply releasing the dataset is not sufficient to call it a benchmark, *and* to count it as a contribution.

3. The approach is tested on very simple settings - 8 error-types and 100 time-steps. Interactive programming assignments typically can consist of an order of magnitude more types of error and number of time-steps. The method would become inapplicable if data requirements also grow proportionally.

## References

[1] Play to Grade: Testing Coding Games as Classifying Markov Decision Process, Allen Nie et al., NeurIPS 2021
questions: ## Questions

**Q1.** **Meta-RL benchmark**: Can the authors provide more details regarding the benchmark?
 
**Q2.** Since the classifier is used to generate the rewards for the exploration agent, how is cold-start problem (i.e. an under trained classifier would provide poor reward signals as well) resolved?

 **Q3.** Regarding Figure 3. In plot 1 the performance decreases between time-steps 4000-5000. That is quite curious. Is there any explanation for this behavior?

**Q4.**  Regarding Figure 3: Why are all the human performance line at the same level? Since they return the list of found errors (cf. appendix C.1) shouldn't the plot be different for at least some error-types? 

**Q5.** Section 5.3: the result of having equally poor performance on training set speeds as on test set speeds indicates perhaps more data could have resolved the issue. Have the authors tried to have more data for this experiment? Does it change the results?

**Q6.** Do the authors have any explanation for why the extension of [1] performs quite poorly?

## Suggestion:

Making Figure 3 color-blind friendly would be nice! Further, it would be valuable to have the y-axis annotated on all the graphs. 

## References

[1] Play to Grade: Testing Coding Games as Classifying Markov Decision Process, Allen Nie et al., NeurIPS 2021
limitations: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a meta exploration reinforcement learning algorithm, dreamgrader, that tries to provide feedback to student-authored interactive programs. The authors conducted experiments on a large real dataset and showed that their approach is effective. 
strengths_and_weaknesses: Strength:
- The overall idea is good: using meta RL to learn how to balance exploration and exploitation for different errors when grading programs
- Experiments are conducted on a large, real dataset, not a synthetic one

Weaknesses:
- Some details of the experiments are unclear. But hopefully this can be resolved through discussions, I'm happy to revisit the rating again depending on what the authors and other reviewers say
- The connection to meta reinforcement learning needs more discussion (essentially only 2 sentences lines 273-275). More discussions would help
questions: 1. If I understood correctly (I may not have), an important benefit of meta learning is generalization to new tasks. A task can be a new student program, which is what this paper studies in its experiments, for sure, but it can be a new error type? What if you leave out an error during training, train on only 7 errors, and see if your learned policy can adapt to the 8th? Like Figure 7 (middle) in the DREAM paper? A discussion or an additional experiment would help a lot here. 

2. Following the previous question - the factorized version of the method learns a separate exploration policy and a separate feedback classification model for each error. This means 8 different sets of models. I don't know if this is any different than the original play to grade approach (1 binary label) since you can treat each error label independently. Which means fundamentally I don't see the improvement here over that paper: feedback simply means doing 8 separate binary classification tasks, so can we do the same with the previous approach just repeated 8 times? 

3. Following the previous question - why are the errors a ordered list (line 121) but treated independently? Intuitively, I guess some errors could be related, which means there could be some shared layers in these models (e.g., for trajectory representations) before going their separate ways. Is there no sweet spot between one unified model for all errors and 8 separate models?

4. The baseline is basically a strawman (one that cannot leverage more than 8 training data points). It's still good to see the direct max version, though, since that illustrate the benefit of credit assignment (which I don't know is the new thing in this paper compared to DREAM). 

5. More experimental details can help: you claim super human performance on some errors, which is understandable, but what is human interreater agreement? How are the ground truth on the 99.5% of the programs not seen during training obtained? Each grader only graded 6 problems and those are averaged in Table 2? Are the results reliable with such a small evaluation set? Any statistical significance?
limitations: The authors did discuss potential limitations in Section 6.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a novel approach called DreamGrader to give feedback on
interactive student programs. In contrast to previous approaches which only gave
binary feedback (correct/incorrect), they assign more fine-grained feedback from
an error rubric of 8 possible errors. They tackle giving feedback as a
meta-exploration problem where each program defines an MDP, and the task is to
explore these efficiently to predict the errors accurately. For this both an
exploration policy and a feedback classifier is learned.

The exploration policy is learned by maximizing the mutual information between
the exploration trajectories and the feedback labels, similarly to how Dream
maximizes the mutual information between the trajectories and the task-relevant
information. Following Dream further, this objective is decomposed according to
the timesteps to yield a shaped reward function which captures how much new
information a transition provides to the feedback classifier on the errors.

strengths_and_weaknesses: ## Strengths

As far as I know, the method is a first of its kind, able to give fine-grained
feedback on interactive programs that correspond to MDPs automatically. It could
have significant impact in computer science education research and applications.

## Weaknesses

I think the main weakness of the paper is that its connection to the Dream
meta-RL algorithm is not clear enough. In my understanding, the main
contribution of Dream is Decoupling the exploRation and ExploitAtion in Meta-RL
(if we read the capitalized letters together we get Dream).
However, the present paper does not make use of this concept, because we already
have the labels for the programs (those correspond to the task-relevant
information in Dream), so it needs to utilize only the exploration part of Dream
along with training the classifier.

Moreover, Section 4.3, which should connect the presented method to Dream does
not really describe Dream but meta-RL in general, which is confusing, as
exploration and exploitation works somewhat differently in Dream.

I think the connection between Dream and the presented method should be much
clearer, and the authors could maybe consider it naming something different
instead of DreamGrader.

Even though the play to grade paper [25] also included just one non-toy task, it
would be good if the results were demonstrated on more than one programming
task.

I find it somewhat of an exaggeration that changing the speed of the ball and
paddle are examples of creativity.

Small typo: line 269 - time time


questions: Table 1 shows $6 \times 5 = 30$ possible error cases, of which 8 are included in
the rubric. I find that some possibly important cases are missing like "When the
ball hits the wall, it incorrectly doesn't bounce". What was the reason for
constraining the error rubric to 8 errors? Could the method scale to 30 error
cases?

In Figure 3, I find the human baselines surprisingly low in some cases. For
example, I don't see how a human grader could miss the "No Ball at Start" error,
as it should be obvious after launching the game. Could you explain?

This is more of a curiosity question: The method works on programs that are
MDPs. What can we do when this assumption does not hold?

limitations: I cannot think of any limitations which are not addressed.
review_title: null

Meta-review: metareview: This work presents dreamgrader that aims to provide feedback to student-authored interactive programs. Reviewers all agreed that this paper presents a novel and original idea, solid experiments on real-world programs, as well as potential impact on MOOCs. There were some minor concerns and most got resolved during the discussion stage. Thus we recommend acceptance. 

confidence: Certain
Decision: Accept