Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors present 3 architectures for learning representations of programs from execution traces. In the variable trace embedding, the input to the model is given by a sequence of variable values. The state trace embedding combines embeddings for variable traces using a second recurrent encoder. The dependency enforcement embedding performs element-wise multiplication of embeddings for parent variables to compute the input of the GRU to compute the new hidden state of a variable. The authors evaluate their architectures on the task of predicting error patterns for programming assignments from Microsoft DEV204.1X (an introduction to C# offered on edx) and problems on the Microsoft CodeHunt platform. They additionally use their embeddings to decrease the search time for the Sarfgen program repair system.

This is a fairly strong paper. The proposed models make sense and the writing is for the most part clear, though there are a few places where ambiguity arises:

- The variable "Evidence" in equation (4) is never defined. 

- The authors refer to "predicting the error patterns", but again don't define what an error pattern is. The appendix seems to suggest that the authors are simply performing multilabel classification based on a predefined set of classes of errors, is this correct? 

- It is not immediately clear from Figures 3 and 4 that the architectures employed are in fact recurrent.

- Figure 5 seems to suggest that dependencies are only enforced at points in a program where assignment is performed for a variable, is this correct?

Assuming that the authors can address these clarity issues, I would in principle be happy for the paper to appear. 
review_title: A solid paper with some clarity issues

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper considers the task of learning program embeddings with neural networks with the ultimate goal of bug detection program repair in the context of students learning to program. Three NN architectures are explored, which leverage program semantics rather than pure syntax.  The approach is validated using programming assignments from an online course, and compared against syntax based approaches as a baseline.

The problem considered by the paper is interesting, though it's not clear from the paper that the approach is a substantial improvement over previous work. This is in part due to the fact that the paper is relatively short, and would benefit from more detail.  I noticed the following issues:

1) The learning task is based on error patterns, but it's not clear to me what exactly that means from a software development standpoint.
2) Terms used in the paper are not defined/explained. For example, I assume GRU is gated recurrent unit, but this isn't stated.
3) Treatment of related work is lacking.  For example, the Cai et al. paper from ICLR 2017 is not considered
4) If I understand dependency reinforcement embedding correctly, a RNN is trained for every trace. If so, is this scalable?

I believe the work is very promising, but this manuscript should be improved prior to publication.
review_title: Interesting application, but lacks clarity

final_score: rating: 7: Good paper, accept
review_content: Summary of paper: The paper proposes an RNN-based neural network architecture for embedding programs, focusing on the semantics of the program rather than the syntax. The application is to predict errors made by students on programming tasks. This is achieved by creating training data based on program traces obtained by instrumenting the program by adding print statements. The neural network is trained using this program traces with an objective for classifying the student error pattern (e.g. list indexing, branching conditions, looping bounds).

---

Quality: The experiments compare the three proposed neural network architectures with two syntax-based architectures. It would be good to see a comparison with some techniques from Reed & De Freitas (2015) as this work also focuses on semantics-based embeddings.
Clarity: The paper is clearly written.
Originality: This work doesn't seem that original from an algorithmic point of view since Reed & De Freitas (2015) and Cai et. al (2017) among others have considered using execution traces. However the application to program repair is novel (as far as I know).
Significance: This work can be very useful for an educational platform though a limitation is the need for adding instrumentation print statements by hand.

---

Some questions/comments:
- Do we need to add the print statements for any new programs that the students submit? What if the structure of the submitted program doesn't match the structure of the intended solution and hence adding print statements cannot be automated?

---

References 

Cai, J., Shin, R., & Song, D. (2017). Making Neural Programming Architectures Generalize via Recursion. In International Conference on Learning Representations (ICLR).
review_title: .

Meta-review: PROS:

1. Interesting and clearly useful idea
2. The paper is clearly written.
3. This work doesn't seem that original from an algorithmic point of view since Reed & De Freitas (2015) and Cai et. al (2017) among others have considered using execution traces. However the application to program repair is novel (as far as I know).
4. This work can be very useful for an educational platform though a limitation is the need for adding instrumentation print statements by hand.

CONS:

1. The paper has some clarity issues which the authors have promised to fix.

---
Decision: Accept (Poster)