Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: JavaScript is the standard programming language of the Web; according to the stats it is used on 95% of the (at least) 1.6 billion websites in the world.  Compared to other programming languages, it posses certain unique characteristics which are also responsible for making it so popular for the Web.   In this interesting work the authors aim to provide a novel data-driven system for detecting and automatically fixing bugs in Javascript.  The authors provide motivating examples behind their research. Indeed, Javascript poses unique challenges as described eloquently in Section 2, and there is a lot of space for improvement. Static analyzers have non-trivial limitations, while there is space for improving data-driven approaches. This is what Hoppity aims to achieve by translating the program into a graph, embedding the nodes and the graph as points in a d dimensional space using graph neural networks, and then using a controller that uses LSTMs decide the action to be taken among a predefined set of possible actions or a single step graph edit. These actions are reasonable, and are able to fix many bugs assuming the right sequence of actions is performed. For the purposes of learning, the author(s) use a corpus crawled and preprocessed from Github that contains more than half a million programs. Overall, I found this paper very interesting to read, and with large potential impact in practice. The paper contains a solid engineering effort, starting from the dataset collection and its preprocessing, to using state-of-the-art machinery to develop Hoppity. Therefore, I support its acceptance. However, some things were not clear from the writeup, and I hope the author(s) of the paper can give some insights. 

- What is the effect of parameter T, i.e., the number of iterations? How is it set in the experiments? Clearly, the authors have a knowledge of what T should be since they have preferred programs with fewer commits.
- Following on my previous point, given the sequence of changes/graph transformations you perform, what is the distribution of the 'edit distance' (i.e., number of hops to fix a bug) in the dataset that you have? While the author(s) have 
already provided a lot of stats in the Appendix, it would be interesting to see such a plot. This distribution could be insightful and serve as a rule of thumb for understanding the effect of T. 
- What is the effect of the beam size? Can you plot  the accuracy  as a function of the beam size? 
- Have you tried points with more than 500 nodes to see how the size of these graphs affect the performance of Hoppity? 
- Can you provide further details on the running times and the GPU specs?   
- The evaluation does not put enough emphasis on false positives/false negative analysis. Is it the case that bug-free programs are treated as such? 
-  Have you tried Hoppity on other programming language(s)?  Do you expect such an improved performance over baselines for other languages (e.g., C++) as well?


review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The paper proposes to learn from bugfixing commits to fix errors in other code. There are several good contributions of the paper, but the main one is to design a language of instructions that fix a program and to formulate the prediction to be a sequence of such instructions. The model, however, is insightful in other ways as well. The lack of naming features or generation of names makes it to point to other identifiers in a program - a major problem for most models for code. Instead, the proposed model only builds embeddings from the structure around the variables.

The work is also well evaluated, on real dataset and also it attempts to compare to the best available static analysis tools. The kind of bugs addressed by the work was also not considered in previous papers. One thing that comes to the examples of the discovered bugs is that even when the fix is shown, the user would still need detailed explanations on why was it a bug.

So, I also have questions for the authors:
1. Do you think it is possible to observe a similar reasoning to the reasoning in your text for why the buggy examples from Figure 1 are wrong, if the activations of the neural network are exposed or with other NN debugging technique.
2. It seems that a specific sequence of actions is provided in the training data and that sequence is left-to-right edits to apply the fix. In this case, doesn’t it make sense to apply restrictions on the Location primitive similar in spirit to the attention masking (see here: http://jalammar.github.io/illustrated-gpt2/#part-2-illustrated-self-attention )

Minor:
- NO_OP was used in some places (pages 4 and 5) and STOP in others (figure 2).
- N_k(v) is not defined.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a graph tranformation-based code repair tool. By representing source code as a graph a network is asked to take a series of simple graph edit operations to edit the code. The authors show that their method better predicts edits from existing code.

Overall, I find the problem interesting and the neural approach of the authors reasonable, principled and interesting. However, the evaluation is either badly written or the authors have not understood the related literature they are comparing to. It is thus unclear how this model compares to alternatives. I believe that this is good work that needs to eventually be published, but it's not ready at this time.

* In 3.2.1 "Value" paragraph "Instead of predicting the replacement value using a language generative model (Chen et al 2018, Allamanis et al 2018), we let the model to choose from either the values appearing in the current file." 
  
   -The Chen et al. (2018) paper is indeed a translation-style model, but thanks to the copying mechanism, the model can also choose from values appearing in the current input.
   - The Allamanis et al. (2018) is *not* a generative model and it can only select a misused variable for those that exist on within the scope (for the Variable Misuse Task)

* The authors compare to the VARNAMING (Section 6.2) task of Allamanis et al. (2018). This is odd, since to my understanding, this task is about alpha-renaming, i.e. semantics preserving renaming of all identifiers of a variable. This is not comparable with the REP_VAL action which simply replaces the value of a single leaf node from the value table. The VARMISUSE objective (picking one variable directly from the value table) is the closest analogue, although it performs no localization. 

Thus, it is not surprising that a char-level language model cannot predict the names of the elements in the value table. So, I don't think that we learn anything from the current comparison.

* It's unclear to me why the REP_TYPE action is relevant to VARMISUSE task of Allamanis et al. (2018). This seems to simply be a classification task (among all known/valid types of the tree) given an existing position.

* Given the above two points, it's unclear what the comparison with the baseline (Table 3) means and what one can deduce from the evaluation: The Allamanis et al. (2018) work tries to find the correct variable at a given location; this work tries to pick a location and find a different value from current one. The Allamanis et al. (2018) and Cvitkovic et al. (2018) uses additional semantic information (as edges), importantly data flow and control flow. This work uses none of those. I am not sure that I can see how the comparison here can work.

* In 3.1 "Unlike previous approaches, we additionally add value nodes that store the actual content of the leaf nodes [...]. The purpose of introducing this additional set of nodes is to provide a name-independent strategy for code representation and modification". I am not sure what's unlike previous approaches. For example, the Allamanis et al. (2018) and the Cvitkovic et al. (2018) work, connect the same variables with various ways (e.g. data-flow edges) in a way that provides a name-independent strategy for code representation. I would fing it surprising that a principled comparison between Allamanis et al. (2018)/ Cvitkovic et al. (2018) and this work will yield any improvements: all papers use the same model (variations of a graph neural network) and this work uses strictly fewer information (no data flow, control flow edges). A comparison would simply compare how the two graph-extraction methods compare, but nothing valuable otherwise.

* The authors of this work frequently refer to Chen et al. (2018), but never compare with it. I would expect that this model beats the Chen et al. (2018) work, but this needs to be shown. In my opinion this is a much more relevant baseline, compared to the comparison with Allamanis et al. (2018). If the authors insist on comparing with that work, then a "fair" comparison needs to be made. Comparing between REP_VAL and VARMISUSE given the location of the node-to-be-repaired, is probably the only reasonable option. Comparing with Vasic et al. (2019) for the REP_VAL/REP_TYPE would also be reasonable.

* The correctness of the NO_OP predictions are never evaluated. Does the model "know" when to terminate or does it keep editing? It's unclear what is the overall accuracy (for a full edit).

Other questions:

* The "ADD" operation predicts two locations: one for the parent and one for the left sibling of the node. Does this mean that for a given node the "ADD" operation will never be able to add a left-most child (since it has no left sibling)?

* It has been recently found that code corpora collected by scraping GitHub may contain a disproportionate amount of duplicates (Lopes et al. 2017, Allamanis 2018). It is unclear if the authors have taken any steps to detect and remove duplicates that would affect their results.

* In Table 2: Does "Beam-3" mean "accuracy in the top 3" or does it mean "accuracy of the top prediction when the beam size is 3"? The difference between "Beam-1" and "Beam-3" is surprisingly large. Can you explain why?

* In the TAJS comparison, it is unclear if any negative examples are added: it's unclear what's the false positive ratio of HOPPITY vs TAJS.

Minor:

* Abstract: "Github" -> "GitHub"
* 3.2 REP_VAL: "Tthe"-> "The"
* Sec 7: "Vasic et al. (2019) present a pointer network..." It is unclear if this work outperforms the Allamanis et al. (2018) work as the comparison is only partial. Another ICLR submission https://openreview.net/forum?id=B1lnbRNtwr suggests that this is not the case.
* The citation of Chen et al. (2018) should be capitalized correctly "Sequencer"->"SequenceR"


## References

Allamanis, Miltiadis. "The Adverse Effects of Code Duplication in Machine Learning Models of Code." arXiv preprint arXiv:1812.06469 (2018).

Cvitkovic, Milan, Badal Singh, and Anima Anandkumar. "Open Vocabulary Learning on Source Code with a Graph-Structured Cache." arXiv preprint arXiv:1810.08305 (2018).

Lopes, Cristina V., et al. "DéjàVu: a map of code duplicates on GitHub." Proceedings of the ACM on Programming Languages 1.OOPSLA (2017): 84.
review_title: Official Blind Review #3

Meta-review: This paper presents a learning-based approach to detect and fix bugs in JavaScript programs. By modeling the bug detection and fix as a sequence of graph transformations, the proposed method achieved promising experimental results on a large JavaScript dataset crawled from GitHub.

All the reviews agree to accept the paper for its reasonable and interesting approach to solve the bug problems. The main concerns are about the experimental design, which has been addressed by the authors in the revision. 

Based on the novelty and solid experiments of the proposed method, I agreed to accept the paper as other revises.

Decision: Accept (Spotlight)