Title: Learning to Represent Programs with Graphs
Authors: Miltiadis Allamanis
Marc Brockschmidt
Mahmoud Khademi
Venue: ICLR 2018
Keywords: programs
source code
graph neural networks

Abstract: Learning tasks on source code (i.e., formal languages) have been considered recently, but most work has tried to transfer natural language methods and does not capitalize on the unique opportunities offered by code's known syntax. For example, long-range dependencies induced by using the same variable or function in distant locations are often not considered. We propose to use graphs to represent both the syntactic and semantic structure of code and use graph-based deep learning methods to learn to reason over program structures.

In this work, we present how to construct graphs from source code and how to scale Gated Graph Neural Networks training to such large graphs. We evaluate our method on two tasks: VarNaming, in which a network attempts to predict the name of a variable given its usage, and VarMisuse, in which the network learns to reason about selecting the correct variable that should be used at a given program location. Our comparison to methods that use less structured program representations shows the advantages of modeling known structure, and suggests that our models learn to infer meaningful names and to solve the VarMisuse task in many cases. Additionally, our testing showed that VarMisuse identifies a number of bugs in mature open-source projects.

TL;DR: Programs have structure that can be represented as graphs, and graph neural networks can learn to find bugs on such graphs

Review discussion:
Summary:  The paper applies graph convolutions with deep neural networks to the problem of "variable misuse" (putting the wrong variable name in a program statement) in graphs created deterministically from source code.  Graph structure is determined by program abstract syntax tree (AST) and next-token edges, as well as variable/function name identity, assignment and other deterministic semantic relations.  Initial node embedding comes from both type and tokenized name information.  Gated Graph Neural Networks (GGNNs, trained by maximum likelihood objective) are then run for 8 iterations at test time.
The evaluation is extensive and mostly very good.  Substantial data set of 29m lines of code.  Reasonable baselines.  Nice ablation studies.  I would have liked to see separate precision and recall rather than accuracy.  The current 82.1% accuracy is nice to see, but if 18% of my program variables were erroneously flagged as errors, the tool would be useless.  I'd like to know if you can tune the threshold to get a precision/recall tradeoff that has very few false warnings, but still catches some errors.
Nice work creating an implementation of fast GGNNs with large diverse graphs.  Glad to see that the code will be released.  Great to see that the method is fast---it seems fast enough to use in practice in a real IDE.
The model (GGNN) is not particularly novel, but I'm not much bothered by that.  I'm very happy to see good application papers at ICLR.  I agree with your pair of sentences in the conclusion: "Although source code is well understood and studied within other disciplines such as programming language research, it is a relatively new domain for deep learning. It presents novel opportunities compared to textual or perceptual data, as its (local) semantics are well-defined and rich additional information can be extracted using well-known, efficient program analyses."  I'd like to see work in this area encouraged.
So I recommend acceptance.  If it had better (e.g. ROC curve) evaluation and some modeling novelty, I would rate it higher still.
Small notes:
The paper uses the term "data flow structure" without defining it.
Your data set consisted of C# code.  Perhaps future work will see if the results are much different in other languages.
Thank you for reviewing our work so kindly. Please note that the evaluation in our submission only covers 2.9M SLOC (not 29M), even though since the submission we have performed additional experiments with similar results on the Roslyn project (~2M SLOC).
We have just updated our submission to also include ROC and PR curves for our main model in the appendix, which show that for a false positive rate of 10%, our model achieves a true positive rate of 73% on the SeenTestProj dataset and 69% on UnseenTestProj. The PR curve indicates that setting a high certainty threshold for such highlighting should yield relatively few false positives. We are working on further improving these numbers by addressing common causes of mistakes (i.e., our model often proposes to use a class field “_field” when the ground truth is the corresponding getter property “Field”; a simple alias analysis can take care of these case).
The paper introduces an application of Graph Neural Networks (Li's Gated Graph Neural Nets, GGNNs, specifically) for reasoning about programs and programming. The core idea is to represent a program as a graph that a GGNN can take as input, and train the GGNN to make token-level predictions that depend on the semantic context. The two experimental tasks were: 1) identifying variable (mis)use, ie. identifying bugs in programs where the wrong variable is used, and 2) predicting a variable's name by consider its semantic context.
The paper is generally well written, easy to read and understand, and the results are compelling. The proposed GGNN approach outperforms (bi-)LSTMs on both tasks. Because the tasks are not widely explored in the literature, it could be difficult to know how crucial exploiting graphically structured information is, so the authors performed several ablation studies to analyze  this out. Those results show that as structural information is removed, the GGNN's performance diminishes, as expected. As a demonstration of the usefulness of their approach, the authors ran their model on an unnamed open-source project and claimed to find several bugs, at least one of which potentially reduced memory performance.
Overall the work is important, original, well-executed, and should open new directions for deep learning in program analysis. I recommend it be accepted.
Thank you for your kind review. We have updated our paper to discuss bugs found by the model in more detail and have privately reported more bugs found in Roslyn to the developers (cf. https://github.com/dotnet/roslyn/pull/23437, and note that this GitHub issue does not de-anonymize the paper authors).
This paper presents a novel application of machine learning using Graph NN's on ASTs to identify incorrect variable usage and predict variable names in context. It is evaluated on a corpus of 29M SLOC, which is a substantial strength of the paper.
The paper is to be commended for the following aspects:
1) Detailed description of GGNNs and their comparison to LSTMs
2) The inclusion of ablation studies to strengthen the analysis of the proposed technique
3) Validation on real-world software data
4) The performance of the technique is reasonable enough to actually be used.
In reviewing the paper the following questions come to mind:
1) Is the false positive rate too high to be practical?  How should this be tuned so developers would want to use the tool?
2) How does the approach generalize to other languages? (Presumably well, but something to consider for future work.)
Despite these questions, though, this paper is a nice addition to deep learning applications on software data and I believe it should be accepted.
Thank you for reviewing our work so kindly. Please note that the evaluation in our submission only covers 2.9M SLOC (not 29M), even though we have performed additional experiments with similar results on the Roslyn project (~2M SLOC).
Regarding your first question: We have just updated our submission to also include ROC and PR curves for our main model in the appendix, which show that for a false positive rate of 10%, our model achieves a true positive rate of 73% on the SeenTestProj dataset and 69% on UnseenTestProj. We expect our system to be most useful in a code review setting, where locations in which the model disagrees with the ground truth are highlighted for a reviewer. The PR curve indicates that setting a high certainty threshold for such highlighting should yield relatively few false positives.
Regarding your second question: We have not tested our model on other languages so far.
However, we expect similar performance on other strongly typed languages such as Java. An interesting research question will be to explore how the model could be adapted to gradually typed (e.g. TypeScript) or untyped (e.g. JavaScript or Python) languages.
We have updated our submission to address some of the comments raised here and to include updated results obtained after the deadline:
- We have included a reference to Bichsel et al. (CCS 2016) and improved
the wording in the related work section to better describe their approach.
- Initial node representations are now computed by a linear layer taking the
concatenation of node label embeddings and type representation as input.
In our experiments, we found this to help with generalization performance.
- We noticed and resolved an issue with the local model (LOC) implementation
on the VarMisuse task and updated Tab. 1 to reflect the results. They are now
substantially better, but the model still performs much worse than all other
models.
- We have updated experimental results for our main GGNN model on the
VarMisuse task to reflect the small model changes and better tuning of
hyperparameters, mainly improving the generalization of GGNNs to unseen
projects (jumping from 68.6% accuracy to 77.9%). We have not updated the
results for all models in the ablation experiments (Table 2) but will do so in a
second update to the submission.
- We have included figures with the ROC and PR curves for our experiments in
the appendix. The key number, requested by the reviewers, is that for the
widely accepted false positive rate of 10% our model achieves a true positive
rate of 73%.
- We have updated the paper to briefly discuss 3 more bugs found in Roslyn,
one with the potential to crash Visual Studio (cf.
https://github.com/dotnet/roslyn/pull/23437, and note that this GitHub issue
does not de-anonymize the paper authors).
We are currently working on an extension of the graph representation of programs that takes conditional dependencies into account (i.e., “variable x is guarded by x != null”) and a refined experimental setup for the VarMisuse task that makes use of an aliasing analysis to further filter the set of candidate variables for each slot. We will rerun all experiments once these changes are finished.
We did not finish the work on an improved data generation procedure yet and thus cannot provide updated experimental results before the end of the official rebuttal phase. We will continue this work and provide updated results here as soon as feasible.

Meta-review:
role: PC
data: comment: There was some debate between the authors and an anonymous commentator on this paper.  The feeling of the commentator was that existing work (mostly from the PL community) was not compared to appropriately and, in fact, performs better than this approach.  The authors point out that their evaluation is hard to compare directly but that they disagreed with the assessment.  They modified their texts to accommodate some of the commentator's concerns; agreed to disagree on others; and promised a fuller comparison to other work in the future.

I largely agree with the authors here and think this is a good and worthwhile paper for its approach.

PROS:
1. well written
2. good ablation study
3. good evaluation including real bugs identified in real software projects
4. practical for real world usage

CONS:
1. perhaps not well compared to existing PL literature or on existing datasets from that community
2. the architecture (GGNN) is not a novel contribution

Decision: Accept (Oral)