Title: Neural Code Completion
Authors: Chang Liu
Xin Wang
Richard Shin
Joseph E. Gonzalez
Dawn Song
Venue: ICLR 2017
Keywords: Deep learning
Applications

Abstract: Code completion, an essential part of modern software development, yet can bechallenging for dynamically typed programming languages.  In this paper we ex-plore the use of neural network techniques to automatically learn code completionfrom  a  large  corpus  of  dynamically  typed  JavaScript  code.   We  show  differentneural networks that leverage not only token level information but also structuralinformation,  and  evaluate  their  performance  on  different  prediction  tasks.   Wedemonstrate that our models can outperform the state-of-the-art approach, whichis based on decision tree techniques, on both next non-terminal and next terminalprediction tasks by 3.8 points and 0.5 points respectively.  We believe that neuralnetwork techniques can play a transformative role in helping software developersmanage the growing complexity of software systems, and we see this work as afirst step in that direction.


Review discussion:
A few questions:
- It seems a strange to have all the EMPTY terminal tokens in the encoding described in section 4. What is the justification for using the described encoding of trees instead of an encoding like in Grammar as a Foreign Language [1]?
- What is the point of predicting non-terminals in a code completion context? To make code completion suggestions, terminals need to be predicted, right?
- Can you elaborate on the training details in 5.2?
Specifically, I didn't understand why it makes sense to only reset hidden states at the beginning of an epoch.
[1] Vinyals, Oriol, et al. "Grammar as a foreign language." Advances in Neural Information Processing Systems. 2015.
We preserve EMPTY terminal tokens to make our work comparable to the PHOG paper (Bielik et al. 2016), which uses the same encoding.
The non-terminal points contains the program structure information that cannot be recovered from only the terminals.
For example, in the dataset that we use, a non-terminal ReturnStatement only has an EMPTY terminal, instead of a “return” keyword. Predicting this non-terminal node is equivalent to predicting “return” for the code-completion task. We have updated our paper (page 4) to give a brief explanation on the reason.
Resetting only at the beginning is a bug that we accidentally introduced during training. We have fixed this bug. We will update the paper to reflect new results within 10 days.
Could the authors give examples, code snippets, which represents the possible use of the various predictors (NT2N, NT2NT, NTN2T)?
Why is it useful to measure the non-terminal prediction accuracy? From the user viewpoint only (visible) terminal prediction accuracy makes sense.
Table 1.: "Overall": should that mean the vocabulary size?
How many queries are used when not all queries are used (programs < 30000 tokens)?
Have the authors tried to use bigger lexicon > 50k? To support the argument to the result in table 3.
Denying prediction: have the authors tried to use 0<\alpha<1 for y_i = UNK, to down weight the possibility of UNK?
Predicting UNK in the test set is not positive results, UNK means out-of-vocabulary word, so it is still a prediction error, because it is not the reference word. Pathological case: only UNK in the vocabulary, would mean 100% accuracy?
The authors might argue that maybe for a user not predicting is less bad than bad prediction.
It would be also interesting to see how many times NTN2T+D reject to predict when it should (not UNK in the reference).
Writing e.g. assignment expression changes the AST -> changes the in-order depth-first sequence -> the whole LSTM should be rerun not just adding one node to the sequence as the author stated.
Alternatively, LSTM states can be cached in nodes of the tree, to reuse the biggest possible common partial-tree.
We will update the paper for the use of different models in the next couple of weeks.
We have updated the paper (page 4) to give an brief explanation why non-terminal is relevant. We also refer to our response to another question above.
Table 1: yes, “overall” means the total vocabulary size.
Programs < 30,000 tokens have ~5.3*10^7 queries, as we have explained in Sec 5.1.
We are trying to increase the vocabulary size. Notice that this will result in a larger model, whose size is beyond our current GPU memory capacity. We may not update any results in this issue in the near future.
But one thing we have tried is to shrink the vocabulary size, and we indeed observed the accuracy drops dramatically.
We view predicting UNK as a sign that the model predicts that the user will input a rare token, so it does not bother to show up suggestions.
In fact, we have different metrics to measure Table 6. For Top-1 and Top-5 accuracy, yes, if only UNK is in the vocabulary, then both these two cases should be 100%.
We have updated the precision and recall of +D models. We hope this can provide more insights about the frequency that NTN2T+D deny prediction when it should not.
For the performance evaluation, yes, the model should be more fine-tuned and many practical optimizations should be implemented to build a full-fledge code completion engine.
However, as we have stated in the paper, “these numbers are just a proof of concept and we do not optimize the code”.
Therefore, we think our evaluation can support our claim that our approaches may have potential practical usage.
Could you explain briefly the motivation behind considering a depth-first serialization of the AST?
Also, it isn't clear to me why a depth-first serialization results in a sequence (N_i, T_i) of interleaving non-terminal N_i and terminal T_i nodes. Wouldn't the depth-first serialization from the root result in sequence of non-terminal nodes, e.g. , interspersed with a terminal nodes, e.g.  N_1, N_2, N_3 T_1, N_4, T_2. Could you clarify? Thanks!
We consider the depth-first serialization of the AST mainly because in doing so, we can compare with the state-of-the-art work on this dataset, i.e., Bielik  et al. 2016 and Raychev et al. 2016. In both these two works, the data is provided as a serialized sequence of the AST, although the tree structure (parent-child links) is preserved.
Both these two works artificially insert an EMPTY terminal as the child of each non-terminal without a real terminal child.
Therefore we follow this procedure to process the data, so that the results can be comparable.
This paper studies the problem of source code completion using neural network models. A variety of models are presented, all of which are simple variations on LSTMs, adapted to the peculiarities of the data representation chosen (code is represented as a sequence of (nonterminal, terminal) pairs with terminals being allowed to be EMPTY).
Another minor tweak is the option to "deny prediction," which makes sense in the context of code completion in an IDE, as it's probably better to not make a prediction if the model is very unsure about what comes next.
Empirically, results show that performance is worse than previous work on predicting terminals but better at predicting nonterminals.
However, I find the split between terminals and nonterminals to be strange, and it's not clear to me what the takeaway is. Surely a simple proxy for what we care about is how often the system is going to suggest the next token that actually appears in the code. Why not compute this and report a single number to summarize the performance?
Overall the paper is OK, but it has a flavor of "we ran LSTMs on an existing dataset". The results are OK but not amazing. There are also some issues with the writing that could be improved (see below). In total, I don't think there is a big enough contribution to warrant publication at ICLR.
Detailed comments:
* I find the NT2NT model strange, in that it predicts the nonterminal and the terminal independently conditional upon the hidden state.
* The discussion of related work needs reworking.
For example, Bielik et al. does not generalize all of the works listed at the start of section 2, and the Maddison (2016) citation is wrong
We thank the reviewer for the useful comments. We have tuned the model, and re-run all experiments. The current results suggest that our approaches can beat the state-of-the-art on both non-terminal and terminal prediction tasks.
We have added a paragraph in Section 3.3 to explain why we think the next node prediction task is more meaningful than the next token prediction task.
Also, the performance of these two tasks are not directly comparable, and recent works have focused more on the node prediction task.
Thus we chose the former rather than the latter.
For NT2NT, we have added an observation in Section 5.4 to show that, even though the model seems to predict the next non-terminal and the next terminal independently (conditioned on the hidden states), our experiment results show that this is not the case.
We have revised the related work section.
Pros:
using neural network on a new domain.
Cons:
It is not clear how it is guaranteed that the network generates syntactically correct code.
Questions, comments:
How is the NT2N+NTN2T top 5 accuracy is computed? Maximizing the multiplied posterior probability of the two classifications?
Were all combinations of NT2N decision with all possible NTN2T considered?
Using UNK is obvious and should be included from the very beginning in all models, since the authors selected the size of the
lexicon, thus limited the possible predictions.
The question should then more likely be what is the optimal value of alpha for UNK.
See also my previous comment on estimating and using UNK.
Section 5.5, second paragraph, compares numbers which are not comparable.
We thank the reviewer for the useful comments. The goal of this work is to enable code completion. With this target application scenario in mind, our latest results (Section 5.3) have shown that over 96% of the time when prediction is restricted to the top 50,000 most frequent tokens in the training set, our approach is able to provide a candidate list so that users can select from the list rather than inputting manually.
Indeed, we agree with the reviewer that our approach does not guarantee the generated code is syntactically correct, but we would like to argue that 96% accuracy indicates that our approach is effective enough for our target application.
In the current version, we have removed the top5 accuracy for Table 5, since these results are lower priority. We will update these results, later, by maximizing the joint posterior probability, and only top5 non-terminals from NT2N and top5 terminals from NTN2T will be considered.
We will update the analysis about UNK tokens and alpha in one to two days, as well as Section 5.5 (which is now removed).
This paper considers the code completion problem: given partially written source code produce a distribution over the next token or sequence of tokens. This is an interesting and important problem with relevance to industry and research. The authors propose an LSTM model that sequentially generates a depth-first traversal over an AST. Not surprisingly the results improve over previous approaches with more brittle conditioning mechanisms (Bielik et al. 2016).
Still, simply augmenting previous work with LSTM-based conditioning is not enough of a contribution to justify an entire paper. Some directions that would greatly improve the contribution include: considering distinct traversal orders, does this change the predictive accuracy? Any other ways of dealing with UNK tokens? The ultimate goal of this paper is to improve code completion, and it would be great to go beyond simply neurifying previous methods.
Comments:
- Last two sentences of related work claim that other methods can only "examine a limited subset of source code". Aside from being a vague statement, it isn't accurate. The models described in Bielik et al. 2016 and Maddison & Tarlow 2014 can in principle condition on any part of the AST already generated. The difference in this work is that the LSTM can learn to condition in a flexible way that doesn't increase the complexity of the computation.
- In the denying prediction experiments, the most interesting number is the Prediction Accuracy, which is P(accurate | model doesn't predict UNK). I think it would also be interesting to see P(accurate | UNK is not ground truth). Clearly the models trained to ignore UNK losses will do worse overall, but do they do worse on non-UNK tokens?
We thank the reviewer for the useful comments. We have updated the related work section to revise the imprecise claims.
We have worked on updated analysis for UNK tokens and deny prediction experiments. We are still waiting for the results, and we will update them within one day.
Dear reviewers,
We have improved our approaches and revised the paper based on the comments. We thank all reviewers for the valuable comments, and hope our work has the chance to be discussed further!
We are still working to update the paper, and some new results will be available by tomorrow.
Here is the list of all changes as of Jan-15-2017:
1) We have fine-tuned the model to achieve better performance. Now, our approaches can achieve a better performance than prior art. We have updated the following related section:
a) Abstract & Introduction
b) Section 5.2. Training details
c) All results in Section 5.3 & 5.4. In 5.3, the baseline is updated to Raychev et al. (2016a).
d) Some results in Section 5.5 are still pending.
So this section is removed right now.
2) The related work section and items in the reference have been revised.
3) We have added a paragraph in Section 3.3 to explain the difference between next node prediction and next token prediction.
The latest revision adds the following content beyond Jan-16's version:
1) We add the results for one single models' performance (rather than an ensemble's performance) for the next non-terminal and terminal predictions in Section 5.3.
2) A new paragraph in Section 5.3 to report the effectiveness of setting the UNK threshold.
3) We add a new subsection, Section 5.5, to report the deny prediction experiments. We also add a paragraph in Sec 5.5 to examine how different choices of alpha affect a model's accuracy.

Meta-review:
role: PC
data: comment: The paper extends existing code completion methods over discrete symbols with an LSTM-based neural network. This constitutes a novel application of neural networks to this domain, but is rather incremental. Alone, I don't think this would be a bad thing as good work can be incremental and make a useful contribution, but the scores do not show an amazingly significant improvement over the baseline. There are many design choices that could be better justified empirically with the introduction of neural benchmarks or an ablation study. We encourage the authors to further refine this work and re-submit.

Decision: Reject