Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: While the overall direction is promising, there are several serious issues with the paper which affect the novelty and validity of the results:

1. Incorrect claims about related work affecting novelty:

  - This work is not the first to explore a deep learning approach to automatic code completion: “Toward Deep Learning Software Repositories”, MSR’15 also uses deep learning for code completion, and is not cited.

  - “Code Completion with Statistical Language Models”, PLDI’14 is cited incorrectly -- it also does code completion with recurrent neural networks.

  - PHOG is independent of JavaScript -- it does representation learning and has been applied to other languages (e.g., Python, see OOPSLA’16 below). 

  - This submission is not the only one that “can automatically extract features”. Some high-precision (cited) baselines do it.

  - “Structured generative models of natural source code” is an incorrect citation. It is from ICML’14 and has more authors. It is also a log-linear model and conditions on more context than claimed in this submission.


2. Uses a non-comparable prediction task for non-terminal symbols: The type of prediction made here is simpler than the one used in PHOG and state-of-the-art (see OOPSLA’16 paper below) and thus the claimed 11 point improvement is not substantiated. In particular, in JavaScript there are 44 types of nodes. However, a PHOG and OOPSLA’16 predictions considers not only these 44 types, but also whether there are right siblings and children of a node. This is necessary for predicting tree fragments instead of a sequence of nodes. It however makes the prediction harder than the one considered here (it leads to 150+ labels, a >3x increase).


3. Not comparing to state-of-the-art: the state-of-the-art however is not the basic PHOG cited here, but “Probabilistic Model for Code with Decision Trees”, (OOPSLA 2016) which appeared before the submission deadline for ICLR’17:

http://dl.acm.org/citation.cfm?id=2984041

On the same dataset, OOPSLA’16 has accuracy of 83.9%, and on the more difficult task than considered here (see above point). Further, state-of-the-art (OOPSLA’16) has 2% higher accuracy than PHOG for tree values, which makes state of the art better than the current sub by 5%.


The work requires a deep revision pass w.r.t to novelty and precision claims which are currently incorrect.

review_title: While the overall direction is promising, there are several serious issues with the paper which affect the novelty and validity of the results.

final_score: rating: null
review_content: Could the authors give examples, code snippets, which represents the possible use of the various predictors (NT2N, NT2NT, NTN2T)?
Why is it useful to measure the non-terminal prediction accuracy? From the user viewpoint only (visible) terminal prediction accuracy makes sense.

Table 1.: "Overall": should that mean the vocabulary size?
How many queries are used when not all queries are used (programs < 30000 tokens)?

Have the authors tried to use bigger lexicon > 50k? To support the argument to the result in table 3.

Denying prediction: have the authors tried to use 0<\alpha<1 for y_i = UNK, to down weight the possibility of UNK?
Predicting UNK in the test set is not positive results, UNK means out-of-vocabulary word, so it is still a prediction error, because it is not the reference word. Pathological case: only UNK in the vocabulary, would mean 100% accuracy?
The authors might argue that maybe for a user not predicting is less bad than bad prediction.
It would be also interesting to see how many times NTN2T+D reject to predict when it should (not UNK in the reference).

Writing e.g. assignment expression changes the AST -> changes the in-order depth-first sequence -> the whole LSTM should be rerun not just adding one node to the sequence as the author stated.
Alternatively, LSTM states can be cached in nodes of the tree, to reuse the biggest possible common partial-tree.



review_title: questions

final_score: rating: null
review_content: Could you explain briefly the motivation behind considering a depth-first serialization of the AST? Also, it isn't clear to me why a depth-first serialization results in a sequence (N_i, T_i) of interleaving non-terminal N_i and terminal T_i nodes. Wouldn't the depth-first serialization from the root result in sequence of non-terminal nodes, e.g. , interspersed with a terminal nodes, e.g.  N_1, N_2, N_3 T_1, N_4, T_2. Could you clarify? Thanks! 
review_title: Clarifications

final_score: rating: 5: Marginally below acceptance threshold
review_content: Pros:
  using neural network on a new domain.
Cons:
  It is not clear how it is guaranteed that the network generates syntactically correct code.

Questions, comments:
  How is the NT2N+NTN2T top 5 accuracy is computed? Maximizing the multiplied posterior probability of the two classifications?
  Were all combinations of NT2N decision with all possible NTN2T considered?

  Using UNK is obvious and should be included from the very beginning in all models, since the authors selected the size of the
  lexicon, thus limited the possible predictions.
  The question should then more likely be what is the optimal value of alpha for UNK.
  See also my previous comment on estimating and using UNK.

  Section 5.5, second paragraph, compares numbers which are not comparable.


review_title: An interesting paper but only initial work about neural network based code completion.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper considers the code completion problem: given partially written source code produce a distribution over the next token or sequence of tokens. This is an interesting and important problem with relevance to industry and research. The authors propose an LSTM model that sequentially generates a depth-first traversal over an AST. Not surprisingly the results improve over previous approaches with more brittle conditioning mechanisms (Bielik et al. 2016). Still, simply augmenting previous work with LSTM-based conditioning is not enough of a contribution to justify an entire paper. Some directions that would greatly improve the contribution include: considering distinct traversal orders, does this change the predictive accuracy? Any other ways of dealing with UNK tokens? The ultimate goal of this paper is to improve code completion, and it would be great to go beyond simply neurifying previous methods.

Comments:

- Last two sentences of related work claim that other methods can only "examine a limited subset of source code". Aside from being a vague statement, it isn't accurate. The models described in Bielik et al. 2016 and Maddison & Tarlow 2014 can in principle condition on any part of the AST already generated. The difference in this work is that the LSTM can learn to condition in a flexible way that doesn't increase the complexity of the computation.

- In the denying prediction experiments, the most interesting number is the Prediction Accuracy, which is P(accurate | model doesn't predict UNK). I think it would also be interesting to see P(accurate | UNK is not ground truth). Clearly the models trained to ignore UNK losses will do worse overall, but do they do worse on non-UNK tokens?
review_title: Great problem, too many decisions taken for granted and not explored

final_score: rating: null
review_content: A few questions:

- It seems a strange to have all the EMPTY terminal tokens in the encoding described in section 4. What is the justification for using the described encoding of trees instead of an encoding like in Grammar as a Foreign Language [1]?

- What is the point of predicting non-terminals in a code completion context? To make code completion suggestions, terminals need to be predicted, right?

- Can you elaborate on the training details in 5.2? Specifically, I didn't understand why it makes sense to only reset hidden states at the beginning of an epoch.


[1] Vinyals, Oriol, et al. "Grammar as a foreign language." Advances in Neural Information Processing Systems. 2015.
review_title: Questions

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper studies the problem of source code completion using neural network models. A variety of models are presented, all of which are simple variations on LSTMs, adapted to the peculiarities of the data representation chosen (code is represented as a sequence of (nonterminal, terminal) pairs with terminals being allowed to be EMPTY). Another minor tweak is the option to "deny prediction," which makes sense in the context of code completion in an IDE, as it's probably better to not make a prediction if the model is very unsure about what comes next.

Empirically, results show that performance is worse than previous work on predicting terminals but better at predicting nonterminals. However, I find the split between terminals and nonterminals to be strange, and it's not clear to me what the takeaway is. Surely a simple proxy for what we care about is how often the system is going to suggest the next token that actually appears in the code. Why not compute this and report a single number to summarize the performance?

Overall the paper is OK, but it has a flavor of "we ran LSTMs on an existing dataset". The results are OK but not amazing. There are also some issues with the writing that could be improved (see below). In total, I don't think there is a big enough contribution to warrant publication at ICLR.

Detailed comments:

* I find the NT2NT model strange, in that it predicts the nonterminal and the terminal independently conditional upon the hidden state.

* The discussion of related work needs reworking. For example, Bielik et al. does not generalize all of the works listed at the start of section 2, and the Maddison (2016) citation is wrong

review_title: Ok paper, but not a big enough contribution

Meta-review: The paper extends existing code completion methods over discrete symbols with an LSTM-based neural network. This constitutes a novel application of neural networks to this domain, but is rather incremental. Alone, I don't think this would be a bad thing as good work can be incremental and make a useful contribution, but the scores do not show an amazingly significant improvement over the baseline. There are many design choices that could be better justified empirically with the introduction of neural benchmarks or an ablation study. We encourage the authors to further refine this work and re-submit.
Decision: Reject