Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper presents a grammar-based generation approach for "slot-filling" style code generation tasks. Given a context AST with opening non-terminal node, the model completes the opening node by predicting a sequence of child nodes, which forms a sub-AST rooted at the original opening node. The proposed model encodes context ASTs using a path-based approach (Alon et al., 2019a), essentially generalizing the previous model of Alon et al., 2019a from a code-to-sequence setting (e.g., generating natural language comments from code) to a "code-to-code" setting (i.e., code completion given contextual snippets). 

Strong Points:

* The paper is very well written. The idea of formalizing code completion as structured language modeling and extending Alon et al., 2019a for the task is natural and well executed, with strong models and significantly improved results on two code completion benchmarks for both Java and C#.

* The authors attempted to establish comparisons with most existing code generation models.

Detailed Review:

*Technical Contribution* I have a very mixed feeling with this paper, while the model registers high empirical performance, the technical contribution is a bit limited, as detailed below:

    - *Path-based Context Encoding* The most important contribution in this submission is the application of path-based AST encoding model of Alon et al., 2019a to encode context (the given contextual and partially generated ASTs) for code generation. While the path-based encoding scheme is indeed a powerful model that intuitively encapsulates and generalizes over most previous approaches (Section 7), applying the model to a different task without significant task-specific adaptation or in-depth analysis might not sound technically novel. Meanwhile, the core idea of modeling/encoding the information flow in both the given context AST and partially generated programs for opening node expansion has already been explored in Brockschmidt et al. (2019a), albeit using a different encoding approach (GNNs) and in a relatively restricted setting of generating arithmetic expressions.

    - *Node-based Tree Generation Model* Apart from the path-based context encoding model, the node-based generation model presented in Section 2 also seems interesting. However, it might take longer time-steps to generate the node sequence instead of the sequence of production rules (composed of multiple child nodes), which could make optimization and inference more difficult. On the other hand, to control arity, the node-based approach need to inject synthetic "EOS" nodes to signal end of generating an argument sequence, while existing production rule-based systems could easily generate arbitrary number of argument nodes using either a transition system (e.g., Yin and Neubig, 2018) or a special neural component to compute end-of-argument-list probability (e.g., Rabinovich et al. (2017)), without using separate production rules of different arity.

    - *Syntactic Copy Mechanism* While the proposed syntactic terminal token copy mechanism (Section 3.3) could be better than the vanilla sequential one, there have already been syntactic copying models capable of copying both terminal tokens and partial ASTs from the context (Yin et al., 2019). 

    How to Improve: to better understand the different technical contributions outlined above and their relative impacts, the following ablation studies would be helpful:

        - Importance of Path-based Context Encoding: the Seq→Path ablation in Table 3 alone might not be adequate to demonstrate the importance of path-based encoding of AST contexts for code generation tasks. The authors should compare with the GNN-based context encoding approach in Brockschmidt et al. (2019a) as this is the most relevant work. The original GNN→NAG model cited in Table 2 used a much simpler copying mechanism and a vanilla production-based tree generation model, and therefore not directly comparable with a tuned SLM.

        - Importance of Node-based Tree Generation Model: If possible, the authors might consider swapping their node-based tree generation model with a state-of-the-art production-based approach (e.g., Yin et al., 2019) to demonstrate its effectiveness.

*Claims* The authors claimed in the beginning of the paper that previous program synthesis approaches are either restricted in domains (e.g., DSLs like SQL) or grammars (e.g., restricted grammar of the full language), therefore coining the proposed approach as "any-code generation". However, there are indeed code generation systems (some of them cited in this paper) that synthesize open-domain code snippets in general-porpuse programming languages without restriction on vocabulary or grammar. To give a few examples, Iyer et al. (2018) generate open-domain Java class member functions; Rabinovich et al. (2017) and Zhao et al. (2019) predict full Python classes or partial snippets, while Yin et al. (2019) synthesize open-domain short C# code diffs observed in GitHub commits. In fact, the the proposed "any-code generation" benchmark is limited to sub-expressions defined within a function, whose scope is more restricted than other benchmarks like CONCODE. 

    How to improve: the authors might present more evidence to substantiate the their claim on the novelty of the AnyGen benchmark compared with existing open-domain, general purpose code synthesis benchmarks, or consider revising the claim and the title.

References:

* Zhao et al., Neural Networks for Modeling Source Code Edits. 2019
* Yin et al., Learning to Represent Edits. 2019

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper proposes a generative task for programming code where an expression from the program is generated given the rest of the program (minus the expression). This is in line with language modeling for natural language. The proposed method generates the AST corresponding to the program by generating one node at the time for the missing/to-be-generated expression by approximating the probabilities of the generated notes. Again, this is similar to the prediction of words in language models.

The method takes into account the AST corresponding to the program. However, when representing the program, the structure of the AST is not preserved, instead, the AST is represented by generating several sequential paths by traversing paths between connected nodes in the tree.

It would be nice if the paper provided some intuition why generating such connecting paths in the tree are relevant for representing the code, specially for nodes that do not have a direct relationship between them (e.g., the nodes are distant enough in the code that the corresponding probabilities of their nodes do not seem/appear related).

The paper presents results for 2 datasets (comparing with various related work methods). The results for the Java dataset improve state of the art by 1-2%, while the results for the restricted C# dataset show a much more significant improvement (in the order of 10-15% improvement, depending on the metric).

I would have liked to see a qualitative analysis of the results. In particular, I would have liked to understand how the predictions differ between acc and tree metrics. In other words, when the prediction looking at the tree structure is correct and the overall prediction is not, what goes wrong?

It was not clear to me why or if all the paths between 2 nodes are necessary when encoding the partial AST and predicting the missing nodes. I was not convinced that the ablation studies were relevant. I would have liked to see ablation studies that considered a subset of the paths in the graph.

The elimination of the methods with more than 20 lines of code seems ad-hoc to me and biases the evaluation with relatively short methods (how many methods were eliminated this way?).

One thing that I struggle with is understanding how useful the proposed task is and how it can be generalized/used in practice for some relevant higher level task in AI4code.

review_title: Official Blind Review #549

final_score: rating: 6: Weak Accept
review_content: The paper proposes a model to address the Any-Code Generation (AnyGen) task, which basically to fill missing code from a given program. The model makes use of partial Abstract Syntax Tree (AST) as input. The model learns representation for partial AST paths and use the learnt representation to generate AST node at masked steps. The conducted experiments show that using AST paths from root and leaves are good for AST node generation, but whether those inputs are robust and sufficient should be further explored.
 
There are some restrictions to the method, for example,  the input is only a single function, and the missing expression is not that complex. Nevertheless this work presents a novel method towards code generation. The paper also introduces a new metric to evaluate the prediction accuracy for generated expressions. Writing is clear. Evaluation is fairly comprehensive.

Questions:
1. Did the author test the method without the camel notation assumption, i.e. the data contains non-camel notation or mixed notations?
2. In the Restrict Code Generation test, it seems that the author filters out non-primitive types and user-defined functions. Therefore, does the experiment on Java-small dataset fully show the proposed model’s strength?
3. Can the author explain why the SLM model fail on Figure 6? Is it because of dividing token into sub tokens?
4. How big is the token vocabulary? How does the vocab size affect the performance?

review_title: Official Blind Review #2

Meta-review: This paper proposes a new method for code generation based on structured language models.

After viewing the paper, reviews, and author response my assessment is that I basically agree with Reviewer 4. (Now, after revision) This work seems to be (1) a bit incremental over other works such as Brockschmidt et al. (2019), and (2) a bit of a niche topic for ICLR. At the same time it has (3) good engineering effort resulting in good scores, and (4) relatively detailed conceptual comparison with other work in the area. Also, (5) the title of "Structural Language Models for Code Generation" is clearly over-claiming the contribution of the work -- as cited in the paper there are many language models, unconditional or conditional, that have been used in code generation in the past. In order to be accurate, the title would need to be modified to something that more accurately describes the (somewhat limited) contribution of the work.

In general, I found this paper borderline. ICLR, as you know is quite competitive so while this is a reasonably good contribution, I'm not sure whether it checks the box of either high quality or high general interest to warrant acceptance. Because of this, I'm not recommending it for acceptance at this time, but definitely encourage the authors to continue to polish for submission to a different venue (perhaps a domain conference that would be more focused on the underlying task of code generation?)
Decision: Reject