Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The system generates long code snippets such as method bodies given their surrounding context. In contrast to modern
code generation models that are supervised by program syntax, it additionally conditions generation on semantic
attributes that are computed on AST nodes by the compiler. The paper shows that this approach generate significantly
more idiomatic code with fewer language errors than syntax-only Transformers or purely-neural neural attribute models.
main_review: 
Strengths:

+ A principled formalism and implementation that incorporates semantic information computed by the compiler into neural
  code generation models. Prior work usually either ignores it (syntactic codegen, either for sequences or for ASTs),
  hard-filters by it at inference time, or incorporates specific instances, like data flow in GNN-based models. Adopting
  it into the well-known formalism of attribute grammars is appreciated.
+ Clear motivating examples and well-explained decision rationale
+ Fair comparison to two most relevant baselines, one syntax-only and one attribute-inspired but not explicitly
  supervised with symbolic attributes.
+ Evaluation along multiple important dimensions: method generation accuracy, small expression generation accuracy (as
  in prior work), and individual language rules.
+ An explicit discussion of limitations is welcome, not discussed often enough in our community.

Weaknesses:

- The paper spends significant space on the NSG formalism and on the particular generative model re-used from Murali et
  al., which forces it to defer its instantiation for Java, choice of attributes, and their encoding to the Appendix.
  As a result, Section 4 ends abruptly, confusing the reader. The "design" paragraph helps set up the scene but, I
  imagine, would not actually be informative to someone without formal methods background, at least without a single
  concrete example.
- Similarly, lack of space likely forced the authors to leave out Appendix H. It provides a valuable complementary
  perspective on the respective strengths and weaknesses of training approaches, and a quick summary of it would be
  welcome in the paper.
- Characterizing [7] as "next token prediction" is unfair, as that model generates whole ASTs and – just like this
  work – supervised with its production rule decisions at every step. The difference is in (a) size of the target AST,
  (b) the amount of available in-method context for conditioning and for copy.
- Phrasing aside, it would be interesting to see how [7] fairs against NSG for _in-method_ expression completion (medium
  horizon). I imagine NSG would still outperform, as it leverages more explicit conditioning on attributes rather than
  their neural abstractions.

Questions & Remarks:

- The general line of reasoning that "semantics are hard to learn from syntax" is not wrong, but:
  - It largely depends on the size of the model and its pretraining data, with improvement following a power law.
  - It further improves with multi-language training as the model pick ups non-formalizable yet similarly expressed
    cross-language cues.  
  
  Both were shown empirically by [Codex](https://arxiv.org/abs/2107.03374), concurrently with this submission. This does
  not diminish the paper's contribution, just shows that the two directions keep improving and could further benefit
  each other.

- A better Transformer baseline would be [GPT-NEO](https://github.com/EleutherAI/gpt-neo) or
  [GPT-J](https://github.com/kingoflolz/mesh-transformer-jax/#gpt-j-6b). Both are pretrained on The Pile corpus, which
  includes a big portion of Github and more natural language. The former was shown to be a strong baseline for code
  generation against Codex and in the [APPS paper](https://arxiv.org/abs/2105.09938), with or without finetuning.

- L404-405: There are many methods to integrate tree structure and generative Transformers. See e.g.:
  - Scholak et al. "DuoRAT: Towards Simpler Text-to-SQL Models", NAACL 2021.
  - Sun et al. "TreeGen: A Tree-Based Transformer Architecture for Code Generation", AAAI 2020.

- How much of attribute vocabulary could be shared cross-language? Most popular imperative languages could implement a
  compiler frontend that computes a version of the attributes in Appendix D. Could this help us effectively train
  cross-language and cross the gap to language model training?

- Is the NSG model robust to presence or absence of attributes? For example, suppose the code/linker is in a broken
  state so that type-based attributes cannot be properly resolved. Does the model break down completely or can it still
  rely on other features?

- Follow-up to previous question: In addition to conditioning on attributes, the model could be trained to predict them
  as an auxiliary task. Multitask learning has improved performance or accelerated training in many domains, code could
  work just as well.

Typo:
- L47: "whose with grammar"
limitations_and_societal_impact: Limitations are explicitly and nicely discussed in the Conclusion.

On societal impact, the authors state "We have proposed an approach to better generation of source code and used open-source programs as training data. We do not anticipate any direct negative societal impact of this work."  
As a counter-point, I recommend reading the thorough and elaborate discussion of possible impacts of such models in the very recent "Evaluating Large Language Models Trained on Code" (Chen et al., 2021).
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a novel approach to conditional program generation that leverages supervision from a static analysis tool during generation. This serves to address a common difficulty in program synthesis using language models, that of generating code without semantic errors. The program generation and static analysis both follow the linear information flow through a program defined by an attribute grammar, with the program generation model having access to the attributes from the analysis. The paper applies this technique to a dataset of 1 million Java programs, comparing primarily with baselines CodeGPT and GNN2NAG. Evaluating both on static checks, overall fidelity metrics, and exact match, the paper finds the proposed NSG method outperforms the baselines considered on nearly all metrics.
main_review: The paper considers the challenging and important task of conditional program generation, an area of increasing interest at NeurIPS in recent years. The main weakness identified, that neural models of source code usually struggle with generation of long sections of code, is important. Being able to generate long semantically coherent programs is a key challenge in the field, and this paper presents an approach to program generation that deals with this effectively.

The method is explained clearly overall, providing background on static analysis with attribute grammars and a clear explanation of the algorithm for using such a static analysis during program generation, compatible with a wide class of models. The appendix provides the details that were omitted from the main paper, including how the training data is prepared, the attributes considered and part of the definition of the attribute grammar, how attributes are encoded, and details about the main and baseline methods.

A key element of the approach is the attribute grammar defines a linear path through the program, allowing the model to be trained using pre-computed attributes and requiring on-the-fly computation of attributes during inference. This is explained best in the training and inference appendix.

A strength of the approach, and an advantage over prior work, is that the method allows for learning constraints and guiding principles from the input corpus, rather than using the static analysis to provide hard constraints. This gives more freedom to the static analysis designer to include attributes that correspond to best-practices or that appear only in certain contexts, without needing to avoid attributes that don't define hard-and-fast rules about programs. Prior work leveraging attribute grammars for program generation used them to enforce hard constraints.

A weakness is that the evaluation static checks match those properties provided by the static analysis attribute grammar. I would be interested to see an evaluation indicating whether the method learns to preserve additional properties beyond those directly measured by the static analysis. Perhaps to that end, please clarify what is meant by 50% evidence. Does this refer to full evidence at 50% of nodes, or 50% evidence at each node? How is the 50% selected? Outside of these concerns, the evaluation is solid. The task and baselines are appropriate, the metrics capture properties of interest, and the evidence from the results tells a quite clear story. Including a measure of spread would improve the results section.

---

Line 71: $\delta$ accepts programs as inputs, but in the prediction error expression $\textbf{E}_{(X,Y)\sim D}[\delta(P_\theta(X|Y), Y)]$ you incorrectly pass $P_\theta(X|Y)$, rather than a program, as the first input to $\delta$.

Nit: Line 165 refers to specification Y rather than specification X.

Line 765 has a broken reference simply appearing as "??".
limitations_and_societal_impact: The authors discuss the main technical limitation of the proposed approach, that the implementation is language-specific, whereas language model baselines are generally language agnostic. The paper does not address (nor, per the checklist, anticipate) potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a new model for automatic code completion of full methods
(rather than simply single lines).  The main innovation is the use of a static
analysis tool which can analyze partially generated programs.  The program is
generated hierarchically from an attribute grammar using a model they call a
Neurosymbolic Attribute Grammer (NSG).  At each step the model has access to:
- The context code surrounding the code to be generated
- All the tree nodes which have been generated so far
- The attributes generated by running the static analyzer over the partially
completed program

Their hypothesis is that the kind of information that the static analyzer
generates is difficult for a neural network model to learn by simply training on
a large dataset of programs.  They show that their model outperforms both a
smallish transformer baseline (~100M parameters) and a Graph Neural Network
baseline. 

main_review: Strengths:
- The paper is well written and relatively easy to understand.
- Their model *significantly* outperforms all of the baseline models on both
their static analysis metrics as well as their whole code metrics.
Specifically, the transformer baseline CodeGPT only generates correct API calls
for 4.7% of test samples while their method generates correct API calls for 43.7% of test samples.
- Tackles the important and immediately applicable task of conditional code
generation.  
- The idea of using on-line static analysis to aid neural models seems very
generally applicable to almost any conditional code generation model, and their
results show that it can make a significant difference.


Weaknesses:
- They only test on relatively small transformer models.  Recent work has
shown that scaling the size of such transformer models significantly improves
their performance.  So it's possible that the gains from the static
analysis will disappear once the transformer models are big enough.  While I don't expect them to scale to GPT-3 results it would have been good to see some analysis of how the performance gains change as the Transformer model size increases.
- As far as I can tell their model is trained on canonicalized versions of
the code where all variable, method and class names are changed to generic names
(i.e. var_0, var_1, etc.), which prevents the model from taking advantage of the
soft information provided in such naming in source code corpora.
- Most of their whole method metrics are AST based metrics which are biased
toward AST base models like theirs rather than sequential metrics like BLEU
which would be biased towards sequential models like CodeGPT (the transformer
baseline).  I understand that code is inherently hierarchical and the AST
metrics may better represent this, but I would have liked to see results for
BLEU score as well (which should be quite easy to compute).
- Their evaluation setup does not include test cases so they cannot directly
execute the code to tell whether the code they generate is semantically correct
(or at least semantically matches the code in the dataset).

Questions:
- Was the context size of 512 tokens that you used for CodeGPT large enough
to include both the relevant specification (surrounding code) as well as the
generated method body for most samples in your training and test sets? What
fraction of samples were truncated in the training and test sets?
- What were you trying to show with the 50% evidence visible setting?  This
doesn't seem like a realistic setting and so I wasn't sure why you included
these results.
limitations_and_societal_impact: I would have liked to see the author acknowledge the fact that their baseline model was a relatively small transformer and that larger transformers may significantly reduce/eliminate the gains provided by their methods.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: Leveraging compiler inferences about code, such as from static analyzers, can improve the ability of neural models to reason about code. This paper introduces a method that relies on attribute grammar coupled with static analysis that operates on partial programs, demonstrating that access to this information greatly improves the quality of generated programs.
main_review: *Note: I appreciate the authors' response; they did not impact my score because I already strongly support acceptance.*

This is a strong paper. It is well-written and easy to follow while introducing a method that bridges several ideas and performs well. The resulting modeling delta compared to previous models is simple enough that it ought to be more broadly useful, although, as the authors point out as well, the need to implement static analyzers is a bit of a hurdle. Given that, my review mostly concerns minor issues and notes.

In terms of evaluation, while the baselines seem adequate, it is a slightly questionable choice to apply CodeGPT in this way -- presenting entire files while removing the method's body. Besides being somewhat removed from its pretraining objective (which the finetuning presumably helped overcome), transformers typically don't scale well to larger inputs. Indeed, the appendix suggests that this model was trained with a sequence length of up to 512 tokens, which tends to be (far) less than the size of typical files. Please clarify how this impacted the model.

Presentation:
- L147: consider splitting out the definition of mu and sigma from this equation, to make it more readable.
- The citation style used in the paper appears to be incorrect for NeurIPS; it uses the numerical references rather than surname-of-first-author.
- Table 1: consider adding a "totals" row, to make it easier to assess the overall performance of these models. Also consider better emphasizing the "intermediate" totals (row 5 & 10).

Typos:
- L47: "and whose with grammar"
- L390: missing semicolon or "and" after "beam search"

limitations_and_societal_impact: The authors have adequately addresses both.
review_title: null

Meta-review: The paper is well written and clearly motivated, with good results outperforming the most relevant baselines.  All reviewers agree this is a good paper and should be accepted.

There are, however, some comments shared by the reviewers that the authors should take into account to improve this work.  Most notably the transformer language model baseline chosen for comparison may have been a bit weak given the recent progress in this direction.
Decision: Accept (Spotlight)