Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper addresses the problem of neural program synthesis, whose aim is to train a neural network to synthesize programs from problem specifications. To this end, the paper proposes a framework that leverages large-scale pre-trained language models and directly optimizes the functional correctness of generated programs using reinforcement learning. The experiments show that the propsoed framework outperforms baselines on the APPS benchmark and demonstrates reasonable generalization on the MBPP benchmark. Ablation studies justify the effectiveness of the design choices and the proposed techniques. I believe this work explores a promising research direction and proposes a reasonable framework to address the problem. Yet, I am concerned with insufficient novelty and seemingly inconclusive experimental results.
strengths_and_weaknesses: ## Paper strengths and contributions
**Motivation and technical contribution**
- The motivation for leveraging pre-trained large language models and RL to improve program synthesis performance is convincing and this paper proposes a reasonable framework to implement this idea.

**Clarity**
The overall writing is clear. The authors utilize figures well to illustrate the ideas.

**Ablation study**
- Ablation studies are comprehensive. The proposed framework consists of multiple components and the provided ablation studies help analyze the effectiveness of each of them, including the cross entropy loss and the RL loss, the design of return estimate, the choice of pre-training approaches, and the critic sampling.

**Experimental results**
- The presentation of the experimental results is clear.
- The experimental results show that the proposed framework outperforms the baselines in terms of both $pass@k$ and $n@k$ on APPS.

**Reproducibility**
Given the clear description in the main paper and the details provided in the appendix, I believe reproducing the results is possible. 

## Paper weaknesses and questions

**Citation format**
The citation format does seem right. Citations in NeurIPS papers are referred to by numbers, not the first authors' last name.

**Novelty & contribution**
Overall, I do not find enough novelty in any aspect while the overall effort of this paper is appreciated. Specifically, 
- Using RL for improving program synthesizers' performance has been studied in "Bridging reinforcement learning and maximum marginal likelihood", "Leveraging grammar and reinforcement learning for neural program synthesis", etc. Adapting this to generating Python code does not seem to be particularly novel.
- Leveraging unit tests and the results to filter and cluster generated programs has been proposed in "Competition-level code generation with AlphaCode". 

**Related work**
The descriptions of the related work are not comprehensive. Several important prior works in neural program synthesis are omitted from the paper, including "Neural Scene De-rendering", "Neural Program Synthesis from Diverse Demonstration Videos", "Learning to Describe Scenes with Programs", "Learning to Infer and Execute 3D Shape Programs", etc.

**Two critics**
The paper mentions that two separate critics are used. I am curious about the intuition behind it and am interested to know whether this contributes to the performance gain.

**CodeContests dataset**
It would be more informative if the authors can also evaluate their proposed framework on the CodeContests dataset, which can be found on [Github](https://github.com/deepmind/code_contests). It would be easier to compare the proposed framework to AlphaCode.

**Low accuracy**
The reported accuracy is rather low (mostly < 10\%) for all the methods. It would make sense to increase $k$ of $pass @ k$ and increase both $n$ and $k$ of $n @ k$.

**Standard deviation of accuracy**
The reported accuracy is rather low for all the methods. In this case, I believe showing the standard deviation of accuracy would be important to compare different methods.

**Comparison to AlphaCode**
AlphaCode paper reports the need for millions of program samples to achieve satifactory performance. It would make sense to increase $k$ of $pass @ k$ and see if the conclusion holds.
questions: Described in the Strengths And Weaknesses section.

limitations: Described in the Strengths And Weaknesses section.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work uses an existing pre-trained language model as an actor that synthesizes code to train a critic based on the results of unit tests.
The actor generates samples and the critic evaluates if unit tests are correct. The paper is well-written.
strengths_and_weaknesses: Strengths:
1. Using unit tests and their results as a reward signal in this fashion is a first.
2. Working with pre-trained models, without retraining, makes the approach independent of the model and usable.
3. The results present an efficient generation process, with an order magnitude fewer programs generated, compared with AlphaCode, 1000 generations instead of 50,000 for reaching a similar result.
4. The experiments reduce compiler errors, push the errors down the pipeline, and increase runtime errors, and breakdown of errors.

Weaknesses:
1. Missing comparison with using unit tests in few-shot learning.
2. Missing comparison with CodeRL component on top of other baselines.
3. Missing comparison with of state-of-the-art model baseline: Incoder, CodeGen, and GPT-NeoX.
4. No results on Codeforces data, though discussed in the introduction.
5. Only Python errors are reported in the results, missing errors in other languages.
questions: 1. Why not use unit tests as examples in the prompt or in few-shot learning?
2. How sensitive is the work to the reward values?
3. Which Codex version is used and with what hyperparameters?
4. Why not compare the same actor-critic formulation using different language models?

limitations: The authors have adequately addressed the limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work builds an actor-critic RL system on top of a pretrained language model for code. The critic is finetuned from unit test execution results of programs sampled from the language model. It's trained to estimate eventual outcome (execution correctness or any errors) at each location of the partial sampled program. The resulting system, when finetuned on the competition dataset APPS, outperforms all known finetuned and few-shot baselines of different sizes. Its performance also transfers zero-shot to a different benchmark MBPP, although with a high degree of potential overlap with finetuning data.
strengths_and_weaknesses: Strength:
- Great idea and execution. The novelty of actor-critic setup is limited but its application to execution-guided code generation for general-purpose programming languages is, as far as I know, the first of its kind.
- Well-conducted execution with multiple metrics and comparable baselines.
- Currently the best system for competitive code generation for both precision (n@k) and recall (pass@k) metrics, at smaller model scale than LM-only competitors.

Weaknesses:
- The effect of scaling is not studied as comprehensively as for baseline language-model only systems. The only two actors evaluated are CodeT5 @ 770M and GPT-J @ 6B, and it's unclear whether the signal learned by CodeRL critic transfers across models and/or model scales. Since training CodeRL is rather resource-intensive, demonstrating that a single critic model could be trained and applied to other generative LMs with potentially different error profiles would be valuable.
- The "Critic Sampling" procedure is rather ad-hoc, and adds a substantial amount of computational overhead for a limited benefit. Its impact is only demonstrated at larger sample sizes (200 or 1000 in Table 4) and adds up to 1.5 points. No ablation data is given for it for the main metrics of Table 1.
- The transfer results of APPS to zero-shot MBPP show an interesting learning (namely, that learning from competition problems of APPS can transfer to an unrelated dataset without finetuning) but the overlap results in Appendix C.2 question this learning.  
Language models are robust to syntactic differences (e.g., internal whitespace, parentheses, semantically equivalent rewrites, and variable names). Even ignoring this, the only-whitespace normalized overlap in Figure 2 of C.2 has 12% of MBPP with more than half of their lines appearing in training data of APPS.  
Hypothetically, if (a) these programs were spread uniformly across MBPP problems, and (b) these 12% problems were the ones newly solved by CodeRL, that would account entirely for the difference between "Raw CodeT5 finetuned" and "CodeT5+CodeRL zero-shot" in Table 4. In practice, these hypotheticals are likely not as severe, but they must both be measured in order to properly appreciate the MBPP transfer result.
questions: 1. Please address the MBPP/APPS overlap described above through a different lens: for a given pass@k metric, among the correct programs produced by CodeRL, which fraction of lines occurs in the training set of APPS?
2. What's the impact of critic sampling on pass@1 and pass@5 metrics?
3. Does the critic learn to correctly predict erroneous locations in wrong programs? Its max-pool mechanism and ability to predict outcomes for partial programs suggests a possibility to interpret the critic as an error localization model. Can you plot how the critic's predictions change over the course of the program's length, and when do they start matching the ground truth?
4. How well does a critic trained on a single model's predictions transfer to other models?
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper addresses the task of synthesizing programs from specifications consisting of natural language and unit tests. It combines unsupervised pre-training on large code datasets crawled from GitHub, supervised fine-tuning (imitation learning) on a dataset of natural language specifications and ground-truth programs, then reinforcement learning using the unit tests from the same dataset. The authors evaluate on the APPS dataset using various baselines and report state-of-the-art results. 
strengths_and_weaknesses: # Originality
Using reinforcement learning for program synthesis is not novel in itself, but as the paper mentions, past works have used domain-specific and toy languages rather than general-purpose languages like Python. This work also differs from others using RL for sequence generation in that it does not rely on smoother measures of quality like BLEU score to have a shaped reward function.

Most of the techniques used in the paper seem to originate from other work, but I think the authors used them competently to achieve an interesting result.

# Quality
The authors report state-of-the-art results on the APPS dataset, which was released in 2021 and contains coding problems of various difficulty. They also show impressive results on the MBPP benchmark which is also a Python dataset of programming problems like APPS, without specific training for the MBPP benchmark, 63.0% pass@80 accuracy, exceeding a much larger 137B-parameter model's 61.4% (which was the best result at the time of the paper submission; since then, [PaLM-Coder](https://arxiv.org/abs/2204.02311) reports 80-85% pass@k for two other, large models). 

One weakness of the work is that they only evaluated using one base, pretrained model (an updated version of CodeT5) and using one fine-tuning dataset (APPS). I think the work would have been stronger if they showed the applicability of the method on more models and on more datasets, but the existing result is enough.

# Clarity
Overall I felt that the paper is clear and well-explained. My main confusion was regarding Critic Sampling, which is one of the main methodologically novel parts of the work. For example, it was not clear how many rounds the sampling approach should run. The mechanism can be explained using pseudocode to increase clarity.

# Significance
The growing scale of transformer models and their pretraining datasets has led to significant increases in their ability to synthesize programs in general-purpose languages, but improvements from larger scale alone are not sustainable. Works like this suggest other directions to improve performance further.

Unfortunately, it was not entirely clear to me from the ablation studies about how much of the improvement should be credited to the use of RL versus the use of critic sampling. The ablation studies on RL versus critic sampling are done on different metrics (pass@1, pass@5 versus pass@200, pass@1000, 1@1000). The significance of the work would be clearer otherwise.

questions: - Why was the critic trained using max pooling over the outputs of all tokens and then used by taking each token's output individually (before pooling)? Aren't there more principled methods to estimate the intermediate returns (i.e. the value function)?
- How many rounds were done for the critic-guided resampling? Which samples are truncated, dropped entirely, or kept unmodified for the final set?
- If the examples in the training set are too hard for the initial model trained with imitation learning (e.g. almost no samples from the initial model pass all unit tests), the reward will always be negative and the reinforcement learning will not work. Can we quantify how much the distribution of different difficulties in the training set helped? For example, what would happen if the model was only trained on competition-level problems?
- It may not always be convenient to have unit tests available as part of the specification. How well would the critic sampling approach work without unit tests for the test-time examples and only using the information from the critic, and how might that affect the results on the pass@k and 1@k metrics?
limitations: I thought the description of societal/broader impacts was sufficient.

I didn't find a description of how long the policy gradient was run and how much computation was used for the RL step. There is also no information on how different random seeds may affect the fine-tuning or reinforcement learning part (I understand that it would be too expensive to repeat the pre-training multiple times). I hope the authors can provide such information in future revisions of the paper.
review_title: null

Meta-review: metareview: All reviewers appreciated the overall idea of using reinforcement learning to generate code from language models leveraging the unit tests and critic scores, especially for real-world languages as opposed to most prior work on using RL for program synthesis that generate code in domain-specific languages. The evaluation with extensive ablations was also greatly appreciated. The reviews have lots of suggestions for improving the presentation of the paper and also additional comparisons, which hopefully the authors can incorporate to improve the paper even further.
confidence: Certain
Decision: Accept