CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
Authors: Hung Le
Yue Wang
Akhilesh Deepak Gotmare
Silvio Savarese
Steven Hoi
Venue: NeurIPS 2022 Accept (2022)
Keywords: code generation
deep reinforcement learning
pretrained language model
program synthesis

Abstract: Program synthesis or code generation aims to generate a program that satisfies a problem specification. Recent approaches using large-scale pretrained language models (LMs) have shown promising results, yet they have some critical limitations. In particular, they often follow a standard supervised fine-tuning procedure to train a code generation model from natural language problem descriptions and ground-truth programs only. Such paradigm largely ignores some important but potentially useful signals in the problem specification such as unit tests, which thus results in poor performance when solving complex unseen coding tasks. We propose “CodeRL” to address the limitations, a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning (RL). Specifically, during training, we treat the code-generating LM as an actor network, and introduce a critic network that is trained to predict the functional correctness of generated programs and provide dense feedback signals to the actor. During inference, we introduce a new generation procedure with a critical sampling strategy that allows a model to automatically regenerate programs based on feedback from example unit tests and critic scores. For the model backbones, we extended the encoder-decoder architecture of CodeT5 with enhanced learning objectives, larger model sizes, and better pretraining data. Our method not only achieves new SOTA results on the challenging APPS benchmark, but also shows strong zero-shot transfer capability with new SOTA results on the simpler MBPP benchmark.

CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
Hung Le, Yue Wang' Akhilesh Deepak Gotmare, Silvio Savarese, Steven C.H. Hoi
Salesforce Research
Abstract
Program synthesis code generation aims generate program that satisfies problem specification. Recent approaches using large-scale pretrained language models (LMs) have shown promising results, yet they have some critical limitations. In particular, they often follow a standard supervised fine-tuning procedure to train code generation model from natural language problem descriptions and ground- truth programs only. Such paradigm largely ignores some important but potentially useful signals in the problem specification such as unit tests, which thus results in poor performance when solving complex unseen coding tasks. To address the limitations, propose "CodeRL", new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning (RL), Specifically, during training, we treat the code-generating LM as an actor network, and introduce a critic network that is trained to predict the functional correctness of generated programs and provide dense feedback signals to the actor. During inference, we introduce new generation procedure with a critical sampling strategy that allows a model to automatically regenerate programs based on feedback from example unit tests and critic scores. For the model backbones, we extended the encoder-decoder architecture CodeT5 with enhanced learning objectives, larger model sizes and better pretraining data. Our method not only achieves new SOTA results on the challenging APPS benchmark, but also shows strong zero-shot transfer capability with new SOTA results on the simpler MBPP benchmark.
Introduction
Considering program synthesis as a sequence- teaecttt-seeeeeee -ossuuenceee task, pretrained language models (LMs) [Hendrycks et al.] 202 Chen et al., [021a| Austin et al.) [021] can be adapted to receive input sequence as problem specification in natural language and generate a sequence of codes as the output program (see Figure [ right, for an example). While these models achieve promising results, especially in basic programming tasks [Chen et al.
al., 2021 we observe that they
still fail generate codes solve complex problems [Hendrycks et al. 2021 Li et al. 2022].
There are two main limitations. First, current models are trained using a conventional next-token prediction (NTP) objective which maximizes the next ground-truth token likelihood. As noted NLP domains [Bengio et al. 2013 Ranzato et al.) 2016), training models only with next-token prediction objective
fracher-forring manner often leads to accumulating errors during test time when tokens are generated by conditioning on previously sampled tokens, not the ground-trute tokens. This issue becomes more serious in the domain of program synthesis. where token-man matehhin scores such as BLEU (Papineni et al. 2002 Ren et al. 2020 are more appropriate in partial program synthesis tasks (i.e. code completion) Husain et al. 2019 have failed to measure the functional correctness
Equal contribution.
Corresponding authors: (hungle, shoi) (oool)BBaaeefocce.com
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Problem Specification
Problem Specification
string a palindrome if reads the same from the loft to the right and from the right to the left....ff there there such substring in Sc5 that not palindrome, Exampie Input and Output:
Policy
such substring...
Pretrained LM (actor)
Input: hhannah'
Cutput:
Values
Action
Solution Program
Input() ans 0
Value Function
Critic
for 1 in rangel range(len (1en(
State
Reward
for rangel
(i: f): sii: [=:-1];
len(s) 1):
Environment
ans max max (ans, 1)
Compiler
print (ans)
Inpul: vutzUw Inpul:: :111a11
Output: Output o..
Unit Tests
I/O
An example program synthesis task (Right): Each task defined by a problem specifica- tion in natural language, often containing example imput and output pairs. The expected output is program to be checked for functional correctness against some unit tests. high-level overview of our CodeRL framework for program synthesis (Left)): treat pretrained code language model (LM) as a stochastic policy, code generations as actions, and rewards can be estimated based on the unit test results of output programs from the compiler (environment).
Figure I:
of complete programs [Hendrycks al.) 2021 Chen al. 2021a Training only with NTP objective is hence, not ideal to tackle full program generation to solve programming problems.
Secondly, current models fail to utilize the potential meaningful signals from unit tests, which directly determine the model performance by the functional correctness of programs. Current approaches neglect this important signal during model optimization well as generation procedure. During optimization, unit tests could be factored into learning objectives to match the final goal of generating semantically correct programs. During inference, since unit tests often parts of problem description (i.e. example unit tests), they are potentially powerful to further improve output programs.
To address the above issues, we introduce "CodeRL", a new framework to improve pretrained LMs for program synthesis tasks through reinforcement learning (see Figure [ left). Specifically, we propose a training strategy that optimizes pretrained LMs for program synthesis tasks in actor-critic approach [Konda and Tsitsiklis
1999 (Sttonn et al. [999]. We treat the pretrained LM as an actor network and synthetically sample sequences from this actor, including both correct and incorrect programs. These program samples are passed to critic model which is trained as an error predictor to assess the functional correctness of these samples. We the token-level hidden states extracted from the learned critic model to estimate the values/scores output tokens of these synthetic samples. The actor network is then finetuned on these synthetic samples weighted by their critic scores. During inference, we introduce a new generation procedure that involves example unit tests and a critic model filter and select sub-sequences. These sub-sequences model to resample new tokens and obtain new output programs. This approach allows the model to automatically refine output programs based their functional correctness during test time.
utilized seeds that condition the
We extend CodeT5 with better pretraining strategies as the foundation model for CodeRL. Our comprehensive experiments show that our models can achieve SOTA performance the challenging APPS benchmark [Hendrycks et al., 2021]. Specifically, our models reach more than 2% pass @ I, 6% pass @ 5, and 19% pass@ @ 1000. Since our RL method is model-agnostic, we apply it to various large-scale models and achieve consistent performance gains. We further test its zero-shot transfer ability on simpler MBPP benchmark (Austin et al.) 2021]. where it sets SOTA result of 63.0% pass@ 80 over a finetuned GPT-137B's ' 61.4%. We release the improved CodeT5 -lrrge (770M) model which outperforms many pretrained LMs of much larger sizes.
Related Work
2.1 Program Synthesis
Program synthesis tasks can date back as early as the early adoption of machine learning research
Waldinger and Lee 1969 Manna and Waldinger] 197T]. Farlier tasks include problem specifications
in the form of input-output (IO) examples [Summers 1977 Gulwani et al. 2012 and synthesis

---

methods are limited to probabilistic approaches Liang et al. 2010 or simple programming concepts (Joulin and Mikolov 2013 Kurach et al. 2015 As deep learning methods became popular. later approaches adopt neural models to induce output programs, assuming an inductive bias given large number of program samples (Parisotto et al., 2016 Balog et al., 2016 Devlin et al. 2017]. More recently, we witnessed the emergence of program synthesis tasks in which output programs are extended to general-purpont purpose programming languages (Yin and Neubig 2017 Xu et al. 201
Chen et al. 2021a] and program specifications are fully described in natural English text [Hendrycks
cet al. |02 Austin et al.) 2021 Poesia et al., 2022]. These extensions have encouraged a rising number of applications of pretrained language models (LMs) to program synthesis to exploit the contextual representations learned from massive data of codes and natural languages [Feng et al.
2020 Clement et al. [2000] Wang et al.) |021] Wang and Komatsuzaki 2021 Chen et al. 2022]
Recently, Nijkamp et al. [2022] proposed a conversational program synthesis approach with large pretrained language models. Despite impressive results in basic programming problems and initial commercial deploymen O existing models still perform poorly against complex problems such as
those from programming competitions Codeforces (Hendrycks et al. 2021
Li et al. 2022).
Program Synthesis in Visual Context. Another related line of research is program synthesis in computer vision domains such as images and videos. Early papers such as [Kulkarni et al. 2015
Yang et al. 2013) introduce inverse graphics networks to infer visual properties such as pose, shape, and lighting, of visual objects. Wu et al. 20177] Liu et al. [2019) Ellis et al. (2018) study the problem of image rendering, which transforms an image to structured and compact representations, i.e. scene programs. Tian et al.] [2019] extends the prior work to render 3D shapes from images through
shape programs, containing features to capture geometric and structural priors. Ganin et al. [2018) introduces an RL-based approach to render realistic images through high-level graphics programs. Sun al. introducss program synthesis from demonstration synthetic videos to summarize the behaviors of the objects in the videos.
While this line of research has remarkable impacts on applications such as image/video editing, captioning, and extrapolating, these approaches are limited to programs of domain-specific languages defined for visual objects. For instance, in [Sun et al. 2018), programming language contains basic functions for object perception, action, and control flows. In our work, we focus on program synthesis from natural language problem specifications and the output programs are in general- purpose languages such as Python. This type of programming task can range from basic programming problems to competition-lewe programming tasks that require a high level of problem-solvin skills.
2.2 Reinforcement Learning for Sequence Generation
Related the program synthesis tasks are research domains of sequence generation, in which RL approaches have demonstrated remarkable achievements, In these domains, RL approaches are used to exploit signals from non- non-differentiable metrics of the task at hand. Earlier work such as (Ranzato
et al. 2016) adopts this strategy with REINFORCE algorithm [Williams] 1992 to directly optimize
models for sequence-basee based test metrics such as BLEU (Papineni et al. 2002 and ROUGE (Lin 2004] scores for translation models. In the same domain, Bahhdnnuu et al. [2016] introduced an
actor-critic framework [Sutton] 1984 Konda and Tsitsiklis 1999]. In visual captioning domains, Rennie et al. [2017). Wang et al. [2018] proposed to use RL to optimize image captioning models using variants of CIDEr scores /eedantam et al. 2015). Alternatively, |en et al. [2017] derived a new goal-oriented return estimate using
embedding. Johnson et al. 2017 Trivedi
ct al. [202]] introduce program generation as an auxiliary task to learn interpretable policies in question-ansswriin and synthetic navigation tasks.
Different from prior domains, in program synthesis, Austin et al. 2202 Chen et al. 2021a et al. [2022] demonstrated very low correlation between token- similarity metrics and functional correctness of programs. Hence, is not trivial to define an appropriate optimization goal in this domain. We propose to exploit unit test signals, which directly exhibit the functional correctness of programs, during both model optimization and test-time generation stages. More related to our work are RL-based program synthesis [Guu et al. 2017 Bunel et al.] 2018 Liang et al. 2018 Zhong et al., 2018) and execution- -uided synthesis approaches |Ellis et al. 2019 Chen et al. 202 1b). However,
these are limited to programming languages defined within specific application domain only.
https://copiioet withup.
3

---

Public Code on Github
Solution Program
Unit Tests
Solution
Problem
Problem
Program
pretraining
Lee
Sampled program
Finetuned
Actor Network
Critic Network
Pretrained LM
LM
0
P6
9
C))(
Figure 2: Overview of our actor-critic framework to optimize pretrained LMs for program synthesis: We treat the LM as an actor network and sample synthetic samples from this actor Another neural network is trained as a critic model to evaluate these synthetic samples based on their probabilities of passing unit tests. The returns estimated based on critic scores and finally factored into the learning objective Lyt to finetune the actor LM network using synthetic samples.
CodeRL
3.1 Program Synthesis Task
Following a soquencceto-sequence approach, the program synthesis task contains problem descrip- tion as an input sequence D and an output sequence of program W (1.1...,, (ir.....))),, di E V that can solve the problem. The output at each decoding step is distribution over the vocabulary V. computed by the softmax function úi softmax(Lincar((... state at decoding step l. Conventionally, during train time, model parameters, 0, are learned by maximizing the likelihood the ground-truth reference programs. Denoting W = (WI,...TT) as the ground-truth program, the objective is to minimize the cross-entropy loss:
where s; is the contextual hidden
Lee (0)
Σ log po (WID) PoogroWW]n
Σ
D)]
where the conditional probability Pa parameterized following the above softmax function. During test time, models generate sequences of programs by autoregressively sampling token w; from the distribution Pp [ D). Models are evaluated against unit tests corresponding to problem. Fach test includes a pair of input and ground-truth output. In real-world program synthesis tasks Henddyyck et al. 2021], example unit tests are often given as parts of the problem specification.
3.2 Pretraining I Language Models on Code
We adopt Transformer models as the backbone of our program synthesis systems. Specifically, this paper extends the CodeT5 model [Wang al. 2021 foundation model for CodeRL.
CodeTS. CodeT5 [Wang et al.) 2021 is multi-lingual code-aware language model pretrained
large-scale source code corpora curated from Github. With a unified encoder-decoder architecture, CodeT5 achieves state-of-the-arr performance in wide range of code intelligence tasks in the CodeXGLUE benchmark [Lu et al. 2021 including both code understanding and generation tasks.
Improving Pretraining Data. large-scale Github Code 5 We have compiled public, non-personal information from GitHub consisting of permissively licensed Python code ("mit", "apache-2", "bsd-3-clause". "bsd-2- 126 clause", "c"0-1.0". "unlicense", "isc"). The resulting Python dataset (GCPY) has 10.5B tokens and 10x larger than the CodeSearchNet (CSN) corpus Husain et al. 2019 used in the original Co [Wang et al. 202T].
We enlarge the Python pretraining dataset using the recently released
Foor simplicity, we use the notation sequence length for all sequences which can actually variable. kattps://hut htttpsss//uugginggaceee co/ /ttaaaat//aaaaaaaaaaaaaaaatst

---

Improving Pretraining Objective. While pretraining tasks in CodeT5 like masked span prediction (MSP) benefit code understanding tasks. they have large discrepancy with program synthesis objectives. To mitigate this gap, we introduce pretraining task of next-token prediction (NTP) into CodeT55 Specifically, we uniformly sample pivot location for each code sample, then pass the content preceding the pivot to the encoder and the remaining to the decoder. To control the length of input and output sequences, we restrict the pivot within 10% to 90% the original sequence.
3.3 Program Synthesis as an RL Problem
We propose to formulate the Program Synthesis as an RL problem and apply an actor-critic RL approach improve the performance pretrained LM by exploiting the unit test signals both model optimization (see Figure 2 and generation procedure (see Figure 3p.
More formally, decides an action as the prediction of each token. Following each action, an LM model updates its hidden state representations which are used by the policy to determine the next action in the next decoding step. At the end of the generation episode (i.e. an <endoftex/> token is observed), the LM model receives relurn measured by the functional correctness of the generated program. The goal of RL finetuning is to minimize the expected return:
view the learned parameters of an LM model,
stochastic policy, which
(,) = -Ew-~pp [r(*))
where W (wf, .... w ) is synthetic sample in which each token w/ is sampled by the LM model at decoding time step l. Following the REINFORCE algorithm (Williams) 1992, Sutton and Barto 2018) and policy gradient theorem (Sutton al. 1999 can define estimate of the gradient (0) of the non-differentiable return r as:
(oCn(()) -Ew-~po (W (W") log (W [D]
Ew Ir (W*)
2 Vo log po (wr w D)]
Defining Return by Unit Test Signals. For each sample sequence W*, the return r can be defined heuristically by checking its functional correctness. We pass generated programs together with the corresponding unit tests to a compiler. From the outputs of the tests, we can determine the return r:
-1.0
if W cannot be compiled (i.e. compile error)
-0.6
W cannot be executed with unit tests (i.c. runtime error)
(5)
r(W))
-0.3
if W failed any unit test
(6)
+1.0
W passed all unit tests
(7)
However, in related domains such text-to-SQL. research [Zhogg et al. 2018 Xu et al. 2018 we note that this approach to estimate returns can lead to an unstable training process with high variance of the gradient estimate following Eq. (3 with mini-batches in training.
Return with a Baseline. In order to alleviate this variance, we adopt a "baseline" [Sutton and Barto 2018). Specifically, we use greedy decoding strategy a baseline and any generated samples that outperform this baseline are given positive return estimation, and negative return estimation otherwise. This relative normalization technique allows models to explore imperfect programs, as long as their returns are better than the baseline's. s. Given a training sample, we denote the return of the baseline rW and the expected gradient is computed as:
V voL.( -Ew+~pp ((r(W((- rr(w)) Σ V log logre((sg (e wf.-- D)]
(8)
Note that each decoding step f, our greedy decoding baseline is independent from the action wr and hence, the expected gradient term (0) from Eq. 3 remains the same in Eq. (8)
Intermediate Return by Critic as Error Predictor. We observe that the above gradient estimate is only based on final return at the end of the decoding process. However, programs often follow fixed syntactical rules in which single token such as an additional white-space character can render program erroncous. Therefore, Eq. (8) becomes too restrictive. A straightforward solution
5

---

Extract example input/output pairs
Problem
Example unit tests
Hidden unit tests
Filter by example unit test results
Finetuned LM
Sample
3a
3b
sub-sequences
Pass
Seed
Seed
Fail
NUM (passed)
by critic scoring
Figure 3: Overview of our Critic Sampling (CS) approach: (1) For each test problem, we first use finetuned LM generate a of solution programs. (2) From the problem description, we extract example unit tests and test against generated solutions. (3) If there are any passed solutions, we pass them to the critic model sample sub-sequences. (4) The sub-sequences from (3) are used seed sequences to condition the LM to regenerate solution programs, repeating the steps from (1) onward.
to use token- token-based similarity scores (Papineni et al. 2002 Ren et al., 2020 between each subsequence Wf. and the ground truth. However, code matching is not ideal return measure due to its poor correlation with program correctness [Hendrycks et al. 2021 Chen et al. 2021a Austin al., 2021] which can only be measured against fully complete programs. Alternatively, we introduce a critic model (see Appendix ?? for an overview). The critic model is parameterized a neural network with parameters that receives inputs as the problem description and sampled program W* = {wi,... w The critic is trained to infer the unit test outcome; one of (CompileError, RuntimeErrorr FailedTesst, PassedTesst) } as described in the return definitions in Eq. to The training objective of the critic ø can be expressed as:
(o))tic(0) = - log (u (WW))
where denotes the ground-trute -tuth unit test outcome given by the compiler. We use Transformer models of smaller sizes than the actor model as the base architecture for the critic model. The contextual hidden states the program tokens (hi,,.., l.T) obtained from the critic model decoder are max-pooled along the sequence length dimension hpool Pooling(hy..... hT).. The critic's prediction on the unit test outcome is computed as â = softmax (Linear(nnnnaannnna
Given learned critic, use the probability distribution è roftmax([inear((] (h+)) )) to estimate the token-level value 9 of w in relation to the ground unit test output (note that we use the token level contextual representation h4 here, before the pooling operation). Specifically, (w()) 0z [u] where t(.] denotes the probability of a specific unit test outcome from the four possible ones. We use this estimate train the actor LM model with intermediate returns:
(0Cr((0) ~ -Ew - rr("')) Σ
(10)
Generating Programs with Example Unit Tests and Critic. We leverage the unit tests provided in the input problem description to improve the generation procedure during inference too (see Figure 3 for an overview and Appendix ?? for step- by--bpytepp explanation). For each problem, we generate N programs, out of which we only select programs that pass example tests (leading set F) and filter out the rest. To improve sample quality, we perform another round of generation where we sub-sequences from these filtered samples prompts (or "seed" sequences) the actor LM. We employ a separate critic model (orea) to guide our choice of sub-sequences from these filtered samples. This critic model is trained with a similar objective Eq. (9) but binary classification setup with [FailedTTest Passed'ees labels.
Let Wfitter {wi...., w denote generated sample that passes the example unit tests. We use the critic model to assign value to each token Tohn (wx) Po (ū = PassedTest//ur D) corresponding to the critic's predicted probability of the sub-sequence till passing the unit tests. We split the sequence position tmax corresponding the highest critic assigned value and use

---

the left split as the seed for the next stage. If this seed sequence till max contains token with Powe (Failed > Po.. (esss) we further chop it this token by removing tokens on the right. This is done to pick prompts that are likely to generate successful programs in the next round.
We use these seeds to initialize and condition the (actor) LM resample new tokens till encounter the <endoftext> token. In this round, each seed sequence can be stacked |/ times for upsampling. This results in the same number of output programs N. We call this generation procedure "Critic Sampling" (CS). We use mini-batch generating to improve efficiency during inference and employ nucleus sampling with a batch size of N 200. While we do incur additional costs to re-sample using the seed sequences, we are only required to generate partial programs in the re- re-generation stage, making this stage less expensive than conventional generating procedures.
4 Experiments
4.1 Experimental Setups and Datasets
Pretraining Setup. We pretrain a Code'T5-large model (770M) from seratch following T5-largess 's
architecture [Raffel et al., 2020]. We follow the pretraining setups in CodeT5 [Wang et al. 2021] with the modifications
proposed in $3.2 We evaluate this new pretrained CodeT5 model on 2021] and achieve new SOTA results (see Appendix ??).
CodeXGLUE [Lu et al.
APPS Benchmark. We choose the challenging APPS program synthesis benchmark (Hendrycks jet alll,, [02T], as it has large coding problems of varying difficulties collected from multiple coding websites. includes training and test splits, each of which has 5000 samples of programming tasks with diverse levels of difficulty, including "Introductory".. "Interview", and "Competition" levels. Each sample includes 20 unit tests on average to validate the functional correctness of programs.
Finetuning Setup. Due to the potential large number of trajectories (i.e. VT) to generate sequence and the unstable feedback loop between actor and critic [Lillicrap et al. [015 Wang et al., 2018). we applied imitation learning to first warm-star a pretrained LM model with Lee only for up to 10 epochs. We then sampled sequences of programs from this actor network to train the critic while keeping the parameters of the actor network frozen. For experiments with CodeTT actor models, we use the Code'T5 -small architecture for the critic model, and GPT2-small critic architecture when the actor models are GPT variants. After training the critic, we then apply both Lce and Lnt with equal weights to finetune the actor network.
Evaluation. We follow [Hendrycks et al., 2021 Chen al. 202 and evaluate the models using the pass@k k metric, which is the percentage of problems solved by using k generated programs per problem. We also follow [L et al. [222] and use n@ metric which only considers subset of candidates from k generated programs per problem. The subset of n candidates are typically selected by filtering method by passing generated programs through example tesls given as part of the problem description [Chen et al. 2021a Li et al. 2022).
For more details of the experimental setup, please refer Appendix ??
4.2 Experimental Results on APPS
Baselines. As reported by Hendrycks et al. [2021), J, we compared our models with several baselines. including GPT2 [Radford et al. 2019], GPT-Neo [Black et al.), and GPT3 [Brown et al. 2020). We also compare the results with Codex [Chen et al. 2021a] and AlphaCode [Li al. 2022 Note that by default, results of pretrained LMs (except for Codex and GPT3) are from models finetuned on APPS using the standard loss Loe only. In our ablations, since CodeRL is model -ggnssi,,, we can also integrate it with GPT variants such GPT-J [Wang and Komatsuzaki 2021 and GPT-Neo.
Overall Results. Firstly, Table Ia shows that the CodeRL with the CodeT5 model can achieve significant performance gains, outperforming many pretrained LMs of much larger sizes. Specifically, our approach achieved new SOTA results of 2.57% pass @ 1, 6.21% pass@5, and 19.36% pass 1000. Table (b]shwss that when evaluating on a subset of filtered code samples, our CodeRL+CodeT3s can achieve SOTA results of 7.83% / @@ and 11.61% 5@ k. Note that while CodeRL incurs additional computation cost during inference with CS, our approach only requires much lower & to achieve comparable performance with other models. Specifically, with & = 1000 only, our model performance is as good AlphaCode with much larger generation budget of k 50000.
7

---

Table 1: Results on APPS: Overall, CodeRL can bring the performance gains on CodeT5 models
and achieves new SOTA on both pass @k and n n@k metrics. "Intro": introductory, "Inter": interview, "Comp": competition-leeel tasks.
Performance by pass @ with {1, 1000}
Model
passi
1000
Intro
Inter
Comp
All
Intro
Inter
Comp
All
Intro
Inter
Comp
All
0.02
0.09
2.25
3.23
AlphaCode
1B
17.67
5.24
7,06
8.09
GPT3
175B
0.20
0.03
0.00
0.06
GPT2
0.18
1.00
0.33
0.00
0.40
2.70
0.73
0.00
1.02
GPT2
1.58
1.30
0.70
0.00
0.68
3.60
1.03
0.00
1.34
25,00
9.27
8,80
12.32
GPT-Neo
1.58
27.90
GPT-J
6B
5.60
1,00
0.50
1.82
9.20
1.73
1.00
3.08
35.20
13.15
13.51
17.63
CodeRL=Code13
770M
6.77
1.80
0.69
2,57
15.27
4.48
2.36
6.21
38.10
14.33
15.70
19.36
(b) Performance by n@k with up 50000 and (1,5}
T@k
5@k
Model
Size
k
Intro
Inter
Comp
Intro
Inter
Comp
All
Codex
1000
22.78
2.64
3.04
6.75
24.52
3.23
3.08
7.46
AlphaCode
1B
1000
14.36
5.63
4.58
7.17
AlphaCode
1B
10000
18.18
8.21
6.65
9.89
AlphaCode
1B
50000
20.36
9.66
7.75
11.42
CodeRL+CodeT5
770M
1000
16.52
6.16
4.15
7.83
24.49
8,58
7.82
11.61
Table 2: Ablation results with variants of return estimates: CodeT5 model that trained with return estimates using baseline (W)) and critic-based return estimates (qe) achieve the best performance. "dist." indicates a rule-based approach that estimates returns following a linear decay by token positions from 1 to T.
#
W
pass I
pass
Intro
Inter
Comp
All
Intro
Inter
Comp
A
4.60
1.10
0.20
1.62
7.10
1.57
0.40
2.44
4.00
0.87
0.20
1.36
5.60
1.30
0.20
1.94
dist.
4.90
1.03
0.20
1.64
7.80
1.60
0.30
2.58
DDD
6,20
1.50
0.30
2.20
9.39
1.90
0.42
3.10
4.3 Ablation Studies
In this section, for a fair comparison between variants of return estimates and learning objectives, report the results of pass@k where = {1,5) with beam search decoding. For larger k we report the results with and without the CS procedure.
trained by different approaches to estimate returns of code samples. Overall, we report that the CodeRL objective with relative token-level return estimates by our critic model (Model D) can achieve the best performance on pass@/ @ / and p.ss@:5. Secondly, we note that using absolute returns without baseline (Model B) could lead to the most performance drop, as this approach heavily penalizes all incorrect samples (even though they might still be better than a naive baseline). Thirdly, without critic model, simply assigning identical rewards to all tokens in code sample (Model A) is disadvantagcous these return estimates are too restrictive to be used as feedback signals for RL training. Finally, we experimented with sistance-based critic which assumes that token values decay linearly from t 1 to t T (Model C). The lower performance suggests the benefit of training a critic network to compute the returns rather than relying rule-based approaches.
Impacts of Return Estimates. Table 2 show the results of
Impacts of Learning Objectives. Table shows the results with different combinations of Lee and Lyt. We experiment with using only L.! and note the problem of vanishing gradients during finetuning [Ranzato et al.] 2016 Bahdanau et al. 2016]. Secondly, we note that by using only Lee for further finetuning, despite improvement losses during training time, the model performance indeed degrades during test time. We expect these models are overfitting to the training data. Interestingly, a naive approach of fae with synthetic samples W all of which are treated as correct codes with r(W*) 1, still leads to some performance improvement with GPT-Neo pass @ 5 (but not in

---

Table 3: Ablation results with different learning objectives: We experiment with both CodeT5 and GPT-Neo with different combinations of cross-entropy loss Lee and reinforcement learning loss L Note that these losses are applied on models that are already warm-started with conventional cross-entropy losses for up to 10 epochs.
pass
pass
Lee
Cri
Intro
Inter
Comp GPT-Neo
Intro
Inter
Comp
/
3.90
0.57
0.00
1.12
5,50
0.80
0.00
1.58
2.70
0.90
0.10
1.10
5.00
1.43
0,30
1.92
((W2)
2.90
0.80
0.30
1.12
5.20
1.57
0.40
2,06
3,30
0.80
0.20
1.18
5.30
1.57
0.20
2.04
4.70
0.73
0.30
1.44
6.58
1.54
0.18
2.28
CodeTS5770M
6.60
1,03
0.30
2.00
8.80
1.67
0.70
2.90
4.60
0.93
0.10
1.50
7.00
1.37
0.20
2.26
((WW
5.10
1.10
0.40
1.76
8.30
1.43
0.70
2.66
5.00
0.90
1.64
7.60
0.60
6.20
1.50
0.30
2.20
9.39
0.42
Table 4: Ablation results of Critic Sampling (CS): Overall, using CS can lead to higher passing
rates as program generations are conditioned on seed sequences filtered by their test results.
Mctrie
Approach
Intro
Inter
Comp
All
without CS
26.79
8.73
7.60
12.12
pass@ 200
with CS
29.10
9.67
9.50
13.52
without CS
35.30
13.33
13.60
17.78
pass@ 1000
with CS
38.10
14.33
15.70
19.36
without CS
16.27
6.00
4.27
7.71
1@ 1000
with CS
16.52
6.16
4.15
7.83
other cases). Finally, we found that using both Lee and Lr results in a more consistent performance improvement overall pass@ and pass @5 for the GPT-Neo and CodeT5 models.
Impact of Critic Sampling Table shows the ablation results of critical sampling (CS) during inference. Overall, we found positive impact of CS for improving pass@ 200 and pass@ 1000 metrics. In addition, the positive impacts of critic sampling pass @) / and pass @.5 5 are indicated by comparing the results Table and Table (row D, in which we only used conventional beam search decoding without critic sampling). We can observe that, using critic sampling, model performance increases from 2.2% pass @1 (3.1% pass 5) to 2.57% pass@ / (6.21% pass@5). Interestingly, from Table 4 we observe that CS does not provide a significant gain the n@ metric. Note that nk measures the solving rate among the subset F filtered from samples. As CS will technically increase the size of this subset, the n@k metric will consider exponentially larger number of options of n samples than before. This will normalize @k by larger pool of n candidate set, resulting in less impact of CodeRL on the results. We recommend additional post-pin processing steps such as candidate ranking [Cobbe et al. 2021 to improve the n @ performance.
Impacts of Pretraining Approaches for CodeT5. Table reports the results of CodeT5 with different configurations of model sizes, pretraining data, and pretraining objectives. For a fair comparison, all models are only finetuned with Lae on APPS. As observed in prior work (Chen et al.) 2021a Austin et al. [021], scaling up the number of model parameters or the size of the pretraining data can significantly improve the model performance that enhancing the pretraining objectives with next token prediction (NTP) is vital for generation tasks, surpassing just masked span prediction (MSP) from the original CodeT5.
downstream synthesis tasks. We also find
4.4 Zero-shot Evaluation on MBPP Benchmark
Finally, we test the zero-shot transfer ability of CodeRL on another smaller and simpler program synthesis benchmark MBPP Austin et al. 2021

---

Table Ablation results of CodeT5 pretrained model variants: We report the results of models pretrained on different configurations by model size, pretraining data, and pretraining task. CSN:
CodeSearchNer, GCPY: Github Code Python, MSP: Masked Span Predition, NTP: Next Token Prediction. For fair comparison, all models are finetuned only with Lee on APPS.
pass
pass
Size
Data
Task
Intro
Inter
Comp
All
Intro
Inter
Comp
All
60M
CSN
MSP
1.40
0.67
0.00
0.68
2.60
0.87
0.10
1.06
220M
CSN
MSP
2.50
0.73
0.00
0.94
3.30
1.10
0.10
1.34
770M
CSN
MSP
3.60
0.90
0.20
1.30
4.30
1.37
0.20
1.72
770M
+GCPY
MSP
4.30
1.10
0.20
1.56
5.60
1.47
0.30
2.06
770M
+GCPY
+NTP
6.60
1.03
0.30
2.00
8.80
1.67
0.70
2.90
Table 6 reports the results of our CodeRL+CodeTS on MBPP benchmark compared with finetuned GPT models of up to 137B size. Our CodeRL+CodeeT5 (ZS) was trained on APPS and then evaluated on MBPP zero-shot setting. We observe that CodeRL with CodeT5 of much smaller model size yields surprisingly good zero-shot performance, setting a new SOTA result of 63.0% pass @ 80 over GPT-137B's 61.4% pass@ 80. This validates the strong zero-shot transfer ability of CodeRL for unseen tasks.
Table Results on MBPP: we test the zero-shot transfer ability of CodeRL. CodeRL+CodeTS (ZS) which was trained on APPS with Lnt and evalu-
ated on MBPP [Austin et al. 2021 in a zero-shot
setting, achieves new SOTA.
Model
Size
pass NO
GPT
224M
7.2
GPT
422M
12.6
GPT
1B
22.4
GPT
4B
33.0
GPT
8B
40,6
GPT
68B
53.6
GPT
137B
61.4
For additional experiments, analysis, and qual- itative results, please refer to Appendix ??.
CodeRL+Code11S (7S)
770M
63.0
5 Limitations and Broader Impacts
One limitation of our approach is the computation cost of training critic model to estimate returns in addition to the original LM (actor network). However, in practice, we found that training good critic model does not require large-scale models to attain a decent performance. For instance, a finetuned critic model initialized from pretrained GPT-2 (small) can achieve over 75% error prediction accuracy on synthetic samples.
Program synthesis can lead to substantial positive social impacts, e.g., transforming future software developing tools, increasing the productivity of developers, and improving the accessibility and quality of programming courses. Yet, some risks and bias issues are still worth considering before deploying such models at scale. For example, training data from public GitHub code repos may contain vulnerabilities and the resulting synthesis models may generate programs with weak security measures [Hammond Pearce et al. 2021].
Conclusion
We present CodeRL, a novel framework for program synthesis, using deep reinforcement learning improve pretrained LMs, by exploiting unit test signals in both training and inference stages. Specifically, we introduce an actor-critic training approach to optimize pretrained LMs with dense feedback signals on synthetic code samples. During inference, we propose new generation procedure with critical sampling, which enables the model to automatically regenerate programs based on feedback from unit tests and critic scores. We integrate CodeRL with the improved CodeT5-large model (770M) and achieve new SOTA results on both the APPS and MBPP benchmarks, surpassing the prior SOTA by massive pretrained LMs of much larger model sizes. Our comprehensive analysis shows that CodeRL achieved consistent improvement upon the conventional pretrained LMs for code generation tasks. CodeRL a general framework that integrates pretrained LMs and RL holistically for program synthesis, and can be extended and improved in various ways. For example, it can be easily integrated with other better pretrained LMs and can be improved with more fine-grained feedback from the environment, such feedback received from static code analyzer.
10

---

References
Austin, Odena, M. Nye, M. Bosma, H. Michalewski, Dohan, E. Jiang, Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108. 07733. 2021.
D. Bahdanau, P. Brakel, K. Xu, A. Goyal, R. Lowe, Pineau, A. Courville, and Y. Bengio. An actor-critic algorithm for sequence prediction. arXiv preprint arXiv: 2016.
M. Balog, A. L. Gaunt, M. Brockschmidt, S. Nowozin, and D. Tarlow. Deepcoder: Learning to write programs, arXiv preprint 1xXiv:6611.01965 2016.
Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
S. Black, G. Leo, P. Wang. C. Leahy, and S. Biderman. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorffoww march 2021. URL https://doi. org/10.5281/zenndd 5297715.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners, Advances in neural information processing systems, 33:1877-1901, 2020.
R. Bunel, M. Hausknecht, Devlin, R. Singh, and Kohli. Leveraging grammar and reinforcement learning for neural program synthesis. In International Conference on Learning Representations, 2018. URL httpst:s:: http://oppeervvieeeee net forttt/foum??i Hw66kRR
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. 2021a.
Q Chen, J. Lacomis, E. J. Schwartz, G. Neubig, B. Vasilescu, and C. Le Goues. VarCLR: Variable semantic representation pre-training via contrastive learning. In International Conference on Software Engineering, ICSE $22, 2022.
X. Chen, D. Song, and Y. Tian. Latent execution for neural program synthesis beyond domain-specifee languages. Advances in Neural Information Processing Systems, 34, 2021b.
C. Clement, D. Drain, J. Timcheck, A. Svyatkovskiy, and N. Sundaresan. PyMT5: multi-mode translation of natural language and python code with transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9052-9065, Online, Nov. 2020. Association for Computational Linguisties. doi: 118883//1/222..mmml 728. URLhttts: //ac:///ccaaaa ho org/ 2020. emn main. 728
K. Cobbe, V. Kosaraju, M. Bavarian, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
J. Devlin, J. Uesato, S. Bhupatiraju, R. Singh, A.-r. Mohamed, and P. Kohli. Robustfill: Neural program learning under noisy i/o. In International conference on machine learning, pages 990-998. PMLR, 2017.
K. Ellis, D. Ritchie, A. Solar-Lezama, and J. Tenenbaum. Learning to infer graphics programs from hand-drawn images. Advances in neural information processing systems, 31, 2018.
K. Ellis, M. Nye, Y. Pu, F. Sosa, J. Tenenbaum, and A. Solar-Lezama. Write, execute, assess: Program synthesis with repl. Advances in Neural Information Processing Systems, 32, 2019.
Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, I.. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou. CodeBERT: pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNIP 2020, pages 1536-1545 Online, Nov. 2020. Association for Computational Linguistics. doi: 18.1..63/v 1/2020 http://........./ssoeooooo findings-emmll.133)) URL htttts://acaatttaaaee wwtp:/gyyyrrg.o.....ooo
org/ 0000 findings emn 1ammmpp11:.. 139
Y. Ganin, T. Kulkarni, I. Babuschkin, S. A. Eslami, and O. Vinyals. Synthesizing programs for images using reinforced adversarial learning. International Conference on Machine Learning, pages 1666-1675. PMLR, 2018.

---

S. Gulwani, W. R. Harris, and R. Singh. Spreadsheet data manipulation using examples. Communica- tions of the ACM. 55(8):97-105, 2012.
K. Guu, P. Pasupat, E. Liu, and P. Liang. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1051-1062, Vancouver, Canada, July 2017, Association for Computational Linguisties. doi: 10.186653/1/1111119999 URL https://aclanthoe
ac www.aaaannthhoooooooo wwt.:/tho.ooooooog11ooo-00 wtt/
B. A. Hammond Pearce, B. Tan, B. Dolan-Gavit, and R. Karri. An empirical cybersecurity evaluation of github copilot's code contributions. arXiv preprint (tt::2008.092900 2021.
D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo. C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Measuring coding challenge competence with apps. NeurIPS, 2021.
H. Husain, H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. CoRR, (et/1999999436 2019.
J. Johnson, B. Hariharan, L. Van Der Maaten, Hoffman, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Inferring and executing programs for visual reasoning. In Proceedings of the IEEE International Conference on Computer Vision, pages 2989-2998, 2017.
A. Joulin and T. Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. Advances in neural information processing systems, 28, 2015.
V. Konda and J. Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
T. D. Kulkarni, W. F. Whitney, P. Kohli, and J. Tenenbaum. Deep convolutional inverse graphics network. Advances neural information processing systems, 28, 2015.
K. Kurach, M. Andrychowicz, and I. Sutskever. Neural random-access machines. arXiv preprint arXiv:151..6633 2015.
Y. Li, Choi, J. Chung, Kushman, Schritwieser,, R. Leblond, Eccles, J. Keeling, Gi- meno, A. D. Lago, et al. Competition-level code generation with alphacode. arXiv preprint arXiv:2203.078. 2022.
C. Liang, M. Norouzi, J. Berant, Q. V. Le, and N. Lao. Memory augmented policy optimization for program synthesis and semantic parsing. Advances in Neural Information Processing Systems, 31, 2018.
P. Liang, M. I. Jordan, and D. Klein. Learning programs: A hierarchical bayesian approach. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 639-646, 2010.
T.P. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02977 2015.
C.-Y. Lin. Rouge: package for automatic evaluation of summaries. Text Summarization Branches Out, 2004.
Y. Liu, J. Wu, Z. Wu, D. Ritchie, W. T. Freeman, and J. B. Tenenbaum. Learning to describe scenes with programs. In International Conference on Learning Representations, 2019. URL
https://openreview net htte://oormmdd//eeeee SyNPK2R9K7 Syyu2221
S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. B. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu. Codexglue: A machine learning benchmark dataset for code understanding and generation. In NewrlPS Datasets and Benchmarks, 2021.
Manna and R. J. Waldinger. Toward automatic program synthesis. Communications of the ACM, 1433):511-165, 1971.
12

---

E. Nijkamp. B. Pang. H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. A conversa- tional paradigm for program synthesis. arXiv preprint arXiv22233.34474 2022.
K. Papineni, S. Roukos, Ward, and W.J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 31 1-318.. Association for Computational Linguistics, 2002.
E. Parisotto, A.rr Mohamed, R. Singh, L. Li, D. Zhou, and P. Kohli. Neuro-symbolic program synthesis, arXiv preprint arXiv; 1et..1//11.00 1.01855, 2016.
G. Poesia, A. Polozov, V. Le, A. Tiwari, G. Soares, C. Meek, and S. Gulwani. Synchromesh: Reliable code generation from pre-trained language models. In International Conference on Learning Representations, 2022. URL https: wwpppppeeeveeey net
VD97J43e
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67 2020.
M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural networks. In Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL /ttt::/paaaate
orq/ abs / 15 1111111111 C6732
S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma. Codebleu: method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.1029 2020.
Ren, X. Wang, Zhang, X. Lv, and L.-J. Li. Deep reinforcement learning-basd based image captioning with embedding reward. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 290-298, 2017.
S. J. Rennie, E. Marcheret, Y. Mrouch, J. Ross, and V. Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7008-7024, 2017.
P. D. Summers. A methodology for lisp program construction from examples. Journal of the ACM (JACM), 24(1):161-175, 1977.
S. Sun, H. Noh, S. Somasundaram, and J. Lim. Neural program synthesis from diverse demonstra- tion videos. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4790- 4799. PMLR, 10-15 Jul 2018. [RLphttss://prrceedl wttp://s.....eeee///////oo/// html
mlr
R. S. Sutton. Temporal credit assignment in reinforcement learning. PhD thesis, University of Massachusetts Amherst, 1984.
R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
R. Sutton, D. McAllester, S. Singh, and Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Y. Tian, A. Luo, X. Sun, K. Ellis, W. T. Freeman, J. B. Tenenbaum, and J. Wu. Learning to infer and execute 3d shape programs. In International Conference Learning Representations, 2019. URL htttps://openrev/
net hww.:neefffortteeeo forumm wtw.rrummmidrrr1122222220
D. Trivedi, J. Zhang, S.-H. Sun, and J. J. Lim. Learning to synthesize programs interpretable and generalizable policies. Advances Neural Information Processing Systems, 34:25146 45-55166 2021.
13

---

R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus- based image description evaluation. Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4576-4575, 2015.
R. J. Waldinger and R. C. Lee. Prow: A step toward automatic program writing. In Proceedings of the Ist international joint conference on Artificial intelligence, pages 241-252, 1969.
B. Wang and A. Komatsuzzki, GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. httpp:://githii wwm www.ooiiiooooooooooooooooooooo.e www.:oo///smsehetteee mts//m/shhhhe tresh-trrnsffrmeee jax May 2021.
X. Wang. W. Chen, J. Wu, Y.F. Wang, and W. Y. Wang. Video captioning via hierarchical rein- forcement learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4213-4222, 2018.
Y. Wang. W. Wang, S. R. Joty, and S. C. H. Hoi. Codet5: Identifie--aware unified pre-trained encoder-decoder models for code understanding and generation. In EMNLP (1), pages 8696-8777 8708. Association for Computational Linguistics, 2021.
R. J. Williams. Simple statistical grolowng- algorithms for connectionist reinforcement learning. Machine learning, (03:229-2666 1992.
J. Wu, J. B. Tenenbaum, and P. Kohli. Neural scene de-rendering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
X. Xu, C. Liu, and D. Song. SQLNet: Generating structured queries from natural language without re- inforcement learning, 2018. URL httppss:/ wtwpee/wwwwww..eee/orraaataaaae. -ttp://oorriiiii--iii.iil.-.ee.-
J. Yang, S. E. Reed, M.-H. Yang, and H. Lee. Weakly-supervised disentangling with recurrent transformations for 3d view synthesis. Advances in neural information processing systems, 28, 2015.
P. Yin and G. Neubig. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 440-450, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 1/6653//1/P1-11000
V. Zhong, C. Xiong, and Socher. Seq2SQL: Generating structured queries from natural lan- guage using reinforcement learning, 2018. URLhttts:://ppenettte wet. htr/fforrm?i Syx6bz-Ab
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section 3 and
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 5
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See Appendix
?? and code at hhttps: /tttts://gitii thubb.b..ttttttt/ com/ fttps/assssccc/oeeellll
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix ??.
14

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] As it is very expensive to experiment with sarge-scale language models, we did not try different random seeds due to the limitation of compu- tation resources.
(d) Did you include the total amount compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See the configurations in Appendix ??.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Appendix ??.
(b) Did you mention the license of the assets? [Yes] See Appendix ??.
(c) Did you include any new assets either in the supplemental material or as a URL? [No] We do not curate any new dataset in this paper. We will release the code and models.
(d) Did you discuss whether and how consent was obtained from people whose data you'l re using/curating? [N/A] All datasets evaluated in our experiments are publicly available for use.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] The data
using code samples from public programming competitions which do not include personally identifiable information or offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "r5igJbHjK0F", "reviewer": "Reviewer_paYh", "summary": "This paper addresses the problem of neural program synthesis, whose aim is to train a neural network to synthesize programs from problem specifications. To this end, the paper proposes a framework that leverages large-scale pre-trained language models and directly optimizes the functional correctness of generated programs using reinforcement learning. The experiments show that the propsoed framework outperforms baselines on the APPS benchmark and demonstrates reasonable generalization on the MBPP benchmark. Ablation studies justify the effectiveness of the design choices and the proposed techniques. I believe this work explores a promising research direction and proposes a reasonable framework to address the problem. Yet, I am concerned with insufficient novelty and seemingly inconclusive experimental results.", "questions": "Described in the Strengths And Weaknesses section.\n", "limitations": "Described in the Strengths And Weaknesses section.\n", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Paper strengths and contributions\n**Motivation and technical contribution**\n- The motivation for leveraging pre-trained large language models and RL to improve program synthesis performance is convincing and this paper proposes a reasonable framework to implement this idea.\n\n**Clarity**\nThe overall writing is clear. The authors utilize figures well to illustrate the ideas.\n\n**Ablation study**\n- Ablation studies are comprehensive. The proposed framework consists of multiple components and the provided ablation studies help analyze the effectiveness of each of them, including the cross entropy loss and the RL loss, the design of return estimate, the choice of pre-training approaches, and the critic sampling.\n\n**Experimental results**\n- The presentation of the experimental results is clear.\n- The experimental results show that the proposed framework outperforms the baselines in terms of both $pass@k$ and $n@k$ on APPS.\n\n**Reproducibility**\nGiven the clear description in the main paper and the details provided in the appendix, I believe reproducing the results is possible. \n\n## Paper weaknesses and questions\n\n**Citation format**\nThe citation format does seem right. Citations in NeurIPS papers are referred to by numbers, not the first authors' last name.\n\n**Novelty & contribution**\nOverall, I do not find enough novelty in any aspect while the overall effort of this paper is appreciated. Specifically, \n- Using RL for improving program synthesizers' performance has been studied in \"Bridging reinforcement learning and maximum marginal likelihood\", \"Leveraging grammar and reinforcement learning for neural program synthesis\", etc. Adapting this to generating Python code does not seem to be particularly novel.\n- Leveraging unit tests and the results to filter and cluster generated programs has been proposed in \"Competition-level code generation with AlphaCode\". \n\n**Related work**\nThe descriptions of the related work are not comprehensive. Several important prior works in neural program synthesis are omitted from the paper, including \"Neural Scene De-rendering\", \"Neural Program Synthesis from Diverse Demonstration Videos\", \"Learning to Describe Scenes with Programs\", \"Learning to Infer and Execute 3D Shape Programs\", etc.\n\n**Two critics**\nThe paper mentions that two separate critics are used. I am curious about the intuition behind it and am interested to know whether this contributes to the performance gain.\n\n**CodeContests dataset**\nIt would be more informative if the authors can also evaluate their proposed framework on the CodeContests dataset, which can be found on [Github](https://github.com/deepmind/code_contests). It would be easier to compare the proposed framework to AlphaCode.\n\n**Low accuracy**\nThe reported accuracy is rather low (mostly < 10\\%) for all the methods. It would make sense to increase $k$ of $pass @ k$ and increase both $n$ and $k$ of $n @ k$.\n\n**Standard deviation of accuracy**\nThe reported accuracy is rather low for all the methods. In this case, I believe showing the standard deviation of accuracy would be important to compare different methods.\n\n**Comparison to AlphaCode**\nAlphaCode paper reports the need for millions of program samples to achieve satifactory performance. It would make sense to increase $k$ of $pass @ k$ and see if the conclusion holds.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ze3TqT4H_4_", "reviewer": "Reviewer_Z533", "summary": "This work builds an actor-critic RL system on top of a pretrained language model for code. The critic is finetuned from unit test execution results of programs sampled from the language model. It's trained to estimate eventual outcome (execution correctness or any errors) at each location of the partial sampled program. The resulting system, when finetuned on the competition dataset APPS, outperforms all known finetuned and few-shot baselines of different sizes. Its performance also transfers zero-shot to a different benchmark MBPP, although with a high degree of potential overlap with finetuning data.", "questions": "1. Please address the MBPP/APPS overlap described above through a different lens: for a given pass@k metric, among the correct programs produced by CodeRL, which fraction of lines occurs in the training set of APPS?\n2. What's the impact of critic sampling on pass@1 and pass@5 metrics?\n3. Does the critic learn to correctly predict erroneous locations in wrong programs? Its max-pool mechanism and ability to predict outcomes for partial programs suggests a possibility to interpret the critic as an error localization model. Can you plot how the critic's predictions change over the course of the program's length, and when do they start matching the ground truth?\n4. How well does a critic trained on a single model's predictions transfer to other models?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- Great idea and execution. The novelty of actor-critic setup is limited but its application to execution-guided code generation for general-purpose programming languages is, as far as I know, the first of its kind.\n- Well-conducted execution with multiple metrics and comparable baselines.\n- Currently the best system for competitive code generation for both precision (n@k) and recall (pass@k) metrics, at smaller model scale than LM-only competitors.\n\nWeaknesses:\n- The effect of scaling is not studied as comprehensively as for baseline language-model only systems. The only two actors evaluated are CodeT5 @ 770M and GPT-J @ 6B, and it's unclear whether the signal learned by CodeRL critic transfers across models and/or model scales. Since training CodeRL is rather resource-intensive, demonstrating that a single critic model could be trained and applied to other generative LMs with potentially different error profiles would be valuable.\n- The \"Critic Sampling\" procedure is rather ad-hoc, and adds a substantial amount of computational overhead for a limited benefit. Its impact is only demonstrated at larger sample sizes (200 or 1000 in Table 4) and adds up to 1.5 points. No ablation data is given for it for the main metrics of Table 1.\n- The transfer results of APPS to zero-shot MBPP show an interesting learning (namely, that learning from competition problems of APPS can transfer to an unrelated dataset without finetuning) but the overlap results in Appendix C.2 question this learning.  \nLanguage models are robust to syntactic differences (e.g., internal whitespace, parentheses, semantically equivalent rewrites, and variable names). Even ignoring this, the only-whitespace normalized overlap in Figure 2 of C.2 has 12% of MBPP with more than half of their lines appearing in training data of APPS.  \nHypothetically, if (a) these programs were spread uniformly across MBPP problems, and (b) these 12% problems were the ones newly solved by CodeRL, that would account entirely for the difference between \"Raw CodeT5 finetuned\" and \"CodeT5+CodeRL zero-shot\" in Table 4. In practice, these hypotheticals are likely not as severe, but they must both be measured in order to properly appreciate the MBPP transfer result.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DkGddxxN9Rc", "reviewer": "Reviewer_5RxT", "summary": "This paper addresses the task of synthesizing programs from specifications consisting of natural language and unit tests. It combines unsupervised pre-training on large code datasets crawled from GitHub, supervised fine-tuning (imitation learning) on a dataset of natural language specifications and ground-truth programs, then reinforcement learning using the unit tests from the same dataset. The authors evaluate on the APPS dataset using various baselines and report state-of-the-art results. ", "questions": "- Why was the critic trained using max pooling over the outputs of all tokens and then used by taking each token's output individually (before pooling)? Aren't there more principled methods to estimate the intermediate returns (i.e. the value function)?\n- How many rounds were done for the critic-guided resampling? Which samples are truncated, dropped entirely, or kept unmodified for the final set?\n- If the examples in the training set are too hard for the initial model trained with imitation learning (e.g. almost no samples from the initial model pass all unit tests), the reward will always be negative and the reinforcement learning will not work. Can we quantify how much the distribution of different difficulties in the training set helped? For example, what would happen if the model was only trained on competition-level problems?\n- It may not always be convenient to have unit tests available as part of the specification. How well would the critic sampling approach work without unit tests for the test-time examples and only using the information from the critic, and how might that affect the results on the pass@k and 1@k metrics?", "limitations": "I thought the description of societal/broader impacts was sufficient.\n\nI didn't find a description of how long the policy gradient was run and how much computation was used for the RL step. There is also no information on how different random seeds may affect the fine-tuning or reinforcement learning part (I understand that it would be too expensive to repeat the pre-training multiple times). I hope the authors can provide such information in future revisions of the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Originality\nUsing reinforcement learning for program synthesis is not novel in itself, but as the paper mentions, past works have used domain-specific and toy languages rather than general-purpose languages like Python. This work also differs from others using RL for sequence generation in that it does not rely on smoother measures of quality like BLEU score to have a shaped reward function.\n\nMost of the techniques used in the paper seem to originate from other work, but I think the authors used them competently to achieve an interesting result.\n\n# Quality\nThe authors report state-of-the-art results on the APPS dataset, which was released in 2021 and contains coding problems of various difficulty. They also show impressive results on the MBPP benchmark which is also a Python dataset of programming problems like APPS, without specific training for the MBPP benchmark, 63.0% pass@80 accuracy, exceeding a much larger 137B-parameter model's 61.4% (which was the best result at the time of the paper submission; since then, [PaLM-Coder](https://arxiv.org/abs/2204.02311) reports 80-85% pass@k for two other, large models). \n\nOne weakness of the work is that they only evaluated using one base, pretrained model (an updated version of CodeT5) and using one fine-tuning dataset (APPS). I think the work would have been stronger if they showed the applicability of the method on more models and on more datasets, but the existing result is enough.\n\n# Clarity\nOverall I felt that the paper is clear and well-explained. My main confusion was regarding Critic Sampling, which is one of the main methodologically novel parts of the work. For example, it was not clear how many rounds the sampling approach should run. The mechanism can be explained using pseudocode to increase clarity.\n\n# Significance\nThe growing scale of transformer models and their pretraining datasets has led to significant increases in their ability to synthesize programs in general-purpose languages, but improvements from larger scale alone are not sustainable. Works like this suggest other directions to improve performance further.\n\nUnfortunately, it was not entirely clear to me from the ablation studies about how much of the improvement should be credited to the use of RL versus the use of critic sampling. The ablation studies on RL versus critic sampling are done on different metrics (pass@1, pass@5 versus pass@200, pass@1000, 1@1000). The significance of the work would be clearer otherwise.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BiIqOEi56U6", "reviewer": "Reviewer_Ukwi", "summary": "This work uses an existing pre-trained language model as an actor that synthesizes code to train a critic based on the results of unit tests.\nThe actor generates samples and the critic evaluates if unit tests are correct. The paper is well-written.", "questions": "1. Why not use unit tests as examples in the prompt or in few-shot learning?\n2. How sensitive is the work to the reward values?\n3. Which Codex version is used and with what hyperparameters?\n4. Why not compare the same actor-critic formulation using different language models?\n", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Using unit tests and their results as a reward signal in this fashion is a first.\n2. Working with pre-trained models, without retraining, makes the approach independent of the model and usable.\n3. The results present an efficient generation process, with an order magnitude fewer programs generated, compared with AlphaCode, 1000 generations instead of 50,000 for reaching a similar result.\n4. The experiments reduce compiler errors, push the errors down the pipeline, and increase runtime errors, and breakdown of errors.\n\nWeaknesses:\n1. Missing comparison with using unit tests in few-shot learning.\n2. Missing comparison with CodeRL component on top of other baselines.\n3. Missing comparison with of state-of-the-art model baseline: Incoder, CodeGen, and GPT-NeoX.\n4. No results on Codeforces data, though discussed in the introduction.\n5. Only Python errors are reported in the results, missing errors in other languages.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept