Execution-based Code Generation using Deep Reinforcement Learning
Authors: Parshin Shojaee
Aneesh Jain
Sindhu Tipirneni
Chandan K. Reddy
Venue: Accepted by TMLR ()

Abstract: The utilization of programming language (PL) models, pre-trained on large-scale code corpora, as a means of automating software engineering processes has demonstrated considerable potential in streamlining various code generation tasks such as code completion, code translation, and program synthesis. However, current approaches mainly rely on supervised fine-tuning objectives borrowed from text generation, neglecting unique sequence-level characteristics of code, including but not limited to compilability as well as syntactic and functional correctness. To address this limitation, we propose PPOCoder, a new framework for code generation that synergistically combines pre-trained PL models with Proximal Policy Optimization (PPO) which is a widely used deep reinforcement learning technique. By utilizing non-differentiable feedback from code execution and structure alignment, PPOCoder seamlessly integrates external code-specific knowledge into the model optimization process. It's important to note that PPOCoder is a task-agnostic and model-agnostic framework that can be used across different code generation tasks and PLs. Extensive experiments on three code generation tasks demonstrate the effectiveness of our proposed approach compared to SOTA methods, achieving significant improvements in compilation success rates and functional correctness across different PLs.

Published in Transactions on Machine Learning Research (07/2023)
Execution-based Code Generation using Deep Reinforce- ment Learning
Parshin Shojaee
parshinshojaee edu aneesh edu sindhuttull edu reddy@ @cs. vt, edu
Aneesh Jain
Sindhu Tipirneni
Chandan K Reddy
Department of Computer Science, Virginia Tech, Arlington, VA
Reviewed on OpenReview: https: // openreveew. net/ forum? id- OXBuazqEeG
Abstract
The utilization of programming language (PL) models, pre-trained large-scale code cor- pora, means of automating software engineering processes has demonstrated consider- able potential in streamlining various code generation lasks such code completion, code translation, and program synthesis. However, current approaches mainly rely on supervised fine-tuning objectives borrowed from text generation, neglecting unique sequence-lovel char- acteristios of code, including but not limited to compilability as well as syntactic and func- tional correctness. To address this limitation, we propose PPOCoder, a new framework for code generation that synergistically combines pre-trained PL models with Proximal Policy Optimization (PPO) which is widely used deep reinforcement learning technique. By uti- lixing non-differentiable feedback from code execution and structure alignment, PPOCoder seamlessly integrates external code-specific knowledge into the model optimization process. It is important to note that PPOCoder is a task-agnostic and model-agnostic framework that can be used across different code generation tasks and PLs. Extensive experiments on three code generation tasks demonstrate the effectiveness of our proposed approach com- pared to SOTA methods, achieving significant improvements in compilation success rates and functional correctness across different PLs. The source code for PPOCoder can be found at
1 Introduction
Recent years have seen surge of attention towards the use of deep learning and neural language models to automate code generation and other software engincering processes, productivity. The software development process encompasses variety of code generation tasks, including code completion (Code2Code) (Li et al. 2018 code translation (Code2Code) (Zhu et al. 2022b), and program sesth (NL2Code) (Li et al. 2022]. Inspired by the great performance of pre-trained neural language models (LMs) in different natural language processing (NLP) tasks, these pre-training techniques have been recently employed on large-scale code corpuses to automate code generation tasks. Examples of such pre-trained models include CodeBERT (Feng et al.] 2020), CodeGPT (Lu al., 2021). PLBART (Ahmnad et al., 2021), and CodeT5 (Wang et al.) 2021). However, the code generation domain brings its readable, unique challenges. While it is essential for code to be human- human-readable, it is imperative that the generated code maintains syntactic and functional correctnoss, i.c., being able to pass the compilation and unit tests.
means to enhance developer
Despite the advancements of pre-trained code models, they are heavily influenced by NLP's self-supervised masked language modeling (MLM) and often struggle to ensure the execution correctness the generated codes. For example, recent works (Chen et al. 2021aa] Li et al. 2022. Tha & Reddy 2023) have shown that a significant number of programs generated from pre-trained PL models using simple sampling methods may not pass the unit tests. To improve code generation towards operational accuracy, several approaches

---

Published in Transactions on Machine Learning Research (07/2023)
Backpropagation
Palicy Model
Protrained PL Model
(Actor)
Values
State
Reward
Value Model (Criii)
Pretrsined PL Model
Generated
Action
Code
Syntactic Match
Executable Target Code
Semantic
Masch Seore
Environment Compiler
Comp Feedback
An overview of the proposed PPOCoder framework. The actor and critic networks are first initialized from the pre-trained PL model for the desired task. Following the sampling of a syuthetic program from the stochastic policy, the reward is determined using the exccution feedback and the ground truth target code. The values are estimated by the critic network. Finally, both actor and critie networks are updated based the obtained values and returns.
Figure 1:
are followed: (i) filtering and repairing the non-executable synthesized programs (Kulal et al. 2019) (ii)
using energy-based generation models with execution constraints (Korbak et al. 2021). and (iii) using reinforcement learning (RL) fine-tuning mechanisms (Wang et al.) 2022 Zhong et al. 2017 Le et al. 2022
However, existing approaches are often tailored to a specific programming language (PL) or task, e.g Le
et al. 2022) is exclusively designed for program synthesis task in Python, and (Roziere et al. 2020 only targets code translation tasks in Python, Java, and C++ PLs. We propose PPOCoder, illustrated in Fig. an RL framework for code gencration that incorporates non-differentiable feedback derived from code execution and structure alignment as external, code-specífic knowledge into the model optimization. al.] (017)) algorithm for RL optimization which based on
PPOCoder utilizes the PPO (Schulman the proximal actor-critic advantage policy gradient objective and a trust region mechanism, making the model optimization more stable and less sensitive to new environments (tasks, PLs, or datasets). Also, PPOCoder integrates discrete compiler feedback with the syntactic and semantic matching scores between the generated codes and executable targets. This integration reduces the sparsity of the reward function, leading to a better guidance of the policy to generate higher-quality code. To the best of our knowledge, our work is the first one to integrate pre-trained PL models with PPO and incorporate non non- con-differenttable execution and code structure elements into the feedback. Inspired by (Ouyang et al., (022), PPOCoder also incorporates the KL-divergence penalty into the reward function to control explorations and prevent catastrophic forgetting, ie., drastic deviations from the priors already acquired by the pre-trained model. Previous works (Le et al.] 2022) have employed token-level matching objectives (i.c., cross-entropy loss) in the fine-tuning stage as well as pre-training of PL models to constrain these drastic deviations; however, overoptimizing for token-level matching frequently results in memorization and restricted performance when faced with new tasks and datasets. We observe that incorporating the KL-divergence penalty instead of token-matching objectives during fino-tuning can effectively minimize the likelihood of memorization (chock zero-shot results of program synthesis in Table fostering more controlled and efficient exploration that generalizes adeptly diverse environments. To summarize, the major contributions of this paper follows:
We present a PPO-based RL framework for code generation, PPOCoder, that utilizes non-differentiable code-specific feedback as the external source of knowledge in model optimization. PPOCoder provides a more stable and generalizable yet accurate model optimization that is less sensitive to new environments (tasks, PLs, or datasets) and generate higher-quality codes.
We develop a new code-specific reward function based on the discrete compiler feedback (compilation or unit test. signal when available) received at the end of the generation episode well as the syntactic and
2

---

Published in Transactions on Machine Learning Research (07/2023)
semantic matching scores between the AST sub-trees and DFG edges of the sampled generations and the correct targets.
We reduce the chance of memorization by incorporating a KI-divergence penally into reward instead of a cross-entropy loss used in earlier works during the fine-tuning phase control explorations and prevent deviations from the pro-trained priors.
We demonstrate the effecti of PPOCoder through an extensive set of experiments across diverse code generation tasks (code completion, code translation, program synthesis) and PLs (C++, Java, Python, C#, PHP; C). PPOCoder outperforms the SOTA basclines, improving the compilation rate and functional correctness over different PLs. We also investigate the benefits of PPOCoder's reward elements and PPO optimization through ablation study.
The organization of the remainder of this paper is as follows: In Section 2 existing code generation meth- ods utilizing pre-trained models, structure-based approaches, and RL methods for sequence generation are summarized. Section g delves into the specifics of our proposed PPOCoder method, including its various components. The experimental evaluation of our method on three code generation lasks: code completion, code translation, and program synthesis tasks, as well as the ablation study and case study, can be found in Finally, the paper concludes in Section 5
Section 4
2
Related Work
2.1
Pre-trained Models for Code Generation
Recent research has focused on using pre-trained language models to automate code generation tasks leverag-
ing large-scale pre-training on the extensive code corpus data available in open-source repositories (Lu et al.
2021
Zan et al. 2022 Niu et al. 2022 Notable examples of these pre-trained models include: (i) CodeBERT
Feng el al. 2020), w which conducts encoder-only pre- tre-training using Masked Language Modeling (MLM) and
Replaced Token Detection tasks; (ii) CodeGPT (Lu et al.) 2021), a comparable decoder-only GPT based pre-trained model introduced alongside the CodeXGL.UE benchmark; (kii) PLBART (Ahmad al. 2021
an encoder-decoder transformer model pre-trained employing the denoising autoencoding (DAE) objective; and (ie) CodeTT5 (Wang et al.) 2021). another encoder-decodee pre-trained model subsequently proposed, built upon the T5 (Ratfel et al. 2020 architecture and trained with code data in eight PLs. However, these
pre-trained PL models tend to rely heavily on self-supervised objectives for text generation, while grappling with maintaining essential sequence-level altributes of code such as syntactic and operational accuracy in the generations.
2.2 Leveraging Structure in Code Generation
Lately, there has been a surge of interest combining PL models with logical constructs such as abstract
syntax trees (ASTs) (Kim al. 2021 Rabinovich al., 2017 Wang & Li 2021 code sketches (Nye et al.
2021 For example, GraphCode- BERT (Guo et al. 2021 uses DFGs to incorporate semantic information into the encoded representations, but its decoder is unaware of the code structures. StructCoder (Tipirneni ct al. 2022 presents a pre-trained structure-aware encoder-decoder architecture, where both the encoder and decoder components are aware of syntax and semantic relations. However, despite these efforts, many of these structure-ane awarr code gen- cration models still struggle to cnsure the syntactic and operational accuracy of the generated codes. The primary reason is the misalignment between code-specific evaluation metrics and model optimization objec- tives. In other words, these models are not optimized for the non-differentiable code-specific objectives such compilability, readability, or passing test cases. Consequently, this results in performance deficiencies.
2019), and data-flow graphs (DFGs) Yasunaga & Liang 2020 Guo et al.)
2.3
RL for Sequence Generation
RL has been used to optimize non-differentiable metrics in sequence generation tasks [Razaaoo et al.) 2016 Bahdanau et al.) 2017), such as using the REINFORCE (Williams 1992) algorithm to improve BLEU 2002a) and ROUGE (Lin 2004) scores in the translation and summarization models.
PPapineni et al. Recently, InstructGPT OOyang et al. 2022 has introduced new Reinforcement with Human Feodback (RLHF) fine-tuning procedure which shows that language models tunetuned with RL can better follow non- differentiable human feedback. Unlike natural language tokens, generated code should follow some unique
3

---

Published in Transactions on Machine Learning Research (07 /023)
Value Function Optimization
Matching
Non-Metching
Value Loss
Policy Loss
PPO Loss
Pre-trained Model
Value Model v, (Critic)
Ne (x)
KL Penalty
KEjita pt))
(d)
Reference Model
p(|xx)
Generation
Compiler Feedback (5)
(a)
Source Data
Data Flow Graph (DFa)
Data Flow Graph
Action (Generated Code
G(Y)
Semantic Match Score
G(y)
(DFB)
Executable
Parg G y)
Count(G(5) ((9))
(ount(flt))
(c)
Code
Target
Abstract Syntax (AST)
Syntex
AST
Syntactic Match Score ASun AST,)
AST
RaudP)g) Count(AST,)
(b)
Pre-trained Model
Policy Model We (Actor)
Policy Gradient Optimization
Figure 2: Overview of the PPOCoder with actor and critic models. The action is sampled from the policy based on the given source data (NL or PT.). Then, reward is obtained for each action to guide and control policy updates. The reward function is composed of four elements: (a) compiler feedback; (b) syntactic matching score based on ASTs; (c) semantic matching score based on DFGs; and (d) KI-divergence penalty between active policy and the reference pre-trained model. The critic model estimates value based on the obtained reward and PPOCoder will be optimized with PPO, which takes into account both value and policy optimization.
code-related attributes. This includes not only syntactic but also functional correctness, as the generated code must. pass compilation and unit tests for machine execution. Recently, execution-guided approaches Ellis et al. 20199 Chen al. (021b) and RL-based fine-tuning mechanisms (Wang al., 2017 Le et al. 2022) used to enhance the quality of generated codes by PL models. For example, (Le et al.
(Chen 2019
2022 Zhong et al.
2022 has recently studied the integration of RI. with unit test signals in the fine-tuning of the program synthesis models. However, existing RL-based methods still encounter several limitations. They are often designed for a particular task (e.g., only program synthesis) or a particular PL (e.g., only Python), receive a sparse and discrete compiler signal only at the end the generation episode, and are susceptible memorization and poor performance on unscen data due the use of token-matching loss the RL fine-tuning step prevent drastic deviations from the pre-trained PL model. Our model, PPOCoder, is designed to be both task- and model-agnostic, allowing the RL framework to be flexible for wide range of code generation tasks and PLs. This is achieved by incorporating PPO-base framework that combines compiler foedback with code structure elements in the reward function to address the sparsity, and employs a KI-divergence penalty to minimize large deviations, while reducing the chance of model memorization.
3 PPOCoder
PPOCoder provides a systematic mechanism for fine-tuning code generation models using deep reinforcement learning (RL) by incorporating compiler feedback and structure alignments as extra knowledge into the model optimization, thereby enhancing the quality of the generated codes terms of code-specific sequence-level features such as syntactic and execution correctness. Fig. shows the overall structure of our proposed PPOCoder model with the policy network (actor) no responsible for code generation actions and the value
4

---

Published in Transactions on Machine Learning Research (07 (07/2023)
function (critic) V responsible for the return estimations. They are both learned with the proximal policy
optimization (PPO) approach taking reward R. As shown in Fig. 2 the total reward is composed of four elements: (i) compiler feedback; (ii) syntactic match score; (iit) semantic match score; and (iv) KL-
divergence penalty. We will provide further details about each these components later in the paper.
3.1 Problem Formulation
The code generation procedure can be formulated as a sequential discrete fmite-horizon Markov Decision Process (MDP) with the use of RL in which an agent interacts with the compiler over discrete horizon 7 which is equivalent to the maximum number of generated code tokens. The proposed PPOCoder is formulated as follows:
State S: The state of environment at each time-step, denoted as s; = (f<1,)), 8t E S, is determined by the source PL/NL data z, as well as the set of generated code tokens before t, Dcc-
Action A: The PL model chooses the action at each time-step, denoted az de, at A, which is equivalent to the generated token at time-step t.
Policy mT(a)(s;): The stochastic policy network parameterized by is the downstream code generation model that predicts the next token conditioned on the previously generated tokens and the source data, so, no mo(0el0ee, 2) : S - (AA) where ((A) denotes the probability distribution over all actions (e-g., larget vo- cabulary). The next action dt will be decided based on the top-k sampling from this probability distribution. Policy is initialized with the pre-trained reference PL model p. i.c., TQ(-) p.
Reward R: The reward R(0, y) will be obtained at the end of the generation episode (i.c., after generating the (endoflokens) Loken) based on the generated code's synlactic and functional correct as well as its structural aligmment with executable codes. The reward function R(-) is composed of different components which are explained in Section 3.2
Advantage År; Inspired by the Generalized Advantage Estimator (GAE) (Schulman et al. 2016), the advantage at time-step & is defined as
A =6 70.+1 + ... 7 - 81- 8, =r; (x(0c, x) V (0cc+1,)),
(1)
where y is the discount rate; rt is the reward time-step t; and V= (st) is the state value function at which can be approximated by dense token-level value head on top of the hidden states of PL model.
Objective: The objective of PPOCoder is to find policy that maximizes the expected reward of generated codes sampled from the policy.
max E (=1)( [R(O, v,,;
(2)
where is the training set of source data; no(-) the policy network; and R(-) is the reward function. We formulate the objective function as maximization of the advantage instead of reward, as shown in Eq. in order to reduce the variability of predictions.
T Σ ((Er,,))
max Ex-XX
(3)
We adopt the policy gradient to estimate the gradient of non-differentiable reward-based objectives in Eqs. and 3 Therefore, updating policy parameters for given source data x be derived as
2
max CP
max E gTuTO
Σ (ogmo(All0ece x) A²)
(4)
T
where VoL.. = Eg
(alogge((ii0000 x) At))
where VoL.G refers to the estimated gradient objective function based on the policy parameterized by 0. In order further reduce the variations and avoid significantly changing the policy at each iteration, the objective function in Eq. (4] will be reformulated as shown Eq. (5), called the conservative policy iteration (CPT).
5

---

Published in Transactions on Machine Learning Research (07 2022))
C8
Eg
Σ
log (De Dce, x) At }
- Egprre
Σ (1))
(=4 24
4
(5)
logTo..d (Del (e
where fod the policy parameters before the update; and (0) is the ratio log- log-probabilitees from new and old policies.
3.2 Reward Function
Figure 2 Pillustrates that the reward of PPOCoder is composed of four different components which are designed to guide and control actions simultaneously towards generating higher-quality codes. These components are designed due to (1) the sparsity of compiler feedback which is only received at the end of code generation episode; and (2) the high chance of policy divergence from the pre-trained PL models. Check the effectiveness
of these reward components in the reward ablation results of Section 4.4
Eq. 6 shows the combination of
these different reward terms in the final reward vector R(g. x. v) length.
R' with T as the generation episode
R(0.2 v) = (re : 1...,...
(6)
T1 [(cond) (D) Raec(D.v) Rarg -
RRa(E, |<t)
+1(-cond) 1- (c()),, cond (ge (endoftokens)
where rt the combined reward at time-step (.) Rasc()), and (-)a(-) are the compiler signal, syntactic match score, and the semantic match score reward terms, respectively. Note that these terms will be received at the end of the generation episode where ýt (endoftokens). The (x, is KL-divergence penalty between the reference pre-trained priors and the active policy which is imposed to the reward function at each time-step to control the corresponding action. B is also the coefficient of penalty to balance the combination of different reward terms.
3.2.1 Compiler Signal
For each source data x, we sample multiple generated codes in the target language based on the current policy network, ÿ ~ <0(1z)) Then, we pass these sampled codes to a compiler and determine the reward based on the parsing signal. In case unit tests are available for the source data, the reward is determined by the functional correctness of generated codes, i.c., passing all unit tests, as shown in Eq. 77. If unit tests are not provided, compiler returns the syntactic correctness of generated codes as shown in Eq. (5) This
reward term is designed to provide guidance to the model, encouraging to take actions that result in the correctness.
generation of higher-quality code in terms of cynnaacticffucctionn
Functional Correctness:
1, if passed all unit tests - 0.3, if 9 failed any unit test 0.6, if received Rame error 1, if received Compile error
Res (i)
(7)
Syntactic Correctness:
+ 1, if ò passed compilation test
(e)
(8)
1, otherwise
3.2.2 Syntactic Matching Score
Since the compiler signal alone is too sparse, also add additional information to better control and guide the structure of policy samples. To do so, we define syntactic matching score Rast(ū.) y) between the generated hypothesis 9 TT(-[x) and the parallel executable larget y. The goal is to maximize this matching score for better compilability or syntactic correctness. We use the abstract syntax tree (AST) to find a tree representation of the code's 's abstract syntax structure. Then, we compare the sub-trocs extracted from the hypothesis and reference target ASTs, respectively, and calculate the syntactic match score as a percentage of matched AST sub-trees.

---

Published in Transactions on Machine Learning Research (07 (07/2023 2203)
Rist v) Count n AST.)/Couun(AATT)
(9)
where CoSnt n ,ST,) is the number of matched AST sub-trees between the hypothesis & and reference yi and Counl(AST.y, is the total number of reference AST sub-trees. While simple token-based comparison can detect some syntactic errors like missing tokens, it might fail to identify the context of these tokens and their structural relationship with cach other in the codc. AST-based analysis can provide more information by recognizing and comparing the struct ures of the code samples beyond just individual lokens. For instance, a missing operator or erroncous type casting- issues often undetected at the token level-would result in different AST sub-trees, flagging these as errors. Therefore, this score takes into account the structural correctness of the entire code syntax which is a richer and more informative measure of syntactic correctness. More details of implementation are provided in Appendix A
3.2.3 Semantic Matching Score
To improve the functional correctness, we need also take into account the semantic matching ween hypothesis and the executable target y, in addition to their syntactic matching. In PLs, code semantics are closely related to the dependencies of its variables. As result, in order to construct semantic matching score, we make use of the data-flow graphs (DFGs), graph representation of code in which the nodes stand in for variables and the edges for the sources of each variable's values. Following (Guo et al., 2021), we denote DFG
code Y as G(Y) = (V; E) where V = {uI..... Jm is the set of variables, and ei, = (1), Uy} is the i + j edge showing that value of the j-th variable originates from the i-th variable. Then, we calculate the semantic match score as a percentage of matched data-flows in DFGs.
(a5a(0.2) 3) (o(nt(G()) (()))/Countt(())))
(10)
where (o) n G(v)) represents the number of matched DFG edges between hypothesis ú and reference y; and (o (v)) represents the lotal number of reference DFG edges. Maximizing this score can guide and control policy to generate codes which are more aligned with executable target codes terms of variable relations, thus, enhancing the semantic quality and logical correctness of the generated codes. More details of implementation are provided in Appendix A
3.2.4 KL-Divergence Constraint
We incorporate negative KT-divergence penalty (x(lIpp) into the reward to prevent the active policy x deviating away from the pre-trained PL model p. The KL-penalty at time t can be approximated as:
Ry (z, )21) =KL (T[IP) = log (-|xr, Dee) log (1 (-1z, (<t)) - log (-1x, (-12,, )ec))
(11)
where log (I (-Ia, I<e)) and log (p (-1z, tcr)) are the log log-probabilitie obtained from the active policy and pre-trained model at time & given source data x and the previously predicted tokens G<-- This reward term can control actions and play the role of entropy bonus in controlling exploration and exploitation where greater 3 in Eq. (6) provides less exploration and more exploitation.
3.3
Loss Function
We employ proximal policy optimization (PPO) (Schulman et al. 2017) and define the loss function of PPOCoder follows.
-- +ack
(12)
cEPI= = Eg
min (c)()) clip (c( -, -c11) 44))
(13)
CVF Eggan
(V>(eee)z) - (AL - Vinsa (0ets
-
(14)
where the loss function Lo the linear combination of surrogate policy objective function CEPT and the value function squared error term CYF. Therefore, minimizing the loss function leads to the maximization
7

---

Published in Transactions on Machine Learning Research (07/2023)
of the surrogate advantage policy objective (actor optimization) well as the minimization of value error (critic optimization). In other words, the actor is guided to maximize the advantage policy objective which is correlated with maximizing the expected reward as explained in Eqs. - and the critic is enforced to minimize the token-level value estimation error which is defined based on the difference between the values of new policy Vi (D<e) and the estimated dense returns of the old policy A + Virna( In Eqs. 12 14
the proximal policy ratio clip range, and the linear combination weight between loss terms actor and critic.
Alg. D provides the pseudocode of PPOCoder. For each source- target pair (I, we sample multiple hypothe- ses from the policy network ~ no (.Irr). After generating each hypothesis, we find the integrated reward based on the reward function defined in Section 3.2 estimate the advantage, calculate the corresponding PPO loss function, and update the policy and value head parameters based on the final gradients (as shown
in lines
Algorithm 1: PPOCoder
Input:
Set. of parallel source-target input data pains (X, y), Pre-trained PT. model Output: Fine-tuned policy with parameter based on RL
1 Initialize policy Ty p
2 for number of epochs until convergence do
for (z,3)= E (x) do
repeat
9 (
Calculate Roward
Compute Rex(0) using Eq.
Eq-
Compute Rast0. using Eq.
9
Compute (a v) using Eq.
10
Compute Ra (zt (.c)) using Eq.
11
11
Ríg.0 a. M) + (rr :t 1,..., T) where
12
re ((cond) (2) + Rust(9 v) (ag((.)) (Ba(a,
+ (-cood) (a (<t]]); using Eq.
6
14
# Estimate Advantage
15
4 + r V/s (0<(,>) + > (B using Eq.
18
Calculate Loss
17
2g
CCPI BB
min (c), (c) c1 c) (tt)
18
CVP
Eg
(V((D<eaa) (At Va (x)),)))))
Lg aC) LCPS
19
20
Update Model Parameters
21
0c 0 VoLo
22
until num_ssmppees
28
end
24 end
4
Experiments
We evaluate PPOCoder on three different code generation tasks: (i) Code Completion automatically com- pletes partial Python code snippets; (ii) Code Translation involves translating between any language-pair among six different PLs (Python, Java, C#, C++, PHP, C); and (iii Program Synthesis (NL2Code) gen- crates a Python function given a natural language (NL) description.
4.1
Code Completion
For this downstream task, we employ the Python corpus in CodeSearchNet (CSN) Husain et al. 2019 We extract 50K compilable Python methods with sufficient length (at least 64 tokens) and randomly split the data to train/val/ted sets with 40K/5K/ 555K samples. We mask the last 25 tokens of the source code and ask the model to complete it. To evaluate the quality of generated codes, three metrics are used: (i) Exact Match (xMatch) which checks if the prediction is the same as the ground truth, (ii) Levenshtein Edit Similarity

---

Published in Transactions on Machine Learning Research (07/2023)
(Edit Sim) (Lu et al. 2021 Svyatkovskiy al. 2020 which is based on the number of single-character edits needed to match the generated code with the correct target, and (iii) Compidation Rate (Comp Rate) (Kulal ett al. 2019) that shows the success rate of compilation among completed programs. Since unit tests are not provided, we focus on the syntactic correctness of the completed codes and take the compiler signal as reward.
Table | shows the results of PPOCoder along with the baselines on the code completion task. In this table, the BiLSTM (Luong al.] 2015) and Transformer (Vaswani et al., 2017 models are not pro-trained. The GPT-2 (Radford et al. 2019 model was pre-trained on text corpus, while CodeGPT (Lu et al. 2021 and
CodeT5 (Wang et al. 2021 models are pre-trained on the large-scale source code corpus. The reported results for these pre-trained models are after the fine-tuning step on the code completion task. More details of the experimental setup are provided in Appendix [A] It can be observed that CodeGPT and CodeTT5 have a compilation rate of 46.84% and 52. 14%, respectively, indicating that about half of the generated codes are not compilable. By employing our proposed PPOCoder framework on the fine-tuned CodcT5 model (PPOCoder + CodeTT5), the compilation rate improves significantly from 52. 14% to 97.68%, demonstrating the importance of incorporating compiler and structure alignment feedback into the model's optimization and the effectiveness of PPOCoder in code completion. We can also see that the PPOCoder performs similarly to other baselines in terms of Edit sim and xMatch scores, showing that the actor model effectively explores without deviating much from the pre-trained priors. In other words, the higher compilation rate achieved by PPOCoder 5 is not due to generating simple, arbitrary codes that are easy to compile.
Table 1: Results on the code completion task for completing the last 25 masked tokens from NodeScarchNet.
Model
TxMatch
Edit Skm
Coomp Rate
BiL.TTM
20.74
55.32
36.34
Transformer
38.91
61.47
40.22
GPT-2
10.13
63.02
43.26
CodeGPT
11.98
61.47
46.81
Code'T5 (220M)
42.61
68.54
52.14
PPOCuder + CodeT5 (220M)
42.63
69.22
97.68
4.2 Code Translation
We use the XLCoST + Zhu et al. 2022a] dataset for the code translation task which is a parallel dataset that includes solutions for problems related to data structures and algorithms in six languages: C++, Java, Python, PHP, C, and C#. In our experiments, we only use the compilable filtered parallel data in source and target language pairs. Table 0 in Appendix shows the detailed statistics of these compilable filtered samples across all six PLs. To evaluate the quality of translated codes, we use two metrics: (i) Comp Rate that measures compilation success rate, and (i) CodeBLEU (Ren et al., 2020) score which combines the weighted BLEU (Papineni et al. 2002b) based on the code-related keywords with the syntactic and semantic alignment measures. As unit tests are not available for parallel language pairs, we focus on syntactic correctness with the help of compiler signal.
Table P presents the results of PPOCoder on XLCOST code translation tasks along with the baselines. Tn this table, column and row headers represent the translation souree and target PLs, respectively. The Naive Copy baseline (Lu et al., 2021) simply copies the source code as the output, showing how similar two PLs are. The reported results of pre-trained CodeBERT and PLBART are after fine-tuning on the XLCoST code translation task for each language pair. More details of the experimental setup and implementation are provided in Appendix A Table 2 demonstrates that incorporating our proposed PPOCoder T5 improves the overall Comp Rate across all language pairs, in comparison to the baseline CodeT5. Specifically, we observe an absolute increase of 9.92%, 22.22%, 21.62%, 13.20%, 7.46%, and 6.11% in the compilation rate for C++, Java, Python, C#, PHP; and C target PLs, respectively. In addition to PPOCoder's 's considcrable improvement in the syntactic correctness of generated codes for different target. PLs, PPOCoder T5 also obtains comparable and mostly better CodeBLEU score to other baselines, meaning that PPOCoder + +Code generated codes maintain the semantic meaning and often align better with the correct targets. This demonstrates that the surge in compilation rate does not derive from generating arbitrary short and
hhttps://githhi http:///tituuuuccccooe/e//////// wwwp://d/eerrddeeeeeeeeee lab- code-ressarcc//t..as
9

---

Published in Transactions on Machine Learning Research (07/2023)
Table 2: Performance comparison of PPOCoder and baselines on XLCoST. The column and row language headers represent the translation source and target languages. The "Overall" column shows the weighted average scores over six PLs. The best results are shown in bold font.
High Reseurvo
Tow Rewouee
Ovemal
Medel
C++
Jawn
Pychon
Ca
PHF
"Coepplatte
[Compieeee
TCesppliste
TGadeliukd
TCreppiiie
"Cveppliste
Teoteliialy
01.56
20125
17.81
31.75
21:25
1935
K21
05.31
4.12
35.155
12.92
02.36
37.12
255.43
20.72
07:12
38.22
12:23
21.51
2.31
45.21
C++
PLBART
T1.21
44.51
60.009
45.92
74.74
31.35
02.35
53.83
52.76
36.22
08.03
46:42
90:17
50.01
72.85
03.82
75.11
141.3]
67 47
65.21
66.172
71.44
71.92
62:35
Coorrh
81.14
70.33
74.03
63.35
72.50
09.18
08.24
$0.02
61.21
79.03
72.11
72.38
Nake Cupy
30.22
14.50
36.51
22.16
02.04
41.05
39.51
211
54.18
2.30
511.39
18:35
Cod-DETT
14.50
55.01
19.53
Javn
PLBART
72.41
12.12
70.31
83.70
76.10
45.75
64.00
21 17
05.2 21
7.22
05.25
67
CadeTT
78 28.52
39.8
75.08
60.66
83.
70
63.54
64.10 87
64.71
07 55
75.15
64.15 73
PPoCCoke CoorTS
79:14
82.80
70.65
92.14
85.66
86.
64.16
90.85
00:32
82.16
73.22
86, 95
Nubon Copy
21.47
20.77
16.77
41291
18:84
45.31
38.135
13:25
Cld-BEAT
76
70.
52
74
Python PLBART
76.40
63.92
07.55
45.55
639 86
66 76
90:15
6.12
62.93
06
70:66
74.12
74.16
622.74
70.54
18:26
70.88
80LIS
56.83
71.51
73.24
60.19
PPOCCooke CoodTS
80.34
88.72
75.12
92.70
70.09
85.33
79.65
911.51
12.35
95,80
72.67
90.81
Nolve Copy
48:31
10.74
7L.81
411/19
37.75
2:41
00:11
4.32
50.85
6.16
Cod-BEIT
61.25
C#
PLBARE
78,28
90.74
57.19
63:43
70:12
54.30
8.141
71161
21.22
CoddT3
$1.49
53.87
84.78
50.73
71.23
56.81
71 15-45
75:12
67:53
102.00
75.29
03.51
Pooe CoorTS
82.04
68.51
85.77
81.92
70.43
78.61
72.06
82.62
68.11
71.00
75.86
76,71
Nomm Copy
20:35
0.12
31.51
10:25
31.05
TC
8:.11
0.41
35.30
0
20:85
Y.77
Cld-REAT
13:21
PHP
70:22
75-21
86.50
06:17
56.23
11
69005
CodeTS
83.45 83
90:00
$0.12
85.62
78
61.70
83:20
65.14
61.52
70.21
78.5 56
PPOC CoinTS
85.55
80.50
82.12
90.31
83:95
82.59
63.88
80.80
65:01
76.02
79.96
85.81
Naive Cupy
[0:41
TOY:
10:12
1027
16:33
2.30
43.54
0
53385
2:50
CokBERRE
22.72
21.19
8.30
7:
20.69
12:06
38.36
34.53
6:12
49.51
20:08
45.17
47.24
15:35
CodeTS
29.18
36.40
74.12
42.80
66.31
41.00
73.21
41.32
64
38,
71.42
42.71
PMMCCaae Cua/TT
82.17
46.40
74.30
63.02
62:15
50.34
71.08
61.72
64.37
42.32
70.92
48.82
easy-to-compile codes, This is due to the KL-divergence penalty in the reward function that controls ex- plorations and ensures the policy does not deviate too far from the pre-trained priors. Chock Appendix Fl for examples of PPOCoder's translated codes. Also, among high-resource languages in Table 2 results show relatively greater compilation rate improvements for Python and Java as target PL. This is likely due to their high-level constructs, such as the absence of pointers and memory management constructs, which can be a source of errors in languages like C++ and C. Additionally, Java and Python feature more lenient compilation process and extensive runtime error checking, resulting in many errors that would cause C++ compilation to fail, being detected only at runtime. The table shows a significantly lower compilation rate for code translation with C target PL among all baselines. This is likely due to the limited number of samples with C a target PL in the XLCoST dataset which is not sufficient for model fine-tuning (as shown in Table e of Appendix B).
We also evaluated the performance of PPOCoder on TransCoder 3 code translation benchmark Roziere
et al. 2020 with unit tests. Check Appendix C for the details.
4.3 Program Synthesis
In this task, we use the APPS (Hendrycks et al., 2021] datasct comprising 10k coding problems of varying difficulty levels, split 50/50 for train/test. sets. The dataset consists of Introductory, Interview, and Com- petition level problems with respective train/test samples of 2639/ 0039/1000, 2000/3000, and 361/1000. Each problem has 23 Python solutions and 21 unit tests on average. To evaluate the generated codes, we employ the pass@k metric, following (Chen et al.) 2021a), which calculates the percentage of problems for which all unit tests are passed using l synthetically generated program samples per problem. Since unit tests are provided in APPS, we use them in the PPOCoder's reward (as defined in Eq. (7).
Table 3 demonstrates the results of program synthesis on the APPS dataset along with other basclines reported in (Hendrycks et al.) 2021) including GPT-2 (Radford al.) 2019), GPT-3 (Brown et al.) 2020
GPT-Neo (Black et al., Codex (Chen et al. 2021a), AlphaCode (Li et al. 2022) and CodeRL (Le et al.
2022
The reported results for various models are post-fine fest-finetuning on APPS, except for GPT-3 and Codex. For the
experimental setup details of all methods, please refer to Appendix
The results indicate that the smaller
10

---

Published in Transactions on Machine Learning Research (07/2023)
Table 3: Results of the program synthesis task on the APPS dataset. The "All" column shows the weighted
average scores over three difficulty levels. The best results are shown in bold font.
!passd]
"pess $2000
Model
Slze
Intro
Inter
Comp
Intro
Inter
Comp
All
Intro
Inter
Comp
Codex
12B
4.14
0.14
0.02
0.92
9.65
0.51
0.09
2.25
25.02
3.70
3.23
7.67
AlphaCode
1B
17.67
5.24
7.06
8.09
GPT-3
175B
0.03
0.00
GPT-2
0.IB
1.00
0.33
0.00
0.40
2.70
0.73
0.00
1.02
GPT-2
1,5B
1.30
0.70
0.00
0.68
3.60
1.03
0.00
1.34
25.00
9.27
8.50
12:32
GPT-Neo
2.7B
3,90
0,57
0.00
1.12
5.30
0.80
0.00
1.58
27.90
9.83
11.40
13.76
CodeT6
GOM
0.00
0.68
2.00
0.87
0.10
1.06
CodeTS
220M
2.50
0.73
0.00
0.91
3.30
1.10
0.10
1.34
CodeT5
770M
3.60
0.90
0.20
1.30
4.30
1.37
0.20
1.72
CodeIL++CodeTTS
770M
0.5
8.60
2.64
3.51
36.10
13.48
17.50
PPOCoder CodeeT5
770M
5.20
1.00
0.5
1.74
9.10
2.50
1.20
3.56
35.20
13.35
13.90
17.77
encoder-decoder architecture of CodeT5 outperforms larger models, and PPOCoder with CodeT5 further improves performance, surpassing even larger pre-trained LMs such as GPTs. As demonstrated in Table B PPOCoder +CodeT5 exhibits comparable even superior pass Ok performance than TodeRL-+CodeTT another RL-based fine-tuning mechanism specifically designed for the program synthesis task.
To further evaluate the generalizability of these models, the zero-shot performance of the APPS fine-tuned models was examined on the MBPP (Austin et al., 2021 program synthesis benchmark, which is a collection of 974 short. (one sentence) problems, each including correct Python solution and 3 corresponding unit tests. Table shows the results of program synthesis on the MBPP benchmark. Both RL-based methods, PPOCoder CodeeT5 and CodeRL+Codee 55, fine-tuned on APPS, exhibit remarkable zero-shot performance MBPP with pass@80 of 63% and 68. 68.2%, respectively, surpassing even the largest GPT-137B's perfor- mance of 61.4%. As observed in Table 4 the proposed PPOCoder + +5 outperforms CodeRT++CodeTT on MBPP by a significant margin of 5.2%. This can be attributed to two factors. Firstly, CodeRL integrates the supervised cross-entropy loss with the RL policy gradient objective during the fine-tuning phase on APPS to prevent deviation from the pre-traine pre-trained priors and avoid catastrophic forgetting. However, overoptimizing the token-matching cross-cntropy during fine-tuning increases the chance of memorization on the APPS data and leads to inferior performance on unseen data like MBPP. PPOCoder regulates deviation by employing the Kivdivereence penalty for generation instead. We can observe that this decreases the likelihood of mem- orization, resulting in improved generalizabilies of APPS fine-tuned model on the unseen MBPP benchmark. Secondly, CodeRL utilizes the actor-critie algorithm with REINFORCE reward policy gradicnt objective, while PPOCoder employs the PPO algorithm with an actor-crilic advanlage policy gradient objective and a trust region mechanism to ensure minimal deviations from the previous policy. This leads to a more stable and generalizable model optimization for new environments. One of the main motivations of PPOCoder is its potential to enhance the competitiveness of smaller models. As demonstrated, even with a model size of 770M, PPOCoder T5 surpasses the pass@80 scores achieved by larger models like GPT3-137B. This highlights the value of leveraging execution and structure alignment signals, which is considerably cheaper than training a model with a significantly larger size. Further comparisons and discussions with SOTA larger models, such as PaI.M Chowdhery et al. 2022 and CodeGen (Nijkamp et al. 2023) at various sizes, are provided in Appendix
4.4 Ablation Study
To investigate the effect of different components of PPOCoder, we conduct ablation experiments with several variants of our model, including different reward terms, RL objective/lose terms, divergence penalty, action space size, and the number of synthetic samples. We take the Java-Python translation as a case study and
present the results in Fig. 3 More ablation studies are provided in Appendix D
Reward Elements. Fig. 3 a) shows the effect of including different reward guidance components in the performance of PPOCoder. Models lested include CodeT5 without RL training, and with RL training utilizing different combinations of reward terms: cs (compiler feedback), dfg (semantic matching score from DFGs), and ast (syntactic matching score from ASTs), assuming the inclusion of divergence penalty by default in all the variants. Fig. f(c) also represents the performance of model with compiler signal feedback (+es) with and without the KT-divergence penalty. Results of Fig. 3(c) demonstrate that upon including
11

---

Published in Transactions on Machine Learning Research (07/2023)
Table 4: Results of the zero-shot transferability on MBPP. Both zero-shot models are finetuned on APPS and evaluated on MBPP in the zero-shot setting.
Model
Size
State
Tasss880
GPT
224M
fine-tuned
GPT
422M
fine- mne-tuned
12.6
GPT
1B
fine-tuned
22.4
GPT
1B
fine-tuned
33.0
GPT
8B
fine-tuned
40.6
GPT
68B
fine-tuned
53.6
GPT
137B
fine-tuned
61.4
CodeT5
60M
fine-tuned
19.2
C5
220M
fine- Ene-tuned
24.0
CodeT5
770M
fine-tuner
32.4
CodeT1
Co
770M
zero-shot
63.0
PPOCoder +odeTT5
770M
zero-shot
68.2
tCs
+G (REINFORCE)
+cs+dfg
+RG+VF (A2C)
+C (PPO)
+ostdigrastt (Ours)
Complatton Rate
5.0
Compitation Rate
0.1
05
03
(a) Reward Terms
(b) Objective Terms
+c5 (wlo
Compilation Rate
#cs (W/
0.0
Complietion Rate
0.00
0.00
0.8
0.6
0.7
0.79
0.6
04
0.00
(6) Divergence Panally
(d) Action Space Size
(e) Synthelic Samples
Figure 3: Ablation experiment results on Java-Python translation with different configurations of (a) reward, (b) loss, (c) divergence penalty, (d) action space size, and (e) number of synthetic samples.
the KL-divergence penalty, we see marked improvement in the compilation rate performance, indicating the necessity of divergence constraint in the RL-based fine-tuning due to the broad exploration space and the high potential of policy deviation from the pre-trained priors (i.e., catastrophic forgetting). Fig, 3 a) also shows that compiler signal is beneficial in the improvement of model performance. However, the improvement obtained by +cs is still minor, potentially due to the sparsity of discrete compiler feedback received at the end of the generation episode. Results show that integrating feedback with the syutactic/ semantic matching score further boosts performance, with the syntactic matching score ( aats) offering more improvement in the compilation rate than the semantic matching score (+cs+dfg) Finally, the best performance is achieved by utilizing all four reward elements, indicating the importance of incorporating multiple sources of feedback in the learning process.
syutactic/semannic
Loss Elements. Fig. 3 represents the results of PPOCoder with different objective configurations. We observe that the policy gradient objective alone (+PG), i.e., the REINFORCE algorithm, can boost the performance the CodeT5 model. The compilation rate further improves by introducing the value function as critic (+PG+VF) i.c., A2C algorithm. Results show that the best performance is achieved by utilizing proximal conservative policy iteration with value optimization (+CPI+VE VF), indicating that the Proximal Policy Optimization (PPO) algorithm performs superior to others in this framework.
12

---

Published in Transactions on Machine Learning Research (07/2023)
Action Space Size. We examine the effectiveness of action space size on PPOCoder's performance by adjusting the k parameter in the top-k policy synthetic sampling. Fig. 3d) shows that when A: = 1, PPOCoder may not be able to have enough exploration for better possible policy updates. On the other hand, when k gets too large, PPOCoder may become overwhelmed by many diffcrent possible actions and struggle to learn the optimal policy, leading to degraded performance. Therefore, results reveal that a small value of k (A: 1) may not provide sufficient exploration, while a large value (A: 50265 (vocab size)) can hinder the learning of optimal policy. In our experiments, we usually use the action space size 5 which provides good balance for optimal exploration in most cases.
No. of Synthetic Samples. The effect of synthetic policy sample size on PPOCoder's performance is examined by modifying the num samples in Alg. | Fig. 3(e) shows that an increase in num samples from to 10 improves performance, but further increases lead to a slight decline in performance. This indicates that while more synthetic samples might improve pattern detection, excessive synthetic samples can reach saturation, becoming less effective and adding computational complexity.
5 Conclusion
We doveloped a PPO-based doep reinforcement learning framework for improving the quality of code gener- ated by PL models. We identified some limitations of traditional objective functions for code generation lasks and designed new optimization framework that is geared towards PLs as opposed to natural language. We incorporated compiler feedback and unit tests along with syntactic and semantic related qualitative feedback into our RL framework to encourage the model to generate more syntactically and logically correct codes. Results of our experiments show the effectiveness of our method compared to baselines in improving the syutactic/funetiee /functionln correctuess of the generated codes. The ablation experiments also show the impact of various model components on the final performance. It is important to note that one of the limitations of PPOCoder the added computational time required for RL-based optimization and the possibility that it may not yicld significant improvements on other evaluation metrics not directly targeted during RL opti- mization. However, PPOCoder is primarily motivated by its ability to enhance the performance of smaller models which is a more cost-effective strategy than training significantly larger ones.
References
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Humon Language Technologies, pp. 2655 2668, Online, June 2021. Association for Computational Linguistics. URLpttps::/// naacl -main.minn....
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint ar Xix:22108.00 07732, 2021.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. In International Con- ference on Learning Representations, 2017. URL /ttpps://pperreviex http::/www...eeeooe/eeeee/aggrae
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow, march 2021. URL https://doi. 2t//10.528/zz000 5297715.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few- shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1901-1901. Curran Associates, Inc., 2020. URL https: .pdf
13

---

Published in Transactions on Machine Learning Research (07 2203))
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Paylov, Alcthea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Eliza- both Barnes, Ariel Voss,, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021a. URL
Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. In International Conference on Learning Representations, 2019. URL /httts:// http:p///..ppaaeee...ee///////an mot./foruurkkd=HHggf0AAA
Xinyun Chen, Dawn Song, and Yuandong Tian. Latent execution for neural program synthesis beyond
domain-specific languages. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 22196- 22208. Curran Associates, Inc., 2021b. URL ma2255226620aaa22766666623333333
pdf
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Scbastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Xir preprinl orxi:22204, 202::2204002311,,, 2022.
Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Joshua B. Tenenbaum, and Armando Solar-Lezama. Write, Execute, Assess: Program Synthesis with REPL. Curran Associates Inc., Red Hook, NY, USA, 2019.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural lan- guages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1536-1541 On- line, November 2020. Association for Computational Linguistics. doi: 10.18653/v1, 11/855/v112020.fiin /tt///000..nning-cmmm/// 139. URL ptttps///aclantholog. wtcpoogg...gg/o// org/2020. f findings-emnlp 139
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie LIU, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. Graphcode (earhcodebbert):: Pro-training code representations with data flow. In International Conference on Learning Representations, 2021. URL htttps://openeriiew.ees forum forum?id=
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Bench- marks Track (Round 2), 2021. URL httpps///pperrvieev..ess. wtt.://tettttooommiddaaaaaaaaata
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXio: 1909.999900 099.09936, 2019.
Akshita Jha and Chandan K Reddy, Codeattack: Code based adversarial attacks for pre-trained program- ming language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 14892 14900, 2023.
Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. Code prediction by feeding trees to transformers. In Proceedings of the 43rd International Conference on Software Engineering, ICSE 221, pp. 150-162. IEEE Press, 2021. ISBN 9781450390859.
doi: 1011000/CSE443022221..0000 URL
1rgg/10.110///0543900 TTT.://EEEE29222000000000 https://dbi.orgg//
14

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Tomasz Korbak, Hady Elsahar, Mare Dymetman, and Germán Kruszewski. Energy-based models for code generation under compilability constraints. arXiv preprint or or 2021.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. Spoc: Search-based pseudocode to code. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems. volume 32. Curran Associates, Inc., 2019. URL httppp///poooddang...eerrrp.o//p pdf.
hww.:oceeddnng..nuriisss.... heerripp.cc/paper/21/////11
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Hoi. CodeRL: Mastering code generation through pretrained models and deep reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
URL https://openreviien
Jian Li, Yue Wang, Michael R. Lyu, and Irwin King. Code completion with neural attention and pointer networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAL'88
pp. 4159 25. AAAI Press, 2018. ISBN 9780999241127.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Koeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092 1097, 2022.
Chin- Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Tert Summarization Branches Out, pp. 81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https: wwcelantholooy.orrg/ggooo
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL httppp///pppaareeeeoeeee/rrrrrrr
Shuai Lu, Daya Guo, Shuo Ren, Junjic Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tu- fano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shu- jie Liu. Codexglue: A machine learning benchmark dataset for code understanding and genera- tion. In Joaquin Vanschoren and Sai-Kit Young (eds.), Proceedings of the Neural Information Process- ing Systems Truck on Dulasels and Benchmarks 1, NeurIPS Dalasels and Benchmarks 2021, Decem- ber 2021, virtual, 2021. URL /tttps://daaasstt--benttt www.hearks- www.eedddgg...urrrs...cpppappepe hash/c16a550 http://cae2eeeeeeeees33sssffeees
hhtml
Thang Luong, Hicu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp.
Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v1 10.18653/v1/155-1166 116.- URL http::///aclaataoooooooo///////
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Leorning Representations, 2023. URL https://opoareviee..r formm?id=iaYc_Kpp22.
Changan Niu, Chuanyi Li, Bin Luo, and Vincent Ng. Deep learning meets software engineering: A survey on pre-trained models of source code. In Lud De Raedt (ed.), Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pp. 5546-5555. International Joint Conferences on Artificial Intelligence Organization, 2022. doi: 10. 10.24963/1jjee 2jccai..022/77.. 24963/1jcai. 2022/775 Survey Track.
URL https://doi. org/10.
Maxwell Nye, Luke Hewitt, Joshua Tenenbaum, and Armando Solar-Lezama.
Learning to infer program sketches. In Kamalika Chaudhuri and Ruslan Salakhutdinov (cds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Reseorch, pp. PMLR, 09 15 Jun 2019. URI httpt:://///ttet httpp:://proceedinggg mlr www.peeeee///ye11...... html
15

---

Published in Transactions on Machine Learning Research (07/2023)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
https://openreviiet http:/iewweee/ffrrreeeee net/forumm waeerrummidTTTGAaaa TGBKACXEON
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311 318, Philadelphia, Pennsylvania, USA, July 2002a. Association for Computational
Linguistics. doi: 10.3115/107308: 1073135. URT. httpps://aclantholoogy wtw.:/toooooooooo/22o22/00000- org/Po2-1040
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadclphia, Ponnsylvania, USA, July 2002b. Association for Computational Linguistics. doi: 10.3115/107308.. 1073135. URL https:://ww.ettt aclweb.. org/ brgg/anthologgg/P221044
Maxim Rabinovich, Mitchell Stern, and Dan Klein, Abstract syntax networks for code generation and seman- tic parsing. In Proceedings of the 55th Annual Meeling of the Associalion for Compulational Linguistics (Volume 1: Long Papers), pp. 1139-1149, Vancouver, Canada, July 2017. Association for Computational org/P17-1105
Linguistics. doi: B URL
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodci, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 188):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21 21(14)):1-67 1-7, 2020. http:/// httpimm.rorrgrpp/eeeeeeeeee orggpapers//21/200077 html
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level training with recurrent neural networks. 2016. Publisher Copyright: C ICLR 2016: San Juan, Puerto Rico. All Rights Reserved.; 4th International Conference on Learning Representations, ICLR 2016; ; Conference date: 02- 05-2016 Through 04-05-2016.
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: method for automatic evaluation of code synthosis. arXiv preprint arXiv:2009.1022 10297, 2020.
Baptiste Roziere, Marie-Aune Lachaux, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (ods.), Advances in Neural Information Processing Systems, volume 33, pp. 20601- 20611. Curran Associates, Inc., 2020. URL ed23fb 11t2sb//Bcc22255///8844444666600
.pdf.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High digh--iimessonal continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun (eds.). 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:://axxi.... org/ www.///////......../// 02438
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint arxiv: 1Ki::170.00. 7777707.3333 06347, 2017.
Alexey kovskk,, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. Intellicode compose: Code gen- eration using transformer. In Proceedings of the 28th ACM Joint Meeting on European Software Engi- neering Conference and Symposium on the Foundations of Software Engineering, ESEC/ 2020, pp. New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450370431. doi: 011145330008004110000 URL httppp////aooo.ogg/...///////6/ 341705
16

---

Published in Transactions on Machine Learning Research (07/2023)
Sindhu Tipirneni, Ming Zhu, and Chandan K Reddy. Structcoder: Structure-aware transformer for code generation. arViv preprint. ar Xiv:22206. 05239, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukass Kaiser, and Miaaa Polosukhin. Attention is all you need. In T. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://procoed// wwwpiiiinnnnnuuiisooo.. nwwripe. /ctrr///.c//pppo///////////// (0tea2/2bb47dee111ftt00d...c2&e Papere
Xin Wang, Yasheng Wang, Yao Wan, Fei Mi, Yitong Li, Pingyi Zhou, Jin Liu, Hao Wu, Xin Jiang, and Qun Liu. Compilable neural code generation with compiler feedback. In Findings of the Association for Compu- tational Linguistics: ACL 2022, pp. 9 19, Dublin, Ireland, May 2022. Association for Computational Lin- guistics. doi: 10.18653/vvl vv 2222 /tt///222222.2....../ findings-acl22 acl.2. URL httpps://acaatthollsaa org/ findings-acil aclanngs-all.... 2,
Yanlin Wang and Hui Li. Code completion by modeling flattened abstract syntax trees graphs Proceedings of the AAAI Conference on Artificial Intelligence, 35(16):14015-144
May 2021. doi: 10.1609/ /aai.//aaii. aaai.
v35116.17650 URL https://ojs wwwi.aaai..... oegai..rg/iii index. www.iinweennniipeeeeeeeeeeeppp http//AA//artticl//ee//////e/
Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Marie-Frande Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 8696 8708. Association for Computational Linguistics, 2021. doi: 10.18653/ /v1 w1/22122..ccm/aa cmnlp-main.683 685. URL
10a//////.... 1165//0//0.11133/////0001111 http:/2222...mlpmppa/e emnlp-main. 685
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., ((3-4))229 256, may 1992. ISSN 0885-6125. doi: 10.1007 J0.1007/BF00999666
URL https:
//doi.org 10. 1007 /FO09996690
Michihiro Yasunaga and Percy Liang. Graph-based, sels-supervised program repair from diagnostic feedback.
In Hal Daumé TIT and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 10799 10808. PMLR, 13 18 Jul 2020. URL httts::/prooceddnng" mlr http:///mw......prrre////////ee/ ppress/1119/asuuag22000 html
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang, and Jian-Guang Lou. When neural model meets nl2code: A survey, 2022. URL lhttps://axxiv orgo atsg///s/2222.00999/ 09420
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXie preprinl "XXx:1700900100 2017.
Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K Reddy. Xleost: A benchmark dataset for cross-lingual code intelligence. arXiv preprint orXiv:2206.00 08474. 2022a.
Ming Zhu, Karthik Suresh, and Chandan K Reddy. Multilingual code snippets training for program transla- tion. Proceedings of the AAAT Conference on Artificial Intelligence, 36(10):11783-11799)) Jun. 2022b. doi:
10.1609/ 10.1609/aaai.vo aaaaaaaiii.... 336110..141 URL https://ojs. aaai htt.g///eer.ppp/////aa/aaaaaaa/a
Appendix
A Implementation Details
In all our experiments, employ batch size of 32, AdamW optimizer with weight decay of 0.05, and a learning rate that warms up from le- to 2 - over the first 1000 steps, then decays based on the inverse square root of the number of steps, as outlined in (Loshchilov & Hutter 2019). We use the tree-si tter parsing library to identify AST sub-trees for code written in different languages. To construct the data- flow graph (DFG), we first identify variables by using the leaves of the AST, and then create directed edges
17

---

Published in Transactions on Machine Learning Research (07/2023)
between the variables based on their relations. For the syntactic (semantic) matching, we follow very simple mechanism adopted in (Ren et al. 2020). After AST creation for both the generated code candidate and the reference target code, we examine each cubbtree/data-fiie reference AST/DFG. If a match is found in the candidate, we increment the matching sub-trces/ sub-troes/data--loow data-flows count and eliminate the current. bub-tree/dataa-loow from the list of all reference ones. All of our experiments are implemented with PyTorch and trained using 4 Quadro RTX 8000 GPUs, with 48GB of RAM.
with the corresponding depth in the
A.1 Code Completion
As explained before, we conduct code completion experiments on the CSN dataset with Python programs. For Python compilation, we adopt py sy_mplll D library. In the experiment result table, The BiLSTM base- line is the Seq2Seq Bi-directional LSTM model taken with the default settings used in (Luong et al. 2015 The transformer baseline is a 6-layer transformer decoder as used in (Vaswani et al., 2017]. BiLSTM and transformer baselines are not pre-trained and will be trained from the random initialization on this task. GPT-2 bascline (Radford ct al. 2019) is a decodcr-only transformer and is pre-trained on a large-scale text corpus. CodeGPT {Lu et al., 2021) and CodeT55 (Wang al.] 2021) baselines are both pre-trained on a large-scale code corpus with the default GPT-2 (decoder-only) and T5 (encoder-decoder) architectures. The pre-trained CodeGPT and CodeT5 models are also fine-tuned on this task, and the PPOCoder +5odeT5 is initialized from the fine-tuned CodeT5. The fine-tuning of the CodeT5 model with token-level matching objectives is stopped when convergence is reached, typically after around 100 epochs. PPOCoder is imple- mented with the discount rate 7 = 1, KL divergence penalty coefficient B = 0.1, policy ratio clip range c c= 0.2 and the value error cocfficient a 0.001. To sample synthetic hypothosis from the stochastic pol- jey, we use the lop-k sampling with A = 5 the action space size. We are training PPOCoder T5 with num haom_ssmmlls___ee 3 as the number of synthetic samples generated for each sample of the CSN dataset. Therefore, PPOCoder observes 40Kx3=120K input-output sample pairs with synthetic outputs during RL optimization for this task. In all code completion experiments on CSN, we set the maximum source and target sequence length as 400, and the maximum number of RL optimization epochs as 6.
A.2 Code Translation
We conduct code translation experiments on the XLCoST dataset with programs in six parallel PLs (C++, Java, Python, C#, PHP, and C). For testing Python compilation, we use the py compile module that comes built-in with Python version 3.8.0. For Java compilation, we use the javac compiler, version 1.8.0. We use gec version 7.5.0 for C and C++ compilations. Syntax checking for PHP is performed using the php -1 command, PHP version 7.2.24. C# compilation is also checked using the Mono C# compiler, version 4.6.2.0. In the result table of code translation experiments on XLCoST, CodeBERT baseline is an encoder-only pre- trained transformer model; PLBART and CodcT5 are encoder-docoder pre-trained transformer models. All these models (i.e., CodeBERT, PLBART, and CodeT5) are first pre-trained on a large-scale code corpus and then fnn-tuned on XLCoST for the code translation task of each language pair. Also, in PPOCoder +CodcT5, the policy network in each language-pale pair translation is initialized from a CodcT5 model finc-tuned that aaggage-pair. Here, in the implementation of PPOCoder, penalty coefficient 8 = 0.01, policy ratio clip range = 0.2 and the value error coefficient = 0.01. To sample from the stochastic policy, we use the top-k sampling with A = 5 as the action space size. Also, we train PPOCoder +ooeeT5 with num sum_samples as 100 for low-resource experiments with C' as the source/target language; 20 for experiments with PHP as source/targe language; and 10 for other translation experiments. For example, for each source codc x in Java-Python translation, we generate 10 synthetic samples basod on the policy, thus, 4, 8,1 x 10 48, 110 (based on Table G) input-output pairs are used in total to train PPOCoder for the Java-Python translation. We also set the maximum number of epochs for RL optimization as 15, and the maximum source and target sequence lengths to 400 for all code translation tasks on XLCoST.
use discount rate y 1, KL divergence
A.3 Program Synthesis
We conduct program synthesis experiments on the APPS dataset with the pair of NL problem descriptions and Python programs in three difficulty levels: Introductory, Interview, and Competition. In the results of
18

---

Published in Transactions on Machine Learning Research (07 (223))
program synthesis experiments APPS, GPT-3, and Codex baselines are tested the APPS dataset in the few-shot settings without fine-tuning, while other baselines are fine-tuned on the APPS with the cross- entropy loss. The PPOCoder +Code15 is also initialized from the fine-tuned Code'I55 on the APPS program synthesis. We use the pass@k metric to evaluate the functional correctnoss of a program, where a code is considered correct if it successfully passes all unit lests designed for the specific problem. In the PPOCoder's implementation on APPS, we use discount rate ? = 1, KL divergence penalty coefficient / = 0.05, policy ratio clip range e = 0.2, and the value error coefficient stochastic policy, use the top-k sampling with k 5 the action space size. We are training PPOCoder = 5 as the number of synthetic samples used for each APPS problem. In all
0.001. To sample synthetic hypothesis from the
T5 with nam_samppl the program synthesis experiments on APPS, we set the maximum source and target sequence lengths as 600 and 512, and the maximum number of RL optimization epochs as 8. We have also evaluated performance of PPOCoder +CodeT5 on the MBPP dataset in a zero-shot setting. We compared the zero-shot performance with GPT and CodeT5 models fine-tuned on MBPP for 60 epochs with maximum source and target sequence lengths of 400.
B Additional XLCoST Dataset Details
We are using the XT.CoST (Zhu et al. 2022a) dataset for code translation experiments with different language pairs among six parallel PLs (C++, Java, Python, C#, PHP, and C). XLCOST has been created by scraping solutions off the popular programming tutorial and interview preparation website GeeksforGocksf
This kind of multilingual parallel data is perfect for training translation models. The dataset statistics are provided in Table 5 This dataset is parallel at both program smippet and full program levels. The upper triangle of Table 5 summarizes snippet-level statistics, and the lower triangle sumnaarizes the program-level statistics. For the purpose of our experiments in the evaluation of cyntactic/functionn use program-level data. Tt noteworthy that all programs this dataset do not successfully compile. Therefore, in our experiments, we only use the compilable filtered parallel data in both source and target language pairs. Our hypothesis is that using ground truth data that is free of compilation errors will provide stronger signal to PPOCoder for correcting syntactic errors than using the full data, which could also contain flawed programs. To do so, we designed an automated compilation pipclinc that evaluates which programs in the dataset compile without any errors. Table g shows statistios of the compilable filtered dataset for all languages (except JavaScript which excluded due to the lack of compilation).
correctness, we only
Table 5: XLCoST dataset statistics. The upper triangle (in bold font) shows the number of parallel code snippets, and the lower triangle shows the number of parallel programs. JS is short for Javascript.
Lang
Java
C#
PHP
train val
8DB-00
80100
85662
69507
17811
3386
C++
4419
3913
4408
3808
923
352
8059
7228
7922
6945
1647
222
train
9450
77769
87066
69841
17868
2996
Java
490
3938
4437
3526
929
tost
901
7250
8011
7005
1672
238
75843
67219
17616
Py
val
468
471
3922
3750
923
311
test
878
882
7215
6861
1656
203
train
0187
9301
9826
68093
17873
2058
C#
val
488
491
470
3826
928
352
1668
train
8482
8470
9182
8367
17117
1876
J8
val
472
475
459
475
921
309
878
864
877
1617
train
3056
3068
3003
3071
2971
856
PHP
tost
303
307
304
307
3012
188
val
50
50
50
50
test
45
49
48
49
49
43
http:://www.eeakaffrreee...
19

---

Published in Transactions on Machine Learning Research (07/2023)
Table 6: XLCoST dataset statistics after filtering for compilable programs. Javascript excluded.
Lang
Javn
Python Cypp
PHP
train
C++
val
Lrain
6251
Java
266
Geel
6200
train
5325
4811
Python
266
250
Sest
529
496
train
5464
5081
5027
C##
279
262
259
bost
503
513
5:2
Erain
1758
1846
1785
1764
PHP
val
91
85
93
92
sest
192
176
201
197
C
vil
34
34
322
34
33
Leel
28
24
29
26
C Additional Evaluation of Code Translation on TransCoder Benchmark
For the code translation task, we additionally evaluate the performance of PPOCoder using the TransCode benchmark (Roziere et al. 2020). This benchmark comprises 852 parallel functions in C++, Java, and Python PTs, along with some unit tests for certain functions to assess the operational accuracy of the generated translations. There are few key points to consider in these experiments: (i) The TransCoder benchmark has 466, 481, and 463 lest functions with unit tests for C++, Java, and Python, respectively. This nunber of samples is inadequate for reinforcement learning (RL)-based model fine-tuning, As depicted in Table 0 of Appendix B XLCoST contains over 5000 training samples for each of these three high-resource languages, which is ample for RL-based fine-tuning. Therefore, we employ the TransCoder data only as an evaluation benchmark for models trained on XLCCST data for each language pair. (ii) We also observed that XT.CoST and TransCoder both stem from the CreesffoGeekss (GfG) website, resulting in some overlap. Consequently, we will assess the performance only on the filtered TransCoder test functions that do not coincide with the XLCoST training data. Upon examination, we discovered that approximately 30% of TransCoder test functions overlap with XLCoST. The detailed statistics for the different PLs are presented in Table 7
Table 7: Statistics of overlapping TransCoder functions with XLCOST.
C++
Java
Python
# of functions with unit tests
466
481
463
# of overlapping functions with XLCoST
137
169
139
% of overlap
29.39
35.13
30.02
Table X displays the evaluation results on the TransCoder benchmark, including metrics such as computa- tional accuracy, compilation error, runtime error, failed unit tests, and timeouts. The evaluation experiments the TransCoder benchmark reveal that PPOCoder+ CCCoodrr-CodTT' CodeT5 enhances Code computational accuracy across various programming languages: 39.9 to 72.1, 36.8 to 57.1, 45.2 to 65.6, 37.4 to 52.1, 26.1 to 39.0, and 15.9 to 51.1 for CJa-,ava, C++-Pyyhon, lava+C++++ Java Java-Python, Python-C+++, and Pathon translations, respectively. Upon examining the failed examples, we found that PPOCoder reduces compila- tion errors from 42.2 to 17.4, 25.2 to 1.1, 47.9 to 19.7, 35.3 to 0.2, 59.9 to 33.5, and 48.3 to 7.8, demonstrating the efficacy of involving compiler signal in the model fine-tuning. The PPOCoder+Codee light several intriguing observations: (1) Many failures are due to compilation errors, particularly when the target language is C+++ followed by Java. (2) PPOCoder achieves greater compilability improvements for Python and Java target PLs as compared to C++. This trend was also observed in the XLCoST dataset translations. As previously mentioned, this is likely attributable to the lower-level constructs in C++, which could introduce more error sources. (3) Runtime errors predominantly occur when Python is the target PL, which can be attributed to Python's more permissive compilation process compared to other languages. Most of the remaining errors arise from the program producing incorrect output (i.e., failing at least one
results also high-
httpp://ggttabb.cc//faaaae httte///sttaaa..o///ceebrrtrrret
20

---

Published in Transactions on Machine Learning Research (07 (07/2023)
unit test). Timeout errors are generally caused by infinite loops, which appear to be more prevalent in translations between Java and Python PLs.
Table 8: Results of code translation evaluation on the TransCoder benchmark. Both models are fine- tuned on XLCoST and evaluated on TransCoder functions with unit tests, The "Overall' column shows the weighted average scores over all language pairs.
Model
Metric
C++ Java C+ I Python Java -C+++
Java -Python
Python +C
Python -Jaa Overall
#Compatationl Accuney
37.
35Comp Error
42.4
25.2
47.9
35.5
30.
48.3
43.2
CodeT3
MRun Time Error
6.8
25.1
1.8
214
144
9.
11.1
FFalled Unit Test
7.0
11.2
4.2
688
11.1
19.4
9.95
TTmmemtt
3.9
4.05
@Cvopataaionll Arcemeyy
72.1
56.3
#CCompee Error
17.4
1.1
19.7
0.2
33.5
13.3
PPOCoder + CodeT5 MHunTTime Time Error
3.2
28.1
2.8
23.1
3.5
11.2
11.5
FFaled Unit Test
0.9
130
11.4
109
252
15.9
13.4
TTneeut
Additional Ablation Experimetns
Similar to the ablation results presented in the Fig. 3 for the Java-Python code translation, Fig. shows the results of additional ablation experiments on the Python-C# translation task for different reward terms, R.. objective terms, action space size, and the number synthetic samples.
CodeT5
33
#csedgg
+PS+VF (AZC)
#cseast
+CPIeVE (PPO)
ass (Ours)
Completion Rate
Complatton Rate
6.7
35
15
(a) Reward Terms
(b) Objective Terme
+cs (wio ki)
0.75
Compitation ftat
0.8
0.3 Complaation
0.3
63
Compliation
0.
0.
1800 3000 30005
Divergence Penalty
(d) Action Space Size
(6) of Synthetic Samples
Ablation experiment results Python-C# translation with different configurations of (a) reward, and (b) loss, (c) divergence penalty, (d) action space size, and (e) number of synthetic samples.
Figure 4:
Fig. shows the additional ablation experiment conducted (on the same Java-Python translation study) for the model variants without the compiler signal (+cs) in reward. Results clearly show that model variants with the compiler signal ddcs+ 0.732, +as+ 0.874, +ss-dfg+ast: 0.921) achieve better compilation rates compared to model variants without the compiler signal (+dfg: 0.674, +ast: 0.801, +asg+ast: 0.879), with the default inclusion of the KT-divergence penalty in all models. Therefore, the sparse compiler signal feedback received at the end of the generation episode not detrimental but, in fact, beneficial when integrated into the reward function for model optimization.
21

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Complttton Rate
64
04
(#) W/ Compiler signal
(b) WO Compier signal
Figure 5: Ablation experiment results on Java-Python translation for the model variants (a) with compiler signal (w/ +cs), and (b) without compiler signal (w/o +cs).
Figure presents the ablation experiment results for the program synthesis task on the MBPP dataset for model with different (a) reward components and (b) RT objective terms. All model variations shown here are fine-tuned on APPS (Hendrycks et al., 2021 and evaluated in the zero-shot setting MBPP. Figure G(a) indicates that using only the compiler signal (+cs) increases pass@80 by approximately 16%. Additionally, incorporating syntactic and semantic matching scores into the reward function can further enhance performance. The results of model variants with different reward components Fig. B are reported with the inclusion of the KL-divergence penalty as default. Figure b( also demonstrates that RL algorithms can effectively improve the base CodeT5 model's performance. The best results are achieved employing proximal conservative policy iteration with value optimization (+CPI+ VF),, indicating the outperformance of using PPO compared to other RL algorithms in this framework.
Codels
Codels
+PG (REWFORCE)
#csedfp
+PG+VE (A2C)
+cseast
+CR+V (PPO)
+ch-dflgeant [Durs]
Objective
Figure 6: Ablation experiment results on MBPP program synthesis with different configurations (a) re- ward, and (b) loss. All model variants are finetuned on APPS and evaluated on MBPP in the zero-shot setting.
Fig. iillstrrates the relationship between the compilation rate and pass@1 performance MBPP program synthe- sis across various model variants with different configura- tions of reward terms. As it can be observed, there is a positive correlation between these two metrics, indicating that an increased model performance terms of pass@1 can be associated with better model performance in terms of compilation rate. Specifically, the base CodeT5 model, with compilation rate of 50.1%, achieved pass@] rate of 3.4. When we added compiler signal (+cs), the com- pilation rale improved significantly to 71.2%, leading to an impressive increase in pass@1 rate to 16.4. A similar positive trend was observed when further reward compo- nents were incorporated. The model variant +dg, which uses data-flow graph, registered higher compila-
Has
soindigrae (Curs)
Compilation Rate
Figure 7: Computation accuracy (pass@1)
vs. compilation rate performance between PPOCoder +5odeT5 with different configura- tions of reward on MBPP program synthesis.
22

---

Published in Transactions on Machine Learning Research (07/202) 2023)
PROCoser-oooT5
770M
DOLEGEN M.ATI 360M
DOCEGEN M.LTI TB
DOEEGEN-MLLTI
DOREGEN-MKNN 3GMM
pass
DOCEGEN-MONO 27B
OODEGEN MONO
DOEESEN-MORO
PaLM
PaLI 623
PaLM
Coder 625
PaLZA Uadde-Uodde bAIM
10
10"
10
10 10
Size (B)
Figure 8: Pareto comparison of performance (pass@1) vs. model size on MBPP between PPOCoder +CodeT5 and larger SOTA models such as CodeGen and PaLM with various size, highlighting the performance- recource trade-off.
tion rate of 75.4%, and correspondingly, a better pass@1 rate of 21.2. Even though the addition of abstract syn- tax trees (+ct+ yielded a higher compilation rate 90.2%, the passu1 rate was slightly lower at 18.3, likely due to other complex factors influencing model performance. Our final proposed model variant, dast-aa-- yielded the best results with the highest compilation rate of 96.5% and the most elevated pass@1 rate of 26.1. Fig. 7 demonstrates that there's generally positive correlation between higher compilation rales and better pass@1 performance.
E
Discussion and Comparison with SOTA Large Models
Fig. S presents a comprehensive comparison between PPOCoder +Code'15 and prominent large language models such as PaLM (Chowdhery et al. 2022 and CodeGen (Nijkamp et al. 2023) for the program synthesis task on the MBPP (Austin et al. 2021 datasel. The vertical axis represents the pass@1, which indicates the percentage of problems for which all unit tests passed after sampling only one program from the model (k == 1). The Pareto chart showcases the performance-resource tradc-off by comparing the model size and the corresponding pass@1 scores. We observe the presence of pre-trained PaLM models, denoted by the PaLM legend, as well fine-tuned PaLM-Codee models. As expected, the fine-tuned PaLM-Coder models exhibit superior performance in MBPP program synthesis compared to the pre-trained PaLM models. PaLM models are available three different sizes: 8B, 62B, and 540B parameters, with performance improving as the model size increases. Notably, the largest 540B model consistently performs the best. Additionally, the chart displays four different sizes of CodeGen models: 350M, 2.7B, 6.1B, and 16.1B parameters. The legend differentiales between multi-lingual pre-trained CodeGen models (CodeGen-Mulli) and Python mono-lingual pre-trained models (CodeGen-Monoo) The CodeGen-Moon Mono models outperform the CodeGen- Multii models in MBPP program synthesis (on Python target PL..
Comparing the performance of PPOCoder T5 (26.1) with other models, we find that surpasses the pass@1 scores all CodeGen- Multi models (7.4, 18.0, 18.3, and 20.9) across varying sizes. Additionally, PPOCoder +CodeT5 outperforms PaLM-8B (5.0). PaLM-62B (21.4), and CodeGen-Monn-B500 (14.5) pass@1. However, it falls short of the performance achieved by the largest PaLM model with 540B parameters (36.8) and the larger CodeGen-Mono models with sizes of 2.7B, 6.1B, and 16. 1B, which exhibit. pass@1
23

---

Published in Transactions on Machine Learning Research (07 (07/2023)
scores of 27.3, 32.4, and 35.2, respectively. Analyzing the performanco-size trade-off highlighted in Fig. it becomes evident that PPOCoder + Co occupies the top-left position, denoting superior performance (higher pass@1) with the smallest possible model size. This placement on the first Pareto front aligns with one the main motivations of PPOCoder, which is to enhance the competitiveness of smaller models. By utilizing execution and structure alignment signals, PPOCoder achieves remarkable performance gains while avoiding the resourco-intensive nature of training significantly larger models. The findings from Fig. g reinforce the effectiveness of PPOCoder T5 and substantiate the advantageous performance-resourrg trade-off it offers.
F Case Studies
Fig. presents case study of PHP to Python translation for both CodeT5 and PPOCoder +CodeT5 models. We observe that CodeT5 is unable to translate the code without compilation errors, and the use of PPOCoder enhances CodeTT5's capability of generating compilable code in the target language while preserving the code fluency and distributions learned by the pre-trained Code For this example, Codel5 faces some problems: (1) "Error: Invalid Syntax": The "for" loop should start from next line (2) "Error: Local variable 'i' referenced before assignment The for loop index "j" but index "i" is called inside the loop; and (3) Semantic Relations: The equivalent PHP INT MAX may be 1000 (or sys maxsize) in Python but CodeT5 translates it to int (n/2) which is logically wrong.
Source (PHP)
function Patriachsinorder ( $ p 5
m [ array (f n F
for
++
For
L (
for ( (
$ L
1f $
continue
5 [ j -HP
for ( 5 ;
[ [
n 1
sizeof
ordnte(MMniiunn number iultiptications
NatriadhadndrderrSarr, Ssize));
Generated (Python)
CodeT5
PPOCoder+CodeT5
del Metre (p.n):
del Malr OChrntOOne (p.n) np.n)
m- [10] [Ol""n "n fork mage (n+
range n,
m - [10 in range [n]]forjin jin [n+1)]
me range (1.
for Linragge n
im -
for Jariin in (_.
for range (2,n):
L=
for range (1,n-L+1:
ift] [t)]==))
j=i+L-1
continue
[[j== n:
[illi]
continue
for kin rangeg
mn [i]li] mim
qm [ +
for range
#(gkm
if( [k]
miiitiiaq
return [1]in-1]
m
"Mirimam VatrfchhainCrder
mutipications
"Wirimam number www.ppttttiiiiaaaooooo MarkchaiinOrder
Compilation? X
Compilation?
Figure 9: Case study example for PHP-Pythan code translation. The erroneous snippets are highlighted in red.
Fig. shows an example of Java to C++ translation for both CodeT5 and PPOCoder +CodeT5. Similar the previous case, it can be observed that the compilation is improved by PPOCoder. For this example, T5's translation has these issues: (1) CodeT5 generates non-standard data type called subset which
24

---

Published in Transactions on Machine Learning Research (07 (07/2023)
takes in pair of integers. The use of the non-standard data-type without importing it defining it causes compilation error, while PPOCoder + +5odeT5 generates the C++ translation using the correct vector data-type corresponding to ArrayList in the source Java; (2) "Error: Local variable 'i' referenced before assignment": for loop index "j" but "i" is used later; and (3) "Error: Invalid Syntax" for missing "(" when calling the function.
Source (Java)
Ieport util CFG static void printvector Arraylist
for 1+ Systes print
printin
static void printvector
int
firdays
public static
String angs {
Arraylist Integer are ArreyList Integer
findayys Care n
Generated (C++)
CodeT5
PPOCoder+Code
Minclude cottslstco++hh hD using homosacco std vold printVactor glsssree
(1)
dinclde >bis//tde++.>>> using namespace std:
!
vold printvector Jisnoor Int
am size [= 13t
If ar: size ()1=1)3 1)
for (incl O: ieam, (1) [++][ cout se am [1] se""""
for (intl O; 1sar size [++][ coul es am [1] [ijsse in
cout and):
cout << andi:
void FindWays vector i in arr Int n
in O privivector aar): int
void finoWhyys vedtrr int int n if -- prntvector
back fndWWays
beok B|
back
fnndWays
intmain
Int
main
fndWayssaar.
vector
fnd/Waysllarr 1.
Compilation? X
Compilation?
Figure 10: Case study example for Java- code translation. The erroneous snippets are highlighted in red.
We also looked into an example of program synthesis from APPS. Fig. 11 illustrates case study of Python program synthesis for a problem description given in NL. Although both CodeT5 and PPOCoder +5odeT5 generate compilable programs, CodeT5's generation cannot pass some hidden unit tests that capture different corners of the problem. These two generated codes are different in the highlighted places. As we can see, the CodeT5's generation (1) initializes the 2D array 'dp' with ones on both dimensions instead of zeros on one dimension; and (2) the for loop inside the nested loop uses the condition >= j instead the correct condition & <- Although these differences do not produce any compilation errors, they result. logical errors and cause unit tests fail.
25

---

Published in Transactions on Machine Learning Research (07/2023)
Source (NL)
You given integers both arrays is equal to m; each element of each array integer between and n (inclusive); ai s bi for any index from to m; array is sorted in non-descending order; array b is sorted in non ascending order. resull be very large, you should print modulo 109
Calculate the number of of arrays such that: the length
Input:
The only line contains two integers m s n 1000, s m 10).
Output: Print one integer the number of arrays and b satisfying the conditions described above modulo 109 +
Generated (Python)
CodeT5
PPOCoder+Code T5
def solve(n,
def solve(n, mi:
dp
rangelm
up [[0]
rangelm
for in range(2, m 1):
in rengel1 n
for range(1,
de
rangauiz
for range/2. m
dp00i = (d:fii) dpli 1IH) % MOD
for in range(11, 1j:
for in range(1. n 1j: (ans dpím]lI) MOD
dplilil = (dplil)) dpfi ([8) % MOD
ens =
range(1, 1}:
ens (ans dp(m]Fl) % MOD
mapdintiinput).spptt)) printsooiver.mm
return
n.m = map(intinnuut) printjsolve(n.m))
Pass Tests?
X
Pass Tests?
Figure 11: Case study example of APPS for Program Synthosis (NL2Code). The problematic snippets are highlighted in red.
26


Reviews:
[{"review_id": "E3cEGWrj9Y", "reviewer": "Reviewer_F6jH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nUsing better RL algorithm and reward functions that are more tailored to code generation is beneficial for improving the performance of language models for code.\n\nWeaknesses:\n\nIn general, I think the novelty of this work is very limited. The approach is mostly a combination of existing techniques, including PPO, compiler feedback, and using AST and dataflow graphs to provide more training signals. Despite that it shows better performance than CodeRL on several benchmarks, the improvement is not significant in many cases.\n\nOn the other hand, some experiments are not convincing, and the setup should be improved.\n\n1. For code translation, I don't think CodeBLEU score is a convincing evaluation metric. It is better to evaluate on benchmarks with unit tests, such as Transcoder [1] and HumanEval-X [2], and report execution accuracies.\n\n2. I would like to see ablation studies of different reward design and RL algorithms on program synthesis tasks. \n\n3. The experimental results are not compared to the SOTA models. Specifically, I feel that the reported accuracies on MBPP are pretty low. For example, according to the Google PaLM paper [3], pass@80 on MBPP at least already reached >84%.\n\n[1] Lachaux et al., Unsupervised Translation of Programming Languages, NeurIPS 2020.\n\n[2] Zheng et al., CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X.\n\n[3] Chowdhery et al., PaLM: Scaling Language Modeling with Pathways.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes PPOCoder, which uses the PPO RL algorithm for code generation. They also design some reward functions based on the compiler feedback, and use AST and dataflow graphs to calculate syntactic and semantic matching scores between the model predictions and the ground truth programs. They evaluate their approach based on the CodeT5 model, and the experiments demonstrate good performance on several code generation and translation tasks. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. For code translation, please evaluate on benchmarks with unit tests, such as Transcoder and HumanEval-X.\n\n2. Add ablation studies of different reward design and RL algorithms on program synthesis tasks. \n\n3. Report MBPP pass@1 accuracies.\n\n4. Add comparison and discussion of experimental results using SOTA pretrained language models for code, such as gpt-3.5-turbo.", "broader_impact_concerns": "No broader impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cl9VupCAg8", "reviewer": "Reviewer_nRFW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n+ Presents the first results of applying PPO to program synthesis, highlighting among others that optimizing for compilabiltiy alone does not tend to yield strong results, perhaps due to the sparsity of this signal.\n+ Results are generally positive, primarily in terms of generating (far) more compilable programs than baselines, and sometimes by improving results on other metrics, such as pass rate on the APPS benchmark.\n\n**Weaknesses:**\n\n- The approach introduces what amounts to strong supervision on the AST and DFG structure of the generated programs relative to a single target. This significantly undermines the point of using reinforcement learning by imposing the need for a supervised dataset and biasing the training signal towards replicating low-level attributes of its training samples that are only very weakly tied to the real objective of compilation. As such, contrary to the statement in the abstract, the resulting model appears to benefit substantially from supervision on the desired program structure. The work couples this supervision based on AST and DFG matching with the goal of achieving \"syntactic and functional correctness\", which further highlights the disconnect: the ground-truth programs are functionally and syntactically correct, but that does not imply that the model should learn to generate programs that have largely similar ASTs and DFGs in order to realize these attributes. One of the key motivations for using the pass@K metric is that valid solutions to program synthesis challenges can be implemented very differently. The work also includes no comparisons with some natural alternatives, including supervised approaches that ignore the compilability score (which seems to hurt the model, Fig. 3(a)) or hybrid ones that involve direct supervision on the AST and DFG while using PPO to optimize for compilability. Finally, the choice of DFG and AST scoring functions seem rather ad-hoc. For instance, using only the target function size as the denominator may heavily incentivize PPOCoder to generate very large functions, to maximize overlap. Another ablation that is missing, given the apparent harm of including the compiler feedback just by itself, is one similar to Fig. 3(a) in which all components are added one by one but without the \"+cs\" part.\n- The evaluation involves some questionable decisions. A prominent example is that PPOCoder was fine-tuned on top of an already fine-tuned CodeT5, seemingly in all conditions. That is, CodeT5 is first fine-tuned on datasets such as CodeSearchNet, and only then is PPOCoder fine-tuned on top of this model. This leads to a skewed comparisons, wherein PPOCoder has consistently been trained on the dataset for longer than its main competitor. Another issue is the absence of code-specific large language models in Tab. 3, such as CodeGen. Even the 2B checkpoint would be a much more reasonable baseline than GPT-Neo, which was mostly not trained on code. The results in this table are also somewhat perplexing, suggesting that both CodeRL and PPOCoder widely outperform far larger baselines such as AlphaCode and Codex. I suppose this may be in part due to the use of fine-tuning for some models while Codex and GPT-3 are only prompted (Appendix A.3).\n- The focus on compilability makes the contribution relatively narrow. The resulting tool often achieves high compilation rates but comparatively much more modest performance improvements on other metrics. Compilability is also not a particularly impactful optimization target; models like CodeT5 tend to generate compilable outputs at high enough rates (in this work, mostly more than 50%) that sampling could help generate a large set of compilable outputs (which is straightforward since compilability is easy to confirm). A much more interesting objective is yielding functionally correct code, as measured by, e.g., passing unit tests (and certainly not by matching DFGs).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work applies PPO to program synthesis. Specifically, it introduces an optimization goal that balances generating programs that compile with imitating the AST and DFG structure of a reference program in order to provide more fine-grained guidance, showing that only optimizing for compilability tends to yield poor results. The results, based on fine-tuning a pretrained CodeT5 model, suggests that the combined model is particularly capable at generating programs that compile, which yields modest benefits along other metrics.\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Considering the weaknesses stated above, the work should position its contributions more conservatively, improve its evaluation, and include more ablations. Primarily, it needs to highlight that the method it proposes involves significant supervision on the program structure at a fairly low level (esp. matching AST subtrees) rather than optimizing for compilability directly. It should also enrich its set of ablations to (among others, see the above) ensure that simple direct supervision (training to replicate the precise AST and DFG) would not work better for this task, given the apparent negative impact of including compilability reported in Fig. 3(a).\n\nSecondly, it should expand the set of baselines compared in at least one place (adding CodeGen to the APPS comparison). It should either undo or note as a potentially significant caveat the choice to fine-tuning PPOCoder on top of its closest competitor (CodeT5) after fine-tuning the latter. Undoing here means fine-tuning PPOCoder on the originally pretrained CodeT5 so it is compared under the exact same conditions as the CodeT5 model being fine-tuned for each task.\n\nFinally, it should articulate more clearly that the performance improvement primarily manifests as a gain in compilation rates, not necessarily in other downstream task specific metrics. This is particularly true in Tab. 1, where the exact match scores are virtually identical with CodeT5 despite having trained longer on this dataset, and Tab. 2, where the global average CodeBLEU rate of CodeT5 is less than 0.1 points below that of PPOCoder, again despite using less training time.\n\n**Other/minor suggested changes:**\n\n- Please correct the citation style in \"Authors of (Chen et al., 2021c)\" to \"Chen et al. (2021c)\"\n- Adjust this statement: \"Unlike text generation, code generation requires not only syntactic but also functional correctness\" -- syntactic correctness is not normally the main objective in text generation, and while functional correctness is not defined in that space, factuality (arguably the closest parallel) certainly is.\n- Please elaborate on the motivation behind, and ideally provide experimental evidence for, the specific choice of reward values in Eq. 8, in particular the use of -0.3 and -0.6.\n- Please justify the choice for using pass@80 specifically in Table 4\n", "broader_impact_concerns": "The work does not include a broader impact statement and I did not note concerns that would warrant adding one.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wRGCUnQbUM", "reviewer": "Reviewer_aDTf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* (~) p1: \"given that the generated code is intended for machine execution as opposed to human comprehension\": I fundamentally disagree with this statement. Any code meant to be used more than once is also intented to be read by other humans, which has substantial consequences for how to evaluate it.\n* (-) I believe Sect. 2.2 to misrepresent the state of the art: (a) a number of models that always produce syntactically valid code exist (e.g., https://arxiv.org/abs/1401.0514, https://proceedings.mlr.press/v48/bielik16.html, https://arxiv.org/abs/1611.01855, ...), but these have been supplanted by approaches based on large language models, and (b) most LLMs have no problem generating syntactically valid code. The only evidence for the claim that this is an important issue is provided in the introduction, in the form of a citation of Sequencer, claiming \"that up to 70% of codes generated by these models can be non-compilable\". However, this is taken from a very specific experiment in the Sequencer paper, which is from 2019 and uses a two layer LSTM; clearly not an LLM.\n* (-) Sect. 4: The results here are very small gains over the CodeT5 baseline (when not losing against it), whose significance cannot be interpreted as no error bars are provided. Given that these experiments are based on sampling from the trained models, error bars are crucial. Finally, the entire section does not discuss the computational cost of the proposed method, and in particular, whether spending equal compute on continued training of the underlying CodeT5 baseline would yield better results.\n* (-) Sect. 4.4: The ablations are performed only on the \"compilation rate\" metric, whose value (for users) is not clear; it is however explicitly built into the reward function, and so it remains unclear how meaningful this is.\n* (-) Sect. 4.4: Fig. 3 tells us that using a reward term only based on whether the predicted program compiles _reduces_ the rate at which predicted programs compile. I would expect some analysis of the causes here, rather than just moving to \"integrating it with the KL-divergence penalty\".\n* (-) Overall writing is poor, with many phrases often repeating and cluttering the text (e.g., that PPOCoder uses \"compiler feedback\" and \"matching scores\" is stated repeatedly, without going into any details)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "An adaptation of PPO to the code generation setting is presented. To this end, a reward function consisting of 4 terms (execution feedback; matching of AST subtrees; matching of data flow edges; divergence from underlying code model) is introduced. Experiments on three datasets (code completion, code translation, program synthesis) show that the method minimally improves on the underlying base model.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- p2: \"eight programming languages However,\" -> Missing \".\"\n- Sect. 2.3 misattributes work (RLHF was not introduced by InstructGPT, as that paper itself says!) and contains sentences that are not parseable (what is \"matching scores between the AST sub-trees and DFG edges\" supposed to mean?). Please rework this.\n- p6 (and following): $R_{dfg}$ has ugly spacing after the $f$. Please use $R_{\\mathit{dfg}}$ (R_{\\mathit{dfg}}).\n- Eq (7): Please reformat so that the is-end-of-token term is on one line, and the other case on the next one - this is unnecessarily hard to parse.\n- Sect. 3.2.1: Given that paper explicitly discusses Python and PHP: What does \"compiler signal\" mean here - that the code is successfully parsed by python/php? Is type checking used?\n- Overall, Sect. 3 spends substantial space on recapping fairly standard RL/PPO bits, and misses many details on how the reward function (the core contribution of the paper!) is actually implemented. Please rearrange this, moving standard components into appendix and expanding on the novel contribution.\n- Sect. 3.2.2: The choice of $R_{ast}$ is not explained - why is token similarity not sufficient? Using the AST explicitly violates the author's aim to be language-agnostic, after all. The given intuition (\"token missing and data type errors\") would apply on the token level as well. Please explain.\n- Sect. 3.2.3: This section misses very important details, foremost how the DFG is computed (given that this is non-trivial in heap-using programs with aliasing) and how DFG edges are matched (finding a graph isomorphism is after all very much non-trivial). Please explain.\n- Sect. 4.1: Details such as the size of the used CodeT5 model (60M? 220M? 770M?) are missing. Please add.\n- p10: \"absence of pointers and memory management constructs, which can be a source of errors in languages like C++ and C#.\" - C# has neither pointers nor explicit memory management, please fix this.\n- Table 3: Explain what the \"All\" column represents.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper isn't perfect: the novelty is limited, and there are a few lingering concerns about the baselines used for comparison. However, the overall idea of augmenting language models of code with RL is promising. The paper also delivers an interesting insight: making an ML
model more likely to generate compilable programs doesn't make it more effective at generating correct programs. Finally, the reviewers
appreciate the additional work that the authors put in after the reviews came in and trust the authors to fix the remaining issues (in particular, the discussion of prior work -- see the reviews for more details). Given all this, I am recommending acceptance. Please make sure to incorporate the feedback in the reviews into the final version.
Decision: Accept (poster)