RLTF: Reinforcement Learning from Unit Test Feedback
Authors: Jiate Liu
Yiqin Zhu
Kaiwen Xiao
QIANG FU
Xiao Han
Yang Wei
Deheng Ye
Venue: Accepted by TMLR ()

Abstract: The goal of program synthesis, or code generation, is to generate executable code based on given descriptions. Recently, there has been an increasing number of studies employing reinforcement learning (RL) to improve the performance of large language models (LLMs) for code. 
However, some of the current representative RL methods have only used offline frameworks, limiting the exploration of new sample spaces. 
Additionally, the utilization of unit test signals is limited, not accounting for specific error locations within the code. 
To address these issues, we proposed RLTF, i.e., Reinforcement Learning from Unit Test Feedback, a novel online RL framework with unit test feedback of multi-granularity for refining code LLMs. 
Our approach generates data in real-time during training and simultaneously utilizes fine-grained feedback signals to guide the model towards producing higher-quality code.
Extensive experiments show that RLTF achieves state-of-the-art performance on the APPS and the MBPP benchmarks.
Our code is available at: \url{https://github.com/Zyq-scut/RLTF}.

Published in Transactions on Machine Learning Research (11/2023)
RLTF: Reinforcement Learning from Unit Test Feedback
Jiate Liu' Tencent
jiateka@@lencenl.cco
Yiqin Zhu' Tencent
yiginsshu@@eeneen..com
Kaiwen Xiao
loktarsiaao@Heeccnll.ccom
Tencent
Qiang Fu Tencent
leomfu@@tencenl.ccmm
Xiao Han
haroldhan@@eencent.ccom
Tencent
Wei Yang Tencent
willyang@@tenncent.comm
Deheng Yo Tencent
dericyee@lencent.coom
Reviewed on OpenReview:
Abstract
The goal of program synthesis, or code generation, is to generate executable code based on given descriptions. Recently, there has been an increasing number of studies employing reinforcement learning (RL) improve the performance of large language models (LLMs) for code. However, some of the current representative RL methods have only used offline frameworks, limiting the exploration of new sample spaces. Additionally, the utilization of unit test signals is limited, not accounting for specific error locations within the code. To ad- dress these issues, we proposed RLTF, i.e., Reinforcement Learning from Unit Test Feedback, a novel online RT. framework with unit test feedback of multi-granularity for refining code LLMs. Our approach generates data in real-time during training and simultaneously utilizes fine-grained feedback signals guide the model towards producing higher-quality code. Ex- lensive experiments show that RLTF achieves state-of-the-arl performance on the APPS and the MBPP benchmarks. Our code is available at: https: :/tg://gttte hegggtttbbb... com/Zyq-ssut/RLTT
1 Introduction
Program synthesis, or code generation, involves creating an executable program that solves a given problem. This rescarch area has gained attention due its potential improve productivity and accossibility programming. An AI model that can generate programs based on human requirements could significantly transform programming tools and workflows (Zan et. al. 2023
"Co-first authors. 'Corresponding author.

---

Published in Transactions on Machine Learning Research (11/2023)
Recently, there has been significant increase in the development of large language models (LLMs) built Transformer (Vaswani et al. 2017) architectures, trained on large unlabelled code datasets. These formidable models possess the power to generate code without explicit programming instructions and have shown remarkable results in program synthesis. Codex (Chen et al. 2021 is a noteworthy example of an LLM with 12 billion parameters that can successfully solve over 70% of complex Python programming challenges created by humans. Morcover, these models have been proven to enhance productivity and coding effectiveness in real-world applications. Since Codex's emergence, many other LLMs tailored for the code domain have appeared, ranging in model size from millions to billions of parameters. AlphaCode (Li et al.; 2022 for instance, aspires to address competitive-level programming challenges, while InCoder (Fried et al. 2022)
enables code insertion at arbitrary junctures utilizing bidirectional contexts. Other acclaimed models include CodeT5 (Wang et. al., 2021), CodeGen (Nijkamp et al., 2022), PaLM-Coder (Chowdhery et al. 2022) PanGu-Coder (Christopoulou et al.) 2022), CodeGeex (Zheng et al. 2023), and SantaCoder (Allal et al., 2023). As the size of these LLMs increases, they demonstrate emergent competencies, including human-like
programming prowess and debugging aptitude (Zhang et al.. 2022 Saunders el al. 2022).
While LLMs have shown promising results in basic programming tasks, there is still progress to be made in tackling more challenging problems such as program competitions. Additionally, pretrained code models can be limited by their reliance on NLP's self-supervised masked language modeling (MLM) and may struggle with ensuring the syntactic and functional correctness of generated code. To improve code generation performance, very recently, reinforcement learning (RL) based algorithms for better utilizing code LLMs have been proposed. For example, CodeRL (Le et al. 2022. has explored the integration of reinforcement learning with unit test signals to fine-tune program synthesis models, and proposes a post-process method called "Critic Sampling" to further improve the performance of the model. PPOCoder (Shojace et al. 2023) uses the Proximal Policy Optimization algorithm to improve CodeRL, although it does not achieve better results on the competition algorithm benchmark APPS, compared to CodeRL.
It is worth noting that some of existing RL-based methods (Le ct al. 2022 are offline, mcaning that they do not interact with the environment dynamically during training, but rather learn from pre-collected data. This can result in RL training quality being limited by the diversity of the dataset and hinder the effective exploration of the environment (Fujimoto et al., 2019 Kumar et al.) 2019). Studies have also shown that offline RL can face issues such as distribution shift (Fujimoto et al.) 2019 Schrittwieser et al., 2021), leading to unstable training and suboptimal policy performance. In contrast, online RI. refers to the scenario where an agent can interact with the environment in real-time, selecting actions and observing their impact to obtain state and reward signals. Compared to offline RL, online RL training is often more stable, enables better exploration of the environment, and leads to the development of more optimal policies. On the other hand, while existing RI-based methods do employ the results of unit tests as fcedback signals, their implereentation is rather simple and coarse-grained. They assign the same reward to the entire episode (a complete code segment) based on the program's 's submission result (true, false, runtime error, or compiler error), without considering that a part of the code may be correct while another part contains bugs. As a result, these approaches fail to capture the nuances in identifying individual code components that contribute to the overall functionality and fixing bugs specific to certain parts of the code. Additionally, in rast. lo domains such as healthcare, autonomous driving, and ct al. 2023), where real-time interaction costs are high or data difficult obtain in real-time, the foodback from unit tests in the program synthesis task does not incur notable costs. This can potentially make the use of an online framework even more beneficial in this context.
alligne in LLMs (Levine et al. 2020 Prudencio
To address the aforementioned issues, we introduce RLTF, a novel online framework with multi-granulart unit test feedback that enhances pretrained LLMs for program synthesis using reinforcement learning. First, we develop an online framework, which consists of two main parts: one for training the model and updating the parameters, and the other for generating programs using the latest model parameters and appending them to the online buffer for the subsequent model training. This framework enables the model to obtain real-time unit test feedback. Additionally, it provides diverse set of samples, enabling the code L.M agent to explore the environment effectively while also promoting training stability. Sccond, we have analyzed the distribution of error types in programs and extracted more detailed information from unit test feedback for training, such as "fine-grained feedback" and "adaptive feedback". Fine-grained feedback categorizes
2

---

Published in Transactions on Machine Learning Research (11/2023)
Table 1: RLTF and PPOCoder adopt an online framework, which allows the model to generate new samples in real-time, while CodeRL rely solely on offiine frameworks. RLTF extracts more detailed information from unit test feedback for training, such as fine-grained feedback and adaptive feedback. Both RLTF and CodeRL use post-processing technique called Critic Sampling, which first introduced in (Le et al. 2022).
Feature
CodeRL Le et al.
2022 PPOCoder (Shojaee al.
2023
RLTF
Framework
ofline
online
online
Supervised Learning
Coarso-grained Feodback
Fine- grae-gr Feedback
Adapative Feedback
Critic Sampling
errors based on their reported information and location, penalizing the specific erroneous parts of the code accordingly. Adaptive feedback provides varying rewards to programs based on the ratio of test cases they pass. More detailed information can be found in Sec. 3.3.2
The main contributions of our paper are as follows:
We propose RLTF, novel online framework for the program synthesis task. By generating new samples in real-time during the training process and leveraging unit test results as feedback signals, it improves overall model performance. This approach enables models to learn the specifies of code errors and enhance their performance accordingly.
Built upon this framework, we develop multi- gulti-granularity feedback that is automatically extracted from unit test. To expand, we introduce coarse-grained and fine-grained feedbacks applicable to programs with errors, aimed al punishing the specific segments code where the errors appear. For programs that do not pass all test cases, we propose an adaptive feedback mechanism that assigns varying penaltics based on the ratio of passed test cases.
For the program synthesis task, our method achieves state-of-the-art results the APPS and MBPP benchmarks, and detailed ablation study demonstrates the effectiveness of our approach. Addi- tionally, we perform tests on different LLMs (c.g., CodcT5, CodeGen), illustrating the robustness of our method and its applicability to different base models.
2
Related Work
2.1
Pretrained LLMs for Program Synthesis
Recent research has delved into leveraging pretrained large language models (LLMs) from the natural lan- guage processing (NLP) field to automate program synthesis tasks, using vast-scale code corpus data mined from open-source repositories. Notably, there are several prominent examples of such pretrained models in- cluding the encoder-only CodeBERT (Feng et al. 2020). decoder-only CodeGPT (Lu et al. 2021 CodeGen NNikaamp et al.) 2022), PaLM-Codeer [Chowdhery et al. 2022). PanGu-Coder (Christopoulou et al. 2022 CodeGeex (Zheng et al. 2023). and SantaCoder [Ala el al.] 2023). as well as encoder-decoder transformer architectures like PLABRT (Ahmad et al.) 2021 and CodeT5 (Wang et al.) (021).. These pretrained prob- abilistic language (PL) models are already capable of generating code that appears visually impressive and well-structured. However, these models still face challenges guarantceing the syntactic and functional
correctness of the generated codes (Chen al. 2021 Hendrycks et al. 2021 Shojace al. 2023
3

---

Published in Transactions on Machine Learning Research (11/2023)
gradient
RL Loss
Online buffer
1, Program Pass!
Problem
LLM
gradient
Loss
Ground Truth Programs
For Problem 1, Generated Program Failure!
shared weight
Unit Test
For Problem 1002, Generated Program 17 Error! Index Error line 27.
Generate Programs
Problem
LLM
For Problem 1078, Generated Program Error! Syntax Error in line 34.
Compiler
(a) Online reinforcement learning framework.
Online buffer
Fresh data
delete
update
(b) Online buffer workflow.
The overall framework of the proposed RLTF. (a) Online reinforcement learning framework: Two LLMs with shared weight are utilized in the training process. One generates the target program and interacts with the compiler to produce a training data pair, which is then stored in the online buffer. The other one utilizes ground truth data and online buffer data to calculate loss and update the model weights through gradient feedback. (b) Online buffer workflow: Online buffers are utilized to retain the newly generated data for RL training. As fresh data is received, old data is deleted. Here SI. refers to supervised learning and RL. refers to reinforcement learning.
Figure 1:
2.2 Reinforcement Learning for Program Synthesis
Reinforcement Learning (RL) is method of learning the optimal policy by obtaining reward signals from the real environment (Sutton et al. 1998) In recent years, RL has shown promise in optimizing non-differentiable metrics for sequence generation lasks, such improving BLEU and ROUGE, scores in summarization and translation models respectively (Husain et al.,
2019 Ren et al. 2020). Unlike text generation, program synthesis requires both syntactic and functional accuracy since the generated code must pass compilation and rnrcution-guiden approaches and RL-based fine-tuning mechanisms have been employed to enhance the quality generated codes. For instance, CodeRL. [Le et al. 2022 investigates the fusion of reinforcement learning and unit test signals enhance program synthesis models, introducing a post-processing technique named Critic Sampling that further boosts model performance. PPOCoder (Shojaee et al.) 2023 employs the Proximal Policy Optimization algorithm from reinforcement lcarning to refine CodeRL. However, existing RI-based methods still face several limitations. First, some of the existing RL methods have only used oflline frameworks, which constrain their ability to explore new sample spaces. Moreover, the current techniques employing unit test signals are relatively simple, neglecting consider specific error information and locations within the code. Our RLTF method adopts an online framework and incorporates multi-graincs feedback for model training. Overall, the differences between RLTF and existing approaches such CodeRL. and PPOCoder summarized in Table 1
unit tests (Hendrycks et al. 2021 Chen et al.
2021 Recently,
4

---

Published in Transactions on Machine Learning Research (11/2023)
3 Approach
3.1 Task Definition
Program synthesis is an automated process that generates computer code W based on a high-level description of desired behavior D. This process aims to increase the likelihood of generaling a code that can solve a given problem, expressed a conditional probability P(W
The task can be expressed as sequencc-to-sequeence problem, and the objective is to maximize the condi- tional probability of generaling the correct output, given the input and the LLM model parameters:
max P(WID,O) max II p(w. D,0, w1+-1)
(1)
where the parameters the LLM model. The LLM model is used to capture the conditional distribution, and it is trained using set of input-output pairs. During training, the parameters are adjusted maximize the likelihood of generating the correct output for a given input.
3.2 Online Reinforcement Learning Framework
While supervised learning can yield valuable outcomes, the vast search space involved in program synthesis make it challenging to acquire comprehensive dataset. As such, propose online reinforcement learning framework, which continuously improves the code ability of LLM through online generated training samples and fine-gratd -re-grained feedback provided by the compiler, to circumvent this predicament. Specifically, our framework incorporates an online buffer during the training process, which crucial in improving the model's effectiveness. The buller maintains a dynamic flow of data pairs that are utilized in reinforcement learning training in real-time. Each data pair comprises a problem description D, the most recent model- generated code W, and feedback from the compiler FBW... This approach allows the model to adapt to changes in the data distribution and facilitate more efficient learning. The present framework incorporates the actor and critic in training, similar to the methodology employed in CodeRI. (Le et al. 2022 and PPOCoder (Shojaee et al. 2023
The overall framework is shown in Fig. I The training process involves two LLMs with shared weights. The first LLM updates model parameters using data obtained from buffer for training. while the second LLM collaborates with the compiler to dynamically generate code, evaluate its soundness, generate data pairs, and continuously refresh buffer in real-time. The incorporation of continuously updated buffer ensures the promptness of the data used for training, thereby enabling the model to explore wider sample space comprchensively.
3.3 Reinforcement Learning from Unit Test Feedback
We propose RLTF, a method to improve the quality of synthesized codes by exploiting unit test feedback to explore the target space. Following the reinforcement algorithm /Sutton et al. (1998] & Barto (0088) we construct reinforcement learning loss with the explored code W as follows:
E
Ltt = RR(W) ( P(WID.O) = - -R(W) Σ logp( 01.-1)
(2)
where R(+) is the reward coefficient, S and E represent the start and end positions of the penalized code snippets. Their values are determined by different feedback categories.
3.3.1 Feedback Category
Unit test feedback FB(W) can be obtained from the program compiler and can be categorized into three main groups, namely Error, Failure, and Pass.
5

---

Published in Transactions on Machine Learning Research (11/2023)
Table 2: Definition of sub-error types in Python. We divide different sub-errors into three categories based on their characteristies:
U Ugtobol
Utines and U Vignore-
Sub-error
Description
Category
Syntax Error
Code contains synlax errors that cause the compilation to fail Index operation is out of bounds
Uline or Uglobat
Index Error
Uiine
Type Error
An operation or function was applied to an object of an inappropriate type An operation or function received an argument with the correct type but with an inappropriate value
Uline
Value Error
Ufiee
EOF Error
The input() function encountered an end-of-file condition (EOF) without reading any data
Utive
Timeout Error
Code execution time exceeds time limit A local or global name is not defined mapping (dictionary) key is not found in the set of existing keys
Ugtobat
Name Error
Utine
Key Error
Uiine
Import Error
The imported package is not found
Uiine
Zivisivision Error
The second argument division or modulo operation is zero
Uiine
Recursion Error
Code execution recursive operation exceeds the maximum limit Triple quotes are incomplete
U global
Tropte-quoted Error
U Vignore
Indentation Error
Wrong indentation format.
Uignore
Else
Utiee
Error refers to situation where the generated code fails to compile results in errors such exceptions that prevent the program from functioning normally. This category can be further subdivided into syntax errors, index errors, type errors, among others.
Failure indicates that the generated code can be compiled or run, but it fails to meet the expected functional or performance requirements. This situation requires additional improvement or optimization of the program. Code belonging to this calegory fails at least one unit test.
Pass implies that the generated code be compiled and run successfully and meets the expected functional and performance requirements. This category indicates that the program has passed all unit tests and can be used without further modification.
3.3.2 Multi-granularity Feedback
Coarse-grained Feedback. The coarse-grained leedback serves as an incentive mechanism for the language model to enhance the probability producing accurate code while reducing the likelihood of generating erroneous code. We use the same setting as CodeRL (Le et al. 2022) and PPOCoder (Shojaee et al., 2023):
1.0, FB(W) is pass
0.3, FB(W' failure
Reoerse(W]
Seaarse 0, Frcarse T
(3)
0.6, FB(W) is error except syntax error
1.0, FB(W) syntax error
The value of Recerse is determined by complier foodback FB(W). The start position Scoarse and end position Eccorse are always and T.
Fine-grained Feedback. The coarse-grained feedback serves to inform the model that it has made error, but does not indicate the specific location of the error. In contrast, the purpose of the fine-grained feedback is to provide the model with information regarding the specific reasons for its errors, with the goal of reducing the likelihood of future errors. The reasons for code errors can vary widely, sometimes stemming from specific errors in the code (such as the use of undefined variables), and at other times from issues with the overall logic of the code (such as recursion errors in Python). For the former, we aim for the model address the error the current line, while for the latter, hope the model re-evaluate and revise the

---

Published in Transactions on Machine Learning Research (11/2023)
code. To this end, we have developed set of rewards for different error subtypes, which can be found in Table 2
Typically, we are able to identify the erroneous code snippets from compiler prompts. We divide the error codes into three categories (Uglobal: Ulanes Uignore) based on their specific subtypes. The gglobal category means that penalties will be applied the entire code generated, usually because of a logic issue that affects the entire code segment, and cannot be exactly identified on a specific line of code. The Utine category applies penalties to the specific line of code that caused the error because fixing this line of code will prevent that error from recurring. The Uignore category temporarily avoids punishment because the reasons for the error are numerous and cannot be accurately located. In cases of Syntax Error, if it is caused by unfinished code due to the maximun token length limit, it is classified as Utoobat, otherwise, it is Utine-
Riine( {
0.0, if W Uignore 0.3, else
tiine start, if W Utine
{ fline end, if W E Utine eless
Syine
,Eiine
0, else
We set Rfine(W) 0.0 for Uignore; otherwisc -0.3. tliee start and fline end are the start and end position of the line that complier leedback.
Adaptive Feedback. Adaptive feedback designed to motivate the model to generate code that can successfully pass maximum number of test samples. To attain this objective, the reward value of adaptive feedback is directly proportional to the number of test samples that the generated code can pass. The specific configuration of the reward is as follows:
N
pass
Radoptive 003 1.3*
Sodoptive
Eadaptive T
(5)
Npase + Nfst
The value Radaptine is positively correlated with the pass rate of unit tests. The start and end position are always and T.
3.4 Optimization Objective
In order to further enhance the performance of the pre-trained LLM via RLTF, the following optimization objectives can be used for fine-tuning:
Iatotal Test + Lecoarse + Lifine + Leadaptive
(6)
where Lst is supervised learning loss which makes the training process stable. We adopt cross-entropy loss as Lal as follows:
Laj - log P(WID,.) = Σ log log (Dr (1
And Lecarse: L Lines Ladaptive are variants of the reinforcement learning loss shown in Eq representing three types of feedback granularity, which follows:

---

Published in Transactions on Machine Learning Research (11/2023)
Brcorse
Lecarse - -Reoarse (W)
Recarse (Whaseline)
logp(10((D, 0,00,,,,001----000 10--))
(8)
tSecarve
Eyine
Lfine = -aBgino(W)
Σ
logp(1.r)D, 012-11)
(9)
Budaptice
Ludaptive (Rodaptive (W) Radaptive Waseline) ))
Σ
logp(10/1D,6) 01,01111)
(10)
Sudaytiive
To address the amplitude fluctuation fine-grained feedback caused by variable values S and E, we use an adaptive weight that balances the equation. This weight is calculated as a Eyine- "rame-Sycne" making the fine-grained leedback equivalent to coarse-grained feedback and adaptive feedback. Here W Wasellnee is the baseline dynamically updated during the training process, which represents the historical optimal code generated by the model under the natural language description D. This item indicates that want the model to continuously outperform its own historical version. For fine-grained leedback, its baseline is hard to define, so we do not add this item to its loss function.
4 Experiments
Our baselines the stale-of-the-ard works on incorporating RL with code LLMs (PPOCoder (Shojaee al. 2023 and CodeRL. (Le et al. 2022 ) and the same benchmarks and setting theirs for evaluation.
4.1
Benchmarks
APPS Benchmark. We first evaluate using the challenging APPS (Automated Programming Progress Standard) program synthesis benchmark presented by (Hendrycks et al. 2021), as it features a diverse range of coding problems from various coding websites, presenting differing levels of difficulty. The benchmark consists of total of 10,000 coding problems, with equal train-test. split. On average, each problem is accompanied by 23.2 accurate Python programs and 21.2 unit tests. The mcan length of each problem is 293.2 words, whereas the average program length is 18.0 lines. The dalasel is classified into three difficulty levels: Introductory (3,699; train/test = 2,639/1, (,,00, Interview (5,000; train/test = 2, 2,000/3,00 (000/,,000), and Competition (1,361; train/test = 361/1 (6,00). Typically, cach sample has 20 unit tests to assess the functional correctness of the programs. We adhere the same preprocessing steps as those in (Hendrycks et al. 2021 for generating input sequences based on problem descriptions.
Note that, during the implementation of our online framework, encountered an issuc in the APPS open- source code, specifically in the unit test portion. If the generated code raised "segmentation fault" error, it could not be bypassed using "try" and "except", resulting in the main program getting stuck and disrupting the sample generation process. To address this issue, we modified all unit test code procedures to be executed through subprocess calls, ensuring the main program does not get stuck and allowing the online sample generation process to run smoothly. We believe it necessary expose this to the community for reproducibility considerations.
For the APPS benchmark, employ the RLTF framework to fine-tune the pretrained CodeT5 model. We utilized machine equipped with 8 NVIDIA V100 GPUs, each with 32GB of memory, for training purposes. Each GPU carried batch size of 32, and a lcarning rate of 2c-6 was employed. The training process look approximately 24 hours. Concurrently, three additional machines with similar 8-card V100 GPU configrratooss were used to generate the latest samples. We updated the online buffer every 50 steps. Following the same approach as CodeRL, half of the stops were for SL. training, while the other half focused on RL training. The length of the online buffer was the initial model, the training process officially commenced, and the online buffer was updated a queue.
6400. After generating 6400 samples using
8

---

Published in Transactions on Machine Learning Research (11/2023)
Table 3: Quantitative evaluation on APPS benchmark.
pass@1
passe
pa0s$11000
Metlod
C8
Iatro
Inter
Comp
all
Intro
Inter
Comp
all
Intro
Inter
Comp
all
Codex
12H
w/o
4.14
0.14
0.02
0.92
9.65
0.51
0.39
2.25
25.02
3.70
3.23
7.87
AlphaCade
w/o
7.06
8.09
GPT3
w/o
9.20
0.03
GPT2
0.1.
w/o
1.00
0.33
0.00
0.40
2.70
0.73
0.00
1.02
GPT2
1.5B
w/o
1:30
0.70
0.00
0.68
3.00
1.03
0.00
1.58
27.
9.53
11:40
13.76
GPT-Neo
2.7B
3.90
0.57
1.12
5.50
0.80
0.00
27.90
9.83
11:40
13.76
CodeRL
770M
w/a
0.78
1.366
17.78
PPOCoder (eriiule(rggnnn)))
770M
w/aa
5.20
1.00
9:30
2.50
1.20
3.56
35.20
13.35
1.3.90
17:77
PPOCoder(pextt)
770M
w/a
4.06
0.79
0.15
1.32
9.977
2.06
0.70
3.37
35.42
13.37
1.3:65
17.84
RLTF
770M
w/o
4.16
0.97
0.20
1.45
10:12
2.65
0.82
3.78
38.30
15.13
13.90
19.92
CodefLL
770M
7.08
1.66
0.75
2.69
16.37
4.95
2.84
6.81
40.00
15.67
17.90
20.98
RLTF
2.28
18.60
5.57
3.70
16.80
20.32
During testing, we employed Nucleus sampling with a top value of 0.95 and set the temperature parameter to 0.6.
MBPP Benchmark. To further evaluate our framework, we also employ an additional, smaller, and simpler Python program synthesis dataset called MBPP (Mostly Basic Programming Problems), introduced by (Austin al. 2021). The dataset consists of 974 instances, with 374 instances for training, 90 instances for validation, and 500 instances for testing, while reserving 10 instances for few-shot learning evaluations. The problems are predominantly brief, often featuring only single sentence of natural language descriptions. Each problem is accompanied by one correct solution (averaging 6.8 lines of code) and three unit tests, which are assert statements to verify functional correctness. In contrast to APPS, MBPP's unit tests are not hidden and are explicilly integrated into the source sequences for program synthesis models. Although this occasionally encourages models to overfit the assert statements through hard-coding an if-expression, we adhere the same source sequence construction as previous work ensure fair comparison with basclines.
Specifically, use the same prompt format (Austin et al. 2021 to prepare the input sequence, which is composed of problem descriptions + "Your code should satisfy these tests:" + three assert statements. We experiment with the MBPP datasct in a zero-shot setting, as the same as CodeRL. While generating samples, we also used Nucleus sampling with a top value of 0.95 and set the temperature parameter to 1.2.
4.2 Quantitative Evaluation APPS
For a fair comparison, we employ the CodeT5 770M as the base model, similarly to CodeRL (Le el al.
2022
PPOCoder(Shh) (ShojeeerSejeeeeeee et al.] 2023). Table 3 presents the overall results of our RLTF approach based on the
CodeT5 model on the APPS benchmark. We compared our method with larger models such as Codex CCen
et al. ). AldhaCode (Li et al. 2022 GPT2 (Radford et al. 2019), GPT3 (Brown et al. 2020 GPT- Neo Black et al. 2021 1, and other CodeT5-based methods like CodeRL. (Te et al. 2022 and PPOCoder
(Shojaee et al.] 2023). Note that by default, results of pretrained LMs (except for Codex and GPT3) are from models fine- Lune on APPS using the standard loss Lee only, and results of CodeRL, PPOCoder, RLTF are from models fine-tuned on APPS using the both loss Lee and Lyt. The results indicate that methods basod on CodeT5 outperform those using other larger-parameter base models such as GPTs. Furthermore, employing the RLTF approach leads to additional performance improvements, surpassing other CodeT5- based methods such CodeRL and PPOCoder, thereby achieving new state-of-the-art (SOTA) in the field. It is important to note that utilized the official open-source CodeRL model and applied nucleus sampling with temperature of 0.6. We noticed that our pass01 score was lower than the result reported in their paper, while our pass55 score was higher. Similar results were also found by others and discussed at /ttps::/gitthie their paper also differ from those the CodeRL paper. Since the authors did not open-source the APPS benchmark code and models, we are uncertain about their exact evaluation process. Therefore, we used the results obtained from the open-source CodeRL model as standard and employed proportional scaling to
Regarding PPOCoder, the results reported in

---

Published in Transactions on Machine Learning Research (11/2023)
Table 4: Ablation studies: Impact of framework.
passo1 I
pass
passo 10
Framework
Intro
Inter
Comp
all
Intro
Inter
Comp
all
Intro
Inter
Comp
all
Offline
3.67
0.84
0.16
1.29
9.25
2.40
0.74
3.43
12.41
3.36
1.33
4.76
Offine+
3.71
0.93
0.19
1.34
9.28
2.51
0.85
3.53
12.60
3.50
1.50
4.92
Online
3.94
0.92
0.17
1.37
9.68
2.43
0.75
3.50
13.01
3.41
1.35
4.92
Muline++RLTH
4.16
0.97
0.20
1.45
10.12
2.65
0.82
3.78
13.59
3.67
1.45
5.21
derive the final results. To provide readers with a clear understanding of the result sources, we concurrently included the results for (oriiie)))))ginal) and PPOCoder(scale in Table Here, PPOCoder(originaal represents the results from the original paper; however, since did not directly compare against it, we have shaded it in gray. PPOCoder(scalees on the other hand, denotes the scaled results. This ensures fair comparison between the different methods, while acknowledging the discrepancies in the reported values. Even without selecting the scaled results, our performance is better than the results reported in the original paper, except for pass@1.
Additionally, we compare the performance of our model with the post-processing approach Critic Sampling, first proposed in (Le al.] 2022). Due to the unavailability of the open-source Critic Sampling code from CodeRL, we reproduced the Critic Sampling based on the details provided in the original paper. We employed both refine and repair and set M, the number of top candidates selected from program samples that fail example unit tests to 1. We also limited the maximum mumber of repair and refine iterations to 1. It is worth not explicitly mentioned in the original paper: the total number
noting that we found a crucial parameter refine/repair /repair step. When measuring pass@l, we set N = 5; for of program samples generated during each refine pass@5, we set N = 20; and for pass@ 1000, we set N 1000. Our results show that Critic Sampling achieves better performance when k 1 5 in pass@k. However, when k is larger, the improvement in performance is relatively small, which differs from the results presented in the original paper. Overall, we think Critic Sampling is post-processing method that focuses how to better use the model rather than how to better train the model. Therefore, comparing our approach with results without Critic Sampling provides a fairer evaluation.
4.3 Ablation Studies
We conduct extensive ablations to understand the effectiveness of the techniques we develop.
Impact of Framework. Table presents the ablation study comparing different training frameworks (online offline) and the use of the RLTF method. All the results have utilized the combined loss Lat and Leearse. The results show that the combination of the online framework and RLTF yields the best performance, while the offline framework without RLTF produces the poorest outcome. Applying either the online framework or RLTF method individually leads to a certain degree of improvement, validating the effectiveness of both approaches. Interestingly, we observe that the performance boost brought by the RLTF method more substantial within the online framework. This can be attributed to the real-time generation of samples in the online setting, which helps the model identify more potential errors in programs and thus facilitates better training.
Impact of Feedback. Table 5 presents the ablation study comparing different feedback combinations during reinforcement learning (RL) training. The results show that using only supervised learning yields the poorest performance. In contrast, employing reinforcement learning and progressively incorporating the rewards mentioned in Section 3 leads to incrcasingly better model performance. The best results are obtained when using combination of coarse-grained, fine-grained, and adaplive feedback, verilying the elfectiveness of these different reward designs. Additionally, we observe that the inclusion of the fine- ane-grained feedback contributes the most significant performance boost among the rewards evaluated. We also evaluate the results with all feedback except the coarse one: pass@1 1.38%, pass@5 3.58%. pass@10 5.05%. The purpose of the coarse feedback is to inform the model about the overall test results of the code (true, false, runtime
10

---

Published in Transactions on Machine Learning Research (11/2023)
Table 5: Ablation: impact of different combinations feedback.
SL
Coarse
Fine
Adaptive
pass@1
pass@5
pass@ 10
pass 100
100
pass 1000
1.30
3.39
4.68
10.51
17.80
1.37
3.50
4.92
10.63
18.31
1.41
3.67
5.10
11.08
19.32
/
1.45
3.78
5.21
11.23
19.92
Table 6: Ablation: impact of the fino-grained
leedback Rfine (W).
Table Ablation: impact of the temperature during training.
Ryine (W)
pass@]
pass
pass@
0.0
1.31
3.42
4.72
Temperature
pass
passt25
pass 10
-0.1
1.39
3.59
4.99
0.2
1.34
3.56
4.99
-0.2
1.38
3.67
5.10
0.6
1.37
3.60
5.06
-0.3
1.40
3.62
5.04
1.0
1.45
3.78
5.21
-0.4
1.39
3.68
5.06
-0.5
1.36
3.57
5.00
error, compile error). Therefore, we think it is also quite important. As can be seen, removing the coarse foedback leads to decline in the performance metrics. It is also worth noting that the performance without coarse leedback is better than the performance with only coarse leedback.
Impact of Rime (W). Table illustrates the impact of different (y) for fine-grained feedback on model training. We can observe that various values (y) result in improved model performance, with each offering better outcomes than those obtained without any penalty.
Impact of Temperature. Table demonstrates the effect of varying the temperature during the training process. A higher lemperature leads to the generation of more diverse samples, while lower temperature results in more conservative samples. We observe that the model performs better when the temperature is set to 1, and comparatively worse when set to 0.2. This suggests that generating more diverse samples during the training process helps the model explore a broader range of sample space, facilitating RL training.
Impact of Language Model. To demonstrate the robustness of the RLTF approach, we conduct exper- iments using another base model, CodeGen 2.7B, in addition to CodeTT5. As shown in Table B the results indicate that applying the RLTF method on CodeGen 2.7B also yields impressive performance, resulting in an almost 1% improvement pass@10. Notably, we find that the larger the base model, the greater the performance boost provided by RLIF. This suggests that the RLTF approach can effectively unleash the potential of different base models to generate better codes, with the impact being more pronounced when the base model size is larger.
4.4 Quantitative Evaluation MBPP
To further demonstrate the effectiveness of RLTF, we evaluate the zero-shot performance of the CodeT5 model trained with RLTF the APPS benchmark when tested the MBPP (Mostly Basic Python Problems) benchmark, shown in Table g For this evaluation, the methods based on the CodeT5 model, including CodeRL, PPOCoder, and RLTF, were asscssed in a zero-shot setting. For the GPT-based models, they were pretrained on 2.93 billion web documents using standard language modeling objective, and fine- tuned on the training set of the MBPP dataset. The results presented in the table were obtained from Figure 3 of the original paper (Austin al., 2021). The results revealed that RLTF outperformed different sizes of GPT models on the MBPP benchmark and attained a new toie---ff-hhhrrrt performance among the CodeT5-based methods.
11

---

Published in Transactions on Machine Learning Research (11/2023)
Table Ablation: different large language models as the backbone.
Language Model
RLTF
passe1
pass@5
pass@
pass 100
pass 1000
CodeT5 770M
w/o
1.30
3.39
4.68
10.51
17.80
Code'155 770M
W
1.45
3.78
5.21
11.23
19.92
CodeGen 2.7B
w/o
1.64
4.23
5.79
12.48
21.44
CodeGen 2.7B
2.04
5.04
6.80
14.17
23.96
Table 9: Quantitative evaluation on the MBPP benchmark (zero-shot).
Method
Size
state
pass
pass080
GPT
224M
fine-tuned
7.2
GPT
442M
fine-tuned
12.6
GPT
1B
fine-tuned
22.4
GPT
4B
fine-tuned
33.0
GPT
8B
fine-tuned
40.6
GPT
68B
fine-Luned
53.6
GPT
137B
fine-tuned
61.4
CodeT5 + CodeRT.
770M
zero-shot
25.7
68.1
CodeT5 + PPOCoder
770M
zero-shot
26.1
68.2
CodeT5 + RLTF
770M
zero-shot
30.4
71.3
4.5 Qualitative Analysis by Unit Test Outcomes
Figure presents the qualitative analysis by unit test outcomes before and after applying the RLTF method on the CodeGen model. For the 5000 problems in the APPS benchmark, we utilize Nucleus sampling generate an average of 20 solutions per problem and recorded their results. Figure 2a depicts the percentages of different unit lest results, as defined in Eq including error, failure, and pass. We observe that the RLTF method can reduce the proportion of programs resulting in errors and increase the proportion of programs that pass, especially for problems with introductory difficulty levels. The obscrved increase in failure rate stems from the fixing of error codes, resulting in either pass or fail outcomes. This illustrate that RLTF is more effective in addressing runtime and compiler errors compared to semantic errors, which remain more challenging. Figure 2b illususttse the percentages of different sub-errors among erroneous results before and after applying the RLTF method. Most errors show decline in proportion after using the RLTF method, particularly syntax errors. It is also noteworthy that the proportion of timeout errors exhibits minor increase, which can be attributed to the correction of other grammar-related errors resulting in timeout errors.
5 Conclusions and Future Work
In this paper, have proposed RLTF (Reinforcement Learning from unit Test Feedback), novel online RL. framework with unit test feedback of multi-granularity, for refining large language models on program synthesis tasks. Compared with existing works, our approach generates data on-the-fly during training and simultancously utilizes finer gramularity feedback signals to guide the model towards producing higher- quality code. Extensive experiments demonstrate that RLTF surpasses existing RL methods for programming and can be applicd various L.MSs, including CodcT5 and CodeGen. Furthermore, achieves stato-of-the- art performance on widely-used benchmarks including APPS and MBPP.
In the future, there are several directions to further improve RLTF. For instance, the input-output examples in the cxisting benchmarks may not be sufficiently diverse, and the programs generated using hidden input- output examples may not be the correct final code versions. Such limitations can restrict the performance of RLTF. Therefore, using LLMs to create a more diverse and accurate set input-output examples is potential rescarch direction worth exploring. Additionally, whether finer-grained feedback signals, such as those from static code analyzers, can further enhance RLTF's performance, is another possible avenue to pursue. Also, note that transferability is importance issue. The manual categorization of sub-error
12

---

Published in Transactions on Machine Learning Research (11/2023)
(9)
Failure
Pass
50.0
90.0
80
40.0
60.0
6.0
30.0
40.0
4.0
20.0
10.0
2.0
0.
0.0
0.0
Intro
Inter
Comp
Incer
Comp
AI
Intro
Inter
Comp
Percentages of unit test results.
SubError
10.0
8.0
2.0
0.0
SyntavEroo
Indextrror
Typetrror
Valuešmor
EOFéror
Timeoutarro
Nomeérror
KeyError
Importzror
Zero Dnvissrnnrrrr
Use
maklo R.IF ww HLTF
(b) Percentages of different sub-errors.
Figure 2: Qualitative analysis by unit test outcomes on CodeGen.
types we employed makes challenging to transfer RLTF to other programming languages, which should considered as another limitation of this work. As a next step, we believe the implementation of automatod calegorization will significantly enhance the applicability of RLTF in broader context.
References
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang, Unified pre-training for program understanding and generation. arXiv preprint arXiv:2103.0622 06333, 2021.
Loubna. Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferran- dis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don't 't reach for the stars! Xr preprint arXiw:2301.0395. 2023.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. anXiv preprint Xrx:222088. 07732, 2021.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale antoregressive language modeling with mesh-tensorflow. If you use this software, please cite il using these meladala, 58, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neurra information processing systems, 33:1877-1901, 2020.
Mark Chen, Jerry Tworek, Hecwoo Jun, Qiming Yuan, Henrique Ponde de Olivcira Pinto, Kaplan, Ed- wards, Burda, Joseph, Brockman, el al. Evaluating large language models trained on c20e. arXiv preprint orXio:2107.03374 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Xviio preprint arXi::204400200 2022.
13

---

Published in Transactions on Machine Learning Research (11/2023)
Fenia Christopoulon, Gerasimos Lampouras, Milan Gritta, Guchun Zhang, Yinpeng Guo, Zhongqi Li, Qi Zhang, Meng Xiao, Bo Shen, Lin Li, et al. Pangu-coder: Program synthesis with function-level language modeling. arXiv preprint arXim:2207.1132 2022.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages.
arXiv preprint arXic:2002.088 08155, 2020.
Daniel Fried, Armon Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXio:2204. 05999, 2022.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052 2062. PMLR, 2019.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint Xiv:22105 09938, 2021.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. Xiv preprinl arXiv 09009,9993,.... 2019.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Hung Le, Yue Wang, Akhilesh Decpak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mas- tering code generation through pretrained models and deep reinforcement learning. Advances in Newral Information Processing Systems, 35:21314 21328,,21 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXie preprinl arXiv:2005.016 200.00005,,643, 2020.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, (7664))111 1097, 2022.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Sryatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset. for code understanding and generation. arXie preprint or arXiv:2102.0466 2021.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. ar preprinl arXiv:2203.1344 2022.
Rafacl Figuciredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline rein- forcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Alcc Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Nya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. arXio preprint urXiv:2009.10299 2020.
William Saunders, Catherine Ych, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike, Self-critiquing models for assisting human evaluators. ar preprint arXi: 05802, 2022.
14

---

Published in Transactions on Machine Learning Research (11/2023)
Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. Advances 2021.
in Neural Information Processing Systems, $4:27580-27855
Parshin Shojace, Aneesh Jain, Sindhu Tipirneni, and Chandan K Reddy. Execution-based code generation
using deep reinforcement learning. arXiv preprint arXt::2301.38116, 2023.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.
Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Yue Wang, Weishi Wang, Shafiq Joly, and Steven CH Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXie preprint orXiv:2109, 000.20099000 2021.
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang, and Jian-Guang Lou. Large language models meet n12code: A survey, 2023.
Jialu Zhang, José Cambronero, Sumit Gulwani, Vu Le, Ruzica Piskac, Gustavo Soares, and Gust Verbruggen. Repairing bugs in python assignments using large language models. arXiv preprint W:Xiv:2209.148777 2022.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. Codegeex: A pre-trained model for code generation with multilingual evaluations humaneval-x. arvvv preprint 2xXi::2303.1750 2023.
15


Reviews:
[{"review_id": "qR0Qz9JtNa", "reviewer": "Reviewer_fu8D", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. I think the overall idea of this work is promising and useful for improving code generation. Unit test execution provides rich fine-grained information, and leveraging such feedback with reinforcement learning is beneficial.\n\n2. Results on both APPS and MBPP are good and outperform prior RL approaches for training code LLMs. \n\n3. The ablation studies are informative and demonstrate the importance of different design choices, such as different feedback types. The error analysis also demonstrates the improvement of RLTF on problems of different difficulty levels, and the distribution of different error types.\n\nWeaknesses\n\n1. First, I would like the authors to clarify the differences between RLTF and prior work in terms of the RL design. To my knowledge, CodeRL and PPOCoder also utilizes the LLM to generate code for computing rewards during training. Therefore, I think the main difference between RLTF and previous work is the reward design, not from the online/offline aspect.\n\n2. The ablation on different feedback combinations is helpful and shows that leveraging finer-grained execution feedback improves the performance. However, I feel that the coarse feedback becomes redundant after adding fine-grained and adaptive feedback. Could the authors try the ablation with all except the coarse feedback?\n\n3. From the error analysis, RLTF seems to increase the failure. What is the explanation for this? Does it mean that RLTF mainly decreases the runtime errors and compiler errors, but does not help reduce the semantic errors?\n\n4. On MBPP, why only pass@80 was reported? It is helpful to also present pass@1 results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes RLTF, a reinforcement learning framework for training code LLMs using fine-grained unit test feedback. Specifically, the reward includes 3 terms: (1) coarse-grained feedback that assigns a positive value for correct code, and different negative values for different errors, same as the CodeRL work. The reward is equally assigned to all code tokens; (2) fine-grained feedback. This feedback comes from the runtime error or compiler error that identify the locations of the errors, thus the negative reward is only applied to the code line that contains errors; and (3) adaptive feedback. This reward term depends on the percentage of passed unit tests. They perform RLTF on both CodeT5 and CodeGen, and evaluate their approaches on APPS and MBPP. They demonstrate that RLTF consistently improves the performance, and outperforms CodeRL and PPOCoder.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please clarify the differences between RLTF and prior work in terms of the RL design. To my knowledge, CodeRL and PPOCoder also utilizes the LLM to generate code for computing rewards during training. Therefore, I think the main difference between RLTF and previous work is the reward design, not from the online/offline aspect.\n\n2. Please add the ablation result with all feedback except the coarse one.\n\n3. Why RLTF increases the failure? Does it mean that RLTF mainly decreases the runtime errors and compiler errors, but does not help reduce the semantic errors?\n\n4. Please also present pass@1 results on MBPP.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FEiDk4cjLq", "reviewer": "Reviewer_g3kU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThe paper achieves empirical results which are reported to exceed that of the prior work. The proposed method is relatively simple to implement, as it largely amounts to using a particular design of the reward function on top of the policy gradient method.\n\n# Weaknesses\nTable 3's results for the prior work don't match what was reported in those papers. For example, I thought that the [CodeRL](https://arxiv.org/pdf/2207.01780v3.pdf) paper's Table 1 (page 12)'s row for \"CodeRL+CodeT5 770M\" should match Table 3's \"CodeRL 770M w/ CS\" row, but the numbers are different. Similarly, the numbers for the [PPOCoder](https://arxiv.org/pdf/2301.13816v4.pdf) paper's Table 3 (page 11)'s row for \"PPOCoder+CodeT5 700M\" don't match the ones in Table 3. The reason for the discrepancy is unclear. \n\nThe paper doesn't use terminology/framing commonly used in other reinforcement learning work, which makes it harder to put into context with the prior work. Examples:\n- The use of \"online\" and \"offline\" seem inconsistent in the paper. In other work, I believe that \"online\" generally corresponds to using an \"on-policy\" method and \"offline\" generally corresponds to using an \"off-policy\" method. Indeed, this paper uses policy gradient/REINFORCE, which is an on-policy RL method. But CodeRL and PPOCoder also use on-policy RL methods and they are marked as \"offline\". It wasn't clear to me what makes this method \"online\" and the others \"offline\".\n- The paper doesn't mention that Equation 2 corresponds to policy gradient/REINFORCE or include any citations to that effect.\n  \nThe use of $S$ and $E$ in don't seem to make theoretical sense, when $U_{line}$ is used and therefore $S$ and $E$ only covers a small portion of the generated program. For example, if a name error occurs on line 10, it could be because the code on line 10 used some variable or function which wasn't defined earlier in lines 1-9. One way to think of this error is that line 10 should be fixed to only refer previously defined variables. But another way is that lines 1-9 could be revised to have defined the name needed on line 10 (for example, by importing that function from a library).\n\nThe description of the \"online buffer\" is quite sparse. Section 3.2 doesn't describe the size of the buffer, when things are added to the buffer, and when things are removed.\n\nTable 1 says that the proposed method employs Critic Sampling, but there appears to be no description later in the paper of how it was incorporated. For example, there is no mention of training a critic model for the programs, as described in Section 3.3.4 of the [CodeRL paper](https://arxiv.org/pdf/2207.01780v3.pdf).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a concrete method for applying reinforcement learning to LLMs pre-trained on code, with a reward function derived from datasets that include natural language program specifications, code which implements those specifications, and unit tests. The method combines supervised fine-tuning using the code provided in the datasets, along with RL from a reward function formed by the use of the unit tests. The authors report that the use of reinforcement learning, and the different components of the reward function, together improve performance on the APPS and MBPP datasets compared to prior work.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would suggest fixing thse minor points:\n- Decoder-only autoregressive LMs don't use \"masked language modeling\". It is inaccurate to refer to them that way.\n- In equation 1, please use `\\max` instead of `max` in your formula so that it looks like $\\max$ and not $max$.\n- In a Markov process, the outcome from a given state depends only on the current state, and not the past history. As such, it seems inaccurate to me to refer to program synthesis with an autoregressive LM as a \"Markov process\", even if it is technically accurate if you define the entire history of the generation so far as the state.\n\nTo addresses the weakness above, I recommend the following changes:\n- Better describe how the comparative methods' results for Table 3 were obtained to explain why they don't match the numbers from the prior papers.\n- Describe the work using more standard RL terminology and either further explain what makes this work \"online\" and CodeRL/PPOCoder \"offline\", or remove that part of the paper.\n- Describe the size of the online buffer, when things are added to it, and when things are removed from it.\n- Describe further how Critic Sampling was used/implemented for this work.", "broader_impact_concerns": "No specific concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7gJrNDYNSs", "reviewer": "Reviewer_fP7d", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The results are ablated and analyzed in substantial detail, which helps the work provide fine-grained insight into the role and impact of loss terms that operate at different semantic levels (e.g., learning from compilation errors, test failures, both binary and in terms pass rate) on code generation performance.\n+ The setup is relatively conventional and easy to follow, allowing for reasonable comparisons with recently published work on RL for code generation.\n\nWeaknesses:\n\n- The work includes a wide array of seemingly hand-picked and unablated constants. Given the narrow scope of the evaluation in terms of datasets assessed (the main results all focus on APPS and MBPP), this raises concerns around overfitting to the evaluation set.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces an online RL approach for improving code generating models based on feedback from unit test executions. It leverages both a coarse (execution level, based on whether the code compiles, passes tests) and fine-grained feedback structure (for code that raises exceptions), as well as a loss term based on test passing rate. Training CodeT5 large with this setup shows promising results on a benchmark of challenging programming problems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In line with the main weakness stated above, the work should evaluate the impact of its many design decisions. Specifically:\n\n- Tab. 2/P6: involves a manual categorization of error types for just a subset of the errors in the Python programming language. This both means that porting this approach to new languages will be time-consuming and that it may not yield the observed improvements in systems with other error types. While those may be rare in interview problems, they abound in practice. The work should discuss this and note it as a limitation. Furthermore, the reason for having a separate U_ignore from U_global/U_whole is unconvincing. The motivation for this category is that \"the reasons for the error are numerous and cannot be accurately located\". That sounds very much like the definition of a global error. Please provide results where this issue type is grouped under global.\n- The work uses 0.3 as a constant in most of its R_* loss terms. Please provide both (a) an analysis where each occurrence of this constant (and others in Eq. 3) is instead tuned on a validation set (a small grid search over permutations suffices) and the optimal values are applied to the test set, as well as (b) an ablation of each coefficient separately other than R_fine in the style of Tab. 6. The goal is to show how sensitive the overall performance is to this coefficient.\n\nPlease also provide a disclaimer on Tab. 3 that many of the baselines there were apparently not fine-tuned on the APPS data the way RLTF was. The ablations in Tab. 4 show that the performance of RLTF is only modestly better than what may be accomplished by naively fine-tuning CodeT5 on task-specific samples (\"Offline\"). At a first read, Tab. 3 gives the impression that the model is far better than (much) larger baselines, so this difference in treatment should be emphasized in the table.\n\nMinor comments:\n- P1: \"up to 70% of the code generated by these models can be non-compilable (Chen et al., 2019)\" is not an appropriate citation given that \"these models\" refers to models introduced at least two years later. The compilation rate is naturally much higher for LLMs than for pre-2020 models. This paper records a rate around 30% in Fig. 2 with a model with just 770M parameters.\n- P4, before Eq. 1: superfluous period before colon in \"parameters.:\"\n- Eq. 1: the notation here is quite odd. Please at least fix `max` to use the non-italic form (LaTeX: \\max) and add the subscript to indicate that it maximizes over the model's parameters.\n- P5: \"manifests itself in this category\" -- unclear, both the phrasing and the implication that failing unit tests might also not be in this category.  As far as I can tell from the evaluation, this category is exclusively reserved for code with failing tests.\n- Tab. 2 uses U_whole but the text on P6 uses U_global. I assume these are the same; please consolidate them.\n- Eq. 4: this equation is not discussed by the text, nor is the S/E notation (also used in Eq. 3). I find this notation to be rather clunky in general and would prefer a simpler solution (at the very least for L_coarse, which also sums from start to end). Please at least briefly introduce the interpretation of this equation in the text.\n- Eq. 7 - 10: normally the cross-entropy loss is averaged over tokens. Was that not done here? That decision interacts with the choice to add an $\\alpha$ term to Eq. 9, which I worry can have an outsized influence on the overall loss when applied to very small snippets.\n- Sec. 4.4: this section needs more information on where the GPT results originate from and/or how they were fine-tuned.\n- Fig. 2: is quite hard to read. The font is small and the colors are very light. Please increase the font size and contrast.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Several recent papers have used RL to finetune LLMs for code, and the paper makes a useful contribution to this space. Specifically, the reviewers liked the reward design ideas introduced in the paper and found the empirical results to be impressive. However, there were concerns about the claim that this is the first "online" RL approach to code model tuning. The authors have since admitted that PPOCoder is also an online method, and the reviewers were happy with their response. However, the early parts of the paper don't reflect this fact appropriately yet, and the paper needs another deep pass over these sections. Given this, I am recommending acceptance with minor revisions.
Decision: Accept (poster)