RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Authors: Jonas Gehring
Kunhao Zheng
Jade Copet
Vegard Mella
Taco Cohen
Gabriel Synnaeve
Venue: ICML 2025 spotlightposter (2025)
Keywords: large language models
multi-turn code generation
reinforcement learning
LLM agents
Primary area: deep_learning->large_language_models
TL;DR: We apply reinforcement learning to ground LLMs in execution feedback for effective multi-turn code generation.

Abstract: Large language models (LLMs) deployed as agents solve user-specified tasks over multiple steps while keeping the required manual engagement to a minimum. Crucially, such LLMs need to ground their generations in any feedback obtained to reliably achieve the desired outcomes. We propose an end-to-end reinforcement learning method for teaching models to leverage execution feedback in the realm of code synthesis, where state-of-the-art LLMs struggle to improve code iteratively compared to independent sampling. We benchmark on competitive programming tasks and achieve large performance gains with both small (8B parameters) and large (70B) models, outperforming previous work while reducing the number of samples required by an order of magnitude. Our analysis of inference-time behavior demonstrates that our method produces LLMs that effectively leverage automatic feedback over multiple steps.

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Jonas Gehring'
Kunhao Zheng'
Jade Copet'
Vegard Mella
Taco Cohen
Gabriel Synnaeve
Abstract
Valld
Test
60
Large language models (LLMs) deployed as agents solve user-specified tasks over multiple steps while keeping the required manual engage- ment minimum. Crucially, such LLMs need to ground their generations in any feedback ob- tained to reliably achieve the desired outcomes. We propose an end-to-end reinforcement learn- ing method for teaching models to leverage ex- ecution feedback in the realm of code synthesis, where state-of-the-art LLMs struggle to improve code iteratively compared to independent sam- pling. We benchmark on competitive program- ming tasks and achieve large performance gains with both small (8B parameters) and large (70B) models, outperforming previous work while re- ducing the number of samples required by an or- der of magnitude. Our analysis of inference-time behavior demonstrates that method produces LLMs that effectively leverage automatic feed- back over multiple steps.
50
Salve Rate 100k
AphaCodium GPT4
AphaCode
AlphaCidum
MapCoder GF7
30
30
20
G714
AlpaCodium 3.
20
ApraCadim GPT-35
GF7:3.5
10
10
10
50 100
300
10
50
100
300
Sample Budget (k)
Sample Budget (k)
Liama 3.1 88 + RLEF
General Purpose
Domainvhdapted
Llama 708
Prompt Engineering
Solve rates of Llama 3.1 Models after RLEF training on CodeContests, compared to previously reported results across sampling budgets (log scale).
Figure 1:
tures from high-level descriptions (Yang et al. 2024
We posit that any decision decision-making agent offering a natural language interface has to possess two skills: First, the ability to accurately deduce a user's intent when prompted; for LLMs, this is typically achieved by fine-tuning to follow instructions according to user preferences (Ouyang) et al. 2022 Rafailov et al. 2023). Second, feedback on intermediate results of the agent's actions has to be taken into account to arrive at the desired outcome. For example, web page that contains a necessary bit of information might have gone offline, requiring another search engine query. In the context of code generation, feedback can provide information about implementation bugs as well as constraints that are inefficient or cumbersome to specify in full detail, e.g., software and hardware platform details or library dependencies. Intermediate feedback is there- fore crucial to ground LLM generations in the concrete situations encountered at inference time.
1. Introduction
The consistent increase in capabilities of Large Language Models (LLMss) has prompted researchers and develop- ers to benchmark and deploy them in increasingly com- plex environments (Brown et al., 2020) OpenAl| 2023]
AI @ Meta 2024). An emerging research direction to employ LLMs as agents solve tasks in multiple steps with little to no human oversight, querying external compu- tation or data sources when needed or as dictated by man- ual scaffolding (Schick et al.] 2023 Kapoor et al. (2024). Such autonomous use of LLMs is of interest for ensuring accurate answers to user queries with up-to-date informa- tion (Mialon et al, 2024 interaction with websites (Yao
al. 2022). generating code to implement software fea-
In this work, we aim to endow pre-trained LLMs with the aforementioned skills task alignment and grounding in inference-time feedback in the domain of code synthe- sis from natural language descriptions (Chen et al.) [021]] Rozière et al., 2023). Here, feedback is naturally provided as the result of the execution of generated code in the form of error messages and unit test results. However, to date,
'Meta FAIR. Correspondence to: Jonas Gchring <igchring@ mota.com>, Gabriel Synnaeve <gab@ meta.com>.
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Natural Language Protrem Description
Natural Lang Problem Desor pror
Provide a Pythan solution for the following competrivve programmingg quest ore Lets cal the string beautfu iiit doee not contain aubstring of lengtt at lasst 2 which is a paindrome. Racall that palindrome [-.]
Fawoution leacltack
Code Solution (uu
LLM
updare
ded nén coot, to, bosuu Eul techecringo
vissa Seuud
found vallo
coe Lengsa in cuage caap bachateanng,
3, 31:
Public Texts: FAIL
Code Solution
PPO
Your code fiiled the following feste:
iinutt '5A bascb 23' faled: Execcution took too long.
Timn lmm
YES
Give anothar try. [--]
Codo Solution (umn
Printee Tests
from firetonte Export THy eache
for: "enach in Hange "eer oe + 13x dop zaage laandas Length 11:
FAIL
Public
PASS
Inference-Time
Training-Time Execution Feedback
Public Tests: PASS
Execution Feedback
Subrritting solution to Priuste Tests Execuutio
Figure 2: Left: Overview of reinforcement learning with execution feedback (RLEF). The LLM is repeatedly prompted to implement code according to a problem description. Each attempt is evaluated on public test set; upon failure, feedback is inserted into the conversation. If public tests are passing, or a specified turn limit is reached, execution on additional, private tests determines the reward signal. The model is then updated to optimize the reward with PPO. Right: Example dialogue with two model responses. Execution feedback hints at an inefficient first solution, which the model responds to utilizing a cache. The code passing the public test sets will be evaluated on the full test set.
utilizing such feedback for code generation with LLMs has failed to yield substantial improvements when taking com- putational demands into account; indeed, obtaining sam- ples independently often results in higher accuracy for a fixed inference budget (Kapoor et al.) [024]] Xia et al.) 2024]. As a test bed to investigate and improve ground- ing execution feedback, we frame code generation as an iterative task, repeatedly asking LLM to produce code according to provided natural language description (Fig- Zr After each generation, code is evaluated on example test cases, and the resulting feedback is provided as addi- tional context for subsequent attempts. We thus obtain an interactive environment where actions correspond to code and observations correspond execution feedback. Im- portantly, such framing permits end-to-end optimization with reinforcement learning (RL) algorithms to maximize reward signal- here, binary reward based on whether the final code solution passes a set of held-out test cases.
LLMs effective in iterative, multi-turn scenarios. Our im- provements from RLEF on CodeContests further general- ize to HumanEval+ and MBPP+, two popular benchmarks for code synthesis, and to increased sample budgets com- pared to training time.
2. Method
2.1. Iterative Code Synthesis
We structure the code synthesis task as multi-turn conver- sation in which an LLM is repeatedly prompted to generate code solution to a natural language problem description. After each solution, we provide an automatically generated response with results obtained by executing the solution's 's code against test cases. This setup is applicable to language models tuned for the use case of interacting with users in chat format and follows previous work on self-repair for code generation (Shinn et al. 2023 Olausson et al. 2024).
We benchmark our training method incorporating repeated code actions and execution feedback in a reinforcement learning context (RLEF) on CodeContests (Li et al. 2022). challenging competitive programming benchmark. Start- ing from Llama 3.1 models (AI @ Meta 2024), we achieve substantial performance improvements, surpassing previ- ous state-of-the-ar results while reducing the number of generations required by an order of magnitude (Figure []]. Our analysis shows that RLEF training unlocks the capabil- ity to leverage inference- peerence-timeennee machine feedback, rendering
Crucially, we utilize two different sets of test cases: a pub- lic test yields execution feedback that can be accessed dur- ing repeated attempts and forms the basis of selecting a final solution, whereas private test set ultimately deter- mines the correctness of the final solution. Separate test sets provide two main benefits. First, if test inputs and out- puts are fixed, held-out tests guard against shortcuts dur- ing the optimization procedure in which an LLM can copy expected test outputs in subsequent answers, based on ex-
2

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforeement Learning
ecution feedback. Second, running full test suite may be computationally demanding, and limited set of public tests can accelerate the iterative code generation procedure. It may, however, be desirable to maximize test coverage for execution feedback at inference time, and we verify that this can indeed improve performance (Appendix (B.3).
and abbreviating previous observations and actions with
Gt = 0o; 00, 01,00 11,0000..000 Ot, our rule-based reward function at step is:
[(ar ct)
R(s; at) = r((), - 3 log p(arlct)) where
1,
if end of episode and all tests pass -1, if end of episode and any test fails
Our conversation flow for code generation is depicted in Figure 2 Concretely, we start the dialogue with the prob- lem description and query the LLM for an initial solution. The solution is verified against the public test set, which yields results in the form of passed and failed test cases, as well as potential syntax or runtime errors. If any public test fails, this execution feedback is formatted and appended to the dialogue. The L.LM is then queried for an updated code solution, with the original problem text, previous solutions, and their respective feedback provided in the prompt. If the solution passes all public tests, or a specified turn limit is reached, it is considered final and will be submitted for evaluation on the private test set. We refer to Appendix C for prompt and execution feedback templates.
r(0:, at) =
0.2, if oz does not contain valid code
with constant 33 trading off between task reward and KL maximization. For PPO, we compute policy gradients by incorporating a concurrently learned value function as a baseline, i.e., we train the policy to maximize the advan- tage A, = V((e) + Σ R(s..,)) see Appendix A.
We note that while the above MDP considers full responses as actions, the underlying policy and value functions implemented as language models outputting single tokens. Selecting suitable action space for optimization hence re- quires consideration in our setup, and a suitable choice may depend on the concrete task at hand. We propose to model the policy at the token level while learning a value func- tion for whole turns; compared to optimizing both mod- els at either the turn token level, this hybrid approach worked best in our early experiments. Hence, we predict the value of response from the last token of its respec- tive prompt, and we use a single advantage value for each token action within a response. Our response-based value estimation is closely related to Zhou et al. (2024); how- ever, we do not train an additional Q-function. For the KL penalty, we found it beneficial to compute the probabilities of responses T (a.le)) as the geometric mean rather than the product of token probabilities. This counteracts a possibly detrimental bias towards shorter generations, in particular for non-final responses.
2.2. Reinforcement Learning with Execution Feedback
The iterative code synthesis described in the previous sec- tion can be seen as a Markov Decision Process (MDP), and the language model as a policy (Sutton & Barto 2018). For generality, we assume a partially observable MDP, as our reward function utilizes held-out, private test set that is not accessible to the policy (unless exact textual repre- sentation of the desired program behavior is provided in the problem description). Observations and actions are pro- vided as tokenized text sequences. Concretely, the initial observation og is the problem description, and actions a, at each step t are textual responses. Successive observations or consist of past observations and actions, including exe- cution feedback obtained by evaluating the previous action a-1 on public tests. Episodes end when public test eval- uation succeeds or a specified step limit is reached. At the end of an episode, a scalar reward is provided correspond- ing to whether all public and private tests are passing. We
3. Experimental Results
3.1. Setup
do not use reward discounting (i.e.. ? = 1).
We perform experiments on the CodeContests benchmark (Li et al. 2022), which requires generating code solution to a problem specified in natural language along with a tex- tual description of public test cases. Problems are of high difficulty and used in human competitive programming with a focus on algorithms, data structures, and runtime efficiency. The correctness of solutions is evaluated with private tests that are hidden from contestants; hence, we present feedback from public tests only. CodeContests consists of a training set and two evaluation sets, "valid" and "test", with 117 and 165 problems, respectively; we use the former for model and hyperparameter selection.
For optimizing policy in the above environment, we employ Proximal Policy Optimization (PPO), a common choice for fine-tuning large language models (Schulman let al.) 2017, Ziegler et al.) 2020) Ouyang et al.) 2022). Following previous work, we include a KL penalty in our reward signal, acting both as an entropy bonus and as regularization towards the distribution of the LLMs we
start from. In initial experiments, we found that possi- ble failure mode concerns the generation of invalid code in finafinal responses, which we address by providing small penalty for invalid responses. Denoting the pol- icy to be optimized with æ and the initial policy with p.
We optimize our models on the training set, from which we discard 669 of the 13,328 problems due to missing
3

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Table I: Results on CodeContests of our initial and RLEF-trained models compared to prior work. The sample budget A in n @ refers to the number of LLM responses, e.g.. 1 1@3 for our results corresponds to a single rollout with up to three model responses. The 70B model obtains state-of-the-arr results after RLEF, and significantly outperforms AlphaCodium and MapCoder generally, and on the test set with a fraction of the samples. The RLEF-trained 8B model outperforms AlphaCodium with 100 samples and MapCoder (gpt-3.5-turbo) with 3 samples.
Model
Source
n n@k
Valid Set
Test Set
AlphaCode 9B
Li al.
2022)
10@ 1000
16.9
13.3
AlphaCode 41B + clustering
et al.
(022)
10@ 1000
21.0
16.4
Code Llama 34B + PPO
Xu et al. (2024)
10@ 1000
19.7
22.4
AlphaCodium gpt-3.5-turbo-Iie 16k
Ridnik et al.
(2024)
5@100 10100
25
17
AlphaCodium gpt-4-0613
Ridnik et al.
(2024)
5@100
44
29
AlphaCodium Llama 3.1 70B Instruct
Ours
5@100
34.2
27.8
MapCoder gpt-3.5-turbo-1100 106
Islam et al.
20244
1 @223
12.7
MapCoder gpt-4-1 10e-ire
Islam et al.
2024
1 @@19
28.5
Llama 3.0 8B Instruct
Ours
1 @@3
4.1
3.2
+ RLEF
Ours
1 1@3
12.5
12.1
Llama 3.1 8B Instruct
Ours
1 @33
8.9
10.5
+ RLEF
Ours
1 @@3
17.2
16.1
Llama 3.1 70B Instruct
Ours
1 @@3
25.9
27.5
+ RLEF
Ours
1 @@3
37.5
40.1
Llama 3.1 8B Instruct
Ours
10@ 100
21.7
24.8
+ RLEF
Ours
10@
29.8
28.7
Llama 3.1 70B Instruct
Ours
10@ 100
50.2
50.3
+ RLEF
Ours
10@ 100
54.5
54.5
public or private test cases. We train all models to output Python code solutions.
3.2. Main Results
In Table 1 list solve rates the CodeContests valid and test sets for iterative code generation with up to three turns, along with previously reported results. When sam- pling from our models, we use temperatures 0,2 for 1 I@3 and 1.0 for 10@ 100, and nucleus sampling with top-p 0.95 in all cases (Holtzman et al.] 2020). Each solve rate estimated on 200 rollouts, using the estimator described by Li al. (2022). We compare against AlphaCode (Li et al. 2022) and PPO with rewards from test execution on the Code Llama 34B model from Xu et al. (2024), both of which report results with large number of samples. AlphaCodium (Ridnik et al.] (024) and MapCoder (Islam) et al. 2024) are high-performing agentic frameworks built on top of the proprietary GPT models and combine chain- of-thought prompting, code execution, program repair, and, in the case of AlphaCodium, automatic test generation.
The Llama 3 family of models (Al @ Meta 2024) com- prises our initial policies, specifically the Instruct 8B and 70B parameter models of the 3.0 and 3.1 release. These models exhibit strong code generation performance out of the box and are able to follow instructions in the prompt, al- leviating the need for initial fine-tuning stage prior to RL training. During training and for evaluations, unless noted, we set the turn limit to allow for 3 LLM attempts at solving each problem. We perform 12,000 and 8,000 updates to the 8B and 70B models, respectively, and select checkpoints based on valid set performance. Hyperparameters and further experimental details are provided in Appendix A
We follow Li et al. 2022) in reporting results as n @ k aver- age solve rates. The n@ k metric represents the expectation that any of n solutions, selected from l samples in total, is correct, i.e., passes all tests. In our multi-turn setup, each turn counts as a sample. This allows for fair comparisons with respect to sample budgets, which is particularly rele- vant when employing large LLMs with high inference cost in agentic scaffoldings (Kapoor et al. 2024
With RLEF training, we improve markedly on the original Llama 3.1 models. Notably, on the test set, the 70B model beats AlphaCodium with GPT-4 with a single rollout compared to 5 solutions from 100 samples (38.0 and 29). Likewise, the 8B model after RLEF is slightly ahead compared to the similar-sized AlphaCode 9B model (16.1 and 13.3), but with a sample budget of 3 in our case and
For simplicity, we consider a full LLM response as a single sample our evaluations. Note that for iterative code generation, the allocated sample budget may not be fully utilized as a success-
ful public test run will result in carly termination of a dialogue.
4

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Errors: Tum 1
Errors Fixed: Turn 2
Errors Fixed: Tum
3
Code Changes
Error Type (8B)
OUTPUT
-500
EXCEPTION
5000
Count
TIMEOUT
2500
OOM
0
1000
2000
3000
0
100
200
20
40
80
0.00
0.25
0.50
0.75
1.00
Error Type (70B)
OUTPUT
6000
EXCEPTION
4000
Count
TIMEOUT
2000
OOM
1000
2000 Count
3000
0
200
400
600
800
100
200
300
0.00
0.25
0.50
0.75
1.00
Count
Count
1-chrf
Llama Instruct
(random feedback)
Figure 3: Behavior analysis of initial and RLEF-trained models with respect to public test results for 8B (top) and 70B (bottom) models. Within 20 rollouts per problem (5640 in total), we count errors in the initial solution (turn 1); errors turned into correct code in turns 2 and 3; and code changes across successive solutions according to the chrf metric. RLEF-trained models make fewer errors initially, can fix errors more reliably, and perform larger code edits; initial models frequently repeat previous solutions. With random execution feedback, error recovery severely impaired.
1,000 for AlphaCode. While we cannot compare directly to the more recent AlphaCode 2, a performance estimate of 34.2 on the valid set for 10@ 100 puts our 70B model ahead (37.5) with just 3 samples When considering larger budget of 100 samples, corresponding 33 rollouts, the 70B Instruct model beats previously reported results, including AlphaCodium on the valid set. Evaluating AlphaCodium with the 70B model results lower scores under a similar budget. With RLEF, we obtain further improvements to 54.5 on the valid and test set. The relative gains over the initial models, while still significant, are reduced in the 10@ 100 setting compared to the @3 setting. Kirk et al. (2024) observe that RL training of LLMs can reduce the diversity of outputs, and we interpret our results as further evidence of their hypothesis.
3.3. Inference-time Behavior
In Table 2 we first take a closer look at single- and multi- turn performance with fixed budget LLM genera- tions (1) This corresponds to our iterative setup with up to three model responses, or three independent re- sponses for single-turn results. We further consider gen- eralization to two popular code generation benchmarks, HumanEval+ and MBPP+ (Liu et al. 2023a), which we modify to match our iterative code generation setup with "base" tests for inference-time execution feedback and "plus" tests for solve rate estimation (see Appendix C.4 for details). Our results demonstrate that, when consid- ering fixed sample budget, base models rarely benefit from access to faulty solutions and execution feedback in the multi-turn code generation setup. This also applies to gpt-4o-2024-05-13, which shows stronger performance when sampling solutions independently on CodeContests and HumanEval+. After RLEF training, the 8B and 70B Llama 3.1 models both benefit from execution feed- back and can therefore achieve larger gains top of im- proved single-turn scores, with the exception of the 8B model on CodeContests and MBPP+ where single-turn per- formance drops. While multi-turn gains from RLEF are most pronounced on CodeContests, the training domain of our models, we also observe notable improvements on Hu- manEval+ and MBPP+. For confidence bounds that con- firm the robustness of our results, well evaluation LiveCodeBench, see Appendix B.I
Table also highlights that the released Llama 3.1 models offer competitive performance on CodeContests from the start, which we attribute focus coding capabilities during instruction tuning (AI @ Meta 2024). However, our method is also highly effective on the previously released 3.0 8B model, improving 1 1@3 solve rates on both the valid (4.1 > 12.5) and test (3.2 + 12.1) set. Thus, RLEF may be useful as a partial substitute for instruction tning for tasks where ground-truth rewards from automatic evaluation are available.
AlphaCode Team 2023) train and evaluate on non con-disclosed competition problems but report sample efficiency increase of 10,000x over AlphaCode, which achieves 10@ 1M solve rate of 34.2 on the valid set.
Next, we seek to determine where the gains of RLEF train- ing stem from. Based on the improved single-turn results in Table 2 we hypothesize that, for the 70B model, these are partly due to training on the specific domain com- petitive programming questions. More importantly, higher
5

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Table 2: 1 @ solve rates in single-turn (ST) and multi-turn (MT) setups for base and RLEF models from the Llama 3.1 (I.3.1) series. On CodeContests, iterative code generation yields modest gains at best and drops in performance at worst, unless RLEF training is employed. Improvements from RLEF on CodeContests in the multi-turn setting carry over to HumanEval+ and MBPP+, which require slightly different execution feedback formatting.
Table 3: 1@3 solve rates starting from Llama 3.1 mod- els. Comparison of different methods for acquiring itera- tive code synthesis capabilities. RLEF is the most effective training method, followed by supervised fine-tuning (SFT). We find few-shot prompting detrimental to Instruct models.
8B Instruct
70B Instruct
Method
Valid
Test
Valid
Test
8.9
10.5
25.9
27.5
Model
CC.
Test
HE+
MBPP+
Few-Shot Prompting
8.5
8.5
22.5
20.3
ST
MT
ST
MT
ST
MT
SFT
10.3
10.0
27.7
27.2
GPT-4o
25.3
24.3
82,8
80.7
68.8
71.7
RLEF
17.2
16.1
37.5
40.1
1.3.1 8B Inst.
11.8
10.5
65.3
63.9
58.3
60.5
+ RLLEF
9.7
16.1
67.5
69.5
57.0
63.1
1.311 70B Inst.
26.2
27.4
73.2
75.0
66.9
70.2
correct final solution, pass@) 10 reflects the ability to recall correct solution (i.e., whether any of 10 solutions passes the private tests). On both valid and test sets, random feed- back results in a drop in pass@ I, which is further amplified as the turn limit increases. This provides further evidence for less targeted repair capabilities with random feedback, as programs can be repaired less reliably. Notably, with ground truth feedback, the pass rate keeps increasing with higher turn limits. For pass@ 10, the difference between true and random execution feedback is less pronounced. As pass@ 10 can be optimized by sampling many diverse candidate solutions within a dialogue, these results indicate that with random feedback, our models resort to sampling succession of diverse, potentially correct solutions.
RLEF
30.1
40.1
78.6
80.4
67.6
72.2
scores in the iterative setting for both the 8B and 70B model could be attributed to either an increased capability of sam- pling diverse solutions within a rollout or more targeted self-repair based on execution feedback. For probing the sensitivity of our models to the observed feedback, we per- form inference- time ablations with random execution feed- back by executing a faulty solution to an unrelated problem, but still end the dialogue if the current solution passes pub- lic tests (details in Appendix C.2
In Figure the execution feedback relates) over 20 rollouts on the valid and test set combined. After RLEF, both the 8B (top row) and 70B (bottom row) models produce fewer wrong out- puts in their initial response but are more prone to exceed- ing the allocated time limit. In subsequent responses, re- covery from all error categories is significantly improved. With random feedback. however, we see a clear impairment of self-repairs, demonstrating that RLEF allows LLMs to effectively leverage the provided feedback. We further gauge changes from one response to the next by comput- ing the character n n-gram F-Score (Popović) 2015) chrF) among successive codes (Figure right). This underscores shortcoming of the Instruct models without RLEF in that they perform only minimal code edits; indeed, we observe that they frequently output the same code solution despite inline feedback pointing out errors.
consider errors on public tests (to which
Finally, we evaluate the generalization across turn limits with respect to a given sample budget. In Figure 4b we perform rollouts with temperature 1.0 to emphasize perfor- mance at higher sample budgets by increasing the diversity of generations. We compute 10@k solve rates by distribut- ing k samples equally across rollouts with different turn limits. For the 8B model (top row), prior to RLEF train- ing, best performance can be obtained with independent samples (1 turn), with the exception of the test set above 30 samples. The initial 70B model performs better with or 5 turns; although, for small budgets, single-turn per- formance is competitive. After RLEF,, we observe that 3, 5, and 10 turns yield a consistent improvement over inde- pendent sampling, with best performance obtained with 5 turns. In all cases, increasing the turn limit to 10 provides no benefits under fixed sample budget.
The analysis above in Figure 3 reveals two key findings about RLEF 1. Samples within rollout are of higher diversity (less similar codes). 2. Edits are also targeted, resulting in fewer successful repairs with random execu- tion feedback. This finding is echoed in Figure 4a| in which we compare models with true and random feedback across different turn limits. Here, we compare pass @ and pass @1 10 metrics, irrespective of different sample bud- gets due to varying turn limits (Chen et al. 202T). While pass@1 I captures the precision with which we arrive at a
3.4. Ablation Studies
3.4.1. LEARNING ITERATIVE CODE SYNTHESIS
We investigate whether LLMs can, apart from our RL train- ing, be effective in multi-turn code generation using few- shot prompting (Brown et al. 2020) and supervised fine- tuning (SFT). Lacking suitable ground truth training exam- ples for SFT, we mine rollouts on the CodeContests train- ing set with Llama 3.1 70B Instruct and filter them based on
6

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Valld
Test Set
Table 4: 1@3 solve rates starting from Llama 3.1 models, temperature 0.2. Conventional single-turn (ST) compared to our multi-turn (M') training method. MT training yields larger improvements compared to ST.
40
40
35
30
25
20
20
RLEF Training
Valid
Test
15
10
Model
ST
MT
ST
MT
8B Instruct
9.4
8.9
11.6
10.5
50
ST
10.3
10.2
9.9
10.9
440
40
MT
16.2
17.2
9.5
16.1
passg
30
70B Instruct
25.6
25.9
25.9
27.5
30
ST
28.3
31.1
27.3
32.9
20
MT
25.8
37.5
30.1
40.1
10
Tumn
8B RLEF
70B RLEF
Thue Foodback
Random Feedback
(a)
model performance on the validation set only; we do not see improvements on the test set. For pre-trained models, improvements from SFT but lower scores compared to instruction-ture models (Appendix B.2). With RLEF, we obtain significantly higher solve rates compared to SFT models, underscoring the efficacy of our RL training loop.
Valid Set
Test Set
30
Solve Rake (1008)
25
2.0
10
10
20
50
100
10
50
100
3.4.2. SINGLE- TRAINING
Salve Rate (100k) 708
-35
55
50
50
In Table 4 we compare our iterative code generation setup to traditional, single-tum generation where the model is not presented with inference-ting feedback. We use the same training loop for single generations, albeit without the penalty for invalid code (Section (.2). as this is subsumed by the reward signal for incorrect solutions. For Llama 3.1 8B Instruct, single-turn training (ST) hurts performance on the test set. The 70B model benefits from single-tum training and improves over multi-turn SFT results in Table 3 Moreover, we observe transfer in that applying the single-turn model in a multi-turn setting improves 1 1@3 solve rates. We attribute this to the existent but comparably weak multi-turn capabilities of the vanilla 70B Instruct model. Overall, we see strongest performance with the RLEF method employing multiple turns training and inference time.
45
80
40
35
35
100
100
Sample Budget (k)
Sample Budget (k)
Turn Limit Model
10
Uama 3.1 Instruct
RLEF
(b)
Figure 4: (a) Pass @1 I and @ass@ 10 across turn limits with RLEF-trained models, providing either true or random exe- cution feedback (temperature 0.2). With random feedback pass@I 1 is reduced while pass @ 10 suffers only slightly, in- dicating that programs can be repaired less consistently. (b) Impact of turn limits 10@k solve rates per sample bud- get (top: 8B model, bottom: 70B model) with temperature 1.0. With RLEF, iterative code generation can leverage up to turns to achieve compute-optimal performance.
Further ablations are in the appendix. In Appendix B.4 we evaluate the effect of training a dedicated repair model on outputs of the single-turn 8B training run in Table 4 similar to Le et al. (2022). Together, the single-turn and repair model obtain 1@3 solve rates of 14.8 on the valid set and 12.6 on the test set; an improvement over the single-turn model alone (10.2 and 10.9) but significantly below the corresponding multi-turn model (17.2 and 16.1), In Appendix B.5 we show that withholding public test execution feedback during training results in significantly worse performance. Finally, in Appendix B.6 we exper- imentally validate the design choice of a tern-level value function (Section 2.2
the correctness of final solutions. We then fine-tune Base and Instruct versions of the Llama 3.1 8B and 70B param- eter models on the mined corpus and also source it for few- shot examples (Appendix(A.. A.33.. The results in Table show that few-shot prompting detrimental to the instruction- tuned models. In Appendix B.2 we report few-shot 1 @3 solve rates for pre-trained models and find that they achieve lower performance compared to zero-shot prompting for instruction models (1.2 and 1.8 for 8B, 4.6 and 5.8 for 70B on valid and test set, respectively). SFT improves Instruct

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforeement Learning
4. Related Work
currently, Kumar al. (2224) propose a two-stage RL method (SCoRe) to improve self-correction by generating two successive solutions. In contrast to our method, SCoRe does not leverage execution feedback at inference time and instead asks the model to reconsider its initial solution. While this approach suits domains lacking automatic feed- back, it cannot benefit from grounding via execution in- formation. Furthermore, inference-time feedback can help the model generalize to new environments after training. Chen et al. (2024a) address code generation with human feedback and develop an appropriate SFT strategy by train- ing a separate code repair model. In our work, we effec- tively leverage automatically generated feedback, format- ted in natural language, with a single model only. More recently, DeephSe---eSS et al. (2025) observe emerging rea- soning capabilities with a large-scale application of GRPO (Shao et al. 2024) to math and code problems and achieve high performance on competitive programming tasks. We thus consider the training of reasoning models with pro- gram execution feedback and, likewise, the introduction of execution feedback to math domains, a promising avenue for future research.
Code Generation from Natural Language A large body of work explores code synthesis from natural language de- scriptions, often leveraging large quantities of source code for pre-training and instruction fine-tuning (Clement et al.)
2020 Chen et al., 2021 Austin et al. 2021; Li et al. 2023
Gunasckar et al. 2023 Rozière et al. 20023] AI @ Meta
2024). This approach has led to notable performance gains on popular benchmarks.
Inference-Time Repair and Scaffolding Recent research shows that techniques such as prompt scaffolding and flow engineering (Ridnik et al.) 2024 Islam et al.) 2024 Zhhnn et al. (024) can improve performance on challeng- ing benchmarks but incur high inference costs by chaining multiple LLM calls. Other work focuses on re-prompting and verifying code through execution (Shinn et al., 2023: Chen et al. 2024b] Zhong et al., 2024). However, re- cent studies suggest that independent sampling of solu- tions is competitive and more efficient (Olausson et al.] 2024 Kapoor et al.) 2024) when considering equal sam- pling budgets. With our method, the self-repair capabilities of LLMS can be dramatically enhanced, resulting in supe- rior performance of iterative code generation for both small and large sample budgets. At the same time, we propose to trade complex, domain- specific prompt engineering and scaffolding for domain-specific reinforcement fine-tuning.
Past work on applying reinforcement learning to LLMs on longer-horizon decision makinn tasks placed an emphasis on the necessary grounding in the environment, such as text-based navigation games (Carta et al. 2023), text games (versus an oracle LLM) and web-shopping Zhou et al.] 2024), and visual observations (Zhai et al. 2024). While our work follows similar motivations, we address funda- mentally different domain, code synthesis, which features significantly larger action space compared to previous work, i.e., the space of valid Python programs.
Fine-Tumne Tine-Tuning/RL with Execution Feedback Fine-tuning LLMs with reinforcement learning has become widely adopted strategy for aligning outputs to desired targets, relying on specialized reward models or automatic sig- nals (Ziegler et al., 2020, Touvron et al., 2023; OpenAl 2023 DeepScek-AI et al., 2024 AI @ Metal 2024). For code synthesis, this reward signal can be derived from executing generated solutions against test cases (Shojaee 2023 Dou et al., 2024) Yu et al. 2024). Specifically, cet al.
5. Conclusion
Limitations. While our results demonstrate effective us- age of inference-time feedback, the code synthesis task we consider is limited to improving single solution to a given problem. Generalizing our method to environments with larger tasks that require decomposition, either via man- ual scaffolding or, eventually, in a self-directed manner, remains the subject of further research. Iterating on the execution results of unit tests naturally requires test cases, which may not be readily available. We regard potential combination with automatic unit test generation (Watson) et al. 2020 Jain et al. 2024a) as an interesting avenue for further experiments.
Le el al. (2022) fine-tune an LLM with policy gradients and next-token loss on execution rewards. To improve performance at test time, they train dedicated models for outcome prediction and program repair ("critic sam- pling"), albeit without incorporating execution outputs in the prompt. Liu et al. (2023b) extend this work with fine-grained reward function. Xu et al.] (2024) fine-tune a stronger, code-specific LLM in a simpler setup with a binary reward from unit tests and observe substantial improvements from RL on CodeContests. In contrast, we expand the iterative environment where execution feedback is not only provided as scalar reward but also in textual form. This allows a single model to learn both code synthesis and code repair capabilities without relying on ground-truth solutions or extra inference scaffolding.
setting to an
In this work, we proposed reinforcement learning from ex- ecution feedback (RLEF), a fine-tuning method for LLMs that endows them with crucial capability for autonomous operation: grounding future generations in environment feedhack. We applied RLEF to iterative code synthesis and obtained substantial improvements in solve rates on the
Concurrent Approaches and Longer-Horine RL Con-
8

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
CodeContests competitive programming benchmark while reducing the required sample budget for inference. The RLEF-trained models further generalize to increased turn limits and to HumanEval+ and MBPP+, two popular code generation benchmarks that exhibit simpler programming questions and different execution feedback formatting. Our in-depth analysis revealed that, while an increase cor- rect first-turn generations and in the diversity of successive generations offers major contribution to performance, our models also meaningfully take execution feedback into ac- count and resolve erross over multiple turns.
Acknowledgements. We thank Quentin Carbonneaux for significant contributions to this work, as well as Chris Cummins, Olivier Duchenne, Fabian Gloeckle, Baptiste Roziere, Sten Sootla, Nicolas Usunier, and Sida Wang for helpful technical contributions, suggestions, and insightful discussions.
Impact Statement
Successful grounding LLMs for code generation execu- tion feedback will amplify their utility when applied im- pactful tasks such assisting software development and performing quality control. In general, however, increas- ing the capabilities of LLMs, now widely deployed in a range of applications, requires quality control and guard- railing to promote safety and minimize potentially harmful output. We limit our study to the generation of source code, where we confine the execution of model- generated output local sandboxes. We believe the framework of [Shavit et al. (023) regarding the governance of AI agents be useful resource for practitioners.

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
References
DeepSeek-AI, Zhu, Q. Guo, D., Shao, Z., Yang. D., Wang. P. Xu, R., Wu, Y., Li, Y., Gao, H., Ma, S., Zeng, W., Bi, X., Gu, Z., Xu, H., Dai, D., Dong, K., Zhang, L.,, Piao, Y., Gou, Z., Xie, Z., Hao, Z., Wang, B., Song, J., Chen, D., Xie, X., Guan, K., You, Y., Liu, A., Du, Q. Gao, W., Lu, X., Chen, Q. Wang, Y., Deng, C., Li, J., Zhao, C., Ruan, C., Luo, F,, and Liang, W. DeepSeck- -ooeekk--odr-V2 Breaking the Barrier of Closed-Soures Models in Code Intelligence. arXiv:2406. / /93/ [cs], Jun 2024.
AI @ Meta, L. T. The Llama 3 Herd of Models. Technical report, 2024.
AlphaCode Team, G. D. AlphaCode 2 Technical Report. Technical report, 2023.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q. and Sutton, C. Program Synthesis with Large Language Models. arXiv:2108.077 2it::2100.07732///1 [cs], Aug 2021.
DeepSeek-AI, Guo, D., Yang. D., Zhang, H., Song, J., Zhang. R., Xu, R., Zhu, Q. Ma, S., Wang, P., Bi, X., Zhang. X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang. B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Ding. H., Xin, H., Gao, H., Qu, H., Li, H., Guo, J., Li, J., Wang. J., Chen, J., Yuan, J., Qiu, J., Li J.. Cai, J. L., Ni, J., Liang, J., Chen, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L., Zhao, L.,, Wang. L.. Zhang, L., Xu, L., Xia, L.., Zhang, M., Zhang. M., Tang, M., Li, M., Wang, M., Li, M., Tian, N., Huang, P., Zhang, P., Wang. Q. Chen, Q.. Du, Q., Ge, R., Zhang, R., Pan, R., Wang, R., Chen, R. J., Jin, R. L., Chen, R., Lu, S., Zhou, S., Chen, S., Ye, S., Wang, S., Yu, S., Zhou, S., Pan, S., Li, S. S., Zhou, S., Wu, S., Ye, S. Yun, T., Pei, T. Sun, T., Wang, T., Zeng, W., Zhao, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Xiao, W. L., An, W., Liu, X., Wang, X., Chen, X., Nie, X.. Cheng, X., Liu, X., Xie, X., Liu, X., Yang, X., Li, X., Su, X., Lin, X., Li, X. Q., Jin, X., Shen, X., Chen, X., Sun, X., Wang, X., Song. X., Zhou, X., Wang, X., Shan, X., Li, Y. K., Wang, Y. Q.. Wei, X. Zhang, Y., Xu, Y., Li, Y., Zhao, Y., Sun, Y., Wang, Y. Yu, Y., Zhang, Y., Shi, Y., Xiong, Y., He, Y. Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y., Ou, Y., Wang, Y., Gong, Y., Zou, Y., He, Y., Xiong. Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zheng, Y., Zhu, Y., Ma, Y., Tang, Y., Zha, Y., Yan, Y. Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z., Zhang, Z., Hao, Z., Ma, Z., Yan, Z., Wu, Z., Gu, Z., Zhu, Z., Liu, Z., Li, Z., Xie, Z., Song, Z., Pan, Z., Huang, Z., Xu, Z., Zhang, Z., and Zhang, Z. DeepSeek-RI: Incen- tivizing Reasoning Capability in LLMS via Reinforce- ment Learning, arXiv:2501.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.. Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J.. Winter, C., Hesse, C.. Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCan- dlish, S., Radford, A., Sutskever, I., and Amodei, D. Language Models are Few-Shot Learners. In NeurlPS, 2020.
Carta, T., Romac, C., Wolf, T., Lamprier, S., Sigaud, O., and Oudeyer, P.-Y. Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning. arXiv:2302.0266 [cs], Sep 2023.
Chen, A., Scheurer, J., Korbak, T., Campos, J. A., Chan, J. S., Bowman, S. R., Cho, K., and Perez, E. Improv- ing Code Generation by Training with Natural Language Feedback. arXiv:2303. 16749, Feb 2024a.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, I.,, Bavar- ian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang. J. Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating I Large I Language Models Trained on Code. WxXiv2207.00377 [cs], Jul 2021.
[cs], Jan 2025.
Dou, S., Liu, Y., Jia H., Xiong, L., Zhou, E., Shan, J., Huang, C., Shen, W., Fan, X., Xi, Z., Zhou, Y., Ji, T., Zheng, R., Zhang, Q., Huang, X., and Gui, T. StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback. arXiv:2402.01391 [cs], Feb 2024.
Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching Large Language Models to Self-Debug. In ICLR, 2024b.
Clement, C. B., Drain, D., Timcheck, J., Svyatkovskiy, A., and Sundaresan, N. PyMTS: Multi-mode translation of natural language and Python code with transformers. In EMNLP, 2020.
Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T.
10

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. S., Wang, X., Bubeck, S., Eldan, R., Kalai, A. T., Lee. Y. T., and Li, Y. Textbooks Are All You Need. arXiv:2306. I 1644 [es], Jun 2023.
D., Reddy, S., Fried, D., Bahdanau, D., Jernite, Y., Fer- randis, C. M., Hughes, S., Wolf, T., Guha, A., Werra, L. V., and de Vries, H. StarCoder: May the source be with you! Transactions on Machine Learning Research, 2023.
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The Curious Case of Neural Text Degeneration. In ICLR, 2020.
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Beeles, T. Keeling, J., Gimeno, F,, Lago, A. D., Hubert, T., Choy, P., d' Autume, C. d. M., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, J., Rob- son, E. S., Kohli, P., de Freitas, N., Kavukcuoglu, K., and Vinyals, O. Competition-Level Code Generation with AlphaCode. arXivv2203.07814 [cs]. 378:1092-1097, Dec 2022.
Islam, M. A., Ali, M. E., and Parvez, M. R. MapCoder: Multi-Agent Code Generation for Competitive Problem Solving. arXiv:2405. 1403 [es], May 2024.
Jain, K. Synnaeve, G., and Rozière, B. TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark. 2rXiv:2410000752 [cs], Oct 2024a.
Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is Your Code Generated by ChatGPT Really Correct? Rigorous Eval- uation Large Language Models for Code Generation. In NeurIPS, 2023a.
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama,, A., Sen, K., and Sto- ica, I. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv:2403.07974 [cs], Mar 2024b.
Liu, J., Zhu, Y., Xiao, K., Fu, Q., Han, X., Yang, W., and Ye, D. RLTF: Reinforcement Learning from Unit Test Feedback. TMLR, 1 11/2023, 2023b.
Kapoor. S., Stroebl, B., Siegel. S., Nadgir, N., and Narayanan, A. AI Agents That Matter. arXiv:2407.01502 [cs], Jul 2024.
Loshchilov, and Hutter, F. Decoupled Weight Decay Reg- ularization. In ICLR, 2019.
Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., and Scialom, T. GAIA: A benchmark for General Al Assistants. In ICLR, 2024.
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Ham- bro, E., Grefenstette, E., and Raileanu, R. Understanding the Effects of RLHF on LLM Generalisation and Diver- sity. In ICLR, 2024.
Olausson, T. X., Inala, J. P., Wang, C., Gao, J., and Solar- Lezama, A. Is Self-Repair Silver Bullet for Code Gen- eration? In ICLR, 2024.
Kumar, A., Zhuang, V., Agarwal, R., Su, Y., Co-Reyes, J. D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roelofs, R., Zhang, L. M., McKinney, K., Shrivastava, D. Paduraru, C., Tucker, G., Precup. D., Behbahani, F. and Faust, A. Training Language Models Self- Correct via Reinforeement Learning. 1rX1v:2409.12917 [cs], Sep 2024.
OpenAL. GPT-4 technical report. arXiv:2303. 2023.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. I.., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F,, Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training Language Models to Follow Instructions with Human Feedback. In NeurlPS, 2022.
Lc, H., Wang, Y., Gotmare, A. D., Savarese, S., and Hoi, S. C. H. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. In NeurlPS, 2022.
Popović, M. chrF: Character n-Gram F-score for automatic MT evaluation. In WMT 2015, pp. 392-395, Lisbon, Portugal, 2015. Association for Computational Linguis- tics.
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D.,. Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., Liu, Q. Zheltonozhskii, E., Zhuo, T. Y., Wang, T., Dehaene,
O., LLmam-mooiiies J., Monteiro, J.. Gontier, N., Yee, M.- H. Umapathi, L. K., Zhu, J., Lipkin, B., Oblokulov, M., Wang, Z., Murthy, R., Stillerman, J. T., Patel, S. S., Abulkhanov, D., Zocca, M., Dey, M., Zhang. Z., Bhat- tacharyya, U., Yu. W., Luccioni, S., Villegas, P., Zh- danoy, F. Lee, T., Timor, N., Ding, J., Schlesinger, C. S., Schoelkopf. H., Ebert, J., Dao, T., Mishra, M., Gu, A., Anderson, C. J., Dolan Solan-Gavitt, B., Contractor,
Rafailov, R. Sharma, A., Mitchell, E., Ermon, S., Man- ning, C. D., and Finn, C. Direct Preference Optimiza- tion: Your Language Model is Secretly a Reward Model. In NeurIPS, 2023.
Ridnik, T., Kredo, D., and Friedman, Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering. arXiv:2401.08500 [cs], Jan 2024.

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I,, Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Fer- rer, C. C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F, Touvron, H., Martin, I.,, Usunier, N., Scialom, T., and Synnaeve, G. Code Llama: Open Foundation Models for Code. arXiv:2308. 12950 [cs], Aug 2023.
Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. Siti::2307.09288, 2023.
Watson, C., Tufano, M., Moran, K., Bavota, G., and Poshy- vanyk, D. On Learning Meaningful Assert Statements for Unit Test Cases. In ICSE, pp. 1398-1409, 2020.
Xia, C. S., Deng, Y., Dunn, S., and Zhang, L.. Agent- less: Demystifying LLM-basse Software Engincering Agents, urXiv:2407.01489 [cs), Jul 2024.
Schick, T., Dwivedi- Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language Models Can Teach Themselves to Use Tools. In NeurlPS, 2023.
Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., Wang, G., Yu, C., and Wu, Y. DPO Superior to PPO for LLM Alignment? Comprehensive Study. In ICML, 2024.
Schulman, J.. Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal Policy Optimization Algorithms. orXiv:1707.066 [cs], Aug 2017.
Yang. J., Jimenez, C. E., Wettig, A., Lieret, K., Yao. S., Narasimhan, K., and Press, O. SWE-agent: Agent- Computer Interfaces Enable Automated Software Engi- neering. arXiv:2405. [es]. May 2024.
Shao, Z., Wang, P., Zhu, Q. Xu, R., Song, J., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. DeepSeekMath: Push- ing the Limits of Mathematical Reasoning in Open Lan- guage Models. 3Xiv:2402.00300 [cs], Feb 2024.
Yao, S., Chen, H. Yang, J., and Narasimhan, K. Web- Shop: Towards Scalable Real-World Web Interaction with Grounded Language Agents, In NeurlPS, 2022.
Shavit, Y., O'Keefe, C.,, Eloundou, T., McMillan, P., Agar- wal, S., Brundage, M., Adler, S., Campbell, R., Lee, T. Mishkin, P., Hickey, A., Slama, K., Ahmad, L., Beutel, A., Passos, A., and Robinson, D. G. Practices for Gov- erning Agentic AI Systems. 2023.
Yu, Z., Tao, Y., Chen, L., Sun, T., and Yang, H. B-Coder: Value-Bass Deep Reinforcement Learning for Program Synthesis. ar 2rXiv:2310.031 173, Mar 2024.
Shinn, N., Cassano, F,, Berman, E., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Lan- guage Agents with Verbal Reinforcement Learning. In NeurIPS, 2023.
Zhai, Y., Bai, H., Lin, Z., Pan, J., Tong, S., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., and Levine, S. Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning. arXiv:2405. 10292 [cs], May 2024.
Shojaee, P., Jain, A., Tipirneni, S., and Reddy, C. K. Execution- based Code Generation using Deep Rein- forcement Learning. TMLR, 07/2023, 2023.
Zhang, Y., Ruan, H., Fan, Z., and Roychoudhury, A. Au- toCodeRover: Autonomous Program Improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024.
Sutton, R. S. and Barto, A. G. Reinforcement Learn- ing: An Introduction. Adaptive Computation and Ma- chine Learning Series. The MIT Press, Cambridge, Mas- sachusetts, second edition edition, 2018. ISBN 978-0- 262-03924-6.
Zhong, L., Wang. Z., and Shang, J. Debug like Human: A Large Language Model Debugger via Verifying Run- time Execution Step-by-step. arXiv:2402.16900 [cs], Jun 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi,
A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P.,
Zhou, Y., Zanette, A., Pan, J., Levine, S., and Kumar, A. ArCHer: Training Language Model Agents via Hierar- chical Multi Turn RL. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024.
Bhosale, S.. Bikel, D., Blecher, L., Canton-Ferree C.,
Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J. Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H.. Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-.,, Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizen- stein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, X., Xu, P., Yan, Z.,,
Ziegler, D. M. Stiennon, N. Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-Tuning Language Models from Human Pref- erences. /oXiv:/909.08555 (cs, stat), Jan 2020.
12

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
A. Experimental Details
A.1. RLEF
We initialize separate policy and value function networks from pre-trained and instruction-tuned LLMs as indicated the respective experiments; for the value function, we replace the output layer with randomly initialized linear projection. For PPO, use AdamW (Loshchilov & Hutter 2019) with learning rate of 2e-7, weight decay of 0.1, and linear
warm-up over 50 steps. We set the KI. regularization factor / of the reward term to 0.05 (Section 2.2 All models
trained with an online, asynchronous training infrastructure that decouples inference and optimization. We incorporate
importance sampling in PPO's clipped surrogate objective (Schulman et al. 2017 Eq.7):
(0) To (e(e()))) Tea ((r|ct)
_ 11)
stop-gree grad min
mx(ap)(t))
L (0) E. min (r,(0),,,, clip
A.)
for model parameters 0, normalized advantage A,, and the behavior policy Tb. We set = 0.2.
For optimizing the value function, we use clipped value loss. With value model parameters and reward function R R(st, ax) (see Section 2.2 have
R₂
Σ R(s;,a));
i=t LV LV(W) (+) E. 2
max ((Vo(c)) RR)) (clip (Vo(c)), Vana (ct) Vona aa)) -
where we the discount factor y and the value clipping threshold a 0.2.
During training, perform inference with temperature of 1.0; collect 1024 rollouts and perform 4 updates on 256 sequences each. Models are evaluated every 800 updates, and we select the final model based on validation set performance. We train our models on NVidia H100 GPUs; a training run takes approx. 20 wall time hours. With the above parameters we use 288 (128 for training, 160 for inference) and 2304 (1024 for training, 1280 for inference) GPUs for 8B and 70B models, respectively.
neither nucleus (top-p) top-k sampling. We
A.2. Code Execution
We evaluate candidate solutions with the accompanying code-base of Li et al. 2022 using Python 3.10. All problems in the validation and test set specify a memory limit, and only a few problems define a time limit. specified, we apply these limits for RLEF training and evaluations; otherwise, we use a 1GB memory limit and maximum wall clock time of 10 seconds per test case.
A.3. Supervised Vine-Tuning
We perform supervised fine-tuning (SFT) for the ablations Section 3.4.1 In order to assemble a training dataset, we perform iterative code generation with our proposed setup on the CodeContests training set with the Llama 3.1 70B Instruct model. We top-p to 0.95 and sample temperature for each response in U(0.1, 1.0). For each problem in the training set we collect 100 multi-turn rollouts and obtain 313,639 successful trajectories.
We fine-tune models for next-token prediction, computing losses on the last response only (i.e., on responses passing both public and private tests); this produced slightly better models compared to training on all responses. We sweep over learning rates 5e 6 and 2e-6, -6 and and epochs with batch size of 64 and sequence length 8192. A linear warmup is performed over 10 steps, and learning rates are annealed according to a cosine schedule. Weight decay is set to 0.1. Models are evaluated after 200 optimizer steps with AdamW and select final parameters based validation performance.
htttps:/ com goog e- deepm deeepmindii http://cooeoooeeeeseeee
13

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Table 5: I @@3 solve rates with 95% confidence intervals on tasks from Table as well as LiveCodeBench up to 10/2024.
Model
Setting
CC. Test
HumanEval+
MBPP+
LiveCodeBench
GPT-4o
single-turn
25.3 (23.2-26.3)
82.8 (81.8-83.2)
68.8 (68. 16..1)
49.9 (48.7-50.0)
multi-turn
24.3 (23.2-25.4)
80.7 (79.8-81.7)
71.7 (71.4-2222
48.4 (48.0-49.0)
Llama 3.1 8B Instruct
single-turn
11.6 (11.3-12.0)
65.3 (63.8-66.2)
58.3 (57.5-58.7)
22.5 (21.6 6-22.7)
multi-turn
10.5 (10.1-10.8)
63.9 (62.8-65.1)
60.4 (59.7-60.9)
23.1 (22.6-23.5)
RLEF
single-turn
9.7 (9.5-10.0)
67.5 (66.7-68.0)
57.0 (56.4-57.2)
22.7 (22.4-22.9)
multi-turn
16.1 (15.8-16.4)
69.5 (68.6-70.5)
63.1 (62.7-63.5)
27.8 (27 (1774-28.1
Llama 3.1 70B Instruct
single-turn
26.2 (25.6-26.6)
73.2 (72.5-73-9)
66.9 (66.1-67.4)
38.1 (37.3-38.4)
multi-turn
27.4 (27.0-27.8)
75.0 (74.3-76.0)
70.2 (69.7-70.8)
40.9 (40.4-41.3)
+ RLEF
single-turn
30.1 (29.7-30.4)
78.7 (78. 1-711)
67.6 (67.2-67.888
36.9 (36.5-37.0)
multi-turn
40.1 (39.7-40.4)
80.4 (79.7-81.0)
72.2 (71.9-2.55)
42.4 (42.2-42.7)
Table 6: (a) @@3 solve rates for few-shot prompting and supervised fine-tuning (SFT) with Llama 3.1 Base models on CodeContests. (b) Further results from Llama 3.1 Instruct 8B (1 @3): withholding execution feedback from public training during RL; learning value function the token level; training dedicated code repair model and applying it to outputs of the single-turn RL model.
Model
Method
Valid
Test
Method (8B)
Valid
Test
8B Base
Few-Shot
1.2
1.8
RLEF
17.2
16.1
SFT
6.9
3.5
No Execution Feedback
12.2
10.9
Token-level Value Function
13.1
13.7
70B Base
Few-Shot
4.6
5.8
SFT
11.1
10.9
Single-turn RI.
10.2
10.9
(a)
Single-turn w/ Repair
14.8
12.6
(b)
B. Additional Experimental Results
B.1. Extended Results on Generalization
Table 2 lists performance of RLEF-trained models and GPT-4o on benchmarks that are not strictly within the competitive programming domain. We report confidence intervals for all results, along with results on LiveCodeBench (Jain et al. 2024b), Table We select LiveCodeBench questions up to October 2024.
We follow Li ei (0222) computing 95% confidence intervals for n@k results: given an evaluation with N samples, draw N results with replacement and estimate 1 @@3 solve rates, repeat this 200 times, and select the 2.5 and 97.5 percentiles lower and upper bounds of the confidence interval.
Table 5 shows that gains from RLEF-training on CodeContests on other tasks are statistically significant in our evaluation setting, where we draw 20 multi-turn samples per model and task combination. For single-turn results, we consider the first generation of each sample only. For consistency, we use 200 samples on CodeContests (except for GPT-4o) as in Table [ Confidence intervals for GPT-4o on CodeContests are therefore wider.
B.2, Pre-trained Models
In Table 6a we list solve rates for few-shot prompting and supervised fine-tuning from pre-trained Llama 3.1 models. We observe significantly lower performance compared to the Instruct models all cases (Table 3).
14

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforeement Learning
B.3. Feedback from Private Tests
Our main evaluations on CodeContests match our training setting, i.e. we provide inference-time feedback on public test cases and estimate solve rates on private (and the dataset's generated) tests. The number of public test cases in the CodeContests validation and test sets vary between 1-7, with median of 1; typically, a higher number of private tests and a large number of generated tests are available per problem.
We verify whether our RLEF-trained models can benefit from larger test sets during inference by including feedback from private and generated tests. Specifically, we test each model response against 20 available test cases, including private tests, and provide execution feedback for up to failed test cases. Comparing @@3 solve rates (temperature 0.2) with a turn limit of 3, the 8B RLEF model can improve from 17.2 18.1 on the valid set, whereas on the test set we see a drop from 16.1 to 14.4. For the 70B RLEF model, validation set performance improves from 37.5 to 40.4, and on the test set obtain 41.2 compared to 40.1 with feedback limited to public tests.
B.4. Extra Repair Model
Le et al. 2022) implement program repair on top of an RL-trained LLM with two extra models: a "critic" predicts the joint outcome of all unit tests (c.g., success, failure, runtime error) and can be used for ranking and determining promising prefixes, and a "rrppir" model maps wrong solutions to ground truth solutions. In this spirit, we evaluate the effect of a dedicated repair model to improve the single-turn 8B model from Section 3.4.2 as follows.
During the RL training procedure, we collect all generations that do not pass the public unit tests. For the training duration of 12,000 gradient steps, this amounts to 1.48M samples. Next we construct training dialogues with the original prompt (as described in Appendix [.T), the wrong generation, and random correct generation for the respective problem from the CodeContests training set. We apply additional processing to the CodeContests solutions by making sure they do indeed pass the provided unit tests and unifying their indentation. We then train repair models via supervised fine-tuning of Llama 3.1 8B Instruct, sweeping over learning rates 5e 6 2e- and le and I or 2 epochs with a batch size of 64 and a sequence length of 8192.
For evaluations, we estimate 1@3 solve rates by generating an initial program with the RL-trained model followed by up to two independent samples from the repair model. Similar to our main RLEF setting, we refrain from (further) repair if the latest solution passes the public tests. We evaluate all models from the sweep in intervals of 400 gradient steps and select the best checkpoint based on validation set performance. This checkpoint achieves, in combination with the single-turn RL model, a 1 @3 solve rate of 14.8 on the validation set and 12.6 on the test set, which is significant increase over the single-turn RL model alone (10.2 and 10.9, respectively; Table but falls short of the corresponding RLEF-trained model which combines code synthesis and code repair (17.2 and 16.1, respectively; Table I
B.5. RL Training Without Public Test Execution Feedback
We validate our setup consisting of inline execution feedback and early stopping based on public tests (Section 2.1 with an ablation where we withhold information from public tests. Concretely, we remove execution feedback from the prompt for subsequent solutions (Appendix C.T), starting directly with "Give it another try". We always ask the model for two follow-up solutions this way (i.e., for a total of three solutions). We do keep our reward definition from Section 2.2 but do not end episodes when public tests are passing.
The resulting model, starting from Llama 3.1 8B Instruct, obtains 1 @@3 solve rate of 12.2 on the validation and 10.9 on the test set (Table 6b). This is better than the initial instruct model (8.9 and 10.2, respectively) but significantly below the corresponding -rained model (17.2 and 16.1, respectively).
B.6. Token-Level Value Function
Here we do not train a value function the level of responses (Section 2.2 Appendix A.1 but rather predict a value for each token of response. Our reward formulation remains unchanged; consequently, due to the discount factor being set to 1, the value function target (reward-to-go for each token of a response is the same. However, we now compute separate per-token advantages.
With this approach and otherwise identical settings, we achieve a 1 @3 solve rate of 13.1 on the validation and 13.7 on the test set, starting from Llama 3.1 8B Instruct (Table 6b). This is below the 17.2 and 16.1 results with the turn-level value
15

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
function (
C. Prompts
C.1. CodeContests
In the initial prompt, we substitute (probbem) by the original problem description as-is.
Initial Prompt
Provide Python solution for the fellewing competitive programming question: / problem).
Your code should be enclosed in triple backticks like so: 'python YOUR CODS HERE "', Use the backticks for your code only.
In the execution feedback prompt below, we show templates for the four different emror types we consider: wrong answer, exception, timeuut, and out of memory. We then show the respective feedback for each failing test.
Execution Feedback
Your code failed the following tests:
input (inputt failed: Expected output (((xpectee_output) but got '$ "s/obseevee_output) input "S(input)" failed:
ratacktrace)
input "S(input)" failed: Execution teck too long. input "((inptt]"" failed: Out of memory.
Give it another try.
Your code should enclosed triple backticks like so: "'ython YOUR CODE HERE "o, Use the backticks for your code cnly.
C.2. Random Feedback Ablation
In Section 3.3 test RLEF-trained models with random execution feedback. For each problem, sample different problem from the respective test set that contains incorrect solutions. We obtain unrelated feedback by evaluating one of these incorrect solutions, chosen at random, against the corresponding public tests and present the resulting feedback to the model. If none of the incorrect solutions fail the public tests, we evaluate raise Not wemplpmmeeeeeeee case, the feedback will contain backtraces pointing to this error. Otherwise our dialog proceeds as usual, i.e., the code solution produced by the LLM passes the true public tests of the problem in questions we stop and evaluate solution on all test cases.
(). In this
C.3. Few-Shot Prompting
For the few-shot ablations in Section 3.4, we select successful trajectories from Llama 3.1 70B Instruct model on prob- lems from the CodeContests training set. We select trajectories with both 2 and successful attempts as demonstrations for successful multi-turn code generation. For instruction models, initialize the dialog with the few-shot examples, separating them with an empty assistant message. For few-shot experiments with pre-trained models (Appendix [B.) we use dialog format in which each message is either prefixed by [USER] or [ASSTSTANT]. The token for l, an invalid symbol in Python, is used as message delimiter.
C.4. HumanEval+
HumanEval problem prompts consist of starter code, with docstring and example tests following the function declaration.
16

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Initial Prompt
Write solution to the following problem and make sure that it passes the testa: (problem)
then provide the problem prompt again at the start of each model response for completion.
The tests in HumanEval+ consist of single function with several assert statements. In order obtain execution feedback for individual tests, extract them from original test function (for computing pass rates, the original test code). We further transform assert statements into matching function calls of Python's 's built-in class. This way, test failures will result in more informative Assert onError exceptions with run-time values; these are provided as asserticn.eree to the template. We also show successful test cases.
TestCase
Execution Feedback
Your code failed test cases:
Failure: (((tes))
1$ [aassertion_erro))
Failure: '9(test)',
(stacktrace)
Failure: 'S(test)";
Execution took too long. Success: "((tet))"
Give it another try.
C.5. MBPP+
Each MBPP prompt consists of problem description and single example test.
Initial Prompt
Provide a Python solution for the follewing problem: ((problem) Your code should pass these tests:
(test)
Your code should be enclosed in triple backticks like so: "'python YOUR CODE HERE "., Use the backticks for your code only,
Execution feedback follows the HumanEval+ format from Appendix C.4 with additional formatting guidelines.
17

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Execution Feedback
Your code failed some test cases:
Failure: "S(test|';
(error))
-Failure: "((test));
ractacktrac)
Failure: "9(tes))";
Execution tock too long.
Success: "{(test)"
Give it another try.
Your code should be enclosed in triple backticks like so: "python YOUR CODE HERE ",, Use the backticks for your code only.
D. Examples
The following examples are selected from the valid set of CodeContests with the RLEF-trained 70B model, using temper- ature 0.2 and top-p 0.95. We apply some light re-formatting of the initial prompts for better readability.
In the first example, the first model response on the right track, but the first print () statement outputs a wrong value. The second response implements the requested algorithm naively, the code inefficient due the loop from I to r; would take about two minutes for the last line in the public test. Finally, the third response contains code that both correct and fast.
Initial Prompt
Provide solution rollowing competitive programinng question: largest possible value As semindor, ned is a romaindor we get when dividirg by b. For example, 26 nod 8 2.
Find
over pairs integere which
Eech test contains maitiple test cases. The first 1ine containa ene posltive Inzegez (1 S 10"41 denoting the nunber of teat cases. Descristion of the test
follows.
only
case
integers 1, $ (0"9).
Output
every case, cutput largest possible
paira (a, b) integera 222222
3xample
1006003000
8 26
995999993
Outpet
699399399
Io the first teat case, the only allee pair is (e, b) (1, 10, For which a nod b mod In the second Lest case, the optinal choiow in paix (190E000000, 9399999991, For which and 1., Your should enciesed triple backticks ""pythoa HESE ..., the backticks code
18

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Model Response
"pytaon
inc Linput (1)
range (C)
1, naptink, Angut O ())
//
primutte // 231
else:
prina Cr
Execution Feedback
lowing
input '4
28
999999999
taille:
Xxpected output
12
499999999
got
13
629899399
Give 1t another try.
Your should enclesed triple backticks Like "python CODE HERE 11, Use the backticks
Model Response
"pytaon
inz (1nput())
For
rangeit):
1, x maprint, Lngut ())
max
for in z1nge(1, F 2):
(ncx_nndd,
print maax_mod)
Execution Feedback
failed the following
iapu: "4
1
699959999 1000000003
999999999
[ailed:
Exeaction
Glow another Lry.
Your code should be enclesed in triple backticks like ""python YOUR CODE HERE **,, Use the backticks for your code only.
19

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Model Response
"pytaoo
inc Linpu. (1)
rango
1,
naptint, Engul O1 ())
prim)(0)
else:
t= // 25 1
ir 1:
prant (D) wlaez
pzint 11
Result
Pesses public
private tests.
20

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
In the second example, the first code solution does not reset the value for Pk- The second and correct attempt fixes this. Note that both solutions contain unused variable, curr in the first and curr in the second response. For this question, approach with dynamic programming that starts initializes pt from P would be preferred; however, the second solution passes all test cases, too.
Initial Prompt
Provide Fychon solut the following coneetitive proceaming questions have areay consiating of positive integers, numbered fron to n. Pefine PLK as 989p_k " E-11 S i, 1 S kI aut nod aLd, vhera x Mondd y donotes tha cenaindez shen divided by You have Find and grint BLR, Vidols, PLnsSS.
The first Line contains the length of the array (2 S 2 10151 contains space-soporazed distinct interars
#F 1
Outpct
Print inbagers pLI, BL2, ... Ra
Xxamples
2
Outpet
12 22
Togust
Ootput
Your code should be enclesed in triple backticks like s0: "python YOUR cope HERE 19,, tee the backticks for your code only.
Model Response
"pyyton
iit Hippt (1)
1ist (map link, Eoput 13 (pplLL
p resslt [1
For k ir rangeitn::
=[k]
tknge
for 5 in range 11:
- ali) a1]1
repeit.appeeddp)
for run in result:
print irun, end
Execution Feedback
failed the following
iiput " 2
ffalled:
Expected output
bbt got 10 2 14 inpuz '3
Expected output 3
bat yot '0 8
Give another try.
Your oode should be unclosed in Leiple backticka 1ike sos "Pyythoa YOUR CODE HERE **", Use the backLicka For your code only.
21

---

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Model Response
"pytton
inc Limput (1)
.aplit
swaull [1
total 0 (a)
surs_-bial
cuEs_á al:k-1]
For i in (k 1):
in
botal 1- wEl) +131
swall. agponstturrrcttttll
cun result:
print Iran, enda" *)
Result
Pesses public bests.
22


Reviews:
[{"review_id": "v8pRZyQw45", "reviewer": "Reviewer_ZGUC", "summary": "This paper proposes an RL training strategy for Code LLMs to enable them to refine generated code using execution feedback besides the capability of following instructions. They present an exhaustive analysis on different aspects of their RLEF-trained models including their inference time behavior, performance gains at different sampling budgets, comparison with other state of the art solutions, and algorithmic alternatives like SFT, single turn RL training, prompting, etc.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough analysis of our paper, their valuable suggestions and stimulating questions.\n\n## Updates\n\nRe. GRPO: We will add the following to the end of our related work section:\n\nMore recently, DeepSeek-AI et al. (2025) observe emerging reasoning capabilities with a large-scale application GRPO (Shao et al., 2024) to math and code problems and achieve high performance on competitive programming tasks. We thus consider the training of reasoning models with program execution feedback and, likewise, the introduction of execution feedback to math domains, a promising avenue for future research.\n\n> Line 244: What does \"stock\" mean here?\n\nThis refers to the officially released Llama models, in this case Llama 3.1 70B Instruct; we will clarify the wording.\n\n## Answers to Questions\n\n### Sensitivity to Execution Feedback\n\nDue to space constraints, we refer the reviewer to our response to Reviewer 3JDi below, where we discuss the choice of random execution feedback and also list solve rates for a \"no execution feedback\" test.\n\n### Granularity of Reward: Turn- vs. Token-Level\n\nWe provide experiments regarding a token-level value function in Appendix B.5. With token-level rewards, we found that the KL penalty biases the model to unreasonably short generations in intermediate turns, and in B.5 we test the combination in which we still provide averaged KL penalties per turn but a learn token-level value function (and hence obtain per-token advantages). We observe worse results with this approach.\n\n### Discarding Rollouts\n\nWe generally do not discard rollouts for evaluation and do not consider public tests in scoring unless noted otherwise. A rollout with 3 turns without a solution passing the private tests will count as a failed sample, similar to rollouts with 1 or 2 turns where public tests are passing. A rollout with a successful result will count as a successful sample. In other words, one rollout corresponds to one sample.\n\nTo compare under equal sampling budgets, we then consider the \"1@1\" and \"1@33\" performances in this setting as \"1@3\" and \"1@100\" solve rates. NB, models will not utilize the full budget due to some rollouts ending early with correct responses. \n\n### Results with Greedy Decoding\n\nWe observe slightly better results with a low temperature compared to greedy decoding.\n\n### Diversity in Outputs\n\nThe question of whether a loss of diversity is a concern depends on the intended applications. Remedies can be found on the RL algorithm level as shown in https://arxiv.org/abs/2503.19595, for example. We found a temperature of 1.0 to deliver best performance in the large-sample regime; this may be linked to the fact that we use a temperature of 1.0 during rollouts as well.\n\nRe Line 286, output diversity depends on the evaluation setting. From the analysis in Table 3 it is clear that base models do not sample diverse solutions *within* a rollout. However, they can produce a large number of different solutions from the initial prompt, in particular with higher temperatures. What makes RLEF-trained models effective is that they can utilize execution feedback to either repair an existing solution or to propose a new approach.\n\n\n### Figure 3\n\nWe selected to show \"errors fixed\" here to highlight differences between the different models and settings. The X-axis scale is different for \"Errors\" and \"Errors Fixed\". While we regard the reported differences as significant, they would be hard to discern visually under the X-axis scale of the leftmost plots. \n\n\n### Gap Between Instruct and RLEF models wrt. Sample Budget\n\nWe did not study the diversity aspect in the limit, i.e., under very large sample budgets. We would expect the gaps to narrow in the limit due to decreased output diversity. An interesting question would however be if this can be offset by sampling longer and longer trajectories.\n\n\n### SFT Dataset\n\nYes, the SFT dataset was obtained solely from the CodeContest training dataset. It consists of 313,639 trajectories. The result that RL can work better than SFT is in line with references discussed in the paper, i.e., Xu et al and Kirk et al, and also the recent results around DeepSeek-R1.\n\n\n### Existing MT capabilities of 70B\n\n> Section 3.4.2 - \"We attribute this to the existent but comparabily weak multi-turn capabilities of the vanilla ...\" I don't agree with this conclusion, and I would instead attribute this to the RLEF (ST) training given that the instruct model doesn't benefit much from MT during inference (25.6 to 25.9), whereas after RLEF the gains are a lot more pronounced (28.3 to 31.1).\n\nWe were referring to the fact that the 70B model already exhibits multi-turn capabilities (although the valid set difference is not statistically significant) when compared to the 8B model, for which performance drops in the multi-turn setting.", "claims_and_evidence": "I find the evidence presented through the experiments in this paper convincing overall, except for one issue when studying the inference time behavior described below.\n\n- Fig 3 / Line 295: The goal here is to study and establish the sensitivity of RLEF trained model to the feedback from execution. I think using no feedback would have made more sense to be studied as a baseline to compare against, as random feedback included in the prompt to an LLM at any turn is very likely to hurt model performance.", "methods_and_evaluation_criteria": "In my assessment, the proposed methods and evaluation criteria make a lot of sense for the problem of code generation from natural language instructions.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes, I find the experimental design and analyses to be very sound.", "relation_to_broader_scientific_literature": "This work is related to prior work on code generation like AlphaCode, AlphaCodium, Mapcoder and CodeRL.", "essential_references_not_discussed": "I understand techniques like GRPO gained prominence only recently with the release of Deepseek-R1, but I believe adding commentary or comparison to such alternatives to PPO and the requirement of the value function network can benefit the placement of the paper in the present day context of LLM research. E.g. algorithms discussed in Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs (Ahmadian et al)", "other_strengths_and_weaknesses": "Strengths:\n\n- Idea is very well-positioned - motivating RL for training agents that can follow instructions and incorporate feedback from environments\n\n- Adaptation of PPO to the setting of code as described in Section 2.2 is non-trivial\n    - Commentary on how they differ from prior work (Le et al (CodeRL)) could help establish their novelty in Section 2 itself, instead of deferring this discussion to the related work.\n\n- Results are convincing\n    - CodeContests is a very challenging benchmark, and their baselines include prominent agentic frameworks\n    - Authors have augmented their findings with good intuition for readers based on their ablation studies (Section 3.3, Line 281 onwards). I particularly liked the thorough nature of the study performed on inference time behavior.\n    - Authors have put in great care in ensuring baselines are reasonable to compare with (for eg - see Line 209)\n\n- Paper is well written and the caveats of the methodology design are sufficiently described", "other_comments_or_suggestions": "Suggestion: Discussion on extension to domains beyond code synthesis is only loosely referred to towards the end of Section 4, but the introduction section seems to offer greater promise. For instance, authors could discuss how execution feedback can be generalized to collect beyond code - on verifiable domains like math related tasks, and how all such signals combined (human, execution environment, math correctness) can lead to training of more capable LLM agents. Positioning this contribution in the context of recent advances like Deepseek-R1 will greatly benefit this paper.\n\nMinor/typos:\n\nLine 306: the a --> the\n\nSection 3.2: Para 2: with a single rollout compared to 5 solutions from 100 samples (38.0 and 29). Did you mean 40.1 here instead of 38? Otherwise the results in Table 1 don't match with this description.\n\nLine 244: What does \"stock\" mean here?", "questions_for_authors": "- In Section 2.2, you mention the choice of granularity of reward / advantage being action level instead of token level. Can you elaborate more on the alternatives to this choice and what impact one could expect from choosing token level reward or advantage?\n\n- Section 3.2 Para 1: \"Each solve rate is estimated on 200 rollouts ...\" Here do you not discard the rollouts where the final solution obtained does not pass public tests?\n\n- Table 1: What would be the impact of sampling temperature on these results? Why was greedy decoding not attempted with the multi-turn single rollout setup? (where 1 rollout <= 3 samples/turns?)\n\n- Line 252: To combat the effect mentioned here (RL training reducing diversity of outputs) can one use higher temperature? What could be other ways of addressing this concern? Is this a concern at all?\n\n- Line 286 - you attribute the higher scores in iterative setting to increased diversity in sampled solutions. This seems to contradict with Line 250 where you cite Kirk et al (2024) who find that RL training can reduce diversity also suggested by your results in Table 1. I am confused with the conclusions from these 2 lines.\n\n- Fig 3: Why do you choose to indicate errors fixed in turns 2 and 3? As a reader I think showing the drop of errors over the turns would be easier to follow through this plot.\n\n- Fig 4b: Can you comment on the gap between the instruct and RLEF models? Does it show any upward or downward trend with the sample budget?\n\n- Section 3.4.1 - What is the scale of the SFT dataset synthesized for this study? I'm not certain if suggesting the ineffectiveness of SFT for iterative code refinement is a valid takeaway. I think this finding needs to be augmented with a solid explanation - which could perhaps be the difficulty in designing synthetically / collecting supervised datasets for this task.\n\n- Section 3.4.2 - Is the single turn SFT dataset coming from Code Contests benchmark? Is the takeaway here that this training set is most effective only when used through the RLEF stage (which uses only the test cases and not the ground truth solutions)? Generally SFT datasets are used to fine-tune a checkpoint, and then environment feedback is used to train in the RL phase, but I see no such comparison here. What'd the result be if RLEF was performed on the SFT (single/multi turn) trained checkpoint?\n\n- Section 3.4.2 - \"We attribute this to the existent but comparabily weak multi-turn capabilities of the vanilla ...\" I don't agree with this conclusion, and I would instead attribute this to the RLEF (ST) training given that the instruct model doesn't benefit much from MT during inference (25.6 to 25.9), whereas after RLEF the gains are a lot more pronounced (28.3 to 31.1).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sE4C3avQoq", "reviewer": "Reviewer_JjJq", "summary": "This paper introduces RLEF - a reinforcement learning method for improving natural language to code generation in an iterative setting. The method treats code generation as a multi-turn conversation, where a language model first produces a program then receives and interprets textual execution feedback to refine solution. This feedback is incorporated as part of the RL training loop. At each turn, the model’s reward depends on whether the final solution passes a private unit tests, while intermediate attempts are guided by test outputs in the prompt based on public tests.\n\nThe model is evaluated on the CodeContests competitive programming benchmark showing that RLEF-trained models achieve notably high accuracy using fewer attempts than earlier approaches such as AlphaCode and more recent GPT-based agentic pipelines. The authors provide analyses showing that the model not only generates correct solutions more often but also reliably fixes its own errors in response to execution messages.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review and valuable feedback.\n\nBased upon their feedback, we investigated the performance of DeepSeek-R1-Distill-Llama-70B in our exact same setting. With a single generation (temp=0.6, prompted with \"<think>\\n\"), we obtained solve rates of 38.5 and 33.9 on the valid and test sets; with execution feedback over three turns, we obtained 41.0 and 37.6, respectively. This places the model in the same ballpark as our RLEF-trained 70B model (37.5 and 40.1).\n\nHowever, we also note that this required a vastly increased inference budget compared to our models, with up to 10k thinking tokens per turn (this limit would often be reached, and we force-close the thinking section in this case). This renders the comparison quite difficult as our paper places a large emphasis on apples-to-apples comparisons, and allowing for large reasoning budgets would effectively shift the goal posts.\n\nThat said, it is evident that DeepSeek-R1 or o1/o3 excel at competitive coding tasks. Therefore, we will update our abstract, intro and experimental work section and do no longer claim state-of-the-art performance in competitive coding tasks.\n\n> AutoCodeRover (Zhang et al., 2024) and other recent automated debugging agents that iteratively fix GH issues using test failures as guidance.\n\nWe thank the reviewer for this pointer; we added a reference in our related work section.\n\n> Limited evaluation scope\n\nWe fully agree that extending our work to open-ended and long-horizon coding tasks SWE-bench is an exciting future direction of our work. For the current paper, however, we consider this out of scope.\n\nFor SWE-bench in particular, current research suggests that substantial domain-specific fine-tuning is required to achieve competitive results. For closed-source frontier models, performance started to increase rapidly once the benchmark was available and the distribution of tasks was known.\n\nWe note that we exclusively train on the CodeContest training set, and we would expect that widening the evaluation scope towards new settings will likely come with new training data requirements.\n\n> No direct comparisons to GPT-4, DeepSeek-R1 or other leading models for code.\n\nWe discuss DeepSeek-R1 at the top of this reply. We include GPT-4o in Table 2, and GPT-4 is used in AlphaCodium and MapCoder (Table 1) which we outperform.\n\n> RLEF uses binary pass/fail rewards, but no ablation tests whether denser rewards (e.g., partial credit for fixing certain test cases) could accelerate training.\n\nWe agree with the author but defer this to further work. Here, we showed that even with a sparse outcome reward we can achieve large gains. We did not experiment with denser rewards.\n\n> RLEF vs. Reflexion or Self-Refine\n\nWe refrained from applying prompting strategies in our work and instead focused on domain-specific fine-tuning. However, CodeTree (https://arxiv.org/abs/2411.04329) contains results on the CodeContests test sets with Reflexion (Table 2). With Llama 3.1 8B, they obtain a solve rate of 13.5 with 20 samples. Our RLEF-trained version obtains 16.0 with a budget of 3 samples.\n\nIn our paper we also experimented with few-shot prompting (Table 3) but found it to not work well for our use-case.", "claims_and_evidence": "Most of the paper’s key claims appear to be backed by concrete evidence. In particular, the central claim that RLEF enables a model to iteratively repair its output and reliably improve code solutions is supported by detailed experiments on CodeContests.", "methods_and_evaluation_criteria": "The proposed method focuses on iterative code generation and repair, where a model generates an initial solution, receives execution feedback (error messages, test results), and refines its solution. The method also makes sense from an agentic AI perspective, as it pushes models toward self-correcting behavior, which is critical for real-world coding applications.\n\nThe evaluation criteria and benchmarks chosen for testing RLEF -- CodeContests, HumanEval+, and MBPP+ -- are reasonable choices, as they represent progressively challenging levels of function synthesis and competitive programming tasks. However, one potential limitation is that these benchmarks focus primarily on single-function correctness and may not fully assess RLEF’s potential for broader software engineering workflows (e.g., multi-file program synthesis, debugging, GitHub issue resolution). While HumanEval+ and MBPP+ demonstrate some generalization beyond competitive programming, additional real-world benchmarks, such as SWE-bench or GitHub Issues, could further validate RLEF’s applicability to industrial coding tasks. Nonetheless, for the specific problem of iterative function-level code improvement, the chosen evaluation framework is sound, and the results convincingly demonstrate the effectiveness of execution feedback-driven RL.", "theoretical_claims": "The paper does not introduce new formal theoretical proofs but relies on established RL theory, specifically PPO. The correctness of PPO’s formulation is well-documented, and the paper applies it appropriately without requiring independent verification.\n\nHowever, the paper makes implicit theoretical claims, such as:\n* Multi-turn execution feedback leads to better policy learning than independent sampling, and\n* Binary pass/fail rewards are sufficient for meaningful RL-based improvement.\n\nWhile these claims are empirically validated, they lack formal guarantees. Additionally, exploring whether dense reward shaping (like, penalizing specific error types differently) would improve learning efficiency remains largely an open question.", "experimental_designs_or_analyses": "The paper’s experimental setup is well-structured, using CodeContests, HumanEval+, and MBPP+ to test RLEF’s ability to improve NL to code generation through execution feedback. Solve rates, pass@k, and sample efficiency comparisons are appropriate metrics, and multi-turn evaluation reflects real-world debugging scenario. Ablation studies (like e.g. random feedback) confirm that the model genuinely learns from execution feedback rather than random resampling.\n\nThe experiments are strong and well-designed, but direct DeepSeek, GPT, etc model benchmarking, real-world coding tasks, and alternative reward structures would further validate RLEF’s generalization and efficiency.", "relation_to_broader_scientific_literature": "The paper builds on reinforcement learning for code generation, drawing from RLHF but replacing human preference signals with automated execution feedback. This aligns with some prior work like CodeRL (Le et al., 2022) and RLTF (Liu et al., 2023), which also optimized LLMs using execution-based rewards. However, RLEF extends this by treating execution feedback as an interactive state rather than a static reward, enabling multi-turn code refinement instead of single-shot optimization.\n\nRLEF also connects to LLM-as-agent frameworks, such as AutoCodeRover (Zhang et al., 2024), which resolves GitHub issues via iterative debugging. Unlike AutoCodeRover, RLEF trains the policy itself, rather than relying on prompting alone. Additionally, it relates to DeepSeek-R1, which also leverages execution-based RL, but RLEF explicitly integrates textual execution feedback into decision-making, making it more akin to self-debugging code models. These connections places RLEF at the intersection of RL-based LLM fine-tuning, execution-grounded AI agents, and automated software repair.", "essential_references_not_discussed": "AutoCodeRover (Zhang et al., 2024) and other recent automated debugging agents that iteratively fix GH issues using test failures as guidance. While they use tool-based error localization, RLEF’s policy learning approach could complement these methods, making their inclusion relevant to agentic coding workflows.", "other_strengths_and_weaknesses": "Strengths:\n* The integration of execution feedback as a dynamic env signal in RL is novel compared to prior one-shot RL for code generation (e.g., CodeRL, AlphaCode).\n* Bridges reinforcement learning and agentic AI, making it relevant for self-improving AI coding assistants, debugging agents, and autonomous software maintenance.\n* The paper is well structured and easy to read.\n\nWeaknesses:\n* Limited evaluation scope: CodeContests is a strong benchmark but focuses on single-function correctness, not full software debugging or open-ended programming tasks. Evaluation on real-world bug-fixing benchmarks (e.g., SWE-bench) would strengthen the paper.\n* No direct comparisons to GPT-4, DeepSeek-R1 or other leading models for code.\n* RLEF uses binary pass/fail rewards, but no ablation tests whether denser rewards (e.g., partial credit for fixing certain test cases) could accelerate training.", "other_comments_or_suggestions": "no", "questions_for_authors": "1) Explore finer-grained rewards:\nRLEF uses a binary pass/fail reward, but no ablation tests whether partial credit for fixing certain test cases or reducing runtime errors could improve the results.\nDid you experiment with denser rewards (weighted scoring based on passing test subsets or fixing specific error types)?\n\n2) How well does RLEF generalize beyond function-level synthesis?\nThe evaluation focuses on competitive programming benchmarks (CodeContests, HumanEval+, MBPP+), which emphasize single-function correctness rather than real-world software debugging or multi-file code synthesis.\nHave you tested RLEF on different prompt styles and benchmarks like SWE-bench, or other multi-file programming tasks?\nIf RLEF does not generalize well beyond competitive programming, its real-world applicability may be more limited than implied. A broader evaluation would solidify its impact for practical software engineering tasks.\n\n3) How does RLEF compare to iterative prompting methods like Reflexion or Self-Refine?\nThe paper compares against standard fine-tuned baselines but not against iterative self-improvement prompting approaches (e.g., Reflexion, Self-Refine).\nHave you tested RLEF against few-shot prompting strategies that also incorporate execution feedback (but without RL-based fine-tuning)?\n\n4) Performance on out-of-distribution tasks\nThe benchmarks used (CodeContests, HumanEval+, MBPP+) are very well established but contain somewhat templated problem formats.\nHave you tested RLEF on entirely novel problem distributions, such as real-world coding tasks and prompt styles not present in training?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HWVzxubaNe", "reviewer": "Reviewer_o59z", "summary": "This paper finetunes LLMs for multi-turn code generation with PPO. The action is code generation/refinement using LLMs. The model is given public test cases for code evaluation and then refinement. The epsidoe ends either when reaching the maximum turn limit or when the generated codes pass the public tests. The reward of the episode is then if the final codes pass the private the test cases. The model is finetuend and evaluated using the train/val/test splits of CodeContests.\n\n## update after rebuttal\n\nIt's a pity that the improvements on livecodebench are not significant.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback and comments and provide the following responses:\n\n> Data Contamination\n\nWe build off the Llama 3.1 models which were originally evaluated on benchmarks like HumanEval and MBPP as well. On top, we add training data from CodeContests exclusively; we hence regard the risk of data contamination as low.\n\n> No baseline with Llama 3.1 Backbone\n\nWe thank the reviewer for this suggestion. We ran AlphaCodium with Llama 3.1 70B Instruct and obtained solve rates of 34.2 on valid and 27.8 on test, i.e., below the corresponding GPT-4 numbers and notably *below* the 10@100 results for this model, which uses an equal sample budget.\n\nWe hence do not think that starting from Llama 3.1 results in an unfair advantage.\n\n> The model is only evaluated in old benchmarks as well. It would be much better to evaluate the model in recent benchmarks such as livecodebench.\n\nWe will add a LiveCodeBench evaluation with questions up to 10/2024 in the Appendix, extending Table 2: https://imgur.com/a/CpsZMJB\n\nWhile we see significant improvements for both model scales, we do not observe gains of the same magnitude as on CodeContests.\n\n> Are there results in more diverse settings, such as more samples per rollout with and without the public test cases?\n\nIn Figure 4 we increase the number of samples per rollout (turns), and we also measure performance when iterating on private tests in Appendix B.3 and observe additional (albeit limited) gains.", "claims_and_evidence": "This paper claims the proposed method, RLEF, is state-of-the-art for improving code generation performance. However, the model is only evaluated on old benchmarks such as CodeContests, HumanEval, and MBPP, suffering the risk of data contamination. Also, there is no baseline performance using Llama-3.1 as the backbone, which raises the concern of unfair comparison. More experimental results are needed to support the claim.", "methods_and_evaluation_criteria": "The method is intuitive, using PPO to tune multi-turn code generation models.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "The baseline models (as listed in Table 1) all use old pretrained models. Llama 3.1-70B, by itself, already bypasses many of them with RLEF. It is hard to tell if RLEF is the key factor for performance improvement. \n\nThe model is only evaluated in old benchmarks as well. It would be much better to evaluate the model in recent benchmarks such as livecodebench.", "relation_to_broader_scientific_literature": "It targets the important multi-turn code generation problem and shows improved performance in several benchmarks.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "* Are there results in newer benchmarks? \n* Are there results of baselines using the same backbone? \n* Are there results in more diverse settings, such as more samples per rollout with and without the public test cases?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "riV96RspLr", "reviewer": "Reviewer_3JDi", "summary": "The paper proposes an end-to-end multi-turn RL framework to teach LLM self-repair/refine based on execution feedback, particularly focused on the code generation domain, where the unit tests and execution feedback is easy to obtain. The main algorithm is PPO with turn-level value function to calculate the advantage, along with a common KL penalty. Empirically, the method shows strong improvement on the CodeContests benchmark using Llama 3.1 models, and sees generalization performance across HumanEval and MBPP. Some interesting ablation studies are also conducted, such as few-shot prompting, SFT on success attempts, single-turn RLEF, etc.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer's comments and feedback on our manuscript.\n\n> the generalization performance to HumanEval and MBPP, can we report CI for the numbers in Table 2 as well? The generalization performance seems worse compared to CC. Test, is the improvement statistically significant?\n\nWe will extend Table 2 with results on LiveCodeBench and confidence intervals for the Appendix: https://imgur.com/a/CpsZMJB\n\nCIs are estimated following AlphaCode, i.e., we repeatedly (200x) sample a subset of the solutions with replacement and estimate 1@3 solve rates. We then take the 2.5% and 97.5% percentiles.\n\n> any hypothesis on why there is more timeout in later turns for the model being RLEFed\n\nOur hypothesis is that this is an effect of solutions passing the public tests but failing under increased problem sizes that may be part of the private test sets.\n\n> could we have a more refined analysis on model is simply re-attempting? or is doing real fixes?\n\n> i think diversity only values when (1). we want to do independent sampling; so we get a higher pass@k as k increases or (2). the diversity at the end of the roll-out, this is the response we will eventually used? In view of the diversity within the rollout, why this is a desirable criteria, i feel more evidence or justification is needed here.\n\nDiversity of solutions within a single rollout can be important. When a proposed solution fails, it may be advantageous to start with a fresh/different approach rather than to address fixes, e.g., if the wrong algorithm was selected, fixing the runtime error would not lead it to be the correct one. Notably, our analysis shows that, without specific training, available LLMs are often not good at refining or changing their solutions.\n\n> In ablation studies, the study of baselines (i.e., few-shot prompting and SFT) is interesting, any idea on why few-shot shows worse performance? is it a more general statement?\n\nWe don't claim that our conclusions regarding the negative effect of few-shot prompting can be readily generalized to other domains. However, we think the general ranking of learning methods as few-shot < SFT < RL is accurate.\n\n> I do like the idea of testing how much the model learns from the execution feedback, but random feedback seems a too weak baseline, it would be ideal to add other more reasonable baselines, such as no execution feedback, give model binary / numerical reward only, or using a subset of public tests?\n\nThe choice of random execution feedback to test the sensitivity is motivated as follows: with our training regime, a trained model can infer that a solution is wrong solely based on the fact that it is prompted for another solution, and the public tests are already included in the initial prompt. Hence, a model insensitive to textual feedback could simply ignore it and propose another solution.\n\nIt is thus hard to arrive at a pure \"no execution feedback\" setting for this experiment, since any re-prompting signals to the model that the previous solution was wrong. Nevertheless, we ran a small study where we replace the textual execution feedback with the string \"Consider if the previous solution is correct and provide a new one if it is not.\" We see small drops in performance for both 8B (17.1 -> 16.8 on valid, 16.0 -> 14.5 on test) and 70B (37.1 -> 34.7, 40.6 -> 40.1) RLEF-trained models for 1@3 solve rates. We get a more pronounced performance reduction from random execution feedback (8B: 17.1 -> 16.0, 16.0 -> 13.8; 70B: 37.1 -> 31.6, 40.6 -> 36.7).\n\nWe would also like to refer to Figure 4(a) in the paper, which shows that, with random execution feedback, models can still propose many solution candidates to the effect that pass@10 scores are closely matched. With execution feedback, solutions proposed in later turns are more targeted, either wrt repairs or in terms of new approaches, which leads to gains in precision (pass@1).\n\n\n> An even more realistic setup in to let LLM self-generate some unit tests, and use it in the multi-turn training process, this might be out of the scope of the current paper, but might be interesting to see.\n\nWe agree, a combination with test generation would be very interesting. We'll hopefully see this in future work.", "claims_and_evidence": "Most of the claims the paper made are from the experiment results (section 3.2), centered around the effectiveness of the multi-turn rl based on execution feedback:\n- the strong performance of RLEF compared with prior work, which supported by comparing with a wide range of baselines such as alphacode, code llama and alphacodium, etc.\n- the generalization performance to HumanEval and MBPP, can we report CI for the numbers in Table 2 as well? The generalization performance seems worse compared to CC. Test, is the improvement statistically significant?\n- RLEF tends to increase the errors fixed in the followup turns, and give much more code changes. This is examined in both the llama 7b and 80b models. Some questions here: (1). any hypothesis on why there is more timeout in later turns for the model being RLEFed. (2). for the eval of code changes (1-chrF), could we have a more refined analysis on model is simply re-attempting? or is doing real fixes? \n- There are some claims about the diversity within a roll-out, L312-L316. I am a little bit confused here, i think diversity only values when (1). we want to do independent sampling; so we get a higher pass@k as k increases or (2). the diversity at the end of the roll-out, this is the response we will eventually used? In view of the diversity within the rollout, why this is a desirable criteria, i feel more evidence or justification is needed here. \n- In ablation studies, the study of baselines (i.e., few-shot prompting and SFT) is interesting, any idea on why few-shot shows worse performance? is it a more general statement?\n- Another claim regarding with the pre-trained models seem to benefit more from IF, than code-specific SFT for code generation performance, it would be give more hypothesis or justification here.", "methods_and_evaluation_criteria": "The method of using end-to-end multi-turn RL to solve the code repair problem is pretty reasonable and solid. Code generation naturally has the execution feedback to be used as intermediate feedback to teach self-repair. \n\nThe evaluation is conducted on common code generation benchmarks, CodeContests, MBPP and HumanEval.", "theoretical_claims": "There is no theoretical claims in the paper, N/A.", "experimental_designs_or_analyses": "The benchmark, baseline and evaluation metrics in general makes sense, some minor issues i saw:\n- I do like the idea of testing how much the model learns from the execution feedback, but random feedback seems a too weak baseline, it would be ideal to add other more reasonable baselines, such as no execution feedback, give model binary / numerical reward only, or using a subset of public tests? An even more realistic setup in to let LLM self-generate some unit tests, and use it in the multi-turn training process, this might be out of the scope of the current paper, but might be interesting to see.", "relation_to_broader_scientific_literature": "This paper is relevant to a broader community such as LLM, multi-turn RL, code generation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n- The paper studies an interesting problem of code-generation using multi-turn RL.\n- The extensive set of ablation studies is interesting in this paper, such as the comparison with few-shot, SFT, single-turn RL. \n\nWeakness:\n- One major weakness I saw from the paper is that the method section is too dense, and it would be great if it could be written into more details, such as the turn-level value function, the task-specific implementations, etc.\n- As mentioned before, it would be interesting to include more realistic case studies regarding to the quality / availability in the unit tests to be used in RLEF.", "other_comments_or_suggestions": "See discussions in previous sections.", "questions_for_authors": "See discussions in previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose an end-to-end RL method to train models to learn from execution feedback for code generation tasks. The authors conducted comprehensive experiments on different model sizes (from 8B to 70B parameters) and demonstrated significant performance gains from the proposed method against strong baselines. The authors include good ablation study (e.g. inference-time behavior) with useful insights for the community. There are some minor issues raised by the reviewers, including the application to more practical tasks e.g. SWEBench, and comparison to more advanced LLMs such as GPT and DeepSeek. 

Overall, I found the proposed method a well designed and elegant solution in the intersection of RL for LLM, AI agentic framework, and self-refining/ self-debugging code generation. I suggest the authors review and incorporate all the feedback into their final revision.
Decision: Accept (spotlight poster)