CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement
Authors: Leitian Tao
Xiang Chen
Tong Yu
Tung Mai
Ryan A. Rossi
Yixuan Li
Saayan Mitra
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) have revolutionized code generation but are require significant resources and tend to over-generalize, limiting their task-specific efficiency. Fine-tuning smaller, open-source LLMs is a cost-effective alternative, yet standard supervised approaches rely solely on correct examples, overlooking valuable insights from failures. We introduce CodeLutra, a new framework that leverages both correct and incorrect code attempts. Instead of purely instructing with correct solutions, CodeLutra uses iterative preference-based refinement, comparing successful and failed outputs to better approximate desired results. This process narrows the performance gap with state-of-the-art, larger models, without requiring massive datasets or auxiliary models. For example, on a challenging data science coding task, using only 500 samples improved Llama-3-8B’s accuracy from 28.2% to 48.6%, approaching GPT-4’s level. By capitalizing on both successes and mistakes, \textsc{CodeLutra} offers a scalable, efficient path to high-quality code generation, making smaller open-source models more competitive with leading closed-source alternatives.

Published in Transactions on Machine Learning Research (06/2025)
CodeLutra:
Boosting
LLM
Code Generation via
Preference-Guided Refinement
Leitian Tao
Ieiliantao C.s.oissssssss....oo wisc. edu
University of Bessonsinn-Madison
Xiang Chen Adobe Research
riangche@adobe.c com
Tong Yu Adobe Research
Iyu hyuDadobe.cm com
Tung Mai Adobe Reseorch
tumaiūadobe..com
Ryan A. Rossi Adobe Research
ryrossi@udabe.com com
Yixuan Li University of Aisconsin-Madison
sharonkifcss.o www.:/ssss.....ee...o.. edu
Saayan Mitra
emitra@addbe..c
Adobe Research
Reviewed on OpenReview: https: // openreview. net/ forum? id= IGaEgWM4 to
Abstract
Large Language Models (LLMs) have revolutionized code generation but require significant resources and tend to over-generalize, limiting their task-specific efficiency. Fine-tuning smaller, open-source LLMs is a cost-effective alternative, yet standard supervised approaches rely solely on correct examples, overlooking valuable insights from failures. We explore re- cent developments preference-based post-training to code generation, leveraging both correct and incorrect code attempts. Instead of solely relying on correct examples, our framework CODELUTRA iteratively refines the model by comparing successful and unsuc- cessful outputs, thereby more accurately aligning the generated code with desired outcomes. This process narrows the performance gap with state-of-the-art, larger models, without re- quiring massive datasets or auxiliary models. For example, on a challenging data science coding task, using only 500 samples improved Llama-3-8B' accuracy from 28.2% to 48.6%, approaching GPT-4's level. CODELUTRA provides scalable and efficient strategy for high- quality code generation, helping smaller open-source models narrow the performance gap with state-of-the-art closed-source systems.
1
Introduction
Large language models (LLMs) have revolutionized numerous domains, consistently delivering great per- formance across different tasks (Brown al., 2020; Achiam al., 2023; Anthropic, 2023; OpenAI, 2023). Among these applications, code generation stands out particularly promising. Models pre-trained on

---

Published in Transactions on Machine Learning Research (06/2025)
LLM's multiple ans
Correct code answers:
IRO0 WiG
small questions:
SELO-T name ThOis meeeeoone crees Wi CCC depetnert SSaes
Evaluation
Eii lare playes Wosiit anacinent Saldd FTDG erpooees Wi "Srer"
Whitant dest wbo divite Taee?
Lun disss sharc
R Rejected code answers:
FTkic iusttee os currpi
Ground truth evaluation:
T
SEL DCT same FRDM analoy aee HRRE ceparmant Saide""
FEAECT sipcceeee E.D eig crpert
Initialization for next iteration of refinement
Paired code ans wers:
Preference-guided refinement
SELECT same FTCCMretollyee WilGtt dcartment Lnf Sala
T&+11
Encourages prefer correct over rejected/ Increases the likelihood of correct
came owg.looggienn reed Draaee' DOLLCCT 7TD0 an engioyeet CCTC Cahess
query
Data science
74.4
Execusionn accursty 1%)
48.6
49
50
Peosig 056
59.3
38.4
LIama-8B
Uiama (B(FF)
Liama- BB(Ours)
CodeX
GPT-4
Llama-88 -8
Liama La5aa-6((FF))
Llama (B(Ours)
CodeX
GPT-4
Figure 1: The proposed CODELUTRA framework (see Section 4) and performance comparison different code generation tasks. The orange dashed box highlights our framework's 's performance relative to GPT-4.
tensive code repositories have demonstrated an impressive capability to solve diverse programming challenges (Li et al., 2022; Zheng et al., 2023; Chen al., 2021b; Wang et al., 2023).
Deploying ulrge closed-source models like GPT-4 (OpenAI, 2023) for code generation challenging due to their enormous resource demands and limited customization options. Fine-tuning smaller, open-source LLMs offers a more flexible and cost-effective alternative, yet code generation requires more than syntactical accuracy-it it demands logical depth and domain-spectt (SFT) on correct code examples can help, it yields only incremental gains and often falls short of state- of-thc-art performance. This is because SFT alone cannot leverage the model's own mistakes, limiting its ability to adapt. and improve diverse programming environments. We illustrate this in Figure 1, where SFT provides minor boost in performance on the challenging data science task, with Llama-3-813 Pass@1 improving from 28.2% to 30.0%. A notable gap remains between the finc-tuned model and GPT-4, which leads with Pass@1 score of 49.4%. To address the issue, prior solutions collect additional training data by leveraging more powerful LLMSs (Shen et al., 2023; Tuo et al., 2023) or by collecting external datasets from public repositories (Lozhkov et al., 2024; Muennighoff et al., 2023). But the big elephant in the room still remains-hove much can we bridge the gap by marimizing the utility of eristing data and model at hand?
understanding. While basic supervised fine-tuning
To address this challenge, we introduce CODELUTRA, a framework that iteratively improves the performance of given LLM without relying on vast external datasets or larger auciliary models. Unlike framework CODELUTRA traditional fine-tuning methods that rely exclusively on correct code solutions, leverages both successful and failed code generated by the model itself, creating self-generated comparative data. While prior works explored using exccution results to enhance code generation, they often require exlensive da datasets to Irain reward models for feedback (Le et al., 2022; Shojaee el. al., 2023) or rely on prompt engineering with ultra-large LLMs (Chen et al., 2023; Ni et al., 2024). In contrast, our work collects successful and failed attempts and enables models iteratively learn, distinguish, and improve from these examples. Our approach demonstrates that even with limited data hand, substantial gains in code generation quality be achieved, closing the gap between smaller fine-tuned models and the top-tier L.Mss. To harness
2

---

Published in Transactions on Machine Learning Research (06/2025)
the comparative data, a key component of CODELUTRA is the preference-guidder refinement mechanism that compares correct and incorrect code snippets, iteratively refining the model's understanding of code quality. With each iteration, the model generates code solutions, evaluates their correctness, and updates its parameters based on the evolving preference dataset. This process allows for continuous improvements of the base model, making the framework effective even with limited initial data (e.g., a few hundred samples). The failed code attempts are invaluable for refining models, since they provide concrete examples of common errors and enable the model to learn strategies for avoiding similar mistakes in future generations.
We comprehensively evaluate the effectiveness of CODELUTRA challenging data query and data science tasks, where the LLM is tasked with generating the correct SQL or Python code to solve a given problem. We compare CODELUTRA with 13 open-source and closed-source LLMs that are competitive in code generation. Notably, on the data query task, our framework allows Llama-3-8B (Dubey et al., 2024) to achieve an execution accuracy of 76.6%, which exceeds GPT-4's 74.4%. Under challenging data science task, we find that using just 500 samples improved Llama-3-8B from an accuracy of 28.2% to 48.6%, approaching the performance of GPT-4. This demonstrates that CODELUTRA achieves strong results even with a limited number of high-quality annotations. Moreover, we observe the consistent performance gains of CODELUTRA on different base models, including Gemma-7B (Team et al., 2024) and StarCoder-7B (Dai & Kumar, 2023). These findings highlight the potential of CODELUTRA in closing the performance gap between open-source and closed-source models. To summarize our key contributions:
We explore brased learning for code generation through CODELUTRA, which leverages self- generated comparalive data from both successful and failed code attempls, enabling low-performing models to rival top-tier solutions without external datasets or ultra-large LLMs feedback.
2. We conduct comprehensive evaluations, comparing CODELUTRA against 13 competitive LLMs spe- cializing in code generation. Results demonstrate CODBLUTRA outperforms both standard fine- tuned LLMs and existing cutting-edge closed-source LLMs.
3. We conduct the in-depth analyses to understand the contribution of failed attempts and likelihood regularization for CODELUTRA, and reveal CODELUTRA improves the performance via reducing syntax errors and improving incorrect answers across iterations.
2 Related Work
Preference learning for LLMs. Preference learning guides language models toward outputs aligned with human preferences. Approaches like RLHF (Ziegler et al., 2019; Ouyang et al., 2022) effectively align LLMs but face inefficiencies and hyperparameter sensitivity (Christiano et al., 2017; Bai et al., 2022). Recent works focus on closed-form loss functions using offline preference data, such as DPO (Rafailov et al., 2023) and its extensions (Liu et al., 2023b; Ethayarajh et al., 2024). Iterative DPO, where models generate their own preference data, shows strong performance in the alignment and math reasoning tasks (Liu et al., 2024; Xiong et al., 2024; Yuan et al., 2024; Pang et. al., 2024). Our work represents the first comprehensive study on iterative preference optimization specifically tailored for code generation, achieving GPTT-4-eeel results with significantly fewer resources. While previous approaches (Le et al., 2022; Shojaee et al., 2023; Ni et al., 2024; Chen et al., 2024) leverage execution results predominantly to refine incorrect code until a correct solution emerges, they often rely heavily on complex reward signals from critic models (Le et al., 2022; Shojaee et al., 2023) or sophisticated prompt engineering (Chen et al., 2024; Ni et al., 2024). The key novelty of our method lies in using execution outcomes to probabilistically differentiate correct from incorrect generations via Bradley-Terry model, representing a fundamental conceptual shift from existing approaches. This conceptual shift allows our framework to remain simpler, more generalizable, and less dependent on proprietary large-scale models: rather than crafting intricate instructions tailored to advanced architectures or relying on external ground truths like unit tests, we directly let the model learn from failures by contrasting them with successos, thereby improving its coding proficiency over itcrative training cycles. Compared to other methods, which introduce complexities that may not scale down to smaller models or open-source environments, our minimalistic strategy consistently demonstrates strong performance.
3

---

Published in Transactions on Machine Learning Research (06/2025)
LLMs for code generation. LLMs trained on large code corpora excel in tasks like code generation (Chen et al., 2021c; Zhang et al., 2022), program repair (Xia & Zhang, 2022), and software testing (Chen et al., 2023). Models like WIZARDCODER (Luo et al., 2023) and DS-CODER (Li et al., 2023b) enhance these techniques (Lu et al., 2022). In-
capabilitics through repository-level organization and retrieval-augmennte struction fine-tuning, as in CODEINSTRUCT (Li et al., 2023a), further improves alignment with human coding preferences. Unlike existing methods that rely on program exccution results (Le et al., 2022; Shojace et al., 2023; Ni et al., 2024; Zhang et al., 2024), which depend on complex reward models or elaborate prompt engi- neering, our novel approach iteratively refines small LLMs using self- self-generan generated preference data, completely eliminating the need for external datasets and larger models. Additionally, we employ a unification of SFT and preference learning through our dual-loss approach, maximizing the likelihood of correct solutions in one stage. Furthermore, although theoretical insights into dual loss objectives have been discussed in other work (Liu et al., 2024), our approach distinctively refines and applies these principles to the uniquely challeng- ing domain of code generation sraatiion-hooing that the integration of execution feedback into a direct, probabilistic learning framework can yield practical gains without the pit.falss of overly elaborate interventions. While prior iterative reasoning methods excel at mathematical tasks (Pang et al., 2024), code generation poses dis- tinet challenges due to strict syntactic and semantic correctness requirements. In mathematical reasoning, a correct final output does not necessarily guarantee an entirely accurate answer (Wang et al., 2024). However, in code generation, even minor errors in the code can result in completely incorrect execution outcomes. The novelty of our proposed framework, CODELUTRA, lies in its unique integration of both successful and failed attempts using a Bradley Terry model, enabling the refinement of outputs without any reliance on explicit feedback or ground-truth labels. By generalizing from errors and iteratively improving attempts, CODE- LUTRA consistently reduces syntactic and logical flaws while maintaining the quality of correct solutions. Empirically, it matches even surpasses advanced models like GPT-4, advancing the democratization of high-quality code generation in domain with strict correctness requirements.
3 Preliminaries
LLMs for code generation. LLMs are pre-trained on diverse datasets encompassing both natural and programming languages. In code generation tasks, an LLM receives a prompt-sude as a natural language description, and generates the corresponding code by predicting the next token in the sequence. Formally, code generation modeled as the conditional probability of code sequence y = (y) .9T) given an input prompt x:
P(y|z) = II P(ullye,,
(1)
where x is the input prompt, is the generated codc sequence of length T, and yer = (y1, U2..... 11-1) represents the tokens generated before time step l.
Supervised fine-tuning of LLMs on task-specific dataset. Pre-trained LLMs can be suboptimal task-specific dataset, necessitating fine-tuning. We consider task-specific dataset D {( v1) H containing examples, where each pair (z,, v.) represents an input prompt z, and its corresponding target code y/. Supervised fine-tuning (SFT) adjusts the model's parameters 0 by maximizing the likelihood of generating correct code sequences ye. The loss is defined as:
SSFT = argmaxx, E( (log To (yilx:)) (z)))
(2)
Verification of code correctness. We verify the correctness of the generated code by running it with the same inputs as the reference code and ensuring that both produce identical outputs.
Limitations of SFT for code generation. Code generation demands not just syntactical correctness but also a deep understanding of logical and domain-specific nuances, which complicates model training. A major limitation of SFT is that it solely mazimizes the likelihood of providing correct code, which restricts the model's ability learn from its own mistakes. Since the training process focuses exclusively on provided examples, the I.I doesn't 't receive the gradient from incorrect suboptimal code. For instance, if the model
4

---

Published in Transactions on Machine Learning Research (06/2025)
only predicts wrongly in the final token in code snippet, the overall probability P(y|æ) might still remain high as the preceding tokens are correct. Correspondingly, the SFT loss is very small in that case. However, unlike natural language generation where minor errors might still preserve meaning, in code generation, this single erroncous token can render the entire code nonfunctional or introduce subtle bugs, significantly alfecting execution correctness despite high likelihood score. This binary nature of code correctness where code either executes perfectly or fails completely-creet creates a unique challenge that standard SFT approaches struggle to address. This reliance on only correct examples limits the model's capacity to identify and recover from errors, reducing its ability to handle more complex or nuanced coding tasks. This motivates our framework CODELUTRA, which leverages both successful and failed code generation attempts.
4 CodeLutra
In this section, we introduce CODELUTRA, framework designed to comparatively learn from both correct and incorrect code generations. CODELUTRA delivers substantial performance gains, achieving performance comparable to more advanced models like GPT-4 (OpenAI, 2023), even with limited initial data. We summarize our algorithm in implementation in the Algorithm 1.
Initialization. We start with an initial base model, denoted as mo, which serves as the starting point for our iterative refinement process. We are provided with an initial training D = {(x1, })) where each æq is natural language query, and zi the corresponding ground truth code solution. Starting with modestly performing model allows to clearly observe improvements attributable to the CODELUTRA framework, ensuring that enhancements result from our methodology rather than inherent model capabilities. Note that at initialization, we only have the correct codes in hand. We describe how to obtain incorrect codes to serve the model refinement.
Generating correct and failed code. At each iteration t, the model He generates M codc responses for each imput v, E D: 0.Th ~ ()(z;) for m E (1,2,..., M.. These responses are execuled to determine correctness, with successful outputs categorized into y:(e) and failures into y,(r), This process introduces output diversity and enables the construction of the preference dataset.
Preference dataset construction. A core aspect of our framework is constructing & preference dataset D; at each iteration t, capturing the relative quality of generated code. For each input 2₁, K preference pairs are created by pairing one correct code 6%* E y:(e) with one rejected code 0 E yy("). Sampling with replacement is used if either set contains fewer than K responses. The resulting dataset, D, = {(,, Di*h 8 }}, contains n x triplets, where n is the size of the initial dataset. This evaluation mechanism offers clear feedback, particularly syntax and execution errors common in code generation tasks, and enables the construction of preference dataset.
Preference-guidedd refinement. While one can di- reetly employ preference optimization approach like DPO (Rafailov et. al., 2023) on our curated dataset Dr this approach presents a notable limitation due to its ten- dency to decrease the likelihood of both correct and re- jected code during training. This is evidenced by the dashed lines in Figure 2, which is also observed in Pal
Log proteblity
-200
et al. (2024); Feng et al. (2024); Liu al. (2024). This occurs because DPO relies proxy rewards from lim- ited data, leading to misdirected gradient updates that cause the model to conservatively reduce likelihood for both response types (Liu et al., 2024). This diminishing likelihood can significantly impact our framework, espe- cially since our correct code are critical for successfully solving the assigned tasks. It is crucial, therefore, to prioritize the likelihood of correct solutions to the task at hand.
Alorr
Loxo May -YL
-500
502
1000
1500
2000
Global step
Figure 2: Efffett of SFT loss to keep the likelihood of correct answers stable.
5

---

Published in Transactions on Machine Learning Research (06/2025)
To address this limitation, we employ dual-loss function integrating DPO with an additional term, which regularizes the training to prevent decreasing likelihood (Liu et al., 2024). Specifically, we maximize the for all 2; E D and k E [K]]. The overall loss
likelihood of correct solutions in dataset D = {(11, 0² function defined as:
logo (8 log To Te( log Te (v (4))
Te (v 22:)
T++1 argmin
Ti (1f
compare correct and incorrect solutions
(E,a)))) (log ne (yf (m:))
(3)
maximize likelihood correct codes
where λ a hyperparameter balancing the contributions of the DPO and SFT losses. The first term facilitates preference-based fine-tuning by optimizing the model to favor correct over incorrect code. Concurrently, the second term enhances the likelihood of generating correct solutions directly to avoid the log probability decrcasing (sce solid line in Figure 2). This dual-loss approach ensures that the model not only ranks correct solutions higher but also increases their generation probability, leading to more consistent high-quality code outputs. We verify the effectiveness of the dual loss empirically in Section 5.3. The refinement process continues until the improvement between consecutive iterations T. and T+41 becomes marginal, indicating convergence.
To our knowledge, this the first work to establish direct link between iterative preference learning and code generation, where execution results serve as the natural source of feedback. Unlike prior studies that rely on model-generated rewards or other large models' feedback (Chen et al., 2024; Xiong et al., 2024; Yuan et al., 2024; Pang et al., 2024; Xie et al., 2024), we leverage execution correctness as a key preference indicator, uniquely suited to code tasks.
5 Experiments
5.1
Experimental Setup
Tasks. To evaluate the effectiveness of our proposed framework, we perform experiments two tasks: Data Query and Data Science. Both tasks reflect common and practical challenges in fields such business, health and scientific computing, where precise code generation is critical for solving data- related problems while receiving limited attention. We provide more details and examples of the two tasks in the Appendix A.3.
Datasets. We conduct our experiments on two cross-domain datasets for data query, Spider (Yu et al., 2018) and BIRD (Li et al., 2024), and a data science dataset, DS-1000 (Lai et al., 2023). We provide more details in the Appendix A.2.
Metrics. For the Data Query lask, we adopt the metrics introduced by Yu el al. (2018): Execution Accuracy (EX), which measures whether the SQL query execution result matches the expected output, and Exact Match (EM), which evaluates whether the generated SQL query exactly matches the reference query in both structure and semantics. For the Data Science task, we use pass@1, following Lai et al. (2023), which indicates the percentage of correct solutions generated by the model on the first attempt.
To evaluate the effectiveness of our method, compare it against three categories of baselines. For fair comparison, all reported results are based on the same prompt. We report the details of baseline setup in Appendix A.
Baselines.
We benchmark our method against competitive open-source LLMs, including models pre-trained on general datasets such as Llama-3-8B (Dubey al., 2024), Gemma-7B (Team et al., 2024), and Blama-3-70B-Instree (Dubey et al., 2024). Additionally, we compare against LLMs pre-
Open-source LLMs:
6

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm CODELUTRA
Require: Training set = (( 1=) initial base model To; responses per input M; preference pairs K; iterations Т; hyper-parameter
Ensure: Fine-tuned model Wy
for f 0 to T - do
Initialize preference datasct Di t- 0 3: for all z; E D do
4: Initialize chosen code set y-(0) +- Initialize rejected code set y-(1)) for k + 1 to M do Generate response O ~ (x+)
7:
8:
if execution result of of matches then
Add i to y else
10:
Add of to y,(r) end if end for
11:
12:
13:
14:
for +- to K do
Randomly sample 0.th from ( (with replacement if [Y,(0) K)
15:
Randomly sample 2hth from ( (with replacement if [r, K) Add (I,, 0.*h 0(+)) to D:
16:
17:
18:
end for
19:
end for
20:
Update model T441 by minimizing the combined loss:
E Ea (.)))))))))))) Tog (s (3 (10g (e (
Te lx;)
M++1 argmin
log
Ti (v/(l;;)
n. (of (x.)
compare and incorrect solutions
AE (a, (log no xx)) maximize likelihood for correct codes
(4)
21: end for
trained specifically on coding datasets, such as Codellama-7B (Roziere et al., 2023), StarCoder-7B (Lozhkov et al., 2024), and Codestral-22B (Mistral AI, 2024).
Pine-tuned LLMs: As supervised fine-tuning on domain-specific datasets popular and effective way to improve LLMs' corresponding performance, we also report the performance of fine-tuned LLMs using standard supervised fine-tuning on the ground-truth solutions training data (Raffel et al., 2020).
Closed-source LLMs: We provide the performance of advanced closed-source LLMs, including Codex (Chen et al., 2021a), ChatGPT (Ouyang et al., 2022), and GPT-4 (OpenAI, 2023).
Experimental setup. For main results, we apply our framework the Llama-3-8B base model (Dubey et al., 2024), denoted To (see Section 5.2 for more backbone results). We use zero-shot prompt containing the question along with reference information. For different answer collections, we employ the best-of-n strategy by sampling 10 responses at the temperature of 1.0. We train epoch per iteration and perform four iterations in lotal, resulting in models {mi, T2, Ta, π4). For verification of code correctness, we employ task- specific approaches: For data query tasks, we execute the generated SQL queries against the same database schema. as the reference solutions and directly compare exccution results, cnsuring functional equivalence regardless of syntactic differences. For data science tasks, we leverage the DS-1000 evaluation framework with standardized test. cases that verify functional equivalence bet generated and reference code. These

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1:
The Execution Accuracy (EX), Exact Match (FM), and Pass for different kinds of models on SPIDER, BIRD, and DS1000. We show the base model (To) without fine-tuning and the model trained with
CODELUTRA in different. iteration ((Ti, n2, n3, ma)). For fair comparison, all reported results in the table use the same prompt. Boldface highlight GPT-4 and our results.
Spider
BIRD
DS1000
Models
EX
EM
EX
EM
Pass@1
Open-source LLMs
Llama-3-8B (Dubey et al., 2024)
59.3
55.1
22.3
19.5
28.2
Codellama-7B (Roziere al., 2023)
57.0
51.4
24.4
18.7
25.6
StarCoder-7B (Lozhkov et al., 2021)
61.2
58.6
25.7
23.0
26.8
Gemma-7B (Team et al., 2024)
19.9
16.7
21.2
19.1
21.2
Codestral-22B (Mistral AI, 2024)
71.3
69.6
12.5
39.9
35.8
Llama-3-70BIInstuuc (Dubey et al., 2024)
68.7
65.4
41.2
39.3
36.4
Fine- tuned LLMs
Llama-3-8B (Dubey et al., 2024)
67.9
64.7
35.6
30.7
30.0
Codellama-7B (Roziere et al., 2023)
67.3
64.3
36.3
30.9
26.8
StarCoder2-7B (Lozhkov al., 2024)
66.9
64.1
36.6
31.1
29.4
Gemma-7B (Team et al., 2024)
65.8
62.8
34.5
29.8
27.4
Closed-Source LLMs
Codex (Chen et al., 2021a)
73.1
70.2
44.7
42.4
38.4
ChatGPT (Ouyang al., 2022)
71.8
68.4
41.3
40.2
38.8
GPT-4 (OpenAI, 2023)
74.4
71.2
46.3
43.2
49.4
CodeLutra (Ours)
Base (mo)
59.3
55.1
22.3
19.5
28.2
Iteration (Ta)
67.8
63.9
37.8
33.2
43.2
Iteration 2 (T2)
72.4
68.3
40.8
36.0
46.8
Iteration 3 (πa)
76.6
72.5
43.1
38.6
48.6
Iteration 4 (Iaa)
76.3
72.1
42.6
38.3
48.2
test cases comprehensively cover edge cases and typical usage patterns to ensure thorough validation. Our correctness verification process is fully automated, allowing efficient processing of thousands code samples during both training and evaluation phases. For more experimental details, please refer to the Appendix A.I.
5.2 Main Results
Results the data query task. We compare CODELUTRA with baselines in code generation for the data query task, shown in Table 1. We found that existing open-source LLMs like Llama-3-8B still have a significant performance gap in code generation for data queries compared to closod-source LLMs like GPT-4. Although supervised fine-tuning can help bridge this gap e.g., SFT increases the EX of Llama- 3-8B on Spider from 59.3% to 67.9% there remains a notable difference with GPT-4's 74.4%. Through our refinement framework, Llama-3-8B after four iterations exceeded SFT performance by 16.9% and even outperforms GPT-4 with an execution accuracy of 76.6%. Additionally, on the more challenging BIRD dataset, after three iterations, CODELUTRA significantly improved the EX of the base model from 22.3 to 43.1, achieving performance very close to GPT-4. The similar performance observed between iterations 3 and can be attributed to model convergence, as the model approaches its optimal state with no further gains in later iterations.
Results on the data science task. Table 1 also presents results for the data science task, where we evaluate both open-source and closed-source LLMs, as well as our method CODELUTRA. On the DS- 1000 dataset, open-source models like Llama-3-8B and Gemma-7B struggle, with significantly lower EM and Pass@1 scores compared to closod-source models like GPT-4. Fine-tuning provides minor boost performance, as seen with Llama-3-8B's @ass@1 improving from 28.2% 30.0%. However, as with the data query task, large performance gap remains between fine-tuned open-source models and closed-source

---

Published in Transactions on Machine Learning Research (06/2025)
(b)
(c)
50
50
50
w negative
9K
ground
w/o negative
1K
w/o ground truth
40
40
Arcuracy
30
30
30
Accu
20
20
20
0
3
lterations
Iterations
Iterations
Figure 3: (a) Ablations on the effects of negative samples training. (b) Ablations on the question number during training. (c) The effects of ground truth for validation during preference datasets collection.
ones, where GPT-4 leads with Pass01 score of 49.4%. Nonetheless, CODELUTRA demonstrates substantial improvements (from 28. 28.2% to 48.6%), offering a promising path for narrowing this gap.
5.3 More experiments
The importance of learning from failed attempts. Our framework CODELUTRA leverages both pos- itive and negative answer pairs to iteratively improve model performance, particularly by minimizing the generation of incorrect responses. But what happens when we omit the negative samples and rely solely on supervised fine-tuning using positive samples generated by the model? In this ablation, we compare the per- formance of our objective 4 with a model trained with (SFT(T; Df). As seen in Figure 3(a), without negative samples (purple linc), the model's performance platcaus across iterations, remaining close the bascline. In contrast, incorporating negative samples (blue line) leads to steady performance improvements over suc- cessive iterations. This ablation confirms that including negative samples is critical to refining the model's ability to distinguish between optimal and suboptimal responses, significantly boosting overall performance.
CodeLutra achieves strong performance under limited training data. The cost of acquiring high- quality question-code pairs can be significant, so examine whether our method truly depends on large datasets. Under the data science code generation task, we found that using just. 500 samples improved Llama-3-8B from an accuracy of 28.2 to 48.6, approaching the performance of GPT-4. This demonstrates that CODELUTRA achioves strong results even with a limited number high-quality annotations. We further verify this with the data query lask by randomly selecting 1K question-code pairs from BIRD's training data and comparing them the full 9K sample set. The results, as shown in Figure 3(b), reveal similar trends, with the two setups reaching peak execution accuracies of 43.1 and 42.4, respectively. This minor difference suggests that CODELUTRA does not heavily rely large volumes of training data and can generalize well with fewer annotations, which is crucial for minimizing the cost of dataset collection.
Importance of SFT regularization during preference optimization. Recall in Section that our loss function in- tegrates DPO with SFT to regularize the training, and prevent decreasing likelihood on the correct solution. In Table 2, we ablate the effect of SFT regularization on both the Spider and DS-1000 datasets. Notably, omitting optimizing the SFT loss on the correct solutions results in a marked decline in model performance, e.g., 459.4% on Spider. This highlights the ef- fectiveness of the dual loss approach, ensuring that the model not only ranks correct solutions higher but also increases their generation probability, leading to more consistent high-quality code outputs.
Table 2: Ablations the SFT on the correct
answers.
Methods
Spider
DS1000
LDPO
17.2
12.4
Ours
76.6
48.6

---

Published in Transactions on Machine Learning Research (06/2025)
CodeLutra remains effective on different base models. To further validate the generalization ca- pability of our framework CODELUTRA, we ex- tend our experiments to two additional open-source base models: Gemma-7B (Team et al., 2024) and StarCoder-7B (Lozhkov al., 2024). As summa- rized in Table 3, we report the results on both the Spider and DS1000 datasets across multiple iter- ations of our refinement process. For Gemma-7B, we observe a significant improvement in Execution Accuracy (EX) on Spider, starting from 49.9% at To (the base model) and reaching 72.6% after four iterations (Ta). similar trend is observed in the DS1000 dataset, where the Pass@1 metrie improves from 24.2% to 44.0%. For StarCoder-7B, the improve- ments are also pronounced, with EX on Spider increasing from 61.2% to 77.5%, and Pass@1 on DS1000 rising from 26.8% to 45.8%. These results demonstrate that our framework is robust across different model architectures, consistently yielding significant performance gains regardless of the underlying base model. Notably, the iterative refinement process of CODELUTRA continues to improve the accuracy and correct of generated code, highlighting the CODELUTRA generalization to different code generation tasks.
Gemma-77
StarCoder-7B
Model
Spider
DS1000
Spider
DS1000
To
49.9
24.2
61.2
26.8
771
63.7
38.8
72.8
39.6
72
69.3
43.6
74.7
42.4
33
71.3
44.4
77.2
45.2
74
72.6
44.0
77.5
45.8
Table 3: Performance with CODELUTRA of Gemma- 7B and StarCoder-7B across Spider and DS1000 benchmarks.
6 Further Analysis on CodeLutra
Is ground truth code necessary for preference datasct collection? Recall that our framework relies on ground truth code evaluate the quality of generated code during the collection of preference datasets. To test the impact of this dependence, general criterion Siiterion-whehher whether the generated code is executable. In the absence of ground truth, consider ex- ecutable answers as chosen and non-executable ones rejected. Applying this approach to the Bird dataset, we observe notable gains despite the absence of ground truth: accuracy rose from 22.3 to 30.9 (see Fig- ure 3(c)). Moreover, the proportion of executable code surged from 59.8% to 89.7%, showing that the model effectively learned to avoid common errors, such as syntax issues or missing database tables. This experi- ment demonstrates that using executability as a metric still enables substantial model improvements, making the method applicable even without high-quality annotations, and highlights the robustness of CODELUTRA under such conditions.
conduct experiments that replace the ground truth with more
CodeLutra helps reduce the syntax errors across iterations. To evaluate whether our method enables LLMs to learn from their mistakes over multiple iterations, we sampled 100 error cases from the test set using models To, nI, n2; n3, trained with CODELUTRA on the BIRD dataset for qualitative analysis. We measure the fraction of executable code generated by each model. As shown in Figure 4, the percentage of non-executable code decreases from 40% 11% when trained with CODELUTRA, indicating that the models have improved in mastering SQL syntax and are better at avoiding basic errors. A qualitative example in Figure 5 highlights this improvement: the base model incorrectly queries the "Currency" column in the wrong table, resulting in an error, while the model trained with CODELUTRA successfully generates the correct SQL query.
CodeLutra improves quality of incorrect answers across iterations. Based on the responses of the initial model To, we divide the test set into correct set and an error set. We track the quality trends of the model in these two sets across iterations. Using the cosine similarity metric based on the BLEURT embedding proposed by (Sellam et al., 2020), we calculate similarities denoted simmr( ygt) for the model fine-tuned over t itcrations. Here denotes the model generation, and ligt is the ground truth solution. As shown in Figure 4(b), we observe that the similarity between the model's output on the correct set and the ground truth remains stable (see purple bars), while with each iteration, the similarity between the model's output on the error set and the ground truth increases significantly- from 0.48 to 0.54. This indicates that CodeLUTRA helps the base model improve outputs on error set, while the outputs on correct cases remain qualitatively stable.
10

---

Published in Transactions on Machine Learning Research (06/2025)
Executable fraction
(b) BLUERT similarity for different models
Executable
Not executable
3
0.70
erfor set
0.65
DLJERT similerty
Model lttrstion
0.55
0.54
0.50
0.48
0.45
30
100
Percentage
Model iteration
Figure 4: (a) Error analysis for different models. (b) BLEURT similarity for different models with the ground truth the correct and divided from the base model.
Different answers from different models for the data query:
Database schema: CREATE TABLE customers ( CustomerID INTEGER UNIQUE not null primary key, Segment TEXT null, Currency TEXT null ); CREATE TABLE "yearmonth" ( CustomerID INTEGER not null references customers update cascade on delete cascade, Date TEXT not mull, Consumption REAL, primary key (Date, CustomerID) );
Question: Among the customers who paid in euro, how many of them have monthly consumption of over 1000?
The answer from To (wrong):
SELECT COUNT (Custtoor))))) FROM yearmonth WHERE Consumption 1000 AND CustomerID IN
SELECT CustomerID FROM customers WHERE Currency 'EUR' );
SQLite error: such column: Currency
The answer from π3 (correct):
SELECT (IU "Number of customers" FROM "yearmonth"
WHERE Consumption > 1000 AND Currency 'EUR' GROUP BY CustomerID
Figure 5: CODELUTRA helps reduce the syntax errors. fails due to a missing column, while TT generates the correct query.
7 Conclusion
We introduced CODELUTRA, a preference-guidee refinement framework that enhances LLMs for code gen- eration using self-generated data from successes and failures, without relying on external datasets larger models. Experiments show CODELUTRA significantly improves base LLMs, with Llama-3-8B outperforming GPT-4 on data query lasks and nearly matching its performance on data science tasks with limited training data. Additionally, CODELUTRA reduces coding errors while improving code quality and accuracy, offering a cost-cfficient, scalable solution for LLM code generation.
Ethics Statement
Our framework CODELUTRA enhances code generation by leveraging both successful and failed attempts. However, it may propagate biases from pretrained datasets, leading to unfair or discriminatory outputs. There is also a risk misuse for generating harmful code, and we advocate for its responsible application. Additionally, the approach relies on self-generated data, which may inadvertently amplify errors if not carefully validated. We encourage users apply CODELUTRA ethically, ensuring fairness, safety, and high- quality outcomes.
11

---

Published in Transactions on Machine Learning Research (06/2025)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprinl KNi:2333308877, 2023.
Anthoopic. Claude: Conversational ai assistant, 2023. URL https://wwy www.aaoooo..ccooooooooiiiii CttL
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforeement learning from human foedback. arXiv preprint arXiv:2204. 05862, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Newral Information Processing Systems, 33:1877-19011, 2020.
Mark Chen, Jacob Tworek, et al. Evaluating large language models trained on code. ar preprinl arXiv:2107.038: 033777 2021a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Har- rison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Fe- lipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizaboth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodci, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, aass/210.003377 2021b.
Achiam,
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint ar Xitx:2107.0377, 2021c.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self- debug. arXiv preprint arXiv:2304.057 05128 2023.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017.
Emily Dai and Ravi Kumar. Starcoder: An open-source code generation language model. arXiv preprint arXiv:2305.0000 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmmd Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21 217881 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela, Kto: Model alignment as prospect theoretic optimization. arXin preprint arXiv:2402.0133 2024.
Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and under- standing the limitations of dpo: A thcoretical perspective. arXiv preprint Axi:22444044222 2024.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Danicl Fried, Sida Wang, and Tao Yu. Ds 100: A natural and reliable benchmark for data science code generation. Tn International Conference on Machine Learning, pp. 98319-18344 PMT.R 2023.
12

---

Published in Transactions on Machine Learning Research (06/2025)
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mas- tering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314 21328, 2022.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Car- bune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprinl ar orXic22309000220 2023.
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024.
Kaixin Li, Qisheng Hu, Xu Zhao, Hui Chen, Yuxi Xie, Ticdong Liu, Qizhc Xie, and Junxian Hc. Instruct- coder: Instruction tuning large language models for code editing. arXiv preprint ar X1v:2330020329, 2023a.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprinl ar arXv:22305.0666 2023b.
Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Ec- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Mas- d' d'Autume, Tgor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esmee Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukenoglu, and Oriol Vinyals. Competition-level code generation with alphacode. CoRR, abs/2203.07814, 2022.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Ts your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Informalion Processing Systems, 36:21558 21572, 2023a.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.068 2023b.
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprinl arXiv:2405.1644 2024.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint Xiv:2400 19179, 2024.
Shuai Lu, Nan Duan, Hojac Han, Daya Guo, Seung-won Hwang, and Alexcy Svyatkovskiy. relrieval-augmented code completion framework. arXi preprinl arXiv:2203.07722 2022.
Reace: A
Ziyang Luo, Can Xu, Pu Zhao, Qingleng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint Xrx:222006. 08568, 2023.
Mistral AI. Codestral-22b-v0.1. leased under the Mistral AI Non-Production License.
2024. Re-
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayan Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.0712. 2023.
Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengeheng Yin. Next: Teaching large language models to reason about code exocution. arXiv preprint arXiv:2404. 14662, 2024.
13

---

Published in Transactions on Machine Learning Research (06/2025)
OpenAL. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. Advances in Newral Information Processing Systems, 35:27730-277 27744, 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint erxiv:240.. 13228, 2024.
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXi preprint arXiu:2444.19993 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea. Finn. Direct preference optimization: Your language model is secretly a reward model. Aduances in Neural Information Processing Systems, 36, 2023.
Colin Raffel, Noam Shazeer, et al. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1 67, 2020.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint XrXi::2308. 12950, 2023.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. Bleurt: Learning robust metrics for text generation. arXiv preprint ar Xio:22004, 04696, 2020.
Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, et al. Pangu-coder2: Boosting large language models for code with ranking feedback. arXiv preprint Xiv::220 14986, 2023.
Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K Reddy. Execution-based code generation using deep reinforcement learning. arXiv preprint arXte:2301.13816 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini rescarch and technology. arXiv preprint arXiv:2403 082255 2024.
Peiyi Wang, Lei Li, Zhihong Shao, Rumxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeling of the Association for Compulational Linguistics (Volume 1: Long Popers), pp- 9426 9439, 2024.
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. Codet5+: Open code large language models for code understanding and generation. CoRR, abs5/2305,7792,, 2023.
Chunqiu Steven Xia and Lingming Zhang. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 959-971, 2022.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shich. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.00// 00,51,, 2024.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference Machine Learning, 2024.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. Xr preprint 40 2018.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. ai preprinl orXiv:2011.0 10020, 2024.
Dylan Zhang, Shizhe Diao, Xueyan Zou, and Hao Peng. Plum: Preference learning plus test cases yields better code language models. arXiv preprint ar TFf:24400008885 2024.
Yuhao Zhang, Yasharth Bajpai, Priyanshu Gupta, Ameya Ketkar, Miltiadis Allamanis, Titus Barik, Sumit Gulwani, Arjun Radhakrishna, Mohammad Raza, Gustavo Soares, et al. Overwatch: Learning patterns in code edit sequences. Proceedings of the ACM on Programming Longuages, 6(00PSLA2):305-423, 2022.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang- Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. CoRR, abs/2303.17566 2023.
Daniel M Ziegler, Nisan Stiemmon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 009903999.00893 2019.
15

---

Published in Transactions on Machine Learning Research (06/2025)
A More information
A.1
Experimental setup
Table 4 summarizes the training hyperparameters used for data query and data science tasks across each iteration. It includes key training parameters such as learning rate, batch size, LoRA rank, etc. We set
Table 4: Summary of training hyperparameters for data query and data science for each iteration.
Parameters
Value
Number of epochs
Learning rate
10
0.1
Batch size
16
Data query
Gradient accumulation steps
Maximum sequence length
2048
DeepSpeed Zero stage
Weight decay
0.0001
LoRA rank X
1.0
Number of epochs
Learning rate
x 10
0.5
Batch size
16
Data science
Gradient accumulation steps
Maximum sequence length
512
DeepSpeed Zero stage
2
Weight decay
0.0001
LoRA rank
0.5
K=10 for each iteration, generating 10 positive and negative sample pairs per question. So for the training set of the preference learning in the data science task, we collect about 5k samples and 100k for the data query task. To maintain quality when selecting incorrect samples, we filter out answers that contain repeated strings.
A.2 Datasets
Spider includes 10,181 questions with 5,693 unique SQL queries across 200 databases in 138 domains, while BIRD contains 12,751 question-SQL pairs across 95 large databases, covering over 37 domains. We utilize DS-1000, which comprises 1,000 data science problems sourced from Stack Overflow, covering seven Python libraries related to analysis in data science. The dataset is designed to minimize memorization risk by modifying original problems and uses a multi-criteria evaluation system assess functional correctness and coding constraints. We split DS- 1000 into 500 samples for training and 500 for evaluation.
A.3 Examples for different datasets
For the Data Query task, the model is given natural language problem description and is tasked with generating the corresponding SQL query for a database using an LLM. For example, given the description "How many heads of the deportments are older than 56?", the model should produce the appropriate SQL query to execute this request. In the Data Science task, the LLM is tasked with generating the correct Python code solve given data science problem. For instance, given the problem "I have 2D array to represent many--mny mapping. What is the quickest way to zero out the second and the first column?" we test
16

---

Published in Transactions on Machine Learning Research (06/2025)
LLM's ability to solve data science problems with numpy. We provide examples that highlight the tasks used to evaluate our framework. The first example illustrates the Data Query task (see Figure 6), where models generate SQL queries from natural language descriptions based on a given database schema. The second example showcases the Data science task (sec Figure 7), in which models write Python code to solve typical data manipulation problems, such as processing a data frame. These examples reflect common real-world applications of language models in both querying databases and performing data science operations.
A.4 Qualitative example
Figure 5 illustrates how CODELUTRA refines incorrect attempts into correct solutions over successive iter- ations. the shown scenario, the model is tasked with querying a database for the number of customers paying euros who have monthly consumption over 1000. Initially, the baseline model mo generates an SQT, query that references a non-existent column Currency directly in the yearmonth table, causing an execution error. By the third iteration, the refined model ma has learned to correctly integrale the Currency condition while preserving syntactic and semantic integrity. It accurately groups results by CustomerID, retrieves the correct subset of customers, and computes the intended count. This example highlights CODELUTRA 's abil- ity to iteratively overcome syntactic and logical pitfalls without explicit feedback, resulting in more reliable and accurate code generation over time.
A.5 Model checkpoints and reproducibility
To ensure reproducibility of our experiments, Table 5 provides the exact model checkpoints used for each model in our evaluation. For proprictary models, we include the specific version and access method.
Table 5: Exact model checkpoints used in our experiments.
Model
Checkpoint/Version
Llama-3-8B
meti-llaama//llann------------a
Llama-3-700
meta-llaama/Lllaa--3-177
CodeLJamaa-7B
Star Claerrder1
bigcode/starrcoder
Gemma-7B
google/gemma--biit
Codex
code-davinci-002 (via Azure OpenAl Service)
GPT-3.5
gpt-3.5-turbo-06f
GPT-4
gpl-4-0613
A.6 Setup of the baseline
To ensure apple-to-apple comparison, all experiments report performance metrics using the prompt, guarantecing the comparability of the model evaluation results. For the sampling hyperparameters during evaluation, we set the temperature to 0.2, the maximum tokens to 512, and top-p to 1.0. The SFT model reported in Table consists of training for epoch with a learning rate of 5 x 10-5, a batch size 16, and gradient accumulation steps set to 1, Regularization is applied with weight decay of 0.0001, while a LoRA rank of 8 is employed to enable low-rank adaptation. Additionally, the setup includes 0.5 and 0.5 to balance specific model training.
B More results
B.1 Assessment ability
Models trained with the DPO loss are capable of assessing the quality of code answers. To prevent data leakage, we utilized the robust open-source model Codestral generate multiple samples on Bird's test set, constructing positive and negative sample pairs based on execution accuracy. We evaluated the fine-tuned
17

---

Published in Transactions on Machine Learning Research (06/2025)
Different answers from different models for the data query:
Database schema:
CREATE TABLE customers ( CustomerID INTEGER UNIQUE not mull
primary key, Segment TEXT mull, Currency TEXT null );
CREATE TABLE gasstations ( GasStationID INTEGER UNIQUE not null primary key,
ChainID INTEGER null, Country TEXT mull, Segment TEXT null );
(0mit other database inf information...)
Ground truth another:
SELECT T2. Consumption FROM transactions 1k AS T1
INNER JOIN yearmonth T2 T1. CustomerID CustomerID
WHERE T1 Price / T1. Amount 29.00
AND T1. IroductC 5 AND T2. Date '201208';
Figure 6: An example from the data query dataset from the BIRD (Li et al., 2024).
An example data science from the DS1000 (Lai al., 2023):
Problem:
have simple dataframe which I would like to bin for every 4 rows.
looks like this:
col1\n0
1n1
1ln2
4
5\n4
1
4)n
and would to turn into this:
coll\no 11km1
5ln
have already posted a similar question here
but have no idea how to port the solution to my current use case.
Can you help me out?
Solution:
def g (df))
return df groupby (df. index // 4) sum()
result copy
Figure 7: An example of data science from the DS1000 (Lai et al., 2023).
LLM's ability to accurately assess code quality by mcasuring the classification accuracy on this datasct. Under the standard supervised fine-tuning (SFT) seuting, the model achieved classification accuracy of 56%, which is close to random guessing and indicates that SFT alone lacks this capability. In contrast, our CODELUTRA attain classification accuracy 79%, demonstrating that our approach enables the model better understand code characteristics and select correct answers. This substantial improvement highlights the potential of CODELUTRA.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Table 6: Code quality assessment accuracy.
Methods
Accuracy (%)
Supervised fine-tuning
56.3
Preference learning
79.6
B.2 Comparison of execution-based methods
To examine the limitations of previous methods on small LLMs, we evaluated SFT, SFT with self-debugging (Chen et al., 2024), and CodeLutra on the Spider benchmark using Llama-3-8B as the base model. The results are summarized Table While self-debugging offers minor improvement over SFT (+1.3%),
Method
Execution Accuracy (%)
SFT
67.9
SFT Self-debug
69.2
CodeLutra (Ours)
76.3
Table 7: Execution accuracy on the Spider benchmark with Llama-3-8B.
CodeLutra performs both, achieving 76.3% execution accuracy. This highlights its ability to leverage exe- cution feedback effectively through probabilistic comparisons, iteratively refining model performance without relying complex reward signals or handcrafted prompts. These results demonstrate the scalability and generalizability of CodeLutra, even with smaller, open-source models.
B.3 Comparision with GPT-4 preference code
To assess the reliability of using execution results versus GPT-4 preferences for constructing pairwise pref- erences, we conducted experiments on DS-1000 with Llama-3-818 as the base model. Specifically, we used GPT-4 to judge which answer in each pair is better, similar to the approach employed by RLAIF (Lee et al., 2023). The results are presented in Table 8.
Preference Source
Accuracy (%)
GPT-4 Preference
27.8
CodeLutra (Execution Results)
48.2
Table 8: Comparison of accuracy using GPT-4 preferences versus execution results for constructing pairwise preferences on DS-1000.
The results indicate that using execution results, as implemented in CodeLutra, achieves a significantly higher accuracy (48.2%) compared to GPT-4 preferences (27.8%). This demonstrates that execution-bassr signals provide more reliable guidance for refining models, particularly for smaller LLMs like Llama-3-8B In contrast, GPT-4 preferences appear to struggle, potentially due to the model's lack of grounding in execution semantics and over-reliance on heuristic judgments.
B.4 Additional Benchmark Results
To demonstrate the generalizability of our approach beyond the primary datasets used in our main ex- periments, we evaluated CODELUTRA on the MBPP benchmark from EvalPlus (Liu et al., 2023a). Using Llama-3-8B as our base model, we compared the performance of the base model, standard supervised fine- tuning (SFT), and our CODELUTRA approach.
These results confirm that CODELUTRA'S effectiveness extends beyond our primary datasets, demonstrating its broad applicability across different code generation tasks and benchmarks.
19

---

Published in Transactions on Machine Learning Research (06/2025)
Methods
Accuracy (%)
Base
52.3
SFT
53.1
CodeLutra
55.4
Table 9: Performance comparison on the MBPP benchmark from EvalPlus using Llama-3-8B.
C Limitation and future work
While CODELUTRA enhances code generation performance by leveraging self-generated data, it exhibits limitations that warrant consideration. The framework focuses on the correctness the generated code, overlooking other vital aspects such as efficiency, readability, and adherence to specific formal specifications, which are essential for practical applications. Additionally, CODELUTRA treats all failed code attempts uni- formly, without distinguishing between different lypes severities of errors, potentially limiting the model's ability to learn from more informative mistakes. Furthermore, pairs could be refined-while we sample with replacement when there are fewer than K examples, statistical theory suggests that sampling with replacement even when more examples are available may be beneficial, as it would better approximate sampling from the underlying distribution (similar to bootstrapping in statis- tics). To address the aforementioned limitations, future research related to CODELUTRA should explore several key dircctions. Expanding the preference-guided refinement mechanism incorporate additional criteria such code efficiency and compliance with formal specifications would enhance the overall quality and utility of the generated code. Developing code attempts based on the type and severity of errors could enable more targeted and effective learning, thereby improving the model's ability to avoid similar mistakes in future generations. Exploring alternative evaluation methods, such as static code analysis or formal verification tools, could reduce the framework's reliance on execution results and broaden its applicability to a wider range of tasks. Additionally, inves- tigating optimal sampling stralegies for preference pairs, including consistent sampling with replacement regardless of sample size, may further improve the statistical properties of our approach.
current sampling strategy for preference
nuanced approach to categorizing and prioritizing failed
20


Reviews:
[{"review_id": "mSkriCwaeI", "reviewer": "Reviewer_LfPu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThis work proposed a novel way to combine DPO and SFT objectives for Code Generation models. The paper is well written and flows naturally. Experiments are thorough, show strong performance, and ablation studies highlight the importance of positive & negative examples as well as the SFT regularization. The method is also tested successfully on various base models.\n\n# Weaknesses\n\nThe approach seems generic enough to be applied to any rule-based environment, not only code. The contribution feels more like a DPO regularization than a code generation-specific method. As such, it would be nice to experiment on other domains with rule-based rewards, and potentially remove the name “Code” from “CodeLutra”.\n\nCan the authors comment on why, in Table 1,  the model at iteration 4 is performing slightly lower than the model at iteration 3? This should be discussed in the main text also.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a new way of finetuning LLMs by combining the Direct Preference Optimization (DPO) loss and the supervised finetuning (SFT) loss. The method is applied to code execution benchmarks, notably Data Query and Data Science benchmarks.\nThe method consists of sampling M solutions per problem with the current model, making a dataset of input & (correct, wrong) outputs; and updating the model weights based on the DPO - $\\lambda$ SFT loss.\n\nExperiments show that this method produces models with performance comparable to large, closed source models such as GPT4 and outperform finetuned opensource models.\n\nAblation studies confirm the importance of the SFT term in the proposed objective, the importance of negative examples, and the effectiveness of the approach on different base models (llama, StarCoder, Gemma).\n\nFurther analysis shows that the proposed method helps reduce syntax errors and improves the quality of incorrect answers across iterations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Nothing major.\n\nSmall things:\n- Explain why Model@4 < Model@3 in Table 1\n- Put a Y axis label on Figure 3", "broader_impact_concerns": "Broad Impacts discussed in the paper are good.\n\nMaybe could add a sentence about the impact that better code models can have on the job market, especially for software engineers.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "j4oSmjLRP7", "reviewer": "Reviewer_EmsM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe experimental results are impressive.\n\n\nWeaknesses:\nThe novelty is limited, as the combination of DPO and SFT loss functions has been previously explored, and the iterative refinement approach is also not new.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper uses iterative preference-based refinement to improve code llms, which achieve good performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In Table 1, the bolded number needs to be corrected.\n\nI suggest elaborating on how the code generation problem differs from related tasks in previous work to better highlight the novelty, as the main methodology is not novel enough.\n\nIt would be helpful to further illustrate/introduce the phenomenon where the model tends to decrease the likelihood of both correct and rejected code during training in detail. \n\nI recommend providing more details on how the test inputs are generated, including how reference code is accessed and how the inputs are designed to meaningfully compare the functionality of the generated code with the reference.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EZz3jrZmSc", "reviewer": "Reviewer_Jkh2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is well written and very easy to follow. Figure 1 gives an exceptionally good overview (although it looks a bit garish on my monitor), and the methodology and experiments are presented very clearly.\n- The authors have carried out a fairly thorough set of experiments. I especially appreciate the ablations in Figure 3 as well as the fact that the results were replicated for 3 base models.\n- The methodology is very straight forward; essentially this is just applying DPO to code generation, and it is interesting to see that this works well given that historically speaking RL for code generation has not been able to live up the expectations.\n\nWeaknesses:\n\n- The choice of datasets is a bit weird to me. SQL and Python code generation is a fine setting, since you can expect to have good test coverage, but I expected to find more established datasets like HumanEval(-++), MBPP, or LiveCodeBench. I think the choice of datasets requires some justification by the authors.\n- The abstract and introduction oversell the novelty of the paper. I don't really see the point of giving your method a new name when it is essentially just taking an old idea (RL for code gen), using a newly developed method (DPO), and finding that it does surprisingly well. To clarify I think that is a very solid contribution to the literature, and definitely of interest to the community, and I understand the desire to give your framework a simple and rememberable name. I just think some of the language in the abstract and introduction is a bit over the top: instead of saying \"We present CodeLutra, a novel framework that iteratively improves LLMs for code generation [...]\" you could have just said \"We apply recent developments in preference-based post-training to code generation, and find that it works surprisingly well\".\n- One hypothesis for the trends you observe is that there are flaws in the execution framework for the benchmarks that are artificially limiting which generations are deemed \"correct\".\nFor example, it may be that the execution engine does not allow for imports of external libraries, or that it does not allow for the use of certain language features, or simply that it expects the answers to be given in a certain format (e.g. defining and calling a function with a particular name vs. top-level scripting). This would be a major confounder in your results, since the large performance gains you observe may then simply boil down to the model learning to avoid these features; indeed this might explain your findings in Figure 3(c), and potentially the very large gap between execution feedback and RLAIF in Table 7. I do not think the paper in its current form has taken enough steps to investigate the impact of this confounding factor. I would recommend that consider doing some more extensive qualitative/manual inspection of the generations, or consider extending your core findings to a broader set of domains. As mentioned above, in particular I would encourage the authors to consider LiveCodeBench, which is a more recent code generation benchmark.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the use of preference-based reinforcement learning for code generation. The core idea is to sample multiple generations from the model during training, classify them into positive and negative examples based on whether they pass the unit tests, and to then use DPO to optimize the likelihood of generating positive samples. The authors use this approach to finetune models based on Llama-3-8b, Gemma-7b and StarCoder-7b on two SQL and one Python data science datasets. They find that with this approach, the models approach the performance of larger closed-source models. Furthermore, through a series of ablations the authors show that the use of preference learning on positive vs. negative examples itself is a critical part of the pipeline, and that doing simple expert iteration (SFT on self-generated positive examples) does not work nearly as well.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- (Critical): Address the possible confounding factor mentioned above. I think it is important to investigate this further, even if just by doing some manual inspection of the data you have already generated (e.g. you may not need to run any new experiments).\n- (Critical): Justify your choice of datasets over other established choices such as MBPP, HumanEval(-++), and LiveCodeBench.\n- (Critical): Please fix your citations. There are multiple inferences of using \\cite instead of \\citep, leading to statements like \"While previous approaches Le et al. (2024)\" instead of \"While previous approaches (Le et al., 2024)\".\n- (Critical): Please add (in the appendix) a table of the exact checkpoints you use for each model. This is important for reproducibility. I am also a bit surprised to see Codex in your experiments given it's now been about 2 years since it was officially retired from the OpenAI API, so to aid reproducibility, please include a statement on how you obtained access to the model (is it still served on Azure?).\n- (Critical): Related to the above, please clarify what you mean by \"ChatGPT\" in the text. \"ChatGPT\" is not a model; it is a web interface. I presume you mean some checkpoint of GPT-3.5, but is not clear which since the models being served on chatgpt have evolved substantially since its initial launch.\n- (Critical): In the last paragraph on page 10 there is an incorrect figure reference: Figure 3(b) should be Figure 4(b).\n- (Not critical): Tone down the language in the abstract and introduction. I think the paper is very solid and interesting, but to me the current language is a bit over the top.\n- (Not critical): Intuitively I expect you should probably be sampling with replacement even when you have more than K examples. This is because, in expectation over the draw of samples, sampling with replacement is equivalent to sampling from the base distribution (see the concept of bootstrapping in statistics). This is a minor point, and doesn't affect the overall validity of your empirical claims, but I thought it was worth mentioning in case you want to consider it in future work.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a finetuning methodology for code generation based on preferences of correct against incorrect code outputs. It achieves this through Direct Preference Optimization (DPO) by comparing successful and unsuccessful outputs to align LLMs to correct outputs, while also combining it with supervised finetuning (SFT) to handle the syntactic and semantic requirements of code.

Strengths
- Experiments are solid and comprehensive to demonstrate the improvements for the proposed method over a variety of backbone models (Reviewer LfPu, Reviewer EmsM, Reviewer Jkh2),.
- Ablation study illustrates the roles of individual pieces of design including positive vs negative examples and SFT (Reviewer LfPu, Reviewer Jkh2).
- The paper is generally well written and the presentation is easy to follow (Reviewer LfPu, Reviewer Jkh2). 

Weaknesses
- The reviewers have mixed opinions over the novelty of the proposed method (Reviewer LfPu deems this novel, while Reviewer EmsM and Reviewer Jkh2 point out the novelty as a limitation of this paper.)
- The approach design seems generic and thus it is not clear how it is different from other related tasks (Reviewer LfPu, Reviewer EmsM). 
- More details can be provided such as training dynamics and test input generation (Reviewer EmsM ), and the execution framework limitations as a confounder (Reviewer Jkh2).

Overall, this is a solid empirical contribution to the code generation for open-source models. Author responses generally resolve the questions from the reviewers. Please incorporate the discussion and required changes by reviewers in the final version, especially the additional results of MBPP.
Decision: Accept (poster)