CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement

## Abstract

Abstract Large Language Models (LLMs) have significantly advanced code generation but often require substantial resources and tend to over-generalize, limiting their efficiency for specific tasks. Fine-tuning smaller, open-source LLMs presents a viable alternative; however, it typically lags behind cutting-edge models due to supervised fine-tuning’s reliance solely on correct code examples, which restricts the model’s ability to learn from its own mistakes and adapt to diverse programming challenges. To bridge this gap, we introduce CodeLutra , a novel framework that enhances low-performing LLMs by leveraging both successful and failed code generation attempts. Unlike conventional supervised fine-tuning, CodeLutra employs an iterative preference-guided refinement mechanism to compare correct and incorrect solutions as well as maximize the likelihood of correct codes. Through continuous refinement, CodeLutra enables smaller LLMs to match or surpass GPT-4’s performance in various code generation tasks without relying on vast external datasets or larger auxiliary models. On a challenging data science coding task, using just 500 samples improved Llama-3-8B’s accuracy from 28.2% to 48.6%, approaching GPT-4’s performance. These results highlight CodeLutra ’s potential to close the gap between open-source and closed-source models, making it a promising approach in the field of code generation.

## 1 Introduction

Large language models (LLMs) have revolutionized numerous domains, consistently delivering great performance across different tasks (Brown et al., ; Achiam et al., ; Bubeck et al., ; Team et al., ; Anthropic, ; OpenAI, ). Among these applications, code generation stands out as particularly promising. Models pre-trained on extensive code repositories have demonstrated an impressive capability to solve diverse programming challenges (Li et al., ; ; Nijkamp et al., ; Zheng et al., ; Fried et al., ; Chen et al., ; Wang et al., ; ).

Despite the promise, deploying ultra-large models for code generation poses significant challenges. Top-tier closed-source models like GPT-4 (OpenAI, ) are often highly resource-intensive and do not offer the flexibility for customization to specific code generation tasks. Fine-tuning smaller, open-sourced LLMs for targeted applications serves as a compelling alternative. Code generation demands not just syntactical correctness but also a deep understanding of logical and domain-specific nuances, which complicates model training. While a straightforward approach is to perform supervised fine-tuning (SFT) on the target task, this typically results in only incremental improvements and frequently lags behind cutting-edge solutions. This limited improvement stems from SFT’s reliance solely on correct code examples, which restricts the model’s ability to learn from its own mistakes and adapt to diverse programming challenges. We illustrate this in Figure 1, where SFT provides a minor boost in performance on the challenging data science task, with LlaMa-3-8B’s Pass@1 improving from 28.2% to 30.0%. A notable gap remains between the fine-tuned model and GPT-4, which leads with a Pass@1 score of 49.4%. To address the issue, prior solutions collect additional training data by leveraging more powerful LLMs (Shen et al., ; Luo et al., ; Wang et al., ; Yang et al., ) or by collecting external datasets from public repositories (Lozhkov et al., ; Muennighoff et al., ). *But the big elephant in the room still remains—how much can we bridge the gap by maximizing the utility of existing data and model at hand?*

To address this challenge, we introduce CodeLutra, a framework that iteratively improves the performance of a given LLM *without relying on vast external datasets or larger auxiliary models*. Our approach demonstrates that even with limited data at hand, substantial gains in code generation quality can be achieved, closing the gap between smaller fine-tuned models and the top-tier LLMs. Unlike traditional fine-tuning methods that rely solely on correct code solutions, CodeLutra constructs and learns from both successful and failed code generated by the current model, which forms self-generated comparative data. The failed code attempts are invaluable for refining models, since they provide concrete examples of common errors and enable the model to learn strategies for avoiding similar mistakes in future generations. To harness the comparative data, a key innovation of CodeLutra is the preference-guided refinement mechanism that compares correct and incorrect code snippets, iteratively refining the model’s understanding of code quality. With each iteration, the model generates code solutions, evaluates their correctness, and updates its parameters based on the evolving preference dataset. This process allows for continuous improvements of the base model, making the framework effective even with limited initial data (e.g., a few hundred samples).

Figure: Figure 1: The proposed CodeLutra framework (see Section 3) and performance comparison on different code generation tasks. The orange dashed box highlights our framework’s performance relative to GPT-4.
Refer to caption: x1.png

We comprehensively evaluate the effectiveness of CodeLutra on challenging data query and data science tasks, where the LLM is tasked with generating the correct SQL or Python code to solve a given problem. We compare CodeLutra with 13 open-source and closed-source LLMs that are competitive in code generation. Notably, on the data query task, *our framework allows Llama-3-8B (Dubey et al., ) to achieve an execution accuracy of 76.6%, which exceeds GPT-4’s 74.4%*. Under a challenging data science task, we find that using just 500 samples improved Llama-3-8B from an accuracy of 28.2% to 48.6%, approaching the performance of GPT-4. This demonstrates that CodeLutra achieves strong results even with a limited number of high-quality annotations. Moreover, we observe the consistent performance gains of CodeLutra on different base models, including Gemma-7B (Team et al., ) and StarCoder-7B (Dai & Kumar, ). These findings highlight the potential of CodeLutra in closing the performance gap between open-source and closed-source models. To summarize our key contributions:

- 1.
We present CodeLutra, a novel framework that iteratively improves LLMs for code generation using self-generated comparative data from both successful and failed code attempts, enabling low-performing models to rival top-tier solutions without external datasets or ultra-large LLMs’ feedback.
- 2.
We conduct comprehensive evaluations, comparing CodeLutra against 13 competitive LLMs specializing in code generation. Results demonstrate CodeLutra consistently outperforms both standard fine-tuned LLMs and existing cutting-edge closed-source LLMs.
- 3.
We conduct a series of in-depth analyses to understand the contribution of failed attempts and likelihood regularization for CodeLutra, and reveal CodeLutra improves the performance via reducing syntax errors and improving incorrect answers across iterations.

## 2 Preliminaries

#### LLMs for code generation.

LLMs are pre-trained on diverse datasets encompassing both natural and programming languages. In code generation tasks, an LLM receives a prompt—such as a natural language description, and generates the corresponding code by predicting the next token in the sequence. Formally, code generation is modeled as the conditional probability of a code sequence ${y}=(y_{1},y_{2},\dots,y_{T})$ given an input prompt ${x}$:

$$ $P({y}|{x})=\prod_{t=1}^{T}P(y_{t}|y_{<t},{x})$ (1) $$

where ${x}$ is the input prompt, ${y}$ is the generated code sequence of length $T$, and $y_{<t}=(y_{1},y_{2},\dots,y_{t-1})$ represents the tokens generated before time step $t$.

#### Supervised fine-tuning on task-specific dataset.

Pre-trained LLMs can be suboptimal on task-specific dataset, necessitating fine-tuning. We consider a task-specific dataset $\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{n}$ containing $n$ examples, where each pair $(x_{i},y_{i})$ represents an input prompt $x_{i}$ and its corresponding target code $y_{i}$. Supervised fine-tuning (SFT) adjusts the model’s parameters $\theta$ by maximizing the likelihood of generating correct code sequences $y_{i}$. The loss is defined as:

$$ $\pi_{\text{SFT}}=\operatorname{argmax}_{\pi_{\theta}}\mathbb{E}_{(x_{i},y_{i}) \sim\mathcal{D}}\left(\log\pi_{\theta}\left(y_{i}|x_{i}\right)\right).$ (2) $$

This process relies on the quality of the dataset, guiding the model to produce more accurate and reliable code based on the given prompts.

#### Verification of code correctness.

We verify the correctness of the generated code by comparing it with the ground truth code ${y}_{\text{gt}}$. We define a verification function $V({y})$ that checks if the output of the generated code matches that of the ground truth. Given an input $\mathbf{I}$, the generated code is correct if it produces the same output $f({y},\mathbf{I})$ as the ground truth code $f({y},\mathbf{I})$. Specifically, $V({y})=1$ if the generated code is correct, and $V({y})=0$ otherwise.

#### Limitations of SFT for code generation.

Code generation
demands not just syntactical correctness but also a deep understanding of logical and domain-specific nuances, which complicates model training. *A major limitation of SFT is that it solely maximizes the likelihood of providing correct code, which restricts the model’s ability to learn from its own mistakes*. Since the training process focuses exclusively on provided examples, the LLM doesn’t receive the gradient from incorrect or suboptimal code. For instance, if the model only predicts wrongly in the final token in a code snippet, the overall probability $P({y}|{x})$ in the Equation 1 might still remain high as the preceding tokens are correct. Correspondingly, the SFT loss is very small in that case. However, this single erroneous token can render the entire code nonfunctional or introduce subtle bugs, significantly affecting execution correctness despite a high likelihood score, which is different from general NLP tasks. This reliance on only correct examples limits the model’s capacity to identify and recover from errors, reducing its ability to handle more complex or nuanced coding tasks. This motivates our framework CodeLutra, which leverages both successful and failed code generation attempts.

## 3 CodeLutra

In this section, we introduce CodeLutra, a framework designed to comparatively learn from both correct and incorrect code generations. CodeLutra delivers substantial performance gains, achieving performance comparable to more advanced models like GPT-4 (OpenAI, ), even with limited initial data. The pseudo-code is provided in the Appendix A.1.

#### Initialization.

We start with an initial base model, denoted as $\pi_{0}$, which serves as the starting point for our iterative refinement process. We are provided with an initial training set $\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{n}$, where each $x_{i}$ is a natural language query, and $y_{i}$ is the corresponding ground truth code solution.
Starting with a modestly performing model allows us to clearly observe improvements attributable to the CodeLutra framework, ensuring that enhancements result from our methodology rather than inherent model capabilities. Note that at initialization, we only have the correct codes in hand. We describe next how to obtain incorrect codes to serve the model refinement.

#### Generating correct and failed code.

At each iteration $t$, the current model $\pi_{t}$ generates multiple code responses for each input query $x_{i}\in\mathcal{D}$. Specifically, for each $x_{i}$, the model produces $M$ distinct code samples:

$$ $\hat{y}_{i}^{m}\sim\pi_{t}(x_{i}),\quad\text{for each }m\in\{1,2,\dots,M\}.$ (3) $$

Generating multiple responses introduces diversity in the model’s outputs, providing a richer dataset for composing preference datasets.
Each generated code snippet $\hat{y}_{i}^{m}$ is then executed to assess its correctness by comparing the execution result against that of the ground truth solution $y_{i}$. Correct executions are categorized into the correct code set $Y_{i}^{(c)}$, while incorrect ones are placed into the rejected set $Y_{i}^{(r)}$. This evaluation mechanism offers clear feedback, particularly syntax and execution errors common in code generation tasks, and enables the construction of preference dataset.

#### Preference dataset construction.

A key of our framework involves constructing a preference dataset $\mathcal{D}_{t}$ that captures the relative quality of generated code. The preference data is updated at every iteration $t$. For each input $x_{i}$, we create $K$ preference pairs by randomly pairing one correct code $\hat{y}_{i}^{c_{k}}\in Y_{i}^{(c)}$ with one rejected code $\hat{y}_{i}^{r_{k}}\in Y_{i}^{(r)}$:

$$ $(\hat{y}_{i}^{c_{k}},\hat{y}_{i}^{r_{k}}),\quad\text{for each }k\in[K].$ (4) $$

If either $Y_{i}^{(c)}$ or $Y_{i}^{(r)}$ contains fewer than $K$ responses, sampling with replacement is employed to maintain consistency. The complete preference dataset at iteration $t$ is thus defined as:

$$ $\mathcal{D}_{t}=\{\left(x_{i},\hat{y}_{i}^{c_{k}},\hat{y}_{i}^{r_{k}}\right) \mid\text{ for all }x_{i}\in\mathcal{D}\text{ and }k\in[K]\},$ (5) $$

which contains $n\times K$ preference triplets. Here, $n$ corresponds to the size of the initial dataset $\mathcal{D}$, which remains constant throughout iterations. This dataset encapsulates nuanced comparisons between correct and incorrect code generations, facilitating the subsequent preference learning step. By systematically pairing correct and rejected responses, the model gains a clearer understanding of high-quality code, enabling targeted improvements.

#### Preference-guided refinement.

Figure: Figure 2: Effect of SFT loss to keep the likelihood of correct answers stable.
Refer to caption: x2.png

While one can directly employ a preference optimization approach like DPO (Rafailov et al., ) on our curated dataset $\mathcal{D}_{t}$, this approach presents a notable limitation due to its tendency to decrease the likelihood of both correct and rejected code during training. This is evidenced by the dashed lines in Figure 2, which is also observed in Pal et al. (); Feng et al. (). This diminishing likelihood can significantly impact our framework, especially since our correct code are critical for successfully solving the assigned tasks. It is crucial, therefore, to prioritize the likelihood of correct solutions to the task at hand.

To address this limitation, we employ a dual-loss function integrating DPO with SFT, which regularizes the training to prevent decreasing likelihood. Specifically, we employ SFT on the dataset with correct solutions, $\mathcal{D}_{t}^{c}=\{\left(x_{i},\hat{y}_{i}^{c_{k}}\right)\mid\text{ for all
}x_{i}\in\mathcal{D}\text{ and }k\in[K]\}$. The overall loss function is defined as:

$$ $\displaystyle\pi_{t+1}=\operatorname{argmin}_{\pi_{\theta}}\Bigg{[}$ $\displaystyle\underbrace{-\mathbb{E}_{(x_{i},y_{i}^{c},y_{i}^{r})\sim\mathcal{ D}_{t}}\left[\log\sigma\left(\beta\left(\log\frac{\pi_{\theta}(y_{i}^{c}|x_{i} )}{\pi_{\text{t}}(y_{i}^{c}|x_{i})}-\log\frac{\pi_{\theta}(y_{i}^{r}|x_{i})}{ \pi_{\text{t}}(y_{i}^{r}|x_{i})}\right)\right)\right]}_{\text{compare correct and incorrect solutions}}$ $\displaystyle-\underbrace{\lambda\mathbb{E}_{(x_{i},y_{i}^{c})\sim\mathcal{D}_ {t}}\left(\log\pi_{\theta}\left(y_{i}^{c}|x_{i}\right)\right)}_{\text{maximize likelihood for correct codes}}\Bigg{]},$ (6) $$

where $\lambda$ is a hyperparameter balancing the contributions of the DPO and SFT losses. The first term facilitates preference-based fine-tuning by optimizing the model to favor correct over incorrect code. Concurrently, the second term enhances the likelihood of generating correct solutions directly to avoid the log probability decreasing (see solid line in Figure 2). This dual-loss approach ensures that the model not only ranks correct solutions higher but also increases their generation probability, leading to more consistent high-quality code outputs. We verify the effectiveness of the dual loss empirically in Section 4.3. The refinement process continues until the improvement between consecutive iterations $\pi_{t}$ and $\pi_{t+1}$ becomes marginal, indicating convergence.

#### Remark.

While DPO-style preference optimization has been studied in recent literature (see review in Section 6), its connection to code generation remains largely unexplored. To the best of our knowledge, our work is the first to establish this critical link between iterative preference learning and code generation. We highlight several novel aspects that differentiate our approach from prior work. First, previous studies mostly focus on natural language generation tasks and rely on model-generated rewards or feedback from other large models (Chen et al., ; Xiong et al., ; Yuan et al., ; Pang et al., ; Xie et al., ). In contrast, we focus exclusively on code generation where the preference labels for iterative refinement come from execution results. This shift is essential and unique for code generation, where execution correctness is a key indicator of preference. Moreover, our work uncovers the insight into the dual-loss mechanism, revealing that it plays a crucial role in improving performance in the context of code generation. From a practical standpoint, our training pipeline bypasses the standalone SFT stage typically required before the preference optimization phase, thereby streamlining the training process. Overall, our work not only establishes a novel link between iterative preference learning and code generation but also introduces practical innovations that enhance performance, marking a significant step forward in the field.

## 4 Experiments

### 4.1 Experimental Setup

#### Tasks.

To evaluate the effectiveness of our proposed framework, we perform experiments on two tasks: Data Query and Data Science. Both tasks reflect common and practical challenges in fields such as business, healthcare, and scientific computing, where precise code generation is critical for solving data-related problems while receiving limited attention. For the Data Query task, the model is given a natural language problem description and is tasked with generating the corresponding SQL query for a database using an LLM. For example, given the description “*How many heads of the departments are older than 56?*”, the model should produce the appropriate SQL query to execute this request.
In the Data Science task, the LLM is tasked with generating the correct Python code to solve a given data science problem. For instance, given the problem “I have a 2D array to represent a many-many mapping. What is the quickest way to zero out the second row and the first column?”, we test LLM’s ability to solve data science problems with numpy. We provide examples of the two tasks of the question and ground truth code in the Appendix A.2

#### Datasets.

We conduct our experiments on two cross-domain datasets for data query, Spider (Yu et al., ) and BIRD (Wang et al., ), as well as a data science dataset, DS-1000 (Lai et al., ). Spider includes 10,181 questions with 5,693 unique SQL queries across 200 databases in 138 domains, while BIRD contains 12,751 question-SQL pairs across 95 large databases, covering over 37 domains. We utilize DS-1000, which comprises 1,000 data science problems sourced from Stack Overflow, covering seven Python libraries related to analysis in data science. The dataset is designed to minimize memorization risk by modifying original problems and uses a multi-criteria evaluation system to assess functional correctness and coding constraints. We split DS-1000 into 500 samples for training and 500 for evaluation.

#### Metrics.

For the Data Query task, we adopt the metrics introduced by Yu et al. (): Execution Accuracy (EX), which measures whether the SQL query execution result matches the expected output, and Exact Match (EM), which evaluates whether the generated SQL query exactly matches the reference query in both structure and semantics. For the Data Science task, we use pass@1, following Lai et al. (), which indicates the percentage of correct solutions generated by the model on the first attempt.

**Table 1: The Execution Accuracy (EX), Exact Match (EM), and Pass@1 for different kinds of models on Spider, Bird, and DS1000. We show the base model ($\pi_{0}$) without fine-tuning and the model trained with CodeLutra in different iteration ($\{\pi_{1},\pi_{2},\pi_{3},\pi_{4}\}$). For fair comparison, all reported results in the table use the same prompt. Boldface highlight GPT-4 and our results.**
| Models | Spider | BIRD | DS1000 |  |  |
| --- | --- | --- | --- | --- | --- |
| EX | EM | EX | EM | Pass@1 |  |
| Open-source LLMs |  |  |  |  |  |
| Llama-3-8B (Dubey et al., ) | 59.3 | 55.1 | 22.3 | 19.5 | 28.2 |
| Codellama-7B (Xu & Zhang, ) | 57.0 | 51.4 | 24.4 | 18.7 | 25.6 |
| StarCoder-7B (Lozhkov et al., ) | 61.2 | 58.6 | 25.7 | 23.0 | 26.8 |
| Gemma-7B (Team et al., ) | 49.9 | 46.7 | 21.2 | 19.1 | 24.2 |
| Codestral-22B (Brown & Lee, ) | 71.3 | 69.6 | 42.5 | 39.9 | 35.8 |
| Llama-3-70B-Instruct (Dubey et al., ) | 68.7 | 65.4 | 41.2 | 39.3 | 36.4 |
| Fine-tuned LLMs |  |  |  |  |  |
| Llama-3-8B (Dubey et al., ) | 67.9 | 64.7 | 35.6 | 30.7 | 30.0 |
| Codellama-7B (Xu & Zhang, ) | 67.3 | 64.3 | 36.3 | 30.9 | 26.8 |
| StarCoder2-7B (Lozhkov et al., ) | 66.9 | 64.1 | 36.6 | 31.1 | 29.4 |
| Gemma-7B (Team et al., ) | 65.8 | 62.8 | 34.5 | 29.8 | 27.4 |
| Closed-Source LLMs |  |  |  |  |  |
| Codex (Chen et al., ) | 73.1 | 70.2 | 44.7 | 42.4 | 38.4 |
| ChatGPT (Ouyang et al., ) | 71.8 | 68.4 | 44.3 | 40.2 | 38.8 |
| GPT-4 (OpenAI, ) | 74.4 | 71.2 | 46.3 | 43.2 | 49.4 |
| CodeLutra (Ours) |  |  |  |  |  |
| Base ($\pi_{0}$) | 59.3 | 55.1 | 22.3 | 19.5 | 28.2 |
| Iteration 1 ($\pi_{1}$) | 67.8 | 63.9 | 37.8 | 33.2 | 43.2 |
| Iteration 2 ($\pi_{2}$) | 72.4 | 68.3 | 40.8 | 36.0 | 46.8 |
| Iteration 3 ($\pi_{3}$) | 76.6 | 72.5 | 43.1 | 38.6 | 48.6 |
| Iteration 4 ($\pi_{4}$) | 76.3 | 72.1 | 42.6 | 38.3 | 48.2 |

#### Baselines.

To evaluate the effectiveness of our method, we compare it against three categories of baselines. For a fair comparison, all reported results are based on the same prompt.

- •
Open-source LLMs: We benchmark our method against competitive open-source LLMs, including models pre-trained on general datasets such as Llama-3-8B (Dubey et al., ), Gemma-7B (Team et al., ), and Llama-3-70B-Instruct (Dubey et al., ). Additionally, we compare against LLMs pre-trained specifically on coding datasets, such as Codellama-7B (Xu & Zhang, ), StarCoder-7B (Lozhkov et al., ), and Codestral-22B (Brown & Lee, ).
- •
Fine-tuned LLMs: As supervised fine-tuning on domain-specific datasets is a popular and effective way to improve LLMs’ corresponding performance, we also report the performance of fine-tuned LLMs using standard supervised fine-tuning methods (Raffel et al., ).
- •
Closed-source LLMs: We provide the performance of cutting-edge closed-source LLMs, including Codex (Chen et al., ), ChatGPT (Ouyang et al., ), and GPT-4 (OpenAI, ).

#### Experimental setup.

For main results, we apply our framework to the Llama-3-8B base model (Dubey et al., ), denoted as $\pi_{0}$ (see Section 4.2 for more backbone results). We use a zero-shot prompt containing the question along with reference information (dataset schema for data query and reference code for data science). For different answer collections, we employ the best-of-$n$ strategy by sampling 16 responses at the temperature of 1.0. We train one epoch per iteration and perform four iterations in total, resulting in models $\{\pi_{1},\pi_{2},\pi_{3},\pi_{4}\}$. These models are evaluated as described in the following sections. For more experimental details, please refer to the Appendix 5.

### 4.2 Main Results

#### Results on the data query task.

We compare CodeLutra with baselines in code generation for the data query task, as shown in Table 1. We found that existing open-source LLMs like Llama-3-8B still have a significant performance gap in code generation for data queries compared to closed-source LLMs like GPT-4. Although supervised fine-tuning can help bridge this gap—e.g., SFT increases the EX of Llama-3-8B on Spider from 59.3% to 67.9%—there remains a notable difference with GPT-4’s 74.4%. Through our refinement framework, Llama-3-8B after four iterations exceeded SFT performance by 16.9% and *even outperforms GPT-4 with an execution accuracy of 76.6%*. Additionally, on the more challenging BIRD dataset, after three iterations, CodeLutra significantly improved the EX of the base model from 22.3 to 43.1, achieving performance very close to GPT-4.

#### Results on the data science task.

Table 1 also presents results for the data science task, where we evaluate both open-source and closed-source LLMs, as well as our method CodeLutra. On the DS-1000 dataset, open-source models like Llama-3-8B and Gemma-7B struggle, with significantly lower EM and Pass@1 scores compared to closed-source models like GPT-4. Fine-tuning provides a minor boost in performance, as seen with Llama-3-8B’s Pass@1 improving from 28.2% to 30.0%. However, as with the data query task, a large performance gap remains between fine-tuned open-source models and closed-source ones, where GPT-4 leads with a Pass@1 score of 49.4%. Nonetheless, CodeLutra demonstrates substantial improvements (from 28.2% to 48.6%), offering a promising path for narrowing this gap further.

### 4.3 More experiments

Figure: Figure 3: (a) Ablations on the effects of negative samples for training. (b) Ablations on the question number during training. (c) The effects of ground truth for validation during preference datasets collection.
Refer to caption: x3.png

#### The importance of learning from failed attempts.

Our framework CodeLutra leverages both positive and negative answer pairs to iteratively improve model performance, particularly by minimizing the generation of incorrect responses. But what happens when we omit the negative samples and rely solely on supervised fine-tuning using positive samples generated by the model? In this ablation, we compare the performance of our objective 7 with a model trained with $\mathcal{L}_{\text{SFT}}(\pi;\mathcal{D}_{t}^{c})$. As seen in Figure 3(a), without negative samples (purple line), the model’s performance plateaus across iterations, remaining close to the baseline. In contrast, incorporating negative samples (blue line) leads to steady performance improvements over successive iterations. This ablation confirms that including negative samples is critical to refining the model’s ability to distinguish between optimal and suboptimal responses, significantly boosting overall performance. Collecting incorrect answers is thus an essential component for preference learning and contributes to greater model accuracy.

#### CodeLutra achieves strong performance under limited training data.

The cost of acquiring high-quality question-code pairs can be significant, so we examine whether our method truly depends on large datasets. Under the data science code generation task, *we found that using just 500 samples improved Llama-3-8B from an accuracy of 28.2 to 48.6, approaching the performance of GPT-4*. This demonstrates that CodeLutra achieves strong results even with a limited number of high-quality annotations. We further verify this with the data query task by randomly selecting 1K question-code pairs from BIRD’s training data and comparing them to the full 9K sample set. The results, as shown in Figure 3(b), reveal similar trends, with the two setups reaching peak execution accuracies of 43.1 and 42.4, respectively. This minor difference suggests that CodeLutra does not heavily rely on large volumes of training data and can generalize well with fewer annotations, which is crucial for minimizing the cost of dataset collection.

#### Importance of SFT regularization during preference optimization.

**Table 2: Ablations on the SFT on the correct answers.**
| Methods | Spider | DS1000 |
| --- | --- | --- |
| $\mathcal{L}_{\text{DPO}}$ | 17.2 | 12.4 |
| Ours | 76.6 | 48.6 |

Recall in Section 3 that our loss function integrates DPO with SFT to regularize the training, and prevent decreasing likelihood on the correct solution. In Table 1,we ablate the effect of SFT regularization on both the Spider and DS-1000 datasets. Notably, omitting optimizing the SFT loss on the correct solutions results in a marked decline in model performance, *e.g.*, $\downarrow$59.4% on Spider. This highlights the effectiveness of the dual loss approach, ensuring that the model not only ranks correct solutions higher but also increases their generation probability, leading to more consistent high-quality code outputs.

#### CodeLutra remains effective on different base models.

**Table 3: Performance with CodeLutra of Gemma-7B and StarCoder-7B across Spider and DS1000 benchmarks.**
| Model | Gemma-7B | StarCoder-7B |  |  |
| --- | --- | --- | --- | --- |
| Spider | DS1000 | Spider | DS1000 |  |
| $\pi_{0}$ | 49.9 | 24.2 | 61.2 | 26.8 |
| $\pi_{1}$ | 63.7 | 38.8 | 72.8 | 39.6 |
| $\pi_{2}$ | 69.3 | 43.6 | 74.7 | 42.4 |
| $\pi_{3}$ | 71.3 | 44.4 | 77.2 | 45.2 |
| $\pi_{4}$ | 72.6 | 44.0 | 77.5 | 45.8 |

To further validate the generalization capability of our framework CodeLutra, we extend our experiments to two additional open-source base models: Gemma-7B (Team et al., ) and StarCoder-7B (Lozhkov et al., ). As summarized in Table 3, we report the results on both the Spider and DS1000 datasets across multiple iterations of our refinement process. For Gemma-7B, we observe a significant improvement
in Execution Accuracy (EX) on Spider, starting from 49.9% at $\pi_{0}$ (the base model) and reaching 72.6% after four iterations ($\pi_{4}$). A similar trend is observed in the DS1000 dataset, where the Pass@1 metric improves from 24.2% to 44.0%. For StarCoder-7B, the improvements are also pronounced, with EX on Spider increasing from 61.2% to 77.5%, and Pass@1 on DS1000 rising from 26.8% to 45.8%. These results demonstrate that our framework is robust across different model architectures, consistently yielding significant performance gains regardless of the underlying base model. Notably, the iterative refinement process of CodeLutra continues to improve the accuracy and correctness of generated code, highlighting the CodeLutra generalization to different code generation tasks.

## 5 Further Analysis on CodeLutra

#### Is ground truth code necessary for preference dataset collection?

Recall that our framework relies on ground truth code to evaluate the quality of generated code during the collection of preference datasets. To test the impact of this dependence, we conduct experiments that replace the ground truth with a more general criterion—whether the generated code is executable. In the absence of ground truth, we consider executable answers as chosen and non-executable ones as rejected. Applying this approach to the Bird dataset, we observe notable gains despite the absence of ground truth: accuracy rose from 22.3 to 30.9 (see Figure 3(c)). Moreover, the proportion of executable code surged from 59.8% to 89.7%, showing that the model effectively learned to avoid common errors, such as syntax issues or missing database tables. *This experiment demonstrates that using executability as a metric still enables substantial model improvements, making the method applicable even without high-quality annotations*, and highlights the robustness of CodeLutra under such conditions.

#### CodeLutra helps reduce the syntax errors across iterations.

To evaluate whether our method enables LLMs to learn from their mistakes over multiple iterations, we sampled 100 error cases from the test set using models ${\pi_{0},\pi_{1},\pi_{2},\pi_{3}}$, trained with CodeLutra on the BIRD dataset for qualitative analysis. We measure the fraction of executable code generated by each model. As shown in Figure 4, the percentage of non-executable code decreases from 40% to 11% when trained with CodeLutra, indicating that the models have improved in mastering SQL syntax and are better at avoiding basic errors. A qualitative example in Figure 5 highlights this improvement: the base model incorrectly queries the “Currency” column in the wrong table, resulting in an error, while the model trained with CodeLutra successfully generates the correct SQL query.

Figure: Figure 4: (a) Error analysis for different models. (b) BLEURT similarity for different models with the ground truth on the correct set and error set divided from the base model.
Refer to caption: x4.png

Figure: Figure 5: CodeLutra helps reduce the syntax errors.$\pi_{0}$ fails due to a missing column, while $\pi_{3}$ generates the correct query.

#### CodeLutra improves quality of incorrect answers across iterations.

Based on the responses of the initial model $\pi_{0}$, we divide the test set into a correct set and an error set. We track the quality trends of the model in these two sets across iterations.
Using the cosine similarity metric based on the BLEURT embedding proposed by Sellam et al. (), we calculate similarities denoted as $\text{sim}{\pi_{t}}(\hat{y},y_{\text{gt}})$ for the model fine-tuned over $t$ iterations. Here $\hat{y}$ denotes the model generation, and $y_{\text{gt}}$ is the ground truth solution. As shown in Figure 3(b), we observe that the similarity between the model’s output on the correct set and the ground truth remains stable (see purple bars), while with each iteration, the similarity between the model’s output on the error set and the ground truth increases significantly—from 0.48 to 0.54. This indicates that CodeLutra helps the base model improve outputs on error set, while the outputs on correct cases remain qualitatively stable.

## 6 Related Work

#### Preference learning for LLMs.

Preference learning aims to guide language models toward generating outputs that align with human preferences and desirable outcomes. A significant body of research has utilized human feedback to refine LLMs across various language tasks (Ziegler et al., ; Ouyang et al., ; Stiennon et al., ; Kreutzer et al., ). The Reinforcement Learning from Human Feedback (RLHF) framework, in particular, has been effective in aligning large pre-trained language models (Christiano et al., ; Ziegler et al., ; Ouyang et al., ; Bai et al., ). However, RLHF can suffer from training inefficiencies and sensitivity to hyperparameters. In response, recent studies have shifted towards closed-form loss functions that directly utilize offline preference data, exemplified by DPO (Rafailov et al., ) and related methodologies (Liu et al., ; Ethayarajh et al., ; Gheshlaghi Azar et al., ; Pal et al., ; Liu et al., ; Xiong et al., ; Tang et al., ; Yu et al., ). While DPO-style models inherently provide rewards, iterative DPO—where the model generates its own pairwise preference data—has demonstrated strong performance and potential (Chen et al., ; Xiong et al., ; Yuan et al., ; Rosset et al., ; Pang et al., ; Xie et al., ). In this work, we introduce iterative preference-guided refinement to code generation for the first time, achieving GPT-4-level results and providing an in-depth analysis that paves the way for future research.

#### LLMs for code generation.

LLMs trained on vast corpora of code have demonstrated remarkable capabilities across a wide range of tasks, including code generation (Chen et al., ; Austin et al., ; Zhang et al., ), program repair (Xia & Zhang, ; Wei et al., ; Xia et al., ; Jiang et al., ; Bouzenia et al., ; Xiong et al., ), and software testing (Chen et al., ; Wang et al., ; Zhou et al., ). Foundational models (Nijkamp et al., ; Wang et al., ; Li et al., ; Roziere et al., ) pre-trained on extensive codebases, have established strong general-purpose capabilities for code generation. Building upon these powerful foundations, more recent models like WizardCoder (Luo et al., ) and DS-Coder (Li et al., ; Bouzenia et al., ) enhance contextual understanding by leveraging repository-level organization of pretraining data and incorporating retrieval-augmented techniques (Borzunov et al., ). Moreover, CodeInstruct (Wang et al., ) capitalize on instruction fine-tuning to improve alignment with human coding preferences. Fine-tuning methodologies and prompt-engineering techniques (Luo et al., ; Chen et al., ; Zhang et al., ) have been crucial in unlocking these models’ full potential for domain-specific tasks, such as security, AI-assisted development, and code synthesis in specialized fields. Zhang et al. () pay attention to preference learning for programming problems. However, they rely on GPT-4 for generating test cases and use preference data in a single offline run. In contrast, we *iteratively* refine a small LLM using self-generated preference data, *without external datasets or larger models*. Moreover, different from Zhang et al. () performing SFT and preference learning in two stages, we introduce the dual-loss to compare correct and incorrect solutions and maximize the likelihood of correct codes in one stage.

## 7 Conclusion

We introduced CodeLutra, a preference-guided refinement framework designed to enhance LLMs for code generation without the need for external datasets or larger models. By utilizing self-generated data from both successful and failed attempts, CodeLutra refines the model’s understanding of code quality. Our experiments on data query and data science tasks demonstrate that CodeLutra significantly boosts the performance of base LLMs. For example, fine-tuning Llama-3-8B with CodeLutra outperforms GPT-4 on data query tasks and nearly matches GPT-4 performance on data science task with small training data. Additionally, CodeLutra reduced common coding errors while improving the quality and accuracy of generated code. These results underline CodeLutra’s potential as a cost-efficient and scalable solution for LLM code generation.

## Appendix A More information

### A.1 Pseudo code of CodeLutra

We summarize our algorithm in implementation in the Algorithm 1. The algorithm operates iteratively, generating code responses for each input and leveraging both correct and incorrect code samples to construct a preference dataset. These comparisons serve as the basis for model refinement, where the model updates its parameters in each iteration. The process ensures that the model learns not only from correct code solutions but also from common mistakes, thereby improving its ability to generate high-quality code across diverse tasks. This iterative refinement, guided by self-generated comparative data, distinguishes CodeLutra from traditional supervised fine-tuning approaches.

Figure: Algorithm 1 CodeLutra

### A.2 Examples for different datasets.

We provide examples that highlight the tasks used to evaluate our framework. The first example illustrates the Data Query task (see Figure 6), where models generate SQL queries from natural language descriptions based on a given database schema. The second example showcases the Data science task (see Figure 7), in which models write Python code to solve typical data manipulation problems, such as processing a data frame. These examples reflect common real-world applications of language models in both querying databases and performing data science operations.

Figure: Figure 6: An example from the data query dataset from the BIRD (Wang et al., ).

Figure: Figure 7: An example of data science from the DS1000 (Lai et al., ).

### A.3 More results

**Table 4: Code quality assessment accuracy.**
| Methods | Accuracy (%) |
| --- | --- |
| Supervised fine-tuning | 56.3 |
| Preference learning | 79.6 |

Models trained with the DPO loss are capable of assessing the quality of code answers. To prevent data leakage, we utilized the robust open-source model Codestral to generate multiple samples on Bird’s test set, constructing positive and negative sample pairs based on execution accuracy. We evaluated the fine-tuned LLM’s ability to accurately assess code quality by measuring the classification accuracy on this dataset. Under the standard supervised fine-tuning (SFT) setting, the model achieved a classification accuracy of 56%, which is close to random guessing and indicates that SFT alone lacks this capability. In contrast, our CodeLutra attain a classification accuracy of 79%, demonstrating that our approach enables the model to better understand code characteristics and select correct answers. This substantial improvement highlights the significant potential of CodeLutra.

### A.4 Experimental setup

Table 5 summarizes the training hyperparameters used for data query and data science tasks across each iteration. It includes key training parameters such as learning rate, batch size, LoRA rank, etc.

**Table 5: Summary of training hyperparameters for data query and data science for each iteration.**
|  | Parameters | Value |
| --- | --- | --- |
| Data query | Number of epochs | 1 |
| Learning rate | $5\times 10^{-5}$ |  |
| $\beta$ | 0.1 |  |
| Batch size | 16 |  |
| Gradient accumulation steps | 1 |  |
| Maximum sequence length | 2048 |  |
| DeepSpeed Zero stage | 2 |  |
| Weight decay | 0.0001 |  |
|  | LoRA rank | 8 |
|  | $\lambda$ | 1.0 |
| Data science | Number of epochs | 1 |
| Learning rate | $5\times 10^{-5}$ |  |
| $\beta$ | 0.5 |  |
| Batch size | 16 |  |
| Gradient accumulation steps | 1 |  |
| Maximum sequence length | 512 |  |
| DeepSpeed Zero stage | 2 |  |
| Weight decay | 0.0001 |  |
|  | LoRA rank | 8 |
|  | $\lambda$ | 0.5 |

We set $K$=10 for each iteration, generating 10 positive and negative sample pairs per question. To maintain quality when selecting incorrect samples, we filter out answers that contain repeated strings.

## Appendix B Limitation and future work

While CodeLutra significantly enhances code generation performance by leveraging self-generated comparative data, it exhibits several limitations that warrant consideration. The current framework primarily focuses on the correctness of the generated code, overlooking other vital aspects such as efficiency, readability, and adherence to specific formal specifications, which are essential for practical applications. Additionally, CodeLutra treats all failed code attempts uniformly, without distinguishing between different types or severities of errors, potentially limiting the model’s ability to learn from more informative mistakes.

To address the aforementioned limitations, future research related to CodeLutra should explore several key directions. Expanding the preference-guided refinement mechanism to incorporate additional criteria such as code efficiency, readability, and compliance with formal specifications would enhance the overall quality and utility of the generated code. Developing a more nuanced approach to categorizing and prioritizing failed code attempts based on the type and severity of errors could enable more targeted and effective learning, thereby improving the model’s ability to avoid similar mistakes in future generations. Exploring alternative evaluation methods, such as static code analysis or formal verification tools, could reduce the framework’s reliance on execution results and broaden its applicability to a wider range of tasks.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - How does CodeLutra perform on longer programs, e.g,, competitive coding? (This is nice to have; the material in the paper is enough for publication.)
- What is the CodeLutra's performance vs program length on the current data sets?
soundness: 3
strengths: - CodeLutra simple yet effective method, with clear articulation of how it differs from related work.

- Impressive results: with only 500 samples, CodeLutra achieves GPT-4-level performance on a base model with just 8 billion parameters. For the Spider benchmark, it improves base model performance from 59.3 to 74.4 in just four iterations, surpassing GPT-4’s 74.4. On BIRD, it increases performance from 22.3 to 42.6 in four iterations, approaching GPT-4’s 46.3.

- Comprehensive evaluation, covering three coding benchmarks (Spider, BIRD, and DS-1000) and three models (Llama-3-8B, Gemma-7B, and StarCode-7B), demonstrating the approach's generalizability.

- Strong ablations that address key questions: (i) dual loss significantly boosts performance, raising it from 17.2 (DPO) to 76.6 on Spider; (ii) negative samples are crucial, as performance increases from 20 to over 40 with their inclusion, while positive samples alone yield minimal improvement.
summary: This paper presents CodeLutra, a supervised fine-tuning (SFT) approach that demonstrates significant improvements on coding tasks. Specifically, CodeLutra achieves GPT-4-level performance fine-tuning an open-source 8-billion-parameter model using as few as 500 samples (with groundtruth solutions). The key idea behind CodeLutra is to use both positive and negative examples to fine-tune the model, creating a hybrid of SFT and DPO techniques. CodeLutra assumes a ground truth to generate these examples: if a sample code produces the same inputs/outputs as the ground truth, it is a positive example; otherwise, it is a negative example.
weaknesses: - Current evaluation focuses on SQL queries and data science problems, which are relatively short (from a few lines of code to several 10s of lines of code). It would be interesting to see how this approach generalizes to longer programs.
- Limited exploration of scenarios without ground truth. In such cases, CodeLutra relies on syntactic error detection, but the results are, as expected, less impressive.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: - In line 150, "if the model only predicts wrongly in the final token in a code snippet, the overall probability P (y|x) in the Equation 1 might still remain high as the preceding tokens are correct". While the hypothesis makes sense, do you really observe this situation in real LLM and dataset? I doubt it.
- Equation 6 is studied in a previous literature "Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer" with theoretical support, but not cited. This also limits the novelty contribution (at least on this part).
- I'm confused with the experimental setup. What is the training dataset? It seems the experiment is using the test dataset to train. Could you clarify?
- Could you explain the setting for SFT in Table 1? One baseline is the SFT model that only uses the groundtruth training solutions, or use the synthetically generated correct solutions. Which one are you using?
- I don't think Table 2 is a right setting, where 17.2 and 12.4 is extremely low for DPO-only method. Normally we do DPO training on top of SFT model. The right setup should be training on top of SFT model. What is the gap between, SFT then DPO training and the SFT regularized preference training?
- 500 samples might mean that 500 prompts or problems. What is the size of generated samples overall?
soundness: 4
strengths: The paper is well-written. The proposed method with training with correct and failed generations iteratively makes sense. Experiments show good improvement on benchmarks.
summary: This paper proposes a method by iteratively generating successful and failed code and training with preference optimization.
weaknesses: Some experimental setup is not clear enough, such as training data, SFT setting, and details of synthetically generated dataset. One of the contribution DPO and SFT loss is studied in previous literature. More experiments might be needed for comparing SFT then DPO with DPO+SFT loss.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: NA
soundness: 2
strengths: NA
summary: The paper introduces CODELUTRA, a framework designed to enhance the performance of LLMs in code generation tasks. However, the method is almost the same as an existing method.
weaknesses: 1. The proposed method closely resembles that presented in [1]. Applying the same approach to a different scenario does not warrant publication, especially since this new scenario is simpler and benefits from execution feedback. 

[1] Iterative Reasoning Preference Optimization. https://arxiv.org/abs/2404.19733

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: * Are the two answers in Figure 5 flipped? Given Currency is in table customers, I feel the first answer is correct, and the second is wrong.
soundness: 2
strengths: * Comprehensive evaluation, ablation, and analysis support the effectiveness of the proposed method. In particular, the necessity of negative training samples and of SFT loss are both well studied.
* The paper is well written and easy to follow.
summary: The paper proposes CODELUTRA, a preference-guided training framework to let code LLMs iteratively refine itself based on execution signals from its own generations. Specifically, given a task-specific training set, at each iteration, the model generates answers which are then evaluated by unit tests. Each correct answer is paired with an incorrect answer to form a preference pair. The preference dataset is then used for DPO training. To address the issue that DPO may reduce the generation probability of both correct and incorrect answers, supervised finetuning loss is added to DPO loss for joint training. Experiments show that CODELUTRA significantly improves performance on SQL and data science tasks, and is much more effective than DPO alone.
weaknesses: * The technical novelty of this paper is somewhat limited. L233-246 claimed two major points of novelty: refinement from execution feedback and dual loss mechanism. First, using feedbacks from program execution to iteratively refine code LLMs is a direction that has been extensively studied (e.g., CodeRL [1], and NExT [2]). However, these works are not discussed in the related work section. Second, the dual loss objective (i.e. adding SFT loss in DPO training) was proposed in [3], known as RPO, which is not cited.
* I find DS-1000 Pass@1 results in Table 1 are inconsistent with the public leaderboard (https://ds1000-code-gen.github.io/model_DS1000.html). In particular, pass@1 of Codestral-22B and Llama-3-70B-Chat is 51.2 and 48.6 respectively in the leaderboard, but 35.8 and 36.4 respectively as reported in the paper.


Reference:

[1] Le, Hung, et al. "Coderl: Mastering code generation through pretrained models and deep reinforcement learning." Advances in Neural Information Processing Systems 35 (2022): 21314-21328.

[2] Ni, Ansong, et al. "NExT: Teaching Large Language Models to Reason about Code Execution." Forty-first International Conference on Machine Learning.

[3] Liu, Zhihan, et al. "Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer." arXiv preprint arXiv:2405.16436 (2024).

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. See weakness 1, 2
2. I am curious that if this method can lead to a model that is generalizable. For example, the authors split DS-1000 for training and evaluation. I wonder how the resulting model would perform on other similar datasets, e.g. MBPP?
soundness: 3
strengths: 1. The paper is well-organized and easy to follow
2. The proposed method can lead to a fine-tuned LLAMA3-8B model which has comparable performance to GPT-4.
3. The authors conduct comprehensive ablation studies that the effect of every component involved in their method is clearly demonstrated.
4. The method can still have good performance with limited annotations or training samples.
summary: The authors proposed a new training framework called CODELUTRA which aims to fine-tune a small CodeLM to match or surpass the closed-source LLMs like GPT-4. CODELUTRA adopts an iterative method to learn by comparing the correct generation and the failed generation. At each iteration, CODELUTRA constructs the preference dataset by classifying the generation codes of the model from the last iteration and employs a dual-loss function that combines DPO with SFT for training. The authors show that their method can achieve a performance comparable to GPT-4 in the data query and data science tasks.
weaknesses: 1. Line 230 states that "The refinement process continues until the improvement between consecutive iteration becomes marginal". However, in the experiments, the authors seem to fix the iteration number to 4. In practice, how do you decide if the improvement between consecutive iterations is marginal?

2. The baseline setup is not clear enough and may not be comprehensive.
a) For closed-source LLMs, it is unknown what prompting method is used. It is also not clearly stated what fine-tuning method is used. From the Appendix, I infer that the LoRA is used in CODELUTRA but is it also used in the fine-tuning baseline?
b) In Table 1, since LLAMA-3 is used as the base model for CODELUTRA, the authors should apply more previous fine-tuning methods in the same setting and compare with them instead of comparing with different open-source CodeLLMs. For example, the related work section mentions other fine-tuning methods (Line 520), e.g. Self-debug and Codefort. The authors should apply them to fine-tune LLAMA3 and compare the results.

3. Paper presentations can be further improved. Specifically, a) Line 142 $f$ is not defined. b) The notations in the legend of Figure 2 are not defined.