CodeDPO: Aligning Code Models with Self Generated and Verified Source Code
Authors: Kechi Zhang, Ge Li, Yihong Dong, Jingjing Xu, Jun Zhang, Jing Su, Yongfei Liu, Zhi Jin
Accepted at: acl25

CodeDPO: Aligning Code Models with Self Generated and Verified Source Code

Kechi Zhang1,2†, Ge Li1,2*, Yihong Dong1,2†, Jingjing Xu3,
Jun Zhang3, Jing Su3, Yongfei Liu3,
Zhi Jin1,2*
1Key Lab of High Confidence Software Technology (PKU), Ministry of Education
2School of Computer Science, Peking University, China
3ByteDance
{zhangkechi, lige, zhijin}@pku.edu.cn

Abstract
Code generation models have shown significant potential for programming tasks. However, existing training methods like supervised fine-tuning face key limitations: they do not effectively teach models to prioritize correct over incorrect solutions in ambiguous situations, nor do they effectively optimize the runtime efficiency of the generated code. To address these challenges, we propose CodeDPO, a framework that integrates preference learning into code generation to improve two key code preference factors: code correctness and efficiency. CodeDPO employs a novel dataset construction method, utilizing a self-generation-and-validation mechanism that simultaneously generates and evaluates code and test cases. The underlying assumption is that test cases executable by multiple code snippets provide more reliable validation, and code that passes more tests is more likely to be correct. Through this self-validation process, our PageRank-inspired algorithm iteratively updates the ranking score of each code snippet, ultimately creating a code preference optimization dataset based on correctness and efficiency. CodeDPO is flexible and scalable, generating diverse preference optimization data without depending on powerful models such as GPT-4. Through comprehensive evaluations of five widely used benchmarks, CodeDPO demonstrates significant improvements in correctness and efficiency compared to existing methods. Our experiments prove that CodeDPO enhances the capabilities of LLMs in code generation and provides a robust foundation for conducting code preference optimization in more complex and challenging real-world scenarios.1

1 Introduction
In recent years, code generation models have gained significant attention for their potential to automate software development (Zhang et al., 2024c). Models such as GPT-4 (GPT-4, 2023), Claude, and open-source alternatives like Phi (Gunasekar et al., 2023; Abdin et al., 2024), DeepSeekCoder (Guo et al., 2024), and StarCoder (Li et al., 2023; Lozhkov et al., 2024) have demonstrated the capability of LLMs to handle complex code generation tasks (Zhang et al., 2024c,b). However, one of the ongoing challenges lies in boosting the correctness and efficiency of the generated code.

To improve code generation models, a common approach is supervised fine-tuning (SFT) (Zhang et al., 2023b), where models are trained on pairs of instructions and correct code snippets. While SFT improves the overall quality of the generated code, it falls short in teaching models to consistently prefer correct solutions over incorrect ones (Hong et al., 2024). Figure 1 illustrates the likelihood of generating code with varying correctness and efficiency during SFT training. When we adopt SFT training on those correct solutions, as the likelihood of preferred outputs increases, the probability of generating undesirable outputs also rises, leading to performance saturation.

To address these limitations, recent research has turned to direct preference optimization (DPO) (Rafailov et al., 2024), a method designed for alignment based on pairwise preference data. DPO allows models to rank different outputs and choose preferable solutions (e.g., more factual or helpful). While DPO has shown success in reasoning tasks like mathematics (Lai et al., 2024; Wu et al., 2024), its application in code generation remains under-explored. Unlike natural language tasks, code generation requires objective metrics, such as executability, which poses challenges for directly applying DPO. In this paper, we first define code preference based on two key factors—Correctness and efficiency. Correctness refers to whether the code solves the problem accurately, while efficiency measures how quickly
Figure 1: Log probabilities for code with varying correctness and efficiency during Phi-2-2.7B model training on our constructed dataset. The traditional SFT strategy struggles to teach models to prefer correct solutions over incorrect or slow ones. In contrast, our CodeDPO approach effectively optimizes for both correctness and efficiency.

the code runs. Existing methods (Gee et al., 2024; Zhang et al., 2024a) rely heavily on high-quality test cases to assess correctness. However, these approaches struggle to fully address correctness and efficiency, facing limitations such as restricted data diversity, an imbalance between positive and negative samples, and insufficient focus on optimizing code efficiency.

In this paper, we introduce CodeDPO, a novel framework that integrates preference learning into code model training to optimize both correctness and efficiency. CodeDPO constructs the dataset from real-world code repositories using a self-generation-and-validation mechanism, where code and test cases are simultaneously generated and evaluated. We assume that tests executable by more code snippets are more reliable, and code that passes more tests is more likely to be correct. To implement this, CodeDPO uses a mutual verification process: each receives an initial self-validation score, which is iteratively updated using a PageRank-inspired (Page, 1999) algorithm. This algorithm adjusts the credibility of each code snippet and tests by considering their relations in cross-verification, prioritizing solutions based on correctness and efficiency. The final preference-optimized dataset is then used to train various code models using the DPO learning algorithm. A key advantage of CodeDPO is its flexibility. Unlike existing methods that rely on high-quality test cases or powerful models to generate them, CodeDPO does not depend on these resources. Its self-generation and validation mechanism supports the scalable creation of diverse and robust preference optimization data. This allows our framework to optimize code models for real-world scenarios where high-quality test data may be sparse.

CodeDPO can serve as a crucial step in the post-training phase of code models. We conduct experiments on five popular benchmarks such as HumanEval (Chen et al., 2021), HumanEval+ (Liu et al., 2024a), MBPP (Austin et al., 2021), MBPP+, and DS-1000 (Lai et al., 2023) with CodeDPO, demonstrating its superiority over existing methods. Notably, we develop a top-performing 6.7B model by building on an existing SFT strategy (Guo et al., 2024; Wei et al., 2023) and further enhancing it with our CodeDPO approach, achieving an impressive 83.5% pass rate on HumanEval. We also conduct ablation studies to investigate the impact of our self-generation-and-validation mechanism and other preference optimization settings. Our findings confirm that CodeDPO enhances the code generation capabilities of LLMs while providing a solid foundation for further research into optimizing code generation for both correctness and efficiency.

2 Related Work

2.1 Large Language Models for Code

Code generation, which automates writing source code from natural language (NL) descriptions, is gaining significant attention. LLMs have shown strong capabilities in this area due to their large-scale training on diverse datasets, such as OpenAI’s GPT-4 (GPT-4, 2023), StarCoder (Li et al., 2023; Lozhkov et al., 2024), Code Llama (Rozière et al., 2023), and DeepSeekCoder (Guo et al., 2024). These models are often fine-tuned further, such as through instruction-supervised fine-tuning (SFT), to maximize their coding potential. Since gathering high-quality data is difficult, researchers use self-instruct methods to generate synthetic instruction data from powerful models like GPT-4 (Wang et al., 2022; Taori et al., 2023; Chaudhary, 2023). Evol-Instruct (Luo et al., 2023) uses more complex prompts for better data generation. OSS-instruct (Wei et al., 2023) allows LLMs to get inspired from real-world code snippets for better qual-
ity in coding tasks. While these SFT methods boost code quality, it does not fully train models to prefer correct solutions over incorrect ones (Hong et al., 2024). Updating training strategies is critical for improving these code models to handle various coding tasks.

2.2 Preference Optimization for Code Models

Preference optimization techniques have recently been used to help LLMs prefer better outputs over weaker ones in various natural language tasks (Rafailov et al., 2024). The Direct Preference Optimization (Rafailov et al., 2024) has been widely applied to LLM alignment due to its convenience and effectiveness. Its objective is defined as:

\[
L_{DPO} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{ref}(y_w \mid x)} \right. \right. \\
\left. \left. - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{ref}(y_l \mid x)} \right) \right]
\]

Compared with the SFT loss, the DPO loss introduces a preference-based mechanism. Instead of merely maximizing the likelihood of ground truth data, as in SFT, DPO optimizes the model to align with human preferences by leveraging both preferred responses (\( y_w \), winning) and dispreferred responses (\( y_l \), losing). While DPO has proven effective in reasoning tasks like mathematics (Lai et al., 2024), its use in code generation is still under-explored. Code generation requires objective measures of correctness and efficiency, unlike natural language tasks where preferences are often more subjective. Some works have simply explored PO (Gee et al., 2024; Zhang et al., 2024a; Ethayarajh et al., 2024; Zhang et al., 2025). Code-Optimize (Gee et al., 2024) builds its dataset from the MBPP-train subset, which includes just 384 problems. PLUM uses GPT-4 to generate tests, which are then used to validate and rank code solutions. PLUM currently achieves state-of-the-art performance in preference optimization for code models. However, PLUM (Zhang et al., 2024a) faces some limitations. It uses a limited number of tests to validate the code, and the resulting dataset is imbalanced due to its validation method, which means it can only use KTO (Ethayarajh et al., 2024) for training. Additionally, PLUM does not consider the code efficiency. This paper introduces CodeDPO, which does not rely on external test cases or powerful models for dataset generation. Our approach uses a self-generation and validation mechanism to create balanced preference pairs, aiming to optimize both correctness and efficiency.

3 CodeDPO: Self-Verified Performance Optimization Code Generation Framework

CodeDPO is designed to integrate preference learning into code generation models, improving both the correctness and efficiency of the generated code. As shown in Figure 2, our method involves four key steps: ① Data Seed Construction with real-world source code: We first collect a data seed from open-source code repositories and generate programming task prompts. ② Correctness Optimization with self-validation score: We generate code and tests simultaneously, using a self-generation-and-validation loop to build a dataset for correctness optimization. The self-validation score is iteratively updated based on whether the generated code passes the tests. We assume that tests executable by multiple code snippets are more reliable, and code that passes more tests is more likely to be correct. As illustrated in the figure, after two iterations, the score of code-1 changes from 1 to 1.75 to 2.6875 (~2.7 in the figure), as it passes more reliable tests and receives higher scores with each update, indicating a greater likelihood of correctness. ③ Efficiency Optimization with execution time: We measure execution time on selected credible test sets to build the dataset for efficiency optimization. In the figure, we select test-1 and test-2 as the credible test set to measure the execution time of each code snippet. ④ Model Optimization Training: We collect the dataset from the previous two stages and use Direct Preference Optimization (DPO) to train various code models.

3.1 Data Seed Construction

The data seed construction for CodeDPO is the first step for initiating the preference learning process to generate programming task prompts. We adopt a method inspired by OSS-instruct (Wei et al., 2023, 2024)\footnote{We follow the implementation provided at https://github.com/bigcode-project/starcoder2-self-align/tree/fd0af77e2773b14696c7cea02a472f9e99d9c4e3.}, which extracts key programming concepts from open-source code repositories. These concepts serve as the foundation for generating various programming task prompts. For
Figure 2: Our CodeDPO involves four steps: ① Data Seed Construction with real-world source code; ② Correctness Optimization with self-validation score (in this figure we set \( T \) to 2 and \( d \) to 0.5. For simplicity, the final score in the figure is rounded to one decimal place. Details are shown in Appendix J.2.1); ③ Efficiency Optimization with execution time on credible tests; ④ Model Optimization Training.

example, a code snippet that performs sorting operations might highlight concepts such as sorting algorithms, data structure traversal, and time complexity. From these concepts, we generate code generation prompts. The data seed thus allows the model to explore a wide range of scenarios.

3.2 Correctness Optimization with Self-Generation and Validation

Central to CodeDPO is the self-generation-and-validation loop, which enables the model to iteratively update the code correctness rank through mutual validation of code and test cases (Chen et al., 2022, 2023; Zhang et al., 2023a). The process begins by generating multiple candidate code snippets based on a prompt. Simultaneously, corresponding test cases are generated to evaluate these snippets. The validation loop follows these steps:
1. **Code Generation**: Given an instruction, the model generates a set of candidate code snippets \( C = \{c_1, c_2, ..., c_n\} \).
2. **Test Case Generation**: Test cases \( T = \{t_1, t_2, ..., t_m\} \) are generated in parallel to validate the candidate snippets.
3. **Validation Process**: Each code snippet is executed against the generated test cases. The validation outcomes are used to update the self-validation scores for both the code snippets and the test cases.

Ranking Code Snippets and Test Cases Using Self-Validation Scores To rank both code snippets and tests, we employ a PageRank-inspired (Page, 1999) iterative algorithm. Initially, each code and test is assigned a self-validation score of 1. Over a fixed number of iterations \( T = 10 \), these scores are updated based on the performance of the snippets and test cases during validation.

The self-validation score for code snippets and test cases is updated using the following formulas:

\[
\text{Score}_t(c_i) = (1 - d) \times \text{Score}_{t-1}(c_i) \\
+ d \times \sum_{t_j} \text{Score}_{t-1}(t_j) \times \text{Link}(t_j, c_i)
\]
\[
\text{Score}_t(t_j) = (1 - d) \times \text{Score}_{t-1}(t_j) \\
+ d \times \sum_{c_i} \text{Score}_{t-1}(c_i) \times \text{Link}(c_i, t_j)
\]

Where \( d \) is the damping factor, and \( \text{Link}(t_j, c_i) \) indicates whether a code snippet \( c_i \) passes the test case \( t_j \). This iterative process is repeated until convergence. After \( T \) iterations, the final rankings reflect the quality of the code snippets and test cases based on the correctness.

3.3 Execution Efficiency Optimization

In addition to ensuring correctness, CodeDPO integrates execution efficiency optimization to ensure that our approach generates functionally correct and efficient code. During the self-validation loop, the execution time for each code snippet is recorded. However, not all test cases accurately reflect the efficiency of the code. To address this, we use the top-performing code from the correctness optimization phase as a reference, assuming the test cases it passes are credible. The total execution time for each code snippet is then measured based on the subset of these credible tests. Code snippets that pass these credible test cases with lower execution times are assigned higher efficiency scores. Finally, we collect both fast and slow code snip-
pets as part of the training dataset for execution efficiency optimization, which is used for further training, encouraging the model to prioritize solutions that are accurate and optimized for speed during code generation.

3.4 Final Dataset and Model Optimization

The final dataset is built from the previous two optimization dataset construction stages, accounting for correctness and execution time. We filter out samples whose ranking scores are identical or too close. The final dataset consists of 93k correctness optimization samples and 21k efficiency optimization samples. Each sample includes a unique code problem prompt with a preferred and a rejected solution. In the subsequent training, we combine both correctness and efficiency data to optimize the model in both aspects simultaneously. In our experiments, we apply Direct Preference Optimization (DPO) (Rafailov et al., 2024) across various code models to facilitate optimization learning. To enhance the stability and robustness of the training process, we employ RPO (Pang et al., 2024; Liu et al., 2024b) format loss, which essentially consists of a weighted SFT loss on the chosen preferences together with the original DPO loss. We utilize both base models and SFT models as the backbone for further training. Our goal is to demonstrate that CodeDPO has the potential to enhance code models at different stages of their training, even for models that have undergone extensive training or fine-tuning. The setup details are provided in Section 4.2.

4 Experiment Setup

In this study, we aim to investigate the following research questions:

RQ1: Does CodeDPO improve the correctness of generated code compared to baseline models on standard benchmarks? We evaluate on HumanEval (Chen et al., 2021), HumanEval+ (Liu et al., 2024a), MBPP (Austin et al., 2021), MBPP+, DS-1000 (Lai et al., 2023) and LiveCodeBench (Jain et al., 2024).

RQ2: Does CodeDPO enhance the execution efficiency of generated code? We measure the execution efficiency of code generated by CodeDPO compared to baseline models.

RQ3: What is the impact of the self-generation-and-validation algorithm on CodeDPO’s performance? We perform ablation studies by removing or modifying the self-generation-and-validation mechanism to assess its contribution to the overall performance.

RQ4: How does the choice of preference optimization strategy affect CodeDPO’s effectiveness? We evaluate different preference optimization strategies, including Direct Preference Optimization (DPO), Kahneman-Tversky Optimization (KTO) (Ethayarajh et al., 2024), and Supervised Fine-Tuning (SFT), to understand their impact on the model’s performance.

RQ5: How does data scaling influence the performance of CodeDPO? We investigate data scaling by varying the amount of training data to show how data size affects its ability.

4.1 Backbone LLMs

We evaluate several widely used LLMs in the code generation domain for our experiments, covering both base models and SFT models at different training stages. For base models, we apply CodeDPO to Phi-2 (2.7B) (Gunasekar et al., 2023), DeepSeekCoder-base (1.3B, 6.7B) (Guo et al., 2024), and StarCoder2-base (7B) (Lozhkov et al., 2024). Additionally, we evaluate our method on several fine-tuned SFT models (Wei et al., 2023), including Magicoder-CL-7B, Magicoder-S-CL-7B, Magicoder-DS-6.7B, and Magicoder-S-DS-6.7B, which are fine-tuned based on CodeLlama-7B and DeepSeekCoder-base-6.7B using state-of-the-art SFT techniques.

While applying the PO phase after SFT is generally recommended (Rafailov et al., 2024), we extend our evaluation to base models as they can generate more diverse code snippets and offer more significant potential for improvement (Wang et al., 2024). We choose all these popular models as the backbone of our experiments to optimize correctness and execution efficiency.

4.2 Training and Inference Settings

For dataset construction, we use DeepSeekCoder-v2 as the data generation model. For each problem prompt, we sample 15 code solutions and test cases from this model with \( temperature = 1.5 \). To construct the preference optimization dataset, we set \( T \) to 10 and \( d \) to 0.85 for the self-validation score. Our practice shows that this parameter configuration quickly yields a stable ranking score. In this paper, we focus on constructing a Python dataset. The total cost of our dataset construction process is nearly 80$. For training, we train each code model
for 10 epochs and select the best-performing model based on the lowest validation loss. We utilize a learning rate of 5e-6 with a linear scheduler and warm-up. For inference, we use greedy search decoding for code generation. We use 16 A100 GPUs for all experiments.

5 Results and Analyses

5.1 Code Correctness (RQ1)

We evaluate the model performance on five widely-used code generation benchmarks: HumanEval, HumanEval+, MBPP, MBPP+, and DS-1000. Following the standard training process (base model → SFT → DPO), we first record the performance of the base model, SFT model, and DPO-aligned model on DeepSeekCoder-6.7B, as shown in Table 1. With the enhancement of our CodeDPO, the final model achieves an 83.5% pass rate on HumanEval. Notably, even after high-quality SFT training, CodeDPO still achieves additional performance improvements. CodeDPO plays a crucial role in the post-training phase of code models, significantly boosting overall performance.

We further evaluate the performance of CodeDPO alongside baselines3 on a wide range of base models and SFT models. As shown in Table 2, CodeDPO achieves the best performance on both HumanEval(+) and MBPP(+). Compared to the baseline models in the first row of each block, we observe that CodeDPO delivers significant improvements across all models, regardless of their initial performance. Notably, we achieve a 36.1% relative improvement on StarCoder2-7B. Additionally, CodeDPO shows remarkable gains on the more challenging HumanEval+, demonstrating its robustness under stricter evaluation. Thanks to CodeDPO’s data construction strategy, we can build a reliable preference dataset that helps the model favour high-quality outputs, leading to more robust and reliable code generation.

For DS-1000, as shown in Table 3 and 9, we further evaluate CodeDPO across different libraries. We did not incorporate prior knowledge of specific Python libraries in our data construction. While we observe slight performance drops in the Torch and TensorFlow settings, this may be due to the relatively low percentage of these libraries in our dataset construction. However, CodeDPO demonstrates overall performance improvements over their respective baselines. DS-1000 differs from benchmarks like HumanEval and MBPP in data format and the coding skills it assesses, and it ensures that it is excluded from nearly all models’ training sets. It proves that CodeDPO can enhance the model’s coding capabilities in more complex and diverse scenarios. Details are in Appendix A.

We conducted additional experiments on LiveCodeBench (Jain et al., 2024) in Table 4, one of the most challenging benchmarks for real-world competitive coding tasks. The results indicate that CodeDPO demonstrates significant performance improvements for both the base model and the supervised fine-tuning (SFT) model across all difficulty levels. The gains are particularly notable in the “medium” and “hard” subsets, which represent some of the most challenging problems in competitive programming tasks. These findings highlight the effectiveness of the proposed framework for real-world, complex coding tasks.

5.2 Code Efficiency (RQ2)

To address RQ2, we follow existing methods (Shypula et al., 2024) by measuring the execution time of the generated code and calculating the speed-up ratio. We also evaluate the percentage of optimized code before and after applying CodeDPO, where a program is considered optimized if it is at least 10% faster than its baseline. These metrics are based on the intersection of solved problems before and after applying CodeDPO. We select HumanEval+ and MBPP+ for evaluation because they significantly expand the diversity of test cases, making them more reliable for measuring the execution efficiency of the generated code under a variety of edge cases. Since runtime environments can affect measurements, we repeat each evaluation five times and show the distribution in Figure 3. It is clear that CodeDPO consistently improves code performance. The speed-up ratio shows that our method speeds up the code by 1.25 to 1.45 times. Additionally, the percentage of optimized code indicates that after applying CodeDPO, around 20%-45% of generated code solutions have been improved, confirming its effectiveness in enhancing code efficiency.

3The baselines have not yet published their datasets. We reproduced the Code-Optimize experiment based on the reported settings. For PLUM, we report results from their paper using models identical to ours, which is why some models do not include PLUM results.
<table>
  <tr>
    <th>Model</th>
    <th>HumanEval</th>
    <th>HumanEval+</th>
    <th>MBPP</th>
    <th>MBPP+</th>
  </tr>
  <tr>
    <td>DeepSeekCoder-6.7B-base</td>
    <td>47.60</td>
    <td>39.60</td>
    <td>70.20</td>
    <td>56.60</td>
  </tr>
  <tr>
    <td>+ SFT (<i>with MagiCoder-OSS-instruct</i>)</td>
    <td>73.17</td>
    <td>68.29</td>
    <td>76.72</td>
    <td>66.67</td>
  </tr>
  <tr>
    <td>+ SFT + Our CodeDPO</td>
    <td><b>83.54</b></td>
    <td><b>76.22</b></td>
    <td><b>80.70</b></td>
    <td><b>70.93</b></td>
  </tr>
</table>

Table 1: Pass rates (%) of code models at different stages on HumanEval(+) and MBPP(+). We track the performance of the base model, SFT model, and DPO-aligned model on DeepSeekCoder-6.7B. Our CodeDPO shows additional improvements, even after high-quality SFT training.

<table>
  <tr>
    <th>Model</th>
    <th>HumanEval (/+)</th>
    <th>MBPP (/+)</th>
  </tr>
  <tr>
    <th colspan="3">SFT Model</th>
  </tr>
  <tr>
    <td>MagiCoder-CL-7B</td>
    <td>51.21 / 48.78</td>
    <td>65.60 / 55.82</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>60.36 / 54.87</b></td>
    <td><b>70.93 / 59.15</b></td>
  </tr>
  <tr>
    <td><i>Code-Optimise</i></td>
    <td>48.78 / 46.95</td>
    <td>67.17 / 57.14</td>
  </tr>
  <tr>
    <td>MagiCoder-S-CL-7B</td>
    <td>67.07 / 61.59</td>
    <td>69.58 / 60.58</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>74.39 / 71.95</b></td>
    <td><b>71.43 / 61.40</b></td>
  </tr>
  <tr>
    <td><i>Code-Optimise</i></td>
    <td>64.63 / 54.88</td>
    <td>69.42 / 60.15</td>
  </tr>
  <tr>
    <td><i>PLUM</i></td>
    <td>73.80 / 69.50</td>
    <td>71.40 / 60.80</td>
  </tr>
  <tr>
    <td>MagiCoder-DS-6.7B</td>
    <td>57.93 / 53.66</td>
    <td>75.93 / 64.02</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td>67.07 / 62.80</td>
    <td><b>81.70 / 68.92</b></td>
  </tr>
  <tr>
    <td><i>Code-Optimise</i></td>
    <td>57.93 / 51.83</td>
    <td>76.19 / 64.91</td>
  </tr>
  <tr>
    <td><i>PLUM</i></td>
    <td><b>71.30 / 65.90</b></td>
    <td>79.60 / 66.70</td>
  </tr>
  <tr>
    <td>MagiCoder-S-DS-6.7B</td>
    <td>73.17 / 68.29</td>
    <td>76.72 / 66.67</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>83.54 / 76.22</b></td>
    <td><b>80.70 / 70.93</b></td>
  </tr>
  <tr>
    <td><i>Code-Optimise</i></td>
    <td>68.90 / 64.63</td>
    <td>78.20 / 67.92</td>
  </tr>
  <tr>
    <td><i>PLUM</i></td>
    <td>80.50 / 73.80</td>
    <td>80.40 / 69.30</td>
  </tr>
  <tr>
    <th colspan="3">Base Model</th>
  </tr>
  <tr>
    <td>Phi-2-2.7B</td>
    <td>48.78 / 46.34</td>
    <td>65.34 / 54.49</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>57.32 / 51.83</b></td>
    <td><b>69.05 / 56.88</b></td>
  </tr>
  <tr>
    <td><i>Code-Optimise</i></td>
    <td>49.39 / 47.56</td>
    <td>67.42 / 55.80</td>
  </tr>
  <tr>
    <td>DeepSeekCoder-1.3B</td>
    <td>31.53 / 28.65</td>
    <td>57.40 / 48.67</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>42.07 / 38.04</b></td>
    <td><b>61.37 / 53.43</b></td>
  </tr>
  <tr>
    <td><i>Code-Optimise</i></td>
    <td>34.15 / 30.49</td>
    <td>59.15 / 49.87</td>
  </tr>
  <tr>
    <td>DeepSeekCoder-6.7B</td>
    <td>47.60 / 39.60</td>
    <td>70.20 / 56.60</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>59.75 / 51.83</b></td>
    <td><b>72.18 / 60.01</b></td>
  </tr>
  <tr>
    <td><i>Code-Optimise</i></td>
    <td>47.56 / 37.20</td>
    <td>72.18 / 57.64</td>
  </tr>
  <tr>
    <td><i>PLUM</i></td>
    <td>56.70 / 48.80</td>
    <td><b>72.90 / 58.90</b></td>
  </tr>
  <tr>
    <td>StarCoder2-7B</td>
    <td>35.40 / 29.90</td>
    <td>54.40 / 45.60</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>48.17 / 34.15</b></td>
    <td><b>58.40 / 49.37</b></td>
  </tr>
  <tr>
    <td><i>Code-Optimise</i></td>
    <td>32.32 / 28.05</td>
    <td>58.90 / 47.89</td>
  </tr>
  <tr>
    <td><i>PLUM</i></td>
    <td>46.30 / 39.60</td>
    <td><b>60.40 / 49.10</b></td>
  </tr>
</table>

Table 2: Pass rate (%) of CodeDPO compared to baseline models on HumanEval and MBPP.

<table>
  <tr>
    <th>Model</th>
    <th>plot</th>
    <th>np</th>
    <th>pd</th>
    <th>torch</th>
    <th>scipy</th>
    <th>sk</th>
    <th>tf</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>Magic-CL-7B</td>
    <td>54.8</td>
    <td>16.4</td>
    <td>16.5</td>
    <td>17.6</td>
    <td>23.6</td>
    <td>29.6</td>
    <td>33.3</td>
    <td>25.5</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td>57.4</td>
    <td>37.3</td>
    <td>22.7</td>
    <td>22.1</td>
    <td>35.8</td>
    <td>31.3</td>
    <td>31.1</td>
    <td><b>34.0</b></td>
  </tr>
  <tr>
    <td>Magic-DS-6.7B</td>
    <td>55.5</td>
    <td>37.7</td>
    <td>28.2</td>
    <td>25.0</td>
    <td>34.0</td>
    <td><b>45.2</b></td>
    <td>33.3</td>
    <td>37.1</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>59.4</b></td>
    <td><b>40.5</b></td>
    <td><b>29.2</b></td>
    <td>23.5</td>
    <td><b>39.6</b></td>
    <td>42.6</td>
    <td>31.1</td>
    <td><b>38.7</b></td>
  </tr>
  <tr>
    <td>Phi-2-2.7B</td>
    <td>42.6</td>
    <td>33.6</td>
    <td>15.5</td>
    <td>16.2</td>
    <td>17.0</td>
    <td>11.3</td>
    <td>17.8</td>
    <td>17.5</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>49.0</b></td>
    <td><b>33.6</b></td>
    <td>16.5</td>
    <td>14.7</td>
    <td><b>20.8</b></td>
    <td>14.8</td>
    <td>13.3</td>
    <td><b>25.3</b></td>
  </tr>
  <tr>
    <td>StarCoder2-7B</td>
    <td>54.2</td>
    <td>37.7</td>
    <td>18.6</td>
    <td>25.0</td>
    <td>31.1</td>
    <td>23.5</td>
    <td>35.6</td>
    <td>31.4</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>56.8</b></td>
    <td><b>38.2</b></td>
    <td><b>18.9</b></td>
    <td>20.6</td>
    <td><b>39.6</b></td>
    <td><b>25.2</b></td>
    <td>31.1</td>
    <td><b>32.6</b></td>
  </tr>
</table>

Table 3: Pass rate (%) of CodeDPO on DS-1000 across seven libraries using greedy decoding.

<table>
  <tr>
    <th>Model</th>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
  </tr>
  <tr>
    <th colspan="4">Base Model</th>
  </tr>
  <tr>
    <td>DeepSeek-Coder-6.7B</td>
    <td>39.9</td>
    <td>7.4</td>
    <td>0.4</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td>51.9</td>
    <td>12.2</td>
    <td>0.7</td>
  </tr>
  <tr>
    <th colspan="4">SFT Model</th>
  </tr>
  <tr>
    <td>MagiCoder-S-DS-6.7B</td>
    <td>48.1</td>
    <td>10.7</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td>53.1</td>
    <td>16.3</td>
    <td>0.7</td>
  </tr>
</table>

Table 4: Performance comparison on LiveCodeBench across difficulty levels.

5.3 Ablation Studies

5.3.1 Self Generation and Validation Algorithm (RQ3)

Correlation between self-validation scores and actual code accuracy using HumanEval ground truth tests To evaluate the effectiveness of our self-generation-and-validation algorithm, we examine the correlation between self-validation scores and actual code accuracy. We use HumanEval for this preliminary experiment. For each problem in HumanEval, we sample 15 code solutions and tests following the setting in Section 4, and then use different strategies to rank these generated codes. To evaluate the rank quality, we execute with the ground truth for each code to get the actual code accuracy. We consider three experimental strategies:

① Self-validation score, which refers to our original method. ② Filter with all tests, which assumes all generated test cases are correct and uses them to judge code correctness. This approach creates passed/non-passed pairs, similar to the baseline PLUM (though PLUM uses GPT-4 for test generation, while we use a more cost-effective model). ③ Sort by number of passed tests, which counts the number of passed tests for each code among all generated tests, using the code with the most and least passed tests as the comparison pair. It is commonly employed in post-processing methods, such as CodeT (Chen et al., 2022).
Table 5 presents the Spearman, Kendall’s Tau, and Normalized Discounted Cumulative Gain (NDCG) metrics for the different ranking strategies. Our experiments show that the self-validation score is highly correlated with actual code accuracy. In contrast, filtering by all tests heavily depends on the quality of the test generation model. Sorting by the number of passed tests treats all tests equally important. However, due to the inherent uncertainty in generated tests, these methods can be vulnerable to low-quality tests. Our proposed self-validation method employs a mutual reinforcement mechanism to update the credibility of both code and tests, effectively mitigating these issues.

<table>
  <tr>
    <th>Method</th>
    <th>Spearman</th>
    <th>Kendall's Tau</th>
    <th>NDCG</th>
  </tr>
  <tr>
    <td>Self-validation score</td>
    <td><b>0.8598</b></td>
    <td><b>0.8047</b></td>
    <td><b>0.9653</b></td>
  </tr>
  <tr>
    <td>Filter with all tests</td>
    <td>0.6114</td>
    <td>0.6114</td>
    <td>0.8753</td>
  </tr>
  <tr>
    <td>Sort by # of passed tests</td>
    <td>0.7724</td>
    <td>0.7250</td>
    <td>0.9162</td>
  </tr>
</table>

Table 5: Correlation between self-validation score and actual code accuracy on HumanEval.

Figure 3: Runtime Speedup and Percentage of Optimized Code on HumanEval+ and MBPP+.

Impact of self-validation score on model performance  We apply these strategies to construct datasets and evaluate the final model performance in code generation. Table 6 presents the model performance across various dataset construction strategies. We introduce a new strategy—random selection—which randomly selects two code solutions from the generated code as the preference pair. The experiment results demonstrate that the self-generation-and-validation algorithm plays an essential role in ensuring the correctness and reliability of the preference dataset construction. Details are shown in Appendix B.

<table>
  <tr>
    <th>Model</th>
    <th>HumanEval (+)</th>
    <th>MBPP (+)</th>
  </tr>
  <tr>
    <td>DeepSeekCoder-1.3B</td>
    <td>31.53 / 28.65</td>
    <td>57.40 / 48.60</td>
  </tr>
  <tr>
    <th colspan="3">Data Construction Strategies</th>
  </tr>
  <tr>
    <td>Filter with all tests</td>
    <td>34.75 / 29.89</td>
    <td>57.40 / 48.80</td>
  </tr>
  <tr>
    <td>Sort by # of passed tests</td>
    <td>37.19 / 31.09</td>
    <td>58.39 / 50.37</td>
  </tr>
  <tr>
    <td>Random selection</td>
    <td>21.34 / 18.29</td>
    <td>48.94 / 38.35</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>42.07 / 38.04</b></td>
    <td><b>61.37 / 53.43</b></td>
  </tr>
  <tr>
    <th colspan="3">Training Strategies</th>
  </tr>
  <tr>
    <td>SFT</td>
    <td>39.02 / 35.36</td>
    <td>59.45 / 50.26</td>
  </tr>
  <tr>
    <td>Our CodeKTO</td>
    <td>40.85 / 35.98</td>
    <td>59.65 / 50.13</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>42.07 / 38.04</b></td>
    <td><b>61.37 / 53.43</b></td>
  </tr>
</table>

Table 6: Ablation study of performance based on DeepSeekCoder-1.3B.

5.3.2 Impact of PO Training Strategy (RQ4)
We explore the impact of different preference optimization strategies (DPO, KTO, and SFT) on model performance. For training, the SFT strategy uses the best code solution from our constructed dataset. In our KTO strategy, we replace DPO with KTO in our framework. As shown in Figure 1, the traditional SFT strategy struggles to guide the model in preferring correct solutions over incorrect or slower ones during training. The results in Table 6 demonstrate that DPO performs best among these strategies. Benefiting from our dataset construction method, we can obtain well-balanced preference pairs, enhancing the contrastive mechanism in DPO. Details are shown in Appendix C.

5.4 Data Scaling Law for CodeDPO (RQ5)
To address RQ5, we explore how scaling the training data affects CodeDPO’s performance. We show the experiment results for RQ5, which can help us explore how scaling the training data affects CodeDPO’s performance. We train the model with varying amounts of data—25%, 50%, and 75%—and evaluate its impact on the model performance. As shown in Table 7, increasing the
data consistently improves model performance, but these improvements gradually plateau as the dataset size grows. For example, HumanEval scores rise from 32.92 (25%) to 41.46 (75%), with similar trends observed on MBPP. In our experiments, we carefully balance performance gains and training costs, ensuring optimal results with CodeDPO. In further research, we plan to expand the current training scale to explore the extreme limits of CodeDPO’s performance.

<table>
  <tr>
    <th>Model</th>
    <th>HumanEval</th>
    <th>HumanEval+</th>
    <th>MBPP</th>
    <th>MBPP+</th>
  </tr>
  <tr>
    <td>DeepSeekCoder-1.3B-base</td>
    <td>31.53</td>
    <td>28.65</td>
    <td>57.40</td>
    <td>48.67</td>
  </tr>
  <tr>
    <td>25%</td>
    <td>32.92</td>
    <td>29.87</td>
    <td>55.13</td>
    <td>47.87</td>
  </tr>
  <tr>
    <td>50%</td>
    <td>36.59</td>
    <td>31.70</td>
    <td>58.14</td>
    <td>49.87</td>
  </tr>
  <tr>
    <td>75%</td>
    <td>41.46</td>
    <td>37.80</td>
    <td>60.65</td>
    <td>52.63</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td>42.07</td>
    <td>38.04</td>
    <td>61.37</td>
    <td>53.43</td>
  </tr>
</table>

Table 7: Model Performances with different data scaling in our CodeDPO.

5.5 Overlap Avoidance with Existing Benchmarks

The seed dataset for CodeDPO was randomly selected from the open-source pretraining dataset The Stack, consisting of approximately 100k functions. This design explicitly considers data decontamination, since the seed dataset has already gone through rigorous data decontamination. It suggests that our dataset is unlikely to introduce additional data leakage beyond the seeds. To ensure quality, we applied a simple filtering process using tools like Tree-sitter and Pyright for static analysis and code formatting.

We intentionally avoided introducing any prior knowledge that might lead to significant overlap with evaluation benchmarks. We also implemented post-sampling data decontamination, similar to MagiCoder and StarCoder. However, given the already low overlap, this process only removed fewer than 30 samples. Thus, we can ensure that there is no risk of the dataset containing examples highly similar to the test sets.

To assess potential overlap for the final dataset with existing benchmarks, we followed the methodology used in MagiCoder. Specifically, we calculated the cosine similarity between HumanEval and the synthetic data generated by different methods. Below are the average similarity scores in Table 8.

These results demonstrate that CodeDPO has a comparable or even lower overlap with HumanEval than most other widely used datasets, ensuring the validity and reliability of our evaluation.

<table>
  <tr>
    <th>Dataset</th>
    <th>Avg Similarity Score</th>
  </tr>
  <tr>
    <td>Self-Instruct</td>
    <td>0.169</td>
  </tr>
  <tr>
    <td>Evol-Instruct</td>
    <td>0.131</td>
  </tr>
  <tr>
    <td>OSS-Instruct</td>
    <td>0.105</td>
  </tr>
  <tr>
    <td>CodeDPO</td>
    <td>0.109</td>
  </tr>
</table>

Table 8: Average similarity scores between datasets and HumanEval.

6 Conclusion

We propose CodeDPO, a preference optimization framework for code models that focuses on both correctness and efficiency. CodeDPO iteratively updates the self-validation score, prioritizing solutions based on correctness and efficiency. Our work technically validates the reliability of self-validation to synthesize preference optimization data, eliminating the need for complex resources such as pre-existing tests or powerful generation models. We hope this work opens new avenues for synthesizing data and implementing large-scale preference optimization for code models.

7 Acknowledgments

This research is supported by the National Key R&D Program under Grant No. 2023YFB4503801, the National Natural Science Foundation of China under Grant No. 62192733, 62192730, 62192731, 62072007 and the Major Program (JD) of Hubei Province (No. 2023BAA024).

Limitation

There are several limitations to our work that we aim to address:

Comparison with Advanced RL Techniques such as DeepSeek-R1 Although our study showcases the effectiveness of CodeDPO, it does not thoroughly compare this method with other advanced reinforcement learning (RL) alignment techniques like DeepSeek-R1 (Guo et al., 2025). Techniques such as GRPO in DeepSeek-R1 are typically designed for online RL alignment, requiring substantial training resources, high-quality datasets, and complex reward environments, which can be prohibitively resource-intensive. In contrast, CodeDPO focuses on offline alignment methods, using approximations and necessary simplifications
to achieve optimization objectives. This allows CodeDPO to deliver results comparable to or even matching those of advanced online RL methods, but with notably reduced resource demands. Our self-validation scores also indicate that our ranking methods are robust even with lower-quality source code and tests, demonstrating consistent performance even with less advanced generation models. Given its low resource requirements and consistent performance, CodeDPO is well-suited to a broad range of code generation scenarios. However, a comprehensive evaluation of how CodeDPO stacks up against these sophisticated RL techniques in terms of performance and efficiency is a potential avenue for future research.

Limitations of Current Correctness Evaluation
Firstly, constrained by current mainstream correctness evaluation methods, the test-case-driven functional correctness DPO is still not enough for code model. Current methods for evaluating correctness heavily rely on high-quality test cases or powerful models (e.g., GPT-4) to generate reliable outputs. To address these limitations, our paper introduces a self-validation data generation method that reduces dependency on such resources while maintaining robustness.

Because our method does not require high-quality test cases or strong external models, it is well-suited for scaling to larger datasets and can be applied to a wide range of code models. This scalability provides a foundation for improving correctness and efficiency across diverse code tasks.

Incorporating Readability and Security
Next, beyond correctness and efficiency, incorporating readability and security metrics into our extended CodeDPO framework is a natural extension: Metrics such as comment-to-code ratio, consistent variable naming, and adherence to coding style guides could be integrated into the preference learning process. For instance, LLMs could act as judges to evaluate readability alongside correctness. Techniques like static code analysis and detection of code smell and common vulnerabilities could help identify and penalize insecure patterns during data construction, contributing to safer code generation. We plan to explore these deeper alignment objectives in future work.

References
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.

Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca.

Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128.

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306.

Leonidas Gee, Milan Gritta, Gerasimos Lampouras, and Ignacio Iacobacci. 2024. Code-optimise: Self-generated preference data for correctness and efficiency. CoRR, abs/2406.12502.

GPT-4. 2023. https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo. OpenAI.

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. arXiv preprint arXiv:2306.11644.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. Deepseek-coder:
When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196.

Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2(4):5.

Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M Zhang. 2024. Effibench: Benchmarking the efficiency of automatically generated code. arXiv preprint arXiv:2402.02037.

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.

Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629.

Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319–18345. PMLR.

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024a. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36.

Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. 2024b. Provably mitigating overoptimization in rlhf: Your sit loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405.16436.

Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtiar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173.

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568.

Lawrence Page. 1999. The pagerank citation ranking: Bringing order to the web. Technical report, Technical Report.

Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. 2024. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.

Alexander G Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob R Gardner, Yiming Yang, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, et al. 2024. Learning performance-improving code edits. In The Twelfth International Conference on Learning Representations.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.

Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. 2024. Planning in natural language improves llm search for code generation. arXiv preprint arXiv:2409.03733.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560.

Yuxiang Wei, Federico Cassano, Jiawei Liu, Yifeng Ding, Naman Jain, Zachary Mueller, Harm de Vries, Leandro Von Werra, Arjun Guha, and Lingming Zhang. 2024. Selfcodealign: Self-alignment for code generation. arXiv preprint arXiv:2410.24198.

Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120.

Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. 2024. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675.

Dylan Zhang, Shizhe Diao, Xueyan Zou, and Hao Peng. 2024a. PLUM: preference learning plus test cases yields better code language models. CoRR, abs/2406.06887.
Kechi Zhang, Ge Li, Jia Li, Yihong Dong, and Zhi Jin. 2025. Focused-dpo: Enhancing code generation through focused preference optimization on error-prone points. arXiv preprint arXiv:2502.11475.

Kechi Zhang, Ge Li, Huangzhao Zhang, and Zhi Jin. 2024b. Hirope: Length extrapolation for code models using hierarchical position. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13615–13627.

Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024c. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13643–13658.

Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023a. Self-edit: Fault-aware code editor for code generation. In The 61st Annual Meeting Of The Association For Computational Linguistics.

Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023b. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792.
A  Code Correctness on DS-1000 (RQ1)
For DS-1000, as shown in Table 9, we further evaluate CodeDPO across different libraries on various backbone models.

B  Self Generation and Validation Algorithm (RQ3)
We apply different strategies to construct datasets and evaluate the final model performance in code generation. We conduct experiments on various models and the detailed results are shown in Table 10.

C  Impact of PO Training Strategy (RQ4)
We explore the impact of different preference optimization strategies (DPO, KTO, and SFT) on model performance. For training, the SFT strategy uses the best code solution from our constructed dataset. In our KTO strategy, we replace DPO with KTO in our framework. We conduct experiments on various models and the detailed results are shown in Table 11.

D  Comparison with CodeT in details
To clarify, CodeT (Chen et al., 2022) evaluates ranking based on the product of the number of test cases a piece of code passes and the number of codes that pass a given test case. While this approach provides a more refined evaluation than relying solely on the number of test cases passed, we observed in our experiments that it still falls short when compared to the fine-grained mechanism of our PageRank-based method. Specifically, CodeT’s verification lacks the granularity offered by our PageRank-based method, particularly when sample size is relatively smaller. In our method, the sample size is set at 15, while CodeT generally uses a sample size of 100 (which is too large for us to follow due to resources). We also give the detailed ablation result based on MagiCoder-S-DS-6.7B in Table 12, where “Sort by |C| * |T|” refers to the implementation of CodeT.

E  CodeDPO Dataset Construction algorithm description
In order to make it clear, we give a formal algorithm description of the CodeDPO construction pipeline in Algorithm 1.

The total pipeline includes four steps as shown in Algorithm 1: ① Data Seed Construction with real-world source code: we extract concepts from code repositories and generate initial dataset; ② Correctness Optimization with self-validation score: we generate initial code snippets and test cases, and update the self-validation score in loop; ③ Efficiency Optimization with execution time on credible tests: we record the execution time for each code snippet for ranking; ④ Model Optimization Training: we use the DPO loss for further training.

F  LLM Prompts for CodeDPO dataset construction

We use the following prompts for dataset seed construction and self-validation. During dataset construction, we first use code snippets from a randomly selected subset of the Stack v1 dataset as input and prompt the LLM to generate the concept (LLM Prompt 1). Based on the concept, we then prompt the LLM to generate the task description (LLM Prompt 2). For the validation process, we directly prompt the LLM with the task description to generate code solutions. Additionally, we prompt the LLM to generate only assertion statements as test cases (LLM Prompt 3). Since our chosen generation LLM is efficient and cost-effective, the entire process of data generation and construction takes around 40 hours on a server with 32 CPUs.

LLM Prompt 1 for Concept Generation
Extract key programming concepts from a given code snippet collected from the open source repositories. Present the concepts as a comma separated list.
{Few-shot Examples}
## Example 2
### Snippet
{Input Code}
### Concepts
{need to generate}
<table>
  <tr>
    <th>Model</th>
    <th>plot (155)</th>
    <th>np (220)</th>
    <th>pd (291)</th>
    <th>torch (68)</th>
    <th>scipy (106)</th>
    <th>sk (115)</th>
    <th>tf (45)</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="9">SFT Model</th>
  </tr>
  <tr>
    <td>Magic-CL-7B</td>
    <td>54.8</td>
    <td>16.4</td>
    <td>16.5</td>
    <td>17.6</td>
    <td>23.6</td>
    <td>29.6</td>
    <td><b>33.3</b></td>
    <td>25.5</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>57.4</b></td>
    <td><b>37.3</b></td>
    <td><b>22.7</b></td>
    <td><b>22.1</b></td>
    <td><b>35.8</b></td>
    <td><b>31.3</b></td>
    <td>31.1</td>
    <td><b>34.0</b></td>
  </tr>
  <tr>
    <td>Magic-S-CL-7B</td>
    <td>52.3</td>
    <td>43.2</td>
    <td>30.6</td>
    <td><b>47.1</b></td>
    <td>34.9</td>
    <td><b>46.1</b></td>
    <td><b>44.4</b></td>
    <td>40.7</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>58.7</b></td>
    <td><b>44.5</b></td>
    <td><b>31.3</b></td>
    <td>38.2</td>
    <td><b>40.6</b></td>
    <td>42.6</td>
    <td>33.3</td>
    <td><b>41.3</b></td>
  </tr>
  <tr>
    <td>Magic-DS-6.7B</td>
    <td>55.5</td>
    <td>37.7</td>
    <td>28.2</td>
    <td><b>25.0</b></td>
    <td>34.0</td>
    <td><b>45.2</b></td>
    <td><b>33.3</b></td>
    <td>37.1</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>59.4</b></td>
    <td><b>40.5</b></td>
    <td><b>29.2</b></td>
    <td>23.5</td>
    <td><b>39.6</b></td>
    <td>42.6</td>
    <td>31.1</td>
    <td><b>38.7</b></td>
  </tr>
  <tr>
    <td>Magic-S-DS-6.7B</td>
    <td>53.5</td>
    <td>49.5</td>
    <td>30.6</td>
    <td><b>47.1</b></td>
    <td>35.8</td>
    <td><b>53.0</b></td>
    <td><b>40.0</b></td>
    <td>42.9</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>59.4</b></td>
    <td><b>50.5</b></td>
    <td><b>31.9</b></td>
    <td>39.7</td>
    <td><b>41.5</b></td>
    <td>47.8</td>
    <td>33.3</td>
    <td><b>43.7</b></td>
  </tr>
  <tr>
    <th colspan="9">Base Model</th>
  </tr>
  <tr>
    <td>Phi-2-2.7B</td>
    <td>42.6</td>
    <td>33.6</td>
    <td>15.5</td>
    <td><b>16.2</b></td>
    <td>17.0</td>
    <td>11.3</td>
    <td><b>17.8</b></td>
    <td>23.5</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>49.0</b></td>
    <td><b>33.6</b></td>
    <td><b>16.5</b></td>
    <td>14.7</td>
    <td><b>20.8</b></td>
    <td><b>14.8</b></td>
    <td>13.3</td>
    <td><b>25.3</b></td>
  </tr>
  <tr>
    <td>DSC-1.3B</td>
    <td><b>36.8</b></td>
    <td>19.5</td>
    <td>10.0</td>
    <td>14.7</td>
    <td>10.4</td>
    <td><b>17.4</b></td>
    <td><b>11.1</b></td>
    <td>17.5</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td>34.8</td>
    <td><b>23.6</b></td>
    <td><b>10.7</b></td>
    <td><b>14.7</b></td>
    <td><b>20.8</b></td>
    <td>13.9</td>
    <td>8.9</td>
    <td><b>18.9</b></td>
  </tr>
  <tr>
    <td>DSC-6.7B</td>
    <td>52.3</td>
    <td>35.5</td>
    <td>20.6</td>
    <td><b>19.1</b></td>
    <td>24.5</td>
    <td><b>37.4</b></td>
    <td><b>22.2</b></td>
    <td>31.1</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>56.8</b></td>
    <td><b>36.4</b></td>
    <td><b>21.6</b></td>
    <td>17.6</td>
    <td><b>34.0</b></td>
    <td>34.8</td>
    <td>20.0</td>
    <td><b>32.8</b></td>
  </tr>
  <tr>
    <td>StarCoder2-7B</td>
    <td>54.2</td>
    <td>37.7</td>
    <td>18.6</td>
    <td><b>25.0</b></td>
    <td>31.1</td>
    <td>23.5</td>
    <td><b>35.6</b></td>
    <td>31.4</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>56.8</b></td>
    <td><b>38.2</b></td>
    <td><b>18.9</b></td>
    <td>20.6</td>
    <td><b>39.6</b></td>
    <td><b>25.2</b></td>
    <td>31.1</td>
    <td><b>32.6</b></td>
  </tr>
</table>

Table 9: More results on Pass rate (%) of CodeDPO on DS-1000 across seven libraries using greedy decoding.

<table>
  <tr>
    <th>Model</th>
    <th>HumanEval</th>
    <th>HumanEval+</th>
    <th>MBPP</th>
    <th>MBPP+</th>
  </tr>
  <tr>
    <td>Phi-2-2.7B</td>
    <td>48.78</td>
    <td>46.34</td>
    <td>65.34</td>
    <td>54.49</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>57.32</b></td>
    <td><b>51.83</b></td>
    <td><b>69.05</b></td>
    <td><b>56.88</b></td>
  </tr>
  <tr>
    <td>Filter with all tests</td>
    <td>49.39</td>
    <td>48.17</td>
    <td>69.17</td>
    <td>55.13</td>
  </tr>
  <tr>
    <td>Sort by # of passed tests</td>
    <td>50.60</td>
    <td>49.39</td>
    <td>67.16</td>
    <td>54.88</td>
  </tr>
  <tr>
    <td>Random selection</td>
    <td>22.56</td>
    <td>18.90</td>
    <td>45.11</td>
    <td>36.59</td>
  </tr>
  <tr>
    <td>DeepSeekCoder-1.3B</td>
    <td>31.53</td>
    <td>28.65</td>
    <td>57.40</td>
    <td>48.60</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>42.07</b></td>
    <td><b>38.04</b></td>
    <td><b>61.37</b></td>
    <td><b>53.43</b></td>
  </tr>
  <tr>
    <td>Filter with all tests</td>
    <td>34.75</td>
    <td>29.89</td>
    <td>57.40</td>
    <td>48.80</td>
  </tr>
  <tr>
    <td>Sort by # of passed tests</td>
    <td>37.19</td>
    <td>31.09</td>
    <td>58.39</td>
    <td>50.37</td>
  </tr>
  <tr>
    <td>Random selection</td>
    <td>21.34</td>
    <td>18.29</td>
    <td>48.94</td>
    <td>38.35</td>
  </tr>
</table>

Table 10: More results on ablations of our self validation score on the trained model performance.

<table>
  <tr>
    <th>Model</th>
    <th>HumanEval</th>
    <th>HumanEval+</th>
    <th>MBPP</th>
    <th>MBPP+</th>
  </tr>
  <tr>
    <td>Phi-2-2.7B</td>
    <td>48.78</td>
    <td>46.34</td>
    <td>65.34</td>
    <td>54.49</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>55.49</td>
    <td>49.22</td>
    <td>66.87</td>
    <td>55.76</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>57.32</b></td>
    <td><b>51.83</b></td>
    <td><b>69.05</b></td>
    <td><b>56.88</b></td>
  </tr>
  <tr>
    <td>Our CodeKTO</td>
    <td>54.88</td>
    <td>51.22</td>
    <td>64.91</td>
    <td>53.63</td>
  </tr>
  <tr>
    <td>DeepSeekCoder-1.3B-base</td>
    <td>31.53</td>
    <td>28.65</td>
    <td>57.40</td>
    <td>48.67</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>39.02</td>
    <td>35.36</td>
    <td>59.45</td>
    <td>50.26</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>42.07</b></td>
    <td><b>38.04</b></td>
    <td><b>61.37</b></td>
    <td><b>53.43</b></td>
  </tr>
  <tr>
    <td>Our CodeKTO</td>
    <td>40.85</td>
    <td>35.98</td>
    <td>59.65</td>
    <td>50.13</td>
  </tr>
  <tr>
    <td>DeepSeekCoder-6.7B-base</td>
    <td>47.60</td>
    <td>39.60</td>
    <td>70.20</td>
    <td>56.60</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>56.09</td>
    <td>46.95</td>
    <td>70.18</td>
    <td>56.88</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td><b>59.75</b></td>
    <td><b>51.83</b></td>
    <td><b>72.18</b></td>
    <td><b>60.01</b></td>
  </tr>
  <tr>
    <td>Our CodeKTO</td>
    <td>54.88</td>
    <td>49.39</td>
    <td>71.93</td>
    <td>58.65</td>
  </tr>
</table>

Table 11: More results on the comparison of preference optimization strategies (DPO vs. KTO vs. SFT).
<table>
  <tr>
    <th>Strategy</th>
    <th>HumanEval</th>
    <th>HumanEval+</th>
    <th>MBPP</th>
    <th>MBPP+</th>
  </tr>
  <tr>
    <td>MagiCoder-S-DS-6.7B</td>
    <td>73.17</td>
    <td>68.29</td>
    <td>76.72</td>
    <td>66.67</td>
  </tr>
  <tr>
    <td>Filter with all tests</td>
    <td>76.22</td>
    <td>68.90</td>
    <td>77.44</td>
    <td>67.17</td>
  </tr>
  <tr>
    <td>Sort by # of passed tests</td>
    <td>78.66</td>
    <td>70.73</td>
    <td>79.19</td>
    <td>68.67</td>
  </tr>
  <tr>
    <td>Sort by \( |C| * |T| \)</td>
    <td>76.22</td>
    <td>71.95</td>
    <td>77.94</td>
    <td>68.17</td>
  </tr>
  <tr>
    <td>Our CodeDPO</td>
    <td>83.54</td>
    <td>76.22</td>
    <td>80.70</td>
    <td>70.93</td>
  </tr>
</table>

Table 12: Dataset Construction Ablations For MagiCoder-S-DS-6.7B

LLM Prompt 2 for Task Prompt Generation
Create a set of independent code instructions that are original, different, diverse, and high-quality, where the properties control an instruction’s category, language, concepts, and difficulty.
{Few-shot Examples}
## Example 2
### Property
{Input Concept}
### Instruction
{need to generate}

LLM Prompt 3 for Test Case Generation
Generate only assertion statements based on the following description. Do not generate any other code:
{Instruction}
Generated Assertions:
assert {need to generate}

G Experiments on Challenging Code Efficiency Tasks
To evaluate code efficiency comprehensively, additional experiments are conducted on EffiBench (Huang et al., 2024). Since the absolute values of the results may vary depending on the specific execution environment, the analysis focuses on the relative improvements achieved by CodeDPO.

The results indicate that CodeDPO significantly reduces execution time and memory usage, both in absolute terms and after normalization, while maintaining comparable maximum memory usage. These improvements highlight the effectiveness of CodeDPO in optimizing code for both computational efficiency and resource usage, ensuring applicability to environments where performance and memory constraints are critical.

H Execution Time for Code Efficiency Experiments
We present the average execution time (in seconds) for experiments conducted with the Phi-2-2.7B model. It is important to note that execution times may vary due to differences in computational resources and runtime conditions. To ensure the reliability of our measurements, repeated experiments are conducted in a stable environment, and the averaged statistics are reported below:

These results demonstrate the consistent improvements in execution efficiency achieved through CodeDPO, highlighting its practical benefits in reducing runtime.

I Ablation on Sample Number for code and test generation
The choice of the sample number and temperature, as described in Section 4.2, is guided by practical considerations to balance the diversity of sampled code solutions and test cases. These parameters are selected based on empirical observations and insights from prior work on data generation. To further investigate this, we conduct a series of ablation studies to evaluate the impact of varying sample numbers. Specifically, we tested sample numbers of 5, 15, and 50, with the experimental setup aligned with the design in Section 5.3.1. Table 15 presents the Spearman correlation between the self-validation score and the actual code accuracy on the HumanEval dataset, and then shows the performance of the Phi-2-2.7B model for varying sample numbers, evaluated on both the HumanEval and HumanEval+ benchmarks. Similar trends are observed for other models. The results suggest that using sample_num=15 achieves a favorable trade-off between diversity and computational feasibility. While larger sample numbers provide marginal gains, they come with increased
Algorithm 1 CodeDPO Dataset Construction Pipeline

1: procedure CodeDPO(model, instruction, max_iterations)
2:   Seed Construction:
3:     Extract key programming concepts from source code repositories
4:     Generate code generation prompts and corresponding test cases
5:     Generate initial dataset (instruction, solutions, testcases)
6:   Initialization:
7:     Generate initial code snippets \( C = \{c_1, c_2, ..., c_n\} \) from the instruction
8:     Generate test cases \( T = \{t_1, t_2, ..., t_m\} \) corresponding to the instruction
9:     Initialize self-validation scores for code snippets and test cases: Score(\(c_i\)) \( \leftarrow 1 \), Score(\(t_j\)) \( \leftarrow 1 \)
10:    Set damping factor \( d \leftarrow 0.85 \)
11:    \( i \leftarrow 0 \)
12:   Self-Validation Loop:
13:    while \( i < \text{max\_iterations} \) do
14:      for each \( c_i \in C \) do
15:        Execute \( c_i \) on test cases \( T \)
16:        for each \( t_j \in T \) do
17:          if \( c_i \) passes \( t_j \) then
18:            Update Score(\(c_i\)) using Equation (1)
19:            Update Score(\(t_j\)) using Equation (2)
20:            Execution Time Optimization:
21:              Record execution time for \( c_i \)
22:              if \( c_i \) fails \( t_j \) then
23:                Set execution time to max penalty to penalize \( c_i \)
24:              end if
25:          end if
26:        end for
27:      end for
28:      \( i \leftarrow i + 1 \)
29:    end while
30:   Final Dataset Collection:
31:   Correctness Optimization:
32:     Select top-ranked code \( c_{top} \) and low-ranked code \( c_{low} \) for each instruction
33:     Store as dataset entries (\(instruction, c_{top}, c_{low}\))
34:   Execution Time Optimization:
35:     Select fastest code \( c_{fast} \) and slowest code \( c_{slow} \) for each instruction
36:     Store as dataset entries (\(instruction, c_{fast}, c_{slow}\))
37:   return final dataset entries
38: end procedure

<table>
  <tr>
    <th>Model</th>
    <th>Total Execution Time</th>
    <th>Normalized Execution Time</th>
  </tr>
  <tr>
    <td>MagiCoder-S-DS-6.7B</td>
    <td>0.29</td>
    <td>2.37</td>
  </tr>
  <tr>
    <td>After CodeDPO</td>
    <td>0.21</td>
    <td>1.58</td>
  </tr>
</table>

<table>
  <tr>
    <th>Model</th>
    <th>Total Max Memory Usage</th>
    <th>Normalized Max Memory Usage</th>
  </tr>
  <tr>
    <td>MagiCoder-S-DS-6.7B</td>
    <td>24.71</td>
    <td>1</td>
  </tr>
  <tr>
    <td>After CodeDPO</td>
    <td>23.48</td>
    <td>1</td>
  </tr>
</table>

<table>
  <tr>
    <th>Model</th>
    <th>Total Memory Usage</th>
    <th>Normalized Memory Usage</th>
  </tr>
  <tr>
    <td>MagiCoder-S-DS-6.7B</td>
    <td>4.57</td>
    <td>2.36</td>
  </tr>
  <tr>
    <td>After CodeDPO</td>
    <td>3.90</td>
    <td>1.93</td>
  </tr>
</table>

Table 13: Performance comparison on EffiBench for execution time and memory usage.

computational costs.
<table>
  <tr>
    <th>Benchmark</th>
    <th>Before CodeDPO (s)</th>
    <th>After CodeDPO (s)</th>
    <th>Average Speedup</th>
  </tr>
  <tr>
    <td>HumanEval+</td>
    <td>0.250</td>
    <td>0.172</td>
    <td>1.45x</td>
  </tr>
  <tr>
    <td>MBPP+</td>
    <td>0.189</td>
    <td>0.137</td>
    <td>1.38x</td>
  </tr>
</table>

Table 14: Average execution time and speedup with CodeDPO.

<table>
  <tr>
    <th>Sample Number (\( n \))</th>
    <th>Spearman Correlation</th>
    <th>HumanEval (%)</th>
    <th>HumanEval+ (%)</th>
  </tr>
  <tr>
    <td>5</td>
    <td>0.7425</td>
    <td>54.88</td>
    <td>49.39</td>
  </tr>
  <tr>
    <td>15</td>
    <td>0.8598</td>
    <td>57.32</td>
    <td>51.83</td>
  </tr>
  <tr>
    <td>50</td>
    <td>0.8613</td>
    <td>57.90</td>
    <td>51.83</td>
  </tr>
</table>

Table 15: Spearman correlation and performance of Phi-2-2.7B for different sample numbers.

J Discussion

J.1 Comparison of Dataset Statistics

Since some baselines have not released their datasets, we rely on statistics reported in their respective papers for comparison. Below is a summary of dataset sizes and the number of unique questions, as both metrics are important—greater diversity in unique questions generally leads to higher dataset quality.

<table>
  <tr>
    <th>Method</th>
    <th>Total Samples</th>
    <th>Unique Questions</th>
  </tr>
  <tr>
    <td>CodeDPO</td>
    <td>114k</td>
    <td>114k</td>
  </tr>
  <tr>
    <td>PLUM</td>
    <td>Up to 120k</td>
    <td>Up to 1,500</td>
  </tr>
  <tr>
    <td>Code-Optimise</td>
    <td>\(~100k\) (extended in our reproduction)</td>
    <td>Up to 584</td>
  </tr>
</table>

Table 16: Comparison of dataset sizes and unique questions across methods.

For SFT datasets, OSS-Instruct often combines multiple data sources. For example, models like MagiCoder-S-DS-6.7B and MagiCoder-S-CL-7B are trained using:

<table>
  <tr>
    <th>SFT Dataset</th>
    <th>Samples</th>
  </tr>
  <tr>
    <td>Magicoder-OSS-Instruct</td>
    <td>\(~75k\)</td>
  </tr>
  <tr>
    <td>Magicoder-Evol-Instruct</td>
    <td>\(~110k\)</td>
  </tr>
  <tr>
    <td>Combined</td>
    <td>Up to 185k</td>
  </tr>
</table>

Table 17: Supervised fine-tuning dataset statistics.

Based on comparisons with other related works, the dataset sizes of CodeDPO appear to be of the same order of magnitude. CodeDPO provides a significantly higher diversity in unique questions compared to baselines like PLUM and Code-Optimise, which heavily reuse prompts and have limited diversity despite similar sample sizes. This diversity ensures a more robust preference optimization process, which is a key advantage over existing approaches.

J.2 Implementation of the Self-Validation Scores

J.2.1 Python Implementation of the Self-Validation Scores

To enhance the understanding of the proposed algorithm, we provide a Python implementation illustrating the calculation process for the case in Figure 4 (specifically, Step 2 in the figure). The code demonstrates the iterative calculation of self-validation scores using a simplified example.

J.2.2 Handling Weak Test Cases

Our designed algorithm is robust. The self-validation scores can reflect the confidence of each code solutions and test cases through the iterative process. Notably, even in the presence of weak test cases (such as assert True), our method handles them robustly. We have carefully considered the impact of weak test cases in our design. We address this issue from two perspectives: ① Natural Suppression of Weak Test Cases in Ranking: Weak test cases are those that almost all code solutions pass. While they contribute to the overall scores of all code solutions, they do not affect the relative differences between code solutions in the ranking process. Since the ranking is based on score differences, weak test cases naturally have minimal impact on the ranking outcomes. ② Filtering Identical or Close Scores: Weak test cases can lead to highly similar scores for multiple code solutions after repeated score updates, diminishing the ability to differentiate between them. To address this, as described in Section 3.4, we implement a filtering mechanism that excludes samples with identical or near-identical ranking scores. This ensures that the influence of weak test cases is mitigated in the final dataset.
Python Implementation of the Self-Validation Scores

import numpy as np

# Example task-solution-test matrix
task_sol_test_matrix = [
    [[1, 1, 0],   # Code1: Test1, Test2
     [1, 0, 0],   # Code2: Test1
     [0, 0, 1]]   # Code3: Test3
]
task_sol_test_matrix = np.array(task_sol_test_matrix)

# Initialize solution and test scores (score=1)
sol_score = np.array([[1, 1, 1]])
test_score = np.array([[1, 1, 1]])

# Define iterative scoring function
def iter_step_page_rank(solution_scores_t_1, \
    test_scores_t_1, beta):
    test_scores_t = test_scores_t_1 * (1 - beta) + \
        np.einsum("PCT,PC->PT", task_sol_test_matrix, \
            solution_scores_t_1) * beta
    solution_scores_t = solution_scores_t_1 * (1 - beta) + \
        np.einsum("PCT,PT->PC", task_sol_test_matrix, \
            test_scores_t) * beta
    return solution_scores_t, test_scores_t

# Perform 2 iterations with beta = 0.5
for i in range(2):
    sol_score, test_score = \
        iter_step_page_rank(sol_score, test_score, 0.5)

# Output final scores
print(sol_score, test_score)

Figure 4: Python Implementation of the Self-Validation Scores in Figure 2.

For example, assume we have 15 code solutions and 15 test cases generated by the model. ① If a weak test case, such as assert True, is passed by all 15 code samples, its score during each update step (as computed by Equation 1) will contribute equally to the scores of all code solutions. As a result, it does not alter the relative ranking of the code solutions. ② If all 15 test cases are similarly weak, the scores of the code solutions will converge to identical or near-identical values after several updates. To mitigate this, we apply a post-processing step (Section 3.4) to filter out such cases, ensuring the integrity of the final rankings. By addressing weak test cases through these mechanisms, our algorithm achieves robustness and maintains the reliability of its outputs, even in challenging scenarios.

Reviewer paper summaries: This paper aims to improve the code generation performance, both correctness and efficiency. The paper proposed a self-generation-and-validation mechanism, which can generate preference-optimized data for code. Based on the assumption that code that passes more tests is more likely to be correct, the method utilizes the PageRank-inspired algorithm to rank the sampled codes and utilize the ranked codes to optimize the Code LLM. Experiments on HumanEval, MBPP, and DS-1000 demonstrate the effectiveness of the proposed method.

This work proposes a data collection approach for DPO-updating code generation models in two aspects: correctness and efficiency. The experiments and discussions look in good shape.
Summary of strengths: 1. The paper is clearly written and well organized.
2. The paper provides an effective preference for optimizing data for the code domain.
3. The experiments in this paper are comprehensive. This paper includes thorough experiments, such as the main experiments, ablation studies, the impact of PO training strategy, and data scaling law.

The page-rank inspired algorithm looks interesting, although I am not sure if it's devised in this work or referenced from some other previous work.

The writing is OK and overall, a relatively easy to follow paper.
Summary of weaknesses: 1. The motivation is a little confusing. The paper claims that "However, these approaches struggle to fully address correctness and efficiency, facing limitations such as restricted data diversity, an imbalance between positive and negative samples, …". However, this paper does not solve the two problems and also faces these problems. It seems that the main difference between this paper and previous work is that this paper focuses on both correctness and efficiency.
2. This paper only compares time optimization performance on HumanEval and MBPP. Datasets that can better reflect the time optimization, such as LiveCodebench, have not been tested.
3. Some analysis experiments (efficiency, ablation study, data scaling experiment, …) are constructed based on the Base Model (Deepseek-Coder-1.3b). However, the DPO is highly dependent on SFT, and the results obtained by directly conducting experiments on the base model have a large bias.

The writing logics needs improvement. For instance, in the intro part at the beginning of Section 3, the authors suddenly mention code-1, test-1 and test-2 without any explanation about what these names mean. I have to jump around to understand what they are about. Not a pleasant way to read a paper.

What's changed during the T iterations in code/test case scoring? In other words, why iterations are needed to compute the scores?
Comments, suggestions and typos: 1. Can you provide a case study of the proposed dataset?
2. “Sort by number of passed tests” is not used in CodeT, which proposed a more effective method, please modify your presentation. Can you compare the final scores of these methods? That is, use these methods to construct the dataset and train the model to get the final score.

I'm willing to improve my score if you can clarify the weaknesses and questions.

Please see in the weaknesses. No typos found
Overall assessment scores: 3

3