CodeDPO: Aligning Code Models with Self Generated and Verified Source Code

## Abstract

Abstract Code generation models have shown significant potential for programming tasks.
However, existing training methods like supervised fine-tuning face key limitations: they do not effectively teach models to prioritize correct over incorrect solutions in ambiguous situations, nor do they effectively optimize the runtime efficiency of the generated code.
To address these challenges, we propose CodeDPO, a framework that integrates preference learning into code generation to improve two key code preference factors: code correctness and efficiency.
CodeDPO employs a novel dataset construction method, utilizing a self-generation-and-validation mechanism that simultaneously generates and evaluates code and test cases.
The underlying assumption is that test cases executable by multiple code snippets provide more reliable validation, and code that passes more tests is more likely to be correct.
Through this self-validation process, our PageRank-inspired algorithm iteratively updates the ranking score of each code snippet, ultimately creating a code preference optimization dataset based on correctness and efficiency.
CodeDPO is flexible and scalable, generating diverse preference optimization data without depending on external resources.
Through comprehensive evaluations of five widely used benchmarks, CodeDPO demonstrates significant improvements in correctness and efficiency compared to existing methods.
Our experiments prove that CodeDPO enhances the capabilities of LLMs in code generation and provides a robust foundation for conducting code preference optimization in more complex and challenging real-world scenarios.

## 1 Introduction

In recent years, code generation models have gained significant attention for their potential to automate software development. Models such as GPT-4 (GPT-4, ), Claude, and open-source alternatives like Phi (Gunasekar et al., ; Abdin et al., ), DeepSeekCoder (Guo et al., ), and StarCoder (Li et al., ; Lozhkov et al., ) have demonstrated the capability of LLMs to handle complex code generation tasks. However, one of the ongoing challenges lies in boosting the correctness and efficiency of the generated code.

To improve code generation models, a common approach is supervised fine-tuning (SFT) (Zhang et al., ), where models are trained on pairs of instructions and correct code snippets.
While SFT improves the overall quality of the generated code, it falls short in teaching models to consistently prefer correct solutions over incorrect ones (Hong et al., ).
Figure 1 illustrates the likelihood of generating code with varying correctness and efficiency during SFT training. When we adopt SFT training on those correct solutions, as the likelihood of preferred outputs increases, the probability of generating undesirable outputs also rises, leading to performance saturation.

Figure: Figure 1: Log probabilities for code with varying correctness and efficiency during Phi-2-2.7B model training on our constructed dataset. The traditional SFT strategy struggles to teach models to prefer correct solutions over incorrect or slow ones. In contrast, our CodeDPO approach effectively optimizes for both correctness and efficiency.
Refer to caption: extracted/5908942/motivation_v2.png

To address these limitations, recent research has turned to direct preference optimization (DPO) (Rafailov et al., ), a method designed for alignment based on pairwise preference data.
DPO allows models to rank different outputs and choose preferable solutions (e.g., more factual or helpful).
While DPO has shown success in reasoning tasks like mathematics (Lai et al., ; Wu et al., ), its application in code generation remains under-explored. Unlike natural language tasks, code generation requires objective metrics, such as executability, which poses challenges for directly applying DPO.
In this paper, we first define code preference based on two key factors—Correctness and efficiency. Correctness refers to whether the code solves the problem accurately, while efficiency measures how quickly the code runs.
Existing methods (Gee et al., ; Zhang et al., ) rely heavily on high-quality test cases to assess correctness. However, these approaches struggle to fully address correctness and efficiency, facing limitations such as restricted data diversity, an imbalance between positive and negative samples, and insufficient focus on optimizing code efficiency.

In this paper, we introduce CodeDPO, a novel framework that integrates preference learning into code model training to optimize both correctness and efficiency.
CodeDPO constructs the dataset from real-world code repositories using a self-generation-and-validation mechanism, where code and test cases are simultaneously generated and evaluated.
We assume that tests executable by more code snippets are more reliable, and code that passes more tests is more likely to be correct. To implement this, CodeDPO uses a mutual verification process: each receives an initial self-validation score, which is iteratively updated using a PageRank-inspired (Page, ) algorithm. This algorithm adjusts the credibility of each code snippet and tests by considering their relations in cross-verification, prioritizing solutions based on correctness and efficiency. The final preference-optimized dataset is then used to train various code models using the DPO learning algorithm.
A key advantage of CodeDPO is its flexibility. Unlike existing methods that rely on high-quality test cases or powerful models to generate them, CodeDPO does not depend on external resources.
Its self-generation and validation mechanism supports the scalable creation of diverse and robust preference optimization data. This allows our framework to optimize code models for real-world scenarios where high-quality test data may be sparse.

CodeDPO can serve as a crucial step in the post-training phase of code models. We conduct experiments on five popular benchmarks such as HumanEval (Chen et al., ), HumanEval+ (Liu et al., ), MBPP (Austin et al., ), MBPP+, and DS-1000 (Lai et al., ) with CodeDPO, demonstrating its superiority over existing methods.
Notably, we develop a top-performing 6.7B model by building on an existing SFT strategy (Guo et al., ; Wei et al., ) and further enhancing it with our CodeDPO approach, achieving an impressive 83.5% pass rate on HumanEval.
We also conduct ablation studies to investigate the impact of our self-generation-and-validation mechanism and other preference optimization settings.
Our findings confirm that CodeDPO enhances the code generation capabilities of LLMs while providing a solid foundation for further research into optimizing code generation for both correctness and efficiency.

## 2 Related Work

### 2.1 Large Language Models for Code

Code generation, which automates writing source code from natural language (NL) descriptions, is gaining significant attention. LLMs have shown strong capabilities in this area due to their large-scale training on diverse datasets, such as OpenAI’s GPT-4 (GPT-4, ), StarCoder (Li et al., ; Lozhkov et al., ), Code Llama (Rozière et al., ), and DeepSeekCoder (Guo et al., ).
These models are often fine-tuned further, such as through instruction-supervised fine-tuning (SFT), to maximize their coding potential.
Since gathering high-quality data is difficult, researchers use self-instruct methods to generate synthetic instruction data from powerful models like GPT-4 (Wang et al., ; Taori et al., ; Chaudhary, ). Evol-Instruct (Luo et al., ) uses more complex prompts for better data generation. OSS-instruct (Wei et al., ) allows LLMs to get inspired from real-world code snippets for better quality in coding tasks.
While these SFT methods boost code quality, it does not fully train models to prefer correct solutions over incorrect ones (Hong et al., ).
Updating training strategies is critical for improving these code models to handle various coding tasks.

### 2.2 Preference Optimization for Code Models

Preference optimization techniques have recently been used to help LLMs prefer better outputs over weaker ones in various natural language tasks (Rafailov et al., ). While DPO has proven effective in reasoning tasks like mathematics (Lai et al., ), its use in code generation is still under-explored.
Code generation requires objective measures of correctness and efficiency, unlike natural language tasks where preferences are often more subjective.
Some works have simply explored PO.
Code-Optimize (Gee et al., ) builds its dataset from the MBPP-train subset, which includes just 384 problems.
PLUM uses GPT-4 to generate tests, which are then used to validate and rank code solutions. PLUM currently achieves state-of-the-art performance in preference optimization for code models.
However, PLUM (Zhang et al., ) faces some limitations. It uses a limited number of tests to validate the code, and the resulting dataset is imbalanced due to its validation method, which means it can only use KTO (Ethayarajh et al., ) for training. Additionally, PLUM does not consider the code efficiency.
This paper introduces CodeDPO, which does not rely on external resources or powerful models for dataset generation. Our approach uses a self-generation and validation mechanism to create balanced preference pairs, aiming to optimize both correctness and efficiency.

## 3 CodeDPO: Self-Verified Performance Optimization Code Generation Framework

Figure: Figure 2: Our CodeDPO involves four steps: ❶ Data Seed Construction with real-world source code; ❷ Correctness Optimization with self-validation score (in this figure we set $T$ to 2 and $d$ to 0.5. For simplicity, the final score in the figure is rounded to one decimal place); ❸ Efficiency Optimization with execution time on credible tests; ❹ Model Optimization Training.
Refer to caption: extracted/5908942/largefigure.png

CodeDPO is designed to integrate preference learning into code generation models, improving both the correctness and efficiency of the generated code.
As shown in Figure 2, our method involves four key steps:
❶ Data Seed Construction with real-world source code: We first collect a data seed from open-source code repositories and generate programming task prompts.
❷ Correctness Optimization with self-validation score:
We generate code and tests simultaneously, using a self-generation-and-validation loop to build a dataset for correctness optimization. The self-validation score is iteratively updated based on whether the generated code passes the tests. We assume that tests executable by multiple code snippets are more reliable, and code that passes more tests is more likely to be correct. As illustrated in the figure, after two iterations, the score of code-1 changes from 1 to 1.75 to 2.6875 ($\sim$2.7 in the figure), as it passes more reliable tests and receives higher scores with each update, indicating a greater likelihood of correctness.
❸ Efficiency Optimization with execution time: We measure execution time on selected credible test sets to build the dataset for efficiency optimization. In the figure, we select test-1 and test-2 as the credible test set to measure the execution time of each code snippet.
❹ Model Optimization Training: We collect the dataset from the previous two stages and use Direct Preference Optimization (DPO) to train various code models.

### 3.1 Data Seed Construction

The data seed construction for CodeDPO is the first step for initiating the preference learning process to generate programming task prompts. We adopt a method inspired by OSS-instruct (Wei et al., ), which extracts key programming concepts from open-source code repositories. These concepts serve as the foundation for generating various programming task prompts.
For example, a code snippet that performs sorting operations might highlight concepts such as sorting algorithms, data structure traversal, and time complexity. From these concepts, we generate code generation prompts. The data seed thus allows the model to explore a wide range of scenarios.

### 3.2 Correctness Optimization with Self-Generation and Validation

Central to CodeDPO is the self-generation-and-validation loop, which enables the model to iteratively update the code correctness rank through mutual validation of code and test cases (Chen et al., ; ; Zhang et al., ).
The process begins by generating multiple candidate code snippets based on a prompt. Simultaneously, corresponding test cases are generated to evaluate these snippets. The validation loop follows these steps:
1. Code Generation: Given an instruction, the model generates a set of candidate code snippets $C=\{c_{1},c_{2},...,c_{n}\}$.
2. Test Case Generation: Test cases $T=\{t_{1},t_{2},...,t_{m}\}$ are generated in parallel to validate the candidate snippets.
3. Validation Process: Each code snippet is executed against the generated test cases. The validation outcomes are used to update the self-validation scores for both the code snippets and the test cases.

##### Ranking Code Snippets and Test Cases Using Self-Validation Scores

To rank both code snippets and tests, we employ a PageRank-inspired (Page, ) iterative algorithm. Initially, each code and test is assigned a self-validation score of 1. Over a fixed number of iterations $T=10$, these scores are updated based on the performance of the snippets and test cases during validation.

The self-validation score for code snippets and test cases is updated using the following formulas:

$$ $\text{Score}_{t}(c_{i})=(1-d)\times\text{Score}_{t-1}(c_{i})+d\times\sum_{t_{j }}\text{Score}_{t-1}(t_{j})\times\text{Link}(t_{j},c_{i})$ (1) $$

$$ $\text{Score}_{t}(t_{j})=(1-d)\times\text{Score}_{t-1}(t_{j})+d\times\sum_{c_{i }}\text{Score}_{t-1}(c_{i})\times\text{Link}(c_{i},t_{j})$ (2) $$

Where $d$ is the damping factor, and $\text{Link}(t_{j},c_{i})$ indicates whether a code snippet $c_{i}$ passes the test case $t_{j}$. This iterative process is repeated until convergence. After $T$ iterations, the final rankings reflect the quality of the code snippets and test cases based on the correctness.

### 3.3 Execution Efficiency Optimization

In addition to ensuring correctness, CodeDPO integrates execution efficiency optimization to ensure that our approach generates functionally correct and efficient code. During the self-validation loop, the execution time for each code snippet is recorded. However, not all test cases accurately reflect the efficiency of the code. To address this, we use the top-performing code from the correctness optimization phase as a reference, assuming the test cases it passes are credible. The total execution time for each code snippet is then measured based on the subset of these credible tests.
Code snippets that pass these credible test cases with lower execution times are assigned higher efficiency scores. Finally, we collect both fast and slow code snippets as part of the training dataset for execution efficiency optimization, which is used for further training, encouraging the model to prioritize solutions that are accurate and optimized for speed during code generation.

### 3.4 Final Dataset and Model Optimization

The final dataset is built from the previous two optimization dataset construction stages, accounting for correctness and execution time. This dual-optimization approach ensures that our CodeDPO dataset can train models to generate not only accurate code but also efficient solutions, addressing both functional and performance challenges in real-world coding tasks. We filter out samples whose ranking scores are identical or too close.
The final dataset consists of 93k correctness optimization samples and 21k efficiency optimization samples. Each sample includes a unique code problem prompt with a preferred and a rejected solution. We carefully avoid overlap between the data seeds of correctness and efficiency samples, ensuring that the constructed dataset captures various problems and instructions. In the subsequent training, we combine both correctness and efficiency data to optimize the model in both aspects simultaneously.

In our experiments, we apply Direct Preference Optimization (DPO) (Rafailov et al., ) across various code models to facilitate optimization learning.
CodeDPO is plug-and-play and can be applied to nearly all code models, regardless of their type or training stage. We utilize both base models and SFT models as the backbone for further training.
Our goal is to demonstrate that CodeDPO has the potential to enhance code models at different stages of their training, even for models that have undergone extensive training or fine-tuning.
The setup details are provided in Section 4.2.

## 4 Experiment Setup

In this study, we aim to investigate the following research questions:

RQ1: Does CodeDPO improve the correctness of generated code compared to baseline models on standard benchmarks? How does CodeDPO compare with other code preference optimization baselines?
We evaluate the pass rate of CodeDPO on benchmarks such as HumanEval (Chen et al., ), HumanEval+ (Liu et al., ), MBPP (Austin et al., ), MBPP+, and DS-1000 (Lai et al., ).
We further compare the performance of CodeDPO with other baselines (Gee et al., ; Zhang et al., ) that also utilize preference optimization techniques.

RQ2: Does CodeDPO enhance the execution efficiency of generated code?
We measure the execution efficiency of code generated by CodeDPO compared to baseline models.

RQ3: What is the impact of the self-generation-and-validation algorithm on CodeDPO’s performance?
We perform ablation studies by removing or modifying the self-generation-and-validation mechanism to assess its contribution to the overall performance.

RQ4: How does the choice of preference optimization strategy affect CodeDPO’s effectiveness?
We evaluate different preference optimization strategies, including Direct Preference Optimization (DPO), Kahneman-Tversky Optimization (KTO) (Ethayarajh et al., ), and Supervised Fine-Tuning (SFT), to understand their impact on the model’s performance.

RQ5: How does data scaling influence the performance of CodeDPO?
We investigate data scaling for CodeDPO by varying the amount of training data to show how data size affects its ability.

### 4.1 Backbone LLMs

We evaluate several widely used LLMs in the code generation domain for our experiments, covering both base models and SFT models at different training stages.
For base models, we apply CodeDPO to Phi-2 (2.7B) (Gunasekar et al., ), DeepSeekCoder-base (1.3B, 6.7B) (Guo et al., ), and StarCoder2-base (7B) (Lozhkov et al., ).
Additionally, we evaluate our method on several fine-tuned SFT models (Wei et al., ), including Magicoder-CL-7B, Magicoder-S-CL-7B, Magicoder-DS-6.7B, and Magicoder-S-DS-6.7B, which are fine-tuned based on CodeLlama-7B and DeepSeekCoder-base-6.7B using state-of-the-art SFT techniques.

While applying the PO phase after SFT is generally recommended (Rafailov et al., ), we extend our evaluation to base models as they can generate more diverse code snippets and offer more significant potential for improvement (Wang et al., ).
Since CodeDPO’s optimization focuses on objective metrics such as code correctness and efficiency, it contrasts with other natural language tasks where preferences are often more subjective. This does not require our backbone model to have a strong ability to follow subjective instructions, allowing CodeDPO to be directly applied to base models.
We choose all these popular models as the backbone of our experiments to optimize correctness and execution efficiency.

### 4.2 Training and Inference Settings

For dataset construction, in order to balance generation speed and cost efficiency, we use DeepSeekCoder-v2 as the data generation model.
For each problem prompt, we sample 15 code solutions and test cases from this model with $temperature=1.5$.
To construct the preference optimization dataset, we set $T$ to 10 and $d$ to 0.85 for the self-validation score. Our practice shows that this parameter configuration quickly yields a stable ranking score. In this paper, we focus on constructing a Python dataset.

For training, we train each code model for 10 epochs and select the best-performing model based on the lowest validation loss. To enhance the stability and robustness of the training process, we employ RPO (Pang et al., ; Liu et al., ) format loss, which essentially consists of a weighted SFT loss on the chosen preferences together with the original DPO loss. We utilize a learning rate of 5e-6 with a linear scheduler and warm-up. The maximum sequence length is set to 2048 tokens.

For inference, we use greedy search decoding for code generation. All evaluations use the framework from bigcode-evaluation-harness (Ben Allal et al., ). We use 16 A100 GPUs for all experiments.

## 5 Results and Analyses

### 5.1 Code Correctness (RQ1)

To answer RQ1, we evaluate the model performance on five widely-used code generation benchmarks: HumanEval, HumanEval+, MBPP, MBPP+, and DS-1000.
Following the standard training process (base model $\rightarrow$ SFT $\rightarrow$ DPO), we first record the performance of the base model, SFT model, and DPO-aligned model on DeepSeekCoder-6.7B, as shown in Table 1.
With the enhancement of our CodeDPO, the final model achieves an 83.5% pass rate on HumanEval. Notably, even after high-quality SFT training, CodeDPO still achieves additional performance improvements. CodeDPO plays a crucial role in the post-training phase of code models, significantly boosting overall performance.

**Table 1: Pass rates (%) of code models at different stages on HumanEval(+) and MBPP(+). We track the performance of the base model, SFT model, and DPO-aligned model on DeepSeekCoder-6.7B. Our CodeDPO shows additional improvements, even after high-quality SFT training.**
| Model | HumanEval | HumanEval+ | MBPP | MBPP+ |
| --- | --- | --- | --- | --- |
| DeepSeekCoder-6.7B-base | 47.60 | 39.60 | 70.20 | 56.60 |
| + SFT (with MagiCoder-OSS-instruct) | 73.17 | 68.29 | 76.72 | 66.67 |
| + SFT + Our CodeDPO | 83.54 | 76.22 | 80.70 | 70.93 |

We further evaluate the performance of CodeDPO alongside baselines(^1^11The baselines have not yet published their datasets. We reproduced the Code-Optimize experiment based on the reported settings. For PLUM, we report results from their paper using models identical to ours, which is why some models do not include PLUM results.) on a wide range of models, including four base models and four SFT models.
As shown in Table 2, CodeDPO achieves the best performance on both HumanEval(+) and MBPP(+). Compared to the baseline models in the first row of each block, we observe that CodeDPO delivers significant improvements across all models, regardless of their initial performance. Notably, we achieve a 36.1% relative improvement on StarCoder2-7B. Additionally, CodeDPO shows remarkable gains on the more challenging HumanEval+, demonstrating its robustness under stricter evaluation.
Thanks to CodeDPO’s data construction strategy, we can build a reliable preference dataset that helps the model favour high-quality outputs, leading to more robust and reliable code generation.

For DS-1000, as shown in Table 3, we further evaluate CodeDPO across different libraries. We did not incorporate prior knowledge of specific Python libraries in our data construction. Thanks to our approach’s flexibility, we can create a wide variety of programming problems and corresponding code pairs. While we observe slight performance drops in the Torch and TensorFlow settings, this may be due to the relatively low percentage of these libraries in our dataset construction. However, CodeDPO demonstrates overall performance improvements over their respective baselines.
It is important to note that DS-1000 differs from benchmarks like HumanEval and MBPP in data format and the coding skills it assesses. The dataset generation process for DS-1000 ensures that it is excluded from nearly all models’ training sets, making the improvements we observe on DS-1000 reliable. These results show that CodeDPO does more than just adapt to standard coding benchmarks like HumanEval. It proves that CodeDPO can enhance the model’s coding capabilities in more complex and diverse scenarios.

**Table 2: Pass rate (%) of CodeDPO compared to baseline models on HumanEval and MBPP.**
| Model | HumanEval | HumanEval+ | MBPP | MBPP+ |
| --- | --- | --- | --- | --- |
| SFT Model |  |  |  |  |
| MagiCoder-CL-7B | 51.21 | 48.78 | 65.60 | 55.82 |
| Our CodeDPO | 60.36 | 54.87 | 70.93 | 59.15 |
| Code-Optimise | 48.78 | 46.95 | 67.17 | 57.14 |
| MagiCoder-S-CL-7B | 67.07 | 61.59 | 69.58 | 60.58 |
| Our CodeDPO | 74.39 | 71.95 | 71.43 | 61.40 |
| Code-Optimise | 64.63 | 54.88 | 69.42 | 60.15 |
| PLUM | 73.80 | 69.50 | 71.40 | 60.80 |
| MagiCoder-DS-6.7B | 57.93 | 53.66 | 75.93 | 64.02 |
| Our CodeDPO | 67.07 | 62.80 | 81.70 | 68.92 |
| Code-Optimise | 57.93 | 51.83 | 76.19 | 64.91 |
| PLUM | 71.30 | 65.90 | 79.60 | 66.70 |
| MagiCoder-S-DS-6.7B | 73.17 | 68.29 | 76.72 | 66.67 |
| Our CodeDPO | 83.54 | 76.22 | 80.70 | 70.93 |
| Code-Optimise | 68.90 | 64.63 | 78.20 | 67.92 |
| PLUM | 80.50 | 73.80 | 80.40 | 69.30 |
| Base Model |  |  |  |  |
| Phi-2-2.7B | 48.78 | 46.34 | 65.34 | 54.49 |
| Our CodeDPO | 57.32 | 51.83 | 69.05 | 56.88 |
| Code-Optimise | 49.39 | 47.56 | 67.42 | 55.80 |
| DeepSeekCoder-1.3B | 31.53 | 28.65 | 57.40 | 48.67 |
| Our CodeDPO | 42.07 | 38.04 | 61.37 | 53.43 |
| Code-Optimise | 34.15 | 30.49 | 59.15 | 49.87 |
| DeepSeekCoder-6.7B | 47.60 | 39.60 | 70.20 | 56.60 |
| Our CodeDPO | 59.75 | 51.83 | 72.18 | 60.01 |
| Code-Optimise | 47.56 | 37.20 | 72.18 | 57.64 |
| PLUM | 56.70 | 48.80 | 72.90 | 58.90 |
| StarCoder2-7B | 35.40 | 29.90 | 54.40 | 45.60 |
| Our CodeDPO | 48.17 | 34.15 | 58.40 | 49.37 |
| Code-Optimise | 32.32 | 28.05 | 58.90 | 47.89 |
| PLUM | 46.30 | 39.60 | 60.40 | 49.10 |

**Table 3: Pass rate (%) of CodeDPO on DS-1000 across seven libraries using greedy decoding.**
| Model | plot (155) | np (220) | pd (291) | torch (68) | scipy (106) | sk (115) | tf (45) | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SFT Model |  |  |  |  |  |  |  |  |
| Magic-CL-7B | 54.8 | 16.4 | 16.5 | 17.6 | 23.6 | 29.6 | 33.3 | 25.5 |
| Our CodeDPO | 57.4 | 37.3 | 22.7 | 22.1 | 35.8 | 31.3 | 31.1 | 34.0 |
| Magic-S-CL-7B | 52.3 | 43.2 | 30.6 | 47.1 | 34.9 | 46.1 | 44.4 | 40.7 |
| Our CodeDPO | 58.7 | 44.5 | 31.3 | 38.2 | 40.6 | 42.6 | 33.3 | 41.3 |
| Magic-DS-6.7B | 55.5 | 37.7 | 28.2 | 25.0 | 34.0 | 45.2 | 33.3 | 37.1 |
| Our CodeDPO | 59.4 | 40.5 | 29.2 | 23.5 | 39.6 | 42.6 | 31.1 | 38.7 |
| Magic-S-DS-6.7B | 53.5 | 49.5 | 30.6 | 47.1 | 35.8 | 53.0 | 40.0 | 42.9 |
| Our CodeDPO | 59.4 | 50.5 | 31.9 | 39.7 | 41.5 | 47.8 | 33.3 | 43.7 |
| Base Model |  |  |  |  |  |  |  |  |
| Phi-2-2.7B | 42.6 | 33.6 | 15.5 | 16.2 | 17.0 | 11.3 | 17.8 | 23.5 |
| Our CodeDPO | 49.0 | 33.6 | 16.5 | 14.7 | 20.8 | 14.8 | 13.3 | 25.3 |
| DSC-1.3B | 36.8 | 19.5 | 10.0 | 14.7 | 10.4 | 17.4 | 11.1 | 17.5 |
| Our CodeDPO | 34.8 | 23.6 | 10.7 | 14.7 | 20.8 | 13.9 | 8.9 | 18.9 |
| DSC-6.7B | 52.3 | 35.5 | 20.6 | 19.1 | 24.5 | 37.4 | 22.2 | 31.1 |
| Our CodeDPO | 56.8 | 36.4 | 21.6 | 17.6 | 34.0 | 34.8 | 20.0 | 32.8 |
| StarCoder2-7B | 54.2 | 37.7 | 18.6 | 25.0 | 31.1 | 23.5 | 35.6 | 31.4 |
| Our CodeDPO | 56.8 | 38.2 | 18.9 | 20.6 | 39.6 | 25.2 | 31.1 | 32.6 |

### 5.2 Code Efficiency (RQ2)

To address RQ2, we follow existing methods (Shypula et al., ) by measuring the execution time of the generated code and calculating the speed-up ratio. We also evaluate the percentage of optimized code before and after applying CodeDPO, where a program is considered optimized if it is at least 10% faster than its baseline. These metrics are based on the intersection of solved problems before and after applying CodeDPO. We select HumanEval+ and MBPP+ for evaluation because they significantly expand the diversity of test cases, making them more reliable for measuring the execution efficiency of the generated code under a variety of edge cases.
Since runtime environments can affect measurements, we repeat each evaluation five times and show the distribution in Figure 3. It is clear that CodeDPO consistently improves code performance. The speed-up ratio shows that our method speeds up the code by 1.25 to 1.45 times.
The range in the figure highlights that most measurements cluster around a significant performance boost.
Additionally, the percentage of optimized code indicates that after applying CodeDPO, around 20%-45% of generated code solutions have been improved, confirming its effectiveness in enhancing code efficiency.

Figure: (a) HumanEval+
Refer to caption: extracted/5908942/speeduphe_v2.png

### 5.3 Ablation Studies

#### 5.3.1 Self Generation and Validation Algorithm (RQ3)

##### Correlation between self-validation scores and actual code accuracy using HumanEval ground truth tests

To evaluate the effectiveness of our self-generation-and-validation algorithm, we examine the correlation between self-validation scores and actual code accuracy. We use a benchmark with pre-existing ground truth test cases, such as HumanEval, for this preliminary experiment.
For each problem in HumanEval, we sample 15 code solutions and tests following the setting in Section 4, and then use different strategies to rank these generated codes. To evaluate the rank quality, we execute with the ground truth for each code to get the actual code accuracy. Then, we calculate the correlation between our self-validation scores and actual code accuracy.

We consider three experimental strategies: ❶ Self-validation score, which refers to our original method. ❷ Filter with all tests, which assumes all generated test cases are correct and uses them to judge code correctness. This approach creates passed/non-passed pairs, similar to the baseline PLUM (though PLUM uses GPT-4 for test generation, while we use a more cost-effective model). ❸ Sort by number of passed tests, which counts the number of passed tests for each code among all generated tests, using the code with the most and least passed tests as the comparison pair.

Table 4 presents the Spearman, Kendall’s Tau, and Normalized Discounted Cumulative Gain (NDCG) metrics for the different ranking strategies. Our experiments show that the self-validation score is highly correlated with actual code accuracy, and its ranking closely reflects true code quality, making it a reliable metric for preference optimization.
In contrast, filtering by all tests heavily depends on the quality of the test generation model. While baselines like PLUM ensure high-quality test generation using GPT-4, our more economical approach highlights that using all tests indiscriminately can introduce noise, as lower-quality tests skew the final ranking and poison the dataset. Sorting by the number of passed tests treats all tests equally important. However, due to the inherent uncertainty in generated tests, this method can be vulnerable to low-quality tests. Our proposed self-validation method employs a mutual reinforcement mechanism to update the credibility of both code and tests, effectively mitigating these issues.

**Table 4: Correlation between self-validation score and actual code accuracy on HumanEval.**
| Method | Spearman | Kendall’s Tau | NDCG |
| --- | --- | --- | --- |
| Self-validation score | 0.8598 | 0.8047 | 0.9653 |
| Filter with all tests | 0.6114 | 0.6114 | 0.8753 |
| Sort by # of passed tests | 0.7724 | 0.7250 | 0.9162 |

##### Impact of self-validation score on model performance

We apply these strategies to construct datasets and evaluate the final model performance in code generation. Table 5 presents the model performance across various dataset construction strategies. In addition, we introduce a new strategy—random selection—which randomly selects two code solutions from the generated code as the preference pair. The experiment results demonstrate that the self-generation-and-validation algorithm plays an essential role in ensuring the correctness and reliability of the preference dataset construction, significantly improving the performance of our CodeDPO framework.

**Table 5: Ablations of our self validation score on the trained model performance.**
| Model | HumanEval | HumanEval+ | MBPP | MBPP+ |
| --- | --- | --- | --- | --- |
| Phi-2-2.7B | 48.78 | 46.34 | 65.34 | 54.49 |
| Our CodeDPO | 57.32 | 51.83 | 69.05 | 56.88 |
| Filter with all tests | 49.39 | 48.17 | 69.17 | 55.13 |
| Sort by # of passed tests | 50.60 | 49.39 | 67.16 | 54.88 |
| Random selection | 22.56 | 18.90 | 45.11 | 36.59 |
| DeepSeekCoder-1.3B | 31.53 | 28.65 | 57.40 | 48.60 |
| Our CodeDPO | 42.07 | 38.04 | 61.37 | 53.43 |
| Filter with all tests | 34.75 | 29.89 | 57.40 | 48.80 |
| Sort by # of passed tests | 37.19 | 31.09 | 58.39 | 50.37 |
| Random selection | 21.34 | 18.29 | 48.94 | 38.35 |

#### 5.3.2 Impact of PO Training Strategy (RQ4)

We explore the impact of different preference optimization strategies (DPO, KTO, and SFT) on model performance. For training, the SFT strategy uses the best code solution from our constructed dataset. In our KTO strategy, we replace DPO with KTO in our framework.
As shown in Figure 1, the traditional SFT strategy struggles to guide the model in preferring correct solutions over incorrect or slower ones during training. The results in Table 6 demonstrate that DPO performs best among these strategies. Benefiting from our dataset construction method, we can obtain well-balanced preference pairs, enhancing the contrastive mechanism in DPO.

**Table 6: Comparison of preference optimization strategies (DPO vs. KTO vs. SFT).**
| Model | HumanEval | HumanEval+ | MBPP | MBPP+ |
| --- | --- | --- | --- | --- |
| Phi-2-2.7B | 48.78 | 46.34 | 65.34 | 54.49 |
| SFT | 55.49 | 49.22 | 66.87 | 55.76 |
| Our CodeDPO | 57.32 | 51.83 | 69.05 | 56.88 |
| Our CodeKTO | 54.88 | 51.22 | 64.91 | 53.63 |
| DeepSeekCoder-1.3B-base | 31.53 | 28.65 | 57.40 | 48.67 |
| SFT | 39.02 | 35.36 | 59.45 | 50.26 |
| Our CodeDPO | 42.07 | 38.04 | 61.37 | 53.43 |
| Our CodeKTO | 40.85 | 35.98 | 59.65 | 50.13 |
| DeepSeekCoder-6.7B-base | 47.60 | 39.60 | 70.20 | 56.60 |
| SFT | 56.09 | 46.95 | 70.18 | 56.88 |
| Our CodeDPO | 59.75 | 51.83 | 72.18 | 60.01 |
| Our CodeKTO | 54.88 | 49.39 | 71.93 | 58.65 |

### 5.4 Data Scaling Law for CodeDPO (RQ5)

To address RQ5, we explore how scaling the training data affects CodeDPO’s performance. We train the model with varying amounts of data—25%, 50%, and 75%—and evaluate its impact on the model performance. As shown in Table 7, increasing the data consistently improves model performance, but these improvements gradually plateau as the dataset size grows. For example, HumanEval scores rise from 32.92 (25%) to 41.46 (75%), with similar trends observed on MBPP.
In our experiments, we carefully balance performance gains and training costs, ensuring optimal results with CodeDPO. In further research, we plan to expand the current training scale to explore the extreme limits of CodeDPO’s performance.

**Table 7: Model Performances with different data scaling in our CodeDPO.**
| Model | HumanEval | HumanEval+ | MBPP | MBPP+ |
| --- | --- | --- | --- | --- |
| DeepSeekCoder-1.3B-base | 31.53 | 28.65 | 57.40 | 48.67 |
| 25% | 32.92 | 29.87 | 55.13 | 47.87 |
| 50% | 36.59 | 31.70 | 58.14 | 49.87 |
| 75% | 41.46 | 37.80 | 60.65 | 52.63 |
| Our CodeDPO | 42.07 | 38.04 | 61.37 | 53.43 |

## 6 Conclusion

We propose CodeDPO, a preference optimization framework for code models that focuses on both correctness and efficiency. CodeDPO introduces a novel dataset construction method that utilizes a self-generation-and-validation mechanism, enabling the simultaneous generation and evaluation of code and test cases to ensure correctness. Our PageRank-inspired algorithm iteratively updates the self-validation score of each code snippet, prioritizing solutions based on correctness and efficiency.
Our work technically validates the reliability of self-validation to synthesize preference optimization data, eliminating the need for complex external resources such as pre-existing tests or powerful generation models. We hope this work opens new avenues for synthesizing data and implementing large-scale preference optimization for code models.

## Appendix A CodeDPO Dataset Construction algorithm description

In order to make it clear, we give a formal algorithm description of the CodeDPO construction pipeline in Algorithm 1.

Figure: Algorithm 1 CodeDPO Dataset Construction Pipeline

## Appendix B LLM Prompts for CodeDPO dataset construction

We use the following prompts for dataset seed construction and self-validation.
During dataset construction, we first use code snippets from a randomly selected subset of the Stack v1 dataset as input and prompt the LLM to generate the concept (LLM Prompt 1). Based on the concept, we then prompt the LLM to generate the task description (LLM Prompt 2).

For the validation process, we directly prompt the LLM with the task description to generate code solutions. Additionally, we prompt the LLM to generate only assertion statements as test cases (LLM Prompt 3).
Since our chosen generation LLM is efficient and cost-effective, the entire process of data generation and construction takes around 40 hours on a server with 32 CPUs.

## Peer Reviews

confidence: 3
contribution: 1
initial_rating: 3
presentation: 2
questions: See weakness, please.
soundness: 1
strengths: 1. This paper focuses on an important topic, LLM-based code generation. 
2. Experimental results show the proposed CodeDPO outperforms existing methods.
summary: This paper proposes CodeDPO, a new framework for improving code generation models by incorporating preference learning to prioritize code correctness and efficiency. Unlike traditional methods, CodeDPO uses a self-generation-and-validation approach, generating and evaluating code with test cases to produce a dataset optimized for code preference. By iteratively ranking code snippets based on test pass rates and using a PageRank-inspired algorithm, CodeDPO enhances model performance on correctness and efficiency without external resources. Evaluations show that CodeDPO significantly outperforms existing methods.
weaknesses: This paper should be rejected for the following reasons:
1. The paper lacks many details, and the explanations of the experiments are insufficiently clear, making it difficult to understand.
2. The paper lacks sufficient novelty and rigor; it does not explain the costs associated with the proposed algorithm.
3. The writing is unrefined and not polished.

Main Argument
First, the paper is based on a variant of DPO and applies it to the code generation domain. However, it does not provide an explanation of DPO, merely mentioning what it is, which makes it hard for readers unfamiliar with DPO to grasp the concepts. Additionally, the paper lacks clear diagrams to illustrate the workflow of the proposed strategy, which further complicates understanding.
Here are specific points I find unreasonable or imprecise in the paper:
1.In terms of correctness within code preference, the paper employs the PageRank algorithm to rank code based on the principles of PLUM, but there are no other innovations related to correctness presented in the paper.
2. The paper does not explain how the test cases in the algorithm are generated.
3. In section 3.2, the choice of 15 code solutions and test cases is mentioned, but no justification is provided for this number. Should there not be comparative experiments? Furthermore, why was a temperature of 1.5 chosen for the model?
4. Does the proposed algorithm incur significant economic costs?
5. The concept of code efficiency is quite broad. In this paper, efficiency is evaluated solely based on the execution time of test cases, which is clearly insufficient. In practical applications, code efficiency is influenced by various factors such as the production environment and tools used. Moreover, the paper does not provide adequate evidence to demonstrate that the execution time of test cases reflects code efficiency.
6. In Chapter 3, the paper should provide a detailed description of the overall process for training the code, rather than only explaining the process after SFT.
7. In Table 2, the pass rates for CODEDPO and PLUM on the more complex benchmarks MBPP and MBPP+ are similar. However, the paper does not adequately explain this point. Why are the results similar? Does this not indicate that CODEDPO does not offer significant improvement?
8. The paper does not report statistics on the average execution time of the test cases. Is the execution time in milliseconds or seconds?
9. The experimental strategy in section 5.3.1 lacks theoretical justification and is difficult to understand.
10. In section 5.3.2, what strategy was used to construct CODEKTO? Why was CODEKTO chosen for comparison, and why not compare against other PO strategies?

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Can you report the number of training samples used in OSS-Instruct, Code-Optimise, and PLUM?

2. Can you elaborate on how the real-world repositories were selected to construct the initial seeds for data generation? Did you perform any data cleaning and decontamination when curating the dataset?

3. Can you report the performance of PLUM for MagiCoder-CL-7B, Phi-2-2.7B, and DeepSeekCoder-1.3B in Table 2? Can you also report the performance of Code-Optimise and PLUM on DS-1000 in Table 3 and the code efficiency results on DS-1000 in Figure 3?

4. Can you compare your PageRank-based algorithm with the scoring algorithms used in CodeT and PLUM?

5. Step 2 in Figure 2 is hard to understand. The authors should redraw that part to illustrate the iterative scoring process.
soundness: 2
strengths: 1. This work presents an interesting investigation of DPO in the domain of code generation. While there have been some similar investigations like Code-Optimise and PLUM, this work involves a couple of new ideas, including the PageRank-inspired algorithm and the integration of code efficiency into the optimization objective. 

2. The authors compared CodeDPO with two similar approaches, Code-Optimise and PLUM, on multiple LLMs and code generation datasets. The results look promising.

3. The authors also did additional experiments to investigate the effectiveness of alternative design choices, such as the code solution scoring algorithm and the optimization strategy.
summary: This paper presents CodeDPO, a framework that improves code correctness and efficiency via direct preference optimization. To construct the preference learning datasets, CodeDPO adopts OSS-Instruct to generate code generation tasks from real-world code repositories. For each code generation task, CodeDPO prompts an LLM to generate multiple code solutions and test cases simultaneously. Since the LLM may generate incorrect solutions and test cases, the authors proposed to use the PageRank algorithm to iteratively score the code solutions and test cases based on the number of test cases each solution passes. CodeDPO further integrates code efficiency into the preference dataset by measuring the execution time of each code solution and selecting both fast and slow solutions. In the end, the authors constructed a dataset with 93K correctness optimization samples and 21K efficiency optimization samples. The authors evaluated CodeDPO on multiple LLMs and code generation datasets. The evaluation results show that CodeDPO outperformed OSS-Instruct and two similar code preference optimization methods, Code-Optimise and PLUM, in most settings. Furthermore, the authors investigated the impact of the PageRank-based scoring algorithm, the preference optimization strategies, and the dataset size.
weaknesses: 1. There is a potential fairness issue in the comparison with baseline methods. The finetuning dataset generated by CodeDPO includes 114K training samples in total. There is no description of the finetuning dataset sizes used in other baseline methods. For instance, if the authors used the original finetuning dataset generated by OSS-Instruct in Table 1, this would be unfair to OSS-Instruct since its dataset only includes 75K training samples. 

2. This paper only describes the final dataset generated by CodeDPO, but it does not describe the initial seeds used to generate the final dataset other than vaguely saying that the initial seeds were from real-world repositories. This raises a concern about whether this dataset may include tasks similar to those of the evaluation benchmarks---HumanEval, MBPP, and DS-1000. 

3. The evaluation results in Table 2 and Table 3 are not complete. Table 2 does not report the performance of PLUM for MagiCoder-CL-7B, Phi-2-2.7B, and DeepSeekCoder-1.3B. Table 3 does not report the performance of Code-Optimise and PLUM on DS-1000. Figure 3 only code efficiency results on HumanEval and MBPP, but not on DS-1000.

4. The design of the PageRank-based scoring algorithm has an issue when scoring test cases. According to Equation 2, if many code solutions pass a test case, the test case will receive a high score. As a result, many weak test cases will receive high scores. For example, if a test case does not check any requirement mentioned in the task description (e.g., "assert true" as an extreme case), this test case will receive a very high score and will be included in the training set. 

5. The evaluation baselines used in RQ3 are weak. CodeT (Chen et al. 2022) uses a dual execution agreement algorithm to rate the correctness of the code solutions and test cases simultaneously generated by CodeT. PLUM uses a self-consistency filtering mechanism to select test cases and code solutions. It is unclear what the advantages of the PageRank-based algorithm are compared with these two SOTA methods.

6. This paper makes several incorrect claims about existing work. First, Section 3.1 says OSS-Instruct extacts key programming concepts from open-source projects and leverages these concepts to generate programming tasks. OSS-Instruct doesn't perform any extraction of program concepts. It directly generates a task description from code fragments in open-source projects. Second, Section 2.2 claims that PLUM generates a limited number of test cases and has an imbalanced dataset. This claim seems unsubstantiated. There is no evaluation of the test cases generated by CodeDPO vs. PLUM. Furthermore, the self-consistency filtering mechanism used in PLUM also looks reasonable. Third, the authors claim that CodeDPO "does not rely on external resources" several times. This sounds wrong since CodeDPO needs to select initial seeds from real-world repositories to generate the finetuning dataset.

7. None of the code generation datasets used in this evaluation are designed for code efficiency. The authors should evaluate CodeDPO on a code efficiency dataset like EffiBench (NeurIPS 2024). https://github.com/huangd1999/EffiBench

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - Could the authors illustrate a bit more what are the main novelties of this work, beyond the combination of implementing existing ideas (See weaknesses)? This will help me better understand the additional value and contribution this work provides to the community, which I might have missed. A better understanding could significantly change my attitude towards this paper.
- Could the author discuss why their results using DPO and KTO contradict with those reported in PLUM? PLUM and CodeDPO have completely opposite conclusions about using DPO and KTO to train code generation models, and it is very confusing which algorithms are more suitable for learning preference with execution feedback and why. I would regard a systematic discussion to deeply understand this perspective as an interesting contribution. Unfortunately, neither PLUM nor CodeDPO provides sufficient discussion on it.
- Can the author provide some thoughts on, besides the straightforward execution signal we can get from the interpreter (e.g., execution correctness and efficiency), what other information can be used to improve the helpfulness, readability, and security of code? Since the test-case-driven functional correctness DPO has been studied by the existing work, it will be interesting to see more insights into the future work of DPO on code.
soundness: 2
strengths: + The paper trains a code generation model with DPO with several pre-trained checkpoints, illustrating the effectiveness of functional-correctness-driven DPO for code generation.
+ The paper proposes a handful of research questions to evaluate CodeDPO on varied benchmarks and compare with multiple baselines
summary: The paper describes CodeDPO, a framework developed to improve code generation models by incorporating preference learning to prioritize both code correctness and efficiency. CodeDPO generates a dataset through a self-generation-and-validation mechanism, where code snippets and test cases are created and iteratively ranked based on performance. Evaluations on five coding benchmarks indicate that CodeDPO offers improvements in model alignment for code correctness and efficiency over existing approaches, providing a dataset that can be used to optimize various code generation models in realistic scenarios.
weaknesses: __Few Novel Ideas Proposed and Few New Insights Concluded.__

The idea of trying DPO for code generation seems to be an intuitive combination for post-SFT phases of training, and such a combination has been tried in varied domains including code. For example, in the latest technical report of Llama-3.1, DPO has become the main algorithm for preference optimization, where they have empirically illustrated DPO's effectiveness in coding, math, natural language understanding, etc. Besides, Llama-3.1 also revealed that DPO has more stable scalability across 8B to 405B than on-policy algorithms, such as PPO. The point I hope to make here is that the effectiveness of DPO on code, as well as other data modalities, has been comprehensively studied and effectively applied to the product-level code LMs. The conclusion that training with SFT+DPO is more effective than SFT only, which is the main observation of CodeDPO, seems obvious nowadays, restricting the value of this paper alone.

The other techniques applied in this work also seem to be very mature and proved to be practical and effective already. For example, the idea of seeding realistic code and sampling synthetic coding problems + solutions using teacher LLM has been well-studied in pioneer work, such as WizardCoder and MagiCoder, and this technique turns out to be very effective and has been used by industry models, including Llama-3.1, Gemma, CodeGemma, Gemini-1.5, etc. Also, the code correctness scoring with LLM-generated tests seems very similar to related work like CodeT[1], and the iterative pipeline has become a popular strategy since the proposal of the Self-Debugging paper[2].

The new thing I noticed is maybe, considering the efficiency of code execution, this perspective has not been soundly studied regarding its standalone effects on other features, like code correctness and helpfulness, so it is not clear how important we should include this feature during DPO. I would encourage the authors to study this new feature in a controlled and isolated way, but this is just a minor suggestion.

Overall, I feel this paper is more of a proof-of-concept implementation of ideas that have been proven to be practically effective. Therefore, unfortunately, I feel the conceptual novelty and the scientific values in terms of new insights concluded from this paper alone are unsatisfactory for top venues like ICLR. However, I still encourage the authors to join the discussion during the rebuttal phase to clarify their novelty and new insights that have __not__ been revealed by the existing works that I might have missed during the review.

__Restricted Evaluation Setting__

All the evaluation settings are still restricted to simple benchmarks like EvalPlus and DS-1000. I am not sure how this simplified code generation setting with the most straightforward reward design might work in more challenging programming tasks, such as repository-level code completion or competition-level programming problems. 

Intuitively, a very capable policy model is the prerequisite for applying preference optimization or RL, and that might be why we could always see DPO/PPO providing additional value on top of HumanEval and MBPP. However, it is unclear but interesting to check that, if the LLM after SFT is not yet promising in the task (e.g., Magicoder reports 70+ accuracy in HumanEval while <30% accuracy in LiveCodeBench), whether DPO could still bring them a significant improvement, or it will hurt the model performance instead. I would encourage the authors to evaluate their model on the latest benchmarks, such as LiveCodeBench[3] and R2E[4].

[1] Chen et al., CodeT: Code Generation with Generated Tests.

[2] Chen et al., Teaching Large Language Models to Self-Debug.

[3] Jain et al., Livecodebench: Holistic and contamination free evaluation of large language models for code

[4] Jain et al., R2E: Turning any Github Repository into a Programming Agent Environment

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Have you performed deduplication of the dataset against the benchmarks? The dataset leakage could be an issue.
- The creation of DPO pairs for correctness and effeciency lack sufficient details. Since the correctness score is no longer 0 and 1, the creation of pairs is not that straightforward. Do you pick the highest score code snippet with the lowest score one to form the pair, or is it another way?
- In table 2, MagiCoder-DS-6.7B, CodeDPO is performing significantly worse than PLUM on HumanEval/HumanEval+. This result is different from the rest of the results. Do you have an explanation?
- In table 4, the correlation analysis is interesting. Do you have access to the unit tests generated from GPT-4 from PLUM? Could you evaluate the correlation of "filter all tests" with these more trustworthy tests from the stronger model?
- In table 6, KTO results are often worse than SFT results. Do you have an explanation for the phenomenon? Is it KTO training just not that good or is it your KTO training is problematic?
soundness: 3
strengths: The proposed method alleviates the problem of wrong tests when using LLMs generated tests, and the data generation does not require a strong teacher LLM. The page-rank-like algorithm is interesting. Extensive experiments have been conducted to show the effectiveness of the approach, and a number of ablation studies have been conducted to showcase the correlation with groundtruth, impact of different preference optimization method and data scaling.
summary: This paper proposes self generation of code solutions and tests, and cross-verify the code and tests via a pagerank-inspired algorithm.
weaknesses: I do not see strong weakness of the paper.
Some section of the paper lack sufficient details. For example, since the correctness score is no longer 0 and 1, the creation of pairs is not that straightforward. Do you pick the highest score code snippet with the lowest score one to form the pair, or is it another way?

Meta-review: The paper proposes CodeDPO, a framework that uses preference learning to improve code generation models in terms of correctness and efficiency. This is an important problem to tackle but, unfortunately, currently the paper is below par for ICLR because of following reasons:

- The proposed approach itself is a straightforward combination of existing methods like DPO. This is not a problem as long as new challenges are mentioned and addressed properly for the specific setting but that doesn't seem to be the case here. 

- The experimental evaluation is lacking in multiple dimensions and requires a lot of work. For example, inconsistent dataset sizes between CodeDPO and baselines make comparisons potentially unfair. Even then, the performance improvements are relatively minor or none over existing baselines. 

- There were lots of missing details and lack of justification about the approach. For example, specific implementation choices and hyper-parameters were not justified. 

Overall, I request the authors to consider improving the paper based on comments from the reviewers for the next cycle.

Decision: Reject