Focused-DPO: Enhancing Code Generation Through Focused Preference Optimization on Error-Prone Points
Authors: Kechi Zhang, Ge Li, Jia Li, Yihong Dong, Jia Li, Zhi Jin
Accepted at: acl25

Focused-DPO: Enhancing Code Generation Through Focused Preference Optimization on Error-Prone Points

Kechi Zhang1,2*, Ge Li1,2*, Jia Li3, Yihong Dong1,2, Jia Li1,2, Zhi Jin1,2*
1Key Lab of High Confidence Software Technology (PKU), Ministry of Education
2School of Computer Science, Peking University, China
3College of AI, Tsinghua University
{zhangkechi,lige,zhijin}@pku.edu.cn

Abstract

Code generation models have shown significant potential for automating programming tasks. However, the challenge of generating accurate and reliable code persists due to the highly complex and long-reasoning nature of the task. Even state-of-the-art models often fail in code generation due to small errors, which can drastically affect the overall functionality of code. Our study identifies that current models tend to produce errors concentrated at specific error-prone points, which significantly impacts the accuracy of the generated code. To address this issue, we introduce Focused-DPO, a framework that enhances code generation by directing preference optimization towards these critical error-prone areas. This approach builds on Direct Preference Optimization, emphasizing accuracy in parts prone to errors. Additionally, we develop a method called Error-Point Identification, which constructs a dataset that targets these problematic points without requiring costly human annotations. Our experiments on benchmarks such as HumanEval(+), MBPP(+), and LiveCodeBench demonstrate that Focused-DPO significantly improves the precision and reliability of code generation, reducing common errors and enhancing overall code quality. By focusing on error-prone points, Focused-DPO advances the accuracy and functionality of model-generated code.

1 Introduction

Code generation has emerged as a pivotal task in artificial intelligence, enabling models to automate essential software development tasks. Code Models (GPT-4, 2023; Guo et al., 2024; Hui et al., 2024) have demonstrated remarkable capabilities in code generation tasks. These advancements have significantly improved developer’s productivity, accelerating software delivery timelines.

Despite their success, generating correct code remains a substantial challenge due to the complex and long-reasoning nature of the task. Writing code necessitates long reasoning, where numerous small decisions about syntax and logic must work together to produce a functional program. Even minor mistakes, such as an incorrect operator, can cause a program to fail. Code generation, therefore, can be viewed as a multi-step long reasoning process. Ensuring the accuracy of every decision in this multi-step process collectively determines the correctness of the resulting output code.

When examining the outputs of current code generation models, we find that errors are not evenly spread across the code. Large language models tend to produce errors concentrated in certain error-prone points, even when sampling multiple times with a high temperature. We illustrate this phenomenon in Figure 1, which shows error-prone points highlighted in yellow. Despite the overall code having similar prefixes and suffixes, differences at these highlighted error points significantly

Question:
Take a 1D array 'arr' and an integer 'n' as input
Return an array of arrays, where each subarray contains n consecutive elements of 'arr' ...

from itertools import groupby

def sol1(arr, n):
    groups = [list(g) for _, g in groupby(enumerate(arr), lambda x: x[0] // n)]
    return groups

The subsequent code accuracy exceeds 90.02%

def sol2(arr, n):
    groups = [list(g) for _, g in groupby(enumerate(arr), lambda x: x[0] % n)]
    return groups

The subsequent code accuracy drops to 3.17%

Figure 1: Error-prone points in generated code from Qwen-2.5-Coder-Instruct-7B. We sample 20 outputs for this question. Outputs have common prefixes and suffixes, differing mainly at yellow-highlighted error points. Continuing generation at these points leads to drastically different accuracies (90.02% vs. 3.17%). This disparity is not seen in non-highlighted parts.
impact the final code accuracy. Generating code from correct outputs at these error-prone points can achieve a final accuracy of up to 90.02%, whereas starting from incorrect outputs reduces accuracy to 3.17%. Parts of the code, such as function headers (usually at the prefix) or return statements (usually at the suffix), often follow familiar patterns. However, some middle parts of the code, which involve more complex reasoning, are more prone to errors. Errors in these parts can disrupt and affect the entire program’s reliability.

It is crucial to address these error-prone points for code generation. However, existing studies on code generation overlook this problem. While standard training approaches such as Supervised Fine-Tuning (SFT) (Wang et al., 2022) help improve overall output quality, they do not specifically focus on the crucial parts necessary for correctness. Methods like Direct Preference Optimization (DPO) (Rafailov et al., 2024) aim to align outputs with preferences (e.g., "chosen" vs. "rejected"), but often overlook fine-grained error-prone points of the code. As a result, these trained models might generate code that appears correct initially but contains critical issues at the error-prone points, ultimately affecting overall accuracy.

To tackle these issues, we introduce Focused-DPO, a framework designed to enhance code generation through focusing preference optimization on error-prone points. Focused-DPO builds on Direct Preference Optimization by emphasizing accuracy improvement in areas where errors are most likely to occur. Unlike traditional methods that treat all parts of the code equally, Focused-DPO specifically targets those error-prone points, which are essential for the overall correctness of the program.

Focused-DPO is a data-driven preference optimization method that relies on a specially constructed dataset with identified error-prone points. We propose a dataset construction method named Error-Point Identification, which includes an automated pipeline to construct paired code preference datasets. This method extracts concepts from real code repositories and synthesizes programming problems. By concurrently generating code and tests, and using a page-rank-inspired algorithm for ranking, we determine the relative performance of all generated code. Error-Point Identification employs common prefix and suffix matching to precisely locate error-prone points. Additionally, our method automatically identifies error-prone code parts, eliminating the need for costly human input, making it scalable and efficient for a variety of programming tasks.

We evaluate Focused-DPO using standard benchmarks such as HumanEval(+) (Liu et al., 2024a), MBPP(+), and LiveCodeBench (Jain et al., 2024), and observe significant improvements over existing methods. Even for models like Qwen2.5-Coder, which already have undergone large-scale alignment training, Focused-DPO still achieves a 42.86% relative improvement on extremely hard competition-level problems in LiveCodeBench. The results show notable increases in the generation quality on error-prone points, highlighting Focused-DPO’s effectiveness in enhancing the accuracy of code generation.

Our contributions are summarized as follows:

• We propose Focused-DPO , a novel framework that enhances code generation by focusing preference optimization on error-prone points, resulting in more accurate codes.

• We introduce a dataset construction method that automatically identifies error-prone points by generating both code and corresponding tests for fine-grained self-verification.

• Experiments on widely-used benchmarks show that Focused-DPO improves the generation quality of code models, even for those that have already undergone extensive post-training on million-level datasets.

2 Related Work

Large language models (LLMs) have made significant progress in generating code from natural language descriptions, showing great potential for automating software development tasks (Zhang et al., 2024d,c). Models(GPT-4, 2023; Li et al., 2023; Hui et al., 2024; Guo et al., 2024; Jiang et al., 2024) have demonstrated strong performance, thanks to extensive training on diverse datasets. To further enhance their capabilities, posting training methods like Supervised Fine-Tuning (SFT) (Luo et al., 2023; Wei et al., 2023b; Zhang et al., 2023) and Direct Preference Optimization (Hui et al., 2024; Zhang et al., 2024b; Dou et al., 2024; Gee et al., 2024; Zhang et al., 2024a) are commonly applied. Preference optimization approaches focus on aligning model outputs with desired outcomes by prioritizing more favorable responses over less favorable
ones. However, existing DPO approaches fail to address one important issue: they do not directly target the most error-prone points in generated code. Errors in these high-impact parts can lead to significant quality and reliability issues in the final output. We aim to address this issue by focusing the preference optimization learning on these error-prone points in the generated code.

Some fine-grained preference optimization methods (Rafailov et al., 2024; Lai et al., 2024; Lu et al., 2024; Zeng et al., 2024; Lin et al., 2024) have shown strong potential in domains like mathematics, which rely heavily on natural language reasoning. Step-DPO (Lai et al., 2024) and Step-Controlled DPO (Lu et al., 2024) propose generating step-wise preference datasets to enable optimization learning based on the standard DPO loss. TDPO (Zeng et al., 2024) enhances the DPO loss by incorporating forward KL divergence constraints at the token level, achieving fine-grained alignment for each token. cDPO (Lin et al., 2024) proposes a tricky method to find the critical token in the thought chain that affects overall accuracy. However, the identified tokens are typical in natural language and the method does not apply to code, which features similar overall patterns but relies on specific key elements in long reasoning processes. However, in the context of code generation, where a small error-prone point can lead to major functional errors, these existing methods often struggle to construct adequate datasets or fail to achieve ideal improvements due to weak fine-grained reward signals. To address this, we propose Focused-DPO, a framework that improves code generation by focusing on optimizing these high-impact parts. Our dataset construction method employs a self-generation and validation process to construct datasets that explicitly identify error-prone points, ensuring the optimization learning process directly enhances the parts of the code that matter most for overall correctness.

3 Focused-DPO

Our proposed Focused-DPO framework aims to enhance code generation by concentrating on error-prone points through focused preference optimization. Building on Direct Preference Optimization, our Focused-DPO specifically targets those high-impact parts of the source code, rather than treating all code parts equally. As illustrated in Figure 2, our method involves three main steps: ① Synthetic Data Generation with Real-World Source Code : We initiate by collecting a seed dataset from open-source code repositories and generate programming task prompts. ② Fine-Grained Verification to Identify Error-Prone Points : We generate both code and tests simultaneously using a self-generation-and-validation loop. We apply a PageRank algorithm to iteratively update scores and rank the outputs, identifying correct and incorrect code samples. By distinguishing between similar versions of correct code and incorrect code, we locate significant parts that highly affect the final correctness and identify these parts as error-prone points, allowing for further fine-grained optimization learning. ③ Focused Preference Optimization Learning : We design a learning optimization algorithm specifically for these critical error-prone points. Using the constructed dataset, our novel training loss helps the model develop a preference for these focused parts within the code, thus optimizing performance more effectively.

![Overview of the Focused-DPO framework. Focused-DPO consists of three key stages: ① Generating synthetic question prompts from real-world code repositories. ② Using a policy model to simultaneously generate code and test cases, applying a page-rank algorithm to identify correct and incorrect samples and locate error-prone points using common prefixes and suffixes. ③ Applying Focused-DPO, which pays special attention on error-prone points as if applying a magnifying glass for focused optimization.](page_1012_186_410_377.png)

Figure 2: Overview of the Focused-DPO framework. Focused-DPO consists of three key stages: ① Generating synthetic question prompts from real-world code repositories. ② Using a policy model to simultaneously generate code and test cases, applying a page-rank algorithm to identify correct and incorrect samples and locate error-prone points using common prefixes and suffixes. ③ Applying Focused-DPO, which pays special attention on error-prone points as if applying a magnifying glass for focused optimization.
3.1 Synthetic Data Generation with Real-World Source Code

The first step in our approach is the construction of a synthetic dataset. We collect a diverse set of programming snippets from open-source repositories to create a seed dataset. Similar to OSS-instruct (Wei et al., 2023a), we use the seed dataset to extract key programming concepts, such as algorithm design and data structure utilization. Then based on these concepts we generate the final prompts. This construction strategy allows the model to explore a broad range of scenarios. The generated question prompts are used in the following stages.

3.2 Fine-Grained Verification to Identify Error-Prone Points

To identify error-prone points, we propose a dataset construction method named Error-Point Identification. Firstly, we use the policy model to simultaneously generate \( k \) output codes and test cases based on the question prompts using a higher-temperature setting. In our experiment, we set \( k = 10 \). Using their execution relationships, we then adopt the ranking method from CodeDPO (Zhang et al., 2024b), a page-rank algorithm to iteratively update scores and rank the outputs:

\[
\text{Score}_t(c_i) = (1 - d) \times \text{Score}_{t-1}(c_i) \\
+ d \times \sum_{t_j} \text{Score}_{t-1}(t_j) \times \text{Link}(t_j, c_i)
\]

\[
\text{Score}_t(t_j) = (1 - d) \times \text{Score}_{t-1}(t_j) \\
+ d \times \sum_{c_i} \text{Score}_{t-1}(c_i) \times \text{Link}(c_i, t_j)
\]

(1)

Where \( d \) is the damping factor, and \( \text{Link}(t_j, c_i) \) indicates whether a code snippet \( c_i \) passes the test case \( t_j \). The ranking score is updated iteratively until the ranking of the code stabilizes.

We consider the test case that the highest-ranked code correctly passes as the ground truth test case for this question. Subsequently, we split all generated codes into two categories: correct code that passes all ground truth test cases and incorrect code that does not. For each pair consisting of a correct code sample and an incorrect code sample, we match their common prefix and suffix to decompose each code snippet into three parts: common_prefix, mid_chosen (or mid_rej), and common_suffix. We then define a Diff function as follows:

\[
\text{Rank(mid)} = \text{Score(common\_prefix, mid, common\_suffix)},
\]
\[
\text{Diff} = \text{Rank(mid\_chosen)} - \text{Rank(mid\_rej)} \\
+ \lambda * (\text{length(common\_prefix)} + \text{length(common\_suffix)}).
\]

(2)

Our constructed Diff function includes two components: \( \bullet \) the difference in rank between the correct and incorrect code, and \( \bullet \) the sum of the lengths of the common prefix and suffix, which ensures that the error-prone points are more concentrated. We maximize Diff to choose the mid_chosen and mid_rej parts that significantly impact the code’s correctness, and identify these as the error-prone points. By focusing on error-prone points, we create training samples that directly address the parts of the code that have significantly impact on correctness. For each policy model, we apply necessary filtering to the generated data, resulting in a final dataset containing 5,000 training samples and 1,000 validation samples. Table 9 presents an example of data statistics.

3.3 Focused Preference Optimization Learning

The core of our method lies in modifying the Direct Preference Optimization (DPO) framework to better enhance code generation by focusing on error-prone points of the code. Given a pairwise preference dataset \( \mathcal{D} = \{ (x_i, y_i^{chosen}, y_i^{ref}) \}_{i=1}^M \), the standard DPO loss (Rafailov et al., 2024) is expressed as:

\[
\ell_{\text{DPO}} = -\mathbb{E}_{(x,y^{chosen},y^{ref}) \sim \mathcal{D}} \left[ \log \sigma \left( \phi(x, y^{chosen}) - \phi(x, y^{ref}) \right) \right],
\]

(3)

where \( \phi(x, y) \) is an implicit reward function. The reward function is defined as:

\[
\phi(x, y) = \beta \cdot \log \frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)} + \underbrace{\beta \cdot \log Z(x)}_{\text{this term can ultimately be reduced}}
\]

(4)

where \( \pi_\theta(y|x) \) represents the probability of a generated response \( y \) under the policy model, and \( \pi_{ref}(y|x) \) is the probability under a reference model, typically the SFT baseline. The goal of DPO loss is to maximize reward difference between the preferred and non-preferred samples.

Reward Function Modification  In its original form, the DPO reward \( \phi(x, y) \) is calculated over the entirety of the sample \( y \), treating all parts of the code equally. However, in the context of code generation, not all parts of the code contribute equally to correctness. Building on our observation that the
middle part (mid) of code—the error-prone point we identify in Section 3.2—should receive more attention, we restructure the reward to reflect the relative importance of different code parts. The reward function is modified to weight the mid part more heavily, reflecting its critical contribution to the correctness of the code. For the preferred sample, the reward function becomes:

\[
\phi_{chosen}(x, y) = \beta \cdot \left( \log \frac{\pi_\theta(\text{prefix}|x)}{\pi_{ref}(\text{prefix}|x)} \right. \\
+ w_{focused} \cdot \log \frac{\pi_\theta(\text{mid}|x, \text{prefix})}{\pi_{ref}(\text{mid}|x, \text{prefix})} \\
\left. + \log \frac{\pi_\theta(\text{suffix}|x, \text{prefix}, \text{mid})}{\pi_{ref}(\text{suffix}|x, \text{prefix}, \text{mid})} \right)
\]

Where \( w_{focused} \) is a weight that amplifies the importance of the mid part.

For the non-preferred sample, we adopt a similar structure but introduce an adjustment to further downweight the contribution of the suffix. This adjustment is based on our observation that regardless of whether the mid part contains errors, the content of the suffix is often the same or similar. Our results in Section 5.1 show that the correlation between the suffix and the overall accuracy of the final code is low, making it less significant in the reward calculation. The reward becomes:

\[
\phi_{rej}(x, y) = \gamma \cdot \left( \log \frac{\pi_\theta(\text{prefix}|x)}{\pi_{ref}(\text{prefix}|x)} \right. \\
+ w_{focused} \cdot \log \frac{\pi_\theta(\text{mid}|x, \text{prefix})}{\pi_{ref}(\text{mid}|x, \text{prefix})} \Bigg)
\]

Final Loss Function Substituting the modified rewards for the preferred (\( y^{chosen} \)) and non-preferred (\( y^{rej} \)) examples into the original DPO loss and simplifying by canceling common terms, we can obtain that:

\[
\Delta_{\text{reward}} = \phi_{chosen}(x, y^{chosen}) - \phi_{rej}(x, y^{rej})
= \sum_{j=k+1}^m \beta \cdot w_{focused} \cdot \log \frac{\pi_\theta(l_{mid, chosen}^j|x, t_{0:k}^{prefix}, t_{k+1:j-1}^{mid, chosen})}{\pi_{SFT}(l_{mid, chosen}^j|x, t_{0:k}^{prefix}, t_{k+1:j-1}^{mid, chosen})}
- \sum_{j=k+1}^n \beta \cdot w_{focused} \cdot \log \frac{\pi_\theta(l_{mid, rej}^j|x, t_{0:k}^{prefix}, t_{k+1:j-1}^{mid, rej})}{\pi_{SFT}(l_{mid, rej}^j|x, t_{0:k}^{prefix}, t_{k+1:j-1}^{mid, rej})}
+ \sum_{j=m+1}^{L_1} \beta \cdot \log \frac{\pi_\theta(l_{suffix}^j|x, t_{0:k}^{prefix}, t_{k+1:m}^{mid, chosen}, t_{m+1:j-1}^{suffix})}{\pi_{SFT}(l_{suffix}^j|x, t_{0:k}^{prefix}, t_{k+1:m}^{mid, chosen}, t_{m+1:j-1}^{suffix})}
= \Delta_{mid} + \Delta_{suffix}
\]

So the final loss function for Focused-DPO is expressed as:

\[
\mathcal{L}_{\text{Focused-DPO}}(\pi_\theta; \pi_{SFT}) =
- \mathbb{E}_{(x, y^{chosen}, y^{rej}) \sim \mathcal{D}} [\log \sigma (\Delta_{mid} + \Delta_{suffix})],
\]

The terms \( \Delta_{mid} \) and \( \Delta_{suffix} \) capture the weighted differences in the probabilities of critical parts between the preferred and non-preferred samples, with greater emphasis focused on the mid parts, which is the error-prone point.

Through this modification, Focused-DPO shifts the focus of optimization toward the error-prone point in the code. By increasing the weight of these parts in the reward calculation, our framework ensures that the model prioritizes improvements where they matter most, leading to higher-quality and more reliable code generation.

4 Experiment Setup

We aim to answer the following research questions:

RQ1: Are there error-prone points in generated code that significantly affect the correctness of the output? This question addresses the core motivation behind Focused-DPO. To investigate this, we construct the validation dataset following Section 3.2. This setup provides empirical evidence supporting the theoretical underpinnings of our Focused-DPO.

RQ2: Can Focused-DPO improve the generation quality of code models, even those that have already been heavily post-trained with alignment techniques such as standard DPO? To explore this, we evaluate Focused-DPO on several widely-used code generation benchmarks, including HumanEval (Chen et al., 2021), HumanEval+ (Liu et al., 2024a), MBPP (Austin et al., 2021), MBPP+, and LiveCodeBench (Jain et al., 2024).

RQ3: How do different components of the Focused-DPO loss formulation affect model performance? Ablation studies include evaluating our dataset construction method, as well as key components in our loss formulation.

4.1 Baselines

We evaluate several widely used large language models (LLMs) in the code generation domain. For base models, we apply Focused-DPO to DeepSeekCoder-base-6.7B (Guo et al., 2024) and Qwen2.5-Coder-7B (Hui et al., 2024). For instruct models , we evaluate on Magicoder-S-DS-6.7B (Wei et al., 2023b) and DeepSeekCoder-instruct-6.7B, which are post-trained from DeepSeekCoder-base-6.7B with large-scale SFT. We further evaluate Qwen-2.5-Coder-Instruct-7B, which is post-
trained from Qwen2.5-Coder-7B on million-level datasets with SFT and DPO.

We compare against several widely used training techniques, including: SFT, standard DPO, Step-DPO (Lai et al., 2024), TDPO (Zeng et al., 2024). SFT trains models only with positive samples, while the other methods utilize a pairwise dataset of preferred and rejected samples.

4.2 Training and Inference Settings

For each backbone LLM, we sample 10 code candidates and corresponding test cases for each problem prompt using temperature=1.5. An example of data statistics is in Table 9. Our analysis shows this configuration results in a stable ranking score and ensures diversity. We focus on Python-based datasets given its widespread use. For training, we train for 10 epochs on 8 NVIDIA V100 GPUs and select the best-performing checkpoint based on the lowest validation loss. We set \( w_{focused} = 2 \) in our experiments. We use a learning rate of \( 5 \times 10^{-6} \) with a linear scheduler and warm-up. We employ greedy search during inference.

5 Results and Analyses

5.1 Exploration of Error-Prone Points in Code (RQ1)

We conduct experiments to validate our motivation:
① Correlation analysis confirms that error-prone points in the code significantly impact correctness, whereas other code parts have minimal effect.
② Generation experiments show that continuing at these points with different content leads to significant differences in overall correctness.
③ Observations reveal that existing code models perform suboptimally at these points.

Correlation Between Different Code Parts and Final Correctness Utilizing the dataset construction pipeline described in Section 3.2, we evaluate the validation dataset based on Qwen2.5-Coder-Instruct-7B. We analyze the relationship between prefix, suffix, two types of mid parts, and the final code correctness, as presented in Table 1.

Results in Table 1 show that common_prefix + chosen_mid appears much more frequently in correct solutions, while common_prefix + rej_mid is prevalent in incorrect solutions. This confirms the critical influence of the mid part, with strong positive and negative correlations respectively, affirming the existence of error-prone points in generated code. In contrast, we find that the prefix and suffix parts have little relation to the correctness of the final answer. It is important to note that in incorrect code, despite the errors in the mid section, the following suffix is not a significant cause of the errors. This observation justifies our decision to exclude the suffix in the reward modification in Section 3.3. These findings provide empirical evidence supporting our hypothesis that focusing on these error-prone points is essential to enhance model performance, which is the core motivation behind our Focused-DPO framework.

<table>
  <tr>
    <th>Segment</th>
    <th>Correct</th>
    <th>Incorrect</th>
    <th>Phi Coefficient</th>
  </tr>
  <tr>
    <td>Common Prefix</td>
    <td>0.7907</td>
    <td>0.7325</td>
    <td style="background-color:#e6f2ea">0.0683</td>
  </tr>
  <tr>
    <td>Common Suffix</td>
    <td>0.8479</td>
    <td>0.7864</td>
    <td style="background-color:#e6f2ea">0.0796</td>
  </tr>
  <tr>
    <td>Common Prefix + Chosen Mid</td>
    <td>0.6367</td>
    <td>0.0911</td>
    <td style="background-color:#e6f2ea">0.5651</td>
  </tr>
  <tr>
    <td>Common Prefix + Reject Mid</td>
    <td>0.0116</td>
    <td>0.5575</td>
    <td style="background-color:#ffe6e6">-0.0685</td>
  </tr>
</table>

Table 1: Relationships between the prefix, suffix, and the two types of mid parts with the final code correctness. The table includes the frequency of each part in correct and incorrect code, as well as their correlation coefficients with overall code correctness.

Accuracy of Continuation at Error-Prone Points
We further generate 20 code solutions based on different contents at error-prone points, to explore the correctness of the final code generated under different conditions in Table 2.

<table>
  <tr>
    <th>Based on Input</th>
    <th>pass@1</th>
    <th>pass@3</th>
    <th>pass@5</th>
    <th>pass@10</th>
  </tr>
  <tr>
    <td>Common Prefix + Chosen Mid</td>
    <td>0.9902</td>
    <td>0.9532</td>
    <td>0.9688</td>
    <td>0.9871</td>
  </tr>
  <tr>
    <td>Common Prefix + Reject Mid</td>
    <td>0.0317</td>
    <td>0.0633</td>
    <td>0.0810</td>
    <td>0.1159</td>
  </tr>
</table>

Table 2: Pass rates based on different content at error-prone points.

The pass rates shown in Table 2 highlight a striking contrast: using chosen_mid at error-prone points results in significantly higher pass rates, reaching around 90% at pass@1, compared to just over 3% for the rej_mid version. This demonstrates the critical importance of accurate content in the error-prone points for determining the correctness of the final generated code.

Based on the above results, we have noticed that the generated content at the error-prone points significantly affects the final outcomes. This leads to a question: how do current code generation models behave at these error-prone points?

Generation Preferences at Error-Prone Points in Code Models We further analyze the Qwen-2.5-Coder-Instruct-7B, which has been post-trained
Figure 3: Generation probability difference (\( p(\text{chosen\_mid}) - p(\text{rej\_mid}) \)) with input.

on million-level datasets using SFT and DPO. We examine the generation preferences of this heavily post-trained model at error-prone points. Specifically, we calculate the probability difference between generating chosen_mid and rej_mid when given the common_prefix as input. The distribution of the difference is shown in Figure 3. The model exhibits little to no clear preference, indicating that existing code generation models lack effective generation capability at these error-prone points. Through this exploration, we confirm that focused preference optimization of error-prone points is crucial for improving the accuracy of code models, addressing RQ1.

5.2 Main Results (RQ2)

Results on benchmarks Tables 3 and 4 summarize the performance of Focused-DPO compared to various baselines, including standard DPO, Step-DPO, TDPO, and SFT. Note that the formulas for standard DPO and Step-DPO are identical, making them equivalent. The relative improvements (Rel) are reported for a clearer comparison.

As shown in Table 3, Focused-DPO consistently outperforms the baseline models across all benchmarks. On the HumanEval(+) and MBPP(+) benchmarks, Focused-DPO improves relative accuracy by 4.79% on average over the baseline. We also evaluate on LiveCodeBench, a challenging benchmark that features iteratively updated, competition-level programming problems sourced from platforms such as LeetCode. The benchmark is divided into three levels of difficulty: Easy, Medium, and Hard. Focused-DPO achieves consistent improvements across all difficulty levels of LiveCodeBench. Notably, on the hardest category (Hard), Focused-DPO can achieve huge relative performance. Focused-DPO entirely outperforms other advanced preference optimization baselines such as Step-DPO and TDPO. These findings highlight the effectiveness of Focused-DPO in challenging code generation scenarios, where optimization on error-prone points of code plays a crucial role in determining final correctness.

Enhancing Heavily Post-trained Models
Focused-DPO can significantly enhance the performance of code models that have already undergone extensive post-training. As demonstrated in Table 5, models like Qwen2.5-Coder-instruct, which have been meticulously optimized using millions of data points from SFT and DPO processes, still exhibit substantial improvements with our Focused-DPO framework. To further illustrate Focused-DPO’s benefits on heavily post-trained models, we conducted an extensive initial DPO training phase. Following the methodology from CodeDPO, we used the model DeepSeekCoder-base-6.7B and a large-scale dataset with 93k samples for DPO training, continued until full convergence. We then apply Focused-DPO for further experiments. This allows us to explore the extent to which Focused-DPO could drive additional improvements, even in models already trained by intensive post-training processes.

As shown in Table 5, we start from the base model and progressively incorporate the SFT stage (Wei et al., 2023b) and the first DPO stage (Zhang et al., 2024b). Finally, applying our Focused-DPO leads to the highest pass rates achieved. These re-

<table>
  <tr>
    <th>Model</th>
    <th>HumanEval</th>
    <th>HumanEval+</th>
    <th>MBPP</th>
    <th>MBPP+</th>
  </tr>
  <tr>
    <th colspan="5">Instruct Model</th>
  </tr>
  <tr>
    <td>Qwen2.5-coder-instruct-7B</td>
    <td>0.915</td>
    <td>0.841</td>
    <td>0.828</td>
    <td>0.714</td>
  </tr>
  <tr>
    <td>+ Our Focused-DPO</td>
    <td><b>0.927</b></td>
    <td><b>0.878</b></td>
    <td><b>0.847</b></td>
    <td><b>0.762</b></td>
  </tr>
  <tr>
    <td>Relative Improvement</td>
    <td>1.29%</td>
    <td>4.14%</td>
    <td>2.24%</td>
    <td>7.11%</td>
  </tr>
  <tr>
    <td>DPO / Step-DPO</td>
    <td>0.921</td>
    <td>0.854</td>
    <td>0.841</td>
    <td>0.743</td>
  </tr>
  <tr>
    <td>Token-DPO</td>
    <td>0.927</td>
    <td>0.872</td>
    <td>0.853</td>
    <td>0.751</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>0.927</td>
    <td>0.872</td>
    <td>0.853</td>
    <td>0.757</td>
  </tr>
  <tr>
    <td>DeepSeekCoder-instruct-6.7B</td>
    <td>0.774</td>
    <td>0.770</td>
    <td>0.761</td>
    <td>0.659</td>
  </tr>
  <tr>
    <td>+ Our Focused-DPO</td>
    <td><b>0.823</b></td>
    <td><b>0.732</b></td>
    <td><b>0.765</b></td>
    <td><b>0.669</b></td>
  </tr>
  <tr>
    <td>Relative Improvement</td>
    <td>6.35%</td>
    <td>4.38%</td>
    <td>1.80%</td>
    <td>1.56%</td>
  </tr>
  <tr>
    <td>DPO / Step-DPO</td>
    <td>0.787</td>
    <td>0.713</td>
    <td>0.751</td>
    <td>0.661</td>
  </tr>
  <tr>
    <td>Token-DPO</td>
    <td>0.799</td>
    <td>0.726</td>
    <td>0.751</td>
    <td>0.661</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>0.787</td>
    <td>0.726</td>
    <td>0.759</td>
    <td>0.667</td>
  </tr>
  <tr>
    <td>MagiCoder-S-DS-6.7B</td>
    <td>0.732</td>
    <td>0.683</td>
    <td>0.767</td>
    <td>0.667</td>
  </tr>
  <tr>
    <td>+ Our Focused-DPO</td>
    <td><b>0.823</b></td>
    <td><b>0.744</b></td>
    <td><b>0.794</b></td>
    <td><b>0.698</b></td>
  </tr>
  <tr>
    <td>Relative Improvement</td>
    <td>12.50%</td>
    <td>8.93%</td>
    <td>3.45%</td>
    <td>4.76%</td>
  </tr>
  <tr>
    <td>DPO / Step-DPO</td>
    <td>0.762</td>
    <td>0.701</td>
    <td>0.772</td>
    <td>0.675</td>
  </tr>
  <tr>
    <td>Token-DPO</td>
    <td>0.811</td>
    <td>0.732</td>
    <td>0.780</td>
    <td>0.680</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>0.738</td>
    <td>0.701</td>
    <td>0.762</td>
    <td>0.653</td>
  </tr>
  <tr>
    <th colspan="5">Base Model</th>
  </tr>
  <tr>
    <td>Qwen2.5-coder-base</td>
    <td>0.835</td>
    <td>0.787</td>
    <td>0.794</td>
    <td>0.685</td>
  </tr>
  <tr>
    <td>+ Our Focused-DPO</td>
    <td><b>0.884</b></td>
    <td><b>0.829</b></td>
    <td><b>0.817</b></td>
    <td><b>0.794</b></td>
  </tr>
  <tr>
    <td>Relative Improvement</td>
    <td>5.89%</td>
    <td>5.37%</td>
    <td>2.95%</td>
    <td>3.03%</td>
  </tr>
  <tr>
    <td>DPO / Step-DPO</td>
    <td>0.848</td>
    <td>0.790</td>
    <td>0.802</td>
    <td>0.688</td>
  </tr>
  <tr>
    <td>Token-DPO</td>
    <td>0.866</td>
    <td>0.799</td>
    <td>0.815</td>
    <td>0.690</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>0.848</td>
    <td>0.805</td>
    <td>0.802</td>
    <td>0.688</td>
  </tr>
  <tr>
    <td>DeepSeekCoder-base-6.7B</td>
    <td>0.476</td>
    <td>0.396</td>
    <td>0.702</td>
    <td>0.566</td>
  </tr>
  <tr>
    <td>+ Our Focused-DPO</td>
    <td><b>0.518</b></td>
    <td><b>0.427</b></td>
    <td><b>0.717</b></td>
    <td><b>0.574</b></td>
  </tr>
  <tr>
    <td>Relative Improvement</td>
    <td>8.89%</td>
    <td>7.79%</td>
    <td>2.13%</td>
    <td>1.43%</td>
  </tr>
  <tr>
    <td>DPO / Step-DPO</td>
    <td>0.488</td>
    <td>0.396</td>
    <td>0.709</td>
    <td>0.569</td>
  </tr>
  <tr>
    <td>Token-DPO</td>
    <td>0.500</td>
    <td>0.421</td>
    <td>0.717</td>
    <td>0.574</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>0.488</td>
    <td>0.396</td>
    <td>0.704</td>
    <td>0.566</td>
  </tr>
</table>

Table 3: Pass Rate on HumanEval(+), MBPP(+)
<table>
  <tr>
    <th>Model</th>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
    <th>Average</th>
  </tr>
  <tr>
    <td colspan="5"><i>Instruct Model</i></td>
  </tr>
  <tr>
    <td><b>Qwen2.5-coder-instruct-7B</b></td>
    <td>0.692</td>
    <td>0.220</td>
    <td>0.034</td>
    <td>0.312</td>
  </tr>
  <tr>
    <td>+ Our Focused-DPO</td>
    <td><b>0.735</b></td>
    <td><b>0.242</b></td>
    <td><b>0.048</b></td>
    <td><b>0.339</b></td>
  </tr>
  <tr>
    <td><i>Relative Improvement</i></td>
    <td>6.22%</td>
    <td>10.04%</td>
    <td>42.86%</td>
    <td>8.44%</td>
  </tr>
  <tr>
    <td>DPO / Step-DPO</td>
    <td>0.685</td>
    <td>0.233</td>
    <td>0.019</td>
    <td>0.310</td>
  </tr>
  <tr>
    <td>Token-DPO</td>
    <td>0.706</td>
    <td>0.239</td>
    <td>0.037</td>
    <td>0.325</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>0.670</td>
    <td>0.208</td>
    <td>0.015</td>
    <td>0.295</td>
  </tr>
  <tr>
    <td><b>DeepSeekCoder-instruct-6.7B</b></td>
    <td>0.453</td>
    <td>0.091</td>
    <td>0.009</td>
    <td>0.181</td>
  </tr>
  <tr>
    <td>+ Our Focused-DPO</td>
    <td><b>0.477</b></td>
    <td><b>0.106</b></td>
    <td><b>0.019</b></td>
    <td><b>0.197</b></td>
  </tr>
  <tr>
    <td><i>Relative Improvement</i></td>
    <td>5.30%</td>
    <td>15.89%</td>
    <td>108.33%</td>
    <td>8.87%</td>
  </tr>
  <tr>
    <td>DPO / Step-DPO</td>
    <td>0.462</td>
    <td>0.094</td>
    <td>0.007</td>
    <td>0.184</td>
  </tr>
  <tr>
    <td>Token-DPO</td>
    <td>0.470</td>
    <td>0.100</td>
    <td>0.019</td>
    <td>0.192</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>0.462</td>
    <td>0.094</td>
    <td>0.004</td>
    <td>0.183</td>
  </tr>
  <tr>
    <td><b>MagiCoder-S-DS-6.7B</b></td>
    <td>0.481</td>
    <td>0.107</td>
    <td>0.001</td>
    <td>0.193</td>
  </tr>
  <tr>
    <td>+ Our Focused-DPO</td>
    <td><b>0.513</b></td>
    <td><b>0.118</b></td>
    <td><b>0.019</b></td>
    <td><b>0.213</b></td>
  </tr>
  <tr>
    <td><i>Relative Improvement</i></td>
    <td>6.56%</td>
    <td>10.12%</td>
    <td>1751.85%</td>
    <td>10.10%</td>
  </tr>
  <tr>
    <td>DPO / Step-DPO</td>
    <td>0.491</td>
    <td>0.109</td>
    <td>0.004</td>
    <td>0.198</td>
  </tr>
  <tr>
    <td>Token-DPO</td>
    <td>0.505</td>
    <td>0.118</td>
    <td>0.015</td>
    <td>0.209</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>0.498</td>
    <td>0.112</td>
    <td>0.004</td>
    <td>0.201</td>
  </tr>
  <tr>
    <td colspan="5"><i>Base Model</i></td>
  </tr>
  <tr>
    <td><b>Qwen2.5-coder-base-7B</b></td>
    <td>0.567</td>
    <td>0.150</td>
    <td>0.017</td>
    <td>0.241</td>
  </tr>
  <tr>
    <td>+ Our Focused-DPO</td>
    <td><b>0.595</b></td>
    <td><b>0.175</b></td>
    <td><b>0.030</b></td>
    <td><b>0.264</b></td>
  </tr>
  <tr>
    <td><i>Relative Improvement</i></td>
    <td>5.00%</td>
    <td>16.47%</td>
    <td>77.78%</td>
    <td>9.23%</td>
  </tr>
  <tr>
    <td>DPO / Step-DPO</td>
    <td>0.577</td>
    <td>0.151</td>
    <td>0.015</td>
    <td>0.244</td>
  </tr>
  <tr>
    <td>Token-DPO</td>
    <td>0.584</td>
    <td>0.163</td>
    <td>0.022</td>
    <td>0.253</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>0.584</td>
    <td>0.157</td>
    <td>0.022</td>
    <td>0.251</td>
  </tr>
  <tr>
    <td><b>DeepSeekCoder-base-6.7B</b></td>
    <td>0.399</td>
    <td>0.074</td>
    <td>0.004</td>
    <td>0.155</td>
  </tr>
  <tr>
    <td>+ Our Focused-DPO</td>
    <td><b>0.423</b></td>
    <td><b>0.085</b></td>
    <td><b>0.011</b></td>
    <td><b>0.169</b></td>
  </tr>
  <tr>
    <td><i>Relative Improvement</i></td>
    <td>6.00%</td>
    <td>14.31%</td>
    <td>177.78%</td>
    <td>9.24%</td>
  </tr>
  <tr>
    <td>DPO / Step-DPO</td>
    <td>0.412</td>
    <td>0.079</td>
    <td>0.004</td>
    <td>0.161</td>
  </tr>
  <tr>
    <td>Token-DPO</td>
    <td>0.419</td>
    <td>0.079</td>
    <td>0.004</td>
    <td>0.164</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>0.419</td>
    <td>0.082</td>
    <td>0.007</td>
    <td>0.166</td>
  </tr>
</table>

Table 4: Pass Rate on LiveCodeBench

<table>
  <tr>
    <th>Model</th>
    <th>HumanEval</th>
    <th>HumanEval+</th>
    <th>MBPP</th>
    <th>MBPP+</th>
  </tr>
  <tr>
    <td>DeepSeekCoder-base-6.7B</td>
    <td>0.4760</td>
    <td>0.3960</td>
    <td>0.7020</td>
    <td>0.5660</td>
  </tr>
  <tr>
    <td>+ SFT Stage<br>(with MagiCoder-OSS-instruct)</td>
    <td>0.7317</td>
    <td>0.6829</td>
    <td>0.7672</td>
    <td>0.6667</td>
  </tr>
  <tr>
    <td>+ First DPO Stage<br>(with CodeDPO-OSS-instruct)</td>
    <td>0.8354</td>
    <td>0.7622</td>
    <td>0.8070</td>
    <td>0.7093</td>
  </tr>
  <tr>
    <td>+ Focused-DPO</td>
    <td><b>0.8719</b></td>
    <td><b>0.7926</b></td>
    <td><b>0.8227</b></td>
    <td><b>0.7275</b></td>
  </tr>
</table>

Table 5: Performance of DeepSeekCoder-6.7B at different training stages. The stages include base model, SFT with MagiCoder, first DPO with CodeDPO, and our Focused-DPO. Focused-DPO achieves additional improvements even after high-quality post-training.

results demonstrate that Focused-DPO effectively boosts the performance of models that have already been extensively post-trained and optimized through previous stages. We further evaluate how Focused-DPO enhances the quality of error-prone points in Appendix 6.2.

5.3 Ablation Study (RQ3)

Dataset Construction Ablations Focused-DPO includes an automated data construction and Error-Prone Identification process. We perform ablation experiments on the dataset construction methods in Table 6. We design two alternative approaches:
① The Step-DPO strategy (Lai et al., 2024) constructs datasets by considering only the common prefix parts, with the rest treated as Error-Prone Points for training. ② Using a git-diff tool 1, we construct datasets where the differences covered by the diff were treated as Error-Prone Points, with the parts following the final diff difference treated as the suffix. Note that Step-DPO dataset construction method is closely tied to the formulation of the Step-DPO loss function, leading to consistent outcomes between the two. However, we observe that

Step-DPO performs suboptimally on code generation tasks. In contrast, the current dataset construction method used in Focused-DPO, which employs a simple yet effective Error-Prone Identification strategy, achieves the best experimental results.

Loss Function Ablations Our Focused-DPO has made appropriate modifications to the calculation of positive and negative rewards. We carry out ablation experiments in Table 6, including trying different values of \( w_{focused} \) and various treatments of the suffix in the reward function. Our findings indicate that increasing or decreasing \( w_{focused} \) leads to a decline in model performance, suggesting that the current value of \( w_{focused} \) is optimal. Additionally, we observe that including the suffix part in the reward function results in degraded performance. Through detailed analysis in Section 5.1, the suffix in incorrect code does not exhibit strong correlations with the overall accuracy. These experiments validate the practical advantages of the design choices in our loss function.

<table>
  <tr>
    <th>Dataset Construction</th>
    <th>HumanEval / HumanEval+</th>
    <th>MBPP / MBPP+</th>
  </tr>
  <tr>
    <td>Focused-DPO</td>
    <td>0.9268 / 0.8780</td>
    <td>0.8466 / 0.7619</td>
  </tr>
  <tr>
    <td>Error Prone Identification</td>
    <td>0.9268 / 0.8780</td>
    <td>0.8466 / 0.7619</td>
  </tr>
  <tr>
    <td>Step-DPO Strategy</td>
    <td>0.9207 / 0.8537</td>
    <td>0.8413 / 0.7434</td>
  </tr>
  <tr>
    <td>Diff-based Strategy</td>
    <td>0.9268 / 0.8598</td>
    <td>0.8439 / 0.7539</td>
  </tr>
  <tr>
    <th colspan="3">Loss Function Setting</th>
  </tr>
  <tr>
    <th>HumanEval / HumanEval+</th>
    <th>MBPP / MBPP+</th>
  </tr>
  <tr>
    <td>Focused-DPO</td>
    <td>0.9268 / 0.8780</td>
    <td>0.8466 / 0.7619</td>
  </tr>
  <tr>
    <td>\( w_{focused} = 2 \), No Suffix in Reject Reward</td>
    <td>0.9268 / 0.8780</td>
    <td>0.8466 / 0.7619</td>
  </tr>
  <tr>
    <td>Decrease Weight<br>\( w_{focused} = 1 \)</td>
    <td>0.9268 / 0.8720</td>
    <td>0.8386 / 0.7487</td>
  </tr>
  <tr>
    <td>Increase Weight<br>\( w_{focused} = 3 \)</td>
    <td>0.9268 / 0.8720</td>
    <td>0.8439 / 0.7566</td>
  </tr>
  <tr>
    <td>\( w_{focused} = 5 \)</td>
    <td>0.8963 / 0.7683</td>
    <td>0.8201 / 0.6878</td>
  </tr>
  <tr>
    <td>Suffix in Reject Reward</td>
    <td>0.9268 / 0.8659</td>
    <td>0.8413 / 0.7487</td>
  </tr>
</table>

Table 6: Dataset Construction and Loss Function Ablation Results based on Qwen2.5-Coder-Instruct-7B

1https://git-scm.com/docs/git-diff
6 Discussions

6.1 Error-Prone Points Identification
In our Focused-DPO method, we introduce a dataset construction technique called Error Prone Identification to automatically identify error-prone points in generated code. To assess the correctness of the code, we employ a self-generation-and-validation mechanism based on PageRank, which captures the relative quality of different code snippets (Zhang et al., 2024b). We are not like approaches such as Magicoder (Wei et al., 2023b), which directly use all test cases as ground truth. In our experiments we use the policy model to generate datasets. Since the policy model’s generation quality is not as robust as that of more powerful models like GPT-4 (used in Magicoder), the PageRank-based method allows us to automatically filter out lower-quality test cases (those with lower scores after iteration), thereby ensuring higher overall dataset quality.

We find that different models exhibit varying levels of accuracy across different problems. Therefore, for each model’s training dataset, we performed necessary filtering by removing code problems with excessively high or low accuracy rates, ensuring a consistent number of code problems in the final dataset. Moreover, we observe that models tend to exhibit similarities in error-prone points when solving the same problems. For example, when comparing the error-prone points identified by DeepSeekCoder-instruct-6.7B and Qwen2.5-Coder-instruct-7B models on the same set of programming problems, we found a 32% overlap. This indicates that there are commonalities in the error-prone points across different models.

In our ablation studies, we compare error-prone points constructed using the git-diff method and the Step-DPO method, noting slight differences in the final results. Balancing effectiveness and efficiency, we use the method based on prefix and suffix, which allows us to identify error-prone points in generated code in a simple yet effective manner. We plan to further explore more identification strategies in future work.

6.2 Improvement in Error-Prone Points
We further evaluate how Focused-DPO enhances the quality in error-prone points. Using our validation dataset (Table 8), we measure the model’s performance on these error-prone parts. The generation probability difference between chosen_mid and reject_mid in error-prone points is illustrated in Figure 4 for Qwen2.5-Coder-Instruct-7B model.

Compared to pre-Focused-DPO results (Figure 3), Focused-DPO demonstrates a strong preference for generating more accurate code at error-prone points. This improvement is particularly critical in complex coding tasks, where precise decisions in error-prone points directly impact the correctness of the generated code. For instance, on the LiveCodeBench-Hard dataset—which consists of challenging, dynamically problems—Focused-DPO achieves a significant improvement of 42.8% in correctness for the Qwen2.5-Coder-Instruct model. Notably, on this dataset, Focused-DPO achieves performance on par with GPT-4o, highlighting its ability to address difficult code generation tasks effectively.

![Distribution of Generation Probability Differences](page_1012_1092_377_246.png)

Figure 4: Generation Probability Difference (p(chose_mid) - p(reject_mid)) after Focused-DPO.

7 Conclusion
We propose Focused-DPO, a framework that improves code generation by focusing on error-prone points. These critical parts significantly impact overall program correctness. Focused-DPO improves Direct Preference Optimization by prioritizing these points, using our Error-Point Identification method to create datasets without costly human annotations. Evaluations show Focused-DPO reduces errors and improves quality, even in heavily post-trained models. This research highlights benefits of focusing on fine-grained preference optimization in AI-driven software development.

8 Acknowledgement
This research is supported by the National Natural Science Foundation of China under Grant Nos. 62192731, 62192730, 62192733, 62072007, the National Key R&D Program under Grant No. 2023YFB4503801, and the Major Program (JD) of Hubei Province (No.2023BAA024).

2This work is based on our prior findings in CodeDPO (Zhang et al., 2024b). We are deeply grateful to Jingjing Xu, Jing Su, and Jun Zhang for their valuable discussions.
Limitation

Despite the contributions of our work, there are several limitations that we aim to address in future research:

Comparison with Advanced RL Techniques
While our study demonstrates the effectiveness of Focused-DPO, we do not extensively compare it with other advanced reinforcement learning (RL) alignment techniques, such as DeepSeek-R1 (Guo et al., 2025). These online RL alignment techniques typically require substantial training resources, high-quality datasets, and complex reward environments, making their application highly resource-intensive. In contrast, offline alignment methods such as Focused-DPO approximate similar optimization objectives while introducing necessary simplifications and derivations. This allows Focused-DPO to achieve comparable or even equivalent optimization results with significantly lower resource requirements. Moreover, we leverage prior knowledge discovered in this work: the insight that only a small part of the generated code—specifically, the Error-Prone Points—plays a critical role in determining the overall correctness of the output. By incorporating this insight into the training loss design, we further enhance training efficiency and effectiveness. Focused-DPO’s low resource requirements and reliable performance make it applicable to a wide range of code generation scenarios. Further exploration of how Focused-DPO compares to these advanced RL techniques in performance and efficiency remains an area for future investigation.

Dataset Construction Strategy
In Focused-DPO, we introduce a dataset construction technique named Error-Prone Identification, which automatically identifies error-prone points in generated code. The primary focus of this paper is on error-prone points associated with correctness in the final output code. However, other factors in source code, such as efficiency, readability, and security, are equally important for optimization. Exploring whether these factors also reveal "Error-Prone Points" in source code is an intriguing direction for future work. For example, techniques like static code analysis, code smells detection, and identification of common vulnerabilities could help identify and penalize insecure patterns during data construction, leading to safer and more robust code generation.

Additionally, our dataset construction pipeline includes specific design choices, such as the use of a page-rank mechanism and the identification of error-prone points based on common prefixes and suffixes. Our preliminary experiments suggest that these settings effectively support the performance of Focused-DPO. Detailed discussions on these designs are provided in Section 6.

References

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Shihao Dou, Yan Liu, Haoxiang Jia, Enyu Zhou, Limao Xiong, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Stepcode: Improving code generation with reinforcement learning from compiler feedback. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 4571–4585.

Leonidas Gee, Milan Gritta, Gerasimos Lampouras, and Ignacio Iacobacci. 2024. Code-optimise: Self-generated preference data for correctness and efficiency. CoRR, abs/2406.12502.

GPT-4. 2023. https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo. OpenAI.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming—the rise of code intelligence. arXiv preprint arXiv:2401.14196.

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, An Yang, Rui Men, Fei Huang, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. Qwen2.5-coder technical report. CoRR, abs/2409.12186.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.

Siyuan Jiang, Jia Li, He Zong, Huanyu Liu, Hao Zhu, Shukai Hu, Erli Li, Jiazheng Ding, Yu Han, Wei Ning, Gen Wang, Yihong Dong, Kechi Zhang, and Ge Li. 2024. aixcoder-7b: A lightweight and effective large language model for code completion. CoRR, abs/2410.13187.

Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629.

Raymond Li, Louba Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161.

Zicheng Lin, Tian Liang, Jiahao Xu, Xing Wang, Ruijin Luo, Chufan Shi, Siheng Li, Yujiu Yang, and Zhaopeng Tu. 2024. Critical tokens matter: Token-level contrastive estimation enhances llm’s reasoning capability. CoRR, abs/2411.19943.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024a. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36.

Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. 2024b. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405.16436.

Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. 2024. Step-controlled DPO: leveraging stepwise error for enhanced mathematical reasoning. CoRR, abs/2407.00782.

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568.

Rafael Rafaïlov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560.

Yuxiang Wei, Federico Cassano, Jiawei Liu, Yifeng Ding, Naman Jain, Harm de Vries, Leandro von Werra, Arijun Guha, and Lingming Zhang. 2023a. Starcoder2-instruct: Fully transparent and permissive self-alignment for code generation. BigCode.

Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023b. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120.

Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. 2024. Token-level direct preference optimization. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024.

Dylan Zhang, Shizhe Diao, Xueyan Zou, and Hao Peng. 2024a. PLUM: preference learning plus test cases yields better code language models. CoRR, abs/2406.06887.

Kechi Zhang, Ge Li, Yihong Dong, Jingjing Xu, Jun Zhang, Jing Su, Yongfei Liu, and Zhi Jin. 2024b. Codedpo: Aligning code models with self generated and verified source code. CoRR, abs/2410.05605.

Kechi Zhang, Ge Li, Huangzhao Zhang, and Zhi Jin. 2024c. Hirope: Length extrapolation for code models using hierarchical position. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13615–13627.

Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024d. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13643–13658.

Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023. Self-edit: Fault-aware code editor for code generation. In The 61st Annual Meeting Of The Association For Computational Linguistics.
A  Data Scaling For Focused-DPO

In our experiments, we use the policy model to sample the dataset for training, with the dataset statistics provided in Table 8. We also explore how scaling the training data affects the final performance of Focused-DPO. Specifically, we investigate two additional settings: doubling the original training dataset to 10k samples and halving the dataset to 2.5k samples, to observe how these changes impact the effectiveness of the model after Focused-DPO training. The experimental results are presented in Table 7. The results indicate that fine-grained preference optimization converges efficiently within our given data range, and increasing the dataset size does not significantly improve the results.

<table>
  <tr>
    <th>Data Scaling</th>
    <th>HumanEval / HumanEval+</th>
    <th>MBPP / MBPP+</th>
  </tr>
  <tr>
    <td>Qwen2.5-coder-instruct-7B</td>
    <td>0.915 / 0.841</td>
    <td>0.828 / 0.714</td>
  </tr>
  <tr>
    <td>Focused-DPO (5k)</td>
    <td>0.926 / 0.878</td>
    <td>0.846 / 0.761</td>
  </tr>
  <tr>
    <td>Decrease to 2.5k</td>
    <td>0.926 / 0.847</td>
    <td>0.830 / 0.719</td>
  </tr>
  <tr>
    <td>Increase to 10k</td>
    <td>0.926 / 0.878</td>
    <td>0.843 / 0.756</td>
  </tr>
</table>

Table 7: Dataset Scaling for Focused-DPO based on Qwen2.5-Coder-Instruct-7B

B  Dataset Statistics

<table>
  <tr>
    <th>Dataset</th>
    <th>Problems</th>
    <th>Avg. Hidden Tests</th>
  </tr>
  <tr>
    <td>HumanEval</td>
    <td>164</td>
    <td>9.57</td>
  </tr>
  <tr>
    <td>HumanEval+</td>
    <td></td>
    <td>748.07</td>
  </tr>
  <tr>
    <td>MBPP</td>
    <td>378</td>
    <td>3.11</td>
  </tr>
  <tr>
    <td>MBPP+</td>
    <td></td>
    <td>105.40</td>
  </tr>
  <tr>
    <td>Easy</td>
    <td>279</td>
    <td>18.07</td>
  </tr>
  <tr>
    <td>LiveCodeBench</td>
    <td>Medium</td>
    <td>331</td>
    <td>21.81</td>
  </tr>
  <tr>
    <td></td>
    <td>Hard</td>
    <td>270</td>
    <td>24.78</td>
  </tr>
</table>

Table 8: Statistics of Evaluation Benchmark.

<table>
  <tr>
    <th colspan="2">Statistics based on Qwen2.5-Coder-Instruct-7B</th>
  </tr>
  <tr>
    <th>Problems</th>
    <th></th>
  </tr>
  <tr>
    <td>Training Set</td>
    <td>5000</td>
  </tr>
  <tr>
    <td>Validation Set</td>
    <td>1000</td>
  </tr>
  <tr>
    <th colspan="2">Average Token Lengths</th>
  </tr>
  <tr>
    <td>Common Prefix</td>
    <td>78.17</td>
  </tr>
  <tr>
    <td>Common Suffix</td>
    <td>33.98</td>
  </tr>
  <tr>
    <td>Chosen Mid</td>
    <td>57.37</td>
  </tr>
  <tr>
    <td>of Total Chosen Code</td>
    <td>34%</td>
  </tr>
  <tr>
    <td>Rejected Mid</td>
    <td>42.63</td>
  </tr>
  <tr>
    <td>of Total Rejected Code</td>
    <td>28%</td>
  </tr>
</table>

Table 9: Training Dataset Statistics based on Qwen2.5-Coder-Instruct-7B

C  Scalability and Dataset Independence

We aim for our proposed method to be scalable and minimally dependent on external prompt resources. Inspired by the self-OSS methodology, we envision that both question generation and solution alignment can be achieved through self-alignment and self-improvement using the policy model itself—a key step toward our ultimate goal.

On the other hand, our method is intentionally designed to avoid being constrained by specific datasets, unlike approaches such as Code-Optimise. Our aspiration is to develop a stable and robust pathway for models to continually enhance their capabilities autonomously.

In our paper, Focused-DPO achieves improvements using preference feedback directly from the model itself. This avoids potential risks associated with externally distilled data, such as unintentionally transferring knowledge from a more capable model to a less capable one.

To further validate the effectiveness of Focused-DPO, we plan to extend our evaluation to existing code datasets. We have adopted the sample datasets from CodeContests. We use the provided test cases to verify the solutions and filter out those samples that have no correct solutions. The final sample number in this dataset setting is nearly 4500. The experiment results are shown in the following Table 10.

<table>
  <tr>
    <th>Model</th>
    <th>HumanEval</th>
    <th>HumanEval+</th>
    <th>MBPP</th>
    <th>MBPP+</th>
    <th>LoveCodeBench</th>
  </tr>
  <tr>
    <td>Qwen2.5-coder-instruct-7B</td>
    <td>0.915</td>
    <td>0.841</td>
    <td>0.828</td>
    <td>0.714</td>
    <td>0.312</td>
  </tr>
  <tr>
    <td>* CodeContests dataset</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Focused-DPO</td>
    <td>0.920</td>
    <td>0.860</td>
    <td>0.844</td>
    <td>0.749</td>
    <td>0.334</td>
  </tr>
  <tr>
    <td>DPO</td>
    <td>0.912</td>
    <td>0.847</td>
    <td>0.833</td>
    <td>0.725</td>
    <td>0.308</td>
  </tr>
</table>

Table 10: Focused-DPO and DPO Results on CodeContests-train

These results demonstrate that Focused-DPO is both dataset-agnostic and scalable. We plan to expand testing across additional datasets like APPs in future work to further validate reproducibility and fair comparisons.

D  Evaluation of k’s Impact

We conducted experiments to measure the consistency of error-prone regions identified across sampled code sequences for varying \( k \) values, as shown in Figure 11. The Phi coefficient results reveal that moderate \( k \) values (e.g., \( k = 10 \)) provide the most consistent identification of error-prone areas.

We further validated the impact of \( k \) by conducting training experiments with different values. The
<table>
  <tr>
    <th>k Value</th>
    <th>Strategy</th>
    <th>Correct</th>
    <th>Incorrect</th>
    <th>Phi Coefficient</th>
  </tr>
  <tr>
    <td rowspan="2">k = 5</td>
    <td>Common Prefix + Chosen Mid</td>
    <td>0.5926</td>
    <td>0.0935</td>
    <td>0.5237</td>
  </tr>
  <tr>
    <td>Common Prefix + Reject Mid</td>
    <td>0.0132</td>
    <td>0.4629</td>
    <td>-0.5314</td>
  </tr>
  <tr>
    <td rowspan="2">k = 10</td>
    <td>Common Prefix + Chosen Mid</td>
    <td>0.6367</td>
    <td>0.0911</td>
    <td>0.5651</td>
  </tr>
  <tr>
    <td>Common Prefix + Reject Mid</td>
    <td>0.0116</td>
    <td>0.5575</td>
    <td>-0.6085</td>
  </tr>
  <tr>
    <td rowspan="2">k = 20</td>
    <td>Common Prefix + Chosen Mid</td>
    <td>0.6521</td>
    <td>0.0975</td>
    <td>0.5710</td>
  </tr>
  <tr>
    <td>Common Prefix + Reject Mid</td>
    <td>0.0102</td>
    <td>0.5473</td>
    <td>-0.6025</td>
  </tr>
</table>

Table 11: Phi Coefficient for different \( k \) settings

results in Figure 12 indicate that moderate \( k \) values yield stable performance improvements across benchmarks.

<table>
  <tr>
    <th>k Value</th>
    <th>HumanEval</th>
    <th>HumanEval+</th>
    <th>MBPP</th>
    <th>MBPP+</th>
    <th>LiveCodeBench Avg.</th>
  </tr>
  <tr>
    <td>k = 5</td>
    <td>0.920</td>
    <td>0.871</td>
    <td>0.835</td>
    <td>0.748</td>
    <td>0.336</td>
  </tr>
  <tr>
    <td>k = 10</td>
    <td>0.927</td>
    <td>0.878</td>
    <td>0.847</td>
    <td>0.762</td>
    <td>0.339</td>
  </tr>
  <tr>
    <td>k = 20</td>
    <td>0.927</td>
    <td>0.878</td>
    <td>0.838</td>
    <td>0.751</td>
    <td>0.339</td>
  </tr>
</table>

Table 12: Benchmark results for different \( k \) settings based on Qwen2.5-coder-instruct-7B

E Suffix in Positive Loss

In the original DPO loss, penalizing inappropriate negative samples tends to significantly harm performance. However, retaining a portion of the positive sample learning component improves the robustness and stability of model training. Several existing works, including (Liu et al., 2024b), have analyzed this phenomenon theoretically. We have also empirically verified this observation in our preliminary experiments. The results, presented in Table 13, demonstrate consistent performance improvements.

<table>
  <tr>
    <th>Model</th>
    <th>HumanEval</th>
    <th>HumanEval+</th>
    <th>MBPP</th>
    <th>MBPP+</th>
    <th>LiveCodeBench</th>
  </tr>
  <tr>
    <td>Qwen2.5-coder-instruct-7B</td>
    <td>0.915</td>
    <td>0.841</td>
    <td>0.828</td>
    <td>0.714</td>
    <td>0.312</td>
  </tr>
  <tr>
    <td>Focused-DPO</td>
    <td>0.927</td>
    <td>0.878</td>
    <td>0.847</td>
    <td>0.762</td>
    <td>0.339</td>
  </tr>
  <tr>
    <td>No Suffix in Positive Reward</td>
    <td>0.927</td>
    <td>0.872</td>
    <td>0.844</td>
    <td>0.760</td>
    <td>0.334</td>
  </tr>
</table>

Table 13: Benchmark Results Comparison for “No Suffix in Positive Reward”

F Case Studies for Error-Prone Points

We show some case studies for error-prone points based on *Qwen2.5-Coder-instruct* in the following Figure 5, 6 and 7.
Question Prompt
Write a Python function that takes a string as input, and returns a valid variable name in Python. The function should perform the following steps:
1. Remove all invalid characters from the string, such as spaces, hyphens, and special characters.
2. Split the string into a list of words.
3. Capitalize the first letter of each word.
4. Join the words back together into a string.
5. Prepend the resulting string with an underscore (_) if it begins with a number.
6. Return the resulting string as a valid variable name.
This problem is useful for converting user-provided input into a valid variable name for use in Python programs.

Correct Solution
def sol1(string):
    string = re.sub(r'[\W\s]', '', string)
    words = string.split()
    words = [word.capitalize() for word in words]
    string = ''.join(words)
    if string[0].isdigit():
        string = "_" + string
    return string

InCorrect Solution
def sol2(string):
    string = re.sub(r'[\W_]+', '', string)
    words = string.split()
    words = [word.capitalize() for word in words]
    string = ''.join(words)
    if string[0].isdigit():
        string = "_" + string
    return string

Figure 5: Case Study: convert to valid variable name

Question Prompt
Write a Python function 'parse_version' that takes a string representing a version number and returns a list of integers representing the version number. Each integer should correspond to a number in the version string. Note: This function should handle both a dot-separated and a dash-separated version string.

Correct Solution
def sol1(version_string):
    version = version_string.replace('-', ':').split(':')
    return [int(v) for v in version]

InCorrect Solution
def sol2(version_string):
    version = version_string.split('-')
    if len(version) == 4:
        version[-1], _ = version[-1].split(':')
    return [int(v) for v in version]

Figure 6: Case Study: parse version

Question Prompt
Implement a Python function to find the maximal value of the product of elements of a list 'array' given a sliding window size 'k'. The function should take two arguments: 'array' and 'k', where 'k' is optional and defaults to 2. The function should return an integer representing the maximal value of the product.

Correct Solution
def sol1(array, k=2):
    if k > len(array):
        raise ValueError("k cannot be greater than the length of the array")
    products = [1] * len(array)
    products[0] = array[0]
    max_product = products[0]
    for i in range(k):
        product = array[i]
        max_product = max(max_product, product)
        for j in range(i+1, len(array)):
            product *= array[j] // array[j - k]
            products[j] = product
            max_product = max(max_product, product)
    return max_product

InCorrect Solution
def sol2(array, k=2) -> int:
    if k > len(array):
        raise ValueError("k cannot be greater than the length of the array")
    products = [1] * len(array)
    for i in range(k):
        products[i] = array[i]
        max_product = max(products[i])
        for j in range(i+1, len(array)):
            products[j] = products[j-1] * array[j] / array[j-k]
            max_product = max(max_product, products[j])
    return max_product

Figure 7: Case Study: max product

Reviewer paper summaries: The paper proposed Focused-DPO, which focused on error-prone points. The method first identifies the error-prone points of negative code samples. Then the method employed a modified DPO loss to fine tune code LLMs. Experiments on HumanEval, MBPP and LiveCodeBench demonstrated the effectiveness of the proposed method.

The aim of the work is to improve code generation model with a focus on tackling error-prone points in generated code. There are mainly two parts in the contribution: 1. identify the error prone points/parts in the generated codes, so as to collect a dataset of paired correct/incorrect codes with only the middle section being different; 2. design a simple DPO-variant, i.e., Focused-DPO to force the model refinement to pay adjustable attention on the difference. The effectiveness of the claim is backed by various experiments.
Summary of strengths: 1. The paper focused on an important field.
2. The writing is clear overall and easy to follow.
3. Extensive experiments demonstrated the effectiveness.

The data generation method as well as the proposed Focused-DPO are both simple and should be easy to reimplement.
The improvement over the baselines seems valid.
In general, the paper is well written and easy to follow, although there are unclear bits here and there which are listed in the weaknesses.
Summary of weaknesses: 1. The code and dataset are not open-sourced.
2. The Section 3.2 is a little confusing. 
  - What is the number of questions in the seed dataset?
  - How did you extract the common prefix and common suffix? just string match? If there was a little difference between the chosen and the rejected code, such as “\n” and “\n\n”, how did you handle it?
  - How did you choose the final 5,000 training sets? question level or pair level?

From my understanding, it is the core contribution of this paper. So I recommend using figures and a case to illustrate the algorithm.

3. The analysis experiments are not constructed on LiveCodeBench, which is a more convincing dataset.
4. It seems your method can be applied to any dataset. Can you test your method on some open-sourced datasets such as APPs and Code Contest, for reproducibility and fair comparison? 

https://huggingface.co/datasets/codeparrot/apps

https://huggingface.co/datasets/deepmind/code_contests

If the suffix is trivial on the overall accuracy of the final code, why only omitting it in the phi_rej not the phi_correct in Focused-DPO? This makes the proposed loss a bit random.

Eq 1 is not explained clearly. For example, what the scores at t = 0? If Link(t_j, c_i) means if c_i passes t_j or not, what does Link(c_i, t_j) mean? Also, what is score on test cases, t_j? I know it's a referenced algorithm, but it should be explained well in this work.

The analysis in the paragraph after Table 1 seems very obvious. Since the error is in the middle section, the suffix section is certainly not going to affect the correctness of the code. But, this doesn't give a convincing answer to my first question.

For Table 5, what kind of data is used for Focused-DPO training? The comparison seems unfair. Ideally, should compare between Focused-DPO and DPO on the exactly same amount of data.

Regarding the experiments on w_focused, could it be some rational number rather than the currently tried integers?
Comments, suggestions and typos: See Weaknesses

Maybe more words on what kind of open-source repos are used for the seed dataset.

If DPO and Step-DPO are identical for this scenario, there's no need to run experiments for both, in my opinion.
Overall assessment scores: 3

3
Metareview score: 3
Metareview: This paper proposes Focused-DPO, a variant of Direct Preference Optimization tailored for code generation. The method first identifies error-prone regions in generated code using a PageRank-inspired algorithm, then applies a modified loss that emphasizes these regions during model training. Experiments on HumanEval, MBPP, and LiveCodeBench show that focusing training on these bottleneck regions improves code generation accuracy.
Summary of reasons to publish: - The paper targets a crucial issue in code generation—localized errors in otherwise correct code. Identifying and training on these error-prone segments is a compelling idea that directly addresses real-world model weaknesses.

- The authors evaluate Focused-DPO across multiple datasets (HumanEval+, MBPP+, LiveCodeBench) and compare against various baselines (Token-DPO, Step-DPO), demonstrating competitive or superior performance.

- Despite some complexity in the reward formulation, the method itself is conceptually intuitive—focusing training on the "weakest links" in generated code—and is built on top of existing frameworks like DPO, which aids reproducibility.