Enhancing Machine Translation with Self-Supervised Preference Data
Authors: Haoxiang Sun, Ruize Gao, Pei Zhang, Baosong Yang, Rui Wang
Accepted at: acl25

Enhancing Machine Translation with Self-Supervised Preference Data

Haoxiang Sun1*, Ruize Gao2, Pei Zhang2, Baosong Yang2†, Rui Wang 1†
1Shanghai Jiao Tong University
2Tongyi Lab, Alibaba Group
1{sunny_sjtu,wangrui12}@sjtu.edu.cn
2{gaoruize.grz, xiaoyi.zp,yangbaosong.ybs}@alibaba-inc.com

Abstract
Model alignment methods like Direct Preference Optimization (Rafailov et al., 2024) and Contrastive Preference Optimization (Xu et al., 2024b) have enhanced machine translation performance by leveraging preference data to enable models to reject suboptimal outputs. During preference data construction, previous approaches primarily rely on humans, strong models like GPT4 (OpenAI, 2023) or model self-sampling. In this study, we first explain the shortcomings of this practice. Then, we propose Self-Supervised Preference Optimization (SSPO), a novel framework which efficiently constructs translation preference data for iterative DPO training. Applying SSPO to 14B parameters large language models (LLMs) achieves comparable or better performance than GPT-4o on FLORES and multi-domain test datasets. We release an augmented MQM dataset in https://github.com/sunny-sjtu/MQM-aug.

1 Introduction
Enhancing the capabilities of open source large language models (LLMs) (Bai et al., 2023; Touvron et al., 2023; Jiang et al., 2023) in machine translation has been extensively explored in previous research. ALMA (Xu et al., 2024a) and Aya 23 (Aryabumi et al., 2024) reach top-tier performance through continued pre-training on large monolingual corpora and supervised fine-tuning (SFT) on high-quality parallel translation data. While SFT lacks a mechanism to prevent the model from rejecting mistakes in translations (e.g. mistranslation, over-translation), model alignment methods like Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022), Direct Preference Optimization (DPO) (Rafailov et al., 2024) and Contrastive Preference Optimization (CPO) (Xu et al., 2024b) further improve machine translation performance by leveraging preference data to enable models to reject suboptimal outputs.

High-quality preference data is crucial for effective model alignment. Current approaches to constructing translation preference data typically rely on human annotations (Xu et al., 2024c; Ramos et al., 2024), model self-sampling (Yang et al., 2024b) or stronger models (Xu et al., 2024b).

This practice faces three major challenges: (1) high cost of querying humans or strong models (2) distributional discrepancy between positive and negative examples from different models, which leads to training instability. (3) insufficient quality contrast between self-sampled positive and negative examples, which weakens reward signals.

To address these challenges, we propose a self-supervised framework for moderate-sized models (~14B parameters) that constructs high-quality preference data without relying on stronger models or human annotations. Our key insight is to equip LLMs with three core capabilities: translation generation, error annotation, and error correction. This allows LLMs to utilize monolingual data by translating, identifying potential errors, and generating corrected versions. The quality gap between the initial and corrected translations naturally forms preference pairs for model alignment. This method also supports continuous improvement through an iterative refinement process. After generating error annotations, we further fine-tune the base model with these examples, creating a specialized error detector which becomes increasingly sensitive to common translation mistakes made by the current model. This enhanced detector provides targeted supervision for the DPO-aligned translation model. As the translation model improves via DPO training, the error detector adapts to new error patterns, progressively enhancing overall translation quality.

Our contributions are summarized as follows:
Figure 1: Overview of SSPO Framework. The process starts by training Model_0 with three types of data: parallel translation pairs, augmented MQM (Multidimensional Quality Metrics) annotations with error explanation and suggested correction, and error correction samples leveraging MQM annotations for corrected translations. The framework then iteratively improves through two paths: (1) SFT with self-generated MQM annotations and corrections to get stronger Error detector Model^E (2) DPO with <intial translation, corrected translation> preference pairs to get stronger Translator Model^T. Each iteration incorporates new monolingual data to expand domain coverage. Deeper model colors indicate enhanced capabilities.

• We propose SSPO, a self-supervised mechanism that enables LLMs to iteratively generate high-quality translation preference data for DPO training. SSPO’s effectiveness is validated across multiple languages, domains and models, achieving consistent improvements in translation performance without relying on external human or model annotations.

• We identify a good strategy for composing translation preference data, showing that integrating model-generated preferences with external high-quality data (from human experts or strong models) during DPO training yields superior performance compared to using either source alone.

• We release an augmented MQM annotation dataset to boost LLMs’ performance in translation-related tasks.

2 Self-supervised Preference Optimization

SSPO is a paradigm designed to generate high-quality translation preference data for iterative preference optimization. Figure 1 provides an overview of SSPO. We begin by describing the initialization of Model_0, the foundation of our framework.

2.1 Initialization of Model_0

Training Set. Model_0 is initialized using three complementary types of training data: parallel translation data, augmented MQM (Multidimensional Quality Metrics) annotations, and error correction data.

MQM offers a detailed assessment of translation quality by identifying specific error types, spans, and severity levels. For each error marked in the MQM annotations, we prompt GPT-4o* (OpenAI, 2024) for correction suggestions.

The error correction data is generated by prompting GPT-4o with the source text, initial translation, and MQM annotations to provide an improved translation. We show our prompts for training data construction in Appendix A.1.

Supervised fine-tuning with these data equips Model_0 with three key capabilities: translation generation, error annotation and error correction.

Design Rationale. We decompose the seemingly continuous chain of error annotation and correction into two separate tasks. This design is motivated by two key observations: (1) generating accurate MQM annotations is the most challenging part, requiring deep understanding of translation errors and improvements. (2) the correction process is straightforward, mainly applying MQM annotations to the initial translation. This separation focuses our iterative refinement on enhancing

*We use gpt-4o-0806 available from the OpenAI API.
the model’s error annotation capabilities, which is more crucial for generating high-quality preference data.

2.2 Self-Supervised Preference Data Construction

Overview. As depicted in Figure 2, Model_0 processes monolingual input through three sequential steps:

1. Translation Generation. For source text \( \mathbf{x} \sim \mathcal{D}_i \), Model_0 generates an initial translation \( \mathbf{y} = \mathrm{Model}_0(\mathbf{x}) \). \( \mathcal{D}_i \) denotes the distribution of monolingual texts in iteration \( i \).

2. Error Annotation. The model then performs error analysis by generating MQM-style annotations \( \{e_1, ..., e_n\} = \mathrm{Model}_0(\mathbf{x}, \mathbf{y}) \). If no errors are detected, it outputs “There is no error in the translation.” and skips the correction step. Otherwise, each identified error \( e_i \) is detailed as a tuple \( \{\mathbf{loc}_i, \mathbf{sev}_i, \mathbf{exp}_i, \mathbf{sugg}_i\} \) compromising:

• \( \mathbf{loc}_i \): Erroneous text span.
• \( \mathbf{sev}_i \): Error severity (major/minor).
• \( \mathbf{exp}_i \): Explanation for the error.
• \( \mathbf{sugg}_i \): Suggested improvement.

3. Error Correction. For translations containing errors, the model generates an improved version: \( \mathbf{y}' = \mathrm{Model}_0(\mathbf{x}, \mathbf{y}, \{e_1, ..., e_n\}) \).

This process yields preference pairs \( (\mathbf{y}, \mathbf{y}') \) for preference optimization.

While these three capabilities are initially unified in Model_0, they are later separated into two specialized models: Model^T for Translation and Model^E for Error annotation and correction. Specifically, for Iter_i (i > 0) and input \( \mathbf{x} \sim \mathcal{D}_i \), Model^T_i generates the initial translation \( \mathbf{y} = \mathrm{Model}^T_i(\mathbf{x}) \). Then Model^E_i identified potential errors \( \{e_1, ..., e_n\} = \mathrm{Model}^E_i(\mathbf{x}, \mathbf{y}) \) and produces an improved version \( \mathbf{y}' = \mathrm{Model}^E_i(\mathbf{x}, \mathbf{y}, \{e_1, ..., e_n\}) \) if errors are detected, forming a preference pair \( (\mathbf{y}, \mathbf{y}') \).

Automatic Filtering & Domain Expansion. A key advantage of our framework is its ability to utilize diverse monolingual data across different domains. In each iteration, we introduce new monolingual texts from varied domains, where Model^T generates translations and Model^E automatically screens for errors. Only translations with identified issues undergo improvement, creating selected and high-quality preference pairs. This approach enables continuous domain knowledge expansion while ensuring efficient preference data generation through automatic error filtering.

2.3 Preference Optimization

Following the construction of preference pairs, the next step is to optimize the translation model using these self-supervised preferences. Our method is compatible with various preference optimization techniques, such as DPO (Rafailov et al., 2024), CPO (Xu et al., 2024b) and SimPO (Meng et al., 2024). We choose DPO for its training stability. Given our self-supervised preference dataset \( P_i \) containing tuples of \( (\mathbf{x}, y_w, y_l) \), where \( y_w \) and \( y_l \) are the better and worse translations, respectively, the final training objective integrates DPO loss with SFT loss:

\[
\mathcal{L}_{\text{final}} = \mathcal{L}_{\text{DPO}} + \alpha \cdot \mathcal{L}_{\text{SFT}},
\]

where the DPO loss and SFT loss are defined as:

\[
\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -E_{(\mathbf{x}, y_w, y_l) \sim P_i}[\log \sigma \\
(\beta \log \frac{\pi_\theta(y_w \mid \mathbf{x})}{\pi_{\text{ref}}(y_w \mid \mathbf{x})} - \beta \log \frac{\pi_\theta(y_l \mid \mathbf{x})}{\pi_{\text{ref}}(y_l \mid \mathbf{x})}],
\]

\[
\mathcal{L}_{\text{SFT}}(\pi_\theta) = -E_{(\mathbf{x}, y_w) \sim P_i}[\log \pi_\theta(y_w \mid \mathbf{x})].
\]

![Self-Supervised Preference Data Construction](page_1016_180_377_563.png)

Figure 2: Self-Supervised Preference Data Construction
Here, \( \pi_\theta \) denotes the translation model Model\(^T_i\) being trained in the current iteration, \( \pi_{ref} \) is the reference model Model\(^{T}_{i-1}\) from the previous iteration. The parameter \( \beta \) is a temperature parameter controlling the sharpness of the preference learning, and \( \alpha_{sft} \) adjusts the weight of SFT loss for training stability. In our preference pairs, \( y_w \) is the improved translation \( y' \) generated by Model\(^{E}_{i-1}\), and \( y_t \) is the original translation \( y \) from Model\(^{T}_{i-1}\).

2.4 Self-Training for Error Annotation

To maintain effective quality assessment as the translation model improves through DPO, we enhance Model\(^E\) via self-training. The training data \( D_{SFT} \) comprises three components: (1) error annotation pairs \( (\mathbf{x}, \mathbf{y}) \to \{e_1, ..., e_n\} \), (2) error correction pairs \( (\mathbf{x}, \mathbf{y}, \{e_1, ..., e_n\}) \to y' \), (3) error-free pairs \( (\mathbf{x}, \mathbf{y}) \to "No\ error" \). The last component prevents over-criticism by helping the model recognize high-quality translations.

This self-training process enables Model\(^E\) to maintain a balanced capability in identifying genuine translation errors and recognizing high-quality translations, ensuring effective quality assessment for the enhanced translation model.

3 Experiment

We carry out comprehensive experiments to demonstrate the effectiveness of SSPO in enhancing LLMs’ machine translation performance.

3.1 Data

We consider 10 translation directions in the paper: da↔en, de↔en, fr↔en, id↔en, zh↔en. As illustrated in Section 2.1, our training dataset consists of three complementary components, detailed data statistics can be found in Appendix A.2-A.4.

Translation Data. For en→de, fr→en,zh↔en, we collect high-quality parallel translation pairs from WMT News Task development and test sets across multiple years. For other language pairs, we sample from News Commentary v18.1 and Europarl v10.

Augmented MQM Annotations. For en→de and zh→en, we collect original MQM-style error annotations from WMT Metrics shared tasks in 2020,2021,2023 (Freitag et al., 2021, 2023). For de→en, fr→en, we collect original MQM-style error annotations from a bio-domain MQM dataset (Zouhar et al., 2024). These data are augmented following the steps in Section 2.1. We provide an example of the augmented MQM annotations in Appendix A.3.

Error Corrections. We construct error correction pairs by sampling from the MQM annotations and manually correcting the identified errors with GPT-4o.

Monolingual Data. We collect monolingual data from open source internet then conduct length and perplexity filtering (detailed in Appendix A.5).

3.2 Models and Training

We apply SSPO to Qwen2.5-14B-Base (Yang et al., 2024a) (14B parameters) and Mistral-Nemo-Base-2407 (Jiang et al., 2023) (12B parameters). The process begins with supervised fine-tuning on our dataset to develop Model\(_0\), which functions as both the initial translation model Model\(^T_0\) and the error detection model Model\(^E_0\). Each iteration \( i \) involves three steps: (1) Model\(^T_i\) generates translations for monolingual inputs, (2) Model\(^E_i\) identifies errors and creates preference pairs for DPO optimization of Model\(^T_{i+1}\), (3) Model\(^E_i\) undergoes self-training to produce Model\(^E_{i+1}\). We adopt LoRA (Hu et al., 2021) in DPO and SFT training. Our prompts, training parameters and implementation environments are provided in Appendix B.1.

3.3 Evaluation

Multi-domain Test sets. For zh→en direction, we employ a 10-domain test suite (Table 1) to evaluate cross-domain generalization. Full test set details are in Appendix B.2.

<table>
  <tr>
    <th>Domain</th>
    <th>Count</th>
    <th>Domain</th>
    <th>Count</th>
  </tr>
  <tr>
    <td>Industry</td>
    <td>3,487</td>
    <td>Finance</td>
    <td>1,322</td>
  </tr>
  <tr>
    <td>Talk</td>
    <td>2,599</td>
    <td>E-commerce</td>
    <td>1,001</td>
  </tr>
  <tr>
    <td>IT</td>
    <td>2,293</td>
    <td>Thesis</td>
    <td>625</td>
  </tr>
  <tr>
    <td>News</td>
    <td>1,875</td>
    <td>Biology</td>
    <td>575</td>
  </tr>
  <tr>
    <td>Literary</td>
    <td>1,514</td>
    <td>Science</td>
    <td>503</td>
  </tr>
</table>

Table 1: Distribution of test samples across different domains for zh→en direction

Multi-lingual Test sets. We evaluate on FLORES-200 test sets (Team et al., 2022) to assess cross-lingual transferability. Full test set details are in Appendix B.3.

Metrics. Following the arguments in CPO Xu et al. (2024b), which demonstrates that
human-written references are not always superior to model outputs and advocates for reference-free evaluation, we adopt reference-free metrics for our evaluation. Specifically, we use: (1) Unbabel/XCOMET-XXL, referred as XCOMET (Guerreiro et al., 2023); (2) Unbabel/wmt23-cometkiwi-da-xxl, referred as KIWI (Rei et al., 2023); and (3) google/metricx-23-qe-xxl-v2p0, referred as METRICX (Juraska et al., 2023).

3.4 Baselines

SoTA Models. We compare with state-of-the-art open-source models including Unbabel’s TowerInstruct (Alves et al., 2024b) and ALMA-13B-R (Xu et al., 2024b) (trained with GPT-4 preference data via CPO). For closed-source models, we benchmark against GPT-4o-0806.

Pipelines. We evaluate our method against three open-source preference data construction pipelines on multi-domain zh→en test set. Instead of starting from Model0, these methods begin with Model-Trans, trained on translation data in our dataset solely for translation generation. (1) Self-Sampling + XCOMET: Generate multiple translations through model self-sampling, then select highest and lowest scoring samples based on XCOMET scores; (2) XCOMET + xTower: Generate initial translations, identify errors using XCOMET, then refine translations with xTower (Treviso et al., 2024) (Unbabel’s 13B correction model trained on GPT-4 annotated data) as positive examples; (3) XCOMET + Qwen2.5-Plus: Similar to (2) but using Qwen2.5-Plus with 5-shot prompting for error correction.

3.5 Multi-domain Results

Multi-domain Evaluation. We systematically evaluate cross-domain generalization using Qwen-2.5-14B-Base on zh → en direction. As shown in Table 2, our iterative optimization achieves consistent gains across all domains. The scores shown are over all domains, with detailed domain-specific results provided in Appendix B.4. These improvements are largely due to the diverse nature of our monolingual data, which enriches the model with domain-specific knowledge.

Comparison with Open-source Pipelines. We conduct comparative experiments to evaluate different preference data construction approaches. Following Section 3.4, we first train Qwen-2.5-14B-Base solely on translation data to obtain Qwen-Trans, which serves as the initial translation model for all pipeline methods. To ensure fair comparison, we use the same set of monolingual data - specifically, the source sentences corresponding to the 12,800 preference data pairs used in training our Qwen-Model\( ^T_1 \).

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">Metrics</th>
  </tr>
  <tr>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
  </tr>
  <tr>
    <th colspan="4">Baseline Models</th>
  </tr>
  <tr>
    <td>Qwen-Trans</td>
    <td>79.32</td>
    <td>90.28</td>
    <td>4.6499</td>
  </tr>
  <tr>
    <td>ALMA-13B-R</td>
    <td>79.19</td>
    <td>91.12</td>
    <td>4.5454</td>
  </tr>
  <tr>
    <td>TowerInstruct</td>
    <td>78.92</td>
    <td>90.22</td>
    <td>4.7780</td>
  </tr>
  <tr>
    <td>GPT-4o-0806</td>
    <td>80.29</td>
    <td>91.54</td>
    <td>4.3310</td>
  </tr>
  <tr>
    <th colspan="4">Pipelines</th>
  </tr>
  <tr>
    <td>Self-sampling + XCOMET</td>
    <td>79.20</td>
    <td>90.43</td>
    <td>4.6686</td>
  </tr>
  <tr>
    <td>XCOMET + xTower</td>
    <td>79.85</td>
    <td><b>92.93</b></td>
    <td>4.2451</td>
  </tr>
  <tr>
    <td>XCOMET + Qwen2.5-Plus</td>
    <td>80.03</td>
    <td>91.64</td>
    <td>4.4125</td>
  </tr>
  <tr>
    <th colspan="4">Our Method</th>
  </tr>
  <tr>
    <td>Qwen-Model<sub>0</sub></td>
    <td>79.42</td>
    <td>90.37</td>
    <td>4.6393</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>1</sub><sup>T</sup></td>
    <td>80.38</td>
    <td>92.40</td>
    <td>4.2382</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>2</sub><sup>T</sup></td>
    <td><b>80.67</b></td>
    <td><b>92.66</b></td>
    <td><b>4.2187</b></td>
  </tr>
</table>

Table 2: Performance comparison on zh→en multi-domain test sets. ↓ means lower is better.

Table 2 presents the results on reference-free metrics. We summarize two key observations:

Integration of error-related data enhances translation performance. Error annotation and correction training data enhances translation performance (Qwen-Model<sub>0</sub> outperforms Qwen-Trans). This improvement can be attributed to the implicit translation knowledge embedded in error-related data.

SSPO generate high-quality preference data. Through a single iteration of SSPO, Qwen-Model<sub>1</sub><sup>T</sup> demonstrates significant improvements compared to Qwen-Model<sub>0</sub> (+0.96 KIWI, +2.03 XCOMET, -0.4 METRICX). Remarkably, it outperforms GPT-4o-0806 across all metrics. In contrast, the self-sampling + DPO approach, which relies on 14B model’s self-sampling for preference data, shows limited effectiveness. Error correction pipelines using xCOMET annotations (XCOMET + xTower and XCOMET + Qwen2.5-Plus) also demonstrate great improvements in translation quality, but they show strong bias towards XCOMET metrics.

We conduct additional LLM evaluation using Claude-3.5* (Anthropic, 2023) to compare Qwen-Model<sub>1</sub><sup>T</sup> with the best-performing open-source pipeline (XCOMET + xTower) through pairwise comparison. To reduce position bias, we per-

*We use claude-3-5-20241022 available from Anthropic API.
form two rounds of evaluations with swapped positions of candidate translations. Qwen-Model\( ^T_1 \) outperforms xTower Pipeline with a net win rate of 8.5%. Evaluation details are listed in Appendix B.5.

3.6 Multi-lingual Results

We conduct 2 iterations of self-supervised optimization on Qwen2.5-14B-Base and Mistral-Nemo-Base-2407 across 10 language directions. For zh→en direction, we evaluate on our multi-domain test sets, while for other language pairs, we use the FLORES200 testset. Primary results for xx→en and en→xx are shown in Figure 3. Following Section 3.3, we report reference-free metrics, full results are detailed in Appendix B.6. Key findings are summarized below.

Progressive Performance Enhancement. Both Qwen and Mistral models show consistent gains across iterations, with Model\( ^T_1 \) outperforming Model\( ^T_0 \) and Model\( ^E_1 \) outperforming Model\( ^E_0 \) across all language pairs. Notably, for xx→en direction, both Qwen-Model\( ^T_2 \) and Mistral-Model\( ^T_2 \) achieve comparable or superior performance to GPT-4o-0806. Although a performance gap remains with GPT-4o-0806 in en→xx direction, our approach still demonstrates substantial improvements (e.g., +0.62 KIWI, +0.37 XCOMET, -0.0526 METRICX when Mistral-Model\( ^T_2 \) v.s. Mistral-Model\( ^E_0 \)). These consistent gains across different translation directions validate the effectiveness of our iterative optimization approach.

Asymmetric Performance Gains Across Directions. The improvement patterns differ between xx→en and en→xx translations, primarily due to the distribution of MQM annotation training data. For xx→en, abundant MQM data enables high-quality error detection in the first iteration, leading to strong preference data and substantial gains. However, the second iteration shows limited improvement due to a growing capability gap: while Model\( ^T_1 \) achieves significant enhancement through DPO training on high-quality preference data, Model\( ^E_1 \), trained solely on self-generated annotations, shows marginal improvement in error detection. Thus it struggles to identify subtle errors in increasingly better translations. Conversely, for en→xx where MQM data is scarce, limited initial error detection capability leads to modest improvements of Model\( ^T_1 \). When Model\( ^E_1 \) is trained on self-generated annotations, its error detection capability improves moderately, still sufficient to identify errors in Model\( ^T_1 \)'s translations, thus enabling further performance gains in the second iteration.

Cross-lingual Transfer of MQM Annotation Ability. Despite our training data only containing MQM annotations for de↔en, fr→en, and zh→en directions, the models successfully generalize to other language pairs, effectively constructing preference data that leads to improved translation performance. This phenomenon is also discovered by Uhlig et al. (2025). We find this generalization ability is influenced by the model’s inherent linguistic capabilities: for language pairs where the base model shows strong performance, high-quality preference data can be generated even without corresponding MQM data. This is exemplified by Mistral’s significant first-iteration improvements in da→en translation.

4 Analyses

We conduct extensive analyses to investigate three critical aspects of our approach. First, we explore how the amount of monolingual data used within a single iteration impacts the optimization results, aiming to determine the optimal data quantity for maximizing translation quality. Second, we examine the impact of error correction strategies when generating preference data, specifically comparing two approaches: correcting all errors (both major and minor) in each iteration versus a progressive strategy that focuses on major errors in the first iteration and addresses all errors in subsequent iterations. Third, we investigate whether incorporating external high-quality preference data can further enhance translation quality. We use Qwen2.5-14B-Base and focus on zh→en direction. These analyses aim to provide deeper insights into the mechanisms and optimization strategies of our approach. We analyze the evolution of Model\( ^E_1 \) in Appendix C and provide an ablation study of our approach in Appendix D.

4.1 Impact of Monolingual Data Amount

In the first iteration of zh→en translation, we observe that model performance plateaus after training on 12,800 preference data pairs, with additional data yielding diminishing returns. To investigate the optimal data utilization strategy, we sys-
Figure 3: Average results for en-xx and xx-en translation directions.

tematically partition the full dataset into 2, 3, 4 equal subsets for multi-iteration training. The key question we want to address is whether to use all available monolingual data in a single iteration until performance saturates, or to distribute it across multiple iterations for gradual improvement.

Results in Table 3 demonstrate clear superiority of the single-iteration strategy. When we split the 12,800 samples into multiple portions, the cumulative improvement after multiple iterations fails to match the performance achieved by using all data at once. This indicates that using the entire data in a single iteration optimizes performance more effectively than incremental updates with smaller portions.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">Metrics</th>
  </tr>
  <tr>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
  </tr>
  <tr>
    <td>Model<sub>0</sub></td>
    <td>79.42</td>
    <td>90.37</td>
    <td>4.6393</td>
  </tr>
  <tr>
    <td>Iter1-1</td>
    <td><b>80.38</b></td>
    <td><b>92.40</b></td>
    <td><b>4.2382</b></td>
  </tr>
  <tr>
    <td>Iter2-1</td>
    <td>79.97</td>
    <td>91.10</td>
    <td>4.4276</td>
  </tr>
  <tr>
    <td>Iter2-2</td>
    <td>80.18</td>
    <td>91.75</td>
    <td>4.3570</td>
  </tr>
  <tr>
    <td>Iter3-1</td>
    <td>79.56</td>
    <td>90.63</td>
    <td>4.5566</td>
  </tr>
  <tr>
    <td>Iter3-2</td>
    <td>79.72</td>
    <td>90.90</td>
    <td>4.5167</td>
  </tr>
  <tr>
    <td>Iter3-3</td>
    <td>79.90</td>
    <td>91.11</td>
    <td>4.4798</td>
  </tr>
  <tr>
    <td>Iter4-1</td>
    <td>79.52</td>
    <td>90.49</td>
    <td>4.5972</td>
  </tr>
  <tr>
    <td>Iter4-2</td>
    <td>79.57</td>
    <td>90.62</td>
    <td>4.5590</td>
  </tr>
  <tr>
    <td>Iter4-3</td>
    <td>79.67</td>
    <td>90.77</td>
    <td>4.5468</td>
  </tr>
  <tr>
    <td>Iter4-4</td>
    <td>79.77</td>
    <td>90.93</td>
    <td>4.5090</td>
  </tr>
</table>

Table 3: Performance comparison of different data utilization strategies. Itern-k denotes the k-th step in the n-iteration setting, where the 12,800 training samples are split into n equal portions.

4.2 Impact of Error Correction Strategy

We try 3 different strategies. "Major-only" means only correcting major errors, "Major&Half Minor" means correcting all major errors and random 50% of minor errors. "Major&Minor" means correcting all errors.

We initially hypothesized that focusing on different types of errors in each iteration might be beneficial, thus exploring various error correction strategies. However, our experimental results suggest otherwise. In our first-round annotations, we identified 15,879 major errors and 15,555 minor errors for 12,800 translations. The results demonstrate that partially or completely omitting minor error corrections during preference data construction leads to missed opportunities for learning important translation patterns, resulting in reduced performance. Therefore, we conclude that correcting all errors in each iteration of prefer-
<table>
  <tr>
    <th rowspan="2">Strategies</th>
    <th colspan="3">Metrics</th>
  </tr>
  <tr>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
  </tr>
  <tr>
    <td>Major-only (Iter 1)</td>
    <td>80.20</td>
    <td>92.13</td>
    <td>4.2573</td>
  </tr>
  <tr>
    <td>Major&Minor (Iter 2)</td>
    <td>80.39</td>
    <td>92.42</td>
    <td>4.2444</td>
  </tr>
  <tr>
    <td>Major&Half Minor (Iter 1)</td>
    <td>80.28</td>
    <td>92.20</td>
    <td>4.2457</td>
  </tr>
  <tr>
    <td>Major&Minor (Iter 2)</td>
    <td>80.42</td>
    <td>92.44</td>
    <td>4.2486</td>
  </tr>
  <tr>
    <td>Major&Minor (Iter 1)</td>
    <td>80.38</td>
    <td>92.40</td>
    <td>4.2382</td>
  </tr>
  <tr>
    <td>Major&Minor (Iter 2)</td>
    <td>80.67</td>
    <td>92.66</td>
    <td>4.2187</td>
  </tr>
</table>

Table 4: Performance comparison of progressive error correction strategies across iterations.

ence data generation is the optimal strategy.

4.3 Benefits of External Preference Data

We examine the impact of preference data composition on DPO effectiveness. For external preference data, we leverage GPT-4o to generate corrections for 12,800 sentences sampled from our augmented MQM training annotations (distinct from the correction data in Model_0 training set). We evaluate three strategies while maintaining a constant total of 12,800 preference pairs: (1) self-supervised preferences only, (2) external preferences only, and (3) an equal mixture of both sources.

<table>
  <tr>
    <th rowspan="2">Data Composition</th>
    <th colspan="3">Metrics</th>
  </tr>
  <tr>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
  </tr>
  <tr>
    <td>Self-supervised</td>
    <td><b>0.8038</b></td>
    <td>0.9240</td>
    <td>4.2382</td>
  </tr>
  <tr>
    <td>External</td>
    <td>0.8032</td>
    <td>0.9231</td>
    <td>4.2416</td>
  </tr>
  <tr>
    <td>Mixed (1:1)</td>
    <td>0.8030</td>
    <td><b>0.9288†‡</b></td>
    <td><b>4.2192†‡</b></td>
  </tr>
</table>

Table 5: Comparison of preference data strategies: self-supervised preferences, external preferences (human annotations with GPT-4o corrections), and their equal mixture. † indicates \( p < 0.05 \) compared to Self-supervised; ‡ indicates \( p < 0.05 \) compared to External.

Results show that the balanced mixture strategy achieves optimal performance with statistically significant improvements in XCOMET and METRICX scores. This success stems from combining two complementary sources: external data provides high-quality reference signals, while self-supervised preferences ensure training stability by aligning with the model’s current capabilities. This complementary combination proves more effective than using either source alone.

5 Related Works

5.1 Translation Error Detection and Correction

In span-level error detection, neural models have proven effective in identifying errors within machine translations, as demonstrated by AutoMQM (Fernandes et al., 2023), InstructScore (Xu et al., 2023), and XCOMET (Guerreiro et al., 2023). For error correction, recent advancements involve prompting large language models (LLMs) to suggest new translations, exemplified by TowerAPE (Alves et al., 2024a) and GPT-4 prompting (Raunak et al., 2023). Ki and Carpuat (2024), Xu et al. (2024d) and Treviso et al. (2024) integrate detailed error feedback into post-editing prompts. Specifically, LLMRefine (Xu et al., 2024d) employs "succinct explanations" of fine-grained errors to guide models towards better translations through iterative refinement. xTower (Treviso et al., 2024) utilize error spans annotated by humans or predicted by XCOMET, first explain these errors then give refine translations. Inspired by these works, we construct augmented MQM data for training, enabling our Model^E to provide error explanations and improvement suggestions alongside identifying errors in a reference-free mode. This approach increases the reliability of the identified errors and reduces the difficulty of error correction.

5.2 Preference Data Construction for Machine Translation

Preference data are triplets consisting of user prompts, user-preferred responses, and non-preferred responses. However, there has been limited exploration of how to construct such preference data specifically for machine translation tasks. Xu et al. (2024b) constructed preference data using GPT-4 (OpenAI, 2023) and gold reference for Contrastive Preference Optimization (CPO). Yang et al. (2024b) generate preference datasets using MBR decoding on Multilingual Large Language Models (MLLMs) to favor higher-ranked translations. Agrawal et al. (2024) collect sentence-level quality assessments from professional linguists on LLMs’ translations and leverage automatic metrics to recover these preferences. They then use this analysis to curate a dataset. While effective, their approaches face the challenges discussed in Sec 1. To address these issues, our work proposes a method to construct translation preference data at scale using monolingual data, tailored to the model’s current capabilities, which effectively enhances the model’s translation ability.
6 Conclusion

In this study, we initially explain the shortcomings of previous approaches to constructing translation preference data. Then, we propose SSPO, a self-supervised mechanism that enables LLMs to iteratively generate high-quality translation preference data for DPO training. Our analysis reveals that combining self-generated preference data with external preference data in DPO training leads to superior translation quality. We validate SSPO’s effectiveness across multiple language pairs, domains and models, demonstrating consistent improvements in translation performance without relying on external human or model annotations. Applying SSPO to 14B parameters large language models (LLMs) achieves comparable or better performance than GPT-4o on FLORES and multi-domain test datasets.

Limitations

We primarily conducted experiments on medium-sized models, while testing on larger or smaller models might reveal different optimization dynamics. Our study also observes relatively few iteration rounds, and a longer-term study could provide deeper insights into the convergence patterns. Additionally, while we employ multiple automatic metrics, the lack of human evaluation means that improvements in metric scores may not perfectly align with actual translation quality as perceived by readers.

Acknowledgments

This paper is supported by the General Program of National Natural Science Foundation of China (62176153). We express our profound appreciation to anonymous reviewers for their helpful suggestions. We also thank Jialong Tang and Haoran Wei for their engaging and valuable discussions that enriched our work.

References

Sweta Agrawal, José G. C. de Souza, Ricardo Rei, António Farinhas, Gonçalo Faria, Patrick Fernandes, Nuno M Guerreiro, and Andre Martins. 2024. Modeling user preferences with automatic metrics: Creating a high-quality preference dataset for machine translation. Preprint, arXiv:2410.07779.

Duarte M Alves, José Pombal, Nuno M Guerreiro, Pedro H Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, et al. 2024a. Tower: An open multilingual large language model for translation-related tasks. arXiv preprint arXiv:2402.17733.

Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins. 2024b. Tower: An open multilingual large language model for translation-related tasks. Preprint, arXiv:2402.17733.

Anthropic. 2023. Claude 3.5.

Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Aeyr Locatelli, Julia Kreutzer, Nick Frost, Aidan Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstünn, and Sara Hooker. 2024. Aya 23: Open weight releases to further multilingual progress. Preprint, arXiv:2405.15032.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhrenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.

Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André F. T. Martins, Graham Neubig, Ankush Garg, Jonathan H. Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. Preprint, arXiv:2308.07286.

Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Preprint, arXiv:2104.14478.

Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Eighth Conference on Machine Translation, pages 578–628, Singapore. Association for Computational Linguistics.
Yuxin Fu, Shijing Si, Leyi Mai, Xi-ang Li, and Yuli An. 2024. Ffn: a fine-grained chinese-english financial domain parallel corpus. arXiv preprint arXiv:2406.18856.

Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2023. xcomet: Transparent machine translation evaluation through fine-grained error detection. Preprint, arXiv:2310.10482.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.

Tianxiang Hu, Pei Zhang, Baosong Yang, Jun Xie, Derek F. Wong, and Rui Wang. 2024. Large language model for multi-domain translation: Benchmarking and domain cot fine-tuning. Preprint, arXiv:2410.02631.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzaadeh, and Markus Freitag. 2023. MetricX-23: The Google submission to the WMT 2023 metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 756–767, Singapore. Association for Computational Linguistics.

Dayeon Ki and Marine Carpuat. 2024. Guiding large language models to post-edit machine translation with error annotations. arXiv preprint arXiv:2404.07851.

Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. Preprint, arXiv:2405.14734.

OpenAI. 2023. GPT-4 technical report. Preprint, arXiv:2303.08774.

OpenAI. 2024. GPT-4o.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Siemens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. Preprint, arXiv:2203.02155.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Preprint, arXiv:2305.18290.

Miguel Ramos, Patrick Fernandes, António Farinhas, and Andre Martins. 2024. Aligning neural machine translation models: Human feedback in training and inference. In Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pages 258–274, Sheffield, UK. European Association for Machine Translation (EAMT).

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, page 35053506, New York, NY, USA. Association for Computing Machinery.

Vikas Raunak, Amr Sharaf, Yiren Wang, Hany Awadalla, and Arul Menezes. 2023. Leveraging GPT-4 for automatic translation post-editing. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12009–12024, Singapore. Association for Computational Linguistics.

Ricardo Rei, Nuno M. Guerreiro, JosÃ© Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, JosÃ© G. C. de Souza, and AndrÃ© Martins. 2023. Scaling up CometKiwi: Unbabel-IST 2023 submission for the quality estimation shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 841–848, Singapore. Association for Computational Linguistics.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejía Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672.

Liang Tian, Derek F. Wong, Lidia S. Chao, Paulo Quaresma, Francisco Oliveira, Yi Lu, Shuo Li, Yiming Wang, and Longyue Wang. 2014. UM-corpus: A large English-Chinese parallel corpus for statistical machine translation. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1837–1842, Reykjavik, Iceland. European Language Resources Association (ELRA).
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint, arXiv:2302.13971.

Marcos Treviso, Nuno M. Guerreiro, Sweta Agrawal, Ricardo Rei, José Pombal, Tania Vaz, Helena Wu, Beatriz Silva, Daan van Stigt, and André F. T. Martins. 2024. xtower: A multilingual llm for explaining and correcting translation errors. Preprint, arXiv:2406.19482.

Kaden Uhlig, Joern Wuebker, Raphael Reinauer, and John DeNero. 2025. Cross-lingual human-preference alignment for neural machine translation with direct quality optimization. Preprint, arXiv:2409.17673.

Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024a. A paradigm shift in machine translation: Boosting translation performance of large language models. Preprint, arXiv:2309.11674.

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024b. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. Preprint, arXiv:2401.08417.

Nuo Xu, Jun Zhao, Can Zu, Sixian Li, Lu Chen, Zhihao Zhang, Rui Zheng, Shihan Dou, Wenjuan Qin, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024c. Advancing translation preference modeling with rlhf: A step towards cost-effective solution. Preprint, arXiv:2402.11525.

Wenda Xu, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Biao Zhang, Zhongtao Liu, William Yang Wang, Lei Li, and Markus Freitag. 2024d. LLMRefine: Pinpointing and refining large language models via fine-grained actionable feedback. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1429–1445, Mexico City, Mexico. Association for Computational Linguistics.

Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li. 2023. Instructscore: Explainable text generation evaluation with finegrained feedback. Preprint, arXiv:2305.14282.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuncheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024a. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115.

Guangyu Yang, Jinghong Chen, Weizhe Lin, and Bill Byrne. 2024b. Direct preference optimization for neural machine translation with minimum bayes risk decoding. Preprint, arXiv:2311.08380.

Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. 2024. Swifta scalable lightweight infrastructure for fine-tuning. Preprint, arXiv:2408.05517.

Vilém Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badeka, Jenyuan Wang, and Brian Thompson. 2024. Fine-tuned machine translation metrics struggle in unseen domains. arXiv preprint arXiv:2306.07899.
A  Datasets Statistics

A.1 Prompts for Producing Error Annotation and Correction Training Data

Error Annotation. As shown in Figure 4, the template includes 3 examples demonstrating various types of errors and their corrections, followed by the actual task structure. Each error analysis includes location, severity, explanation, and improvement suggestions. For each language pair, we carefully curate examples with diverse error types and varying complexity levels to stimulate the model’s error annotation capabilities.

Error Correction. When doing error correction, we use 1-shot prompting in Figure 5. The example should be relatively long with multiple types of errors, serving to stimulate the model’s ability to fully correct the translation.

A.2 Translation Data

<table>
  <tr>
    <th>Language Pair</th>
    <th>Quantity</th>
    <th>Source</th>
  </tr>
  <tr>
    <td>da → en</td>
    <td>5000</td>
    <td>Sample from Europarl v10</td>
  </tr>
  <tr>
    <td>en → da</td>
    <td>5000</td>
    <td>Sample from Europarl v10</td>
  </tr>
  <tr>
    <td>de → en</td>
    <td>5000</td>
    <td>Sample from News Commentary v18.1</td>
  </tr>
  <tr>
    <td>en → de</td>
    <td>25227</td>
    <td>WMT dev test</td>
  </tr>
  <tr>
    <td>fr → en</td>
    <td>22074</td>
    <td>WMT dev test</td>
  </tr>
  <tr>
    <td>en → fr</td>
    <td>5000</td>
    <td>Sample from news commentary v18.1</td>
  </tr>
  <tr>
    <td>id → en</td>
    <td>5000</td>
    <td>Sample from news commentary v18.1</td>
  </tr>
  <tr>
    <td>en → id</td>
    <td>5000</td>
    <td>Sample from news commentary v18.1</td>
  </tr>
  <tr>
    <td>zh → en</td>
    <td>16587</td>
    <td>WMT dev test</td>
  </tr>
  <tr>
    <td>en → zh</td>
    <td>11050</td>
    <td>WMT dev test</td>
  </tr>
</table>

Table 6: Translation Data Statistics

A.3 MQM Data

Table 7 shows the source and distribution of the orginal MQM data.

<table>
  <tr>
    <th>Language Pair</th>
    <th>Quantity</th>
    <th>Source</th>
  </tr>
  <tr>
    <td>da → en</td>
    <td>0</td>
    <td>\</td>
  </tr>
  <tr>
    <td>en → da</td>
    <td>0</td>
    <td>\</td>
  </tr>
  <tr>
    <td>de → en</td>
    <td>3667</td>
    <td>(Zouhar et al., 2024)</td>
  </tr>
  <tr>
    <td>en → de</td>
    <td>7865</td>
    <td>WMT Metrics Shared Task</td>
  </tr>
  <tr>
    <td>fr → en</td>
    <td>3029</td>
    <td>(Zouhar et al., 2024)</td>
  </tr>
  <tr>
    <td>en → fr</td>
    <td>0</td>
    <td>\</td>
  </tr>
  <tr>
    <td>id → en</td>
    <td>0</td>
    <td>\</td>
  </tr>
  <tr>
    <td>en → id</td>
    <td>0</td>
    <td>\</td>
  </tr>
  <tr>
    <td>zh → en</td>
    <td>41943</td>
    <td>WMT Metrics Shared Task</td>
  </tr>
  <tr>
    <td>en → zh</td>
    <td>0</td>
    <td>\</td>
  </tr>
</table>

Table 7: MQM Data Statistics

Original MQM data contains severity, location and specific category for each error. i.e. each error \( e_i^{\text{original}} \) conatins \( \text{loc}_i, \text{sev}_i, \text{cat}_i \).

We augment MQM data by prompting GPT-4o to produce explanation and correct suggestions for each error. In this process, we intentionally omitted the error type information from the original MQM annotations to simplify the data structure and reduce the complexity of model learning. Each error \( e_i^{\text{augmented}} \) conatins \( \text{loc}_i, \text{sev}_i, \exp_i, \text{sugg}_i \).

Fig 6 shows an example of the augmented MQM data.

Our augmentation manner implicitly incorporates a Chain-of-Thought (CoT) mechanism: by requiring the model to first explain the error (\( \exp_i \)) before generating a correction suggestion (\( \text{sugg}_i \)), we enforce a step-by-step reasoning process. The model must understand the error (e.g., semantic mismatch, grammatical flaw, or cultural mistranslation) before proposing a fix, mirroring the "diagnose-then-correct" workflow of human experts. This idea is also used in the design of xTower(Treviso et al., 2024).

A.4 Error Correction Data

We only construct zh → en and de → en error since it is not a hard task.

<table>
  <tr>
    <th>Language Pair</th>
    <th>Quantity</th>
    <th>Source</th>
  </tr>
  <tr>
    <td>zh → en</td>
    <td>9322</td>
    <td>GPT-4o prompting</td>
  </tr>
  <tr>
    <td>de → en</td>
    <td>9786</td>
    <td>GPT-4o prompting</td>
  </tr>
</table>

Table 8: Error Detection Data Statistics

A.5 Monolingual Data

We employ a two-stage dynamic filtering approach to curate high-quality monolingual data for iterative training:

Length Filtering.

• Procedure: When generating preference dataset \( P_0 \) from \( \mathcal{D}_0 \) via Model_0, we compute the mean token count \( \bar{T}_0 \) of source sentences in \( P_0 \). This identifies the typical sentence length where Model_0 makes errors.

• Application: For Model_1^{T'}'s training, we sample monolingual sentences longer than \( \bar{T}_0 \) for translation.

• Rationale: Longer sentences offer two advantages: (i) they capture diverse linguistic phenomena (e.g., discourse coherence, idiomaticity), ensuring the model encounters challenging cases; (ii) they mitigate trivial corrections
from short, error-free translations that offer no training signal.

Perplexity (PPL) Filtering.

• Procedure: After length filtering, we compute the perplexity of the remaining monolingual corpus using the current iteration’s translation model (e.g., Model\( ^I \)). We retain sentences with \( \mathrm{PPL} \leq \mu + 2\sigma \), where \( \mu \) denotes the mean PPL and \( \sigma \) the standard deviation.

• Rationale: This threshold serves dual purposes: (i) it excludes outliers such as overly complex or noisy sentences beyond the model’s current capability; (ii) it balances difficulty by ensuring sentences are challenging yet interpretable, avoiding degenerate cases such as garbled text.

B Experimental Details

Here we list additional experimental details for our implementation and experiments.

B.1 Training Configurations

Here we detail our prompts, training parameters and implementation environment.

B.1.1 Prompts

The followings are our prompts during model training and inference.

Translation Generation Prompt
Translate the following {src_lang} text into {tgt_lang}.
{src_lang}: {src}
{tgt_lang}:

Error Annotation Prompt
Based on the {src_lang} source, identify the major and minor errors in the {tgt_lang} translation. For each error, please provide explanation and improvement.
{src_lang} source:{src}
{tgt_lang} translation:{trans}
Errors:

Error Correction Prompt
Given the {src_lang} source text, the initial {tgt_lang} translation, and the list of identified errors with explanations and suggested improvements, improve the initial {tgt_lang} translation to make it accurate, fluent, and true to the meaning and tone of the original text.
{src_lang} source:{src}
Initial {tgt_lang} translation:{initial_trans}
Errors:{errors}
Improved Translation:

B.1.2 DPO Training
We use SWIFT framework (Zhao et al., 2024) with the following parameter setting in Table 9. We use 8 80G A100 GPUs for 50 hours DPO training for all our models.

<table>
  <tr>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>Training Type</td>
    <td>DPO with LoRA</td>
  </tr>
  <tr>
    <td>DPO Config</td>
    <td>\( \beta=0.1, \alpha=1.0 \)</td>
  </tr>
  <tr>
    <td>LoRA Config</td>
    <td>rank=128, \( \alpha=16 \), dropout=0.1</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>1e-5 (cosine schedule)</td>
  </tr>
  <tr>
    <td>Sequence Length</td>
    <td>1024</td>
  </tr>
  <tr>
    <td>Optimization</td>
    <td>weight_decay=0.1, max_grad_norm=1.0</td>
  </tr>
</table>

Table 9: DPO Training Configuration

B.1.3 SFT Training
We use Deepspeed framework (Rasley et al., 2020) with the following parameter setting in Table 10. We use 8 80G A100 GPUs for 100 hours training for all our models.

<table>
  <tr>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>Training Type</td>
    <td>SFT with LoRA</td>
  </tr>
  <tr>
    <td>Deepspeed Config</td>
    <td>zero_stage=0</td>
  </tr>
  <tr>
    <td>LoRA Config</td>
    <td>rank=128, \( \alpha=32 \), dropout=0.1</td>
  </tr>
  <tr>
    <td>Sequence Length</td>
    <td>1024</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>1e-4(cosine schedule)</td>
  </tr>
  <tr>
    <td>Optimization</td>
    <td>weight_decay=0.1, max_grad_norm=1.0</td>
  </tr>
</table>

Table 10: SFT Training Configuration

B.2 zh → en Multi-domain Test set
We collect multi-domain dataset from various sources, as shown in Table 11

<table>
  <tr>
    <th>Domain</th>
    <th>Count</th>
    <th>Source</th>
  </tr>
  <tr>
    <td>Industry</td>
    <td>3,487</td>
    <td>(Hu et al., 2024)</td>
  </tr>
  <tr>
    <td>Talk</td>
    <td>2,599</td>
    <td>IWSLT 16,17</td>
  </tr>
  <tr>
    <td>IT</td>
    <td>2,293</td>
    <td>(Hu et al., 2024)</td>
  </tr>
  <tr>
    <td>News</td>
    <td>1,875</td>
    <td>WMT22 New Task Test Set</td>
  </tr>
  <tr>
    <td>Literary</td>
    <td>1,514</td>
    <td>WMT24 Literary Task Test Set</td>
  </tr>
  <tr>
    <td>Finance</td>
    <td>1,322</td>
    <td>(Fu et al., 2024)</td>
  </tr>
  <tr>
    <td>E-commerce</td>
    <td>1,001</td>
    <td>(Hu et al., 2024)</td>
  </tr>
  <tr>
    <td>Thesis</td>
    <td>625</td>
    <td>Sample from (Tian et al., 2014)</td>
  </tr>
  <tr>
    <td>Biology</td>
    <td>575</td>
    <td>WMT Biomedical Translation Task</td>
  </tr>
  <tr>
    <td>Science</td>
    <td>503</td>
    <td>Sample from (Tian et al., 2014)</td>
  </tr>
</table>

Table 11: Distribution of test samples across different domains for zh→en direction

B.3 Multi-lingual Test Set

B.4 Full Multi-domain Results
We show the detailed zh → en multi-domain results in Table 13.
<table>
  <tr>
    <th colspan="3">FLORES-200 Test Set</th>
  </tr>
  <tr>
    <th>Source → Target</th>
    <th>Language Pair</th>
    <th>Size</th>
  </tr>
  <tr>
    <td>da ↔ en</td>
    <td>Danish ↔ English</td>
    <td>1,012</td>
  </tr>
  <tr>
    <td>de ↔ en</td>
    <td>Germen ↔ English</td>
    <td>1,012</td>
  </tr>
  <tr>
    <td>fr ↔ en</td>
    <td>French ↔ English</td>
    <td>1,012</td>
  </tr>
  <tr>
    <td>id ↔ en</td>
    <td>Indonesian ↔ English</td>
    <td>1,012</td>
  </tr>
  <tr>
    <td>en → zh</td>
    <td>English → Chinese</td>
    <td>1,012</td>
  </tr>
</table>

Table 12: Test set composition from FLORES-200 across different language directions

B.5 Model Evaluation

Evaluation Prompt We use the following the following prompt on Claude-3.5 to evaluate translation quality.

Translation Evaluation Prompt
Chinese Source: {src}
Translation 1: {trans1}
Translation 2: {trans2}

Please evaluate the translation quality from the following aspects:
1. Accuracy: Whether the translation accurately conveys the meaning of the original text.
2. Fluency: Whether the translation is natural and idiomatic English.
3. Fidelity: Whether the translation is faithful to the original without adding or omitting information.
4. Style and Tone: Whether the translation maintains the style and tone of the original text.

After considering all these factors, please indicate which translation is better:
Reply '1' if Translation 1 is better
Reply '2' if Translation 2 is better
Reply '0' if they are equally good

Please only reply with the number without any explanation.

Evaluation Results As shown in Table 14, Qwen-Model\( T_1 \) outperform xTower Pipeline in both evaluation orders, reaching a net win rate of 8.5%.

B.6 Full Multilingual Result

Full results for xx → en are in Table 15. Full results for en → xx are in Table 16.

C Evolvement of Model\(^E\)

We focus on zh → en language pair (as our multi-domain test set provides more representative results) and analyze the evolvement of Model\(^E\) from three perspectives. We use Qwen series for our analysis.

C.1 Error Rates During SSPO Iterations

We calculated the error rates identified during the two iterations of SSPO training. As shown in Table 18, the error rate decreases from **48.4%** in the first iteration to **37.0%** in the second iteration. *This reduction can be attributed to the improved translation quality of Qwen-Model\(^T_1\)*.

C.2 MQM Pattern Error Detection

We evaluate Model\(^E\)'s error detection performance using the WMT22 zh → en MQM dataset, which consists of 29,579 sentences. We employ Qwen-Model\(_0\), Qwen-Model\(^E_1\), and Qwen-Model\(^E_2\) as the error detection models to annotate errors then calculate precision, recall, and F1 scores. We calculate by the following settings.

**TP (True Positive):** The model correctly identifies errors or agrees with the test set that the translation is error-free.

**FP (False Positive):** The model predicts an error where none exists.

**FN (False Negative):** The model fails to detect an actual error.

Results in Table 18 reveals significant evolution for the error detection ability of Model\(^E\):

• **Substantial improvement in precision:** From 34.30% in Model\(_0\) to 51.90% in Model\(^E_2\) (+17.60%), indicating a more accurate understanding of "actual translation error."

• **Steady growth in recall:** From 50.28% in Model\(_0\) to 61.09% in Model\(^E_2\) (+10.81%), suggesting the model's ability to identify a wider range of translation errors.

• **Overall improvement in F1 score:** From 40.78% in Model\(_0\) to 56.12% in Model\(^E_2\) (+15.34%), reflecting a better balance between precision and recall.

We find that the training MQM data for Model\(_0\) is imbalanced, with error-containing samples significantly outnumbering error-free ones. This leads to over-predictions. *The error annotation datasets curated from our SSPO framework is much more balanced, mitigates the over-prediction issue and lead to more calibrated predictions.*

C.3 Multi-domain Error Detection And Correction

We use Qwen2.5-14B-Instruct as the translation model to generate initial translations on our multi-domain Zh-En test set. We then apply Qwen-Model\(_0\), Qwen-Model\(^E_1\), and Qwen-Model\(^E_2\) to an-
<table>
  <tr>
    <th rowspan="2">Domain</th>
    <th colspan="3">KIWI</th>
    <th colspan="3">XCOMET</th>
    <th colspan="3">METRICX↓</th>
  </tr>
  <tr>
    <th>Model<sub>0</sub></th>
    <th>Model<sup>E</sup><sub>1</sub></th>
    <th>Model<sup>T</sup><sub>2</sub></th>
    <th>Model<sub>0</sub></th>
    <th>Model<sup>E</sup><sub>1</sub></th>
    <th>Model<sup>T</sup><sub>2</sub></th>
    <th>Model<sub>0</sub></th>
    <th>Model<sup>E</sup><sub>1</sub></th>
    <th>Model<sup>T</sup><sub>2</sub></th>
  </tr>
  <tr>
    <td>E-commerce</td>
    <td>63.90</td>
    <td>64.89</td>
    <td>65.12</td>
    <td>77.68</td>
    <td>84.86</td>
    <td>85.53</td>
    <td>6.6998</td>
    <td>6.1222</td>
    <td>6.0991</td>
  </tr>
  <tr>
    <td>Industry</td>
    <td>81.67</td>
    <td>82.63</td>
    <td>82.79</td>
    <td>91.46</td>
    <td>93.33</td>
    <td>93.47</td>
    <td>4.2090</td>
    <td>3.6912</td>
    <td>3.7428</td>
  </tr>
  <tr>
    <td>IT</td>
    <td>84.06</td>
    <td>84.49</td>
    <td>84.75</td>
    <td>94.61</td>
    <td>95.66</td>
    <td>95.92</td>
    <td>3.5555</td>
    <td>3.2356</td>
    <td>3.2253</td>
  </tr>
  <tr>
    <td>Literary</td>
    <td>72.96</td>
    <td>74.66</td>
    <td>75.01</td>
    <td>86.04</td>
    <td>88.23</td>
    <td>88.49</td>
    <td>5.8166</td>
    <td>5.4268</td>
    <td>5.3986</td>
  </tr>
  <tr>
    <td>Science</td>
    <td>86.13</td>
    <td>86.60</td>
    <td>86.98</td>
    <td>96.55</td>
    <td>97.06</td>
    <td>97.26</td>
    <td>3.5208</td>
    <td>3.2702</td>
    <td>3.2741</td>
  </tr>
  <tr>
    <td>Thesis</td>
    <td>79.43</td>
    <td>80.24</td>
    <td>80.37</td>
    <td>88.97</td>
    <td>90.86</td>
    <td>90.95</td>
    <td>4.0493</td>
    <td>3.5893</td>
    <td>3.6219</td>
  </tr>
  <tr>
    <td>News</td>
    <td>77.18</td>
    <td>78.65</td>
    <td>78.96</td>
    <td>91.06</td>
    <td>93.39</td>
    <td>93.71</td>
    <td>4.6709</td>
    <td>4.1914</td>
    <td>4.1423</td>
  </tr>
  <tr>
    <td>Bio</td>
    <td>83.70</td>
    <td>84.75</td>
    <td>84.93</td>
    <td>91.61</td>
    <td>93.06</td>
    <td>93.28</td>
    <td>4.0794</td>
    <td>3.7218</td>
    <td>3.7051</td>
  </tr>
  <tr>
    <td>Talk</td>
    <td>82.59</td>
    <td>83.82</td>
    <td>84.25</td>
    <td>96.43</td>
    <td>97.09</td>
    <td>97.29</td>
    <td>5.0868</td>
    <td>4.7863</td>
    <td>4.6798</td>
  </tr>
  <tr>
    <td>Finance</td>
    <td>82.56</td>
    <td>83.07</td>
    <td>83.35</td>
    <td>89.32</td>
    <td>90.42</td>
    <td>90.73</td>
    <td>4.7052</td>
    <td>4.3470</td>
    <td>4.2984</td>
  </tr>
</table>

Table 13: Performance comparison across different domains and iterations on zh → en (Qwen-Model<sub>0</sub>, Qwen-Model<sup>E</sup><sub>1</sub>, Qwen-Model<sup>T</sup><sub>2</sub>).

<table>
  <tr>
    <th>Order</th>
    <th>Win</th>
    <th>Tie</th>
    <th>Loss</th>
  </tr>
  <tr>
    <td>xTower Pipeline v.s. Qwen-Model<sup>E</sup><sub>1</sub></td>
    <td>3.870</td>
    <td>5.444</td>
    <td>6.477</td>
  </tr>
  <tr>
    <td>Qwen-Model<sup>E</sup><sub>1</sub> v.s. xTower Pipeline</td>
    <td>4.747</td>
    <td>6.371</td>
    <td>4.673</td>
  </tr>
</table>

Table 14: LLM evaluation results between Qwen-Model<sup>E</sup><sub>1</sub> and xTower Pipeline

notate errors and refine the translations. We compare the translation quality and the number of major and minor errors detected by each model.

Results in Table 19 shows that translation quality improves consistently across all metrics from Model<sub>0</sub> to Model<sup>E</sup><sub>1</sub>. Besides, error annotation becomes more precise, as later models identify fewer errors yet achieve better translations. This suggests that Model<sup>E</sup> are learning to generalize across domains, identifying more meaningful errors and reducing over-prediction.

D Ablation Study

D.1 Necessity of Model Separation

As depicted in Fig 1, SSPO algorithm separates the initial model Model<sub>0</sub> into two specialized models: Model<sup>E</sup> for error detection and Model<sup>T</sup> for translation generation. To validate this design, we conducted a controlled experiment by training a hybrid model Qwen-Model<sup>E+T</sup> that combines both capabilities, using the same DPO training data from the first iteration.

Table 20 compares the performance of three models on the zh→en multi-domain test set. The results show a clear performance hierarchy: Qwen-Model<sup>E</sup><sub>1</sub> outperforms Qwen-Model<sup>E+T</sup>, which in turn surpasses Qwen-Model<sub>0</sub>. While the hybrid model improves upon the baseline, it falls short of the specialized translation model.

This performance gap stems from the conflicting optimization objectives: DPO for translation versus SFT for error correction. Training a single model for both tasks risks catastrophic forgetting, where improving one capability degrades the other. Therefore, separating Model<sub>0</sub> into task-specific models (Model<sup>E</sup> and Model<sup>T</sup>) proves essential for optimal performance.

D.2 Effectiveness of DPO Training

To validate the effectiveness of our DPO process, we use the positive example in our preference data to SFT Qwen-Trans (the model trained solely on the translation data in A.2) and get Qwen-Trans-SFT<sub>1</sub>. We compare its performance with our Qwen-Model<sup>E</sup><sub>1</sub>. This experiment is carried out on zh → en direction with the preference data used in the first iteration.

Results in Table 21 demonstrate that our method outperforms SFT method.

E Potential Risks

This paper presents work whose goal is to advance the field of Machine Translation and Large Language Model. We used open-source data and models to do machine translation tasks without other intend. We don’t include offensive information in our data. We think we don’t have risks include potential malicious or unintended harmful effects and uses.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">da→en</th>
    <th colspan="3">de→en</th>
    <th colspan="3">fr→en</th>
  </tr>
  <tr>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
  </tr>
  <tr>
    <td>TowerInstruct</td>
    <td>90.80</td>
    <td>94.05</td>
    <td>4.8012</td>
    <td>89.31</td>
    <td>90.50</td>
    <td>5.0228</td>
    <td>88.86</td>
    <td>89.14</td>
    <td>5.0336</td>
  </tr>
  <tr>
    <td>GPT-4o-0806</td>
    <td><b>93.49</b></td>
    <td><b>96.96</b></td>
    <td><b>4.4658</b></td>
    <td>91.46</td>
    <td>96.77</td>
    <td>2.2697</td>
    <td>92.27</td>
    <td>95.69</td>
    <td>4.7092</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>0</sub></td>
    <td>92.46</td>
    <td>95.36</td>
    <td>4.6772</td>
    <td>91.19</td>
    <td>96.33</td>
    <td>2.3186</td>
    <td>92.23</td>
    <td>95.26</td>
    <td>4.8869</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>1</sub><sup>F</sup></td>
    <td>92.70</td>
    <td>95.73</td>
    <td>4.6444</td>
    <td>91.41</td>
    <td>96.52</td>
    <td>2.2997</td>
    <td>92.66</td>
    <td>95.71</td>
    <td>4.7578</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>2</sub><sup>F</sup></td>
    <td>93.38</td>
    <td>96.81</td>
    <td>4.5120</td>
    <td>91.63</td>
    <td>96.78</td>
    <td>2.2801</td>
    <td><b>92.83</b></td>
    <td><b>95.89</b></td>
    <td><b>4.6820</b></td>
  </tr>
  <tr>
    <td>Mistral-Model<sub>0</sub></td>
    <td>91.15</td>
    <td>94.14</td>
    <td>4.7889</td>
    <td>91.36</td>
    <td>96.67</td>
    <td>2.3068</td>
    <td>92.12</td>
    <td>94.81</td>
    <td>4.9447</td>
  </tr>
  <tr>
    <td>Mistral-Model<sub>1</sub><sup>F</sup></td>
    <td>92.87</td>
    <td>96.44</td>
    <td>4.5971</td>
    <td>91.58</td>
    <td>96.82</td>
    <td>2.2710</td>
    <td>92.31</td>
    <td>95.09</td>
    <td>4.7447</td>
  </tr>
  <tr>
    <td>Mistral-Model<sub>2</sub><sup>F</sup></td>
    <td><b>93.05</b></td>
    <td><b>96.69</b></td>
    <td><b>4.5230</b></td>
    <td><b>91.73</b></td>
    <td><b>96.98</b></td>
    <td><b>2.2523</b></td>
    <td>92.54</td>
    <td>95.43</td>
    <td>4.6980</td>
  </tr>
  <tr>
    <th colspan="3">id→en</th>
    <th colspan="3">zh→en</th>
    <th colspan="3">Avg.</th>
  </tr>
  <tr>
    <th>Models</th>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
  </tr>
  <tr>
    <td>TowerInstruct</td>
    <td>85.06</td>
    <td>83.06</td>
    <td>5.8541</td>
    <td>78.92</td>
    <td>90.22</td>
    <td>4.7780</td>
    <td>86.59</td>
    <td>89.39</td>
    <td>5.0979</td>
  </tr>
  <tr>
    <td>GPT-4o-0806</td>
    <td>89.39</td>
    <td>96.27</td>
    <td>4.7982</td>
    <td>80.29</td>
    <td>91.54</td>
    <td>4.3310</td>
    <td>89.38</td>
    <td>95.45</td>
    <td>4.1148</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>0</sub></td>
    <td>89.48</td>
    <td>96.16</td>
    <td>4.8467</td>
    <td>79.42</td>
    <td>90.37</td>
    <td>4.6393</td>
    <td>88.96</td>
    <td>94.70</td>
    <td>4.2737</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>1</sub><sup>F</sup></td>
    <td>89.49</td>
    <td>96.43</td>
    <td>4.7885</td>
    <td>80.38</td>
    <td>92.40</td>
    <td>4.2382</td>
    <td>89.33</td>
    <td>95.36</td>
    <td>4.1457</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>2</sub><sup>F</sup></td>
    <td><b>89.65</b></td>
    <td><b>96.58</b></td>
    <td><b>4.7120</b></td>
    <td><b>80.67</b></td>
    <td><b>92.66</b></td>
    <td><b>4.2187</b></td>
    <td><b>89.63</b></td>
    <td><b>95.74</b></td>
    <td><b>4.0810</b></td>
  </tr>
  <tr>
    <td>Mistral-Model<sub>0</sub></td>
    <td>89.23</td>
    <td>96.02</td>
    <td>4.8796</td>
    <td>79.16</td>
    <td>90.31</td>
    <td>4.7038</td>
    <td>88.6</td>
    <td>94.39</td>
    <td>4.3248</td>
  </tr>
  <tr>
    <td>Mistral-Model<sub>1</sub><sup>F</sup></td>
    <td>89.28</td>
    <td>96.13</td>
    <td>4.8100</td>
    <td>79.91</td>
    <td>92.77</td>
    <td>4.4024</td>
    <td>89.19</td>
    <td>95.45</td>
    <td>4.1650</td>
  </tr>
  <tr>
    <td>Mistral-Model<sub>2</sub><sup>F</sup></td>
    <td>89.41</td>
    <td>96.32</td>
    <td>4.7290</td>
    <td>80.29</td>
    <td>93.08</td>
    <td>4.3354</td>
    <td>89.40</td>
    <td>95.7</td>
    <td>4.1075</td>
  </tr>
</table>

Table 15: Performance comparison on xx→en translation across different language pairs.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">en→da</th>
    <th colspan="3">en→de</th>
    <th colspan="3">en→fr</th>
  </tr>
  <tr>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
  </tr>
  <tr>
    <td>TowerInstruct</td>
    <td>86.69</td>
    <td>94.81</td>
    <td>1.3383</td>
    <td>86.19</td>
    <td>98.17</td>
    <td>0.8424</td>
    <td>90.79</td>
    <td>96.43</td>
    <td>1.1191</td>
  </tr>
  <tr>
    <td>GPT-4o-0806</td>
    <td><b>92.11</b></td>
    <td><b>97.53</b></td>
    <td><b>0.9415</b></td>
    <td>87.06</td>
    <td>98.33</td>
    <td>0.8053</td>
    <td><b>90.88</b></td>
    <td><b>96.57</b></td>
    <td><b>1.1087</b></td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>0</sub></td>
    <td>75.37</td>
    <td>92.73</td>
    <td>1.4860</td>
    <td>85.04</td>
    <td>97.94</td>
    <td>0.8839</td>
    <td>89.35</td>
    <td>95.53</td>
    <td>1.2164</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>1</sub><sup>F</sup></td>
    <td>83.24</td>
    <td>92.99</td>
    <td>1.5154</td>
    <td>85.13</td>
    <td>98.04</td>
    <td>0.8772</td>
    <td>89.87</td>
    <td>95.73</td>
    <td>1.2060</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>2</sub><sup>F</sup></td>
    <td>84.62</td>
    <td>93.53</td>
    <td>1.3920</td>
    <td>85.32</td>
    <td>98.18</td>
    <td>0.8630</td>
    <td>90.36</td>
    <td>96.01</td>
    <td>1.1680</td>
  </tr>
  <tr>
    <td>Mistral-Model<sub>0</sub></td>
    <td>88.44</td>
    <td>96.08</td>
    <td>1.1123</td>
    <td>86.61</td>
    <td>98.41</td>
    <td>0.8076</td>
    <td>90.23</td>
    <td>96.08</td>
    <td>1.1450</td>
  </tr>
  <tr>
    <td>Mistral-Model<sub>1</sub><sup>F</sup></td>
    <td>89.13</td>
    <td>96.21</td>
    <td>1.0786</td>
    <td>86.84</td>
    <td>98.39</td>
    <td>0.7931</td>
    <td>90.16</td>
    <td>96.21</td>
    <td>1.1407</td>
  </tr>
  <tr>
    <td>Mistral-Model<sub>2</sub><sup>F</sup></td>
    <td><b>89.65</b></td>
    <td><b>96.49</b></td>
    <td><b>1.0230</b></td>
    <td><b>87.21</b></td>
    <td><b>98.54</b></td>
    <td><b>0.7800</b></td>
    <td><b>90.58</b></td>
    <td><b>96.44</b></td>
    <td><b>1.1190</b></td>
  </tr>
  <tr>
    <th colspan="3">en→id</th>
    <th colspan="3">en→zh</th>
    <th colspan="3">Avg.</th>
  </tr>
  <tr>
    <th>Models</th>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
  </tr>
  <tr>
    <td>TowerInstruct</td>
    <td>84.48</td>
    <td>92.23</td>
    <td>1.7846</td>
    <td>86.19</td>
    <td>90.66</td>
    <td>1.5572</td>
    <td>86.87</td>
    <td>94.46</td>
    <td>1.3283</td>
  </tr>
  <tr>
    <td>GPT-4o-0806</td>
    <td><b>90.43</b></td>
    <td><b>95.97</b></td>
    <td><b>1.2112</b></td>
    <td><b>86.90</b></td>
    <td>91.57</td>
    <td><b>1.4737</b></td>
    <td><b>89.48</b></td>
    <td><b>95.99</b></td>
    <td><b>1.1081</b></td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>0</sub></td>
    <td>87.45</td>
    <td>94.12</td>
    <td>1.4011</td>
    <td>85.60</td>
    <td>91.19</td>
    <td>1.5656</td>
    <td>84.56</td>
    <td>94.3</td>
    <td>1.3106</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>1</sub><sup>F</sup></td>
    <td>88.26</td>
    <td>94.49</td>
    <td>1.4094</td>
    <td>85.89</td>
    <td>91.49</td>
    <td>1.4094</td>
    <td>86.48</td>
    <td>94.55</td>
    <td>1.2835</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>2</sub><sup>F</sup></td>
    <td><b>88.73</b></td>
    <td><b>94.82</b></td>
    <td><b>1.3680</b></td>
    <td><b>86.35</b></td>
    <td><b>91.72</b></td>
    <td><b>1.3720</b></td>
    <td><b>87.08</b></td>
    <td><b>94.85</b></td>
    <td><b>1.2326</b></td>
  </tr>
  <tr>
    <td>Mistral-Model<sub>0</sub></td>
    <td>87.45</td>
    <td>93.86</td>
    <td>1.4874</td>
    <td>85.62</td>
    <td>90.40</td>
    <td>1.5505</td>
    <td>87.67</td>
    <td>94.97</td>
    <td>1.2206</td>
  </tr>
  <tr>
    <td>Mistral-Model<sub>1</sub><sup>F</sup></td>
    <td>87.35</td>
    <td>93.95</td>
    <td>1.4808</td>
    <td>85.88</td>
    <td>90.64</td>
    <td>1.5212</td>
    <td>87.87</td>
    <td>95.08</td>
    <td>1.2029</td>
  </tr>
  <tr>
    <td>Mistral-Model<sub>2</sub><sup>F</sup></td>
    <td><b>87.79</b></td>
    <td><b>94.23</b></td>
    <td><b>1.4350</b></td>
    <td><b>86.21</b></td>
    <td><b>90.98</b></td>
    <td><b>1.4830</b></td>
    <td><b>88.29</b></td>
    <td><b>95.34</b></td>
    <td><b>1.1680</b></td>
  </tr>
</table>

Table 16: Performance comparison on en→xx translation across different language pairs.

<table>
  <tr>
    <th>Iteration</th>
    <th>#Monolingual</th>
    <th>#Erroneous</th>
    <th>#Error-free</th>
    <th>Error Rate</th>
  </tr>
  <tr>
    <td>1 (Qwen-Model<sub>0</sub> annotate Qwen-Model<sub>0</sub>)</td>
    <td>39880</td>
    <td>19290</td>
    <td>20590</td>
    <td>48.4%</td>
  </tr>
  <tr>
    <td>2 (Qwen-Model<sub>1</sub><sup>F</sup> annotate Qwen-Model<sub>1</sub><sup>F</sup>)</td>
    <td>25533</td>
    <td>9465</td>
    <td>16088</td>
    <td>37.0%</td>
  </tr>
</table>

Table 17: Error rates during actual SSPO iterations. #Monolingual represents the total number of monolingual sentences processed in each iteration. #Erroneous shows the count of sentences identified as containing translation errors. #Error-free indicates the number of sentences identified to have acceptable translations without errors.
# Example1
## Question
source: 'It is particularly satisfying that Ruby has produced a female calf – these are vital for the development of the breeding programme and Stella will eventually move to another collection to hopefully become a successful mother herself.'
translation: Es ist besonders befriedigend, dass Ruby ein weibliches Kalb hervorgebracht hat – diese sind für die Entwicklung des Zuchtprogramms von entscheidender Bedeutung – und Stella wird schließlich in eine andere Kollektion wechseln, um hoffentlich selbst eine erfolgreiche Mutter zu werden".
errors: There is a Major error at "Kollektion". There is a Minor error at "Bedeutung –". There is a Minor error at "-". There is a Minor error at "".
This is an English-to-German translation error annotation task. Only the location and severity of errors are provided. Please analyze each error by considering the context in the translation field and the English source text, and provide explanations and improvement suggestions that conform to German grammar, conventions and idiomatic usage.

## Answer
[{
    "location": "Kollektion",
    "severity": "major",
    "explanation": "The term 'Kollektion' is not appropriate for referring to a group or place where animals are kept or bred. In the context of animal breeding programs, 'Sammlung' or 'Zuchtbestand' would be more suitable.",
    "improvement": "Change 'Kollektion' to 'Sammlung' or 'Zuchtbestand'."
},
{
    "location": "Bedeutung –",
    "severity": "minor",
    "explanation": "The use of a dash here is not typical in German writing and may disrupt the sentence structure. A comma or simply ending the sentence would be more appropriate.",
    "improvement": "Replace the dash with a comma or end the sentence after 'Bedeutung'."
}]

# Example2
...

# Example3
...

# Your task
## Question
source: {source}
translation: {translation}
errors: {errors}
This is an English-to-German translation error annotation task. Only the location and severity of errors are provided. Please analyze each error by considering the context in the translation field and the English source text, and provide explanations and improvement suggestions that conform to German grammar, conventions and idiomatic usage.

## Answer

Figure 4: Prompt template for en → de translation error annotation. The template demonstrates various error types including terminology misuse, punctuation errors, and grammatical issues, with detailed explanations and improvements for each error.

<table>
  <tr>
    <th>Model</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>TP</th>
    <th>FP</th>
    <th>FN</th>
  </tr>
  <tr>
    <td>Qwen-Model<sub>0</sub></td>
    <td>0.3430</td>
    <td>0.5028</td>
    <td>0.4078</td>
    <td>18473</td>
    <td>35381</td>
    <td>18267</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>E</sub><sup>F</sup></td>
    <td>0.5119</td>
    <td>0.5935</td>
    <td>0.5497</td>
    <td>22638</td>
    <td>21588</td>
    <td>15505</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>E</sub><sup>F</sup><sub>2</sub></td>
    <td><b>0.5190</b></td>
    <td><b>0.6109</b></td>
    <td><b>0.5612</b></td>
    <td><b>22954</b></td>
    <td><b>21270</b></td>
    <td><b>14623</b></td>
  </tr>
</table>

Table 18: Error detection results on WMT22 zh → en MQM test set.
## Question
Given the English source text, the initial German translation, and the list of identified errors with explanations and suggested improvements, improve the initial German translation to make it accurate, fluent, and true to the meaning and tone of the original text.

English source: Google is marking its own birthday today with a doodle. The doodle features a photo of a 90s’ style computer with Google search page and date stamp for September 27, 1998, surrounded by confetti in Google colours. A smaller doodle with number 21 as candles, as a part of Google logo, shows on the search results page. The search giant was founded 21 years ago in 1998 by Larry Page and Sergey Brin, then students at Stanford University in California. September 27 is, however, not Google’s actual birthday.

Initial German translation: Google feiert heute seinen eigenen Geburtstag mit einem Doodle. Das Doodle zeigt ein Foto eines Computers im Stil der 90er mit Google-Suchseite und Datumsstempel für den 27. September 1998, umgeben von Konfetti in Google-Farben. Ein kleineres Doodle mit der Nummer 21 als Kerzen als Teil des Google-Logos zeigt sich auf der Suchergebnisseite. Der Suchriese wurde 1998 vor 21 Jahren von Larry Page und Sergey Brin, damals Studenten an der Stanford University in Kalifornien, gegründet.

Errors: [
{
    "location": "90er",
    "severity": "minor",
    "explanation": "While '90er' is commonly used, it should be followed by 'Jahre' to fully convey the meaning of '90s' style.",
    "improvement": "Change '90er' to '90er Jahre' for clarity."
},
{
    "location": "gegründet.",
    "severity": "minor",
    "explanation": "The final sentence about September 27 not being Google’s actual birthday is missing.",
    "improvement": "Add 'Der 27. September ist jedoch nicht Googles eigentlicher Geburtstag.' at the end."
}]

Improved Translation: Google feiert heute seinen eigenen Geburtstag mit einem Doodle. Das Doodle zeigt ein Foto eines Computers im Stil der 90er Jahre mit Google-Suchseite und Datumstempel für den 27. September 1998, umgeben von Konfetti in Google-Farben. Ein kleineres Doodle mit der Nummer 21 als Kerzen als Teil des Google-Logos zeigt sich auf der Suchergebnisseite. Der Suchriese wurde 1998 vor 21 Jahren von Larry Page und Sergey Brin, damals Studenten an der Stanford University in Kalifornien, gegründet. Der 27. September ist jedoch nicht Googles eigentlicher Geburtstag.

# Your task
Given the English source text, the initial German translation, and the list of identified errors with explanations and suggested improvements, improve the initial German translation to make it accurate, fluent, and true to the meaning and tone of the original text.

English source: {source}
Initial German translation: {translation}
Errors: {errors}
Improved Translation:

Figure 5: Prompt template for translation improvement task. The template shows how to analyze translation errors and make improvements while maintaining accuracy and fluency in German. The example demonstrates handling of missing content and stylistic refinements.

<table>
  <tr>
    <th>Model</th>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
    <th>#Major</th>
    <th>#Minor</th>
  </tr>
  <tr>
    <td>Qwen2.5-14B-Instruct</td>
    <td>78.52</td>
    <td>90.54</td>
    <td>4.8249</td>
    <td>\</td>
    <td>\</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>0</sub> Refine</td>
    <td>78.53</td>
    <td>91.28</td>
    <td>4.7158</td>
    <td>7906</td>
    <td>11927</td>
  </tr>
  <tr>
    <td>Qwen-Model<sup>E</sup><sub>1</sub> Refine</td>
    <td>79.00</td>
    <td>91.64</td>
    <td>4.6537</td>
    <td>4340</td>
    <td>3856</td>
  </tr>
  <tr>
    <td>Qwen-Model<sup>E</sup><sub>2</sub> Refine</td>
    <td><b>79.21</b></td>
    <td><b>91.82</b></td>
    <td><b>4.6329</b></td>
    <td>4832</td>
    <td>3319</td>
  </tr>
</table>

Table 19: Error annotation and improvement results of different Model<sup>E</sup> on the initial translations generated by qwen2.5-14b-instruct for the zh→en multi-domain test set.
{
    "source": "市内一座商场同样倒塌，数百名居民赶到现场，等候亲友的音讯。",
    "translation": "A shopping mall collapsed, and hundreds of residents rushed to the scene, waiting for the audio of friends and relatives.",
    "original errors": "There is a major error at \"audio\"",
    "augmented errors": [
        {
            "location": "audio",
            "severity": "major",
            "explanation": "The term '音讯' in the original text is misinterpreted as 'audio'. In this context, '音讯' means 'news' or 'information' regarding the safety of friends and relatives, not 'audio'.",
            "improvement": "Change 'audio of friends and relatives' to 'news of friends and relatives'."
        }
    ]
}

Figure 6: An example of the augmented MQM data. Compared to the orginal MQM data, wo add explanations for each error and provide improvement suggestions.

<table>
  <tr>
    <th>Model</th>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
  </tr>
  <tr>
    <td>Qwen-Model<sub>0</sub></td>
    <td>79.42</td>
    <td>90.37</td>
    <td>4.6393</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>1</sub><sup>E+T</sup></td>
    <td>80.01</td>
    <td>91.50</td>
    <td>4.4495</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>1</sub><sup>T</sup></td>
    <td><b>80.38</b></td>
    <td><b>92.40</b></td>
    <td><b>4.2382</b></td>
  </tr>
</table>

Table 20: Performance comparison of model separation strategy.

<table>
  <tr>
    <th>Model</th>
    <th>KIWI</th>
    <th>XCOMET</th>
    <th>METRICX↓</th>
  </tr>
  <tr>
    <td>Qwen-Trans-SFT<sub>1</sub></td>
    <td>80.22</td>
    <td>91.56</td>
    <td>4.3690</td>
  </tr>
  <tr>
    <td>Qwen-Model<sub>1</sub><sup>T</sup></td>
    <td><b>80.38</b></td>
    <td><b>92.40</b></td>
    <td><b>4.2382</b></td>
  </tr>
</table>

Table 21: Performance comparison between Qwen-Trans-SFT<sub>1</sub> and Qwen-Model<sub>1</sub><sup>T</sup>. The best results are in bold.

Reviewer paper summaries: This paper proposes Self-Supervised Optimization (SSPO), a framework which constructs translation preference data for iterative DPO training.
Summary of strengths: 1. The writing is good.

2. This paper presents a lot of analysis of the experiment.
Summary of weaknesses: 1. A highly-similar work is not mentioned or cited, "Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Level in EMNLP 2024," which also proposes a translation refinement data construction strategy and employs an iterative training process.

2. Technical novelty is limited: The pipeline is constructed with a supervised finetuned translation refinement model for iterative DPO.

3. The design of SSPO is confusing: It constructs a preference dataset containing tuples of (x, y_w, y_l), where y_w and y_l are the better and worse translations, respectively. Why not directly use y_gt as y_w? I think the training dataset includes annotations for y_gt.

4. Missing ablation study for the proposed Model_E: Could the model be replaced by existing models? Could we directly use a reward model to annotate y_w and y_l? It seems on-policy optimization would lead to better optimization.
Comments, suggestions and typos: See Weakness.
Overall assessment scores: 2