CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation
Authors: Guofeng Cui, Pichao WANG, Yang Liu, Zemian Ke, Zhu Liu, Vimal Bhat
Accepted at: acl25

CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation

Guofeng Cui1*, Pichao Wang2, Yang Liu2, Zemian Ke2, Zhu Liu2, Vimal Bhat2

1Rutgers University, 2Amazon
gc669@cs.rutgers.edu, {pichaowang,kezemian}@gmail.com, {yangliu,zhuzliu,vimalb}@amazon.com

Abstract
Large language models (LLMs) have shown great potential in natural language processing tasks, but their application to machine translation (MT) remains challenging due to pretraining on English-centric data and the complexity of reinforcement learning from human feedback (RLHF). Direct Preference Optimization (DPO) has emerged as a simpler and more efficient alternative, but its performance depends heavily on the quality of preference data. To address this, we propose Confidence-Reward driven Preference Optimization (CRPO), a novel method that combines reward scores with model confidence to improve data selection for fine-tuning. CRPO selects challenging sentence pairs where the model is uncertain or underperforms, leading to more effective learning. While primarily designed for LLMs, CRPO also generalizes to encoder-decoder models like NLLB, demonstrating its versatility. Empirical results show that CRPO outperforms existing methods such as RS-DPO, RSO and MBR score in both translation accuracy and data efficiency.

1 Introduction
Recent advances in decoder-only large language models (LLMs), such as GPT series (Achiam et al., 2023), LLaMA (Touvron et al., 2023; Dubey et al., 2024), and Falcon (Almazrouei et al., 2023), have showcased their outstanding ability to understand context and perform various natural language processing (NLP) tasks. However, applying LLMs to machine translation (MT) remains a challenging endeavor, especially due to their pretraining on predominantly English-centric datasets. This limitation has generated significant interest in aligning LLMs for translation tasks using further training methods, with particular attention to enhancing their multilingual performance.

To mitigate the linguistic bias inherent in LLMs, instruction tuning has become a widely adopted approach. Instruction tuning fine-tunes LLMs using multilingual datasets and translation-specific instructions, with the goal of expanding linguistic diversity and improving translation quality (Yang et al., 2023b; Chen et al., 2023; Zhu et al., 2023b; Zhang et al., 2023). Despite these efforts, gaps remain between the performance of LLMs and specialized machine translation models (Zhu et al., 2023a). To address these challenges, approaches such as reinforcement learning from human feedback (RLHF) (Christiano et al., 2017) have been explored. RLHF allows LLMs to align with human preferences by training a reward model on human-annotated preference data and fine-tuning the LLM to maximize the predicted reward for translation quality. For example, Xu et al. (2024b) construct a preference translation dataset using multilingual books and fine-tune LLaMA-2 with RLHF to optimize translation performance.

However, RLHF introduces several complexities that hinder its efficiency. These include the need for multiple components—a reward model, a policy model, a reference policy, and a value model—which significantly increase memory and computational overhead. Additionally, the robustness of RLHF is a concern due to the disjoint training of the reward and policy models. To address these limitations, Direct Preference Optimization (DPO) (Rafailov et al., 2024) and SLiC (Zhao et al., 2022, 2023) have emerged as more efficient alternatives. These methods directly fine-tune LLMs using human preference data, bypassing the complexity of RLHF. By optimizing the model through closed-form solutions of preference objectives, DPO and SLiC have shown promise in machine translation tasks, particularly in reducing computational complexity while maintaining strong performance (Zeng et al., 2024; Wu et al., 2024).
Figure 1: Comparison among RSO, RS-DPO and CRPO. RSO scores candidate responses with reward and applies statistical reject sampling for preference dataset. RS-DPO accepts sentence pairs with reward difference surpass a preset threshold. Instead, CRPO evaluates sentence pairs with both reward difference and policy confidence.

Despite these advancements, a critical challenge persists in the selection of high-quality preference data, which is essential for offline methods like DPO and SLiC. Recent works such as LLaMA-3 (Dubey et al., 2024) emphasize the importance of careful data selection and cleaning. LLaMA-3 collects preference data from models trained on diverse data mixes, and further refines the data by filtering based on quality, difficulty, and removing semantically redundant sentence pairs. These iterative data processing steps are crucial for preventing distribution shifts and ensuring high-quality data for each round of DPO fine-tuning. However, such exhaustive data cleaning procedures come at the cost of high memory and time complexity, making them less scalable for large-scale translation tasks.

To address these limitations, recent research has explored more flexible and efficient data selection strategies. RSO (Liu et al., 2023) proposes a statistical rejection sampling method to subsample preference data from the target optimal policy, effectively estimating the optimal policy distribution. RS-DPO (Khaki et al., 2024), on the other hand, utilizes a simpler approach by selecting preference pairs based on the reward difference between sentences. RS-DPO scores a fixed number of responses for each prompt using a point-wise reward model, maintaining only those pairs with reward differences above a predefined threshold. Although these methods improve the efficiency of data selection, they primarily focus on reward values and fail to consider the model’s confidence in its predictions, which can be critical for determining which sentence pairs offer the most learning potential.

Our method, Confidence-Reward driven Preference Optimization (CRPO), as illustrate in Figure 1, aims to address these gaps by jointly considering both reward scores and model confidence for data selection, comparing with RSO that statistically selects candidate translations based on reward scores and RS-DPO that keeps sentence pairs with large reward differences. This approach allows CRPO to select data where the model struggles the most—sentence pairs with high reward differences but where the model is uncertain or incorrect—leading to more effective fine-tuning. By incorporating both of these factors, CRPO ensures that the selected data is not only high-quality but also maximizes the model’s learning potential.

While CRPO was primarily designed for LLMs, its application is not limited to decoder-only architectures. Our method also extends to encoder-decoder models, such as NLLB (No Language Left Behind) (Costa-jussà et al., 2022), which have shown strong performance in multilingual translation tasks. NLLB, which is designed to handle over 200 languages, benefits from similar preference optimization techniques, where challenging sentence pairs are selected based on both reward differences and model uncertainty. The success of CRPO on NLLB further demonstrates the method’s versatility, as it effectively addresses the challenges of both LLM-based and encoder-decoder-based translation models.

2 Preliminaries

To fine-tune LLMs with human preference annotation on machine translation, translation sentence pair given each source sentence is collected from reference policy \( \pi_{ref} \), larger LLMs such as GPT-4 or human annotator. We define the human preference dataset as \( \mathcal{D}_w = \{ x^{(i)}, y_w^{(i)}, y_l^{(i)} \}_{i=1}^N \), where \( x^{(i)} \) refers to the \( i \)th source sentence, \( y_w^{(i)} \) and
\( y_l^{(i)} \) are prefered and disprefered sentence respectively annotated by either human or reward model, and the dataset contains \( N \) sentence pairs in total. In this paper, we build a candidate set by sampling \( K \) sentences output from reference policy as \( \{y^{(ij)}\}_{j=1}^K \sim \pi_{ref}(y|x^{(i)}) \) and then score each sentence \( y^{(ij)} \) with point-wise reward model as \( r^{(ij)} = R(x^{(i)}, y^{(ij)}) \). To further construct \( \mathcal{D}_r \), sentence pairs are selected from the candidate set. Two recent preference data selection methods are shown below.

**RSO** (Liu et al., 2023) approximates optimal policy \( \pi^* \) with \( \pi_{ref} \) as:

\[
\pi^* = \frac{1}{Z(x)} \pi_{ref}(y|x) \exp(\frac{1}{\beta} r(x, y))
\] (1)

and applies statistical rejection sampling for data selection. The expected acceptance rate for each candidate sentence \( y^{(ij)} \) is \( \mathbb{E}_{y^{(ij)} \sim \pi_{ref}}[\exp(\frac{1}{\beta} : (r^{(ij)} - r_{max}))] \), where \( r_{max} \) refers to the maximum reward among candidate sentences and the reward value is the main consideration for acceptance decision.

**RS-DPO** (Khaki et al., 2024) calculates the reward difference between each sentence pair and accept a sentence pair when \( \sigma(\frac{r^{(ij)} - r_{max}}{\tau}) > \eta \) where \( \eta \) is the threshold defined as a hyperparameter and \( \sigma(\cdot) \) is the sigmoid function with the formula of \( \sigma(x) = \frac{1}{1+e^{-x}} \).

Given the preference dataset, both DPO and RLHF fine-tune LLMs to optimize the following objective:

\[
\max_{\theta} \mathbb{E}_{x^{(i)} \sim P, y^{(i)} \sim \pi_\theta}[R(x^{(i)}, y^{(i)})] - \beta \mathbb{D}_{KL}[\pi_\theta(y|x)|\pi_{ref}(y|x)]
\] (2)

where \( \theta \) refers to the parameter of current policy \( \pi_\theta \) and \( \pi_{ref} \) refers to the reference policy. DPO calculates the closed form solution of Eq. 2 and defines the loss function with Bradley-Terry (BT) model (Bradley and Terry, 1952) as:

\[
\mathcal{L}_{DPO}(\pi_\theta; \pi_{ref}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_r}
[\log \sigma(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)})]
\] (3)

which directly fine-tunes LLM on the preference dataset.

Xu et al. (2024a) propose contrastive preference optimization (CPO) to set the reference policy as uniform prior \( U \) for efficiency and act as an upper bound of DPO loss, the format of which is defined to be:

\[
\mathcal{L}_{CPO}(\pi_\theta; U) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_r}
[\log \sigma(\beta \log \pi_\theta(y_w|x) - \beta \log \pi_\theta(y_l|x))]
\] (4)

To further enhance LLMs performance, SFT term is added to CPO to clone the behavior of preferred sentences. Moreover, considering the importance of data quality to offline training, Xu et al. (2024a) construct preference dataset with translation sentences from GPT4, human annotators and pretrained policy, named as triplet dataset.

Sentences with higher quality or reward could benefit the training of preference optimization, which is the main concern of RSO, RS-DPO and the triplet dataset used in CPO. But the performance of policy \( \pi_\theta \) is also important and not considered in these methods. Although RSO jointly leverages the distribution of \( \pi_{ref} \) and optimal policy, the acceptance rate is mainly based on reward.

3 **CRPO: Confidence-Reward Driven Preference Optimization**

Instead of reward, we consider the acceptance of sentence pairs with the value of optimization loss in two ways, loss value and loss change. A higher loss value indicates that the policy achieves worse performance and the information of related data has not been learnt sufficiently. Similarly, a large loss change during training indicates that the policy extracts useful information from the related data to reduce the prediction confusion or even correct the error prediction. Thus the sentence pairs with either high loss value or loss change are potential to benefit model fine-tuning. In this section, we analyze these two terms on DPO loss and derive two formulations of Confidence-Reward Score (CR-Score) for data selection respectively, Confidence-Reward Plus (CR+) to measure loss change and Confidence-Reward Multiplication (CR×) to measure loss value. Although the derivation is different, we will show that both these two scores share the idea of combining model confidence with sentence reward.

3.1 **CR+: Derivation from Loss Change**

We start with the derivation from loss change. For the reason that both \( \log \) and \( \sigma \) are monotonic increasing functions, we simplify the loss change as the difference of minus term inside \( \sigma \) function of
Eq. 3 during training. Formally, taking two parameters \( \theta_1 \) and \( \theta_2 \), the loss change is defined as:

\[
\Delta_{\theta} \mathcal{L} := [\log \pi_{\theta_2}(y_w|x) - \log \pi_{\theta_2}(y_l|x)] + [\log \pi_{\theta_1}(y_l|x) - \log \pi_{\theta_1}(y_w|x)]
\] (5)

where \( \theta_1 \) is the parameter before the fine-tuning and we set \( \pi_{\theta_1} \) to be \( \pi_{ref} \). \( \theta_2 \) is the parameter after certain steps of fune-tuning, which is hard to be calculated specifically. One potential way to approximate \( \pi_{\theta_2} \) is to use the target optimal policy \( \pi^* \) in Eq. 1 and the loss change will be driven into:

\[
\Delta_{\theta} \mathcal{L} := \frac{R(x, y_w) - R(x, y_l)}{\beta}
\] (6)

which is exactly the selection metric used in RS-DPO. But in practise, the translation ability is the main concern during inference which is measured by the reward. As also mentioned in Meng et al. (2024), we expect the trained policy to have higher probability to generate high-reward sentence which is different from the distribution of the optimal policy. Noted that \( \Delta_{\theta} \mathcal{L} \) is defined for data selection which should serve our practical purpose, so we directly approximate \( \pi_{\theta_2} \) as the distribution following reward value as:

\[
\pi_{\theta_2}(y|x) := \frac{1}{Z_r(x)} \exp(K \cdot R(x, y))
\] (7)

Compared with the optimal policy, \( \pi_{ref} \) is not included and \( \phi \) is a hyperparameter that represents how much we trust the reward model and does not necessarily equal to \( \frac{1}{2} \). As a result, we define the CR+ as \( \Delta_{\theta} \mathcal{L} \) with the formulation derived from Eq. 5 and Eq. 7 as:

\[
\text{CR+} := \phi \cdot [R(x, y_w) - R(x, y_l)] + \underbrace{[\log \pi_{ref}(y_l|x) - \log \pi_{ref}(y_w|x)]}_{\text{Reward}} \underbrace{[\log \pi_{ref}(y_l|x) - \log \pi_{ref}(y_w|x)]}_{\text{Confidence}}
\] (8)

where the first term is about reward difference between sentence pairs and the second term is the likelihood difference of \( \pi_{ref} \) prediction. The larger value of second term indicates that \( \pi_{ref} \) has higher confidence on generating \( y_l \) rather than \( y_w \). Additionally, the two terms in CR+ conflict each other that the reward term encourages a larger reward for \( y_w \) while the confidence term prefers smaller likelihood of \( y_w \) generation. In other words, CR+ selects sentence pairs with larger reward difference and worse performance for \( \pi_{ref} \) to distinguish their qualities. Intuitively, CR+ tends to select dispreferred sentences that \( \pi_{ref} \) tends to generate but refuses to generate after training and preferred sentences that \( \pi_{ref} \) fails to generate but tends to generate after fine-tuning, which thus leads to large behavior change of policy.

3.2 CR×: Derivation from Loss Value

We then consider the derivation from loss value. For the reason that in the first iteration \( \pi_{\theta} \) is always set to be \( \pi_{ref} \), \( \mathcal{L}_{DPO} \) is a constant and cannot be used for data selection. So we base on \( \mathcal{L}_{CPO} \) instead to evaluate sentence pairs. Similar to CR+, we only consider the minus term inside \( \sigma \) function of Eq. 4 and construct a more general format as follow:

\[
\mathcal{L}(\pi_{\theta}) = \gamma(R(x, y_w) - R(x, y_l)) \cdot [\log \pi_{\theta}(y_w|x) - \log \pi_{\theta}(y_l|x)]
\] (9)

where \( \gamma(\cdot) \) is a mapping function measuring the correlation between reward score \( R(x, y) \) and the translation quality of sentence. For CPO loss, \( \gamma(\cdot) \) is set to be the following format:

\[
\gamma(\cdot) = \begin{cases}
1 & \text{for } R(x, y_w) > R(x, y_l) \\
-1 & \text{otherwise}
\end{cases}
\] (10)

where the reward model is totally trusted. In this case, a sentence with higher reward is considered to have sufficient quality advantage compared to that with lower reward, regardless of the reward difference. This format of \( \gamma(\cdot) \) does not fit our goal for two reasons. Firstly, error exists in reward model and a margin needs to be maintained for reward difference. Secondly, it is not reasonable to force the policy to separate sentence pairs with small reward difference. As a result, we need the \( \gamma(\cdot) \) to represent the reward gap in order to measure the quality and trustiness of sentence pair. In practise, with point-wise reward model outputting reward within the range of [0, 1], we set:

\[
\gamma(R(x, y_w) - R(x, y_l)) = R(x, y_w) - R(x, y_l)
\] (11)

the value of which also falls within the range of [0, 1] when \( R(x, y_w) > R(x, y_l) \). Specially, for a ground truth and an irrelevant sentence outputs, the ideal value of \( R(x, y_w) - R(x, y_l) \) is closed to 1. For the reason that a larger \( \mathcal{L}_{CPO} \) desire a smaller minus term inside \( \sigma \) function, we define CR× as
the minus of \( \mathcal{L} \) as:

\[
\text{CR}\times := -\mathcal{L}(\pi_{ref})
= [R(x, y_w) - R(x, y_l)] \cdot \\
\underbrace{\frac{\log \pi_{ref}(y_l|x) - \log \pi_{ref}(y_w|x)}}_{\text{Confidence}}
\]

which is the multiplication of the reward term and the confidence term. Similar to CR+, CR× also tends to select sentence pairs with larger reward difference and worse performance of \( \pi_{ref} \). Intuitively, CR× selects sentence pairs with sufficient quality gap while \( \pi_{ref} \) fails to distinguish, leading to trustworthy large loss value.

3.3 Further Discussion

Comparison between CR+ and CR×. Although CR+ is derived from loss change and CR× from loss value, both scores incorporate reward and confidence terms, aiming to maximize the discrepancy between the reward and the policy \( \pi_{ref} \). While it is possible to multiply CR+ with an additional reward term, as in CR×, this would introduce redundancy, as CR+ already contains a reward component. The key distinction between the two lies in the way they handle the magnitude difference between the reward and confidence terms. In CR+, this difference necessitates careful tuning of the hyperparameter \( \phi \), which not only adjusts for the reliability of the reward model but also bridges the gap between the reward and confidence scales. In contrast, CR× naturally balances the two terms through multiplication, eliminating the need for such manual adjustments. In practice, for a specific task and LLM, we estimate \( \phi \) by selecting reward and confidence values that best approximate a balanced contribution from both terms, ensuring robustness across various settings.

Why CR-Score? In machine translation, methods like DPO and CPO have proven effective for fine-tuning LLMs, but the challenge of selecting high-quality preference data remains unresolved. CR-Score offers a systematic approach to evaluate the potential contribution of sentence pairs before the actual model training, thereby guiding more informed data selection. Unlike RS-DPO and RSO, which focus primarily on reward scores, CR-Score incorporates the likelihood of sentence generation by the LLM. This enables the exclusion of "easy" sentence pairs—those where the model already performs well—focusing instead on pairs where the model is uncertain, maximizing the impact of each data point on fine-tuning.

CRPO Algorithm. The algorithm for data selection with CR-Score is outlined in Appendix A. Instead of evaluating all sentence pairs, we begin by selecting the sentence with the highest reward score, \( y_w \), to ensure a baseline of sentence quality. This approach, similar to that used in CPO and recent DPO applications (e.g., LLaMA-3 (Dubey et al., 2024)), enhances fine-tuning by prioritizing high-reward sentences. Moreover, we filter out sentence pairs with negative CR-Score, ensuring that only the most informative data is retained in the preference dataset. By combining confidence-reward-driven data selection with preference optimization, CRPO creates a more effective fine-tuning strategy that balances model confidence and reward, enhancing overall model performance.

4 Related Works

Preference Optimization for Machine Translation. To align LLMs with human preference and enhance their translation ability, RLHF is introduced to fine-tune the language model (Christiano et al., 2017). In order to improve the robustness and efficiency of RLHF, DPO (Rafailov et al., 2024) calculates closed-form solution on RLHF object to optimize BT model and directly train LLMs on preference dataset. CPO (Xu et al., 2024a) develops upon DPO to release the complexity caused by the requirement of reference model and add SFT term for behavior cloning. Although these preference optimization methods achieve dramatic success, the offline training strategy causes their sensitivity toward the quality of preference data. To address this problem, we analyze the loss value and loss change and propose CR-Score to effectively select essential sentence pairs to reach the DPO objective.

Rejection Sampling. To select preference data for alignment, rejection sampling is a widely adopted method. RSO (Liu et al., 2023) introduces statistical rejection sampling (Neal, 2003) and sample preference sentences from target policy distribution. ReST (Gulcehre et al., 2023) iteratively increase reward threshold and apply rejection sampling to select higher quality sentence for further RLHF step. RS-DPO (Khaki et al., 2024) instead sets the threshold of reward difference between sentence pairs and only maintains those with large enough preference difference. Specifically for the
machine translation task, the MBR score (Yang et al., 2023a; Finkelstein et al., 2023) leverages reference-based metric to estimate the expected utility of each candidate translation in relation to the set of pseudo-references. To reduce the computational complexity, Finkelstein et al. (2023) further consider to score translations with QE metric and fine-tune LLM with the best translation result. However, these sampling methods only focus on reward value neglecting the performance of the pretrained policy. Instead, our proposed CR-Score considers reward and policy confidence together.

5 Experiments

We evaluate CRPO on machine translation task and compare it with five baselines, evaluated with COMET and BLEURT (Sellam et al., 2020) metrics. Moreover, we adopt ablation studies to consider more data selection strategies and the effect of reward and confidence term on CR-Score.

5.1 Dataset

Following CPO (Xu et al., 2024a), we consider 10 translation directions in this paper: \( en \leftrightarrow zh \), \( en \leftrightarrow de \), \( en \leftrightarrow cs \), \( en \leftrightarrow is \), \( en \leftrightarrow ru \). Our preference training dataset of machine translation task is derived from FLORES-200 dataset (Costajussà et al., 2022) with the same source sentences applied to fine-tune ALMA (Xu et al., 2023) in CPO. In the training dataset, 3,065 source sentences are contained in each of \( en \leftrightarrow zh \) and \( en \leftrightarrow de \), and 2,009 source sentences are contained in each of other translation directions. In total, 24,314 source sentences are included. Note that ALMA is also pretrained on a subset of FLORES-200 dataset, we collect 64 candidate sentences for each source sentence with the pretrained ALMA to release distribution shift problem which results in 784,640 candidate translation sentences. The sampling temperature is set to be 0.9 and top-p is set to be 0.9. To evaluate the quality of translation sentences during preference dataset construction, we use two 3.5B COMET models, Unbabel/XCOMET-XL (Guerreiro et al., 2023) and Unbabel/wmt23-cometkiwi-da-xl (Rei et al., 2023), as reward models and average the two output scores from them as the final reward of translation sentences. Moreover, we follow CPO to extract data of \( en \leftrightarrow is \) from WMT21 (Freitag et al., 2021) and data of the other 8 translation directions from WMT22 (Freitag et al., 2022) as test set, resulting in 17,471 translation pairs in total.

5.2 Experiment Setup

We train the ALMA-7B in a many-to-many multilingual translation manner, starting with ALMA-7B-Pretrain-LoRA as initial checkpoint. Then we sample preference dataset with CR-Score and apply DPO to fine-tune the pretrained ALMA-7B model on preference dataset. Then we evaluate the translation results with COMET models. Besides the reward models XCOMET and KIWI-XL, we also utilize COMET-22 (Unbabel/wmt22-comet-da) and KIWI-22 (Unbabel/wmt22-cometkiwi-da) for fair comparison which are not involved in either data selection or model fine-tuning. During inference, we generate the final output of ALMA-7B with beam search, setting beam size to be 5 and maximum sequence length to be 512 tokens. For more details, please refer to the Appendix B.

5.3 Baselines

We compare CRPO with five baselines, QE Fine-tuning (Finkelstein et al., 2023), RSO (Liu et al., 2023), RS-DPO (Khaki et al., 2024), MBR Score (Yang et al., 2023a) and Triplet dataset (Xu et al., 2024a). As an additional comparison, we also calculate the evaluation score of gold reference sentences from the WMT dataset.

QE Fine-tuning. We choose the sentence with the highest QE reward score from the candidate set as the target sentence to fine-tune policy.

RSO. We statistically sub-sample 8 sentences from the candidate dataset. The acceptance rate for each sentence is \( \exp(\frac{1}{\beta} \cdot (r_{ij} - r_{max})) \).

RS-DPO. For convenience in hyperparameter tuning, we replace \( \sigma(\frac{r_1 - r_2}{\tau}) \) with \( r_1 - r_2 \) for RS-DPO. For the reason that the translation direction task \( * \rightarrow en \) is harder than \( en \rightarrow * \), we set a larger \( \eta \) for the former cases. In general, we consider two groups of values for \( \eta \) for a fair comparison, specifically 0.6 for \( en \rightarrow * \) and 0.5 for \( * \rightarrow en \), 0.65 for \( en \rightarrow * \) and 0.55 for \( * \rightarrow en \).

MBR Score. We calculate MBR score for candidate translation sentences with BLEURT-20 (Sellam et al., 2020) Metric. Specifically, we consider MBR-BW as selecting the best and worst translation sentences and MBR-BMW as selecting the best, middle, and worst translation sentences.

Triplet Dataset. We reuse the preference dataset from Triplet Dataset that is used for CPO training.
Figure 2: Visualization of reward score and log \( \pi_{ref} \) for the entire candidate dataset, as well as preferred and dis-preferred sentences selected by CRPO+, CRPO×, RS-DPO, RSO and MBR-BW.

Table 1: Average results on ten translation directions. **Black bold font** refers to the best result and gray bold font refers to the second best result. RS-DPO-1 (RS-DPO-2) refers to RS-DPO with \( \eta = 0.6 \) (\( \eta = 0.65 \)) for \( en \rightarrow * \) and \( \eta = 0.5 \) (\( \eta = 0.55 \)) for \( * \rightarrow en \).

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">Average</th>
  </tr>
  <tr>
    <th>KIW122</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
  </tr>
  <tr>
    <td>ALMA-7B</td>
    <td>0.8140</td>
    <td>0.8559</td>
    <td>0.9203</td>
    <td>0.7306</td>
  </tr>
  <tr>
    <td>QE Ft.</td>
    <td>0.8149</td>
    <td>0.8563</td>
    <td>0.9243</td>
    <td>0.7338</td>
  </tr>
  <tr>
    <td>Goal Ref.</td>
    <td>0.8098</td>
    <td>-</td>
    <td>0.9118</td>
    <td>0.7268</td>
  </tr>
  <tr>
    <td>RSO</td>
    <td>0.8197</td>
    <td>0.8598</td>
    <td>0.9277</td>
    <td>0.7403</td>
  </tr>
  <tr>
    <td>RS-DPO-1</td>
    <td>0.8134</td>
    <td>0.8547</td>
    <td>0.9189</td>
    <td>0.7299</td>
  </tr>
  <tr>
    <td>RS-DPO-2</td>
    <td>0.8140</td>
    <td>0.8553</td>
    <td>0.9205</td>
    <td>0.7311</td>
  </tr>
  <tr>
    <td>Triplet</td>
    <td>0.8168</td>
    <td>0.8581</td>
    <td>0.9274</td>
    <td>0.7371</td>
  </tr>
  <tr>
    <td>MBR-BW</td>
    <td>0.8174</td>
    <td>0.8589</td>
    <td>0.9240</td>
    <td>0.7357</td>
  </tr>
  <tr>
    <td>MBR-BMW</td>
    <td>0.8167</td>
    <td>0.8588</td>
    <td>0.9248</td>
    <td>0.7356</td>
  </tr>
  <tr>
    <td><b>CRPO+</b></td>
    <td><b>0.8218</b></td>
    <td><b>0.8618</b></td>
    <td><b>0.9311</b></td>
    <td><b>0.7462</b></td>
  </tr>
  <tr>
    <td><b>CRPO×</b></td>
    <td>0.8217</td>
    <td>0.8612</td>
    <td>0.9307</td>
    <td>0.7451</td>
  </tr>
</table>

5.4 Experiment Results

The average results for ten translation directions are shown in Table 1, where CRPO with CR+ achieves the best performance and CR× gets the second best results. As RSO, RS-DPO and MBR Score mainly select the preference dataset based on sentence reward, the evaluation results emphasize the benefit of adding the confidence term to consider policy behavior. Triplet dataset performs worse than RSO and CRPO, which is mainly caused by distribution shift between \( \pi_{ref} \) and response sentences from other resources. Although RS-DPO also constructs preference dataset from candidate sentences, the main reason for its worse performance we think is the performance gap of policy among different translation directions even when we already set different \( \eta \) for \( en \rightarrow * \) and \( * \rightarrow en \). For example, setting \( \eta = 0.6 \), on average around three sentence pairs will be maintained for each source sentence in \( en \rightarrow zh \) direction while only 20% of source sentences are maintained in \( en \rightarrow de \) direction. A large \( \eta \) causes the lack of information in difficult translation directions while small \( \eta \) maintains relatively useless information in easy translation directions. A potential solution is to set specific \( \eta \) for each translation direction while leading to higher computational cost. On comparison, although hyperparameter \( \phi \) also need to be set in CR+, we only need to consider the magnitude gap between reward and confidence and set one value for all translation directions which is more straightforward. We show more results in Appendix C, where CRPO achieves the best performance on almost all translation directions, which empirically proves the robustness of our method and the significant role of confidence term. Additionally, we evaluate CRPO with non-COMET family metric BLEURT in Appendix C.3 to address the concern of correlation between reward model and evaluation metric and show that CRPO also achieves the best result.

For further comparison, we visualize the distribution of reward score and log \( \pi_{ref} \) for preferred and dis-preferred sentences selected by different methods in \( en \rightarrow * \) translation directions in Figure 2 where preferred sentences always have better reward scores. Specifically for RS-DPO, as only sentence pairs with high reward gap are selected, the reward difference of sentence pairs are more obvious than other methods. However, preferred sentences tend to have higher generation likelihood than dis-preferred sentences, leading to "easy" data that model already performs well. Similar problem also exists in RSO and MBR-BW. In comparison, CRPO+ and CRPO× select preferred
Table 2: Average results on ten translation directions for NLLB. \( \eta \) is set to be 0.82 for RS-DPO.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">Average</th>
  </tr>
  <tr>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
  </tr>
  <tr>
    <td>NLLB-1.3B</td>
    <td>0.8009</td>
    <td>0.8362</td>
    <td>0.8947</td>
    <td>0.7001</td>
  </tr>
  <tr>
    <td>QE Ft.</td>
    <td>0.7890</td>
    <td>0.8201</td>
    <td>0.8670</td>
    <td>0.6770</td>
  </tr>
  <tr>
    <td>Goal Ref.</td>
    <td>0.8098</td>
    <td>-</td>
    <td><b>0.9118</b></td>
    <td><b>0.7268</b></td>
  </tr>
  <tr>
    <td>RSO</td>
    <td>0.8142</td>
    <td>0.8466</td>
    <td>0.9066</td>
    <td>0.7183</td>
  </tr>
  <tr>
    <td>RS-DPO</td>
    <td>0.8078</td>
    <td>0.8406</td>
    <td>0.8972</td>
    <td>0.7084</td>
  </tr>
  <tr>
    <td>Triplet</td>
    <td>0.8138</td>
    <td>0.8465</td>
    <td>0.9025</td>
    <td>0.7163</td>
  </tr>
  <tr>
    <td>MBR-BW</td>
    <td>0.8104</td>
    <td>0.8447</td>
    <td>0.9026</td>
    <td>0.7134</td>
  </tr>
  <tr>
    <td>MBR-BMW</td>
    <td>0.8073</td>
    <td>0.8421</td>
    <td>0.9005</td>
    <td>0.7080</td>
  </tr>
  <tr>
    <td>CRPO+</td>
    <td><b>0.8149</b></td>
    <td><b>0.8469</b></td>
    <td>0.9074</td>
    <td>0.7207</td>
  </tr>
  <tr>
    <td>CRPO×</td>
    <td>0.8148</td>
    <td>0.8467</td>
    <td>0.9063</td>
    <td>0.7202</td>
  </tr>
</table>

Table 3: Average results on ten translation directions for ablation study.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">Average</th>
  </tr>
  <tr>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
  </tr>
  <tr>
    <td>ALMA-7B</td>
    <td>0.8140</td>
    <td>0.8539</td>
    <td>0.9203</td>
    <td>0.7306</td>
  </tr>
  <tr>
    <td>QE Ft.</td>
    <td>0.8149</td>
    <td>0.8563</td>
    <td>0.9243</td>
    <td>0.7338</td>
  </tr>
  <tr>
    <td>MinMaxR</td>
    <td>0.8194</td>
    <td>0.8594</td>
    <td>0.9251</td>
    <td>0.7387</td>
  </tr>
  <tr>
    <td>MinMaxP</td>
    <td>0.8178</td>
    <td>0.8592</td>
    <td>0.9265</td>
    <td>0.7371</td>
  </tr>
  <tr>
    <td>MinMaxPO</td>
    <td>0.8081</td>
    <td>0.8508</td>
    <td>0.9137</td>
    <td>0.7184</td>
  </tr>
  <tr>
    <td>TopScores</td>
    <td>0.8183</td>
    <td>0.8592</td>
    <td>0.9260</td>
    <td>0.7380</td>
  </tr>
  <tr>
    <td>CRPO+</td>
    <td><b>0.8218</b></td>
    <td><b>0.8618</b></td>
    <td><b>0.9311</b></td>
    <td><b>0.7462</b></td>
  </tr>
  <tr>
    <td>CRPO×</td>
    <td>0.8217</td>
    <td>0.8612</td>
    <td>0.9307</td>
    <td>0.7451</td>
  </tr>
</table>

sentences with worse log \( \pi_{ref} \) and dispreferred sentences with higher log \( \pi_{ref} \) which are more difficult for policy and have better impact to model fine-tuning.

5.5 Experiment for NLLB

To evaluate the generalization of CRPO, we extend CRPO to encoder-decoder model, NLLB-1.3B (Costa-jussà et al., 2022), and compare CRPO with RSO, RS-DPO (\( \eta = 0.82 \)), Triplet dataset and MBR score. Similar to the setting of ALMA experiment, for the 10 translation directions, we leverage the pretrained NLLB model (checkpoint facebook/nllb-200-1.3B) to collect 64 candidate sentences for each source sentence from FLORES-200 dataset and then apply data selection methods to construct preference dataset. We fine-tune NLLB model with DPO and evaluate it on the same WMT dataset. For more details of experiment setting, please refer to the Appendix.

The experimental results are shown in Table 2. Although worse than goal reference translation results for XCOMET and KIWI-XL scores, CRPO+ achieves better performance than other data selection methods, which indicates that CRPO generalizes well to encoder-decoder translation model.

5.6 Ablation Study

In the ablation study, we compare CRPO with more data selection methods to evaluate the effect of reward term and confidence term. We consider some questions that can be potentially raised from CRPO in the Appendix C.5. Specifically, we consider the following methods:

MinMaxR. To further evaluate the contribution of confidence term in CR-Score, we drop it from the CR+ and only maintain the reward difference as the score. In another word, the sentence with maximum reward score is selected as preferred sentence and the sentence with minimum reward score is selected as dispreferred sentence.

MinMaxP. Similarly, we evaluate the contribution of reward term by setting \( \phi = 0 \) in CR+ and selecting the sentence pairs with maximum value of \( [\log \pi_{ref}(y_l|x) - \log \pi_{ref}(y_w|x)] \). For fair comparison, the sentence pairs resulting in negative value are dropped. Noted that sentence pair with higher value of MinMaxP gets larger CPO loss value.

MinMaxPO. As a comparison with MinMaxP, we select the sentence with maximum likelihood and the sentence with minimum likelihood, among which the sentence with higher reward is set as \( y_w \).

TopScores. RSO utilizes statistical reject sampling to sample sentences from candidate set. We instead consider reject sampling directly based on the acceptance rate, that is only keeping sentences with top reward scores. Among these selected sentences, we set the one with highest reward score as preferred sentence and the one with smallest reward score as dispreferred sentence.

The average experiment results on ten translation directions are shown in Table 3. Dropping confidence term from CRPO, MinMaxR only considers the sentence quality from reward while neglecting the information policy could learn from related sentences, resulting in worse performance compared with CRPO. On the contrary, MinMaxP and MinMaxPO ensure the diversity of sentences by likelihood difference while neglecting the sentence quality, also gets lower evaluation scores. The above experiments empirically prove the necessity of combining sentence reward with prediction likelihood for sentence selection. Comparing with RSO, TopScores selects sentence with top scores rather than applying statistical rejection sampling and gets worse results, the reason of which might related to sentence diversity we think.
6 Conclusion

In this paper, we argue that sentence pairs with large loss value or loss change during training contains information the model has not yet learn and could benefit the model fine-tuning. We analyze loss value and loss change based on DPO and find them to be controlled by confidence terms measuring the prediction likelihood difference and reward terms measuring reward difference of sentence pair, based on which CR+ and CR× are designed for data selection. With experiment results, we empirically prove that CRPO outperforms previous data sampling method in machine translation tasks. And based on ablation study, we show the necessity of considering the two terms together for fine-tuning.

Limitation

In CRPO, we only select the sentence pair with maximum CR-Score, which will discard high quality data with slightly smaller CR-Score. Potential solution to this limitation includes leveraging pre-set threshold, or CR-Score distribution analysis.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Ebtessam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cocoraru, Merouane Debbaah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, et al. 2023. Falcon-40b: an open large language model with state-of-the-art performance.

Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345.

Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. 2023. Improving translation faithfulness of large language models via augmenting instructions. arXiv preprint arXiv:2308.12674.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30.

Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Mara Finkelstein, Subhajit Naskar, Mehdi Mirzazadeh, Apurva Shah, and Markus Freitag. 2023. Mbr and qe finetuning: Training-time distillation of the best and most expensive decoding methods. arXiv preprint arXiv:2309.10966.

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Koomei, George Foster, Alon Lavie, and André FT Martins. 2022. Results of wmt22 metrics shared task: Stop using bleu–neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46–68.

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021. Results of the wmt21 metrics shared task: Evaluating metrics with expert-based human evaluations on ted and news domain. In Proceedings of the Sixth Conference on Machine Translation, pages 733–774.

Nuno M Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André FT Martins. 2023. xcomet: Transparent machine translation evaluation through fine-grained error detection. arXiv preprint arXiv:2310.10482.

Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. 2023. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998.

Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. 2024. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. arXiv preprint arXiv:2402.10038.

Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 388–395.

Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. 2023. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657.

Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734.

Radford M Neal. 2003. Slice sampling. The annals of statistics, 31(3):705–767.
Rafael Rafaílov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506.

Ricardo Rei, Nuno M Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José GC de Souza, and André FT Martins. 2023. Scaling up cometkiwi: Unbabel-ist 2023 submission for the quality estimation shared task. arXiv preprint arXiv:2309.11925.

Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of ACL.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Qiyu Wu, Masaaki Nagata, Zhongtao Miao, and Yoshimasa Tsuruoka. 2024. Word alignment as preference for machine translation. arXiv preprint arXiv:2405.09223.

Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2023. A paradigm shift in machine translation: Boosting translation performance of large language models. arXiv preprint arXiv:2309.11674.

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024a. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417.

Nuo Xu, Jun Zhao, Can Zu, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024b. Advancing translation preference modeling with rlhf: A step towards cost-effective solution. arXiv preprint arXiv:2402.11525.

Guangyu Yang, Jinghong Chen, Weizhe Lin, and Bill Byrne. 2023a. Direct preference optimization for neural machine translation with minimum bayes risk decoding. arXiv preprint arXiv:2311.08380.

Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023b. Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages. arXiv preprint arXiv:2305.18098.

Jiali Zeng, Fandong Meng, Yongjing Yin, and Jie Zhou. 2024. Teaching large language models to translate with comparison. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19488–19496.

Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, et al. 2023. Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models. arXiv preprint arXiv:2306.10968.

Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalmans, Mohammad Saleh, and Peter J Liu. 2023. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425.

Yao Zhao, Mikhail Khalmans, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. 2022. Calibrating sequence likelihood improves conditional language generation. In The eleventh international conference on learning representations.

Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023a. Multilingual machine translation with large language models: Empirical results and analysis. arXiv preprint arXiv:2304.04675.

Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023b. Extrapolating large language models to non-english by aligning languages. arXiv preprint arXiv:2308.04948.
A Algorithm

The algorithm for data selection with CR-Score is outlined in Alg 1 where the inputs are source sentence, \( \pi_{ref} \) and reward model. In line 3-6, we sample candidate translation sentences from \( \pi_{ref} \) and then calculate their reward score and generation likelihood. In line 7, we select the sentence with highest reward score as preferred sentence \( y_w \), to ensure a baseline of sentence quality. In lines 9-13, we filter out sentence pairs with negative CR-Score and select the sentence pair with maximum CR-Score, ensuring that only the most informative data is retained in the preference dataset.

Algorithm 1 Data Selection with Confidence-Reward Score

1: Input: Source Sentence \( x^{(i)} \), Reference Policy \( \pi_{ref} \), Reward Model \( R \).
2: Output: Preference Data \( \mathcal{D}_{\simeq}^{(i)} \).
3: Set \( \mathcal{D}_{\simeq}^{(i)} \) as empty, Set \( s_{best} = 0 \);
4: Sample candidate sentences:
\( \mathcal{Y} = \{y^{(ij)}\}_{j=1}^K \sim \pi_{ref}(y|x^{(i)}) \);
5: Collect probabilities:
\( \mathcal{P} = \{p^{(ij)} = \pi_{ref}(y^{(ij)}|x)\}_{j=1}^K \);
6: Collect rewards:
\( \mathcal{R} = \{r^{(ij)} = R(x^{(i)}, y^{(ij)})\}_{j=1}^K \);
7: \( j_{max} = \arg\max_j \mathcal{R} \);
8: for each sentence \( y^{(ij)} \) in \( \mathcal{Y} \) do
9:   if \( p^{(ij)} - p^{(ij_{max})} + \epsilon > 0 \) then
10:     Calculate CR-Score for \( (y^{ij_{max}}, y^{ij}) \):
\( s = \mathrm{CR+} \) or \( s = \mathrm{CR\times} \);
11:     if \( s > s_{best} \) then
12:       \( \mathcal{D}_{\simeq}^{(i)} = (x^{(i)}, y^{(ij_{max})}, y^{(ij)}) \);
13:     end if
14:   end if
15: end for
16: Return \( \mathcal{D}_{\simeq}^{(i)} \);

B Experimental Details

In this section, we provide more details for experiment setting of ALMA and NLLB fine-tuning.

B.1 Experiment Setting for ALMA

During the training phase, we train the ALMA-7B in a many-to-many multilingual translation manner, starting with ALMA-7B-Pretrain-LoRA as initial checkpoint. For model fine-tuning, we focus on updating the weights of added LoRA parameters which have a rank of 16 and only add an additional 7.7M parameters to the original 7B size of the model. We follow Xu et al. (2024a) to set the default \( \beta \) value to be 0.1, the batch size of ALMA-7B in fine-tuning process to be 16, a warm-up ratio to be 0.01 and learning rate to be 0.0001. For all experiments, ALMA-7B is fine-tuned with one single epoch and the maximum length of accommodating sequence is set to be 512 tokens. For CR+, we set \( \phi \) to be 50 to bridge the magnitude different between reward and confidence terms. To optimize training efficiency, we implement the model fine-tuning with deepspeed tool (Rasley et al., 2020). For machine translation, we follow ALMA to set the prompt as “Translate this from <source language> to <target language>; <source language>: <source sentence>; <target language>:” and exclude the token sequence of prompt for loss calculation. Moreover, during training, we follow Dubey et al. (2024) to drop the EOS tokens at the end of translation outputs for both preferred and dispreferred sentences to avoid repeated tail content and add SFT term for DPO with coefficient weight set to be 1 for performance enhancement.

B.2 Experiment Setting for NLLB

We also train NLLB-1.3B in many-to-many multilingual translation manner, starting with facebook/nllb-200-1.3B as initial checkpoint. During fine-tuning, we add LoRA parameters to liner layers of NLLB model with additional 27.7M parameters which are the only trainable parameters and we train the model for 2 epochs. Similar to ALMA, we set the \( \beta \) value for DPO as 0.1, the batch size in fine-tuning process to be 16, a warm-up ratio to be 0.01 and learning rate to be 0.0001. For CR+, we again set \( \phi \) to be 50 to bridge the magnitude difference between reward and confidence terms. Following Costa-jussà et al. (2022), we manually set the BOS tokens for both encoder input sentence and decoder output sentence as related language index. We also add SFT term for DPO with coefficient weight set to be 1 for fair comparison. But different from ALMA, we retain the EOS tokens for preference sentence pairs which would not cause the tail content redundancy problem as in ALMA.

C More Experimental Results

C.1 Result on Each Translation Direction

To further evaluate the performance of CRPO, we show the evaluation results on \( en \rightarrow * \) in Table 4
Table 4: Experiment results on translation directions of \( en \rightarrow * \). The average result over these 5 translation directions are shown in the column of \( en \rightarrow * \).

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">en → zh</th>
    <th colspan="4">en → de</th>
  </tr>
  <tr>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
  </tr>
  <tr>
    <td>QE Ft.</td>
    <td>0.8099</td>
    <td>0.8596</td>
    <td>0.8611</td>
    <td>0.7387</td>
    <td>0.8265</td>
    <td>0.8574</td>
    <td>0.9645</td>
    <td>0.7392</td>
  </tr>
  <tr>
    <td>Goal Ref.</td>
    <td>0.8093</td>
    <td>-</td>
    <td>0.8594</td>
    <td>0.7402</td>
    <td>0.8268</td>
    <td>-</td>
    <td>0.9640</td>
    <td>0.7475</td>
  </tr>
  <tr>
    <td>RSO</td>
    <td>0.8160</td>
    <td>0.8631</td>
    <td>0.8694</td>
    <td>0.7488</td>
    <td>0.8310</td>
    <td>0.8604</td>
    <td>0.9663</td>
    <td>0.7456</td>
  </tr>
  <tr>
    <td>RS-DPO (\( \eta=0.60/0.50 \))</td>
    <td>0.8081</td>
    <td>0.8574</td>
    <td>0.8556</td>
    <td>0.7339</td>
    <td>0.8265</td>
    <td>0.8558</td>
    <td>0.9631</td>
    <td>0.7369</td>
  </tr>
  <tr>
    <td>RS-DPO (\( \eta=0.65/0.55 \))</td>
    <td>0.8069</td>
    <td>0.8564</td>
    <td>0.8560</td>
    <td>0.7314</td>
    <td>0.8254</td>
    <td>0.8546</td>
    <td>0.9626</td>
    <td>0.7345</td>
  </tr>
  <tr>
    <td>Triplet Dataset</td>
    <td>0.8089</td>
    <td>0.8581</td>
    <td>0.8690</td>
    <td>0.7373</td>
    <td>0.8278</td>
    <td>0.8591</td>
    <td>0.9663</td>
    <td>0.7424</td>
  </tr>
  <tr>
    <td>MBR-BW</td>
    <td>0.8106</td>
    <td>0.8596</td>
    <td>0.8571</td>
    <td>0.7371</td>
    <td>0.8287</td>
    <td>0.8588</td>
    <td>0.9636</td>
    <td>0.7422</td>
  </tr>
  <tr>
    <td>MBR-BMW</td>
    <td>0.8098</td>
    <td>0.8604</td>
    <td>0.8613</td>
    <td>0.7392</td>
    <td>0.8280</td>
    <td>0.8595</td>
    <td>0.9662</td>
    <td>0.7419</td>
  </tr>
  <tr>
    <td>CRPO+</td>
    <td><b>0.8194</b></td>
    <td><b>0.8656</b></td>
    <td><b>0.8706</b></td>
    <td><b>0.7548</b></td>
    <td><b>0.8326</b></td>
    <td><b>0.8622</b></td>
    <td><b>0.9673</b></td>
    <td><b>0.7533</b></td>
  </tr>
  <tr>
    <td>CRPO×</td>
    <td>0.8178</td>
    <td>0.8639</td>
    <td>0.8719</td>
    <td><b>0.7555</b></td>
    <td><b>0.8329</b></td>
    <td><b>0.8637</b></td>
    <td><b>0.9675</b></td>
    <td><b>0.7530</b></td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">en → cs</th>
    <th colspan="4">en → is</th>
  </tr>
  <tr>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
  </tr>
  <tr>
    <td>QE Ft.</td>
    <td>0.8397</td>
    <td>0.8941</td>
    <td>0.9246</td>
    <td>0.7556</td>
    <td>0.8124</td>
    <td>0.8574</td>
    <td>0.9000</td>
    <td>0.7557</td>
  </tr>
  <tr>
    <td>Goal Ref.</td>
    <td>0.8319</td>
    <td>-</td>
    <td>0.8964</td>
    <td>0.7405</td>
    <td>0.8051</td>
    <td>-</td>
    <td>0.8872</td>
    <td>0.7559</td>
  </tr>
  <tr>
    <td>RSO</td>
    <td>0.8455</td>
    <td>0.8980</td>
    <td>0.9292</td>
    <td>0.7648</td>
    <td>0.8135</td>
    <td>0.8584</td>
    <td>0.9008</td>
    <td>0.7602</td>
  </tr>
  <tr>
    <td>RS-DPO (\( \eta=0.60/0.50 \))</td>
    <td>0.8369</td>
    <td>0.8923</td>
    <td>0.9151</td>
    <td>0.7491</td>
    <td>0.8108</td>
    <td>0.8527</td>
    <td>0.8890</td>
    <td>0.7499</td>
  </tr>
  <tr>
    <td>RS-DPO (\( \eta=0.65/0.55 \))</td>
    <td>0.8389</td>
    <td>0.8931</td>
    <td>0.9194</td>
    <td>0.7522</td>
    <td>0.8100</td>
    <td>0.8549</td>
    <td>0.8923</td>
    <td>0.7533</td>
  </tr>
  <tr>
    <td>Triplet Dataset</td>
    <td>0.8421</td>
    <td>0.8957</td>
    <td>0.9306</td>
    <td>0.7620</td>
    <td>0.8121</td>
    <td>0.8575</td>
    <td>0.8968</td>
    <td>0.7562</td>
  </tr>
  <tr>
    <td>MBR-BW</td>
    <td>0.8431</td>
    <td>0.8983</td>
    <td>0.9232</td>
    <td>0.7585</td>
    <td>0.8161</td>
    <td>0.8596</td>
    <td>0.8994</td>
    <td>0.7599</td>
  </tr>
  <tr>
    <td>MBR-BMW</td>
    <td>0.8408</td>
    <td>0.8964</td>
    <td>0.9256</td>
    <td>0.7568</td>
    <td>0.8161</td>
    <td>0.8601</td>
    <td>0.9020</td>
    <td>0.7608</td>
  </tr>
  <tr>
    <td>CRPO+</td>
    <td><b>0.8483</b></td>
    <td><b>0.9019</b></td>
    <td><b>0.9379</b></td>
    <td><b>0.7780</b></td>
    <td><b>0.8199</b></td>
    <td><b>0.8646</b></td>
    <td><b>0.9088</b></td>
    <td><b>0.7688</b></td>
  </tr>
  <tr>
    <td>CRPO×</td>
    <td><b>0.8489</b></td>
    <td>0.9015</td>
    <td>0.9372</td>
    <td>0.7769</td>
    <td>0.8176</td>
    <td>0.8627</td>
    <td>0.9051</td>
    <td>0.7655</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">en → ru</th>
    <th colspan="4">en → *</th>
  </tr>
  <tr>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
  </tr>
  <tr>
    <td>QE Ft.</td>
    <td>0.8099</td>
    <td>0.8758</td>
    <td>0.9335</td>
    <td>0.7665</td>
    <td>0.8265</td>
    <td>0.8689</td>
    <td>0.9167</td>
    <td>0.7511</td>
  </tr>
  <tr>
    <td>Goal Ref.</td>
    <td>0.8297</td>
    <td>-</td>
    <td>0.9241</td>
    <td>0.7598</td>
    <td>0.8206</td>
    <td>-</td>
    <td>0.9062</td>
    <td>0.7488</td>
  </tr>
  <tr>
    <td>RSO</td>
    <td>0.8373</td>
    <td>0.8792</td>
    <td>0.9395</td>
    <td>0.7488</td>
    <td>0.8287</td>
    <td>0.8718</td>
    <td>0.9210</td>
    <td>0.7456</td>
  </tr>
  <tr>
    <td>RS-DPO (\( \eta=0.60/0.50 \))</td>
    <td>0.8292</td>
    <td>0.8726</td>
    <td>0.9257</td>
    <td>0.7581</td>
    <td>0.8223</td>
    <td>0.8662</td>
    <td>0.9097</td>
    <td>0.7456</td>
  </tr>
  <tr>
    <td>RS-DPO (\( \eta=0.65/0.55 \))</td>
    <td>0.8302</td>
    <td>0.8729</td>
    <td>0.9284</td>
    <td>0.7618</td>
    <td>0.8223</td>
    <td>0.8664</td>
    <td>0.9117</td>
    <td>0.7466</td>
  </tr>
  <tr>
    <td>Triplet Dataset</td>
    <td>0.8326</td>
    <td>0.8759</td>
    <td>0.9388</td>
    <td>0.7704</td>
    <td>0.8247</td>
    <td>0.8693</td>
    <td>0.9203</td>
    <td>0.7537</td>
  </tr>
  <tr>
    <td>MBR-BW</td>
    <td>0.8332</td>
    <td>0.8776</td>
    <td>0.9349</td>
    <td>0.7683</td>
    <td>0.8262</td>
    <td>0.8709</td>
    <td>0.9156</td>
    <td>0.7531</td>
  </tr>
  <tr>
    <td>MBR-BMW</td>
    <td>0.8329</td>
    <td>0.8784</td>
    <td>0.9334</td>
    <td>0.7670</td>
    <td>0.8257</td>
    <td>0.8709</td>
    <td>0.9177</td>
    <td>0.7532</td>
  </tr>
  <tr>
    <td>CRPO+</td>
    <td><b>0.8398</b></td>
    <td><b>0.8815</b></td>
    <td><b>0.9441</b></td>
    <td><b>0.7821</b></td>
    <td><b>0.8320</b></td>
    <td><b>0.8752</b></td>
    <td><b>0.9257</b></td>
    <td><b>0.7674</b></td>
  </tr>
  <tr>
    <td>CRPO×</td>
    <td>0.8397</td>
    <td><b>0.8820</b></td>
    <td><b>0.9442</b></td>
    <td>0.7818</td>
    <td>0.8314</td>
    <td>0.8748</td>
    <td>0.9252</td>
    <td>0.7660</td>
  </tr>
</table>

and \( * \rightarrow en \) in Table 5. CRPO achieves the best results on almost all directions which empirically proves the robustness of our method and the importance of leveraging confidence term to measure the policy performance for different translation directions.

C.2 Significance Test

To evaluate the reliability of performance improvement achieved by CRPO, we perform paired bootstrap test (Koehn, 2004) to compare CRPO with RSO, RS-DPO and Triplet Dataset. Specifically, we set the total sample times to be 10,000 and sample rate to be 0.5 in all language pairs.

On ALMA-7B model, for XCOMET score, we reject null hypothesis and accept CRPO+ to be better than RSO with \( p = 0.001 \) and Triplet Dataset with \( p = 0.001 \). For all other evaluation metrics and baselines, we accept CRPO+ and CRPO× to be better with \( p = 0.000 \). On NLLB-1.3B model, we accept CRPO+ and CRPO× to be better than RS-DPO and Triplet Dataset with \( p = 0.000 \). The comparison between CRPO and RSO are shown in Table 6, where CRPO+ and CRPO× always achieve better performance than RSO.

As the commonly used level of reliability of the result is 95 (\( p \leq 0.005 \)), we conclude that CRPO significantly outperforms RSO, RS-DPO and Triplet Dataset.

C.3 Non-COMET Metric

Due to the similar training procedure of COMET metrics, concerns may arise that the results in Section 5 and Appendix C.1 could be highly correlated with the reward model - COMET metrics. To address this concern, we also consider BLEURT-20 (Sellam et al., 2020) for evaluation, which is a non-COMET and neural-based metric. We compare CRPO with RSO, RS-DPO, MBR Score and Triplet Dataset for both ALMA-7b and NLLB-1.3B
Table 5: Experiment results on translation directions of * → en. The average result over these 5 translation directions are shown in the column of * → en.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">zh → en</th>
    <th colspan="4">de → en</th>
  </tr>
  <tr>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
  </tr>
  <tr>
    <td>QE Ft.</td>
    <td>0.7759</td>
    <td>0.8017</td>
    <td>0.8690</td>
    <td>0.6721</td>
    <td>0.8108</td>
    <td>0.8437</td>
    <td>0.9702</td>
    <td>0.7349</td>
  </tr>
  <tr>
    <td>Goal Ref.</td>
    <td>0.7709</td>
    <td>-</td>
    <td>0.8420</td>
    <td>0.6674</td>
    <td>0.7874</td>
    <td>-</td>
    <td>0.9476</td>
    <td>0.6975</td>
  </tr>
  <tr>
    <td>RSO</td>
    <td>0.7868</td>
    <td>0.8093</td>
    <td>0.8739</td>
    <td>0.6871</td>
    <td>0.8129</td>
    <td>0.8463</td>
    <td>0.9710</td>
    <td>0.7372</td>
  </tr>
  <tr>
    <td>RS-DPO (\( \eta = 0.60/0.50 \))</td>
    <td>0.7771</td>
    <td>0.8019</td>
    <td>0.8631</td>
    <td>0.6741</td>
    <td>0.8093</td>
    <td>0.8427</td>
    <td>0.9683</td>
    <td>0.7312</td>
  </tr>
  <tr>
    <td>RS-DPO (\( \eta = 0.65/0.55 \))</td>
    <td>0.7790</td>
    <td>0.8045</td>
    <td>0.8671</td>
    <td>0.6771</td>
    <td>0.8095</td>
    <td>0.8430</td>
    <td>0.9677</td>
    <td>0.7321</td>
  </tr>
  <tr>
    <td>Triplet Dataset</td>
    <td>0.7835</td>
    <td>0.8079</td>
    <td>0.8728</td>
    <td>0.6827</td>
    <td>0.8119</td>
    <td>0.8462</td>
    <td>0.9711</td>
    <td>0.7372</td>
  </tr>
  <tr>
    <td>MBR-BW</td>
    <td>0.7850</td>
    <td>0.8099</td>
    <td>0.8745</td>
    <td>0.6834</td>
    <td>0.8110</td>
    <td>0.8454</td>
    <td>0.9692</td>
    <td>0.7330</td>
  </tr>
  <tr>
    <td>MBR-BMW</td>
    <td>0.7821</td>
    <td>0.8081</td>
    <td>0.8722</td>
    <td>0.6805</td>
    <td>0.8102</td>
    <td>0.8449</td>
    <td>0.9697</td>
    <td>0.7341</td>
  </tr>
  <tr>
    <td>CRPO+</td>
    <td><b>0.7883</b></td>
    <td><b>0.8106</b></td>
    <td><b>0.8780</b></td>
    <td><b>0.6905</b></td>
    <td><b>0.8132</b></td>
    <td><b>0.8469</b></td>
    <td><b>0.9719</b></td>
    <td><b>0.7398</b></td>
  </tr>
  <tr>
    <td>CRPO×</td>
    <td><b>0.7889</b></td>
    <td><b>0.8098</b></td>
    <td><b>0.8767</b></td>
    <td><b>0.6898</b></td>
    <td><b>0.8141</b></td>
    <td><b>0.8459</b></td>
    <td><b>0.9717</b></td>
    <td><b>0.7390</b></td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">cs → en</th>
    <th colspan="4">ts → en</th>
  </tr>
  <tr>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
  </tr>
  <tr>
    <td>QE Ft.</td>
    <td>0.8197</td>
    <td>0.8614</td>
    <td>0.9383</td>
    <td>0.7160</td>
    <td>0.8108</td>
    <td>0.8639</td>
    <td>0.9307</td>
    <td>0.7413</td>
  </tr>
  <tr>
    <td>Goal Ref.</td>
    <td>0.8209</td>
    <td>-</td>
    <td>0.9325</td>
    <td>0.7150</td>
    <td>0.8089</td>
    <td>-</td>
    <td>0.9240</td>
    <td>0.7379</td>
  </tr>
  <tr>
    <td>RSO</td>
    <td><b>0.8239</b></td>
    <td><b>0.8650</b></td>
    <td>0.9410</td>
    <td>0.7210</td>
    <td>0.8138</td>
    <td>0.8674</td>
    <td>0.9326</td>
    <td>0.7431</td>
  </tr>
  <tr>
    <td>RS-DPO (\( \eta = 0.60/0.50 \))</td>
    <td>0.8179</td>
    <td>0.8613</td>
    <td>0.9365</td>
    <td>0.7122</td>
    <td>0.8073</td>
    <td>0.8615</td>
    <td>0.9244</td>
    <td>0.7363</td>
  </tr>
  <tr>
    <td>RS-DPO (\( \eta = 0.65/0.55 \))</td>
    <td>0.8199</td>
    <td>0.8623</td>
    <td>0.9374</td>
    <td>0.7147</td>
    <td>0.8079</td>
    <td>0.8621</td>
    <td>0.9245</td>
    <td>0.7364</td>
  </tr>
  <tr>
    <td>Triplet Dataset</td>
    <td>0.8213</td>
    <td>0.8633</td>
    <td>0.9427</td>
    <td>0.7187</td>
    <td>0.8130</td>
    <td>0.8662</td>
    <td>0.9322</td>
    <td>0.7412</td>
  </tr>
  <tr>
    <td>MBR-BW</td>
    <td>0.8220</td>
    <td>0.8633</td>
    <td>0.9386</td>
    <td>0.7171</td>
    <td>0.8123</td>
    <td>0.8664</td>
    <td>0.9308</td>
    <td>0.7401</td>
  </tr>
  <tr>
    <td>MBR-BMW</td>
    <td>0.8219</td>
    <td>0.8652</td>
    <td>0.9403</td>
    <td>0.7165</td>
    <td>0.8106</td>
    <td>0.8643</td>
    <td>0.9275</td>
    <td>0.7391</td>
  </tr>
  <tr>
    <td>CRPO+</td>
    <td><b>0.8240</b></td>
    <td><b>0.8644</b></td>
    <td><b>0.9442</b></td>
    <td><b>0.7241</b></td>
    <td><b>0.8152</b></td>
    <td><b>0.8682</b></td>
    <td><b>0.9338</b></td>
    <td><b>0.7450</b></td>
  </tr>
  <tr>
    <td>CRPO×</td>
    <td>0.8238</td>
    <td>0.8633</td>
    <td>0.9422</td>
    <td>0.7220</td>
    <td><b>0.8160</b></td>
    <td><b>0.8682</b></td>
    <td>0.9366</td>
    <td><b>0.7457</b></td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">ru → en</th>
    <th colspan="4">* → en</th>
  </tr>
  <tr>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
  </tr>
  <tr>
    <td>QE Ft.</td>
    <td>0.8121</td>
    <td>0.8480</td>
    <td>0.9512</td>
    <td>0.7184</td>
    <td>0.8059</td>
    <td>0.8437</td>
    <td>0.9319</td>
    <td>0.7165</td>
  </tr>
  <tr>
    <td>Goal Ref.</td>
    <td>0.8074</td>
    <td>-</td>
    <td>0.9403</td>
    <td>0.7066</td>
    <td>0.7991</td>
    <td>-</td>
    <td>0.9173</td>
    <td>0.7048</td>
  </tr>
  <tr>
    <td>RSO</td>
    <td>0.8158</td>
    <td>0.8511</td>
    <td>0.9528</td>
    <td>0.7223</td>
    <td>0.8106</td>
    <td>0.8478</td>
    <td>0.9343</td>
    <td>0.7217</td>
  </tr>
  <tr>
    <td>RS-DPO (\( \eta = 0.60/0.50 \))</td>
    <td>0.8109</td>
    <td>0.8489</td>
    <td>0.9483</td>
    <td>0.7168</td>
    <td>0.8045</td>
    <td>0.8433</td>
    <td>0.9281</td>
    <td>0.7141</td>
  </tr>
  <tr>
    <td>RS-DPO (\( \eta = 0.65/0.55 \))</td>
    <td>0.8122</td>
    <td>0.8493</td>
    <td>0.9497</td>
    <td>0.7176</td>
    <td>0.8057</td>
    <td>0.8442</td>
    <td>0.9293</td>
    <td>0.7156</td>
  </tr>
  <tr>
    <td>Triplet Dataset</td>
    <td>0.8148</td>
    <td>0.8506</td>
    <td>0.9538</td>
    <td>0.7224</td>
    <td>0.8089</td>
    <td>0.8468</td>
    <td>0.9345</td>
    <td>0.7204</td>
  </tr>
  <tr>
    <td>MBR-BW</td>
    <td>0.8131</td>
    <td>0.8500</td>
    <td>0.9482</td>
    <td>0.7178</td>
    <td>0.8087</td>
    <td>0.8470</td>
    <td>0.9323</td>
    <td>0.7183</td>
  </tr>
  <tr>
    <td>MBR-BMW</td>
    <td>0.8133</td>
    <td>0.8516</td>
    <td>0.9500</td>
    <td>0.7185</td>
    <td>0.8076</td>
    <td>0.8468</td>
    <td>0.9319</td>
    <td>0.7177</td>
  </tr>
  <tr>
    <td>CRPO+</td>
    <td><b>0.8172</b></td>
    <td><b>0.8517</b></td>
    <td><b>0.9547</b></td>
    <td><b>0.7259</b></td>
    <td><b>0.8116</b></td>
    <td><b>0.8484</b></td>
    <td><b>0.9365</b></td>
    <td><b>0.7251</b></td>
  </tr>
  <tr>
    <td>CRPO×</td>
    <td>0.8171</td>
    <td>0.8510</td>
    <td>0.9539</td>
    <td>0.7247</td>
    <td><b>0.8120</b></td>
    <td>0.8476</td>
    <td>0.9362</td>
    <td>0.7242</td>
  </tr>
</table>

Table 6: Significance test between CRPO and RSO on NLLB-1.3B model.

<table>
  <tr>
    <th>Comparison</th>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
  </tr>
  <tr>
    <td>CRPO+ vs. RSO</td>
    <td>0.025</td>
    <td>0.000</td>
    <td>0.034</td>
    <td>0.009</td>
  </tr>
  <tr>
    <td>CRPO× vs. RSO</td>
    <td>0.035</td>
    <td>0.000</td>
    <td>0.016</td>
    <td>0.013</td>
  </tr>
</table>

Table 7: Evaluation Results based on BLEURT-20 metric for ALMA-7B and NLLB-1.3B.

<table>
  <tr>
    <th>Method</th>
    <th>ALMA-7B</th>
    <th>NLLB-1.3B</th>
  </tr>
  <tr>
    <td>RSO</td>
    <td>0.7451</td>
    <td>0.7305</td>
  </tr>
  <tr>
    <td>RS-DPO (\( \eta = 0.60/0.50 \))</td>
    <td>0.7401</td>
    <td>0.7154</td>
  </tr>
  <tr>
    <td>RS-DPO (\( \eta = 0.65/0.55 \))</td>
    <td>0.7404</td>
    <td>0.7158</td>
  </tr>
  <tr>
    <td>Triplet Dataset</td>
    <td>0.7444</td>
    <td>0.7212</td>
  </tr>
  <tr>
    <td>MBR-BW</td>
    <td>0.7463</td>
    <td>0.7281</td>
  </tr>
  <tr>
    <td>MBR-BMW</td>
    <td>0.7460</td>
    <td>0.7251</td>
  </tr>
  <tr>
    <td>CRPO+</td>
    <td><b>0.7497</b></td>
    <td><b>0.7317</b></td>
  </tr>
  <tr>
    <td>CRPO×</td>
    <td>0.7490</td>
    <td>0.7314</td>
  </tr>
</table>

models in Table 7, where CRPO+ and CRPO× achieve the best score indicating the robustness and high performance of combining confidence and reward terms.

C.4 Visualization of Reward and Confidence

To further represent the CRPO strategy, we visualize the correlation between reward scores and log \( \pi_{ref} \) for candidate dataset in Figure 3 and for selected dataset in Figure 4.

As the ALMA-7B checkpoint already achieves outstanding performance, Figure 3 shows that before fine-tuning the policy has high probability to generate high reward translation sentences and sentence pairs with low reward difference also tend to have low log \( \pi_{ref} \) difference. This also explains that in Figure 4, for all data selection methods, sentences with high reward tend to have high generation likelihood. But comparing with RS-DPO, RSO and MBR-BW, dis-preferred sentences selected by
Figure 3: Reward score vs. log \( \pi_{ref} \) for entire dataset.

Table 8: Average results of CRPO on mixed dataset.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">Average</th>
  </tr>
  <tr>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
  </tr>
  <tr>
    <td>CRPO+</td>
    <td>0.8218</td>
    <td>0.8618</td>
    <td>0.9311</td>
    <td>0.7462</td>
  </tr>
  <tr>
    <td>CRPO×</td>
    <td>0.8217</td>
    <td>0.8612</td>
    <td>0.9307</td>
    <td>0.7451</td>
  </tr>
  <tr>
    <td>Triplet</td>
    <td>0.8168</td>
    <td>0.8581</td>
    <td>0.9274</td>
    <td>0.7371</td>
  </tr>
  <tr>
    <td>CRPO+*</td>
    <td><b>0.8223</b></td>
    <td>0.8622</td>
    <td>0.9299</td>
    <td>0.7458</td>
  </tr>
  <tr>
    <td>CRPO×*</td>
    <td><b>0.8221</b></td>
    <td><b>0.8626</b></td>
    <td><b>0.9319</b></td>
    <td><b>0.7465</b></td>
  </tr>
</table>

CRPO generally get higher value of log \( \pi_{ref} \) which refers to difficult or error predicted translation sentences for the policy. Moreover, as CRPO only selects sentence pairs with positive CR-Score, only sentence pairs with negative log \( \pi_{ref} \) difference are remained in Figure 4.

C.5 Potential Questions.

We further evaluate CRPO by considering potential questions could be raised from CR-Score and attempt to answer them with existed or novel experiment results.

Is it possible to apply CRPO on sentences from extra resources with higher reward? To answer this question, we compose a new candidate set by mixing Triplet Dataset with our generated candidate sentences from reference policy. We then apply CR-Score to construct preference dataset and fine-tune the policy with DPO. The results are shown in Table 8 where mixing Triplet Dataset (CRPO+* and CRPO×*) increases the overall performance of CRPO. Note that although fine-tuning Triplet Dataset gets worse evaluation scores, CRPO selects sentences from extra resource only when they could provide useful information for the policy and thus achieves higher performance.

Will Triplet Dataset achieves better result when trained with CPO? For the reason that reference policy is dropped in CPO, distribution shift problem might be released in CPO and the quality of sentence pairs should be more important. To answer the question, we provide additional experiment to fine-tune the policy on CPO, the results of which are shown in Table 9. CRPO with CPO fine-tuning still achieves better overall performance compared with Triplet Dataset. Moreover, it is interesting to show that although Triplet Dataset achieves better result with CPO compared with DPO, CPO on preference dataset constructed by CR-Score does not achieves better performance compared with DPO. We think the reason is that preference dataset from CRPO already provides enough information to increase the reward that policy could achieve and dropping the KL divergence term in RLHF objective such as CPO would not further improve the performance. The better way to increase fine-tuning result of CRPO is to increase the quality of sentences, such as mixing with Triplet Dataset.

Table 9: Average results of CRPO and Triplet Dataset on CPO.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">Average</th>
  </tr>
  <tr>
    <th>KIWI22</th>
    <th>COMET22</th>
    <th>XCOMET</th>
    <th>KIWI-XL</th>
  </tr>
  <tr>
    <td>Triplet (CPO)</td>
    <td>0.8175</td>
    <td>0.8587</td>
    <td>0.9284</td>
    <td>0.7389</td>
  </tr>
  <tr>
    <td>CRPO+ (CPO)</td>
    <td><b>0.8214</b></td>
    <td><b>0.8607</b></td>
    <td><b>0.9306</b></td>
    <td><b>0.7451</b></td>
  </tr>
  <tr>
    <td>CRPO× (CPO)</td>
    <td><b>0.8214</b></td>
    <td>0.8604</td>
    <td>0.9302</td>
    <td>0.7450</td>
  </tr>
</table>
(a) Preferred Sentences.

(b) Dis-preferred Sentences.

(c) Reward Difference vs. log \( \pi_{ref} \) Difference.

Figure 4: Distribution of reward scores and log \( \pi_{ref} \).

Metareview score: 3
Metareview: The paper introduces new methods that leverage model confidence of the reference model and reward scores for selecting preference pairs when aligning MT systems with preference data. The authors apply their proposed approach to an LLM-based MT system and a standard encoder-decoder based translation model.
Summary of reasons to publish: Results show the efficacy of their approach over several preference example selection methods: Rejection sampling (best-worst, RSO), rejection sampling with accept-reject criteria (RS-DPO), MBR decoding (best-middle-worst) and QE reranking (1-best output).