Venue: NAACL 2024

# Improving Machine Translation with Human Feedback:

An Exploration of Quality Estimation as a Reward Model

 Zhiwei He\({}^{1*}\) Xing Wang\({}^{2\dagger}\) Wenxiang Jiao\({}^{2}\) Zhuosheng Zhang\({}^{1}\)

Rui Wang\({}^{1\dagger}\) Shuming Shi\({}^{2}\) Zhaopeng Tu\({}^{2}\)

###### Abstract

Insufficient modeling of human preferences within the reward model is a major obstacle for leveraging human feedback to improve translation quality. Fortunately, quality estimation (QE), which predicts the quality of a given translation without reference, has achieved impressive alignment with human evaluations in the last two years. In this work, we investigate the potential of employing the QE model as the reward model to predict human preferences for feedback training. We first identify the _overoptimization_ problem during QE-based feedback training, manifested as an increase in reward while translation quality declines. We examine the problem and argue that the vulnerability of the QE model might lead to high rewards for incorrect translations, resulting in overoptimization and error propagation. To address the problem, we adopt a simple yet effective method that uses heuristic rules to detect the incorrect translations and assigns a penalty term to the reward scores of them. Experimental results show that the proposed QE-based feedback training achieves consistent and significant improvements across various settings, further verified through human preference studies. Our subsequent analysis demonstrates the high data efficiency of the proposed QE-based feedback training: it outperforms systems using larger parallel corpora by a small amount of monolingual data. Our code is available at: [https://github.com/zwhe99/FeedbackMT](https://github.com/zwhe99/FeedbackMT)

+
Footnote †: This work was done when Zhiwei He were interning at Tencent AI Lab.

\({}^{\dagger}\) Xing Wang and Rui Wang are co-corresponding authors.

\({}^{Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics:

Human Language Technologies (Volume 1: Long Papers), pages 8164-8180

June 16-21, 2024 0204 Association for Computational Linguistics

## 1 Introduction

Human feedback has greatly contributed to recent advances in large language models (LLMs), aligning model behavior with human preferences and thereby enhancing the helpfulness and harmlessness of LLMs Dong et al. (2023); Yuan et al. (2023); Zhao et al. (2023); Rafailov et al. (2023). The common practice involves using human evaluation data to train a reward model as a proxy for human preferences, followed by feedback training to fine-tune the LLM and maximize the reward score.

Early efforts in neural machine translation (NMT) also attempted to integrate feedback to improve translation quality. Most works used similarity scores (such as sentence-level BLEU; Papineni et al. 2002) between the predicted translation and a reference translation to simulate feedback rather than employing feedback from humans Sokolov et al. (2016a, b); Kreutzer et al. (2017); Sokolov et al. (2017); Lawrence et al. (2017); Nguyen et al. (2017); Wu et al. (2018); Wieting et al. (2019). Few attempts to use real human feedback have been made, and these efforts either used implicit feedback in limited scenarios (e.g., e-commerce; Kreutzer et al. 2018) or relied only on a minimal amount of human feedback data Kreutzer et al. (2018). Therefore, the integration of real human feedback in NMT has been constrained by inadequate modeling of human preferences.

Fortunately, the field of MT has seen substantial advancements in quality estimation (QE; Rei et al. 2021, 2022b; Wan et al. 2022). A QE model offers a reference-free estimation of translation quality and has been facilitated by the growing availability of human evaluation data Specia et al. (2020, 2021); Zerva et al. (2022) and the development of pre-trained language models Devlin et al. (2019); Conneau et al. (2020). Typically, given a source sentence and its translation, a sentence-level QE model can provide a numerical score to indicate the quality of the translation. The most advanced QE models to date have achieved impressive alignment with human evaluations Freitag et al. (2022) and have been used to guide the decoding process Fernandes et al. (2022); He et al. (2024).

In light of this progress, we explore the potential of utilizing QE models as proxies of human preferences and functioning them as reward models in feedback training for the first time. Firstly, we

identify the _overoptimization_ problem in feedback training, manifested as an increase in reward while translation quality declines. Our analysis reveals that the underlying issue lies in the vulnerability of QE-based reward models, which, in rare instances, assign high scores to patently incorrect translations. As a result, these flawed patterns spread through subsequent training, leading to divergence from human preferences and a training collapse. However, the reward keeps rising throughout the whole process. This phenomenon aligns with the observation from Fernandes et al. (2023) that "overoptimizing" against an imperfect reward model can lead to systems "that receive good feedback from the model, but not humans".

Through manual examination of these flawed patterns, we categorize the most prevalent errors into two groups: length ratio errors and off-target errors (i.e., not the desired target language). Guided by this observation, we propose a simple yet effective solution to mitigate overoptimization. We first detect these errors and then add a negative penalty to the reward for these erroneous translations. We show that this approach significantly alleviates the overoptimization problem and results in notable improvements across various settings, which are further verified by human preference studies1

Footnote 1: Automatic metrics and human judgement might be not consistent under feedback training as discussed in Limitations.

In summary, the contributions of this work are detailed as follows:

\(\bullet\) We identify the _overoptimization_ problem when using QE-based reward models for feedback training and verify the ubiquity of this phenomenon across comprehensive settings (12 in total): 3 QE models \(\times\) 2 model architectures (decoder-only LLM and encoder-decoder NMT models) \(\times\) 2 resource settings (high-resource and low-resource).

\(\bullet\) By addressing the overoptimization with a simple yet effective method, we successfully integrate the QE model into feedback training for the first time, achieving remarkable improvements across various settings.

\(\bullet\) Through further analysis, we demonstrate the high data efficiency of using the QE-based reward model for feedback training, showing that it can outperform systems using larger parallel corpora by only a small amount of monolingual data.

\(\bullet\) We investigate the influence of the base model on feedback training, finding that stronger base models (larger in size and pretrained) yield greater improvements after feedback training. We also examine the effect of crucial hyperparameters.

## 2 Feedback Training

### Formulation

We denote an MT model as \(M=P(y|x;\theta)\), with model parameters \(\theta\). It takes a source sentence \(x\) (or a prompt) as input and generates a target sentence \(y\) according to the distribution \(P_{T}(y|x;\theta)\), where \(T\) is a temperature parameter to control the diversity. We consider the QE-based reward model as \(r(x,y)\) whose value indicates the quality of \(y\) as the translation of \(x\). Taken \(\mathcal{D}\) as the training distribution of \(x\), the optimization objective is:

\[\max_{\theta}\mathbb{E}_{x\sim\mathcal{D},y\sim P(y|x;\theta)}r(x,y). \tag{1}\]

We adopt the local ranking version of Reward rAnked FineTuning (RAFT; Dong et al., 2023) to train the model \(M\). The basic idea of RAFT is to rate the generated candidates from a prompt using the reward model and just learn from the best one of them. It has been proved to be more stable and efficient than Proximal Policy Optimization (PPO; Schulman et al., 2017). Algorithm 1 shows the details of RAFT.

```
0:Training set \(\mathcal{X}\), reward function \(r(x,y)\), initial model \(M_{0}=P(y|x;\theta_{0})\), batch size \(b\), temperature \(T\), the number of candidate \(k\)
1:for iteration \(i\) in \(0,1,\dots,N-1\)do
2:\(D_{i}\leftarrow\text{SampleBatch}(\mathcal{X},b)\)
3:\(\mathcal{B}=\emptyset\)
4:for\(x\in D_{i}\)do
5:\(y_{1},\dots,y_{k}\sim P_{T}(y|x;\theta_{i})\)
6:\(y^{*}=\arg\max_{y_{j}\in\left\{y_{1},\dots,y_{k}\right\}}r(x,y_{j})\)
7:\(\mathcal{B}=\mathcal{B}\cup\left\{(x,y^{*})\right\}\)
8: Fine-tune \(\theta_{i}\) on \(\mathcal{B}\) to obtain \(M_{i+1}=P(y|x;\theta_{i+1})\).
```

**Algorithm 1** RAFT

### Addressing Overoptimization Problem

OveroptimizationOur preliminary experiment observed _that as the reward increases, the translation performance deteriorates_ (shown in Figure 1). This phenomenon is dubbed as _overoptimization_ by Gao et al. (2023). The underlying reason for overoptimization is that the reward model does not serve as a perfect proxy for human preferences. Hence, overoptimizing rewards could steer 

the model's behavior away from human preferences. This aligns with Goodhart's Law, which states, "When a measure becomes a target, it ceases to be a good measure."

Footnote 1: [https://www.facebook.com/en-us/2018/03/04/05/06/07/09](https://www.facebook.com/en-us/2018/03/04/05/06/07/09) 

model, we carry out the following steps: (1) Supervised fine-tuning (SFT), where we utilize parallel data to fine-tune the pretrained base model, thereby obtaining an initial MT model. (2) Feedback training, where we train the model using RAFT/RAFT+ to maximize the reward from a QE model. Note that this stage only uses monolingual data.

ModelFor the base models, we adopt:

* Llama-2-7B [35]: a decoder-only LLM primarily trained in English, with the objective to predict the next token.
* Nllb-200-1.3B [46]: an encoder-decoder model that trained for multilingual translation across 200 languages. After SFT, it will be adapted to the language pairs we considered.

For QE models, we use:

* Comet-qe-da [8]: A QE model trained on _Direct Assessments_ data.
* Comet-qe-mqm [9]: Fine-tuned Comet-qe-da model using _Multidimensional Quality Metric_ (MOM) data.
* UniTE-MUP [26]: A unified translation evaluation framework that is jointly trained for reference-only, source-only, and source-reference-combined evaluation. We only use its source-only evaluation.

DataWe consider the settings for both high- and low-resource language pairs. In the high-resource setting, we focus on English\(\Leftrightarrow\)Chinese and English\(\Leftrightarrow\)German. In the low-resource setting, our focus is on English\(\Leftrightarrow\)Ukrainian and Ukrainian\(\Leftrightarrow\)Czech. It is important to note that both settings are multilingual, where all four translation directions are trained concurrently within one model, and we ensure the balance of all directions to avoid introducing other factors. That is, every direction has an equal number of training samples. We adopt WMT2 [46, 8] as the test sets for all language pairs, which is latest enough to avoid data contamination [27, 28]. For development sets, high resource setting uses WMT21, and low resource setting uses Flores200 [45]. Table 2 lists the detail of the data we used.

Training detailsFor SFT, we conducted training for one epoch. For RAFT/RAFT+, we trained Llama-2-7B for 10 iterations using a learning rate of 2e-6, and for Nllb-200-1.3B, we trained for 20 iterations with a learning rate of 2e-5. We set the batch size \(b\) to 1024, the number of candidates \(k\) to 8, and the temperature \(T\) to 0.85.

EvaluationWe use COMET [8] and BLEURT [21] to assess translation quality. These neural metrics show superiority over string-based metrics like BLEU [11, 46, 47]. We use Unbabel/wmt22-comet-da and BLEURT-20 checkpoints for these two metrics. It is worth noting that we also observe the overoptimization problem on COMET, that is, COMET increased but the actual translation quality decreased (see SS 3.2). Therefore, we use BLEURT as the main metric, but still report COMET as a reference and conduct human evaluation in SS 4.1. We select the best checkpoint based on the performance on the dev sets, and report the final results on test sets using beam search (beam size = 4).

### Results

Training curvesFigure 3 illustrates the training curves of reward and BLEURT on the development sets when using Comet-qe-da as the QE-based reward model. Our observations are:

\(\bullet\)**Overoptimization is a phenomenon of high frequency when using vanilla RAFT.** Three out of the four settings (Figure 2(a), c, d) have the situation that the reward shows an increase while the BLEURT declines. We further verify this phenomenon in Appendix C when using Comet-qemqm orUnite-MUP as the reward model.

\(\bullet\)**The severity of overoptimization varies under different settings.** Figure 2(d) represents the most severe overoptimization, where the BLEURT

\begin{table}
\begin{tabular}{c l l l} \hline \hline  & & **High-res** & **Low-res** \\ \cline{2-4}  & SFT (para) & WMT (2M) & Wiki (200K) \\
**Train** & FB (mono) & CC (85K) & Wiki (84K) \\ \hline \multirow{2}{*}{**Dev**} & WMT21 & Flores200 \\  & Test & WMT22 & WMT22 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Data used in the different phases of our pipeline. SFT uses parallel data, while FB uses monolingual data. Numbers indicate the number of samples. “FB” denotes feedback training. “CC” and “Wiki” denote CCMatrix [46]and WikiMatrix [46]. WMT and CC are both general domain.

 

score starts decreasing from the onset of the training process. Figure 2(a) and Figure 2(c) exhibit a trend of initial increase followed by a decrease. In such scenarios, a relatively good checkpoint can be chosen based on the performance on the development set. Conversely, Figure 2(b) does not display overoptimization. We conjecture that the severity of overoptimization could be related to multiple factors, including language pairs, the reward model, and the SFT model.

\(\bullet\)**RAFT+ alleviates overoptimization effectively.** The BLEURT scores of RAFT+ in the four settings consistently increase as the training progresses. At the same time, the growth rate of the reward scores in RAFT+ is significantly slower than that in RAFT. We even witness a situation where the reward decreases in Figure 2(c). Moreover, in Figure 2(b) where overoptimization did not occur originally, RAFT+ can still achieve a performance close to that of RAFT. However, it is worth noting that the overoptimization problem has not been completely solved, as some errors, like hallucinations, are challenging to identify. Furthermore, we cannot guarantee generalization to all other language pairs since language identification is not reliable for extremely low-resource language Aji et al. (2022), indicating room for improvement. We leave a more comprehensive study to future work.

Main resultsTable 3 shows the main results on test sets when using Comet-qe-da or Comet-qemqm as the QE-based reward model. We present the results of UnitTE-MUP in Appendix D and use chrF as the evaluation metric in Appendix E.

\(\bullet\)**Feedback training brings significant improvements in general.** Regardless of whether the resource setting is high or low and irrespective of the variation in base models, RAFT+ tends to deliver notable enhancements (though still underperforms the WMT22 Best systems), particularly pronounced in the low resource setting. This suggests that current QE models are already equipped with the capability to function as reward models after addressing the overoptimization problem. In contrast, while achieving good gains in most settings, RAFT can suffer from severe overoptimization that might lead to failed training.

\(\bullet\)**The performance of the reward model varies under different resource settings.** In the high-resource setting, Comet-qe-mqm outperforms Comet-qe-da, while the opposite is true in the low-resource setting, which suggests that QE models still have a lot of room for improvement.

\(\bullet\)**Remarkably, even COMET, a reference-based metric, can be overoptimized.** In RAFT training of Nllb-200-1.3B under low-resource setting (bottom of Table 2(b)), there is a marginal increase in COMET scores, yet a significant drop in BLEURT, which is unusual. By inspecting the outputs, we found that severe off-target errors significantly hamper the model's performance. One plau

Figure 3: Training curves under various settings. The metrics are average values for all language pairs on the development set. The QE-based reward model is Comet-qe-da.

 

sible reason might be the similarities between the COMET and COMET-QE models, both of which may be susceptible to these off-target errors. This finding underscores the necessity of treating automatic metrics with caution and the importance of employing multiple metrics simultaneously for a more comprehensive evaluation. Especially for feedback training using a reward model, the increasing of automatic metrics might not indicate true translation quality improvement. We therefore

\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \multirow{2}{*}{Method} & \multicolumn{3}{c}{De\(\Rightarrow\)En} & \multicolumn{3}{c}{En\(\Rightarrow\)De} & \multicolumn{3}{c}{Zh\(\Rightarrow\)En} & \multicolumn{3}{c}{En\(\Rightarrow\)Zn} & \multicolumn{3}{c}{**Average**} \\ \cline{2-10}  & comet & bleurt & comnet & bleurt & comnet & bleurt & comnet & bleurt & comnet & bleurt \\ \hline WMT22 Best & 85.0 & 73.8 & 87.4 & 77.9 & 81.0 & 68.9 & 86.8 & 72.8 & 85.1 & 73.4 \\ \hline \hline \multicolumn{10}{l}{LlamA+2-7B} \\ SFT & 82.5 & 70.5 & 80.7 & 68.2 & 76.1 & 62.3 & 84.9 & 69.3 & 81.0 & 67.6 \\ \hline Reward model: Comet-eq-da & & & & & & & & & \\ RAFT & 83.7 & 72.1 & 82.8 & 71.1 & 78.7 & 65.3 & 85.9 & 70.1 & 82.8\({}_{1:7.7}\) & 69.7\({}_{1:2.1}\) \\ RAFT+ & 83.6 & 72.1 & 84.4 & 73.9 & 79.0 & 66.1 & 85.4 & 69.3 & **83.1\({}_{1:2.1}\)** & **70.3\({}_{2:7.7}\)** \\ \hline \hline \end{tabular} \begin{tabular}{l l l l l l l l l l l} \hline \multirow{2}{*}{Reward model: Comet-ug-mq} & & & & & & & & & & & & \\ RAFT & 83.3 & 72.0 & 84.8 & 75.1 & 77.8 & 64.3 & 86.1 & 70.4 & 83.0\({}_{j:2.0}\) & 70.5\({}_{j:2.9}\) \\ RAFT+ & 83.7 & 72.4 & 85.6 & 75.7 & 78.6 & 65.6 & 85.8 & 70.0 & **83.4\({}_{j:2.4}\)** & **70.9\({}_{i:3.3}\)** \\ \hline \hline \multicolumn{10}{l}{Niller-200-1.3B} \\ SFT & 70.9 & 52.5 & 85.3 & 74.8 & 66.0 & 48.4 & 83.7 & 69.1 & 76.5 & 61.2 \\ \hline \multicolumn{10}{l}{Reward model: Comet-gie-da} \\ RAFT & 73.2 & 52.2 & 85.8 & 75.1 & 67.9 & 50.5 & 84.2 & 68.9 & 77.8\({}_{1:3.3}\) & 61.7\({}_{i:0.5}\) \\ RAFT+ & 74.2 & 56.7 & 85.8 & 75.2 & 69.0 & 52.6 & 84.0 & 67.9 & **78.2\({}_{1:7.7}\)** & **63.1\({}_{1:9.9}\)** \\ \hline \multicolumn{10}{l}{Reward model: Comet-ug-mom} \\ RAFT & 82.8 & 71.3 & 83.9 & 73.4 & 76.1 & 62.3 & 84.6 & 68.6 & 81.8\({}_{i:3.3}\) & 68.9\({}_{i:7.7}\) \\ RAFT+ & 83.3 & 71.8 & 84.6 & 74.4 & 76.7 & 62.9 & 84.6 & 68.4 & **82.3\({}_{i:8.8}\)** & **69.4\({}_{i:8.2}\)** \\ \hline \hline \end{tabular} 
\begin{tabular}{l l l l l l l l l l l l} \hline \multirow{2}{*}{Method} & \multicolumn{3}{c}{En\(\Rightarrow\)Uk} & \multicolumn{3}{c}{Uk\(\Rightarrow\)En} & \multicolumn{3}{c}{Uk\(\Rightarrow\)Cs} & \multicolumn{3}{c}{Cs\(\Rightarrow\)Uk} & \multicolumn{3}{c}{**Average**} \\ \cline{2-10}  & comet & bleurt & comnet & bleurt & comnet & bleurt & comnet & bleurt & comnet & bleurt & comnet & bleurt \\ \hline WMT22 Best & 87.8 & 76.5 & 85.9 & 76.6 & 92.2 & 82.8 & 91.6 & 80.3 & 89.4 & 79.1 \\ \hline \multicolumn{10}{l}{LIlamA-2-7B} \\ SFT & 79.2 & 64.0 & 76.7 & 66.0 & 70.0 & 53.2 & 71.2 & 51.3 & 74.3 & 58.6 \\ \hline \multicolumn{10}{l}{Reward model: Comet-de-da} \\ RAFT & 82.3 & 68.0 & 81.4 & 71.1 & 82.5 & 69.5 & 84.3 & 69.9 & **82.6\({}_{i:8.3}\)** & **69.6\({}_{i:11.0}\)** \\ RAFT+ & 82.0 & 67.8 & 81.5 & 71.2 & 82.2 & 68.8 & 84.5 & 70.1 & **82.6\({}_{i:8.3}\)** & 69.5\({}_{i:10.9}\) \\ \hline \multicolumn{10}{l}{Reward model: Comet-qi-mq} \\ RAFT & 80.7 & 65.5 & 76.7 & 66.0 & 75.7 & 59.9 & 75.2 & 54.8 & 77.1\({}_{i:2.8}\) & 61.5\({}_{i:2.9}\) \\ RAFT+ & 81.2 & 67.0 & 79.2 & 68.9 & 77.3 & 62.3 & 78.8 & 60.7 & **79.1\({}_{i:4.8}\)** & **64.8\({}_{i:2.2}\)** \\ \hline \hline \multicolumn{10}{l}{Niller-200-1.3B} \\ SFT & 83.1 & 70.2 & 71.1 & 62.7 & 73.2 & 61.5 & 57.3 & 43.4 & 71.2 & 59.4 \\ \hline \multicolumn{10}{l}{Reward model: Comet-qie-da} \\ RAFT & 85.2 & 72.5 & 64.7 & 33.2 & 70.5 & 29.7 & 73.8 & 30.1 & 73.6\({}_{i:2.4}\) & 41.4\({}_{i:18.0}\) \\ RAFT+ & 84.5 & 71.3 & 77.7 & 67.0 & 83.1 & 70.3 & 72.0 & 55.1 & **79.3\({}_{i:8.1}\)** & **65.9\({}_{i:6.6}\)** \\ \hline \multicolumn{10}{l}{Reward model: Comet-quism} \\ RAFT & 85.8 & 73.2 & 67.5 & 50.0 & 71.1 & 41.6 & 71.1 & 42.7 & 73.9\({}_{i:2.7}\) & 51.9\({}_{i:7.5}\) \\ RAFT+ & 84.5 & 71.8 & 76.4 & 66.1 & 82.1 & 69.9 & 71.4 & 54.5 & **78.6\({}_{i:7.4}\)** & **65.6\({}_{i:6.2}\)** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Translation performance on the test sets under various settings, using Comet-eq-da and Comet-eq-mq as reward models. Results when UnITE-MUP is used as the reward model are presented in Appendix D. Bold indicates that the average performance of the method exceeds that of SFT and RAFT/RAFT+ within the same QE model. The subscripts indicate the change in performance relative to the SFT. “WMT22 Best” means competition winners for each direction as reported in WMT22 correspond to those used by Hendy et al. (2023).

 

conducted a human evaluation in SS 4.1 and have a discussion in Limitations. It also stresses the significance of recognizing a metric's vulnerabilities during its development, including adjustments for prevalent translation inaccuracies such as length-ratio, off-target errors, hallucinations.

## 4 Analysis

### Human evaluation

As discussed in SS 3.2, automatic metrics may not correlate perfectly with actual translation quality. Therefore, we perform human preference studies on En\(\Leftrightarrow\)Zh test sets. For each test sample, our annotators (professional translators) were presented with a source sentence and two possible translations generated by either the SFT or RAFT+ model (based on Llama-2-7B). They were then tasked with selecting the superior translation or determining that neither translation was better than the other. Figure 4 shows the results of human preference studies. We find that RAFT+ achieves better or equal translations in 92.5% of the cases for En-Zh and 87.3% for Zh-En, compared to SFT, confirming the effectiveness of feedback training.

### Data Efficiency of Feedback Training

Data efficiency refers to the capacity to achieve good performance with less training data. For low-resource language pairs, data efficiency is important since it is costly to annotate high-quality parallel data He et al. (2022b). Feedback training provides an alternative path where humans only need to evaluate rather than generate translations.

We examine the data efficiency of feedback training in the low-resource setting. We collected an extra 3M of parallel data from WikiMatrix Schwenk et al. (2021) and continued training the SFT model (based on Llama-2-7B) using data of different sizes. We adopted two continued training strategies: full parameter fine-tuning and parameter-efficient fine-tuning, specifically employing LoRA Hu et al. (2022). To be fair, we filtered samples with length-ratio and off-target errors in the parallel data. On the other hand, RAFT+ only consumes 10K monolingual data (102atch size \(\times\) 10 iterations). Figure 5 depicts their average performance on test sets. Unexpectedly, the continuous training with increasing amounts of parallel data fails to yield consistent improvements. This observation aligns with a similar phenomenon reported by Zhou et al. (2023). A plausible explanation could be the low quality of the crawled data for low-resource languages. Conversely, RAFT+ performs markedly better using merely 10K monolingual data, exhibiting high data efficiency. It is also worth noting that in RAFT+ the model is only trained on self-generated translations. Therefore, we conjecture that the SFT model already has strong translation potential inherently, and RAFT+ can bring this potential to fruition.

### Effects of Scaling Model Size and Pretraining

In this section, we examine the effects of two dimensions of the base model on feedback training: model size and the presence of pretraining. We conduct experiments under the high-resource setting, using Comet-ge-mqm as the reward model. For model size, we consider NLLB-200-(600M, 1.3B, 3.3B) as base models; for pretraining, we randomly initialize NLLB-200-600M and perform the SFT from scratch with 80M parallel data, which we call Random-600M. From Figure 6, we have

Figure 4: Human preference evaluation, comparing RAFT+ to SFT model on En\(\Leftrightarrow\)Zh test sets.

Figure 5: Comparison between RAFT+ and continuous training in the low-resource setting.

 

two obvious phenomena: (1) a larger base model size results in a more significant enhancement from feedback training; (2) feedback training is effective only when the base model has undergone pretraining. Combining the two, we deduce that a stronger base model results in more significant improvements from feedback training. A plausible rationale for this could be that a well-established base model inherently possesses great potential, which can be further unlocked by suitable feedback.

### Effects of Hyperparameters

The sampling temperature and the number of candidates are two crucial hyperparameters for RAFT+. Both influence the diversity of the candidates. Intuitively, having more diverse candidates may provide richer feedback signals, potentially leading to better final performance. However, as illustrated in Figure 7, (1) continuously increasing the temperature leads to performance degradation; (2) the boost from continuously increasing the number of candidates approaches saturation. This means that the increase in candidate diversity has reached its upper limit.

## 5 Related Work

### Feedback Training in MT

Many early works have attempted to use various forms of feedback to improve MT, where the model has no access to the gold references but receives partial feedback on its output. Depending on the source of the feedback, they can be categorized into simulated or human feedback.

Simulated feedbackThis type of work utilizes a measure of similarity between the model's output and the reference translation to simulate feedback, e.g., sentence-BLEU and cross-entropy loss Sokolov et al. (2016a, b). Kreutzer et al. (2017) lifts linear bandit learning to neural sequence-to-sequence learning. Lawrence et al. (2017) demonstrates the possibility of counterfactual learning from deterministic bandit logs. Nguyen et al. (2017) propose a training algorithm combining the advantage actor-critic algorithm with the attention-based neural encoder-decoder architecture. Wieting et al. (2019) proposes to use semantic similarity rather than BLEU as simulated feedback. Wu et al. (2018) discusses how to train the MT model using feedback effectively. The main drawback of simulated feedback is that there is no real human involvement, so the metrics being optimized, such as BLEU, do not correlate well with human preferences Freitag et al. (2022). In addition, the need for references presents a challenge for low-resource language pairs.

Human feedbackKreutzer et al. (2018a) utilizes implicit human feedback in a constrained e-commerce scenario, which cannot be applied to general MT. Kreutzer et al. (2018b) investigates how to improve translation using human evaluation data but is limited by the small data size (1K). Concurrently with this work, Finkelstein and Freitag (2023); Ramos et al. (2023) and Gulcehre et al. (2023) also consider using QE models to provide reward signals in the training of MT models. Xu et al. (2024) points out that the references of training data might be worse than model outputs in quality, which aligns with our findings in SS 4.2.

Figure 6: Training curves of RAFT+ (high-resource, Comet-qe-mqm) under different base models. We report the change in BLEURT score for each checkpoint relative to the SFT model.

Figure 7: Effects of temperature \(T\) and the number of candidates \(k\) on the final performance (high-resource setting, Llama-2-7B, Comet-qe-da).

 



### Aligning LLMs with Human

Pretrained on raw text from the internet, LLMs might generate toxic, inaccurate, and unhelpful content (Fernandes et al., 2023). To mitigate these issues, researchers have employed human feedback to better align the behavior of LLMs with human preferences, thereby enhancing their helpfulness and reducing potential harm (Ouyang et al., 2022; Stiennon et al., 2020; Bai et al., 2022). The basic idea involves learning a reward function that captures human preferences and optimizing the LLM using proximal policy optimization (PPO; Schulman et al., 2017) to maximize the reward. Lighter weight training strategies, such as RAFT (Dong et al., 2023), RRHF (Yuan et al., 2023) and DPO (Rafailov et al., 2024), use training data ranking based on rewards or human annotations to align the LLM. LIMA (Zhou et al., 2023) underscores the importance of high-quality supervised data for effective alignment. For a comprehensive understanding of alignment, we recommend readers refer to survey papers: Fernandes et al. (2023) and Wang et al. (2023).

## 6 Conclusion

We explore the potential of using the current QE model as a reward model. By identifying, analyzing, and mitigating the overoptimization problem, we successfully integrate the QE model into feedback training to refine translation quality. We validate its effectiveness across various settings, including human evaluation. Further analysis demonstrates the high data efficiency of feedback training using the QE-based reward model. Lastly, we delve into the impact of the base model and hyperparameters on feedback training.

## Acknowledgements

Zhiwei and Rui are with MT-Lab, Department of Computer Science and Engineering, School of Electronic Information and Electrical Engineering, and also with the MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai 200204, China. Rui and Zhiwei are supported by the Tencent Open Fund (RBFR2023012), the National Natural Science Foundation of China (62176153), and the Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102).

## Limitations

An increase in automatic metrics do not necessarily indicate a true improvement in translation qualityWhen we conducting feedback training, we need a QE model to provide reward score and a metric model to evaluate the final performance. Though they play different roles in the whole pipeline, they share the same function of assessing translation quality. Unfortunately, current open-sourcing QE and metric models that show strong correlation with human mostly rely on human evaluation data in WMT, which indicating the overlap between their training data. Therefore, overoptimizing the QE model might lead to unconsciously overoptimizing the metric model, as we observed in SS 3.2. This is the reason why we conducted a human preference study to double check the improvement. However, human annotation is always costly and not scalable, and we only conduct it on En\(\Leftrightarrow\)Zh.

Training AlgorithmThis work only considers a simple yet stable training algorithm, RAFT, in favor of its ease of use and reduced computational requirements. We also implemented the Minimum Risk Training (MRT) algorithm (Shen et al., 2016) in Appendix B, but found it difficult to train stably. While other more commonly used reinforcement learning (RL) algorithms such as PPO exist, their training stability can be influenced by a variety of factors and often demands substantial GPU memory (Dong et al., 2023; Zheng et al., 2023). Hyperparameters, stability, and computational constraints led us to consciously limit our exploration to RAFT, thereby not investigating other potentially effective but more complex training algorithms. This choice simplifies the training process but might limit the general applicability of the approach in different contexts. Future work may explore these alternative algorithms, acknowledging the trade-offs between complexity, stability, and performance.

Granularity of Quality EstimationThis work exclusively focuses on sentence-level QE, neglecting other granularities, such as word-level or document-level QE. The choice of granularity inherently limits the scope of our insights and applications. Understanding how word-level QE feedback might be utilized, or how QE could be employed to enhance document translation quality, presents exciting avenues for future research

Imbalance in Multilingual NMT - DataThis work does not consider the imbalances present in multilingual NMT. In real-world scenarios, the quantity of available corpora varies significantly across different languages, which might lead to biased or suboptimal results. However, considering unbalanced scenarios introduces many additional factors, such as different ways of sampling training data. Therefore, we constrain our focus to the balanced situation in this work, recognizing the need for future research to address this complexity and explore methods that can better handle the imbalances inherent in multilingual contexts.

Imbalance in Multilingual NMT - RewardAlthough we kept the amount of data the same for different language pairs, we still observed training imbalance problems due to "uneven" distribution of rewards. Specifically, in Table 3a, neither RAFT nor RAFT+ consistently improves NLLB in En\(\rightarrow\)De and En\(\rightarrow\)Zh (both are _from-English_ direction) at the high resource setting. On the other hand, the performance of the other two _to-English_ directions (De\(\rightarrow\)En and Zh\(\rightarrow\)En) has been significantly improved by notable margins (BLEURT ! De\(\rightarrow\)En: 53.5\(\rightarrow\)71.3, Zh\(\rightarrow\)En: 48.4\(\rightarrow\)62.9). This means that during training, the model allocates more "capacity" to the _to-English_ directions than to the _from-English_ directions, i.e., they are not balanced. The intuitive reason is that the model "finds" optimizing the _to-English_ directions offers a quick boost in reward, thereby "ignoring" the _from-English_ directions which have less room for improvement. This involves _how the reward can be reasonably distributed among different language pairs_, which is less discussed in this work, but a meaningful and underexplored future direction.

Applicability of the Method to Mitigate OveroptimizationThe current method for mitigating overoptimization focuses on detecting relatively easy-to-identify errors such as len-ratio and off-target errors. More elusive mistakes, such as hallucinations, remain unaddressed and might potentially lead to overoptimization as well Guerreiro et al. (2023). In addition, current language detectors are not reliable for extremely low-resource languages Aji et al. (2022), limiting the applicability in these contexts.

## References

* Aji et al. (2022) Alham Fikri Aji, Genta Indra Winata, Fajri Koto, Samuel Cahyawijaya, Ade Romadhomy, Rahmad Mahendra, Kemal Kurniawan, David Moeljadi, Rativdo Eko Prasojo, Timothy Baldwin, Jey Han Lau, and Sebastian Ruder. 2022. One country, 700+ languages: NLP challenges for underrepresented languages and dialects in Indonesia. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 7226-7249, Dublin, Ireland. Association for Computational Linguistics.
* Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. _arXiv preprint arXiv:2204.05862_.
* Bawden and Yvon (2023) Rachel Bawden and Francois Yvon. 2023. Investigating the translation performance of a large multilingual language model: the case of bloom. _arXiv preprint arXiv:2303.01911_.
* Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Costa-jussas et al. (2022) Marta R Costa-jussas, James Cross, Onur Celebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. _arXiv preprint arXiv:2207.04672_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dong et al. (2023) Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. Raft: Reward ranked finetuning for generative foundation model alignment. _arXiv preprint arXiv:2304.06767_.
* Fernandes et al. (2022) Patrick Fernandes, Antonio Farinhas, Ricardo Rei, Jose G. C. de Souza, Perez Ogaroya, Graham Neubig, and Andre Martins. 2022. Quality-aware decoding for neural machine translation. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: 

Human Language Technologies_, pages 1396-1412, Seattle, United States. Association for Computational Linguistics.
* Fernandes et al. (2023) Patrick Fernandes, Aman Madaan, Emmy Liu, Antonio Farinhas, Pedro Henrique Martins, Amanda Bertsch, Jose GC de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, et al. 2023. Bridging the gap: A survey on integrating (human) feedback for natural language generation. _arXiv preprint arXiv:2305.00955_.
* Finkelstein and Freitag (2023) Mara Finkelstein and Markus Freitag. 2023. Mbr and qe finetuning: Training-time distillation of the best and most expensive decoding methods. _arXiv preprint arXiv:2309.10966_.
* neural metrics are better and more robust. In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Gao et al. (2023) Leo Gao, John Schulman, and Jacob Hilton. 2023. Scaling laws for reward model overoptimization. In _International Conference on Machine Learning_, pages 10835-10866. PMLR.
* Goyal et al. (2022) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzman, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. _Transactions of the Association for Computational Linguistics_, 10:522-538.
* Guerreiro et al. (2023) Nuno M. Guerreiro, Elena Voita, and Andre Martins. 2023. Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 1059-1075, Dubrovnik, Croatia. Association for Computational Linguistics.
* Gulcehre et al. (2023) Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Maosen Wang, Chenjie Gu, et al. 2023. Reinforced self-training (rest) for language modeling. _arXiv preprint arXiv:2308.08998_.
* He et al. (2024) Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. 2024. Exploring Human-Like Translation Strategy with Large Language Models. _Transactions of the Association for Computational Linguistics_, 12:229-246.
* shanghai jiao tong university low-resource translation system for the WMT22 translation task. In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 260-267, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* He et al. (2022b) Zhiwei He, Xing Wang, Rui Wang, Shuming Shi, and Zhaopeng Tu. 2022b. Bridging the data gap between training and inference for unsupervised neural machine translation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 6611-6623, Dublin, Ireland. Association for Computational Linguistics.
* Hendy et al. (2023) Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. _arXiv preprint arXiv:2302.09210_.
* Hu et al. (2022) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In _International Conference on Learning Representations_.
* Kocni et al. (2022) Tom Kocni, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novak, Martin Popel, and Maja Popovic. 2022. Findings of the 2022 conference on machine translation (WMT22). In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 1-45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Kocmi et al. (2021) Tom Kocni, Christian Federmann, Roman Grundkiewicz, March Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In _Proceedings of the Sixth Conference on Machine Translation_, pages 478-494, Online. Association for Computational Linguistics.
* Louisiana. Association for Computational Linguistics.
* Kreutzer et al. (2017) Julia Kreutzer, Artem Sokolov, and Stefan Riezler. 2017. Bandit structured prediction for neural sequence-to-sequence learning. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1503-1513, Vancouver, Canada. Association for Computational Linguistics 

* Kreutzer et al. (2018b) Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. 2018b. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1777-1788, Melbourne, Australia. Association for Computational Linguistics.
* Lawrence et al. (2017) Carolin Lawrence, Artem Sokolov, and Stefan Riezler. 2017. Counterfactual learning from bandit feedback under deterministic logging : A case study in statistical machine translation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2566-2576, Copenhagen, Denmark. Association for Computational Linguistics.
* Nguyen et al. (2017) Khanh Nguyen, Hal Daume III, and Jordan Boyd-Graber. 2017. Reinforcement learning for bandit neural machine translation with simulated human feedback. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1464-1474, Copenhagen, Denmark. Association for Computational Linguistics.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In _Advances in Neural Information Processing Systems_.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. _arXiv preprint arXiv:2305.18290_.
* Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. _Advances in Neural Information Processing Systems_, 36.
* Ramos et al. (2023) Miguel Moura Ramos, Patrick Fernandes, Antonio Farinhas, and Andre FT Martins. 2023. Aligning neural machine translation models: Human feedback in training and inference. _arXiv preprint arXiv:2311.09132_.
* Rei et al. (2022a) Ricardo Rei, Jose G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andre F. T. Martins. 2022a. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 578-585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Rei et al. (2021) Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, Andre F. T. Martins, and Alon Lavie. 2021. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In _Proceedings of the Sixth Conference on Machine Translation_, pages 1030-1040, Online. Association for Computational Linguistics.
* Rei et al. (2022b) Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jose G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and Andre F. T. Martins. 2022b. CometKiwN: IST-unbabel 2022 submission for the quality estimation shared task. In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 634-645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _arXiv preprint arXiv:1707.06347_.
* Schwenk et al. (2021a) Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzman. 2021a. WikitaTatrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1351-1361, Online. Association for Computational Linguistics.
* Schwenk et al. (2021b) Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021b. CCMatrix: Mining billions of high-quality parallel sentences on the web. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6490-6500, Online. Association for Computational Linguistics.
* Sellam et al. (2020) Thibault Sellam, Dipjanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892, Online. Association for Computational Linguistics.
* Sennrich et al. (2017) Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Laubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a toolkit for neural machine translation. In _Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics_, pages 65-68, Valencia, Spain. Association for Computational Linguistics.

 

Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016Minimum risk training for neural machine translation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1683-1692, Berlin, Germany. Association for Computational Linguistics.
* Sokolov et al. (2016a) Artem Sokolov, Julia Kreutzer, Christopher Lo, and Stefan Riezler. 2016a. Learning structured predictors from bandit feedback for interactive NLP. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1610-1620, Berlin, Germany. Association for Computational Linguistics.
* Sokolov et al. (2016b) Artem Sokolov, Julia Kreutzer, Stefan Riezler, and Christopher Lo. 2016b. Stochastic structured prediction under bandit feedback. In _Advances in Neural Information Processing Systems_, volume 29. Curran Associates, Inc.
* Sokolov et al. (2017) Artem Sokolov, Julia Kreutzer, Kellen Sunderland, Pavel Danchenko, Witold Szymaniak, Hagen Furstenau, and Stefan Riezler. 2017. A shared task on bandit learning for machine translation. In _Proceedings of the Second Conference on Machine Translation_, pages 514-524, Copenhagen, Denmark. Association for Computational Linguistics.
* Specia et al. (2020) Lucia Specia, Frederic Blain, Marina Fomicheva, Erick Fonseca, Vishrav Chaudhary, Francisco Guzzan, and Andre F. T. Martins. 2020. Findings of the WMT 2020 shared task on quality estimation. In _Proceedings of the Fifth Conference on Machine Translation_, pages 743-764, Online. Association for Computational Linguistics.
* Specia et al. (2021) Lucia Specia, Frederic Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary, and Andre F. T. Martins. 2021. Findings of the WMT 2021 shared task on quality estimation. In _Proceedings of the Sixth Conference on Machine Translation_, pages 684-725, Online. Association for Computational Linguistics.
* Stahl (2023) Peter M. Stahl. 2023. lingua-py.
* Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. _Advances in Neural Information Processing Systems_, 33:3008-3021.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Wan et al. (2022) Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek Wong, and Lidia Chao. 2022. UniTE: Unified translation evaluation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8117-8127, Dublin, Ireland. Association for Computational Linguistics.
* Wang et al. (2023) Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Aligning large language models with human: A survey. _arXiv preprint arXiv:2307.12966_.
* Wieting et al. (2019) John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. 2019. Beyond BLEU:training neural machine translation with semantic similarity. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4344-4355, Florence, Italy. Association for Computational Linguistics.
* Wu et al. (2018) Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and Ti-Yan Liu. 2018. A study of reinforcement learning for neural machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3612-3621, Brussels, Belgium. Association for Computational Linguistics.
* Xu et al. (2023) Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2023. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. _arXiv preprint arXiv:2401.08417_.
* Yan et al. (2023) Yiming Yan, Tao Wang, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Mingxuan Wang. 2023. BLEURT has universal translations: An analysis of automatic metrics by minimum risk training. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 5428-5443, Toronto, Canada. Association for Computational Linguistics.
* Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses to align language models with human feedback without tears. _arXiv preprint arXiv:2304.05302_.
* Zerva et al. (2022) Chrysoula Zerva, Frederic Blain, Ricardo Rei, Piyawat Lervittyakumiporn, Jose G. C. de Souza, Steffen Eger, Diptsh Kanojla, Duarte Alves, Constantin Orsan, Marina Fomicheva, Andre F. T. Martins, and Lucia Specia. 2022. Findings of the WMT 2022 shared task on quality estimation. In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 69-99, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Zhao et al. (2023) Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. 2023. Calibrating sequence likelihood improves conditional language generation. In _The Eleventh International Conference on Learning Representations_ 

Rui Zheng, Shihan Dou, Songyang Gao, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Limao Xiong, Lu Chen, et al. 2023. Secrets of r hhf in large language models part i: Ppo. _arXiv preprint arXiv:2307.04964_.
* Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Pavin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avika Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. _arXiv preprint arXiv:2305.11206_.
* Zhu et al. (2023a) Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shuijian Huang. 2023a. Multilingual machine translation with large language models: Empirical results and analysis. _arXiv preprint arXiv:2304.04675_.
* Zhu et al. (2023b) Wenhong Zhu, Hongkun Hao, Zhiwei He, Yunze Song, Yumeng Zhang, Hanku Hu, Yiran Wei, Rui Wang, and Hongyuan Lu. 2023b. Clean-eval: Clean evaluation on contaminated large language models. _arXiv preprint arXiv:2311.09154_.

 

Error Detection and Penalty

We penalize for len-ratio errors and off-target errors. We consider translations whose length ratios do not belong to \([L,U]\) as len-ratio errors. We computed the distribution of length ratios of the SFT training data for each language pair (using the tokenizer of the corresponding model), taking \([L,U]\) so that it covers the top 50% of the most frequent length ratios. We adopt language detector from Stahl (2023) to detect off-target errors.

## Appendix B MRT Implementation

MRT (Shen et al., 2016) is designed to directly optimize the model with respect to evaluation metrics. However, we failed to get meaningful improvements when using MRT.

The _risk_ of MRT can be formulate as:

\[{\cal R}(\theta)=\sum_{(x,y^{*})\in{\cal D}}\mathbb{E}_{y\sim P( y|x;\theta)}\left[\Delta\left(y,y^{*}\right)\right], \tag{3}\]

where \(y^{*}\) represents the ground-truth translation. The \(\Delta\) function serves as the loss function and can be instantiated using various evaluation metrics, for example, \(1-\mathrm{BLEU}(y,y^{*})\). The objective or MRT is to minimize \({\cal R}(\theta)\).

In out setting, \({\cal D}\) is the training distribution of \(x\), i.e., monolingual data, and we use QE-based reward \(r(x,y)\) to indicate the quality of \(y\) as the translation of \(x\). Therefore, we modify the _risk_ as:

\[{\cal R}(\theta)=\sum_{x\in{\cal D}}\mathbb{E}_{y\sim P(y|x;\theta)}\left[1- r(x,y)\right]. \tag{4}\]

In practice, we sample \(k\) candidates and normalize the probabilities to approximate the expectation. Let \(S_{k}\) denote a set containing \(k\) candidates drawn from the distribution \(P_{T}(y|x;\theta)\), where \(T\) is the sampling temperature. \(R(\theta)\) is approximated as:

\[\tilde{R}(\theta)=\sum_{x\in D}\sum_{y\in S_{k}}\tilde{P}(y|x;\theta)[1-r(x,y)], \tag{5}\]

where

\[\tilde{P}(y|x;\theta)=\frac{P(y|x;\theta)^{\alpha}}{\sum_{y^{\prime}\in S_{k}} P(y^{\prime}|x;\theta)^{\alpha}}, \tag{6}\]

and \(\alpha\) is a hyperparameter. Algorithm 2 shows the details of MRT. Similar to RAFT+, when adding a penalty term to reward, we call it MRT+. We set the penalty term \(P=1\) and \(\alpha=0.005\lx@math@degree\), and leave other hyperparameters the same as RAFT.

```
0: Training set \({\cal X}\), reward function \(r(x,y)\), initial model \(M_{0}=P(y|x;\theta_{0})\), batch size \(b\), temperature \(T\), the number of candidate \(k\)
1:for iteration \(i\) in \(0,1,\ldots,N-1\)do
2:\(D_{i}\leftarrow\text{SampleBatch}({\cal X},b)\)
3:\({\cal B}=\emptyset\)
4:for\(x\in D_{i}\)do
5:\(y_{1},\dots,y_{k}\sim P_{T}(y|x;\theta_{i})\)
6:\(S_{k}=\{y_{1},\dots,y_{k}\}\)
7:\({\cal B}={\cal B}\cup\{(x,S_{k})\}\)
8: Fine-tune \(\theta_{i}\) on \({\cal B}\) to obtain \(M_{i+1}=P(y|x;\theta_{i+1})\) using MRT risk: \(\tilde{R}(\theta_{i})\).
```

**Algorithm 2** MRT

Figure 8 shows that vanilla MRT suffers from the overoptimization problem, manifested as an increase in reward while translation quality declines. Additionally, MRT+ also poses challenges in achieving stable convergence.

## Appendix C More Training Curves

Figure 9 shows the training curves when using Comet-qe-mqm and UnitTE-MUP as the reward models. Consistent with Figure 3, vanilla RAFT suffers from severe overoptimization problems in most cases, which are greatly alleviated by RAFT+.

## Appendix D UniTE-MUP as the Reward Model

Table 4 presents the test results when UniTE-MUP functions as the reward model. The main trends are consistent with those shown in Table 3. We also observe that neither RAFT nor RAFT+ achieves positive improvement in high-resource language pairs when Nllb-200-1.3B is the base model. We speculate that this lack of improvement

Figure 8: Training curves of MRT and MRT+ under high-resource setting. NLLB-200-1.3B and Comet-qe-mqm are used as base model and QE-based reward model, respectively.

 

is due to overoptimization problems caused by the presence of errors that are not related to length ratio and off-target errors.

## Appendix E chrF as the Evaluation Metric

Table 5 shows the chrF values for main results.

Figure 9: Training curves under various settings when using Comet-qe-mqm and UniTE-MUP as reward models. The metrics are average values for all language pairs on the development set.

 



\begin{table}
 (a) High-resource language pairs

 (b) Low-resource language pairs

\end{table}
Table 4: Test results under various settings when UniTE-MUP functions as the reward model. Bold indicates that the average performance of the method exceeds that of SFT and RAFT/RAFT+ within the same QE model. The subscripts indicate the change in performance relative to the SFT.