Venue: NAACL 2024

# A Preference-driven Paradigm for Enhanced Translation with Large Language Models

Dawei Zhu\({}^{1,2*}\) Sony Trenous\({}^{1}\) Xiaoyu Shen\({}^{1}\) Dietrich Klakow\({}^{2}\)

Bill Byrne\({}^{1}\) Eva Hasler\({}^{1}\)

\({}^{1}\)Amazon AGI

\({}^{2}\)Saarland University, Saarland Informatics Campus

{daweizhu,trenous,willbyrn,ehasler}@amazon.com

Work done during an internship at Amazon.Now with Eastern Institute of Technology, Ningbo.Our dataset, code, and models are available at: [https://github.com/amazon-science/preference-driven-mt](https://github.com/amazon-science/preference-driven-mt)

###### Abstract

Recent research has shown that large language models (LLMs) can achieve remarkable translation performance through supervised fine-tuning (SFT) using only a small amount of parallel data. However, SFT simply instructs the model to imitate the reference translations at the token level, making it vulnerable to the noise present in the references. Hence, the assistance from SFT often reaches a plateau once the LLMs have achieved a certain level of translation capability, and further increasing the size of parallel data does not provide additional benefits. To overcome this plateau associated with imitation-based SFT, we propose a preference-based approach built upon the Plackett-Luce model. The objective is to steer LLMs towards a more nuanced understanding of translation preferences from a holistic view, while also being more resilient in the absence of gold translations. We further build a dataset named MAPLE to verify the effectiveness of our approach, which includes multiple translations of varying quality for each source sentence. Extensive experiments demonstrate the superiority of our approach in "breaking the plateau" across diverse LLMs and test settings. Our in-depth analysis underscores the pivotal role of diverse translations and accurate preference scores in the success of our approach.1

Footnote 1: Work done during an internship at Amazon.

+
Footnote †: Now with Eastern Institute of Technology, Ningbo.

+
Footnote †: Our dataset, code, and models are available at: [https://github.com/amazon-science/preference-driven-mt](https://github.com/amazon-science/preference-driven-mt)

## 1 Introduction

The emergence of Large Language Models (LLMs) has significantly transformed the landscape of NLP, showcasing outstanding capabilities in a spectrum of NLP tasks Brown et al. (2020); Scao et al. (2022); Chowdhery et al. (2023); Touvron et al. (2023a). This transformation extends to machine translation (MT) OpenAI (2023); Jiao et al. (2023b); Hendy et al. (2023). Through supervised fine-tuning (SFT) using a small amount of parallel data, LLMs demonstrate the capability to compete with established commercial translation services such as Google Translate, particularly in high-resource languages Jiao et al. (2023a); Zhang et al. (2023b).

Nevertheless, SFT trains the model to imitate reference translations token by token, making it vulnerable to the noise present within the data Ott et al. (2018); Zhou et al. (2023); Touvron et al. (2023b). The noise can stem not only from the lack of attention by annotators, but also from the inherent challenge of achieving perfect translations due to the intricate interplay of language, culture, and vocabulary. As an adept translator requires not only linguistic proficiency but also a deep understanding of cultural contexts and nuances in both the source and target, it is nearly unattainable to gather extensive parallel translations of top-notch equality Khayrallah and Koehn (2018); Herold et al. (2022); Maillard et al. (2023). As a result, the performance enhancement achieved through SFT often quickly reaches a plateau. Further increasing the volume of parallel translations typically yields minimal additional benefits, and may instead impair the translation capabilities of LLMs Xu et al. (2023).

To alleviate aforementioned limitation of SFT, endeavors have been made to provide LLMs with holistic assessment of contrasting examples rather than token-level limitations. Jiao et al. (2023a); Chen et al. (2023) add a flawed translation to the reference translation in the model input, encouraging the target LLM to recognize their quality difference. Zeng et al. (2023) also use a pair of translations, but they additionally optimize the LLM to favor better translations through ranking loss. Nevertheless, these works have shared limitations. First, the flawed translations are either generated by adding artificial noise to the reference translations or by other (smaller) MT systems. These imperfections in translations can be obvious and easy for

LLM to distinguish, weakening the learning signal. Second, they only provide the relative ranking of the two translations, without quantifying the extent of their quality differences.

In this work, we present a framework based on the Plackett-Luce model to explicitly align the generation probability of the target LLM with human preferences (Plackett, 1975). Instead of using artificial noise, we collect contrasting translations generated by our target LLM, directing our optimization efforts toward "hard negative examples" (Robinson et al., 2021). Human preferences are denoted with precise scores rather than general ranking orders to teach LLMs about the nuances in different translations. LLMs are then trained to enhance their capabilities incrementally from the learnt nuances without depending solely on the existence of "gold references", so as to effectively break the plateau associated with SFT.

We build a dataset, which we refer to as MAPLE, to facilitate preference learning. It equips each source sentence with five translations in diverse quality, scored by professional translators. By performing preference learning on MAPLE, our final MT model outperforms other MT models based on the same foundation LLM by up to 3.96 COMET score. We further show that while the intention of creating MAPLE is to enhance our target LLM, it can be reused to improve other LLMs, helping them break the performance plateau with up to 1.4M parallel data. Finally, we analyze the key factors that make preference learning effective.

Our contributions are as follows. **(1)** We leverage preference learning to teach LLMs a holistic notion of translation quality. Extensive experiments show that our model consistently outperforms strong baselines on two test sets across four translation directions. **(2)** We revisit the underlying modelling assumptions leading to the Bradley-Terry and Plackett-Luce ranking models and discuss how preference distances can be incorporated directly into the ranking models. **(3)** We meticulously construct an MT-oriented preference dataset, MAPLE, employing professional human translators to obtain quality scores for multiple translations corresponding to the same source sentence. We release our dataset to facilitate future MT research. **(4)** Our indiepth analysis reveals that high-contrast pairs and accurate quality scores are crucial in enhancing the effectiveness of our approach, providing guidance for maximizing the benefits of preference learning.

## 2 Related Work

LLM-based MT.One simple and effective approach to use LLMs for translation tasks is through prompting. Research in this field involves examining the impact of model sizes, the number of examples ("shots") used, and template choices (Zhang et al., 2023; Bawden and Yvon, 2023; Mu et al., 2023; Zhang et al., 2024). Moreover, (Ghazvininejad et al., 2023; He et al., 2023) highlight that better translations can be achieved by adding supplementary information to prompts, or engaging LLMs in related tasks prior to translation. Alternatively, another research direction seeks to fully tailor LLMs for MT tasks. Jiao et al. (2023a); Zeng et al. (2023); Chen et al. (2023); Alves et al. (2023); Zhang et al. (2023b) further train LLMs on parallel data via (parameter-efficient) fine-tuning. Xu et al. (2023) show that increasing the size of parallel data may not further improve LLM. The diminished returns from increasing data volume are likely due to data noise. Recent analyses suggest that quality trumps quantity when it comes to data effectiveness (Zhu et al., 2023; Zhou et al., 2023). Leveraging these insights, we goes beyond merely fitting the reference translations. Instead, we aim to enhance the LLM's ability to discern translations of varying quality, encouraging the generation of more precise translations while suppressing flawed outputs.

Human preference alignment.Ouyang et al. (2022) align LLMs with human intentions and values by training a reward model for preference ranking and optimizing the LLMs through the PPO algorithm (Schulman et al., 2017). However, the online reinforcement learning nature of PPO leads to considerable computational costs and is known for its sensitivity to hyperparameters (Islam et al., 2017; Huang et al., 2022). To ease the alignment, Hu et al. (2023); Dong et al. (2023) suggest offline RL algorithms where samples are pre-generated. Further research goes a step beyond by directly employing the target LLMs as reward models. Yuan et al. (2023) use a ranking loss to steer LLMs towards generating helpful responses and avoiding harmful ones. In a similar vein, Rafailov et al. (2023); Song et al. (2023); Hejna et al. (2023) use the Plackett-Luce model (Plackett, 1975) to capture human preferences in alignment. In this work, we adopt the Plackett-Luce model to MT, teaching the model to discern nuances in different translations and to prefer accurate translations 

Methodology

We aim to enhance LLM in MT tasks via a two-stage optimization process. We first fine-tune the target LLM with a small set of high-quality parallel data to elicit its translation ability (Section 3.1). This mirrors the supervised fine-tuning approach used in prior work, where LLMs were tailored to follow instructions Taori et al. (2023); Zheng et al. (2023). We then use preference learning to guide the LLM to prioritize the generation of accurate translations over flawed ones (Section 3.2).

### Supervised fine-tuning

We begin with optimizing our target LLM on parallel data to specialize it for translation. Let \(x\) and \(y\) denote the source and target sentence, respectively. Following Jiao et al. (2023) we first construct a prompt by applying an instruction template \({\cal I}\) to \(x\). The instruction template is randomly sampled from an instruction pool for each training sample. The target LLM, denoted by \(\pi_{\theta}\) is optimized through the log-likelihood loss:

\[{\cal L}_{SFT}(\pi_{\theta}) =-\log\pi_{\theta}(x,y)\] \[=-\sum_{t}\log P_{\pi_{\theta}}(y_{t}|y_{1,\cdots,t-1},{\cal I}( x)) \tag{1}\]

where \(\pi_{\theta}(x,y)\) denotes the likelihood of \(\pi_{\theta}\) generating output \(y\) given input \(x\). Note that in a standard implementation, a decoder-only LLM will also predict tokens within \({\cal I}(x)\), we zero-out the loss on these tokens as our main goal is to teach translation, not to model the input distribution Touvron et al. (2023).2

Footnote 2: As per Ouyang et al. (2022), we use the term “SFT” which is interchangeably referred to as “instruction-tuning” or simply “fine-tuning” in current literature to convey the same concept.

### Preference learning

The goal of the preference learning stage is to explicitly optimize the target LLM to favor accurate translations over erroneous ones. Formally, consider a set of translations \(y^{1},\cdots,y^{L}\) corresponding to a source sentence \(x\). We assume that these translations are ordered by preference: \(y^{i}\succ_{x}y^{j}\) for \(i<j\). That is, translation \(y^{i}\) is preferred over \(y^{j}\) as a translation of the source sentence \(x\). We further assume that there is some underlying reward model \(r^{*}\) that reflects the quality of the translations, which we cannot access but which we can approximate. Under the Plackett-Luce ranking model Plackett (1975), the distribution of preferences can be formulated as follows:

\[p^{*}(y^{1:L}_{\succ_{x}}|x)=\prod_{i=1}^{L-1}\frac{\exp(r^{*}(x,y^{i}))}{\sum _{j=i}^{L}\exp(r^{*}(x,y^{j}))} \tag{2}\]

where \(y^{1:L}_{\succ_{x}}\) is a shorthand for the complete preference ranking \(y^{1}\succ_{x},\cdots,\succ_{x}y^{L}\). In practice, given a training set \({\cal D}\) with translations equipped with a preference ranking, a reward model \(r_{\theta}\) can be trained via maximum likelihood estimation Cheng et al. (2010):

\[{\cal L}_{PL}(r_{\theta})=-\mathbb{E}_{x,y^{1:L}_{\succ_{x}}\in{ \cal D}}\sum_{i=1}^{L-1}\big{[}r_{\theta}(x,y^{i})-\\ \log\sum_{j=i}^{L}\exp(r_{\theta}(x,y^{j}))\big{]} \tag{3}\]

Following recent work Rafailov et al. (2023); Song et al. (2023); Hejna et al. (2023), we parameterize the reward model using the target LLM \(\pi_{\theta}\) and rewrite the above objective as:

\[{\cal L}_{PL}(\pi_{\theta})=-\mathbb{E}_{x,y^{1:L}_{\succ_{x}}\in{\cal D}} \sum_{i=1}^{L-1}\log\frac{\pi_{\theta}(x,y^{i})}{\sum_{j=i}^{L}\pi_{\theta}(x ,y^{j})} \tag{4}\]

where \(r_{\theta}\coloneqq\log(\pi_{\theta})\). Through optimizing Equation 4, we explicitly align the LLM generation probability with the translation quality.

A caveat when optimizing Equation 4 is that the ranking information omits any measure of absolute translation quality, which may lead to inadvertent suppression of the likelihood of good translations. Consider a case where we have a pair of translations, \(y^{1}\) and \(y^{2}\), which are both acceptable translations but have different word orders that causes minor difference in preference. Optimizing Equation 4 may cause the model to raise the probability of \(y^{1}\) and to suppress the probability \(y^{2}\), which may damage the model.3 To address this issue, we follow Song et al. (2023) to consider the preference 

distance in \({\cal L}_{PL}\):

\[{\cal L}_{PLD}(\pi_{\theta})=\] \[-\mathbb{E}_{x,y^{1;L}_{>x}\in{\cal D}}\sum_{i=1}^{L-1}\log\frac{ \pi_{\theta}^{d_{i}}(x,y^{i})}{\sum_{j=i}^{L}\pi_{\theta}^{d_{j}^{i}}(x,y^{j})}\] where \[d_{i}^{j} =r^{*}(x,y^{i})-r^{*}(x,y^{j}),\text{for $j>i$}\] \[d_{i}^{i} =\max_{j>i}(d_{i}^{j}) \tag{5}\]

We obtain the ground truth preference value \(r^{*}(x,y)\) through human annotation, which will be detailed in Section 4. Finally, we combine a SFT loss calculated on the best translation \(y^{1}\) with \({\cal L}_{PLD}\), making the complete loss function:

\[{\cal L}={\cal L}_{PLD}+\beta{\cal L}_{SFT} \tag{6}\]

where the hyperparameter \(\beta\) balances the strengths of preference learning and SFT. We use **PL** as an abbreviation of our preference learning method (i.e., optimizing Equation 6) in the subsequent text.

We now provide some justification for directly incorporating preference distances into the Plackett-Luce model by studying the original derivation of the binary case (\(L=2\)) Thurstone (1927); Mosteller (1951); Bradley (1953); Hamilton et al. (2023). Denote the preferences for \(y^{i}\) and \(y^{j}\) by random variables \(X_{i}\) and \(X_{j}\) such that the probability that \(y^{i}\) is preferred to \(y^{j}\) is \(\pi_{ij}=P(X_{i}>X_{j})\). Assuming that \(X_{i}\) and \(X_{j}\) follow Gumbel distributions4 with locations \(s_{i}\) and \(s_{j}\) and a common scale parameter \(\gamma\), the difference between the two random variables \(d_{ij}=X_{i}-X_{j}\) follows a logistic distribution with location \(s_{i}-s_{j}\) and scale \(\gamma\):

Footnote 4: Assuming preferences arise from a large number of i.i.d. contributions, a normal distribution results in the limit if these are averaged while the Gumbel distribution results from taking their maximum Hamilton et al. (2023).

\[d_{ij}\sim\frac{1}{4\gamma}\operatorname{sech}^{2}(\frac{d_{ ij}-(s_{i}-s_{j})}{2\gamma}) \tag{7}\]

By defining \(\pi_{i}=e^{s_{i}}\), it follows that

\[\pi_{ij} =P(d_{ij}>0)\] \[=\int_{0}^{\infty}\frac{1}{4\gamma}\operatorname{sech}^{2}( \frac{d_{ij}-(s_{i}-s_{j})}{2\gamma})dd_{ij}\] \[=\frac{\pi_{i}^{\frac{1}{\gamma}}}{\pi_{i}^{\frac{1}{\gamma}}+ \pi_{j}^{\frac{1}{\gamma}}} \tag{8}\]

Usually the scale parameter \(\gamma\) is set to \(1\) which yields the Bradley-Terry model Bradley and Terry (1952) (and Equation 13 of Bradley (1953)).

To introduce distance information for the binary preference case, we first note that \(d_{1}^{1}=d_{1}^{2}\) for \(L=2\) (from Equation 5). We then take \(\gamma=\frac{1}{d_{1}^{2}}\) and \(\pi_{i}=\pi_{\theta}(x,y^{i})\), which yields:

\[\pi_{12}=\frac{\pi_{\theta}^{d_{1}^{2}}(x,y^{1})}{\pi_{\theta}^{ d_{1}^{1}}(x,y^{1})+\pi_{\theta}^{d_{1}^{2}}(x,y^{2})} \tag{9}\]

This shows that, for the binary case, preference distances based on the ground truth preferences can be incorporated exactly into the Bradley-Terry distribution by assuming that the \(X_{1}\) and \(X_{2}\) have Gumbel distributions with location parameters \(s_{i}=\log\pi_{\theta}(x,y^{i})\) and scale parameter \(\gamma=\frac{1}{r^{*}(x,y^{1})-r^{*}(x,y^{2})}\).

We derive and discuss the more general case of Equation 5 (\(L>2\)) in Appendix A.

Connections with DropThe preference learning framework investigated here shares a common origin with DPO Rafailov et al. (2023) in the Bradley-Terry and Plackett-Luce models over rankings (Equation 2, and Equation 18 of Rafailov et al. (2023)). Here, the target LLM \(\pi_{\theta}\) serves directly as the reward function (\(r_{\theta}=\log(\pi_{\theta}))\), whereas the DPO reward function also includes a reference distribution \(\pi_{ref}\) that arises from the KL-divergence constraint term in its RL objective function. By contrast, regularization in this work is through an external SFT term (Equation 6) distinct from the reward function. We note also that the use of distance functions based on ground truth reference values brings additional information into our ranking model beyond preference order alone.

## 4 Human preference data collection

We build MAPLE (MAchine translation dataset for Preference LEarning), a dataset derived from WMT20/21 test sets. It contains multiple translations per source sentence, each assigned a real-valued human preference score. MAPLE covers four translation directions: German-to-English (de\(\rightarrow\)en), Chinese-to-English (zh\(\rightarrow\)en), English-to-German (en\(\rightarrow\)de), and English-to-Chinese (en\(\rightarrow\)zh). For each direction, 1.1K source sentences are sampled from the test sets of WMT20/21. Each source sentence is associated with five translations, including one reference translation from 

WMT20/21, and four translations generated by VicunaMT, our target LLM that we aim to improve through preference learning (see training details of VicunaMT in Section 5.1). Among the four translations, one is generated using beam search with a beam size of four, and three translations are obtained through nucleus sampling Holtzman et al. (2020) with \(p=0.9\). We also build a development set containing 200 source sentences per direction sourced from News Crawl 2022. Altogether, MAPLE contains 5.2K source sentences and 26K translations with preference scores. See Appendix B.1 for more detail on the translation collecting process.

Annotation guidance.We send both the source sentence and the corresponding five translations to a panel of translators for evaluation. Each example (source sentence and its translations) is assigned to two different professional translators. They observe the source and the five translations at the same time, and assign scores between 1 (worst) and 6 (best) in increments of 0.2 using a slider. See Appendix B.2 for the full scoring rubric.

Dataset statistics.The score distribution is shown in Fig. 1. The left side shows the score distribution by rank, and we can see MAPLE contains translations that exhibit a wide range of qualities. The right side shows the score distribution by translation type, and as expected the reference is ranked highest, followed by the beam search and the nucleus samples. Nonetheless, there is considerable overlap in the score distributions, and we find that in 21% of the cases, the beam search predictions are scored higher than the reference translation. Table 1 shows an example where the reference translation contains an error.

## 5 Experiments

In this section, we present our MT model trained using the proposed two-stage framework and compare it with strong LLM-based MT systems.

Datasets.We train and evaluate the model on data on four translation directions: en\(\leftrightarrow\)de and en\(\leftrightarrow\)zh. In the SFT stage, we use high-quality test sets from WMT17/18/19 for training, containing 30K parallel sentences in total across the four directions. The WMT21 test set is used for validation. In the preference learning stage, we train on MAPLE, and validation is done on the remaining data from WMT20/21 test sets which was not selected for inclusion in MAPLE. We evaluate trained models on the test sets of WMT22 Kocmi et al. (2022) and FLORES-200 Costa-jussa et al. (2022). Refer to Appendix C.1 for detailed data statistics.

Training.In both SFT and PL stages, we use a learning rate of 5e-6, an effective batch size of 96, and a linear learning rate schedule with a warmup ratio of 0.1. For each training instance, one MT instruction is randomly selected from an instruction pool containing 31 MT instructions. See Appendix C.2 for the complete list of instructions.

Evaluation.At inference time, a fixed MT translation instruction is used. The maximum generation length is set to 512. We use a beam size of 4 for decoding and report BLEU Papineni et al. (2002) and COMET Rei et al. (2022) scores.

### SFT makes good translation models

The SFT stage seeks to train a well-performing foundation MT model using parallel data. When ap

Figure 1: Human score distribution of translations by rank (left) and source (right).

\begin{table}
\begin{tabular}{c p{142.3pt}} \hline \hline Source & Zu einem grossen Tuning-Treflen ist es am Samtagabend (25. Juli 2020) in Numberger Sudstadt gekomen. _(A large tuning meeting took place on Saturday evening July 25, 2020) in Nuremberg’s Salstadt district.)_ \\ \hline Reference translation & A large tuning meetup took place in a city south of Nürnberg this Saturday evening. \\ \hline Best translation & On Saturday evening (25th July 2020) a large tuning meeting took place in Nuremberg’s south district. \\ \hline \hline \end{tabular}
\end{table}
Table 1: An example where the reference translation is less accurate than the best model prediction. More examples are in Appendix B.4.

 

plying SFT, we can either select a pre-trained LLM, or its instruction-tuned version. Prior research uses both types of LLMs interchangeably, leaving it unclear which is preferable in practice. To address this gap, we explore three popular families of open-access LLMs, performing SFT on both their raw (i.e., only pre-trained) and instructed-tuned versions. Specifically, we consider LLaMA-1 [25], Mistral [30] and BLOOM [46]; and their instructions-tuned versions, which are Vicuna [30], Mistral-Instruct, and BLOOMZ [28]. The 7B parameter variants of these models are used here.

Results.Table 2 presents the results before and after SFT. It can be seen that LLMs without instruction-tuning, e.g., BLOOM, perform poorly; we observe that they tend to overgenerate and repeat tokens in the source sentences.5 In contrast, instruction-tuned models work out-of-the-box and exhibit decent performance. It can be also observed that SFT dramatically boosts the performance of raw LLMs, and slightly benefits instruction-tuned LLMs. For BLOOM and Mistral, the performance gap between raw and instruction-tuned models is mostly lost after SFT. An interesting case is Vivuna, where there is a considerable improvement on en\(\rightarrow\)zh over its base model LLaMA-1. This implies that instruction-tuned LLMs may serve as a better base model for SFT. In addition, different LLMs excel in diverse translation directions and their instruction-tuned versions do not deviate from this pattern. For example, both BLOOM and BLOOMZ perform quite well on en\(\rightarrow\)zh, but have a deficiency in en\(\rightarrow\)de. For LLaMA-based models, the opposite holds. This could be due to the fact that German and Chinese are not included (at least, not intentionally) in BLOOM's and LLaMA's pre-training corpora, respectively.

Footnote 5: Overgeneration is also noticed in [9], while it can be partially alleviated by prompt engineering and text post-processing [27], enhancing LLMs’ zero-shot performance is not our primary focus.

The Vicuna+SFT model has the best overall performance and so we select it as our target LLM to be improved through preference learning. We call this model **VicunaMT**. The generated translations in the MAPLE dataset are produced by this model.

### Refining through preference learning

Baselines.We continue training our VicunaMT model on MAPLE through preference learning and compare it with the following competitive systems from recent work: (1) **ParroT**[30] adds a "Hint" field to the model input, prompting the model to generate both correct and incorrect translations. At inference time, the "correct" version of the translations is used for evaluation. (2) **TIM**[30] incorporates standard SFT with a ranking loss computed on a pair of correct and incorrect translations. (3) **SWIE**[31] proposes to attach an instruction adapter to enhance LLMs' long-term attention for better translation. (4) **ALMA**[25] first continues pre-training the LLM on monolingual data, followed by performing SFT on parallel data. Furthermore, as the preference learning stage introduces additional data, a performance gain could be trivial by exposing the model with more samples. To establish a fair comparison, we design two additional

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & de\(\rightarrow\)en & en\(\rightarrow\)de & en\(\rightarrow\)zh & zh\(\rightarrow\)en & Avg. \\ \hline \multicolumn{7}{c}{_WMT22_} \\ BLOOM & 49.86 & 41.95 & 51.59 & 55.21 & 49.65 \\ +_SFT_ & 77.21 & 69.17 & 84.60 & 78.76 & 77.44 \\ BLOOMZ & 74.58 & 62.52 & 83.10 & 78.29 & 74.62 \\ +_SFT_ & 77.24 & 69.32 & **84.95** & **78.77** & 77.57 \\ Mistral & 54.18 & 49.08 & 49.10 & 55.47 & 51.96 \\ +_SFT_ & 83.15 & 81.10 & 81.48 & 78.05 & 80.95 \\ Mistral-Ins. & 82.45 & 80.39 & 76.57 & 77.73 & 79.28 \\ +_SFT_ & 82.68 & 81.23 & 82.49 & 77.73 & 81.03 \\ LLaMA-1 & 63.29 & 55.29 & 45.80 & 55.17 & 54.89 \\ +_SFT_ & 83.30 & 82.54 & 77.58 & 75.78 & 79.80 \\ Viconna & 82.55 & 82.02 & 81.42 & 74.81 & 80.20 \\ +_SFT_ & **83.55** & **82.79** & 81.27 & 77.39 & **81.25** \\ \hline \multicolumn{7}{c}{_FLORES-200_} \\ BLOOM & 55.03 & 42.36 & 53.82 & 60.25 & 52.86 \\ +_SFT_ & 83.69 & 67.43 & **86.06** & **85.45** & 80.66 \\ Mistral & 42.36 & 32.74 & 33.35 & 42.10 & 37.64 \\ +_SFT_ & 88.63 & 84.49 & 80.97 & 85.17 & 84.81 \\ Mistral-Ins. & 88.04 & 82.55 & 73.20 & 83.70 & 81.87 \\ +_SFT_ & 88.21 & 83.73 & 82.41 & 84.77 & 84.78 \\ LLaMA-1 & 58.89 & 52.71 & 42.77 & 49.92 & 51.07 \\ +_SFT_ & 88.50 & 84.82 & 76.73 & 83.09 & 83.26 \\ Vicuna & 87.82 & 84.17 & 81.52 & 81.53 & 83.76 \\ +_SFT_ & **88.66** & **86.27** & 80.62 & 84.44 & **85.00** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Model performance (in COMET score) before and after performing SFT on parallel data. Rows in blue indicate instruction-tuned LLMs. Best results are in **bold**. Instruction-tuned LLMs yield high COMET scores even without SFT. Raw LLMs benefit the most from SFT. Vicuna performs the best on average on both test sets. We exclude BLOOMZ on FLORES-200 as it is a part of BLOOMZ’s training data. Performance measured by BLEU score is reported in Appendix D.

 

baselines: (5) **REF** trains VicunaMT with the reference translations in MAPLE. (6) **BEST** trains VicunaMT with the translations that are scored highest by our annotators. See Table 1 for an example comparison of the reference and best translations. All aforementioned baselines are performed on 7B LLMs (based either on BLOOM-7B or LLaMA-7B). Finally, we also compare our model against commercial LLMs, including ChatGPT and GPT-4.

Results.We report the MT performance of various baselines in Table 3. It can be seen that our VicunaMT model performs well compared to recent MT systems. PL further increases the performance advantage. Our final model, VicunaMT+PL, achieves the highest average performance (83 on WMT22 and 86.3 on FLORES-200), consistently outperforming all LLaMA-1 based models across all directions, with the largest improvement being a 3.96 increase in COMET score. (en\(\rightarrow\)zh on WMT22). Notably, LLaMA-based models are originally much weaker in directions involving Chinese. Through preference learning, VicunaMT reaches a translation performance close to BLOOM-based LLMs. This becomes practically significant when the goal is to deploy a single LLM to handle multiple translation directions. Also, the PL model scores higher than VicunaMT models fine-tuned on the reference and best translations, indicating that the performance gain does not just come from having more data. Compared to the ALMA model, which is based on LLaMA-2 [23], a widely recognized superior open access LLM, our model demonstrates only a slight deficit of 0.59 COMET scores. Note that our strategy is orthogonal to ALMA's approach, which leverages monolingual data. Combining both strategies should lead to even better performance.

We supplement our assessment with a human evaluation, contrasting VicunaMT+PL with SFT-only Vicuna variations including VicunaMT and VicunaMT+REF, as illustrated in Table 4. The human evaluation confirms the trend observed with automatic metrics, where PL substantially outperforms SFT-only variations.

## 6 Analysis

Rouse of preference data.MAPLE contains the translations generated by VicunaMT, which is also the target LLM we aim to improve. There would be additional value if this data could be reused to improve other LLMs. To investigate this, we train both Mistral-Instruct and BLOOMZ on MAPLE using PL. As shown in Table 5, PL improves both models, suggesting that the MAPLE is not limited for use with VicunaMT and can be reused for im 

proving other LLMs.

Limited gains with additional parallel data.Section5.2 shows that the MAPLE dataset, which contains 4.4K preference examples, can be more valuable than an equivalent amount of parallel data with either the reference or the best translations. A natural follow-up question is whether adding more parallel data can close the gap. To answer this question, we collect more data by concatenating WMT20, WMT21 test data with News Commentary v16, making 1.4M parallel sentences in total.6 We fine-tune VicunaMT and Mistral-InstructMT (i.e., Mistral-Instruct after SFT stage) on different proportions of this data and plot the performance curve in Figure2. In both cases, similar to observations in Xu et al. (2023), adding more parallel data does not always improve these models and they never attain the performance level reached by using PL with MAPLE.

Footnote 6: We select News Commentary for its high-quality, domain-matching parallel data to WMT test data. WMT20/21 are included as MAPLE is built on a subset from them.

Diverse translations help more.By default, we perform PL using all five translations provided by MAPLE. We now study the relation between the final model performance and the number of preference translations used. We select \(K=\{2,3,4\}\) translations and rerun the PL algorithm on VicunaMT and Mistral-InstructMT. We explore two selection modes for selecting \(K\) translations. Given five translations sorted by human preference scores in descending order, the _forward_ mode selects the first \(K\) translations (i.e., the best \(K\)), while the _reverse_ mode select the first and last \(K-1\) translations. We compare both modes varying \(K\) and present the results in Figure3. There is a clear disparity in performance with these two selection modes. The reverse mode consistently outperforms the forward mode given the same number of translations, with a larger advantage in low-resource cases, such as when \(K=2\). This is intuitive since the reverse mode always includes the highest- and lowest-scored translations and thus, PL may have a better chance to see "hard negatives" which have low human preference score but high generation probability. The general trend shows that including more preference samples is better, and using all available samples yields the best performance.

Distance information is crucial.Our framework considers the distance information in preference scores (Equation5). We now investigate if this information can be replaced by simply using the ranking information. That is, we set \(d^{j}_{i}=1\) for all translations and rerun the PL algorithm. Table6 shows that when the distance information is available, excluding the SFT loss does not harm

Figure 2: Performance comparison between PL using 4.4K examples from MAPLE and SFT, employing up to 1.4M parallel data. Evaluation is done on WMT22, and COMET scores are averaged across four translation directions. Performing SFT on more parallel data does not always lead to performance gain. PL consistently outperforms SFT in all cases.

 

the performance much. In fact, we achieve the best performance when setting \(\beta=0\) for VicunaMT. However, when the distance information is withheld, we see a clear degradation in performance. We find that a larger \(\beta\) value is required when relying only on the ranking information, but this makes the PL algorithm closer to SFT. As a result, when only the ranking information is provided, VicunaMT performs similarly to the \({\cal L}_{SFT}\) only baseline. Finally, disabling both \(L_{SFT}\) and distance cause a large performance drop.

Better Model Calibration.In our preference learning framework, the model learns both translation and the ability to differentiate between different translation quality. We analyze if PL has successfully transferred human preference to the model. Using the held-out set of MAPLE, we examine the sentence-level correlation between the scores assigned by the human annotators and model generation probability. Specifically, we compute the average Pearson and Kendall's tau correlation varying the number of preference samples (reverse mode). The results are presented in Figure 4. Compared to the SFT baseline, VicunaMT, PL substantially improves the correlation, suggesting that our final model aligns better with human preference.

## 7 Conclusion

We present a preference learning framework to break the performance plateau faced when performing SFT. It enhances the translation capabilities of LLMs by motivating them to differentiate the nuances in different translations. To support this framework, we have carefully curated a preference dataset, named MAPLE, featuring translations of varying quality, each scored by professional translators. Extensive experiments, including human evaluations, confirm the effectiveness of this framework. In addition, we demonstrate that MAPLE can be reused to enhance other LLMs, further bolstering its practical usability. Future research could consider extending our framework into an iterative process for continuous improvement of LLMs' translation capabilities.

### Limitations

This work demonstrates that preference learning can effectively improve LLMs' translation capabilities. However, our study is not exhaustive and has the following limitations.

Low-resource languages.This work centers on translation directions involving high-resource languages where LLMs already exhibit proficiency. The extent to which translations for low-resource languages can leverage our framework remains uncertain. Nevertheless, it is important to emphasize that our framework is language- and model-agnostic, implying its potential applicability to low-resource languages. We leave the investigation into

Figure 3: Sentence-level correlation between model generation probability and human preference scores varying number of translations (\(K\)). PL helps the model align better with human judgement.

\begin{table}
\begin{tabular}{l c c} \hline \hline  & VicunaMT & Mistral-InstrctMT \\ \hline SFT stage & 81.25 & 81.03 \\ PL stage & 83.00 & 82.56 \\ w/o \({\cal L}_{SFT}\) & 83.00 & 82.54 \\ w/o distance & 82.22 & 81.92 \\ w/o \({\cal L}_{SFT}\)/dist. & 74.65 & 60.70 \\ \({\cal L}_{SFT}\) only & 82.07 & 81.79 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Ablation study. PL is less sensitive to \({\cal L}_{SFT}\) than the distance information. Disabling both factors leads to substantial model degradation.

 

this aspect to future work.

Annotation cost.Assigning preference scores for five translations per sentence can be costly, which may hinder the scaling up of the preference dataset. However, as we show in Section 6, a preference learning dataset such as MAPLE offers distinct learning signal that is not covered by massive parallel data. In addition, we highlight that the preference data can be reused to benefit other LLMs. Thus, the collected data is a valuable and reusable resource, rather than a one-time expense.

Noise in human judgement.Inevitably, human preference scores can be subjective, and annotators may not always agree. Additionally, there is a risk of annotators finding shortcuts in the annotation process Ipeirotis et al. (2010); Hosking et al. (2023). To reduce the potential annotation mistakes, we average the scores of two translators for each sample and all translators we employ are experienced in translation assessment.

## Acknowledgements

Bill Byrne holds concurrent appointments as an Amazon Scholar and as Professor of Information Engineering at the University of Cambridge. This paper describes work performed at Amazon.

We thank Felix Hieber, Lei Sun, and Tobias Domhan for their thoughtful advice, in-depth discussions, and implementation support. We would also like to thank our anonymous reviewers for their constructive feedback.

## References

* M. Alves, N. Guerreiro, J. Alves, J. Pombal, R. Rei, J. G. C. de Souza, P. Colombo, and A. F. T. Martins (2023)Steering large language models for machine translation with finetuning and in-context learning. CoRRabs/2310.13448. External Links: Link Cited by: SS5.
* R. Bawden and F. Yvon (2023)Investigating the translation performance of a large multilingual language model: the case of bloom. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023, Tampere, Finland, 12-15 June 2023, pp. 157-170. External Links: Link Cited by: SS5.
* R. Allan Bradley (1953)Some statistical methods in taste testing and quality evaluation. Biometrics9 (1), pp. 22-38. External Links: ISSN 0001-9002, Document, Link Cited by: SS5.
* R. Allan Bradley and M. E. Terry (1952)Rank analysis of incomplete block designs: i. the method of paired comparisons. Biometrika39 (3/4), pp. 324-345. External Links: Link Cited by: SS5.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020)Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. External Links: Link, Document Cited by: SS5.
* Y. Chen, Y. Liu, F. Meng, Y. Chen, J. Xu, and J. Zhou (2023)Improving translation faithfulness of large language models via augmenting instructions. arXiv preprint arXiv:2308.12674. Cited by: SS5.
* W. Cheng, K. Dembczynski, and E. Hullermeier (2010)Label ranking methods based on the blackett-luce model. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), Haifa, Israel, pp. 215-222. External Links: ISBN 978-3-030-1673-3, Link Cited by: SS5.
* W. Cheng and E. Hullermeier (2008)Learning similarity functions from qualitative feedback. In LNAI 5239 Advances in Case-Based Reasoning: The 9th European Conference on Case-Based Reasoning (ECCBR-08), Trier, Germany, pp. 129-134. External Links: Link, Document Cited by: SS5.
* A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsuysachenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhikaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. DeV, L. Michaelski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. Sankaranarayana Pilali, A. Lewkowycz, E. Moreira, R. Child, O. Podozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Friedel (2023)Palm: Scaling language modeling with pathways. Journal of Machine Learning Research24 (240), pp. 1-113. External Links: ISSN 0021-9002, Document, Link Cited by: SS5.
* M. R. Costa-jussa, J. Cross, O. Celebi, M. Elbayad, K. Heafield, K. Heinferman, E. Kalbassi, J. Lam, D. Licht, J. Maillard, A. Sun, S. Wang, G. Wenzek, A. Youngblood, B. Akula, L. Barrault, G. Meja Gonzalez, P. Hansanti, J. Hoffman, S. Jarrett, K. Ram Sadagopan, D. Rowe, S. Spruit, C. Tran, P. Andrews, N. Fazil Ayan, S. Bhosale, S. Edunov, A. Fan, C. Gao, V. Goswami, F. Guzman, P.

 

Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. _CoRR_, abs/2207.04672.
* Dong et al. (2023) Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. 2023. Steerlm: Attribute conditioned FFT as an (user-steerable) alternative to RLHF. _CoRR_, abs/2310.05344.
* neural metrics are better and more robust. In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Ghazvininejad et al. (2023) Marjan Ghazvininejad, Hila Gonen, and Luke Zettlemoyer. 2023. Dictionary-based phrase-level prompting of large language models for machine translation. _CoRR_, abs/2302.07856.
* Hamilton et al. (2023) Ian Hamilton, Nick Tawn, and David Frirth. 2023. The many routes to the ubiquitous Bradley-Terry model. (arXiv:2312.13619).
* He et al. (2023) Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. 2023. Exploring humanlike translation strategy with large language models. _CoRR_, abs/2305.04118.
* Hejna et al. (2023) Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. 2023. Contrastive preference learning: Learning from human feedback without RL. _CoRR_, abs/2310.13639.
* Hendy et al. (2023) Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hank Hassan Awadalla. 2023. How good are GPT models at machine translation? a comprehensive evaluation. _arXiv preprint arXiv:2302.09210_.
* Herold et al. (2022) Christian Herold, Jan Rosendahl, Joris Vanvinckenroye, and Hermann Ney. 2022. Detecting various types of noise for neural machine translation. In _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_, pages 2542-2551. Association for Computational Linguistics.
* Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Hosking et al. (2023) Tom Hosking, Phil Blunsom, and Max Bartolo. 2023. Human feedback is not gold standard. _CoRR_, abs/2309.16349.
* Hu et al. (2023) Jian Hu, Li Tao, June Yang, and Chandler Zhou. 2023. Aligning language models with offline reinforcement learning from human feedback. _CoRR_, abs/2308.12050.
* Huang et al. (2022) Shengyi Huang, Rouslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. 2022. The 37 implementation details of proximal policy optimization. In _ICLR Blog Track_. [https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/).
* Ipeirotis et al. (2010) Panagiotis G Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality management on Amazon Mechanical Turk. In _Proceedings of the ACM SIGKDD workshop on human computation_, pages 64-67.
* Islam et al. (2017) Riashat Islam, Peter Henderson, Maziar Gomrockhi, and Doina Precup. 2017. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. _CoRR_, abs/1708.04133.
* Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devndra Singh Chaplot, Diego de Las Casas, Florian Bressand, Giannac Lengyel, Guillaume Lample, Lucile Saulnier, Lelio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2023. Mistral 7b. _CoRR_, abs/2310.06825.
* Jiao et al. (2023a) Wenxiang Jiao, Jen-tse Huang, Wenxuan Wang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023a. Parrot: Translating during chat using large language models. _arXiv preprint arXiv:2304.02426_.
* Jiao et al. (2023b) Wenxiang Jiao, Wenxuan Wang, JT Huang, Xing Wang, and ZIP Tu. 2023b. Is ChatGPT a good translator? yes with GPT-4 as the engine. _arXiv preprint arXiv:2301.08745_.
* Khayrallah and Koehn (2018) Huda Khayrallah and Philipp Koehn. 2018. On the impact of various types of noise on neural machine translation. In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, NMT@ACL 2018, Melbourne, Australia, July 20, 2018_, pages 74-83. Association for Computational Linguistics.
* Kocni et al. (2022) Tom Kocni, Rachel Bowden, Ondrej Bojar, Anton Dworkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grudkiewicz, Barry Hadlow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novak, Martin Popel, and Maja Popovic. 2022. Findings of the 2022 conference on machine translation (WMT22). In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 1-45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Luce (1959) R. Duncan Luce. 1959. _Individual choice behaviour_. John Wiley.

 

Jean Maillard, Cynthia Gao, Elahe Kalbassi, Kaushik Ram Sadagopan, Vedanuj Goswami, Philipp Roehn, Angela Fan, and Francisco Guzman. 2023. Small data, big impact: Leveraging minimal data for effective machine translation. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2740-2756, Toronto, Canada. Association for Computational Linguistics.
* Maystre and Grossglauser [2015] Lucas Maystre and Matthias Grossglauser. 2015. Fast and accurate inference of Plackett-Luce models. In _Advances in Neural Information Processing Systems_, volume 28.
* Mosteller [1951] Frederick Mosteller. 1951. Remarks on the method of paired comparisons: I. the least squares solution assuming equal standard deviations and equal correlations. _Psychometrika_, 16(1):3-9.
* Mu et al. [2023] Yongyu Mu, Abudurexiti Reheman, Zhiquan Cao, Yuchun Fan, Bei Li, Yinqiao Li, Tong Xiao, Chunliang Zhang, and Jingbo Zhu. 2023. Augmenting large language model translators via translation memories. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 10287-10299, Toronto, Canada. Association for Computational Linguistics.
* Muennighoff et al. [2023] Niklas Muennighoff, Thomas Wang, Lintang Suatwika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 15991-16111. Association for Computational Linguistics.
* OpenAI [2023] OpenAI. 2023. GPT-4 technical report. _CoRR_, abs/2303.08774.
* Ott et al. [2018]Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018_, volume 80 of _Proceedings of Machine Learning Research_, pages 3953-3962. PMLR.
* Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In _Advances in Neural Information Processing Systems_, volume 35, pages 27730-27744. Curran Associates, Inc.
* Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Plackett [1975] Robin L Plackett. 1975. The analysis of permutations. _Journal of the Royal Statistical Society Series C: Applied Statistics_, 24(2):193-202.
* Rafailov et al. [2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. _CoRR_, abs/2305.18290.
* Rei et al. [2022] Ricardo Rei, Jose G. C. de Souza, Duarte M. Alves, Chrysoula Zerva, Ana C. Farinaha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andre F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In _Proceedings of the Seventh Conference on Machine Translation, WMT 2022, Abu Dhabi, United Arab Emirates (Hybrid), December 7-8, 2022_, pages 578-585. Association for Computational Linguistics.
* Robinson et al. [2021] Joshua David Robinson, Ching-Yao Chuang, Suvrit Sha, and Stefanie Jegelka. 2021. Contrastive learning with hard negative samples. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
* De Scao et al. [2023] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagne, Alexandra Sasha Luccioni, Francois Yvon, Matthias Galle, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammamananchion, Thomas Wang, Benoit Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwasac, Rachel Baowden, Staas Behman, Angheliana McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulinier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurencon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aiotor Soroa, Alham Fikri Ajil, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emerze, Christopher Klamm, Colin Leoneg, Daniel van Strien, David Ifeoluuwa Adelani, and et al. 2022. BLOOM: A 176b-parameter open-access multilingual language model. _CoRR_, abs/2211.05100.
* Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _CoRR_, abs/1707.06347.
* Song et al. [2023] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. 2023. Preference ranking optimization for human alignment. _CoRR_, abs/2306.17492.
* Srivastava et al. [2023] Saurabh Srivastava, Chengyue Huang, Weiguo Fan, and Ziyu Yao. 2023. Instance needs more care: Rewriting 

prompts for instances yields better zero-shot performance. _CoRR_, abs/2310.02107.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An instruction-following LLaMA model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Thurstone (1927) Louis L Thurstone. 1927. Psychophysical analysis. _The American journal of psychology_, 38(3):368-389.
* Tuvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Bapitste Rozirez, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. LLaMA: Open and efficient foundation language models. _CoRR_, abs/2302.13971.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyn Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Sagabr Hosseini, Rui Hou, Hakan Inan, Marcia Kardas, Viktor Kerkez, Madian Khabsa, Isabel Klounman, Artem Korenev, Punit Singh Korua, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghail Lu, Yuning Mao, Xavier Martinet, Todor Mihaloylov, Pushkar Mishra, Igor Molbyog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Runga, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Pavxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models. _CoRR_, abs/2307.09288.
* Train (2003) Kenneth Train. 2003. _Discrete Choice Method With Simulation_. Cambridge University Press.
* Xu et al. (2023) Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2023. A paradigm shift in machine translation: Boosting translation performance of large language models. _CoRR_, abs/2309.11674.
* Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. RRHF: rank responses to align language models with human feedback without tears. _CoRR_, abs/2304.05302.
* Zeng et al. (2023) Jiali Zeng, Fanding Meng, Yongjing Yin, and Jie Zhou. 2023. Tim: Teaching large language models to translate with comparison. _arXiv preprint arXiv:2307.04408_.
* Zhang et al. (2023a) Biao Zhang, Barry Hadlow, and Alexandra Birch. 2023a. Prompting large language model for machine translation: A case study. In _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_, volume 202 of _Proceedings of Machine Learning Research_, pages 41092-41110. PMLR.
* Zhang et al. (2024) Maioran Zhang, Vagrant Gautam, Mingyang Wang, Jesusjob O. Alabi, Xiaoyu Shen, Dietrich Klakow, and Marius Mosbach. 2024. The impact of demonstrations on multilingual in-context learning: A multidimensional analysis.
* Zhang et al. (2023b) Xuan Zhang, Navid Rajabi, Kevin Duh, and Philipp Koehm. 2023b. Machine translation with large language models: Prompting, few-shot learning, and fine-tuning with QLoRA. In _Proceedings of the Eighth Conference on Machine Translation_, pages 466-479, Singapore. Association for Computational Linguistics.
* Zheng et al. (2023) Linamin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. _CoRR_, abs/2306.05685.
* Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezehe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. _arXiv preprint arXiv:2305.11206_.
* Zhu et al. (2023) Dawei Zhu, Xiaoyu Shen, Marius Mosbach, Andreas Stephan, and Dietrich Klakow. 2023. Weaker than you think: A critical look at weakly supervised learning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 14229-14253, Toronto, Canada. Association for Computational Linguistics.

 

Incorporating multiple preferences with distance information

In Section 3.2, we demonstrated how the distance information of two preferences can be integrated into preference modeling, as illustrated in Equation 9. A similar analysis can be done for the Plackett-Luce ranking model to incorporate distance metrics across multiple preferences. Specifically, we model the probability of a particular ordering \(X_{1},\cdots,X_{L}\) as follows:

\[P(X_{1}\geq X_{2}\cdots \geq X_{L})\\ =\prod_{i=1}^{L-1}P_{i}(X_{i}>X_{j},\forall j>i)\]

For each distribution \(P_{i}\), let \(X_{j}=s_{j}+\varepsilon_{j}\) for \(j\geq i\), with \(\varepsilon_{j}\sim\) standard Gumbel and independent so that (following Train (2003), Section 3)

\[P_{i}(X_{i}>X_{j},\forall j>i)=\frac{e^{s_{i}}}{\sum_{j\geq i}e^{s_{j}}}\]

This ranking can be interpreted as a sequence of \(L-1\) independent choices: choose the first item, then choose the second among the remaining alternatives, etc. (Maystre and Grossglauser, 2015). It is usually assumed that each independent choice is made by the same judge whose underlying preferences do not change. If we assume \(s_{j}=\log\pi_{\theta}(x,y^{j})\) for this judge then Equation 4 results.

Suppose instead that, rather than a single judge, a succession of \(L-1\) different judges each make one of the sequence of independent choices. The distributions \(P_{i}\) should change to reflect the changing preferences of the judges. In particular, if we introduce the preference distances \(d_{i}^{j}\) for the \(i^{th}\) judge, then we obtain Equation 5 if for each \(P_{i}\) the location parameters are set to \(s_{j}=d_{i}^{j}\log\pi_{\theta}(x,y^{j})\) for \(j\geq i\). We find that this modified version of the Placket-Luce model can work well in practice although we note that these modifications may violate Luce's Choice Axiom (Luce, 1959; Hamilton et al., 2023).

Consider the case of \(L=3\). The Choice Axiom requires the odds of choosing \(X_{2}\) over \(X_{3}\) are independent of the presence of \(X_{1}\) as an option, i.e. that the odds should not depend on whether this is a choice for the first or the second position

\[\frac{P_{1}(X_{2}>X_{j},j=1,3)}{P_{1}(X_{3}>X_{j},j=1,2)}=\frac{P_{2}(X_{2}>X_ {3})}{P_{2}(X_{3}>X_{2})}\]

With the location parameters from above, the Choice Axiom requires

\[\frac{\pi_{\theta}(x,y^{2})^{d_{1}^{2}}}{\pi_{\theta}(x,y^{3})^{d_{1}^{3}}}= \frac{\pi_{\theta}(x,y^{2})^{d_{2}^{2}}}{\pi_{\theta}(x,y^{3})^{d_{2}^{3}}}\]

or that \(\pi_{\theta}(x,y^{2})^{(d_{1}^{2}-d_{2}^{2})}=\pi_{\theta}(x,y^{3})^{(d_{1}^{ 2}-d_{2}^{3})}\). This holds for the default setting, \(d_{i}^{j}=1\), leading to Equation 4, but appears not to hold in general.

We find that the ground truth preference values can be introduced as preference distances in the binary comparison case, but that doing so in the more general case, while useful, may not satisfy the Axiom of Choice.

## Appendix B More details on Maple

### Data Construction

The source sentences in the training data of MAPLE are sampled from the test sets of WMT20 and WMT21. As mentioned in Section 4, four of the five translations are produced by VicunaMT. Considering that VicunaMT is already a strong MT system, often providing accurate translations free of mistakes, randomly selecting source sentences from WMT data could predominantly yield translations that are trivial for VicunaMT to translate, resulting in the collection of many uninformative samples with high human preference scores. To mitigate this, we prioritize source sentences that present difficulties for VicunaMT. Specifically, we use reference translations as a proxy to assess the quality of the model translations through COMET scores. We give priority to samples where the beam search output falls within a COMET score range of [75,85] and where there is a significant standard deviation in COMET scores among the four translations. Following these criteria, we select 1.1K samples for each translation direction. For the development set in MAPLE, we use monolingual data from News Crawl 2022. The sampling and selection process are the same as that of the training set, except that we do not have reference translations, instead, we use a strong commercial MT system to generate pseudo "reference" translations.

### Scoring Rubric

The annotators are asked to judge the translation on a scale of 1 to 6, following the guidelines outlined in the following scoring rubric. They can assign scores in increments of 0.2, allowing for more detailed assessments, such as 1.2, 1.4, and so 

* **Score it a 1** when the translation has nothing to do with the source; or when the translation has many unknown words; or when the translation looks like word salad.
* **Score it a 2** when you can understand why some of the words in the translation are there, but when the meaning of the source sentence is lost.
* **Score it a 3** when you understand why all or almost all the words in the translation are there and when some of the meaning of the source sentence are adequately transferred into the target language, but when the main meaning of the source sentence is lost.
* **Score it a 4** when the meaning of the source sentence is generally preserved, but when the translation is mechanical and possibly has vocabulary, grammatical, or date / numbering errors.
* **Score it a 5** when the meaning of the source sentence is fully preserved and the translation has no grammatical errors, but when the translation does not sound like the translation a native target language speaker would produce given the style and register of the source sentence.
* something a professional translator/interpreter would come up with when she understands well the context in which the source sentence was produced.

### Annotation UI

The UI shows the different translations in a blind and randomized order. All translations are scored simultaneously. A screenshot of the UI is shown in Figure 5.

### More Examples

Table 7 shows two additional examples in which the model's translation scores higher than the reference translation. This once again highlights the presence of noise in parallel datasets.

## Appendix C More implementation details

### Dataset statistics

The data statistics are presented in Table 8. We use different validation sets in different training stages because MAPLE contains a subset of the parallel data in WMT20/21.

### Prompt format

For each source sentence, we attach a MT instruction asking the LLM to generate the translation. The MT instructions come from a instruction pool based on the list of MT instructions released by Jiao et al. (2023)7. We list all 31 instructions in our instruction pool in Table 9. During training (in both SFT and PL stages), an instruction is randomly sampled from the instruction pool. During evaluation, the first instruction from Table 9 is always used. In addition to instructions, instruction-tuned models like Vicuna requires specific prompt formats. Table 10 presents a depiction of the conversion process from raw data points to the final model input.

Footnote 7: [https://github.com/wxjiao/ParroT](https://github.com/wxjiao/ParroT)

### Hyper-parameter search

Hyper-parameter search is done for \(\beta\in[0.0,0.05,0.1]\), and best values are selected according to the validation loss.

\begin{table}
\begin{tabular}{p{71.1pt} p{284.5pt}} \hline \hline Source & Other MPs criticised Twitter for allowing the tweets to remain visible. \\ \hline Reference translation & \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\##\#\#\#\#\#\#\#\#\#\##\#\#\#\#\#\#\##\#\#\#\#\#\##\#\#\#\#\##\#\#\#\#\##\#\#\#\##\#\#\#\#\##\#\#\#\##\#\#\#\##\#\#\#\##\#\#\##\#\#\##\#\#\#\##\#\#\##\#\#\##\#\#\##\#\#\##\#\#\##\#\#\##\#\#\##\#\#\##\#\#\##\#\#\##\#\#\##\#\##\#\#\##\#\#\##\#\#\##\#\#\##\#\##\#\#\##\#\##\#\#\##\#\##\#\#\##\#\#\##\#\##\#\##\#\#\##\#\##\#\##\#\#\##\#\##\#\##\#\#\##\#\##\#\##\#\##\#\##\#\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\#\##\#\##\#\##\#\##\#\##\#\##\#\##\#\#\##\#\##\#\##\#\##\#\#\##\#\##\#\##\#\#\##\#\##\#\#\##\#\##\#\#\##\#\##\#\#\##\#\#\##\#\#\##\#\##\#\#\##\#\#\##\#\#\##\#\#\##\#\#\##\#\#\##\#\#\##\#\#\##\#\#\##\#\#\#\##\#\#\#\##\#\#\#\#\##\#\#\#\##\#\#\#\##\#\#\#\#\#\##\#\#\#\#\#\#\# 



### Evaluation packages

We use the Unbabel/wmt22-comet-da model8 to compute the COMET scores and sacreBLEU9 for computing BLEU scores. The signature of the sacreBLEU package is nrefs:1, case::mixed, efff:no, tok:13a, smooth:exp, version:2.0.0 for all translation directions but en\(\rightarrow\)zh, in which we use tok:zh.

Footnote 8: [https://github.com/Unbabel/COMET](https://github.com/Unbabel/COMET)

Footnote 9: [https://github.com/mjpost/sacrebleu](https://github.com/mjpost/sacrebleu)

### Hardware specifications and runtime

All experiments are either run on a host with eight NVIDIA A100-40GB GPUs or with eight H100-80GB GPUs. Mixed precision with bfloat16 is used in both SFT and PL. Deepspeed10 zero-stage 3 is used when running PL with five preference samples. Each experiment runs no longer than 15 minutes on H100 GPUs.

Footnote 10: [https://github.com/microsoft/DeepSpeed](https://github.com/microsoft/DeepSpeed)

## Appendix D SFT results in BLEU score

We present model performance after SFT stage measured by BLEU score in Table 11. While the general trend remains consistent in comparison to the performance evaluated by COMET, there are some exceptions. For example, although VicunaMT still achieves the top average score on FLORES-200, it is outperformed by MistralMT (i.e., Mistral + SFT) on WMT22.

## Appendix E Model comparison in BLEU score

We present model performance measured by BLEU score in Table 12. In this case, there is no clear

Figure 5: User interface of translation assessment.



winner. Interestingly, VicunaMT+PL attains lower BLEU scores than VicunaMT on en\(\rightarrow\)de and zh\(\rightarrow\)en when evaluated on WMT22. However, both COMET score and our human evaluation in Table 4 show the opposite, highlighting that BLEU scores may less correlated to human judgement, as also noticed in Freitag et al. (2022).

## Appendix F Data reuse in BLEU score and Results on FLORES-200

We reuse MAPLE to enhance BLOOMZMT and MistralInstructMT (i.e., BLOOMZ and MistralInstruct after the SFT stage) and report model performance on WMT22 in BLEU score in Table 13. In addition, we evaluate MistralInstructMT on FLORES and present the results in Table 14.

 



\begin{table}
\begin{tabular}{l l



\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{6}{c}{WMT22} & \multicolumn{6}{c}{FLORES-200} \\ \cline{3-8}  & de\(\rightarrow\)en & en\(\rightarrow\)de & en\(\rightarrow\)zh & zh\(\rightarrow\)en & Avg. & \multicolumn{1}{c}{_COMET_} & \multicolumn{1}{c}{_COMET_} & \multicolumn{1}{c}{_COMET_} \\ \cline{2-9} Mistral-Ins.\({}^{\dagger}\) & 88.21 & 83.73 & 82.41 & 84.77 & 84.78 \\ +REF & 88.10 & 85.04 & 83.59 & 84.74 & 85.37 \\ +BEST & 88.41 & 84.55 & 83.46 & 84.94 & 85.34 \\ +PL & **85.56** & **84.98** & **83.86** & **85.34** & **85.67** \\ \hline \multicolumn{8}{c}{_BLEU_} & \multirow{6}{*}{39.16} \\ Mistral-Ins.\({}^{\dagger}\) & 39.16 & 27.79 & 29.77 & 23.10 & 29.96 \\ +REF & 38.10 & **28.39** & **31.24** & 23.09 & 30.21 \\ +BEST & 39.35 & 28.33 & 30.46 & 22.98 & 30.28 \\ +PL & **39.80** & 27.97 & 31.00 & **23.44** & **30.55** \\ \hline \end{tabular}
\end{table}
Table 14: Model performance on FLORES-200 in COMET and BLEU scores. Best results are in **bold**. \({}^{\dagger}\): SFT stage has already been applied to these models.

\begin{table}
\begin{tabular}{l l c c c c c c c c} \hline \hline System & \multicolumn{6}{c}{WMT22} & \multicolumn{6}{c}{FLORES-200} \\ \cline{2-9}  & de\(\rightarrow\)en & en\(\rightarrow\)de & en\(\rightarrow\)zh & zw\(\rightarrow\)en & Avg. & \multicol