Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation
Authors: yingfeng luo, Tong Zheng, Yongyu Mu, Bei Li, Qinghong Zhang, Yongqi Gao, Ziqiang Xu, Peinan Feng, Xiaoqian Liu, Tong Xiao, JingBo Zhu
Accepted at: acl25

Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation

Yingfeng Luo 1, Tong Zheng 1, Yongyu Mu 1, Bei Li 1, Qinghong Zhang 1, Yongqi Gao 1, Ziqiang Xu 1, Peinan Feng 1, Xiaoqian Liu 1, Tong Xiao 1,2 *, Jingbo Zhu 1,2,
1 School of Computer Science and Engineering, Northeastern University, Shenyang, China
2 NiuTrans Research, Shenyang, China
luoyingfeng.neu@outlook.com
{xiaotong,zhujingbo}@mail.neu.edu.cn

Abstract

The field of neural machine translation (NMT) has changed with the advent of large language models (LLMs). Much of the recent emphasis in natural language processing (NLP) has been on modeling machine translation and many other problems using a single pre-trained Transformer decoder, while encoder-decoder architectures, which were the standard in earlier NMT models, have received relatively less attention. In this paper, we explore translation models that are universal, efficient, and easy to optimize, by marrying the world of LLMs with the world of NMT. We apply LLMs to NMT encoding and leave the NMT decoder unchanged. We also develop methods for adapting LLMs to work better with the NMT decoder. Furthermore, we construct a new dataset involving multiple tasks to assess how well the machine translation system generalizes across various tasks. Evaluations on the WMT and our datasets show that results using our method match or surpass a range of baselines in terms of translation quality, but achieve 2.4 ~ 6.5× inference speedups and a 75% reduction in the memory footprint of the KV cache. It also demonstrates strong generalization across a variety of translation-related tasks. Our code, datasets, and models are publicly available at:

NiuTrans/LaMaTE

1 Introduction

The last decade has yielded remarkable breakthroughs in machine translation (MT) through the use of deep neural networks, scaled dramatically in both model parameters and training data. During this period, early methods, known as neural machine translation (NMT), were largely based on the encoder-decoder architecture (Sutskever et al., 2014; Bahdanau et al., 2015). In NMT, the machine translation problem is commonly treated as a sequence-to-sequence task, where the input sequence is first encoded into an intermediate representation, and the output sequence is then generated based on this representation, as illustrated in Figure 1 (a). Models of this kind are typically trained on bilingual text in a supervised manner, and their inference on modern GPUs is efficient. However, like their predecessors in past decades (e.g., statistical machine translation models), NMT models are generally developed for specific tasks, such as translation in a specific genre or domain.

Everything changed in NLP, with the success of LLMs through large-scale self-supervised learning

![Architectures of machine translation models. x = source-language token sequence, y_i = target-language token predicted as position i , y_{<i} = target-language tokens preceding i, and c = prompt.](page_1012_384_393_627.png)

Figure 1: Architectures of machine translation models. \( x \) = source-language token sequence, \( y_i \) = target-language token predicted as position \( i \) , \( y_{<i} \) = target-language tokens preceding \( i \), and \( c \) = prompt.
(Brown et al., 2020). In the LLM paradigm, the translation problem is framed as the token prediction problem in language modeling, as illustrated in Figure 1 (b). Such an approach greatly simplifies the modeling. We can now pre-train the LLM on large amounts of text via self-supervision so as to produce a single model that can be fine-tuned and prompted for various translation-related tasks, such as constrained translation and post-editing. Nevertheless, as a consequence of using large-scale neural networks, LLMs are computationally expensive and pose challenges for applications requiring low latency and a small memory footprint.

The fundamental change brought by LLMs has led most of the field to focus on replacing previous NLP systems with LLMs. However, machine translation researchers might still expect that a system could be efficient and easy to optimize, and, at the same time, could generalize across various sorts of tasks.

In this paper, we examine how the world of LLMs can be married with the world of NMT, thereby benefiting from both paradigms. One simple approach is to use an LLM as the encoder in NMT (call it LaMaTE — Large Language Models as Machine Translation Encoders). As both the encoder and decoder are essentially based on language models, we can view the whole system as a single language model. The decoupling of encoding and decoding from language modeling confers great modeling flexibility. A direct result of this is that we can employ a deep, complex encoder and a lightweight decoder, as illustrated in Figure 1 (c). This heterogeneous architecture is particularly well-suited for machine translation, where we can use a powerful encoder to understand the input text, while generating high-quality translations at a lower decoding cost.

In order to examine to what extent the model can generalize, we develop a new benchmark, called the Comprehensive Machine Translation benchmark (ComMT). It consists of several diverse tasks that evaluate different aspects of a translation model. While there have been datasets developed for fine-tuning LLMs for multiple translation tasks (Alves et al., 2024), our focus is on a broader range of application scenarios. We hope that such a benchmark can be adopted to provide a systematic evaluation of machine translation systems, and that this, in turn, will encourage practitioners to pay more attention to the issue of generalization when developing these systems.

We conduct extensive experiments and evaluate various models including NMT, LLMs, and our LaMaTE method. Our results show that the LaMaTE model achieves comparable or better performance than a range of baseline systems on several tasks, but runs 2.4 ~ 6.5 times faster and reduces the memory footprint of the KV cache by 75%. Evaluations on the ComMT dataset also demonstrate the strong generalization capabilities of the LaMaTE model, showing significant improvements over baseline systems. These results are not surprising, but intriguing, as findings in NMT remain applicable in the era of LLMs, for instance, scaling up the encoding network is still beneficial for machine translation tasks. This suggests an interesting direction for future work, where we could develop a powerful yet efficient system by using a strong model for language understanding and a lightweight model for language generation.

2 Related Work

Designing widely applicable models for MT has been an active area of NLP research for decades. Although MT models have evolved significantly over time, most of them still operate on an “analyze-then-generate” paradigm (Brown et al., 1993; Koehn et al., 2003; Bahdanau et al., 2015). For example, in statistical MT, the source-language sentence is parsed into either syntactic or non-syntactic forms, and the translation is generated by mapping these parsed forms to target-language constructions (Chiang, 2005). LLMs can broadly be categorized as following a similar design: they first compute the key-value cache for each input sequence, and then produce output tokens in a left-to-right manner based on this cache. From a modeling perspective, therefore, decoupling the encoding and decoding processes is a natural design choice for both traditional MT and LLMs.

In NLP, a large body of work has focused on pre-training and applying text encoders, such as the BERT series of models (Devlin et al., 2019), which are primarily designed to address language understanding problems. More recently, there have been attempts to use LLMs as text encoders, though these models are more commonly used for generating text. For example, BehnamGhader et al. (2024) and Muennighoff et al. (2024) fine-tuned LLMs for text encoding and showed that LLMs can produce high-quality representations of text in various embedding tasks. However, it is rare to see studies on
incorporating LLMs in the encoding for NMT or other language generation tasks.

In fact, the focus of research in MT has shifted. Recent studies in this field are now more concerned with the adaptation of LLMs, rather than the improvement of model architectures. One strand of research aims to enable LLMs to translate via prompting techniques, including designing better prompts (Zhang et al., 2023a; Zhu et al., 2024), introducing translation demonstrations (Mu et al., 2023; Chitale et al., 2024), and using chain-of-thought reasoning (Lu et al., 2024). Another strand focuses on developing MT-specific LLMs via fine-tuning (Xu et al., 2023; Yang et al., 2023b; Alves et al., 2024; Guo et al., 2024; Zheng et al., 2025); we have followed this approach, but with a focus on improving the model architecture, as well as developing a new benchmark for evaluating universal MT models.

This work is also related to efficient methods for LLMs. For example, one can compress the model using quantization and pruning techniques (Xiao et al., 2023; Ma et al., 2023), and speed-up inference using speculative decoding algorithms (Leviathan et al., 2023; Kim et al., 2023). But these methods and ours are in no way contradictory. Since our model follows standard encoding (or prefilling) and decoding frameworks, it can be easily combined with various efficient methods to further improve efficiency.

3 LaMaTE

In this section, we introduce LaMaTE and its training method.

3.1 Model Architecture

We begin by outlining the basic concepts and notation needed for our description. There are three networks that we consider here.

• NMT Encoder Enc(x). It is a standard Transformer encoder, consisting of an embedding layer and a number of stacked Transformer layers. The output of the NMT encoder is a bidirectional representation of the input token sequence x, denoted by \( \mathbf{H}^1 \).

• NMT Decoder Dec(H, y). It shares the same architecture as the NMT encoder, with an additional cross-attention sublayer added in each Transformer layer. The NMT decoder accepts two inputs — one from the encoder for cross-attention, and another as the regular sequential input. In the self-attention mechanism, each position can only attend to the preceding positions. So its output representation is unidirectional. Typically, a Softmax layer is added on top of the decoder to generate distributions of tokens.

• LLM Decoder Dec(__, [c, x, y]). It operates on token sequences only, without the need for input from the encoder. Thus, the cross-attention sublayers are removed, and the first parameter of the above function is left blank. Here [c, x, y] denotes the concatenation of the prompt c, the input sequence x, and the translation y.

For NMT models, we can simply connect the NMT encoder and decoder together. The probability of token prediction can be expressed as

\[
\begin{align*}
\Pr(y_i|x, y_{<i}) &= \operatorname{Softmax}(\mathbf{WS})_i \\
\mathbf{S} &= \operatorname{Dec}(\mathbf{H}, y_{<i}) \\
\mathbf{H} &= \operatorname{Enc}(x)
\end{align*}
\]

where \( y_i \) denotes the target-language token at position i, \( y_{<i} \) denotes the target-language tokens that precede position i, and \( \operatorname{Softmax}(\mathbf{WS})_i \) denotes the Softmax function that computes the distribution of tokens at position i. \( \mathbf{WS} \) maps the decoder output \( \mathbf{S} \) to a representation space of the vocabulary size using the linear mapping matrix \( \mathbf{W} \).

The LaMaTE model follows this encode-decode architecture, but with an LLM decoder replacing the NMT encoder, given by

\[
\begin{align*}
\mathbf{S} &= \operatorname{Dec}_\phi(\mathbf{H}', y_{<i}) \\
\mathbf{H}' &= \operatorname{Dec}_\theta(__, [c, x])
\end{align*}
\]

Here we use the subscripts \( \phi \) and \( \theta \) to denote the parameters for the NMT decoder and the LLM decoder, respectively.

The function \( \operatorname{Dec}_\theta(__, [c, x]) \) serves as a unidirectional encoder. Although it is generally thought that bidirectional encoders can make better use of contextual information, our experiments in Section 6.4 demonstrate that using such unidirectional encoders based on LLMs is also very effective. Given that both the encoder and decoder are unidirectional, we can roughly think of the whole model as a hybrid language model: a large, powerful
model is used for prefilling, and a small, efficient model is used for decoding.

The use of pre-trained LLMs for encoding allows the model to have a stronger understanding of the input sequence, which has been found to be very beneficial in NMT systems (Dou et al., 2018; Wang et al., 2019). As another bonus, one can easily adapt the model via fine-tuning and prompting, thereby deploying a single such system for many different tasks. For example, by giving an appropriate prompt c, we can guide the system to produce outputs following specific needs or contexts. This gives LaMaTE a big advantage over traditional NMT models.

One problem with the model described in Eqs. (4-5) is that the LLM decoder may have a larger hidden size than that of the NMT decoder. In this case, we add an adaptor to the output of the LLM decoder to reduce its dimensionality, so that it matches the NMT decoder. We can then redefine \( \mathbf{H}' \) as

\[
\mathbf{H}' = F_\omega(\mathrm{Dec}_\theta(__, [\mathbf{c}, \mathbf{x}]))
\]

where \( F_\omega(\cdot) \) is the adaptor with the parameters \( \omega \).

3.2 Adaptor
One simple form of adaptors is a linear mapping of the output of the LLM decoder, that is, we linearly transform representations from \( \mathrm{Dec}_\theta(\cdot) \) to lower-dimensional representations required by \( \mathrm{Dec}_\phi(\cdot) \). But we find that this method falls short of our expectations because LLMs are typically too big that simply transforming their output representations into lower-dimensional ones makes it difficult for untrained smaller models to use them directly.

Instead, we design an adaptor that makes better use of the LLM outputs for the NMT decoder. It involves three components. The complete model architecture is illustrated in Figure 2

• Instead of considering only the output of the final layer of the LLM, we fuse the outputs of the intermediate layers to form a finer-grained output of the LLM decoder. Such methods have been found to be very beneficial to deep-to-shallow architectures in NMT (Wang et al., 2019; Yang et al., 2023a). To do this, we divide the LLM layers into K groups, and fuse their outputs in the following form

\[
\mathbf{H}_{\text{fuse}} = \text{LayerNorm}\left( \frac{1}{K} \sum_{k=1}^K w_k g_k \right)
\]

where LayerNorm(\(\cdot\)) is the layer normalization function, \( g_k \) is the output of the k-th layer group, and \( w_k \) is the learnable weight for \( g_k \). We use the last layer’s hidden states of each group as that group’s hidden states.

• We place a 2-layer MLP after layer fusion to reduce the hidden size of \( \mathbf{H}_{\text{fuse}} \) (denoted by \( d_1 \)) to the hidden size of the NMT decoder (denoted by \( d_2 \)). This network is given by

\[
\mathbf{H}_{\text{mlp}} = \text{GELU}(\mathbf{H}_{\text{fuse}} \mathbf{W}_1) \mathbf{W}_2
\]

where GELU(\(\cdot\)) is the activation function, and \( \mathbf{W}_1 \in \mathbb{R}^{d_1 \times d_2} \) and \( \mathbf{W}_2 \in \mathbb{R}^{d_2 \times d_2} \) are the linear mapping matrices.

• It is also possible to learn bidirectional representations from unidirectional representations. A common method is to incorporate some Transformer encoder layers. Thus the

![The architecture of LaMaTE, where the Adaptor consists of three components: Fusion combines the representations of layer groups g_k, MLP reduces the representations's dimensionality, and EncStack learns bidirectional representations. The training process consists of two stages: the first stage trains the Adaptor and Decoder, and the second stage trains all model parameters.](page_176_120_1096_312.png)

Figure 2: The architecture of LaMaTE, where the Adaptor consists of three components: Fusion combines the representations of layer groups \( g_k \), MLP reduces the representations's dimensionality, and EncStack learns bidirectional representations. The training process consists of two stages: the first stage trains the Adaptor and Decoder, and the second stage trains all model parameters.
self-attention models can help generate bidirectional representations. The final output of the adaptor is then given by

\[
\mathbf{H}' = \mathrm{EncStack}(\mathbf{H}_{\mathrm{mlp}})
\]

where \( \mathrm{EncStack}(\cdot) \) is a stack of Transformer encoder layers. Note that EneStack(\(\cdot\)) consists of only a few layers, and thus adds very small computational overhead. This step of bidirectional representation learning is optional, and one can choose whether to adopt it in practice.

3.3 Model Training
The LaMaTE model has three sets of parameters: \( \theta \) for the LLM decoder, \( \phi \) for the NMT decoder, and \( \omega \) for the adaptor. Here, \( \theta \) is initialized with the pre-trained parameters of the LLM, while \( \phi \) and \( \omega \) need to be trained from scratch.

Optimizing these parameters on bitext seems straightforward, but poses practical challenges. Since fine-tuning pre-trained LLMs is costly, using less labeled data is generally favorable. However, we found that this approach can lead to inadequate learning of the new parameters \( \phi \) and \( \omega \). On the other hand, extensive fine-tuning may cause the LLM to forget its original knowledge encoded in \( \theta \). Similar findings have been reported in the literature (Xu et al., 2023).

Here, we present a two-stage training method that performs more efficient learning for different types of parameters. In the first stage, we freeze \( \theta \) and pre-train both \( \phi \) and \( \omega \). The task of pre-training is a standard translation task. We train the model to translate source-language sequences to corresponding target-language sequences. Therefore, the adaptor and the NMT decoder can learn to map from source-language representations to translations. Since the parameters of the LLM decoder are frozen, this process requires only the forward pass of this large network, without the need for the backward pass. It is thus less computationally expensive than LLM fine-tuning, making it possible to scale up the pre-training to large datasets.

The second stage is a fine-tuning stage, where \( \theta \), \( \phi \), and \( \omega \) are fine-tuned together on various tasks. For each task, as with instruction fine-tuning for LLMs, we provide LaMaTE with task-specific instructions, inputs and outputs, and optimize all the parameters end-to-end. In this way, the model is adapted to follow these instructions and can be deployed as a single, universal model to handle a variety of translation-related problems. We use our ComMT dataset for fine-tuning in this work.

4 ComMT
Many translation-related tasks, such as document-level translation, rely on both understanding complex inputs and generating coherent, contextually appropriate outputs, which are not captured by commonly used sentence-level translation tasks. To generalize LaMaTE to diverse tasks and evaluate it on these tasks, we developed a new evaluation benchmark called ComMT, constructed from existing publicly available translation datasets. It comprises five tasks:

• General Translation. This is a standard sentence in, sentence out task.

• Document-level Translation. This task extends translation from sentence-level to document-level.

• Domain Translation. This task focuses on domain-specific translation.

• Terminology-constrained Translation. In this task, the system is required to produce translations that follow the given terminology translation requirements.

• Automatic Post-editing. This task aims to automatically correct errors in translations.

We collected data mainly from public resources, driven by two goals: (i) to find as many data sources as possible in order to increase data diversity, and (ii) to collect as much high-quality, manually annotated translation data as possible. ComMT is multilingual and supports four languages: German, Czech, Russian, and Chinese. The reader can refer to Appendix Section B for more details of ComMT.

Ultimately, we created a training set of 239k samples, as shown in Figure 7, and the test set statistics illustrated in Table 5. Overall, ComMT exhibits significant diversity and broad applicability, providing a well-curated data resource for developing and evaluating universal translation models.

5 Experimental Setup
5.1 Data and Evaluation Metrics
We trained our model on multilingual translations between English (En) and four languages: German (De), Czech (Cs), Russian (Ru), and Chinese
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">General Trans</th>
    <th colspan="2">Domain Trans</th>
    <th colspan="2">Doc-level Trans</th>
    <th colspan="2">Terminology-con Trans</th>
    <th colspan="3">Automatic Post-edition</th>
  </tr>
  <tr>
    <th>COMET</th><th>BLEU</th>
    <th>COMET</th><th>BLEU</th>
    <th>COMET</th><th>BLEU</th>
    <th>COMET</th><th>BLEU</th>
    <th>TSR↑</th><th>COMET</th><th>BLEU</th><th>HTER↓</th>
  </tr>
  <tr>
    <th colspan="13">Decoder-only Models</th>
  </tr>
  <tr>
    <td>ALMA-7B*</td><td><b>83.23</b></td><td>31.49</td><td>83.93</td><td>31.94</td><td>82.03</td><td>18.59</td><td>81.84</td><td>24.89</td><td>43.20</td><td>85.25</td><td>40.91</td><td>54.42</td>
  </tr>
  <tr>
    <td>BigTranslate*</td><td>77.84</td><td>22.04</td><td>77.53</td><td>21.99</td><td>72.36</td><td>9.18</td><td>77.29</td><td>18.56</td><td>33.10</td><td>78.84</td><td>29.82</td><td>69.04</td>
  </tr>
  <tr>
    <td>Aya-23-8B</td><td>82.72</td><td>32.11</td><td>84.07</td><td>33.67</td><td><b>84.98</b></td><td><b>32.19</b></td><td>81.47</td><td>31.14</td><td>74.15</td><td>87.68</td><td>67.92</td><td>36.23</td>
  </tr>
  <tr>
    <td>TowerInstruct-7B</td><td><b>83.12</b></td><td><b>32.75</b></td><td>83.65</td><td>33.25</td><td>81.61</td><td>28.22</td><td>82.57</td><td>31.84</td><td>76.98</td><td><b>88.91</b></td><td><b>77.41</b></td><td><b>23.49</b></td>
  </tr>
  <tr>
    <td>Bayling-13B</td><td>74.81</td><td>21.34</td><td>76.50</td><td>24.07</td><td>74.10</td><td>19.46</td><td>77.01</td><td>25.08</td><td>71.94</td><td>73.90</td><td>29.67</td><td>114.80</td>
  </tr>
  <tr>
    <td>Llama3-8B-Base</td><td>79.03</td><td>26.89</td><td>81.21</td><td>28.29</td><td>80.44</td><td>26.77</td><td>81.81</td><td>29.84</td><td>79.20</td><td>86.26</td><td>61.43</td><td>38.93</td>
  </tr>
  <tr>
    <td>Llama3-8B-Inst</td><td>73.56</td><td>23.20</td><td>73.04</td><td>22.69</td><td>81.79</td><td>27.22</td><td>71.36</td><td>24.41</td><td>86.46</td><td>68.05</td><td>25.70</td><td>101.85</td>
  </tr>
  <tr>
    <td>Llama3-8B-SFT</td><td>82.41</td><td>32.42</td><td><b>84.60</b></td><td><b>36.92</b></td><td>83.79</td><td><b>31.75</b></td><td><b>84.01</b></td><td><b>36.77</b></td><td><b>87.52</b></td><td>88.57</td><td><b>70.13</b></td><td><b>31.57</b></td>
  </tr>
  <tr>
    <th colspan="13">Encoder-Decoder Models</th>
  </tr>
  <tr>
    <td>NLLB-3.3B*</td><td>81.30</td><td>31.63</td><td>81.38</td><td>32.66</td><td>73.60</td><td>11.72</td><td>80.21</td><td>26.32</td><td>41.25</td><td>84.97</td><td>46.42</td><td>52.42</td>
  </tr>
  <tr>
    <td>NMT-8-8*</td><td>79.70</td><td>30.08</td><td>80.63</td><td>32.76</td><td>73.98</td><td>10.11</td><td>77.03</td><td>25.54</td><td>40.56</td><td>84.50</td><td>47.52</td><td>50.21</td>
  </tr>
  <tr>
    <td>NMT-40-8*</td><td>80.89</td><td>32.05</td><td>81.50</td><td>34.06</td><td>74.74</td><td>12.39</td><td>77.19</td><td>25.80</td><td>41.15</td><td>85.10</td><td>47.57</td><td>50.26</td>
  </tr>
  <tr>
    <td>mT5-Large</td><td>81.26</td><td>29.34</td><td>82.06</td><td>30.26</td><td>77.69</td><td>14.43</td><td>81.80</td><td>30.34</td><td>76.93</td><td>85.88</td><td>62.77</td><td>45.10</td>
  </tr>
  <tr>
    <td>LaMaTE (Ours)</td><td><b>82.32</b></td><td><b>33.85</b></td><td><b>84.69</b></td><td><b>37.49</b></td><td><b>84.34</b></td><td><b>31.69</b></td><td><b>83.35</b></td><td><b>34.76</b></td><td><b>75.64</b></td><td><b>88.60</b></td><td><b>69.10</b></td><td><b>33.42</b></td>
  </tr>
</table>

Table 1: Performance on ComMT test set. Bold numbers represent the highest scores in each category, while underlined numbers indicate the second highest scores. Models marked with "*" cannot handle additional inputs for the terminology-constrained translation and automatic post-editing tasks, we only use the source sequence as input.

(Zh), resulting in a total of eight translation directions. The parallel corpus for training stage 1 was sourced from WMT2023, as detailed in Table 6. We sampled 10M bilingual sentence pairs for each language, yielding a total of 40M pairs for training stage 1. The training stage 2 utilizes data from the ComMT training set with 239k samples.

We conducted tests on ComMT and evaluated the model’s translation performance across all tasks using COMET (wmt22-comet-da) (Rei et al., 2020) and SacreBLEU (Post, 2018). Additionally, we used Terminology Success Rate (TSR) for the terminology-constrained translation task and Human Translation Edit Rate (HTER) for the automatic post-editing task.

5.2 Training Setup

Our multilingual translation experiments necessitate a language model with multilingual capabilities for better encoding. We evaluated various popular multilingual LLMs (see Appendix C.2 for details) and selected Llama3-8B-base (Dubey et al., 2024) as our encoder, which has a model dimension of 4096 and consists of 32 layers.

For the decoder, we set the dimension to 1024 with 8 layers, which also applies to the EncStack of the adaptor, with the adaptor and decoder together introducing fewer than 500M parameters. Hyperparameters for training are detailed in Table 7.

5.3 Models for Comparison

We create two categories of models for comparison:

Decoder-only Models. This includes: BigTranslate (Yang et al., 2023b), Bayling-13B (Zhang et al., 2023b), Aya-23-8B (Üstün et al., 2024), ALMA-7B (Xu et al., 2023) and TowerInstruct-7B (Alves et al., 2024). Also included are variants of Llama3-8B: Llama3-8B-Base (3-shot in-context learning), Llama3-8B-Inst (general instruction-tuned), and Llama3-8B-SFT (fine-tuned on ComMT).

Encoder-Decoder Models. This includes: (i) NLLB-3.3B (Costa-jussà et al., 2022): a comprehensive multilingual model. (ii) Traditional encoder-decoder models: NMT-40-8 (40 encoder layers, 8 decoder layers) and NMT-8-8 (8 encoder layers, 8 decoder layers), both with a model dimension of 1024, trained from scratch on 161M bilingual sentences from combined parallel data of four languages. (iii) Fine-tuned encoder-decoder models: mT5-Large (1.2B) (Xue et al., 2021), aligned with our method’s data settings.

6 Results and Analyses

6.1 Results

The averaged performance across all directions is shown in Table 1, with more detailed results provided in Table 14.

Comparison with Encoder-Decoder Models. In comparison to encoder-decoder models, LaMaTE
Figure 3: Comparison of decoder-only (Llama3-8B and TowerInstruct-7B) and encoder-decoder (mT5-large and LaMaTE) models on off-target rate (OTR), unaligned source words (USW), and unaligned target words (UTW).

<table>
  <tr>
    <th>Models</th>
    <th>HRL</th>
    <th>MRL</th>
    <th>LRL</th>
    <th>XRL</th>
  </tr>
  <tr>
    <td>NMT-40-8</td>
    <td>30.13</td>
    <td>30.28</td>
    <td>21.02</td>
    <td>16.59</td>
  </tr>
  <tr>
    <td>mT5-large</td>
    <td>29.12</td>
    <td>30.14</td>
    <td><b>27.46</b></td>
    <td><b>24.64</b></td>
  </tr>
  <tr>
    <td>LaMaTE (Ours)</td>
    <td><b>32.11</b></td>
    <td><b>32.13</b></td>
    <td>26.20</td>
    <td>21.70</td>
  </tr>
</table>

Table 2: BLEU performance evaluated on the FLORES-200 devtest set, with models trained on OPUS-100 data. HRL, MRL, LRL, and XRL represent high, medium, low, and very low-resource languages, respectively.

demonstrates stronger performance, surpassing NLLB-3.3B, NMT, and mT5-Large across all evaluated tasks. This result highlights the potential of LLMs as high-capacity encoders, capable of producing more expressive and informative representations. These enhanced representations provide a solid foundation for the decoder, ultimately improving translation quality. Moreover, LaMaTE’s consistent performance across multiple tasks underscores its strong generalization capability.

Comparison with Fine-Tuned LLMs. When compared with fine-tuned LLMs, LaMaTE achieves comparable overall performance to Llama3-8B-SFT. While Llama3-8B-SFT holds a slight advantage in the terminology-constrained translation task with higher TSF scores, there is no significant difference in other tasks. Additionally, LaMaTE outperforms earlier fine-tuned LLMs like Bayling-13B and remains competitive with more advanced models such as TowerInstruct-7B. This further demonstrates that LLMs can serve as effective MT encoders, offering an effective and computationally efficient alternative to the approach of fine-tuning LLMs as direct generators.

Misalignment Evaluation. To further investigate translation quality, we analyze misalignment issues commonly observed in LLM-based generation (Zhang et al., 2024; Zeng et al., 2024), where models may generate output that diverges from the source text. We compare decoder-only models (Llama3-8B and TowerInstruct-7B) and encoder-decoder (mT5-large and LaMaTE) across three metrics: off-target rate, unaligned source words, and unaligned target words in En → X direction. As shown in Figure 3, encoder-decoder models consistently exhibit lower scores across all three dimensions, indicating they reduce misalignment compared to decoder-only models. LaMaTE follows the encoder-decoder paradigm, which also helps mitigate misalignment issues as an added bonus. We speculate that this improvement can be attributed to the cross-attention mechanism in the NMT decoder, which allows direct token-wise interaction between the target and the source sequence, thereby enhancing source-target alignment. In contrast, LLMs rely on a single attention mechanism to jointly process the concatenated source and target text, which could potentially cause a loss of attention focus (Zhang et al., 2024).

Large-Scale Multilingual Translation. To further validate whether our method can benefit a broader range of languages, we conducted extensive multilingual translation directions (99 languages), with the detailed description in Appendix D.1. As shown in Table 2, LaMaTE significantly outperforms the NMT-40-8 model across all resource levels and surpasses the mT5-large model in high and medium language resources. This highlights the effectiveness of leveraging LLMs as encoders for multilingual translation, as their rich representations can boost performance. However, our model underperforms compared to mT5-large in low and very low-resource languages, likely due to limited pre-training data for these languages in the Llama3-base model. Using a model with stronger multilingual capabilities as the encoder could further improve performance.
Figure 4: Comparison of efficiency: The left chart displays the decoding speedup ratio of LaMaTE versus Llama3-8B under varying source sequence lengths and batch sizes, and the right chart shows the theoretical KV cache size factor for each model.

6.2 Efficiency Analysis

Figure 4 illustrates the efficiency comparisons between LaMaTE and other models, with their configuration details are provided in Table 10. The results highlight two key improvements: **decoding speedup** and **memory efficiency**.

The left chart in Figure 4 demonstrates that as the source sequence length and batch size increase, LaMaTE achieves a **2.4× to 6.5× speedup** over Llama3-8B (see Table 13 for details). Furthermore, as shown in the right chart, LaMaTE significantly **reduces KV cache memory by 75%** compared to Llama3-8B. This reduction in memory usage enhances scalability, enabling larger batch sizes and longer source sequences. For instance, at an 80G memory budget, Llama3-8B encounters out-of-memory issues at a batch size of 24 for source lengths of 300-400, whereas LaMaTE still works normally. This improvement is particularly beneficial for large-scale inference, where optimizing throughput is crucial for real-world deployment. Overall, LaMaTE significantly enhances computational efficiency without compromising translation quality, demonstrating its effectiveness in balancing performance and compute resources.

6.3 Depth vs Performance

To analyze how the depth of the EncStack and decoder affects performance and efficiency, we first fix the decoder depth and vary the EncStack depth, then fix the EncStack depth and adjust the decoder depth. We evaluate translation performance in both En → X and X → En directions and measure decoding speed, as shown in Figure 5.

The left panel presents the effect of increasing the number of EncStack layers. The results show that adding more EncStack layers leads to modest improvements in both translation directions, but the gains are relatively small compared to variations in the decoder. Additionally, since the EncStack operates only on the encoder side, its impact on decoding efficiency is minimal.

The right panel illustrates the effect of varying decoder depth. Unlike the encoder, increasing decoder layers significantly influences both translation performance and efficiency. While deeper decoders consistently improve translation quality, particularly in the En → X direction, they also introduce a notable trade-off: decoding speed drops sharply as depth increases, with a 39% reduction when moving from 8 to 16 layers. This suggests that scaling the encoder offers a more effective trade-off between performance gains and computational efficiency. Alternatively, the decoder can be scaled using the Mixture of Experts (MoE) technique, which enables capacity expansion without an increase in inference cost. We leave further exploration of these strategies for future work.

6.4 Ablation Study

We present ablation studies in Table 3. First, training the model with only stage 1 (W/o S2) results in a substantial performance drop. Incorporating stage 2 while freezing LLM parameters yields no-
<table>
  <tr>
    <th>Models</th>
    <th>COMET</th>
    <th>BLEU</th>
  </tr>
  <tr>
    <td>LaMaTE (Ours)</td>
    <td>82.32</td>
    <td>33.85</td>
  </tr>
  <tr>
    <td>W/o S2</td>
    <td>80.07\(^{(-2.25)}\)</td>
    <td>29.67\(^{(-4.18)}\)</td>
  </tr>
  <tr>
    <td>W/ S2 & Frozen LLM</td>
    <td>82.16\(^{(-0.16)}\)</td>
    <td>33.73\(^{(-0.12)}\)</td>
  </tr>
  <tr>
    <td>W/o Layer Fusion</td>
    <td>82.08\(^{(-0.24)}\)</td>
    <td>33.14\(^{(-0.71)}\)</td>
  </tr>
  <tr>
    <td>W/o EncStack</td>
    <td>82.02\(^{(-0.30)}\)</td>
    <td>33.32\(^{(-0.53)}\)</td>
  </tr>
  <tr>
    <td>Concat Decoder</td>
    <td>81.52\(^{(-0.80)}\)</td>
    <td>31.51\(^{(-2.34)}\)</td>
  </tr>
  <tr>
    <td>Prefix Decoder</td>
    <td>81.66\(^{(-0.66)}\)</td>
    <td>32.38\(^{(-1.47)}\)</td>
  </tr>
</table>

Table 3: Ablation studies on training methods, adaptor design, and decoder variants. The numbers in the bottom right represent the performance gap relative to the complete model (LaMaTE).

ticeable improvements, but still falls short of LaMaTE’s performance. This underscores the importance of both high-quality data and LLM parameter tuning in maximizing model performance.

For the adaptor, removing the layer fusion leads to a performance drop, indicating that fusing intermediate LLM layers provides a more informative representation than relying solely on the final layer. Similarly, removing EncStack results in a slight degradation. These findings confirm that the adaptor enhances the native representations of LLMs, rendering them more suitable for the NMT decoder.

Besides, we explore two decoder variants—Concat Decoder and Prefix Decoder—both of which remove the cross-attention layer in the standard decoder. For a detailed description of these variants, see Appendix D.3. As shown in the results, the standard decoder proves superior to alternative designs. Together with the observations from Figure 3, we argue that maintaining cross-attention in the decoder is particularly beneficial for translation tasks that demand strong alignment.

7 Conclusion

In this paper, we explore the connection between the two worlds of LLMs and NMT, presenting LaMaTE—a method that leverages LLMs as MT encoders and pairs them with lightweight decoders. We design an adaptor better to align LLM’s representations for the decoder and propose a two-stage training strategy to develop a universal translation model. Additionally, we introduce ComMT, a new dataset suite encompassing diverse translation-related tasks, facilitating the development and evaluation of universal translation models. Experiments on ComMT demonstrate LaMaTE’s strong performance and generalization ability, while significantly improving computational efficiency—achieving 2.4× to 6.5× faster decoding speeds and reducing KV cache memory usage by 75%. We hope this study will provide valuable insights and inspire further exploration into optimizing LLMs and expanding their role in NLP tasks.

Limitations

Our approach requires pre-training the decoder’s parameters using large-scale bilingual data in the first stage, as the decoder is randomly initialized. This process may not be the most efficient. Future work could explore initializing these parameters from the encoder or directly leveraging a small pre-trained language model to reduce the state gap between the encoder and decoder. Additionally, since our decoder is lightweight, it may become a bottleneck when generating translations into many target languages, so expanding its capacity is essential for better performance. A more effective strategy might involve scaling the decoder’s capacity with Mixture-of-Experts (MoE) instead of adding more layers, thereby boosting performance without compromising efficiency.

Acknowledgments

This work was supported in part by the National Science Foundation of China (Nos. 62276056 and U24A20334), the Fundamental Research Funds for the Central Universities, the Yunnan Fundamental Research Projects (No. 202401BC070021), and the Program of Introducing Talents of Discipline to Universities, Plan 111 (No.B16009).

References

David Ifeoluwa Adelani, Md Mahfuz Ibn Alam, Antonios Anastasopoulos, Akshita Bhagia, Marta R. Costa-jussà, Jesse Dodge, Fahim Faisal, Christian Federmann, Natalia Fedorova, Francisco Guzmán, Sergey Koshchev, Jean Maillard, Vukosi Marivate, Jonathan Mbuya, Alexandre Mourachko, Safiyyah Saleem, Holger Schwenk, and Guillaume Wenzek. 2022. Findings of the wmt’22 shared task on large-scale machine translation evaluation for african languages. In WMT, pages 773–800. Association for Computational Linguistics.

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. GQA: training generalized multi-query transformer models from multi-head checkpoints. In EMNLP, pages 4895–4901. Association for Computational Linguistics.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. The falcon series of open language models. CoRR, abs/2311.16867.

Duarte M. Alves, José Pombal, Nuno Miguel Guerreiro, Pedro Henrique Martins, João Alves, M. Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins. 2024. Tower: An open multilingual large language model for translation-related tasks. CoRR, abs/2402.17733.

Antonios Anastasopoulos, Alessandro Cattelan, Zi-Yi Dou, Marcello Federico, Christian Federmann, Dmitriy Genzel, Francisco Guzmán, Junjie Hu, Macduff Hughes, Philipp Koehn, Rosie Lazar, William Lewis, Graham Neubig, Mengmeng Niu, Alp Oktem, Eric Paquin, Grace Tang, and Sylwia Tur. 2020. TICO-19: the translation initiative for covid-19. In Proceedings of the 1st Workshop on NLP for COVID-19@ EMNLP 2020, Online, December 2020. Association for Computational Linguistics.

Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. On the cross-lingual transferability of monolingual representations. CoRR, abs/1910.11856.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.

Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. Llm2vec: Large language models are secretly powerful text encoders. CoRR, abs/2404.05961.

Peter F Brown, Stephen A Della Pietra, Vincent J Della Pietra, and Robert L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2):263–311.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurIPS.

Kaiyan Chang, Songcheng Xu, Chenglong Wang, Yingfeng Luo, Xiaoqian Liu, Tong Xiao, and Jingbo Zhu. 2024. Efficient prompting methods for large language models: A survey. arXiv preprint arXiv:2404.01077.

David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of the 43rd annual meeting of the association for computational linguistics (acl’05), pages 263–270.

Pranjal A. Chitale, Jay P. Gala, and Raj Dabre. 2024. An empirical study of in-context learning in llms for machine translation. In ACL (Findings), pages 7384–7406. Association for Computational Linguistics.

Shamil Chollampatt, Raymond Hendy Susanto, Liling Tan, and Ewa Szymanska. 2020. Can automatic post-editing improve nmt?. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 2736–2746. Association for Computational Linguistics.

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

Marta R. Costa-jussà, James Cross, Onur Celebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Y. Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejía Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedant Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. CoRR, abs/2207.04672.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), pages 4171–4186. Association for Computational Linguistics.

Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In NeurIPS, pages 13042–13054.

Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, pages 2112–2128. Association for Computational Linguistics.

Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Shuming Shi, and Tong Zhang. 2018. Exploiting deep representations
for neural machine translation. In EMNLP, pages 4253–4262. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aoob Yang, Archi Mitra, Archie Sravanakumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolidakis, Damien Alionsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esibou, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vraneš, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasudevanwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. 2024. The llama 3 herd of models. CoRR, abs/2407.21783.

Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70–74. Association for Computational Linguistics.

Christian Federmann, Tom Kocmi, and Ying Xin. 2022. NTREX-128 – news test references for MT evaluation of 128 languages. In Proceedings of the First Workshop on Scaling Up Multilingual Evaluation, pages 21–24. Online. Association for Computational Linguistics.

Christian Federmann and William D. Lewis. 2017. The microsoft speech language translation (MSLT) corpus for chinese and japanese: Conversational test data for machine translation and speech recognition. In Proceedings of Machine Translation Summit XVI, Volume 1: Research Track, MTSummit 2017, September 18-22, 2017, Nagoya, Aichi, Japan, pages 72–85.

Marina Fomicheva, Shuo Sun, Erick Fonseca, Frédéric Blain, Vishrav Chaudhary, Francisco Guzmán, Nina Lopatina, Lucia Specia, and André F. T. Martins. 2020. MLQE-PE: A multilingual quality estimation and post-editing dataset. arXiv preprint arXiv:2010.04480.

Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Statistical power and translationese in machine translation evaluation. In EMNLP (1), pages 72–81. Association for Computational Linguistics.

Nuno M. Guerreiro, Elena Voita, and André Martins. 2023. Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia. Association for Computational Linguistics.

Jiaxin Guo, Hao Yang, Zongyao Li, Daimeng Wei, Hengchao Shang, and Xiaoyu Chen. 2024. A novel paradigm boosting translation capabilities of large language models. CoRR, abs/2403.11430.

Kazuma Hashimoto, Raffaella Buschiazzo, James Bradbury, Teresa Marshall, Richard Socher, and Caiming Xiong. 2019. A high-quality multilingual dataset for structured documentation translation. In Proceedings of the Fourth Conference on Machine Translation, WMT 2019, Florence, Italy, August 1-2, 2019 - Volume 1: Research Papers, pages 116–127. Association for Computational Linguistics.

Jie He, Tao Wang, Deyi Xiong, and Qun Liu. 2020. The box is in the pen: Evaluating commonsense reasoning in neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 3662–3672. Association for Computational Linguistics.

Julian Hitschler, Shigehiko Schamoni, and Stefan Riezler. 2016. Multimodal pivots for image caption translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.

Zhiyuan Hu, Yuliang Liu, Jinman Zhao, Suyuchen Wang, Yan Wang, Wei Shen, Qing Gu, Anh Tuan Luu, See-Kiong Ng, Zhiwei Jiang, et al. 2024. Longrecipe: Recipe for efficient long context generalization in large language models. arXiv preprint arXiv:2409.00509.

Yuchen Eleanor Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Ryan Cotterell, and Mrinmaya Sachan. 2023. Discourse centric evaluation of machine translation with a densely annotated parallel corpus. In Proceedings of the 2023 Conference of the Association for Computational Linguistics: Human Language Technologies, pages 1550–1565, Toronto, Canada. Association for Computational Linguistics.

Wenxiang Jiao, Jen-tse Huang, Wenxuan Wang, Zhiwei He, Tian Liang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023. Parrot: Translating during chat using large language models tuned with human translation and feedback. In EMNLP (Findings), pages 15009–15020. Association for Computational Linguistics.
Linghao Jin, Jacqueline He, Jonathan May, and Xuezhe Ma. 2023a. Challenges in context-aware neural machine translation. In EMNLP, pages 15246–15263. Association for Computational Linguistics.

Linghao Jin, Jacqueline He, Jonathan May, and Xuezhe Ma. 2023b. Challenges in context-aware neural machine translation. In Empirical Methods in Natural Language Processing (EMNLP).

Yimin Jing, Deyi Xiong, and Yan Zhen. 2019. Bi-par: A bilingual parallel dataset for multilingual and cross-lingual reading comprehension on novels. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3–7, 2019, pages 2452–2462. Association for Computational Linguistics.

Marzena Karpinska and Mohit Iyyer. 2023. Large language models effectively leverage document-level context for literary translation, but critical errors persist. In Proceedings of the Eighth Conference on Machine Translation, pages 419–451, Singapore. Association for Computational Linguistics.

Marzena Karpinska, Nishant Raj, Katherine Thai, Yixiao Song, Ankita Gupta, and Mohit Iyyer. 2022a. Demetr: Diagnosing evaluation metrics for translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

Marzena Karpinska, Katherine Thai, Kalpesh Krishna, John Wieting, Moira Inghilleri, and Mohit Iyyer. 2022b. Par3.

Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. 2023. Speculative decoding with big little decoder. In NeurIPS.

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin Popel, Maja Popovic, and Mariya Shmatova. 2023. Findings of the 2023 conference on machine translation (WMT23): llms are here but not quite there yet. In WMT, pages 1–42. Association for Computational Linguistics.

Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (HLT-NAACL 2003), pages 48–54. Association for Computational Linguistics.

Abdullatif Köksal and Arzucan Özgür. 2020. The RELX dataset and matching the multilingual blanks for cross-lingual relation classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 340–350, Online. Association for Computational Linguistics.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In SOSP, pages 611–626. ACM.

Samuel Läubli, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen, and Antonio Toral. 2020. A set of recommendations for assessing human-machine parity in language translation. J. Artif. Intell. Res., 67:653–672.

Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 19274–19286. PMLR.

Yunlong Liang, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. 2021. Modeling bilingual conversational characteristics for neural chat translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5711–5724, Online. Association for Computational Linguistics.

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuhui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona T. Diab, Veselin Stoyanov, and Xian Li. 2021. Few-shot learning with multilingual language models. CoRR, abs/2112.10668.

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuhui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona T. Diab, Veselin Stoyanov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In EMNLP, pages 9019–9052. Association for Computational Linguistics.

Boxiang Liu and Liang Huang. 2021. Paramed: a parallel corpus for english-chinese translation in the biomedical domain. BMC Medical Informatics Decis. Mak., 21(1):258.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In NeurIPS.

Hongyuan Lu, Haoran Yang, Haoyang Huang, Dongdong Zhang, Wai Lam, and Furu Wei. 2024. Chain-of-dictionary prompting elicits translation in large language models. In EMNLP, pages 958–976. Association for Computational Linguistics.

Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. In NeurIPS.
Sameen Maruf, Fahimeh Saleh, and Gholamreza Haffari. 2022. A survey on document-level neural machine translation: Methods and evaluation. ACM Comput. Surv., 54(2):45:1–45:36.

Yongyu Mu, Abdurexiti Reheman, Zhiquan Cao, Yuchun Fan, Bei Li, Yingqiao Li, Tong Xiao, Chun-liang Zhang, and Jingbo Zhu. 2023. Augmenting large language model translators via translation memories. In ACL (Findings), pages 10287–10299. Association for Computational Linguistics.

Niklas Muenninghoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2024. Generative representational instruction tuning. CoRR, abs/2402.09906.

Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation. In WMT 2018, Brussels, Belgium. Association for Computational Linguistics.

Maria Nädejde, Anna Currey, Benjamin Hsu, Xing Niu, Marcello Federico, and Georgiana Dinu. 2022. CoCoA-MT: A dataset and benchmark for Contrastive Controlled MT with application to formality. In Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, USA. Association for Computational Linguistics.

Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Gorri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative LLM inference using phase splitting. In ISCA, pages 118–132. IEEE.

Aleksandr Perevalov, Dennis Diefenbach, Ricardo Usbeck, and Andreas Both. 2022. Qald-9-plus: A multilingual dataset for question answering over dbpedia and wikidata translated by native speakers. In 2022 IEEE 16th International Conference on Semantic Computing (ICSC), pages 229–234.

Edoardo Maria Ponti, Goran Glavas, Olga Majewska, Qianchu Liu, Ivan Vulic, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 2362–2376. Association for Computational Linguistics.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In WMT, pages 186–191. Association for Computational Linguistics.

Alec Radford. 2018. Improving language understanding by generative pre-training.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: memory optimizations toward training trillion parameter models. In SC, page 20. IEEE/ACM.

Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In EMNLP (1), pages 2685–2702. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In EMNLP (1), pages 4512–4525. Association for Computational Linguistics.

Parker Riley, Timothy Dozat, Jan A. Botha, Xavier Garcia, Dan Garrette, Jason Riesa, Orhan Firat, and Noah Constant. 2022. FRMT: A benchmark for few-shot region-aware machine translation. arXiv preprint.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Illic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Lucioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muenninghoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Sorroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emeneau, Christopher Klamn, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. 2022. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100.

Priyanka Sen, Alham Fikri Aji, and Amir Saffari. 2022. Mintaka: A complex, natural, and multilingual dataset for end-to-end question answering. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1604–1619, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.

Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.

David Stap, Eva Hasler, Bill Byrne, Christof Monz, and Ke Tran. 2024. The fine-tuning paradox: Boosting translation quality without sacrificing llm abilities. Preprint, arXiv:2405.20089.

Ralf Steinberger, Andreas Eisele, Szymon Klocek, Spyridon Pilos, and Patrick Schlüter. 2013. DGT-TM: A freely available translation memory in 22 languages. CoRR, abs/1309.5226.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS, pages 3104–3112.

Kenan Tang. 2022. PETCI: A parallel english translation dataset of chinese idioms. CoRR, abs/2202.09509.

Liang Tian, Derek F. Wong, Lidia S. Chao, Paulo Quaresma, Francisco Oliveira, and Lu Yi. 2014. Um-corpus: A large english-chinese parallel corpus for statistical machine translation. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26-31, 2014, pages 1837–1842. European Language Resources Association (ELRA).

Ahmet Üstün, Virata Aryabumi, Zheng Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalka Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargas, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction finetuned open-access multilingual language model. In ACL (1), pages 15894–15939. Association for Computational Linguistics.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998–6008.

Elena Voita, Rico Sennrich, and Ivan Titov. 2019. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1198–1212, Florence, Italy. Association for Computational Linguistics.

Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019. Learning deep transformer models for machine translation. In ACL (1), pages 1810–1822. Association for Computational Linguistics.

Rachel Wicks and Matt Post. 2023. Identifying context-dependent translations for evaluation set production. In Proceedings of the Eighth Conference on Machine Translation, WMT 2023, Singapore, December 6-7, 2023, pages 452–467. Association for Computational Linguistics.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 38087–38099. PMLR.

Tong Xiao and Jingbo Zhu. 2025. Foundations of large language models. arXiv preprint arXiv:2501.09223.

Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2023. A paradigm shift in machine translation: Boosting translation performance of large language models. CoRR, abs/2309.11674.

Mingzhou Xu, Longyue Wang, Derek F. Wong, Hongye Liu, Linfeng Song, Lidia S. Chao, Shuming Shi, and Zhaopeng Tu. 2022. Guofeng: A benchmark for zero pronoun recovery and translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11266–11278. Association for Computational Linguistics.

Weijia Xu and Marine Carpuat. 2021. Rule-based morphological inflection improves neural terminology translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 5902–5914. Association for Computational Linguistics.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In NAACL-HLT, pages 483–498. Association for Computational Linguistics.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024a. Qwen2 technical report. CoRR, abs/2407.10671.

Jian Yang, Yuwei Yin, Liqun Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Furu Wei, and Zhoujun Li. 2023a. Gtrans: Grouping and fusing transformer layers for neural machine translation. IEEE ACM Trans. Audio Speech Lang. Process., 31:1489–1498.

Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023b. Bigtrans: Augmenting large language models with multilingual translation capability over 100 languages. CoRR, abs/2305.18098.

Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification. In Proc. of EMNLP.

Yuming Yang, Wantong Zhao, Caishuang Huang, Junjie Ye, Xiao Wang, Huiyuan Zheng, Yang Nan, Yuran Wang, Xueying Xu, Kaixin Huang, Yunke Zhang, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024b.
Beyond boundaries: Learning a universal entity taxonomy across datasets and languages for open named entity recognition. CoRR, abs/2406.11192.

Jiali Zeng, Fandong Meng, Yongjing Yin, and Jie Zhou. 2024. Improving machine translation with large language models: A preliminary study with cooperative decoding. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 13275–13288. Association for Computational Linguistics.

Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a. Prompting large language model for machine translation: A case study. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 41092–41110. PMLR.

Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In ACL, pages 1628–1639. Association for Computational Linguistics.

Hongbin Zhang, Kehai Chen, Xuefeng Bai, Yang Xi-ang, and Min Zhang. 2024. Paying more attention to source context: Mitigating unfaithful translations from large language model. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 13816–13836. Association for Computational Linguistics.

Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, and Yang Feng. 2023b. Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models. CoRR, abs/2306.10968.

Shaozhui Zheng, Zhixu Li, Jian Wang, Jianfeng Qu, An Liu, Lei Zhao, and Zhigang Chen. 2023. Long-document cross-lingual summarization. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM 2023, Singapore, 27 February 2023 - 3 March 2023, pages 1084–1092. ACM.

Tong Zheng, Yan Wen, Huiwen Bao, Junfeng Guo, and Heng Huang. 2025. Asymmetric conflict and synergy in post-training for llm-based multilingual machine translation. arXiv preprint arXiv:2502.11223.

Wenhao Zhu, Shujian Huang, Tong Pu, Pingxuan Huang, Xu Zhang, Jian Yu, Wei Chen, Yanfeng Wang, and Jiajun Chen. 2021. Fdmt: A benchmark dataset for fine-grained domain adaptation in machine translation. arXiv preprint arXiv:2012.15717.

Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual machine translation with large language models: Empirical results and analysis. In NAACL-HLT (Findings), pages 2765–2781. Association for Computational Linguistics.

A Encoder-Decoder vs Decoder-Only

Recent advancements in NLP have been profoundly shaped by Transformer models (Vaswani et al., 2017; Radford, 2018; Devlin et al., 2019), which have revolutionized both model design and task handling. The original Transformer (Vaswani et al., 2017) was designed for sequence-to-sequence (seq2seq) tasks, utilizing an encoder-decoder architecture where the encoder encodes the input sequence and the decoder generates the output. Subsequent models like GPT (Radford, 2018) leverage only the Transformer’s decoder, omitting the cross-attention layer, for language modeling tasks. Both architectures are illustrated in Figure 6. Although originally designed for different purposes, many NLP tasks can be framed as either seq2seq or language modeling problems (Xiao and Zhu, 2025; Chang et al., 2024).

From a macroscopic perspective, the encoder-decoder architecture employs an explicit encoder module to encode the input, after which the decoder generates output based on both the source representation from the encoder’s top layer and the target representation from preceding layer. Given the source sequence \( X \) and target sequence \( Y \), this process can be represented as follows 2:

\[
X^l = \mathrm{FFN}(\mathrm{SAtt}(X^{l-1}, M_f)) \tag{10}
\]
\[
Y^l = \mathrm{FFN}(\mathrm{CAtt}(\mathrm{SAtt}(Y^{l-1}, M_c), X^L, M_f)) \tag{11}
\]

where \( l \) denotes the layer index, \( X^l \) and \( Y^l \) represent the representations of the source and target at \( l \)-th layer, respectively, with \( X^0 \) and \( Y^0 \) indicating the embeddings. Each layer of the encoder first performs self-attention (SAtt) with a fully-visible mask pattern \( M_f \), indicating that all tokens in the source can attend to each other, followed by a feed-forward neural network (FFN). The encoder generates its final representation \( X^L \) parallely by stacking L such layers. The decoder is similar to the encoder but differs in that: (i) due to its autoregressive nature, its self-attention uses a causal mask pattern \( M_c \), allowing attention only to historical tokens; (ii) it incorporates an additional cross-attention (CAtt) layer to integrate representations from the source, where the mask \( M_f \) is a fully-visible mask pattern, enabling the target to attend to the entire source.

Contrastingly, the decoder-only architecture processes both the source and target sequences within a single module:

2For simplicity, we omit layer normalization and residual connections in the following formulas and descriptions.
![The Encoder-Decoder and Decoder-only architecture.](page_186_120_1077_349.png)

Figure 6: The Encoder-Decoder and Decoder-only architecture.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Connection Type</th>
    <th colspan="2">Fuse Type</th>
    <th rowspan="2">Src Mask</th>
    <th rowspan="2">Enc-Dec Parameter Sharing</th>
  </tr>
  <tr>
    <th>TopOnly</th>
    <th>Layer-Wise</th>
    <th>Cross Attention</th>
    <th>Concat Attention</th>
  </tr>
  <tr>
    <td>Encoder-Decoder</td>
    <td>✓</td>
    <td></td>
    <td>✓</td>
    <td></td>
    <td>Fully-visible</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>Causal LM</td>
    <td></td>
    <td>✓</td>
    <td></td>
    <td></td>
    <td>Causal</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Prefix LM</td>
    <td>✓</td>
    <td></td>
    <td>✓</td>
    <td>✓</td>
    <td>Fully-visible</td>
    <td>✓</td>
  </tr>
</table>

Table 4: Comparision of different architecture by analyzing their structural elements.

\[
[X^l, Y^l] = \text{FFN}(\text{SAtt}([X^{l-1}, Y^{l-1}], M_c))
\]

In practice, due to the nature of causal attention, the representation of \( X \) is independent of \( Y \). This allows us to encode \( X \) independently to obtain representations at each layer, as shown in Equation (10). Subsequently, the representations of \( X^l \) at each layer are concatenated with the corresponding target representations, enabling self-attention computations of the target across both source and target sequences, as shown in Figure 6(b). It is important to note that: (i) when encoding \( X \), we can modify its original causal attention mask \( M_c \) to be fully-visible mask \( M_f \), similar to used in the encoder-decoder architecture, thus creating a variant of the CausalLM model known as the prefixLM model (Dong et al., 2019; Raffel et al., 2020); (ii) the fusion of source and target information is achieved by computing attention on concatenated representations of both, distinct from cross-attention, which we refer to in this paper as concat attention; (iii) the interaction between \( X \) and \( Y \) is Layer-Wise, rather than using only the top-layer representation of X as in the encoder-decoder architecture (TopOnly).

The overall comparison of these architectures is shown in Table 4. In practice, some modern deployment frameworks of LLMs (decoder-only models) explicitly separate encoding (prefilling) and decoding processes across distinct computational resources, making the architecture structurally resemble encoder-decoder models (Patel et al., 2024). In this light, the so-called decoder-only model can be considered a variant of the encoder-decoder model, wherein the encoding function is implicitly integrated through shared parameters with the decoder. Conversely, one may view the encoder-decoder model as an extension of PrefixLM, with a more explicit division between encoding and decoding stages.

B ComMT

B.1 Data Collection, Categorization and Processing

Our data collection and curation efforts are guided by two key principles: ensuring high quality and maintaining diversity across translation tasks. To this end, we extensively gathered publicly available datasets from the research community, focusing primarily on well-established repositories such as OPUS 3, WMT 4, and Papers with Code 5. These sources have been widely used in MT research.

To uphold data quality, we retained only those datas that had undergone manual annotation, cul-

3https://opus.nlpl.eu/
4https://www.statmt.org/wmt23/
5https://paperswithcode.com/datasets
ComMT

<table>
  <tr>
    <th>General Translation(GT)</th>
    <th>Doc-level Translation(DLT)</th>
    <th>Domain Translation(DT)</th>
  </tr>
  <tr>
    <td>WMT 14-23<br>FLORES-200<br>CommonMT</td>
    <td>WMT 23<br>IWSLT 15-17<br>DiscoMT</td>
    <td>WMT 14, 16, 18-22<br>UM-corpus<br>IWSLT 14</td>
  </tr>
  <tr>
    <td>XNLI-15way<br>NTREX-128<br>PAWS-X</td>
    <td>BiPar<br>mZPRT<br>News-commentary</td>
    <td>DGT-TM<br>BMELD<br>MSLT</td>
  </tr>
  <tr>
    <td>FRMT<br>RELX</td>
    <td>BWB<br>Par3</td>
    <td>BiPaR<br>NEJM</td>
  </tr>
  <tr>
    <th>Terminology-con Translation(TCT)</th>
    <th>Automatic Post-edition(APE)</th>
    <th>In-context Learning(ICL)</th>
  </tr>
  <tr>
    <td>WMT 21, 23<br>Health Terminology<br>...</td>
    <td>WMT 16-21<br>IWSLT 15<br>CommonMT</td>
    <td>IdiomsInCxt-MT<br>PETCI<br>Good-translation-wrong-in-context<br>...</td>
  </tr>
  <tr>
    <td>Synthetic_term</td>
    <td>DEMETR<br>MLQE-PE<br>ContraPro</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>SubEdits<br>PETCI</td>
    <td></td>
  </tr>
</table>

![Pie chart showing distribution of tasks in ComMT](page_320_629_708_312.png)

<table>
  <tr>
    <th>Task</th>
    <th>Subtask</th>
    <th>De</th>
    <th>Ru</th>
    <th>Zh</th>
    <th>Cs</th>
    <th>All</th>
  </tr>
  <tr>
    <td rowspan="2">GT</td>
    <td>-</td>
    <td>38483</td>
    <td>40862</td>
    <td>56918</td>
    <td>21058</td>
    <td>157321</td>
  </tr>
  <tr>
    <td>DLT</td>
    <td>2481</td>
    <td>3323</td>
    <td>2447</td>
    <td>2207</td>
    <td>10458</td>
  </tr>
  <tr>
    <td rowspan="4">DT</td>
    <td>Medical</td>
    <td>3764</td>
    <td>929</td>
    <td>2682</td>
    <td>788</td>
    <td>8163</td>
  </tr>
  <tr>
    <td>Law</td>
    <td>1500</td>
    <td>417</td>
    <td>1466</td>
    <td>3383</td>
    <td></td>
  </tr>
  <tr>
    <td>IT</td>
    <td>2411</td>
    <td>862</td>
    <td>1694</td>
    <td>357</td>
    <td>3524</td>
  </tr>
  <tr>
    <td>Colloquial</td>
    <td>5116</td>
    <td>3122</td>
    <td>1310</td>
    <td>-</td>
    <td>9748</td>
  </tr>
  <tr>
    <td>Literature</td>
    <td>731</td>
    <td>3561</td>
    <td>2220</td>
    <td>95</td>
    <td>6607</td>
  </tr>
  <tr>
    <td>TCT</td>
    <td>-</td>
    <td>5318</td>
    <td>4850</td>
    <td>2615</td>
    <td>-</td>
    <td>12783</td>
  </tr>
  <tr>
    <td>APE</td>
    <td>-</td>
    <td>15314</td>
    <td>1646</td>
    <td>3646</td>
    <td>-</td>
    <td>20606</td>
  </tr>
  <tr>
    <td>ICL</td>
    <td>-</td>
    <td>1794</td>
    <td>1535</td>
    <td>1470</td>
    <td>469</td>
    <td>5268</td>
  </tr>
  <tr>
    <td>ALL</td>
    <td>-</td>
    <td>76912</td>
    <td>60690</td>
    <td>75619</td>
    <td>26440</td>
    <td>239661</td>
  </tr>
</table>

Figure 7: Our comprehensive translation dataset, ComMT, includes diverse translation-related tasks. The table presents the training set statistics for ComMT.

minating in a collection of over 50 high-quality datasets covering a diverse range of domains, and translation scenarios. For specialized tasks like document-level translation with limited annotated data, we sampled from resources such as news-commentary. To further refine our dataset and remove potentially low-quality samples, we employed the COMETkiwi (Rei et al., 2020) model for filtering. We set the filtering threshold at 0.55 for literature and doc-level translation tasks and 0.75 for all other tasks.

After filtering, we carefully organized and categorized each dataset based on its characteristics according to our classification protocol. All collected datasets are presented in Table 15. The principles for processing each task are as follows:

General Translation. This essential and well-researched task in the MT field emphasizes sentence-level translation in general domains, serving as the foundation for more specialized tasks. Datasets lacking distinctive features are classified as general sentence-level tasks. We included a small amount of domain-specific data to enhance generalizability, making this task the largest part of ComMT.

Document-level Translation. Document-level translation focuses on maintaining coherence and context in extended texts, rather than merely achieving sentence-level accuracy. This requires the model to consider a broader context and capture nuances across sentences (Maruf et al., 2022; Hu et al., 2024). Our document-level translation tasks mainly involve general long texts, such as news articles. Jin et al. (2023a) indicates that 3 preceding sentences are usually sufficient to disambiguate most discourse phenomena. Based on this finding and practical considerations, we restricted our document-level translation tasks to texts under 500 words and organized the data accordingly.

Domain Translation. Domain translation focuses on specialized domains, ensuring the accurate use of domain-specific terminology and expressions. Based on real-world applications and existing research, we identified five initial domains for study: Medical, Law, Information Technology(IT), Colloquial, and Literature. For the colloquial domain, we focus on informal, non-written text data, mainly from social media platforms, dialogues, and subtitles. For the literature domain, considering the
<table>
  <tr>
    <th rowspan="2">Task</th>
    <th rowspan="2">Subtask</th>
    <th colspan="2">De</th>
    <th colspan="2">Ru</th>
    <th colspan="2">Zh</th>
    <th colspan="2">Cs</th>
  </tr>
  <tr>
    <th>De2En</th>
    <th>En2De</th>
    <th>Ru2En</th>
    <th>En2Ru</th>
    <th>Zh2En</th>
    <th>En2Zh</th>
    <th>Cs2En</th>
    <th>En2Cs</th>
  </tr>
  <tr>
    <td rowspan="3">General Translation</td>
    <td>WMT23</td>
    <td>549</td>
    <td>557</td>
    <td>1723</td>
    <td>2074</td>
    <td>1976</td>
    <td>2074</td>
    <td>-</td>
    <td>2074</td>
  </tr>
  <tr>
    <td>WMT22</td>
    <td>1984</td>
    <td>2037</td>
    <td>2016</td>
    <td>2037</td>
    <td>1875</td>
    <td>2037</td>
    <td>1448</td>
    <td>2037</td>
  </tr>
  <tr>
    <td>FLORES-200</td>
    <td>-</td>
    <td>1012</td>
    <td>-</td>
    <td>1012</td>
    <td>-</td>
    <td>1012</td>
    <td>-</td>
    <td>1012</td>
  </tr>
  <tr>
    <td>Doc-level translation</td>
    <td>-</td>
    <td>500</td>
    <td>547</td>
    <td>-</td>
    <td>500</td>
    <td>415</td>
    <td>500</td>
    <td>500</td>
    <td>519</td>
  </tr>
  <tr>
    <td rowspan="4">Domain Translation</td>
    <td>Medical</td>
    <td>748</td>
    <td>744</td>
    <td>513</td>
    <td>1772</td>
    <td>580</td>
    <td>2094</td>
    <td>999</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Law</td>
    <td>-</td>
    <td>969</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>959</td>
  </tr>
  <tr>
    <td>IT</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>650</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>519</td>
  </tr>
  <tr>
    <td>Colloquial</td>
    <td>517</td>
    <td>595</td>
    <td>564</td>
    <td>507</td>
    <td>520</td>
    <td>652</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Literature</td>
    <td>506</td>
    <td>-</td>
    <td>506</td>
    <td>-</td>
    <td>401</td>
    <td>550</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>Terminology-constrained Translation</b></td>
    <td>-</td>
    <td>2948</td>
    <td>775</td>
    <td>-</td>
    <td>500</td>
    <td>1638</td>
    <td>798</td>
    <td>-</td>
    <td>2898</td>
  </tr>
  <tr>
    <td><b>Automatic Post-edition</b></td>
    <td>-</td>
    <td>1980</td>
    <td>1000</td>
    <td>526</td>
    <td>1023</td>
    <td>-</td>
    <td>1000</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 5: Statistics for the ComMT Test Set. We ensure that the source language of the test set is original, so not all language directions have corresponding data for each task.

![Bar charts comparing COMET and BLEU scores for En→X and X→En across WMT17-20, TowerBlock, and ComMT datasets](page_1012_1042_627_246.png)

Figure 8: Comparison of performance across three datasets—WMT17-20, TowerBlock, and ComMT—fine-tuned on Llama3-8B and evaluated on the WMT23 test set.

complex discourse phenomena in literary texts, sentence-level translation is unsuitable; thus, we construct this task as multi-sentences form. We categorize sentences into varying lengths of [100, 200, 300, 400] words, distributed in proportions of [0.2, 0.3, 0.3, 0.2], to enhance generalizability.

<b>Terminology-constrained Translation.</b> This task requires models to adhere to predefined terminology pairs for accurate conversion of terms, which is crucial for professional translations. Due to the limited availability of terminology translation data, we sampled data and used the B2NERD (Yang et al., 2024b) model to extract term pairs. Note that each sample may contain one or more terminology pairs in our data.

<b>Automatic Post-editing.</b> APE focuses on enhancing the quality of preliminary machine translation outputs by automatically correcting errors in grammar, spelling, and style in the raw output. We focused on collecting authentic data for this task rather than using synthetic data due to its potential lack of accuracy in representing real post-editing scenarios.

<b>In-context Learning Translation.</b> 6 We designed this task mainly to leverage the inherent In-context learning capabilities of LLMs, which potentially enable models to adapt on-the-fly. We extracted 5% of the data from each task category to create few-shot datasets. We structured these into 1-shot, 2-shot, and 3-shot data sets in proportions of [0.3, 0.3, 0.4].

6We created this few-shot training data to enhance data diversity for the other five primary tasks. This task is used solely for training and is not included in the evaluation.
B.2 Construction of Training and Test Sets

For train sets, to manage the diverse data sources and prevent excessive data accumulation, we have set a cap of 5,000 samples per dataset. The training set statistics are displayed in Figure 7. The sample display of each task is shown in Table 16.

For test sets, we ensure that the source language data is "source-original" to closely simulate real-world scenarios and prevent "translationese" effects that could negatively impact evaluation accuracy (Graham et al., 2020; Läubli et al., 2020). For general translation tasks, we keep data from WMT22 (Adelani et al., 2022), WMT23 (Kocmi et al., 2023), and Flores-200 (Costa-jussà et al., 2022), and use wmt23 as the default test set. For other tasks such as terminology-constrained translation and automatic post-editing, we either use standard test sets (e.g., WMT17–WMT23) or construct custom test sets by sampling from multiple publicly available datasets. We strive to maintain a minimum of 500 examples per test set to ensure robust evaluation. For under-resourced tasks, test sets are currently unavailable due to data scarcity, but we plan to continue expanding coverage in future work.

B.3 Quality Verification

To verify the quality of our dataset, we fine-tuned the Llama3-8B model (Dubey et al., 2024) and compared it against two other datasets commonly used in LLM-based MT research: (i) a merged news test set from WMT17 to WMT20 with 61k samples, widely adopted data setting in previous studies (Xu et al., 2023; Jiao et al., 2023; Guo et al., 2024); (ii) the TowerBlock dataset (Alves et al., 2024), which comprises 638k samples across translation tasks, named entity recognition (NER), general dialogue, and other tasks. We evaluated the resulting models on the WMT23 test set, with the results shown in Figure 8.

The results clearly demonstrate that ComMT surpasses WMT17–WMT20 in both the En → X and X → En directions. However, while it excels in the En → X direction, it underperforms in the X → En direction compared to TowerBlock. This discrepancy may be attributed to TowerBlock’s larger dataset size and its inclusion of additional general task data, predominantly in English, which may specifically enhance performance when translating into English. We intend to investigate this further in future research.

C Experimental Setup

C.1 Data and Hyperparameter

<table>
  <tr>
    <th>Language Pair</th>
    <th>De-En</th>
    <th>Cs-En</th>
    <th>Ru-En</th>
    <th>Zh-En</th>
  </tr>
  <tr>
    <th>Before Clean</th>
    <td>50.1M</td>
    <td>56.3M</td>
    <td>39.2M</td>
    <td>40.8M</td>
  </tr>
  <tr>
    <th>After Clean</th>
    <td>46.4M</td>
    <td>50.4M</td>
    <td>30.8M</td>
    <td>33.6M</td>
  </tr>
</table>

Table 6: Statistics on the use of parallel data from WMT2023. Note that due to the extensive bilingual data in the En-De CommonCrawl corpus, we only sampled a portion and merged it with other data to create a dataset of 50M. For En-Cs, we excluded the CzEng 2.0 dataset due to licensing issues.

While traditional MT research predominantly employs beam search for decoding, LLM-based generation often utilizes sampling strategies for faster and more diverse outputs. This raises the question of whether LLMs, when used for translation, should follow the conventional beam search approach or adopt sampling-based decoding. To investigate this, we conducted a comparative analysis of beam search (beam size = 5) and sampling (temperature = 0.7, top-k = 50, top-p = 0.8) in terms of translation quality and decoding speed (batch size = 4). As shown in Table 8, while beam search is significantly slower, it consistently yields higher translation quality. Based on these observations, we adopt beam search as the primary decoding method in this paper.

C.2 Evaluating the Translation Capabilities of Multilingual LLMs

To assess the fundamental translation capabilities of state-of-the-art multilingual LLMs, we conducted a comprehensive evaluation across several widely used models, including XGLM-7.5B (Lin et al., 2022), Bloom-7B (Scalo et al., 2022), Falcon-7B (Almazrouei et al., 2023), Qwen2-7B (Yang et al., 2024a), Llama3-8B (Dubey et al., 2024). These models were evaluated in both zero-shot and three-shot settings across English ↔ German, Czech, Russian, and Chinese translation tasks.

The averaged performance in in Figure 9 show that Llama3-8B consistently outperformed the other models in both En → X and X → En translation directions. Its superior performance across different language pairs suggests that it possesses stronger multilingual capabilities. Based on these results, we selected Llama3-8B as our backbone model for subsequent experiments.
<table>
  <tr>
    <th>Hyperparameter</th>
    <th>Stage1</th>
    <th>Stage2</th>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>5e-4</td>
    <td>2e-5</td>
  </tr>
  <tr>
    <td>Adam \( \beta \)</td>
    <td>(0.9, 0.999)</td>
    <td>(0.9, 0.999)</td>
  </tr>
  <tr>
    <td>LR Scheduler</td>
    <td>inverse_sqrt</td>
    <td>cosine</td>
  </tr>
  <tr>
    <td>Number of Epochs</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Global Batch Size</td>
    <td>2,560</td>
    <td>384</td>
  </tr>
  <tr>
    <td>Train Steps</td>
    <td>30,000</td>
    <td>1,200</td>
  </tr>
  <tr>
    <td>Warmup Ratio</td>
    <td>0.01</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>Weight Decay</td>
    <td>0.01</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>Decoding Method</td>
    <td colspan="2">beam search</td>
  </tr>
  <tr>
    <td>Beam Size</td>
    <td colspan="2">5</td>
  </tr>
</table>

Table 7: Hyperparameter configuration during two-stage training and decoding. In the first training stage, we use pure data parallelism because the LLM parameters are frozen. In the second stage, we employed DeepSpeed ZeRO-2 (Rajbhandari et al., 2020) for full parameters training.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Decoding Method</th>
    <th colspan="6">De2En</th>
    <th colspan="2">Avg.</th>
    <th rowspan="2">Speed (tokens/s)</th>
  </tr>
  <tr>
    <th>Ru2En</th>
    <th>Zh2En</th>
    <th>En2De</th>
    <th>En2Cs</th>
    <th>En2Ru</th>
    <th>En2Zh</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>TowerInstruct-7B</td>
    <td>Beam Search</td>
    <td>85.15</td>
    <td>83.18</td>
    <td>80.36</td>
    <td>83.10</td>
    <td>78.88</td>
    <td>85.39</td>
    <td>85.93</td>
    <td>83.16</td>
    <td>114</td>
  </tr>
  <tr>
    <td></td>
    <td>Sampling</td>
    <td>84.98</td>
    <td>82.49</td>
    <td>79.79</td>
    <td>82.03</td>
    <td>68.97</td>
    <td>84.14</td>
    <td>84.39</td>
    <td>81.15</td>
    <td>167</td>
  </tr>
  <tr>
    <td>Llama3-8B-SFT</td>
    <td>Beam Search</td>
    <td>83.76</td>
    <td>81.66</td>
    <td>76.97</td>
    <td>82.66</td>
    <td>85.80</td>
    <td>82.42</td>
    <td>80.80</td>
    <td>84.03</td>
    <td>123</td>
  </tr>
  <tr>
    <td></td>
    <td>Sampling</td>
    <td>83.48</td>
    <td>80.58</td>
    <td>76.63</td>
    <td>80.01</td>
    <td>81.74</td>
    <td>82.21</td>
    <td>82.03</td>
    <td>80.87</td>
    <td>151</td>
  </tr>
  <tr>
    <td>LaMaTE</td>
    <td>Beam Search</td>
    <td>83.90</td>
    <td>81.46</td>
    <td>79.13</td>
    <td>80.73</td>
    <td>86.81</td>
    <td>82.71</td>
    <td>84.41</td>
    <td>82.59</td>
    <td>296</td>
  </tr>
  <tr>
    <td></td>
    <td>Sampling</td>
    <td>82.82</td>
    <td>80.53</td>
    <td>77.97</td>
    <td>80.73</td>
    <td>86.81</td>
    <td>82.71</td>
    <td>84.41</td>
    <td>80.57</td>
    <td>379</td>
  </tr>
</table>

Table 8: Comparison of the effectiveness and efficiency of beam search versus sampling.

![Bar charts comparing BLEU and COMET scores for different models in 0-shot and 3-shot settings](page_1012_1012_627_246.png)

Figure 9: Evaluation of LLM translation capabilities in 0-shot and 3-shot settings using the WMT23 test set.

D Experiment Results

D.1 Results

We present a detailed performance of each translation task and language pair in the ComMT benchmark in Table 14.

Misalignment Evaluation. To assess the translation misalignment issue, we used three metrics: off-target rate, unaligned source words, and unaligned target words. Off-target refers to instances where machine-generated translations contain segments from incorrect languages or exhibit code-switching. We utilize langdetect\footnote{https://github.com/Mimino666/langdetect} to determine the language of each translation. The off-target rate for a translation is calculated by subtracting the probability of the predicted target language from 1. We then average this rate across all sentences. Unaligned source words (USW) refer to words in the source sentence that do not have a corresponding translation in the target sentence. Conversely, unaligned target words (UTW) capture instances where words appear in the translation without clear support from the source sentence, indicating potential insertions or hallucinations. We employ awesome-align (Dou and Neubig, 2021) to obtain word alignments. Table 9 displays the results of these three indicators.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">Off-target Rate ↓</th>
    <th colspan="4">USW Rate ↓</th>
    <th colspan="4">USW Rate ↓</th>
  </tr>
  <tr>
    <th>De</th><th>Cs</th><th>Ru</th><th>Zh</th>
    <th>De</th><th>Cs</th><th>Ru</th><th>Zh</th>
    <th>De</th><th>Cs</th><th>Ru</th><th>Zh</th>
  </tr>
  <tr>
    <td>Llama3-8B-SFT</td><td>5.63</td><td>9.45</td><td>2.92</td><td>3.76</td><td>12.03</td><td>17.59</td><td>17.13</td><td>19.36</td><td>14.16</td><td>14.80</td><td>14.10</td><td>22.59</td>
  </tr>
  <tr>
    <td>Tower-7B</td><td>6.07</td><td>10.41</td><td>3.49</td><td>2.46</td><td>10.78</td><td>17.65</td><td>15.79</td><td>18.72</td><td>12.39</td><td>15.51</td><td>13.28</td><td>23.93</td>
  </tr>
  <tr>
    <td>mT5-large</td><td>6.52</td><td>9.14</td><td>2.53</td><td>3.50</td><td>10.00</td><td>15.03</td><td>15.31</td><td>16.88</td><td>11.60</td><td>11.45</td><td>12.51</td><td>18.27</td>
  </tr>
  <tr>
    <td>LaMaTE</td><td>4.97</td><td>7.93</td><td>2.06</td><td>3.17</td><td>10.74</td><td>16.11</td><td>16.34</td><td>17.57</td><td>11.82</td><td>11.10</td><td>12.61</td><td>17.51</td>
  </tr>
</table>

Table 9: Results of the Decoder-only and Encoder-Decoder models on three indicators of misalignment.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">Decoder-only</th>
    <th colspan="6">Encoder-Decoder</th>
  </tr>
  <tr>
    <th>Llama2-7B</th><th>Llama3-8B</th><th>Llama2-13B</th><th>NMT-40-8</th><th>mT5-large</th><th>NLLB-3.3B</th><th>LaMaTE (Ours)</th>
  </tr>
  <tr>
    <td>Dim</td><td>4096</td><td>4096</td><td>5120</td><td>1024</td><td>1024</td><td>2048</td><td>4096-1024</td>
  </tr>
  <tr>
    <td>Encoder layer</td><td>-</td><td>-</td><td>-</td><td>40</td><td>24</td><td>24</td><td>32-8</td>
  </tr>
  <tr>
    <td>Decoder layer</td><td>32</td><td>32</td><td>40</td><td>8</td><td>24</td><td>24</td><td>8</td>
  </tr>
  <tr>
    <td>Vocab size</td><td>32k</td><td>128k</td><td>32k</td><td>128k</td><td>256k</td><td>256k</td><td>128k</td>
  </tr>
  <tr>
    <td>Params</td><td>6.73B</td><td>8.01B</td><td>13.01B</td><td>0.77B</td><td>1.23B</td><td>3.34B</td><td>8.5B</td>
  </tr>
  <tr>
    <td>KV Cache (Kb)</td><td>524b(s+t)</td><td>131b(s+t)</td><td>819b(s+t)</td><td>32b(s+t)</td><td>98b(s+t)</td><td>196b(s+t)</td><td>32b(s+t)</td>
  </tr>
</table>

Table 10: Details on parameter and theoretical KV cache size of the compared models in our work. In the KV Cache, b, s, and t denote the batch size, source sequence length, and target sequence length, respectively. Note that Llama3-8B uses GQA (Ainslie et al., 2023), resulting in smaller KV cache usage compared to Llama2-7B.

Large-Scale Multilingual Translation. To further validate whether our method can benefit a broader range of languages, we conducted extensive multilingual translation. Specifically, we utilized OPUS-100 (Zhang et al., 2020) as our training set, which includes bilingual data for 99 languages paired with English, totaling 55M sentences. Table 12 shows the classification of resource levels for these languages. We trained the models on the X → En direction and evaluated with Flores-200 dev-test set (Costa-jussà et al., 2022). For our method, we froze the LLM parameters during training and only trained the parameters of the adaptor and decoder. Due to the high cost of fine-tuning LLMs on such a large-scale dataset, we did not compare the LLM-SFT model.

D.2 Efficiency Analysis

We first summarize the key architectural details of the evaluated models in Table 10. Next, we evaluate the decoding efficiency of these models in Table 13 by comparing their decoding speed across different sequence lengths and batch sizes. VLLM (Kwon et al., 2023) accelerates LLM decoding but has limited beam search support and compatibility issues with some models. To ensure fair and consistent evaluation, we use the original Transformers framework across all models. The results demonstrate that LaMaTE consistently outperforms Llama3-8B in decoding speed, particularly as batch sizes and input lengths increase, achieving a speedup ranging from 2.4× to 6.5×.

D.3 Decoder Variants Comparision

The decoder of the original transformer utilizes cross attention to integrate the encoder’s representation, as shown in Equation (11). We refer to this standard decoder Cross Decoder. We propose two variants of the decoder that omit the cross-attention layer.

The first variant, referred to as the Concat Decoder, handles the encoder’s representation \( H_E \) by incorporating it directly into the self-attention layers of the decoder. Specifically, in the self-attention computation, the keys and values are computed from \([H_E, Y^{l-1}]\), while the queries are derived solely from \(Y^{l-1}\). Thus, this allows target tokens to integrate source and target information within a single attention computation:

\[
Y^l = \mathrm{FFN}(\mathrm{SAtt}([H_E, Y^{l-1}]))
\]

The second variant, taking inspiration from recent research in multimodal language model (Liu et al., 2023), referred to as Prefix Decoder, where the encoder representations are concatenated directly with the decoder’s embeddings \(Y_0\) before being fed to the upper decoder layers:

\[
Y^l = \mathrm{FFN}(\mathrm{SAtt}(Y^{l-1})), \quad Y^0 = [H_E, Y^0]
\]

We adopt a masking strategy similar to PrefixLM, ensuring that source-side tokens retain their bidirectional feature.
Y^1 = FFN(CAtt(SAtt(Y^{l-1}), X^l))
Cross Decoder

Y^1 = FFN(SAtt([X^l, Y^{l-1}]))
Concat Decoder

Y^1 = FFN(SAtt(Y^{l-1})), Y^0 = [X^l, Y^0]
Prefix Decoder

Figure 10: Three variants of decoders: Cross Decoder is the standard decoder, while Concat Decoder and Prefix Decoder remove the cross-attention sublayer, integrating source information through self-attention and early fusion methods, respectively.

<table>
  <tr>
    <th rowspan="2">Decoder</th>
    <th rowspan="2">Training Stage</th>
    <th colspan="3">De</th>
    <th colspan="3">Cs</th>
    <th colspan="3">Ru</th>
    <th colspan="3">Zh</th>
    <th colspan="3">Avg.</th>
  </tr>
  <tr>
    <th>COMET BLEU</th>
    <th>BLEU</th>
    <th>COMET BLEU</th>
    <th>BLEU</th>
    <th>COMET BLEU</th>
    <th>BLEU</th>
    <th>COMET BLEU</th>
    <th>BLEU</th>
    <th>COMET BLEU</th>
    <th>BLEU</th>
    <th>COMET BLEU</th>
    <th>BLEU</th>
  </tr>
  <tr>
    <td>Cross Decoder</td>
    <td>1</td>
    <td>79.93</td>
    <td>31.68</td>
    <td>85.31</td>
    <td>33.22</td>
    <td>79.38</td>
    <td>26.67</td>
    <td>78.64</td>
    <td>29.92</td>
    <td>80.07</td>
    <td>29.67</td>
  </tr>
  <tr>
    <td></td>
    <td>2</td>
    <td>82.22</td>
    <td>39.50</td>
    <td>86.54</td>
    <td>35.34</td>
    <td>81.71</td>
    <td>28.61</td>
    <td>81.46</td>
    <td>34.00</td>
    <td>82.32</td>
    <td>33.85</td>
  </tr>
  <tr>
    <td>Concat Decoder</td>
    <td>1</td>
    <td>79.45</td>
    <td>30.50</td>
    <td>85.02</td>
    <td>32.89</td>
    <td>79.45</td>
    <td>26.40</td>
    <td>78.99</td>
    <td>30.15</td>
    <td>80.01</td>
    <td>29.33</td>
  </tr>
  <tr>
    <td></td>
    <td>2</td>
    <td>79.89</td>
    <td>32.38</td>
    <td>85.97</td>
    <td>34.04</td>
    <td>81.63</td>
    <td>28.47</td>
    <td>81.14</td>
    <td>33.49</td>
    <td>81.52</td>
    <td>31.51</td>
  </tr>
  <tr>
    <td>Prefix Decoder</td>
    <td>1</td>
    <td>79.12</td>
    <td>30.04</td>
    <td>85.37</td>
    <td>32.73</td>
    <td>79.71</td>
    <td>26.73</td>
    <td>79.24</td>
    <td>30.21</td>
    <td>80.11</td>
    <td>29.25</td>
  </tr>
  <tr>
    <td></td>
    <td>2</td>
    <td>81.17</td>
    <td>37.56</td>
    <td>85.54</td>
    <td>32.06</td>
    <td>81.11</td>
    <td>27.54</td>
    <td>81.07</td>
    <td>33.10</td>
    <td>81.66</td>
    <td>32.38</td>
  </tr>
</table>

Table 11: A comparison of decoder variants.

A comparative overview of the three variants is presented in Figure 10. Table 11 displays their performances. As shown, the Cross Decoder, i.e., the standard decoder, achieves the best overall performance.
<table>
  <tr>
    <th>Resource</th>
    <th>Languages</th>
  </tr>
  <tr>
    <td>High (>1%)</td>
    <td>Arabic (ar), German (de), English (en), Spanish (es), French (fr), Italian (it), Japanese (ja), Dutch (nl), Polish (pl), Portuguese (pt), Russian (ru), Turkish (tr), Chinese (zh)</td>
  </tr>
  <tr>
    <td>Medium (>0.1%)</td>
    <td>Bulgarian (bg), Bengali (bn), Catalan (ca), Czech (cs), Danish (da), Modern Greek (el), Estonian (et), Persian (fa), Finnish (fi), Hindi (hi), Hungarian (hu), Indonesian (id), Korean (ko), Lithuanian (lt), Latvian (lv), Norwegian (no), Romanian (ro), Slovak (sk), Slovene (sl), Swedish (sv), Thai (th), Ukrainian (uk), Vietnamese (vi)</td>
  </tr>
  <tr>
    <td>Low (>0.01%)</td>
    <td>Afrikaans (af), Azerbaijani (az), Belarusian (be), Basque (eu), Galician (gl), Gujarati (gu), Modern Hebrew (he), Armenian (hy), Icelandic (is), Georgian (ka), Kazakh (kk), Kannada (kn), Macedonian (mk), Malayalam (ml), Mongolian (mn), Marathi (mr), Nepali (ne), Albanian (sq), Serbian (sr), Tamil (ta), Telugu (te), Urdu (ur)</td>
  </tr>
  <tr>
    <td>Very-Low (<0.01%)</td>
    <td>Amharic (am), Aragonese (an), Assamese (as), Avaric (av), Tibetan (bo), Breton (br), Bosnian (bs), Welsh (cy), Dzongkha (dz), Esperanto (eo), Western Frisian (fy), Irish (ga), Gaelic (gd), Hausa (ha), Croatian (hr), Igbo (ig), Central Khmer (km), Kurdish (ku), Kirghiz (ky), Limburgish (li), Malagasy (mg), Malay (ms), Maltese (mt), Burmese (my), Norwegian Bokmål (nb), Norwegian Nynorsk (nn), Occitan (oc), Oriya (or), Panjabi (pa), Pashto (ps), Kinyarwanda (rw), Northern Sami (se), Serbo-Croatian (sh), Sinhala (si), Tajik (tg), Turkmen (tk), Tatar (tt), Uighur (ug), Uzbek (uz), Walloon (wa), Xhosa (xh), Yiddish (yi), Yoruba (yo), Zulu (zu)</td>
  </tr>
</table>

Table 12: We categorized 99 languages in OPUS-100 (Zhang et al., 2020) into four resource levels—high, medium, low, and very low—based on the proportion of data available on the internet.
<table>
  <tr>
    <th rowspan="2">Batch Size</th>
    <th rowspan="2">Length</th>
    <th colspan="4">Decoder-only</th>
    <th colspan="3">Encoder-Decoder</th>
    <th rowspan="2">Speedup↑</th>
  </tr>
  <tr>
    <th>Llama2-7B</th>
    <th>Llama3-8B</th>
    <th>Llama2-13B</th>
    <th>NMT-40-8</th>
    <th>mT5-large</th>
    <th>NLLB-3.3B</th>
    <th>LaMaTE (Ours)</th>
  </tr>
  <tr>
    <td rowspan="4">1</td>
    <td>0-100</td>
    <td>35</td>
    <td>32</td>
    <td>28</td>
    <td>170</td>
    <td>43</td>
    <td>48</td>
    <td>77</td>
    <td>2.41</td>
  </tr>
  <tr>
    <td>100-200</td>
    <td>33</td>
    <td>33</td>
    <td>23</td>
    <td>167</td>
    <td>44</td>
    <td>42</td>
    <td>83</td>
    <td>2.52</td>
  </tr>
  <tr>
    <td>200-300</td>
    <td>31</td>
    <td>34</td>
    <td>21</td>
    <td>162</td>
    <td>43</td>
    <td>38</td>
    <td>83</td>
    <td>2.44</td>
  </tr>
  <tr>
    <td>300-400</td>
    <td>29</td>
    <td>32</td>
    <td>19</td>
    <td>160</td>
    <td>42</td>
    <td>36</td>
    <td>82</td>
    <td>2.56</td>
  </tr>
  <tr>
    <td>400-500</td>
    <td>29</td>
    <td>33</td>
    <td>18</td>
    <td>160</td>
    <td>43</td>
    <td>36</td>
    <td>81</td>
    <td>2.45</td>
  </tr>
  <tr>
    <td rowspan="4">2</td>
    <td>0-100</td>
    <td>67</td>
    <td>63</td>
    <td>51</td>
    <td>310</td>
    <td>84</td>
    <td>77</td>
    <td>152</td>
    <td>2.41</td>
  </tr>
  <tr>
    <td>100-200</td>
    <td>53</td>
    <td>62</td>
    <td>34</td>
    <td>294</td>
    <td>83</td>
    <td>63</td>
    <td>156</td>
    <td>2.52</td>
  </tr>
  <tr>
    <td>200-300</td>
    <td>45</td>
    <td>58</td>
    <td>29</td>
    <td>289</td>
    <td>81</td>
    <td>55</td>
    <td>154</td>
    <td>2.66</td>
  </tr>
  <tr>
    <td>300-400</td>
    <td>38</td>
    <td>52</td>
    <td>24</td>
    <td>281</td>
    <td>80</td>
    <td>52</td>
    <td>157</td>
    <td>3.02</td>
  </tr>
  <tr>
    <td>400-500</td>
    <td>38</td>
    <td>51</td>
    <td>24</td>
    <td>283</td>
    <td>79</td>
    <td>51</td>
    <td>155</td>
    <td>3.04</td>
  </tr>
  <tr>
    <td rowspan="4">4</td>
    <td>0-100</td>
    <td>114</td>
    <td>123</td>
    <td>76</td>
    <td>515</td>
    <td>164</td>
    <td>181</td>
    <td>296</td>
    <td>2.41</td>
  </tr>
  <tr>
    <td>100-200</td>
    <td>57</td>
    <td>87</td>
    <td>35</td>
    <td>472</td>
    <td>141</td>
    <td>112</td>
    <td>276</td>
    <td>3.17</td>
  </tr>
  <tr>
    <td>200-300</td>
    <td>47</td>
    <td>77</td>
    <td>OOM</td>
    <td>453</td>
    <td>133</td>
    <td>94</td>
    <td>272</td>
    <td>3.53</td>
  </tr>
  <tr>
    <td>300-400</td>
    <td>38</td>
    <td>66</td>
    <td>OOM</td>
    <td>421</td>
    <td>123</td>
    <td>82</td>
    <td>263</td>
    <td>3.98</td>
  </tr>
  <tr>
    <td>400-500</td>
    <td>38</td>
    <td>65</td>
    <td>OOM</td>
    <td>422</td>
    <td>123</td>
    <td>79</td>
    <td>259</td>
    <td>3.98</td>
  </tr>
  <tr>
    <td rowspan="4">6</td>
    <td>0-100</td>
    <td>144</td>
    <td>172</td>
    <td>90</td>
    <td>668</td>
    <td>230</td>
    <td>217</td>
    <td>416</td>
    <td>2.42</td>
  </tr>
  <tr>
    <td>100-200</td>
    <td>59</td>
    <td>100</td>
    <td>OOM</td>
    <td>565</td>
    <td>183</td>
    <td>116</td>
    <td>363</td>
    <td>3.63</td>
  </tr>
  <tr>
    <td>200-300</td>
    <td>49</td>
    <td>87</td>
    <td>OOM</td>
    <td>549</td>
    <td>171</td>
    <td>95</td>
    <td>355</td>
    <td>4.08</td>
  </tr>
  <tr>
    <td>300-400</td>
    <td>39</td>
    <td>73</td>
    <td>OOM</td>
    <td>495</td>
    <td>155</td>
    <td>84</td>
    <td>335</td>
    <td>4.59</td>
  </tr>
  <tr>
    <td>400-500</td>
    <td>39</td>
    <td>72</td>
    <td>OOM</td>
    <td>500</td>
    <td>154</td>
    <td>83</td>
    <td>334</td>
    <td>4.64</td>
  </tr>
  <tr>
    <td rowspan="4">8</td>
    <td>0-100</td>
    <td>162</td>
    <td>210</td>
    <td>100</td>
    <td>789</td>
    <td>289</td>
    <td>232</td>
    <td>532</td>
    <td>2.53</td>
  </tr>
  <tr>
    <td>100-200</td>
    <td>59</td>
    <td>109</td>
    <td>OOM</td>
    <td>643</td>
    <td>216</td>
    <td>119</td>
    <td>446</td>
    <td>4.09</td>
  </tr>
  <tr>
    <td>200-300</td>
    <td>OOM</td>
    <td>93</td>
    <td>OOM</td>
    <td>617</td>
    <td>198</td>
    <td>99</td>
    <td>432</td>
    <td>4.65</td>
  </tr>
  <tr>
    <td>300-400</td>
    <td>OOM</td>
    <td>78</td>
    <td>OOM</td>
    <td>553</td>
    <td>177</td>
    <td>87</td>
    <td>404</td>
    <td>5.18</td>
  </tr>
  <tr>
    <td>400-500</td>
    <td>OOM</td>
    <td>77</td>
    <td>OOM</td>
    <td>556</td>
    <td>175</td>
    <td>82</td>
    <td>397</td>
    <td>5.16</td>
  </tr>
  <tr>
    <td rowspan="4">16</td>
    <td>0-100</td>
    <td>167</td>
    <td>278</td>
    <td>95</td>
    <td>956</td>
    <td>446</td>
    <td>281</td>
    <td>816</td>
    <td>2.94</td>
  </tr>
  <tr>
    <td>100-200</td>
    <td>OOM</td>
    <td>118</td>
    <td>OOM</td>
    <td>728</td>
    <td>269</td>
    <td>OOM</td>
    <td>631</td>
    <td>5.35</td>
  </tr>
  <tr>
    <td>200-300</td>
    <td>OOM</td>
    <td>99</td>
    <td>OOM</td>
    <td>690</td>
    <td>244</td>
    <td>OOM</td>
    <td>589</td>
    <td>5.95</td>
  </tr>
  <tr>
    <td>300-400</td>
    <td>OOM</td>
    <td>82</td>
    <td>OOM</td>
    <td>609</td>
    <td>207</td>
    <td>OOM</td>
    <td>535</td>
    <td>6.52</td>
  </tr>
  <tr>
    <td>400-500</td>
    <td>OOM</td>
    <td>81</td>
    <td>OOM</td>
    <td>602</td>
    <td>206</td>
    <td>OOM</td>
    <td>533</td>
    <td>6.58</td>
  </tr>
  <tr>
    <td rowspan="4">24</td>
    <td>0-100</td>
    <td>130</td>
    <td>308</td>
    <td>OOM</td>
    <td>976</td>
    <td>550</td>
    <td>307</td>
    <td>1050</td>
    <td>3.41</td>
  </tr>
  <tr>
    <td>100-200</td>
    <td>OOM</td>
    <td>121</td>
    <td>OOM</td>
    <td>739</td>
    <td>263</td>
    <td>OOM</td>
    <td>728</td>
    <td>6.02</td>
  </tr>
  <tr>
    <td>200-300</td>
    <td>OOM</td>
    <td>102</td>
    <td>OOM</td>
    <td>708</td>
    <td>250</td>
    <td>OOM</td>
    <td>667</td>
    <td>6.54</td>
  </tr>
  <tr>
    <td>300-400</td>
    <td>OOM</td>
    <td>OOM</td>
    <td>OOM</td>
    <td>624</td>
    <td>OOM</td>
    <td>OOM</td>
    <td>594</td>
    <td>-</td>
  </tr>
  <tr>
    <td>400-500</td>
    <td>OOM</td>
    <td>OOM</td>
    <td>OOM</td>
    <td>619</td>
    <td>OOM</td>
    <td>OOM</td>
    <td>593</td>
    <td>-</td>
  </tr>
</table>

Table 13: Evaluate model decoding speed (tokens/s) across various batch sizes and source sequence lengths. Speedup indicates the decoding speedup ratio of LaMaTE versus Llama3-8B.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">En2De</th>
    <th colspan="2">En2Cs</th>
    <th colspan="2">En2Ru</th>
    <th colspan="2">En2Zh</th>
    <th colspan="2">De2En</th>
  </tr>
  <tr>
    <th>COMET</th><th>BLEU</th>
    <th>COMET</th><th>BLEU</th>
    <th>COMET</th><th>BLEU</th>
    <th>COMET</th><th>BLEU</th>
    <th>COMET</th><th>BLEU</th>
  </tr>
  <tr><td>NLLB-3.3B</td><td>79.62</td><td>33.51</td><td>88.06</td><td>36.79</td><td>83.74</td><td>29.03</td><td>79.66</td><td>34.84</td><td>81.35</td><td>35.42</td></tr>
  <tr><td>BigTranslate</td><td>70.27</td><td>14.59</td><td>83.45</td><td>23.50</td><td>79.46</td><td>19.04</td><td>80.60</td><td>30.90</td><td>79.36</td><td>26.43</td></tr>
  <tr><td>Aya-23-SB</td><td>82.27</td><td>34.13</td><td>87.25</td><td>32.02</td><td>83.92</td><td>26.41</td><td>84.03</td><td>41.39</td><td>84.29</td><td>41.24</td></tr>
  <tr><td>TowerInstruct-7B</td><td>83.10</td><td>35.00</td><td>78.88</td><td>17.41</td><td>85.39</td><td>29.81</td><td>85.93</td><td>42.66</td><td>85.15</td><td>44.33</td></tr>
  <tr><td>Bayling-13B</td><td>70.03</td><td>15.97</td><td>68.15</td><td>14.34</td><td>64.22</td><td>12.38</td><td>81.88</td><td>37.89</td><td>81.39</td><td>27.85</td></tr>
  <tr><td>NMT-8-8</td><td>77.18</td><td>29.16</td><td>85.54</td><td>35.94</td><td>79.48</td><td>25.43</td><td>80.29</td><td>38.78</td><td>81.33</td><td>33.33</td></tr>
  <tr><td>NMT-40-8</td><td>78.42</td><td>31.81</td><td>86.91</td><td>38.04</td><td>80.65</td><td>26.07</td><td>82.05</td><td>40.73</td><td>82.85</td><td>36.99</td></tr>
  <tr><td>mT5-Large</td><td>78.83</td><td>30.74</td><td>84.22</td><td>30.08</td><td>81.62</td><td>24.10</td><td>83.59</td><td>36.89</td><td>82.93</td><td>37.12</td></tr>
  <tr><td>Llama3-SB-Base, 3-shot</td><td>73.82</td><td>18.61</td><td>81.63</td><td>24.34</td><td>80.17</td><td>23.20</td><td>81.89</td><td>39.51</td><td>80.45</td><td>32.84</td></tr>
  <tr><td>Llama3-SB-Inst, 0-shot</td><td>78.81</td><td>29.01</td><td>77.46</td><td>22.29</td><td>80.21</td><td>22.67</td><td>67.46</td><td>18.51</td><td>74.40</td><td>33.20</td></tr>
  <tr><td>Llama3-SB-SFT</td><td>82.66</td><td>37.76</td><td>85.80</td><td>29.64</td><td>83.94</td><td>27.03</td><td>83.72</td><td>40.66</td><td>83.76</td><td>40.62</td></tr>
  <tr><td>LaMaTE-s1</td><td>77.73</td><td>29.96</td><td>85.31</td><td>33.22</td><td>79.19</td><td>24.16</td><td>80.88</td><td>39.61</td><td>82.12</td><td>33.39</td></tr>
  <tr><td>LaMaTE-s2</td><td>80.68</td><td>37.59</td><td>86.54</td><td>35.34</td><td>82.45</td><td>26.93</td><td>84.05</td><td>45.98</td><td>83.75</td><td>41.41</td></tr>
</table>

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Ru2En</th>
    <th colspan="2">Zh2En</th>
    <th colspan="2">Avg. En → X</th>
    <th colspan="2">Avg. X → En</th>
  </tr>
  <tr>
    <th>COMET</th><th>BLEU</th>
    <th>COMET</th><th>BLEU</th>
    <th>COMET</th><th>BLEU</th>
    <th>COMET</th><th>BLEU</th>
  </tr>
  <tr><td>NLLB-3.3B</td><td>80.70</td><td>31.59</td><td>77.44</td><td>22.14</td><td>82.77</td><td>33.54</td><td>79.83</td><td>29.72</td></tr>
  <tr><td>ALMA-7B</td><td>82.46</td><td>31.52</td><td>79.45</td><td>22.77</td><td>84.28</td><td>31.33</td><td>82.18</td><td>31.66</td></tr>
  <tr><td>BigTranslate</td><td>76.84</td><td>23.43</td><td>75.48</td><td>16.29</td><td>78.45</td><td>22.03</td><td>77.23</td><td>22.05</td></tr>
  <tr><td>Aya-23-SB</td><td>81.38</td><td>29.92</td><td>77.54</td><td>21.03</td><td>84.37</td><td>33.49</td><td>81.07</td><td>30.73</td></tr>
  <tr><td>TowerInstruct-7B</td><td>83.18</td><td>34.54</td><td>80.36</td><td>23.99</td><td>83.33</td><td>31.22</td><td>82.90</td><td>34.29</td></tr>
  <tr><td>Bayling-13B</td><td>77.34</td><td>21.12</td><td>76.89</td><td>18.59</td><td>71.07</td><td>20.15</td><td>78.54</td><td>22.52</td></tr>
  <tr><td>NMT-8-8</td><td>78.84</td><td>29.13</td><td>76.15</td><td>21.04</td><td>80.62</td><td>32.33</td><td>78.77</td><td>27.83</td></tr>
  <tr><td>NMT-40-8</td><td>79.83</td><td>30.28</td><td>76.63</td><td>22.56</td><td>82.01</td><td>34.16</td><td>79.77</td><td>29.94</td></tr>
  <tr><td>mT5-Large</td><td>80.42</td><td>27.85</td><td>77.97</td><td>19.73</td><td>82.07</td><td>30.45</td><td>80.44</td><td>28.23</td></tr>
  <tr><td>Llama3-SB-Base, 3-shot</td><td>80.17</td><td>30.14</td><td>75.38</td><td>19.08</td><td>79.38</td><td>26.42</td><td>78.67</td><td>27.35</td></tr>
  <tr><td>Llama3-SB-Inst, 0-shot</td><td>70.55</td><td>21.49</td><td>68.44</td><td>15.12</td><td>75.99</td><td>23.12</td><td>71.13</td><td>23.27</td></tr>
  <tr><td>Llama3-SB-SFT</td><td>81.66</td><td>31.88</td><td>76.97</td><td>20.70</td><td>84.03</td><td>33.77</td><td>80.80</td><td>31.07</td></tr>
  <tr><td>LaMaTE-s1</td><td>79.56</td><td>29.18</td><td>76.40</td><td>20.22</td><td>80.78</td><td>31.74</td><td>79.36</td><td>27.60</td></tr>
  <tr><td>LaMaTE-s2</td><td>80.97</td><td>30.29</td><td>78.87</td><td>22.02</td><td>83.43</td><td>36.46</td><td>81.20</td><td>31.24</td></tr>
</table>

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Medical</th>
    <th colspan="2">Law</th>
    <th colspan="2">IT</th>
  </tr>
  <tr>
    <th>De2En</th><th>En2De</th>
    <th>Ru2En</th><th>En2Ru</th>
    <th>En2De</th><th>En2Cs</th>
    <th>Ru2En</th><th>En2Cs</th>
  </tr>
  <tr><td>NLLB-3.3B</td><td>87.32</td><td>86.65</td><td>84.49</td><td>88.71</td><td>84.29</td><td>85.29</td><td>88.20</td><td>86.89</td><td>89.88</td><td>86.81</td><td>92.09</td></tr>
  <tr><td>ALMA-7B</td><td>87.06</td><td>86.10</td><td>84.56</td><td>88.02</td><td>84.83</td><td>85.66</td><td>87.57</td><td>85.45</td><td>88.94</td><td>86.60</td><td>92.35</td></tr>
  <tr><td>BigTranslate</td><td>84.79</td><td>82.63</td><td>81.04</td><td>80.14</td><td>81.70</td><td>84.17</td><td>85.61</td><td>83.26</td><td>88.03</td><td>78.57</td><td>84.08</td></tr>
  <tr><td>Aya-23-SB</td><td>87.15</td><td>85.88</td><td>84.08</td><td>88.04</td><td>84.82</td><td>86.49</td><td>87.71</td><td>86.31</td><td>89.24</td><td>87.19</td><td>92.97</td></tr>
  <tr><td>TowerInstruct-7B</td><td>87.51</td><td>86.89</td><td>85.44</td><td>87.77</td><td>85.40</td><td>87.08</td><td>87.15</td><td>86.27</td><td>82.14</td><td>85.27</td><td>84.86</td></tr>
  <tr><td>Bayling-13B</td><td>85.62</td><td>80.11</td><td>82.47</td><td>64.29</td><td>83.00</td><td>85.03</td><td>84.12</td><td>75.39</td><td>70.65</td><td>66.26</td><td>75.52</td></tr>
  <tr><td>mT5-Large</td><td>86.37</td><td>85.01</td><td>83.24</td><td>86.63</td><td>82.90</td><td>85.86</td><td>86.56</td><td>84.16</td><td>86.85</td><td>84.93</td><td>91.49</td></tr>
  <tr><td>NMT-8-8</td><td>87.12</td><td>86.69</td><td>83.85</td><td>86.93</td><td>83.61</td><td>85.46</td><td>87.76</td><td>87.98</td><td>91.14</td><td>83.17</td><td>92.73</td></tr>
  <tr><td>NMT-40-8</td><td>87.50</td><td>87.03</td><td>84.74</td><td>87.79</td><td>84.42</td><td>86.09</td><td>88.21</td><td>88.28</td><td>91.78</td><td>83.03</td><td>93.25</td></tr>
  <tr><td>Llama3-SB-Base, 3-shot</td><td>85.14</td><td>83.32</td><td>83.11</td><td>84.61</td><td>82.46</td><td>83.96</td><td>86.57</td><td>79.46</td><td>82.13</td><td>84.43</td><td>88.48</td></tr>
  <tr><td>Llama3-SB-Inst, 0-shot</td><td>73.47</td><td>84.79</td><td>74.20</td><td>85.72</td><td>74.58</td><td>75.20</td><td>66.78</td><td>83.59</td><td>81.95</td><td>84.20</td><td>86.25</td></tr>
  <tr><td>Llama3-SB-SFT</td><td>86.65</td><td>85.86</td><td>83.95</td><td>88.94</td><td>84.85</td><td>87.69</td><td>88.07</td><td>85.53</td><td>88.18</td><td>88.88</td><td>92.84</td></tr>
  <tr><td>LaMaTE</td><td>87.10</td><td>86.39</td><td>84.36</td><td>88.95</td><td>84.36</td><td>88.73</td><td>87.68</td><td>87.70</td><td>91.36</td><td>89.48</td><td>93.86</td></tr>
</table>
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">Colloquial</th>
    <th colspan="6">Literature</th>
    <th colspan="2">Avg. COMET</th>
    <th colspan="2">Avg. COMET</th>
  </tr>
  <tr>
    <th>De2En</th><th>En2De</th><th>Ru2En</th><th>En2Ru</th><th>Zh2En</th><th>En2Zh</th>
    <th>De2En</th><th>Ru2En</th><th>Zh2En</th><th>En2Zh</th>
    <th>En → X</th><th>X → En</th>
    <th>En → X</th><th>X → En</th>
  </tr>
  <tr>
    <td>NLLB-3.3B</td><td>87.09</td><td>81.32</td><td>82.75</td><td>78.94</td><td>80.69</td><td>85.04</td><td>63.65</td><td>65.98</td><td>59.98</td><td>65.94</td><td>84.32</td><td>78.44</td><td>85.76</td><td>82.09</td>
  </tr>
  <tr>
    <td>ALMA-7B</td><td>88.65</td><td>82.55</td><td>85.86</td><td>84.26</td><td>83.13</td><td>86.86</td><td>74.75</td><td>72.17</td><td>72.27</td><td>76.54</td><td></td><td></td><td>85.76</td><td>82.09</td>
  </tr>
  <tr>
    <td>BigTranslate</td><td>82.68</td><td>69.91</td><td>74.72</td><td>66.70</td><td>79.57</td><td>82.05</td><td>67.78</td><td>57.47</td><td>62.22</td><td>72.72</td><td>79.30</td><td>75.76</td><td></td><td></td>
  </tr>
  <tr>
    <td>Aya-23-SB</td><td>89.18</td><td>83.50</td><td>86.61</td><td>84.90</td><td>83.18</td><td>87.56</td><td>74.77</td><td>72.45</td><td>68.68</td><td>77.01</td><td>86.28</td><td>81.86</td><td></td><td></td>
  </tr>
  <tr>
    <td>TowerInstruct-7B</td><td>88.48</td><td>81.73</td><td>86.66</td><td>82.12</td><td>82.45</td><td>86.79</td><td>75.44</td><td>73.29</td><td>75.38</td><td>79.37</td><td>84.57</td><td>82.72</td><td></td><td></td>
  </tr>
  <tr>
    <td>Bayling-13B</td><td>85.94</td><td>71.40</td><td>81.97</td><td>59.95</td><td>80.22</td><td>82.73</td><td>73.92</td><td>69.36</td><td>71.30</td><td>73.87</td><td>73.20</td><td>79.79</td><td></td><td></td>
  </tr>
  <tr>
    <td>mT5-Large</td><td>87.08</td><td>78.05</td><td>83.92</td><td>80.69</td><td>80.73</td><td>86.85</td><td>71.22</td><td>69.09</td><td>68.30</td><td>75.36</td><td>84.17</td><td>79.94</td><td></td><td></td>
  </tr>
  <tr>
    <td>NMT-8.8</td><td>87.79</td><td>78.19</td><td>83.20</td><td>77.54</td><td>79.55</td><td>84.52</td><td>62.26</td><td>64.20</td><td>51.11</td><td>71.86</td><td>84.20</td><td>77.05</td><td></td><td></td>
  </tr>
  <tr>
    <td>NMT-40.8</td><td>88.98</td><td>79.76</td><td>84.45</td><td>79.28</td><td>80.53</td><td>86.09</td><td>63.82</td><td>66.56</td><td>48.39</td><td>75.10</td><td>85.23</td><td>77.76</td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3-SB-Base, 3-shot</td><td>86.74</td><td>75.46</td><td>85.39</td><td>80.81</td><td>78.91</td><td>79.51</td><td>75.56</td><td>72.82</td><td>70.58</td><td>76.32</td><td>81.68</td><td>80.73</td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3-SB-Inst, 0-shot</td><td>61.97</td><td>73.28</td><td>62.26</td><td>81.34</td><td>57.89</td><td>58.92</td><td>70.70</td><td>69.35</td><td>72.04</td><td>60.14</td><td>77.76</td><td>68.32</td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3-SB-SFT</td><td>89.20</td><td>85.02</td><td>86.75</td><td>85.46</td><td>81.00</td><td>84.73</td><td>77.33</td><td>75.62</td><td>72.26</td><td>79.75</td><td>86.63</td><td>82.57</td><td></td><td></td>
  </tr>
  <tr>
    <td>LaMaTE</td><td>88.76</td><td>83.01</td><td>85.58</td><td>82.84</td><td>82.75</td><td>87.53</td><td>75.17</td><td>72.15</td><td>74.54</td><td>78.51</td><td>87.12</td><td>82.25</td><td></td><td></td>
  </tr>
</table>

Domain Translation(BLEU)

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">Medical</th>
    <th colspan="6">Law</th>
    <th colspan="2">IT</th>
  </tr>
  <tr>
    <th>De2En</th><th>En2De</th><th>Ru2En</th><th>En2Ru</th><th>Zh2En</th><th>En2Zh</th>
    <th>De2En</th><th>En2De</th><th>Ru2En</th><th>Zh2En</th><th>En2Zh</th>
    <th>En2Cs</th><th>En2Cs</th>
  </tr>
  <tr>
    <td>NLLB-3.3B</td><td>42.09</td><td>33.26</td><td>44.36</td><td>33.54</td><td>31.29</td><td>36.81</td><td>46.54</td><td>48.74</td><td>50.43</td><td>35.83</td><td>33.94</td><td></td><td></td>
  </tr>
  <tr>
    <td>ALMA-7B</td><td>41.25</td><td>30.20</td><td>41.06</td><td>28.97</td><td>33.32</td><td>36.75</td><td>42.46</td><td>36.61</td><td>35.95</td><td>33.21</td><td>31.69</td><td></td><td></td>
  </tr>
  <tr>
    <td>BigTranslate</td><td>32.69</td><td>22.00</td><td>29.48</td><td>18.81</td><td>23.56</td><td>31.56</td><td>37.05</td><td>30.83</td><td>37.70</td><td>20.34</td><td>20.5</td><td></td><td></td>
  </tr>
  <tr>
    <td>Aya-23-SB</td><td>41.82</td><td>30.05</td><td>41.01</td><td>28.83</td><td>32.63</td><td>39.09</td><td>43.92</td><td>42.46</td><td>42.64</td><td>35.59</td><td>34.20</td><td></td><td></td>
  </tr>
  <tr>
    <td>TowerInstruct-7B</td><td>43.83</td><td>33.71</td><td>46.31</td><td>32.37</td><td>36.13</td><td>42.04</td><td>43.18</td><td>41.83</td><td>19.25</td><td>33.04</td><td>19.03</td><td></td><td></td>
  </tr>
  <tr>
    <td>Bayling-13B</td><td>35.04</td><td>23.28</td><td>32.82</td><td>13.71</td><td>27.43</td><td>36.40</td><td>31.26</td><td>25.50</td><td>18.74</td><td>19.31</td><td>15.14</td><td></td><td></td>
  </tr>
  <tr>
    <td>mT5-Large</td><td>38.44</td><td>27.88</td><td>36.80</td><td>27.38</td><td>28.16</td><td>34.86</td><td>38.61</td><td>39.24</td><td>39.42</td><td>34.08</td><td>33.81</td><td></td><td></td>
  </tr>
  <tr>
    <td>NMT-8.8</td><td>43.71</td><td>34.21</td><td>42.18</td><td>31.68</td><td>33.36</td><td>38.55</td><td>43.68</td><td>49.85</td><td>52.75</td><td>34.06</td><td>38.86</td><td></td><td></td>
  </tr>
  <tr>
    <td>NMT-40.8</td><td>44.14</td><td>35.43</td><td>43.40</td><td>32.48</td><td>35.28</td><td>39.82</td><td>46.12</td><td>51.68</td><td>54.17</td><td>33.26</td><td>40.33</td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3-SB-Base, 3-shot</td><td>37.89</td><td>25.20</td><td>37.57</td><td>24.07</td><td>29.33</td><td>30.84</td><td>39.71</td><td>30.64</td><td>26.62</td><td>31.98</td><td>23.74</td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3-SB-Inst, 0-shot</td><td>27.21</td><td>28.33</td><td>28.12</td><td>26.47</td><td>22.31</td><td>18.24</td><td>22.53</td><td>35.31</td><td>29.54</td><td>32.75</td><td>25.40</td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3-SB-SFT</td><td>41.70</td><td>30.53</td><td>42.70</td><td>36.18</td><td>36.24</td><td>45.76</td><td>49.07</td><td>41.08</td><td>39.90</td><td>40.96</td><td>36.77</td><td></td><td></td>
  </tr>
  <tr>
    <td>LaMaTE</td><td>41.98</td><td>32.74</td><td>42.04</td><td>37.64</td><td>32.76</td><td>48.97</td><td>45.16</td><td>47.96</td><td>50.83</td><td>42.23</td><td>45.64</td><td></td><td></td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">Colloquial</th>
    <th colspan="6">Literature</th>
    <th colspan="2">Avg. BLEU</th>
    <th colspan="2">Avg. BLEU</th>
  </tr>
  <tr>
    <th>De2En</th><th>En2De</th><th>Ru2En</th><th>En2Ru</th><th>Zh2En</th><th>En2Zh</th>
    <th>De2En</th><th>Ru2En</th><th>Zh2En</th><th>En2Zh</th>
    <th>En → X</th><th>X → En</th>
    <th>En → X</th><th>X → En</th>
  </tr>
  <tr>
    <td>NLLB-3.3B</td><td>48.09</td><td>42.90</td><td>32.44</td><td>20.69</td><td>20.94</td><td>33.72</td><td>14.51</td><td>20.69</td><td>5.24</td><td>11.76</td><td>34.69</td><td>30.62</td><td></td><td></td>
  </tr>
  <tr>
    <td>ALMA-7B</td><td>49.19</td><td>38.16</td><td>37.63</td><td>26.41</td><td>22.20</td><td>32.15</td><td>22.77</td><td>24.04</td><td>12.13</td><td>13.88</td><td>31.27</td><td>32.61</td><td></td><td></td>
  </tr>
  <tr>
    <td>BigTranslate</td><td>32.92</td><td>20.54</td><td>19.81</td><td>8.45</td><td>15.15</td><td>24.08</td><td>14.91</td><td>7.68</td><td>3.74</td><td>10.05</td><td>22.27</td><td>21.70</td><td></td><td></td>
  </tr>
  <tr>
    <td>Aya-23-SB</td><td>50.86</td><td>42.54</td><td>40.04</td><td>28.84</td><td>21.36</td><td>35.65</td><td>23.65</td><td>25.38</td><td>10.57</td><td>16.48</td><td>34.22</td><td>33.12</td><td></td><td></td>
  </tr>
  <tr>
    <td>TowerInstruct-7B</td><td>53.77</td><td>42.61</td><td>39.68</td><td>26.95</td><td>22.65</td><td>34.11</td><td>24.35</td><td>27.79</td><td>15.06</td><td>18.49</td><td>31.22</td><td>35.28</td><td></td><td></td>
  </tr>
  <tr>
    <td>Bayling-13B</td><td>43.73</td><td>28.55</td><td>27.45</td><td>12.25</td><td>20.46</td><td>30.57</td><td>20.23</td><td>18.10</td><td>9.61</td><td>13.41</td><td>21.53</td><td>26.61</td><td></td><td></td>
  </tr>
  <tr>
    <td>mT5-Large</td><td>49.23</td><td>36.53</td><td>32.27</td><td>24.05</td><td>21.39</td><td>31.40</td><td>19.02</td><td>20.99</td><td>9.37</td><td>13.18</td><td>31.08</td><td>29.43</td><td></td><td></td>
  </tr>
  <tr>
    <td>NMT-8.8</td><td>50.55</td><td>41.63</td><td>32.31</td><td>23.74</td><td>21.32</td><td>33.79</td><td>11.61</td><td>16.73</td><td>4.74</td><td>11.34</td><td>35.50</td><td>30.02</td><td></td><td></td>
  </tr>
  <tr>
    <td>NMT-40.8</td><td>54.16</td><td>43.43</td><td>34.32</td><td>24.53</td><td>21.83</td><td>33.75</td><td>13.06</td><td>18.45</td><td>4.60</td><td>13.52</td><td>36.58</td><td>31.54</td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3-SB-Base, 3-shot</td><td>46.09</td><td>28.92</td><td>36.14</td><td>23.83</td><td>18.16</td><td>25.50</td><td>24.24</td><td>25.13</td><td>11.30</td><td>14.75</td><td>26.01</td><td>30.56</td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3-SB-Inst, 0-shot</td><td>22.55</td><td>32.94</td><td>16.55</td><td>25.04</td><td>9.10</td><td>11.21</td><td>21.02</td><td>24.38</td><td>11.75</td><td>7.75</td><td>24.82</td><td>20.55</td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3-SB-SFT</td><td>55.53</td><td>49.83</td><td>41.79</td><td>31.01</td><td>22.28</td><td>28.71</td><td>30.50</td><td>36.33</td><td>16.36</td><td>21.76</td><td>36.59</td><td>37.25</td><td></td><td></td>
  </tr>
  <tr>
    <td>LaMaTE</td><td>53.41</td><td>48.91</td><td>36.94</td><td>29.80</td><td>24.42</td><td>38.41</td><td>25.60</td><td>26.40</td><td>16.49</td><td>21.82</td><td>40.45</td><td>34.52</td><td></td><td></td>
  </tr>
</table>
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Doc-level Translation</th>
    <th colspan="4"></th>
    <th colspan="4"></th>
  </tr>
  <tr>
    <th colspan="2">De2En</th>
    <th colspan="2">En2De</th>
    <th colspan="2">En2Ru</th>
    <th colspan="2">Zh2En</th>
    <th colspan="2">En2Zh</th>
  </tr>
  <tr>
    <th></th>
    <th>d-COMET</th>
    <th>d-BLEU</th>
    <th>d-COMET</th>
    <th>d-BLEU</th>
    <th>d-COMET</th>
    <th>d-BLEU</th>
    <th>d-COMET</th>
    <th>d-BLEU</th>
    <th>d-COMET</th>
    <th>d-BLEU</th>
  </tr>
  <tr><td>NLLB-3.3B</td><td>74.01</td><td>11.75</td><td>73.19</td><td>14.56</td><td>78.10</td><td>7.37</td><td>68.34</td><td>11.14</td><td>70.17</td><td>18.58</td></tr>
  <tr><td>ALMA-7B</td><td>83.73</td><td>25.58</td><td>81.10</td><td>20.14</td><td>85.93</td><td>14.29</td><td>76.53</td><td>20.97</td><td>80.56</td><td>16.26</td></tr>
  <tr><td>BigTranslate</td><td>74.77</td><td>9.04</td><td>68.47</td><td>9.80</td><td>71.34</td><td>5.61</td><td>68.01</td><td>8.48</td><td>80.24</td><td>22.46</td></tr>
  <tr><td>Aya-23-8B</td><td>86.10</td><td>39.87</td><td>85.19</td><td>28.92</td><td>89.15</td><td>25.79</td><td>76.61</td><td>28.05</td><td>82.31</td><td>33.04</td></tr>
  <tr><td>TowerInstruct-7B</td><td>85.86</td><td>37.74</td><td>83.41</td><td>25.61</td><td>80.93</td><td>18.57</td><td>82.53</td><td>38.88</td><td>81.47</td><td>22.22</td></tr>
  <tr><td>Bayling-13B</td><td>84.71</td><td>30.56</td><td>72.55</td><td>17.50</td><td>45.83</td><td>4.82</td><td>78.62</td><td>23.69</td><td>79.92</td><td>20.90</td></tr>
  <tr><td>mT5-Large</td><td>79.29</td><td>17.04</td><td>71.22</td><td>10.69</td><td>78.99</td><td>7.28</td><td>74.71</td><td>18.11</td><td>80.97</td><td>22.73</td></tr>
  <tr><td>NMT-8-8</td><td>74.80</td><td>10.94</td><td>71.76</td><td>9.78</td><td>74.49</td><td>3.58</td><td>69.76</td><td>16.00</td><td>74.33</td><td>11.18</td></tr>
  <tr><td>NMT-40-8</td><td>76.69</td><td>14.04</td><td>71.40</td><td>11.78</td><td>74.33</td><td>4.72</td><td>69.89</td><td>17.26</td><td>75.75</td><td>13.37</td></tr>
  <tr><td>Llama3-8B-Base, 3-shot</td><td>34.42</td><td>81.76</td><td>19.31</td><td>75.99</td><td>15.26</td><td>81.27</td><td>29.39</td><td>76.52</td><td>31.71</td><td>83.67</td></tr>
  <tr><td>Llama3-8B-Inst, 0-shot</td><td>83.01</td><td>33.97</td><td>82.89</td><td>24.09</td><td>84.27</td><td>19.32</td><td>81.91</td><td>36.35</td><td>70.12</td><td>15.48</td></tr>
  <tr><td>Llama3-8B-SFT</td><td>84.87</td><td>35.07</td><td>81.62</td><td>26.35</td><td>85.25</td><td>21.78</td><td>80.62</td><td>40.85</td><td>86.25</td><td>38.08</td></tr>
  <tr><td>LaMaTE</td><td>83.52</td><td>32.69</td><td>82.17</td><td>27.45</td><td>87.02</td><td>22.56</td><td>82.68</td><td>39.65</td><td>83.50</td><td>37.56</td></tr>
</table>

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">En2Cs</th>
    <th colspan="4">Cs2En</th>
    <th colspan="2">Avg. En → X</th>
    <th colspan="2">Avg. X → En</th>
  </tr>
  <tr>
    <th>d-COMET</th>
    <th>d-BLEU</th>
    <th>d-COMET</th>
    <th>d-BLEU</th>
    <th>d-COMET</th>
    <th>d-BLEU</th>
    <th>d-COMET</th>
    <th>d-BLEU</th>
    <th>d-COMET</th>
    <th>d-BLEU</th>
    <th>d-COMET</th>
    <th>d-BLEU</th>
  </tr>
  <tr><td>NLLB-3.3B</td><td>78.43</td><td>10.41</td><td>74.34</td><td>9.24</td><td>74.97</td><td>12.73</td><td>72.23</td><td>10.71</td></tr>
  <tr><td>ALMA-7B</td><td>84.74</td><td>12.38</td><td>82.69</td><td>17.71</td><td>83.08</td><td>15.77</td><td>80.98</td><td>21.42</td></tr>
  <tr><td>BigTranslate</td><td>68.86</td><td>3.98</td><td>74.65</td><td>6.17</td><td>72.23</td><td>10.46</td><td>72.48</td><td>7.90</td></tr>
  <tr><td>Aya-23-8B</td><td>88.44</td><td>24.36</td><td>88.33</td><td>41.14</td><td>86.27</td><td>28.03</td><td>83.68</td><td>36.35</td></tr>
  <tr><td>TowerInstruct-7B</td><td>66.40</td><td>8.22</td><td>87.13</td><td>36.69</td><td>78.05</td><td>18.66</td><td>85.17</td><td>37.77</td></tr>
  <tr><td>Bayling-13B</td><td>62.70</td><td>7.80</td><td>85.48</td><td>24.23</td><td>65.25</td><td>12.76</td><td>82.94</td><td>26.16</td></tr>
  <tr><td>mT5-Large</td><td>77.80</td><td>7.24</td><td>80.39</td><td>15.45</td><td>77.25</td><td>11.99</td><td>78.13</td><td>16.87</td></tr>
  <tr><td>NMT-8-8</td><td>77.24</td><td>7.73</td><td>75.92</td><td>9.48</td><td>74.46</td><td>8.07</td><td>73.49</td><td>12.14</td></tr>
  <tr><td>NMT-40-8</td><td>78.54</td><td>10.54</td><td>76.81</td><td>12.75</td><td>75.01</td><td>10.10</td><td>74.46</td><td>14.68</td></tr>
  <tr><td>Llama3-8B-Base, 3-shot</td><td>13.67</td><td>77.69</td><td>36.83</td><td>85.34</td><td>19.99</td><td>79.66</td><td>33.55</td><td>81.21</td></tr>
  <tr><td>Llama3-8B-Inst, 0-shot</td><td>83.53</td><td>18.99</td><td>85.23</td><td>34.60</td><td>80.20</td><td>19.47</td><td>83.38</td><td>34.97</td></tr>
  <tr><td>Llama3-8B-SFT</td><td>83.21</td><td>19.49</td><td>84.98</td><td>35.28</td><td>84.08</td><td>26.43</td><td>83.49</td><td>37.07</td></tr>
  <tr><td>LaMaTE</td><td>86.57</td><td>22.92</td><td>85.36</td><td>34.95</td><td>84.82</td><td>27.62</td><td>83.85</td><td>35.76</td></tr>
</table>

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Terminology-constrained Translation</th>
    <th colspan="3"></th>
    <th colspan="3"></th>
    <th colspan="3"></th>
  </tr>
  <tr>
    <th>De2En</th>
    <th>En2De</th>
    <th>En2Ru</th>
    <th>Zh2En</th>
    <th>En2Zh</th>
  </tr>
  <tr>
    <th></th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>TSR</th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>TSR</th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>TSR</th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>TSR</th>
  </tr>
  <tr><td>NLLB-3.3B</td><td>79.50</td><td>21.18</td><td>36.03</td><td>89.59</td><td>48.37</td><td>77.52</td><td>89.77</td><td>30.02</td><td>57.83</td><td>65.76</td><td>8.74</td><td>8.91</td></tr>
  <tr><td>ALMA-7B</td><td>79.49</td><td>21.41</td><td>36.38</td><td>88.49</td><td>40.64</td><td>69.95</td><td>88.98</td><td>24.98</td><td>55.01</td><td>73.29</td><td>12.33</td><td>24.08</td></tr>
  <tr><td>BigTranslate</td><td>77.88</td><td>18.06</td><td>32.25</td><td>83.85</td><td>29.16</td><td>58.41</td><td>81.29</td><td>17.44</td><td>40.16</td><td>66.59</td><td>7.16</td><td>12.55</td></tr>
  <tr><td>Aya-23-8B</td><td>80.95</td><td>31.71</td><td>67.25</td><td>83.35</td><td>39.86</td><td>85.78</td><td>86.17</td><td>27.29</td><td>70.55</td><td>71.99</td><td>13.40</td><td>71.18</td></tr>
  <tr><td>TowerInstruct-7B</td><td>81.35</td><td>31.09</td><td>66.27</td><td>88.85</td><td>46.57</td><td>87.54</td><td>89.39</td><td>30.85</td><td>73.50</td><td>73.28</td><td>15.43</td><td>78.84</td></tr>
  <tr><td>Bayling-13B</td><td>81.33</td><td>29.76</td><td>70.28</td><td>81.58</td><td>33.65</td><td>84.10</td><td>66.44</td><td>12.46</td><td>51.24</td><td>70.96</td><td>12.03</td><td>74.29</td></tr>
  <tr><td>mT5-large</td><td>81.74</td><td>33.70</td><td>76.34</td><td>87.50</td><td>42.76</td><td>88.30</td><td>87.79</td><td>25.71</td><td>66.55</td><td>71.20</td><td>12.44</td><td>76.15</td></tr>
  <tr><td>NMT-8-8</td><td>78.80</td><td>21.26</td><td>35.62</td><td>89.37</td><td>47.79</td><td>77.14</td><td>87.34</td><td>27.85</td><td>55.24</td><td>55.58</td><td>6.16</td><td>8.53</td></tr>
  <tr><td>NMT-40-8</td><td>79.35</td><td>21.75</td><td>36.56</td><td>89.45</td><td>47.99</td><td>76.83</td><td>88.64</td><td>27.92</td><td>55.24</td><td>54.37</td><td>5.97</td><td>8.80</td></tr>
  <tr><td>Llama3-8B-Base, 3-shot</td><td>81.73</td><td>34.34</td><td>77.08</td><td>85.18</td><td>38.04</td><td>88.30</td><td>86.72</td><td>22.60</td><td>71.85</td><td>73.40</td><td>13.87</td><td>76.49</td></tr>
  <tr><td>Llama3-8B-Inst, 0-shot</td><td>64.85</td><td>20.68</td><td>84.67</td><td>85.99</td><td>40.56</td><td>95.03</td><td>84.68</td><td>24.76</td><td>80.92</td><td>57.72</td><td>7.96</td><td>83.35</td></tr>
  <tr><td>Llama3-8B-SFT</td><td>82.53</td><td>37.28</td><td>87.15</td><td>89.85</td><td>49.62</td><td>94.27</td><td>89.84</td><td>31.70</td><td>84.69</td><td>74.56</td><td>17.06</td><td>82.18</td></tr>
  <tr><td>LaMaTE</td><td>81.83</td><td>33.03</td><td>71.05</td><td>90.17</td><td>52.12</td><td>90.90</td><td>89.53</td><td>32.59</td><td>72.79</td><td>72.71</td><td>13.91</td><td>75.05</td></tr>
</table>
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">En2Zh</th>
    <th colspan="3">En2Cs</th>
    <th colspan="3">Avg. En → X</th>
    <th colspan="3">Avg. X → En</th>
  </tr>
  <tr>
    <th>COMET</th><th>BLEU</th><th>TSR</th>
    <th>COMET</th><th>BLEU</th><th>TSR</th>
    <th>COMET</th><th>BLEU</th><th>TSR</th>
    <th>COMET</th><th>BLEU</th><th>TSR</th>
  </tr>
  <tr><td>NLLB-3.3B</td><td>85.89</td><td>40.52</td><td>64.84</td><td>85.89</td><td>31.77</td><td>39.87</td><td>87.79</td><td>37.67</td><td>60.02</td><td>72.63</td><td>14.96</td><td>22.47</td></tr>
  <tr><td>ALMA-7B</td><td>85.95</td><td>38.38</td><td>62.56</td><td>85.77</td><td>27.63</td><td>37.15</td><td>87.30</td><td>32.91</td><td>56.17</td><td>76.39</td><td>16.87</td><td>30.23</td></tr>
  <tr><td>BigTranslate</td><td>83.19</td><td>32.19</td><td>50.24</td><td>80.99</td><td>19.24</td><td>26.39</td><td>82.33</td><td>24.51</td><td>43.80</td><td>72.24</td><td>12.61</td><td>22.40</td></tr>
  <tr><td>Aya-23-8B</td><td>87.37</td><td>46.84</td><td>88.37</td><td>88.95</td><td>44.92</td><td>71.59</td><td>86.46</td><td>39.73</td><td>79.07</td><td>76.47</td><td>22.56</td><td>69.22</td></tr>
  <tr><td>TowerInstruct-7B</td><td>87.42</td><td>48.68</td><td>90.45</td><td>85.67</td><td>35.56</td><td>74.10</td><td>87.83</td><td>40.42</td><td>81.40</td><td>77.32</td><td>23.26</td><td>72.56</td></tr>
  <tr><td>Bayling-13B</td><td>85.46</td><td>42.96</td><td>92.87</td><td>77.95</td><td>27.96</td><td>58.11</td><td>77.86</td><td>29.26</td><td>71.58</td><td>76.15</td><td>20.90</td><td>72.29</td></tr>
  <tr><td>mT5-Large</td><td>86.40</td><td>40.66</td><td>85.81</td><td>86.79</td><td>41.25</td><td>69.73</td><td>87.12</td><td>37.60</td><td>77.60</td><td>76.47</td><td>23.07</td><td>76.25</td></tr>
  <tr><td>NMT-8-8</td><td>84.48</td><td>38.79</td><td>60.07</td><td>86.25</td><td>35.04</td><td>43.65</td><td>86.86</td><td>37.37</td><td>59.03</td><td>67.19</td><td>13.71</td><td>22.08</td></tr>
  <tr><td>NMT-40-8</td><td>85.10</td><td>39.34</td><td>61.52</td><td>86.83</td><td>35.70</td><td>44.88</td><td>87.51</td><td>37.74</td><td>59.62</td><td>66.86</td><td>13.86</td><td>22.68</td></tr>
  <tr><td>Llama3-8B-Base, 3-shot</td><td>85.62</td><td>40.65</td><td>90.59</td><td>86.68</td><td>40.98</td><td>75.64</td><td>86.05</td><td>35.57</td><td>81.60</td><td>77.57</td><td>24.11</td><td>76.79</td></tr>
  <tr><td>Llama3-8B-Inst, 0-shot</td><td>72.46</td><td>32.78</td><td>95.36</td><td>82.60</td><td>39.90</td><td>84.29</td><td>81.43</td><td>34.50</td><td>88.90</td><td>61.29</td><td>14.32</td><td>84.01</td></tr>
  <tr><td>Llama3-8B-SFT</td><td>89.44</td><td>56.76</td><td>96.06</td><td>88.69</td><td>47.34</td><td>86.45</td><td>89.46</td><td>46.36</td><td>90.37</td><td>78.55</td><td>27.17</td><td>84.67</td></tr>
  <tr><td>LaMaTE</td><td>89.72</td><td>58.44</td><td>87.68</td><td>88.29</td><td>41.01</td><td>61.54</td><td>89.43</td><td>46.04</td><td>78.23</td><td>77.27</td><td>23.47</td><td>73.05</td></tr>
</table>

Automatic Post-edition

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">De2En</th>
    <th colspan="3">En2De</th>
    <th colspan="3">Ru2En</th>
    <th colspan="3">En2Ru</th>
  </tr>
  <tr>
    <th>COMET</th><th>BLEU</th><th>HTER</th>
    <th>COMET</th><th>BLEU</th><th>HTER</th>
    <th>COMET</th><th>BLEU</th><th>HTER</th>
    <th>COMET</th><th>BLEU</th><th>HTER</th>
  </tr>
  <tr><td>NLLB-3.3B</td><td>89.18</td><td>52.90</td><td>38.94</td><td>85.99</td><td>51.55</td><td>38.70</td><td>85.53</td><td>48.71</td><td>42.47</td><td>85.92</td><td>40.49</td><td>53.89</td></tr>
  <tr><td>ALMA-7B</td><td>88.12</td><td>45.56</td><td>44.08</td><td>84.24</td><td>42.71</td><td>46.84</td><td>86.74</td><td>47.01</td><td>42.97</td><td>85.93</td><td>35.14</td><td>58.49</td></tr>
  <tr><td>BigTranslate</td><td>85.70</td><td>42.44</td><td>51.04</td><td>75.70</td><td>24.55</td><td>68.72</td><td>78.43</td><td>30.62</td><td>59.87</td><td>77.21</td><td>21.55</td><td>75.29</td></tr>
  <tr><td>Aya-23-8B</td><td>89.08</td><td>75.51</td><td>22.83</td><td>84.71</td><td>63.41</td><td>29.53</td><td>90.93</td><td>75.46</td><td>19.22</td><td>88.49</td><td>54.35</td><td>40.16</td></tr>
  <tr><td>TowerInstruct-7B</td><td>88.98</td><td>80.62</td><td>17.06</td><td>85.64</td><td>71.84</td><td>20.87</td><td>91.64</td><td>81.49</td><td>14.41</td><td>91.13</td><td>75.32</td><td>19.14</td></tr>
  <tr><td>Bayling-13B</td><td>84.89</td><td>39.59</td><td>63.14</td><td>74.17</td><td>29.85</td><td>76.56</td><td>78.77</td><td>28.56</td><td>92.13</td><td>51.54</td><td>7.89</td><td>218.99</td></tr>
  <tr><td>mT5-large</td><td>88.34</td><td>73.97</td><td>21.57</td><td>85.15</td><td>70.13</td><td>22.17</td><td>89.25</td><td>70.37</td><td>26.23</td><td>81.60</td><td>35.31</td><td>86.20</td></tr>
  <tr><td>NMT-8-8</td><td>89.46</td><td>52.67</td><td>36.74</td><td>84.50</td><td>50.25</td><td>39.38</td><td>85.05</td><td>51.81</td><td>37.90</td><td>84.70</td><td>43.46</td><td>49.92</td></tr>
  <tr><td>NMT-40-8</td><td>89.60</td><td>53.55</td><td>36.43</td><td>84.88</td><td>50.39</td><td>39.55</td><td>86.10</td><td>53.23</td><td>36.46</td><td>85.12</td><td>39.98</td><td>52.69</td></tr>
  <tr><td>Llama3-8B-Base, 3-shot</td><td>88.15</td><td>61.72</td><td>33.26</td><td>84.06</td><td>59.78</td><td>32.26</td><td>86.97</td><td>66.00</td><td>29.09</td><td>88.84</td><td>62.54</td><td>32.92</td></tr>
  <tr><td>Llama3-8B-Inst, 0-shot</td><td>61.70</td><td>23.09</td><td>94.49</td><td>78.79</td><td>36.02</td><td>59.94</td><td>55.35</td><td>17.51</td><td>112.49</td><td>79.42</td><td>20.86</td><td>95.40</td></tr>
  <tr><td>Llama3-8B-SFT</td><td>90.08</td><td>78.05</td><td>19.53</td><td>86.56</td><td>66.46</td><td>26.00</td><td>90.34</td><td>75.15</td><td>20.76</td><td>91.36</td><td>69.48</td><td>25.77</td></tr>
  <tr><td>LaMaTE</td><td>90.50</td><td>76.86</td><td>21.03</td><td>86.07</td><td>64.63</td><td>25.80</td><td>91.04</td><td>75.13</td><td>19.88</td><td>90.89</td><td>68.24</td><td>26.72</td></tr>
</table>

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">En2Zh</th>
    <th colspan="3">Avg. En → X</th>
    <th colspan="3">Avg. X → En</th>
  </tr>
  <tr>
    <th>COMET</th><th>BLEU</th><th>HTER</th>
    <th>COMET</th><th>BLEU</th><th>HTER</th>
    <th>COMET</th><th>BLEU</th><th>HTER</th>
  </tr>
  <tr><td>NLLB-3.3B</td><td>75.87</td><td>34.04</td><td>99.76</td><td>82.59</td><td>42.03</td><td>64.12</td><td>87.36</td><td>50.81</td><td>40.71</td></tr>
  <tr><td>ALMA-7B</td><td>79.05</td><td>28.71</td><td>90.65</td><td>83.07</td><td>35.52</td><td>65.33</td><td>87.43</td><td>46.29</td><td>43.52</td></tr>
  <tr><td>BigTranslate</td><td>73.91</td><td>23.21</td><td>103.89</td><td>75.61</td><td>23.10</td><td>82.63</td><td>82.07</td><td>36.53</td><td>55.45</td></tr>
  <tr><td>Aya-23-8B</td><td>82.84</td><td>63.29</td><td>84.56</td><td>85.35</td><td>60.35</td><td>51.42</td><td>90.01</td><td>75.49</td><td>21.03</td></tr>
  <tr><td>TowerInstruct-7B</td><td>85.73</td><td>74.12</td><td>53.72</td><td>87.50</td><td>73.76</td><td>31.24</td><td>90.31</td><td>81.06</td><td>15.74</td></tr>
  <tr><td>Bayling-13B</td><td>72.20</td><td>38.00</td><td>160.32</td><td>65.97</td><td>25.25</td><td>151.96</td><td>81.83</td><td>34.08</td><td>77.64</td></tr>
  <tr><td>mT5-Large</td><td>82.15</td><td>54.63</td><td>90.52</td><td>82.97</td><td>53.36</td><td>66.30</td><td>88.80</td><td>72.17</td><td>23.90</td></tr>
  <tr><td>NMT-8-8</td><td>75.99</td><td>34.69</td><td>99.98</td><td>81.73</td><td>42.80</td><td>63.09</td><td>87.26</td><td>52.24</td><td>37.32</td></tr>
  <tr><td>NMT-40-8</td><td>77.08</td><td>34.85</td><td>100.00</td><td>82.36</td><td>41.74</td><td>64.08</td><td>87.85</td><td>53.39</td><td>36.44</td></tr>
  <tr><td>Llama3-8B-Base, 3-shot</td><td>81.94</td><td>54.66</td><td>74.88</td><td>84.95</td><td>58.99</td><td>46.68</td><td>87.56</td><td>63.86</td><td>31.17</td></tr>
  <tr><td>Llama3-8B-Inst, 0-shot</td><td>74.51</td><td>36.41</td><td>145.27</td><td>77.57</td><td>31.10</td><td>100.20</td><td>58.53</td><td>20.30</td><td>103.49</td></tr>
  <tr><td>Llama3-8B-SFT</td><td>82.91</td><td>55.08</td><td>77.20</td><td>86.94</td><td>63.67</td><td>42.99</td><td>90.21</td><td>76.60</td><td>20.14</td></tr>
  <tr><td>LaMaTE</td><td>82.35</td><td>53.76</td><td>86.66</td><td>86.43</td><td>62.21</td><td>46.39</td><td>90.77</td><td>76.00</td><td>20.45</td></tr>
</table>

Table 14: Performance on ComMT benchmark.

<table>
  <tr>
    <th>Data Set</th>
    <th>Language</th>
    <th>Task</th>
  </tr>
  <tr>
    <td>WMT 14-23-news<sup>8</sup></td>
    <td>Cs, De, Ru, Zh</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>WMT 16-21-ape<sup>9</sup></td>
    <td>De, Ru, Zh</td>
    <td>APE</td>
  </tr>
  <tr>
    <td>WMT 20.22-chat<sup>10</sup></td>
    <td>De</td>
    <td>DT(colloquial)</td>
  </tr>
</table>

8https://www.statmt.org/wmt21/translation-task.html.
9https://www.statmt.org/wmt21/apetask.html.
10https://www.statmt.org/wmt20/chat-task.html.
<table>
  <tr>
    <th>Data Set</th>
    <th>Language</th>
    <th>Task</th>
  </tr>
  <tr>
    <td>WMT 14, 18-22-medical<sup>11</sup></td>
    <td>Cs, De, Ru, Zh</td>
    <td>DT(medical)</td>
  </tr>
  <tr>
    <td>WMT 16-it<sup>12</sup></td>
    <td>Cs, De</td>
    <td>DT(it)</td>
  </tr>
  <tr>
    <td>WMT 20-robustness<sup>13</sup></td>
    <td>De</td>
    <td>DT(colloquial)</td>
  </tr>
  <tr>
    <td>WMT 21, 23-terminology<sup>14</sup></td>
    <td>Cs, De, Ru, Zh</td>
    <td>TCT</td>
  </tr>
  <tr>
    <td>WMT 23-literary</td>
    <td>Zh</td>
    <td>DLT</td>
  </tr>
  <tr>
    <td>IWSLT 14-17<sup>15</sup></td>
    <td>Cs, De, Ru, Zh</td>
    <td>DLT</td>
  </tr>
  <tr>
    <td>IWSLT 16<sup>16</sup></td>
    <td>De</td>
    <td>APE</td>
  </tr>
  <tr>
    <td>IWSLT23_OPUS_OpenSubtitles<sup>17</sup></td>
    <td>De</td>
    <td>GT, DT(colloquial)</td>
  </tr>
  <tr>
    <td>news-commentary-v18<sup>18</sup></td>
    <td>Cs, De, Ru, Zh</td>
    <td>DCL</td>
  </tr>
  <tr>
    <td>GlobalVoices<sup>19</sup></td>
    <td>Cs, De, Ru</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>DiscoMT<sup>20</sup></td>
    <td>De</td>
    <td>DCL</td>
  </tr>
  <tr>
    <td>frmt (Riley et al., 2022)</td>
    <td>Zh</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>PAWS-X (Yang et al., 2019)</td>
    <td>De, Zh</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>XNLI-15way (Conneau et al., 2018)</td>
    <td>De, Ru, Zh</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>NTREX128 (Federmann et al., 2022)</td>
    <td>Cs, De, Ru, Zh</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>CommonMT (He et al., 2020)</td>
    <td>Zh</td>
    <td>GT, APE</td>
  </tr>
  <tr>
    <td>BMELD (Liang et al., 2021)</td>
    <td>Zh</td>
    <td>DT(colloquial)</td>
  </tr>
  <tr>
    <td>par3 (Karpinska et al., 2022b)</td>
    <td>Cs, De, Ru, Zh</td>
    <td>DLT, DT(literature)</td>
  </tr>
  <tr>
    <td>BWB (Jiang et al., 2023)</td>
    <td>Zh</td>
    <td>DLT, DT(literature)</td>
  </tr>
  <tr>
    <td>UM-corpus (Tian et al., 2014)</td>
    <td>Zh</td>
    <td>GT, DT(law, literature)</td>
  </tr>
  <tr>
    <td>mZPRT (Xu et al., 2022)</td>
    <td>Zh</td>
    <td>DLT, DT(colloquial, literature)</td>
  </tr>
  <tr>
    <td>tico19 (Anastasopoulos et al., 2020)</td>
    <td>Ru, Zh</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>FGraDA (Zhu et al., 2021)</td>
    <td>Zh</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>NLLB (Costa-jussà et al., 2022)</td>
    <td>Ru</td>
    <td>GT, DT(colloquial)</td>
  </tr>
  <tr>
    <td>MULTI30k (Elliott et al., 2016)</td>
    <td>Cs, De</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>FLORES-200 (Costa-jussà et al., 2022)</td>
    <td>Cs, De, Ru, Zh</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>localization-xml-mt (Hashimoto et al., 2019)</td>
    <td>De, Ru, Zh</td>
    <td>DT(it)</td>
  </tr>
  <tr>
    <td>DEMETR (Karpinska et al., 2022a)</td>
    <td>De, Ru, Zh</td>
    <td>APE</td>
  </tr>
</table>

<sup>11</sup>https://www.statmt.org/wmt21/biomedical-translation-task.html.
<sup>12</sup>https://statmt.org/wmt16/it-translation-task.html.
<sup>13</sup>https://statmt.org/wmt20/robustness.html.
<sup>14</sup>https://statmt.org/wmt21/terminology-task.html.
<sup>15</sup>https://wit3.fbk.eu/2014-01.
<sup>16</sup>https://wit3.fbk.eu/2016-02.
<sup>17</sup>https://iwslt.org/2024/subtitling.
<sup>18</sup>https://data.statmt.org/news-commentary/v18.1/training/.
<sup>19</sup>https://opus.nlpl.eu/GlobalVoices/corpus/version/GlobalVoices.
<sup>20</sup>https://www.idiap.ch/webarchives/sites/www.idiap.ch/workshop/DiscoMT/.
<table>
  <tr>
    <th>Data Set</th>
    <th>Language</th>
    <th>Task</th>
  </tr>
  <tr>
    <td>mlqe-pe (Fomicheva et al., 2020)</td>
    <td>De, Ru, Zh</td>
    <td>APE</td>
  </tr>
  <tr>
    <td>XQUAD (Artetxe et al., 2019)</td>
    <td>De, Ru, Zh</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>p2p-data (Jin et al., 2023b)</td>
    <td>Zh</td>
    <td>DLT, DT(literature)</td>
  </tr>
  <tr>
    <td>NEJM (Liu and Huang, 2021)</td>
    <td>Zh</td>
    <td>DT(medical)</td>
  </tr>
  <tr>
    <td>DGT-TM (Steinberger et al., 2013)</td>
    <td>Cs, De</td>
    <td>DT(law)</td>
  </tr>
  <tr>
    <td>health_term (Xu and Carpuat, 2021)</td>
    <td>De</td>
    <td>TCT</td>
  </tr>
  <tr>
    <td>XCOPA (Ponti et al., 2020)</td>
    <td>Zh</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>MINTAKA (Sen et al., 2022)</td>
    <td>De</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>MGSM (Shi et al., 2023)</td>
    <td>De, Ru, Zh</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>MSLT (Federmann and Lewis, 2017)</td>
    <td>Zh</td>
    <td>DT(colloquial)</td>
  </tr>
  <tr>
    <td>Perseus (Zheng et al., 2023)</td>
    <td>Zh</td>
    <td>GT, DLT, DT(it, medical)</td>
  </tr>
  <tr>
    <td>BiPaR (Jing et al., 2019)</td>
    <td>Zh</td>
    <td>DLT, DT(literature)</td>
  </tr>
  <tr>
    <td>XStoryCloze (Lin et al., 2021)</td>
    <td>Ru, Zh</td>
    <td>GT, DT(literature)</td>
  </tr>
  <tr>
    <td>RELX (Köksal and Özgür, 2020)</td>
    <td>De</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>PETCI (Tang, 2022)</td>
    <td>Zh</td>
    <td>GT, ICL</td>
  </tr>
  <tr>
    <td>QALD-9-Plus (Perevalov et al., 2022)</td>
    <td>De, Ru</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>SubEdits (Chollampatt et al., 2020)</td>
    <td>De</td>
    <td>APE</td>
  </tr>
  <tr>
    <td>hallucinations-in-nmt (Guerreiro et al., 2023)</td>
    <td>De</td>
    <td>APE</td>
  </tr>
  <tr>
    <td>good-translation-wrong-in-context (Voita et al., 2019)</td>
    <td>Ru</td>
    <td>GT, ICL, DT(colloquial)</td>
  </tr>
  <tr>
    <td>CoCoA-MT (Nädejde et al., 2022)</td>
    <td>De, Ru</td>
    <td>GT, DT(colloquial)</td>
  </tr>
  <tr>
    <td>unfaithful(Zhang et al., 2024)</td>
    <td>De, Zh</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>ContraPro (Müller et al., 2018)</td>
    <td>De</td>
    <td>GT, APE, DT(colloquial)</td>
  </tr>
  <tr>
    <td>LiteraryTranslation (Karpinska and Iyyer, 2023)</td>
    <td>Cs, De, Ru, Zh</td>
    <td>DLT, DT(literature)</td>
  </tr>
  <tr>
    <td>ctxpro (Wicks and Post, 2023)</td>
    <td>De, Ru</td>
    <td>GT, ICL, DT(colloquial)</td>
  </tr>
  <tr>
    <td>DeCOCO (Hitschler et al., 2016)</td>
    <td>De</td>
    <td>GT</td>
  </tr>
  <tr>
    <td>IdiomsInCtx-MT (Stap et al., 2024)</td>
    <td>De, Ru</td>
    <td>GT, ICL</td>
  </tr>
  <tr>
    <td>Books<sup>21</sup></td>
    <td>De, Ru</td>
    <td>DLT, DT(literature)</td>
  </tr>
  <tr>
    <td>EUbookshop<sup>22</sup></td>
    <td>Cs, De, Ru</td>
    <td>DLT</td>
  </tr>
  <tr>
    <td>TED2020 (Reimers and Gurevych, 2020)</td>
    <td>Cs, De, Ru</td>
    <td>DLT</td>
  </tr>
</table>

Table 15: Data composition of our ComMT. We compiled translation datasets and manually categorized them based on predefined characteristics. Task abbreviations: GT - general translation, DLT - doc-level translation, DT - domain translation, TCT - terminology-constrained translation, APE - automatic post-editing, ICL - in-context learning.

<sup>21</sup>https://opus.nlpl.eu/Books/corpus/version/Books
<sup>22</sup>https://opus.nlpl.eu/EUbookshop/corpus/version/EUbookshop
<table>
  <tr>
    <th>General translation</th>
  </tr>
  <tr>
    <td><b>Prompt:</b> Translate the following text from English into Chinese.<br>
<b>English:</b> I know that advertising is how they make their money, but all that garbage seems counterproductive if it drives people away.<br>
<b>Chinese:</b> 我知道广告是为他们创收的一种方式，而如果大量的广告让观众反感离开，似乎会适得其反。</td>
  </tr>
</table>

<table>
  <tr>
    <th>Doc-level translation</th>
  </tr>
  <tr>
    <td><b>Prompt:</b> Translate the following text from English into Chinese.<br>
<b>English:</b> The outliers tend to be those who die young, so that typical (median) life expectancy is higher than average life expectancy. This means that raising the average HLE can be achieved by raising the HLE of those at the bottom of the health distribution to that of the typical (median) person. This not only makes targeting inequality more attractive, but does not require path-breaking medical innovations to achieve longer lifespans – just the achievement of typical outcomes for more people. With this in mind, it is urgent to close the sizeable rich-poor life-expectancy gap – around 15 years – in the United States. As a metric for economic and social progress, targeting HLE implicitly acknowledges that aging is malleable (if it wasn’t, it wouldn’t be a viable target). It turns out that a range of behaviors and policies, as well as the environment we inhabit, influence how we age and how long we live. It is estimated that our genetics account for only one-quarter of the factors contributing to how we age. Given this malleability, it is crucial that governments focus on HLE for the maximum number of people. Such a focus would also help governments confront one of the biggest challenges of the future: societal aging. Given that every country in the world is expected to experience societal aging, focusing on how well we age becomes paramount. This age malleability requires drawing a distinction between chronological and biological measures of age and focusing on the latter.<br>
<b>Chinese:</b> 反常之处便是那些年纪轻轻便死去的人，他们让典型（中位）寿命预期长于平均寿命预期。这意味着提高平均健康寿命预期可以通过将位于健康分布底层的人变成典型（中位）健康寿命预期的人来实现。这不仅让针对不平等性问题变得更有吸引力，也不必一定需要突破性的医学创新才能实现生命周期的延长——而只需要让更多人实现典型结果。基于此，现在迫切需要缩小美国庞大的贫富寿命预期差距——大约在15年左右。作为一个经济和社会进步的指标，健康寿命预期间接承认衰老具有可塑性（若非如此的话，这就不是一个可行的目标）。一系列行为和政策，以及我们所居住的环境，都影响着我们如何变老和如何延长我们的寿命。据估计，我们的器官大约占我们衰老的四分之一因素。考虑到这一可塑性，政府必须关注最大数量人口的见刊寿命预期。关注这一点也有助于政府面对未来最巨大的挑战之一：社会老龄化。世界上每一个国家都会经历社会老龄化，关注我们以多么优秀的方式衰老变得至关重要。这一年龄可塑性需要区分年龄的时间和生物指标，专注于后者。</td>
  </tr>
</table>
<table>
  <tr>
    <th colspan="2">Domain translation</th>
  </tr>
  <tr>
    <td>Domain:</td>
    <td>Medical</td>
  </tr>
  <tr>
    <td>Prompt:</td>
    <td>Translate the following text from English into Chinese.</td>
  </tr>
  <tr>
    <td>English:</td>
    <td>The median age of the 30 patients was 56.5 (28-80) years old, among them, 25 patients were primary plasma cell leukemia, and 5 patients were secondary plasma cell leukemia.</td>
  </tr>
  <tr>
    <td>Chinese:</td>
    <td>30例PCL患者中位年龄56.5（28-80）岁，25例为原发性浆细胞白血病，5例为继发性浆细胞白血病。</td>
  </tr>
  <tr>
    <td>Domain:</td>
    <td>Law</td>
  </tr>
  <tr>
    <td>Prompt:</td>
    <td>Translate the following text from English into Chinese.</td>
  </tr>
  <tr>
    <td>English:</td>
    <td>Article 8 Small and medium-sized enterprises shall observe the laws and regulations of the State on labor safety, occupational health, social security, protection of resources and environment, quality, taxation and finance, etc. and shall operate and manage according to law, and may not infringe upon the legitimate rights and interests of the employees or damage the public interests.</td>
  </tr>
  <tr>
    <td>Chinese:</td>
    <td>第八条 中小企业必须遵守国家劳动安全、职业卫生、社会保障、资源环保、质量、财政税收、金融等方面的法律、法规，依法经营管理，不得侵害职工合法权益，不得损害社会公共利益。</td>
  </tr>
  <tr>
    <td>Domain:</td>
    <td>IT</td>
  </tr>
  <tr>
    <td>Prompt:</td>
    <td>Translate the following text from English into Chinese.</td>
  </tr>
  <tr>
    <td>English:</td>
    <td>If you are using a Customer Portal and want to allow self-registration, follow these steps:</td>
  </tr>
  <tr>
    <td>Chinese:</td>
    <td>如果您正使用客户入口网站并希望允许自助注册,请按以下步骤操作:</td>
  </tr>
  <tr>
    <td>Domain:</td>
    <td>Colloquial</td>
  </tr>
  <tr>
    <td>Prompt:</td>
    <td>Translate the following text from English into Chinese.</td>
  </tr>
  <tr>
    <td>English:</td>
    <td>Believe me, I’m gonna take care of you and he’s gonna be OK.</td>
  </tr>
  <tr>
    <td>Chinese:</td>
    <td>相信我,我会照顾好你们,他会没事的.</td>
  </tr>
  <tr>
    <td>Domain:</td>
    <td>Literature</td>
  </tr>
  <tr>
    <td>Prompt:</td>
    <td>Translate the following text from English into Chinese.</td>
  </tr>
  <tr>
    <td>English:</td>
    <td>The President required the name of that citizen. The accused explained that the citizen was his first witness. He also referred with confidence to the citizen’s letter, which had been taken from him at the Barrier, but which he did not doubt would be found among the papers then before the President.</td>
  </tr>
  <tr>
    <td>Chinese:</td>
    <td>庭长问那公民是谁。被告说那公民便是他的第一个证人。他还很有把握地提起那人的信，那是在城门口从他身上取走的，他相信可以在庭长的卷宗中找到。</td>
  </tr>
</table>
<table>
  <tr>
    <th colspan="2">Terminology-constrained translation</th>
  </tr>
  <tr>
    <td><b>Prompt:</b> Translate the following text from <a href="https://en.wikipedia.org/wiki/English_language">English</a> into <a href="https://en.wikipedia.org/wiki/Chinese_language">Chinese</a> using the provided terminology pairs, ensuring the specified terms are accurately translated as indicated.<br>
    <b>Terminology pairs:</b> "National Football League" = "国家橄榄球联盟"<br>
    <b>English:</b> Tim’s younger brother, Tod Leiweke, is currently the chief operating officer of the National Football League since 2015.<br>
    <b>Chinese:</b> 蒂姆的弟弟托德·莱维克自 2015 年以来担任国家橄榄球联盟的首席运营官。</td>
  </tr>
  <tr>
    <th colspan="2">Automatic post-edition</th>
  </tr>
  <tr>
    <td><b>Prompt:</b> Improve the following machine-generated translation from <a href="https://en.wikipedia.org/wiki/English_language">English</a> to <a href="https://en.wikipedia.org/wiki/Chinese_language">Chinese</a>. Correct errors and generate a more accurate translation.<br>
    <b>English:</b> unfazed. Matteo hires an assassin to bomb the resort to create chaos and mayhem.<br>
    <b>Chinese:</b> 马特奥不慌不忙地雇用了一名刺客来轰炸制造混乱和混乱的手段.<br>
    <b>Improved translation:</b> 马特奥不慌不忙地雇用了一名刺客来轰炸度假村，从而引起混乱。</td>
  </tr>
</table>

Table 16: Examples from ComMT.

Reviewer paper summaries: The authors propose taking an existing decoder-only LLM as a base model and train an encoder-decoder translation model on top. The output of the LLM serves as the input to the encoder of a traditional Neural Machine Translation (NMT) model. The model is trained and tested on a diverse set of existing datasets. The evaluation shows that the model is on par with the LLM being fine-tuned for translation, while offering higher translation throughput, as the encoder-decoder model is smaller than the base LLM.

This paper introduces an LLM-based architecture for machine translation. The model uses LLM for encoding and NMT decoder to improve the performance.
Summary of strengths: The paper provides datapoints on the use of LLMs as encoders of dedicated NMT systems. With LLMs a lot of the work has shifted towards decoder-only models, despite encoder-decoder models previously working better for translation. It is interesting to see that an encoder-decoder model can be trained and run on top of representations produced by a LLM. In terms of results the authors show similar quality of an NMT system trained on LLM encoders at a reduce inference cost compare to an LLM fine-tuned on translation data. One some benchmarks one system works better on others the other. While the results in terms of translation quality are comparable to other approaches, results on sequence to sequence models on top of LLMs could be interesting to the larger research community.  The evaluation is thorough, including a range of different evaluation “tasks” covering several translation settings and languages.

The paper is easy to follow.
Summary of weaknesses: In terms of quality the method is comparable to fine-tuning an LLM for translation tasks. With the advantage of the method mainly coming from efficiency it would have been interesting to see further methods for efficiency LLM inference applied to the baseline system. Of course there may also be other advantages that are mentioned in the paper, such as the reduced off-target translation rate, but the main advantage seems to be efficiency. One baseline efficiency method that could have served as a baseline is using a smaller LLM that is fine-tuned for translation as a drafting model.

While it is great to see a diverse set of translation related tasks for evaluation, it is not entirely clear if the results can be reproduced based on the description in the paper. From the description in the paper it is unclear which test sets were used for the terminology constrained translation as well as for the automatic post-editing. The training data and the remainder of the test data is derived from existing datasets, so that likely this also applies to these test sets, but for reproducibility it is essential to describe which ones and if they are newly created test sets how they were created and whether they will be released as well.

There are several instances in the paper where vague subjective terms are used without being quantified and backed up by data. For example in the conclusions the “impressive performance” or in the abstract the model is described as “universal”, “easy to optimize”. It is unclear what makes the model universal or easy to optimize. I would suggest to focus the scientific writing on data and facts in a neutral tone when describing the model instead of using vague terms.
You write "demonstrate the strong generalization capabilities of the LaMaTE model". It would be good to clarify what experimental results demonstrate this.

There are some problems in this paper.

1. Although the authors introduce this model clearly, the motivation of this paper is still unclear. There is no clear description of the motivation in the Introduction or Abstract. Besides, the authors claim that using LLM for encoding can improve the performance in the Introduction, while it cannot be viewed as the motivation of the work. 

2. The name of the LLM component is confusing. LLM is a decoder for encoding, while it is much better to make LLM an encoder without any confusing description. 

3. Results of experiments show that the model improves the performance not significantly. Table 1 shows that the model still cannot outperform LLM while the size of the model is larger than other LLMs.
Comments, suggestions and typos: For the training data the paper describes "we construct a new dataset". From the Appendix and description it sounds like a collection of existing datasets was used for training. It should be clarified in the paper whether the "new" dataset is just a name of a collection of existing datasets or any data is new. If data is new it would be great to clarify if and what data will be released.

Line 508: Instead of calling it "tasks" it would be easier to understand "translation directions" if that is what is meant.

There’s a description of In-context Learning Translation in the Appendix. What section in the paper does this refer to? If this is for results that were excluded, why is that? I would suggest to remove it if no longer relevant.

Figure 4: it would be much easier to get this point across if you use the same y axis scaling for the left and right subplot.

N/A
Overall assessment scores: 3

2.5
Metareview score: 3.5
Metareview: This paper proposes LaMaTE, a machine translation architecture that employs a pre-trained decoder-only large language model (LLM) as an encoder, combined with a lightweight neural machine translation (NMT) decoder. The model is trained using a two-stage fine-tuning approach and is evaluated on a diverse benchmark (ComMT). The results demonstrate that the model achieves translation quality comparable to fine-tuned LLMs, while significantly improving inference efficiency (decoding speed and KV cache memory).
Summary of reasons to publish: 1. The paper is well-structured and easy to follow.
2. The proposed method offers a good balance between translation quality and computational efficiency, achieving faster decoding and reduced memory usage.
3. The evaluation is thorough, covering a wide range of translation tasks, domains, and languages.
4. The work demonstrates the potential of LLMs for high-quality text representation within encoder-decoder frameworks, with practical advantages for real-world applications.