Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation
Authors: Zhi Qu, Yiran Wang, Jiannan Mao, Chenchen Ding, Hideki Tanaka, Masao Utiyama, Taro Watanabe
Accepted at: acl25

Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation

Zhi Qu*1 Yiran Wang2 Jiannan Mao2
Chenchen Ding1,2 Hideki Tanaka2 Masao Utiyama2 Taro Watanabe1
1Nara Institute of Science and Technology, Japan.
2National Institute of Information and Communications Technology, Japan.
qu.zhi.pv5@is.naist.jp

Abstract

The multilingual neural machine translation (MNMT) aims for arbitrary translations across multiple languages. Although MNMT-specific models trained on parallel data offer low costs in training and deployment, their performance consistently lags behind that of large language models (LLMs). In this work, we introduce registering, a novel method that enables a small MNMT-specific model to compete with LLMs. Specifically, we insert a set of artificial tokens specifying the target language, called registers, into the input sequence between the source and target tokens. By modifying the attention mask, the target token generation only pays attention to the activation of registers, representing the source tokens in the target language space. Experiments on EC-40, a large-scale benchmark, show that our method advances the state-of-the-art of MNMT. We further pretrain two models, namely MITRE (multilingual translation with registers), by 9.3 billion sentence pairs across 24 languages collected from public corpora. One of them, MITRE-913M, outperforms NLLB-3.3B, achieves comparable performance with commercial LLMs, and shows strong adaptability in fine-tuning. Finally, we open-source our models to facilitate further research and development in MNMT: https://github.com/zhiqu22/mitre.

1 Introduction

Multilingual neural machine translation (MNMT) aims to enable arbitrary translations across multiple languages. Traditionally, training models specific to MNMT using parallel data was highly appealing, not only because such MNMT-specific models maintain a minimal number of parameters (Firat et al., 2016; Fan et al., 2021; NLLB Team, 2022),

but also due to their potential for zero-shot translation, i.e., translating language pairs unseen during training, which helps address data scarcity in certain translation directions (Johnson et al., 2017; Fan et al., 2021; Zhang et al., 2020). However, the current mainstream solution for MNMT relies on large language models (LLMs), as the performance of MNMT-specific models has lagged behind that of LLMs (Zhu et al., 2024; Xu et al., 2024). Recent analyses (Chen et al., 2023; Tan and Monz, 2023) identify the off-target problem, i.e., translations fail to reach the intended target language, as a key factor causing the under-performance of MNMT-specific models. Moreover, Rios et al. (2020); Chen et al. (2023) show that constraining the target token generation to the target language space can alleviate the off-target problem.

In this work, we present registering, a simple yet effective method designed for MNMT-specific models based on the decoder-only architecture (Dec-only) without introducing additional parameters. Specifically, we insert a set of artificial tokens between the source and target tokens, called registers, which indicate only the target language without any semantics. The registers are designed to have the same length as the source tokens, because each register is expected to capture the semantics of its positionally-aligned source token and then

![Illustration of the attention view among different architectures. "Token" refers to the representation corresponding to the token.](page_1012_384_384_192.png)

Figure 1: Illustration of the attention view among different architectures. "Token" refers to the representation corresponding to the token.
represent it in the target language space. As illustrated in Figure 1, by modifying the attention mask, the generation of target tokens no longer follows the attention mechanism of encoder-decoder (Encdec) or Dec-only architectures. Instead, it relies solely on the registers located in the representational space of the intended target language.

We conduct two sets of experiments, evaluated with four automatic metrics: spBLEU (NLLB Team, 2022), chrF++ (Popović, 2015, 2017), COMET (Rei et al., 2020), and off-target ratio (Zhang et al., 2020). First, we experiment with EC-40 (Tan and Monz, 2023), a large-scale benchmark designed to assess zero-shot translation capability. Experimental results show that, compared to strong baselines, our method improves spBLEU scores by up to 71% on average across 1,640 directions with fewer parameters and drastically reduces the off-target ratio from 26.69% to 3.65%. Second, we collect 9.3 billion sentence pairs across 24 languages by sampling from the NLLB open dataset (NLLB Team, 2022) with the bridge language strategy (Fan et al., 2021). We then pre-train two models, MITRE-466M and MITRE-913M (multilingual translation with registers). One of them, MITRE-913M, not only outperforms NLLB-3.3B (NLLB Team, 2022) and GPT-3.5 Turbo (Brown et al., 2020) but also achieves competitive performance with GPT-4o mini (OpenAI, 2024). Also, we fine-tune the pre-trained models with full parameters and LoRA (Hu et al., 2022) in three distinct scenarios, demonstrating the superior adaptability of MITRE in fine-tuning. Finally, by analyzing the attention mechanisms and the representation distribution in translation instances at the token level, we confirm that the register mirrors the corresponding source token in the target language space.

2 Related Work

Johnson et al. (2017) proposed adding a language tag as a translation instruction at the beginning of the input sequence, marking the beginning of MNMT-specific models. Recent analyses (Chen et al., 2023; Tan and Monz, 2023) show that addressing the off-target problem is key to improving zero-shot translation. Early works (Rios et al., 2020; Qu and Watanabe, 2022) tried to use language-specific dictionaries or components to isolate generation across languages, but this was costly and hindered knowledge sharing (Zhang et al., 2021). As a compromise, Chen et al. (2023) proposed adding language-specific subsets to a shared dictionary to mitigate the off-target problem. Beyond the explicit addition of language-specific parameters, optimizing internal representations implicitly improves zero-shot translation. Specifically, aligning semantic information across languages (Pan et al., 2021; Bu et al., 2024) and strengthening the translation instruction toward the target language (Stap et al., 2023; Qu et al., 2024a; Sun et al., 2024) help mitigate the off-target problem. Our proposed method is a combination of explicit and implicit strategies, as it separates generation-related representations by language.

In methodology, the translation instruction used in MNMT-specific models (Johnson et al., 2017) is similar to the artificial tokens in prefix-tuning (Li and Liang, 2021). In fact, we are methodologically inspired by gisting, a variation of prefix-tuning proposed by Mu et al. (2023). Specifically, they modified the attention mask to compress information into a set of artificial tokens, used in generation to eliminate the need for the original sequence. However, the difference in concept is that our proposed method, registering, aims to transfer each source token's semantics into the positionally-aligned register, which can be regarded as a representation-level container pointing to the target language. In other words, registering is conceptually similar to chain-of-thought (Wei et al., 2022), where the process represents "rethinking" the source tokens from the perspective of the target language.

Additionally, given that LLMs already exhibit strong MNMT capabilities (Zhu et al., 2024), fine-tuning LLMs into MNMT-specific models has become a popular direction of exploration. However, the results in this direction are still limited, such as performance still falling behind commercial LLMs (Yang et al., 2023) and fewer supported languages, e.g., 5 in Xu et al. (2024) and 10 in Alves et al. (2024). In this work, we demonstrate the potential of directly training an MNMT-specific model with parallel data only, aiming to drive further discussion on MNMT.

3 Multilingual Translation With Registers

3.1 Multilingual Neural Machine Translation

Given a multilingual corpus \( \mathbb{C} \) spanning multiple translation directions, each instance in \( \mathbb{C} \) is defined as \( (\boldsymbol{x}, \boldsymbol{y}) \), where \( \boldsymbol{x} \) consists of a set of source tokens \( x = x_1, \ldots, x_I \) and a set of target tokens \( y = y_1, \ldots, y_J \). Also, we introduce a set of lan-
Figure 2: Illustration of registering. The example depicts a translation from English to German. The illustrated model stacks N layers, each following the Transformer decoder layer structure (Vaswani et al., 2017) with pre-normalization (Xiong et al., 2020). Notably, each circle represents a token and its representation in the generation.

guage tags \( \mathbb{L} = \{l_1, \ldots, l_K\} \), which are artificial tokens, each corresponding to one of the \( K \) languages in \( \mathbb{C} \). Following Johnson et al. (2017); Wu et al. (2021), we add a tag indicating the language of \( y \) at the beginning of \( x \) as the translation instruction for multilingual neural machine translation (MNMT), denoted by \( l_y \). Consequently, the input fed into the MNMT-specific model becomes \( x' = l_y, x_1, \ldots, x_T \). Formally, we train the model over all instances of \( \mathbb{C} \) by optimizing the following cross entropy loss:

\[
\mathcal{L}_{ce} = - \sum_{x', y \in \mathbb{C}} \sum_{j=1}^J \log p(y_j \mid x', y_{<j}),
\]

where \( p(y_j \mid x', y_{<j}) \) represents the probability for generating \( y_j \) by MNMT-specific model. The current state-of-the-art models (Fan et al., 2021; NLLB Team, 2022) utilize the encoder-decoder architecture (Enc-dec), where the generation of \( y_j \) can be expressed as:

\[
y_j = \text{decoder}(\text{encoder}(x'), y_{<j}).
\]

Also, Gao et al. (2022); Zhang et al. (2022) show that the MNMT-specific model can be implemented with a decoder-only architecture (Dec-only). In this setup\( ^1 \), the generation can be described as:

\[
y_j = \text{decoder}(x', y_{<j}).
\]

3.2 Registering

Gu et al. (2019); Qu et al. (2024b) suggest that \( l_y \) is a key factor causing the off-target problem. Specifically, although \( l_y \) is a translation instruction distinctly representing the target language, generation cannot strictly depend on \( l_y \) due to the dilution of other source tokens in the attention mechanism. Therefore, registering is proposed to address this problem by constraining the generation within the target language space, where all registers used in generation have the same function as \( l_y \).

Given a Dec-only model\( ^2 \), we begin by initializing a set of artificial tokens corresponding to the target language, denoted by \( r = r_1, \ldots, r_{l+1} \), matching the length of \( x' \). Notably, since \( l_y \) has the same function as we design for \( r \), \( r \) is initialized by duplicating \( l_y \). We then insert \( r \) into the input sequence between \( x' \) and \( y \), thus reformulating the generation process of Equation 3 as:

\[
y_j = \text{decoder}(x', r, y_{<j}).
\]

The key step of registering is modifying the Transformer attention mask (Vaswani et al., 2017) to remove source tokens from the view of target tokens. As shown in Figure 2, we initialize the attention mask based on the prefix Dec-only scheme (Dong et al., 2019), where the source tokens compute attention for each other bidirectionally, while the target tokens only compute attention for the

\( ^1 \)We follow Gao et al. (2022) to train a Dec-only MNMT-specific model with Equation 1 rather than using a language modeling loss (Radford et al., 2018).

\( ^2 \)Intuitively, Dec-only is more parameter-efficient than Enc-dec, as separate components in the latter process source and target tokens. Therefore, given that registering can address the off-target problem, Dec-only is the preferable choice. Supporting experiments are provided in Appendix J.
<table>
  <tr>
    <th rowspan="2">#params</th>
    <th rowspan="2">Method</th>
    <th colspan="3">High</th>
    <th colspan="3">Med</th>
    <th colspan="3">Low</th>
    <th colspan="3">Extra Low</th>
    <th>sup.</th>
    <th>zero.</th>
    <th>avg.</th>
    <th>off.(%)</th>
  </tr>
  <tr>
    <th>→</th><th>←</th><th>→</th><th>←</th><th>→</th><th>←</th><th>→</th><th>←</th><th>→</th><th>←</th><th>→</th><th>←</th><th>→</th><th>←</th><th>→</th><th>←</th>
  </tr>
  <tr>
    <td rowspan="4">Enc-dec</td>
    <td>vanilla</td><td>9.46</td><td>11.05</td><td>7.49</td><td>9.80</td><td>5.03</td><td>3.95</td><td>5.41</td><td>2.59</td><td>29.06</td><td>5.86</td><td>6.99</td><td>48.40</td>
  </tr>
  <tr>
    <td>+CL</td><td>14.21</td><td>14.19</td><td>12.19</td><td>14.18</td><td>7.89</td><td>7.55</td><td>7.66</td><td>6.04</td><td>29.03</td><td>9.74</td><td>10.68</td><td>19.08</td>
  </tr>
  <tr>
    <td>+LCS</td><td>10.44</td><td>13.67</td><td>9.34</td><td>13.17</td><td>8.73</td><td>6.07</td><td>8.49</td><td>4.10</td><td>29.18</td><td>8.35</td><td>9.37</td><td>22.43</td>
  </tr>
  <tr>
    <td>+LAWS</td><td>9.71</td><td>12.22</td><td>7.74</td><td>11.19</td><td>5.71</td><td>3.98</td><td>6.37</td><td>2.99</td><td>29.07</td><td>6.40</td><td>7.54</td><td>42.98</td>
  </tr>
  <tr>
    <td rowspan="3">Dec-only</td>
    <td>vanilla</td><td>11.57</td><td>11.51</td><td>9.48</td><td>11.03</td><td>6.01</td><td>6.06</td><td>5.74</td><td>4.20</td><td>28.61</td><td>7.30</td><td>8.34</td><td>22.21</td>
  </tr>
  <tr>
    <td>+TDO</td><td>13.33</td><td>13.50</td><td>10.53</td><td>12.75</td><td>7.13</td><td>6.88</td><td>6.74</td><td>4.60</td><td>28.84</td><td>8.62</td><td>9.61</td><td>27.18</td>
  </tr>
  <tr>
    <td>+Ours</td><td><b>15.43</b></td><td><b>15.46</b></td><td><b>13.88</b></td><td><b>14.62</b></td><td><b>8.94</b></td><td><b>8.99</b></td><td><b>8.76</b></td><td><b>7.94</b></td><td><b>28.90</b></td><td><b>11.05</b></td><td><b>11.92</b></td><td><b>4.65</b></td>
  </tr>
  <tr>
    <td rowspan="4">Enc-dec</td>
    <td>vanilla</td><td>12.66</td><td>15.02</td><td>10.86</td><td>14.50</td><td>7.40</td><td>5.04</td><td>7.12</td><td>3.47</td><td>30.28</td><td>8.64</td><td>9.69</td><td>26.69</td>
  </tr>
  <tr>
    <td>+CL</td><td>15.89</td><td>15.97</td><td>13.67</td><td>16.15</td><td>8.36</td><td>8.16</td><td>8.32</td><td>5.96</td><td>30.54</td><td>10.79</td><td>11.76</td><td>19.99</td>
  </tr>
  <tr>
    <td>+LCS</td><td>10.79</td><td>16.19</td><td>10.00</td><td>15.31</td><td>9.99</td><td>5.39</td><td>9.41</td><td>3.70</td><td>30.33</td><td>9.25</td><td>10.28</td><td>23.47</td>
  </tr>
  <tr>
    <td>+LAWS</td><td>14.03</td><td>16.39</td><td>12.50</td><td>16.31</td><td>8.61</td><td>6.31</td><td>8.45</td><td>3.57</td><td>30.20</td><td>10.03</td><td>11.01</td><td>21.73</td>
  </tr>
  <tr>
    <td rowspan="3">Dec-only</td>
    <td>vanilla</td><td>14.57</td><td>15.07</td><td>12.25</td><td>15.02</td><td>8.27</td><td>7.40</td><td>7.71</td><td>5.11</td><td>29.97</td><td>9.84</td><td>10.52</td><td>19.01</td>
  </tr>
  <tr>
    <td>+TDO</td><td>15.27</td><td>15.79</td><td>12.83</td><td>15.56</td><td>8.44</td><td>7.96</td><td>8.15</td><td>5.40</td><td>30.23</td><td>10.40</td><td>11.37</td><td>23.14</td>
  </tr>
  <tr>
    <td>+Ours</td><td><b>16.81</b></td><td><b>16.98</b></td><td><b>15.25</b></td><td><b>16.57</b></td><td><b>10.10</b></td><td><b>9.88</b></td><td><b>9.64</b></td><td><b>8.37</b></td><td><b>29.88</b></td><td><b>12.26</b></td><td><b>13.12</b></td><td><b>3.65</b></td>
  </tr>
</table>

Table 1: Averaged spBLEU scores of results on EC-40, the last column (off.) lists the off-target ratio averaged from all directions, the scores of chrF++ and COMET are reported in Tables 10 and 11, as discussed in Appendix H. We report scores by grouping languages that have the same resource tier. Then, → includes directions translating from the corresponding group to languages out of this group, and ← includes directions translating to the corresponding group. sup., zero, and avg. abbreviate the average of supervised translations, the average of zero-shot translations, and the average of all translations, respectively. The best score in each column of a block is in bold.

previous ones. We then adjust the mask to control token-level representation according to the following rules: (1) \( r \) pays attention to \( x' \); (2) \( r \) computes attention bidirectionally within \( r \); (3) \( y_j \) pays attention to \( r \) and \( y_{<j} \). With this design, the generation of \( y \) can solely rely on the activation of \( r \), where the activation of \( r_i \) not only functions as a representational container of the target language but also carries the semantics of \( x_i \). As a result, the generation is strictly constrained to the target language space to effectively address the off-target problem.

4 Experiment: On Benchmark

4.1 Dataset and Evaluation

We conduct the first set of experiments on a large zero-shot translation benchmark, EC-40 (Tan and Monz, 2023), consisting of 120 million translation instances spanning 41 languages across five language families in the training data\footnote{Details of EC-40 are described in Appendix A.}. Except for English, each family includes eight languages, categorized into four resource tiers, namely, High, Medium, Low, and Extra Low, corresponding to 5M, 1M, 100K, and 50K sentence pairs, respectively. For testing the zero-shot translation capability, all training directions in EC-40 involve English, either as the source or target language, resulting in 80 supervised directions. Then, we evaluate the trained models on all supervised and zero-shot directions, i.e., 1,640 directions. Unlike the original setup in Tan and Monz (2023), we follow NLLB Team (2022); Cao et al. (2024) to standardize the validation and testing processes using Flores\footnote{https://github.com/openlanguagedata/flores.}, which is a high-quality parallel dataset available for over 200 languages. Specifically, we use the dev and devtest sets of Flores for validation and testing, containing 997 and 1,012 sentences per language, respectively.

In the evaluation, we set the beam size to 5 in inference. We employ four automatic metrics to evaluate inference results on the test set: (1) spBLEU (NLLB Team, 2022), a variant of BLEU (Papineni et al., 2002; Post, 2018) used for Flores, unifies tokenization across languages through an open-source tokenizer\footnote{https://tinyurl.com/flores200sacrebleuspm.}; (2) chrF++ (Popović, 2015, 2017) assesses character-level overlap and balances precision with recall; (3) COMET\footnote{All COMET scores are computed using Unbabel/wmt22-comet-da (Rei et al., 2022).} (Rei et al., 2020) evaluates quality by comparing generated translations, reference translations, and source sentences at a representation level; (4) we report the off-target ratio (Zhang et al., 2020) as a supplementary metric, because the testing tool\footnote{https://github.com/LlmKira/fast-langdetect.} is not fully accurate as it relies on recognizing language-specific tokens. Since COMET and off-target ratio evaluations lack support for certain languages, we compute these scores only for supported languages, as listed in Appendix G.
4.2 Configuration and Baseline

The modeling follows the manner of Transformer (Vaswani et al., 2017) with an embedding size of 1,024, an inner size of 4,096, and 16 attention heads. We divide the models into two configurations based on model depth. We first stack 12 layers and balance the number of layers between the encoder and decoder in Enc-dec, resulting in 242M parameters for Enc-dec and 217M for Dec-only. Then, models include 24 layers in the second configuration, yielding 418M parameters for Enc-dec and 368M for Dec-only.

Apart from the vanilla Enc-dec and Dec-only, we also reproduce four related methods mentioned in Section 2 as baselines: (1) LAVS (Chen et al., 2023) adds language-specific tokens to the shared dictionary; (2) CL (Pan et al., 2021) aligns sentence-level semantic representations across languages using the encoder output; (3) LCS (Sun et al., 2024) strengthens translation instructions for Enc-dec by biasing token representations in the encoder with a target language embedding. This mechanism resembles \( r \) but uses a different operation; (4) TDO (Qu et al., 2024b) strengthens translation instructions for Dec-only by dividing the process of Dec-only into two phases, specifically, encoding source tokens with stronger target language features in the first phase, and then, concatenating the encoded source tokens and target tokens to feed into Dec-only models. We list all implementation and training details of these baselines in Appendix C.

4.3 Result

Experimental results shown in Table 1 exhibit consistent trends across both configurations, where our method consistently performs the best. The most notable improvement is in the off-target ratio, where the metric reduces from 48.40% to 4.65% in 12-layer models and from 26.69% to 3.65% in 24-layer models. These results indicate that registering nearly resolves the off-target problem. Moreover, although our method does not achieve the highest performance in supervised translation, this is not a drawback, because the higher supervised performance of vanilla models is attributed to the overfitting (Gu et al., 2019; Liu et al., 2021). Additionally, we observe that registering significantly outperforms LAVS. Based on our discussion of LAVS and its underlying methods in Section 2, we argue that simply adding language-specific parameters is not a cost-efficient solution.

<table>
  <tr>
    <th>Family (Group)</th>
    <th>Languages</th>
    <th>Bridge</th>
  </tr>
  <tr>
    <td>English*</td>
    <td>en</td>
    <td>en</td>
  </tr>
  <tr>
    <td>Germanic</td>
    <td>de, nl, sv, da, af</td>
    <td>de, nl</td>
  </tr>
  <tr>
    <td>Romance</td>
    <td>fr, es, it, pt, ro</td>
    <td>fr, es</td>
  </tr>
  <tr>
    <td>Slavic</td>
    <td>ru, cs, pl, bg, uk</td>
    <td>ru, cs</td>
  </tr>
  <tr>
    <td>Malayo-Polynesian</td>
    <td>id, ms, jv, tl</td>
    <td>id, ms</td>
  </tr>
  <tr>
    <td>Asian*</td>
    <td>ja, zh, ko, vi</td>
    <td>ja, zh</td>
  </tr>
</table>

Table 2: Languages in data collection. Languages are shown by their ISO 639-1 codes. Decoration with * indicates a language group instead of a language family.

We also observe that the gains of spBLEU scores from related methods tend to diminish as the number of model parameters increases. In the 12-layer models, the highest gain among four related methods over vanilla models in zero-shot translation is 3.88. In 24-layer models, the improvement decreases to 2.15. However, our method achieves more consistent improvements with gains of 5.19 and 3.62 in 12-layer and 24-layer models, respectively. From this comparison, we can conclude that registering demonstrates superior scalability.

5 Experiment: Pre-trained Models

5.1 Data Collection with Bridge Languages

A robust and practical MNMT-specific model requires training across multiple directions rather than English-involved directions only (Zhang et al., 2020; Eriguchi et al., 2022). However, collecting data for every possible translation direction is infeasible, as the number of directions grows exponentially with the number of supported languages. In this work, limited by our computational resources, we adopt the Bridge Language strategy (Fan et al., 2021) to collect data across 24 languages spanning more than five language families.

As shown in Table 2, we group languages by family except for English. The Germanic, Romance, and Slavic groups belong to European language families, and the Malayo-Polynesian differs significantly from these European languages. In addition, we define a special group, Asian, which includes four languages predominantly spoken in the Asian continent: ja, zh, ko, and vi. While these languages belong to different families, they share certain similarities due to their geographic proximity. We designate the two most resource-rich languages in each group as bridge languages and follow these rules for data collection: (1) en connects with all languages; (2) bridge languages connect with each other; (3) bridge languages connect with the re-
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">English</th>
    <th colspan="2">Germanic</th>
    <th colspan="2">Romance</th>
    <th colspan="2">Slavic</th>
    <th colspan="2">Mal.-Polyn.</th>
    <th colspan="2">Asian</th>
    <th rowspan="2">avg.</th>
  </tr>
  <tr>
    <th>→</th><th>←</th>
    <th>→</th><th>←</th>
    <th>→</th><th>←</th>
    <th>→</th><th>←</th>
    <th>→</th><th>←</th>
    <th>→</th><th>←</th>
  </tr>
  <tr>
    <td>483M</td><td>30.36</td><td>31.92</td><td>24.40</td><td>22.58</td><td>24.01</td><td>25.81</td><td>22.59</td><td>23.40</td><td>17.94</td><td>16.50</td><td>18.30</td><td>18.37</td><td>22.10</td>
  </tr>
  <tr>
    <td>M2M<br>615M</td><td>30.69</td><td>31.98</td><td>26.35</td><td>25.56</td><td>25.47</td><td>27.52</td><td>24.02</td><td>24.77</td><td>20.11</td><td>17.49</td><td>19.31</td><td>19.09</td><td>23.65</td>
  </tr>
  <tr>
    <td>1.2B</td><td>35.92</td><td>35.14</td><td>29.51</td><td>26.82</td><td>28.40</td><td>28.38</td><td>26.58</td><td>26.19</td><td>18.09</td><td>17.57</td><td>15.48</td><td>20.07</td><td>24.69</td>
  </tr>
  <tr>
    <td>615M</td><td>35.85</td><td>41.04</td><td>28.13</td><td>27.41</td><td>27.46</td><td>29.09</td><td>25.40</td><td>25.33</td><td>25.39</td><td>24.35</td><td>20.72</td><td>19.42</td><td>26.05</td>
  </tr>
  <tr>
    <td>NLLB<br>1.3B</td><td>38.08</td><td>43.42</td><td>30.52</td><td>30.17</td><td>29.63</td><td>31.42</td><td>27.84</td><td>28.25</td><td>28.08</td><td>26.87</td><td>23.50</td><td>21.06</td><td>28.51</td>
  </tr>
  <tr>
    <td>3.3B</td><td>39.80</td><td>45.08</td><td>31.93</td><td>31.77</td><td>30.88</td><td>32.62</td><td>29.29</td><td>30.13</td><td>29.81</td><td>28.08</td><td>25.18</td><td>22.56</td><td>30.01</td>
  </tr>
  <tr>
    <td>GPT<br>3.5 turbo<br>4o mini</td><td>38.27</td><td>42.37</td><td>31.01</td><td>31.02</td><td>30.09</td><td>32.73</td><td>28.56</td><td>27.85</td><td>26.75</td><td>22.81</td><td>23.61</td><td>24.08</td><td>28.66</td>
  </tr>
  <tr>
    <td></td><td>41.49</td><td>43.97</td><td>33.09</td><td>31.92</td><td>31.40</td><td>34.03</td><td>30.54</td><td>30.69</td><td>31.01</td><td>27.20</td><td>26.34</td><td>27.53</td><td>31.09</td>
  </tr>
  <tr>
    <td>MITRE<br>466M</td><td>40.20</td><td>42.60</td><td>32.14</td><td>31.51</td><td>31.32</td><td>33.26</td><td>29.36</td><td>29.80</td><td>28.46</td><td>26.16</td><td>24.05</td><td>23.56</td><td>29.77</td>
  </tr>
  <tr>
    <td>913M</td><td>41.16</td><td>44.17</td><td>33.34</td><td>32.95</td><td>32.53</td><td>34.23</td><td>30.74</td><td>31.26</td><td>29.90</td><td>27.22</td><td>25.93</td><td>25.58</td><td>31.15</td>
  </tr>
</table>

Table 3: Averaged spBLEU scores comparing MITRE with baselines. The off-target ratio is not reported due to the near-zero values in these large-scale models. chrF++ and COMET scores are provided in Tables 12 and 13, as discussed in Appendix 1. Mal.-Polyn. abbreviates Malayo-Polynesian, and other abbreviations follow Table 1. Prompts used for GPT are reported in Appendix F. Additionally, we use green boxes to highlight scores exceeding NLLB-3.3B and blue boxes for those surpassing GPT-4o mini, where blue box has the priority.

maining languages within their respective groups. Given that ms cannot meet (2) and (3) due to its low resource, we collect additional data for ms. Based on the above strategy, out of 552 possible translation directions, we collect data from the reproduced version of the NLLB dataset8 (NLLB Team, 2022) for a total of 194 directions, resulting in 9.3 billion sentence pairs for our pre-training.9

5.2 Configuration and Baseline
We begin by training a vocabulary of 160,000 tokens using SentencePiece (Kudo and Richardson, 2018) on 150 million sentences randomly sampled from the training set. We then pre-train two models, named MITRE (multilingual translation with registers), on 80 V100 GPUs with 466 million and 913 million parameters, respectively. We report complete details of modeling and training in Appendix D. The validation and testing process aligns with Section 4.1.

We compare our model against not only state-of-the-art MNMT-specific models, but also commercial LLMs, because commercial LLMs present the upper limit of fine-tuning open-sourced LLMs into MNMT-specific models (Section 2). First, the MNMT models include three versions of M2M (483M10, 615M, and 1.2B) (Fan et al., 2021) and three versions of NLLB (615M-distilled, 1.3B, and 3.3B) (NLLB Team, 2022). Also, we include commercial LLMs, GPT-3.5 Turbo11 (Brown et al., 2020) and GPT-4o mini12 (OpenAI, 2024). Meanwhile, we include these NLLB models as baselines13 in our fine-tuning experiments. Specifically, we create three scenarios randomly selecting 5, 25, and 100 translation directions from the possible directions. Then, we perform fine-tuning with full parameters and with LoRA (Hu et al., 2022) on the Flores dev, which contains 997 sentence pairs per direction. Fine-tuning settings are provided in Appendix E.

5.3 Main Results
The experimental results comparing MITRE with baselines are shown in Table 3. We observe that although NLLB-3.3B surpasses MITRE-913M by 0.91 spBLEU points for translations into English and by 0.86 points for Malayo-Polynesian languages, MITRE-913M consistently achieves higher scores in other translation directions, with an overall average gain of 1.14 points. Given that NLLB even surpasses GPT-4o mini by 1.11 points in English translation, we infer that MITRE, a Dec-only model with registering, demonstrates better generalization than NLLB, based on Enc-dec. Notably, scaling parameters of NLLB from 1.3B to 3.3B yields only a gain of 1.50 points, while MITRE attains a comparable gain of 1.38 points with an additional 450M parameters. Furthermore, the alignment of training and validation loss for two MITRE models (Appendix D) reinforces our conclusion in Section 4.3 that registering provides superior scalability. Finally, based on all experimental results,

8https://opus.nlpl.eu/NLLB/corpus/version/NLLB
9Appendix B reports the data distribution at the language-family and language level.
10The official name is M2M-418M, however, this model actually has 483M parameters.
11Version is gpt-3.5-turbo-0125.
12Version is gpt-4o-mini-2024-07-18.
13Due to the limitation of computational resources, NLLB-3.3B is not included in fine-tuning with full parameters.
<table>
  <tr>
    <th rowspan="2">model</th>
    <th colspan="2">5-direction</th>
    <th colspan="2">25-direction</th>
    <th colspan="2">100-direction</th>
  </tr>
  <tr>
    <th>spB.</th><th>com.</th><th>spB.</th><th>com.</th><th>spB.</th><th>com.</th>
  </tr>
  <tr>
    <td>N.-615M</td>
    <td>pre.</td><td>24.00</td><td>82.91</td><td>25.88</td><td>83.71</td><td>25.37</td><td>83.35</td>
  </tr>
  <tr>
    <td></td>
    <td>lora</td><td>24.68</td><td>83.23</td><td>26.84</td><td>84.10</td><td>26.41</td><td>84.00</td>
  </tr>
  <tr>
    <td></td>
    <td>f.t.</td><td>25.59</td><td>83.80</td><td>27.32</td><td>84.29</td><td>26.70</td><td>84.17</td>
  </tr>
  <tr>
    <td></td>
    <td>pre.</td><td>25.59</td><td>84.66</td><td>28.35</td><td>85.74</td><td>27.62</td><td>85.75</td>
  </tr>
  <tr>
    <td>N.-1.3B</td>
    <td>lora</td><td>27.39</td><td>85.20</td><td>29.49</td><td>85.87</td><td>29.18</td><td>85.87</td>
  </tr>
  <tr>
    <td></td>
    <td>f.t.</td><td>28.50</td><td>85.78</td><td>30.13</td><td>86.09</td><td>29.61</td><td>86.05</td>
  </tr>
  <tr>
    <td></td>
    <td>pre.</td><td>27.95</td><td>85.60</td><td>29.70</td><td>86.16</td><td>29.51</td><td>85.96</td>
  </tr>
  <tr>
    <td>N.-3.3B</td>
    <td>lora</td><td>29.05</td><td>86.08</td><td>31.23</td><td>86.63</td><td>30.98</td><td>86.71</td>
  </tr>
  <tr>
    <td></td>
    <td>pre.</td><td>24.57</td><td>83.75</td><td>28.71</td><td>85.41</td><td>29.07</td><td>85.56</td>
  </tr>
  <tr>
    <td>M.-466M</td>
    <td>lora</td><td>26.37</td><td>84.47</td><td>29.97</td><td>86.09</td><td>30.42</td><td>86.27</td>
  </tr>
  <tr>
    <td></td>
    <td>f.t.</td><td>28.19</td><td>85.28</td><td>30.61</td><td>86.36</td><td>30.81</td><td>86.45</td>
  </tr>
  <tr>
    <td></td>
    <td>pre.</td><td>25.52</td><td>84.56</td><td>29.95</td><td>86.07</td><td>30.57</td><td>85.92</td>
  </tr>
  <tr>
    <td>M.-913M</td>
    <td>lora</td><td>28.14</td><td>85.59</td><td>31.68</td><td>86.90</td><td>32.33</td><td>87.15</td>
  </tr>
  <tr>
    <td></td>
    <td>f.t.</td><td>30.09</td><td>86.45</td><td>32.47</td><td>87.23</td><td>32.73</td><td>87.35</td>
  </tr>
</table>

Table 4: Averaged spBLEU and COMET scores of results on three fine-tuning scenarios, where the specific translation directions are listed in Appendix E. N., M., pre., and f.t. abbreviate NLLB, MITRE, pre-trained models, and fine-tuning with full parameters, respectively. The best score is in bold, blue boxes highlights the largest gain in f.t. relative to pre., and green boxes highlights the largest gain in lora.

we conclude that MITRE-466M performs competitively with NLLB-3.3B, while MITRE-913M not only outperforms NLLB-3.3B but also competes with GPT-4o mini14, showing the practical potential of our models.

5.4 Fine-tuning Results

Table 4 shows the fine-tuning results. By comparing NLLB and MITRE, we observe that MITRE outperforms NLLB in both scenarios: fine-tuning on a few translation directions and fine-tuning on multiple translation directions simultaneously. Specifically, we find that performance gains from fine-tuning increase with model size, and our MITRE-913M shows the highest improvement in both full parameter fine-tuning and LoRA-based fine-tuning. Additionally, since pre-trained models of both NLLB and MITRE achieve near-zero off-target ratios, these gains can be attributed to increased quality rather than addressing the off-target problem. This suggests that MITRE has a higher performance ceiling, likely due to our cost-effective data collection strategy, which may have constrained MITRE from reaching its theoretical maximum. Therefore, due to MITRE’s superior fine-tuning capability, we reaffirm that its practical potential is remarkable.

14Table 13 shows that the COMET scores of MITRE-913M are lower than GPT-4o mini. Therefore, although spBLEU and chrF++ scores of MITRE-913M are higher, we only claim that MITRE-913M performs competitively with GPT-4o mini. We provide an additional discussion in Appendix I.

<table>
  <tr>
    <th>#layer</th>
    <th>register</th>
    <th>mask</th>
    <th>spB.↑</th>
    <th>chrF↑</th>
    <th>com.↑</th>
    <th>off.↓</th>
  </tr>
  <tr>
    <td>12</td>
    <td>✗</td>
    <td>✗</td>
    <td>8.34</td>
    <td>22.34</td>
    <td>55.71</td>
    <td>22.21</td>
  </tr>
  <tr>
    <td></td>
    <td>✓</td>
    <td>✗</td>
    <td>8.19</td>
    <td>22.96</td>
    <td>55.72</td>
    <td>32.11</td>
  </tr>
  <tr>
    <td></td>
    <td>✓</td>
    <td>✓</td>
    <td>11.92</td>
    <td>29.02</td>
    <td>61.19</td>
    <td>4.65</td>
  </tr>
  <tr>
    <td>24</td>
    <td>✗</td>
    <td>✗</td>
    <td>10.82</td>
    <td>26.04</td>
    <td>59.95</td>
    <td>19.01</td>
  </tr>
  <tr>
    <td></td>
    <td>✓</td>
    <td>✗</td>
    <td>8.91</td>
    <td>24.06</td>
    <td>57.60</td>
    <td>34.22</td>
  </tr>
  <tr>
    <td></td>
    <td>✓</td>
    <td>✓</td>
    <td>13.12</td>
    <td>30.51</td>
    <td>63.54</td>
    <td>3.65</td>
  </tr>
</table>

Table 5: Averaged scores of ablation study on EC-40. Here, register means adding registers, and mask means modifying the attention mask.

![The spBLEU score variations on EC-40 where the x-axis is len(x')/len(r), where only the length of r is changed and x' is fixed.](page_1012_1042_388_246.png)

Figure 3: The spBLEU score variations on EC-40 where the x-axis is len(x')/len(r), where only the length of r is changed and x' is fixed.

6 Discussion

6.1 Ablation Study

We conduct two ablation studies to measure the impact of registering. First, we decompose registering into two steps: (1) adding registers and (2) modifying the attention mask. As shown in Table 5, merely adding registers reduces the performance of vanilla Dec-only models; registering only becomes effective after modifying the attention mask. This result aligns with expectations, i.e., the model without constraints on generation defaults to relying directly on source tokens instead of registers.

In Section 3.2, we state that the lengths of r and x are matched to ensure a one-to-one correspondence between registers and source tokens. To validate this design, we vary the length of r while keep x' fixed to observe performance trends. Specifically, a ratio less than 1.0 means that r is an augmenting of x', a ratio greater than 1.0 means r is a compressing of x', and a ratio of 1.0 means the registering. The trend illustrated in Figure 3 empirically supports that registering is the optimal mechanism.

To analyze the mechanism differences among augmenting, registering, and compressing, we analyze the attention alignment between registers and
<table>
  <tr>
    <th>Ratio</th>
    <th>Mechanism</th>
    <th>T.1 S.</th>
    <th>T.2 S.</th>
    <th>Dist.</th>
    <th>Entropy↑</th>
  </tr>
  <tr>
    <td>0.75</td>
    <td>augmenting</td>
    <td>1.68</td>
    <td>0.72</td>
    <td>2.67</td>
    <td>5.15</td>
  </tr>
  <tr>
    <td>1</td>
    <td>registering</td>
    <td>1.80</td>
    <td>0.78</td>
    <td>2.09</td>
    <td><b>5.25</b></td>
  </tr>
  <tr>
    <td>1.25</td>
    <td>compressing</td>
    <td>2.13</td>
    <td>0.94</td>
    <td>1.53</td>
    <td>4.91</td>
  </tr>
  <tr>
    <td>1.5</td>
    <td>compressing</td>
    <td>2.15</td>
    <td>0.94</td>
    <td>1.62</td>
    <td>4.73</td>
  </tr>
</table>

Table 6: Attention mechanism across different ratios, i.e., \( \text{len}(x)/\text{len}(r) \). T.1 S. and T.2 S. denote average top-1 and top-2 attention scores; Dist. is the average positional distance between the top-2 source tokens; Entropy measures the diversity of source token selection. Higher entropy indicates more diverse attention; lower values suggest focus on a narrower set of tokens.

source tokens based on models trained on EC-40. For each register, we extract the source tokens with the highest and the second-highest attention scores. We then measure the entropy of the source token selection distribution for each sentence, i.e., how frequently each source token is selected as the top-1 attention. Results based on 100 random instances are summarized in Table 6. Among the three mechanisms, registering yields the highest entropy, indicating a more diverse use of source tokens, i.e., each token is likely to be selected as a top attention target at least once. Augmenting, by contrast, results in lower scores and longer distances, suggesting redundancy and diffused attention. Compressing, while producing high scores, focuses on short contiguous spans and exhibits reduced entropy, implying potential neglect of broader contextual information. These findings supplement the empirical results in Figure 3 and reinforce registering as the optimal mechanism.

6.2 Mechanism: registering source tokens to target language spaces

We first reveal the registering mechanism at the token level from two perspectives: (1) representations of registers are located in the intended target language space, and (2) registers carry the semantics of the positionally-aligned source token.

We analyze token representations by randomly selecting 100 translation instances from three translation directions and applying t-SNE (van der Maaten and Hinton, 2008) to reduce them to two dimensions. Figure 4 shows the representation distribution in the final layer. First, source tokens, registers, and target tokens are clearly separated into distinct spaces, indicating that the model can distinguish their different functions. Additionally, source token representations from different languages cluster, suggesting that the model processes them in a language-agnostic manner. Most importantly, for registers and target tokens, token representations for the three translation directions cluster in separate spaces. This supports our design, where the register representation is located in the intended target language space.

The relationship between source tokens and registers can be exhibited by analyzing the attention weights in generation based on a simple and interesting grammar of de. Figure 5 shows two translation instances translated from de to en, which have the same semantics. We observe that the target tokens in both examples are identical, while their source tokens differ only in the verb form (highlighted with a purple border). Specifically, in Figure 5a, the verb is a single token, “öffne”, whereas in Figure 5b, it consists of two distant tokens, “mache ... auf”. Despite this difference, “öffne” and “mache ... auf” share the same semantics and can be replaced by each other. Figure 5 presents the attention weights for each token representation. In Figure 5a, the highest attention weight of the verb in the target tokens, i.e., “open”, comes from \( r_1 \); in Figure 5b, “open” pays the highest attention weight to \( r_1 \) and \( r_6 \). Meanwhile, \( r_1 \) in Figure 5a aligns positionally with “öffne”, and in Figure 5b, \( r_1 \) and \( r_6 \) align positionally with “mache ... auf”. Additionally, we observe that the highest attention weight of a register always comes from its positionally-aligned source token15.

![2D distribution of token-level representations extracted from the output of the 24th layer of a model trained on EC-40. Each class listed in the legend contains 300 randomly sampled tokens. Appendix L shows the representational distributions from other layers.](page_1012_186_410_312.png)

Figure 4: 2D distribution of token-level representations extracted from the output of the 24th layer of a model trained on EC-40. Each class listed in the legend contains 300 randomly sampled tokens. Appendix L shows the representational distributions from other layers.

15 Appendix K shows more examples in Russian, Chinese, and Japanese, where all instances follow this statement.
Figure 5: Token-level attention weights illustration, where the weight of each token is averaged across all heads of a model trained on EC-40. 5a and 5b illustrate two instances translated from de to en. The top-3 attention directions for each token are labeled, with darker colors indicating higher attention weights. Note that while the target tokens for these two instances are identical, their source tokens are not, because the verbs in 5a and 5b are semantically equivalent but have different forms. To aid understanding, we highlight the verbs with purple borders: “öffne” in 5a and “mache ... auf” in 5b both correspond to the target verb “open”. Then, the registers with the highest attention weights associated with these verbs are also marked with purple borders.

Figure 6: Layer-wise cosine similarity among sentence-level representations of source tokens, registers, and target tokens. Results are averaged over 10 random instances processed by a model trained on EC-40.

Beyond theorizing the mechanism through token-level analyses, we also conduct a sentence-level representation analysis16 to validate the mechanism. As shown in Figure 6, sentence-level cosine similarity between source and target is the highest in the lower layers, reflecting strong semantic alignment due to the use of gold translations and indicating that lower layers primarily encode shared semantic content. As depth increases, source-target similarity drops noticeably, suggesting that upper layers capture more language-specific features. Mean-

16We follow Liu et al. (2021) to apply mean-pooling over the token representations to obtain sentence representations, and then compute cosine similarity between them. For each translation instance, we extract representations for source tokens, registers, and target tokens.

while, source-register similarity remains relatively stable until the top layers, where it drops sharply. In contrast, register-target similarity gradually decreases in the middle layers but rises sharply in the top layers. In the final layer, registers are most similar to the target, followed by those to the source, while source-target similarity is lowest. These trends indicate that registers transition toward the target space while maintaining semantic ties to the source, supporting our token-level analyses. Based on the above, we conclude that the register’s activation represents the target language and carries the semantics of the positionally-aligned token, namely, registers act as “rethinking” the source from the perspective of the target language.

7 Conclusion

In this work, we present registering to address the off-target problem in MNMT-specific models. By introducing registers and modifying the attention mask, our method ensures that the generation of target tokens depends solely on the activation of registers. Analytical experiments demonstrate that the activation of registers carries the semantics of source tokens within the target language spaces. Using this method, we develop and open-source two MNMT-specific models, MITRE-466M and MITRE-913M, supporting translation across 24 languages. Experimental results show that MITRE performs competitively with commercial LLMs, setting a new state-of-the-art in MNMT.
Limitations

A key concern is our limited computational resources. Given that the training of MITRE-913M has already required 80 Tesla V100 GPUs for one month, MITRE supports 24 languages, and we cannot further increase the supported languages. Although this number is far greater than the latest research in the community, e.g., 10 languages of Alves et al. (2024) and 5 languages of Xu et al. (2024), this number is fewer than the number of supported languages of M2M, NLLB, and commercial LLMs. However, the comparison in Section 5 is relatively fair. Specifically (using NLLB as an example), first, our training data is collected from the reproduced version of the NLLB dataset, which includes fewer samples per translation direction than those used for training NLLB models. Second, as described in Section 5.1, our Bridge Language strategy results in fewer supervised translation directions, whereas NLLB is trained on as many directions as possible. Moreover, NLLB incorporates additional engineering strategies, e.g., back-translation (Edunov et al., 2018) and distillation (Hinton et al., 2015), whereas MITRE only iterates over the training set. Also, we directly compare MITRE-466M and MITRE-913M to NLLB-3.3B, where the parameter size difference helps offset the disparity in supported languages. Finally, we conduct fine-tuning experiments to compare MITRE and NLLB with the same settings.

Another limitation of our approach is the additional computational cost introduced by registers, as they double the number of source tokens. Based on our measurements on EC-40 using a Tesla V100 GPU, the training time for models with registers is 1.34 times that of the vanilla decoder-only model, 1.63 times that of the vanilla encoder-decoder model, and approximately equivalent (1.01 times) to the previous state-of-the-art method, CL (Pan et al., 2021). At inference time, thanks to the KV cache, the model with registers incurs only a linear and affordable increase in inference cost, comparable to other MNMT-specific models. To validate this, we randomly translate 100 sentences using publicly available implementations of M2M, NLLB, and MITRE from HuggingFace (with batch size 1 and beam size 5), and repeat each experiment 10 times. Results in Table 7 confirm our claim. Additionally, in practical usage of MITRE, the inference cost is substantially lower than that of LLMs due to the smaller number of parameters.

<table>
  <tr>
    <th>Model</th>
    <th>#Tokens</th>
    <th>Times (s)</th>
    <th>Times per Token (s)</th>
  </tr>
  <tr>
    <td>MITRE-466M</td>
    <td>3667</td>
    <td>64.95</td>
    <td>0.0177</td>
  </tr>
  <tr>
    <td>MITRE-913M</td>
    <td>3626</td>
    <td>85.65</td>
    <td>0.0236</td>
  </tr>
  <tr>
    <td>M2M-483M</td>
    <td>4226</td>
    <td>52.22</td>
    <td>0.0124</td>
  </tr>
  <tr>
    <td>M2M-1.2B</td>
    <td>4234</td>
    <td>89.75</td>
    <td>0.0212</td>
  </tr>
  <tr>
    <td>NLLB-600M</td>
    <td>3990</td>
    <td>41.29</td>
    <td>0.0103</td>
  </tr>
  <tr>
    <td>NLLB-1.3B</td>
    <td>4104</td>
    <td>74.37</td>
    <td>0.0181</td>
  </tr>
  <tr>
    <td>NLLB-3.3B</td>
    <td>3837</td>
    <td>82.15</td>
    <td>0.0214</td>
  </tr>
</table>

Table 7: Generation cost measurement. Tokens refer to the total tokens generated during inference, and Times refer to the time cost of inference, which is counted by seconds.

Ethical Considerations

Although our training data is collected from public datasets, MITRE has not been evaluated for toxicity or has undergone detoxification. Thus, while we open-source MITRE, we recommend its use primarily for research purposes or in applications only after thorough appropriate processing.

References

Duarte Miguel Alves, José Pombal, Nuno M Guerreiro, Pedro Henrique Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and Andre Martins. 2024. Tower: An open multilingual large language model for translation-related tasks. In First Conference on Language Modeling.

Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. Preprint, arXiv:1907.05019.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.

Mengyu Bu, Shuhao Gu, and Yang Feng. 2024. Improving multilingual neural machine translation by utilizing semantic and linguistic features. In Findings of the Association for Computational Linguistics ACL 2024, pages 10410–10423, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
Zhe Cao, Zhi Qu, Hidetaka Kamigaito, and Taro Watanabe. 2024. Exploring intrinsic language-specific subspaces in fine-tuning multilingual neural machine translation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21142–21157, Miami, Florida, USA. Association for Computational Linguistics.

Liang Chen, Shuming Ma, Dongdong Zhang, Furu Wei, and Baobao Chang. 2023. On the off-target problem of zero-shot multilingual neural machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9542–9558, Toronto, Canada. Association for Computational Linguistics.

Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.

Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. Association for Computational Linguistics.

Akiko Eriguchi, Shufang Xie, Tao Qin, and Hany Hassan. 2022. Building multilingual machine translation systems that serve arbitrary XY translations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 600–606, Seattle, United States. Association for Computational Linguistics.

Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2021. Beyond english-centric multilingual machine translation. J. Mach. Learn. Res., 22(1).

Orhan Firat, Baskaran Sankaran, Yaser Al-onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. 2016. Zero-resource translation with multi-lingual neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 268–277, Austin, Texas. Association for Computational Linguistics.

Yingbo Gao, Christian Herold, Zijian Yang, and Hermann Ney. 2022. Is encoder-decoder redundant for neural machine translation? In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 562–574, Online only. Association for Computational Linguistics.

Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O.K. Li. 2019. Improved zero-shot neural machine translation via ignoring spurious correlations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1258–1268, Florence, Italy. Association for Computational Linguistics.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. Preprint, arXiv:1503.02531.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.

Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339–351.

Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. Preprint, arXiv:1412.6980.

Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics.

Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, Online. Association for Computational Linguistics.

Danni Liu, Jan Niehues, James Cross, Francisco Guzmán, and Xian Li. 2021. Improving zero-shot translation by disentangling positional information. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1259–1273, Online. Association for Computational Linguistics.

Jesse Mu, Xiang Lisa Li, and Noah Goodman. 2023. Learning to compress prompts with gist tokens. In Thirty-seventh Conference on Neural Information Processing Systems.

NLLB Team. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672.
OpenAI. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.

Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. 2021. Contrastive learning for many-to-many multilingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 244–258, Online. Association for Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.

Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.

Zhi Qu, Chenchen Ding, and Taro Watanabe. 2024a. Languages transferred within the encoder: On representation transfer in zero-shot multilingual translation. Preprint, arXiv:2406.08092.

Zhi Qu, Yiran Wang, Chenchen Ding, Hideki Tanaka, Masao Utiyama, and Taro Watanabe. 2024b. Improving language transfer capability of decoder-only architecture in multilingual neural machine translation. Preprint, arXiv:2412.02101.

Zhi Qu and Taro Watanabe. 2022. Adapting to non-centered languages for zero-shot multilingual translation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5251–5265, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, and 1 others. 2018. Improving language understanding by generative pre-training.

Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.

Annette Rios, Mathias Müller, and Rico Sennrich. 2020. Subword segmentation and a single bridge language affect zero-shot neural machine translation. In Proceedings of the Fifth Conference on Machine Translation, pages 528–537, Online. Association for Computational Linguistics.

David Stap, Vlad Niculae, and Christof Monz. 2023. Viewing knowledge transfer in multilingual machine translation through a representational lens. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14973–14987, Singapore. Association for Computational Linguistics.

Zengkui Sun, Yijin Liu, Fandong Meng, Jinan Xu, Yufeng Chen, and Jie Zhou. 2024. LCS: A language converter strategy for zero-shot neural machine translation. In Findings of the Association for Computational Linguistics ACL 2024, pages 9201–9214, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Shaomu Tan and Christof Monz. 2023. Towards a better understanding of variations in zero-shot neural machine translation performance. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13553–13568, Singapore. Association for Computational Linguistics.

Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579–2605.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.

Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Aligning large language models with human: A survey. Preprint, arXiv:2307.12966.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems.

Liwei Wu, Shanbo Cheng, Mingxuan Wang, and Lei Li. 2021. Language tags matter for zero-shot neural machine translation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3001–3007, Online. Association for Computational Linguistics.

Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. 2020. On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10524–10533. PMLR.

Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024. A paradigm shift in machine translation: Boosting translation performance of large language models. In The Twelfth International Conference on Learning Representations.

Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023. Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages. Preprint, arXiv:2305.18098.

Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. 2021. Share or not? learning to schedule language-specific capacity for multilingual translation. In International Conference on Learning Representations.

Biao Zhang, Behrooz Ghorbani, Ankur Bapna, Yong Cheng, Xavier Garcia, Jonathan Shen, and Orhan Firat. 2022. Examining scaling and transfer of language model architectures for machine translation. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 26176–26192. PMLR.

Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1628–1639, Online. Association for Computational Linguistics.

Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual machine translation with large language models: Empirical results and analysis. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2765–2781, Mexico City, Mexico. Association for Computational Linguistics.
A  Description of EC-40

EC-40 is an English-centric dataset introduced by Tan and Monz (2023). In addition to English, it includes 40 languages spanning five language families, with each family containing eight languages. These languages are categorized into four tiers based on data availability: High, Medium, Low, and Extra Low. Each non-English language is paired with English, resulting in 80 supervised translation directions used for training and 1,560 zero-shot translation directions. Details of this dataset are summarized in Table 14.

B  Description of Pre-training Dataset

Our pre-training dataset comprises 24 languages, as detailed in Table 8. As described in Section 5.1, our data collection strategy results in 9.3 billion translation instances across 194 translation directions. The data distribution is visualized at the family level in Figure 9a and at the language level in Figure 9b. Additionally, Figure 9b highlights which translation directions are supervised and which are zero-shot. Notably, translation directions involving ms are also indicated in Figure 9b.

C  Training Details of EC-40

Training configurations  We employ Fairseq (Ott et al., 2019), an open-source toolkit, to implement our models with methods mentioned in Section 4.2. First, we directly reuse the vocabulary and binary training data provided by Chen et al. (2023)17. Note that we include only supervised translation directions in validation. We train on 8 Tesla V100 GPUs, setting memory-efficient-fp16 in Fairseq, with a maximum input of 2048 source tokens per GPU and a gradient accumulation of 16 steps. Both input and output token lengths are limited to 256, and we share the embedding layer between the encoder and decoder. We use a seed of 1234, a learning rate of 0.0005 with the inverse square root schedule and a warmup of 4000 steps, the Adam optimizer (Kingma and Ba, 2017), dropout of 0.1, attention dropout of 0.1, a label smoothing rate of 0.1, no weight decay, and the temperature sampling with \( T = 5 \) (Arivazhagan et al., 2019). Finally, we train for 200,000 steps, averaging the last 5 checkpoints, saving by epoch.

Configurations of Related Methods  Generally, both our method and these related methods share the same hyper-parameters as the vanilla models. However, there are some method-specific configurations we have to notice. (1) For LAVS (Chen et al., 2023), we directly reuse their code18 and add around 12k language-specific tokens into the shared vocabulary, resulting in 12.8M additional parameters in modeling; (2) For CL (Pan et al., 2021), we directly reuse their code19 and set the contrastive learning temperature to 0.1, which is the optimal setting according to their reports; (3) For LCS (Sun et al., 2024), the model follows another translation instruction strategy of Fan et al. (2021) by adding a source language tag at the beginning of the source tokens and a target language tag at the beginning of the target tokens. we reimplement their code and, in the case of 12-layer models where the encoder has 6 layers, apply LCS biasing at the 5th encoder layer; For models with 12 encoder layers, we apply it at the 8th encoder layer; (4) For TDO (Qu et al., 2024b), we also reuse their code20 and, based on their ablation study, set the number of layers for the first stage to 3 in 12-layer models and to 6 in 24-layer models to allow stronger zero-shot translation ability.

D  Training Details of MITRE

We employ Fairseq to implement MITRE mentioned in Section 5.2, and two versions of MITRE have the different configurations in modeling and have the same configuration in training. Specifically, MITRE-466M is configured with an embedding size of 1,024, an inner size of 4,096, 16 attention heads, and 24 layers. MITRE-913M,

![The training and validation loss in pre-training MITRE. We report the first 100 epochs, each with approximately 2262 steps.](page_1012_120_388_246.png)

Figure 7: The training and validation loss in pre-training MITRE. We report the first 100 epochs, each with approximately 2262 steps.

17https://github.com/Smu-Tan/ZS-NMT-Variations
18https://github.com/PKUnlp-icler/Off-Target-MNMT
19https://github.com/PANxiao1994/mRASP2
20https://github.com/zhiqu22/PhasedDecoder
<table>
  <tr>
    <th>Family</th>
    <th>ISO code</th>
    <th>Flores code</th>
    <th>Language</th>
    <th>script</th>
  </tr>
  <tr>
    <td rowspan="4">Germanic</td>
    <td>en</td>
    <td>eng_Latin</td>
    <td>English</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>de</td>
    <td>deu_Latin</td>
    <td>German</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>nl</td>
    <td>nld_Latin</td>
    <td>Dutch</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>sv</td>
    <td>swe_Latin</td>
    <td>Swedish</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>da</td>
    <td>dan_Latin</td>
    <td>Danish</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>af</td>
    <td>afr_Latin</td>
    <td>Afrikaans</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td rowspan="6">Romance</td>
    <td>fr</td>
    <td>fra_Latin</td>
    <td>French</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>es</td>
    <td>spa_Latin</td>
    <td>Spanish</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>it</td>
    <td>ita_Latin</td>
    <td>Italian</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>pt</td>
    <td>por_Latin</td>
    <td>Portuguese</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>ro</td>
    <td>ron_Latin</td>
    <td>Romanian</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td rowspan="5">Slavic</td>
    <td>ru</td>
    <td>rus_Cyril</td>
    <td>Russian</td>
    <td>Cyrillic</td>
  </tr>
  <tr>
    <td>cs</td>
    <td>ces_Latin</td>
    <td>Czech</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>pl</td>
    <td>pol_Latin</td>
    <td>Polish</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>bg</td>
    <td>bul_Cyril</td>
    <td>Bulgarian</td>
    <td>Cyrillic</td>
  </tr>
  <tr>
    <td>uk</td>
    <td>ukr_Cyril</td>
    <td>Ukrainian</td>
    <td>Cyrillic</td>
  </tr>
  <tr>
    <td rowspan="3">Malayo-Polynesian</td>
    <td>id</td>
    <td>ind_Latin</td>
    <td>Indonesian</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>ms</td>
    <td>zsm_Latin</td>
    <td>Malay</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>jv</td>
    <td>jav_Latin</td>
    <td>Javanese</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>tl</td>
    <td>fil_Latin</td>
    <td>Filipino</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td rowspan="4">Asian*</td>
    <td>ko</td>
    <td>kor_Hang</td>
    <td>Korean</td>
    <td>Hangul</td>
  </tr>
  <tr>
    <td>vi</td>
    <td>vie_Latin</td>
    <td>Vietnamese</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>ja</td>
    <td>jpn_Jpan</td>
    <td>Japanese</td>
    <td>Kanji; Kana</td>
  </tr>
  <tr>
    <td>zh</td>
    <td>cnn_Hans</td>
    <td>Chinese</td>
    <td>Chinese</td>
  </tr>
</table>

Table 8: Details of the dataset in our pre-training. The decoration * on Asian means a group instead of a language family. We not only list the ISO 639-1 code for each language but also list the Flores code to help search corresponding resources from Flores+.

a larger model with expanded width and depth, has an embedding size of 1,280, an inner size of 5,120, 20 heads, and 36 layers. In training, we first train a shared SentencePiece vocabulary (Kudo and Richardson, 2018) with a size of 160,000 by 150 million sentences randomly sampled from the training set. We include only supervised translation directions in validation. Then, we train on 80 Tesla V100 GPUs, setting memory-efficient-fp16 in Fairseq, with a maximum input of 1408 source tokens per GPU and a gradient accumulation of 10 steps. In practice, this setup results in each batch containing approximately 0.91 million source tokens. Given the large batch size, we set the learning rate of 0.002 with the inverse square root schedule and the warmup of 8000 steps. We also use a seed of 42, the Adam optimizer (Kingma and Ba, 2017), dropout of 0.1, attention dropout of 0.1, a label smoothing rate of 0.1, no weight decay, and the temperature sampling with \( T = 1 \) (Arivazhagan et al., 2019). We train for 300,000 steps and save a checkpoint per 10,000 steps. Finally, we average the last 5 checkpoints. Figure 7 shows the variations of training and validation loss. We can observe that the trends of MITRE-466M and MITRE-913M are highly consistent.

E Training Details of Fine-tuning

Selecting Directions  We use random.sample in Python to randomly select translation directions for fine-tuning, setting the seed to 0. We define three scenarios, including 5, 25, and 100 translation directions. It is important to note that random.sample causes the 5 and 25 directions to be subsets of the 100 directions. Specifically, the first 5 and first 25 directions in the 100-direction set correspond to the other two scenarios. The 100 directions are: jv→sv, ms→id, de→tl, ru→pl, ko→jv, zh→bg, ms→en, pl→ru, zh→af, uk→ko, pt→jv, ko→ro, fr→da, cs→pl, fr→af, da→fr, ru→sv, fr→pl, pl→tl, da→ro, sv→es, bg→jv, zh→en, da→cs, uk→ms, tl→es, bg→de, pt→nl, vi→bg, tl→id, ru→bg, nl→ms, en→uk, da→sv, jv→ms, en→nl, zh→vi, bg→ja, ro→ja, bg→ru, nl→tl, vi→es, ja→pt, cs→uk, da→ko, af→it, jv→zh, zh→cs, sv→da, ko→pt, cs→nl, pt→vi, nl→en, vi→ja, es→nl, tl→ru, ru→es, ja→jv, ro→zh, nl→ro, fr→jv, cs→fr, fr→cs, uk→jv, ko→bg, cs→da, es→ro, ms→sv, ja→cs, cs→en, da→pl, jv→tl, pl→pt, zh→sv, pl→de, fr→ro, pt→zh, zh→id, pl→fr, ko→ru, it→bg, es→de, cs→tl, af→pt, fr→ru, da→nl, da→af, ms→fr, ko→cs, en→jv, p1→uk, bg→uk, af→tl, ro→bg, de→pl, de→vi, uk→nl, id→ja, nl→zh, zh→pl

Fine-tuning Configurations  All fine-tuning experiments use the same settings. We conduct experiments on 8 Tesla V100 GPUs, with a maximum of 1024 tokens per GPU and a gradient accumulation of 2 steps. Based on the pre-trained model, we set the learning rate to 0.0001 with a warmup step of 1 (for launching the inverse square root schedule), and train for 10 epochs. Finally, we use the last epoch for testing.

LoRA Configurations  We adopt the setting of Hu et al. (2022) to implement LoRA components for pre-trained models. Specifically, LoRA is only implemented for Query and Value in the attention mechanism with a rank of 8. As a result, the learnable parameters of NLLB series are 1.18M, 2.36M, and 4.72M, respectively, and the learnable parameters of MITRE series are 0.78M and 1.47M, respectively.

F Prompts for GPT

Our prompts for GPT series follow: Translating the following sentence from [SRC] to [TGT]: [INPUT]. Here, [SRC] and [TGT] are the source and target
language names following Table 8, and [INPUT] is the source sentence. We find that GPT occasionally repeats [INPUT] in the output. Once it happens, we manually remove the [INPUT] before evaluation.

G  Details of Evaluation Metrics

In evaluating the performance of models trained on EC-40, some languages lack support from COMET (Unbabel/wmt22-comet-da) and the off-target ratio (fast-langdetect). Notably, fast-langdetect operates by word recognition, so we also exclude certain supported languages that exhibit low recognition success rates. We list the supported languages in this section.

Languages in COMET: en, bg, so, ca, da, be, bs, es, uk, am, hi, ro, no, de, cs, pt, nl, mr, is, ne, ur, ha, sv, gu, ar, fr, ru, it, pl, sr, sd, he, af, kn, bn.

Languages in Off-target Ratio: en, bg, da, es, uk, hi, ro, de, cs, pt, nl, mr, ur, sv, gu, ar, fr, ru, it, pl, he, kn, bn, be, mt, am, is, sd.

H  Supplementary Results of EC-40

In Section 4.3 and Table 1, we report spBLEU scores and off-target ratio. In this appendix, we report chrF++ and COMET scores in Tables 10 and 11, respectively. Overall, four metrics show consistent trends across this benchmark.

I  Supplementary Results of MITRE

In Section 5.3 and Table 3, we report spBLEU scores and do not report the off-target ratio, because the values are near zero across those large-scale pre-trained models. In this appendix, we report chrF++ and COMET scores in Tables 12 and 13, respectively. However, when comparing MITRE and commercial LLMs, COMET reveals a different trend: MITRE-913M underperforms GPT-4o mini, despite similar trends in spBLEU and chrF++ scores, which show MITRE-913M as superior to GPT-4o mini. This suggests that while MITRE generates more accurate sequences relative to the test set, GPT-4o mini produces more fluent and natural output. This is expected, as commercial models are aligned with human-like styles (Wang et al., 2023), whereas MITRE follows the training data’s style. Further supporting this, GPT-4o mini shows a significant improvement over GPT-3.5 turbo (Table 13). Therefore, based on the results across all three metrics, our claim is not that MITRE-913M outperforms GPT-4o mini, but rather that MITRE-913M outperforms NLLB 3.3B and can compete with GPT-4o mini.

J  Supporting Experiments for Comparing Enc-dec and Dec-only

In Section 3.2, we explain our reason for implementing registering in Dec-only. Specifically, Dec-only offers better parameter efficiency than Enc-dec, where the encoder learns the representation of input tokens while the decoder learns the generated tokens (as illustrated in Figure 1, where encoded source tokens are shown in gray). In contrast, Dec-only utilizes all parameters for both encoding and

![Three cases of attention analysis on MITRE-466M. Details of this illustration, e.g. colors, classes, and arrows, follow that of Figure 5.](page_1012_180_377_1012.png)

Figure 8: Three cases of attention analysis on MITRE-466M. Details of this illustration, e.g. colors, classes, and arrows, follow that of Figure 5.
<table>
  <tr>
    <th>Methods</th>
    <th>#layer</th>
    <th>spBLEU</th>
    <th>chrF++</th>
    <th>COMET</th>
    <th>Off-target(%)</th>
  </tr>
  <tr>
    <td>Enc-dec</td>
    <td></td>
    <td>6.99</td>
    <td>19.68</td>
    <td>53.72</td>
    <td>48.4</td>
  </tr>
  <tr>
    <td>+registering</td>
    <td></td>
    <td>10.55</td>
    <td>27.29</td>
    <td>58.07</td>
    <td>7.5</td>
  </tr>
  <tr>
    <td>Dec-only</td>
    <td>12</td>
    <td>8.34</td>
    <td>22.34</td>
    <td>55.71</td>
    <td>22.21</td>
  </tr>
  <tr>
    <td>+registering</td>
    <td></td>
    <td><b>11.92</b></td>
    <td><b>29.02</b></td>
    <td><b>61.19</b></td>
    <td><b>4.65</b></td>
  </tr>
  <tr>
    <td>Enc-dec</td>
    <td></td>
    <td>9.69</td>
    <td>23.95</td>
    <td>58.08</td>
    <td>26.69</td>
  </tr>
  <tr>
    <td>+registering</td>
    <td></td>
    <td>11.13</td>
    <td>27.91</td>
    <td>58.67</td>
    <td>3.91</td>
  </tr>
  <tr>
    <td>Dec-only</td>
    <td>24</td>
    <td>10.82</td>
    <td>26.04</td>
    <td>59.95</td>
    <td>19.01</td>
  </tr>
  <tr>
    <td>+registering</td>
    <td></td>
    <td><b>13.12</b></td>
    <td><b>30.51</b></td>
    <td><b>63.54</b></td>
    <td><b>3.65</b></td>
  </tr>
</table>

Table 9: Results on EC-40. For convenience, the score in this table is averaged from all 1,640 translation directions. Enc-dec and Dec-only indicate the vanilla models without registering. The best score is in bold.

generation. Based on this, we employ Dec-only as the backbone of our implementation.

To further support our statement, we provide supporting experiments on EC40, following the experimental setup described in Section 4.2. As shown in Table 9, registering also significantly improves the performance of Enc-dec. However, the gains are more pronounced in Dec-only, suggesting that registering is particularly well-suited for this architecture.

Additionally, we propose an insight beyond the scope of this work. In Section 3.2, we mention a potential cause of the off-target problem: the dilution of translation instruction attention by other tokens (Gu et al., 2019; Qu et al., 2024b). This theory may explain why Dec-only tends to underperform Enc-dec in MNMT (Gao et al., 2022; Zhang et al., 2022), as the effectiveness of \( l_y \) is further diluted by both source and target tokens in the attention mechanism. The results in Table 9 further support this explanation, as registering achieves a much larger performance gain in Dec-only than in Enc-dec.

K Supplementary Analysis of Attention

To further support our analysis in Section 6.2, we examine additional cases in MITRE-466M where the source and target sentences exhibit significant structural differences. In all cases, the attention relationship between registers and source tokens remains consistent, i.e., one-to-one attention weights being the most prominent. Next, we observe the following patterns: (1) As shown in Figure 8a, in Japanese, the attention for “drink” points to \( r_6 \), while “friend” points to \( r_1 \) and \( r_2 \). (2) As shown in Figure 8b, “friend” points to \( r_2 \), and “book” points to \( r_5 \). (3) As shown in Figure 8c, “book” points to \( r_0 \), and “student” points to \( r_4 \). Given that the attention weights between registers and target tokens highlight the structural differences between source and target sentences, we can state again that registers mirror the corresponding source tokens.

L Supplementary Analysis of Representation

We present Figure 10 to supplement the analysis in Section 6.2 on representation distributions, where Figure 4 focuses specifically on the representation state in the 24th layer. Our observations are as follows: (1) In the embedding layer and the 1st layer output (Figures 10a and 10b), source and target token representations are loosely distributed, while registers form three compact clusters based on language. This is because registers lack semantic content and are distinguished only by positional encoding. (2) Starting from the 6th layer (Figure 10c), source tokens begin to become distinguishable by language, and registers start to shift within the representation space toward the source tokens. By the 12th layer (Figure 10d), registers and source tokens are entirely separated in the representation space. (3) By the 18th layer (Figure 10e), target tokens become clearly separated in the representation space, registers’ distribution becomes more diffuse, and the distribution of source tokens becomes more concentrated. These trends culminate in the state observed in the 24th layer (Figure 10f), as described in Section 6.2. These findings suggest two key phenomena: (1) registers progressively reinforce the semantic information they carry as they propagate through the layers; and (2) the representations of target tokens reflect their predicted state only in the higher layers.
<table>
  <tr>
    <th>#params</th>
    <th>Method</th>
    <th colspan="2">High</th>
    <th colspan="2">Med</th>
    <th colspan="2">Low</th>
    <th colspan="2">Extra Low</th>
    <th colspan="2">sup.</th>
    <th colspan="2">zero.</th>
    <th colspan="2">avg.</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
  </tr>
  <tr>
    <td></td>
    <td>vanilla</td>
    <td>23.27</td><td>25.61</td>
    <td>20.43</td><td>24.61</td>
    <td>16.39</td><td>14.25</td>
    <td>17.73</td><td>13.34</td>
    <td>49.27</td><td>18.16</td>
    <td>19.68</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Enc-dec</td>
    <td>+CL</td>
    <td>31.64</td><td>30.87</td>
    <td>28.65</td><td>31.27</td>
    <td>21.38</td><td>21.66</td>
    <td>21.46</td><td>19.33</td>
    <td>49.20</td><td>24.86</td>
    <td>26.04</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>+LCS</td>
    <td>25.34</td><td>31.05</td>
    <td>24.13</td><td>30.90</td>
    <td>24.81</td><td>19.84</td>
    <td>24.77</td><td>17.45</td>
    <td>49.34</td><td>23.70</td>
    <td>24.95</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>+LAVS</td>
    <td>24.99</td><td>26.66</td>
    <td>22.33</td><td>25.77</td>
    <td>17.55</td><td>15.54</td>
    <td>17.77</td><td>14.64</td>
    <td>49.29</td><td>19.38</td>
    <td>20.88</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Dec-only</td>
    <td>+TDO</td>
    <td>27.26</td><td>27.10</td>
    <td>24.42</td><td>27.16</td>
    <td>18.30</td><td>18.01</td>
    <td>18.46</td><td>16.17</td>
    <td>48.44</td><td>21.00</td>
    <td>22.34</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>+Ours</td>
    <td>33.74</td><td>33.55</td>
    <td>31.77</td><td>32.47</td>
    <td>24.87</td><td>24.50</td>
    <td>24.82</td><td>24.68</td>
    <td>48.87</td><td>28.00</td>
    <td>29.02</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Enc-dec</td>
    <td>+CL</td>
    <td>33.02</td><td>32.52</td>
    <td>29.85</td><td>32.84</td>
    <td>20.99</td><td>22.05</td>
    <td>21.84</td><td>18.30</td>
    <td>50.41</td><td>25.51</td>
    <td>26.73</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>+LCS</td>
    <td>24.67</td><td>33.81</td>
    <td>23.88</td><td>32.99</td>
    <td>26.30</td><td>18.10</td>
    <td>25.71</td><td>16.87</td>
    <td>50.16</td><td>24.26</td>
    <td>25.57</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>+LAVS</td>
    <td>29.69</td><td>34.49</td>
    <td>27.61</td><td>33.90</td>
    <td>22.66</td><td>18.62</td>
    <td>18.75</td><td>15.77</td>
    <td>49.89</td><td>25.02</td>
    <td>26.18</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Dec-only</td>
    <td>+TDO</td>
    <td>32.12</td><td>32.22</td>
    <td>28.43</td><td>32.24</td>
    <td>21.64</td><td>20.85</td>
    <td>22.25</td><td>18.32</td>
    <td>50.04</td><td>25.23</td>
    <td>26.44</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>+Ours</td>
    <td>35.24</td><td>35.00</td>
    <td>33.31</td><td>34.56</td>
    <td>26.41</td><td>26.01</td>
    <td>26.22</td><td>25.61</td>
    <td>49.69</td><td>29.35</td>
    <td>30.51</td><td></td><td></td><td></td>
  </tr>
</table>

Table 10: Averaged chrF++ scores of results on EC-40. All notations and abbreviations follow Table 1.

<table>
  <tr>
    <th>#params</th>
    <th>Method</th>
    <th colspan="2">High</th>
    <th colspan="2">Med</th>
    <th colspan="2">Low</th>
    <th colspan="2">Extra Low</th>
    <th colspan="2">sup.</th>
    <th colspan="2">zero.</th>
    <th colspan="2">avg.</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
  </tr>
  <tr>
    <td></td>
    <td>vanilla</td>
    <td>50.21</td><td>49.61</td>
    <td>42.07</td><td>44.93</td>
    <td>28.62</td><td>28.15</td>
    <td>32.94</td><td>31.17</td>
    <td>77.26</td><td>52.29</td>
    <td>53.72</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Enc-dec</td>
    <td>+CL</td>
    <td>55.33</td><td>52.42</td>
    <td>46.33</td><td>47.69</td>
    <td>30.00</td><td>31.42</td>
    <td>37.17</td><td>34.40</td>
    <td>77.33</td><td>56.75</td>
    <td>57.93</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>+LCS</td>
    <td>50.27</td><td>52.22</td>
    <td>43.19</td><td>47.73</td>
    <td>31.88</td><td>30.49</td>
    <td>37.09</td><td>32.50</td>
    <td>77.52</td><td>55.44</td>
    <td>56.71</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>+LAVS</td>
    <td>52.89</td><td>51.57</td>
    <td>44.30</td><td>46.13</td>
    <td>28.81</td><td>28.71</td>
    <td>32.97</td><td>32.16</td>
    <td>77.46</td><td>54.64</td>
    <td>56.08</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Dec-only</td>
    <td>+TDO</td>
    <td>54.39</td><td>52.48</td>
    <td>45.22</td><td>47.40</td>
    <td>30.09</td><td>30.76</td>
    <td>34.53</td><td>33.59</td>
    <td>77.13</td><td>56.16</td>
    <td>57.56</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>+Ours</td>
    <td>56.94</td><td>54.87</td>
    <td>48.74</td><td>49.59</td>
    <td>32.61</td><td>33.83</td>
    <td>37.10</td><td>37.11</td>
    <td>77.12</td><td>60.23</td>
    <td>61.19</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Enc-dec</td>
    <td>+CL</td>
    <td>58.06</td><td>55.20</td>
    <td>48.40</td><td>49.96</td>
    <td>30.79</td><td>32.53</td>
    <td>35.50</td><td>35.06</td>
    <td>78.90</td><td>59.26</td>
    <td>60.38</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>+LCS</td>
    <td>51.52</td><td>55.63</td>
    <td>44.84</td><td>50.74</td>
    <td>33.84</td><td>30.80</td>
    <td>38.91</td><td>33.94</td>
    <td>78.62</td><td>57.61</td>
    <td>58.81</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>+LAVS</td>
    <td>54.18</td><td>56.30</td>
    <td>46.24</td><td>50.73</td>
    <td>33.02</td><td>31.60</td>
    <td>37.64</td><td>32.46</td>
    <td>76.29</td><td>57.90</td>
    <td>58.95</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Dec-only</td>
    <td>+TDO</td>
    <td>57.49</td><td>55.44</td>
    <td>48.34</td><td>50.48</td>
    <td>31.77</td><td>32.57</td>
    <td>36.61</td><td>35.72</td>
    <td>78.48</td><td>59.77</td>
    <td>60.84</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>+Ours</td>
    <td>58.98</td><td>56.92</td>
    <td>50.66</td><td>51.82</td>
    <td>33.91</td><td>35.37</td>
    <td>38.60</td><td>38.04</td>
    <td>78.12</td><td>62.66</td>
    <td>63.54</td><td></td><td></td><td></td>
  </tr>
</table>

Table 11: Averaged COMET scores of results on EC-40. All notations and abbreviations follow Table 1.

<table>
  <tr>
    <th>Model</th>
    <th colspan="2">English</th>
    <th colspan="2">Germanic</th>
    <th colspan="2">Romance</th>
    <th colspan="2">Slavic</th>
    <th colspan="2">Mal.-Polyn.</th>
    <th colspan="2">Asian</th>
    <th colspan="2">avg.</th>
  </tr>
  <tr>
    <td></td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
  </tr>
  <tr>
    <td>M2M</td>
    <td>483M</td><td>50.43</td><td>54.36</td>
    <td>44.84</td><td>46.24</td>
    <td>43.96</td><td>48.26</td>
    <td>43.36</td><td>43.06</td>
    <td>37.54</td><td>41.77</td>
    <td>39.83</td><td>27.85</td>
    <td>42.53</td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>615M</td><td>49.97</td><td>54.74</td>
    <td>46.77</td><td>48.62</td>
    <td>45.34</td><td>49.83</td>
    <td>44.70</td><td>44.42</td>
    <td>40.17</td><td>43.59</td>
    <td>41.04</td><td>28.44</td>
    <td>44.12</td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>1.2B</td><td>54.80</td><td>54.44</td>
    <td>49.02</td><td>47.06</td>
    <td>47.49</td><td>48.04</td>
    <td>45.87</td><td>43.35</td>
    <td>32.78</td><td>40.68</td>
    <td>30.90</td><td>28.01</td>
    <td>42.56</td><td></td>
  </tr>
  <tr>
    <td>NLLB</td>
    <td>615M</td><td>55.54</td><td>61.73</td>
    <td>48.48</td><td>50.39</td>
    <td>47.38</td><td>51.54</td>
    <td>46.38</td><td>45.34</td>
    <td>45.98</td><td>50.96</td>
    <td>42.56</td><td>29.73</td>
    <td>46.70</td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>1.3B</td><td>56.82</td><td>63.35</td>
    <td>50.27</td><td>52.18</td>
    <td>48.80</td><td>53.19</td>
    <td>47.98</td><td>47.46</td>
    <td>47.92</td><td>52.49</td>
    <td>44.70</td><td>30.98</td>
    <td>48.40</td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>3.3B</td><td>57.88</td><td>64.27</td>
    <td>50.75</td><td>51.36</td>
    <td>49.63</td><td>53.92</td>
    <td>48.91</td><td>48.76</td>
    <td>49.06</td><td>53.28</td>
    <td>45.71</td><td>31.97</td>
    <td>49.35</td><td></td>
  </tr>
  <tr>
    <td>GPT</td>
    <td>3.5 turbo</td><td>53.30</td><td>61.60</td>
    <td>49.39</td><td>52.16</td>
    <td>48.31</td><td>53.34</td>
    <td>47.54</td><td>46.35</td>
    <td>45.64</td><td>48.05</td>
    <td>43.46</td><td>31.20</td>
    <td>47.41</td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>40 mini</td><td>58.03</td><td>62.85</td>
    <td>51.26</td><td>52.90</td>
    <td>49.33</td><td>54.43</td>
    <td>49.12</td><td>48.62</td>
    <td>49.39</td><td>52.25</td>
    <td>45.86</td><td>34.11</td>
    <td>49.48</td><td></td>
  </tr>
  <tr>
    <td>MITRE</td>
    <td>466M</td><td>58.11</td><td>62.77</td>
    <td>51.40</td><td>53.41</td>
    <td>50.06</td><td>54.46</td>
    <td>49.18</td><td>48.62</td>
    <td>47.83</td><td>52.04</td>
    <td>45.10</td><td>32.41</td>
    <td>49.29</td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>913M</td><td>58.84</td><td>64.01</td>
    <td>52.40</td><td>54.59</td>
    <td>51.03</td><td>55.36</td>
    <td>50.32</td><td>49.84</td>
    <td>48.97</td><td>52.88</td>
    <td>46.65</td><td>33.88</td>
    <td>50.42</td><td></td>
  </tr>
</table>

Table 12: Averaged chrF++ scores of results for comparing MITRE and baselines. All notations and abbreviations follow Table 3.

<table>
  <tr>
    <th>Model</th>
    <th colspan="2">English</th>
    <th colspan="2">Germanic</th>
    <th colspan="2">Romance</th>
    <th colspan="2">Slavic</th>
    <th colspan="2">Mal.-Polyn.</th>
    <th colspan="2">Asian</th>
    <th colspan="2">avg.</th>
  </tr>
  <tr>
    <td></td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
    <td>→</td><td>←</td>
  </tr>
  <tr>
    <td>M2M</td>
    <td>483M</td><td>81.63</td><td>81.40</td>
    <td>78.90</td><td>77.03</td>
    <td>80.48</td><td>78.49</td>
    <td>79.85</td><td>80.31</td>
    <td>68.34</td><td>72.75</td>
    <td>78.67</td><td>78.57</td>
    <td>77.74</td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>615M</td><td>81.16</td><td>82.15</td>
    <td>81.42</td><td>79.98</td>
    <td>82.00</td><td>80.50</td>
    <td>81.29</td><td>82.43</td>
    <td>72.56</td><td>74.62</td>
    <td>80.02</td><td>79.96</td>
    <td>79.79</td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>1.2B</td><td>85.93</td><td>85.17</td>
    <td>84.15</td><td>82.87</td>
    <td>84.46</td><td>83.12</td>
    <td>83.87</td><td>85.41</td>
    <td>75.91</td><td>77.83</td>
    <td>82.95</td><td>82.58</td>
    <td>82.66</td><td></td>
  </tr>
  <tr>
    <td>NLLB</td>
    <td>615M</td><td>86.61</td><td>86.76</td>
    <td>84.06</td><td>82.77</td>
    <td>84.50</td><td>83.05</td>
    <td>83.41</td><td>84.39</td>
    <td>81.26</td><td>83.51</td>
    <td>82.12</td><td>82.02</td>
    <td>83.33</td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>1.3B</td><td>87.76</td><td>87.63</td>
    <td>85.72</td><td>84.76</td>
    <td>85.93</td><td>84.76</td>
    <td>85.07</td><td>86.82</td>
    <td>83.57</td><td>84.93</td>
    <td>84.36</td><td>83.52</td>
    <td>85.13</td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>3.3B</td><td>88.22</td><td>88.09</td>
    <td>85.49</td><td>85.86</td>
    <td>85.45</td><td>85.84</td>
    <td>87.96</td><td>84.63</td>
    <td>85.45</td><td>85.42</td>
    <td>84.56</td><td>85.96</td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>GPT</td>
    <td>3.5 turbo</td><td>87.67</td><td>88.02</td>
    <td>86.26</td><td>85.80</td>
    <td>86.62</td><td>85.96</td>
    <td>85.91</td><td>87.50</td>
    <td>83.09</td><td>82.09</td>
    <td>85.45</td><td>85.77</td>
    <td>85.66</td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>40 mini</td><td>89.59</td><td>88.64</td>
    <td>87.50</td><td>86.38</td>
    <td>87.56</td><td>86.57</td>
    <td>87.08</td><td>88.90</td>
    <td>85.89</td><td>86.03</td>
    <td>86.99</td><td>87.47</td>
    <td>87.16</td><td></td>
  </tr>
  <tr>
    <td>MITRE</td>
    <td>466M</td><td>87.87</td><td>87.29</td>
    <td>85.99</td><td>84.98</td>
    <td>86.49</td><td>85.14</td>
    <td>85.88</td><td>87.19</td>
    <td>82.24</td><td>83.41</td>
    <td>84.38</td><td>84.29</td>
    <td>85.19</td><td></td>
  </tr>
  <tr>
    <td></td>
    <td>913M</td><td>88.11</td><td>87.81</td>
    <td>86.54</td><td>85.61</td>
    <td>86.96</td><td>85.70</td>
    <td>86.16</td><td>88.03</td>
    <td>83.15</td><td>83.80</td>
    <td>85.52</td><td>85.35</td>
    <td>85.88</td><td></td>
  </tr>
</table>

Table 13: Averaged COMET scores of results for comparing MITRE and baselines. All notations and abbreviations follow Table 3. Given the different trend compared to Tables 3 and 12, we not only mention it in Section 5.3, but also provide an additional discussion in Appendix I.
Figure 9: Data distribution of our pre-training dataset. Figure 9a shows data size distribution at the family level, while Figure 9b displays data size at the language level. In 9a, non-zero values are scaled by log10 and adjusted by subtracting 7 for clearer visualization. In 9b, non-zero values are also scaled by log10 and shifted by subtracting the minimum value to enhance illustration clarity.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3">Germanic</th>
    <th colspan="3">Romance</th>
    <th colspan="3">Slavic</th>
    <th colspan="3">Indo-Aryan</th>
    <th colspan="3">Afro-Asiatic</th>
  </tr>
  <tr>
    <th>code</th>
    <th>Language</th>
    <th>Script</th>
    <th>code</th>
    <th>Language</th>
    <th>Script</th>
    <th>code</th>
    <th>Language</th>
    <th>Script</th>
    <th>code</th>
    <th>Language</th>
    <th>Script</th>
    <th>code</th>
    <th>Language</th>
    <th>Script</th>
  </tr>
  <tr>
    <td>High<br>(5 million)</td>
    <td>de</td>
    <td>German</td>
    <td>Latin</td>
    <td>fr</td>
    <td>French</td>
    <td>Latin</td>
    <td>ru</td>
    <td>Russian</td>
    <td>Cyrillic</td>
    <td>hi</td>
    <td>Hindi</td>
    <td>Devanagari</td>
    <td>ar</td>
    <td>Arabic</td>
    <td>Arabic</td>
  </tr>
  <tr>
    <td></td>
    <td>nl</td>
    <td>Dutch</td>
    <td>Latin</td>
    <td>es</td>
    <td>Spanish</td>
    <td>Latin</td>
    <td>cs</td>
    <td>Czech</td>
    <td>Latin</td>
    <td>bn</td>
    <td>Bengali</td>
    <td>Bengali</td>
    <td>he</td>
    <td>Hebrew</td>
    <td>Hebrew</td>
  </tr>
  <tr>
    <td>Med<br>(1 million)</td>
    <td>sv</td>
    <td>Swedish</td>
    <td>Latin</td>
    <td>it</td>
    <td>Italian</td>
    <td>Latin</td>
    <td>pl</td>
    <td>Polish</td>
    <td>Latin</td>
    <td>kn</td>
    <td>Kannada</td>
    <td>Devanagari</td>
    <td>mt</td>
    <td>Maltese</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td></td>
    <td>da</td>
    <td>Danish</td>
    <td>Latin</td>
    <td>pt</td>
    <td>Portuguese</td>
    <td>Latin</td>
    <td>bg</td>
    <td>Bulgarian</td>
    <td>Cyrillic</td>
    <td>mr</td>
    <td>Marathi</td>
    <td>Devanagari</td>
    <td>ha</td>
    <td>Hausa*</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td>Low<br>(100 thousand)</td>
    <td>af</td>
    <td>Afrikaans</td>
    <td>Latin</td>
    <td>ro</td>
    <td>Romanian</td>
    <td>Latin</td>
    <td>uk</td>
    <td>Ukrainian</td>
    <td>Cyrillic</td>
    <td>sd</td>
    <td>Sindhi</td>
    <td>Arabic</td>
    <td>ti</td>
    <td>Tigrinya</td>
    <td>Ethiopic</td>
  </tr>
  <tr>
    <td></td>
    <td>lb</td>
    <td>Luxembourgish</td>
    <td>Latin</td>
    <td>oc</td>
    <td>Occitan</td>
    <td>Latin</td>
    <td>sr</td>
    <td>Serbian</td>
    <td>Latin</td>
    <td>gu</td>
    <td>Gujarati</td>
    <td>Devanagari</td>
    <td>am</td>
    <td>Amharic</td>
    <td>Ethiopic</td>
  </tr>
  <tr>
    <td>Extra Low<br>(50 thousand)</td>
    <td>no</td>
    <td>Norwegian</td>
    <td>Latin</td>
    <td>ast</td>
    <td>Asturian</td>
    <td>Latin</td>
    <td>be</td>
    <td>Belarusian</td>
    <td>Cyrillic</td>
    <td>ne</td>
    <td>Nepali</td>
    <td>Devanagari</td>
    <td>kab</td>
    <td>Kabyle*</td>
    <td>Latin</td>
  </tr>
  <tr>
    <td></td>
    <td>ic</td>
    <td>Icelandic</td>
    <td>Latin</td>
    <td>ca</td>
    <td>Catalan</td>
    <td>Latin</td>
    <td>bs</td>
    <td>Bosnian</td>
    <td>Latin</td>
    <td>ur</td>
    <td>Urdu</td>
    <td>Arabic</td>
    <td>so</td>
    <td>Somali</td>
    <td>Latin</td>
  </tr>
</table>

Table 14: Details of non-English languages in EC-40. This table is duplicated from Tan and Monz (2023). Numbers in the table represent the number of sentences paired to the English. Two exceptions are Hausa and Kabyle, where their data sizes are 334,000 and 18,448, respectively.
Figure 10: 2D distributions of token-level representations extracted from the different layers of a model trained on EC-40. This illustration complements Figure 4.

Reviewer paper summaries: In this paper, the authors introduce a "registering" method that allows small multilingual neural machine translation (MNMT) models (<1B) to compete with large language models (LLMs) like GPT-4o-mini. This method inserts artificial tokens indicating the target language into the input sequence between source and target tokens. Combined with a modified attention mask, it effectively mitigates the off-target problem.  
Experimental results show that this approach advances the state of the art in MNMT. Additionally, the authors pre-trained two models using their proposed methods on 9.3 billion sentence pairs across 24 languages from a public dataset. Both models demonstrated strong performance.

This paper introduces a method for small MNMT-specific models by modifying the attention mask.
Summary of strengths: 1. The author's presentation and writing are good. The motivation, method, experiments, and analysis are well presented.
2. The experimental results show that the author's method outperforms the baseline significantly across multiple evaluation metrics. Given the model's size, flexibility, and low cost, this work could be valuable for practical applications.
3. This paper's experiments are solid. They compare existing methods and commercial models, analyze pretraining and fine-tuning, and use reasonable and comprehensive evaluation metrics.

The idea is simple to understand. The paper is easy to follow. The description of the method is easy to follow.
Summary of weaknesses: 1. I'm not entirely sure what the author means by emphasizing that their trained model reaches a level comparable to GPT-4o-mini. I understand they want to highlight the practical value of their work, but GPT-4o-mini hasn't been specifically optimized for translation (e.g., fine-tuned with the same translation data).
2. If a direct metric for inference cost were provided, the experiment would be more complete (though not necessary). The author shows model size, but since their method requires extra tokens and modifies attention, inference costs may increase. Showing inference costs directly, such as FLOPs or inference time, would make this clearer.

1. Authors only evaluate this method on EC-40 in Table 1, while it is better to evaluate the method on more datasets and language-pairs instead of EC-40 only.

2. Authors put the results of the models in Table 1 while splitting the tables into two parts which is confusing. It is better to put the results of the same method together.

3. Some tables and figures are important and should be moved to Section 4. For example, Table 8-11 should be put in Section 4 after Table 1 and Table 3.
Comments, suggestions and typos: 1. In line 50, the word "however" also has a hyperlink to a reference source.

N/A
Overall assessment scores: 4

4
Metareview score: 4
Metareview: This paper introduces “registering,” a mechanism of inserting artificial target-language tokens (“registers”) in the input sequence for decoder-only multilingual NMT. Through targeted attention masking, each register encodes a corresponding source token in the target language space, aiming to mitigate off-target translations. The submission reports strong empirical results on a large-scale EC-40 benchmark, including zero-shot directions. The authors also release two pre-trained models (MITRE)—demonstrating that a ~913M model can outperform the 3.3B NLLB model and, in some cases, be competitive with certain commercial LLMs.
Summary of reasons to publish: 1.  The proposed "registering" technique offers a conceptually simple yet effective approach to improve MNMT, particularly in decoder-only models, by explicitly bridging the source and target language spaces. It demonstrably reduces off-target translations, a known challenge in MNMT.
2.  The paper presents compelling results on the large-scale EC-40 benchmark. The MITRE models achieve state-of-the-art performance compared to previous MNMT-specific models and show competitive results against significantly larger models (NLLB-3.3B, GPT-4o-mini), highlighting the efficiency and effectiveness of the approach. The gains are consistent across resource levels and language families.
3.  The authors provide solid analysis, including ablations on the register mechanism, comparisons against strong baselines (including commercial LLMs), evaluations across multiple metrics (BLEU, chrF++, COMET, off-target ratio), and assessments of both pre-training and fine-tuning adaptability. The rebuttal further added valuable analyses on computational cost, register length justification, and sentence-level representations.
4.  The release of the pre-trained MITRE models (MITRE-466M and MITRE-913M) is a significant contribution to the community, providing strong, efficient, and adaptable MNMT models that can facilitate further research and practical applications, particularly where deployment cost is a concern.
5.  The paper is well-written, clearly motivated by the need for efficient yet powerful MNMT alternatives to large LLMs, and the proposed method is well-explained.