THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation
Authors: Yunlong Liang, Fandong Meng, Jie Zhou
Accepted at: acl25

THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation

Yunlong Liang, Fandong Meng*, Jie Zhou
Pattern Recognition Center, WeChat AI, Tencent Inc
{yunlonliang,fandongmeng,withtomzhou}@tencent.com

Abstract

The sparse Mixture-of-Experts (MoE) has achieved significant progress for neural machine translation (NMT). However, there exist two limitations in current MoE solutions which may lead to sub-optimal performance: 1) they directly use the task knowledge of NMT into MoE (\emph{e.g.}, domain/linguistics-specific knowledge), which are generally unavailable at practical application and neglect the naturally grouped domain/linguistic properties; 2) the expert selection only depends on the localized token representation without considering the context, which fully grasps the state of each token in a global view. To address the above limitations, we propose THOR-MoE via arming the MoE with hierarchical task-guided and context-responsive routing policies. Specifically, it 1) firstly predicts the domain/language label and then extracts mixed domain/language representation to allocate task-level experts in a hierarchical manner; 2) injects the context information to enhance the token routing from the pre-selected task-level experts set, which can help each token to be accurately routed to more specialized and suitable experts. Extensive experiments on multi-domain translation and multilingual translation benchmarks with different architectures consistently demonstrate the superior performance of THOR-MoE. Additionally, the THOR-MoE operates as a plug-and-play module compatible with existing Top-\(k\) (Shazeer et al., 2017a) and Top-\(p\) (Huang et al., 2024) routing schemes, ensuring broad applicability across diverse MoE architectures. For instance, compared with vanilla Top-\(p\) (Huang et al., 2024) routing, the context-aware manner can achieve an average improvement of 0.75 BLEU with less than 22% activated parameters on multi-domain translation tasks.

1 Introduction

The rapid advancement of neural machine translation (NMT; Zhang et al. (2020); Fan et al. (2020)) has been significantly propelled by sparse Mixture-of-Experts (MoE) architectures (Shazeer et al., 2017a; Lepikhin et al., 2021a; Fedus et al., 2022), which increase model capacity while maintaining computational efficiency through conditional computation. In the literature, existing work can be roughly classified into two categories: (1) some studies mainly focus on how to effectively incorporate task information (\emph{e.g.}, domain/linguistics-specific knowledge) into MoE models designing task-specific modules (Pham et al., 2023; Jiang et al., 2024); (2) another line of work concentrates on how to improve the training and inference efficiency via reducing the activated experts in MoE (Jawahar et al., 2023a; Elbayad et al., 2023; Wang et al., 2024; Zeng et al., 2024).

Specifically, in (1), Kudugunta et al. (2021) routes input examples to different experts based on translation language representations. Li et al. (2023c) explores multiple language-group-specific routers to incorporate language group knowledge into models. Zhao et al. (2024b) takes one step further by designing hierarchical language-guided token routing. Although intuitive and effective, they heavily rely on explicit task-specific knowledge (\emph{e.g.}, domain or language labels), which is often unavailable in real-world scenarios. This reliance not only restricts generalization but also overlooks the intrinsic hierarchical grouping of domain and linguistic properties inherent in multilingual or multi-domain translation tasks. In (2), Zhao et al. (2024a) distills the knowledge of unselected experts to the selected one and thus reduces the number of experts without impacting performance; Li et al. (2023a) and Huang et al. (2024) decrease the number of experts by designing a threshold based on expert probability distribution; Zhao et al.
(2024b) adapts the number of experts by the difficulty of language. Generally, the global context grasp the overall situation and might know whether each token is difficult or not (Gloeckle et al., 2024). However, previous routing strategies select experts solely based on localized token-level representations, neglecting the broader contextual dependencies that govern token interactions. This myopic view limits the model’s ability to allocate experts optimally in a globally coherent manner.

In this work, to address the above issues, we propose a innovative hierarchical context-responsive routing (THOR-MoE) framework for neural machine translation. Specifically, the THOR-MoE first predicts which domain/language the input belongs to and then extracts the mixed task\(^1\) representation to allocates experts in a hierarchical manner. It is not only flexible in practice but also effectively infuses the domain knowledge into the task-level routing via encouraging experts to specialize in certain domains/languages. Furthermore, before each token routing, the THOR-MoE injects the context information to help accurately assign the experts from the pre-selected task-level experts set for each token in a global perspective. This enables the model to adaptively capture evolving contextual dependencies, such as long-range syntactic structures or discourse coherence. Crucially, the THOR-MoE operates as a plug-and-play module compatible with existing Top-\(k\) (Shazeer et al., 2017a) and Top-\(p\) (Huang et al., 2024) routing schemes, ensuring broad applicability across diverse MoE architectures.

We validate our proposed THOR-MoE framework on the commonly-used multi-domain NMT benchmark (Aharoni and Goldberg, 2020) and multilingual NMT benchmark (Zhang et al., 2020), which contains 5 domains and 16 languages, respectively. Extensive experiments show that the hierarchical context-responsive routing is pivotal for unlocking the full potential of MoE systems.

In summary, our main contributions are:

• We propose two components to expand MoE: (i) a hierarchical design to enable language- and domain-specific expert routing; (ii) enabling the use of context information during expert selection, which is important for translation.
• Extensive experiments on both multi-domain and multilingual translation consistently show the effectiveness and generalization of THOR-MoE. For example, our model achieves consistent improvements of an average improvement of 0.75 BLEU with less than 22% activated parameters over vanilla Top-p routing in multi-domain translation tasks.

2 Background

2.1 Neural Machine Translation

Given an input sentence in the source language \( X = \{ x_i \}_{i=1}^{|X|} \), the goal of the model is to produce its translation in the target language \( Y = \{ y_i \}_{i=1}^{|Y|} \). The conditional distribution of the model is:

\[
\mathcal{L}_{\text{NMT}} = - \sum_{t=1}^{|Y|} \log(p(y_t | X, y_{1:t-1})),
\]

where \( y_{1:t-1} \) is the partial translation.

2.2 Sparse Mixture-of-Experts

The vanilla MoE model can be seen the variant of Transformer model via replacing the feed-forward (FFN) sub-block of the transformer block with MoE layer, in which per token selects fixed number of experts (Lepikhin et al., 2021b). In each MoE layer, there are \( N \) experts, denoted as \( \mathcal{E} = \{ e_1, e_2, \ldots, e_N \} \). An input \( \mathbf{x} \) is dispatched to these experts, and the output of the MoE layer is computed as the weighted average of the outputs from the experts:

\[
\text{MoE}(\mathbf{x}) = \sum_{i=1}^N g_i(\mathbf{x}) * e_i(\mathbf{x}),
\]

where \( g_i(\mathbf{x}) \) is computed by a router that predicts the contribution of each expert to the final output. Given the computing efficiency, the MoE assigns each token to limited experts (\( e.g. \), 1 or 2).

To derive \( g_i(\mathbf{x}) \), we generally compute the probability \( \mathbf{P} \) of each expert being selected for the input \( \mathbf{x} \) as follows:

\[
\mathbf{P} = \text{Softmax}(\mathbf{W}_r \cdot \mathbf{x}^T),
\]

where \( \mathbf{W}_r \in N \times d \) serves as a learnable parameter and \( d \) denotes the dimension of the input \( \mathbf{x} \). The vector \( \mathbf{P} \), with a size of \( N \), encapsulates the probabilities associated with the selection of each expert. And \( P_i \) indicates the likelihood of choosing the \( i^{th} \) expert \( e_i \) to calculate the input \( \mathbf{x} \).

\footnotetext{1Note that the task generally denotes the domain/language-specific knowledge in this work.}
Figure 1: The overview of the proposed THOR-MoE. 1) vanilla token routing; 2) hierarchical task-guided routing; 3) context-responsive token routing. The ‘Emb.’ and ‘Rep.’ denotes the embedding and representation, respectively. The hierarchical manner denotes that the task-guided routing firstly selects different task-specific experts sets for different queries (e.g., \( \mathcal{E}_t \) for query 1). Then the context-responsive token routing assigns experts from \( S^t \) for each token in query 1.

2.2.1 Top-k Routing
Top-\( k \) routing (Shazeer et al., 2017b; Lepikhin et al., 2021b; Zuo et al., 2022) entails the selection of the top \( k \) experts, which correspond to the \( k \) highest probabilities within the vector \( \mathbf{P} \). Then, the probabilities of the chosen experts are re-scaled through normalization, while the probabilities of the non-selected experts are set to zero, signifying their deactivation. The resultant computation for \( g_i(\mathbf{x}) \) proceeds as below:

\[
g_i(\mathbf{x}) = \begin{cases}
\frac{P_i}{\sum_{j \in \text{TopK}(\mathbf{P})} P_j}, & i \in \text{TopK}(\mathbf{P}) \\
0, & i \notin \text{TopK}(\mathbf{P})
\end{cases}
\]

where TopK(\( \mathbf{P} \)) returns the indices of the k largest elements in \( \mathbf{P} \).

2.2.2 Top-p Routing
Top-\( p \) routing is initially proposed by (Li et al., 2023a; Huang et al., 2024), which aims to overcome the drawback of Top-\( k \) activating the fixed number of experts overlooking the variability in difficulty across different inputs. We refer to readers to Huang et al. (2024) for more details.

Formally, Top-\( p \) first sorts the elements in \( \mathbf{P} \) from highest to lowest, returning in a sorted index list \( I \). Subsequently, we identify the minimal subset of experts for which the aggregate probability surpasses the threshold \( p \). The count of selected experts, denoted as \( t \), is then determined by:

\[
t = \arg\min_{k \in \{1,...,N\}} \sum_{j < k} P_{i,j} \geq p,
\]

where \( p \) is the threshold that controls how confident the model should be when stopping adding more experts. \( p \) is a hyper-parameter whose range is from 0 to 1. The higher the \( p \) is, the more experts will be activated.

In dynamic routing mechanism, the calculation of \( g_i(\mathbf{x}) \) is:

\[
g_i(\mathbf{x}) = \begin{cases}
P_i, & e_i \in S \\
0, & e_i \notin S
\end{cases}
\]

where \( S \) is the set of selected experts controlled by \( t \) in Equation 5:

\[
S = \{e_{I_1}, e_{I_2},...e_{I_t}\}.
\]

3 THOR-MoE

In this section, we introduce the proposed hierarchical context-responsive routing framework as shown in Figure 1, including two components: *hierarchical task-guided routing* (\S 3.1) and *context-responsive token routing* (\S 3.2). Finally, we describe the training objectives (\S 3.3) in detail.
3.1 Hierarchical Task-Guided Routing
Our hierarchical task-guided routing consists of three stages, as shown in 2) of 1. The first stage is task prediction, which aims to automatically obtain the task knowledge by predicting which domain/language the input belongs to. The second stage is to inject the mixed task representation to allocates experts in a hierarchical task-guided manner (third stage).

Task Predictor. During real testing for a query, it is hard to reach its domain or language category. Therefore, to automatically obtain the domain knowledge or language knowledge, we add a special token [CLS] in front of the input. Then, it represents the global sentence information and thus can effectively reflect which domain/language the input belongs to (Devlin et al., 2019). Finally, we transform the representation \( \mathbf{H}^{cls} \) into a fixed-size vector through Maxpooling and apply a fully-connected layer to predict suitable labels:

\[
\mathbf{H}^{max} = \text{Maxpooling}(\mathbf{H}^{cls}), \quad \mathbf{H}^{max} \in \mathbb{R}^d,
\]
\[
\mathcal{P}^t = \text{Softmax}(\mathbf{W}^p \mathbf{H}^{max}),
\]

where \( \mathbf{W}^p \in \mathbb{R}^{K \times d} \) is trainable weight and \( K \) denotes the number of domains or languages and \( d \) is the model dimension.

Mixed Task Representation. Theoretically, we can directly take the predicted task label to extract corresponding task representation. However, in some cases in practice, it is hard to judge since some sentences belongs to multiple domains. In language cases, there are some code-mixed sentences, which is also confused to decide. Therefore, directly use the given label in existing work or the predicted label cannot accurately help model select the specialized experts for such cases and thus limits their potential, as shown in § 6.2. To address this issue, we use the mixed task representation as follows:

\[
\mathbf{E}_p = \sum (\mathcal{P}^t \cdot \text{EMB1}), \quad \mathbf{E}_p \in \mathbb{R}^d,
\]

where \( \mathbf{E}_p \) denotes the mixed task representation generated by the weighted sum of the task distribution \( \mathcal{P} \) (predicted by the task predictor as Eq. 8) and the task parameter matrix EMB1. The dimension of EMB1 is \( num \) (number of tasks/language groups) * \( d \) (embedding dimension), which is randomly initialized and then optimized with the training objective.

Hierarchical Task-Guided Routing. With the mixed task representation, we design a task router \( g^t \) at the task level. In each MoE layer, \( g^t \) takes the mixed task representation \( \mathbf{E}_p \) as input and outputs a task-dependent expert vector. Then, based on it, we use TopK function to select task-specific candidate experts as \( \mathcal{S}^t \) from all experts. Finally, based on the selected task-specific candidate experts \( S_t \), we can conduct general token routing at the token level with Top-k or Top-p routing as described in § 2.2.1 and § 2.2.2, respectively. The final routing policy can be written as \( \sum_i^{|S^t|} \sum_j^{|E|} g_i^t \cdot g_j \cdot e_{i,j}(\mathbf{x}) \).

However, current expert selection at the token level only depends on the current token representation without considering the complete word meaning and global context. Generally, the global context grasp the overall situation and might know which token is difficult or not. Thus, the context can schedules the token to route to suitable experts in a global view. To this end, we propose context-responsive experts selection at the token level.

3.2 Context-Responsive Routing
As shown in 3) of 1, before the experts selection at the token level, we explicitly inject the context into the token representation. Since it is hard to determine for a code-mixed input or a query that can belongs to several domains, which thus limits its application in practice. Therefore, we use the averaged context representation as the context representation. \( \mathbf{H}_{ctx} = \frac{1}{|\mathcal{Y}_{t:t-1}|} \sum_{l=t-1}^{t-1} \mathbf{h}_l^L \) where \( \mathbf{h} \) is the hidden state of the \( t \)-th token at the \( L \) layer. At each decoding step, the context used for each input token is the prefix of the generated next token. That is, the context is dynamically updated with decoding.

To effectively incorporate the context representation into the routing process, we design a gate to dynamically control the contribution of these information:

\[
\mathbf{x}_i = g \odot \mathbf{x}_i + (1 - g) \odot \mathbf{H}_{ctx},
\]
\[
g = \sigma([\mathbf{x}_i; \mathbf{H}_{ctx}] \mathbf{W}^g + \mathbf{b}^g),
\]

where \( \mathbf{W}^g \) and \( \mathbf{b}^g \) are the trainable parameters.

In this manner, we can inject the context information into each token to accurately assign specialized experts to suitable tokens and unlock the potential of the MoE.

3.3 Training Objectives
Our training objectives consist of four parts in Top-k routing (translation loss \( \mathcal{L}_{NMT} \), task prediction loss \( \mathcal{L}_{tp} \), load balance loss at the task level \( \mathcal{L}_{bd} \
and token level \( \mathcal{L}_{bt} \)) and five parts in Top-\( p \) routing (\( \mathcal{L}_{NMT}, \mathcal{L}_{tp}, \mathcal{L}_{bd}, \mathcal{L}_{bt} \) and dynamic routing loss \( \mathcal{L}_d \) in Top-\( p \)).

Task Prediction Loss. The task prediction loss is defined as:

\[
\mathcal{L}_{tp} = \min(-\log(\mathcal{P}^t[t_g])),
\]

where \( t_g \) is the golden class label. \( \mathcal{P}^t \) is calculated as in Eq. 8.

Load Balance Loss at the Task Level. To fully encourage the experts to specialize in certain tasks, we propose a task-guided routing loss. It aims to achieve the number of tasks processed by different experts to be roughly the same. Therefore, the task-guided routing loss is defined as:

\[
\mathcal{L}_{bd} = N * \sum_{i=1}^{N} F_i^t * Q_i^t,
\]

where \( F_i^t \) represents the proportion of tasks selecting expert \( e_i \), and \( Q_i^t \) denotes the proportion of the router’s probability allocated to expert \( e_i \). For \( K \) tasks, \( F_i^t \) and \( Q_i^t \) are defined as: \( F_i^t = \frac{1}{K} \sum_{j=1}^{K} 1\{e_i \in S^{t,j}\} \) and \( Q_i^t = \frac{1}{K} \sum_{j=1}^{K} P_j^i \) where \( S^{t,j} \) is the set of activated experts for task \( j \), which is calculated by Equation 7, and \( P_j^i \) is the probability of selecting each experts for task \( j \), calculated by Equation 3.

Load Balance Loss at the Token Level. This loss is the similar to the vanilla load balance loss with a slight difference that the experts candidates is pre-selected by the hierarchical task-guided routing. Formally, the load balance loss at the token level is defined as:

\[
\mathcal{L}_{bt} = |S^t| * \sum_{k=1}^{|S^t|} F_k^b * Q_k^b,
\]

where \( F_k^b \) represents the proportion of tokens selecting expert \( e_k \), and \( Q_k^b \) denotes the proportion of the router’s probability allocated to expert \( e_k \). For a sequence comprising \( M \) tokens, \( F_k^b \) and \( Q_k^b \) are defined as: \( F_k^b = \frac{1}{M} \sum_{\ell=1}^{M} 1\{e_k \in S^{\ell,t}\} \) and \( Q_k = \frac{1}{M} \sum_{\ell=1}^{M} P_\ell^k \) where \( S^{\ell,t} \) is the set of activated experts for token \( \ell \) from pre-selected experts set \( S^{d,\ell} \), which is calculated by Equation 7, and \( P_\ell^k \) is the probability of selecting each experts for token \( \ell \), calculated by Equation 3.

Dynamic Routing Loss in Top-\( p \). The dynamic routing loss aims to prevent dynamic routing from using too many parameters to cheat and losing its ability to selectively choose experts. Therefore, Huang et al. (2024) introduce a constraint on \( \mathbf{P} \) and minimize the entropy of the distribution \( \mathbf{P} \), ensuring that every token can focus on as less specific experts as possible, which is formalized as:

\[
\mathcal{L}_{topp} = - \sum_{i=1}^{N} P_i * \log(P_i).
\]

Final Loss. Our approach can be flexibly applied in Top-\( k \) and Top-\( p \) routing strategies. Therefore, we have two final loss functions for Top-\( k \) and Top-\( p \), respectively. It is a combination of the translation loss, task prediction loss, load balance loss at the task level and at the token level, and dynamic loss:

\[
\mathcal{J}_{topk} = \mathcal{L}_{NMT} + \alpha \mathcal{L}_{dp} + \beta \mathcal{L}_{bd} + \gamma \mathcal{L}_{bp},
\]
\[
\mathcal{J}_{topp} = \mathcal{J}_{topk} + \delta \mathcal{L}_{topp},
\]

where \( \alpha, \beta, \gamma, \) and \( \delta \) are hyper-parameters to adjust the contribution among these loss functions, respectively.

4 Experiments

4.1 Datasets and Metric

Datasets. We use the multi-domain translation dataset proposed by Koehn and Knowles (2017). The dataset mainly covers five diverse domains: IT, Koran, Law, Medical, and Subtitles, which are available in OPUS (Aulamo and Tiedemann, 2019). Following previous work (Gu et al., 2022; Liang et al., 2024), we use the new data splitting released by Aharoni and Goldberg (2020), and perform German to English translation (De→En). We use the OPUS-16 for multilingual translation following (Zhao et al., 2024b). The OPUS-16 comes from OPUS-100 (Zhang et al., 2020), which includes 16 languages (8 high resource (> 0.9M), 4 medium resource, and 4 low resource (< 0.1M)). Please refer to Tab. 8 of § 8 for detailed data statistics.

Metric. For a fair comparison, we follow previous work (Gu et al., 2022; Zhao et al., 2024b) and adopt the 4-gram case-sensitive BLEU with the SacreBLEU tool2 (Post, 2018) and report the statistical significance test (Koehn, 2004). For

2BLEU-case.mixed+numrefs.1+smooth.exp-tok.13a+version.1.4.13
<table>
  <tr>
    <th>Models</th>
    <th>IT</th>
    <th>Koran</th>
    <th>Medical</th>
    <th>Law</th>
    <th>Subtitles</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th>Dense</th>
    <td>SFT-3B</td>
    <td>40.65</td>
    <td>20.4</td>
    <td>51.4</td>
    <td>54.8</td>
    <td>28.33</td>
    <td>39.12</td>
  </tr>
  <tr>
    <td rowspan="3">MoE</td>
    <td>Trim-MoE (Top-1)</td>
    <td>40.03</td>
    <td>19.55</td>
    <td>51.80</td>
    <td>55.83</td>
    <td>25.50</td>
    <td>38.54</td>
  </tr>
  <tr>
    <td>Trim-MoE (Top-2)</td>
    <td><b>45.10</b></td>
    <td>22.68</td>
    <td>51.84</td>
    <td>57.12</td>
    <td><b>29.02</b></td>
    <td>41.15</td>
  </tr>
  <tr>
    <td>Trim-MoE (Top-\( p \), \( p=0.5 \))</td>
    <td>39.39</td>
    <td>19.21</td>
    <td>55.67</td>
    <td>60.18</td>
    <td><b>29.21</b></td>
    <td>40.73</td>
  </tr>
  <tr>
    <td rowspan="3">THOR-MoE</td>
    <td>THOR-MoE (Top-1)</td>
    <td>43.96<sup>†</sup></td>
    <td>21.93<sup>†</sup></td>
    <td>52.52<sup>†</sup></td>
    <td>56.25<sup>†</sup></td>
    <td>28.41<sup>†</sup></td>
    <td>40.61<sup>†</sup></td>
  </tr>
  <tr>
    <td>THOR-MoE (Top-2)</td>
    <td><b>46.00<sup>†</sup></b></td>
    <td><b>23.35</b></td>
    <td><b>55.79</b></td>
    <td><b>61.06</b></td>
    <td>28.23</td>
    <td><b>42.89</b></td>
  </tr>
  <tr>
    <td>THOR-MoE (Top-\( p \), \( p=0.5 \))</td>
    <td>44.63<sup>‡</sup></td>
    <td>22.53<sup>‡</sup></td>
    <td>53.58</td>
    <td>58.65</td>
    <td>27.99</td>
    <td>41.48<sup>‡</sup></td>
  </tr>
</table>

Table 1: BLEU score on multi-domain translation benchmarks with decoder-only architecture. “†” and “‡” denote that statistically significant better than the best result of the counterpart (e.g., THOR-MoE (Top-2) vs. Trim-MoE (Top-2)) with t-test \( p < 0.05 \) and \( p < 0.01 \) hereinafter, respectively. The best and second best results are bold and underlined, respectively.

multilingual translation, the scores encompass the average ratings across all language pairs, such as English→Any (En→XX), and Any→English (XX→En) on the OPUS-16 dataset.

4.2 Implementation Details

In multi-domain translation, we use decoder-only Transformer architecture. Specifically, we use Qwen1.5-MoE-A2.7B (Team, 2024a). It has 14.3B parameters (60 non shared experts + 4 shared experts) in total and 2.7B activated parameters (4 non shared experts + 4 shared experts) during runtime. Due to limited GPU resource, we used a trimmed version, which has 3.5B parameters (8 non shared experts + 4 shared experts) in total and 2.3B activated parameters (2 non shared experts + 4 shared experts). To keep its capacity, we initialize the trimmed model with Qwen1.5-MoE-A2.7B and denote it as Trim-MoE. During training, we use Llama-Factory (Zheng et al., 2024) to instruct-tune LLMs. All LLMs are tuned on an 8×NVIDIA A100 GPUs (40G) with 1e-5 learning rate. We set gradient accumulation to 16 and batch size to 1, which gives us 2*8*16*1 batch in total. We use the DeepSpeed optimization (Rasley et al., 2020), and set ZeRO-3 optimization. Following Qin et al. (2024), we set the number of training epochs to 3. We set hyper-parameters \( \alpha, \beta, \gamma, \) and \( \delta \) to 1e-2, 1e-2, 1e-2, and 1e-4, which are determined by a grid search.

In multilingual translation, we follow (Zhao et al., 2024b) and compare our method with the Transformer-Base model (as Dense) and its MoE variants that have 6 encoder and decoder layers, 32 experts. The input and hidden dimensions of all feed-forward networks are 512 and 2048. We set the training processes to 35K iterations with a learning rate of 5e-4, which follows the inverse square root with 4,000 warm-up steps. To keep balanced training, we use temperature-based data sampling strategy with temperature 1.5. We set hyper-parameters \( \alpha, \beta, \gamma, \) and \( \delta \) to 1e-2, 5e-2, 5e-2, and 1e-4, which are determined by a grid search.

4.3 Comparison Models

Our comparison models mainly include two types: Decoder-only and Encoder-Decoder based.

Decoder-only. We fine-tune two dense models based on Qwen2.5-3B (Team, 2024b), which have similar parameters with the activated parameter and denoted as SFT-3B. Besides, based on the Trim-MoE-A2.3B model, we also fine-tuned three MoE models with Top-1, Top-2, and Top-\( p \) routing strategy.

Encoder-Decoder. There are three types of comparison models: vanilla dense model, vanilla MoE model, and enhanced MoE models via language knowledge. We use the Transformer-Base model as Dense model. For vanilla MoE model, the Switch Transformer (Fedus et al., 2022) with a top-1 token-based routing (as ST-MoE) and GShard (Lepikhin et al., 2021a) with a top-2 token-based routing (as GS-MoE) are adopted. For Language-specific MoE models, the LS-MoE with fixed routing inspired by (Pires et al., 2023), assigning 2 non-overlapping experts for tokens according to their source language in the encoder and target language in the decoder; Hybrid-MoE (Kudugunta et al., 2021), with a top-2 token routing in the encoder and a top-2 target language routing in the decoder side; Residual-MoE (Elbayad et al., 2023; Rajbhandari et al., 2022; Zhang et al., 2021) that augments each MoE layer with a shared feed-forward network through a binary gate function; Lingual-
MoE (Zhao et al., 2024b) stands for an MoE model with linguistic-guided routing and dynamic expert allocation, where the first-level language router selects the top 8 experts and the second-level token router activates the dynamical number of experts.

5 Main Results

Table 1 shows main results on the multi-domain translation benchmark with decoder-only architecture. Table 2 presents main results on the multilingual translation with encoder-decoder architecture. For a fair comparison, all models are trained and assessed on the OPUS-16 dataset with Transformer-Base as the backbone architecture.

5.1 Results on Multi-Domain Translation

Table 1 shows that the Trim-MoE-based models generally surpass the dense one. For example, the Trim-MoE (Top-2) outperforms SFT-3B model by average 2 BLEU scores where the activated parameters is less than dense model (2.3B vs. 3B), showing the effectiveness of MoE model. Furthermore, the proposed THOR-MoE significantly and consistently surpasses the counterpart of Trim-MoE-based one. For instance, the THOR-MoE (Top-2) outperforms the Trim-MoE (Top-2) by averaged 1.74 BLEU scores. This clear advantage confirms the superiority of incorporating domain knowledge and context knowledge into the routing. What’s important, the proposed approach is compatible with the Top-\( k \) (Shazeer et al., 2017a) and Top-\( p \) (Huang et al., 2024) routing strategies, validating the generalization of THOR-MoE, which can be a plug-and-play module. Besides, we find that the scores on the same domain (\( e.g. \), Law) change largely. The reason may be that the data distribution is unbalanced. We also list the results in terms of COMET (Rei et al., 2020) score in Table 3 and we can conclude the similar findings.

5.2 Results on Multilingual Translation

Table 2 summarizes the results and we can conclude several observations:

THOR-MoE vs. Dense and Vanilla MoE Baselines. The THOR-MoE substantially surpasses the Dense and vanilla MoE multilingual translation baselines with a large margin. Specifically, compared with Dense and ST-MoE, the THOR-MoE (Top-\( p \)) achieves {5.39%, 3.99%, 4.70%} and 2.46%, 0.55%, 1.51% averaged improvement in terms of BLEU score for Avg1., Avg2., and All Avg., respectively. This significant margin demonstrates the effectiveness of hierarchical language-guided routing and context-responsive routing.

THOR-MoE vs. Language-Guided MoE models. The THOR-MoE also consistently outperforms language-guided MoE models, including LS-MoE, hybrid-MoE, Residual-MoE, and Lingual-MoE. It shows again that the superiority of hierarchical language-guided routing and context-responsive routing. In detail, the Lingual-MoE performs the highest in baselines and it also employs a hierarchical language-group-guided and dynamical routing, which introduces the hard language id embedding and fails to consider code-mixed cases and that the token in different language can be the same (\( e.g. \), ‘Internet’ in German and English). In contrast, THOR-MoE applies a mixed language representation where the language ids are predicted, which comprehensively incorporates the languages knowledge. Furthermore, the dynamical routing in Lingual-MoE does not incorporate the context information. The context generally knows which token is difficult or not in a global view. The THOR-MoE fully considers the above issues and obtains better results.

6 Analysis

6.1 Ablation Study

We conduct ablation studies to investigate how well hierarchical context-responsive routing of THOR-MoE works. We conclude two findings from the results in Tab. 4.

(1) “w/o hierarchical task-guided routing”: \( i.e. \), without using any task-related routing and decaying to vanilla manner (select experts from full set of experts), the model performance greatly degrades on both translation tasks. It shows the necessity of using task-guided routing in a hierarchical manner.

(2) “w/o context-responsive routing”: the model performance becomes worse on both tasks when removing context. This shows that our context-responsive routing indeed can enhance the routing effectiveness and guarantee the token can be assigned to suitable and specialized experts with the indication of context, which thus benefits the model performance on both translation tasks.

6.2 Comparison among Different Task Representations

In this section, we aim to investigate the impact of task representation in different manners. Table 5
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="4">En→XX</th>
    <th colspan="4">XX→En</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>high</th><th>medium</th><th>low</th><th>Avg1.</th>
    <th>high</th><th>medium</th><th>low</th><th>Avg2.</th>
  </tr>
  <tr>
    <td>Dense</td>
    <td>Transformer-base</td>
    <td>25.37</td><td>39.12</td><td>14.78</td><td>26.16</td>
    <td>28.81</td><td>39.24</td><td>24.21</td><td>30.27</td>
    <td>28.21</td>
  </tr>
  <tr>
    <td></td>
    <td>GS-MoE</td>
    <td>25.55</td><td>40.71</td><td>18.97</td><td>27.70</td>
    <td>28.77</td><td>41.70</td><td>29.25</td><td>32.12</td>
    <td>29.91</td>
  </tr>
  <tr>
    <td></td>
    <td>ST-MoE</td>
    <td>26.55</td><td>43.04</td><td>20.23</td><td>29.09</td>
    <td>29.94</td><td>44.00</td><td>30.94</td><td>33.71</td>
    <td>31.40</td>
  </tr>
  <tr>
    <td></td>
    <td>LS-MoE</td>
    <td>20.06</td><td>37.31</td><td>11.72</td><td>22.29</td>
    <td>25.69</td><td>42.99</td><td>32.01</td><td>30.60</td>
    <td>26.44</td>
  </tr>
  <tr>
    <td></td>
    <td>Hybird-MoE</td>
    <td>24.56</td><td>31.70</td><td>13.38</td><td>23.55</td>
    <td>29.59</td><td>41.74</td><td>27.81</td><td>32.18</td>
    <td>27.85</td>
  </tr>
  <tr>
    <td>MoE</td>
    <td>Residual-MoE</td>
    <td>26.52</td><td>43.49</td><td>21.58</td><td>29.53</td>
    <td>29.84</td><td>43.94</td><td>31.25</td><td>33.72</td>
    <td>31.62</td>
  </tr>
  <tr>
    <td></td>
    <td>Lingual-MoE</td>
    <td>27.11</td><td>46.24</td><td>23.34</td><td>30.95</td>
    <td>29.87</td><td>43.49</td><td>32.02</td><td>33.81</td>
    <td>32.38</td>
  </tr>
  <tr>
    <td></td>
    <td>THOR-MoE (Top-1)</td>
    <td>27.21</td><td>46.37</td><td>23.88</td><td>31.17</td>
    <td>29.68</td><td>44.01</td><td>32.58</td><td>33.99</td>
    <td>32.57</td>
  </tr>
  <tr>
    <td></td>
    <td>THOR-MoE (Top-2)</td>
    <td>27.88<sup>†</sup></td><td>47.29<sup>†</sup></td><td>24.85<sup>†</sup></td><td>31.98<sup>†</sup></td>
    <td>30.38<sup>†</sup></td><td>44.52<sup>†</sup></td><td>33.26<sup>†</sup></td><td>34.64<sup>†</sup></td>
    <td>33.31<sup>†</sup></td>
  </tr>
  <tr>
    <td></td>
    <td>THOR-MoE (Top-p)</td>
    <td>27.63<sup>†</sup></td><td>46.82<sup>†</sup></td><td>24.15<sup>†</sup></td><td>31.55<sup>†</sup></td>
    <td>29.97</td><td>44.18<sup>†</sup></td><td>32.96<sup>†</sup></td><td>34.26</td>
    <td>32.91<sup>†</sup></td>
  </tr>
</table>

Table 2: Averaged BLEU scores on multilingual translation with encoder-decoder architectures. “†” denotes that statistically significant better than Lingual-MoE with t-test \( p < 0.05 \).

<table>
  <tr>
    <th>Models</th>
    <th>IT</th>
    <th>Koran</th>
    <th>Medical</th>
    <th>Law</th>
    <th>Subtitles</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>SFT-3B</td>
    <td>86.39</td><td>73.17</td><td>85.43</td><td>87.19</td><td>78.52</td><td>82.14</td>
  </tr>
  <tr>
    <td>THOR-MoE (Top-p)</td>
    <td>87.31</td><td>74.09</td><td>86.09</td><td>87.88</td><td>79</td><td>82.87</td>
  </tr>
</table>

Table 3: COMET scores on multi-domain translation benchmarks with decoder-only architecture.

<table>
  <tr>
    <th></th>
    <th colspan="2">Multi-Domain</th>
    <th colspan="2">Multilingual</th>
  </tr>
  <tr>
    <td></td>
    <td>Multi-Domain</td>
    <td>Multi-Domain</td>
    <td>Multi-Domain</td>
    <td>Multi-Domain</td>
  </tr>
  <tr>
    <td>THOR-MoE (Top-p)</td>
    <td>41.48</td><td>32.91</td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>w/o hierarchical task-guided routing</td>
    <td>30.16</td><td>31.37</td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>w/o context-responsive routing</td>
    <td>40.74</td><td>32.12</td>
    <td></td><td></td>
  </tr>
</table>

Table 4: Ablation Study with Avg. results.

<table>
  <tr>
    <th></th>
    <th>Multi-Domain</th>
    <th>Multilingual</th>
  </tr>
  <tr>
    <td>baseline (w/o any task Rep.)</td>
    <td>40.74</td>
    <td>32.12</td>
  </tr>
  <tr>
    <td>Non-Mixed Rep.</td>
    <td>40.95</td>
    <td>32.44</td>
  </tr>
  <tr>
    <td>Golden Rep.</td>
    <td>41.32</td>
    <td>32.78</td>
  </tr>
  <tr>
    <td>THOR-MoE (Mixed manner)</td>
    <td>41.48</td>
    <td>32.91</td>
  </tr>
</table>

Table 5: The Avg. results of different representation (Rep.) manner during hierarchical task-guided routing.

<table>
  <tr>
    <th></th>
    <th>Multi-Domain</th>
    <th>Multilingual</th>
  </tr>
  <tr>
    <td>baseline (w/o any task Rep.)</td>
    <td>40.74</td>
    <td>32.12</td>
  </tr>
  <tr>
    <td>Infuse task Rep. into token Rep.</td>
    <td>40.95</td>
    <td>32.44</td>
  </tr>
  <tr>
    <td>THOR-MoE (Hierarchical manner)</td>
    <td>41.48</td>
    <td>32.91</td>
  </tr>
</table>

Table 6: The investigation of why hierarchical manner.

![Average activated experts number across training steps on the multi-domain translation task.](page_1012_1342_388_282.png)

Figure 2: Average activated experts number across training steps on the multi-domain translation task.

shows the results where the ‘Non-Mixed Representation’ denotes that directly uses the automatically predict task label to extract corresponding task representation and ‘Golden Representation’ indicates using the golden task label to extract corresponding representation. We conclude that the task knowledge indeed has a positive impact on translation performance (vs. baseline). We also observe that using the automatically predicted task labels (actually the mixed task representation) shows better results than using ground truth and Non-Mixed manner in terms of Avg. BLEU scores. The reason may be that the mixed task representation has certain fault tolerance. We also analyze the accuracy of task prediction to further show its quality in Appendix A.

6.3 Why Hierarchical Design?

In this section, we aim to investigate the manner of using mixed task representation. Table 6 shows that introducing task knowledge indeed helps tokens route well and achieve improvement (vs. baseline). The hierarchical manner greatly outperforms the direct infusion with token representation before routing, proving the effectiveness of hierarchical design, which unlock the potential of MoE with the help of the mixed task knowledge.
<table>
  <tr>
    <th>Domains</th>
    <th>IT</th>
    <th>Koran</th>
    <th>Medical</th>
    <th>Law</th>
    <th>Subtitles</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th>Top-<i>p</i></th>
    <td>1.87</td>
    <td>1.95</td>
    <td>1.82</td>
    <td>1.92</td>
    <td>1.77</td>
    <td>1.87</td>
  </tr>
  <tr>
    <th>Top-<i>p</i> w/ ctx</th>
    <td>1.37</td>
    <td>1.61</td>
    <td>1.42</td>
    <td>1.56</td>
    <td>1.31</td>
    <td>1.45</td>
  </tr>
</table>

Table 7: Average activated experts (AE) in different domain translation tasks. ‘Top-<i>p</i> w/ ctx’ denotes the proposed method to infuse the context during routing.

6.4 Analysis of Efficient Training and Inference

To further explore whether our proposed method is efficient in training and inference, we calculate the average number of experts activated by the model on multi-domain translation tasks. Figure 2 and Table 7 shows the average number of experts activated per token across various translation tasks during training and inference. The result is averaged across all the layers of transformers.

During training, we can see that our method converges faster to use less activated experts than the vanilla Top-<i>p</i> (Huang et al., 2024). During inference, we can observe that across all five tasks, the number of activated experts is less than original Top-<i>p</i> routing, averaging 1.45 activated experts (less than 22% activated parameters) with better performance. Both findings show that the context plays a key role in guiding token routing.

7 Related Work

Neural Machine Translation. The NMT have received remarkable attention in the era of LLMs (Jawahar et al., 2023b; Zhou et al., 2023). Previous work mainly focuses on the continual learning of new domains (Gu and Feng, 2020; Gu et al., 2022; Liang et al., 2024), introducing ready-made task-related (linguistics) knowledge via adapter (Zhang et al., 2020, 2021), parameter sharing (Aharoni et al., 2019), or task-specific modules (Pires et al., 2023), and multilingual representation learning (Yang et al., 2021).

Mixture-of-Experts. Jacobs et al. (1991) first proposes the concept of MoE, which consists of a series of network sub-modules. The Sparsely-gated MoE, as a variant, which only activates a few expert networks for each input, has been shown its superiority in diverse NLP and computer vision applications(Shazeer et al., 2017b; Zoph et al., 2022). In the context of NMT, most of previous work focuses on addressing the over/under-fitting (Elbayad et al., 2023) via regularization strategies (Elbayad et al., 2023) or modularizing MoE (Li et al., 2023c; Zhang et al., 2024), and incorporating task-related knowledge (language, domain etc) (Kudugunta et al., 2021; Zhao et al., 2024b; Pham et al., 2023; Gururangan et al., 2022; Li et al., 2023b), achieving impressive performance. Besides, there are some studies aim to improve the training/inference efficiency of routing via adaptive computation (Jawahar et al., 2023a) or reducing the number of activated experts (Li et al., 2023a; Huang et al., 2024).

Different from these prior works that directly use the task-related knowledge, we propose a hierarchical context-responsive routing method where we automatically extract corresponding domain/language knowledge and design a hierarchical network to guide the task-level routing. Besides, we aim to help each token accurately select specialized and suitable experts and thus we incorporate the context to guide the token routing rather than relying on the localized token only in existing work. In this manner, the context-responsive routing can improve the training or inference efficiency, which is compatible with previous Top-<i>k</i> and Top-<i>p</i> routing policies.

8 Conclusion

In this paper, we propose a new hierarchical task-guided and context-responsive routing framework for NMT. To automatically obtain the task knowledge, we propose to predict it and then use mixed task representation. Consequently, we design a hierarchical routing at the task level and the token level. Further, we propose to inject context to enhance the effectiveness of token routing. Extensive experiments on multi-domain and multilingual translation benchmarks show the superiority and generalization of our proposed approach.

Limitations

While we introduce the mixed domain/language and context knowledge into routing in hierarchical manner and achieve good results, there are some limitations worth considering to study in future work: (1) In this study, the design relies on the prior (the number of tasks and language groups), which may limit its extention to broader topics (Li and Zhou, 2025); (2) This work only conduct experiments on translation tasks and does not conduct experiments other generation or discriminative tasks.
References

Roee Aharoni and Yoav Goldberg. 2020. Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7747–7763, Online. Association for Computational Linguistics.

Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874–3884. Minneapolis, Minnesota. Association for Computational Linguistics.

Mikko Aulamo and Jörg Tiedemann. 2019. The OPUS resource repository: An open package for creating parallel corpora and machine translation services. In Proceedings of the 22nd Nordic Conference on Computational Linguistics, NoDaLiDa 2019, Turku, Finland, September 30 - October 2, 2019, pages 389–394.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. Minneapolis, Minnesota. Association for Computational Linguistics.

Maha Elbayad, Anna Sun, and Shruti Bhosale. 2023. Fixing MoE over-fitting on low-resource languages in multilingual machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 14237–14253, Toronto, Canada. Association for Computational Linguistics.

Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Çelebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020. Beyond english-centric multilingual machine translation. CoRR, abs/2010.11125.

William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39.

Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. 2024. Better & faster large language models via multi-token prediction. Preprint, arXiv:2404.19737.

Shuhao Gu and Yang Feng. 2020. Investigating catastrophic forgetting during continual training for neural machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4315–4326, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Shuhao Gu, Bojie Hu, and Yang Feng. 2022. Continual learning of neural machine translation within low forgetting risk regions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1707–1718. Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2022. DEMix layers: Disentangling domains for modular language modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5557–5576, Seattle, United States. Association for Computational Linguistics.

Quzhe Huang, Zhenwei An, Nan Zhuang, Mingyu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. 2024. Harder task needs more experts: Dynamic routing in MoE models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12883–12895, Bangkok, Thailand. Association for Computational Linguistics.

Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. 1991. Adaptive mixtures of local experts. Neural Comput., 3(1):79–87.

Ganesh Jawahar, Subhabrata Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks Lakshmanan, V.S., Ahmed Hassan Awadallah, Sébastien Bubeck, and Jianfeng Gao. 2023a. AutoMoE: Heterogeneous mixture-of-experts with adaptive computation for efficient neural machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9116–9132, Toronto, Canada. Association for Computational Linguistics.

Ganesh Jawahar, Subhabrata Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Ahmed Hassan Awadallah, Sébastien Bubeck, and Jianfeng Gao. 2023b. Automoe: Heterogeneous mixture-of-experts with adaptive computation for efficient neural machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 9116–9132. Association for Computational Linguistics.

Songtao Jiang, Tuo Zheng, Yan Zhang, Yeying Jin, Li Yuan, and Zuozhu Liu. 2024. Med-MoE: Mixture of domain-specific experts for lightweight medical vision-language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3843–3860, Miami, Florida, USA. Association for Computational Linguistics.

Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of EMNLP, pages 388–395.
Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, NMT@ACL 2017, Vancouver, Canada, August 4, 2017, pages 28–39.

Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, and Orhan Firat. 2021. Beyond distillation: Task-level mixture-of-experts for efficient inference. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3577–3599, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021a. (GS)hard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations.

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021b. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.

Jiamin Li, Qiang Su, Yitao Yang, Yimin Jiang, Cong Wang, and Hong Xu. 2023a. Adaptive gating in mixture-of-experts based language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3577–3587, Singapore. Association for Computational Linguistics.

Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. 2023b. Branch-train-merge: Embarrassingly parallel training of expert language models.

Shangjie Li, Xiangpeng Wei, Shaolin Zhu, Jun Xie, Baosong Yang, and Deyi Xiong. 2023c. MMNNMT: Modularizing multilingual neural machine translation with flexibly assembled MoE and dense blocks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4978–4990. Singapore. Association for Computational Linguistics.

Ziyue Li and Tianyi Zhou. 2025. Your mixture-of-experts LLM is secretly an embedding model for free. In The Thirteenth International Conference on Learning Representations.

Yunlong Liang, Fandong Meng, Jiaan Wang, Jinan Xu, Yufeng Chen, and Jie Zhou. 2024. Continual learning with semi-supervised contrastive distillation for incremental neural machine translation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10914–10928, Bangkok, Thailand. Association for Computational Linguistics.

Hai Pham, Young Jin Kim, Subhabrata Mukherjee, David P. Woodruff, Barnabas Poczos, and Hany Hassan. 2023. Task-based MoE for multitask multilingual machine translation. In Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL), pages 164–172, Singapore. Association for Computational Linguistics.

Telmo Pires, Robin Schmidt, Yi-Hsiu Liao, and Stephan Peitz. 2023. Learning language-specific layers for multilingual machine translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14767–14783, Toronto, Canada. Association for Computational Linguistics.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of WMT, pages 186–191.

Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. 2024. OI replication journey: A strategic progress report–part 1. arXiv preprint arXiv:2410.18982.

Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. 2022. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. Preprint, arXiv:2201.05596.

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506.

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.

Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017a. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017b. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.

Qwen Team. 2024a. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters".

Qwen Team. 2024b. Qwen2.5: A party of foundation models.
An Wang, Xingwu Sun, Ruobing Xie, Shuaiqin Li, Ji-aiqi Zhu, Zhen Yang, Pinxue Zhao, J. N. Han, Zhanhui Kang, Di Wang, Naoaki Okazaki, and Cheng zhong Xu. 2024. Hmoe: Heterogeneous mixture of experts for language modeling. Preprint, arXiv:2408.10681.

Junhong Wu, Yuchen Liu, and Chengqing Zong. 2024. F-MALLOC: Feed-forward memory allocation for continual learning in neural machine translation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7180–7192, Mexico City, Mexico. Association for Computational Linguistics.

Yilin Yang, Akiko Eriguchi, Alexandre Muzio, Prasad Tadepalli, Stefan Lee, and Hany Hassan. 2021. Improving multilingual translation by representation and gradient regularization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7266–7279, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Zihao Zeng, Yibo Miao, Hongcheng Gao, Hao Zhang, and Zhijie Deng. 2024. AdaMoE: Token-adaptive routing with null experts for mixture-of-experts language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6223–6235, Miami, Florida, USA. Association for Computational Linguistics.

Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. 2021. Share or not? learning to schedule language-specific capacity for multilingual translation. In International Conference on Learning Representations.

Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1628–1639, Online. Association for Computational Linguistics.

Fan Zhang, Mei Tu, Song Liu, and Jinyao Yan. 2024. A lightweight mixture-of-experts neural machine translation model with stage-wise training strategy. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2381–2392, Mexico City, Mexico. Association for Computational Linguistics.

Hao Zhao, Zihan Qiu, Huijia Wu, Zili Wang, Zhaofeng He, and Jie Fu. 2024a. HyperMoe: Towards better mixture of experts via transferring among experts. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10605–10618, Bangkok, Thailand. Association for Computational Linguistics.

Xinyu Zhao, Xuxi Chen, Yu Cheng, and Tianlong Chen. 2024b. Sparse moe with language guided routing for multilingual machine translation. In The Twelfth International Conference on Learning Representations.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 400–410, Bangkok, Thailand. Association for Computational Linguistics.

Chulun Zhou, Yunlong Liang, Fandong Meng, Jinan Xu, Jinsong Su, and Jie Zhou. 2023. RC3: Regularized contrastive cross-lingual cross-modal pre-training. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11747–11762, Toronto, Canada. Association for Computational Linguistics.

Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906.

Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. 2022. Taming sparsely activated transformer with stochastic experts. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
<table>
  <tr>
    <th>Tasks/Groups</th>
    <th>Train</th>
    <th>Valid</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>Multi-Domain Translation Dataset (De→En)</td>
    <td>IT<br>Koran<br>Law<br>Medical<br>Subtitles</td>
    <td>0.22M<br>18K<br>0.47M<br>0.25M<br>0.5M</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Multilingual Translation Dataset</td>
    <td>16</td>
    <td>17,559,950</td>
    <td>30*1000</td>
    <td>30*1000</td>
  </tr>
</table>

Table 8: The data statistic of the multi-domain translation dataset and multilingual translation (OPUS-16) dataset. The number in Train/Valid/Test columns denotes the number of sentence pairs in each domain/language pair.

<table>
  <tr>
    <th>Models</th>
    <th>IT</th>
    <th>Koran</th>
    <th>Medical</th>
    <th>Law</th>
    <th>Subtitles</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Qwen1.5-MoE-A2.7B (SFT, Top-8)</td>
    <td>45.59</td>
    <td>23.31</td>
    <td>55.67</td>
    <td>60.18</td>
    <td>29.21</td>
    <td>42.79</td>
  </tr>
  <tr>
    <td>THOR-MoE (Top-8, p=0.75)</td>
    <td>47.75</td>
    <td>25.57</td>
    <td>57.97</td>
    <td>62.29</td>
    <td>30.29</td>
    <td>44.02</td>
  </tr>
  <tr>
    <td>Qwen1.5-MoE-A2.7B (SFT, Top-8)</td>
    <td>87.43</td>
    <td>74.24</td>
    <td>85.85</td>
    <td>87.78</td>
    <td>79.46</td>
    <td>83.97</td>
  </tr>
  <tr>
    <td>THOR-MoE (Top-8, p=0.75)</td>
    <td>87.45</td>
    <td>74.58</td>
    <td>86.45</td>
    <td>88.75</td>
    <td>80.23</td>
    <td>83.99</td>
  </tr>
</table>

Table 9: BLEU (top block) / COMET (below block) scores on multi-domain translation benchmarks with decoder-only architecture. Note that top-8 means it activates 4 shared experts and 4 non-shared experts out of 60 experts.

<table>
  <tr>
    <th></th>
    <th>IT</th>
    <th>Koran</th>
    <th>Medical</th>
    <th>Law</th>
    <th>Subtitles</th>
  </tr>
  <tr>
    <td>IT</td>
    <td>0.833</td>
    <td>0.027</td>
    <td>0.055</td>
    <td>0.0219</td>
    <td>0.065</td>
  </tr>
  <tr>
    <td>Koran</td>
    <td>0.012</td>
    <td>0.817</td>
    <td>0.003</td>
    <td>0.019</td>
    <td>0.149</td>
  </tr>
  <tr>
    <td>Medical</td>
    <td>0.111</td>
    <td>0.004</td>
    <td>0.737</td>
    <td>0.135</td>
    <td>0.013</td>
  </tr>
  <tr>
    <td>Law</td>
    <td>0.142</td>
    <td>0.008</td>
    <td>0.032</td>
    <td>0.798</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td>Subtitles</td>
    <td>0.058</td>
    <td>0.076</td>
    <td>0.009</td>
    <td>0.006</td>
    <td>0.851</td>
  </tr>
</table>

Table 10: BLEU (top block) / COMET (below block) scores on multi-domain translation benchmarks with decoder-only architecture. Note top-8 means it activates 4 shared experts and 4 non-shared experts out of 60 experts.

A Task Prediction

We also evaluate the performance of the Task Predictor to show whether the classifier can accurately predict suitable labels. The results are 82.45% and 64.89% for domain and language prediction, respectively. It suggests that our classifier can predict suitable labels and further provide effective mixed task representation for routing.

B Comparison to Larger Model

We implemented our proposed method based on a large-scale Qwen1.5-MoE-A2.7B (totally 14B, activated 2.7B, denoted as THOR-MoE). Besides, we also fine-tuned the vanilla Qwen1.5-MoE-A2.7B model as the baseline. The averaged COMET and BLEU results shown in Table 9 demonstrate that the THOR-MoE model at the 14B level also achieve better results than fine-tuned Qwen1.5-MoE-A2.7B model in terms of BLEU and COMET scores.

C Analysis of Domain Similarity

Following previous work (Wu et al., 2024), we conduct some analysis for domain similarity. Specifically, we sum the predicted soft label and then normalize it. The vertical axis and horizontal axis are unsupervised cluster label (clustering with BERT-base and k=5) and predicted soft labels, respectively. The similarity matrix are shown in Table 10.

The results show that there exist some sentences that are assigned to a cluster of another domain (i.e., hard cases). This makes sense as the mixed representation can capture such domain information, making the proposed approach works (the reason).

Reviewer paper summaries: In this paper, the authors point out the following weaknesses of translation using MoE.

(1) Even if multiple experts are used, it is not possible to use it in the real world because it is dependent on specific domains and languages

(2) It only uses local word information, and it is not possible to use the context information that is important for translation

To overcome these weaknesses, the authors propose a hierarchical context-sensitive routing framework. By making MoE hierarchical, they can select language- and domain-specific experts, solving the problem in (1). Next, they solve the problem in (2) by incorporating averaged context representations into the routing process.
Experiments were conducted on five domains and 16 language pairs, and the method was more effective than conventional methods.
Only transformer-base size models were tested, and no testing was done on large-scale language models.

This paper proposes THOR-MoE, a novel MoE arch designed for NMT. It introduces a hierarchical expert-selecting strategy, which first predicts the task and then uses context information to select an expert. Experiments on multi-domain and multi-lingual translation tasks, demonstrate the effectiveness of the proposed approach.

This paper introduces one MoE with hierarchical task-guided and context-responsive routing policies. The method firstly predicts the domain/language label and then extracts mixed domain/language representations.
Summary of strengths: - It overcomes the weaknesses of translation using MoE. In particular, the fact that it has expanded MoE, which could only use local word information, to enable the use of context information, which is important for translation, is a clear contribution.
- The experiment was conducted under multiple conditions, and it is expected to be effective.
- The paper was easy to read.

1. The proposed hierarchical MoE arch is novel and intuitive, utilizing task and contextual information in expert selection is a good idea.
2. Experimental results are quite strong and t-tests are conducted to ensure the significance.
3. The paper is well-written and easy to follow.

The method introduced in this paper is clearly described and the idea is good. Results show that this method also works well.
Summary of weaknesses: - When incorporating MoE into large-scale language models, issues such as computational cost are often cited. This proposed method combines multiple MoE in a hierarchical manner. Is it actually possible to incorporate this method into large-scale language models? Is it also possible to combine it with FasterMoE or Mixture of Parameter-Efficient Experts (MoPE)?

1. The task prediction module limits the generalization of the proposed approach, as the task predictor and embedding layer require pre-defined task labels for supervised training.
2. The training objective is complex and hard to optimize, this could lead to unstable training.
3. Some of the experimental results need further explanation. In Table 1, the performance advantages are mainly gained from the Medical and Law domains, could the author explain why Trim-MoE underperforms in these domains? 
4. The analysis is mainly conducted on final translation performance. The reported performance of task prediction is relatively low for a simple classification task. What would be the performance if the ground truth label is given in inference? 
5. Is it really necessary for multi-lingual MT to predict the target language? Generally, one would get the target language id in a translation request.

1. Authors should evaluate this method on some LLMs and compare this method with other LLM-based methods. Authors have evaluated this method on decoder-only models while it is better to evaluate this method on other models. 

2. Authors report the BLEU score performance of models in Table 1 and 2, while it is better to give more results in other translation metrics instead of BELU only.

3. There is only an experiment of THOR-MoE with Top-p in Table 2 while Top-1 and Top-2 in Table 1. The authors should evaluate THOR-MoE with Top-1 and Top-2 and give the results in Table 2.

4. There are some typos in this paper. For example, in line 213, the authors use Equation 5, while in 262, authors use Eq. 8. Line 281, schedules->schedule.

5. Authors should give more figures to describe their method instead of Figure 1 only.  Beside, authors put Figure 1. 2 below the Figure 1.3 which is confusing.
Comments, suggestions and typos: I don't think that the content of L124 to L134 can be called a 'contribution'. Of course, I know that this format is popular. If you are describing it as a 'contribution to the NLP community', then the content of L124-134 is an explanation of your method, not a 'contribution'.I think that your contribution is only L135-141.

1. There are many typos in loss functions.
2. Currently the evaluation is conducted with BLEU only, I suggest the authors add the COMET score to their evaluation.

N/A
Overall assessment scores: 4

3.5

3.5