HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts (2024)

Abstract: The Mixture of Experts (MoE) for language models has been proven effective in augmenting the capacity of models by dynamically routing each input token to a specific subset of experts for processing. Despite the success, most existing methods face a challenge for balance between sparsity and the availability of expert knowledge: enhancing performance through increased use of expert knowledge often results in diminishing sparsity during expert selection. To mitigate this contradiction, we propose HyperMoE, a novel MoE framework built upon Hypernetworks. This framework integrates the computational processes of MoE with the concept of knowledge transferring in multi-task learning. Specific modules generated based on the information of unselected experts serve as supplementary information, which allows the knowledge of experts not selected to be used while maintaining selection sparsity. Our comprehensive empirical evaluations across multiple datasets and backbones establish that HyperMoE significantly outperforms existing MoE methods under identical conditions concerning the number of experts. Our code is publicly available at https: //github.com/Bumble666/Hyper_MoE

heading: 1 Introduction
text: The accelerated advancement of large language models has culminated in their widespread application across various domains, including healthcare, education, and social interactions (Brown et al., 2020; Achiam et al., 2023; Touvron et al., 2023). The remarkable capabilities of these models are attributed to the enhancements in their scale. Nevertheless, the scaling of dense models is often hampered by significant computational demands, posing a challenge to developing the Natural Language Processing (NLP) community. In response,
* Corresponding authors.
sparse activation models have emerged as a solution (Artetxe et al., 2022; Du et al., 2022), activating only a subset of parameters for different inputs, thus mitigating computational costs. One of the most representative methods is the Mixture of Experts (MoE, Shazeer et al. (2017)), which routers different inputs to specific groups of experts, thereby enlarging the model’s capacity without increasing computational burdens.
The key to effectively reducing computational costs lies in the sparsity of expert selection, with the number of experts selected for each token being kept at a lower level. In practical applications or experiments, existing works (Roller et al., 2021a; Fedus et al., 2022; Rajbhandari et al., 2022; Xue et al.,
10605
2023) usually select only one or two experts per input. However, increasing the number of selected experts per token can enhance the availability of expert knowledge and improve the performance of downstream tasks (Yang et al., 2019; Shazeer et al., 2017; He et al., 2023). This scenario positions MoE model in a predicament akin to a zero-sum game: a choice between increasing the number of available experts to improve performance or preserving a lower level of available experts to ensure sparsity, as depicted in Figure 1.
To mitigate this contradiction, one solution would be to use the knowledge of other experts to assist the sparsely selected experts. This is similar to multi-task learning, which transfers knowledge among related tasks. Some works (Karimi Mahabadi et al., 2021; Ivison and Peters, 2022; Zhao et al., 2023) suggest using hypernetworks (Ha et al., 2017) to generate task-specific knowledge to enhance positive transfer between tasks. Inspired by this, we aim to increase the availability of expert knowledge by transferring the knowledge of unselected experts while sparsely selecting experts.
In this paper, we propose HyperMoE, a novel MoE framework built upon hypernetworks, which captures the information from every expert by leveraging expert-shared hypernetwork while achieving positive expert transfer by generating conditioned modules individually. We refer to the information as cross-expert information. Specifically, a HyperMoE consists of HyperExperts, which are generated based on the information of unselected experts and serve as supplementary information for selected experts while maintaining sparsity.
We further improve upon this by introducing the concept of cross-layer Hypernetworks: A hypernetwork is shared among all transformer layers, which enables information flow among MoEs in different layers. This brings additional efficiency in terms of parameters and computational costs: Despite the additional computation, our method only experienced a decrease1 of approximately 15% in training speed and 10% in inference speed compared to the standard MoE.
We evaluate HyperMoE on 20 representative NLP datasets across diverse tasks: sequence classification, extractive question answering, summariza-
1The degree of decline in speed is related to the scale of the Hypernetworks and the bottleneck size in the generated HyperExpert (similar to r in LoRA). For various tasks, these hyperparameters can be dynamically adjusted to control the delay.
tion, and text generation. Extensive experimental results show that HyperMoE outperforms baselines, including Switch Transformer (Fedus et al., 2022) with MoE architecture. This demonstrates the effectiveness of our method in transferring knowledge to experts, which increases the utilization of expert knowledge while keeping the number of experts selected at a low level.
To summarise, our core contributions are:
• We propose a novel HyperMoE architecture with HyperExpert for MoE framework, which resolves the inherent tension between maintaining sparse expert selection and ensuring sufficient expert availability within MoE.
• HyperMoE outperforms baselines based on Switch Transformer across a diverse set of NLP tasks, confirming our approach’s effectiveness.
• We show the relevance between selection embeddings, which are based on the context of unselected experts, and selected experts, indicating that the selection embeddings effectively encode the information of knowledge that the currently selected experts need.

heading: 2 Background

heading: 2.1 Mixture of Expert
text: A Mixture of Experts (MoE) typically consists of two parts: the gate model G and a set of expert models E1, E2, · · · , EN . The gate model is used to dynamically select and combine the outputs of the expert models based on the input x. As a result, each input will be determined by the collective participation of multiple expert models to obtain the output y:
y =
N∑
i=1
G(x)iEi(x). (1)
The gate model G(·) is a Noisy Top-K Network (Shazeer et al., 2017) with parameters Wg and Wnoise. This gating method introduces adjustable noise and then retains the top-k values as the final output:
G(x) =TopK (Softmax (xWg
+N (0, 1) Softplus (xWnoise ))) , (2)
where TopK(·) denotes selecting the largest K elements.
MoE allows for flexible adjustment of the contribution of expert models in different input scenarios, thereby improving the overall performance and adaptability of the model.

heading: 2.2 HyperNetworks
text: Hypernetwork (Ha et al., 2017) can generate parameters to be used by target networks or modules. Specifically, a hypernetwork with independent parameters ϕ denoted as hϕ, leverages an context information z to generate the target parameters θ for the primary network fθ and the primary network with an input x is redefined as:
output = fθ(x) = fhϕ(z)(x). (3)
This method of flexibly adjusting the parameters of the target network to adapt to different input scenarios is widely used in multi-task learning (Karimi Mahabadi et al., 2021; Üstün et al., 2022) and few-shot learning (Ponti et al., 2021). While generating condition-specific parameters, these parameters also benefit from shared knowledge (Pfeiffer et al., 2023).

heading: 3 Method
text: Overview. Taking inspiration from knowledge transferring between different tasks in multi-task learning, we propose HyperMoE. The key idea of HyperMoE is to enhance the availability of knowledge for the current input through positive knowledge transfer between experts. Through the condition input we designed, the relevant knowledge within the cross-expert information captured by the hypernetwork is encoded into HyperExpert, serving as supplementary information for the currently selected experts. In this work, we introduce conditional expert, in which we use shared hypernetworks to generate the expert weights based on the information of the unselected experts. The hypernetworks capture information across experts and transfer relevant knowledge to the selected experts by conditional generation.

heading: 3.1 Conditional Expert
text: In the transformer model based on the MoE structure, the experts E1, E2, · · · , EN in MoE are typically denoted as a group of parallel FFNs. For an input x ∈ Rh, the output y ∈ Rh can be calculated by the FFN layer as follows:
y = FFN(x) = σ(xW1)W2, (4)
where W1 ∈ Rh×b and W2 ∈ Rb×h are weight matrices. σ(·) denotes a non-linear activation function.
In our approach, the matrices W1 and W2 are generated by a hypernetwork as described in Section 3.2. In addition, we adopt a bottleneck structure for the conditional expert to improve parameter efficiency inspired by the Adapter (Houlsby et al., 2019). Specifically, the bottleneck dimension b satisfies b ≪ h in our method.

heading: 3.2 HyperExpert
text: These works (Karimi Mahabadi et al., 2021; He et al., 2022; Phang et al., 2023; Ivison et al., 2023) indicate that hypernetworks can learn the parameter information of the main neural network under different input scenarios and efficiently adjust the parameters of the target network to adapt to this information.
Consequently, we propose a novel design called HyperExpert, which captures beneficial knowledge from cross-expert through conditional generation to serve as auxiliary information for the selected experts involved in the computation, as shown in Figure 2. This also results in the extra parameters increasing sub-linearly with the number of layers, enhancing the parameter efficiency of the model. Selection Embedding. We define the selection embedding to encode the information of experts not selected for each token. Let pi ∈ Rt denote the selection embedding for i-th token and t denotes the dimension. To calculate the selection embedding efficiently and achieve better generalization, we introduce a group of expert embedding {Sm}Nm=1, where Sm ∈ Rt′ represents the m-th expert out of N experts. The computation process is as follows:
Ẑi = I − Zi = I −G(xi), (5)
pi = MLP(
N∑
j=1 Sj zi,j∑N j=1zi,j ), (6)
where G(·) denotes Noisy Top-K Network as described in Section 2.1. The vector Zi ∈ R|N | represents token-expert allocations: each element zi,j is a binary scalar indicating if the expert embedding Sj is active for the input token xi. I is an identity vector. MLP(·) is consisting of two feed-forward layers and a ReLU non-linearity. HyperExpert. We use a hypernetwork He(·) to construct HyperExpert Ê based on the conditional information of the unselected experts. To better
share information across different layers and improve parameter efficiency, we share the hypernetwork among all layers. Additionally, we define the layer embeddings lτ ∈ Rt′ for the τ -th Transformer layer. After that, we feed a concatenation of selection embedding and layer embedding to a project network to acquire final embedding kiτ = h(pi, lτ ), which is the input to hypernetwork He(·) to generates the weight matrices Dτi and W τi for HyperExpert:
(Dτi , U τ i ) = He(k i τ ) = (W D,WU )kiτ . (7)
The weight matrices of hypernetworks WD and WU are used to generate the down-projection matrix Dτi ∈ Rh×b and the up-projection matrix U τi ∈ Rb×h in the HyperExpert Êi for i-th token at τ -th transformer block.
Finally, we insert HyperExpert into the expert layer of MoE in parallel and calculate the output of i-th token as follows:
Êi(xi) = Relu(D τ i x)U τ i , (8)
yi = N∑
r=1
G(xi)Er(xi) + Êi(xi). (9)
In this way, the hypernetwork acts as an information capturer across experts, while the selection embeddings efficiently extract knowledge of experts suitable for the current token selection from the
hypernetwork and generate HyperExpert to reduce the transfer of negative knowledge in cross-expert information.

heading: 4 Experiments

heading: 4.1 Datasets
text: We evaluate HyperMoE on 20 NLP datasets across diverse tasks including sequence classification, question answering, summarization, and text generation. GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) benchmarks are widely used evaluation datasets for assessing natural language understanding capabilities. Both of them are a collection of text classification tasks: sentence similarity (STS-B; Cer et al., 2017), (MRPC; Dolan and Brockett, 2005), (QQP; Wang et al., 2018), question-answering (BoolQ; Clark et al., 2019), (MultiRC; Khashabi et al., 2018), (RECORD; Zhang et al., 2018), sentiment analysis (SST-2; Socher et al., 2013), sentence acceptability (CoLA; Warstadt et al., 2019), natural language inference (MNLI; Williams et al., 2018), (QNLI; Demszky et al., 2018), (RTE; Giampiccolo et al., 2007), (CB; De Marneffe et al., 2019), word sense disambiguation (WIC; Pilehvar and Camacho-Collados, 2019), coreference resolution (WSC; Levesque et al., 2012) and sentence completion (COPA; Roemmele et al., 2011). For the question-answering task, we consider SQuAD
v1.1 (Rajpurkar et al., 2016), a collection of question-answer pairs derived from Wikipedia articles, with each answer being a text span from the corresponding reading passage. For the summarization task, we use Xsum (Narayan et al., 2018) and CNN/Daily Mail(CNNDM) (Hermann et al., 2015) to test the model’s ability to summarize articles. And finally, the WikiText-2 dataset (Merity et al., 2017) is used to measure the ability of long-range dependencies generation.

heading: 4.2 Experiments Details
text: Following (He et al., 2023), we fine-tune pretrained MoE models on downstream tasks and report results from the last checkpoint. Unless otherwise specified, Our base model primarily uses Switch Transformer-base-8, which is an MoE model built on T5-base (Raffel et al., 2020) with 8 available experts, having a total number of parameters of 620M. For the WikiText dataset, we employ GPT-2 small (Radford et al., 2019) as the base model and expand it into the MoE structure by duplicating the weights of the feed-forward layer. In addition, we also use Switch Transformer-base16/32 to explore the effect of expert numbers on our method. To achieve a fair comparison, all methods in our paper employ the same Top-1 routing and auxiliary loss. For different data scales, we grid-search the training epoch and batch size from {10, 15, 20}, and {8, 16, 32, 64}, respectively. The learning rate is grid-search from {1e-5, 5e-5, 1e4, 5e-4} with Adam optimizer and the first 10% warm-up steps. We set the maximum token length to 1024 for WikiText datasets, 348 for SQuAD, and 256 for all other datasets except for the summarization task. For Xsum and CNNDM, we set the max
length of source articles to be 1024 and the max length of the target summary to be 128. As for All experiments run for 3 times with different seeds and we report the average for each result.

heading: 4.3 Baselines
text: Our approach is built upon Switch Transformer (Fedus et al., 2022), a well-known MoE model using Top-1 routing. Consequently, we primarily compare our approach with the following baselines: (1) MoE, fully finetuning switch transformer model. (2) MoE-Share, as it is a relevant baseline that does not exploit the inductive bias of the relationship between selected and unselected experts in the process of computation: add an MLP network that is shared among all experts in the MoE layer of a switch transformer, which has the same size as the experts in MoE.

heading: 4.4 Results and Analysis

heading: 4.4.1 Main Results
text: GLUE and SuperGLUE. Table 1 shows the results of various methods applied to the tasks within GLUE and SuperGLUE. Overall, our method improves significantly compared to both MoE and MoE-Share. Specifically, compared to MoE, our method shows a +0.48% and +0.84% increase on the GLUE and SuperGLUE benchmarks, respectively. This enhancement underscores the advantage of adopting expert knowledge transfer in improving the performance of MoE models. It’s noteworthy that MoE-Share is relevant to ours, but performs worse than MoE on certain datasets such as STS-B, CoLA, and BoolQ. A potential reason is that the cross-expert information captured through a shared network cannot achieve effective positive
transfer, adversely impacting MoE-Share’s effectiveness on these datasets. In contrast, our method maintains a lead on these datasets while also performing well on most other datasets. This underscores the effectiveness of our conditional generation strategy: selectively transferring knowledge by leveraging expert selection information during the computation process. Other Tasks. Table 2 displays the performance of various methods across question-answering tasks, summarization tasks, and text-generation tasks. In addition to achieving outstanding performance on Natural Language Understanding (NLU) tasks represented by GLUE and SuperGLUE, our method also excels in Natural Language Generation (NLG) tasks. Experimental results show that our method outperforms baseline methods across all NLG tasks. Specifically, in extractive question-answering tasks, our method shows improvements of 0.50% and 0.64% over MoE and MoE-Share, respectively. Like the NLU tasks, MoE-Share again underperforms, indicating that the extra networks may not effectively learn information useful to experts without the expert selection inductive bias. Furthermore, our method still performs well in summarization tasks involving long-text inputs. This demonstrates that our method can still effectively enhance the availability of expert knowledge through knowledge transfer under complex input conditions, suitable for tasks of various text lengths. Lastly, our method also achieves considerable improvement on Wikitext. These results demonstrate the effectiveness of HyperMoE in various tasks.
Additionally, we compared our method with a related baseline (Do et al., 2023) that uses a hypernetwork to generate conditional parameters for the routing of MoEs, aiming to address the routing allocation problem in SMoE methods, where all
experts tend to have similar representations. Experimental results can be found in Appendix B.

heading: 4.4.2 Ablation Study
text: We conduct an ablation study on the SQuAD to evaluate the effectiveness of the proposed modules. The embedding design is removed to verify the effect of using external information as embeddings. As shown in Table 3 (row 1), when the embedding and hypernet are removed, our method is equivalent to MoE. Table 3 (row 2) omits the embedding design, directly using the sample’s hidden state as input to the hypernetwork. This results in a marked decrease in performance, even falling below that of MoE. This suggests that conditioning the hypernetwork on the sample enlarges the parameter search space and is difficult to optimize. In an additional experiment, we use a depthwise separable convolutional network (Howard et al., 2017) with kernels of sizes 5×5 and 3×3 to compress and reduce the dimensions of the experts’ weights, obtaining expert embeddings. More details are in Appendix A. The selection embeddings are then computed and input into the hypernetwork as described in Section 3.2. Empirically, expert weights can better represent the information of experts. However, as shown in Table 3 (row 3), this strategy leads to a slight drop in performance, defying expectations. A potential explanation is the substantial information loss associated with compressing expert weights, resulting in a loss of specific information details. We leave the exploration of this strategy to future work.

heading: 4.4.3 Performance in Scaling the Number of Experts and Model Size
text: To explore the impact of the variation in the number of experts on our method, we fine-tuned on the SQuAD dataset using Switch Transformer-base16/32 as pre-trained models. These models possess 16 and 32 experts in each MoE layer, respectively. As demonstrated in Figure 3, every method
achieves performance enhancement across models featuring a diverse number of experts. Notably, our method exhibits consistent superior growth and outperforms the others. This indicates that the proposed conditional generation strategy can still effectively benefit from knowledge transfer as the number of experts increases.
In addition, to verify the effectiveness of our method on larger parameter-scale language models, we fine-tuned DeepSeek-MoE (Dai et al., 2024), which has 16 billion parameters, on the Open-Platypus (Lee et al., 2023) dataset and tested the fine-tuned model on MMLU (Hendrycks et al., 2020), GSM8K (Cobbe et al., 2021), and BBH (Suzgun et al., 2023). As shown in Table 4, our method consistently achieved improvements. This indicates that our method can be widely applied to language models ranging from hundreds of millions to tens of billions of parameters.

heading: 4.4.4 Investigating of Selection Embedding.
text: The unselected expert embeddings are more informative than selected expert embeddings. Empirically, by conditioning on the information of unselected experts, specific knowledge can be extracted from cross-expert knowledge, which selected experts do not possess, thereby aiding the selected experts. To verify this idea, we input embeddings of both selected and unselected experts into a hypernetwork, comparing their performance on the Xsum and CNNDM datasets. As shown in Figure 4, using unselected expert information as conditional input can achieve comparable results. This implies that the conditional information of unselected experts can generate more beneficial knowledge for the selected experts through a shared hypernetwork. Expert embeddings and the selection embeddings have a corresponding relationship. In addition, to explore whether the embeddings encode the information in our proposed method, we provide visualizations of the expert embeddings and computed selection embeddings within the final MoE layer of Switch Transformer-base-8 learned on CNNDM. Figure 5 reveals that both sets of embeddings exhibit sparse distributions, suggesting that the embeddings encode some specific non-relevant information. We also observe a correlation between the distances among selection embeddings and the distances among expert embeddings, such as between 4-5-6, 1-2-8, 7-3. This correlation implies that the information of the unselected experts encoded by the selection embeddings depends on the information of the selected experts, further illustrating that the selection embeddings effectively capture the information of the knowledge the currently selected experts need.

heading: 4.4.5 Impact of Additional Computation
text: Although our method achieves significant performance improvements compared to the original MoE structure, it introduces additional networks, which inevitably slightly reduces the inference speed of HyperMoE. We evaluate the number of
samples per second that our method can train/infer based on Switch Transformer-base-8. The methods in each task employ the same batch size. As shown in Table 5, our method’s training/inference speed is only reduced by about 15% and 10% compared to MoE, respectively. This suggests that our approach can enhance the availability of expert knowledge more effectively without significantly increasing computational costs while maintaining sparsity during expert selection.

heading: 5 Related Work

heading: 5.1 Mixture of Expert
text: Shazeer et al. (2017) introduces Mixture-of-Expert layers for LSTM language modeling and machine translation. These layers are inserted between the standard layers of the LSTM model. Subsequent work primarily builds on Transformers, where expert layers often replace dense layers. Lepikhin et al. (2021) first introduces MoE layers into Transformers and studies them in the context of machine translation. With the release of Gshard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2022), MoE models are scaled up to new heights by introducing thousands of small-scale experts. In terms of routing, Shazeer et al. (2017) use routing to the top k experts out of k > 1. Hazimeh et al.
(2021) propose DSelect-k, a smoothed version of the top-k routing algorithm that improves upon standard top-k routing. Fedus et al. (2022), Clark et al. (2022) and Xue et al. (2023) demonstrate that top-1 routing can also achieve competitive results. Hash Layer (Roller et al., 2021b) and StableMoE (Dai et al., 2022) employ fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert selection routing strategy where each token can be assigned to a different number of experts. Qiu et al. (2023) demonstrates that a standard LM can be fine-tuned as its Mix of Experts (MoE) counterpart, effectively improving downstream in-domain and out-of-domain generalization capabilities. Rajbhandari et al. (2022) and Dai et al. (2024) isolate general knowledge from experts using shared experts from engineering and algorithm perspectives, respectively, to promote expert specialization.
In contrast to previous work, our work mainly focuses on the knowledge transfer between experts in MoE. This provides a solution for improving the availability of expert knowledge in MoE while maintaining sparsity.

heading: 5.2 HyperNetwork
text: Hypernetworks (Ha et al., 2017) are widely used in multi-task learning due to their ability to avoid negative interference of corresponding modules by soft parameter sharing and generating module parameters conditioned on the shared parameters. The most common approach usually takes task (Karimi Mahabadi et al., 2021; Zhao et al., 2023) or language embeddings (Üstün et al., 2020; Baziotis et al., 2022) as contextual information to generate corresponding module parameters, such as adapter layers (Üstün et al., 2020; Ansell et al., 2021; Karimi Mahabadi et al., 2021), classifier heads (Ponti et al., 2021), and continuous prompts (He et al., 2022). In addition, hypernetwork-based approaches have also been very successful in zero-shot and few-shot scenarios (Deb et al., 2022; Phang et al., 2023; Ivison et al., 2023).
In the field of NLP, hypernetworks are mainly used to improve the generalization (Volk et al., 2022; Zhang et al., 2023) and applicability (Wullach et al., 2022; He et al., 2022; Tan et al., 2023) of dense models. HyperRouter (Do et al., 2023) chooses to solve the routing allocation problem in SMoE methods by using a fixed HyperNetwork to generate routing weights. Our work explores
the integration of hypernetworks with sparse MoE. We propose to input the expert selection status of tokens as information into the hypernetwork and generate module parameters that correspond to the respective tokens. To the best of our knowledge, this is the first time that hypernetworks have been introduced in the MoE structure, this is the first time HyperNetwork has been applied to the expert structure in MoEs, introducing inductive bias through the hypernetwork: information from unselected experts is reused, which extends the application scope of hypernetworks and provides new insights for knowledge transferring in MoE.

heading: 6 Conclusion
text: In this work, we introduce HyperMoE, a novel Mixture of Experts (MoE) architecture. Inspired by the concept of knowledge transfer in multi-task learning, we propose a method to facilitate knowledge transfer between experts through conditional generation. Our method enhances expert knowledge availability while maintaining expert selection’s sparsity. We show the effectiveness of our approach across a wide range of NLP tasks. Experimental results demonstrate that our method exhibits excellent performance compared to the conventional MoE. Furthermore, our analysis shows that without any measures, there could be negative knowledge transfer across experts when transferring knowledge to specific experts. Our approach mitigates this issue by capturing the contextual information of experts. We explore the feasibility of knowledge transfer between experts in MoE, providing a new perspective for future improvements in MoE architectures.
Limitations
Despite our work has demonstrated strong experimental results, there are several limitations: (1) In this work, we utilize end-to-end training to learn expert embeddings. Incorporating prior knowledge, such as expert weights, into the embedding learning process may improve efficiency and performance. We will improve upon this in future work. (2) We insert HyperExpert into the expert layer of MoE in parallel. This incurs additional computational overhead. Mitigating this issue could be achieved by employing some parameter-efficient methods (such as LoRA (Hu et al., 2022) and (IA)3 (Liu et al., 2022)) to insert HyperExpert into MoE. (3) Current experiments mainly focus on fine-tuning
the pre-trained MoE model. Utilizing our proposed method to train a large-scale MoE from scratch will be the emphasis of our future work.

heading: Acknowledgements
text: This work is supported by the National Key R&D Program of China (Grant No. 2021ZD0110100), National Natural Science Foundation of China (Grant No. 62176025, U21B2045), Beijing Nova Program (Grant No.20220484161), and Themebased Research Scheme (T45-205/21-N) from Research Grants Council of Hong Kong and Development Centre from InnoHK.

heading: A Depthwise Separable Convolutional Networks Details
text: For every expert weight in each MoE layer, we use the same convolutional network to reduce its dimensionality. First, we stack them so that their dimensional form is three-dimensional, similar to images. Then, we perform convolution on them. Our experiments used depthwise separable convolutions, with specific parameters as shown in Table 6.
The compressed expert weights are used as expert embeddings in subsequent computations as described in Section 3.2.

heading: B Additional Results
text: Here, We conducted additional experiments to demonstrate the superiority of our method over similar methods. The experiment is based on SwitchTransformer and GPT2-MoE, with all methods having the same number of parameters and experts. Moreover, we trained GPT2-MoE from scratch on WikiText-103, and the experimental results are as shown in Table 7.
The results show that our method achieves the best performance both scratch training and finetuning. The way HyperRouter generates router weights causes the model to lose its original pretrained weights, making HyperRouter ineffective in fine-tuning scenarios. The reason HyperRouter’s performance is inconsistent with the original re-
sults under the scratch training scenario might be because we fixed the number of experts chosen during the training process, instead of gradually increasing the number of chosen experts during training as in the original setup. In the original setting, the computational cost of HyperRouter is significantly higher than the method with a fixed number of experts. For a fair comparison, we chose the same expert selection setting. Experimental results show that HyperMoE can efficiently transfer knowledge between different experts in MoE when choosing the same number of experts.

heading: Future Work
text: We explore the feasibility of knowledge transfer between experts in MoE, providing a new perspective for future improvements in MoE architectures. Limitations
Despite our work has demonstrated strong experimental results, there are several limitations: (1) In this work, we utilize end-to-end training to learn expert embeddings. Incorporating prior knowledge, such as expert weights, into the embedding learning process may improve efficiency and performance. We will improve upon this in future work. (2) We insert HyperExpert into the expert layer of MoE in parallel. This incurs additional computational overhead. Mitigating this issue could be achieved by employing some parameter-efficient methods (such as LoRA (Hu et al. , 2022) and (IA)3 (Liu et al. , 2022)) to insert HyperExpert into MoE. (3) Current experiments mainly focus on fine-tuning
the pre-trained MoE model. Utilizing our proposed method to train a large-scale MoE from scratch will be the emphasis of our future work