Venue: NAACL 2025

# CartesianMoE: Boosting Knowledge Sharing among Experts

via Cartesian Product Routing in Mixture-of-Experts

 Zhenpeng Su\({}^{1,2}\) Xing Wu\({}^{1,2}\) Zijia Lin\({}^{3+}\) Yizhe Xiong\({}^{3}\) Minxuan Lv \({}^{1,2}\) Guangyuan Ma\({}^{1,2}\) Hui Chen \({}^{3+}\) Songlin Hu\({}^{1,2*}\) Guiguang Ding\({}^{3}\)

\({}^{1}\)Institute of Information Engineering, Chinese Academy of Sciences

\({}^{2}\)University of Chinese Academy of Sciences

\({}^{3}\)Tsinghua University

suzhenpeng130163.com {wuxing,maguanguyan,lvmixuan,husonglin}@iie.ac.cn linzjiao7@tsinghua.org.cn dingggetsinguua.edu.cn, {xiongyzhee20801,huichen}@gmail.com

Corresponding authors.This work was supported by Beijing Natural Science Foundation (L247026) and National Natural Science Foundation of China (No 62441235).

###### Abstract

Large language models (LLM) have been attracting much attention from the community recently, due to their remarkable performance in all kinds of downstream tasks. According to the well-known scaling law, scaling up a dense LLM enhances its capabilities, but also significantly increases the computational complexity. Mixture-of-Experts (MoE) models address that by allowing the model size to grow without substantially raising training or inference costs. Yet MoE models face challenges regarding knowledge sharing among experts, making their performance somehow sensitive to routing accuracy. To tackle that, previous works introduced shared experts and combined their outputs with those of the top \(K\) routed experts in an "addition" manner. In this paper, inspired by collective matrix factorization to learn shared knowledge among data, we propose CartesianMoE, which implements more effective knowledge sharing among experts in more like a "multiplication" manner. Extensive experimental results indicate that CartesianMoE outperforms previous MoE models for building LLMs, in terms of both perplexity and downstream task performance. And we also find that CartesianMoE achieves better expert routing robustness.

## 1 Introduction

Large language models (LLM) have demonstrated impressive performance across various downstream natural language tasks Touvron et al. (2023); Dai et al. (2022); Brown et al. (2020); Anil et al. (2023); Chowdhery et al. (2022); Radford et al. (2019); Rae et al. (2021); Biderman et al. (2023). Moreover, the well-known scaling law suggests that, as the model size increases, the model capabilities will continue to improve Kaplan et al. (2020); Hoffmann et al. (2022). However, for dense LLMs, the computational costs of scaling up their model sizes can become prohibitively high. To tackle that, sparse activation networks are proposed Child et al. (2019); Du et al. (2022). They reduce computational costs by activating only a subset of parameters for each input. A prominent approach among them is the mixture-of-experts (MoE) Lepikhin et al. (2021); Du et al. (2022); Dai et al. (2024); Fedus et al. (2022); Roller et al. (2021), which involves training multiple experts but using only a subset to process each input, with each expert generally being a feed-forward network (FFN). Compared to dense LLMs of equivalent sizes, MoE LLMs effectively reduces computational costs while delivering comparable results, in terms of both perplexity (PPL) and downstream task performance Lepikhin et al. (2021); Du et al. (2022); Dai et al. (2024); Su et al. (2024); Huang et al. (2024b); Yang et al. (2024); Zhao et al. (2024a).

Conventional MoE models, like Lepikhin et al. (2021); Fedus et al. (2022); Du et al. (2022), activate the top \(K\) routed exerts among the total \(N\) experts. Due to the independent training of all experts, they rarely share learned knowledge, and thus routing fluctuations can affect the output substantially, making the performance of such MoE models somehow sensitive to the routing accuracy. To tackle that, Dai et al. (2024); Rajibhandari et al. (2022) suggests using several fixed-activated shared experts to store shared knowledge, in addition to the top\(K\) routed experts. And it has been well-validated to improve MoE model performance. With shared experts, Dai et al. (2024); Yang et al. (2024) further split full-sized experts into more fine-grained experts to enhance representation specialization and gain additional performance improvement, where the \(N\) experts are split into \(mN\) smaller ones, and the top \(mK\) routed ones of them are activated.

The remarkable shared-expert method essentially merges the shared knowledge (i.e., outputs of shared experts) with the specific knowledge (i.e.,

outputs of the routed experts) in an "addition" manner. For instance, with a shared expert FFN\({}_{a}\) and several routed experts FFN\({}_{b}\), FFN\({}_{c}\), and FFN\({}_{d}\), the knowledge sharing among experts can be represented as: FFN\({}_{a}\) + FFN\({}_{b}\), FFN\({}_{a}\) + FFN\({}_{c}\), and FFN\({}_{a}\) + FFN\({}_{d}\). Inspired by collective matrix factorization to learn shared knowledge among data Singh and Gordon (2008), in this paper we propose to represent knowledge sharing among experts in an alternative "multiplication" manner, i.e., FFN\({}_{a}\cdot\text{FFN${}_{b}$}\), FFN\({}_{a}\cdot\text{FFN${}_{c}$}\), and FFN\({}_{a}\cdot\text{FFN${}_{d}$}\). Specifically, by defining two sets of sub-experts {FFN\({}^{1}_{a}\), FFN\({}^{1}_{b}\), ...} and {FFN\({}^{2}_{a}\), FFN\({}^{2}_{b}\), ...}, we derive each expert to be the combination of any two sub-experts from both sets respectively, like FFN\({}_{aa}\) = FFN\({}^{1}_{a}\cdot\text{FFN${}^{2}_{a}$ or FFN${}_{ab}$}=\text{FFN${}^{1}_{a}$}\cdot \text{FFN${}^{2}_{b}$}\). In that sense, each expert share an identical sub-expert with many others. It can also be seen that, all the experts can be derived by the Cartesian product between both sub-expert sets, and thus we term our proposed method as CartesianMoE. Specifically, in our proposed CartesianMoE, we replace the conventional MoE layer as a _Cartesian Product Layer_, which consists of two sequential MoE sub-layers, each denoting a set of sub-experts, as illustrated in Fig. 1. Then the routing process to select routed experts is also divided into the two MoE sub-layers, termed as Cartesian Product routing.

Extensive experiments on building MoE LLMs show that CartesianMoE yields superior performance than previous counterparts, using the same number of model parameters and activated parameters. CartesianMoE also shows better routing robustness. We argue that the superiority of CartesianMoE comes from its more fine-grained knowledge sharing among experts. Specifically, compared to the shared-expert method that requests all routed experts to always share the same global knowledge held by the fixed shared experts, CartesianMoE allows to divide experts into groups with each sharing some group-wise knowledge. In that sense, CartesianMoE is supposed to be also equipped with shared experts, so as to form a "global shared knowledge + group-wise shared knowledge + expert-specific knowledge" system.

Our contributions are summarized as follows:

* Inspired by collective matrix factorization to learn shared knowledge among data, we analyze the feasibility of enabling knowledge sharing among experts in a "multiplication" manner, an alternative to the "addition" manner proposed by the shared-expert method.
* We propose CartesianMoE, which derives experts via the Cartesian Product of two sub-expert sets. CartesianMoE enables group-wise knowledge sharing among experts and helps to build a more complete knowledge sharing system with shared experts equipped.
* We validate the effectiveness of the proposed CartesianMoE with extensive experiments. Experimental results show that it consistently outperforms previous MoE models, and shows better routing robustness.

## 2 Related Work

The concept of MoE models was first introduced by Jacobs et al. (1991). Then, Eigen et al. (2013) extended the MoE model to multiple layers. Later,

Figure 1: Illustration of CartesianMoE. Subgraph a) represents the conventional top-2 routing for full-sized experts, subgraph b) illustrates the top-4 routing for half-sized fine-grained experts, and subgraph c) shows the top-4 Cartesian Product routing (i.e., top-2 routing for each sub-layer) in the proposed CartesianMoE. All subgraphs share the same numbers of model parameters and activated parameters.

 

(Shazeer et al., 2017) extended that idea to Long Short-Term Memory (LSTM) networks Graves (2013), training an LSTM model with up to 137 billion parameters. With the advent of the Transformer architecture Vaswani et al. (2017); Devlin et al. (2019), the Gshard model Lepikhin et al. (2021) applied MoE techniques to Transformers, paving the way for the development of more advanced MoE models like GLaM Du et al. (2022) and Switch Transformer Fedus et al. (2022).

In early works Zoph et al. (2022); Fedus et al. (2022); Du et al. (2022); Lepikhin et al. (2021); Roller et al. (2021); Dai et al. (2022), when extending dense models to MoE models, the MoE layer of the Transformer consists of multiple FFNs that are of the same size as those in the dense models. Recent works Muennighoff et al. (2024); Dai et al. (2024); Yang et al. (2024) show that splitting a fully-sized FFN into several smaller, fine-grained experts facilitates representation specialization. Additionally, shared experts are commonly adopted Dai et al. (2024); Rajbhandari et al. (2022); Su et al. (2024), to enhance knowledge sharing among experts for performance improvement.

The shared-expert method combines shared knowledge (i.e., the outputs of the shared experts) with specialized knowledge (i.e., the outputs of the routed experts) in an "addition" manner. Inspired by collective matrix factorization to learn shared knowledge among data, here we propose an alternative "multiplication" manner to share expert knowledge, which demonstrates superiority over previous MoE methods.

## 3 Background

### Large Language Models

For simplicity, here we focus on the mainstream _generative LLM_ with the Transformer backbone. Given a sequence of tokens \({\bf x}=(x_{1},x_{2},\ldots,x_{T})\), a generative LLM iteratively produces a probability distribution \({\bf p}\) over the vocabulary for each token, conditioning on its preceding tokens. Usually, the cross-entropy loss function is employed to optimize the predicted probability w.r.t the ground-truth token \(x_{t}\). And thus in total, the training loss \(\mathcal{L}_{lm}\) for the generative LLM can be expressed as:

\[\begin{split}&\mathcal{L}_{lm}=-\sum_{t=1}^{T-1}\log({\bf P}_{x_{ t+1},t})\\ \text{s.t.,}&{\bf P}_{\cdot,t}=\text{softmax}(W{ \bf H}_{\cdot,t}^{L})\\ &{\bf H}^{L}=\text{Transformer}(x_{1},x_{2},\ldots,x_{T-1})\end{split} \tag{1}\]

Here, \(L\) is the number of blocks in the Transformer backbone. \({\bf P}_{\cdot,t}\) and \({\bf H}_{\cdot,t}^{L}\) represent the \(t\)-th column of the matrices \({\bf P}\) and \({\bf H}^{L}\), respectively, corresponding to \(x_{t}\). \({\bf H}^{L}=[{\bf h}_{1}^{L},{\bf h}_{2}^{L},\ldots,{\bf h}_{T-1}^{L}]\) denotes the hidden states of the last layer, and \({\bf P}_{x_{t+1},t}\) denotes the predicted probability w.r.t the ground-truth token \(x_{t+1}\) in \({\bf P}_{\cdot,t}\). Here the linear projection layer \(W\) takes \({\bf H}_{\cdot,t}^{L}\) as input to compute the probability distribution \({\bf P}_{\cdot,t}\) across the vocabulary.

In the Transformers backbone, each layer features a multi-head self-attention (MHA) module and a feed-forward network (FFN), with the FFN typically comprising two fully connected layers. Formally,

\[\hat{{\bf h}}_{t}^{l} =\text{MHA}([{\bf h}_{1}^{l-1},{\bf h}_{2}^{l-1},\ldots,{\bf h}_{t }^{l-1}]) \tag{2}\] \[{\bf h}_{t}^{l} =\text{FFN}(\hat{{\bf h}}_{t}^{l})\]

where \(l\) denotes the \(l\)-th block in the Transformer backbone.

### Mixture-of-Experts

MoE methods typically replace the dense model's FFN module with an MoE module composed of multiple FFNs, each being an _expert_. The outputs of these FFNs are combined using a routing function, \({\bf r}(\cdot)\), referred to as the _router_. Formally,

\[{\bf h}_{t}^{l}=\sum_{i=1}^{N}{\bf r}_{i}(\hat{{\bf h}}_{t}^{l})\cdot\text{FFN }_{i}(\hat{{\bf h}}_{t}^{l})\quad\text{s.t. $|{\bf r}(\hat{{\bf h}}_{t}^{ l})|_{0}=K$} \tag{3}\]

where \(N\) is the number of experts in a single MoE module, \(K\) is the number of activated experts, \({\bf r}_{i}\) represents the routing outcome for the \(i\)-th expert, and \(|\cdot|_{0}\) denotes the \(L_{0}\)-norm, i.e., the number of non-zero elements. With \(K\ll N\), only a small subset of experts is activated. And thus increasing the total number of experts in MoE models does not significantly increase computational time.

For fine-grained experts Muennighoff et al. (2024); Dai et al. (2024); Yang et al. (2024), each of the original \(N\) experts is split into \(m\) equal parts, resulting in \(mN\) fine-grained experts in total. In that case, the intermediate size of the fine-grained 

experts is \(\frac{1}{m}\) of the original full-sized experts. To maintain a constant number of activated parameters, the number of activated experts is usually adjusted to \(mK\) as well. Formally,

\[\mathbf{h}_{t}^{l}=\sum_{i=1}^{mN}\mathbf{r}_{i}(\mathbf{\hat{h}}_{ t}^{l})\cdot\text{FFN}_{i}(\mathbf{\hat{h}}_{t}^{l})~{}~{}~{}\text{s.t.}~{}| \mathbf{r}(\mathbf{\hat{h}}_{t}^{l})|_{0}=mK \tag{4}\]

## 4 Method

### Proposed CartesianMoE

As mentioned above, the current MoE models either rarely share learned knowledge among experts or only apply shared experts to share global knowledge. We propose CartesianMoE, as shown in Fig. 1, to facilitate more thorough expert sharing.

As shown in Figure 1(c), the proposed CartesianMoE introduces a _Cartesian Product Layer_, and also employs fine-grained experts in its two MoE sub-layers, denoted as \(A\) and \(B\). Then CartesianMoE combines the fine-grained sub-experts across the two MoE sub-layers to derive real experts. Formally,

\[A\times B =\{(a,b)~{}|~{}a\in A~{}\text{and}~{}b\in B\}\] s.t. \[A =\{\text{FFN}_{1},\ldots,\text{FFN}_{e}\}, \tag{5}\] \[B =\{\text{FFN}_{e+1},\ldots,\text{FFN}_{2e}\}.\]

where \(e\) is the number of sub-experts in each MoE sub-layer. To maximize the diversity of \(A\times B\), we set \(e=mN/2\) experts, with \(mN\) being the number of all fine-grained sub-experts. Specifically, the computation of the _Cartesian Product Layer_ is formulated as follows.

\[\mathbf{\tilde{h}}_{t}^{l} =\sum_{i=1}^{e}\mathbf{r}_{i}^{1}(\mathbf{\hat{h}}_{t}^{l})\cdot \text{FFN}_{i}(\mathbf{\hat{h}}_{t}^{l}) \tag{6}\] \[\mathbf{\bar{h}}_{t}^{l} =\mathbf{\tilde{h}}_{t}^{l}+\mathbf{\hat{h}}_{t}^{l}\] (7) \[\mathbf{h}_{t}^{l} =\sum_{i=e+1}^{2e}\mathbf{r}_{i}^{2}(\mathbf{\bar{h}}_{t}^{l}) \cdot\text{FFN}_{i}(\mathbf{\bar{h}}_{t}^{l})\] (8) s.t. \[|\mathbf{r}^{1}(\mathbf{\hat{h}}_{t}^{l})|_{0}=|\mathbf{r}^{2}( \mathbf{\bar{h}}_{t}^{l})|_{0}=k \tag{9}\]

where \(\mathbf{r}^{1}\) and \(\mathbf{r}^{2}\) represent the _routers_ corresponding to the 1st and 2nd MoE sub-layers of the _Cartesian Product Layer_, respectively. Note that we also add a residual connection between the two MoE sub-layers, to ensure that tokens exceeding the capacity of a sub-expert in the 1st MoE sub-layer can be directly passed to the 2nd MoE sub-layer, i.e., "token droppable" in Fedus et al. (2022) to balance optimization among experts. In order to maintain a consistent total number of activated parameters as previous fine-grained MoE methods, the number of activated experts per MoE sub-layer, i.e., \(k\) in Eq. 9, is also reduced by half, i.e., \(k=mk/2\). We term such a routing process as Cartesian Product routing. Through such a two-layer structural design, the Cartesian Product mechanism is natively implemented, and is supposed to facilitate knowledge sharing among experts.

Then following Transformer Vaswani et al. (2017), we add \(\mathbf{\bar{h}}_{t}^{l}\) to \(\mathbf{h}_{t}^{l}\) to serve as the input for the next block with a skip connection. Formally,

\[\mathbf{h}_{t}^{l}\leftarrow\mathbf{h}_{t}^{l}+\mathbf{\bar{h}}_{t}^{l} \tag{10}\]

### Load Balance Loss

LLMs are typically trained in a distributed manner, which can lead to load imbalances in MoE models Lepikhin et al. (2021); Fedus et al. (2022); Dai et al. (2022), where a minority of experts handle the majority of tokens and meanwhile the majority of experts remain idle. Such imbalances can adversely affect the training efficiency. To address that issue, a load balancing loss is commonly introduced in the training of MoE models. We follow Huang et al. (2024); Fedus et al. (2022) and employ a balanced loss function by summing the routing losses of both MoE sub-layers within a _Cartesian Product Layer_:

\[\mathcal{L}_{bal} =\sum_{i=1}^{e}w_{i}^{1}R_{i}^{1}+\sum_{i=e+1}^{2e}w_{i}^{2}R_{i}^ {2} \tag{11}\] \[\text{s.t.}, w_{i}^{k} =\frac{1}{B}\sum_{j=1}^{B}\mathbb{I}\left\{\text{argmax}(\mathbf{ r}_{..j}^{k})=i\right\}\] \[R_{i}^{k} =\frac{1}{B}\sum_{j=1}^{B}\mathbf{r}_{i,j}^{k}\] \[\forall k\in\{1,2\}\]

where \(B\) represents the number of tokens in a mini-batch, \(k\in\{1,2\}\) denotes the sub-layer index within a _Cartesian Product Layer_, \(\mathbf{r}_{..j}^{k}\) denotes the routing output probability distribution for the \(j\)-th token in the 1st (\(k=1\)) or 2nd (\(k=2\)) MoE sub-layer, and \(\mathbf{r}_{i,j}^{k}\) represents the specific probability value with respect to the \(i\)-th expert in either the 1st (\(k=1\)) or 2nd (\(k=2\)) MoE sub-layer.

Our final loss is a combination of the language model loss and the load-balance loss:

\[\mathcal{L}=\mathcal{L}_{lm}+\alpha\mathcal{L}_{bal}\] (12) 

where \(\alpha\) is a hyperparameter.

### Relations to Flattened Fine-grained Experts

As detailed above, the proposed CartesianMoE leverages two layers of fine-grained sub-experts to build a _Cartesian Product Layer_. Then it would be interesting to see its relations to the flattened fine-grained experts proposed in Dai et al. (2024).

Suppose the number of fine-grained experts/sub-experts is \(2e\) for both methods, same as before. The output of the MoE module in Dai et al. (2024), together with that of the residual connection, is formulated as below:

\[{\bf h}^{l}_{t}=\hat{\mathbf{h}}^{l}_{t}+\sum_{i=1}^{e}{\bf r}^{1}_{ i}(\hat{\mathbf{h}}^{l}_{t})\cdot\text{FFN}_{i}(\hat{\mathbf{h}}^{l}_{t})\] (13) \[~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}

 

shared knowledge + group-wise shared knowledge + expert-specific knowledge" system mentioned before. We compare the proposed CartesianMoE with 6 remarkable baselines Touvron et al. (2023); Fedus et al. (2022); Roller et al. (2021); Huang et al. (2024a); Dai et al. (2024) in our experiments. The respective parameter settings for each compared model are provided in Appendix 9.1. Considering that shared experts are commonly included in MoE models Dai et al. (2024); DeepSeek-AI et al. (2024); Zhao et al. (2024b, 2022); Rajbandari et al. (2022); Su et al. (2024), all compared baselines include shared experts to gain further performance improvement unless otherwise noted.

### Main Results

We first present the model's perplexity (PPL) on the Pile validation set. Then, following Touvron et al. (2023); Brown et al. (2020); Su et al. (2024); Dai et al. (2024), we evaluate the model performance on various downstream benchmarks, including zero-shot tests for HellaSwag Zellers et al. (2019), LAMBADA Paperno et al. (2016), PIQA Bisk et al. (2020), StoryCloze Mostafazadeh et al. (2016), and WinograndeWino (Sakaguchi et al. (2020), in terms of _accuracy_. In addition, following Touvron et al. (2023); Su et al. (2024), we conduct 5-shot evaluations on TriviaQA Joshi et al. (2017), WebQuestions WebQs Berant et al. (2013), and Natural QuestionsNaturalQs Kwiatkowski et al. (2019) using the _exact match_ metric.

#### 5.3.1 Perplexity Results

Table 1 shows the perplexity (PPL) of language modeling on the Pile validation set. With the _same number of activated parameters_, the MoE models (_MoE-Base/MoE-Large_) consistently outperform the dense models (_Base/Large Model_) with significantly reduced PPL. Furthermore, CartesianMoE exhibits a substantial performance improvement over other models, in both _MoE-Base_ and _MoE-Large_ settings. The result presents the superiority of CartesianMoE, which is equipped with complete "global shared knowledge + group-wise shared knowledge + expert-specific knowledge".

Note that _Fine-grained Routing_ with flattened fine-grained exerts exhibits inconsistent improvements across different model sizes. In the _MoE-Base_ setting, it significantly outperforms _SMoE-Share_, but in the _MoE-Large_ setting, it performs slightly worse than _SMoE-Share_. In contrast, CartesianMoE demonstrates consistent performance improvements across different settings, highlighting its consistent superiority.

#### 5.3.2 Benchmark Results

As shown in Table 2, we present the model's performance on downstream tasks. We can also observe that the MoE models achieve performance improvements over the dense counterpart in those benchmark tasks. More importantly, the proposed CartesianMoE stands out among all MoE models, in both _MoE-Base_ or _MoE-Large_ settings. Specifi

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline Model & Configuration & SE & FOE & Params & Activated Params & Pile PPL (\(i\)) \\ \hline Base Model & d=768, D=3072 & N/A & N/A & 162M & 162M & 8.55 \\ Large Model & d=1024, D=d096 & N/A & N/A & 468B & 468M & 6.95 \\ \hline _MoE-Base_ & & & & & & \\ \hline SMoE-Share & d=768, D=3072, top\(K\)=2 & True & False & 842M & 247M & 7.37 \\ SMoE-Top3 & d=768, D=3264, top\(K\)=3 & False & False & 842M & 258M & 7.40 \\ Hash Layer & d=768, D=3072, top\(K\)=2 & True & False & 842M & 247M & 7.47 \\ Fine-grained Routing & d=768, D=1536, top\(K\)=4, & True & True & 842M & 247M & 7.33 \\ TopP Routing & d=768, D=3072, top\(F\)=0.4 & True & False & 842M & 247M & 7.41 \\ CartesianMoE & d=768, D=1526, top\(K\)=(2+2) & True & True & 842M & 247M & **7.19** \\ \hline _MoE-Large_ & & & & & & \\ \hline SMoE-Share & d=1024, D=d096, top\(K\)=2 & True & False & 2.88B & 770M & 6.13 \\ SMoE-Top3 & d=1024, D=4352, top\(K\)=3 & False & False & 2.88B & 808M & 6.18 \\ Hash Layer & d=1024, D=4096, top\(K\)=2 & True & False & 2.88B & 770M & 6.28 \\ Fine-grained Routing & d=1024, D=2048, top\(K\)=4 & True & True & 2.88B & 770M & 6.16 \\ TopP Routing & d=1024, D=4096, top\(F\)=0.4 & True & False & 2.88B & 770M & 6.14 \\ CartesianMoE & d=1024, D=2048, top\(K\)=(2+2) & True & True & 2.88B & 770M & **6.08** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Perplexity (PPL) results of language modeling. The best score is marked in **bold**. SE indicates whether to use shared experts, FGE indicates whether to use fine-grained experts, d represents the hidden state dimensionality, D represents the intermediate size of each FFN, and top\(K\) refers to the number of experts activated for each token. For CartesianMoE, top\(K\)=(2+2) means that each of the two sub-layers activates two sub-experts. For TopP Routing, top\(P\) is the threshold that controls how many experts should be activated to reach it.

 

cally, compared to other MoE models, CartesianMoE yields the best performance for 7 of 8 benchmarks in the _MoE-Base_ setting and 6 of 8 benchmarks in the _MoE-Large_ setting.

Particularly, against _Fine-grained Routing_ with flattened fine-grained experts, CartesianMoE excels in 7 of 8 benchmarks with the _MoE-Base_ setting and also excels in all benchmarks with the _MoE-Large_ setting. That further verifies our analysis above that CartesianMoE can enjoy deeper representations of input and more flexible routing than the flattened counterpart. It also demonstrates the effectiveness of introducing the _Cartesian Product Layer_ for group-wise knowledge sharing.

## 6 Analyses

### Impact of Fixed-Activated Shared Expert

Under the _MoE-Large_ setting, we remove the fixed-activated shared experts from CartesianMoE to investigate its impact on the model performance.

As shown in Table 3, after removing the fixed-activated shared experts (i.e., _w/o Shared Expert_), CartesianMoE yields slightly better performance than _Fine-grained Routing_ equipped with shared experts. The result well reflects the effectiveness of group-wise knowledge sharing among experts proposed by CartesianMoE, which is equally important as global knowledge sharing introduced by shared experts. Moreover, when CartesianMoE is equipped with shared experts as by default, its performance is substantially enhanced, which further demonstrates the effectiveness of forming a "global shared knowledge + group-wise shared knowledge + expert-specific knowledge" system, as proposed by CartesianMoE.

### Analysis on Expert Routing Robustness

To analyze the expert routing robustness of different MoE models, we disable the top-\(1\) routed expert and then evaluate the PPL variance brought by such a routing change on the Pile validation set. Specifically, for each token, we mask the expert with the highest routing probability and then select the top \(K\) experts from the remaining ones. Since each _Cartesian Product Layer_ in CartesianMoE has two MoE sub-layers, we randomly select one sub-layer each time and mask the corresponding top-\(1\) expert.

As shown in Table 4, even with the top-\(1\) routed expert disabled, CartesianMoE still yields the lowest PPL, and enjoys a much smaller PPL variance, compared to other MoE methods. That well indicates the superior routing robustness of CartesianMoE. And we attribute it to the more thorough knowledge sharing among experts in CartesianMoE, which includes both global and group-wise knowledge sharing.

### Training with More Tokens

The previous experiments are conducted using 100B tokens. To investigate whether the superiority of the proposed CartesianMoE can be maintained after training with more tokens, here we continue to train CartesianMoE and the most competitive baseline _Fine-grained Routing_Dai et al. (2024)

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline Model & Hellaswag & LAMBADA & PIQA & StoryLoe & Wino & TriviaQA & WebQs & NaturalQs \\ \hline Base Model & 32.54 & 39.22 & 62.52 & 58.52 & 50.75 & 2.99 & 2.22 & 0.72 \\ Large Model & 40.73 & 52.55 & 67.62 & 63.55 & 53.75 & 7.44 & 4.97 & 1.78 \\ \hline \multicolumn{8}{l}{_MoE-Base_} \\ \hline SMoE-Share & 37.87 & 47.35 & **66.76** & 61.52 & 51.14 & 6.11 & **4.18** & 1.39 \\ SMoE-Top3 & 37.43 & 46.42 & 66.38 & 61.25 & 51.14 & 5.82 & 4.04 & 1.16 \\ Hash Layer & 33.39 & 43.08 & 61.59 & 57.88 & 50.20 & 2.85 & 1.57 & 0.61 \\ Fine-grained Routing & 37.50 & 45.10 & 66.49 & 61.20 & 51.22 & 6.12 & 3.79 & 1.30 \\ TopP Routing & 35.10 & 45.02 & 63.76 & 58.10 & 51.46 & 4.12 & 2.41 & 0.55 \\ CartesianMoE & **38.17** & **48.17** & 65.83 & **62.53** & **51.62** & **7.23** & **4.18** & **1.58** \\ \hline \multicolumn{8}{l}{_Mee-Large_} \\ \hline SMoE-Share & 48.25 & 58.80 & **79.51** & 66.49 & 55.56 & 14.91 & 9.06 & 3.99 \\ SMoE-Top3 & 47.59 & 57.60 & 70.40 & 65.95 & 53.75 & 14.46 & 7.92 & 4.24 \\ Hash Layer & 42.80 & 56.45 & 66.32 & 64.08 & 52.33 & 6.66 & 4.33 & 1.86 \\ Fine-grained Routing & 48.10 & 58.32 & 69.86 & 64.89 & 55.80 & 14.40 & 8.42 & 3.77 \\ TopP Routing & 44.31 & **59.85** & 67.46 & 63.92 & 54.54 & 11.02 & 6.15 & 2.41 \\ CartesianMoE & **49.14** & 59.17 & 70.24 & **67.02** & **56.51** & **15.09** & **9.15** & **4.85** \\ \hline \end{tabular}
\end{table}
Table 2: Performances of language models on downstream tasks. The best score is marked in **bold**.

\begin{table}
\begin{tabular}{l l} \hline \hline Model & PPL \\ \hline SMoE-Top3 & 6.18 \\ Fine-grained Routing & 6.16 \\ CartesianMoE & **6.08** \\ _w/o Shared Expert_ & 6.15 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Impact of the fixed-activated shared expert.

 

until 400B tokens, and compare their performance in the _MoE-Large_ setting.

As shown in the left part of Table 5, on the Pile validation set, the PRL of _Fine-grained Routing_ converged to 5.78, while that of CartesianMoE further decreases to 5.69. And on downstream tasks, CartesianMoE also outperforms _Fine-grained Routing_ in 6 out of 8 benchmarks. The full changing curves for PRL and benchmark performance of both MoE models are provided in Figure 2 and Figure 4 in the Appendix, respectively. It can be seen that even trained on more tokens, CartesianMoE consistently maintains superior performance, well demonstrating its effectiveness.

### Scaling Up the Model Size

To investigate the performance of the proposed CartesianMoE with a larger model size, we follow the setting of Muennighoff et al. (2024) to train CartesianMoE and the most competitive baseline _Fine-grained Routing_, with 7.25B parameters and 1.61B activated parameters. The specific parameter settings are provided in Section 9.2.

As shown in the right part of Table 5, on the Pile validation set, the PPL of _Fine-grained Routing_ converged to 4.99, while that of CartesianMoE decreases to 4.92. And on all downstream tasks, CartesianMoE outperforms _Fine-grained Routing_. The full changing curves for PPL and downstream tasks of both MoE models are also provided in Figure 3 and Figure 5 in the Appendix. The experimental results further demonstrate the superiority and scalability of CartesianMoE.

### Training in Different Expert Granularities

The experiments above use half-sized FFNs as fine-grained experts in CartesianMoE. It would be interesting to see whether CartesianMoE can maintain its superiority with more finer-grained experts. Suppose we have \(N\) full-sized experts. As mentioned before, to keep the numbers of total parameters and activated parameters unchanged, we equally split each full-sized expert into \(m\) fine-grained experts via splitting its FFN intermediate size into \(m\) equal parts, with \(m\) being the splitting factor, and the number of activated fine-grained experts would also be scaled up by \(m\). It can be seen \(m=1\) for full-sized experts, and experiments above use \(m=2\) for CartesianMoE. Here we further conduct experiments with \(m=4\), for both CartesianMoE and the most competitive baseline _Fine-grained Routing_, to further validate CartesianMoE.

As is seen in Table 6, in both \(m=2\) and \(m=4\) settings, CartesianMoE consistently outperforms _Fine-grained Routing_ in terms of PPL on the Pile validation set, which further demonstrates its superiority and robustness across different expert granularities. We also find that increasing \(m\) may not lead to better performance, as over-fine-grained experts can encounter underfitting.

## 7 Conclusions

Inspired by collective matrix factorization to capture shared knowledge within data, we introduce CartesianMoE, a "multiplication"-manner knowledge sharing method among experts in MoE models. CartesianMoE categorizes fine-grained subexperts into two distinct sets, and uses their Cartesian product to build experts that facilitate group

\begin{table}
\begin{tabular}{l l c l r} \hline \hline D & \multicolumn{1}{c}{\(K\)} & \multicolumn{1}{c}{\(m\)} & Mode & PPL \\ \hline
3072 & 2 & 1 & Full-sized & 7.37 \\
1536 & 4 & 2 & Fine-grained Routing & 7.33 \\
768 & 8 & 4 & Fine-grained Routing & 7.34 \\
1536 & 24:2 & 2 & CartesianMoE & **7.19** \\
768 & 4+4 & 4 & CartesianMoE & 7.26 \\ \hline \hline \end{tabular}
\end{table}
Table 6: PPL on the Pile validation set, with different expert granularity. \(D\) indicates the FFN intermediate size, \(K\) denotes the number of activated experts, and \(m\) denotes the splitting factor.

\begin{table}
\begin{tabular}{l c c} \hline \hline  & PPL & PPL(disable top-1) \\ \hline SMoE-Share & 6.13 & 7568.85 \\ SMoE-Top3 & 6.18 & 847.46 \\ Fine-grained Routing & 6.16 & 3095.44 \\ TopP Routing & 6.14 & 84.42 \\ CartesianMoE & **6.08** & **27.72** \\ \hline \end{tabular}
\end{table}
Table 4: PPL on the Pile validation set, with the top-1 routed expert disabled. The baseline _Hash Layer_ is excluded here, as the experts for each input in it are fixedly assigned.

\begin{table}
\begin{tabular}{l c c|r c} \hline \hline  & \multicolumn{2}{c|}{MoE-Large} & \multicolumn{2}{c}{7.25B Parans} \\  & CartesianMoE & Fine-grain & CartesianMoE & Fine-grain \\ \hline Hellaswag & **54.28** & 52.10 & **56.96** & 55.64 \\ LAMBADA & **63.05** & 61.55 & **63.50** & 62.83 \\ PIQA & **71.71** & 71.21 & **73.99** & 71.87 \\ StoryCloze & 67.93 & **67.97** & **69.37** & 69.11 \\ Wino & **56.74** & 55.88 & **58.17** & 58.16 \\ TriviaQA & 20.03 & **21.40** & **24.87** & 22.82 \\ WebQx & **10.58** & 9.01 & **11.96** & 11.32 \\ NaturalQs & **5.87** & 5.54 & **6.57** & 5.54 \\ \hline PPL(↓) & **5.69** & 5.78 & **4.92** & 4.99 \\ \hline \hline \end{tabular}
\end{table}
Table 5: The performance comparison after training 400B tokens with different model sizes, with _Fine-grain_ being short for the baseline _Fine-grained Routing_. The best score in each setting is marked in **bold**.

 

wise knowledge sharing. Equipped with shared experts as previous works, CartesianMoE builds a more thorough knowledge sharing system among experts, i.e., "global shared knowledge + group-wise shared knowledge + expert-specific knowledge". Extensive experiments well demonstrate that CartesianMoE outperforms previous MoE models across various settings, in terms of language modeling perplexity and downstream task performance. It also presents much better routing robustness due to enhanced knowledge sharing.

## 8 Limitations

We only perform Cartesian product computations between two MoE sub-layers. In fact, the Cartesian product can be extended to more than two sub-layers. However, Dubey et al. (2024) has shown that increasing the number of model sub-layers requires a corresponding increase in hidden state dimensionality to ensure training effectiveness. And thus we leave the exploration of extending to more MoE sub-layers for future work.

## References

* Anil et al. (2015) Rohan Anil, Sebastian Borgeaud, Yonghui Wu, JeanBaptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Milican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angelik Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Eric Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anais White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. 2023. Gemini: A family of highly capable multimodal models. CoRRabs/2312.11805.
* Berant et al. (2013) JonathanBerant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1533-1544. ACL.
* Biderman et al. (2020) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397-2430. PMLR.
* Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432-7439. AAAI Press.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems _33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Child et al. (2019) Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. CoRR, abs/1904.10509.
* Chowdhery et al. (2019)Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Mayenze, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazezer, Vinodokumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipao De, Henry Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Lu, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pilali, Marie Pellart, Aitor Lewkowycz, Eric Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuexi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Field. 2022. Palm: Scaling language modeling with pathways. CoRR, abs/2204.02311 

Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wentfeng Liang. 2024. DeepseeKmo: Towards ultimate expert specialization in mixture-of-experts language models. CoRRabs/2401.06066.
* Dai et al. (2022) Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Stable-moe: Stable routing strategy for mixture of experts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085-7095. Association for Computational Linguistics.
* DeepSeek et al. (2022) DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongji Ji, Erhang Li, Fangyun Lin, Full Luo, Guangbo Hao, Guanting Chen, Guowei Li, Hao Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Ley Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaqium Wang, Mingchuan Zhang, Minghua Tang, Mingming Li, Ning Tian, Papan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shanghai Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shihoring Ma, Shiyu Wang, Shuang Zhou, Shuping Yu, Shunfeng Zhou, Size Zheng, Tao Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotaohei, and Xiaowen Sun. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. CoRRabs/2405.04434.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171-4186. Association for Computational Linguistics.
* Du et al. (2022) Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P. Bosma, Zongewiet Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. 2022. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547-5569. PMLR.
* Dubey et al. (2021) Abhimanyu Dubey, Abhinav Jaahri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhl Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Harshorn, Aoobo Yang, Archi Mitra, Archine Sarvankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austeen Gregerson, Ava Spataru, Baptiste Roziere, Bethabyon Birno, Binth Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Trouret, Chunyang Wu, Corinine Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egot Lakomkin, Ehab AlBadawly, Eliana Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Malion, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannak Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Klourann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Janna Vranes, Jason Park, Jay Mahadeckar, Jeeth Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstrun, Joshua Saxe, Junteng Jia, Kalyan Vasudeau Ahvala, Kartikeyas Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. 2024. The llama 3 herd of models. CoRRabs/2407.21783.
* Eigen et al. (2013) David Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. 2013. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314.
* Fedus et al. (2022) William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23:120:1-120:39.
* Gao et al. (2021) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The pile: An 800gb dataset of diverse text for language modeling. CoRRabs/2101.00027.
* Graves (2013) Alex Graves. 2013. Generating sequences with recurrent neural networks. CoRRabs/1308.0850.
* Hoffmann et al. (2017) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan 

Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models. CoRRabs/2203.15556.
* Huang et al. (2024) Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. 2024a. Harder tasks need more experts: Dynamic routing in moe models. CoRRabs/2403.07652.
* Huang et al. (2024b) Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. 2024b. Harder task needs more experts: Dynamic routing in MoE models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12883-12895, Bangkok, Thailand. Association for Computational Linguistics.
* Jacobs et al. (1991) Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. 1991. Adaptive mixtures of local experts. Neural Comput.3(1):79-87.
* August 4, Volume 1: Long Papers, pp. 1601-1611. Association for Computational Linguistics.
* Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. CoRRabs/2001.08361.
* Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452-466.
* Lepikhin et al. (2021) Dmitry Lepikhin, HyoukJoog Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
* Lewis et al. (2021) Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021. BASE layers: Simplifying training of large, sparse models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 6265-6274. PMLR.
* Lian et al. (2024) Haoran Lian, Yizhe Xiong, Jianwei Niu, Shasha Mo, Zhenpeng Su, Zijia Lin, Peng Liu, Hui Chen, and Guiguang Ding. 2024. Scaffold-bpc: Enhancing byte pair encoding with simple and effective scaffold token removal. arXiv preprint arXiv:2404.17808.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
* Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucvy Vanderwende, Pushmeet Kohli, and James F. Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. CoRRabs/1604.01696.
* Muennighghoff et al. (2024) Niklas Muennighghoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Wejia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. 2024. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060.
* Paperno et al. (2016) Denis Paperno, German Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Austin 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog.1(8):9.
* Rae et al. (2019) Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Eisen, Sidhdant M. Jayakumar, Elena Buchauskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganianiani, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kumcoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliik Lazaridou, Arthur Mensch, Jean-Baptise Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sotitauux, Mantus Rajarskas, Toby Pollen, Zhitao Gong, Daniel Toyama, Cypien de Masson d'Autume, Yuja Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A.

 

Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Dennis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2021. Scaling language models: Methods, analysis & insights from training gopher. CoRRabs/2112.11446.
* Rajbhandari et al. (2022) Samyam Rajbhandari, Conglong Li, Zhewei Yao, Mingjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. 2022. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332-18346. PMLR.
* Roller et al. (2021) Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. 2021. Hash layers for large sparse models. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 17555-17566.
* Sakaguchi et al. (2020) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732-8740. AAAI Press.
* Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
* Singh and Gordon (2008) Ajit P. Singh and Geoffrey J. Gordon. 2008. Relational learning via collective matrix factorization. In _Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, KDD '08, page 650-658, New York, NY, USA. Association for Computing Machinery.
* Su et al. (2024a) Zhenpeng Su, Zijia Lin, Xue Bai, Xing Wu, Yizhe Xiong, Haoran Lian, Guangyuan Ma, Hui Chen, Guiguang Ding, Wei Zhou, et al. 2024a. Maskmoe: Boosting token-level learning via routing mask in mixture-of-experts. arXiv preprint arXiv:2407.09816.
* Su et al. (2024b) Zhenpeng Su, Zijia Lin, Baixue Baixue, Hui Chen, Songlin Hu, Wei Zhou, Guiguang Ding, and Xing W. 2024b. MiLe loss: a new loss for mitigating the bias of learning difficulties in generative language models. In _Findings of the Association for Computational Linguistics: NAACL 2024_, pages 250-262, Mexico City, Mexico. Association for Computational Linguistics.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRRabs/2302.13971.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* 16, 2023.
* Xiong et al. (2024) Yizhe Xiong, Xiansheng Chen, Xin Ye, Hui Chen, Zija Lin, Haoran Lian, Jianwei Niu, and Guiguang Ding. 2024. Temporal scaling law for large language models. arXiv preprint arXiv:2404.17785.
* Yang et al. (2024) Yuanhang Yang, Shiyi Qi, Wenchao Gu, Chaozheng Wang, Cuiyun Gao, and Zenglin Xu. 2024. XMoE: Sparse models with fine-grained and adaptive expert selection. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 11664-11674, Bangok, Thailand and virtual meeting. Association for Computational Linguistics.
* Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_, pages 4791-4800. Association for Computational Linguistics.
* Zhao et al. (2024) Hao Zhao, Zihan Qiu, Huijia Wu, Zili Wang, Zhaofeng He, and Jie Fu. 2024a. HyperMoE: Towards better mixture of experts via transferring among experts. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 10605-10618, Bangokok, Thailand. Association for Computational Linguistics.
* Zhao et al. (2024b) Hao Zhao, Zihan Qiu, Huijia Wu, Zili Wang, Zhaofeng He, and Jie Fu. 2024b. Hypermoe: Towards better 

mixture of experts via transferring among experts. _arXiv preprint arXiv:2402.12656_.
* Zoph et al. (2022) Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. St-moe: Designing stable and transferable sparse expert models. _arXiv preprint arXiv:2202.08906_.

## 9 Appendix

### Compared Models

The model settings we compare are as follows. For MoE models, following Lepikhin et al. (2021); Yang et al. (2024), unless otherwise specified, each layer of the MoE has 16 experts, with the top-2 experts activated.

* **Dense** represents a standard Transformer language model.
* **SMoE-Share** denotes an MoE model similar to Lepikhin et al. (2021); Fedus et al. (2022), without fine-grained splitting of experts. Additionally, each MoE layer in _SMoE-Share_ includes 1 shared expert.
* **SMoE-Top3** denotes an MoE model with top-3 routing and no shared experts. To maintain the total number of parameters after removing the shared expert, _SMoE-Top3_ increases the intermediate dimensionality of each expert's FFN, which results in slightly more activated parameters compared to other models and acts as a stronger baseline for comparison.
* **Hash Layer**Roller et al. (2021) signifies a method without _router_ parameters, where each token is fixedly assigned to two experts using a random hash. The model also has a shared expert for fair comparison with the other models.
* **Fine-grained Routing** denotes an MoE model that employs a _Fine-grained Routing_ strategy Dai et al. (2024). For both routing and shared experts, we split the fully-sized FFNs into 2 half-sized FFNs, resulting in 32 fine-grained experts per MoE layer. To maintain the total number of activated parameters consistent, the _Fine-grained Routing_ strategy uses top-\(4\) routing and includes 2 fixed-activated shared experts for each MoE layer.
* **TopP Routing**Huang et al. (2024a) is a routing strategy that dynamically adjusts the number of activated experts based on the difficulty of tokens. It selects the top experts until their cumulative confidence exceeds the pre-set confidence threshold top\(P\). Following Huang et al. (2024a), we set top\(P\) as \(0.4\). Similarly, each MoE layer includes one shared expert to enable fair comparison with other models 



### Training Configuration

\begin{table}
\begin{tabular}{l c c} \hline  & **CartesianMoE** & _Fine-grained Routing_ \\ \hline
**Hidden Size** & 2.048 & 2.048 \\ Activation & SwiGLU & SwiGLU \\
**Intermediate Size** & 2.048 & 2.048 \\
**Attn heads** & 16 & 16 \\
**Num layers** & 16 & 16 \\ Layer norm type & RMSNorm & RMSNorm \\
**Pos emb.** & RoPE & RoPE \\
**MoE layers** & Every & Every \\
**Shared Experts** & True & True \\
**Fine-grained Experts** & True & True \\ Max seq len & 4096 & 4096 \\
**MoE sub-layers** & 2 & N/A \\
**\# Experts** & 32 & 32 \\
**\# Activated Expert** & TopK=(2+2) & TopK=4 \\
**\# Params** & 7.25B & 7.25B \\
**\# Activated Params** & 1.61B & 1.61B \\ \end{tabular}
\end{table}
Table 7: Configurations of CartesianMoE and _Fine-grained Routing_ with 7.25B parameters.

Figure 3: PPL changing curves during language model training with 400B tokens for CartesianMoE and _Fine-grained Routing_ with 7.25B parameters.



Figure 4: Changing curves of downstream task performance during language model training with 400B tokens for CartesianMoE and _Fine-grained Routing_ in _MoE-Large_ setting.

 

Figure 5: Changing curves of downstream task performance during language model training with 400B tokens for CartesianMoE and _Fine-grained Routing_ with 7.25B parameters.