Autonomy-of-Experts Models
Authors: Ang Lv
Ruobing Xie
Yining Qian
Songhao Wu
Xingwu Sun
Zhanhui Kang
Di Wang
Rui Yan
Venue: ICML 2025 poster (2025)
Keywords: Mixture-of-Experts
Autonomy-of-Experts
Language models
Primary area: deep_learning->large_language_models

Abstract: Mixture-of-Experts (MoE) models mostly use a router to assign tokens to specific expert modules, activating only partial parameters and often outperforming dense models. We argue that the separation between the router's decision-making and the experts' execution is a critical yet overlooked issue, leading to suboptimal expert selection and learning. To address this, we propose Autonomy-of-Expert (AoE), a novel MoE paradigm in which experts autonomously select themselves to process inputs. AoE is based on the insight that an expert is aware of its own capacity to effectively process a token, an awareness reflected in the scale of its internal activations. In AoE, routers are removed; instead, experts pre-compute internal activations for inputs and are ranked based on their activation norms. Only the top-ranking experts proceed with the forward pass, while the others abort. The overhead of pre-computing activations is reduced through a low-rank weight factorization. This self-evaluating-then-partner-comparing approach ensures improved expert selection and effective learning. We pre-train language models having 700M up to 4B parameters, demonstrating that AoE outperforms traditional MoE models with comparable efficiency.

Autonomy-of-Experts Models
Ang Lv!
Ruobing Xie?
Yining Qian Songhao Wu
Xingwu Sun 24
Zhanhui Kang²
Di Wang
Rui Yan 156
Abstract
Mixture-of-Expert (MoE) models mostly use a router to assign tokens to specific expert mod- ules, activating only partial parameters and often outperforming dense models. We argue that the separation between the router's decision-making and the experts' execution is critical yet over- looked issue, leading to suboptimal expert selec- tion and ineffective learning. To address this, we propose Autonomy-of-Expers (AoE), a novel MoE paradigm in which experts autonomously select themselves to process imputs. AoE based the insight that an expert is aware of its own capacity to effectively process token, aware- ness reflected in the scale of its internal acti- vations. In AoE, routers are removed; instead, experts pre-compute internal activations for in- puts and are ranked based on their activation norms. Only the top-ranking experts proceed with the forward pass, while the others abort. The overhead of pre-computing activations is duced through a low-rank weight factorization. This self- cellu-eallutiig-ttee thennngttheentge teeetthee---ttree wartner-comparty ap- proach ensures improved expert selection and ef- fective learning. We pre-train language models having 700M up to 4B parameters, demonstrat- ing that AoE outperforms traditional MoE models with comparable efficiency. The code is avail- able at https: http//gttttt/at htt.uu..ccmtttttttttteeee Automomy-of f of-xeetsss Experts
W
Expert 1
Expert 2
Expert
SILU
Softmax
W
Router
Inpul
(a) Mixture-of- of-ppets
Export
Expert
Expert
W
SILU
Norm.
Softmax(
Input
(b) Automomy--f-Fxperts
Figure Comparison between traditional MoE and AoE. Arrows indicate data flow. while shadowed modules represent unused pa- rameters or variables.
(a) Traditional MoE models use a router to assign tokens specific experts. This separation between the router's 's maaiiinn-making and experts' execution leads to suh- optimal expert selection and ineffective learning. (b) In AoE model, experts operate autonomously. They are ranked based on their internal activation norms, and only the activvtiddedd experts continue processing, while the others are terminated. The AoE expert architecture is modified to maintain efficiency.
"Gaoling School of Artificial Intelligence, Renmin University China Large Language Model Department, Tencent 'Southcast University, China "University of Macau 'ngineering Research Center of Nexi-Generation Intelligent Search and Recommenda- tion, MoE "School of Artificial Intelligence, Wuhan University. Correspondence Ruobing Xie <xrbsnowing 163.com>, Rui Yan <ruiyan ruc.edu.en>..
1. Introduction
Large language models (LLM) built on Mixture-of--xxprrts techniques (MoE, Shazeer et al., 2017: Lepikhin et al., 2021; Fedus et al., 2022) have gained increasing research and in- dustrial attention (Jiang et al., 2024; Dai al., 2024; Team,
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

-utonosy-of-Experts Models
2024; Sun et al., 2024). The core idea of MoE in LLMs involves dividing large feed-forward network (FFN) into smaller FFNs, known as experts, and activating different ex- perts' parameters for different inputs. The decision on which experts process which inputs are made by a router, typically an MLP-based classifier. Compared to dense models, MoE models are more efficient due to their sparse activation, and their ability to flexibly combine expert knowledge enhances downstream performance.
Algorithm 1 A working pipeline of an MoE layer
imput A hidden state x E Rddunt number of experts n. output The layer output h E Rddaa,, initialized as zeros. I: R(x)
// P E R / R
2: I = argtopK(p))
P = so Stftmaxx (p)
" VPERA R
for i to do
if I then
6:
h -- p[i] Ex)
A critical issue in MoE is often overlooked: the separa- tion between the router's 's pecision-making and the experts' execution. The router cannot directly assess the experts' abilities, making its selection of an expert essentially a pre- diction without available labels. If the router makes an incorrect prediction, the chosen expert may struggle to pro- cess the tokens effectively, leading to increased training loss. To reduce the loss, the expert might adapt its parameters to handle these tokens, potentially conflicting with its original expertise. Alternatively, the router must learn to make bet- ter decisions through trial and error, as it lacks awareness of which experts are best suited for specific tasks, thereby wasting many training steps.
7: end if
8:
end for
the model to process inputs effectively by activating only the experts with the necessary capabilities, thereby improving efficiency. Some studies (Chen et al., 2024; Lin et al., 2024) dense MoE do not reduce the parameter activation ratio, which is not the primary concern of this paper. In this paper, when we refer to MoF, we mean sparse MoE.
MoE-based LLMS (Jiang et al., 2024; Dai et al., 2024; Team, 2024; Lenz et al., 2025; Sun et al., 2024; Abdin et al., 2024) typically follow the FFN design in the Llama models (Tou- vron et al., 2023) as an expert module. The i-th expert within specific layer can be formulated as:
To address these challenges, we propose novel MoE MarliimmAAttoomy--ooii
(AoE). AoE allows ex- perts to decide whether to process inputs themselves. This design is based on the observation that experts are aware of their ability to handle inputs, an awareness reflected in the scale of their internal activations. Building on this in- sight, we enable all experts an AoE layer to process every token and cache their internal activations. For each token, experts are ranked by their internal activation norms, with only the top-ranked experts continuing to process the token using the cache, while the others terminate the process. The additional overhead from caching and computations of un- used experts is mitigated by factorizing the experts' weights, which compresses the inputs into low-dimensional vectors for efficient caching. Due the autonomy of the experts, the router is eliminated. Figure I presents comparative overview of traditional MoF and AoE models.
Fx(x) = (SIIU(XW,) o (xW))) W
(1)
where x Russt is the input hidden state; W W Remanta x xds, and W E R.d Xdhaas are the expert weights. This paper focuses on this classical FFN formulation.
g' p
A router (or gate) R determines which expert processes which hidden state. Many studies have proposed var- ious routing strategies, such as token choosing top ex- perts (Shazeer et al., 2017: Lepikhin et al., 2021), expert choosing top tokens (Zhou et al., 2022; 2023), dynamic ex- pert calls (Raposo et al., 2024; Gong et al., 2024), and refining expert selection by solving mathematical prob- lems (Lewis et al., 2021; Clark et al., 2022), among others. Without loss of generality, our discussion focuses on token choosing the Top- K experts (Shazeer et al., 2017; Lepikhin et al., 2021), but our experiments consider various strategies. Algorithm 1 presents a working pipeline of an MoE layer with a total of n experts. The "[i]" notation the algorithm follows Python syntax, indicating the selection of the i-th element in vector matrix.
We pre-train AoE language models with up to billion parameters, and they outperform traditional MoE models on downstream tasks with comparable efficiency. We provide a comprehensive analysis of AoE to highlight its advantages. These advantages include improved expert selection, more specialized experts, and more effective training, all of which contribute to better downstream performance.
2. Background: Mixture-of-Expert (MoE)
A challenge faced by MoE is the imbalanced expert load. MoE routers tend to disproportionately favor specific ex- perts, resulting in suboptimal parameter utilization. Fedus et al. (2022) incorporate a load- wrad-balancing loss, controlled by hyperparameter weight, O to ensure that each expert
We focus on sparse MoE models, treating each feed-forward network (FFN) module as an expert. Each FFN, expert, is expected to possess diverse and distinct abilitics, enabling
2

---

Autonomy-of-Experts Models
Table 1. We remove routers from pre-trained MoF-ILLMs and select experts during inference based on the internal activation norms of specific nodes in the computational graph. The accuracy on two challenging tasks is reported, along with the time cost (in minutes) for 8xA800-800 800 GPUs, which is given in parentheses. Without parameter updates, we can largely preserve accuracy under certain nodes, but this rudimentary approach requires significant improvements in efficiency.
Node for Norm Calculation
MMLU (5-shot)
ARC-C C (5-shot)
Mixtral x 7B
Phi-3.5-MoE-
Mixtral x 7B
Phi-3.5-MoE-inn
XW xWr X 9) SiLU(xW,, b) X P Experts' Final Outputs
64.23 (42.70)
29.43 (33.05)
50.43 (4.40)
28.84 (3.47)
62.06 (42.73)
34.60 (33.05)
53.41 (4.40)
40,36 (3.47)
61.71 (43.88)
38.03 (34.32)
58.79 (4.51)
47.53 (3.60)
66.64 (75.53)
27.89 (52.60)
58.79 (6.27)
35.32 (5,42)
66.66 (76.15)
29.69 (69.20)
58.62 (7.42)
36.35 (7.07)
Performance W. Router
70.35 (24.30)
78.20 (14.53)
62.12 (2.50)
67.41 (1.60)
receives a similar load for a batch B with T tokens:
3.1. An Insight: Experts "Know" What They Know
We present the experiment that motivated the development of AoE models.
L = O'axx
T
Σ f. P. where
Geva et al. (2021) interpret FFN layers as key-value mem- ory networks, where inputs are projected into "key" vec- tor (e.g., (SiLU(xWg) @ (xWp))). The "key" vector retrieves knowledge or abilities stored in the parameters through a key-value matching mechanism (e.g., multiplying by W.). If the experts can effectively handle the input, the "key" should be highly activated, allowing for effective retrieval. Note that this example is purely analogical; there are no defined rules to determine which internal activations behave more like the "key" and which behave more like the "value," as models are not trained with constraints that would regularize these roles.
f,
Lifie argtopk
T
XEB
P. Softmax (eettmax((())))) [x]
XEB
Several variants of this auxiliary loss are proposed (Zuo et al., 2022; Wang et al., 2024a;b; Huang et al., 2024), shar- ing the same Load-balancing goal. Therefore, our discussion focuses on the balancing loss presented above.
Several studies (Roller et al., 2021; Gururangan et al., 2022: Ren et al., 2023; Fan et al., 2021) classify tokens based on prior knowledge-suu as domain, language, or hash mapping-amn assign them to fixed experts. While they do not use explicit routers, they differ significantly from AoE in many respects. Most importantly, their expert selection is not determined by the experts themselves, leaving the sepa- ration between decision maaiing and execution unaddressed. Pham et al. (2024) use the norm of expert final outputs the label for router logits. This method shares the concept with ours, where the activation norm represents expertise: however, it incurs dense activation across all experts and does not address the separation issue we highlighted.
Inspired by (Geva et al., 2021), we conducted preliminary experiments to explore whether experts in pre-trained MoE- L.MSs "know" their capabilities-thatt is, whether the scale of their activation norms reflects their ability to handle spe- cific inputs. Specifically, for a given pre-trained MOE-LLM, we remove all routers and let every expert within a layer to process each input up to a specific "pause" node in the computational graph (e.g., after x is multiplied by W,). We then ranked the experts based on the La norm of their activa- tions at the node.' I The top- K experts continue the forward pass from the pause node to generate the final MoE outputs, while the others are terminated. We conducted 5-shot tests Mixtral 8 x 7B (Jiang et al., 2024) and Phi-3.5- instruct (Abdin et al., 2024) using MMLU (Hendrycks et al., 2021) and ARC-Challenge (Clark et al., 2018), and investi- gated how much of the performance of these LLMs can be preserved using this expert-selection strategy.
3. Method
We begin by introducing preliminary experiments that mo- tivate the development of Autonomy-of- (AoH) in Section 3. In Section 3.2, we refine the straightforward implementation from the preliminary experiments, improv- ing the expert architecture to address efficiency concerns and, finally, deriving the AoE method.
Regarding which node use for calculating the activation norm, we conducted several trials. The accuracy scores un-
We also evaluated the Lt and L norms, but these performed worse than the L² norm, as detailed in Appendix A.
3

---

Autonomy-of-Expertt Models
Algorithm 2 A working pipeline of an AoE layer
ally, large dimeennnnioo activations (14,336 for Mixtral 8 7B and 6,400 for Phi-3.5-MoF) at the pause node cached, leading to significant memory usage.
input A hidden state E Riduutt number of experts n. Initialize the activation cache C E Raxddw and p E R" as all zeros.
A factorization of the W, matrix can address these two issues. We decompose W g into two low-rank matrices: W downn E &Rdans X dhw and W up E IR Xdea, where dhow < dmodel < dwide. The i-th AoE expert can be formulated as:
output The layer output h E IR initialized as zeros.
1: // In practice, we replace the following loop with a
2: // single matrix multiplication (see Eq. 4) for efficiency.
3: for i = 1 to n do
4: Cli] = xW down
1/ C[i] dhe
Ex(x) (SiLU (x down W up) O (xW;) W
(3)
5: end for
6: p = L2 L2-Norm(CC C, dim=-1)
// P E R"
where Wi
p E IDdunsa xdea, and W Rd x Xduau
7: I = orgtopk(p)
// I E RK
Algorithm 2 formulates the pipeline within an AoE layer. In each expert, Wdown first compresses the input vectors into low-dimensional activations. These activations are cached as C, and their L norms are used to rank the experts. Given an input, the experts with the top- K norms use the cache to continue the forward computation within the expert, while unchosen experts abort processing. The compressed ac- tivations significantly reduce both the cache size and the computational overhead from unselected experts. This fac- torization does not impair the model's 's expressiveness, as the weights are inherently low-rank in large language mod- cls (Li et al., 2018; Aghajanyan et al., 2021; Hu et al., 2022).
8: P-- = So (oftmax(p(I)
1/ P E IR
9: for = 1 to n do
10: i E I then
11: h += p[j] (., ()))))
12: end if
13: end for
der various setups are shown in Table 1. We also report the time taken on A8800-88 The test code is based on the LM Evaluation Harness (Gao et al., 2024) with a batch size of 50. Experiments across different models and tasks reveal that the optimal nodes for preserving the performance of a pre-trained LLM vary. This finding supports the earlier as- sertion that, in a pre-trained LLM, there is no predetermined node whose norm best reflects experts' underlying abilities. Notably, this experiment does not update any parameters and is conducted under out-of-distribttion inference behavior, i.e, without routers. Despite this, performance preservation reaches up to 95% for Mixtral and 71% % for Phi-3.5.
Furthermore, to enhance efficiency, the loop for calculating the activation cache (Line 2 in Algorithm 2) can be elimi- nated by combining the W down matrices of all experts into single large matrix. This allows the cache to be obtained through single multiplication:
W down
W down
W down E R Amaca X (
(4)
C = W down-
These preliminary results motivate us to train an MoE model from scratch with an explicit designation of the node for expert selection. We expect that the model will naturally learn to represent its awareness of its capabilities through the norm of the designated node. Such an approach could effec- tively address the separation between the router's decision- making and the experts' execution- challenge inherent in traditional MoE models.
The resulting C E Rundw is then reshaped into an dhow matrix for subsequent computations.
In Section 4.1, we demonstrate that an AoE model achieves up to 97% of the throughput of traditional MoE model while also delivering superior downstream performance.
4. Experiments
3.2. Autonomy-of-Expers (AoE)
We begin by providing a detailed analysis of our method through ablation experiments on pre-trained small language models using AoE and traditional MoE. These experiments enable us to answer key research questions related to AoE. Based on the insights gained, we scale up the language mod- els 4 billion parameters, demonstrating AoE's s scalability.
The following paper centers on using the norm of xW, y to guide expert selection in our new MoE language models pre-trained from scratch. There is no technical difference or challenge in applying our method to any other node, regardless of the architecture. However, utilizing nodes other than x xW, or xW, p is not cost-effective.
4.1. Method Analysis through Small Language Models
The efficiency of the rudimentary method in Section 3.1 must be improved. The primary overhead arises from all ex- perts computing activations for a given token, even though not all results contribute to the final MoB output. Addition-
I. GENERAL SETUP
We train small language models consisting of 12 layers, each containing 12 attention heads. Each layer contains
4

---

Autonomy-of-Experts Models
Table Ablations were performed on 732M- parameeer language models (with 247M active parameters). Fach model trained on 100 billion tokens. The results, highlighted in color, emphasize superior performance compared to configuration @2, the most common MoE setup. Bold text indicates that configuration outperforms the best traditional MoE variant terms of average performance.
Contiguration
ARC-E
PIQA
SIQA
WINO
HELLA
MNLI
QNLI
SST2
AVG.
Traditional MoE
39.90
58.43
35.67
52.09
27.98
33.09
49.28
49.66
43.28
+ Luxx
40.74
58.49
36.13
51.30
28.11
32.67
50.23
51.83
43.68
+ Lax + Factorized W
40.45
58.65
36.75
52.09
28.03
32.55
50.08
51.03
43.70
+ Loux Large Router
41.41
57.62
36.64
52.33
28.34
33.18
49.53
50.69
43.71
AoE idam 64)
39.77
58.71
35.31
52.33
28.29
32.78
50.27
52.98
43.81
+ Leg
42.17
57.67
36.75
50.75
28.15
34.06
50.49
53.10
44.12
AoE (dhm 128)
40.70
59.41
36.64
52.09
28.06
34.38
50.69
53.21
44.39
+ Lax
41.33
58.65
36.80
50.75
28.40
33.71
49.55
53.10
44.04
AoE (dlm
41.08
58.81
36.44
51.70
28.23
32.24
50.54
53,90
44.12
Laux
41.16
58.32
36.80
53.04
28.37
32.78
50.61
54.59
44.46
AoE (dio 512)
40.57
57.89
36.75
50.59
28.38
32.71
49.72
53.56
43.77
Laux
41.16
57.83
36.75
52.09
28.30
34.92
50.67
50.92
44.08
8 experts, with the top-K 2 experts selected. Models use the Llama (Touvron et al., 2023) vocabulary of size 32,000 and the same Rre-RMSNomm (Zhang & Sennrich, 2019) module. We set dmodel = 768 and din 3,072 for traditional MoE models, while the values of diow and dwide for AoE models are variable. Specifically, in experiments below, to ensure that the total number of parameters in an AoE model is comparable that of MoE model, when adjust diows dwide is set as follows:
Configs (AB foad
Mol clow
2.70
dlow 120
2.68
00 dlow 256
2.66
die
2.64
2.62
2.60
2.58
3 dmudeel dii diow dmnndel
14000
16000
18000
20000
22000
24000 [TA4RS
dwide
(5)
Training Step
diow + 2 dmodel
Figure 2. Pre- Pre-training NLL losses. All configurations shown are trained with Cax. though its value is included in the figure.
The total number of model parameters 732 million, and the number of activated parameters 247 million.
We train models 100 billion tokens from RedPa- jama (Computer, 2023), with batch size of 4.2 million tokens, learning rate of 2 x 10 10-4, and linear warmup over the first 4,800 steps, followed by a cosine decay sched- ule that reduces the learning rate to 1.28 x 10-5 (Tow et al., 2024). The AdamW optimizer (Loshchilov & Hutter, 2019) employed with (31, (2) (0.9, 0.95), gradient norm clipping threshold of and weight decay of 0.1.
4.1.2. RESOLVING QUESTIONS REGARDING AoE
We investigate the following questions related to AoE through series ablation experiments.
How does the downstream performance of AoE compare with traditional MoE models? We evalu- ated various configurations of AoE (Configs. 5 to (2) and traditional MoE models (Configs. ! to 4. Every AoE setup outperforms the hest-performing MoE setup in terms of average accuracy across eight tasks. Notably, AoE with- out any auxiliary loss surpasses traditional MoE models, which enhances the simplicity of training an MoE model. Additionally, AoE exhibits lower training loss, suggesting more efficient training. We elaborate this in Question 2.
Question 1:
We conduct comprehensive evaluation of language mod- els across range of widely used tasks, including ARC- easy (Clark et al., 2018), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), Winogrande (Sakaguchi et al., 2019), HellaSwag (Zellers al., 2019), MNLI (Williams al., 2018), MRPC (Dolan & Brockett, 2005), QNLI (Wang et al., 2019), QQP (Wang et al., 2019), and SST-2 (Socher et al., 2013). The first five tasks are evaluated zero-shot, while the remaining tasks are tested three-shot because mod- els exhibit unstable performance in zero-shot scenarios, with most errors arising from incorrect answer formats. The ac- curacy is reported Table
Question 2: What is the impact of varying dion? ? We ad- justed dio to values of 64, 128, 256, and 512, corresponding to Configs. @. ④, L., and @, respectively. The combined impact of Laux and dio will be discussed the next ques-
5

---

Autonomy-of-Expertt Models
(a) Traditional MoF
(c) Traditional MoF: Laxx
Fapert and Diaribation f
Expen Load Distribution f
Entwst
0.14
0.04
0.03
0.50
04
0.03
0.02
0.22
1:45
1.59
0:13
0.05
0.07
0.25
0.14
0:11
0.10
0.14
1.99
0.01
0:13
0.00
0.00
0.32
0.18
0.05
1.50
1.10
0:15
2.00
0.11
0.02
0:42
0:14
0.02
0.09
0.12
0.08
1.72
1.20
0:21
0.14
0.12
0:13
0.07
0.11
0.11
0.12
2.03
1.29
0.01
0.02
0.03
0.07
0.03
1.29
1.20
0.14
0.10
0.13
0.10
0.19
0:12
0:15
0.07
2.04
1.01
0.05
0.07
0:33
0:31
0.00
0.13
0:40
0.11
1.74
1.07
0.14
0.14
0.15
0:10
0.11
0.14
0.14
0.39
2.06
Layer
0.05
0.18
0.03
0.13
0.08
0.43
0.02
0.08
1.67
1.00
0.13
0.05
0.17
0.15
0.09
0.07
0.15
2.00
1.12
055
2,05
0.13
0.06
0:27
0.08
0.09
0.06
0:22
0.08
0.05
0:14
0:12
0:12
0:13
0:12
0:15
0.16
0.02
0:28
0.06
0.07
0.20
0.32
0.04
0.01
1.66
1.28
0.09
0.18
0.06
0.15
0.13
0:28
0.06
0.14
2.01
1.08
0.02
0:30
0.06
0.05
0.07
0.10
0.32
0.07
0.13
0:30
0:13
0.14
0:17
2.03
fruonbary
0.00
0.06
0:13
0:14
0:13
0.12
0.07
0.35
1.79
1.11
0.17
0.05
0.07
0.10
0.17
0.13
0.15
0.16
2.02
1.22
Color
0:26
0:19
0:05
0.03
1:23
0:34
0.16
0.09
2.03
1.27
11
0:17
0.03
0.03
0.07
0.16
0.20
0.23
0.13
1.86
1.29
11
0.10
0.06
0.19
0.14
0:16
0.21
0.10
0.04
1.48
5
2
3
Exper: Index
Expert Index 1
Color Bat of 1u1
Bar of Entropy
(b) AoE
(d) AoE
Lau
Expert Load Distribution
Entrasd
Entoast
Expert Load Distribution F
Entlat
0:02
0.05
0.
0:41
0:05
0.C1
0.03
0.35
1:45
169
0
0.23
0.18
0.10
0.10
0.08
0.08
0:12
0.10
2.00
1.0
0:36
0.06
12
3.06
0:10
0.12
0.13
0.10
1.95
1:20
0.16
0.12
0.12
0.16
0.16
0.08
0.09
0.11
2.05
1.26
0:16
0.04
0:17
0.04
1.90
1.01
2
0.09
0.17
0.10
0.13
0.12
0.12
0:10
0.16
2.06
1.12
0:13
0.22
05
0.15
0:19
0.06
0:13
0.66
1.96
1:04
0.09
0.17
0.09
0.13
0:15
0:10
0:16
0.11
2.05
1.13
0:
0:11
0:15
0:02
0:14
1.99
0:10
0:13
0.17
0:16
0.07
0.09
0:13
2.02
Lager
0.20
0.05
C2
0.02
0.13
0.14
0.24
0:18
1.85
0.97
0.20
0.05
0.07
0.12
0:34
0.16
0.10
0.16
2.01
0.95
C4
0.23
0.05
1.90
0.10
0.12
0:12
0:14
2.04
0.00
0.22
0.08
C5
0:11
0.13
0.C3
0.21
0.17
1.93
0:93
0.09
0.13
0.07
0.13
0.36
0.05
0.17
0.10
2.00
0.60
03
0.02
0.12
0.12
0.77
0.15
0:13
0.06
0:12
0.18
2.02
9-
0.17
0.15
0.17
0:12
0.09
0.15
0.06
0.08
2.02
0.79
9
0.10
0.10
0.09
0.10
30
0.12
0.09
0.11
2.04
0.85
0.14
0:13
0.14
0.11
0.14
0.11
0.10
2.07
0.70
0.20
0.16
0.05
1.96
0.74
11
0:18
0.18
0:
0.02
0.19
0.18
0 06
0.C8
0.13
0.11 17
0:12
0.05
1.19
0.15
0.16
0.00
2.02
0.78
5
2
3
8
Export index
Expert Index i
Figure 3. Statistical analysis of expert load. The figure reveals several key insights: (1) Cam. enhances load balancing in both traditional MoE and AoE. (2) AoEs generally exhibit more balanced load distributions compared to their traditional MoE counterparts, as indicated by higher Entead values. (3) AoEs also demonstrate greater confidence in expert selection, reflected by lower Enteout values.
tion. All of these variants outperform the traditional MoE model in downstream performance. The performance dif- ferences among these configurations are relatively small. The maximum performance gain occurs when dow is ap- proximately one-third of dmedel (256/768). Both smaller and larger values of dhow result in lower performance, though they still surpass the baselines. The suboptimal performance with smaller deow may be due to the factorization of W, into Woown up being lossy approximation when diow is below the true rank of W Conversely, larger diow introduce more noise into the activation, potentially hindering the effective- ness of the norm- based selection measure.
Finding 3.1: AoE improves load balancing compared to traditional MoE models, with or without Laur-
AoE can incorporate Laux with minor modifications to Eq. 2, as shown below:
Laux = aaaxx 7 Σ f. P., where
(6)
XEB f,
1 T
Σ
1 1i argtopk (122 (x ( Woomn))
}
In Figure 2, we present the negative log- likelihood (NLL) loss during training for traditional MoE (Config. 2) and AoE models (Configs. @, @, 1. and @). AoE models ex- hibit more effective expert learning, as evidenced by lower loss values. However, when diow = 64 (Config. @), the loss is comparable to that of traditional MoE models, suggest- ing that smaller dhow values hinder AoE performance. In contrast, dwo 256 (Config. 10) results in the lowest train- ing loss overall, reinforcing the finding that setting djow to approximately one-third of dmustel yields the most benefits.
P
1 T
Σ XEB
So softmax (12-Norm (x Wi Wowwn) [0]
and aaux is determined using a validation set comprising billion tokens from (Gokaslan & Cohen, 2019). Experi- ments indicate that Qaux 0.01 is effective for both tradi- tional MoE and AoE models. We adopted this value across all configurations without further hyperparameter tuning.
Figure 3 illustrates expert load statistics on the SST-2 dataset (Socher et al., 2013) for Configs. D. ② (Tradi- tional MoE with and without Caux). 2. and 10 (AoE with and without ) We report both the load distribution f,
Question 3: How is the load balancing of AoE?
There are three main findings regarding load balancing.
6

---

Autonomy-of-Expert, Models
(as defined in Eqs. 2 and6), representing the percentage of tokens processed by expert i, and the entropy of the load distribution within each layer:
AoE
10
15
Entroad
Σ i=1
f log f.
(7)
12.5
Norm
Higher entropy values indicate more balanced load distri- butions across experts. Comparing Figures 3(a) and 3(b), without aux: AoE achieves a more balanced load distribu- tion in 11 out of 12 layers. Comparing Figures 3(c) and 3(d), with Laxx, AoE maintains a superior overall balance. For reference, the average Enteed values for subfigures (c) and (d) are 2.015 and 2.023, respectively.
10
Lo9g
7.5-
5-
2.5-
20
100
Finding 3.2: AoE models exhibit stronger confidence in expert selection.
Tokens (Billion)
AoE Laux
We introduce the confidence entropy, denoted as Enteoar- For each layer, we have:
10
18
16
Entcoaf
P: log Ps:
14
i=1
(8)
Norm.
Softmax (12-Norm (. down)) for AoE
12-
P
Softmax (R(x)), for traditional MoE
10
8-
This entropy quantifies the confidence in expert selection: lower entropy indicates a distribution closer to one-hot vec- tor, signifying more confident expert selection, while higher entropy reflects greater uncertainty in expert decisions. AoE exhibits significantly lower entropy, demonstrating stronger confidence in selecting experts. Furthermore, its confidence increases from shallow deep layers, aligning with the intuitive inductive bias that shallow layers perform funda- mental, non-specialized functions, whereas deeper layers handle specialized and abstract tasks (Wang et al., 2023; Lv et al., 2024). In contrast, MoE models do not display this trend, potentially suggesting more homogeneous expertise within and across layers (Wang al., 2024a).
20
80
100
Tokens (Billion)
Figure 4. Average activation norm dynamics during training. Each plot represents expert. distinguished by color according its layer. Experts within the same layer achieve similar activation scales, indicating that their self-evaluation criteria for determining whether they are capable of processing inputs are aligned.
Finding 3.3: Beyond improved load balancing, AoE with Lou achieves better downstream performance.
experts' weight matrix on performance by comparing Con- figurations ④ and @. The factorization does not signifi- cantly influence performance, expected in Section 3.2, based on findings that the weights of LLMs are inherently low-rank (Li et al., 2018; Aghajanyan et al., 2021; Hu et al., 2022). Therefore, the improvements observed with AoE are not attributed to the factorization of model weights.
In general, faux benefits both traditional MoE and AoE mod- els. However, when d,w 128, applying Laux results a decrease in accuracy, which we attribute to task-specific vari- ations. In conclusion, as addressed in response Question AoE exhibits strong potential for advancing MoB-based LLMs, owing to its improvements in both load balancing and downstream performance.
Question 5: Does the improvement of AoE come from involving more parameters in expert selection? We in- creased the size of the router in MoE to include n-djow "dmoded parameters, ensuring that the number of parameters involved in expert selection remains consistent with that of AoE mod- els. Note that in this setup, traditional MoE models have more activated parameters total. Comparing Config. ! and ②. the larger router provides a slight performance bene-
Question 4: Do improvements stem from the factoriza- tion of W We examined the impact of factorizing the
dum has minimal impact on the statistical metries discussed in Question 4. As result, we do not provide analysis for other configurations, as they offer little additional insight.

---

Autonomy-of-Expertt Models
Table 3. Comparison of traditional MoR and AoF: models trained using alternative expert-sclection strategies. For the Top- P strategy, the number of activated parameters is input-dependent but nearly the same between the two models, whereas the expert-choice strategy activates 247 out of 732M parameters.
Strategy
Model
ARC-E
PIQA
SIQA
WINO
HELLA
MNLI
QNLI
SST2
AVG.
Top-P
Traditional MoE
41.08
57.96
37.46
50.36
28.25
32.79
50.39
52.64
43.87
(Huang et al., 2024)
AoE
41.04
58.65
36.39
51.07
28.35
32.96
51.46
54.36
44.29
Expert-Choice (Zhou al., 2022)
Traditional MoF. AoE
40.91
59.09
37.26
50.75
28.09
32.11
50.12
52.75
43.89
41.58
58.22
37.21
53.04
28.44
33.83
50.54
50.46
44.17
fit. However, every AoE setup still outperforms this config- uration. Thus, the improvement in AoE is not primarily due to involving more parameters in expert selection.
Table 4. Throughput and memory usage comparison among several configurations. Auxiliary losses do not impact efficiency.
Question 6: How aligned are the self-evaluation criteria among experts? In AoE models, each expert independently develops self-evaluation criteria for processing tokens, as reflected in their activation scales. This might raise concerns that some experts could become overly "egoistic," meaning their internal activations are consistently larger than those of others. For example, one expert might produce activations with norms ranging from 10 to 20, while an "ego"" expert produces activations with norms from 20 to 30, leading to biased selections that favor the "ego" expert.
Configuration
TP. (K/s)/ Mem. (GB)
Traditional MoE
51,42 / 50.61
AoE (dhus 64)
49.79 59.39
AoE idus 128)
49.42 / 57.86
AoE Idua 206)
47.98 / 57.32
AoE tdin 512)
46.07 55.90
in the shape Tf x n (i.c., the router outputs in traditional MoE or the activation norms in AoE). The token-choice strategy applies the Top-K operator along the n dimension, whereas expert-choice applies it along the T dimension. Models trained using the expert-choice strategy do not re- quire auxiliary losses. We set the "ccaacitty factor" to 2 (see (Zhou et al., 2022) for details), allowing each expert to process 25% of the tokens in batch.
We track dynamics of activation norms during pre-training- Figure 4 shows the details for Configs. ! and 10. Except for the very initial period, experts' self-evaluation criteria are well aligned, as evidenced by clusters of same-colored plots (representing experts within the same layer). In the early stages of training without the auxiliary loss, some middle- to-upper-layer experts exhibit significantly lower activation. However, AoE naturally resolves this imbalance in activa- tion scales during training. Alternatively, Laux can address this imbalance earlier because it acts as regularizer for activation norms, increasing the norm scales of underactive experts and ensuring they are used more often.
Results are shown in Table 3, where djow in these experi- ments is 256. AoE outperforms traditional MoE models, demonstrating its generality across various expert-selection strategies.
Question 8: How Efficient is AoE? Table 4 shows the maximum training throughput (tokens processed per sec- ond per GPU) and memory usage for both traditional MoE models and various AoE models. Here are the key findings:
Question 7: Is AoE compatible with other expert- selection strategies? We also train language models using the Top-P token-choice (Huang et al., 2024) and the Top-K expert-choice strategy (Zhou et al., 2022).
Finding 8.1: AoE achieves up to 97% of the throughput of the traditional MoE model, with the added cost of memory.
For Top- P token-choice (Huang et al., 2024), we replace the Top-K = 2 strategy with Top-P = 0.6 following (Wang et al., 2024a). Models utilizing the Top-P strategy require an additional auxiliary loss equivalent to minimizing our introduced Entcont (Eq. 8). This ensures that the model does not learn shorteuts by assigning uniform probabilities to all experts, which would activate too many parameters to achieve lower loss. Following (Huang et al., 2024), we set the weight of this regularization term to 10
Additionally, note that experts in our experiments work se- quentially within the same layer but in practical deployments of MoE-LLMs, experts are typically distributed across dif- ferent devices and operate parallel. Consequently, experts must wait for the most loaded expert to finish computation, resulting in idle time that can be quantified by the difference between the maximum and minimum expert loads. The total differences across layers are 1.49 for Figure 3(c) (tra- ditional MoE) and 1.41 for Figure 3(d) (AoE). In this case, AoE can achieve an additional time reduction equivalent to processing 8% of the total tokens through a single MoE
Expert-choice (Zhou et al., 2022) is similar to the Top-K token-choice strategy. Consider an expert-selection matrix
8

---

Autonomy-of-Expers Models
Table For 4B-parameter L.MSs (with 1.18B active parameters), AoF exhibits better downstreamm performance than MoE models.
Model
ARC-E
PIQA
SIQA
WINO
HELLA
MNLI
QNLI
SST2
AVG.
Traditional MoE
53.70
65,40
39.10
51.54 52.57
35.80
32.19
49.77
57.00
48.06
AoE
55.98
65.61
39.87
36.77
35.39
50.05
61.93
49.80
layer. Assuming an ideal load distribution where each of the 8 experts processes 12.5% of the total tokens, this reduc- tion translates to a 64% decrease in the running time of one MoE layer. This advantage, however, is not reflected in the reported efficiency metries.
widely overlooked issue: the separation between the router's s decision-making and the experts' execution, which leads to suboptimal expert selection and learning. AoE selects experts based on their internal activation scales. Several ar- chitectural modifications ensure efficiency. Language mod- els based on AoE outperform traditional MoE models in many aspects. This paper highlights the advantages of en- abling MoE experts to self-select and aims to inspire the community to develop more powerful Moß-like models.
Finding 8.2: In AoE, memory usage and throughput are inftuenced by drow presenting trade-offs.
In terms of incremental memory, a smaller dow requires larger dwide. thereby increasing the memory consumption of xW up to T dwide. where T is the number of tokens. Conversely, a larger diow results in a larger activation cache, raising memory usage to T dlww- For Configs. ④ to 10 n diow < dwide. making the primary memory cost stem from the larger up-pr ap-projection. In contrast, Config. ① and L satisfy diow > dwide, meaning the increased memory usage is more attributable to the larger activation cache. In terms of throughput reduction, a smaller daw requires more computational resources for the up-projection, while larger diow leads a higher unused activation cache. worth noting that the efficiency of AoE diminishes the number of experts increases and sparsity grows. We are actively working on further optimizing AoE's efficiency under these conditions.
A Acknowledgement
Ang Lv supported by the Outstanding Innovative Talents Cultivation Funded Programs 2023 of Renmin University of China and CIE-Tennent Doctoral Student Research In- centive Program (HunYuan Large Language Model Special Project). Ruobing Xie is supported by the Young Elite Sci- entists Sponsorship Program by CAST (2023QNRC001) )). This work is also supported by the Public Computing Cloud, Renmin University of China and by fund for building world- class universities (disciplines) of Renmin University of China.
Impact Statement
4.2. Pre-training Large Language Models
Training large language models can generate content with ethical implications. Many effective techniques can align the preferences and values of L.LSs to mitigate these concerns. Beyond this, believe that our work does not introduce additional societal ethical issues.
We pre-train LLMs with total of 4 billion parameters, of which 1.18B are activated. The initial learning rate is 3.2 x 10-1 (Tow et al., 2024). Each model has 24 layers, with 20 attention heads per layer. For traditional MoE models, we set dmodel = 1,280 and dm = 5,0.0. Considering the trade- offs between efficiency overhead and performance gain, we set dhow 400 and, according to Eq. 5, derive dwide 6,470. Other settings follow those in Section 1. Both models are enhanced by Laux with aux 0.01. Table 5 demonstrates that AoE outperforms traditional MoE models as they scale, with the performance improvement being more pronounced in LLMs compared to smaller models. This highlights the potential of AoE to drive advancements larger and more powerful MoE-based LLMss.
References
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., Benhaim, A., Bilenko, M., Bjorek, J., Bubeck, S., Cai, M., Cai, Q., Chaudhary, V., Chen, D., Chen, D., Chen, W., Chen, Y.-C., Chen, Y.-L., Cheng, H., Chopra, P., Dai, X., Dixon, M., Eldan, R., Fragoso, V., Gao, J., Gao, M., Gao, M., Garg. A., Giorno, A. D., Goswami, A., Gunasekar, S., Haider, E., Hao, J., Hewetr, R. J., Hu, W., Huynh, J., Iter, D., Jacobs, S. A., Javaheripi, M., Jin, X., Karampatziakis, N., Kauffmann, P., Khademi, M., Kim, D., Kim, Y. J., Kurilenko, L., Lee, J. R., Lee, Y. T., Li, Y., Li, Y., Liang, C., Liden, L., Lin, X., Lin, Z., Liu, C., Liu, L., Liu, M., Liu, W., Liu, X., Luo, C., Madan, P., Mahmoudzadeh,
5. Conclusion
We introduce Autonomy-of-Expee (AoE), novel Mixture-
of Experts (MoE) paradigm that addresses a crucial yet
9

---

Autonomy-of-Expert Models
A., Majercak, D., Mazzola, M., Mendes, C. T. Mitra, A., Modi, H., Nguyen, A., Norick, B., Patra, B., Perez- Becker, D., Portet, T., Pryzant, R., Qin, H., Radmilac, M., Ren. L., de Rosa, G., Rosset, C., Roy, S., Ruwase, O., Saarikivi, O., Saied, A., Salim, A., Santacroce, M., Shah, S., Shang, N., Sharma, H., Shen, Y., Shukla, S., Song. X., Tanaka, M., Tupini, A., Vaddamanu, P., Wang, C., Wang, G., Wang, L., Wang, S., Wang, X., Wang, Y., Ward. R., Wen, W., Witte, P., Wu, H., Wu, X., Wyatt, M., Xiao, B., Xu, C., Xu, J., Xu, W. Xue, J., Yadav, S., Yang. F., Yang, J.. Yang, Y., Yang. 7., Yu, D., Yuan, I.., Zhang. C.,, Zhang, C., Zhang. J.. Zhang, L. L., Zhang, Y., Zhang, Y., Zhang. Y., and Zhou, X. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https: paattps:/aaxiii org/ abs/ 2404. 14219.
chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 4057-4086. PMLR, 17-23 Jul mlr, /
2022. URL httptttpsss/ppcccciipppee
v162/ (162/clar222a.hre html.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the al2 reasoning challenge, 2018. URL https:/ /ttp///xxvvv..//////eee/e/eeellt orgg ors//bs/ 05457.
Computer. T. Redpajama: An open source recipe to reproduce llama training dataset, 2023, URL
Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W.. Yu, X., Wu, Y., Xie, Z., Li, Y. K., Huang, P., Luo, F., Ruan, C., Sui, Z., and Liang, W. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models, 2024. URL https: / //arxiv. org/ / 2202240111110 06066.
Aghajanyan, A., Gupta, S., and Zettlemoyer, L.. Intrin- sic dimensionality explains the effectiveness of language model fine-tuning, In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.). Proceedings of the 59th Anmual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing ( (Volume 1: Long Papers), pp. 7319-7328, Online, August 2021. Association for Com- putational Linguistics. doi: 10. 10018663/1122002acc-oomm 568. URL https: /tlppps///caaaaiee htt////aaaattlloogo.ooo org/2021. hel-long.55 568.
Dolan, W. B. and Brockett, C. Automatically construct- ing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). 2005. URL https: happa:///caanhhlooyyy org/105-5005
Bisk, Y., Zellers, R., Le bras, R., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural lan- guage. Proceedings of the AAAI Conference on Artificial Intelligence, 34((5):7432-7433) aaai.v34i05.62399 URL hwtph:ttp///t/ /ojs///jj/././//// /tap://ssaaaa.rroo............oa index. php AAAII httpa////aatiteeeeeeeeeeeeeeeeee 6239.
Fan, A., Bhosale, S., Schwenk, H., Ma, Z., El-Kishky, A., Goyal, S., Baines, M., Celebi, O., Wenzek, G., Chaudhary, V., Goyal, N., Birch, T., Liptchinsky, V., Edunov, S., Grave, E., Auli, M., and Joulin, A. Beyond english- centric multilingual machine translation. J. Mach. Learn. Res., 22(1), January 2021. ISSN 1532-4435.
Apr. 2020. doi: 10.1609/
Chen, Y., Lv, A. Lin, T.-E., Chen, C., Wu, Y., Huang, F., Li, Y., and Yan, R. Fortify the shortest stave in attention: En hancing context awareness of large language models for effective tool use. In Ku, L.W,., Martins, A., and Sriku- mar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11 11160-1111 1114,, 174, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10. 10.186653/111220224eee 601. URL https: //aclanthology org/ 2224 acl- long. 601.
Fedus, W., Zoph, B., and Shazeer, N. Switch transform- ers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022. URL http:/ /jmrr. ozq/papers/v23 2ttps//////2211999//0 htmll.
Gao, L., Tow, Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, I.., Hsu, J., Le Noac'h, h, A., Li,
H., McDonell, K., Muennighoff, N., Ociepa, C.,, Phang,
J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https: /ttp///eeoooooo/ wwwpooddooorgg//eoeoeooooooooooo arg/records/ 12608602.
Clark, A., De Las Casas, D., Guy, A., Mensch, A., Pa- ganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., Borgeaudd, S., Van Den Driessche, G. B., Rutherford, E., Hennigan, T., Johnson, M. J., Cassirer, A., Jones, C., Buchatskaya, E.,, Budden, D., Sifre, L., Osindero, S., Vinyals, O., Ranzato, M., Rae, J., Elsen, F.,, Kavukcuoglu, K., and Simonyan, K. Unified scaling laws for routed language models. In Chaudhuri, K., Jegelka, S., Song, L.. Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the 39th International Conference on Ma-
Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t (eds.), Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, On- line and Punta Cana, Dominican Republic, November
10

---

Autonomy-of-Expers Models
2021. Association for Computational Linguisties. doi: 1.11666/v1122211emmppmmaa main.m...m..mmm.....m.m URL https:/ aclanthology org/2022 oc.atthoooory.oor/222111.. emnlp- 446.
Lavaud, L. R., Saulnier, L.., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L.,, Gervet, T., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mixtral of experts, 2024. BRLLhtt:://axa. org/ /aabb//////// 2401. 04088.
Gokaslan, A. and Cohen, V. Openwebtext cor- pus.
http://Skyliont wtw.skyy111000.11o11 www.:077...tt...bb.. / OpenWebTexxtCorpus 2019.
Lenz, B., Lieber, O., Arazi, A., Bergman, A., Manevich, A., Peleg, B., Aviram, B., Almagor, C., Fridman, C., Padnos, D., Gissin, D., Jannai, D., Muhlgay, D., Zimberg, D., Ger- ber, E. M., Dolev, E., Krakovsky, E., Safahi, E., Schwartz, E., Cohen, G., Shachaf, G., Rozenblum, H., Bata, H., Blass, I., Magar, I., Dalmedigos, I., Osin, J., Fadlon, J.. Rozman, M., Danos, M., Gokhman, M., Zusman, M., Gidron, N., Ratner, N., Gat, N., Rozen, N., Fried, O., Leshno, O., Antverg, O., Abend, O., Dagan, O., Cohavi, O., Alon, R., Belson, R., Cohen, R., Gilad, R., Glozman, R., Lev, S., Shwarz-S S., Meirom, S. H., Delbari, T., Ness, T., Asida, T. Gal, T. B.,, Braude, T., Pumer- antz, U., Cohen, J., Belinkov, Y., Globerson, Y., Levy, Y. P., and Shoham, Y. Jamba: Hybrid transformer-mamba language models. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https: //openreview net / forumm pididdd=FPpppppppp
Gong, Z., Lv, A., Guan, J., Wu, W., Zhang, H., Huang, M., Zhao, D., and Yan, R. Mixture-of of ofotuue-o-mmodule:: Reinvent- ing transformers dynamic assemblies of modules. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 20924-20938, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi: 10.18653/112222.mmmmll main. 1164. URL htwp::ttts/..... a/ aaccclnhhologg. org/2044 emnlp ma in.i 1164.
Gururangan, S., Lewis, M., Holtzman, A., Smith, N. A., and Zettlemoyer, L. DEMix layers: Disentangling do- mains for modular language modeling. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V. (eds.), Pro- ceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguis- ties: Human Language Technologies, pp. 5557-5576, Seattle, United States, July 2022. Association for Compu- tational Linguistics. doi: 10. 10.186553/1120022maalll 407. URL https:/ naacl -ma mmman..000 407.
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang. Y., Krikun, M., Shazeer, N., and Chen, Z. (GS) Seal- ing giant models with conditional computation and auto- matic sharding. In International Conference Learning Representations, 2021. URL https: openreview....eeeee net
Hendrycks, D., Burns, C.,, Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https:// openreview. / forum? iddd7k=dkk jm
Lewis, M., Bhosale, S., Dettmers, T., Goyal, N., and Zettle- moyer, L. Base layers: Simplifying training of large, sparse models. In Meila, M. and Zhang, T. (eds.), Pro- ceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6265-6274 PMLR, 18-24 Jul 2021. httpttps: /tttttts://ppoeeeee http///pcceeenggeeeee/ep mlr, press/ v1 v139/ 1139/lewis211 s2wa..sll... html.
Hu, F. J., Shen, Y., Wallis, P., Allen-Zhu, Z.,, Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https:/ openrevicn www..iiwwww.... net. forum? wwwtorrmmmiiiineiieeeeee innndddeeeeeeeeeeeee
Li, C., Farkhoor, H. Liu, R., and Yosinski, J. Measur- ing the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018. URL https:/ www.oeeeiew.nneee.eee.ee forum? id=ryup8-WCw.
Huang, Q., An, Z., Zhuang, N., Tao, M., Zhang, C., Jin, Y., Xu, K., Xu, K., Chen, L., Huang, S., and Feng, Y. Harder task needs more experts: Dynamic routing in MoE models. In Ku, L.WW., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12883-12895, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https://
Lin, H., Lv, A., Chen, Y., Zhu, C., Song, Y., Zhu, H., and Yan, R. Mixture of in-context experts enhance LLMs' long context awareness. In The Thirty-eighth Annual Conference Neural Information Processing Systems, net/ forum?
2024. URL
doi: 10. 10..18633w11122244.c444oog..... aclant aclantholocy.org/00000 aon-on 696/.
id=RcP HboficN.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019. URL https:://// //pp://////ee..e... net/ met./forumiid-ae 6Ric)77
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G.,

---

Autonomy-of-Expert Models
Lv, A., Chen, Y., Zhang, K., Wang, Y., Liu, L.,, Wen, J.-R., Xie, J., and Yan, R. Interpreting key mechanisms of fac- tual recall in transformer-based language models, 2024. URL https: / arxiv. org/ abs/ 2200./243... 19521.
and Bethard, S. (eds.), Proceedings of the 2013 Confer- ence on Empirical Methods in Natural Language Process- ing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguisties. URL https: /tc//aallca aeeelaathhlll ogy htt.eeeeooooo.o...o/.....1--11-. ortg//3111111111100
Pham, Q. Do, G., Nguyen. H., Nguyen, T., Liu, C., Sartipi, M., Nguyen, B. T., Ramasamy, S., Li, X., Hoi, S., and Ho, N. Competesmoe effective training of sparse mixture of experts via competition, 2024.
Sun, X., Chen, Y., Huang, Y., Xie, R., Zhu, J. Zhang, K., Li, S., Yang, Z., Han, J., Shu, X., Bu, J., Chen, Z., Huang. X., Lian, F. Yang. S., Yan, J., Zeng, Y., Ren, X., Yu, C., Wu, L., Mao, Y., Xia, J., Yang. T., Zheng, S., Wu, K., Jiao, D., Xue, J., Zhang, X., Wu, D., Liu, K., Wu, D., Xu, G., Chen, S., Chen, S., Feng, X., Hong, Y., Zheng, J., Xu, C., Li, Z., Kuang, X., Hu, J., Chen, Y., Deng, Y., Li, G., Liu, A., Zhang, C., Hu, S., Zhao, Z., Wu, Z., Ding, Y.. Wang, W., Liu, H., Wang. R., Fei, H., Yu, P., Zhao, Z., Cao, X., Wang, H., Xiang. F., Huang, M., Xiong. Z., Hu, B., Hou, X., Jiang. L., Ma, J.. Wu, J., Deng, Y., Shen, Y. Wang. Q.. Liu, W., Liu, J., Chen, M., Dong, I.,, Jia, W., Chen, H., Liu, F., Yuan, R., Xu, H., Yan, Z., Cao, T., Hu, Z.,, Feng, X., Du, D., Yu, T., Tao, Y., Zhang, F., Zhu, J., Xu, C., Li, X., Zha, C., Ouyang. W., Xia, Y., Li, X., He, Z., Chen, R., Song, J., Chen, R., Jian, F., Zhao, C., Wang, B., Gong, H., Gan, R., Hu, W., Kang, Z., Yang. Y., Liu, Y., Wang. D., and Jiang, Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent, 2024. URL https: /ttp/aaxvvvv/////////////..//aaa wwwpxii.orgg/bbo.. wrg.abs.///oooo.o.ooo.o......... 2aabbs/2410 221221111111 02265.
Raposo, D., Ritter, S., Richards, B., Lillicrap. T., Humphreys, P. C., and Santoro, A. Mixture-of-depths: Dynamically allocating compute in transformer-based language models, 2024.
Ren, X., Zhou, P., Meng, X., Huang, X., Wang, Y., Wang, W., Li, P., Zhang, X., Podolskiy, A., Arshinov, G., Bout, A., Piontkovskaya. I., Wei, J., Jiang, X., Su, T., Liu, Q., and Yao, Pangu-sigma: Towards trillion parameter lan- guage model with sparse heterogeneous computing, 2023. URL htw.h:tts:///et http://axxvvvv/. org.ii..rrraaaae wttp:///.........././ 10845.
Roller, S., Sukhbaatar, S., Szlam, A., and Weston, J. E. Hash layers for large sparse models. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https:://///// https///pppeeeeeeeeeeeeee www.eeewweee/oooee net/forum? id=1MgDDWb1UlM
Team, Q. Owen1.5-moe: Matching 7b model per- formance with I/3 activated parameters", February 2024. hwtphhtpp////eeeee wawieenmm.iituubb...... github. io/blog/ qwen qwen-moe/.
Sakaguchi, K., Bras, R. L.,, Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https:/ // awwaaiivvvo... org/ abs/ 1 1907. 10641.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation lan- guage models, 2023. URL https: http://axivvvorr abs/2302.13971.
Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social IQa: Commonsense reasoning about social interactions. In Inui, K., Jiang, J., Ng. V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP,, pp. 4463-4473, Hong Kong, China, November 2019. Association for Compu- tational Linguistics. doi: 1/118663/1/D11-144 URL https: aclaccclnni wwwaaaaanncllgggoo www.aaathhhooo.o..oooo-ea-- 1454.
Tow, J. Bellagente, M. Mahan, D., and Riquelme, C. Stablelm 3b 4elt. 2024. URL [https: pottps::/hugger httphuugiinfaa...//ttaaaiiiiiiia stable 4elt] hhttps:/ co/stabillyyii/taaaee
3b- 4elt).
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. URL https: http:///eeeeeeeeeeeeeeee net net/ffrunm id=
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. Hinton, G., and Dean, Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. URL https: / wptp//wwwwweeeeeeee/peeeeeo net/ forum? id=B1ckMDQ1g.
Wang. A., Sun, X., Xie, R., Li, S., Zhu, Yang. Z., Zhao, P., Han, J. N., Kang, Z., Wang, D., Okazaki, N., and zhong Xu, C. Hmoe: Heterogeneous mixture of experts for language modeling, 2024a. URL https: //arxiar..vv org/ /albs///408.0mm 10681.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Yarowsky, D., Baldwin, T., Korhonen, A., Livescu, K.,
12

---

Autonomy-of-Expert Models
Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretabilityy in the wild: circuit for indi- rect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL, https: htw.:tttsss///eeeeeeeeeeeeee www.rppeeeeeeeee forrmmmmmm id-NpsVSN6o4ul.
former with stochastic experts. In International Confer- ence Learning Representations, 2022. URL https:
/ Bttp://www..t//ooooo/od////////
Wang. L., Gao, H., Zhao, C.. Sun, X., and Dai, D. Auxiliarylloss-free load balancing strategy for mixture-of- experts, 2024b. URL https: /tapps://arxi..../ org/ abs/ 2408. 15664.
Williams, A., Nangia, N., and Bowman, S. A broad- coverage challenge corpus for sentence understanding through inference. Walker, M., Ji, H., and Stent, A. (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume I (Long Papers), pp- 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653//1/N18-11 1001. URL https: //aclanthology org/ N18 111p//1181111111 1101.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sen- tence? In Korhonen, A., Traum, D., and Marquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791- 4800, Florence, Italy, July 2019. Association for Compu- tational Linguistics. doi: 10. 10.18653/11/119-1444 URL https: / aclaccllnno aclanthologyyyy
/P19 1472/.
Zhang, B. and Sennrich, R. Root mean square layer nor- malization. Curran Associates Inc., Red Hook, NY, USA, 2019.
Zhou, Y., Lei, T. Liu, H., Du, N. Huang, Y., Zhao,
V. Y., Dai, A. M., Chen, 7., Le, Q. V., and Laudon, J. ofxtere-of-experts with expert choice routing. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL httphhtt////// httpt:/tt////aeee /wtps///eeeeeeeeeeeeemm net/ forum? id=jdJolHIVinI.
Zhou, Y., Du N., Huang, Y., Peng, D., Lan, C., Huang, D.,
Shakeri, S., So, D., Dai, M., Lu, Y., Chen, Z., Le, Q Cui, C., Laudon, J., and Dean, Brainformers: Trading simplicity for efficiency. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Ma- chine Learning, volume 202 Proceedings of Machine Learning Research, pp. 42531-42542. PMLR, 23-29 Jul 2023. Uttsttps: http///poceenngg...e mlr press/ v202 hht./ccoo2cccccc html.
Zuo, S., Liu, X., Jiao, J., Kim, J., Hassan, H., Zhang, R., Gao, J., and Zhao, Taming sparsely activated trans-
13

---

Autonomy-of-Experts Models
A. Re-running Experiments in Section 3.1 Using Alternative Expert-Selection Metrics
also use Lt and - norms as expert-selection metrics in pre-trained LLMs, which resulted in poorer performance preservation compared the L norm. The time costs for each configuration are identical to those presented in Table and therefore omitted here for clarity. The results are shown below.
Table 6. Preliminary study results pre-trained MoE-LLMs, selecting experts by L' norn of internal activation.
Node for Norm Calculation
MMLU (5-shot)
ARC-C (S-shot)
Mixtral x
Phi-3.5-Mof.
Mixtral Phi-3.5-MoE-ins.
X xW xWp X SiLU(XW,) ,) xww,
51.14
24.15
41.98
29.01
39.79
35.87
40.19
36.35
47.29
26.37
45.73
36.09
54.37
26.95
50.09
33.79
Experts' Final Outputs
57.84
26.56
52.73
31.31
Performance w. Router
70.35
78.20
62.12
67.41
Table Preliminary study results on pre-trained MoE-LLMs, selecting experts by - norm of internal activation.
Node for Norm Calculation
MMLU (5-shol)
ARC-C (S-shot)
Mixtral 8 x 7B
Phi-3.5 Phi-3.5-MoE-ins.
Mixtral 8 x 7B
Phi-3.5-MoE-inss
xW xW W ) SiLU(xW,) ,) O xW,
16
29,28
43.77
35.92
50.43
34.78
49.49
40.02
54.30
36.38
47.95
50.85
50.72
26.43
46.08
33.02
Experts' Final Outputs
51.03
23.64
53.16
30.12
Performance w. Router
70.35
78.20
62.12
67.41
B. Additional Interpretation of AoE's Advantage
We provide some intuitive insights into AoE's strengths by developing fully controlled classification task and monitoring training dynamics of both tiny AoE and MoE models. We provide details here for interested readers. This experiment is of toy nature and not intended as major claim or contribution.
In our setup, inputs are multivariate Gaussian vectors belonging to three classes. Classes one and two have distinct positive and negative means, respectively, while class three has a zero mean. We adjust their standard deviations to ensure no overlap within a three-sigma range. Initially, we train both tiny AoE and MoE classifiers to distinguish between classes one and two; this is referred training stage one. After convergence, we introduce class three into the training process and continue training, referred to training stage two. The classifiers consist of single layer with experts. Throughout training, monitor expert behaviors, such as internal activation scales and token load. Figure 5 illustrates the pipeline and results of this toy experiment.
During training stage one, we observed that MoE classifiers assign class one and class two to different experts. This suggests that the classification role is primarily handled by the router, while the experts perform post-prose post-processing contrast, AoE uses only one expert to process all inputs during training stage one. Early in training, one expert identifies that the two classes are separable and develops the capability for binary classification. As training progresses, this expert's ability (reflected in increasingly larger activation norms) causes the other expert to remain naturally idle.
In training stage two, MoE evenly assigns inputs from the newly added class three to both experts. This occurs because the router has been trained for binary classification and lacks the capacity to handle distribbitiiitoonn inputs, leading to equal prediction distribution across experts. This exacerbates the issue of homogeneous experts the MoE classifier, as the capability to classify class three is also distributed across all experts. Conversely, in the AoE classifier, the expert handling classes one and two exhibits low activation when presented with third-class inputs. Its activation is even lower than that
14

---

-eapoomy--o--xpperr Models
Truining Stuge One
Training Siage Two
They determine if the arc positive negalive
They posi-process inputs.
Class assigned evenly experts.
Export inputs.
Expert 1
Expert
Expert 1
Expert 2
Expert
Expert 2
Expert
Expert
Class 2
Class
Class
Closs
Class 3 Inputs
Class
Class
Class Inputs
Class 2
Inputs
Inpuls
Traditional MoE Classifier
AoE Classifier
Traditional MoE Classifie
AoE Classifier
Figure 5. The overview of our experiments training tiny AoE and traditional MoE classifiers.
of the idle expert, which lacks specialization and does not resist class three inputs. As result, the idle expert naturally handles all class three inputs. This results in heterogeneous experts within the AoE classifier: one expert manages the negative-positive classification,, while the other processes zero-mean inputs.
Notably, in these toy experiments, the expert load during the first training stage is not balanced AoE. In contrast, real-world pre-trained language models do not exhibit this imbalance, as shown in Figure The reason is that the classification of input features in practical scenarios is far more complex, with greater number of classes involved. As an evidence, when class three is added during training, AoE achieves balanced expert load.
Comparing token assignments between the two models reveals several drawbacks of traditional MoF models:
(1) Sub-optimal expert selection: The binary classification task of distinguishing between classes one and two, which relatively easy, could be effectively managed by a single MLP (i.e., one expert). However, MoE classifiers utilize both experts due to the router's classification behavior. This leads to under-exploitation of parameters and highlights the sub-optimal selection of experts in traditional MoE models, resulting from "the separation between the router's decision and the experts' execution."
(2) Distributed expertise: The ability to perform binary classification is distributed across two experts, preventing specializa- tion.
The observation holds and near-zero loss is achieved as long as there is no overlap within a three-sigma range. In our experiments, we tested input dimensions and the model's dm and dwide parameters within the range of 32 256. When the input dimension small relative to the model dimension, the task becomes too easy to learn, and the above behavior is not observed. Conversely, if the input dimension is too large, the task becomes difficult, preventing the loss from decreasing and rendering observed behavior uninformative.
15


Reviews:
[{"review_id": "g1LCmAcaMS", "reviewer": "Reviewer_yvWh", "summary": "This paper proposes a new scheme to select expert sublayer in the mixture-of-experts (MoE) language model. Rather than empoying a router layer to choose the expert to process incoming embedding, the method utilizes a factorized subcomponent of feed-forward layer to calculate the importance score (\"norm\" in the paper) that are fed to a softmax and top-k to choose the expert to be activated. Experiments show that the proposed method outperforms the traditional MoE with router sublayer with a better outcomes with a load balancing loss.\n\n## update after rebuttal:\n\nThank you for your comments on my review! The additional results would be useful to discuss about this study more in-detail.\nHowever, I don't have strong reason to revise my overall judgment, so I will leave the overall score as it is.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive suggestions and valuable comments! We hope our rebuttal helps address your concerns. If so, we would be grateful if you could consider increasing the overall recommendation.\n&nbsp;\n\n---\n&nbsp;\n\n# Comparison with more MoE works\n\nWe trained the Hash-Layer model (Roller et al.) on 100B tokens, using the same hyperparameters and model setup as our other models. Unlike other methods cited in the last paragraph of Section 2, which require domain-labeled data, Hash-Layer does not—hence our decision to include only this method for comparison. Here are the results:\n### Task Performance\n|100B tokens|ARC-E|PIQA|SIQA|WINO|HELLA|MNLI|QNLI|SST2|AVG.|\n|-|-|-|-|-|-|-|-|-|-|\n|MoE (Config.2)|40.74|58.49|36.13|51.30|28.11|32.67|50.23|51.83|43.68|\n|Hash-Layer|41.12|57.62|36.18|50.91|28.82|32.54|50.30|51.03|43.57|\n|AoE (Config.10)|41.16|58.32|36.80|53.04|28.37|32.78|50.61|54.59|44.46|\n\nOur results align with Roller et al. (p.7), who note that \"Switch Transformer and Hash perform similarly in multi-layer routing.\" Hash-Layer's key advantage lies in its balanced load distribution.\n\nOur AoE model not only outperforms both Hash-Layer and traditional MoE in downstream tasks, but also demonstrates improved load balancing compared to traditional MoE.\n\n&nbsp;\n\n---\n&nbsp;\n\n# Please compare other expert selection mechanisms\n\nThank you for the suggestion. We have added a new baseline—the Hash-Layer—as noted above. In Table 2, we report results using the _top-K token choice_, while Table 3 includes experiments with _top-P token choice_ and _top-K expert choice_ selection mechanisms. We hope these results address your concern.\n\nIf you have further suggestions for additional baselines, we’d be glad to explore them or discuss more!\n\n&nbsp;\n\n---\n&nbsp;\n\n# Try factorizing $W_p$ or other trivial tweaks\n\nThank you for your insightful question. We trained new models by factorizing $W_p$ instead of $W_g$ ($d_{low}$ = 256, using 100B tokens with $L_{aux}$), but observed a performance drop.\n### Task Performance\n|100B tokens|ARC-E|PIQA|SIQA|WINO|HELLA|MNLI|QNLI|SST2|AVG.|\n|-|-|-|-|-|-|-|-|-|-|\n|MoE (Config.2)|40.74|58.49|36.13|51.30|28.11|32.67|50.23|51.83|43.68|\n|AoE (Config.10, Factorized $W_p$ )|40.95|58.22|36.54|50.67|28.08|32.97|50.21|53.67|43.91|\n|AoE (Config.10)|41.16|58.32|36.80|53.04|28.37|32.78|50.61|54.59|44.46|\n\nWe have a few hypotheses regarding the performance drop:\n\n1) Factorizing $W_p$ might create a bottleneck in this branch, while the activation function in the $W_g$ branch may already act as a bottleneck. Since both pathways in the FFN are constrained in this case, it could lead to the performance drop. If this is the case, we would suggest always factorizing $W_g$ in AoE or encourage future work to further develop expert architectures.\n\n2) Alternatively, the optimal value for $d_{low}$ might change when factorizing $W_p$.\n\nDue to time constraints, we cannot provide a definitive explanation at this moment. However, these aspects of model architecture are valuable areas for future research, and we will discuss this point in the paper.\n\nWe also experimented with computing norms using only the first _n_ elements of the intermediate layer ($n=256$), training with 50B tokens due to time constraints. Compared to standard AoE ($d_{low}=256$) and MoE, this method underperformed because of insufficient activation information for expert selection:\n### Task Performance\n|50B tokens|ARC-E|PIQA|SIQA|WINO|HELLA|MNLI|QNLI|SST2|AVG.|\n|-|-|-|-|-|-|-|-|-|-|\n|MoE (Config.2)|38.76|56.26|35.67|50.12|27.34|33.05|50.23|50.80|42.78|\n|AoE (Split, n=256)|39.14|55.71|36.03|50.75|27.55|33.05|50.27|49.31|42.72|\n|AoE (Config.10)|39.27|57.13|35.98|51.70|27.40|33.32|50.87|50.34|43.25|\n\nWe will include these points in the paper, as they offer insights that may be useful for future research. Thank you again for your insightful question!", "claims_and_evidence": "The claim of the paper is based on assumption that the model separation between each expert and router should be hermful. The paper said this characteristics is \"overlooked\", but the paper contains basically only empirical results on the proposed model that compares downstream scores, meaning that it is not sure to me to say that the features the paper focused on is necessary or not.", "methods_and_evaluation_criteria": "The proposed method is very intuitive and it should work if the large amount of training iteration have processed (like other MoE models).\nFactorizing gate layers in the expert layer significantly lose information from the expert layer, but is an acceptable solution to yield low-dimensional vector for score calculation if we relied on references cited on top-right of p.4. Table 2 also shows its impact is negligible.", "theoretical_claims": "The proposed method is just an empirical method based on hypothesis that the expert sublayer itself is more aware of selecting tokens to process.", "experimental_designs_or_analyses": "Evaluations are employed only for comparing the proposed method with traditional MoE. Though the results show better outcomes on the proposed method, it is still unclear whether the proposed method is still better against other strategies to select experts (e.g., those cited in the last paragraph in the section 2.", "relation_to_broader_scientific_literature": "Not sure", "essential_references_not_discussed": "Not sure", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "(1) please add experiments to compare other expert selection mechanisms.\n(2) The proposed method introduced factorization of the gate layer in the expert. Could you show some results when other subpart is employed for factorization (especially $W_p$) or other trivial tweaks (e.g., splitting first n-th element of intermediate layer to calculate norms)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tnfy5Hwu4N", "reviewer": "Reviewer_44vv", "summary": "This paper introduces Autonomy-of-Experts (AoE), a novel approach to Mixture-of-Experts (MoE) models that addresses a critical issue in traditional MoE architectures: the separation between routing decisions and expert execution. In traditional MoE, a router decides which experts process which inputs, creating potential misalignment between routing decisions and expert capabilities.\n\nThe core innovation of AoE is to eliminate the router entirely and allow experts to autonomously determine whether they should process inputs. I quite like the author's initial findings in existing findings that justify the development of AoE and also the toy illustration of AoE behaviors in the appendix which is very cute.\n\nOverall, the authors justify the claims by extensive pre-training and ablation experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive suggestions and valuable comments! We hope our rebuttal helps address your concerns. If so, we would be grateful if you could consider increasing the overall recommendation.\n&nbsp;\n\n---\n&nbsp;\n# To discuss training time more explicitly\n\nHere are the total machine hours (1 machine = 8 GPUs):\n||Machine Hours\n|-|-\n|MoE|72.73\n|AoE ($d_{low}$=64)|76.15\n|AoE ($d_{low}$=128)|76.58\n|AoE ($d_{low}$=256)|77.56\n|AoE ($d_{low}$=512)|81.34\n\nWe will include these results. Thank you for the feedback.\n\n&nbsp;\n\n---\n&nbsp;\n# Small Model's Performance gains are relatively modest on some tasks\nThere are two key factors to consider:\n\n1. Larger models are stronger. With the same number of tokens, a large AoE shows more noticeable gains over MoE.\n2. The \"spiral rise\" in performance during pre-training. Both AoE and MoE show steady overall improvements, but task-specific performance fluctuates across checkpoints. We show results at 50B, 80B, and 100B tokens (AoE: Config.10, MoE: Config.2) to illustrate this.\n\n\tFor example, at 100B tokens, MoE slightly outperforms AoE on PIQA due to a larger gain from 80B to 100B. A similar trend occurs with HELLA from 50B to 80B, though AoE regains the lead at 100B. Despite fluctuations, AoE consistently achieves higher AVG. scores across checkpoints.\n\n### Task Performance over tokens\n|Tokens|Models|ARC-E|PIQA|SIQA|WINO|HELLA|MNLI|QNLI|SST2|AVG.\n|-|-|-|-|-|-|-|-|-|-|-\n|50B|MoE|38.76|56.26|35.67|50.12|27.34|33.05|50.23|**50.80**|42.78\n||AoE|**39.27**|**57.13**|**35.98**|**51.70**|**27.40**|**33.32**|**50.87**|50.34|**43.25**\n|80B|MoE|40.45|57.45|36.39|50.20|**27.93**|33.32|50.28|50.57|43.32\n||AoE|**41.79**|**57.67**|**36.44**|**51.30**|27.83|**34.51**|**50.38**|**51.49**|**43.93**\n|100B|MoE|40.74|**58.49**|36.13|51.30|28.11|32.67|50.23|51.83|43.68\n||AoE|**41.16**|58.32|**36.80**|**53.04**|**28.37**|**32.78**|**50.61**|**54.59**|**44.46**\n\nWe'll include detailed task accuracy development plots to highlight the consistent advantages of 732M AoEs.\n\n&nbsp;\n\n---\n&nbsp;\n# Can AoE insights benefit traditional MoEs?\n \nWhile AoE shows that routers can be removed, we try to use expert norms as labels for router training. The router receives gradient-detached inputs to avoid interfering with AoE learning. During inference, expert selection is performed solely by the router to reduce memory usage. This setup led to a performance drop (50B tokens, $d_{low}$=256, with $L_{aux}$):\n\n### Task Performance\n|50B tokens|ARC-E|PIQA|SIQA|WINO|HELLA|MNLI|QNLI|SST2|AVG.\n|-|-|-|-|-|-|-|-|-|-\n|MoE (Config.2)|38.76|56.26|35.67|50.12|27.34|33.05|50.23|50.80|42.78\n|AoE w. router|38.24|56.37|35.88|51.30|27.36|32.32|50.27|50.11|42.73\n|AoE (Config.10)|39.27|57.13|35.98|51.70|27.40|33.32|50.87|50.34|43.25\n\nThese results suggest that—even with supervision—routers struggle to learn effective expert selection, highlighting the limitation of separating routers and experts. We will discuss this in the paper and hope it inspires further research. Thank you for the thoughtful question.\n\n&nbsp;\n\n---\n&nbsp;\n# Effects of $d_{low}$\n\nWhen $d_{low}$ = 64 or 128, AoE with $L_{aux}$ results in lower performance compared to $d_{low}$ = 256 (Config.10). Additionally, as shown in Figure 2, $d_{low}$ = 64 results in slower convergence, similar to traditional MoEs.\n\nWe also tested an extreme case where $d_{low}$ = 8. In this case, the approximation of $W_g$ is too lossy, leading to poor performance and high loss. Due to limited time, we trained with only 50B tokens. We will include this experiment and make our statement more accurate. Thank you for your valuable feedback!\n### Task Performance\n|50B tokens|ARC-E|PIQA|SIQA|WINO|HELLA|MNLI|QNLI|SST2|AVG.\n|:-:|-|-|-|-|-|-|-|-|-\n|MoE (Config.2)|38.76|56.26|35.67|50.12|27.34|33.05|50.23|50.80|42.78\n|AoE ($d_{low}$=8)|37.67|56.31|34.80|50.36|27.21|32.27|50.08|50.34|42.38\n|AoE (Config.10)|39.27|57.13|35.98|51.70|27.40|33.32|50.87|50.34|43.25\n### Loss over Tokens\n||MoE|AoE ($d_{low}$=8)|AoE (Config.10)\n|-|:-:|:-:|:-:\n|10B|3.59|3.70|3.59\n|20B|3.09|3.17|3.08\n|30B|2.92|3.02|2.91\n|40B|2.84|2.91|2.82\n|50B|2.77|2.86|2.75\n\n---\n&nbsp;\n# The uncertainty of the accuracies\n\nFirst, we note that AoE outperforms MoE across various setups presented in our paper, with the testing process using greedy decoding to eliminate generation randomness.\n\nFor each configuration, we aggregated the outputs from all tasks into a sequence of 1s (correct) and 0s (incorrect), paired with the outputs of traditional MoE (Config. 2), and performed a McNemar test. The improvement of AoE over MoE is significant (p < 0.05) across all configurations in Tables 2, 3, and 4. We will highlight the significance of this improvement.", "claims_and_evidence": "The authors claims are quite clear and supported by convincing evidence.", "methods_and_evaluation_criteria": "The methods proposed in the paper are appropriate for addressing the identified problem:\n\n1. The authors first validate their core insight through preliminary experiments on pre-trained MoE models before developing their full AoE approach.\n2. The low-rank factorization to reduce computational overhead is well-motivated and effectively implemented.\n3. The evaluation criteria are appropriate.", "theoretical_claims": "The paper does not present formal theoretical proofs but provides conceptual explanations for why AoE is effective.", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "The paper appropriately situates AoE within the broader MoE literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Overall, I find this paper presents a solid technical improvement, though I have a few remaining questions:\n\n1. The computational overhead analysis focuses on throughput but could discuss training time more explicitly.\n\n2. While AoE outperforms traditional MoE, the performance gains on the 732M parameter models are relatively modest on some tasks. Can you explain it?\n\n3. The paper could benefit from more discussion on whether the insights from AoE could be incorporated into traditional MoE architectures without completely replacing the router.\n\n4. The paper mentions that a smaller d_low may be a lossy approximation when below the true rank of Wg. d_low seems to have no effects to the final performance in 732M model.\n\n5. What about the uncertainty of the accuracies in the tables?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lU7h5IyLOP", "reviewer": "Reviewer_BPoe", "summary": "The authors introduce a new Mixture of Experts (MoE) paradigm called Autonomy-of-Experts (AoE), where experts independently decide whether to process inputs. The foundation of AoE lies in the understanding that an expert can gauge its ability to effectively handle a token by observing its internal activations. In AoE, routers are eliminated; experts pre-calculate internal activations for inputs and are ranked according to their activation norms. Only the highest-ranking experts continue with the forward pass, while the others are terminated. The overhead associated with pre-computing activations is mitigated through low-rank weight factorization. This method of self-assessment followed by comparison with peers leads to enhanced expert selection and effective learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive suggestions and valuable comments! We hope our rebuttal helps address your concerns. If so, we would be grateful if you could consider increasing the overall recommendation.\n&nbsp;\n\n---\n&nbsp;\n\n# If the baselines should be allowed to pre-train for additional steps\n\nWe would like to clarify that in our paper, we ensured fairness by using the same model size and training on the same number of tokens. The difference in throughput (tokens processed per second) results in slightly longer training times of AoE models but does not indicate that AoE has learned additional knowledge. If baselines are trained for additional steps, they would acquire knowledge that AoE does not. If you have any further experimental suggestions, we would be happy to discuss or explore them!\n\n&nbsp;\n\n---\n&nbsp;\n\n# To train Switch Transformer with multiplicative noise\n\nThank you for your valuable advice. We trained a traditional MoE model with multiplicative jitter noise (0.01), applied to the layer inputs using 100B tokens with $L_{aux}$. As shown by the Switch Transformer, jitter noise encourages expert exploration, which is beneficial for MoE models, but it still cannot outperform AoE models with $L_{aux}$.\n\n||ARC-E|PIQA|SIQA|WINO|HELLA|MNLI|QNLI|SST2|AVG.|\n|-|-|-|-|-|-|-|-|-|-|\n|MoE (Config.2)|40.74|58.49|36.13|51.30|28.11|32.67|50.23|51.83|43.68|\n|MoE (Config.2 + noise)|40.95|58.43|36.75|51.07|28.34|32.96|50.01|51.72|43.78\n|AoE (Config.10)|41.16|58.32|36.80|53.04|28.37|32.78|50.61|54.59|44.46|\n\nWe will include these experiments in our paper. Thank you again for your helpful suggestions!", "claims_and_evidence": "Yes, the claims appear to be substantiated by the evidence.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical analysis", "experimental_designs_or_analyses": "Experimental design and analysis appears to be valid.", "relation_to_broader_scientific_literature": "The contributions are contextualized properly in the context of broader scientific literature.\nThe idea of using internal activations of experts in an efficient manner for pre-training is novel in the context of Sparse MoEs.", "essential_references_not_discussed": "Related works that are essential to understanding the (context for) key contributions of the paper are discussed.", "other_strengths_and_weaknesses": "Strengths:\n- The paper is written clearly\n- The idea of using internal activations of experts in an efficient manner is novel\n- The pre-training experiments and ablation studies are thorough.\n\nWeaknesses:\n- Given there is a 3% difference in throughput between MoE and AoE, it would be interesting to see if the baselines should be allowed to pre-train for additional steps (for a fair comparison in wall-clock time) and whether some of the gain still holds.\n- Typically, multiplicative noise is added to the input before computing gate scores e.g., in the Switch Transformer paper, which can improve performance. Can the authors compare against that version of the gate?", "other_comments_or_suggestions": "None", "questions_for_authors": "See weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qGTUQn6krc", "reviewer": "Reviewer_JD26", "summary": "This work proposes Autonomy of Experts (AoE) to replace the traditional MoE model. Instead of employing a router to select the experts, AoE computes the internal activations of all experts and select the best one to proceed. The authors conduct the pre-training experiments to investigate various properties of AoE and demonstrate promising results compared to the conventional MoE.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable comments! We hope our rebuttal helps address your concerns. If so, we would be grateful if you could consider increasing the overall recommendation.\n\n---\n&nbsp;\n\n# Contribution and Novelty\nThank you for listing [Pham et al.]. Our paper fundamentally differs from it. We believe our AoE presents a solid novelty and technical contribution. Here are the differences.\n\n### 1. Motivation, Architecture, and Method\n\n- AoE identifies and addresses a widely overlooked issue, i.e., the separation between the router's decision-making and the execution of experts. AoE is ***a novel MoE paradigm where the router is removed, allowing the experts to autonomously select themselves.***\n\n- In contrast, Pham et al. ***aim to enhance the router***. They take the experts' output as additional supervision signals to the router logits, which requires a complex training objective. Additionally, because they sometimes compute **every expert's *final* outputs**, this results in ***dense activation, making the model not technically even an SMoE.***\n\n### 2. Use of Activation Norms\n\n- [1] proposed that activation norms can be used to measure the knowledge of modules. Our approach is inspired by [1] and we cite them in Line 147. Pham et al. do not cite this paper or other related interpretability works. ***This perspective is not Pham et al.'s innovation, nor do we claim it as ours.***\n\n- One of our contributions lies in a novel MoE paradigm that selects experts without routers, using only intermediate activations, along with architecture designs for enhanced efficiency. ***This is not simply a matter of selecting which nodes to use for norm calculations or improving results.*** AoE avoids the need to compute every expert's final output, preserving the sparse activation characteristic of SMoE and making AoE significantly more efficient and practical.\n\n### 3. Efficiency\n\n- AoE achieves similar pre-training throughput of an SMoE.\n\n- Pham et al. did not report efficiency results. Since their model is sometimes densely activated, assuming the common 8-select-2 setting, ***AoE can be up to four times more efficient during pretraining.***\n\n### 4. Effectiveness\n\n- Pham et al. developed a complex training pipeline and only tested models up to **36M** parameters, with evaluation limited to BPC and PPL metrics.\n\n- AoE supports simple end-to-end training, validated at scale with models up to **4B** parameters and evaluated across various downstream tasks.\n\nAdditionally, ***Pham et al. publically acknowledged that their complex router-enhancement can not be reproduced***, making it impossible to use it as a baseline for comparison. This is stated in the first line of their README: https://github.com/giangdip2410/CompeteSMoE\n\nWe would be truly grateful if you could kindly re-assess our novelty and contribution.\n\n### References\n\n[1] Transformer Feed-Forward Layers Are Key-Value Memories, EMNLP 2021\n\n&nbsp;\n\n---\n&nbsp;\n\n# Response to Concern 1\nMost of this concern has been addressed in our initial response. Here, we address your question that \"the expert with the highest intermediate norm is not guaranteed to have the highest output norm.\"\n\n*A neural network works according to how it is trained. Since AoE is not trained like a traditional MoE, this guarantee is unnecessary.* As mentioned in the last paragraph of Sec.3.1, we train AoE to represent its awareness of its capabilities through the norm of the intermediate node. To support this claim, we present a new experiment: during inference, when using Final Output Norms  (FON) to select experts in a pre-trained AoE (Config.10), rather than the intermediate norms used during training, there is a performance drop across all tasks:\n||ARC-E|PIQA|SIQA|WINO|HELLA|MNLI|QNLI|SST2|AVG.\n|-|-|-|-|-|-|-|-|-|-\n|AoE|41.16|58.32|36.80|53.04|28.37|32.78|50.61|54.59|44.46\n|AoE (FON)|37.33|56.75|35.41|51.54|27.77|32.00|50.16|54.47|43.18\n\nThis is a valuable question, as many readers who are accustomed to the traditional MoE models may also have similar concerns. We will include this response in the paper. Thank you very much.\n\n&nbsp;\n\n---\n&nbsp;\n# Response to Concern 2\nThe word \"decompose\" in Line 202 might be ambiguous. We mean that we modify the model architecture, replacing the $W_g$ in traditional MoE with two low-rank matrices. The architecture is illustrated in Fig.1(b), which does not require dynamic adjustment during runtime. We will improve the clarity in Line 202. Thank you for your feedback.\n\n&nbsp;\n\n---\n&nbsp;\n# Response to \"Relation To Broader Literature\"\nMoE is the foundation of advanced LLMs, such as DeepSeek-R1, Qwen-2.5-Max, Grok, etc. A deeper understanding and innovation in MoE could help advance future LLMs. We will highlight the necessity and potential impact of studying MoE in the NLP field.\n\n&nbsp;\n\n---\nIf you have further questions, please leave more comments. We are committed to addressing them. Your assessment is very important to us! Thank you very much!", "claims_and_evidence": "The claims are generally supported.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "- The paper does not contain any theoretical claims", "experimental_designs_or_analyses": "The experimental designs are sound. The authors conducted the pre-training experiments to train AoE and SMoE from scratch, and evaluated them on several downstream tasks. This is the standard experimental setting.", "relation_to_broader_scientific_literature": "The scope of this work is quite limited as it only focuses on the design of the MoE expert selection mechanism. The paper did not discuss how this idea can impact related research disciplines.", "essential_references_not_discussed": "While the AoE architecture design is relatively new, its expert selection mechanism has been observed in a previous study of CompeteSMoE [1], which also proposed to selected the best experts via its activation norm.\n\n[1] Pham, Quang, et al. \"CompeteSMoE--Effective Training of Sparse Mixture of Experts via Competition.\" arXiv preprint arXiv:2402.02526 (2024).", "other_strengths_and_weaknesses": "I have two major concerns regarding AoE\n\n- First, while using the expert's activation norm for selection has been explored, AoE instead uses the intermediate's activation norm. Could the authors provide more justification into why this design could achieve a better result? Specifically, it is not guaranteed that expert with the highest intermediate norm will also have the highest output norm. \n- Second, the training pipeline for AoE is not clear. Since AoE requires decomposing the expert weight into an up and down projections, how are the parameters updated throughout training, and does AoE decompose the expert weight after each update?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- Please clarify the key contribution in the expert selection strategy in AoE compared to previous studies such as CompeteSMoE.\n- Please clarify the advantages of using the intermediate norm for expert selection and the training dynamic of AoE, as elaborated in the Other Strength and Weaknesses section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Paper #2647
This paper proposes a new scheme to select expert sublayer in the mixture-of-experts (MoE) language model. Rather than empoying a router layer to choose the expert to process incoming embedding, the method utilizes a factorized subcomponent of feed-forward layer to calculate the importance score ("norm" in the paper) that are fed to a softmax and top-k to choose the expert to be activated. [Reviewer yvWh]
The core innovation of AoE is to eliminate the router entirely and allow experts to autonomously determine whether they should process inputs. I quite like the author's initial findings in existing findings that justify the development of AoE and also the toy illustration of AoE behaviors in the appendix [Reviewer 44vv]
In AoE, routers are eliminated; experts pre-calculate internal activations for inputs and are ranked according to their activation norms. Only the highest-ranking experts continue with the forward pass, while the others are terminated [Reviewer BPoe]
The authors conduct the pre-training experiments to investigate various properties of AoE and demonstrate promising results compared to the conventional MoE. [Reviewer JD26]

Though the results show better outcomes on the proposed method, it is still unclear whether the proposed method is still better against other strategies to select experts (e.g., those cited in the last paragraph in the section 2. [Reviewer yvWh]
The paper does not present formal theoretical proofs but provides conceptual explanations for why AoE is effective. [Reviewer 44vv]

All reviewers acknowledge the authors rebuttals.
Decision: Accept (poster)