Unveiling Super Experts in Mixture-of-Experts Large Language Models
Authors: Zunhai Su
Qingyuan Li
HaoZhang
Weihao Ye
Qibo Xue
Yulei Qian
Ngai Wong
Kehong Yuan
Venue: ICLR 2026 Poster (2026)
Keywords: MoE
LLM
compression
attention
Primary area: foundation or frontier models, including LLMs
TL;DR: In this work, we present the first identification and systematic study of a distinct subset of experts, termed Super Experts.  We analyze their characteristics, distributions, and critical functional roles within MoE LLMs.

Abstract: Leveraging the intrinsic importance differences among experts, recent research has explored expert-level compression techniques to enhance the efficiency of Mixture-of-Experts (MoE) large language models (LLMs).
However, existing approaches often rely on empirical heuristics to identify critical experts, while lacking a deeper understanding into the heterogeneous importance of experts and the inner workings of MoE LLMs.
In this study, we report, for the first time, the discovery and systematic investigation of a distinct subset of experts that play a pivotal role in the model's forward inference.
These experts are prevalent in open-source MoE LLMs, and despite their extremely limited number, pruning them results in a substantial decline in model performance (e.g., prune just three out of 6,144 causes Qwen3-30B-A3B to generate repetitive and uninformative outputs).
We refer to these experts as Super Experts (SEs).
Our comprehensive analysis provides progressively deeper insights into SEs: 
(i) SEs are characterized by rare but extreme activation outliers in the output of the down\_proj, which give rise to massive activations in the
hidden states between decoder layers.
Moreover, the distribution of SEs is model-specific, data-agnostic, and remains unaffected by post-training processes.
(ii) By pruning SEs, we assess their significance across a variety of tasks, revealing their considerable impact on the model's overall performance, particularly in mathematical reasoning.
(iii) We further investigate why compressing SEs exerts such a pronounced impact. 
We show that, in MoE LLMs, SEs serve as the primary source of the systematic outlier mechanism in Transformers, and that compressing them profoundly disrupts this process, ultimately causing the collapse of attention sinks. 
These findings advance the understanding of the internal dynamics of MoE LLMs, filling an important gap in the current knowledge.
In addition, we developed an automated tool for rapid and accurate SE profiling.
The code is provided in the supplementary materials.

Published as a conference paper at ICLR 2026
UNVEILING SUPER EXPERTS IN MIXTURE-OF- EXPERTS LARGE LANGUAGE MODELS
Zunhai Su', Qingyuan LiP, Hao Zhang?, Weihao Ye Qibo Xue Yulei Qian', Yuchen Xie?,
Ngai Wong', Kehong Yuan'
TTsinghua University MMeituan 'Xiamen University 'The University of Hong Kong
*Nanjing University
ABSTRACT
Leveraging the intrinsic importance differences among experts, recent research has explored expert-level compression techniques enhance the efficiency of Mixture-of-Expers (MoE) large language models (LLMs). However, existing approaches often rely on empirical heuristics identify critical experts, while lacking a deeper understanding into the heterogeneous importance of experts and the inner workings of MoE L.MMs. this study, we report, for the first time, the discovery and systematic investigation of distinct subset of experts that play a pivotal role the model's forward inference. These experts are prevalent open- source MoE LLMs, and despite their extremely limited number, pruning them results a substantial decline in model performance (e.g., prune just three out 6,144 causes Swen3-30B-AAe to generate repetitive and uninformative outputs). We refer to these experts as Super Experts (SEs). Our comprehensive analysis provides progressively deeper insights into SEs: (i) SEs are characterized by rare but extreme activation outliers in the output of the down_proj which give rise massive activations in the hidden states between decoder layers. Moreover, the distribution of SEs is model-specific, data-agnostic, and remains unaffected by post-training processes. (ii) By pruning SEs, we assess their significance across a variety of tasks, revealing their considerable impact on the model's 's overall per- formance, particularly mathematical reasoning. (iii We further investigate why compressing SEs exerts such pronounced impact. We show that, in MoE LLMs, SEs serve as the primary source of the systematic outlier mechanism in Trans- formers, and that compressing them profoundly disrupts this process, ultimately causing the collapse of attention sinks. These findings advance the understanding of the internal dynamics of MoE LLMs, filling an important gap in the current knowledge. In addition, we developed an automated tool for rapid and accurate SE profiling. The code is provided in http:://giiuhh.cmm//uuaaSSuuuuuu Experss Profilling
INTRODUCTION
Sparsely activated Mfxture-of- Ex-perss (MoE) models employ dynamic routing and sparse activa- tion, demonstrating significant potential in enhancing the learning capacity of large language models
(LLMs) (Cai et al. 2024 Mu & Lin 2025). This paradigm has led to the development of state-of-
the-art MoE LLMs, including DeepSeek (Guo et al.] 2025 Liu et al. 2024). Qwen (Yang et al.]
2025a). LongCat-Flash (Team al. 2025b ab and others. Despite their potential, significant chal-
lenge stems from their large parameter size and high computational cost (Li et al. 2023 Lu et al.)
2024 Chowdhury et al. 20024), which present considerable obstacles for deployment. Model com-
pression techniques, such as quantization (Frantar et al.) [2022] Xiao et al. 2023a Su et al. 2025a)
pruning (Frantar& & Alistarh 2002] (Snn al.) (202)) and others (Zhu al. 2024 Wang al. 2024
enable the development of more compact and computationally efficient models.
Beyond LLM-oriented compression approaches, expert-level compression methods have been de-
veloped by leveraging the structural characteristics of MoE models and the uneven importance of
experts induced by training strategies (Chowdhury et al. 2024 Chi et al.) 2022 Lu et al. 2024).

---

Published as a conference paper at ICLR 2026
Specifically, it employs various expert impor- tance metrics to guide the pruning, merging, or
Super Experts
(num=3, PPL-59.86)
skipping of less critical experts (Lu et al.) [024]
Jeg/Porplexte
Non-Super Experts
Huang et al.) 2025 Xie et al. 2024). priori- tize more important ones by assigning higher bit budgets during quantization (Duanmu et al. 2025| Li et al. 2024), and allocate more ranks in low-rank decomposition (Yang et al. 2024] Li et al.] 2023). For instance, several works evaluate expert importance by measuring acti- vation frequency or by analyzing router scores produced within MoE layers (I.i et al.) [024] 2023 Huang et al., 2025). Additionally, recon-
A5
Baseline:
(num-0, PPL-8.70)
(num-1000) PPL-10.
200
300
700
890
900
1999
Number of Pruned Experts
Figure 1: Analysis of experts pruning on Qwen3- 30B-A3B using the WikiText-2 dataset. Pruning three Super Experts results a significant degra- dation in Perplexity (PPL).
struction loss and other similarity-basee met- rics have been utilized in previous studies (Lu
al. 2024 Duanmu al.
2025 Zhang al. 2024
Analyzing expert importance not only facilitates model compression but also provides deeper in- sights into the inner workings of MoE LLMs (Zhang al. (2026). However, existing approaches often rely on empirical criteria to identify critical experts, lacking a deeper exploration and under- standing of the heterogencous importance among experts. In this study, we address a fundamental yet previously overlooked question: Is there a small subset of distinct experts that plays an excep- tionally critical role in the underlying mechanisms of MoE LLMs?
Through comprehensive analysis of various open-source MoE LLMs, we consistently confirm the existence of such experts. Despite their extreme limited number, pruning these experts leads to a significant collapse in model performance. As shown in Figure I pruning just three experts from Owen3-30B-A3B leads to significant degradation model performance, while randomly pruning other experts results in a considerably smaller impact. We refer to these experts as Super Experts (SEs), and our comprehensive analysis provides progressively deeper insights into SEs.
In Section 3 we first characterize SEs and analyze their distribution across various models and input data domains. SEs are identified by extreme activation outliers in the output of the down-proj, which induce massive activations (MAs) (Sun et al., 2024). Intriguingly, the distribution of SEs remains model-specific, aata-agnostic, and the SEs in the base model maintain consistency after post-training processes. In Section 4 assess the importance of SEs by quantifying performance degradation following their dynamic pruning. Notably, pruning SEs leads to a complete performance collapse with Pass @ dropping to nearly zero on tasks such as AIME and Math-500 (AIME) 2024]
2025 Lightman et al. 2023) for reasoning LLMs. In Section we further deepen our understanding of SES by revealing their origin in the behavior of systematic outliers mechanism in Transformers
(Su & Yuan 2025; An et al. 2025). Our findings confirm that MoE LLMs rely on SEs to induce attention sinks, which are crucial for the distribution of attention scores and must be preserved during
sparse attention or KV compression (Xiao et al. 2023b Su et al. 2025a).
The main contributions of this work are summarized as follows:
We provide the first comprehensive characterization of Super Experts (SEs) in MoE LLMs, an exceptionally rare yet fundamentally critical subset of experts, thereby filling significant gap in the existing understanding of MoE LLMs. Extensive analyses across multiple models and tasks reveal key properties of SEs, including their stable distribution and critical impact on model performance.
We demonstrate that SEs are the primary drivers of systematic outliers in Transformers. In MoE L.Mss, their strong activation on attention sink tokens makes them the fundamental source of these outliers, and compressing them severely disrupts this process, ultimately leading to the collapse of attention sinks.
Our findings on SEs provide new insights into the internal dynamics of MoB LLMs and the het- erogeneous importance of experts. These insights serve as a foundation for designing more expert- balanced pre- pre-training regimes and for advancing robust expert compression strategies.
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARIES ON MOE LLMS
MoE LLMs. LLMs are typically structured as a stack Transformer decoder blocks (Vaswani al. 2017), each consisting of multi-head self-attention (MHSA) layer and feed-forward net- work (FFN) layer. In MoE LLMs, the FFN layers are replaced by MoE layers, where each layer consists of multiple experts, each represented by a FFN. A concise overview of MoE LLMs is pre- sented in Figure Let H E R= represent the input to the first decoder, where is the embedding dimension, and n the length of the tokenized input sequence. Then, the output of the I-th decoder block, H' E IR is given by:
H'= = MoE [LN (H'))
+ H",
(1)
H" = H' = MHSA (LN (LNmhso mhsa (H' 1H)
(2)
where 1 S/S L, with L denoting the total number of blocks. LN refers to layer normalization, O representing the output of the MHSA, and H² denoting residual summations after the MHSA.
MoE Layer. The hidden representation after MHSA, H" passes through LN and then fed into the MoE layer. First, the router network determines which experts activate and how to scale their outputs through the weight matrix Wa. The routing weights G E gnx are com- puted as:
out
down
Attention
RoPE
Zop-A
router
gate
LayerNorm
LayerNorm
softmax HH" WG).
(3)
Then, sparse activation the experts is achieved by selecting the top-k routing weights for each input token. The output of the activated experts is scaled by the routing weights and aggre- gated to form the output of the MoE layer:
Figure 2: Decoder Architecture of MoE. LLM.
Σ
G.. FFN (ANmoc((H))) Vj = In..n,
FFF
(4)
GG
where Top-k(G,) denotes the indices the top-k routing weights for the j-th input token. The FFN is defined as:
FFN (X) = (a (XW2) @ XW.) Wa.
(5)
where Wg, W., and Wa the weight matrices for the gating, up-projection, and down-projection, respectively, denotes the activation function, and @ represents the Hadamard product.
3 SUPER EXPERTS: DISCOVERY AND LOCALIZATION
In this section, we first demonstrate the discovery process SEs using Qwen3-30B-Ar 30B-BBBBBBBBBB- as an example. Next, analyze SEs across different MoE LLMs and data domains to examine their distribution patterns and highlight the widespread presence SEs.
3.1 SUPER EXPERTS INDUCE MASSIVE ACTIVATIONS
Recent research has explored a distinct class of extreme activation outliers in LLMs, which appear in the hidden states between decoder layers and are known massive activations (MAs) (Sun al.) Guo et al. 2024). They are limited number, yet their values are orders magnitude larger than those of other activations (c.g., up to 100,000 times larger). The discovery of SEs arises from an exploration and analysis of the formation MAs (Sun et al. 2024) in MoE LLMs. Existing research has yet to clarify how these MAs arise in MoE LLMs. Do these activations arise from the collective activity all activated experts, are they primarily driven by some specific experts, or are they instead caused by other components of the model?
2024
Through analysis of several prominent ooun-source MoE LLMs (e.g., Qwen series, DeepSeek series, Mistral), surprisingly find that a small subset of experts consistently produces extreme activation
3

---

Published as a conference paper at ICLR 2026
Layer 1
Gitipe Magalledas
Layer 2
+
Mas Gusper Naguitada
Layer
Ghipps Megnttaicoll
E
Expert
&
Expert
Expert
Layenfoom
Layeniarm
Figure 3: SEs mechanism in Owen3-33B-AR 30B-A3B. The line plots depict the maximum output magni- tudes of down. proj for experts 68/92/82 across layers. Massive activation is gradually amplified through expert 68 layer 1, expert 92 layer 2, and expert 82 in layer Extreme activation out- liers from these SEs are propagated into the hidden states between decoders via residual summation, progressively leading to massive activation.
Rise
2d
Miaa
get Rie
1" Rie
Ris
Prane
Laner
Prane Six Laver
Original Model
Prune the SEs in Lover
18288
Figure 4: Impact of SEs pruning MAs in Qwen3-30B-A3B MAs are computed using 100 input samples from the C4 (Raffel et al. 2020) dataset, each with length of 2K.
outliers in the output of their down_proj layers. These outliers are subsequently passed onto the hidden states via residual summation after the MoE layers, leading to MAs. The entire process is illustrated Figure using Qwen3-30B-A33 as example. This phenomenon typically occurs in a single layer (e.g., Mixtral) in just few layers (e.g.. Qwen3-30B-A3B, starting from the initial decoder layers, ultimately leading to stable MAS across nearly all subsequent layers. To directly validate this mechanism, also perform ablation experiments by dynamic pruning the SEs in Qwen3-30B-A3B As illustrated in Figure [ pruning SEs from single layer effectively eliminates their contribution to MAs. Furthermore, when all SEs pruned, MAs are completely eliminated, confirming that they are directly generated by SEs.
3.2
LOCALIZATION OF SUPER EXPERTS
3.2. SUPER EXPERTS PROFILING
Given that SEs are defined by their influence the formation of MAs through the extreme acti- vation outliers they generate, we propose the following broad yet effective quantitative definition. Specifically, we compute the maximum output magnitudes to the down_pro> for all experts across all layers. Let denote the set of layers responsible for the formation of MAs. Let ar,e denote maximum output magnitude to the down-pro of expert in layer I, and let A {ar,e) } be the set of all such values across the entire model. An expert in layer classified as a SE if:
a1,e Poo.5 and 0t,e
10 Gmax and l L
where Po9.5 = Percentilego. (A) and Omax = max A. This criterion motivated by the heavy- tailed distribution of dt.e and effectively identifies the experts of interest across various MoE LLMs, as highlighted in bold in Tablc[]] Additional analyses are provided in Appendix[[ No specific dataset designated for identifying SEs, since we later demonstrate that their distribution remains stable across different input datasets. The pseudocode of SEs profiling presented in Appendix L We have developed an automated tool for rapid and precise SE profiling based this definition. The wwt.://wwww..oouuuuuuuuuuuuueeee Ewwertspprofiiiiiieeee
code is provided in
3.2.2 DISTRIBUTION OF SUPER EXPERTS ACROSS MODELS AND DATA DOMAINS
We select three representative MoE LLMs with distinct designs for analysis: Qwen3-30B- A3B, DeepSeek-V2-Lite--ee and Mixtral-sx7B--Instnttt--...1 We also include the base

---

Published as a conference paper at ICLR 2026
Table I: Activations identified as SEs are highlighted in bold, based on results from the C4 dataset.
Model
Total
SE Count
Proportion
Top
Top
Top
0.5%
...1
Owen 3308-A3B
DeepSock- -R
[6677
10
0.066%
616.0
536.0
171.0
143.0
143.0
67.8
36.75
61.6
DeepSeck-V BeeySeek-VV-Lite-CCC Vi-uitt-Chr
1782
2
0.11%
1424.0
462.0
112.5
89.5
37.5
24.0
34.5
142.4
Bixtral-Bx7--nnstrut--O00
1030
1900
Layer
600
500
Expert
127
Expert
127
(a) Qwen3-3085 Beenn-30B--3BB
(b) Qwen3-30BB 1aan3-30B-ABBB 30B-A3B-BBse
0
500
Layer
Layer
500
400
420
200
200
Expert
Expert
(c) DeepSeek- V2-Lite-Chat
(d) DeepSeek V2-Lite.
1030
1000
800
Layer
800
500
400
31
31
Expert
Expert
(e) Mixtral-8x7B-vO, 1-...7---......Itreee
(I) Mixtral-&x7B-vO.1.
Figure 5: Heatmap visualizations of the maximum output magnitudes from the down.proj for each expert across layers. SEs are highlighted with arrows.
models of these three LLMs to illustrate the impact of post-training processes. Al- though of these models are MoE LLMs, they exhibit distinct design differences.
For instance, Qwen3 and Mixtral do not em- ploy shared experts, whereas DeepSeek does. DeepSeck-VV-Lite adopts hybrid architec- ture. wherein the first layer utilizes dense MIPs, while the remaining layers are based MoE blocks. Through the proposed SE profil- ing tool, we identify the SEs these models using the C4 (Raffel et al. 2020) dataset, A summary of the SES is provided in Table and heatmap visualizations of the maximum output magnitudes from the down proj j are shown in
Table 2: SEs of several MoE LLMs.
Super Experts
Bwen3-30B-A3B
Layer Expert 68, Layer Exper 92, Layer Expert 82
Soon3-30B-A3B-BBB
DeepSeck-V2-Lie-000 V2-aite-hht
Layer Expert 54, Layer Expert 38
DeepSeck-V22 V2-Lite
Battral-8x7B-nnstrccse
Layer Expert 3
Mixtral 8x7--v0.
Figure [ The key conclusions regarding SEs are summarized as follows: (i) SEs are consistently present across the investigated models, accounting for less than 0.5% of all experts. (ii) After post- training processes, the distribution of SEs remains unchanged compared to the base model. Addi- tional results on the distribution of SEs across training stages is provided in Appendix K Moreover, some experts in the final layers also exhibit extreme activation outliers. However, since they do not contribute to the formation of MAs, they do not hold the same level of significance as SEs. Additional results are available in Appendix C
In addition to the C4 dataset, we also analyze SE distributions across several other datasets, including WikiText-2 (Merity et al. 2016), C-Eval (Huang et al. 2023), GSM8K (Cobbe et al. 2021). and

---

Published as a conference paper at ICLR 2026
Table 3: Evaluation of the importance of SEs in non-reasoning models, The results of random pruning are obtained by averaging the performance over five runs.
Hella Swag
Open BookQA
Wino Grande
Wiki
Model
Setting
Arg.
ARC-c
ARC-e
BoolQ
GSMBK
MMLU
PIQA
PPL
Baseline
70.22
52.65
79.50
88.72
89.61
59.63
77.82
34.20
79.33
70.56
8.70
Prune SEs
55.00
46.08
76.05
70.73
42.38
39.31
56.03
29.80
72.52
62,12
59.86
Qwen3
Drop Rate (%)
20.28%
28.004
12.87%
8.58%
30B-A38
Random
70.36
52.73
79.46
88.59
89.84
59,50
77,84
34.00
79.76
71,51
8.71
Drop Rate (%)
-0.20%
-0.15%
0.05%
0.15%
-0.26%
0.22%
-0.03%
0.58%
40.54%
-1.35%
LC Random
70.21
52.60
79.55
88.63
89.55
59.45
77,80
34.00
79.43
70.86
8.70
Ratc (%)
0.01%
0.09%
-0.06%
0.10%
0.07%
0.30%
0.03%
0.58%
-0.13%
-0.43%
Baseline
60.27
46.59
78.37
79.79
37,83
58.75
55.03
34.60
80.30
71.19
6.31
Prune SEs
43.90
29.27
54,92
68.62
9.78
43,72
41.77
21.00
68.28
57.7
10.75
DeepSeek
Drop Rate
27.17%
37.18%
29.92%
14.00%
74.15%
25.56 25.58%
24. 10%
39.31%
14.97%
18.95%
V2-Lite
Random
60.30
46.50
78.45
80.37
37.38
58:77
55.10
34.40
80.
71.59
6.31
Drop Rate (%)
-0.05%
0.19%
-0.10%
-0.73%
1.19%
-0.03%
-0.13%
0,58%
0,20%
-0.36%
Random
60.18
79.83
55.05
6.32
Drop Rate (%)
0.15%
0.21%%
0.20%
00.0%%
1.61%
0.07%
-0.4%
0.58%
0.07%
-0.10%
85.02
67.83
35.60
$2.48
3.84
Prane SEs
49 38
36.01
64.44
75.66
24.34
50.6
42,47
20.60
73.12
57.22
6.23
Mixtral
Drop Rate (%)
27.21 %
36.34%
23.52%
11.01%
57.54%
22.02%
37.39%
42.13%
11.35%
25.26%
8x7B-v0.1
Random
67.82
56.57
84.09
85,23
58.15
64.92
68.08
35.00
82.21
76.16
3.86
Drop Rate (%)
0.02%
0.009
0.20%
-0.25%
-1.45%
-0055%
0.37%
1.69%
0.33%
0.52%
Random Drop Rute (%)
56.48
84.16
85,
57.83
64.73
67.52
81.14
76.55
3.85
0.24%
16%
0.12%
-0.14%
-0.8%%
0.25%
0.46%
0.00%
1.62%%
0.01%
Table 4: Evaluation of the importance of SEs in DeepSeek- R1.
Model
Setting
Avg.
GPQA Diamond
Math-500
AIME 2024
AIME 2025
LiveCodeBench
Wiki
Pass
Pass
Pass
Pass#1
Pss
Raseline
75.64
71.50
97.60
79.33
66.33
63.44
3.33
Prune SFs
1.81
5.05
4.00
0.00
0.00
0.00
5.18
Drop Rate (%)
97.61%
93.0%
95.9%
100%
100%
100%
DeepSeek RI
Random Pruning
75.53
72.63
98.00
77.67
67.00
62.37
3.35
Drop Rate (%)
0.15%
-1.58%
-0.41%
2.09%
-1.01%
1.69%
LC Random Pruning 75.31
71.30
98.00
78.67
67.00
62.37
3.36
Drop Rate (%)
0.17%
0.00%
-0.41%
0.83%
-1.01%
1.96%
HumanEval (Chen et al. 2021). As shown in Appendix
D
the distribution of SEs remains highly
stable, regardless of variations in the input data domain.
4 THE IMPORTANCE OF SUPER EXPERTS
In this section, we assess the importance of SEs by measuring the performance drop caused by dynamically pruning them (i.e., skipping the experts when selected by the router). We use the original model and results from random pruning of an equivalent number of experts as baselines. Random pruming is implemented in two ways: globally across all layers, or within the same layers as the SEs, which we refer to as layer-controlled (L.C) random pruning. To more effectively evaluate the importance of SEs, we utilize distinct benchmark types for non-reasoning and reasoning models.
4.1 IMPACT NON-REASONING MODELS
For non-reasoning models, we select three models: the non-thinking mode of Qwen3-30B-4 A3B, DeepSeek-V2.2 V2-Lite and Mixtral-&x7B-vi))
We utilize the datasets listed below and conduct eval- uations using Im-eval (Gao et al.) 2024b), including ARC-challenge (ARC-c). ARC-easy (ARC-e) Clark et al. 2018), BoolQ (Clark al. (01 GSM8K (Cobbe et al.) 202 HellaSwag (Zellers) et al. 2019) MMLU (Hendrycks et al. 2021). OpenBookQA (Mihaylov et al. 2018), PIQA (Bisk et al. 2020). and WinoGrande (Keisuke et al. 2019). As shown in Table 3 pruning only a few SEs leads to significant degradation across all tasks, with average accuracy dropping by 21.68% to 27.21% %. In particular, for GSM8K, the degradation ranges from 52.71 1 % to 74.15%. In contrast,
random pruning has negligible impact, underscoring the crucial role of SEs.

---

Published as a conference paper at ICLR 2026
Table 5: Evaluation of the importance SEs in Qwen3-30B-A3B.
Model
Setting
Avg.
GPQA Diamond
Math-500
AIME
AIME 2025
HumanEval
Wiki
Puss
Pass@
Pass I
Puss
Pus
PPL
Baseline
69.37
61.62
88.00
80.00
73.33
43.90
8.70
Prune Stis
4,02
18.69
1.40
0.90
0,00
0.00
59.86
Drop Rate (%)
93.62%
69.7%
98.4%
100%
100%
100%
Qwen1-30R.A3BB
Random Pruning
69.33
61.62
89.00
80.00
73.33
42.70
8.71
Drop Rate (%)
0.06%
0.00%
-1.10%
0.00%
0.00%
2.7%
I.C. Random Pruning Drop Rate (%)
68.97
61.62
88.00
80.00
73.33
41.90
8.72
0.58%
0.00%
0.00%
0.00%
0.00%
4.56%
4.2 IMPACT ON REASONING MODELS
For evaluating the importance of SEs in reasoning models, we select DeepSeek-RI and the thinking mode of Qwen3-30B-A3B We select benchmarks more suitable for testing reasoning models and conduct evaluations based on the EvalScope (Team] 2024). The generation configurations align with the corresponding technical reports of the models. These benchmarks are: (i) General Tasks: We use GPQA-Diamond under a 5-shot setting. GPQA (Rein et al. (022) is a challenging dataset of multiple-choice questions authored by domain-specific multidisciplinary experts.
(it) Math &
Text Reasoning: To evaluate mathematical and logical reasoning skills, we use high-level math benchmarks, including MATH-500 (Lightman al. 2023), AIME'24 24 (AIME) 2024). and AIME'25 (AIME) 2025). (iii) Agent & Coding: To test the model's proficiency in coding and agent-based
tasks, we use LivcCh (Jain et al. 2024) and HumanEval (Chen al. 2021).
The results, presented in Tables Talles4anne and show that pruning the SEs causes significant performance degradation, while random pruning has almost no impact. The Pass@1 scores for most tasks drop to zero, highlighting the critical role of SEs. During the review of model responses the Math-500 benchmark, we made a striking observation: after pruming the SEs, the model consistently generated repetitive responses in nearly every test, continuing until it reached the maximum output length, as shown in Table TT and [2] This behavior suggests that the model loses its ability to reason and solve problems entirely after SE pruning, with additional discussion this part provided in Appendix H More results are in Appendix E
5 UNDERSTANDING THE IMPACT OF SUPER EXPERTS COMPRESSION
Why SEs so critical to MoE LLMs? In this section, first reveal SEs the primary source of systematic outliers in MoE LLMs. Then, we examine how compressing SEs affects the attention mechanism, providing both an in-depth understanding and quantitative analysis.
5.1 SUPER EXPERTS AS THE ORIGIN OF SYSTEMATIC OUTLIERS IN MOE LLMS
Previous studies (Su & Yuan 2025 An al. 2025 have shown that Transformer-based dense L.MSs exhibit systematic outliers. These outliers appear in multiple forms, including weight outliers (also referred to as super weights YYu al. 2024)), activation outliers (encompassing both activation spikes and MAs (Yang et al. 2025b Sun et al. 20244), and attention outliers (commonly known as attention sinks (ASs) (Xiao et al.) (023b)). Importantly, such outliers emerge, stabilize, and vanish in systematic fashion, and they are crucial to the model's 's overall performance (Yu et al. 2024 Sun al., 2024; Xiao al. 2023b). A detailed discussion of related work on systematic outheers in Transformers provided in Appendix B
Building on prior research and our own findings on SEs, we demonstrate that SEs constitute the fundamental source of systematic outliers in MoE LLMs. Specifically, using Qwen3-10B-A3B as an example, the router scores assigned to SEs for the first token (which also serves as the atten- tion sink token) are exceptionally large, whereas for non-sink tokens the scores are more evenly distributed across experts, as shown in the visualization calibrated on the C4 dataset in Figure 6 This routing behavior of SEs ensures that the attention sink token is strongly activated the SEs. Notably, this behavior is independent of the input dataset (see Appendix F). which also explains why SES distributions model-specifie. The sink token subsequently produces activation outliers

---

Published as a conference paper at ICLR 2026
Oweer 208 Layeet Adlaabe Duter Stee
See
24-
(a) Sink token in Qwen3-30B-A3B.
(b) Non-sink tokens in Owen3-30B-A3BB
be Tsper De Yeas 303)
Seade Ieow Stannt Eateatn Tean
(c) Sink token in DeepSeek- V2-k V2-LLite Lite.
(d) Non-sink tokens in DeepSeck- V2-Lite.
Marral ecto aver Exper Aotvattor O butor of Boen
aborra Aborat Layer Exper Activatce Dicributces ofaan Sinc Beanic4t
(c) Sink token in Mixtral-8x7B-vdl
(f) Non-sink tokens in Mixtral-&x7B-vO..t
Figure 6: Expert router score distributions for sink and non-sink tokens, based on calibration using the C4 dataset. Additional experimental results are provided in Appendix
layer in
Exp 292
Exp dowa_sot
layer_our
Fiit
Exp 492 down
Layer 2
Layer
Figure 7:
Systematic outlier mechanism in a single layer of Qwen3-30n Seon333B--AB. using the input: "Green trees sway under gentle wind." The complete illustration is provided in Figure 22
in the output through weight outliers in the down-proj. Through the residual connections, these outliers propagate into the hidden states as MAs. At the attention layers, such tokens then attract dis- proportionate attention and ultimately emerge attention sinks. Unlike dense LLMs, where such behavior typically occurs within single layer (An et al.) 2025; Su & Yuan 2025). MoE models exhibit the progressive formation of systematic outliers by SEs across multiple layers. The overall process is illustrated in Figure 7 More detailed analyses of this process are presented in Appendix G Additional cross-domain analyses of SEs and attention sink tokens are provided in Appendix
Within the dynamics of systematic outliers in MoE LLMs, SEs constitute the primary source, MÁs act as the intermediate bridge, and ASs manifest their effects within the attention mecha- nism. This analysis underscores the pivotal role of SEs in the internal mechanisms of MoE LLMs and elucidates the distinctive manifestation of systematic outliers Transformers within the MoE paradigm. Further weight-level experiments and interpretability analyses SEs presented in Appendix H
5.2 SUPER EXPERTS COMPRESSION DISRUPTS ATTENTION SINKS
Given that SEs act the primary source of systematic outliers and ASs embody their final mani- festation, we posit that compressing SEs interferes with AS formation, thereby causing significant deterioration in model performance. StreamI (Xiao al. 2023b) identified ASs in LLMs, in
8

---

Published as a conference paper at ICLR 2026
Layer 25 Head 10
35 Heed 10
Layer 25 Head 10
Layar Hewd 10
(a) Layer 25 Head 10.
(b) Layer Head 10.
(c) Layer Head
(d) Layer 35 Head 10.
Figure 8: Attention scores of Swen3-30B-A3B Figures (a) and (b) depict the attention maps of the original model, where the first token clearly functions as an AS, consistently attracting the majority of attention. Figures (c) and (d) illustrate the attention scores following SE pruning, where the AS completely disappears.
which large fraction of attention is drawn to only few sink tokens (typically the first token). Although ASs often emerge at semantically insignificant tokens (Gu et al. [024]] Guo et al. 2024).
the mechanism itself is critical for model performance. In efficient LLM techniques such as sparse
attention and KV cache compression (Xiao et al. 2023b] Su & Yuan 2025 Su et al. 2025b: Xiong
al. 2025), maintaining ASs is essential for preventing undesirable distributional shifts of attention scores.
To validate this insight and quantitatively assess the impact of SEs compression, we introduce Attention Sinks Decay Rate, denoted siink It is defined as the average decay rate of ASs across all heads:
Atteation Sink Decay Rate (Qwen3-30B-A3B)
100%
90%
After Pruning Super Experts
60%
40%
Eies pf H h=1 EiESP P
H Σ
20%
Dsink = 1
10
Layer
where И is the total number of heads, p rep- resents the attention score between the Query Figure 9: Daink of Qwen3-30B-AA. across layers. token t and the Key token before SEs pruning, p denotes the attention score after SEs pruning. and S refers to the set of sink tokens. We evaluate Diink Qwen3-30B-A3B with the C4 dataset, identifying the first token the attention sink token. As shown in Figure 9 after SE pruning, the Dann remains consistently high, at approximately or exceeding 90%, demonstrating a substantial disruptive effect ASs. Figure visualizes the attention scores for several heads before and after pruning SEs, highlighting the complete disap- pearance of ASs following SE pruning. Notably, ASs introduce implicit attention biases (Sun al., 2024 An et al. 2025) that persist across all subsequent tokens and may encode global or other crit- ical information (Dareet et al.] (023). Consequently, the impact of SEs compression on attention computation remains both continuous and significant.
6 RELATED WORK ON EXPERT-LEVER COMPRESSION
M-SMoB (Li et al. 2023) performs expert merging by using activation frequencies to consolidate less significant experts. while also applying low-rank techniques to the merged experts achieve further compression. NAEE (Lu et al. 2024) introduces alug-and-play pruning and skipping meth- ods that leverage reconstruction loss to selectively compress less critical experts. MC (Huang al.) 2025) harnesses the significance of both experts and tokens to perform mixed-precise
quantization and dynamic expert pruning, achieving extreme compression. MC-Suite (Jaiswal et al. 2025) re- views various empirical criteria for identifying critical experts, considering four dimensions: weight, expert behavior, intermediate activations, and gradient behavior. Besides pruning-base based methods, there also few works that specifically study quantization in MoE LLMs (Duanmu et al. 2025 Zheng et al. 2025 Hu et al. 2025). While these methods examine expert importance from various perspectives to optimize expert compression, they lack deeper exploration and understanding of the

---

Published as a conference paper at ICLR 2026
mechanistic importance of specific experts. This study constitutes the first systematic characteriza- tion of SEs, analyzing their properties, functional impact on attention mechanisms, and contribution to systematic outliers, thereby filling critical gap in current understanding of MoE LLMs.
CONCLUSION AND FUTURE WORK
In this work, present the first systematic identification and comprehensive characterization of a distinct and exceptionally rare subset of experts, termed Super Experts. We thoroughly examine their distributions, intrinsic properties, and critical functional roles driving systematic outliers. While these findings provide essential insights into the internal mechanisms of MoE LLMs, several important research directions remain open for further exploration. Specifically, future investigations will explore leveraging SEs for improved post-training compression and studying their formation during training dynamics, with the objective of mitigating extreme imbalances among experts.
ETHICS STATEMENT
This research adheres to established ethical standards in the field. All data used in experiments obtained from publicly available sources with appropriate permissions, and sensitive or personally identifiable information was utilized. LLMs were employed exclusively as linguistic aids for text refinement, including grammar and stylistic improvements, and did not contribute to the design, execution, analysis, or conclusions of the study. The authors have taken care to ensure that the research findings are accurate, unbiased, and presented responsibly, with consideration for potential societal impacts.
REPRODUCIBILITY STATEMENT
All models, datasets, experimental setups, and hyperparameters used in this work are thoroughly documented. Key code components are provided in the supplementary materials, and the full algo- rithmic procedures are detailed in the appendix. Together, these details provide sufficient informa- tion for other researchers to independently verify and reproduce the results reported in this work.
REFERENCES
AIME. Aime problems and solutions.
2024.
AIME. Aime problems and solutions.
https///aroffprcc.eemaavvv...... Index. wwtpps/pp//AMMPM.aa... rob www.errooeemsooottttu..o.o
2025.
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Systematic outliers in large language models. arXiv preprint arXiv:2502.064/5 2025.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Joseph Bloom. Curt Tigges, Anthony Duong, and David Chanin. Saclens. com/ jbl oomAus SaannAuuSS/ALLnSSS SAELens 2024.
https://github.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Remov- ing outliers by helping attention heads do nothing. Advances in Neural Information Processing Systems, 36775067-75066 2023.
Weilin Cai, Juyong Jiang. Fan Wang. Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint arXiv:2407.06244 2024.
Mark Chen, Jerry Tworek, Heewoo Jun Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and Alex Ray. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
10

---

Published as a conference paper at ICLR 2026
Shu Chen, Zeqian Ju, Xiangyu Dong, Hongchao Fang, Sicheng Wang, Yue Yang, Jiaqi Zeng, Ruisi Zhang, Ruoyu Zhang, Meng Zhou, Penghui Zhu, and Pengtao Xie. Meddialog: large-scale medical dialogue dataset. arXiv preprint arXiv:2004.03329, 2020.
Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, et al. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems, 35:34600-33613,, 2022.
Mohammed Nowaz Rabbani Chowdhury, Meng Wang, Kaoutar EI Maghraoui, Naigang Wang, Pin- Yu Chen, and Christopher Carothers. A provably effective method for pruning experts in fine- tuned sparse mixture-of-experts. arXiv preprint arXiv:2405. 16646, 2024.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 19X1::9905.10044 2019a.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. arXiv preprint (oXiv:/906600341,, 2019b.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.0543
2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 10.1:2110.11166 2021.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309. 16588, 2023.
Jacob Devlin, Ming-Woi Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers). pp. 4171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbor,, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Haojie Duanmu, Xiuhong Li, Zhihang Yuan, Size Zheng, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. Mxmoe: Mixed-precision quantization for moe with accuracy and performance co- design. arXiv preprint arXiv:2505.05799 2025.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024a. URL https:
// //arxiv.org abs
wwa.aa2a /2002 04093
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024b. URL hRRhttts.. : zenodo. org/ rttp/ds.orrd/////// 602
Xiangming Gu. Tianyu Pang, Chao Du Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arXiv preprint arXiv:2410. 1078, 2024.
11

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Tianyu Guo, Druv Pai, Yu Bai, Jiantao Jiao, Michael Jordan, and Song Mei. Active-dormant attention heads: Mechanistically demystifying extreme-token phenomena in llms. arXiv preprint arXiv:2410. 13835, 2024.
Peter Henderson Mark S. Krass* Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Juraf- sky, and Daniel E. Ho. Pile of law: Learning responsible data filtering from the law and a 256gb open-souree legal dataset, 2022. URL /ttttttt////xxiiii orq/ abs/2207, 00220
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Xing Hu, Zhixuan Chen, Dawei Yang. Zukang Xu, Chen Xu, Zhihang Yuan, Sifan Zhou, and Jiangy- ong Yu. Moequant: Enhancing quantization for mixture-of -exppettof-expetts large language models via expert-balancee sampling and affinity guidance. arXiv preprint arXiv:2505.03804 2025.
Wei Huang, Yue Liao, Jianhui Liu, Ruifci He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and Xiaojuan Qi. Mixture compressor for mixture-of-experts Ilms gains more. In The Thirteenth International Conference on Learning Representations, 2025.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08 08322, 2023.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint N0V::240307777 2024.
Ajay Jaiswal, Jianyu Wang, Yixiao Li, Pingzhi Li, Tianlong Chen, Zhangyang Wang, Chong Wang, Ruoming Pang, and Xianzhi Du. Finding fantastic experts in moes: A unified study for expert dropping strategies and observations. arXiv preprint arXiv:2504.055: 005586, 2025.
Sakaguchi Keisuke, Le Bras Ronan, Bhagavatula Chandra, and Choi Yejin. Winogrande: An adver- sarial winograd schema challenge at scale. Communications the ACM, 2019.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. arXiv preprint arXiv: 2019.
Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient smoe with hints from its routing policy. arXiv preprint arXiv:2310.01334 2023.
Pingzhi Li, Xiaolong Jin, Yu Cheng, and Tianlong Chen. Examining post-training quantization for mixture-of-exppers: A benchmark. arXiv preprint arXiv:2406. 08155, 2024.
Hunter Lightman, Vincet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepscek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang. Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of -fxeeerrss large language models. arXiv preprint 2402 14800, 2024.
12

---

Published as a conference paper at ICLR 2026
Callum McDougall. SAE Visualizer. vis
2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
Siyuan Mu and Sen Lin. comprchensive survey of mixture-of-experts: Algorithms, theory, and applications. arXiv preprint arXiv:2503.07137, 2025.
Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, et al. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-feeee arXiv preprint arXiv:2505.06777 2025.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(14)): 1-11116672222 2020.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In First Conference on Language Modeling, 2024.
Zunhai Su and Kehong Yuan. Kvsink: Understanding and enhancing the preservation of attention sinks in kv cache quantization for lims. arXiv preprint
2025.
Zunhai Su, Zhe Chen, Wang Shen, Hanyu Wei, Linge Li, Huangqi Yu, and Kehong Yuan. Rotatekv: Accurate and robust 2-bit kv cache quantization for Ilms via outlier-aware adaptive rotations. arXiv preprint 2rXhv:2501.1638) 2025a.
Zunhai Su, Wang Shen, Linge Li, Zhe Chen, Hanyu Wei, Huangqi Yu, and Kehong Yuan. Akvq-vl: Attention-aware kv cache adaptive 2-bit quantization for vision-language models. arXiv preprint arXiv:2501.15021, 2025b.
Mingjic Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17766 2024.
Meituan LongCat Team, Anchun Gui, Bei Li, Bingyang Tao, Bole Zhou, Borun Chen, Chao Zhang, techni-
Chengeheng Han, Chenhui Yang. Chi Zhang, et al. Introducing mangcat-fash-thiniee cal report. arXiv preprint arXiv:2509.1880
2025a.
Meituan LongCat Team, Bei Li, Bingye Lei, Bo Wang, Bolin Rong, Chao Wang, Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, et al. Longcat-flash technical report. arXiv preprint arXiv:2509.01322, 2025b.
ModelScope Team. EvalScope: Evaluation framework for large models, 2024. URL
https:
//github.com hwt.ha/..ccmmmoeeeeeeee////eeeee
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307. 092888 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Wenxiao Wang. Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He. Model compression and efficient inference for large language models: A survey. arXiv preprint 0927422402.09748, 2024.
13

---

Published as a conference paper at ICLR 2026
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 2e122339977775 17453, 2023b.
Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu. Moe-pruner: Pruning mixture-of cexture-of-expert -epeeetstteessseeeeeeeeeee large language model using the hints from its router. arXiv preprint arXiv:2410. 2024.
Jing Xiong, Liyang Fan, Hui Shen, Zunhai Su, Min Yang, Lingpeng Kong, and Ngai Wong. Dope: Denoising rotary position embedding. arXiv preprint arXiv:2511.0911 2025.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang. Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. Moe-i2: Compressing mixture of experts models through inter-expert pruning and intra-cxpert low-rank decomposition. arXiv preprint arXiv:2411.01006. 2024.
Jaewoo Yang, Hayun Kim, Junyung Ji, and Younghoon Kim. Mitigating quantization emrors due to activation spikes in gated linear unit-based large language models. Future Internet, 18(,) 2025b.
Mengxia Yu, De Wang, Qi Shan, and Alvin Wan. The super weight large language models. arXiv preprint arXiv:2411.0719/ 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Hengyuan Zhang, Zhihao Zhang. Mingyang Wang, Zunhai Su, Yiwei Wang, Qianli Wang, Shuzhou Yuan, Ercong Nie, Xufeng Duan, Qibo Xue, et al. Locate, steer, and improve: A practi- cal survey of actionable mechanistic interpretability in large language models. arXiv preprint arXiv:2601.14004, 2026.
Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. Diversifying the expert knowledge for task-agnostic pruning sparse mixture-of-expert.. arXiv preprint arXiv:2407.09590, 2024.
Zihao Zheng, Xiuping Cui, Size Zheng, Maoliang Li, Jiayu Chen, Xiang Chen, et al. Moqa: Re- thinking moe quantization with multi-stage data-model distribution awareness. arXiv preprint arXiv:2503.22 1355, 2025.
Xunyu Zhu, Jian Li. Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12: 1556-1577, 2024.
Zayd MK Zuhri, Erland Hilman Fuadi, and Alham Fikri Aji. Softpick: No attention sink, no massive activations with rectified sofumax. arXiv preprint arXiv:2504.20966, 2025.
14

---

Published as a conference paper at ICLR 2026
A STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
In the preparation of this paper, LLMs were employed solely as linguistic aids to enhance clarity, correctness, and readability, including grammar refinement and stylistic improvement.
B RELATED WORK ON SYSTEMATIC OUTLIERS IN TRANSFORMERS
Previous studies (Su & Yuan 2025 An et al. 2025) have shown that Transformer-base dense L.Ms exhibit systematic outliers. These outliers appear in multiple forms, including weight outliers (also referred to as super weights Yu et al. 2024)). activation outliers (encompassing both activation spikes and MAs (Yang et al. 2025b Sun al. 2024)), and attention outliers (commonly known as attention sinks (ASs) (Xiao et al.) 2023b)). This phenomenon is not confined to LLMs but is also observed in other Transformer- based architectures, including BERT (Devlin al. 2019 Kovaleval et al.
2019:
Clark et al. (219b)). Vision Transformer (ViT) (Dosovitskiy et al.] 2020 Bondarenko
ct al.
2023 Sun al.] 2024).
Quantizable Transformers (Bondarenko et al., 2023); as pioneering study, identified the bottleneck activation quantization of T'Tnstoorrrsss caused by extreme outliers and revealed the intrinsic rela- tionship between attention focus patterns and these outliers. The study further showed that attention focus emerge as attention heads attempt to perform a "no-op" or a partial update the residual. In this process, strong activation outliers arise due the limitations of the softmax function, which can- not produce exact zeros or ones. Consequently, Transformers learn a workaround which attention disproportionately concentrates on small set of fixed tokens, whose corresponding Value States typically have small norms. As a result, the attention output remains small, effectively endowing the model with no-op capability.
Building on this insight, recent research has shown that enhancing softmax attention can substan- tially mitigate or eliminate systematic outliers during pretraining, thereby enabling more accurate low-precision quantization. Quantizable Transformers (Bondarenko et al. (203)) demonstrate that pretraining with clipped softmax and gated attention produces significantly smaller outliers while preserving, and in some cases even enhancing, floating-poie task performance. Qwen Team (Qiu al., 2025) finds that applying a head-specific sigmoid gate after the Scaled Dot-Product Atten- tion (SDPA) consistently improves performance and eliminates systematic outliers. Softpick (Zuhrij cct al. 2025) introduces a rectified drop-in replacement for softmax in Transformer attention that relaxes the sum-to-one constraint, effectively climinates attention sink and massive activations, and holds strong promise for advancing quantization, low-precision training, sparsity optimization, prun- ing, and interpretability. By revealing SEs as the root source of systematic outliers, this work pro- vides the first comprehensive characterization of such phenomena in MoE LLMs and establishes a foundation for future advances in outlier mitigation.
C FURTHER ANALYSIS OF OUTLIER EXPERTS IN FINAL LAYERS
Some experts in the final layers also exhibit extreme activation outliers in the output of the down_proj, apart from the SEs in the shallower layers. We refer to these experts as outlier experts. Based on extensive additional experiments and findings, we do not consider outlier experts to have the same mechanistic significance as SEs:
(i) We performed PPL evaluations after pruning outlier experts, and as shown in Table they do not significantly affect the model's ''s performance in the same way as the SEs.
(ii) In both 30em3-30B-A3BB and DeepSeck-R1, pruning outlier experts does not result in repetitive outputs on reasoning benchmarks such as Math-500 (Lightman et al.] 2023). whereas pruning SEs does, shown in Tables 11 and 12 The pruned SEs and outlier experts shown in Table
(iii) We observed on Qwen3-30B-AB that the distribution of outlier experts varies with the input dataset, while the distribution of SEs remains quite stable, illustrated in Figure 14and [5
(iv) The router scores in the final layer are relatively evenly distributed across all experts for both sink and non-sink tokens, as shown in Figure TO In contrast, sink-token router logits are strongly
15

---

Published as a conference paper at ICLR 2026
Opeet 100 A.B Deart Albuste thitt
(a) Layer sink token.
(b) Layer 1 non-sink tokens.
De to Atluabe Duter
Rec Ztret to totee Deatt oa
(c) Layer 2 sink token.
(d) Layer non-sink token.
Oherd 201-0006----00 Layer' Layer Exain Actvatton Diento stoe lank (C4)
4500
0p0 Lager Exper wees Dembitinn honiine 1C
(c) Layer 3 sink token.
(f) Layer 3 non-sink tokens.
O0ees- 423 Layer Sopar Aetivtion buree of's Blen
Oiann 300-20 Laper Expen Achorie Der butte hes Raens
Espert
(g) Layer 47 sink token.
(h) Layer 47 non-sink tokens.
Figure 10: Expert router score distributions for sink and non-sink tokens in Qwen3-30B-A3B based on calibration using the C4 dataset.
skewed toward SEs, indicating that the final-layer outlier experts fundamentally different from SEs.
We infer that since MAs occur in the shallower layers, these outlier experts not involved in the formation of MAs. Therefore, these experts do not operate under the same mechanism as SEs and do not hold the same level of significance.
D DISTRIBUTION OF SUPER EXPERTS ACROSS VARIOUS DATA DOMAINS
In addition to amalyzing the distribution of SEs across different models based on the C4 dataset (Raffel al. 2020). we also examine their distribution patterns across various input data domains. We assess the impact of diverse language inputs on SEs using the WikiText-2 (Merity et al. 2016) and C-Eval (Huang et 2023) datasets. Furthermore, we investigate the influence data from the mathematics and coding domains using the GSM8K (Hendrycks et al. (02T) and HumanEval (Chen et al. 202I) datasets. As shown in Figures [4 15
17 18 and 19 the distribution of SEs remains highly stable, regardless of variations in the input data domain.
16
E ADDITIONAL RESULTS OF REASONING MODELS AFTER SUPER EXPERTS PRUNING
After pruning SEs, consistently observed repetitive output and loss of reasoning ability in both
Awen3-30B-A3B and DeepSeck-RL The pruned SEs are shown in Table 7 and additional examples from the Math-500 (Lightman al. 2023 benchmark presented in Tables 11 and 12
16

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of expert pruning, with PPL evaluated using the WikiText-2 dataset.
Model
Prune Experts
PPL
Super Experts
Original Model
8.70
Qwen3-30B-ABe
Layer Expert 68, Layer 2 Expert 92, Layer Expert 82
59.86
Yes
Layer 47 Expert 8, Layer 47 Expert 48, Layer 47 Expert 100
8.71
No
Original Model
6.31
DeepScek-V2-B1 V2-LLiee
Layer Expert 54, Layer 4 Expert 38
10.75
Yes
Layer 25 Expert 11, Layer 25 Expert 39
6.32
No
Table 7: Super Experts and Outlier Experts in Qwen3-30B-A3B and DeepSeek-RI models.
Model
Super Experts
Outlier Experts
Layer Expert Layer Expert
Layer Expert 8, Layer 47 Expert Layer 47 Expert 100
Qwen3-30B-A3B
Layer Expert 82
Layer 8 Expert 24, Layer 8 Sexeer_expert
Layer 60 Expert 81, Layer 60 Expert 92 Layer 60 Expert 231, Layer 60 Seared_expert
Layer Expert 190, Layer 13 Expert 64
DeepScek-RI
Layer Expert 202, Layer 14 Shared_expert
Layer 60 Expert 121, Layer 60 Expert 0
Layer 22 Shared_expert, Layer 33 Expert 64
Layer 60 Expert 60, Layer 60 Expert 237 Layer 60 Expert 53, Layer 60 Expert 117
Layer 33 Shared_expert, Layer 35 Shared_expert
F ADDITIONAL RESULTS ON ROUTER SCORE DISTRIBUTIONS
The router score distributions calibrated on the C4 dataset are shown in Figure 20 Similarly, the router score distributions calibrated on the Wikitext-2 dataset are presented in Figure 21 Notably, in both datasets, SEs are consistently and strongly activated on the sink tokens.
G MORE ANALYSIS ON SUPER EXPERTS MECHANISM
Figure 22 provvids comprehensive illustration of the systematic outlier mechanism in Qwen3-30B-
A3B, showing the stepwise formation process across three layers. Furthermore, the mapping from massive activations to attention sinks remains consistent even after multiple transformations. such as layer normalization (LN) and QKV projections. Drawing on prior research (Sun et al. 20244] Su & Yuan 2025). we offer more detailed analysis elucidate this mechanism. Specifically. this process is governed by two key mechanisms:
QKV suppression. The presence of massive activations with large magnitudes results in substan- tially smaller normalized values for the corresponding tokens after LN, as dictated by the RMSNorm process. This reduction in norm is preserved throughout the QKV states. shown in Figures the Queries, Keys, and Values of sink tokens consistently exhibit significantly smaller norms com- pared non-sink tokens.
High cosine similarity of QK. Despite the reduced norms of Queries and Keys, the cosine similarity between the Queries of non-sink tokens and the Keys of sink tokens remains high (Gu et al. 2024).
leading to disproportionately large attention scores, as illustrated in Figure Ila
These intermediate mechanisms ensure that massive activations remain aligned with attention sink tokens, thereby establishing their participation in systematic outliers.
H WEIGHT-LEVEL ANALYSES OF SUPER EXPERTS
While the preceding discussion primarily addressed the role of experts in expert-level compression, this section provides more granular, weight-level analysis of SEs. Conducting analyses at the weight level offers several key advantages: (i) it clarifies the specific sources of SEs' importance, (ii) it facilitates the investigation of analogous patterns in dense models, and (ii) it enables the
17

---

Published as a conference paper at ICLR 2026
congg byler heado
query states layer
key states layor
Ta
Channel
Channel
(a) Cosine similarity
(b) Query states
(c) Key states
olta viice acoha loyer 3ead
bnye
cupput layer
Taken
Channel
Channel
(d) Attention scores
Value states
Attention outputs
11c), and Ie illustrate QKV suppression. 11a) highlights the high cosine similarity of QK. IIf) visualizes the attention output. Visualizations use the following input from MMLU (Hendrycks et al., 2021), evaluated on Llama-2-7B: "The following are multiple-choice questions (with answers) about machine learning. 'n'n..."
Figure 11: (1b).
Table 8: Super weights of several models.
Models
Total Layers
Emergence Layer
Super Experts
Shape of down.proj
Super Weights
-I.VA-VI.5-7B
0-31
(4096, 1 10088
(1415, 7890), (2533, 7890)
(400, 1417), (698, 1417), (2029, 1417), (1159, 1417)
JJama-3.2-18
0-15
1
(2048,8192)
1
(940, 711)
Qwen3-30B-A3B
0-47
2
(2048,768)
(940, 59)
3 82
(940, 423)
application of interpretability tools, such sparse autoencoders BBloom et al. 2024 Gao et al. 2024a),, to examine individual weights or neurons.
Specifically, we detect extreme activation outlier channels in the down_proj inputs and outputs and map them to their corresponding weights, following the methodology used to identify Super Weights (SWs) in (Yu et al., 2024). For convenience, we refer to these weights as SWs. The SWs utilized in our experiments are listed Table 8
These SWs are subsequently pruned during inference, and experiments conducted on LLaVA- V1.5-7B (Liu et al.] 2202).. 2lama3.22-1 (Touvron et al.] (023). and Qwen3-30B-AM A3BB spanning dense LLMs, vision-language models (VL.Ms,, and MoF. I.MMs. Tables 13 that removing the SW's consistently leads to repetitive and uninformative outputs across all mod- els. demonstrating that these weights are critical contributors to SE importance. This outcome is expected, as it is consistent with our analysis of SEs' role in driving systematic outliers in Trans- formers. Interestingly, in dense LLMs, the single FFN layer exhibiting massive activations can be seen as playing role analogous to that of an SE.
14 15
and 16 show
Why does the model exhibit substantial performance gap before and after SWs pruning? We then investigate the underlying causes of this discrepancy. As model neurons are often polysemantic (Gao et al. 2024a), directly analyzing the semantics SWs is challenging due to their intrinsic polysemanticity. To address this, we employ Sparse Autoencoders (SAEs) (Bloom et al., 2024 Gao et al. 2024a), an unsupervised method designed to extract interpretable features from LLMS by reconstructing activations through sparse bottleneck layer. Using SAEs, we can decompose the semantics of polysemantic neurons into more discrete, monosemantic features and conduct semantic analyses the features most strongly correlated with SWs. We then train our SAE using activations from the Layer FEN outputs of 1lama-3.2-13 1B with the C4 dataset (Raffel et al., 2020). The loss function consists of two components: a reconstruction loss and sparsity penalty loss (Bloom al.
18

---

Published as a conference paper at ICLR 2026
Qwen3
DeepSeek-V2-u
g=1.50,
0-1.37.
on 1.
mmm=140.00
mmn3000.0
CCDF
CCDF
Activations
Activations
Activations
Figure 12: CCDF analysis of the distribution of A for Owen3-30B-A3B. Mixtral-8 and DeepSeek- Dee-Leek-V2-Lite on the C4 dataset. Across all three models, the fitted CCDF tails exhibit clear power-law form, indicating that only small subset of experts produce exceptionally large activa- tions. For each model, the estimated tail exponent is below 2, placing the distribution within the heavy-tailed regime. This establishes the presence of extreme yet statistically stable structured acti- vation outliers in A.
2024 The reconstruction loss is defined as
mselloss
ND
1
ΣΣ
(sae_out:,) sae_in:.))
N 1=1
where N denotes the batch size and D) the dimensionality of the activations. The sparsity penalty loss is given by
K Σ
1/p
11.loss X-
feature acts; | wedeccll
N
where À is hyperparameter controlling sparsity, K is the number of features in the bottleneck layer, fcature_acts;, denotes the activation of feature k for sample i. W_dec; is the corresponding decoder weight vector, and p specifies the norm used for aggregation.
After achieving satisfactory performance with the SAE, we apply the TopK algorithm to the de- coder weight matrix to extract the ten features most strongly associated with the SWs neurons. Each feature is interpreted via forward pass to infer its semantic meaning (McDougal] (024). As illus- trated in 23 the top features associated with the SWs neuron consistently exhibit pronounced activation at the end of text. token, a pattern that is rarely observed in other neurons. This find- ing provides a plausible explanation for the behavior L.MMs, which repeatedly generate text until reaching the maximum output length following the pruning of SEs or SWs. When SEs are pruned, MoE LLMs lose the ability to recognize sentence boundaries and generate text continuously until reaching the output length limit. This indicates that SEs play critical role in regulating sentence length and termination. We further hypothesize that SEs may contribute to additional model capa- bilities, which will explore in future research.
1 ANALYSIS OF THRESHOLD-BASEE SUPER EXPERTS IDENTIFICATION
In this section, we demonstrate that SEs reflect an intrinsic property of MoE LLMs, with threshold- based identification serving practical and principled detection method. We first provide jus- tification based on the heavy-tailed distribution of A {ate } (as discussed in Section 3.2.1 and then analyze the robustness of the threshold- based SE identification.
The identification of SEs is motivated by the heavy-tailed nature of extreme activations. While most experts produce modest responses, a small subset consistently generates extreme values (as illustrated in the heatmaps in Figure and Appendix (D). forming distinct long tail that is naturally present in MoE LLMs. To mathematically validate this observation, we analyze the tail of the activation distribution using standard heavy-tail analysis techniques, namely the complementary cumulative distribution function (CCDF) and pawer-law fitting. The CCDF a random variable X is defined as
CCDF(#) P(X > s),
(10)
19

---

Published as a conference paper at ICLR 2026
Table 9: Sensitivity analysis of SE identification across a range of threshold settings for multiple MoE LLMs and input domains.
Models
Number of Identified Experts
C4
Pas
Pou
Pa0.5
Po9.9
WikiText-2
Pos
Pae
Paa.5
9a0.9
0.1 Gmax
0.1 Omax
3
3
3
Qwen3-30B-A3B
0.09 Omax
0.09 * Omax
3
0.08 * Omax
0.08 * Amax
3
0.07 * Omax
3
0.07 * Omax
3
3
3
C4
Pos
Pgg
P99.5
P99.9
WikiText-2
Pos
Pgp
Pys.5
P99.9
0.1 Gmax
0.1 Omax
2
DeepSeek- V-----2
0.09 * Omax
0.09 * Omax
2
0.08 Omax
2
2
2
0.08 *
2
2
2
2
0.07 Omax
0.07 Omax
2
C4
Pos
Pea
P30.5
Po0.9
WikiText-2
Pos
Poe
Poa.
Pao.9
0.1 Gmax
0.1 # amax
Mixtral-8x7B
0.09 * Omax
0.09 # Gmax
1
0.08 Omax
0.08 Omax
0.07 Omax
0.07 + Omax
representing the prohability that X exceeds given value Plotting the CCDF log-log scale provides an initial visual assessment of the tail behavior. To quantitatively characterize the tail, we fit it to power-law model
P(X > x x Tmins
(1I)
where is the tail exponent and Tmin denotes the minimum value above which the power-law be- havior holds. The exponent a estimated using maximum likelihood estimation (MLE). As shown Figure 12, the CCDF tail is consistent with power-law model. This long-tail behavior indi- cates that only a few experts dominate the extreme activations, naturally separating them from the majority and confirming that extreme activations follow an inherent heavy-tailed distribution rather than arising from random noise. Threshold based methods thus provide a practical and principled approach for identifying this small, statistically significant subset experts.
To further assess the robustness of SE identification, we perform a sensitivity analysis over rea- sonable range of thresholds across different MoE I.LMs and input domains. As shown in Table SE identification remains consistent, confirming that extreme activation outliers reflect intrinsic property MoE LLMs rather than an artifact of any specific threshold choice. Together, these anal- yses demonstrate that the threshold-basee criterion is both robust and scientifically justified, and that SEs constitute an inherent feature of MoE LLMs.
CROSS-DOMAIN ANALYSIS OF SUPER EXPERTS AND ATTENTION SINK TOKENS
In this section, we conduct cross-domain analysis of SEs and attention sink tokens using inputs drawn from diverse domains, including out-o--diitribution (OoD) inputs that differ substantially from the training datasets. Each domain includes visualizations of SE heatmaps, attention sink patterns, and router score distributions both sink and non-sink tokens.
We use C4, C-Eval, GSM8K, and HumanEval (Raffel et al. 2020 Huang et al. 2023 Hendrycks al. 202T Chen et al.) (02T), covering range domains including English, Chinese, mathemat- and code. For the OoD datasets, we adopt Pile-of-Law and MedDialog (Henderson al. 2022 Chen et al. 2020), which are legal and medical datasets and differ significantly from the pretraining corpus of the tested MoE LLMs in content and domain.
As shown in Figures across these varied domains OoD datasets. Moreover, the sink tokens consistently appear as the first token. Their router scores on SEs are markedly larger, whereas the router scores of non- sink tokens remain relatively uniform across experts. These observations reinforce claim that
26
28 and 29 the distribution of SEs remains highly consistent
20

---

Published as a conference paper at ICLR 2026
1030
Layer
950
500
650
600
400
400
200
47
47
Expert
Expert
(a) Qwen3-30B-A3B.
(b) Owen3-30B-A3B-Base
1030
1000
Layer
Layer
800
Expert
127
Expert
127
(c) Qwen3-300B-AAB--nsstuttt22
(d)
Figure 13: Heatmap visualizations of the maximum output magnitudes from the down.pro for
each expert across layers based on C4 dataset. SEs are highlighted with arrows.
Table 10: Super Experts in Qwen3-30B-A3B.
Models
Super Experts
Qwen3-30B-A33
Layer Expert 68, Layer2 Expert 92, Layer3 Expert 82
Swen3-30B-A3B--o 13B-Instruet-250
Owen3-30B-AN 30B-ABB- Thinking 2507
SEs reflect an intrinsic property of MoE LLMs, rather than arising from over-exposure to domain- specific tokens, and are not affected by OoD inputs.
K DISTRIBUTION OF SUPER EXPERTS ACROSS TRAINING STAGES
To study the distribution of SEs across training stages, we identified SEs in four officially open- sourced Owen3-30B-AA3 models at different training stages: Qwen3-30B-AN Base, Qwen3-30B. A3B, Qwen3-30n -0B-A3B-- Inst-nnt-ttt555 - 2507, and Owen3-30B-A33 --B---iinniig------ -507.. The indices of the SEs are shown in Table TO and the SE heatmaps are presented in Figure [3] demonstrating fully consistent across all models and training stages. These results support the conclusion that SEs are persistent features of MoE LLMs.
21

---

Published as a conference paper at ICLR 2026
L ALGORITHM FOR PROFILING SUPER EXPERTS
The detailed procedure for profiling Super Experts is illustrated in Algorithm
Algorithm Calibration- based Super Experts Profiling
Input: Model with E experts per layer, calibration dataset D
2: Output: Set of Super Experts S
3: Stage 1: Calibration of MA-formation Layers
4: L +- 0
5: for each batch E D do
6: for each layer I the model do Compute hidden activations (x) 8: if LA pattern detected in (x) then
7:
9:
+ LU (1)
10: end if
end for
12: end for
13: Stage 2: Identification of Sxper-Experts
14: A 0
15: for each batch E D do for each layer 1 do for each expert in layer do Compute output (1)) before down.proj
16:
18:
At,e max (x)) Whe
19:
down
A t AU fay.c
21: end for
22: end for
23: end for
24: Po9.5 + Percentilega. (AA)
25: 0max f- max(.A)
26: S 5+0
27: for each e) with atee E A do
28: if ata Po9.s and 0t.e 10 Amax then
29:
5 S U e(I,e)
30: end if
31: end for
return S
22

---

Published as a conference paper at ICLR 2026
1000
1080
Layer
Layer
410
2141
Expert
Expert
Qwen3-30B-A3B (WikiText-2).
(b) Qwen3-30B-A3B (C-Eval).
5000
1080
Layer
Jayer
700
47
Expert
127
Expert
(c) Qwen3 30B-A3B (GSM8K).
(d) Qwen3-30B-Ae 3000--30-AABB (HumanEval).
Figure 14: Heatmap visualizations of the maximum output magnitudes from the down_proj for
each expert in Swen3-30B-A3BB across multiple datasets. SEs are highlighted with arrows.
1000
1000
Layer
Laver
410
200
47
Expert
Expert
(a) Owen3-30B-AA3B-Bas (WikiText-2).
(b) Qwen3-30B-A33B-BB (C-Eval)..
8000
1080
Layer
400
210
Expert
Expert
(c) S0ee3-300--33--BB (GSM8K).
(d) Bowe33-00B-A3B-BB- (HumanEval),
Figure 15: Heatmap visualizations of the maximum output magnitudes from the down.pe proj for
each expert in Qwen3-30B-A3I across multiple datasets. SEs are highlighted with arrows.
23

---

Published as a conference paper at ICLR 2026
1000
1080
Layer
Layer
400
2141
Expert
Expert
DeepSeek-V2-LLite-CCe V2-Lite-Chat (Wik
(h) DeepSeck- DeepSeck-V2-Lite-Chee (C-Eval).
1000
5000
Layer
Laayer
700
Expert
Expert
(c) DeepSeek V2-Lite- (GSM8K).
(d) DeepSeek Mart-eekk---Liite V2-Lite Chat (HumanEval),
Figure 16: Heatmap visualizations of the maximum output magnitudes from the down_proo for
each expert in DeepSeek- Sat--eeee-V--Lt--CC-- V2-Lite-Chat across multiple datasets. SEs are highlighted with arrows.
1000
1000
Layer
Layer
410
200
Expert
Expert
(a) DeepSeek- V2-Lite (WikiText-2).
DeepSeek- V2-Lkk--2--iieeee (C-Eval).
0000
1080
Layer
Layer
400
210
Expert
Expert
DeepScck-V2-42 (GSM8K).
(d) DeepSeck- V2-Litc (HumanEval),
Figure 17: Heatmap visualizations of the maximum output magnitudes from the down_prot for each expert in DeepSeek- --V--eeek----iiee V2-Lite across multiple datasets. SEs are highlighted with arrows.
24

---

Published as a conference paper at ICLR 2026
1000
1080
Layer
Layer
410
2141
Expert
Expert
(a) Mixtral-8877--nnstrrreel) (WikiTexx-2).
Mixtral-$x7B--nssmu--00) (C-Eval).
1000
0000
Layer
Layer
700
Expert
Expert
Bestral-8xB-Innttueee (GSM8K).
(d) Mixtral-8xx Hixtral-8x7B-Inssee Ix7B-Insttuct-vO.. (HumanEval),
Figure 18: Heatmap visualizations of the maximum output magnitudes from the down_proo for
each expert in Mixtral-8x7BB -0----x-xttttttt-O00 across multiple datasets. SEs are highlighted with arrows.
1000
1000
Layer
Layer
410
210
Expert
Expert Mixtral-8x7B-vO. (C-Eval).
Mixtral-8x7B-v0.1 (WikiText-2).
8000
8080
Layer
Layer
400
210
Expert
Expert
(c) Mixtral-8x7B-vO.1 (GSM8K).
Mixtral-8x7B-v0.1 (HumanEval),
Figure 19: Heatmap visualizations of the maximum output magnitudes from the down_prot for
each expert in Mixtral-&x7B-v..1 across multiple datasets. SEs are highlighted with arrows.
25

---

Published as a conference paper at ICLR 2026
Table 11: Responses of DeepSeek-R1 in the MATH-500 benchmark.
DeepSeek-R1
Input
Repeating
Auswer
Okay, sa need to find the domain of the function f(x) (2 xx log(2 - log(x 2)). Hmmm, let see. The domain of function includes all the real numbers x
Original Model
What is the domain of the function
no
((r log log(t-ka(x=21) k2))=20)
where log is the base logarithm function?
Express your answer in interval notation.
Alright, let's try to first. the functioo f(x) which given that need to the domain of this function that's 's the which the of a that's the given this, that's is the the the the the the the the the the the the the the the the. that's is the. that's 's the, that's the. that's 'ss the that's the.
Please reason step by step.
pat your final answer within (boxad {]-
Prune Super Experts
Regeal ing
that's is the. that's 's the. that's the. that's 's the that's the.
Okay, let's try ligure the domain of this function
Prune Outlier Experts Last Layer
(ar) logi2 loste 27. Hmm, domains can be tricky because there are multiple layers
Alright, so 1 need in find the domain of the function [(x] (2 x) log(2 log(s 2)). UImm, okky, Let start by recalling that the domain of
Random Pruning
no
Table 12: Responses of Qwen3-30B-A3B in the MATH-500 benchmark after SE pruning.
Qwen3-30B-A3B
Iuput
Repeating
Answer
Okay, so need to find the value p(8) where p(x) degree polynomial that satisfies p(n) = (02 for o= 2, 3,
Original Model
p(a)) be polynomiaa of degree such pin) _²"_ for 4, ..... 7. Find p(S).
Please reason step by stop, and final answer within (boxed().
Okay. so need to find p(8) where is polynomial. The problem says that for each n, p(n) equal to p(8). The question from competition, so it's probably from Olympiad, the answer reason step by step. and put my final answer within (boxed(]. Airight,
some fraction,, and
integer.
Prune Super Experts
yes
start by understanding the problem. It says that for each p(n) is equal to some fraction. But problem is from competition. so it's probably from an Olympiad, and the reason step by step, and put my final answer within Repest Ing
p(8). question
is integer. But need to
Okay, so need to find the value of p(8) where p(x) is dogree polynomial that satisfies p(t) (n2- for n 3, 5, Hmm, let me think about how
Prune Outlier Experts I Last Layer
DO
Okny, so need to find the value of p(8) where p(x) dayree polynomial that sati p(n) = n/ in 1) for n= 2, 3,4..... 7.
Random Pruning
26

---

Published as a conference paper at ICLR 2026
O 308 ADB Eean Aclvtton Dutrzuden Bar 1541
Oven ADB Loyer Expe vollen Dearbo Sen Teerr
(a) Layer sink token.
(b) Layer non-sink tokens.
Qwerb Aittieer Actratie Diotil
Qpech 350 w30 Soper waes Demiiicton TAR0L
(c) Layer 2 sink token.
(d) Layer 2 non-sink token.
Qwent upe. syer Arteatee Bratds
Gaees www Eoger Newere temte ofn Yereurr
(e) Layer 3 sink token.
(I) Layer 3 non-sink tokens.
Figure 20: Expert router score distributions for sink and non-sink tokens Qwen3-30B-AD3 based on calibration using the C4 dataset.
Dhend 30003-30.4300 Zagnt dcteciion of Dem
20003-00-00 Exant Atbaube Duttrautee Send Teeera Ma11
(a) Layer sink token.
(b) Layer non-sink tokens.
Oven) 300 Lagar Sipart Activdion demiitationn sine Salan (wic)
Ower' D0l Layer Casee acivatton Dranttution hars Stane Nild)
(c) Layer 2 sink token.
(d) Layer non-sink token.
Quenh 300 Logar Sipar Antivaion demiiitiio sie Saan
Owerc 206 Ailiiaeer oayer' Essser acivatio Erent attie vars Sianne Mild)
(e) Layer 3 sink token.
(I) Layer 3 non-sink tokens.
Figure 21: Expert router score distributions for sink and non-sink tokens Qwen3-30B-A based on calibration using the Wikitext-2 dataset.
27

---

Published as a conference paper at ICLR 2026
layer Jo
Exp
Exp
layer_ns
ainn
MoF
Exp tus dawa, proj
Layer 1
Layer
layor
#92
slin pruds
Atteation
MoF:
Firpy
Exp down, proj
Layer 2
Layer
layer_n
Exp 282 doma,
Exp A82 dows,
layer
siin pmnds
Attention
MoE
Exp proj
Layer 3
Layer
Figure 22: Systematic outlier mechanism of Qwen3-3nn0 Q3wen3-30-AA3B. using the input: "Green trees sway under gentle wind."
28

---

Published as a conference paper at ICLR 2026
Table 13: Responses of LLaVA-V1.5-7B after SWs pruning.
LLVVA-V1.-7B
Input
Repeating
Answer
The player holding the baseball bat is now wearing the number 44.
Original Model
no
Prune SWs Neurons
yes
What number is the player holding the baschall hat now?
The player holding the baseball bat is now wearing the number 44,
Random Pruning
Table 14: Responses of LLaVA-V1.5-2 after SWs pruning.
LLVVA-VI.57B
Input
Repeating
Answer
The traffic sign says "You Can't Stop Obama."
Original Model
no
6 SUAATT STOP OBAMA
Prune SWs Neurons
yes
F1t 00
The traffic sign says "You Can't Stop Ohama."
Random Pruning
What does the traffic sign say?
TOP ACTIVATIONS
MAX 0,005
TOP ACTIVATIONS
noan taon" Stend seattl (jovd teat leed of zwet ofone
MAX 0.406
tirce Henttttt tentt un just (land keet clund
government not fercing this upan people.
It' top -olling mittlpe sclerosts beatii lend cleed staperss
baet pland zeedfocl and Boet
spensored legialatizn) abous foreenn all of baxic lomen
pay evertine rates canvet werkers to wore nccessive hours
Hery cPanel ande and of beet Hlant pestiie and bodi londd swcils
provistoos
ancing bonst for
deiivee
cocestial fir eagiget workrne gloee teat moat of of rriday Donild Tranodlent teet of clat cfantee
You may be trying to force sonething that lon'it ready
that foore churches
(a) The top features associated with the SWs neu- ron consistently exhibit pronounced activation at the end_of.text token.
atblid mechines then always be transitioning
(b) The top features in other neurons.
Figure 23: Top features identified in SW neurons versus other neurons using the trained Sparse Autoencoder.
29

---

Published as a conference paper at ICLR 2026
Table 15: Responses of 1lama-3.2-1B after SWs pruning.
Llama-3.2-1B
Input
Repeating
Answer
3x = 18. x Step 1: Write down the equation given in the problem. 7 Step 2: Subtract from both sides of equation isolate term with x. 3x 25 which simplifies
Original Model
18. Step 3: Divide hoth sides the equation by solve for 3x / 18 3. which simplifies
Solve for 3x + 25.
The final answer is:
Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy
Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy
Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy
Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy
Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy
Prune SWs Neurons
yes
Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy
Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy
Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy
Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy
Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy
Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy Quy
18. x Step Write down the equation given problem.
Step Subtract from both 7 25 which simplities
sides of equation isolate term with 3x
Random Pruning
18. Step 3: Divide
both sides of the equation by solve for x. 3x /3 18 / 3, which simplifies x = 6. The final answer is:
Table 16: Responses of Qwen3-30B-A3B after SWs pruning.
Qwen3-30B-A3B
Input
Repeating
Answer
Okay, need to solve the equation 3x 25.
Let me think about how approach this. remember from algebra that the goal isolate x, which means getting x by itself the equation. First, I should probably get rid of that that's being added 3x. To do that, can subtract from both sides the equation. That
side
Original Model
no
Solve for x: 3x + 7 25.
Okay, so I need to solve this equation: 3x + 7 25. Hmm, right. Let me think. So, first, maybe should subtract 7 from both sides? Wait, no, that's not. Wait, no, that's not. Wait, no, that's not. Wait, no, that's not. Wait, no, that's 's not. Wait, no. that's 's not. Wait, no, that's not. Wait, no, that's not. Wait, no, that's not. Wait, that's 's not. Wait, no. that's 's not. Wait, no, that's not. Wait, no, that's not.
Prune SWs Neurons
...
Okay, so need to solve the equation 3x 25. Let me think about how to approach this. remember from algebra that the goal to isolate x, which means getting x by itself on one side the equation. First, should probably get rid that that's being added to 3x. To do that, I can subtract 7 from both sides of the equation. That
Random Pruning
30

---

Published as a conference paper at ICLR 2026
1000
Head
Head
Layer
500
47
Expert
127
(a) SEs heatmap.
(b) Attention sinks.
(c) Attention sinks.
Dwe12:00-88Beet Exant Atlauton Distnutee 3ew
9 Loye Towe Dest bf Sem Macu
(d) Router scores distribution of sink token.
(e) Router scores distribution non-sink tokens.
Figure 24: SE heatmaps, attention sink visualizations, and router score analyses for sink and non-
sink tokens in Qwen3-30B- 30-n--30BBBBB on the C4 dataset. SEs are highlighted with arrows.
Layer 25 Head 10
Layer 35 Head 10
Layer
5/0
400
Expert
127
SEs heatmap.
(b) Attention sinks.
Attention sinks.
USSUSEM
Expat
(d) Router scores distribution sink token.
(e) Router scores distribution non-sink tokens.
Figure 25: SE heatmaps, attention sink visualizations, and router score analyses for sink and non- sink tokens in Qwen3-30B-A32 based on the C-Eval dataset. SEs are highlighted with arrows.
1000
Layer
Head
Layer er
200
47
Expert
127
(a) SEs heatmap.
(b) Attention sinks,
(c) Attention sinks.
Ovan3-36-400 Asthrtior butber
Cowe3-38--22 Expert Actrttr Durbutes Wan Sink Rohen (ogren
(d) Router scores distribution of sink token.
(e) Router scores distribution of non-sink tokens.
Figure 26: SE heatmaps, attention sink visualizations, and router score analyses for sink and non- sink tokens in Owen3-30B-AAe based the GSM8K dataset. SEs are highlighted with arrows.
31

---

Published as a conference paper at ICLR 2026
1000
Head
Head
Layer
500
47
Expert
127
(a) SES heatmap.
(b) Attention sinks.
(c) Attention sinks.
Deecs-308-438 Leont Ioent fucnd Dtttt fee Thon Dte
Oward Se- Dnant Anl Oater Sander Toer Ptesttt
(d) Router scores distribution of sink token.
(e) Router scores distribution non-sink tokens.
Figure 27: SE heatmaps, attention sink visualizations, and router score analyses for sink and non-
sink tokens in Qwen3-30B- 30-B-33BBBBBB based on the HumanEval dataset. SEs are highlighted with arrows.
Layer 25 Head 10
Layer 25 Head 10
Layer
500
400
Expert
127
SEs heatmap.
(b) Attention sinks.
Attention sinks.
Expat
Exper
(d) Router scores distribution sink token.
(e) Router scores distribution non-sink tokens.
Figure 28: SE heatmaps, attention sink visualizations, and router score analyses for sink and non- sink tokens in Qwen3-30B-A3B based on the Pile-of-Law dataset. SEs are highlighted with arrows.
1000
Layer
Head
Layer
200
0
Expert
127
(a) SEs heatmap.
(b) Attention sinks,
(c) Attention sinks.
Oven3-300438 Accverin Clatriuution (Medile
Ower 208-420 Enaent Azivaten Distouttin hand Tchera (PedDerigg
(d) Router scores distribution of sink token.
(e) Router scores distribution of non-sink tokens.
Figure 29: SE heatmaps, attention sink visualizations, and router score analyses for sink and non- sink tokens in Owen3-30B-AA3 based the MedDialog dataset. SEs are highlighted with arrows.
32


Reviews:
[{"review_id": "YHdhfawtDT", "reviewer": "Reviewer_r91y", "summary": "This work identifies a rare subset of Super Experts in MoE LLMs, which generate extreme activation outliers that propagate through the model and trigger massive hidden state activations. The study further reveals that SEs serve as the primary source of systematic outliers by strongly activating attention sink tokens; compressing them severely disrupts attention sinks, ultimately causing repetitive and uninformative model outputs.", "questions": "see weakness-1", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents the first rigorous identification and mechanistic explanation of \"Super Experts\" (SEs)—the primary source of systematic outlier phenomena, whose removal severely impairs performance, particularly in reasoning tasks.\n\n2. By linking expert-level routing dynamics to Transformer-wide outlier mechanisms in MoE models, this work offers a principled account of known empirical phenomena and reveals critical vulnerabilities in current expert compression methods.", "weaknesses": "The identification of SEs heavily relies on an empirical threshold defined in Equation (6), where experts are selected based on exceeding the 99.5-percentile activation magnitude. This criterion appears rather extreme and ad-hoc, raising concerns that the observed stability of SEs may stem from the definition itself rather than reflecting an intrinsic property of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YbxgBri9er", "reviewer": "Reviewer_vNxZ", "summary": "This paper investigates the heterogeneous importance of experts in Mixture-of-Experts (MoE) large language models. The authors report the discovery of a small, distinct subset of experts, which they term \"Super Experts\" (SEs). The central claim is that these SEs, despite being extremely rare (e.g., 3 out of 6,144 in Qwen3-30B-A3B), are fundamentally critical to the model's performance.\n\nThe authors provide strong empirical evidence showing that pruning just these few SEs causes a catastrophic collapse in model performance, including massive increases in perplexity and near-zero accuracy on reasoning tasks. In contrast, pruning a large number (e.g., 1000) of non-SEs has a negligible impact.\n\nMechanistically, the paper links SEs to the phenomenon of \"massive activations\" (MAs). It proposes that SEs are characterized by rare but extreme activation outliers in their down_proj output. These outliers propagate through residual connections, creating MAs in the hidden states. The paper further hypothesizes that these SE-generated MAs are the primary source of the systematic outlier mechanism in Transformers, and that their function is to create and maintain \"attention sinks\" (ASs). Pruning the SEs breaks this mechanism, causing the attention sink to collapse and, consequently, the model's performance to degrade, often resulting in repetitive and uninformative output. The authors also find that SE distribution is model-specific, data-agnostic, and stable through post-training.", "questions": "1. The analysis in Appendix H on \"Super Weights\" (SWs) is fascinating and, in my view, central to the paper's claim. It shows that pruning just a few neurons can replicate the catastrophic failure. To de-conflate the \"expert\" vs. \"neuron\" contributions, have the authors considered two complementary experiments? a) prune only the SWs identified in Appendix H, but leave the rest of the SEs' parameters intact. b) prune the entire SE except for the identified SWs. These experiments would clarify whether the model relies on the entire expert or just on these specific activation-generating channels.\n\n2. Would the authors consider a more descriptive and less misleading name for this phenomenon to more accurately reflect the paper's findings? \n\n3. Can the authors please clarify the definition of the layer set $L$ (layers responsible for MA formation) in Algorithm 1? How is this set determined a priori? How does this definition mechanistically exclude the final layers, even if they produce large outliers? Is the only reason for their exclusion the post-hoc empirical finding that pruning them does not hurt performance?\n\n4. Could authors share insights on why the last layer produces more outliers in the expert output?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Interesting Empirical Finding**: The paper identifies a clear and striking phenomenon: a tiny, identifiable subset of experts is responsible for the vast majority of model stability. The empirical evidence, particularly the stark contrast between pruning SEs versus a large number of random experts (Fig. 1, Tables 3-5), is convincing.\n\n2. **Plausible Mechanistic Explanation**: The paper provides a commendable in-depth analysis of why these experts are so critical. It successfully links the MoE structure to the known \"massive activation\" and \"attention sink\" phenomena in dense Transformers. \n\n3. **Thorough Investigation**: The authors analyze this phenomenon across multiple models (Qwen, DeepSeek, Mixtral) and datasets, demonstrating that the presence of SEs is a consistent pattern. The analysis of router scores (Fig. 6), showing that sink tokens are preferentially routed to SEs, is a key piece of evidence for the proposed mechanism.", "weaknesses": "1. **Is this a new discovery or a restatement?** The paper's core finding is that SEs cause MAs, which in turn create attention sinks. The phenomenon and importance of massive activation are already known in the established literature. The paper's main contribution seems to be locating the source of MAs within specific experts in MoE models. In appendix H, the author also provides analysis on locating superweights within super experts, causing massive activation. It's unclear to me whether \"Super Expert\" is a new concept or just \"the expert containing the MA neurons.\" The framing suggests a novel discovery, but it might be more of a localization and restatement of an existing concept\n\n2. **\"Super” experts could be misleading**: My **most significant concern** is the framing of the discovery. The term \"Super Expert\" implies these experts are qualitatively \"better,\" \"stronger,\" or has a superior capability. However, the paper's own evidence strongly suggests they are highly specialized components. Their function appears to maintain the attention sink. Their \"importance\" stems from the fact that the model's internal mechanism is critically dependent on them; pruning them removes attntion sink, causing collapse. This is different from removing the \"smartest\" expert. This naming convention is misleading and obscures the paper's core finding, which is about a specialized mechanism, not capability.\n\n3. **Expert-level or neuron-level phenomena**: The paper frames this as an expert-level discovery. However, the mechanism is attributed to \"extreme activation outliers\" which, as investigated in Appendix H, can be traced to specific \"Super Weights\" (SWs) or neurons. This raises the question: is this an expert-level phenomenon, or are we simply observing the location of a few critical neurons (which cause massive activation)? The findings in Appendix H, where pruning just the neurons causes a similar collapse, suggest the latter. This point significantly muddies the central claim that this is about \"experts\" as a unit.\n\n4. **Excluding final-layer activations is unconvincing**: The paper notes (primarily in Appendix C) that other experts, particularly in the final layers, also exhibit extreme activation outliers. These experts are excluded from \"SEs\" because they \"do not contribute to the formation of MAs\" and pruning them does not impact performance (Table 6). This reasoning feels post hoc and incomplete. In my opinion, it seems that removing MA in the final layer does not destroy the attention sink and thus does not destroy the performance. But this also corresponds to weakness 2; the findings in this paper should not be framed as identifying the super expert in the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zob6fJlHBo", "reviewer": "Reviewer_Jn1W", "summary": "The authors introduce the concept of Super Experts (SEs) in Mixture-of-Experts (MoE) Large Language Models (LLMs), identifying them as the main source of activation outliers, a phenomenon analogous to the massive activations (MA) seen in dense LLMs. These SEs are few in number, yet their pruning severely degrades performance. The extreme activation outliers causing this behavior are found to arise specifically from the `down_proj` operation within the expert layers. A series of experiments confirmed that SEs share key characteristics with MA, such as being model-specific, data-agnostic, and having a relationship with Attention Sink (AS) tokens. Furthermore, their distributions are noted to remain unaffected after post-training, and their impact was demonstrated across both reasoning and non-reasoning tasks.", "questions": "1. Since Super Experts (SEs) and Massive Activations (MAs) are hypothesized to function as an implicit, input-independent bias term, one would expect them to be primarily localized in global components, such as the shared expert in models like DeepSeek-R1, which is universally activated by all tokens. Therefore, could the authors provide a detailed explanation as to why SEs appear in both normal (token-specific) and shared experts, as demonstrated by the distribution shown in Table 7?\n\n2. The original research on Massive Activations (MAs) in dense LLMs demonstrated that their presence could be eliminated by introducing an explicit, input-independent bias term to the hidden states. Have the authors investigated whether implementing a comparable, explicit bias term technique in MoE models can eliminate the appearance of Super Experts (SEs) or significantly mitigate their performance-critical reliance on them?\n\n3. A key contribution of this work is the distinction between activation outliers in MoE and dense LLMs. The observation that \"Unlike dense LLMs, where such behavior typically occurs within a single layer, MoE models exhibit the progressive formation of systematic outliers by SEs across multiple layers.\" (illustrated in Figures 7 and 19) is particularly intriguing but lacks detailed discussion. Could the authors elaborate on this \"progressive formation\" phenomenon, providing a comparative analysis of why this multi-layer progression is necessary or emerges in the MoE architecture, while the analogous behavior in dense LLMs appears to be self-contained within a single layer?\n\n4. Could the authors include an extra section that summarizes the similarities and differences between the activation outlier phenomena in dense LLMs and MoE models (maybe in the appendix)? This section should detail their shared characteristics, such as both being model-specific, data-agnostic, and having a relationship with Attention Sink (AS) tokens, while clearly highlighting key distinctions, such as the outlier source transitioning from a single-layer event in dense models to a multi-layer progressive formation in MoE architectures, and so on.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work is the first to analyze the source of massive activations (MA) in Mixture-of-Experts (MoE) LLMs, providing a quantitative definition for this source by introducing the concept of Super Experts (SEs). The paper is well-written and easy to follow, and its core claims are supported by extensive experimental evidence.", "weaknesses": "See questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PzjAVBwz4f", "reviewer": "Reviewer_mM7h", "summary": "This paper investigates “super experts\", specialized experts within large Mixture-of-Experts language models that disproportionately influence performance. The authors identify such experts through gradient-based sensitivity analysis and routing statistics, arguing that a small subset of experts accounts for the majority of downstream quality.", "questions": "See weakness.  \n\nAre “super experts” persistent across training epochs, or do they shift over time?\n\nIt would be valuable to provide per-domain heatmaps showing which experts dominate specific token types or tasks.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[1] Introduces the super-expert hypothesis, offering a conceptual lens distinct from purely performance-based pruning. Connects findings to neural specialization, offering a biological analogy that enriches interpretability discussions.  \n[2]  Strong empirical evidence across diverse MoE models (e.g., Mixtral-8×7B, DeepSeek-R1) and tasks (MMLU, GSM8K, CodeEval).   Good balance between theoretical framing and empirical results.    \n[3] Clear motivation and well-structured narrative.", "weaknesses": "[1]  If “super experts” arise due to overexposure to specific domain tokens, the effect might reflect data imbalance rather than emergent specialization. The paper does not control for or analyze this possibility.    \n[2] Results focus on held-out benchmarks similar to pretraining data. It is unclear if retaining only super experts maintains out-of-distribution robustness.   \n[3]  There is little theoretical explanation for why certain experts become super experts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)