Understanding Cross-layer Contributions to Mixture-of-Experts Routing in LLMs
Authors: Wengang Li
Lingqi Zhang
Toshio Endo
Mohamed Wahib
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Mixture-of-Experts
Routing Mechanism
Primary area: interpretability and explainable AI

Abstract: Mixture-of-Experts (MoE) has been a prevalent method for scaling up large language models at a reduced computational cost. Despite its effectiveness, the routing mechanism of MoE still lacks a clear understanding from the perspective of cross-layer mechanistic interpretability. We propose a light-weight methodology at which we can break down the routing decision for MoE to the contribution of model components, in a recursive fashion. We use our methodology to dissect the routing mechanism by decomposing the input of routers into model components. We study how different model components contribute to the routing in different widely used open models. Our findings on four different LLMs reveal patterns such as: a) MoE layer outputs usually contribute more than attention layer outputs to the routing decisions of subsequent layers, b) *MoE entanglement* at which MoE firing up in layers consistently correlate with firing up of MoE in subsequent layers, and c) some components can persistently influence the routing in many following layers. Our study also includes findings on how different models have different patterns when it comes to long-range and short-range inhibiting/promoting effects that components can have over MoE in subsequent layers. Our results indicate the importance of quantifying the impact of components across different layers on MoE to understand the routing mechanism.

Published as a conference paper at ICLR 2026
UNDERSTANDING CROSS-LAYER CONTRIBUTIONS TO MIXTURE-OF-EXPEE ROUTING IN LLMs
Wengang Li', Lingqi Zhang', Toshio Endo', Mohamed Wahib
Institute of Science Tokyo RRIKEN
ABSTRACT
Mixture-of-Expee (MoE) has been a prevalent method for scaling up large lan- guage models at a reduced computational cost. Despite its effectiveness, the routing mechanism of MoE still lacks clear understanding from the perspective of cross-layer mechanistic interpretability. We propose light-weight methodology at which we can break down the routing decision for MoE to the contribution of model components, in a recursive fashion. We use our methodology to dissect the routing mechanism by decomposing the input of routers into model components. We study how different model components contribute to the routing in different widely used open models. Our findings on four different LLMs reveal patterns such as: a) MoE layer outputs usually contribute more than attention layer outputs the routing decisions of subsequent layers, b) MoE entanglement at which MoE firing up in layers consistently correlate with firing up of MoE in subsequent layers, and c) some components can persistently influence the routing many following layers. Our study also includes findings on how different models have different patterns when it comes long-range and short-range inhibiting/promotii that components can have over MoE in subsequent layers. Our results indicate the importance of quantifying the impact components across different layers on MoE to understand the routing mechanism.
effects
INTRODUCTION
Transformer-based Large Language Models (LLMs) (Vaswani et al.
2017) have demonstrated
powerful and versatile capabilities in recent years (Achiam et al. 2023 Comanici et al. 2025). To improve their capabilities. researchers have attempted to increase the model size as encouraged by scaling laws (Kaplan al. 2020) and emergent abilities (Wei et al. 2022) of LLMss. However, this can lead to high computational cost. Mixture-of-HE -ixture-of-Exxpetss (MoE) (Jacobs et al. 1991; Shazeer et al. 2017, al. (2022) has been applied L.MMS as an effective method to scale up models and alleviate those unfavorable effects, as it can reduce computation by routing the input to subset of experts instead of using all model parameters to process
Although MoE have achieved great success in many advanced LLMs such Grok-1 (xAl 2024)
and Gemini 2.5 (Comanici et al.) 2025). there is still a lack of understanding of how components in different layers affect how the routing mechanism works, from the perspective of cross-layer mechanistic interpretability. Previous studies mainly investigated the routing mechanism at the expert-level. Muennighoff et al. (2025) and Tiang al. (2024) performed analyses on the domain or token specialization of experts. Muennighoff et al. (2025) studied the co-activation of experts in the same layer. Lo al. (2025) inspected the weights of MoE (including routers and experts), gate scores, and expert outputs using similarity and norm metrics. These studies heavily investigated the correlation between experts or between experts and tokens and overlooked the interaction between routers and other components in the model 2
In this work, we aim to explore and understand the routing mechanism by recursively decomposing the input of routers into components and studying how they contribute to the routing. In summary, our observations and conclusions, from studying four different models, are as follows:
Cooee available at: II this work, we mainly use the word "component" in layer to refer any of the following: individual attention head, attention layer, individual expert in an MoE, and full MoF FFN inclusive of all its experts.

---

Published as a conference paper at ICLR 2026
Our results reveal that MoE routing cannot be understood as purely local process; rather, it emerges from intricate cross-layer interactions among model components.
MoE layer output usually has the strongest influence the routing rather than other components except for the first few layers. However, tokens and attention layer output may have a stronger influence in small group area such as the bottom and the top layers.
Routing is shaped not just by local computations, but also by long-range cross-layer entanglements, challenging the assumption that routing decisions are primarily local.
few experts have a significant influence on the routing decisions, consistent with the effect reported in (Su et al. 2025).
2 RELATED WORK
Mixture of Experts. Mfxture-of-Exxee (MoE) was first introduced in Jacobs et al. (1991) and has been studied for decades. [Shazeer et al. 2017)) proposed sparsely-gated MoE (SMoEs) method to scale up deep learning models with reduced inference overheads. They introduced the top-K routing algorithm, which has become dominant paradigm nowadays for its simplicity and effectiveness. Researchers have put much effort in optimizing the design of SMoEs (Fedus et al. 2022). ). Recent studies have also attempted to understand the mechanism of the MoE layer (Chen al. 2022) and
have discovered the routing scores can be applied in model compression (Li et al.) 2024) or used as an embedding model (Li & Zhou| 2025). Lo al. (2025) made an early attempt to analyze MoE-based language models by observing the correlations and norms of some components related to the MoE layer, such as experts and gate scores.
Decomposition of Transformers. combination of outputs of its internal components, facilitating the dissection and understanding of Transformer-based language models (Elhage et al. 2021 Geva et al., 2021 Yu & Ananiadou 2024] Ferrando & Voita 2024). These methods decompose the outputs of attention or Feed ferw-Forwwrd Net- work (FFN) layers into smaller component vectors that can be further studied. The assignment scores assigned to experts can also be decomposed as the sum of sub-scores assigned by the components. Hence, we can study the distribution of the sub-scores understand how these components influence routing decisions.
The output of Transformer blocks can be decomposed as linear
BACKGROUND
In this section, we present recursive decomposition of the architecture of the MoE-basee Transformer into components that together comprise the assignment score of MoE routing.
MoE-based Transformer. An MoE-based decoder-only Transformer consists of L blocks. Each block consists of an attention layer, followed by a Mixture-of-Expeee token embedding sequence T [41, (2,2,,,,, ..., tu. to the model, the first layer input a RRde the token embedding t, where de is embedding dimension. The block output yout, (Token i, Block E) formulated as follows:
(MoE) layer. Given an input
#out., = a'n., + abut. + T where wf.,; is the input of Block (xf := x f- out,i for l > 0), a and m m are the outputs of attention and MoE Layer respectively. The final block output Fout, is normalized by layer normalization and then projected onto the vocabulary space yield the probability distribution of the next token.
(I)
Attention Layer. The attention layer output a E Rd can be decomposed into the linear combination of the head outputs out,, which can be further decomposed at the token level:
H
H
aout,
aout.
A
vp
J.
(2)
h=
k=1p=1
The element of attention map c Buxu is computed by:
9. A.A.
so soflmer 1SPSi
p p
A f.h
V4z
(3)
0
p>i
2

---

Published as a conference paper at ICLR 2026
where dj is key dimension, qi f.h kit R wh Id are query, key, value vectors, respec- tively. Mathematically, qi We an ke = W
K Ginps & W af din.p where wo w E gd xds, w E gd xde, whth E idd are query, key, value and output weight matrices of Head h in attention Layer E, dn is head dimension, a is the attention layer input:
(4)
where (.) is layer normalization. RMS layer normalization is applied in the tested models in this znd
work, hence LN (z) : RMS(*)' where 2 E Rd is the input vector, RMS(-) is the root mean square function, and xf E I is a learnable parameter.
MoE Layer The MoE layer consists of router and N experts, i.e. N parallel sub-FFN layers. The router assigns a score to each expert and selects the top-K experts to process the MoE layer input m where K is hyperparameter. Mathematically,
m., (N' + a.uu.s).
(5)
Typically, the scores of all N experts computed by Wom where W E Nxx, is the routing weight matrix F Each row vector g Rdd of the routing weight matrix We corresponds to one expert. We call these row vectors "routing weight vectors". The assignment score S of Expert n4, i.e., Expert n in MoE Layer f, is essentially the dot product of its corresponding routing weight vector gf and the MoE layer input m
Sig fm min.a) =g minn.i.
(6)
The assignment scores are passed through a Softmax function to yield expert weights. The MoE layer output but, is the weighted sum of outputs of selected experts:
Moutt
(miin De
OEJ
where (.)( softmor(S(g is the expert weight, out, is the expert output, and J is the jEJ
set of indices of selected experts.
METHODOLOGY
For each given input token, the MoE router assigns scores experts and selects the top-K experts process the input. Assignment scores have two determinants: the routing weight vectors and the MoE layer inputs (Equation 6). Intuitively, assignment scores are approximately decomposable since the MoE layer input can be decomposed into components (Equations I and 5}. To understand the routing mechanism, we can study the patterns of scores "assigned" by the components In this section, we delineate the decomposition method in Section 4.1 and discuss some basics of scoring patterns in Section
4.1 DECOMPOSITION OF EXPERTS ASSIGNMENT SCORES
In this section, we discuss how the assignment score of an expert can be decomposed into sub-scores assigned by components of different granularities, from entire layers to individual neurons. We define an approximation operation (c) = RMS(=) where c is an arbitrary component and z is the original input (z). This approximation distributes the same normalization factor across all
end
"Soe MoE models also applicable those models, but for simplicity, discuss only the most widely used implementation of the routing mechanism.
sophisticated mechanisms such MLP layers obtain scores. Our method is
*When we say the score assigned by component to expert, we refer to the portion of the expert's score contributed by the component.
3

---

Published as a conference paper at ICLR 2026
(0 Assignment expert decompased scores
assigned by all previous components le., tolkens, attention FPN layers.
Ancation Layer
Mok Layer
Aftention Layer
MoE Layer -
Attention Layer
Moll Layer
Export
Embeddiing
MeE
Scores can be further decompased into fine-grained level, heads scoring behavier components.
Attention Heads
Experts
the
Figure 1: Overview of the decomposition of experts assignment scores.
components. We suggest that this is acceptable definition for the contribution of to IN (c) in the context of studying routing decisions since it preserves the direction of the projected component and keeps their relative magnitudes faithful to what the router actually receives. We can recursively apply Equations
! and 5 to decompose the score assigned to the Expert (e, n):
Sig min,))
Min.i =g g (N' + dint.s))
git (N aout,
+
Σ mout.))
(8)
LN on
LN LN (a +
LN (m
Equation g indicates that the assignment score can be decomposed into multiple sub-scores, i.e..
the scores assigned by the token embedding (S(gein, x g LN (=!n, ()), attention layer outputs, and MoE layer outputs. Both attention and MoE layer outputs can be further decomposed. According Equation| 2 the score S(g".". f" a.u,,) assigned by attention layer output af can be further decomposed as follows:
Sig aout,s) gf fm LN: =
LN A0p (9) (N = fen ΣΣ hm1pm1
1m
From this perspective, we can regard the score assigned by an attention layer output to an expert as the sum of the scores assigned by tuples like (head, query, key) (Equations and 9.. Similarly, we can decompose the scores S m assigned by the MoE layer output m into the scores assigned by the selected experts (Fquation
Sig m = f.a LN, (m = g.n
> (r (m out.
(10)
yel
It is possible to further decompose the expert (i.e., sub-FFN) outputs at the neuron level (Geva et al. 2021
Dai et al. 2022 Geva et al. 2022h:
W WaG
(o m...)) (W (W m)..)))
((3)))
(11)
where all of the matrices correspond to Expert (e,j), W fy is the z-th column of down
(1.e)
matrix, W Ed and W &J
(2.1) the z-th row of gating matrix and up-projection matrix, respectively,
(2.)) a(.) an activation function. We leave for further study, considering its potential complexity.
4.2 BASICS OF SCORING
In Section we proposed method for determining the scores of experts assigned by the compo- nents. In this section, we discuss what can be learned from the scores assigned by components. The proofs of our propositions in Appendix A
4

---

Published as a conference paper at ICLR 2026
Proposition 1: Variance of scores assigned by component measures its influence on the routing decision. Suppose a component assigns a constant score to all experts, which means its scoring variance is 0, then it does not influence the routing decisions because if the scores it assigns dropped, the differences between the scores of experts are unchanged. Intuitively, we posit that a component with higher scoring variance has stronger influence on the routing decisions and thus is more important. Based on this postulate, we can further infer that the length (i.e., L2-norm) of the component influences the upper bounds of the variance of scores assigned by it, which indicates that strong influences on the routing decisions tend to be associated with components with large norm.
Positive scores promote experts, negative scores inhibit experts. The degree is measured by the score magnitude. Since the scoring operation is a dot product of a gating weight vector and a component vector (Equations 6 8 and 10P, the angle between them controls the sign of the score. When the angle is acute (obtuse), the score assigned by the component to the corresponding expert positive (negative), indicating the component promotes (inhibits) the expert to be selected. When the two vectors orthogonal, the score is zero, indicating the component has no bias on the corresponding expert. The magnitude of the score is controlled by the length of the two vectors and the angle between them. Hence, if we fix a gating weight vector, then component with a smaller angle and larger length will assign a higher score to the corresponding expert.
Proposition 2:
5 SCORE DISTRIBUTION
In this section, we show empirical results on the assignment score distribution of experts assigned by tokens, attention layer outputs, and MoE layer outputs, respectively.
5.1 EXPERIMENTAL SETUP
Models and dataset. We adopt four MoE-based language models, scaling from OLMoE (Muen
nighoff et al. 2025). DeepSeek- V2-Lite (Liu et al. 2024). Qwen3-30B-An (Yang al.
2025).
to Mixtral-8x 7B (Jiang et al.] 2024). Their basic information is summarized in Appendix B We
randomly select samples with least 32 tokens from C4 dataset (Raffel et al.) (020) and truncate each
to the first 32 tokens to simplify the experiments. We use 1000 and 5000 samples for the experiments
in Section 5.1 and 5.3] respectively. In this section, report the results from OLMoE in the main text. The results of the other three models are reported in Appendix D
Metrics. We use the variance of scores assigned by component to the experts an MoE layer to measure the contribution of the component to the routing decisions of that layer: The variance of scores 81, s., 8N is N Es 1a)², where (promotion or inhibition) of component to set of specified experts, we use the average positive score (APS) and average negative score (ANS) assigned to those experts by the component:
N E 8a. To find the scoring tendency
1 APS
1 N Σ
N Σ S(g, c) ANS
(12)
where N the number of experts and is the indicator function. We separate the positive and negative scores to avoid cancellation. Variance, APS, and ANS are computed per token position, then averaged over non-lead tokens, and then averaged over samples. We report the results related the lead position in Appendix since we observe that they have a different behavior, compared with tokens on other positions.
5.2 RESULTS ON TOKEN SCORING
Token scoring is influenced by its part-of-speech (POS). We pack the scores of all experts assigned by each token into vector and apply t-SNE(Maaten & Hinton 2008) to visualize them. From Figure[]. we find that most scores assigned by tokens are clustered according to the POS tags of tokens, which implies that the POS influences token scoring. Furthermore, most function words are clustered into isolated lumps. In contrast, content words are entangled, which is possibly because
'If word is split into multiple tokens, each token inherits the POS of the word.
5

---

Published as a conference paper at ICLR 2026
10+
10
Rocciv Mae Layar
(a) Token Part-of- Speeech Distribution
(b) Token Position Distribution
(c) Token Routing Variance
Figure 2: (a) t-SNE of scores assigned by token embeddings, colored by POS. (b) t-SNE of scores assigned by token embeddings, colored by position. (c) Average variances of scores assigned by tokens to MoF layers.
function words may have multiple POS, e.g., "drink" can be noun or a verb, whereas the semantics of function words is more stable.
The lead token is special. In Figure Db the lead tokens of each prompt appear clustered, whereas other tokens, in comparison, are not, indicating that the lead position has noticeable influence on scoring. We find the scoring distribution of attention and MoE layers at the lead position also has typical pattern. We speculate that the attention layers capture the position information, lead to the norm of MoE layer output at the lead position to being typically larger than that at other tokens, which influences the RMS term of layer normalization and finally influences the scoring distribution. In light of this, we report the results excluding the lead position in the main text henceforth and report the results related to the lead position Appendix D
In Figure Z the average variance of scores assigned by tokens decreases rapidly as the layer goes deeper, showing that the influence of tokens on routing decisions decreases rapidly. Furthermore, the variance of scores in the first two MoE layers assigned by tokens is typically high, indicating the tokens have a strong influence on the routing decisions in them, which is in line with intuition. The magnitude of ANS and APS also decreases as the layer goes deeper (see Appendix D).
5.3 RESULT ON ATTENTION/MOE LAYER OUTPUTS
In this section, we analyze the average scoring variance of the attention and MoE layer outputs and the average positive/negative scores assigned by them, to study their influence on the routing decisions.
Average variance. As shown in Figure 3p. the highest variance of attention layer scoring occurs at AO + MO, i.e., the assignment from sending attention Layer 0 to receiving MoE Layer The variance diminishes gradually the receiving layer appears deeper. Comparatively. the first two attention layers (AO and A1) exhibit a higher variance than the intermediate sending attention layers, although the magnitude of their scoring variance is still low. We compare the four models tested and find that the early sending attention layers generally have a high average variance to their neighbor receiving layers. Surprisingly. few MoE layers have pronounced entangled influence on the routing decisions on following MoE layers (Figure 3d): Sending layers MI and M4 have a notably higher variance compared with others, indicating they have higher importance. It also unique that their variance does not decrease monotonically. We will show that two experts with typically high variance in these two layers cause the "stripes" (Section 77]. We also find such stripes DeepSeek, Qwen and Mixtral, but some of them may just have apparently higher variance compared with neighbor sending layers, instead of having an increasing average variance on the receiving layers. The stripes also occur in sending attention layers in Qwen. Finally. the variance of scores assigned by sending MoE layers is usually comparatively higher than that assigned by sending attention layers, except for the bottom receiving layers M0~M33 indicating that the MoE layer output usually has stronger influence on the routing decisions than the attention layer output.
These findings suggest two key implications. First, load balancing of expert parallelism can be improved by prefetching and preloading experts high-variance layers, leveraging cross-layer entanglement reduce contention. Second, post-NAS approximation strategies Gu et al. (2025)

---

Published as a conference paper at ICLR 2026
Sending Layer
5
15
10
10
15
-0.8
Attention-Mot Receiving Layer
0.6
5
0.10
0.6
0.4
10
0.05
0.2
0.2
15
(a) Varlance
(b) APS
(c) ANS
0.20
MoE-MoE Receiving Layer
0.15
5:
0.4
0..5
0.10
0.10
10
0.2
0.05
0.05
15
(d) Variance
(e) APS
(f) ANS
Figure 3: Scores assigned by attention Layer x to MoE Layer y: (a) Variance, (b) Average positive scores (APS), (c) Average negative scores (ANS); Scores assigned by MoE Layer x to MoE Layer y: (d) Variance, (c) APS, (f) ANS.
selectively compress low-variance attention layers while preserving influential ones, enabling more efficient yet accurate architectures.
Scoring pattern of layers. We conducted an experiment to analyze the scores assigned to all the experts, rather than the selected ones. By observing ANS and APS, we can learn which components promote or inhibit the experts in specific layer. We first look at the scoring pattern of sending attention layers. Comparing Figure and c, we note that the positive and negative scores oceur in different areas: positive scores occur at the early sending layers i.e., left side of x-axis (AO and A1), whereas negative scores occur at AO, and most areas on or near the diagonal, and the magnitude increases as the sending layer goes deeper. In other words, promotion assigning positive scores to experts-tos -end to be local, whereas inhibition effects are more global, with components exerting stronger inhibition the receiving MoE layer appears deeper in the model. Comparing the tested models, we find that the negative scores tend to occur at the bottom or top sending attention layers. The magnitude of APS is generally smaller than that of ANS, although Mixtral seems to be an exception.
components
The scoring pattern of sending MoE layers is shown in Figure and Sending M1 and M4 have relatively high positive and negative scores, respectively, which conforms to the variance matrix (Figure 3t.. The positive M14) and the diagonal, whereas the negative scores appear at sending MI, M4, and the area near the diagonal. We also find the APS and ANS matrices in Qwen have multi-stripe pattern, ie.. more prominent entanglement effect, whereas other tested models show only few or no stripes.
mainly appear sending MI, M4, deeper sending layers (M9
SCORING OF ATTENTION HEADS
In this section, we investigate the scores assigned by attention heads to the experts in different layers. We continue to use C4 dataset observe the general behavior of the scoring, Additionally, we use Indirect Object Identification (IOI) (Wang et al. 2023) task to study how function heads influence the routing decisions and how the variance of scores relate to the scoring patterns.
6.1 EXPERIMENTAL SETUP
General test. We conduct the general test OLMoE and DeepSeck (Appendix G). which have fewer heads (256 and 432) and thus facilitate our analysis. We follow the setting in Section [.] and use 5000 samples for experiments.
7

---

Published as a conference paper at ICLR 2026
Attention Head
10
15
5
10
0
10
15
Sending /Receiiiig Layer
Attention Head-MoB
0.
1
0.20
- -0255
- -.20
5
0.15
0.15
0.10
10
0.10
10"4
0.05
-0.05
15
(a) Variance
(b) APS
(c) ANS
Figure 4: Scores assigned by heads to the experts in the same block: (a) Variance, (b) Average positive scores (APS), (c) Average negative scores (ANS).
IOT task is to predict the next token of a prompt like "When Mary and John went to the store, John gave a drink clause is expected to be the prediction result, i.e., "Mary". We adopt this task to observe if function heads have a noticeable influence on the routing decisions. We use path patching (Wang et al., 2023 We can regard the scores assigned by head as
where the name that exists in the first clause but does not appear the second
to find portion of the function heads in OLM [ "score map" once we determine metric for tuples like (head, query, key). We use the variance of scores assigned by the tuple (head, query, key) to all experts in MoE layer the metric. We refer to it as the score variance map and compare it with the results from path patching and the attention map.
6.2 RESULTS ON GENERAL TEST
Since attention layer outputs usually exert the strongest influence on the routing decisions in the same block in OLMoE, examine the variance and mean of scores assigned by heads to the experts in the same block to facilitate the observation (Figure D. Henceforth, we use "AxHy" to denote Head attention Layer x. Heads with high variance mainly occur at the early layers (typically AO and AI), conforming to the results in 3a. By comparing Figures 4p and we find that many attention heads tend to assign negative scores to the experts, and a small proportion of heads tend generally assign positive scores, such A7H15 and A9H6. The scores assigned by some heads are apparently polarized, such as AOHI. Finally, we observe that a head has a relatively higher scoring variance, the absolute magnitude of scores assigned by the head is usually larger.
6.3 RESULTS ON IOI TASK
We employ IOI task as an example to study the scoring pattern of function heads. The visualized results are shown in Appendix F Our main findings this task follows:
Function heads have a noticeable influence on the routing decisions. We find that function heads, ie.. attention heads that contribute to finish the IOI task, usually have higher scoring variance, compared with the heads do not show any clear function in the task, indicating that the function heads tend to have stronger influence the routing decisions than the other heads.
Scores variance of attention heads correlate with attention maps. We compare the "attention map" A (where query token t, and key token t, are fixed, Layer and Head y are variables), with the "score variance map", that is, variance of scores assigned by tuples (head=AxHy, query=t, key=t,), (
in other words, terms LN (W?:9) A " 2) for AxHy (Equation 9 to the experts in a specific layer e(c z). We find that they have simillar pattern, which is in line with our intuition since attention map can manipulate the attention output and thus influence the scoring.
For simplicity, we do not aim discover all the function heads activated in the IOI task in this work.
8

---

Published as a conference paper at ICLR 2026
M1E9
M1E68
M24E111
5-
MB1E56
M4E14
M3E82
MB3E69
Other Experts
0.6
M21E69
Other Experts
Variance
M22E92
3-
2-
0.2
1-
10
15
0
10
20
50
Receiving Layer (a) OLMoE
Receiving Layer
(b) Qwen
Figure 5: The variance of scores assigned by experts to the following layers.
SCORING OF EXPERTS
In Section 7.1 use variance measure the influence of experts on routing decisions in subsequent layers. In Section 5.3 we have seen that the variance of scores assigned by MI and M4 in OLMoE has an unusual phenomenon: the variance does not decrease monotonically. We find that two experts contribute to the phenomenon. Furthermore, small subset of experts has strong influence on the routing decisions in OLMoE and Qwen3. Intriguingly, among these experts, some maintain their influence on the routing decisions till the last layers rather than decreasing significantly. We further propose a new metric named "average absolute ranking variation of top-K experts (AARV)" and use to conduct small causal analysis to observe large-variance expert found in Section 7.1 case study in Section 7.2
7.1 MAIN FINDINGS
OLMoE (p). MIE9 (Experts 9 at MoE Layer 1) and M4E14 exhibit an unusual phenomenon where their influence peaks around MoE Layer 6, followed by secondary peak near MoE Layer 10, then consistently decreases (with MIE9 dropping below M4E14 by the end). In contrast, M2E30 shows a steadily increasing influence that reaches its peak in the final layer.
Qwen (Figure 5p). M1E68 (circled in red) exhibits a strong but localized influence primarily on the immediate next layer. M2E92 consistently maintains the highest impact across all subsequent layers. M3E82 also shows persistent influence throughout, though with smaller magnitude. Other highlighted experts (M21E69, M22E92, M24E11 11, M31E56, M33E69) from middle and late layers generally follow the typical pattern of gradually building to peak influence before fading out.
We find some of these experts are the "Super Experts" identified by Suu al. (2025), which have an output of extreme magnitude. We summarize the rank of variance of scores assigned by the Super Experts the experts the next MoE layers in Appendix H We find that not all the Super Experts have typically large scoring variance, such as M2E54 in DeepSeek. Although M1E68 in Qwen has the rank variance among the experts in Layer 1, but its scoring variance decreases drastically (Figure [p)). The Super Experts in DeepSeek do not have top rank nor a persistent high scoring variance. The scoring variance of Super Expert Mixtral (M1H3) has a unique distribution: has relatively high variance in the layer 28 ~ 31 (Appendix
7.2 CAUSAL ANALYSIS OF M1E9 IN OLMoE
In this section, we analyze MIE9 to study how it perturbs the top-K experts with a physical metric- "average absolute ranking variation of top-K experts (AARV)" in an MoE layer as a case study to directly justify that variance can influence the selection of top-K experts. We define the metric as follows:
1 AARV
Σ eEtop-K
rankurig (e) rankped (c)):
(13)
K

---

Published as a conference paper at ICLR 2026
Averagg absolute rank variation of tup-k axports
Receiving Layer
M1E9
M10
M15
MIE9
Expert in
Figure 6: The AARV of M5, M10, and M15 when the scores assigned by an expert in MI set to zero.
where are top-K experts in an MoE layer, rankorgg (e) is the original rank before any perturbations on the net assignment scores of Expert e, rankpen(e) the perturbed rank after we set the score assigned by a component to zero.
We the scores assigned by each expert in M1 to zero and show AARV of top-K experts in M5, M10, and M15 in Figure We observe that M1E9 has dominant influence on the routing decisions in these three layers, whereas the other experts in M1 weakly nudge the top-K decisions. The Spearman's 's coefficient results of the data in Figure i provided in Appendix N
We further conduct an ablation experiment and identify that M M1E9 controls the activation of M4E14 and they need to coexist to exert prominent influence on the routing decisions in MoF Layer 5 and the following layers (Appendix]) I
8 SUMMARY OF FINDINGS AND CONCLUSION
In this work, we proposed recursive decomposition framework quantify how different components contribute to routing decisions in expert assignment scores into contributions from tokens, attention layers, MoE outputs, and attention heads, we provided the first cross-layer perspective on routing interpretability. Our analysis across four production MoE models (OLMoE, DeepSeek-Vi Veee-iite.---..ee Qwen3-30B-A33 and Mixtral-8x7B) revealed several consistent patterns. First, MoE outputs exert the strongest and most persistent influence on downstream routing, while attention layers and tokens have more localized effeets, especially in the bottom and top layers. Second, routing decisions are shaped by both promotion and inhibition: positive contributions typically act locally, while negative contributions inhibit experts across longer ranges. Third, identified cross-layer entanglement phenomena, where certain MoE layers (e.g.. M1, M4 in OLMoE) disproportionately affect routing in much deeper layers, forming "stripes" of influence. At finer granularity, found that small of experts and attention heads dominate the routing behavior, with some maintaining their impact throughout the model. Notably, not all previously identified "Super Experts" exhibit strong influence under our vuriance-based analysis. These findings demonstrate that MoE routing is not solcly local mechanism, but instead emerges from a complex interplay of components across layers. Our work shows that the routing decisions can be studied in decomposition perspective to understand its underlying mechanism.
(MoE) language models. By breaking down
USE OF LARGE LANGUAGE MODELS (LLMs)
We do not use LLM in idea formation, experimental result analysis, or paper writing in this work. The authors take full responsibility for the contents in this paper.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding the mixture-of-experts layer in deep learning. Advances in neural information processing systems, 35: 23049-23062, 2022.
Gheorghe Comanici. Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.0626/ 2025.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8493-8502, 2022.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 1-11 2022.
Javier Ferrando and Elena Voita. Information flow routes: Automatically interpreting language models at scale. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 17432-17445, 2024.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.
2021.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 30-45, 2022.
Yuxian Gu, Qinghao Hu, Shang Yang, Haocheng Xi, Junyu Chen, Song Han, and Han Cai. Jet-
nemotron: Efficient language model with post neural architecture search, 2025. URL https
//arxiv.org/aatt http:/bb/225550 2508 15884
HuggingFace. Openrl math 220k, February 2025. URL https::/hhugiingfee co/
datasets/open
-aatnRiiMMMttt -Math-22 -200k]
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.0400: 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08.8 083361 2020.
Pingzhi Li, Zhenyu Zhang. Prateek Yadav, Yi-Lin Sung. Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient smoe with hints from its routing policy. In The Twelfth International Conference on Learning Representations, 2024.
Ziyue Li and Tianyi Zhou. Your mixture-ou of ex-eexssts 1lm is secretly an embedding model for free. In The Thirteenth International Conference Learning Representations, 2025.
11

---

Published as a conference paper at ICLR 2026
Aixin Liu, Bei Feng, Bin Wang. Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: strong, economical, and efficient mixture-of- experts language model. arXiv preprint arXiv:2405.0444 444404400444 2024.
Ka Man Lo, Zeyu Huang. Zihan Qiu, Zili Wang, and Jie Fu. A closer look into mixture-of-experts in large language models. In Findings of the Association for Computational Linguisties: NAACL 2025, pp. 4427-4447, 2025.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9NNov):257--2605, 2008.
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo. Jacob Morrison, Sewon Min, Weijia Shi, Evan Pete Walsh, Oyvind Tafjord, Nathan Lambert, et Olmoe: Open mixture-of-experts language models. In The Thirleenth International Conference on Learning Representations, 2025.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gate mixture-of-expprrs layer. In International Conference on Learning Representations, 2017.
Zunhai Su, Qingyuan Li, Hao Zhang, YuL.i Qian, Yuchen Xie, and Kehong Yuan. Unveiling super experts in ofxture-of-expprts large language models. arXiv preprint arXiv:2507.23273 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt.
Interpretability in the wild: circuit for indirect object identification in GPT-2 small. In
The Eleventh International Conference on Learning Representations, 2023. URL https
7/ /openreview..oo wtt.eeewweeeeeeeeeeeeeeeeeeeeeee http://rrrmmiiaassssoooooo-o (ogensssiei.ns//ffooraaaaaaaaaaa
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
xAI. Grok-1, March 2024. URL https:://ithubb.. xai grok-1
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint XXXx:2555..0388 2025.
Zeping Yu and Sophia Ananiadou. Neuron-level knowledge attribution in large language models. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 3267-3280, 2024.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A PROOF OF PROPOSITIONS
Variance of scores assigned by a component measures its influence on the routing decision.
Proposition (Variance of component-contribuice scores and norm of component). Fix layer and position Let 2, E RRd denote the contribution a single component (e.g., token embedding, an attention head output, or a previous MoE. block output) to the MoF-input before normalization. Define
(2.) = RMS
Ze
(I a aouts)
where A E Rede x is a fixed linear map for this (f, t). Let E RNde stack the routing weight
(g(2)T
vectors rows, G
The vector of expert scores contributed by this component is then
(T
((0) (L.)(Z)) GA E RNN, i.e., ( (A zc.
gfg
(i) Zero-variance implies no routing influence. (()) is constant across experts, i.e., s( _..-=
N then adding removing this component shifts all experts by the same constant. For any pair n, m,,
sotal 88
+
total sotal
+ c)
Sn
total
sm
total
so the ordering is unchanged. Hence constant-scor component has variance 0 and no influence routing.
(ii) Variance influence measure. Define Var(s(())= == N Σm (0 g(0)),, where z(o)
N En sh Then Var(s! 0 iff gl is constant. Moreover, for any a E R
Var(GA(az.)) = a² Var(GAze),
so larger component magnitudes yield quadratically larger variance, hence stronger influence on routing.
(iii) Norm-controlledd bounds. By Cauchy-Schwarz
| 1(AT (T
zel 1 | T n |
Let M max, | gf Then
E [-|Mlllll, M 1e] Vn,
so the range of component-contribure scores is bounded by 2M z.M]2z.ll2. Consequently.
(ar(s < IGA 1
showing that variance (and hence influence) is upper-bounded by constant depending on the router and normalization. times the squared L2 norm of the component.
Conclusion. Components that assign constant scores exert influence, while those with larger norms admit wider score ranges and potentially larger variance across experts, thereby possessing greater capacity to alter expert rankings and influence routing.
13

---

Published as a conference paper at ICLR 2026
Proposition 2 (Sign and magnitude of component-contributee scores). Fix a layer e, position i, and an expert n. Let the component contribution before routing be 2e E R and let u := LN, (ze) Rd. denote its normalized contribution this (f, (cf. Eq. 8p. The score contributed by this component to expert n is
g u.
Let 0 E [0, π| be the angle between g gf and u.
(i) Sign determines promotion vs. inhibition. By the cosine formula for the dot product,
[c) = l1g f1 |/2/////2 cos An.
Hence sh > O iff 0n E (0, (acute), sh < 0 iff 0n (², (obtuse), and 8% = 0 iff 0n = (orthogonal). Since (a) top-K selection depends only on score orderings and (b) the softmax used to form expert weights is strictly increasing in each coordinate, adding a component with sn (c) > 0
increases expert n's total score and softmax weight (promotes selection), while (c) 0 decreases them (inhibits selection); s leaves them unchanged.
(ii) Magnitude quantifies degree of influence. The absolute score satisfies
[ = |lg 112 ||//|2 cos I
llg ||u|22
with equality iff gfv and colinear. For fixed gmm the dependence angle and component length is monotone:
Osh
Osn (c)
00,, = - - ||ul|2 sin On. I 0,
||g cos On.
Thus, for 8n E [0, )), decreasing the angle (better alignment) or increasing the component norm strictly increases on
(c) for 6, (2. n). the same operations make s(s) more negative, strengthening inhibition. Consequently, the degree of promotion//nhibitonn is exactly captured by the magnitude
s(a) which grows with both alignment (via cos 0nl) and component length |/uu|2 (and hence with | up to the fixed normalization factor at (f, i)).
Conclusion. The sign of the component-expert dot product governs whether the component pro- motes or inhibits that expert's ' selection, while its maagitude--lgg" f ||u||2 cos 0, quaatiffie the strength of this effect.
B BASIC INFORMATION OF TESTED MODELS
Table 1: Basic information of tested models.
Information
OLMoE
DeepSeck-V22 V2-Lite
Qwen3-30B-A3B
Mixtral-8x7B
Total Params
7B
16B
30B
47B
Activated Params
IB
3B
3B
7B
Number of Layers
16
27
48
32
Number of Routed Experts
64
64
128
8
Top-K
8
6
8
2
DeepSeek- V2-Lite has an FFN layer instead of an MoE layer in Layer O. Each MoE layer has two shared experts DeepSeek-V2 V2-Lite.
NOTE:
C BRIEF INTRODUCTION TO IOI TASK
We apply the method (path patching) and the metric (logit difference) from (Wang al. 2023 to reproduce the experiment on OLMoE, and identify four types of heads which are active at END token, follows:
14

---

Published as a conference paper at ICLR 2026
Name Mover Heads attend to the previous name tokens. They promote the name tokens.
Negative Name Mover Heads similar to Name Mover Heads but inhibit the name tokens as the prediction result.
S-Inhibition Heads inhibit token, and influence Name Mover Heads and Negative Name Mover Heads.
Backup Name Mover Heads active when the Name Mover Heads ablated. They also show a weak influence when the regular Name Mover Heads work normally.
We adapt the code provided in the original paper to generate 5000 samples for the 101 task experi- ments.
D SUPPLEMNTARY RESULTS ON THE DECOMPOSITION AT THE LAYER LEVEL
We find that the scoring distribution at the lead tokens is completely different from other tokens (Figures 7 ~ I0). For example, OLMoE, the score assignment from sending attention layers to receiving MoE layers, the high variance occurs at the bottom sending layers to their neighbor receiving layers, and the last receiving layer. However, when sending and receiving layers are both MoE layers, the high variance occurs at some sending MoE layers and the last receiving MoE layer. The highest APS occur A15 M15, and M14 M15. The highest ANS occur at A14 - M15 (AO + MO), and M2 - M15. The four models at the lead token have different variance patterns. We can observe that there are strides (e.g., Figure () in the score variance distribution, indicating that some sending layers more influential and have persistent influence on the routing decisions. The results at other tokens on DeepSeek, Qwen, and Mixtral are presented in Figures respectively.
The results the scoring of tokens are presented in Figure 14
Sending Layer
15
5
15
5
Attention-Moo Receiving Layer
10
5-
0.6
10-3
0.4
10:
10-5
-02
15
0.0
(a) Variance
(b) APS
(c) ANS
10a
Receiving Layer
MoE-MMoE
5
10 2
1.5
10-4
1.0
0.5
-0-6
15
0.0
(d) Variance
(e) APS
(f) ANS
Figure 7: Scoring distribution at the lead tokens, OLMoE
15

---

Published as a conference paper at ICLR 2026
Sending Layer
10
15
20
25
Attention MoE
Receiving Layer
10
-0.4
0.8
-0.3
10
0.6
0.4
10-3
0.2
-0.1
25
0.0
(a) Variance
(b) APS
(c) ANS
Receiving Layer
5
10-1
0.5
0.25
MoE-MMEE
0.4
-0.20
10
10
0.3
-015
15
10
0.2
20
0.1
0.05
25
0.0
0.00
(d) Variance
(e) APS
(f) ANS
Figure 8: Scoring distribution at the lead tokens, DeepSeek
Sending Layer
0 1015202530354045
1015 25 30 40
5 1015202530354045
10
-4
Attention-Mok
Receiving Layer
10
0.04
10
15
0.03
20
10
25
30
10-4
35
10-5
0.01
40
45
10-6
0.00
(a) Variance
(b) APS
(c) ANS
01
0.8
Receiving Layer
10
5
10-1
6
MoE-MoE
15
0.6
20
10-1
25
0.4
30
35
5
0.2
40
45
0.0
(d) Variance
(e) APS
(f) ANS
Figure 9: Scoring distribution at the lead tokens, Qwen
16

---

Published as a conference paper at ICLR 2026
Sending Layer
5
10
15
20
25
0
15
20
25
30
5
10
15
20
25
30
0.4
Attention- MME
Receiving Layer
10
0.3
10
10
0.2
0.2
10-5
20
0.1
-0.1
25
10-7
30
0.0
(a) Varlance
(b) APS
(c) ANS
10
-0.8
Receiving Layer
MoE--MoE
10
10
2
-0.6
10 4
00.4
20
10-6
25
0.2
10-1
30
(d) Variance
(a)
ANS
Figure 10: Scoring distribution at the lead tokens, Mixtral
Sending Layer
0
5
10
15
20
25
5
10
15
20
25
5
10
15
20
1.0
Attention- MoE
Receiving Layer
10-1
0.3
10:
10-2
0.2
15
0.4
10-3
20
25
(a) Variance
(b) APS
(c) ANS
Receiving Layer
10
0.25
MoE-MoE
10-2
10:
0.15
15
10-1
0.10
20
10-4
0.05
25
(d) Variance
(e)
Figure 11: Scoring distribution at other tokens (except the lead tokens), DeepSeek
17

---

Published as a conference paper at ICLR 2026
Sending Layer
5 1015500530355405
5 1015202530354045
0 5 2015502533334405
0.3
Attention- MoE
Receiving Layer
10-1
10
20
15
10
0.2
25
-2
30
10-1
0.1
-1
10-4
45
0.0
(a) Varlance
(b) APS
(c) ANS
10
1.25
Receiving Layer
10
0.06
MoE-MMoE
15
10-2
-1.00
20
0.04
-075
30
10
0.50
35
40
-025
45
10-4
0.00
0,00
(d) Variance
(e) APS
(f) ANS
Figure 12: Scoring distribution at other tokens (except the lead tokens), Qwen
Sending Layer
0
5
10
15
20
25
30
10
15
20
25
30
0
5
10
15
20
25 30
0.3
Attention-MoE
Receiving Layer
10
-0.3
10
2
0.2
15
0.2
20
10-3
-0.1
25
10-4
30
0.0
(a) Variance
(b) APS
(c) ANS
0.25
Receiving Layer
10
-0.25
MoE-MMEE
10
0.20
-020
10-2
15
0.15
-0.5
10-1
0.10
0.10
25
10-4
0.05
0.05
30
0.00
0.00
(d) Variance
(e) APS
(f) ANS
Figure 13: Scoring distribution at other tokens (except the lead tokens), Mixtral
18

---

Published as a conference paper at ICLR 2026
without TO
with
without TO
with TO
without TO
with TO
0.4
0.15
-0.3
(35 layers)
OLMoE
0.3
0.10
0.2
0.1
0.05
-0.1
0.0
0.00
0.0
12
12
0.20
0.6
(48 layers)
0.15
Owen
0.2
-0.2
24
16
32
16
32
40
0.6
0.3
DeepSeck
layers
0.4
0.2
-02
0.2
-0.1
0.0
6
12
18
24
12
18
24
12
10
24
0.4
Nixtral
(32 layers)
0.3
-0.
0.5
0.2
0.1
-0.1
0.0
0.0
0.0
16
24
16
24
Recelving McE Layer Varlance
Receiving MoE Layer
Receiving MoE Layer ANS
Figure 14: Scoring distribution of tokens.
19

---

Published as a conference paper at ICLR 2026
E RESULTS OF T-SNE ON SCORES ASSIGNED BY TOKENS (QWEN)
Content words
Noun
Adjective
Verb
Adverb
Number
Function words
Conjunction
Determiner
Adposition
Punctuation
Pronoun
Particle
Other words
Foreign word, typo, abbr.
Figure 15: t-SNE of scores assigned by token embeddings in Qwen, colored by POS.
Position token
0
10
15
20
Figure 16: t-SNE of scores assigned by token embeddings in Qwen, colored by position.
20

---

Published as a conference paper at ICLR 2026
F
RESULTS ON IOI TASK
Head
20
20
15
20
25
10
23
X10
Method
Layer
10
14.4
222
455
(a) Path Patching
(b) Our Mathod 10
(4) Our Mathod 51
(d) Our Mathod 52
Attention Map
Layer
18-
-22
10
20
10
(a) Attention Map 10
(f) Attention Map 51
(g) Attention Map 52
Figure 17: Path patching: (a) I0 to logits (logit difference). Variance scores assigned by: (b)
(Head, Query=END, Key=S2), (c) (Head, END. S1), (d) (Head, END, 10) to the experts in the last block. Attention map: (e) (Query=END, Key=S2), (f) (END, S1), (g) (END,IO)
We follow Wang et al. (2023) to use "IO" to denote the indirect object, "S1" and "S2" denote the first and second occurrences of the subject, "END" denote the last token of the prompt. Additional details are provided in Appendix [
We apply path patching for the pathway h - logits at END token for each head / and compute the variation logit difference (ogit(1O) (S), which shown in Figure 17p We observe that the Name Mover Heads (e.g., A12H12, A A13H8, A A13H10, A A13H1I) and Negative Name Mover Heads (A13HI, A13H2, A A13H5) have pronounced influence on the variation of logit difference, compared with other heads in the same layer. Since these function heads directly focus on the name tokens, speculate that they have a relatively larger influence on the routing decisions through some specific terms, i.e., (W A oe (Equation 9p. where query t, END, key tp E {IO, S1,
Given that the identified function heads have a high influence on the logits, we observe the variance of scores assigned to the experts MI5 (the last MoE layer) by tuples (head=AxHy, query=t,, key=t,). example. We find that the identified function heads tend to more influence the routing decisions, compared with the idle heads. The Name Mover Heads exhibit apparent "activation"
ie., terms LN, (Wo: A, "z""))
[query=ENID, key=1O] and very weak "activation" [END, S1] (Figures 17p and c), indicating the scores assigned by tuples (A12H12, END, IO), (A13H8, END, IO), (A13HI0, END, IO), and (A13H11, END, IO) have comparatively high influence on the routing decisions in M15. We also identify a Name Mover Heads (A1 A1H11 that have weaker influence on both IOI circuits (Figure 17p) and routing decisions (Figure 17p), which could be Backup Name Mover Head. In contrast, the Negative Name Mover Heads influence more in the variance score maps of [END, S2] (Figure I7d). A12H3 is noticeable for its influence on routing decisions (Figure [7p), which is likely to be S-inhibition head as it assigns higher attention weight on IO instead of ST and (Figures b~g).
Finally, we compare the score variance maps (Figure 17p-d)) with the attention maps (Figure 17 p7)- We note that they resemble each other, particularly for the function heads. Although attention maps have some small non-zero activations in the early and intermediate layers, they usually do not have strong contribution on the routing decisions in the last layers, which may due to the natural influence decay in deep layers, the value vector cancels the effect. In conclusion, the function heads tend to have a comparatively noticeable influence the variance of assignment scores, which correlates with the corresponding attention map.
21

---

Published as a conference paper at ICLR 2026
G
RESULTS ON SCORING OF ATTENTION HEADS IN DEEPSEEK
Attention Head
Attention Head MoE
0.100
10-2
-0.10
Layer
10
0.075
0.050
10-3
0.05
0.025
20
0.000
0.00
(a) Variance
(b) APS
(c) ANS
Figure 18: DeepSeck: (a) Variance of scores assigned by heads to experts in the same block. (b) Average positive scores (APS) the heads. (c) Average negative scores (ANS) of the heads. Please note that Layer of DeepSeek an FFN layer, not MoE layer.
H SUPPLEMENTARY RESULTS ON SCORING OF EXPERTS. RESULTS
Table The rank of variance of scores assigned by Super Experts to the experts in the next MoE layers.
Model
Experts
DeepSeek
M2E54 (#55), M3H38 (#7)
OLMoE
Not Available
Qwen
M1H68 (# M2E92 (#3),
M3E82 (#2)
Mixtral
M1E3 (#8)
M1E3
0.5
0.0
10
20
23
30
Receiving Layer
Figure 19: Mixtral, MIE3 found to be a "Super Expert", but it has a high variance at the top layers, especially the last layer (circled in red).
22

---

Published as a conference paper at ICLR 2026
I
ABLATION STUDY OF M1E9, M4E14, AND MIE18 IN OLMOE
Sending Layer
5
15
15
5
15
0.6
Receiving Layer
MoE-MoE
0.4
10
0.2
15
0.0
(a)
M1E9
(b) M4E14
(c) M1E18
5
15
0
5
15
0
5
15
0.6
Receiving Layer
MoE-MoE
5
0.4
10
0.2
0.0
(d) M1E9
(e) M4E14
(f) M1E18
Figure 20: Variance of scores assigned by MoE Layer to MoE Layer y. After setting the scores assigned by the specific expert to zero to perturb the routing decisions, we pad the output of MoE layers with original output: (a) MIE9, (b) M4E14, (c) MIE18. We do not perturb the routing decisions but pad the output of the specific expert as zero: (d) M1E9, (e) M4E14, (f) M1E18.
In this section, conduct zero-ablation experiment Experts M1E9, M4E14, MIE18 in OLMoE as a case study to observe the influence on the routing decisions of M1E9 and M4E14, which are identified large-varte variance experts in Figure MIE18 is the expert with second large variance in the score assignment from MI to M10, which is used for comparison.
We first set the scores assigned by the specific expert zeros to observe how it perturbs the variance, and then pad the MoE layer output with the original output (Figure (00p~e). In Figure 20 we observe that the stripe at sending layer MI vanishes after Similarly, in Figure 20b, the stripe at sending layer M4 vanishes after we set the scores assigned by M4E14 as zeros. However, after we the score zeros assigned by MIEI8. the two stripes persist (Figure 20 >. The results indicate that both M1E9 and M4E14 4 have a significant influence on the contribution of M1 and M4 the routing decisions the layers in the stripe, respectively.
the scores assigned by M1 zeros.
Now we do not perturb the routing decisions but just pad the output of the specific expert zero vector (Figure 2f. In Figure 20d. both stripes vanish after we apply the operation on M1E9, which indicate that the contribution of M4 is significantly influenced by M1E9, not just the contribution of M1. If we apply the operation on M4E14, we can observe that there only a faint stripe corresponding to MI (Figure 20 We suggest that the output of M4E14 14 support the contribution of MIE9, which may also attribute to the attention layer 5 or the layer normalization operation after it. In Figure 200; again observe that the two stripes persist after applying the operation MIE18, which indicates MIE18 does not have an apparent influence on the contribution MI and M4, compared with M1E9 and M4E14.
We summarize that: MIE9 first activates M4E14, and then they need to coexist to exert a significant influence the following layers. This also explain why the stripes start from MoE Layer 5.
23

---

Published as a conference paper at ICLR 2026
J RESULTS ON MATH REASONING TASK
In this section, we provide the experimental results on a subset of I (Hugging 2025), math reasoning task dataset, investigate the generalizability of the scoring distribution pattern found in the main text. The results (Figures 21 22 and 23 reveal that the scoring distribution
pattern consistent across different English corpora, shown by comparing with the results in Figures] and with minor change in the magnitude.
Sending Layer
5
10
15
10
15
0
5
10
15
Attention MoE Receiving Layer
0.6
0.8
0.10
5
- 0.6
0.4
- 0.4
0.05
0.2
-02
15
(a) Varlance
(b) APS
(c) ANS
-0.20
MoE-MMoE Receiving Layer
0.15
00.5
5-
0.10
0.10
0.2
0.05
-0.05
15
(d) Variance
(e) APS
(f) ANS
Figure 21: (Math reasoning task) Scores assigned by attention Layer x to MoE Layer y: (a) Variance,
(b) Average positive scores (APS), (c) Average negative scores (ANS); Scores assigned by MoE
Layer MoE Layer y: (d) Variance, (e) APS, ANS.
Attention Head
Bunding/Receiving Layer
Attention Head-MoE
0
-0.3
0.20
10
5
10
0.15
0.2
10
0.10
10
-0.
10
0.05
15
0.00
(a) Varlance
(b) APS
(c) ANS
Figure 22: (Math reasoning task) Scores assigned by attention Layer to MoE Layer y: (a) Variance of scores assigned by heads to experts in the same block. (b) Average positive scores (APS) of the heads. (c) Average negative scores (ANS) of the heads.
24

---

Published as a conference paper at ICLR 2026
M1E9
5
M2E30
Experts
4-
Variance
0
10
15
Receiving Layer
Figure 23: (Math reasoning task) The variance of scores assigned by experts to the following layers.
K
L2-NORM HEATMAP
Sending Layer
16
12
Receiving
10
4
2
15
(a) Attention Layer Output
(b) McE Layer Output
Figure 24: The 1.2-norm of projected sending layer output in different receiving layers. (a) attention layer output, (b) MoE layer output.
To study the difference of 1.2-norm of a component and the variance of scores assigned by the component, illustrate the L.-norm of components in the general experiment C4 dataset in Figure In Figure 24p. the norm of projected sending attention layer output gradually decreases in the deeper receiving MoE layers. In Figure 24p, again observe the existence of stripes at receiving layers M1 and M4. Both Figure 24h and b reveals that the large norm tend to occur in the deep sending layers. By Comparing Figure 24 and 33 and d, conclude that projected component with large norm does not necessarily to have a high scoring variance.
L SENSITIVITY TO HYPERPARAMETER K (OF THE TOP-K)
In this section, we show that the cross-layer contribution is not sensitive to the hyperparemter K in OLMoE (original K=8, 64 experts in total) (Figure We change K and renun on C4 dataset with the same setting adopted in the main text. The pattern of score variance persists when K is changed, although magnitude varies slightly.
25

---

Published as a conference paper at ICLR 2026
Sending Layer
5
10
15
0
10
15
0
5
10
15
0.8
Attention-Mon Receiving Layer
0.6
5-
0.4
10
0.2
15
0.0
(a) K=4
(b) K=16
(c) 6=
0
0.8
Receiving Layer
MoE-MoE
0.6
5
0.4
10
0.2
15:
0.0
(d) K=4
(e) K=16
(f) K=64
Figure 25: Variance of scores assigned by attention Layer to MoE Layer y: (a) K=4, (b) K=16, (c)
K=64; Variance of scores assigned by MoE Layer x to MoE Layer y: (d) K=4, (e) K=16, (f) K=64.
M
GAP BETWEEN EXPERTS OF RANK K AND RANK (K+1)
Sending Layer
0
5
10
15
5
10
15
0
5
10
15
Receiving Layer
0.5
0.5
MoE-MoE
5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
(a) K+1
(b) K+2
(c) Last
Figure 26: (Swapping rank K expert for an unselected expert) Variance of scores assigned by MoE
layer x to MoE layer y: (a) K+1, (b) K+2, (c) Last ranked expert.
In this section, we show that the score gap between experts rank K and is not evidently important for the routing decisions. We again conduct the experiment C4 dataset but swap the rank the expert of rank K for the expert of rank K+1, K+2 or the last ranked expert (rank 64) during the routing decision (Figure 26 in OLMoF, and observe the impact the variance. We can find that the gap between rank K and K+1 (or other unselected experts) has fairly weak impact the routing decisions in the following MoE layers. We suggest that the redundancy of selected experts lead to this phenomenon.
N RESULT OF SPEARMAN'S RANK CORRELATION
In this section, give the Spearman coefficient of the cases have studied in Section 7.2 (Figure 6). We measure the monotonic association between the variance of a component's s score contributions and AARV of the receiving layer (defined in Section 7.2) using Spearman's rank correlation. We
26

---

Published as a conference paper at ICLR 2026
observe Spearman coefficients of 0.595 (MI-MM.) 0.708 (MI- and 0.580 (M1 (M1-M155) These results indicate moderate to strong monotonic correlation, with the correlation peaking at 0.708 for M1-+M10).
207
Arerage absolus rane vantatinnnn of experts
Recelving Layer
M10. pm0.708
M15, pm1 500
20"
10
10
10:7
10"
10a
Varance of scores assigned by MI to the recelving layer
Figure 27: The variance scores assigned by experts MI and the averange absolute rank variation of top-K experts. The Spearsman coefficient p is shown in the legend, all the p-values < 0.001.
O MODEL. DESIGN IMPLICATIONS
a) Non-uniform MoE placement. Our variance maps show that a few MoE layers consistently dominate the routing landscape, while others have comparatively modest contribution. This suggests exploring heterogeneous depthwise allocation, where MoE capacity (number of layers, width, or top-K) concentrated in high-influence regions, and reduced low-influence regions.
removed in consistently
b) Tiered expert capacity. At the expert level, we observe that only small subset of experts main- tains strong contributions across many layers. This motivates architectures with tiered experts-for example, mix of "high-capacity" experts in high-influence layers and cheaper experts elsewhere, rather than a uniform design.
Cross-layer- moos-layer-aware -ware routing heuristies. Since routing decisions are not purely local, one could use our cross-layer contribution maps as prior for inference-ting time scheduling: c.g., prefetching experts that are systematically promoted by influential upstream layers, or biasing training-time regularizers to stabilize or diversify those long-range pathways.
P A BRIEF MECHANISTIC EXPLANATION OF THE OBSERVED PHENOMENA
a) Why MoE outputs persist more than attention outputs? MoE FFN outputs have larger post- normalization magnitudes and expert-specific projections. Since router scores are dot products with normalized inputs, this gives MoE components much larger capacity introduce variance across experts. Attention outputs are redistributive and tend to decay across depth, leading to more local effects.
b) Why "entanglement stripes" appear? Once a high-impact expert fires in layer L, its output is repeatedly re- re-projected through subsequent blocks. When this direction aligns with router weight vectors in deeper layers, the influence accumulates rather than dissipates. In addition, the results indicate that router weights across layers not random but partially aligned, creating shared routing subspaces and the observed stripes.
Why cross-layer coupling is strong in MoEs? Top-K sparsity stabilizes early expert selections, and expert-specific nonlinear transformations generate distinctive activation signatures that persist across depth. These signatures repeatedly bias downstream routing. contrast, attention layers mainly remix existing signals and do not introduce expert-specific basis directions, which explains their weaker long-range persistence.
27


Reviews:
[{"review_id": "KMOP1HUhAw", "reviewer": "Reviewer_Dj7p", "summary": "The paper analyzes the routing mechanism in Mixture-of-Experts (MoE) language models from a cross-layer mechanistic interpretability perspective.\n\nThe authors propose to understand routing by recursively decomposing the expert assignment score. The score for a given expert is a dot product of its weight vector and the MoE layer's input vector (Eq. 6). The paper's key insight is to decompose this input vector into a sum of its constituent parts: the original token embedding plus the outputs from all preceding attention and MoE layers (Eq. 8).\n\nsome of the key findings include:\n\n- MoE layer outputs generally have a stronger and more persistent influence on downstream routing decisions than attention layer outputs.\n- The paper finds that routing is not a purely local decision. Certain MoE layers (and specific experts within them, e.g., M1E9 and M4E14 in OLMOE) exhibit high influence on routing decisions many layers deeper, creating \"stripes\" of influence (visible in Fig 3d).", "questions": "Could you elaborate on the `LN_bar`? from Eq 8, it seems that The methodology's decomposition (Eq. 8) relies on attributing scores to normalized components (`LN_bar`). However, Layer Normalization (LN) is a non-linear function applied to the sum of all components. $LN(x+y) ≠ LN(x) + LN(y)$. The paper defines $LN_bar(z) = \\frac{(z * γ)}{RMS(total input)}$, which means the \"contribution\" of one component is scaled by a denominator that depends on all other components. So is such decomposition really make sense?\n\nOn Model Design Implications: Your findings suggest some layers/experts are far more important to routing. Based on this, would you recommend alternative MoE architectures? For instance, any new architecture designs or distribution of MoE layers in the network?\n\nOn Training Dynamics: This analysis is a snapshot of pre-trained models. Have you investigated how these influence maps evolve during training? Does the long-range \"entanglement\" (the stripes) emerge early, or is it a late-stage phenomenon as the model refines its pathways?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper's primary strength is its shift in perspective. Moving from \"what do experts specialize in?\" to \"what components influence the choice of expert?\" is a significant contribution. The method of decomposing the router's input vector is elegant and interpretable.\n\n- Clear and Justified Metrics: The use of variance as a proxy for \"influence\" (Proposition 1) and APS/ANS for \"tendency\" (Proposition 2) is well-defined and effective. This provides a clear, quantitative framework to move beyond simple co-activation analysis.\n\n- The discovery of \"MoE entanglement\" (the long-range \"stripes\" in Fig 3d and 5a) is a major finding. It compellingly argues that MoE routing is a complex, non-local phenomenon, which has significant implications for how these models are understood.", "weaknesses": "Correlation vs. Causation: The analysis is entirely correlational. While it shows that the output of expert M1E9 correlates with routing decisions in layer 10, it does not prove a causal link. The paper lacks interventional experiments (e.g., ablating a high-variance expert and measuring the actual change in downstream routing) to confirm that these high-variance components are truly driving the decisions.\n\nUnderdeveloped Link to \"Super Experts\": The paper attempts to connect its high-influence experts to the \"Super Experts\" identified by Su et al. (2025) but finds the link is weak (Section 7, Appendix H). This is an interesting anti-finding, but it feels incomplete. It raises the question: which metric matters more? This paper's routing \"influence\" (variance) or the \"Super Expert\" output magnitude? The paper opens this door but doesn't walk through it.\n\nSome of the notation are made deliberately complicate, which is hard to read at first, especially in section 3 and 4.1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wlcbvC9AXv", "reviewer": "Reviewer_tjsZ", "summary": "This paper proposes a lightweight recursive decomposition method for quantifying the cross-layer contributions of different model components to MoE routing decisions. Through experiments on four production-level MoE models, the authors find:\n\n  -Dominant Role of MoE Layer Outputs: The outputs of MoE layers exert the strongest and most persistent influence on the routing decisions of subsequent layers.\n\n  -Cross-Layer Entanglement: Routing decisions are influenced by long-range cross-layer interactions, challenging the assumption that routing is a local process.\n\n  -Component Effects: Attention layer outputs exhibit local facilitative effects in the bottom and top layers, while MoE layer outputs demonstrate global suppression in the middle layers.\n\n  -Impact of Key Experts and Attention Heads: A small number of super experts have a sustained impact on routing.\nThese findings highlight the importance of cross-layer interpretability and provide new opportunities for the efficient design of MoEs.", "questions": "Same as weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-Lightweight Design: The recursive decomposition method requires no architectural modifications or retraining, achieving decomposition solely through input manipulation with low computational overhead. This offers a new paradigm for MoE interpretability research.\n\n  -Model Diversity: Covers four mainstream MoE architectures, validating the universality of the conclusions. Supplementary evidence in the appendix, including t-SNE clustering and heatmaps, provides intuitive visualization of the spatial and hierarchical distribution of component contributions.\n\n  -Theoretical Rigor: Propositions 1 and 2 prove the rationality of using variance as an influence metric, establishing a mathematical foundation for the analysis.\n\n  -Discovery of Super Experts and Long-Range Suppression: Identifies super experts and long-range suppression effects, offering new ideas for optimizing MoE systems.", "weaknesses": "-Limited Model Scope: Experiments are confined to open-source models, excluding systems like GPT-4 MoE or other closed-source implementations. The generalizability of conclusions to larger-scale or heterogeneous architectures remains questionable.\n\n  -Task Simplicity: Only the C4 and IOI tasks are used, lacking validation on complex downstream tasks such as reasoning or multimodal tasks. This makes it difficult to prove the task invariance of routing patterns.\n\n  -Missing Attribution Method Comparison: No comparison is made with gradient-based attribution methods or causal intervention methods. The advantages of the variance metric are not fully substantiated.\n\n  -Sensitivity to Hyperparameters: The analysis does not examine the impact of routing hyperparameters like the Top-k value or number of layers on the conclusions. For instance, Mixtral's Top-k value of 2 might amplify locality effects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9IkSziZEJO", "reviewer": "Reviewer_T74p", "summary": "This paper investigates the routing mechanism in MoE-LLMs from a cross-layer mechanistic interpretability perspective. The authors propose a lightweight, recursive decomposition methodology to attribute the routing score assigned to each expert back to the contributions of preceding model components (token embeddings, attention layers/heads, and MoE layers/experts). Applying this methodology to four different large-scale MoE models (OLMOE, DeepSeek-V2-Lite, Qwen3, Mixtral), the study identifies several common patterns: (1) MoE layer outputs generally contribute more significantly and persistently to downstream routing decisions compared to attention layer outputs; (2) Evidence of \"MoE entanglement,\" where the activation of experts in earlier layers correlates with the activation of experts in later layers; (3) Certain components exhibit long-range influence, affecting routing decisions many layers downstream; (4) Models exhibit distinct patterns regarding short-range versus long-range promoting or inhibiting effects from components on subsequent routing.", "questions": "1. Could you elaborate on the validity of the $\\overline{LN}_{i}^{l}(z)$ approximation used in Equation 8, particularly for RMSNorm? \n\n2. Regarding Proposition 1, how strongly does the variance of a component's score contributions correlate with its actual impact on changing the set of top-k selected experts?\n\n3. You suggest using cross-layer contribution insights for expert prefetching. Could you sketch a potential algorithm or strategy for how the variance or entanglement patterns identified could be used to predict which experts are likely to be needed in future layers, thereby informing a prefetching policy?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper addresses a timely and important gap in understanding MoE models by focusing on the cross-layer dynamics influencing routing decisions.\n2. The empirical results reveal non-trivial cross-layer interactions, such as the stronger, persistent influence of MoE outputs compared to attention outputs, the \"entanglement\" effect, and long-range influences. These findings challenge simplistic, localized views of routing and provide valuable insights into the complex dynamics within these models.", "weaknesses": "1. Equation 8 uses an approximation ($\\overline{LN}_{i}^{l}(z)$) where the RMS term is calculated over the *total* layer input, but the contribution of a *single* component $z$ is scaled. Given the non-linear nature of RMSNorm (depending on the norm of the entire input vector), the accuracy and potential biases introduced by this approximation are unclear and not quantified. This could affect the relative attribution scores between components.\n\n2. While the paper successfully identifies several interesting cross-layer patterns (e.g., MoE persistence, entanglement stripes), it provides limited mechanistic explanations for *why* these patterns emerge. For instance, what structural or functional properties of MoE layers enable their outputs to have a more persistent influence compared to attention layers? What specific information pathways lead to the observed \"entanglement\" between expert activations across layers?\n\n3. The paper could be clearer about how metrics like variance, APS, and ANS are aggregated (e.g., averaged across which tokens, which samples?) when presenting layer-level heatmaps (e.g., Figure 3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XPQVsPg1Gh", "reviewer": "Reviewer_NVZU", "summary": "This paper investigates the routing mechanisms in Mixture-of-Experts (MoE) LLMs from a cross-layer perspective. The core idea is to recursively decompose the expert assignment score, i.e., the result of a dot product between an expert's routing weight vector and the MoE layer's input, into a linear sum of contributions from all preceding model components. These components include the initial token embedding, the outputs of all previous attention layers, and the outputs of all previous MoE layers.\n\nThe authors define a metric for \"influence\" based on the variance of the scores a single component contributes to all experts in a receiving MoE layer; high variance implies a strong influence on the routing decision. Using this framework, the paper analyzes OLMOE, DeepSeek-V2-Lite, Qwen3-30B-A3B, and Mixtral-8x7B.  Here are some of their key findings:\n\nThe variance of the initial token embedding scores decays rapidly with layer depth.\n\nThe authors identify a \"cross-layer entanglement\" phenomenon, where specific MoE layers exhibit a strong, persistent, long-range influence on routing decisions in many subsequent layers.\n\nThe methodology could be extended to finer granularities, showing that specific attention heads or a small subset of experts are disproportionately influential on routing.", "questions": "1. Can the authors provide a layer-level heatmap (identical in structure to Figure 3) but plotting the average L2-norm of the component outputs instead of the score variance? If this norm-heatmap is nearly identical to the variance-heatmap, the paper's claim to novelty is significantly weakened.\n\n2. Could the authors comment on the limitations of variance as a metric, specifically its insensitivity to the score \"gaps\" between the Top-K and non-Top-K experts and selected expert’s contribution? Do we need additional metrics for this?\n\n3. Assuming the variance metric is not just a proxy for norm (as per Question 1), what is the authors' primary hypothesis for the observed divergence between high-norm and high-influence components? \n\n4. Have the authors investigated whether these cross-layer influence patterns remain consistent during autoregressive generation on specific tasks (e.g., question-answering or MATH reasoning)? Is it possible that the observed routing patterns are an artifact of the C4 dataset and the forward-pass-only analysis?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Intuitive and straightforward method: The paper presents a simple and computationally light method for MoE interpretability. Shifting the focus from expert specialization (what experts do) to the composition of the router's input (why a router chooses) is a good contribution to the field. \n\n- Interesting Findings: The findings of this paper clearly demonstrate that MoE routing is not just shaped by the precedent attention layer but also previous transformer blocks.", "weaknesses": "1. **My greatest concern** for this paper is whether variance is a good metric for an expert’s contribution. The paper's metric (variance) measures \"flatness\" of gating scores, but this is an imperfect proxy for influence on the Top-K routing decision (expert selection and weight). For example, a component's contribution that changes scores (assuming selecting 2 experts out of 4 experts) from [10, 9, 1, 0] to [12, 7, 2, -1] with contribution [2, -2, 1, -1] (Top-K={E1, E2}) would have a high variance. A different component that changes scores from [5, 4.9, 4.8, 4.7] to [6, 3.9, 4.8, 4.7] (Top-K={E1, E3, a changed routing decision) with contribution[1, -1, 0, 0]. The latter alter the routing results but have lower variance.  The metric is not sensitive to the \"gaps\" between the K-th and (K+1)-th expert, which is what actually determines the decision. Mean while, the variance also does not reflect the final contribution/weight of each expert. For example, a component contribute [100, 100, 100, 100] to the gating score and another component contribute [1, 1, 0, 0]. The first component’s contribution is zero accroding to variance. but it actually make the final selective expert’s contribution more flat, which is quite important and cannot be ignored when understanding routing mechanism in MoE.  \n\n2. **Does this paper re-find the massive activation phenomenon?** The observations discovered by authors closely relates to massive activation and attention sink. I hope authors could discuss the relation with these two phenomenons.  Furthermore, the author should to provide the most direct piece of evidence for illustrating the contribution of each components: a heatmap of component L2-norms, directly comparable to Figure 3. The findings in Figure 3 (e.g., high influence from M1, M4) could very well be a simple re-discovery of massive activation of that layer. Without this comparison, the paper's core claim of novelty is undefended. \n\n3. **Fragmented Narrative**: I feel that the paper's presentation is highly fragmented. This is a major weakness because it buries the paper's core discovery. Making readers easy to conclude that the findings are just a restatement of known concepts (i.e., massive activation). By the way, the title in the PDF is different from the submission.\n\n4. **Speculative Practical Applications**: The abstract and conclusion claim the findings \"highlight the opportunities... for effective model design and model serving.\" However, these claims are speculative and undeveloped.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)