Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysis
Authors: Junzhuo Li, Bo Wang, Xiuze Zhou, Peijie Jiang, Jia Liu, Xuming Hu
Accepted at: acl25

Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysis

Junzhuo Li12, Bo Wang1, Xiuze Zhou12, Peijie Jiang3, Jia Liu3 and Xuming Hu12*
1The Hong Kong University of Science and Technology (Guangzhou)
2 The Hong Kong University of Science and Technology
3Ant Group
{jz.li, bo.wang, xz.zhou}@connect.hkust-gz.edu.cn
xuminghu@hkust-gz.edu.cn

Abstract

The interpretability of Mixture-of-Experts (MoE) models, especially those with heterogeneous designs, remains underexplored. Existing attribution methods for dense models fail to capture dynamic routing-expert interactions in sparse MoE architectures. To address this issue, we propose a cross-level attribution algorithm to analyze sparse MoE architectures (Qwen 1.5-MoE, OLMoE, Mixtral-8x7B) against dense models (Qwen 1.5-7B, Llama-7B, Mistral-7B). Results show MoE models achieve 31% higher per-layer efficiency via a “mid-activation, late-amplification” pattern: early layers screen experts, while late layers refine knowledge collaboratively. Ablation studies reveal a “basic-refinement” framework—shared experts handle general tasks (entity recognition), while routed experts specialize in domain-specific processing (geographic attributes). Semantic-driven routing is evidenced by strong correlations between attention heads and experts (\( r = 0.68 \)), enabling task-aware coordination. Notably, architectural depth dictates robustness: deep Qwen 1.5-MoE mitigates expert failures (e.g., 43% MRR drop in geographic tasks when blocking top-10 experts) through shared expert redundancy, whereas shallow OLMoE suffers severe degradation (76% drop). Task sensitivity further guides design: core-sensitive tasks (geography) require concentrated expertise, while distributed-tolerant tasks (object attributes) leverage broader participation. These insights advance MoE interpretability, offering principles to balance efficiency, specialization, and robustness.

significantly enhanced task performance by scaling up parameters, yet at the cost of prohibitive computational demands. To balance efficiency and performance, Mixture-of-Experts (MoE) architectures (Fedus et al., 2021; Lepikhin et al., 2021; Shen et al., 2023; Jiang et al., 2024; Dai et al., 2024; Sun et al., 2024) activate subsets of experts per input, reducing overhead while maintaining capability. While existing research has prioritized optimizing routing mechanisms and expert module efficiency (e.g., top-\( k \) gating (Fedus et al., 2021; Lepikhin et al., 2021)), a critical question remains unanswered: How do experts collaboratively process and refine knowledge within MoE models?

This gap in interpretability is particularly pronounced in models with heterogeneous designs, such as shared expert modules (Dai et al., 2024). Despite their widespread adoption (DeepSeek-AI et al., 2024; Team, 2024), the roles of shared experts—whether they serve as universal feature extractors or redundant backups—lack empirical validation. The opacity of expert collaboration not only hampers credibility assessment (e.g., verifying if experts specialize in specific knowledge domains) but also forces model designers to rely on trial-and-error strategies, limiting systematic improvements. For instance, fundamental issues like the distribution of shared experts across layers and their impact on knowledge consistency remain unexplored.

Two key challenges compound this problem: (1) Existing attribution methods (Geva et al., 2021; Dai et al., 2022a; Geva et al., 2022; Wu et al., 2023) are designed for dense models and fail to capture dynamic interactions between routed experts and shared modules. (2) Comparative studies on heterogeneous MoE architectures are scarce, leaving functional assumptions (e.g., “shared experts handle general tasks”) unverified.

In this paper, we address these gaps by proposing a cross-level attribution algorithm for MoE models, extending dense model methods to ana-

1 Introduction

In recent years, transformer-based large language models (LLMs) (Vaswani, 2017; Brown et al., 2020; Touvron et al., 2023b; Bai et al., 2023) have
Figure 1: Schematic of the MoE expert layer: Shared experts perform foundational tasks (e.g., the Capital Identifier recognizes “capital,” the Country Identifier recognizes “Canada”), while routed experts execute refinement steps—generating candidate capitals, matching them to “Canada,” and selecting the final answer (“Ottawa”).

lyze both macro architectures (Mixtral-8x7B (Jiang et al., 2024), Qwen 1.5-MoE (Team, 2024), OL-MoE (Muenninghoff et al., 2024)) and micro behaviors. Our experiments uncover four pivotal insights. First, MoE models significantly outperform dense counterparts in per-layer efficiency—Qwen 1.5-MoE achieves a layer efficiency of 0.294 with 24 layers, surpassing Qwen 1.5-7B (0.213 with 32 layers), a 37% improvement attributed to concentrated knowledge processing in early layers. Second, we identify a hierarchical "basic-refinement" collaboration: shared experts act as generalists for foundational tasks like entity localization, while routed experts specialize in refining domain-specific attributes (e.g., linking “Canada” to “Ottawa”). Third, semantic-driven routing is validated through strong temporal correlations between attention heads and expert selection (\( r = 0.68, p < 0.0014 \)), indicating that attention mechanisms guide experts toward task-relevant features. Finally, robustness varies starkly by architecture: blocking top-10 experts in deep Qwen 1.5-MoE causes a 43% MRR drop in geographic tasks, whereas shallow OLMoE suffers a 76% decline, underscoring the critical role of depth and shared experts in redundancy design.

Contributions Our main contributions are as follows:

• A cross-level attribution algorithm for MoE models, bridging the gap in interpreting sparse architectures by dynamically weighting expert and attention contributions.

• Empirical validation of MoE efficiency and collaboration patterns, including layer efficiency gains, a “mid-activation, late-amplification” processing paradigm, and semantic-driven expert routing.

• Design implications for robust MoE architectures, demonstrating that deep models with shared experts balance specialization and redundancy, particularly critical for complex tasks like geographic reasoning.

2 Methodology

In this section, we first describe the MoE model architecture, followed by the knowledge attribution method. More details for model settings and experimental settings (i.e., dataset, evaluation metrics) can be found in Appendix A.

2.1 MoE Architecture

In the decoder-only MoE architecture, each Transformer decoder block replaces the conventional Feed-Forward Network (FFN) with an MoE layer, enhancing model capacity via dynamic routing. Given an input sequence \( X = (x_1, x_2, \ldots, x_T) \), the tokens are embedded to produce the initial hidden state \( h^0 = \mathrm{Embed}(X) \). We omit bias and layer normalization (Ba et al., 2016) following Geva et al. (2022).

Each decoder layer consists of a self-attention mechanism and an MoE layer. At layer \( l \), the attention output for token \( x_i \) is computed as:

\[
A_i^l = \sum_{j=1}^H \mathrm{ATTN}_j^l(h_1^{l-1}, h_2^{l-1}, \ldots, h_T^{l-1}),
\]

where \( H \) is the number of attention heads. The intermediate representation \( u_i^l = A_i^l + h_i^{l-1} \) is routed to \( N \) experts via a gating network, which computes the routing probabilities:

\[
g_i^l = \mathrm{softmax}(\mathbf{W}_g^l u_i^l + b_g^l),
\]

where \( \mathbf{W}_g^l \) and \( b_g^l \) are learnable parameters. Each expert \( \mathcal{E}_j^l \) is a feed-forward network:

\[
\mathcal{E}_j^l(u_i^l) = W_{j,2}^l \phi(W_{j,1}^l u_i^l + b_j^l),
\]
<table>
  <tr>
    <th>Model</th>
    <th>HIT@10</th>
    <th>MRR</th>
    <th>Total FFN Gain</th>
    <th>Total Attn Gain</th>
    <th>Peak Gain Layer Position (Rel. Position%)</th>
    <th>Layer Efficiency (FFN Gain / Layers)</th>
  </tr>
  <tr>
    <td>Llama-7B</td>
    <td>0.90</td>
    <td>0.70</td>
    <td>5.16</td>
    <td>4.03</td>
    <td>24.82 (77.6%)</td>
    <td>0.161</td>
  </tr>
  <tr>
    <td>Qwen 1.5-7B</td>
    <td>0.79</td>
    <td>0.62</td>
    <td>6.49</td>
    <td>4.14</td>
    <td>27.08 (90.2%)</td>
    <td>0.203</td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>0.88</td>
    <td>0.71</td>
    <td>6.74</td>
    <td>2.96</td>
    <td>26.64 (83.2%)</td>
    <td>0.211</td>
  </tr>
  <tr>
    <td>Qwen 1.5-MoE</td>
    <td>0.85</td>
    <td>0.63</td>
    <td>7.36</td>
    <td>3.18</td>
    <td>20.36 (84.8%)</td>
    <td>0.307</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>0.83</td>
    <td>0.64</td>
    <td>4.98</td>
    <td>4.40</td>
    <td>13.53 (84.6%)</td>
    <td>0.311</td>
  </tr>
  <tr>
    <td>Mixtral-8x7B</td>
    <td>0.90</td>
    <td>0.73</td>
    <td>6.79</td>
    <td>3.03</td>
    <td>26.66(83.3%)</td>
    <td>0.212</td>
  </tr>
</table>

Table 1: Comparison of model architectures on performance metrics, component contributions, and layer efficiency across dense and MoE models. Rel.:Relative.

where \( \phi(\cdot) \) is a nonlinear activation function (Nair and Hinton, 2010; Maas et al., 2013; Hendrycks and Gimpel, 2016). The final output for token \( x_i \) at layer \( l \) is computed by a weighted sum of the top \( k \) experts’ outputs:

\[
F_i^l = \sum_{j=1}^k g_{i,j}^l \cdot \mathcal{E}_j^l(u_i^l).
\] (4)

Some MoEs include shared experts that are always selected (Dai et al., 2024), which results in:

\[
F_i^l = \sum_{j=1}^k g_{i,j}^l \cdot \mathcal{E}_j^l(u_i^l) + g_{i,s}^l \cdot \mathcal{E}_s^l(u_i^l).
\] (5)

Finally, the output at position \( i \) of the \( l \)-th layer is:

\[
h_i^l = h_i^{l-1} + A_i^l + F_i^l.
\] (6)

The final hidden states \( h_i^L \) need to go through the unembedding process to map the high-dimensional representations back to the vocabulary space. Specifically, for each position \( i \), the final output token \( y_i \) is obtained by multiplying the hidden state \( h_i^L \) with the unembedding matrix \( W_{unembed} \) and passing it through a softmax function.

2.2 Neuron-Level Attribution

We adopt the neuron-level attribution methods from Yu and Ananiadou (2024) for the FFN and attention layers and extend them for MoE models. For attention neurons \( v_A^l \), a vector (neuron) in \( l \)-th Attention layer, the importance score is:

\[
\mathcal{I}(v_A^l) = \log p(y_i|v_A^l + h^{l-1}) - \log p(y_i|h^{l-1}),
\] (7)

For a token \( x_i \) The importance score for FFN neurons is defined as the log probability increase:

\[
\mathcal{I}(v_F^l) = \log p(y_i|v_F^l + u^l) - \log p(y_i|u^l),
\] (8)

where \( v_F^l \) is a vector (neuron) in \( l \)-th FFN layer.

In MoE models, we extend these methods to account for dynamic routing and expert contributions. The importance score for MoE expert neurons is computed as:

\[
\mathcal{I}(v_{E_j}^l) = \log p(y_i|g_{i,j}^l v_{E_j}^l + u^l) - \log p(y_i|u^l),
\] (9)

where \( g_{i,j}^l \) is the gating probability for expert \( E_j \), and \( v_{E_j}^l \) is the neuron from expert \( E_j \) of \( l \)-th layer.

Through these modifications, we are able to accurately attribute importance not only to the conventional FFN and attention neurons but also to the individual experts and their specific contributions within the MoE framework. This extension allows us to capture the complex interactions between experts and tokens, providing deeper insights into the behavior of MoE models and how they utilize expert knowledge.

The importance score represents the influence of a neuron on the final prediction, reflecting the extent of its contribution during the model’s inference process. More specifically, it can be regarded as the gain introduced by the module to the overall model performance, revealing the role of different neurons or experts in the model’s output. This metric aids in understanding how the model utilizes the knowledge of various components and the significance of each module for specific tasks.

3 Comparative Analysis: Efficiency and Knowledge Processing in MoE vs. Dense Models

In this section, we compare three MoE architectures (Qwen 1.5-MoE, Mixtral, and OLMoE) with two dense models (Qwen 1.5-7B, Llama-7B, and
Mistral) across several key metrics. Our analysis highlights significant differences in parameter efficiency, knowledge localization, and FFN gain evolution—differences that underscore the unique interpretability advantages of MoE models.

3.1 Layer Efficiency and Knowledge Concentration in MoE Models

Table 1 summarizes the performance of the four models, revealing that MoE architectures achieve notably higher per-layer gains. For example, Qwen 1.5-MoE attains a layer efficiency of 0.307 using only 24 layers, whereas its dense counterpart, Qwen 1.5-7B, requires 32 layers to reach an efficiency of just 0.203. Despite a slightly lower overall FFN gain, OLMoE—by virtue of its shallower design—achieves the highest layer efficiency at 0.311, demonstrating a marked optimization in computational density.

The data further indicate that MoE models concentrate their knowledge processing at earlier stages. Specifically, Qwen 1.5-MoE and OLMoE reach their peak FFN gains at 84.8% and 84.6% of the network depth, respectively. This “early routing, late refinement” pattern enables these models to perform initial expert filtering in mid-layers and then focus on intensive knowledge refinement in later layers. In contrast, dense models such as Qwen 1.5-7B and Llama-7B display a more gradual accumulation of gains—peaking at 90.2% and 77.6% of the network depth—which, while beneficial for cross-layer integration, tends to blur the functional distinctions among layers and reduce interpretability.

Furthermore, a comparison of attention versus FFN gains underscores divergent processing strategies. Qwen 1.5-MoE, for instance, exhibits a pronounced reliance on specialized expert networks, with an FFN gain of 7.36 contrasted against an attention gain of only 3.18. Dense models, by comparison, distribute gains more evenly (with attention gains of 4.14 and 4.03 for Qwen 1.5-7B and Llama-7B, respectively), complicating the attribution of specific processing roles to individual layers. Notably, OLMoE achieves nearly identical attention and FFN gains (4.98 and 4.40, respectively), reflecting a balanced strategy that integrates both attention mechanisms and expert routing.

3.2 FFN Gain Evolution: Mid-Stage Activation and Late-Stage Amplification

Figure 2 illustrates the evolution of FFN gains across layers in Llama-7B, Qwen 1.5-MoE, Qwen 1.5-7B, OLMoE, Mistral, and Mixtral. The analysis reveals distinct processing patterns shaped by architectural designs:

Qwen 1.5-MoE exhibits a three-phase dynamic: (1) Early stage (Layers 1–13): Parallel initialization of experts extracts basic features (e.g., entity localization), contributing 6.1% of the total gain. (2) Mid-stage (Layers 14–19): Dynamic routing activates expert screening, where the top-4 gating mechanism selects task-relevant specialists. This phase contributes 43.5% of the total gain while minimizing redundant computations. (3) Late stage (Layers 20–24): Collaborative refinement between shared experts (e.g., broad knowledge integration) and routed experts (e.g., attribute association) amplifies gains, accounting for 50.4% of the total.

In contrast, Qwen 1.5-7B relies on linear accumulation across deeper layers (25–32), gradually integrating knowledge without explicit stage specialization. OLMoE, as a shallow architecture, rapidly peaks at layer 12 (mid-stage) but lacks sustained late-stage refinement due to limited depth, leading to a gradual performance decay.

Notably, Mistral diverges significantly from typical dense models: While other dense architectures exhibit diminished mid-to-late-stage gains, Mistral sustains high FFN activation through later layers, achieving gain levels comparable to MoE models. Mixtral shows strong similarity to Mistral, with both models peaking sharply at layer 28 before experiencing rapid performance decay. This parallel trajectory—absent in other architectures—suggests potential initialization-driven dynamics influencing their gain structures.

This structured processing enables MoE models
Figure 3: Cumulative FFN layers contribution curves.

to achieve higher per-layer efficiency. For example, Qwen 1.5-MoE’s late-stage layers (20–24) generate a gain of 1.45 per layer, compared to Qwen 1.5-7B’s 0.68 per layer in layers 25–32. The efficiency stems from task-specialized routing and cross-expert collaboration, contrasting with dense models’ uniform parameter utilization. The exceptional late-stage performance of Mistral and its resonance with Mixtral, however, indicate that initialization strategies (Lo et al., 2025) and architectural synergies may further modulate gain dynamics beyond conventional dense/MoE dichotomies.

3.3 Cross-Model Comparison of Stage Contributions

Having established the stage-wise processing patterns, we next compare their contributions across architectures. To quantify architectural differences, Figure 3 and Table 1 compare cumulative contributions across early, mid-, and late stages.

Mid-stage dominance in MoE  Qwen 1.5-MoE achieves 41.7% mid-stage contribution (absolute gain: 2.93), surpassing Qwen 1.5-7B’s 37.5% (2.56). OLMoE’s shallow design further amplifies mid-stage reliance (52.2% contribution), but limited depth restricts total gain (4.79 vs. Qwen 1.5-MoE’s 7.05).

Late-stage amplification in deep MoE  Qwen 1.5-MoE’s 24-layer architecture enables extended refinement, with 50.7% late-stage gain (3.57) versus Qwen 1.5-7B’s 47.6% (3.25). OLMoE’s 16-layer design results in rapid late-stage decay (12.5% contribution post-layer 12), highlighting the trade-off between depth and efficiency.

<table>
  <tr>
    <th>Model Config</th>
    <th>HIT@10</th>
    <th>MRR</th>
  </tr>
  <tr>
    <td>Only Top1</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Only Top2</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Only Shared</td>
    <td>0.03</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>Shared + 2nd</td>
    <td>0.04</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>Shared + Top1</td>
    <td>0.82</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>Shared + Top2</td>
    <td>0.83</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>Shared + Top4 (default)</td>
    <td>0.85</td>
    <td>0.63</td>
  </tr>
</table>

Table 2: Experimental results for different Qwen 1.5-MoE configurations, highlighting the effects of the shared expert and top-1 routing.

<table>
  <tr>
    <th>Model Config</th>
    <th>HIT@10</th>
    <th>MRR</th>
  </tr>
  <tr>
    <td>Top0</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>Top1</td>
    <td>0.86</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>Top2 (default)</td>
    <td>0.90</td>
    <td>0.73</td>
  </tr>
</table>

Table 3: Experimental results for different Mixtral configurations.

Architectural impact on processing strategies
Deep MoE models (e.g., Qwen 1.5-MoE) prioritize collaborative refinement in late layers, while shallow MoE architectures (e.g., OLMoE) focus on rapid mid-stage decisions. Dense models’ dispersed gains hinder task specialization—Qwen 1.5-7B requires 32 layers to achieve comparable total gains, indicating inefficiency in cross-layer integration.

These results underscore that MoE’s efficiency arises from stage-specialized processing, yet its effectiveness depends on balancing depth and expert collaboration. A detailed analysis of robustness implications (e.g., expert blocking effects) is provided in § 5.2.

4 Expert Collaboration:
Basic-Refinement via Semantic Routing

This section investigates the collaborative mechanisms between shared and routed experts in MoE models. Through systematic ablation experiments, we validate the necessity of expert collaboration for robust knowledge processing (§ 4.1) and reveal how semantic-driven routing coordinates attention heads and experts into a hierarchical framework (§ 4.2).
Figure 4: OLMoE’s performance on knowledge prediction tasks when using different routing strategies.

4.1 Expert Role Specialization

Our ablation studies demonstrate that effective collaboration between shared and routed experts is fundamental to MoE performance. In both Qwen 1.5-MoE (Table 2)—which combines a shared expert with routed specialists—and OLMoE (Figure 4) and Mixtral (Table 3)—which relies solely on routed experts—isolating individual experts severely degrades model capability. For example, when only a single expert is activated, OLMoE’s HIT@10 drops to a range of 0.15–0.19 with near-zero MRR, while Qwen 1.5-MoE achieves a mere HIT@10 of 0.03 when restricted to its shared experts and completely fails when limited to the top routed expert. These results underscore that complex knowledge tasks cannot be managed by isolated experts; instead, they require synergistic interactions between generalists and specialists.

A critical observation is the diminishing returns of increasing expert participation. In OLMoE, activating three experts elevates HIT@10 from 0.18 (single expert) to 0.81, but adding a fourth expert yields only a marginal gain (0.81 to 0.83). Similarly, Qwen 1.5-MoE sees limited improvement when expanding beyond its shared experts and top routed expert—HIT@10 rises from 0.82 to 0.85 and MRR from 0.59 to 0.63. This suggests the existence of an “effective expert threshold”: core knowledge is predominantly captured by a small subset of experts, with additional modules contributing minimally.

Architectural differences further amplify these patterns. Qwen 1.5-MoE’s shared experts handles cross-domain fundamentals—such as entity recognition and syntactic parsing—freeing routed experts to specialize in domain-specific refinement. This division allows Qwen 1.5-MoE to achieve high performance with minimal experts while maintaining fault tolerance; blocking a routed expert reduces HIT@10 by only 3% (0.85 to 0.82). In contrast, OLMoE’s lack of a shared expert forces it to activate multiple routed experts (typically three) for comparable accuracy, making it vulnerable to routing failures. For instance, blocking a critical expert in OLMoE degrades MRR by over 30%, as shown in Figure 4.

These findings collectively support a “basic-refinement” framework for MoE design (as shown in Figure 1): (1) Shared experts act as generalists, providing robust foundational processing. (2) Routed experts specialize in refining domain-specific knowledge. This hierarchical collaboration not only enhances efficiency but also improves interpretability, offering practical guidance for optimizing MoE architectures under computational constraints. For instance, prioritizing shared experts in early layers and routed experts in deeper layers could balance performance and resource usage—a strategy validated by Qwen 1.5-MoE’s success.

4.2 Semantic-Driven Attention Head and Expert Selection Mechanism

In large-scale MoE models, the interaction between attention heads and experts operates as a tightly coordinated, semantic-driven collaboration. This mechanism ensures that task-specific knowledge is dynamically routed and refined across layers (Figure 1). Take Qwen 1.5-MoE as an example, the process begins in the early to mid-layers (layers 18–20), where attention heads such as \( \mathcal{H}_{10}^{18} \) focus on encoding foundational semantic features—for example, identifying entities like “Canada” as a country. These features are stored and processed by activated experts (e.g., \( \mathcal{E}_{38}^{18} \)), which serve as repositories of domain-specific knowledge.

As processing progresses to layer 20, the routing mechanism enters a critical phase. Here, attention heads like \( \mathcal{H}_{12}^{19} \) align token-level semantics with expert capabilities, selectively activating the top-4 specialists through gating probabilities (Eq. 2). For instance, the head \( \mathcal{H}_{12}^{19} \) may prioritize expert \( \mathcal{E}_{59}^{20} \) for tasks involving geographic relations, leveraging its stored knowledge of country attributes. This semantic alignment minimizes redundant computations while maximizing task relevance.

In the deeper layers (21–23), the collaboration shifts toward refinement. Specialized experts such as \( \mathcal{E}_{48}^{21} \) work under the guidance of attention heads like \( \mathcal{H}_{13}^{22} \) to generate precise outputs—for exam-
ple, linking “Canada” to its capital “Ottawa”. To validate this hierarchical pipeline, we analyzed temporal correlations between attention heads and experts. The results reveal statistically significant synchronization: the correlation coefficient between \( \mathcal{E}^{18}_{10} \) and \( \mathcal{E}^{20}_{59} \) reaches \( r = 0.68 \quad (p < 0.001) \), while \( \mathcal{H}^{23}_{15} \) and \( \mathcal{E}^{23}_{20} \) exhibit \( r = 0.62 \quad (p < 0.01) \) (Table 4). These findings confirm that attention heads actively steer expert selection in a semantics-aware manner, contrasting sharply with dense models’ static parameter usage.

4.3 Causality Analysis
While the above temporal correlations provide strong evidence of semantic coordination between attention heads and experts, we recognize that correlation alone does not establish a causal relationship. To verify that attention heads actively drive expert selection, we designed and ran three targeted interventions:

Attention Head Suppression We selectively suppressed the activity of head \( \mathcal{H}^{10}_{13} \) in Qwen during inference. This intervention reduced the average gating probability for the top-4 experts by 54% and led to a 29% drop in mean reciprocal rank (MRR), demonstrating that disabling a single key head disrupts downstream expert routing.

Expert Forcing In a complementary experiment, we forcibly activated expert \( \mathcal{E}^{20}_{59} \) regardless of the gating probabilities. This intervention recovered 85% of the original MRR, indicating that reinstating the correct expert can largely restore model performance even when upstream signals are perturbed.

Path Analysis via Integrated Gradients We applied Integrated Gradients to trace the decision path from attention heads to experts. On average, 28% of the attribution for expert activations (e.g. the route \( \mathcal{H}^{22}_{13} \rightarrow \mathcal{E}^{24}_{41} \)) could be directly linked back to specific attention heads, quantifying their direct influence in the routing computation.

Together, these causal interventions confirm that attention heads do not merely co-activate with experts but in fact guide their selection in a semantics-aware, hierarchical pipeline.

5 MoE Knowledge Localization: Trade-offs and Robustness
In this section, we investigate the phenomenon of neuron-level knowledge localization in MoE models. For each knowledge type, identify the 100 most important neurons, determine which experts they belong to, and then we list the top 5 experts (i.e. the five experts containing the greatest number of those 100 neurons). Here, our analysis is more concentrated on fine-grained experts, whose specialization trends are particularly pronounced (Wang et al., 2024); for coarse-grained MoE models such as Mixtral, we provide a detailed description in Appendix B. We provide insights into expert specialization (\S 5.1) and investigate how architectural design affects the robustness and redundancy of the model, especially under expert failure scenarios (\S 5.2). Through systematic expert-blocking experiments, we reveal critical trade-offs between specialization depth, task sensitivity, and fault tolerance.

<table>
  <tr>
    <th>Relation</th>
    <th>High-Imp. Attention Head</th>
    <th>High-Freq. Routed Expert</th>
  </tr>
  <tr>
    <td>capital</td>
    <td>\( \mathcal{H}^{22}_{13}, \mathcal{H}^{18}_{10}, \mathcal{H}^{23}_{15} \)</td>
    <td>\( \mathcal{E}^{20}_{59}, \mathcal{E}^{21}_{48}, \mathcal{E}^{21}_{14} \)</td>
  </tr>
  <tr>
    <td>language</td>
    <td>\( \mathcal{H}^{22}_{13}, \mathcal{H}^{18}_{10}, \mathcal{H}^{23}_{15} \)</td>
    <td>\( \mathcal{E}^{20}_{59}, \mathcal{E}^{23}_{20}, \mathcal{E}^{21}_{20} \)</td>
  </tr>
  <tr>
    <td>birthplace</td>
    <td>\( \mathcal{H}^{22}_{13}, \mathcal{H}^{18}_{10}, \mathcal{H}^{19}_{12} \)</td>
    <td>\( \mathcal{E}^{20}_{59}, \mathcal{E}^{24}_{48}, \mathcal{E}^{18}_{38} \)</td>
  </tr>
</table>

Table 4: Top 3 attention heads and experts in Qwen 1.5-MoE.

5.1 Expert Specialization and Architectural Trade-offs
As shown in Table 5, our findings reveal that certain experts consistently rank among the top five across various knowledge categories such as name_birthplace, country_capital_city, and country_language.

The specialization of experts in MoE models is profoundly influenced by architectural depth. In deeper architectures like Qwen 1.5-MoE, experts adopt a clear hierarchical division of labor. Take “The capital of Canada is the city of” as an example. Early layers host generalists such as \( \mathcal{E}^{20}_{59} \), which extract broad semantic features—for example, recognizing “Canada” as a geographic entity. Subsequent layers deploy specialists like \( \mathcal{E}^{21}_{20} \), which refine these features into precise attributes (e.g., mapping “Canada” to “Ottawa”). This stratification is evident in tasks such as geographic reasoning: when processing the input “The capital of Canada is the city of”, \( \mathcal{E}^{20}_{59} \) first isolates the country context, while \( \mathcal{E}^{21}_{20} \) generates the answer “Ottawa” through attribute association.

In contrast, shallow architectures like OLMoE
<table>
  <tr>
    <th>Relation</th>
    <th>Model</th>
    <th>Top5 Routed Experts</th>
    <th colspan="3">MRR (Excl. Top*n Experts)</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th>-</th>
    <th>Top1</th>
    <th>Top5</th>
    <th>Top10</th>
  </tr>
  <tr>
    <td rowspan="2">name_birthplace</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{59}^{20}, \mathcal{E}_{20}^{23}, \mathcal{E}_{42}^{19}, \mathcal{E}_{37}^{22}, \mathcal{E}_{13}^{16} \)</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.83</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{56}^{13}, \mathcal{E}_{49}^{14}, \mathcal{E}_{42}^{14}, \mathcal{E}_{40}^{12}, \mathcal{E}_{47}^{12} \)</td>
    <td>0.82</td>
    <td>0.80</td>
    <td>0.80</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td rowspan="2">country_capital_city</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{20}^{22}, \mathcal{E}_{48}^{22}, \mathcal{E}_{27}^{21}, \mathcal{E}_{48}^{18} \)</td>
    <td>0.71</td>
    <td>0.68</td>
    <td>0.68</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{56}^{13}, \mathcal{E}_{40}^{12}, \mathcal{E}_{52}^{12}, \mathcal{E}_{12}^{12} \)</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>0.76</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td rowspan="2">country_language</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{16}^{22}, \mathcal{E}_{20}^{20}, \mathcal{E}_{20}^{23}, \mathcal{E}_{20}^{19} \)</td>
    <td>0.94</td>
    <td>0.92</td>
    <td>0.92</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{7}^{13}, \mathcal{E}_{3}^{14}, \mathcal{E}_{49}^{10}, \mathcal{E}_{56}^{13} \)</td>
    <td>0.96</td>
    <td>0.92</td>
    <td>0.87</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td rowspan="2">fruit_inside_color</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{15}^{19}, \mathcal{E}_{22}^{22}, \mathcal{E}_{20}^{20}, \mathcal{E}_{18}^{18} \)</td>
    <td>0.74</td>
    <td>0.68</td>
    <td>0.63</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{11}^{17}, \mathcal{E}_{8}^{14}, \mathcal{E}_{15}^{15}, \mathcal{E}_{32}^{12}, \mathcal{E}_{59}^{13} \)</td>
    <td>0.76</td>
    <td>0.60</td>
    <td>0.52</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td rowspan="2">object_superclass</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{22}^{22}, \mathcal{E}_{25}^{21}, \mathcal{E}_{20}^{20}, \mathcal{E}_{19}^{19} \)</td>
    <td>0.83</td>
    <td>0.82</td>
    <td>0.80</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{8}^{11}, \mathcal{E}_{59}^{10}, \mathcal{E}_{38}^{13}, \mathcal{E}_{53}^{13}, \mathcal{E}_{40}^{10} \)</td>
    <td>0.80</td>
    <td>0.75</td>
    <td>0.70</td>
    <td>0.66</td>
  </tr>
</table>

Table 5: Top 5 experts for each knowledge type, along with the model’s MRR after excluding the top 1, top 5, and top 10 of those experts. The complete experimental results can be found in Appendix B.

face inherent constraints. With only 16 layers, experts such as \( \mathcal{E}_{13}^{56} \) must balance multiple roles, handling both geographic tasks (e.g., country-language mappings) and object attributes (e.g., fruit_inside_color). While this promotes cross-task generalization, it sacrifices fine-grained precision. For instance, \( \mathcal{E}_{8}^{14} \) in OLMoE processes both fruit colors and object hierarchies, leading to blurred functional boundaries compared to Qwen 1.5-MoE’s specialized experts.

These architectural differences further impact robustness. Deep models leverage shared experts and layered redundancy to mitigate failures: blocking \( \mathcal{E}_{20}^{21} \) in Qwen 1.5-MoE causes only a marginal MRR drop (0.85 → 0.84) for geographic tasks. Shallow models, however, lack such safeguards. Blocking the critical expert \( \mathcal{E}_{13}^{56} \) in OLMoE degrades MRR by 30%, underscoring the vulnerability of architectures that prioritize breadth over depth. This trade-off highlights a fundamental design principle: deeper MoE models achieve robustness through hierarchical specialization, while shallower models rely on expert versatility at the cost of precision.

5.2 Robustness and Redundancy in MoE Architectures

Table 5 shows the changes in MRR of Qwen 1.5-MoE and OLMoE models after blocking top-n experts under different knowledge types. Experimental results show that as the Top-1, Top-5, and Top-10 experts are blocked, Qwen 1.5-MoE and OLMoE show different robustness in different tasks.

The interplay between expert specialization and architectural depth profoundly impacts model robustness. For Qwen 1.5-MoE, its 24-layer design and shared expert integration enable distributed knowledge storage, buffering against expert failures. Blocking the top 10 experts in general tasks like name_birthplace causes only a marginal MRR drop (0.85 → 0.81), as redundant general experts (e.g., \( \mathcal{E}_{20}^{20} \)) compensate for losses. However, geographic tasks like country_capital_city exhibit fragility: blocking core experts (e.g., \( \mathcal{E}_{21}^{21} \)) slashes MRR by 43% (0.71 → 0.40), highlighting their reliance on specialized modules.

In contrast, OLMoE’s shallow 16-layer architecture lacks such safeguards. While its routing mechanism dynamically selects alternative experts to maintain performance when blocking top 1–5 experts (MRR remains 1.0 for country_capital_city), deeper dependencies overwhelm its redundancy capacity. Blocking the top 10 experts degrades MRR to 0.76, with geographic tasks like country_language suffering a 30% decline (0.96 → 0.68). This stark contrast underscores that depth enables hierarchical redundancy, whereas shallow models trade transient adaptability for long-term fragility.

Furthermore, task sensitivity further dictates robustness requirements. Core-sensitive tasks (e.g., geography) demand concentrated expertise. Qwen 1.5-MoE’s late-layer specialists (e.g., \( \mathcal{E}_{48}^{21} \)) are irreplaceable, necessitating redundancy through shared experts. Distributed-tolerant tasks (e.g., object attributes) leverage broader expert participation. Qwen 1.5-MoE’s MRR drops mildly (0.83 → 0.79) when blocking experts for object_superclass, while OLMoE plummets (0.80 → 0.66) due to overburdened generalists.

These findings inform actionable design princi-
ples: (1) Deep MoE architectures should prioritize shared experts in early layers for cross-task redundancy and allocate specialists to late layers for core-sensitive tasks. (2) Shallow MoEs must balance expert versatility and routing adaptability, though their limited depth inherently restricts robustness. (3) By aligning architectural depth with task requirements, MoEs can optimize both performance and reliability—a strategy exemplified by Qwen 1.5-MoE’s resilience and OLMoE’s adaptability trade-offs.

6 Related Work

Mixture of Experts The Mixture of Experts (MoE) framework, introduced by Jacobs et al. (1991), dynamically activates specialized sub-networks for inputs. Early applications in language modeling used LSTMs (Shazeer et al., 2017), but recent work integrates MoE into Transformer FFN layers (Vaswani, 2017), scaling parameters via sparse routing (e.g., top-k gating (Fedus et al., 2021; Lepikhin et al., 2021; Roller et al., 2021; Dai et al., 2022b; Zeng et al., 2024; Xue et al., 2024)). Innovations like DeepSeekMoE (Dai et al., 2024) and Qwen-MoE (Team, 2024) enhance specialization through shared experts, balancing efficiency and performance.

Knowledge Attribution Studies on how models store knowledge reveal that Transformer MLPs act as key-value memories (Geva et al., 2021; Dai et al., 2022a; Meng et al., 2022; Wu et al., 2023; Xu et al., 2023; Meng et al., 2023), with knowledge accumulating across layers (Geva et al., 2022; Lv et al., 2024). While methods like knowledge circuits (Yao et al., 2024) and neuron-level attribution (Yu and Ananiadou, 2024) interpret dense models, sparse MoE architectures—with dynamic routing and expert collaboration—remain underexplored. Our work bridges this gap, offering the first comprehensive analysis of MoE interpretability.

7 Conclusion

In this paper, we propose a cross-level knowledge attribution algorithm for MoEs, bridging the interpretability gap between sparse and dense architectures. Through comparative analysis, we uncover MoE’s unique efficiency patterns—such as “mid-stage activation, late-stage amplification” and a “basic-refinement” collaboration framework—where shared experts handle general tasks while routed experts specialize in refinement. We further validate a semantic-driven routing mechanism: attention heads coordinate expert selection via high temporal correlations, enabling task-aware processing.

Our findings offer actionable guidelines for MoE design:

• Prioritize shared experts in early layers to provide cross-task redundancy and foundational processing.

• Allocate routed experts to deeper layers for domain-specific refinement, balancing specialization and efficiency.

• Optimize routing depth based on task sensitivity: Deploy deeper architectures for core-sensitive tasks (e.g., geography) and shallower models for distributed-tolerant tasks (e.g., object attributes).

• Enhance semantic alignment between attention heads and experts to improve routing robustness.

These principles address critical challenges in MoE scalability and reliability, offering a pathway to deploy efficient, interpretable models in resource-constrained scenarios.

Acknowledgments

This work was supported by the CAAI-Ant Group Research Fund; Guangdong Provincial Department of Education Project (Grant No.2024KQNCX028); Scientific Research Projects for the Higher-educational Institutions (Grant No.2024312096), Education Bureau of Guangzhou Municipality; Guangzhou-HKUST(GZ) Joint Funding Program (Grant No.2025A03J3957), Education Bureau of Guangzhou Municipality.

Limitations

Our analysis is confined to 7B-parameter, static-routing MoE models, so it remains to be seen whether much larger (e.g., 100 B+) (DeepSeek-AI et al., 2024; Sun et al., 2024) or fully dynamic MoE (Huang et al., 2024; Guo et al., 2025) architectures will exhibit the same specialization patterns. We also omit retrieval-augmented MoEs (e.g., Monet (Park et al., 2025)), whose external-knowledge coupling adds confounding factors beyond our scope.
References

Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.06450.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Owen technical report. arXiv preprint arXiv:2309.16609.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.

Damai Dai, Chengqi Deng, Chenggang Zhao, R.x. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.k. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280–1297, Bangkok, Thailand. Association for Computational Linguistics.

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022a. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493–8502, Dublin, Ireland. Association for Computational Linguistics.

Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. 2022b. Stable-MoE: Stable routing strategy for mixture of experts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7085–7095, Dublin, Ireland. Association for Computational Linguistics.

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Leam Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qiao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuan Chen, Shaqiong Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaoban Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437.

William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint.

Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30–45, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Mor Geva, Roei Schuster, Jonathan Berant, and Omer
Levy, 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Yongxin Guo, Zhenglin Cheng, Xiaoying Tang, Zhaopeng Tu, and Tao Lin. 2025. Dynamic mixture of experts: An auto-tuning approach for efficient transformer models. Preprint, arXiv:2405.14297.

Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415.

Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2024. Linearity of relation decoding in transformer language models. In The Twelfth International Conference on Learning Representations.

Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. 2024. Harder task needs more experts: Dynamic routing in MoE models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12883–12895, Bangkok, Thailand. Association for Computational Linguistics.

Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. 1991. Adaptive mixtures of local experts. Neural Computation, 3(1):79–87.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mistral of experts. CoRR, abs/2401.04088.

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations.

Ka Man Lo, Zeyu Huang, Zihan Qiu, Zili Wang, and Jie Fu. 2025. A closer look into mixture-of-experts in large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4427–4447, Albuquerque, New Mexico. Association for Computational Linguistics.

Ang Lv, Yuhuan Chen, Kaiyi Zhang, Yulong Wang, Lifeng Liu, Ji-Rong Wen, Jian Xie, and Rui Yan. 2024. Interpreting key mechanisms of factual recall in transformer-based language models. arXiv preprint arXiv:2403.19521.

Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. Proceedings of the 30th International Conference on Machine Learning (ICML-13), pages 338–345.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35.

Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. Preprint, arXiv:2210.07229.

Niklas Muenninghoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wetting, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. 2024. Olmoe: Open mixture-of-experts language models. Preprint, arXiv:2409.02060.

Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 807–814.

Jungwoo Park, Ahn Young Jin, Kee-Eung Kim, and Jae-woo Kang. 2025. Monet: Mixture of monosemantic experts for transformers. In The Thirteenth International Conference on Learning Representations.

Stephen Roller, Sainbayar Sukhbaatar, arthur szlam, and Jason Weston. 2021. Hash layers for large sparse models. In Advances in Neural Information Processing Systems, volume 34, pages 17555–17566. Curran Associates, Inc.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.

Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, et al. 2023. Mixture-of-experts meets instruction tuning: A winning combination for large language models. arXiv preprint arXiv:2305.14705.
Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaiqip Li, Zhen Yang, Jonny Han, Xiaoobo Shu, Jiahao Bu, Zhongzhi Chen, Xuemeng Huang, Fengzong Lian, Saiyong Yang, Jianfeng Yan, Yuyuan Zeng, Xiaoqin Ren, Chao Yu, Lulu Wu, Yue Mao, Tao Yang, Suncong Zheng, Kan Wu, Dian Jiao, Jinbao Xue, Xipeng Zhang, Decheng Wu, Kai Liu, Dengpeng Wu, Guanghui Xu, Shaohua Chen, Shuang Chen, Xiao Feng, Yigeng Hong, Junqiang Zheng, Chengcheng Xu, Zongwei Li, Xiong Kuang, Jiangu Hu, Yiqi Chen, Yuchi Deng, Guiyang Li, Ao Liu, Chenchen Zhang, Shihui Hu, Zilong Zhao, Zifan Wu, Yao Ding, Weichao Wang, Han Liu, Roberts Wang, Hao Fei, Peijie She, Ze Zhao, Xun Cao, Hai Wang, Fusheng Xiang, Mengyuan Huang, Zhiyuan Xiong, Bin Hu, Xuebin Hou, Lei Jiang, Ji-ajia Wu, Yaping Deng, Yi Shen, Qian Wang, Weijie Liu, Jie Liu, Meng Chen, Liang Dong, Weiwen Jia, Hu Chen, Feifei Liu, Rui Yuan, Hulin Xu, Zhenxiang Yan, Tengfei Cao, Zhichao Hu, Xinhua Feng, Dong Du, Tinghao She, Yangyu Tao, Feng Zhang, Jianchen Zhu, Chengzhong Xu, Xirui Li, Chong Zha, Wen Ouyang, Yinben Xia, Xiang Li, Zekun He, Rongpeng Chen, Jiawei Song, Ruibin Chen, Fan Jiang, Chongqing Zhao, Bo Wang, Hao Gong, Rong Gan, Winston Hu, Zhanhui Kang, Yong Yang, Yuhong Liu, Di Wang, and Jie Jiang. 2024. Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent. Preprint, arXiv:2411.02265.

Qwen Team. 2024. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters".

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. Preprint, arXiv:2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajijwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems.

Zihan Wang, Deli Chen, Damai Dai, Runxin Xu, Zhushou Li, and Yu Wu. 2024. Let the expert stick to his last: Expert-specialized fine-tuning for sparse architectural large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 784–801, Miami, Florida, USA. Association for Computational Linguistics.

Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. 2023. DEPN: Detecting and editing privacy neurons in pre-trained language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2875–2886, Singapore. Association for Computational Linguistics.

Shaoyang Xu, Junzhuo Li, and Deyi Xiong. 2023. Language representation projection: Can we transfer factual knowledge across languages in multilingual language models? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3692–3702, Singapore. Association for Computational Linguistics.

Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. 2024. Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402.01739.

Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, and Huajun Chen. 2024. Knowledge circuits in pretrained transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Zeping Yu and Sophia Ananiadou. 2024. Neuron-level knowledge attribution in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3267–3280, Miami, Florida, USA. Association for Computational Linguistics.

Zhiyuan Zeng, Qipeng Guo, Zhaoye Fei, Zhangyue Yin, Yunhua Zhou, Linyang Li, Tianxiang Sun, Hang Yan, Dahua Lin, and Xipeng Qiu. 2024. Turn waste into worth: Rectifying top-k router of MoE. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13305–13320, Miami, Florida, USA. Association for Computational Linguistics.
A  Experiment Settings

Models  We evaluate multiple models to ensure robust conclusions. These include three dense models (Llama-7B (Touvron et al., 2023a), Qwen 1.5-7B (Bai et al., 2023)) and Mistral-7B (Jiang et al., 2023) and three Mixture-of-Experts (MoE) models (Qwen 1.5-MoE (Team, 2024), OLMoE (Muenninghoff et al., 2024) and Mixtral 8x7B (Jiang et al., 2024)). Qwen 1.5-MoE is a 24-layer model where each MoE layer comprises a shared expert and 60 routing experts, employing a Top-4 routing strategy during inference. Mixtral has 32 layers, 8 experts per layer, no shared experts, employing a Top-2 routing strategy. In contrast, OLMoE is a shallower model with 16 layers, where each MoE layer contains 64 routing experts and employs a Top-8 routing mechanism. These structural differences among the MoE models provide complementary perspectives, enriching our analysis. Table 6 lists the specific configuration comparison of each model.

Datasets  We use the dataset provided by LRE (Hernandez et al., 2024), which contains a diverse set of knowledge types, including linguistic patterns, common sense, factual information, and biases. A total of 12 distinct knowledge categories are selected for our experiments from the dataset (i.e., adjective_antonym, word_first_letter, word_last_letter relations, object_superclass, fruit_inside_color, work_location relations, country_language, country_capital, name_birthplace, name_religion, occupation_age, and occupation_gender). This dataset serves two purposes: it provides explicit knowledge for the model and is also used to evaluate its zero-shot performance across varied knowledge domains.

Metrics  We assess model performance using two standard metrics: HIT@10 and Mean Reciprocal Rank (MRR). HIT@10 measures the proportion of test instances for which the correct answer is ranked among the top 10 predictions:

\[
\text{HIT}@10 = \frac{1}{|V|} \sum_{i=1}^{|V|} \mathbb{I}(\text{rank}_i \leq 10),
\]

where \( \mathbb{I}(\cdot) \) is the indicator function, returning 1 if the condition is satisfied and 0 otherwise, and \( \text{rank}_i \) denotes the rank position of the correct answer for the \( i \)-th instance.

![HIT@10 scores of four models on 12 different relations.](page_1012_186_393_393.png)

Figure 5: HIT@10 scores of four models on 12 different relations.

MRR computes the average of the reciprocal ranks of the correct answers:

\[
\text{MRR} = \frac{1}{|V|} \sum_{i=1}^{|V|} \frac{1}{\text{rank}_i}.
\]

While HIT@10 emphasizes the model’s ability to retrieve correct answers within the top predictions, MRR provides a more sensitive measure by giving higher scores to cases where the correct answer is ranked highly. Together, these metrics offer complementary insights into the overall ranking performance of the model.

Results  Figure 5 shows the HIT@10 of different models on 12 tasks. It can be seen that the performance of different models varies greatly in some types. Therefore, when analyzing the experimental results, we mainly focus on the relations with similar performance to avoid the influence of the model’s internal parameter knowledge on the effect of knowledge attribution.

B  Detailed Analysis of Coarse/Fine-Grained Routed Experts

We report the full expert-isolation results in Table 8. Figure 7 visualizes the heat map of the MRR drop of the models under different settings. As expected, the two fine-grained MoE models suffer precipitous drops in performance when their top specialists are blocked—reaffirming the strong localization of functionality we described in § 5.2. By contrast, Mixtral—a coarse-grained MoE with only eight experts per layer and a top-2 routing policy—maintains near-identical accuracy even when its ten most “important” experts are silenced.
<table>
  <tr>
    <th>Model Name</th>
    <th>Layers</th>
    <th>FFN / Expert Dimension</th>
    <th>Number of Experts</th>
    <th>Routing Strategy</th>
    <th>Shared Expert</th>
    <th>Shared Expert Dimension</th>
  </tr>
  <tr>
    <td>LLama-7B</td>
    <td>32</td>
    <td>11008</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Qwen 1.5-7B</td>
    <td>32</td>
    <td>11008</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>32</td>
    <td>14336</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Qwen1.5-MoE-A2.7B</td>
    <td>24</td>
    <td>1408</td>
    <td>60</td>
    <td>Top-4</td>
    <td>Yes</td>
    <td>5632</td>
  </tr>
  <tr>
    <td>OLMoE-1B-7B</td>
    <td>16</td>
    <td>1024</td>
    <td>64</td>
    <td>Top-8</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Mixtral-8x7B</td>
    <td>32</td>
    <td>14336</td>
    <td>8</td>
    <td>Top-2</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 6: Model Configuration.

![Pie chart showing type distribution of neurons in the top 100, 200, and 300 of Qwen 1.5-MoE importance scores.](page_120_393_420_312.png)

Figure 6: Type distribution of neurons in the top 100, 200, and 300 of Qwen 1.5-MoE importance scores.

![Heatmap comparing robustness across different tasks and expert routing strategies](page_670_393_420_312.png)

Figure 7: Robustness comparison heat map. The darker the color, the greater the drop in MRR.

This resilience stems from Mixtral’s relatively dense routing: with two experts activated per token, each expert has a 1/4 chance of selection, compared to just 1/15 in Qwen 1.5-MoE and 1/8 in OLMoE. Over long sequences, this routing density means that almost every expert participates, distributing representational capacity broadly rather than concentrating it in a few specialists. Since the Mixtral authors do not explicitly detail their expert initialization, we tentatively infer—based on related research (Lo et al., 2025)—that its experts may be seeded by copying FFN weights from the dense Mistral model, a practice that would likely homogenize their behaviors and impede the development of fine-grained specialization.

Taken together, these findings highlight a fundamental trade-off in MoE design. Coarse-grained architectures like Mixtral effectively amplify model capacity and fault tolerance by maximizing activation coverage, but they do so at the expense of individual expert specialization. Fine-grained MoEs, on the other hand, cultivate highly specialized experts—yielding sharp performance gains on focused tasks—but are more vulnerable to expert failure. Future work might explore hybrid schemes that combine a base level of broad activation with targeted specialization, for instance by increasing expert count, adjusting routing sparsity, or employing specialized initialization and regularization to encourage functional diversity among experts.

C Cross-Knowledge Neuron Dynamics

In the context of shared experts, we observe a significant overlap of neurons for different samples within the same relation (Table 7). For instance, in the "capital" task, neurons responsible for predicting the capitals of various countries often share common activations. This suggests that neurons
<table>
  <tr>
    <th>Category</th>
    <th>Top-5 Routed Shared Expert Neurons</th>
  </tr>
  <tr>
    <td>birthplace</td>
    <td>\( \mathcal{N}_{1044}^{20}, \mathcal{N}_{201}^{16}, \mathcal{N}_{211}^{21}, \mathcal{N}_{123}^{17}, \mathcal{N}_{91}^{19} \)</td>
  </tr>
  <tr>
    <td>capital</td>
    <td>\( \mathcal{N}_{5}^{20}, \mathcal{N}_{1270}^{12}, \mathcal{N}_{415}^{15}, \mathcal{N}_{1060}^{18}, \mathcal{N}_{637}^{18} \)</td>
  </tr>
  <tr>
    <td>language</td>
    <td>\( \mathcal{N}_{18}^{18}, \mathcal{N}_{232}^{23}, \mathcal{N}_{682}^{68}, \mathcal{N}_{647}^{67}, \mathcal{N}_{322}^{32} \)</td>
  </tr>
  <tr>
    <td>fruit</td>
    <td>\( \mathcal{N}_{20}^{20}, \mathcal{N}_{113}^{16}, \mathcal{N}_{175}^{15}, \mathcal{N}_{1158}^{19}, \mathcal{N}_{900}^{19} \)</td>
  </tr>
  <tr>
    <td>object</td>
    <td>\( \mathcal{N}_{32}^{32}, \mathcal{N}_{1145}^{145}, \mathcal{N}_{631}^{63}, \mathcal{N}_{1262}^{126}, \mathcal{N}_{359}^{35} \)</td>
  </tr>
</table>

Table 7: Top-5 shared expert neurons for each relation. For example, \( \mathcal{N}_{1044}^{20} \) represents the 1044th neuron of the shared expert in the 20th layer.

can store knowledge specific to a given relation and share it across different instances. However, the overlap of neurons across different relations remains minimal. For example, the shared neuron overlap between the "capital" and "language" tasks is typically less than 5%. This indicates that the role of shared experts is task-specific, with distinct mechanisms for knowledge storage and processing across tasks.

Further analysis, as shown in Table 8, reveals that the effect significantly increases after blocking the top 10 experts for certain relations (e.g., the MRR for the occupation_age relation in Qwen 1.5-MoE changes from 0.22 to 0.36). This counterintuitive phenomenon contrasts with the behavior observed in other relations, where blocking commonly used experts leads to a decrease in performance or fluctuations around baseline values. However, it is important to note that the model’s baseline MRR on this particular task is quite low and close to random, meaning that the observed increase does not indicate an issue with our method.

Moreover, when analyzing the top 100 neurons, we find that approximately 25% of these neurons belong to routed experts, while the remaining 75% are associated with shared experts, as shown in Figure 6. However, as we expand the analysis to the top 300 neurons, the proportion of routed experts increases to 38%. This suggests that more expert collaboration is required for handling complex tasks, with the routing mechanism dynamically expanding its activation range. As task complexity increases, the routing mechanism activates additional experts, creating a dynamic collaboration chain that connects core experts with auxiliary experts. This mechanism enables the model to flexibly coordinate knowledge from a broader set of experts, ultimately improving its performance on more complex tasks.
<table>
  <tr>
    <th>Relation</th>
    <th>Model</th>
    <th>Top5 Routed Experts</th>
    <th colspan="3">MRR (Excl. Top\( n \) Experts)</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th>-</th>
    <th>Top1</th>
    <th>Top5</th>
    <th>Top10</th>
  </tr>
  <tr>
    <td rowspan="3">country_capital_city</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{29}^{20}, \mathcal{E}_{18}^{21}, \mathcal{E}_{22}^{22}, \mathcal{E}_{21}^{21}, \mathcal{E}_{38}^{18} \)</td>
    <td>0.71</td>
    <td>0.68</td>
    <td>0.68</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{56}^{56}, \mathcal{E}_{40}^{40}, \mathcal{E}_{14}^{14}, \mathcal{E}_{52}^{52}, \mathcal{E}_{12}^{12} \)</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>0.76</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td>Mixtral</td>
    <td>\( \mathcal{E}_{21}^{21}, \mathcal{E}_{20}^{20}, \mathcal{E}_{22}^{22}, \mathcal{E}_{0}^{20}, \mathcal{E}_{19}^{19} \)</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>0.93</td>
  </tr>
  <tr>
    <td rowspan="3">country_language</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{16}^{20}, \mathcal{E}_{20}^{20}, \mathcal{E}_{20}^{20}, \mathcal{E}_{50}^{50}, \mathcal{E}_{13}^{13} \)</td>
    <td>0.94</td>
    <td>0.92</td>
    <td>0.92</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{13}^{13}, \mathcal{E}_{14}^{14}, \mathcal{E}_{49}^{49}, \mathcal{E}_{12}^{12}, \mathcal{E}_{56}^{56} \)</td>
    <td>0.96</td>
    <td>0.92</td>
    <td>0.87</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Mixtral</td>
    <td>\( \mathcal{E}_{5}^{5}, \mathcal{E}_{4}^{4}, \mathcal{E}_{6}^{6}, \mathcal{E}_{29}^{29}, \mathcal{E}_{21}^{21} \)</td>
    <td>0.98</td>
    <td>0.98</td>
    <td>1.0</td>
    <td>0.97</td>
  </tr>
  <tr>
    <td rowspan="3">adj_antonym</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{20}^{20}, \mathcal{E}_{22}^{22}, \mathcal{E}_{22}^{22}, \mathcal{E}_{16}^{16}, \mathcal{E}_{21}^{21} \)</td>
    <td>0.60</td>
    <td>0.57</td>
    <td>0.50</td>
    <td>0.53</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{24}^{24}, \mathcal{E}_{40}^{40}, \mathcal{E}_{32}^{32}, \mathcal{E}_{26}^{26}, \mathcal{E}_{25}^{25} \)</td>
    <td>0.67</td>
    <td>0.64</td>
    <td>0.65</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Mixtral</td>
    <td>\( \mathcal{E}_{0}^{20}, \mathcal{E}_{9}^{19}, \mathcal{E}_{25}^{25}, \mathcal{E}_{6}^{6}, \mathcal{E}_{21}^{21} \)</td>
    <td>0.75</td>
    <td>0.75</td>
    <td>0.74</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td rowspan="3">word_first_letter</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{19}^{19}, \mathcal{E}_{11}^{11}, \mathcal{E}_{49}^{49}, \mathcal{E}_{18}^{18}, \mathcal{E}_{45}^{45} \)</td>
    <td>0.96</td>
    <td>0.95</td>
    <td>0.93</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{39}^{14}, \mathcal{E}_{17}^{17}, \mathcal{E}_{45}^{45}, \mathcal{E}_{32}^{32}, \mathcal{E}_{60}^{60} \)</td>
    <td>0.79</td>
    <td>0.75</td>
    <td>0.68</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Mixtral</td>
    <td>\( \mathcal{E}_{7}^{30}, \mathcal{E}_{29}^{29}, \mathcal{E}_{28}^{28}, \mathcal{E}_{4}^{4}, \mathcal{E}_{28}^{28} \)</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>0.99</td>
  </tr>
  <tr>
    <td rowspan="3">word_last_letter</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{7}^{19}, \mathcal{E}_{11}^{11}, \mathcal{E}_{49}^{49}, \mathcal{E}_{45}^{45}, \mathcal{E}_{41}^{18} \)</td>
    <td>0.27</td>
    <td>0.25</td>
    <td>0.12</td>
    <td>0.12</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{14}^{14}, \mathcal{E}_{12}^{12}, \mathcal{E}_{17}^{17}, \mathcal{E}_{11}^{11}, \mathcal{E}_{10}^{10} \)</td>
    <td>0.12</td>
    <td>0.12</td>
    <td>0.10</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>Mixtral</td>
    <td>\( \mathcal{E}_{7}^{30}, \mathcal{E}_{29}^{29}, \mathcal{E}_{0}^{4}, \mathcal{E}_{4}^{4}, \mathcal{E}_{7}^{7} \)</td>
    <td>0.95</td>
    <td>0.95</td>
    <td>0.91</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td rowspan="3">name_birthplace</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{59}^{20}, \mathcal{E}_{20}^{23}, \mathcal{E}_{19}^{22}, \mathcal{E}_{16}^{16}, \mathcal{E}_{13}^{13} \)</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.83</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{56}^{56}, \mathcal{E}_{49}^{49}, \mathcal{E}_{2}^{40}, \mathcal{E}_{47}^{47} \)</td>
    <td>0.82</td>
    <td>0.80</td>
    <td>0.80</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>Mixtral</td>
    <td>\( \mathcal{E}_{18}^{18}, \mathcal{E}_{22}^{22}, \mathcal{E}_{3}^{25}, \mathcal{E}_{4}^{29}, \mathcal{E}_{31}^{31} \)</td>
    <td>0.87</td>
    <td>0.85</td>
    <td>0.83</td>
    <td>0.87</td>
  </tr>
  <tr>
    <td rowspan="3">name_religion</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{30}^{30}, \mathcal{E}_{50}^{30}, \mathcal{E}_{13}^{13}, \mathcal{E}_{23}^{23}, \mathcal{E}_{18}^{18} \)</td>
    <td>0.36</td>
    <td>0.45</td>
    <td>0.32</td>
    <td>0.37</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{13}^{13}, \mathcal{E}_{12}^{12}, \mathcal{E}_{13}^{13}, \mathcal{E}_{13}^{13}, \mathcal{E}_{49}^{49} \)</td>
    <td>0.44</td>
    <td>0.46</td>
    <td>0.45</td>
    <td>0.44</td>
  </tr>
  <tr>
    <td>Mixtral</td>
    <td>\( \mathcal{E}_{19}^{19}, \mathcal{E}_{20}^{20}, \mathcal{E}_{22}^{22}, \mathcal{E}_{18}^{18} \)</td>
    <td>0.87</td>
    <td>0.86</td>
    <td>0.84</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td rowspan="3">occupation_age</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{46}^{20}, \mathcal{E}_{19}^{19}, \mathcal{E}_{3}^{21}, \mathcal{E}_{10}^{10}, \mathcal{E}_{22}^{22} \)</td>
    <td>0.22</td>
    <td>0.21</td>
    <td>0.36</td>
    <td>0.36</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{27}^{27}, \mathcal{E}_{33}^{33}, \mathcal{E}_{50}^{50}, \mathcal{E}_{53}^{53} \)</td>
    <td>0.36</td>
    <td>0.40</td>
    <td>0.37</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>Mixtral</td>
    <td>\( \mathcal{E}_{20}^{20}, \mathcal{E}_{0}^{20}, \mathcal{E}_{6}^{21}, \mathcal{E}_{4}^{4} \)</td>
    <td>0.35</td>
    <td>0.37</td>
    <td>0.49</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td rowspan="3">occupation_gender</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{22}^{22}, \mathcal{E}_{17}^{17}, \mathcal{E}_{19}^{19}, \mathcal{E}_{18}^{18} \)</td>
    <td>0.95</td>
    <td>0.96</td>
    <td>0.94</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{27}^{12}, \mathcal{E}_{14}^{14}, \mathcal{E}_{1}^{26}, \mathcal{E}_{14}^{14} \)</td>
    <td>0.95</td>
    <td>0.86</td>
    <td>0.84</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Mixtral</td>
    <td>\( \mathcal{E}_{23}^{23}, \mathcal{E}_{20}^{20}, \mathcal{E}_{19}^{19}, \mathcal{E}_{29}^{29} \)</td>
    <td>0.97</td>
    <td>1.0</td>
    <td>0.97</td>
    <td>0.97</td>
  </tr>
  <tr>
    <td rowspan="3">fruit_inside_color</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{15}^{15}, \mathcal{E}_{22}^{22}, \mathcal{E}_{20}^{20}, \mathcal{E}_{18}^{18} \)</td>
    <td>0.74</td>
    <td>0.68</td>
    <td>0.63</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{12}^{12}, \mathcal{E}_{14}^{14}, \mathcal{E}_{13}^{13}, \mathcal{E}_{59}^{59} \)</td>
    <td>0.76</td>
    <td>0.60</td>
    <td>0.52</td>
    <td>0.41</td>
  </tr>
  <tr>
    <td>Mixtral</td>
    <td>\( \mathcal{E}_{4}^{4}, \mathcal{E}_{20}^{20}, \mathcal{E}_{3}^{25}, \mathcal{E}_{29}^{29} \)</td>
    <td>0.74</td>
    <td>0.75</td>
    <td>0.60</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td rowspan="3">object_superclass</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{11}^{22}, \mathcal{E}_{25}^{21}, \mathcal{E}_{25}^{21}, \mathcal{E}_{1}^{19}, \mathcal{E}_{15}^{15} \)</td>
    <td>0.83</td>
    <td>0.82</td>
    <td>0.80</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{8}^{8}, \mathcal{E}_{40}^{38}, \mathcal{E}_{33}^{33}, \mathcal{E}_{13}^{13}, \mathcal{E}_{13}^{13} \)</td>
    <td>0.80</td>
    <td>0.75</td>
    <td>0.70</td>
    <td>0.66</td>
  </tr>
  <tr>
    <td>Mixtral</td>
    <td>\( \mathcal{E}_{6}^{6}, \mathcal{E}_{3}^{21}, \mathcal{E}_{21}^{21}, \mathcal{E}_{19}^{19}, \mathcal{E}_{24}^{24} \)</td>
    <td>0.76</td>
    <td>0.75</td>
    <td>0.73</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td rowspan="3">work_location</td>
    <td>Qwen 1.5-MoE</td>
    <td>\( \mathcal{E}_{51}^{51}, \mathcal{E}_{27}^{27}, \mathcal{E}_{47}^{47}, \mathcal{E}_{74}^{74} \)</td>
    <td>0.45</td>
    <td>0.47</td>
    <td>0.51</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>OLMoE</td>
    <td>\( \mathcal{E}_{43}^{14}, \mathcal{E}_{12}^{12}, \mathcal{E}_{63}^{13}, \mathcal{E}_{13}^{13}, \mathcal{E}_{25}^{25} \)</td>
    <td>0.52</td>
    <td>0.48</td>
    <td>0.45</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>Mixtral</td>
    <td>\( \mathcal{E}_{21}^{21}, \mathcal{E}_{20}^{20}, \mathcal{E}_{23}^{23}, \mathcal{E}_{24}^{24}, \mathcal{E}_{18}^{28} \)</td>
    <td>0.61</td>
    <td>0.61</td>
    <td>0.59</td>
    <td>0.58</td>
  </tr>
</table>

Table 8: Top-5 most frequently selected experts for each knowledge type, along with the MRR of the model after excluding the top 1, 5, and 10 experts.

Reviewer paper summaries: The paper proposes a cross-level attribution technique to analyze knowledge attribution in MoE language models. The authors apply this technique to models like Qwen 1.5-MoE and Olmoe and compare them against dense models like Qwen 1.5-7B and find that the MoEs achieve 37% higher per-layer efficiency through a mid-activation, late-amplification processing pattern. They further study the contribution of shared experts and identify a basic-refinement collaboration pattern where the handle general tasks like entity recognition while routed experts specialize in domain-specific knowledge, like geographical attributes. They also show that the architectural depth significantly impacts robustness, with deeper models like Qwen 1.5-MoE better at compensating for expert failures through redundancy, while shallow architectures like Olmoe suffer severe performance degradation when important experts are blocked. The authors suggest that these findings could offer principled guidelines for designing more efficient, specialized, and robust MoE architectures.
Summary of strengths: 1. There is comparatively less work on interpreting internals of MoEs, the results from this work could provide actionable guidelines for designing MoE architectures and could be interesting to the larger MoE community if shown to generalize across multiple tasks.
2. The authors showcase interesting phenomena in both specialized experts and the shared experts across a variety of different model architectures
3. The authors show strong evidence for their claims of efficiency gains and robustness characteristics for the tasks chosen.
Summary of weaknesses: 1. The paper examines only two MoE architectures (Qwen 1.5-MoE and Olmoe), which is a subset of various MoE architectures that have been recently proposed.
2. While the authors use 12 knowledge categories, these are primarily factual retrieval tasks, and they may not comprehensively test all aspects of language model capabilities.
3. The findings of the paper might be specific to the particular implementations studied rather than universal principles applicable to all MoE architectures.
4. While correlations between attention heads and expert selection are a good indication of the proposed phenomena, the authors do not show rigorous causal relevance of the identified components.
Comments, suggestions and typos: 1. Recent work from Hase et. al (Does Localization Inform Editing?) hints that attribution-based approaches may not point to the modules in the network that are causally relevant. It might be worthwhile to address that in the paper.
2. The authors should apply their proposed approach to other MoEs, such as Switch Transformer or Monet
Overall assessment scores: 4