Mixture of Experts Made Intrinsically Interpretable
Authors: Xingyi Yang
Constantin Venhoff
Ashkan Khakzar
Christian Schroeder de Witt
Puneet K. Dokania
Adel Bibi
Philip Torr
Venue: ICML 2025 poster (2025)
Keywords: Mixture of Expert; Interpretability; Polysemanticity
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: We present **MoE-X**, a mixture-of-experts (MoE) language model designed to be *intrinsically* interpretable.

Abstract: Neurons in large language models often exhibit \emph{polysemanticity}, simultaneously encoding multiple unrelated concepts and obscuring interpretability. 
Instead of relying on post-hoc methods, we present \textbf{MoE-X}, a mixture-of-experts (MoE) language model designed to be \emph{intrinsically} interpretable. Our approach is motivated by the observation that, in language models, wider networks with sparse activations are more likely to capture interpretable factors. however, directly training such large sparse networks is computationally prohibitive. MoE architectures offer a scalable alternative by activating only a subset of experts for any given input, inherently aligning with interpretability objectives. In MoE-X, we establish this connection by rewriting  the MoE layer as an equivalent sparse, large MLP. This approach enables efficient scaling of the hidden size while maintaining sparsity. To further enhance interpretability, we enforce sparse activation within each expert and redesign the routing mechanism to prioritize experts with the highest activation sparsity. These designs ensure that only the most salient features are routed and processed by the experts. We evaluate MoE-X on chess and natural language tasks, showing that it achieves performance comparable to dense models while significantly improving interpretability. MoE-X achieves a perplexity better than GPT-2, with interpretability surpassing even sparse autoencoder (SAE)-based approaches.

Mixture of Experts Made Intrinsically Interpretable
Xingyi Yang Constantin Venhoff'
Ashkan Khakzar' Christian Schroeder de Witt Puneet K. Dokania
Adel Bibi
Philip Torr
Abstract
Dense MLP
Wide & Sparse MLP
Neurons in large language models often exhibit polysemanticity, simultancously encoding multi- ple unrelated concepts and obscuring interpretabil- ity. Instead of relying on post-hoc methods, we present MoF-X, a Mfxture-of-Experss (MoF) lan- guage model designed to be intrinsically inter- pretable. Our approach is motivated by the ob- servation that, in language models, wider net- works with sparse activations more likely to capture interpretable factors. However, directly training such large sparse networks is computa- tionally prohibitive. MoF. architectures offer a scalable alternative by activating only subset of experts for any given input, inherently aligning with interpretability objectives. In MoE-X, we establish this connection by rewriting the MoE layer as an equivalent sparse, large MLP. This approach enables efficient scaling of the hidden size while maintaining sparsity. To further en- hance interpretability, we enforce sparse activa- tion within each expert and redesign the routing mechanism to prioritize experts with the highest activation sparsity. These designs ensure that only the most salient features routed and processed by the experts. We evaluate MoE-X on chess and natural language tasks, showing that it achieves performance comparable to dense models while significantly improving interpretability. MoE-X achieves perplexity better than GPT-2, with in- terpretability surpassing even sparse autoencoder (SAE)-based approaches.
Interpeetabllity
Sparsified Expert & Routing
Sparsity Aware
MoE-X Layer
Figure I. MoE-X introduces sparse and wide network architec- ture designed for interpretability. Compared to dense MLPs, incorporates both sparsity and wider structure. Unlike traditional MoF. models, enforces sparsity within each expert and routes tokens the sparsest experts.
workings remain poorly understood. This lack of under- standing often leads to unexpected and potentially harmful behaviors (Hendrycks et al., 2023; Ngo et al., 2022), posing risks in their deployment. To address this, mechanistic inter- pretability (Blhage al., 2022c) seeks to uncover how these models process information and reduce potential risks.
A central obstacle to mechanistic interpretability is polyse- manticity, where individual neurons encode multiple, unre- lated concepts (Olah et al., 2020). Specifically, we refer to the hidden neurons from the multi-layer perceptron (MLPs) in Transformers. Such polysemantic neurons lack clear, singular roles, making it difficult identify disentangled features or factors in neural networks.
A common strategy to address this issue is to decompose en- tangled neuron activity into interpretable vectors using post- hoc methods like Sparse Auco-Ensoders (SAEs) (Huben et al., 2023). However, these approaches are computation- ally expensive (Gao et al., 2024: Lieberum et al., 2024). require additional analysis after training, and often do not explain all the features of the model (Menon et al., 2024).
1. Introduction
Transformer-based large language models (LLMs) (Rad- ford et al., 2019; Brown et al., 2020; Waswani et al., 2017) have achieved remarkable progress. However, their internal
University of Oxford National University of Singapore.
Instead, we advocate for designing interpretability directly into the model architecture in which the resultant discour- ages polysemanticity during training. While some works explore architectural changes for interpretability, they often
Proceedings of the 4 International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Mixture of Experts Made Intrinsically Interpretable
focus on toy-scale tasks (Pearce et al., 2024; Agarwal et al., 2021; Sharkey, 2023; Jermyn et al., 2022) or compromise performance (Elhage et al., 2022a).
semantically meaningful internal representations. Our ex- periments on chess and language tasks confirm that MoB-X matches or exceeds the performance of dense transformers while eliminating the need for expensive post-hoe inter- pretability methods.
To achieve built-in interpretability, as shown in Figure 1, we identify two key factors that influence it, (1) increasing the size of the MLP layers, i.e., number of hidden activations, and (2) increasing the sparsity of these activations. That is to say, making the MLP layers transformer wider and spanser should encourage more disentangled internal rep- resentations. To test this beyond toy experiments (Jermyn et al., 2022; Elhage et al.. 2022b), we conduct experiments on GPT-2-like (Radford et al., 2019) models on chess game- play data (Karvonen et al., 2024). Chess provides an ex- cellent natural "ground truth" tM for interpretability because the board state can be used a reference for understand- ing how each neuron represents and predicts chess moves. Our experiments show that sufficiently wide-and-sspase MLP in transformer indeed yields more interpretable neu- rons, leading to a 25% increase in Fl score for chess move prediction, supporting our hypothesis.
In summary, our main contributions are:
a) We systematically analyze how architectural particll-y width and ipafsity-influene interpretability in transformer-basee language models.
b) We introduce MoE-X, a redesigned MoE layer that func- tions as wide, sparse, and more interpretable MLP within large language models.
We incorporate ReLU Experts and Sparsity-Aware Rout- ing to tightly connect gating decisions with the expected sparsity of activations.
d) Our experiments chess tasks demonstrate that MoE-X models achieve strong performance while offering clear and interpretable representations.
Motivated by these findings, we propose to leverage mixture- of-experts (MoE) architectures (Fedus al., 2022; Shazeer et al., 2017) natural fit for intrinsic interpretability. While different from a standard MLP, MoE can be rewritten as wide and sparse MLP, whose neurons are each expert's s neuron weighted by the sparse gating scores. This structure allows MoE to increase the width while maintaining con- trolled sparsity, leading to inherently interpretable models.
2. Related Work
Mechanistic Interpretability & Polysemantics. Mecha- nistic interpretability (Olah, 2022) aims understand deep neural networks by analyzing individual units (e.g.. neu- rons) reverse-engineering their computations (Elhage et al., 2021). Although this approach provides insights into large language models (LLMs) (Zhong et al., 2024; Wang et al., 2022), many neurons remain polysemantie, activating for multiple concepts (Elhage et al., 2022b), making interpreta- tion difficult. Post-hoe hoc methods like Sparse Auto-Encoders (SaEs)(Gao et al., 2024; Huben et al., 2023) attempt to address this but are computationally expensive and incom- plete (Menon et al., 2024). In this paper, we introduce a MoE architecture to reduce polysemanticity. This approach promotes more interpretable internal representations with- out the need for extensive post-hoc methods.
Yet, typical MoE models are not perfect for interpretability. First, each expert is still dense MLP, which can suffer from polysemanticity. Second, standard lop-k routing (Fedus et al., 2022) primarily targets performance rather than expert sparsity. As a result, the gating decisions made by the routing mechanism are often misaligned with the goals of interpretability.
To bridge this gap and align MoE with interpretability, we propose MoE-X, which includes two key designs:
ReLU Experts. We use ReLU activation within each expert. This simple yet effective modification promotes intrinsic activation sparsity in the expert (Awasthi et al., 2024), which in turn helps to disentangle the feature representations.
Intrinsic Interpretability. Intrinsic interpretability aims to design neural networks that are inherently easier to un- derstand without sacrificing performance. These methods enforce sparsity, modularity, and monosemanticity through architectural and training constraints. For example, (Liu et al., 2023a;b) use brain-inspired modular training to en- hance anatomical modularity RNNs, while (Jermyn et al., 2022; Elhage al., 2022a) explore structural choices for monosemanticity, and (Sharkey, 2023) employs bilinear layers for interpretability. In language modeling, some ap- proaches incorporate concepts like bag-of-words (Hewitt et al., 2023) or utilize prototypes (Xie et al., 2023) to achieve this. We take a different approach by leveraging MoE for intrinsic interpretability.
Sparsity-Aware Routing. We introduce gating function that predicts which experts would produce the most sparse activations. To avoid expensive computations, we develop method to estimate each expert's sparsity without explicitly computing all activations. This ensures that the sparsest experts are chosen during inference, promoting disentangled representations.
Together, these modifications enable MoB-X maintain competitive performance while providing more transparent,
2

---

Mixture of Experts Made Intrinsically Interpretable
Mixture of Experts. Mixture-of-Experts (MoE) models dynamically route input specialized "experts" reduce computation (Jacobs et al., 1991; Shazeer et al., 2017). Recent work focus on replacing MLP layers in LLMs with MoF. layers, achieving better performance at lower cost (Jiang et al., 2024; Fedus et al.. 2022). A major chal- lenge in MoE is designing the routing function (Zhou et al., 2022), which typically requires an auxiliary loss to balance expert usage. Regarding interpretability, previous studies observed that MoE models tend to exhibit increased monose- manticity (Park et al., 2024; Oldfield et al., 2024). but these studies offer limited explanations for why this occurs. In this work, we clarify its underlying mechanisms and propose a redesigned routing function that prioritizes experts with more interpretable activations, rather than focusing solely on performance.
Willoe Koerscn
30000
MiDhoput
Averae
Roctaale Gare Revakue
Atteniinn
1ibd fiss 1u0 Noa Ros pete
Figure 2. Illustration of using chess game evaluate the LLM's interpretability..
Coverage of Chess Board State
ta 15.d 5121
0.40
Ja- 6,d 2045
0.35
0.25
0.20
Fixed =
3. Preliminary Study: What Architectural Choices Enhance Interpretability?
Fixed 512
5121
SAE
50
1.00
200
250
300
330
Model Size (MB)
To design more interpretable architectures, it essential to identify what is the key influencing factors.
Figure 3. Comparision BSP Coverage score v.s. Model size.
In this section, we conduct series of toy experiments by training LLMs on chess gameplay data and evaluating their interpretability. Through extensive ablations of various de- sign choices, we identify two key factors that significantly enhance inherent interpretability in language models:
After training, we calculated the BSP Coverage Score on layer-6 MLP hidden activation. This score measures the av- crage best Fl score when each feature is used as a classifier for BSPs. More information can be found in Section 5.1 and Appendix. We report the core observations on model hidden size and activation sparsity.
a) MLP Hidden Size: Larger hidden states result better interpretability.
b) Sparsity of Hidden Activations: Lower numbers of non- zero neurons lead to more interpretable representations.
Study I: Model Hidden Size. this study, we train mod- els with different MLP hidden widths D. As shown in Figure 2, the width is determined by two factors: the input dimension of the MLP as d, and the hidden size multiplier
In the next section, we will use these findings to design intrinsic interpretable architectures.
a. Together, the hidden size is D = ad. We vary either d or a. With a = fixed, we test D E (256, 512, 1024, 2048). With d 512 fixed, we test a E {2, 8, 16). Figure compares BSP coverage for fixed and fixed a, with model size on the x-axis. As baseline, we plot the SAE score with dictionary size of 4096, trained post-res for model (ar = 4, d 512). This results in nearly 3x increase in model parameters.
3.1. Measuring interpretability on Chess Games
Designing and evaluating the interpretability of language models is challenging due to the absence of universal metric. In our experiments, we to assess interpretability (McGrath et al., 2022; Toshniwal et al., 2022; He et al., 2024). Specifically, we measure how well the model's internal activations align with semantically meaningful chess board state properties (BSP), using the metries described in (Karvonen et al., 2024).
chess game dataset
We make three key observations. First, increasing both d and a improves the interpretability, as indicated by a higher coverage score. Second, scaling a is better than scaling d. This implies that, instead of increasing the overall model size, can efficiently scale the hidden size multiplier for better interpretability. Third, by increasing the model size, we can eventually outperform the SAE features in terms of interpretability. For example, model with (a = 16, d = 512) achieves score of 0.53, compared to 0.45 for (a= 4, d 512)+SAF, with a similar overall parameter.
Dataset and Metrics. As shown in Figure 2, we trained LLMs chess Portable Game Notation (PGN), treating it language, and analyzed the interpretability of MLP hidden activations. Specifically, we trained an 8-layer GPT2- like model (Radford et al., 2019). Each character in the PGN is treated as a token, and we conducted next-toker prediction training.
3

---

Mixture of Experts Made Intrinsically Interpretable
Coverage of Chess Board State
excellent candidate for such design. In this section, we first explain how SMoE works and demonstrate how it aligns with wide-and-sparse principles. Building on top of this, we propose a new MoE layer designed to further enhance interpretability.
0.50
0.45
0.40
0.35
0.30
4.1. Preliminary: Sparse Mixture of Experts
0.25
0.20
ReLU
Top-k
SMoE improves efficiency by activating only a subset of computations per input. Unlike traditional models that use a single MLP for each LLM layer, SMoE employs multiple parallel MLPs, referred to as "experts". Each token routed to a subset of experts, and their predictions are combined using weights determined dynamically based on the input.
0.15
am88
a=16
Baseline
10a
10
L0 (Lower is Sparser)
Figure 4. Comparing BSP Coverage score v.s. L-0 norm of the hidden.
Formally, SMoE comprises two key components: experts and a gating network.
Study II: Activation Sparsity. In this study, we explore how the activation sparsity affects the model interpretabil- ity. we explore different techniques to control the sparsity of the hidden activations in the MLP. Specifically, we ei- ther use ReLUU activation instead of GELU, or apply Top-k activation (Makhzani & Frey, 2013; Gao et al., 2024)'.
Experts: Let I be the input vector of one token. A SMoE layer consists of M experts, each represented as fx() for j E {1,2, MM. Typically, each expert is small MLP
[xx;0,) = whi (1) 2(100 - (w(Llx) (h()).
We visualize the results in Figure 4. We find that, for various model sizes, imposing sparsity typically improves coverage scores (difference between the dashed blue and green line). Moreover, these gains become more pronounced in larger models. However, the most extreme sparsity does not always yield the best interpretability, and identifying the optimal sparsity level remains difficult.
where W E RdxDD and wif) E RDd are weight. dec h() and z(3 are pre-activation and post-activation vec- tors. Here, D represents the hidden dimension of each expert.
Gating Network: The gating network g(x; () generates set of weights w = [1], 42,..., wwrl R. with each wi indicating the contribution of expert the output. These weights are computed using learnable matrix W 9 E M as follows:
Analysis and Understanding. We hypothize wide and sparse neural networks are more interpretable as they min- imize feature superposition (Elhage et al., 2022b). Width provides sufficient capacity for the model to assign distinct neurons to specific features. Sparse activations ensure that only small subset of relevent neurons is active for given input, which reduces interference between features.
= g(x; o) Softmax (TopK (Wxx)),
(I)
Although similar architectural properties have been ex- plored in non-language tasks and non-transformer archi- tecture (Jermyn et al., 2022; Elhage et al., 2022b), or SAE features (Gao et al., 2024), evaluating those properties on language model pretraining was previously unexplored.
Only top-k values are retained and normalized using soft- max (Shazeer et al., 2017), while the rest are set to zero.
The final output of the MoE model, ý, is weighted sum of the expert outputs y = Σe= wify (x; 0y). Since most wy are zero, this model is referred to as a "Sparse" MoF.
4. Mixture of Experts for Intrinsic Interpretability
4.2. SMoE is a Natural Fit for Interpretability
SMoE naturally aligns with our identified interpretable ar- chitecture, as it is both wide and sparsely activated. To see this, consider a "mega-decoder" by concatenating all expert decoder matrices Was concat([W:)) dec:.... w(A))) E Rd We can also define new hidden code as z = concat([w)z ,.... wy (M)) E MD. Here, each w,z()) is the scaled activation from j-th expert. Notably, decoding z through Wate dec exactly the same as SMoE output (Liu et al.,
Building on the observations in Section 3. we aim to de- sign architectures that are both wide and sparsely activated, ensuring better interpretability. Sparse Mixture of Experts (SMoE) naturally aligns with these properties, making it
We also experimented with & regularization but found it in- effective for reducing the &a norm and challenging to tune, excluded it.

---

Mixture of Experts Made Intrinsically Interpretable
2023c)
for ||20/ based on probabilistic assumptions about the
encoder weights w
M y = (x;0)x = Σ wll dec (w)z((0 = WW d
M
Approximate Sparsity via Gaussian Assumptions. For expert j. assume each column of the encoder weight matrix
(2)
Whit is drawn ii.d. from the same Gaussian distribution, (Wm,3 }m= ~ N(u,)) ( (a()))) Then each component h( of the pre-activation vector h is a sum of Gaussian random variables and thus also follows Gaussian distribu- tion:
In other words. SMoE acts like larger MLP whose hidden layer is each expert's 'ss activations, scaled by its gating score. Because only top-k non non-zero W are retrained, z is structured sparse. When wij 0, all elements in w,z5) are zero Con- sequently, SMoE is "wide", as its hidden dimension is M., but also "sparse", as activations are restricted to a subset of experts. In this way. SMoE satisfies the criteria of wide and sparsely activated MLP that supports interpretabilityy
1P N (MR (0))))))) =N Euu Di EioP)
(4)
Thus, the probability that 11() is positive is given as Prob Prob(h!)) 0) + ( fo()) where (bh) is the CDF
SMoE is not perfect for interpretability. Despite its inher- ent sparsity and modularity, two key issues remain. First, activations in each expert are still dense, which can still lead to polysemantic features. Second, the gating function is trained purely for performance, so its values may not reflect interpretable expert properties. We address these issues in the following sections.
of the normal distribution. Since each elements in h fol- lows the same distribution, the fo-norm could be estimated as the expected value of
( Ph (
||20 = ΣBI
4 2 20] = DΦ
(5)
4.3. Designing SMoE for Greater Interpretability
Hence, we pick expert(s) that minimize ||z i.e., those with the lowest probability of being positive. Router Implementation. In practice, we estimate u6 and
To address the issues discussed above, we redesign both the expert architecture and the routing function to enforce neuron-level sparsity within each expert.
(2)
()) efficiently using column- wise statistics of W and oh
4. RELU EXPERT FOR ACTIVATION SPARSITY
That is to say, for expert
To address the first challenge, we adopt the ReLU function as the non-linear activation (-) for each expert MLP. Em pirically, we observe that experts trained with ReLU exhibit high degree of activation sparsity, which helps disentan- gle features while maintaining strong performance. While intrinsic activation sparsity has been studied in the context of efficiency (Zhang et al., 2024; Mirzadeh et al., 2023; Awasthi et al., 2024), its role in enhancing interpretability is less explored.
1
w
( [m,:]
D
m=1
1 D
Σ
(W
[m. ())²
where have that 14 nو) and 0h ()) o r()) (x²).
Using the CDF approximation (h) ~ 1 } + erf(h/ (2)))
we compute the gating weights via a TopK So ftmax op- eration
4.3.2. SPARSITY-AWARE ROUTING
S PB
TTop (-erf"
W x)x::))=Soo))
- Softmax
To tackle the second issue, we aim to route each input token to the expert fj. which produces sparsest activation (i.e., fewest non-zero entries). Formally, if 25) = Re then the sparsity of (()) can be discribed using its Eo -nom
V2ok
(6)
Here, erf(-) is the error function. The resulting gating scores select the experts and provide an inexpensive estimate of each expert's 's activation sparsity.
| 10 = Σ 10h
( 2 0),
(3)
Routing Regularizes Sparsity. Since we do not ach the gradient of ( enc. the gating function implicitly regular-
where IT()) stands for the indicator function. The simplest approach to do gating is to evaluate z(i for all experts and select the one with the fewest positive elements. However, this contradicts the SMoE principle of limiting computa- tion to a subset of experts. Instead, we use cheap proxy
izes the experts to produce sparser activations. Specifically, when the model learns favor an expert j by increasing its
gating score wy. it simultaneously updates W enc, to encour- age activation ( to be sparser.
5

---

Mixture of Experts Made Intrinsically Interpretable
ally, we report the BSP Coverage and Reconstruction score defined in (Karvonen et al., 2024) assess interpretability.
Table Comparison with baseline method by keeping model acti-
vated parameters the same.
Model
Val Loss
Coveraget
Reconstruction
We compared our proposed MoE-X X against three families of models. The first is dense baseline model similar to GPT-2. The second includes models with activation func- tions designed for better interpretability, such as bilinear layers like GEGLU (Pearce al.. 2024; Shazeer, 2020) and SoLU (Elhage al., 2022a). The third consists of MoE models, including fine-grat tne-grained MoEs like Monet (Park et al., 2024) and PEER (He, 2024), as well standard MoEs like the Switch Transformer (Fedus et al., 2022). For Switch Transformer and MoE-X, we use experts, with experts activated at a time. For MoE models, we explain the scaled hidden representation z defined Section 4.2. This avoids using the raw activations from each expert.
GELU (GPT-2)
0.213
0.356
0.608
Activation Function
ReLU
0.215
0.312
0.581
GEGLU
0,209
0.255
0.394
SoLU
0,216
0.306
0.343
Mixture-of- -xxpers
Monet-HD
0.210
0.312
0.528
Monet-Vi VD
0.212
0.283
0.482
PEER
0.214
0,323
0.426
Switch
0.212
0.424
0.734
MoE-X
0.211
0.428
0.840
Reconbhuttin of Soord State
Coverge of Biond State
All models have 8 layers and are trained for 60k iterations with batch size of 100. We use the Adam W optimizer with an initial learning rate of 3e-4 and cosine scheduling to reduce the learning rate to le-4 in the end. We train MoE-X by upcycling the weights (Komatsuzaki et al., 2022) from dense model. Additionally, we applied a load balance loss with a value of X 0.001. All experiments were conducted on 4 NVIDIA A40 GPUs. More details are listed in Appendix.
GPT2 SAE
GPT2 SAE
GFT2
GF12
Sw las
Active Porommeters (01)
Active Parameters
Figure BSP Coverage and Reconstruction score of different model sizes.
Computational Complexity. Given input x Nxd, standard top-k gating has complexity Md). Naively computing all activations and their fu-norm costs MDd.. In contrast, our Sparsity-Aware Routing only requires Dd) for column-wise statistics plus and Md) for imner products, resulting a total complexity of O((N + Dd). This design scales efficiently with large numbers of experts and high-dimensional inputs.
MoE Achieves Better Interpretability. We present the interpretability scores of different models in Table 1. To ensure fair comparison, we strictly match the number of activated parameters between dense models and MoE models. For dense models, we use an MLP hidden size of D 4096, while for MoE models, activate 2 experts, each with 2048 hidden neurons.
5. Experiments
Several key observations emerge from the results. First, MoF models demonstrate superior interpretability. Swieth transformer readily improve the interpretability score, and our proposed MoE-X achieves the best Reconstruction Score of 0.84.
In this section, we conduct experiments on the chess and language datasets validate the design of MoE-X, focusing on both performance and interpretability.
5.1. Chess Play Experiments
Second, prior architecture designs claiming improved inter- pretability do not perform well in practice. For example, SoLU's scores is even lower than the GELU-based GPT-2 baseline. Similarly, recent MoE models claiming to improve monosemanticity, such as Monet (Park et al., 2024), do not do well. We hypothesize that this is due to the use of product key quantization (Lample et al., 2019), which relies on the Cartesian product. This method makes expert gating scores interdependent, preventing experts from functioning inde- pendently-a key requirement for interpretability. These findings call for a thorough re-evaluation of this field, as many claims of improved interpretability lack strong empir- ical support.
Experimental Setup. For chess experiments, we train models on lichess 6gb² (Karvonen, 2024), a 16 mil- lion games from the public Lichess chess games database. The input to the model is chess PGN string (1.e4 e5 2. Nf3 .) of maximum length of 1023 characters, with each character representing an input token. The model's 's vocabulary consists of the 32 characters necessary to construct chess PGN strings. We split the dataset into 99% of training corporse and validation on 1% of validation set, report validation loss test performance. Addition-
games
6

---

Mixture of Experts Made Intrinsically Interpretable
Switch Transformer
Switch ReLU Expert
MoE-X
Expert
Expert
Export
Export
Export
Expert
Expert
Expert
Figure 6. t-SNE projections of encoder weights for original MoB layer, MoE with ReLU experts, and without full MoE-X X layers, trained on Chess datasct.
MoE-X Scales Interpretability Faster. We evaluate in- terpretability across different model sizes, comparing dense GPT-2, Switch Transformers, and MoE-X. To compare the
"Against" in spurt gumes Expert 4, 2813
Cl./Claa 4/% Male scnalieed pane aeaned St. Thetaar Xioe E.dd
[-] strunl foottall game agatnd Crimace Knight Cool crrkd consrrmaa hompiees Lenyse muech Rramml
Celestial Objects Expert 0, 539
of tee scnd lod
the Mixes mest (..)
size fairly, for dense models, we fix D = 512 and vary {4, 8, 16). For MoEs, we set 512 and
G.o) Their sock intlamnoo life Earth (..)
for each expert, while varying the number of activated experts 1 E (1,2, 4).
Roman numerals Expert 0, 12
(. oon he lad la your grow Ill
(-2 aarooo bon @ Thiich lus (..)
(radppier
First name "Jeff" Expert 3,0 1273
C.3il Old Bill Boo al
< S..oorresk Anthores Bike Aves boct Katee
As shown in Figure 5, interpretability scores improve signif- icantly as model size increases. With the same number of activated parameters during inference, MoE-X consistently outperforms alternatives, particularly in the BSP Reconstruc- tion Score.
"Bimding" Expert 4, 491
[ 1G2 FW
Jesesa poly
Figure 7. Activated tokens for experts in MoB-X small on RedPajama-v2 validation dataset. Their interpretations were iden- tified using the auto-interpretation.
MoE-X beats SAE with Greater Faithfulness. We com- pare M with SAE trained on GPT-2-small post-res with a SAE hidden size of 4096. As shown in Figure 5, MoE-X achieves better interpretability than SAE with the same total parameters (GPT-2 + SAE).
5.2. Interpretability for Natural Language
Experimental Setup: For natural language models, pretrain on the 10BT subset of Fine (Penedo et al., 2024). We use a batch size of 320, context length of 1024 tokens per sentence, and train all models for 100k gradient steps. We evaluate the models on Open WebTWe (Gokaslan et al., 2019), LAMBADA (Paperno et al., 2016), Wiki- Text103, and WikiText2 (Merity al., 2016), and reported the perplexity (PPL) score to show the performance.
Moreover, MoE-X is inherently more faithful due its intrinsic interpretability. Unlike SAE, which relies on post- hoc decomposition to approximate features, MoE-X directly learns interpretable features. As a result, SAE always suffers some performance loss (~ 96% validation loss), while MoE- achieves perfect fidelity (100% loss recovery).
In addition to performance evaluation, we measure the in- terpretability by rumning the nute-interpretabiiity pipeline and report the Detection Accuracy² defined in (Paulo et al., 2024). To obtain this score, we collect the activations of the target MLP over 10M tokens from RedPajama-v2 (Weber et al., 2024). The activated contexts are then fed into an explainer LLM, which provides short interpretation for the corresponding neuron. A scorer LLM is asked to do binary classification to determine whether a whole sequence activated a hidden neuron given an interpretation and test text. We report the accuracy of this classification. We use Llama 3. 3.1b 70b instruct as both the scorer and the explainer model. More detail in Appendix.
MoE-X Expert cluster features. To better understand the models, visualize the encoder weights of different MoE models trained on chess data using t-SNE (Van der Maaten & Hinton, 2008) projections. We treat each row data point, and apply t-SNE to
from W! enc is treated project them onto 2D plot. As shown in Figure our MoE- X effectively clusters vectors for expert 0, 4, 6, 7, capturing topics related to interpretable factors. In contrast, vanilla MoE models, such as the Switch Transformer, use routing functions optimized solely for performance, which fail to form meaningful cluster of features.

---

Mixture of Experts Made Intrinsically Interpretable
Table 2. Language modeling performance for different architectures. For PPI., lower better.
Model
Open Web (PPL))
LAMBADA (PPL)
WikiTText103 (PPL).
WikiTexf2 (PPL).
GPT-2 Small
22.83
32.71
49.89
44.36
GPT-2 Small SAE
31.60
38.21
55.33
49.16
Switch-S (8x 124M)
18.36
27.63
45.22
38,90
MoE-X-S (8x 124M)
19.42
28.11
43.80
42.58
GPT-2 Medium
17.19
24.31
37.87
35.70
Switch-M (8x354M)
15.43
20.82
35.41
34.71
MoE-X-M (8x354M)
14.78
21.34
35.01
35.16
We compare MoE-X with GPT-2 and Switch-Trame Transformer. For GPT-2, we trained small (124M) and medium (354M) models. Similarly, for Switch-Transfomen and MoE-X, created small and medium configurations with 8 experts each. During inference, 2 experts are active, resulting in 180M and 555M active parameters for small and medium models. We evaluate interpretability at layer 8 for small models and layer 16 for medium models. GPT-2 with SAE also evaluated post-res res of layer
SAE
0.8
GPT2-Sma
MoE-X Small
Accuracy
0.7
MoE-X Medium
0.6
0.4
0.3
Quantitative Experiments. Table presents the language modeling performance for different models. We observe that MoE models outperform dense model like GPT-2, with Switch Transformer slightly ahead of MoE-X but at com- parable level. Notably, GPT-2 performance drop signifi- cantly when running with SAE. This is because post-hoc explainations like SAE simply fail to capture all crucial features. results in reduced performance and less faithful explanations.
0.2
Not
Q1
Q2
Q3
Q6
Q7
Q8
Q9
Q10
Quantiles
Figure 8. Automated Interpretability Detection Results in 8th Layer Hidden Activation Quantiles 1000 Random Features with 95% Confidence Intervals. Not indicates non-activating text.
Table 3. Ablation study of Routing and Expert Choice.
ReLU Expert
Sparsity Router Coverage
Reconstruction
For interpretability, we report the Detection Score for 1,000 randomly selected features. Each feature is scored with 100 activating and 100 cot-activating examples. The activat- ing examples are chosen via stratified sampling such that there are always 10 examples from each of the 10 deciles of the activation distribution. Figure 8 illustrates the overall accuracy. GPT+SAE serves strong baseline for inter- pretability, which MoE-X Small already matches. When the model size is increased to MoE-X Medium, interpretability improves further, surpassing SAF.
X
0.424
0.734
x
0,404
0.740
/
0.418
0.829
0.428
0,840
Sparsity-Aware Routing. We evaluate our gating func- tion from two perspectives: (1) whether selects sparser perts, and (2) whether it improves interpretability. Figure 9 shows the fn-norm of each expert's activations across 5,000 sentences, alongside gating scores from both a standard top- approach and our sparsity- -based method. While standard gating often misestimates sparsity, our gating scores exhibit strong negative correlation (r -0.95) with the actual expert sparsity and consistently selects sparser experts. As shown in Table 3, applying sparsity-aware gating top of ReLU experts further boosts interpretability.
Qualitative Experiments. We show some auto-interp re- sults on MoE-X small and its top-activant aetiiitittedd context in Fig- ure 7. More results included in Appendix 5. MoE-X successfully identifies interpretable concepts.
5.3. Ablation Study and Analysis
6. Conclusion
ReLU Expert. We verify the benefit of use ReLU experts by replacing it with default GELU function, and train the chess dataset using 2-of-8 expert setup. Since ReLU zeros out negative values while GELU does not, this replace- ment increases the average l, norm of hidden activations from 166 to 4091. Besides, as shown in Table 3 applying ReLU significantly improves the reconstruction score. Those experiments show that applying ReL.U induces sparser and more interpretable features.
This paper addresses the challenge of improving inter- pretability in LLMs by introducing MoE-X, mixture-of- experts architecture designed for intrinsie transparency. Our key finding is that sparsity and width are essential for inter- pretability, By structuring MoE as wide and sparse MLP layers, we show that it naturally enhances interpretability. To further improve this, we use ReLU-based experts and sparsity- routing to reduce polysemanticity and create
8

---

Mixture of Experts Made Intrinsically Interpretable
Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
orgi cating 1Wgpe
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield- Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. A math- ematical framework for transformer circuits. Trans- former Circuits Thread, 2021. https:/ttransfommrr
Expert LG [Lower Sparse)
Expari LD Looer is Sparser)
302
Top: Experts
252
213.4
41 205.0
166.6
153
103
Original Gating (W,x) Ours (-orf) Me 1)) 12n.
Ground-Truth
Elhage, N., Hume, T., Olsson, C.,, Nanda, N., Henighan, T., Johnston, S., EIShowk, S., Joseph, N., DasSarma, N., Mann, B., Hernandez, D., Askell, A., Ndousse, K., Jones, A., Drain, D., Chen, A., Bai, Y., Ganguli, D., Lovitt, L., Raddield-Dodds Z., Kermion, J., Conerly, T., Kravec, S., Fort, S., Kadavath, S., Jacobson, J., Tran- Johnson, E., Kaplan, J., Clark, J., Brown, T., McCan- dlish, S., Amodei. D., and Olah, C. Softmax linear units. Transformer Circuits Thread, 2022a. https//trrassormmrr cireuisspub/2222//oo//naddd
Figure 9. Comparison between TopK gating and our Sparsity rout- ing. Our score identifies a more sparse set of experts.
sparser internal representations. Experiments on chess and language tasks demonstrate that MoE-X performs on par with dense Transformers while providing more interpretable outputs.
Impact Statement
This paper introduces MoE-X, scalable and interpretable language model designed promote trust and reliability in AI systems. By improving transparency with sparse activations and efficient routing, MoB-X helps make AI decisions easier to understand. It can be especially useful in fields like healthcare and education, where trust is critical. While there is some risk of misuse or bias, these can addressed through careful and ethical use. Overall, this work aims to advance AI by providing more transparent and reliable approach to large-scale models.
Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Saufield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S. Kaplan, J., Amodei, D., Wattenberg. M. and Olah, C. Toy models of superposition. Trans- former Circuits Thread, 2022b. https:/transformer cireuits.pub/2022/og model/index.html.
Elhage, N., Olsson, C., Nanda, N., and Others. A mathematical framework for transformer circuits. https: //::://wwww http://wwwwttttttttt////tt//a.st foran-former-d reuits.puia 2022 htt.mmechhaaee 1e1o2m/mchinntep--oe
Acknowledgements
2022c.
Fedus, W., Zoph. B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
This work is supported by the UKRI grant: Turing Al Fel- lowship EP/W002981/1.
References
Gao, L., Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scal- ing and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04mm 2024.
Agarwal, R., Melnick, L., Frosst, N., Zhang, X., Lengerich, B., Caruana, R., and Hinton, G. E. Neural additive models: Interpretable machine learning with neural nets. Advances in neural information processing systems, 34: 4699-4711, 2021.
Gokaslan, A.. Cohen, V., Pavlick, E., and Tellex. S. Open- webtext corpus. http://Skyl 1ooyy111oo00...m 1on007.github. gith.b. io to/Openteebeextoorpuss 2019.
Awasthi, P., Dikkala, N., Kamath, P., and Meka, R. Learning neural networks with sparse activations. In The Thirty Seventh Annual Conference on Learning Theory, pp. 406- 425. PMLR, 2024.
He, X. O. Mixture of million experts. arXiv preprint arXiv:2407.0411 2024.
Brown, T,, Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
He, Z., Ge, X., Tang, Q., Sun, T., Cheng, Q., and Qiu, X. Dietionary learning improves patch-free circuit discovery

---

Mixture of Experts Made Intrinsically Interpretable
in mechanistic interpretability: A case study on othello- gpt. arXiv preprint (0K.i:::202.2220000 2024.
Langley, Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Hendrycks, D., Mazeika, M., and Woodside, T. An overview of catastrophic ai risks. arXiv preprint arXiv:2306.12000 2023.
Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L.,, Sonnerat, N., Varma, V., Kramár, J., Dragan, A., Shah, R., and Nanda, N. Gemma scope: Open sparse autoen- coders everywhere all at once on gemma arXiv preprint arXiv:2408.05:1 2024.
Hewitt, J., Thickstun, J., Manning, C., and Liang, Back- pack language models. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 9103-9125, Toronto, Canada, July 2023, Association for Compu- tational Linguistics. doi: 10.186653w12223..4--onng 506. URL https: http://ccaannoooo..oogg/// acl 1all-long onn..066/
Liu, Z., Gan, E., and Tegmark, M. Seeing is believing: Brain-inspired modular training for mechanistic inter- pretability, Entropy, 3611) 2023a.
Liu, Z., Khona, M., Fiete, I. R., and Tegmark, M. Grow- ing brains: Co-emergence of anatomical and functional modularity in recurrent neural networks. arXiv preprint arXiv:2310.07711, 2023b.
Huben, R., Cunningham, H., Smith, L. R., Ewart, A., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2023.
Liu, Z. L., Dettmers, T., Lin, X. V., Stoyanov, V., and Li, X. Towards unified view of sparse feed-forward net- work in pretraining large language model. arXiv preprint arXiv:2305. 13999, 2023c.
Jacobs, R. A., Jordan, M. I,, Nowlan, S. J., and Hinton, G. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Makhzani, A. and Frey, B. K-sparse autoencoders. arXiv preprint arXiv:1312.5665 2013.
Jermyn, A. S., Schiefer, N., and Hubinger, E. Engineer- ing monosemanticity in toy models. arXiv preprint arXiv:2211.09169, 2022.
McGrath, T., Kapishnikov, A., Tomašev, N., Pearce, A., Wattenberg, M., Hassabis, D., Kim, B., Paquet, U., and Kramnik, V. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119 (47)xe220662519, 2022.
Jiang, A. Q.. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint ar wxiv:2401.000088 2024.
Karvonen, A.
Emergent world models and latent vari- able estimation in chess-playing language models. arXiv preprint arXiv:2403.15493 2024.
Menon, A., Shrivastava, M., Krueger, D., and Lubana, E. Analyzing (in) abilities of saes via formal languages. arXiv preprint arXiv:2410.11767 2024.
Karvonen, A., Wright, B., Rager, C., Angell, R., Brinkmann, J., Smith, L. R., Verdun, C. M., Bau, D., and Marks, S. Measuring progress dictionary learning for lan- guage model interpretability with board game models. In The Thirty-eighth Annual Conference on Neural In- formation Processing Systems, 2024. URL https: //openrevies eww..w.ww....................... net net/ffrum forum? wwwpammmd=ssccdddocc d=dSdddoGg...........o
Merity, S., Xiong, C., Bradbury, J., and Socher, Pointer sentinel mixture models, 2016.
Mirzadeh, I., Alizadeh, K., Mehta, S., Del Mundo, C. C.,
Tuzel, O., Samei, G., Rastegari, M., and Farajtabar, M. Relu strikes back: Exploiting activation sparsity in large language models. arXiv preprint arXiv:2310.04564. 2023.
Komatsuzaki, A., Puigcerver, J., Lee-Thorp J., Ruiz, C. R., Mustafa, B., Ainslic, J., Tay. Y., Dehghani, M., and Houlsby, N. Sparse upcycling: Training mixture- of-experts from dense checkpoints. arXiv preprint arXiv:2212.05055. 2022.
Ngo, R., Chan, L., and Mindermann, S. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.000 00626,, 2022.
Olah, C. Mechanistic interpretability, variables, and the importance of interpretable bases, 2022. URL
Lample, G.. Sablayrolles. A., Ranzato, M., Denoyer, L., and Jégou, H. Large memory layers with product keys. Advances in Neural Information Processing Systems, 32, 2019.
2022 18.
Accessed: 2025-01-
10

---

Mixture of Experts Made Intrinsically Interpretable
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. Zoom in: An introduction to circuits. Distill, 5(3):00024-001, 2020.
Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, Interpretability in the wild: cireuit for indirect object identification in gpt-2 small. arXiv preprint wrXi:2221100092 2022.
Oldfield, J., Georgopoulos, M., Chrysos, G. G., Tzelepis, C., Panagakis, Y., Nicolaou, M. A., Deng, J., and Pa- tras, I. Multilinear mixture of experts: Scalable ex- pert specialization through factorization. arXiv preprint orXiv:2402.125500 2024.
Waswani, A., Shazeer, N., Parmar, N., Uszkoreit, Jones, L.,, Gomez, A., Kaiser, I.., and Polosukhin, I. Attention all you need. In NIPS, 2017.
Weber, M., Fu, D., Anthony, Q. Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., al. Redpajama: open dataset for training large lan- guage models. arXiv preprint arXiv:2411.12372 2024.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint wrXvv:/606.0003 06031, 2016.
Xie, S., Vosoughi, S., and Hassanpour, Im: A prototypical network-based framework for built- in interpretability in large language models. In Bouamor, H.,, Pino, J., and Bali, K. (eds.), Find- ings of the Association for Computational Linguis- tics: EMNLP 2023, pp. 3964-3979, Singapore, De- cember 2023. Association for Computational Lin- finn3..iinn/. 261. URL https: /ttp///aaaatahheee/ee/ee f Lindings-emnly 261/.
Proto-
Park, J., Ahn, Y. J., Kim, K.-E., and Kang, Monet: Mix- ture of monosemantic experts for transformers. arXiv preprint arXiv:2412.0411 2024.
Paulo, G., Mallen, A., Juang, C., and Belrose, N. Automati- cally interpreting millions of features large language models. arXiv preprint arXiv:2410.13928, 2024.
guistics. doi:
org/2023.
Pearce, M. T., Dooms, T., Rigg, A., Oramas, M., and Sharkey, L. Bilinear mips enable weight-based mecha- nistic interpretability, arXiv preprint arXiv:2410.08417, 2024.
Zhang, Z., Song, Y., Yu, G., Han, X., Lin, Y., Xiao, C., Song, C., Liu, Z., Mi, Z., and Sun, M. Relu 2 wins: Discovering efficient activation functions for sparse llms. arXiv preprint arXiv:2402.00 038004 2024.
Penedo, G., Kydlíček, H., allal, Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirtt-eight Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track, 2024. URL https:://///....... http:://pppeeiieeeeeeee n forum? id-n6SCkn2QaG.
Zhong, Z., Liu, 2., Tegmark, M., and Andreas, J. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems, 36, 2024.
Zhou, Y., Lei, T., Liu, H., Du, N., Huang, Y., Zhao, V., Dai, A. M., Le, Q. V., Laudon, J., et al. Mixture-of- experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7 11710077114, 2022.
Radford, A.,
Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Sharkey, L. A technical note on bilinear layers for inter- pretability. arXiv preprint arXiv:2305.0344 2023.
Shazeer, N. Glu variants improve transformer. arXiv preprint arXiv:2002.05202. 2020.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. Hinton, G., and Dean, Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 011117011.65533 2017.
Toshniwal, S., Wiseman, S., Livescu, K., and Gimpel, K. Chess as testbed for language model state tracking. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pp. 11385-11393, 2022.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008

---

Mixture of Experts Made Intrinsically Interpretable
In the appendix, provide additional details to complement our paper. Section explores how interpretability scores evolve during training and across different model layers. Section B defines the metrics used to assess model interpretability. Section D describes the auto-interpretabiiity experiment setup and presents newly identified interpretable features in the MoE-X small model. Section E provides full derivation of how Sparse MoE can be formulated as an MLP layer and our sparse-aware gating. Finally, Section F details the model training configurations.
A. Interpretability Dynamics
We evaluate types of interpretability dynamics in a language model trained chess. First, study how interpretability evolves over the number of training steps. Second, we examine how interpretability varies across different layers of the language model. For our experiments, we use 8-layer GPT-style model with configuration (ax 4, d = 512).
Coverage of Chess Board State
0.46
0.22
0.44
0.20
0.42
Loss
0.38
0.16
0.36
0.34
0.14
0.32
0.12
100000
200000 300000
400000
500000
600000
Training Iterations
Figure 10. Dynamies of BSP Coverage Score and Validation Loss over Training Steps.
Training Iterations & Interpretability. As shown in Figure 10, the interpretability coverage score generally increases as training progresses. However, the trend does not exactly mirror the validation loss. Even when the validation loss plateaus, the coverage score continues to increase, indicating ongoing improvements in interpretability.
This observation motivates decision upcycle model weights from dense model, following (Komatsuzaki al., 2022). We find that longer training leads higher interpretability scores. However, in MoE models, training is inherently less efficient per expert. Given total of T iterations, each expert in a MoE model is only activated and trained for Tk/M iterations, where k is the number of selected experts per step. As result, each expert is effectively under-trained compared to dense model, making direct interpretability comparisons unfair.
To validate this and assess the impact weight upcycling, we conduct an experiment comparing different training strategies. Specifically, we train MoE both from scratch and with upcycled dense weights while also continuing the training of dense model for the same number of iterations.
The results, shown in Table 4, indicate that upcycling significantly improves interpretability. The MoE trained from scratch achieves higher coverage score than the dense model, but its reconstruction performance lags behind. In contrast, the upcycled MoE not only outperforms the dense models in interpretability but also shows the best reconstruction score, demonstrating the benefits of leveraging pre-trained dense weights.
Method
Coverage
Reconstruction
Dense
0.356
0.608
Dense (Continued Training)
0.377
0.674
MoE-X (Seratch)
0.398
0.657
MoE-X (Up-cycle)
0.428
0.840
Table 4. Comparison of interpretability scores for different training methods.
12

---

Mixture of Experts Made Intrinsically Interpretable
Chess Board Reconstruction F1 Score
MLP Activations
0.2
0.1
0.0
Layer Number
Figure 11. BSP Reconstruction Score at different language model layers.
Layer number & Interpretability. In Figure 11, we show the BSP Reconstruction Score across different layers of the language model. The interpretability score increases initially, peaks at layer and then decreases. Based on this observation, we evaluate layer in our 8-layer transformer. Similarly, we select moderate layers (e.g., layer 8 for 12-layer model and layer 16 for 24-layer model) for other model depths in natural language experiments.
B. Metrics Definitions
B.1. Board State Properties in Chess
We define board state property (BSP) as a function [game board] - (0, 1}, which evaluates specific characteristics of a board state. In this work, we focus on interpretable classes of BSPs that capture fundamental game properties.
One such class, Goaard states includes BSPs that determine whether specific piece is present a given board square. Chess an 8 x board. In chess, consider the full board for all twelve distinct piece types (e.g., white king, white queen, black king), resulting in total of 8 x 12 BSPs.
B.2. Coverage
The coverage metric evaluates how well the features learned by Sparse Autoencoder (SAE) align with a given set of Board State Properties (BSPs). Let G be collection of BSPs, and let {fr) denote the set of features learned by the SAE. For each feature fi, we define binary classifier Φ based on a threshold E [0. 1]:
(),((x)= - [[f((x)) > J!
where:
fa(x) is the activation feature for input.or
mmax max ((() maximum activation of f; over the dataset D,
[[-] is the indicator function, which outputs 1 the condition is true and 0 otherwise.
For given BSP y G, the F1-score of dy., as classifier for g denoted by F Fi( g). The coverage of the SAE with respect to G is then defined as:
Cov([f,),G)
max max (((
G
f.
IEG
13

---

Mixture of Experts Made Intrinsically Interpretable
In words, for each BSP g. select the feature and threshold | that maximize the F1-score for classifying g. The coverage score is the average of these maximal F1-scores across all BSPs in G. coverage score of indicates that the SAE has least one feature that perfectly classifies every BSP in G.
B.3. Board Reconstruction
The board reconstruction metric measures the ability of an SAE to recover the complete state of chessboard from its feature activations in a numan-interpretabll way. Let G' be set of BSPs, and let {fi} denote the set of SAE features. For each feature fi we identify the subset of BSPs E G' for which f,, is a nigh-precisson classifier (precision 2 0.95) on training dataset Drrain-
For given activation z, the predicted state of a BSP g E G is determined by the rule:
11, if (),((x) for any /; that is high-precision for on Dirain 0, otherwise.
Po((((())))
The full predicted board state is represented P(([(z))) (Pa(C)(())) which contains predictions for all 64 squares of the chessboard. The quality of the reconstruction is evaluated using the Fl-score of the predicted board state P({f((z))) compared to the true board state b, denoted as (((((((()))):b)
The board reconstruction score is then computed as the average F1-score over all board states in a test dataset Drest
Rec((/s) Drea) =
Σ A(P(U(()))))))))) xEDi
Drest
where (xx) the true board state corresponding to activation æ. This metric reflects how well the SAE's feature activations can be combined reconstruct the full board state, emphasizing interpretability and precision.
C. Routing Regularizes Sparsity
Our sparsity-aware gating significantly reduces expert sparsity. In a 2-out-of-8 MoE setup with top-k gating and ReLU experts, the in norm of the experts is approximately 313. With our sparsity-aware gating, this value decreases to around 166. This demonstrates that sparsity-aware gating greatly enforces sparsity.
D. Auto-Interpretablliy
We conduct an auto-interpretability experiment following the approach described in (Paulo et al., 2024).
For MLP layer, We collected latent activations from the MLP over 10M token sample of caoPajama-v22(Webbe et al., (BOS) token.
2024). The activations are gathered from batches of 256 tokens, each starting with a
To interpret these activations, we use Llama 3.1 70B Instruct as the explainer model. is presented with 20 activating examples, each consisting of 32 tokens, where the activating tokens can appear at any position. These examples are randomly selected from larger dataset to ensure diversity activation patterns.
For evaluation, we use the detection score, where scorer model identifies which sequences activate given latent based on an interpretation. In this setup, the model is shown five examples at time, each with an independent probability of activating the latent, regardless of the others. Each latent is evaluated using 100 activating and 100 non-activating examples. The activating examples are selected through stratified sampling, ensuring that 10 examples are drawn from each the 10 deciles of the activation distribution.
Example. Besides the sample showed in the main paper, show more results in the Table 5
14

---

Mixture of Experts Made Intrinsically Interpretable
Auto-Interp Meaning
Location
Example
Time of day in expressions
Expert 2, #457
"We went for a walk in the evening." " "The meeting scheduled for aftermoon."
"She always exercises in the morning." "She explained the concept using e.g. as example."
Abbreviations with dots
Expert 5, #89
"You must submit all forms by Friday, i.e.,
tomorrow."
"Common abbreviations include a.m. and p.m. for time."
Capitals the start of acronyms
Expert 6, #1601
"The NASA mission successful."
"The company developed cutting-edge AI systems."
"Students use PDF documents for submis- sions."
Ordinal numbers in sentences Hyphenated compound words Currency symbols preceding numbers Parentheses around numbers or letters
Expert 3, #412
"He finished place."
Expert #187
"This is well-being initiative."
Expert #273
"The total cost $100."
Expert 6, #91
"Refer to section (a) for details."
Ellipsis usage
Expert 0,, #55
"He paused and said, I'll think about it."
Measurements followed by units Dates in numeric formats Repeated punctuation marks Hashtags in text Uppercase words for emphasis Colon in timestamps
Expert 0, #384
"The box weighs 5 kg.
Expert 7, #401
"The deadline is 2025-01-29."
Expert 128
"What happening ???"
Expert 4, #340
"Follow the trend at #trending."
Expert #278
"The sign read, STOP immediately!"
Expert 3, #521
"The train arrives at 12:30." "I can't do this alone."
Contractions with apostrophes
Expert 6, #189
Table Sampled Activated Tokens and Contexts for Neurons MoE-X Small. The meanings are identified by the Auto-interp process.
E. Derivation
E.1. MoE Layer Sparse MLP
In this section, we demonstrate how a Mixture-of-Experts (MoE) layer can be reformulated as a large and sparse Multi-Layer Perceptron (MLP).
The output the MoE layer is expressed as weighted sum of the expert outputs:
M
y
M
Wi
(w(1) dec
(W(())
(8)
j=1
wi(i) dec
wirr
(9)
M
(i dec
w1,(1ر)
(10)
where W/ is the gating weight for the j-th expert, and (0) = o (
x)) is the hidden representation after the activation function a in the j-th expert. Since wy scalar, can be factored out before multiplication with (i) des
15

---

Mixture of Experts Made Intrinsically Interpretable
To simplify this representation, define "mega-decoder" by concatenating all expert decoder matrices:
Wdec dec concat ([ ) E MDdx
Similarly, concatenate the scaled hidden representations of all experts:
2 concat([w]2" Wwz (M0 E RAD.D
With these definitions, the MoE output can be reformulated as:
y d.
This reformulation demonstrates that MoE layer is equivalent to wide and sparse MLP, where sparsity is induced by the selective activation of only subset of experts for given input.
Interestingly, a similar derivation is mentioned in (Liu et al., 2023c). However, their work focuses on building efficient and sparse neural networks, while ours emphasizes interpretability.
E.2. Swarsity-Aware Gating
W1,1
Wi,d
I1
Suppose we have E experts, each accosiated W.n. enc
R Dxd and input x
E Rd..
The
WDD1
WD.d
Id
hidden activation is
Σ, W1it
21
2
W eneX
(1I)
Σ, WD
2D
we assume that each rows of W ene is i.dd from a Gaussian distribution, {wg,)s=1 ~ NN of), Then we can see each element of from a mixture of gaussian distribution
2j Σ wryc ~ (2) = =N(> M:I; Σ of
(12)
If apply ReLU function on top of this hidden, the probability that 2, is positive is P(z₁ > 0). P(z, 0) directly corresponds to the sparsity of ReI.(zz) Given the Gaussian assumption,
Uz 02
= + Pz =
1
x²
P(2 > 0) 4
exp
du
(13)
2m
2
Where ((2) CDF $ is
P(Z < 2) the CDF of the normal distribution. In practice, a common closed-for form approximation for the
((2) ~ _ erf( (2f(2222))
(14)
The larger the sparsity, the less non-zero values, and the P(zj > 0) gets smaller. Because we want to select the expert with the largest sparsity
Therefore, we select the experts with the smallest P(2j > 0)
arg max
SSprrsiy (Worc)
arg mrg [(ss) 0)]
(15)
In terms of the Gaussian CDF
M2
1 (1 erf( Mz H=²) =arg arg max - erf( #1z
arg min +
arg max
2
/20=
/20=
(16)
We this as the gating function for mixture-of-expert
w =[(x; = Coftmax((Topk (-erf( uz -)))
(17)
/20g
16

---

Mixture of Experts Made Intrinsically Interpretable
F. Training Details
The training configuration and hyperparameters are presented Table and Table
Table 6. MoF GPT-2 Training Configuration for Chess Dataset.
Parameter
Value
Num layer
8
Num head
Num embd
512
dropout
0.0
Init learning rate
3e-4
Min Ir
3e-5
Lr warmup iters
2000
Max iters
600000
optimizer
Adam W
batch size
100
context len
1023
Num experts
Num experts per Token
2
grad_clip
1.0
Table 7. MoE & GPT-2 Small Training Configuration for FineWeb Language Tasks.
Names
Small
Medium
Num layer
12
24
Num head
12
16
Num embd
768
1024
dropout
0.0
0.0
Init learning rate
3e-4
3e-4
Min Ir
3e-5
3e-5
Lr warmup iters
5000
5000
Max iters
100000
100000
optimizer
AdamW
Adam W
batch size
320
320
context len
1024
1024
Num experts
8
Num experts per Token
2
grad_clip
1.0
1.0
Load Balance Loss. To ensure a balanced distribution of tokens across experts, we use an auxiliary loss borrow from (Fedus et al., 2022). This auxiliary loss is added to the total model loss during training.
Given experts indexed by to N and a batch B containing tokens, the auxiliary loss is defined as the scaled dot product between the token distribution vector and the router probability vector P:
N
Lbalance = a N
P
(18)
where f, represents the fraction of tokens assigned to expert
17

---

Mixture of Experts Made Intrinsically Interpretable
fi
Σ
Ifarg maxpmax p(x) = i}
(19)
T
#EB
and P. denotes the fraction the router probability allocated to expert e:
P.
Σ p=(a)) #EB
(20)
T
Since we aim for uniform token routing across all N experts, both f and P should ideally have values close to 1/N, For all MoE model training in this paper, we set the load balancing weight to À = 0.001.
18


Reviews:
[{"review_id": "3fmnxF0ZyQ", "reviewer": "Reviewer_a7ZR", "summary": "This paper proposes a variant of an MoE architecture called MoE-X that makes design decisions that boosts the mechanistic interpretability of the model, while largely preserving quality. The authors authors motivate this with a preliminary study on the importance of MLP hidden size and sparsity of hidden activations for interpretability, using Chess board state prediction as an example. They then draw connections to why MoEs are essentially a special case of a large, sparse MLPs. They then design MoE-X which extends the MoE with various design decisions to boost interpretability: notably, the use of ReLU activation, and a novel sparsity-aware routing scheme. They show that on chess, MoE-X is more interpretable, than dense, MoE, and post-hoc interpretability baselines (SAE). The same holds true when pretraining the model and baselines over natural language (FineWeb): MoE-X is more interpretable than GPT-2 quantitatively, while achieving similar perplexity evals to normal MoE models. Qualitatively MoE-X activations mirror that typically obtained via more involved automated interpretability detection methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate R-a7ZR's thoughtful comments and suggestions. \n\n`>>> Q1`**Evaluation Setup and Additional Experiments**\n\n`>>> A1`We truly appreciate the suggestion! As R-a7ZR mentioned, our primary focus is on **interpretable model design**. In line with this goal, we use small-scale datasets, guided by 3 key factors:\n1. **Comparison with Prior Work**: We follow the same evaluation protocol used in prior studies (SAE on GPT-2 Small) for fair comparison. \n2.  **Interpretability Evaluation**: We need a benchmark that evaluates both interpretability and performance. Small, controlled dataset like chess is good for this.\n3. **Computational Constraints**: Our resources are limited to an 8×48G GPU server, which supports prototyping but not large-scale experiments.\n\n**Large-scale Evaluations**\n\nWhile large-scale training isn't feasible within the rebuttal period, we follow the reviewer’s suggestion and evaluate our trained checkpoints on additional standard few-shot benchmarks. The results are shown in the table below.\n\nThough preliminary (due to the model’s small size), our MoE-X model performs comparably to the Switch Transformer. We plan to scale up the model in future work.\n\n|Metric|GPT-Neo 125M|Switch-S (8×124M)|MoE-X-S (8×124M)|\n|-|-|-|-|\n|Avg.|25.8|**29.3**|28.6|\n|ARC (25-shot)|22.9|27.3|**28.3**|\n|HellaSwag (10-shot)|30.3|**34.1**|33.2|\n|MMLU (5-shot)|26.0|**30.4**|29.3|\n|TruthfulQA (0-shot)|45.6|**52.3**|49.5|\n|Winogrande (5-shot)|51.8|**54.6**|54.1|\n|GSM8K (5-shot)|0.3|**1.3**|1.2|\n|DROP (3-shot)|3.7|5.1|**6.3**|\n\n`>>> Q2`**Fair Comparison between GPT-2 and MoE-X**\n\n`>>> A2`Thank R-a7ZR for this thoughtful suggestion! We completely agree that a parameter-matched comparison is critical for fairness. \n\nAs suggested, we re-train MoE-X Small with 1 activated expert (124M active) to match GPT-2 Small’s size. To speed up training, we fine-tune the old top-2 MoE-X, but activating only 1 expert per token. The training took ~8 hours. This top-1 MoE-X (124M) has the same active parameters as GPT-2 (124M).\n\n**Apples-to-Apples Comparison**:\nWe report performance using perplexity. As shown below, MoE-X-S (1 expert active) outperforms GPT-2 Small across all datasets while using ​**identical active parameters**. But as expected not as good as the MoE-X-S (2 from 8×124M) with around 50% more parameters.\n|Model|OpenWeb (PPL)↓|LAMBADA (PPL)↓|WikiText103 (PPL)↓|WikiText2 (PPL)↓|\n|-|-|-|-|-|\n|GPT-2 Small|22.83|32.71|49.89|44.36|\n|MoE-X-S (1 activated)|21.36|30.92|47.17|44.07|\n|MoE-X-S (2 activated)|**19.42**|**28.11**|**43.80**|**42.58**|\n\n​**Interpretability Gains Persist**:\nWe evaluate interpretability using the *Detection Score* defined in the paper. Still, with the same activated parameters, MoE-X-S (1 from 8×124M) demonstrates a clear improvement in identifying more accurate concepts compared to GPT2-Small.\n|Quantiles/Accuracy|GPT2-Small|MoE-X-S (1 activated)|\n|-|-|-|\n|Not|**0.88**|0.84|\n|Q1|0.223|**0.265**|\n|Q2|0.233|**0.281**|\n|Q3|0.249|**0.301**|\n|Q4|0.230|**0.302**|\n|Q5|0.265|**0.342**|\n|Q6|0.311|**0.357**|\n|Q7|0.302|**0.369**|\n|Q8|0.315|**0.379**|\n|Q9|0.321|**0.388**|\n|Q10|0.354|**0.403**|\n\nThese results highlight that MoE-X’s advantages stem not just from scale but from its *structured sparse design*. We deeply appreciate the suggestion and will include in the revised paper!\n\n`>>> Q3`**Switch Transformer in Figure 8**\n\n`>>> A3`Great suggestion! As suggested, we incorperate Switch Transformer into Figure 8. We re-run the auto-interp experiment for Switch-S and compare with MoE-X-S using the average *Detection Score*. \n\nWhile both models perform well, MoE-X-S consistently outperforms the standard MoE in interpretability across all quantiles.\n\n| Quantiles | Switch-S | MoE-X-S |\n|-|-|-|\n|Not|0.84|0.82|\n|Q1|0.305|**0.320**|\n|Q2|0.321|**0.331**|\n|Q3|0.330|**0.339**|\n|Q4|0.318|**0.330**|\n|Q5|0.356|**0.379**|\n|Q6|0.378|**0.396**|\n|Q7|0.436|**0.448**|\n|Q8|0.448|**0.461**|\n|Q9|0.515|**0.531**|\n|Q10|0.685|**0.699**|\n\n\n\n`>>> Q4`**SAE applied to MoEs**\n\n`>>> A4`That’s really an interesting direction! SAE can indeed be applied to MoEs, including MoE-X.\n\nDue to time constraints, we conduct a quick experiment by training SAE on MoE-X and Switch Transformer using the Chess dataset for post-training interpretability. The SAE is placed at `post-res` layer 6 with a hidden size of 4096.\n\n|Model|Val Loss|Coverage|Reconstruction|\n|-|-|-|-|\nSwitch |0.212| 0.424| 0.734| \nSwitch + SAE|0.222|0.430|0.824\nMoE-X|**0.211**|0.428|0.840|\nMoE-X + SAE|0.220| **0.433**|**0.857**|\n\nThree key observations:\n1. SAE improves interpretability for MoE but increases validation loss.\n2. Switch + SAE slightly outperforms MoE-X alone. But MoE-X + SAE fights back.\n3. Improvements are modest due to SAE's hidden size matching MoE's channel width.\n\n`>>> Q5`**Typos and formating**\n\n`>>> A5`We sincerely thank the reviewer for the careful proofreading! We will correct the typos in the references and ensure proper formatting for terminology in the revision.", "claims_and_evidence": "The claims in this work are sound and are supported by clear and convincing evidence. The experimental settings certainly have limitations, but the authors present the results clearly and straightforwardly, and do not overclaim. For the most part, both the qualitative examples, and quantitative results provide convincing evidence that MoE-X exhibits improved interpretability over baselines while maintaining quality.", "methods_and_evaluation_criteria": "Evaluation criteria and methodology in this work is on the smaller scale end, focusing on a toy example (chess) and small scale language modeling, with evaluation only on perplexity. \n\nIf the intention of this work is to spark interest in the field of interpretable model design, this may be sufficient. However, most architecture proposals typically strive to validate that the quality of the model is either neutral or better across a wide array of settings, in order to prove to the reader that the architecture is worth adopting. This paper would be significantly stronger if it showed that on common pretraining evaluations, such as few-shot learning, the proposed architecture did not regress over an identical MoE parameterization without these modifications. It is difficult to judge how realistically this model could be used via Table 1 and 2 in this work, which are quite limited evaluation of LM performance.", "theoretical_claims": "The paper doesn't make any proofs or theoretical claims. I am familiar with the theoretical connection between wide MLPs and MoEs presented in the motivation of the work, and their description is sound.", "experimental_designs_or_analyses": "Apart from the limited evaluations noted above, my primary concern with this paper lies in the comparison between GPT-2 and MoE-X in this paper. While MoE-X and Switch are parameterized identically, and thus are comparable, the comparison against GPT-2 is a bit unfair as the small and medium parameterizations for MoE is slightly larger in terms of activated params / FLOPs than the dense baseline. Although in absolute terms the difference is not that large, a small scales like those studied in this paper 124M vs 180M or 354M vs 555M is a meaningful relative difference of ~50%. This could contribute to both the quality gains and the increased interpretability (extra MLP params) when compared against the Dense baseline.\n\nIt seems then the true baseline for MoE-X is more similar to Switch Transformer. On chess this comparison is shown well, but Figure 8 does not show this comparison w.r.t. interpretability for Switch Transformer. This somewhat calls into question the validity of some of the comparisons made in the work. In general, the comparison to SAE could be applied more rigorously -- how does this method compare to SAE as applied to MoEs?", "relation_to_broader_scientific_literature": "This paper makes an interesting contribution to the field as it explores how to make an existing state-of-the-art architecture (token-routed MoEs) more interpretable while maintaining its quality. This is in opposition to post-hoc interpretability techniques over such models, which might be suboptimal. Figure 5 provides compelling evidence this is the case. This is a compelling direction as if a modeling proposal is neutral but improves interpretability, then it could be adopted almost for free. This work's novel sparsity-aware routing seems to be a meaningful step towards this goal.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- The design of sparsity-aware routing for interpretability is novel, interesting, and appears effective. It honestly could be worth a rigorous study / paper on its own. \n- It's positioning / contribution to the field is meaningful and valuable as noted above.\n\nWeaknesses:\n- Presentation generally needs to be polished.", "other_comments_or_suggestions": "Paper contains multiple typos, such as in the intro \"Waswani et al.\" instead of \"Vaswani\", and in the section \"Study II: Activation Sparsity\". Stylization of certain methods could be improved, such as \"AdamW\" instead of \"Adamw\".", "questions_for_authors": "Do you have a Sparse Transformer baseline for Figure 8? How does it compare against MoE-X?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lZHgM8tVA0", "reviewer": "Reviewer_K7C2", "summary": "The paper introduces MoE-X, a Mixture-of-Experts (MoE) language model designed to be intrinsically interpretable. This is different from the recent trend of using Sparse Autoencoders to interpret the model representations at post-doc. The proposed method addresses the challenge of polysemanticity in large language models (LLMs) representations, where individual neurons encode multiple unrelated concepts, making post-hoc interpretability difficult. The authors propose a novel architecture by leveraging MoE’s sparsity and width to encourage disentangled representations. Extensive experiments on both chess and language tasks prove the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly thank the R-K7C2 for the nice comments.\n\n`>>> Q1`**Additional Application Scenarios**\n\n`>>> A1`We sincerely appreciate the suggestion! Expanding to the medical domain is valuable, but a key challenge is finding a benchmark that **evaluates both interpretability and performance**. While many medical datasets focus on performance, few assess interpretability, highlighting the need for a dedicated benchmark.\n\nAs a first step, we tested the small MoE-X model (8×124M) on MMLU’s medical subset using 5-shot prompting, comparing it to a similarly sized dense model (GPT-Neo 125M).\n\n| Metric                 | GPT-Neo 125M |  MoE-X-S (8×124M) |\n|------------------------|--------------|------------------|\n| MMLU (5-shot)         | 26.0        | **29.3**            |\n\nWhile MoE-X-S performs good, we’d need more research to evaluate how interpretable it is in specialized domains.\n\n`>>> Q2`**Human Evaluation**\n\n`>>> A2`Thanks for the suggestion! We conducted a blinded human evaluation to assess neuron monosemanticity in MoE-X compared to baseline models (GPT-2 and GPT-2 with SAE).\n\nFollowing GatedSAE’s approach, 5 raters evaluated 20 random features from each model, presented in random order. Each rater evaluated 12 features (20 × 3 / 5). For each feature, raters were shown the top 5 activating samples. Raters classified each feature as: `Yes` (clearly monosemantic), `Maybe` (partially interpretable), `No` (not monosemantic).\n\n| Model | Yes | Maybe | No|\n|--|--|--|--|\n|GPT-2|8|4|8|\n|GPT-2 + SAE| 12 |3|5|\n|MoE-X | 14|3|3\n\nMoE-X’s features were rated as ​most interpretable, with 70% (_14/20_) labeled `Yes`. While this is a small-scale study, it provides a promising preliminary insight.\n\n`>>> Q3`**Validate the Gaussian assumption**\n\n`>>> A3`Very luckily, a recent arXiv paper [A] has conducted a formal study on investigates the distribution of weights in LLMs. The authors find that the **weights of all major open-source LLMs closely follow a Gaussian distribution**, including LLaMA, Qwen, and the Vicuna family. They provide both statistical evidence and theoretical explanations for this phenomenon. We will cite this work in the revised version!\n\n[A] Unveiling the Mystery of Weight in Large Foundation Models: Gaussian Distribution Never Fades https://arxiv.org/abs/2501.10661\n\n\n`>>> Q4`**MoE Feature Steering**\n\n`>>> A4`We truly appreciate the suggestion. \nWhile we cannot perform a full qualitative comparison within the rebuttal period, we provide an initial analysis by applying feature steering to MoE-X and include the results below.\n\n**Love − Hate Steering**  \n\nBuilding on the **Activation Addition** method [B], we demonstrate how feature steering can influence model completions. Using MoE-X-S, we first extract feature vectors for the contrasting concepts `\"Love\"` and `\"Hate\"`. \n\nThese vectors are then used to compute a steering adjustment\n `v_Steered = v_Unsteered + coefficient * (v_\"Love\" − v_\"Hate\")`. We apply the steering at **MoE hidden feature at layer 6** with a coefficient of **5**.  \n\nThe table below compares unmodified model completions (left) with steered ones (right). The bolded text represents the prompt.  \n\n| **Unsteered Completions** | **Steered Completions** |\n| :-- | :-- |\n| **I hate you because** you're spineless. know what I hate? People who believe the only way to accomplish something is to go through the motions without real effort. Those who hesitate to take meaningful action are... | **I hate you because** you're an incredible person. The reason I'm here is that I truly care about you. I don't want to be distant anymore. I want to embrace the present and move forward with hope... |  \n\nThis demonstrates how feature addition can also steer model behavior in our MoE-X.\n\n[B] Turner, Alexander Matt, et al. \"Activation addition: Steering language models without optimization.\" _arXiv e-prints_ (2023): arXiv-2308.\n\n`>>> Q5`**Formatting issue**\n\n`>>> A5`We sincerely apologize for the formatting and line number issues! We will make sure to correct them in the revision.\n\n`>>> Q6`**Impact of number of experts**\n\n`>>> A6`Thanks for the insightful question! As suggested, we explore two ways to increase the number of experts, and both show promising improvements:\n\n1.  **Activating More Experts (with a fixed total number)**: In fact, this has been shown in `Fig 5` in the paper, which we varying the number of activated experts\n$k \\in \\{1,2,4\\}$. It shows that increasing the number of activated experts leads to better interpretability.\n\n2) **Expanding Total Experts (keeping the number of activated experts constant)**: We increased the total experts from 8 to 16. This improved interpretability. However, the improvements were modest compared to increasing activated experts, likely because the per-inference cost remained the same.\n\n|Model |Val Loss&#8595;|Coverage&#8593;|Reconstruction&#8593;|\n|--|--|--|--|\nMoE-X (2 from 8, in paper) | 0.211 | 0.428| 0.840|\nMoE-X (2 from 16) | **0.207** | **0.442** | **0.853**|", "claims_and_evidence": "The major claims made by the paper are:\n1. MLP Hidden Size: Larger hidden states result in better interpretability.\n2. Sparsity of Hidden Activations: Lower numbers of nonzero neurons lead to more interpretable representations.\n\nOverall, the claims are well-supported by experiments.", "methods_and_evaluation_criteria": "The proposed evaluation methods are reasonable.\n1. Chess dataset: Used as a structured benchmark with board state properties as a ground truth, which is appropriate.\n2. Automated interpretability pipeline on natural language tasks: This is a standard method used in other related works.", "theoretical_claims": "The theoretical claims focus on how MoE can be reformulated as a sparse MLP and how routing can be modified to enforce sparsity. The derivations appear correct.", "experimental_designs_or_analyses": "As in *Methods and Evaluation Criteria,* the experimental setup is reasonable, and the results support the claims.", "relation_to_broader_scientific_literature": "Different from the recent trend of using Sparse Autoencoders to interpret the LLM representations at post-doc, the authors borrow ideas from MoE to achieve intrinsically interpretable LLM, which is novel.", "essential_references_not_discussed": "The paper covers the most relevant references.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to follow.\n2. The proposed method is novel and technically sound.\n\nWeaknesses:\n1. The paper would benefit from additional application scenarios, such as medical NLP benchmarks. And also human evaluations.\n2. It would be more rigorous to validate the Gaussian assumption in sparsity-aware routing via experiments.\n3. It would be more interesting to see if we can leverage the proposed method to steer the model behaviors or correct wrong/biased predictions.", "other_comments_or_suggestions": "Is the formatting a big issue? Since this paper does not use a template that has line numbers and indicating under review in ICML.", "questions_for_authors": "Minor: Will increasing the number of experts significantly impact interpretability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YRx37An3La", "reviewer": "Reviewer_tfxC", "summary": "The paper presents MoE-X, a novel Mixture-of-Experts (MoE) architecture designed to enhance the interpretability of large language models (LLMs) while maintaining competitive performance. The authors explore the challenge of polysemanticity in neurons and its relationship to the model's architecture. They address this through architectural modifications that promote sparsity and width in the network. Key contributions include redesigning the MoE layer as a wide, sparse MLP with ReLU experts and sparsity-aware routing. The paper demonstrates through experiments on chess and natural language tasks that MoE-X achieves performance comparable to dense models while significantly improving interpretability metrics.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer tfxC's suggestions. We have carefully incorporated them into the revised manuscript.\n\n`>>> Q1`**Related Work on MoE**\n\n`>>> A1`We truly appreciate R-tfxC for bringing the ReMoE [1] paper on sparse MoE to our attention. We will definitely include a citation in the revised version.\n\nWhile both work utilize the properties of ReLU in MoE, there are key differences:\n1. **Purpose of study**: ReMoE aims to make the routing function fully differentiable, while our focus is on enhancing the interpretability of the MoE.\n2. **Position where ReLU used**: In our work, ReLU is applied within each expert, whereas ReMoE uses it in the gating function.\n\n`>>> Q2`**Feature Streering Experiments**\n\n`>>> A2`We truly appreciate the suggestion. \nWhile we cannot perform a full qualitative comparison within the rebuttal period, we provide an initial analysis by applying feature steering to MoE-X and evaluating the results.\n\n**Love − Hate Steering**  \n\nBuilding on the **Activation Addition** method [A], we demonstrate how feature steering can influence model completions. Using MoE-X-S, we first extract feature vectors for the contrasting concepts `\"Love\"` and `\"Hate\"`. \n\nThese vectors are then used to compute a steering adjustment\n `v_Steered = v_Unsteered + coefficient * (v_\"Love\" − v_\"Hate\")`. We apply the steering at **MoE hidden feature at layer 6** with a coefficient of **5**.  \n\nThe table below compares unmodified model completions (left) with steered ones (right). The bolded text represents the prompt.  \n\n| **Unsteered Completions** | **Steered Completions** |\n| :-- | :-- |\n| **I hate you because** you're spineless. You know what I hate? People who believe the only way to accomplish something is to go through the motions without real effort. Those who hesitate to take meaningful action are... | **I hate you because** you're an incredible person. The reason I'm here is that I truly care about you. I don't want to be distant anymore. I want to embrace the present and move forward with hope... |  \n\nThis demonstrates how targeted feature modifications can also steer model behavior in our MoE-X.\n\n> [A] Turner, Alexander Matt, et al. \"Activation addition: Steering language models without optimization.\" _arXiv e-prints_ (2023): arXiv-2308.\n\n`>>> Q3`**Interpretability and Performance**\n\n`>>> A3`The review is absolutely right -- there is indeed a trade-off between interpretability and performance. If we enforce extreme sparsity (e.g., pushing the  $l_0$​  norm to zero), the model would have no activations and thus fail to learn.\n\nTo address this, we use  **ReLU-based sparsity**  rather than hard constraints like $l_1$​ regularization (as in SAEs). This allows the model to  *adaptively learn sparse patterns*  while still maintaining strong performance. As a result, our approach preserves interpretability without sacrificing the model's ability to fit the data effectively.", "claims_and_evidence": "The claims about improved interpretability and maintained performance are supported by experimental results on chess and language tasks. The authors provide evidence through quantitative metrics (perplexity, BSP coverage score) and qualitative analysis (t-SNE visualizations, auto-interpretability examples). The discussion of architectural factors influencing interpretability is well-supported by preliminary studies and ablation analyses. However, the claim that MoE-X completely eliminates polysemanticity might be overstated, as some level of polysemanticity is inherent in neural networks, especially in language models where words have multiple meanings in different contexts", "methods_and_evaluation_criteria": "The proposed methods (ReLU experts, sparsity-aware routing) and evaluation criteria (BSP coverage score, reconstruction score, detection accuracy) make sense for the problem of improving interpretability in language models. The chess dataset provides a clear ground truth for evaluating interpretability, and the natural language experiments use standard benchmarks.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental designs appear sound. The authors compare MoE-X against several baselines (dense models, other MoE variants) and use appropriate metrics for both performance and interpretability. The ablation studies help isolate the contributions of different components.", "relation_to_broader_scientific_literature": "The paper builds on prior work in mechanistic interpretability, sparse autoencoders, and MoE architectures. \n\nThe paper could benefit from citing recent work on differentiable MoE routing mechanisms, such as ReMoE[1], which also uses ReLU-based routing but focuses on differentiability and load balancing.\n\n[1] Wang, Ziteng, Jianfei Chen, and Jun Zhu. \"ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing.\" arXiv preprint arXiv:2412.14711 (2024).", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "It will make this paper more convincing if the feature streering experiments are involved.", "other_comments_or_suggestions": "No.", "questions_for_authors": "I would expect some degree of conflict between interpretability and performance. What specific strategies did you employ to balance interpretability gains against potential loss in performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pP8IjNrxAJ", "reviewer": "Reviewer_nXvb", "summary": "The paper proposes a novel method for intrinsically interpretable LLMs, called MoE-X. It's goal is to achieve better interpretability than sparse autoencoder by leveraging mixture of experts and providing sparse explanation without polysemanticity of activations. To do so there is a proposal to use wide and sparse experts and a routing to properly handle the computations. The experiments are performed on Chess benchmark, and OpenWebText, WikiText103 and WikiText2. Perplexicity is used as a metric.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank R-nXvb for their valuable questions!\n\n`>>> Q1`**Evaluation Metrics**\n\n`>>> A1`The reviewer noted that `metrics for interpretability is perplexity only`. We appreciate the feedback, but would like to clarify two critical points\n\n1. **Misconception About Perplexity**:  Perplexity is **not an interpretability metric**; it measures language modeling accuracy.\n2. **Interpretability Metrics**:  Our work evaluates interpretability using **three distinct metrics**\n   - *Coverage/Reconstruction Score* (Table 1): Measures the alignment between the hidden neurons and human-defined chess states.\n   - *Auto-Interp Score* (Figure 8): Assesses the accuracy of predicting activations for unseen samples using extracted explanations.\n   - *Case Studies* (Figure 7): Provide qualitative validation of interpretability.\n  These metrics are commonly used in previous research.\n\nAdditionally, both `R-K7C2` and `R-tfxC` acknowledged that our evaluation is reasonable. Based on this, we believe our evaluation is sufficient.\n\n`>>> Q2`**Comparing with Proto-lm [A]**\n\n`>>> A2`Really thank the reviewer for bring this great work! We will cite Proto-lm in our revision. While both methods share similar motivation, key differences make direct comparison challenging:\n1. **Base Models**: Proto-lm focuses on *encoder-only models* (e.g., BERT), while MoE-X is designed for *decoder-only causal LLMs* like GPT.\n2. **Tasks**: Proto-lm targets text classification (e.g., SST2), whereas MoE-X is for general language generation. So metrics like *Simulatability* cannot be directly applied to our model.\n3. **Layers**. Proto-lm explains only the *final layer's embeddings*, while MoE-X provides interpretability across all layers.\n4. **Interpretability Categorization**. Proto-lm uses **prototype-based explanations** for local, per-sample explaination. On the other hand, MoE-X focuses on **mechanical interpretability**, offering a global explanation of how the network functions by identifying the meaning of individual neuron.\n\n`>>> Q3`**User Study**\n\n`>>> A3`Thanks for the suggestion! As requested, we conduct a blinded human evaluation to assess neuron monosemanticity in MoE-X compared to GPT-2 and GPT-2 + SAE.\n\nSpecifically, 5 raters evaluate 20 random features from each model. Based on the top 5 activating samples, raters classified each feature as: `Yes` (clearly monosemantic), `Maybe` (partially interpretable), `No` (not monosemantic). \n\nResults showed MoE-X’s features were most interpretable, with 70% (_14/20_) labeled as `Yes`.\n\n|Model|Yes|Maybe|No|\n|-|-|-|-|\n|GPT-2|8|4|8|\n|GPT-2 + SAE|12|3|5|\n|MoE-X|14|3|3\n\n`>>> Q4`**Faithfulness and Simulatability Score**\n\n`>>> A4`As noted in `Q2`, directly comparing *Faithfulness* and *Simulatability* is not feasible due to differences in model, task, and setup. However, as suggested, we retrained the model and defined new evaluation metrics to make this comparison possible.\n\nSpecifically, we fine-tune MoE-X-Medium on SST2 using the last token for classification. We report the interpretability on the final layer. For Proto-lm, scores are **extracted from the raw figure in its paper**, which may have minor discrepancies. This provides *reasonable, though not exact*, comparability for interpretability analysis.\n\n**Faithfulness**\n\nTo evaluate faithfulness, we define two metrics inspired by [A]. \n-   **Comprehensiveness (Comp)**: Measures the decrease in model confidence when top $k$% activated neurons are removed.\n-   **Sufficiency (Suff)**: Measures the change in confidence when only top $k$% activated neurons are preserved.\n\nWe do evaluations on the SST2, with with $k\\in$ {1,5,10, 20, 50}.\n\n**Results**:  The table shows *Comp* and *Suff* scores. Notably, for $k\\geq 5$%, MoE-X **achieves perfect scores** (Comp=100%, Suff=0%) due to its extreme sparsity: MoE-X activates <100 neurons out of 8,192 in a layer. This ensures faithful explanations, outperforming Proto-lm.\n\n|k%|Comp&#8593;||Suff&#8595;||\n|-|-|-|-|-|\n||Proto-lm|MoE-X  |Proto-lm|MoE-X |\n|1|69|**87**|36|**9**|\n|5|71|**100**|31|**0**|\n|10|81|**100**|27|**0**|\n|20|85|**100**|35|**0**|\n|50|88|**100**|50|**0**|\n\n**Simulatability**\n\nAfter training MoE-X on SST-2, we identify each neuron's concept and select the top 5 activated concepts for each sample. Following the evaluation setup in [A], 3 evaluators categorize 50 SST-2 questions based on these concepts.\n\n**Results**: MoE-X performs slightly worse than Proto-LM but outperforms other methods in [A]. Note that differences in question selection and human judgment make results not directly comparable to [A].\n\n|Method|SST-2| \n|-|-|\n|Random|42.3%|\n|LIME |87.3%| \n|Integrated Gradient|84.7%| \n|Proto-lm|**90.0%**|\n|**MoE-X**|88.7%|\n\n> Special Note: The detection score in paper is a Simulatability score. It predicts activation patterns (not classes) using an LLM, not humans.\n\n[A] \"Proto-lm: A prototypical network-based framework for built-in interpretability in large language models.\" EMNLP (2023).", "claims_and_evidence": "claims are presented clearly", "methods_and_evaluation_criteria": "methods and evaluation criteria are presented clearly", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The metrics for interpretability is perplexity only, this is strongly limiting the work. Additionally, there is no user study showcasing that those explanations are better comprehensible by the users. \n\nTo fill the gap of metrix, authors should also consider reporting faithfulness score, and Simulatability. As well as conduct a user study showcasing their superiority.", "relation_to_broader_scientific_literature": "They are related works and broader aspects of the field correctly.", "essential_references_not_discussed": "all essentials works are cited. However, there is no discussion with alternative to MoE solutions based on prototypical parts. and comparison, e.g. with work of: \n\nXie, Sean, Soroush Vosoughi, and Saeed Hassanpour. \"Proto-lm: A prototypical network-based framework for built-in interpretability in large language models.\" EMNLP (2023).", "other_strengths_and_weaknesses": "well written", "other_comments_or_suggestions": "I do not see a time to provide additional results in a author response time.", "questions_for_authors": "Could you provide comparison of your method to Proto-lm? \n\nAlso can you provide evaluation results of other metrics widely used in XAI such as faithfullness?\n\nI have read the rebuttal and increased my score accordingly. Good job for the authors to provide comparison! It was a challenging task given the time.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Legal Compliance (e.g., GDPR, copyright, terms of use)"], "ethical_review_concerns": "It is an XAI method for LLMs, and the usage of LLMs should be monitored."}}]

Author rebuttal:

Meta-review: In the paper, the authors propose MoE-X, an interpretable mixture of experts large language model, while maintaining competitive performance. All the reviewers are positive about the contributions of the papers, including: (1) the novelty of the proposed MoE-X; (2) the experiments are sufficient to support the claims about the interpretability of the MoE-X; (3) the paper is written and presented clearly. After the rebuttal, most of the concerns of the reviewers were addressed, and all the reviewers are happy with the current stage of the paper.

In my opinion, the contributions and originality of the proposed MoE-X are sufficient for acceptance at ICML. Therefore, I recommend accepting it in its current form. However, I encourage the authors to address the reviewers’ suggestions and integrate their feedback into the camera-ready version of their paper.
Decision: Accept (poster)