Beyond Redundancy: Diverse and Specialized Multi-Expert Sparse Autoencoder

Title: 1 Introduction
ArXiv: 2511.05745v1
Version: v1
Sections: 31
Estimated tokens: 11.0k

## Contents
- 1 Introduction
- 2 Methodology
  - 2.1 Preliminary
  - 2.2 The Scale Sparse Autoencoder
    - 2.2.1 Multiple Expert Activation
    - 2.2.2 Feature Scaling
- 3 Experiment
  - 3.1 End-to-end Performance
  - 3.2 Ablation Study
    - 3.2.1 Multiple Expert Activation
    - 3.2.2 Feature Scaling
  - 3.3 What Makes Scale SAE Work?
    - 3.3.1 Expert specialization
    - 3.3.2 Diversity of neuronal activation
  - 3.4 Feature similarity
- 4 Related Works
- 5 Conclusion and Discussion
- Acknowledgements
- Appendix
  - Appendix A Detailed Evaluation Metrics
    - A.1 Reconstruction Mean Squared Error (MSE)
    - A.2 Loss Recovered
    - A.3 Automated Interpretability Score
    - A.4 Average Neuronal Activation Similarity
  - Appendix B Case Study: Interpreting the Token "apples"
    - B.1 Multi-Faceted Interpretation
    - B.2 Discussion and Limitations
  - Appendix C Further Study on Feature Decomposition Methods
    - C.1 Alternative Decomposition Strategies
    - C.2 Discussion of Results
  - Appendix D Distribution of neuron activation

### 1 Introduction

The rapid advancement of Large Language Models (LLMs) has intensified the need for interpretability to address concerns regarding their safety, reliability, and fairness (bereska2024mechanisticinterpretabilityaisafety; 10.1145/3639372; rai2025practicalreviewmechanisticinterpretability). A primary approach for understanding these models involves analyzing the function of individual neurons. However, this effort is fundamentally hindered by polysemanticity, a phenomenon where a single neuron is activated by multiple, seemingly unrelated concepts, rendering its role ambiguous (olah2020zoom).
This phenomenon is explained by the superposition hypothesis (elhage2022toymodelssuperposition), which posits that polysemanticity arises from the compression of a vast number of real-world features into a finite, high-dimensional activation space. This compression compels individual neurons to represent multiple concepts simultaneously. Consequently, the function of any single neuron becomes entangled and intractable to interpret directly, which fundamentally obstructs our ability to understand the model’s internal mechanisms.

Sparse Autoencoders (SAEs) have been proposed to address the challenge of polysemanticity by decomposing model activations into a sparse, monosemantic feature dictionary (cunningham2023sparseautoencodershighlyinterpretable; NEURIPS2024_c212c1b8; bussmann2024batchtopksparseautoencoders). Traditional SAEs employ an overcomplete hidden layer and an $L_{1}$ penalty to learn this dictionary (NIPS2007_4daa3db3; 6639343), approximating the decomposition of polysemantic neurons into more interpretable, orthogonal feature directions. This line of research has seen rapid progress, with recent work (bricken2023monosemanticity; rajamanoharan2024improvingdictionarylearninggated) successfully scaling SAEs to millions of features and applying them to state-of-the-art models such as Claude 3 Sonnet (anthropic2024claude) and GPT-4 (achiam2023gpt).
Despite these advances, the practical adoption of SAEs is hindered by a significant scalability bottleneck (mudide2025efficientdictionarylearningswitch; shu2025surveysparseautoencodersinterpreting). Achieving sufficient sparsity requires an extremely wide hidden layer, resulting in computational costs that grow linearly with the model’s hidden dimension. Furthermore, because a trained SAE is layer-specific (shu2025surveysparseautoencodersinterpreting), a separate one must be trained for each layer of interest, adding significant computational overhead.

The integration of Mixture of Experts (MoE) architectures into SAEs has emerged as a recent strategy to alleviate this computational cost (shazeer2017outrageouslylargeneuralnetworks; lepikhin2020gshardscalinggiantmodels; switchtransformer). These models, exemplified by the Switch SAE (mudide2025efficientdictionarylearningswitch), partition a large autoencoder into smaller expert subnetworks and dynamically route activations to reduce computational costs while maintaining performance. However, our empirical analysis reveals a fundamental limitation: experts often fail to specialize, collapsing into redundant ensembles where the same features are duplicated across multiple experts. This lack of specialization leads to a recurrence of polysemanticity within each expert; for instance, a single feature might be strongly activated by disparate concepts such as "travel" and "boxer." This expert-level redundancy not only wastes model capacity but also undermines interpretability, as the mapping between learned features and human-understandable concepts becomes fragmented and unclear.

In this work, we introduce Scale Sparse Autoencoder, a dual-mechanism framework designed to enforce both expert specialization and neuron activation diversity. The first mechanism, Multiple Expert Activation, addresses specialization at the specialist level. Instead of activating a single expert, our method dynamically selects and activates a subset of experts for each input. This approach encourages structured specialization by routing the distinct semantic components of a polysemantic neuron to different features within separate experts, which leads each expert to develop a characteristic sensitivity to a particular conceptual domain. The second mechanism, Feature Scaling, operates at the feature level. Inspired by signal decomposition, this technique adaptively amplifies the high-frequency components of the encoder’s features in a learnable manner. Our empirical findings show that the model always learns to enhance these high-frequency components. This results in more monosemantic features, an improvement evidenced by a lower similarity to the features and more diverse neuron activation patterns. Extensive experiments validate the superiority of Scale SAE over existing algorithms across three key axes: reconstruction fidelity, functional faithfulness, and feature redundancy. Ablation studies further confirm that these benefits are directly attributable to our two core, mutually reinforcing mechanisms, as ablating either one results in a significant degradation across the aforementioned performance metrics.
The primary contributions of this work are as follows.

- 1.
We introduce Scale SAE, a novel framework that integrates two synergistic mechanisms: Multiple Expert Activation and Feature Scaling (Sections [2.2.1](https://arxiv.org/html/2511.05745v1#S2.SS2.SSS1) and [2.2.2](https://arxiv.org/html/2511.05745v1#S2.SS2.SSS2)).
- 2.
We carry out end-to-end experiments showing that Scale SAE substantially outperforms existing methods, achieving a lower reconstruction error, higher Loss Recovered, enhanced interpretability, and reduced feature similarity [3.1](https://arxiv.org/html/2511.05745v1#S3.SS1).
- 3.
Our ablation studies validate the synergistic contribution of both mechanisms to the model’s overall performance (Section [3.2](https://arxiv.org/html/2511.05745v1#S3.SS2)).
- 4.
We provide a detailed mechanistic analysis that explains these performance gains, establishing that Multiple Expert Activation directly improves expert specialization. At the same time, Feature Scaling reduces feature redundancy by activating more diverse features (Sections [3.3](https://arxiv.org/html/2511.05745v1#S3.SS3)).

### 2 Methodology

#### 2.1 Preliminary

Dense SAE. A traditional sparse autoencoder, such as the TopK variant (gao2024scalingevaluatingsparseautoencoders), processes input through a single encoder-decoder pair. The training objective is to minimize the reconstruction mean square error (MSE) $\|\mathbf{x}-\mathbf{\widehat{x}}\|^{2}$, where the reconstructed vector is $\widehat{\mathbf{x}}=\mathbf{W}^{\mathrm{dec}}\mathbf{z}+{\mathbf{b}_{\mathrm{pre}}}$ and the sparse code is $\mathbf{z}=\mathrm{TopK}(\mathbf{W}^{\mathrm{enc}}(\mathbf{x}-{\mathbf{b}_{\mathrm{pre}}}))$.

MoE SAE. The MoE SAE architecture improves this design for computational efficiency by employing $N$ distinct "expert" autoencoders. For each input $\mathbf{x}\in\mathbb{R}^{d_{\text{model}}}$, a router selects a single expert $i^{*}$ based on a learned gating distribution: $i^{*}=\mathrm{argmax}_{i}(p_{i}(\mathbf{x}))$, where ${p(\mathbf{x})}=\mathrm{softmax}(\mathbf{W}_{g}\mathbf{x})$. Each expert can generate an individual reconstruction $E_{i}(\mathbf{x})$:

$$ $E_{i}(\mathbf{x})=\mathbf{W}^{\mathrm{dec}}_{i}\mathrm{TopK}(\mathbf{W}^{\mathrm{enc}}_{i}\mathbf{x})$ (2.1) $$

The expert chosen then produces the final reconstruction $\widehat{\mathbf{x}}=p_{i^{*}(\mathbf{x})}E_{i^{*}(\mathbf{x})}(\mathbf{x}-\mathbf{b}_{\mathrm{pre}})+\mathbf{b}_{\mathrm{pre}}$.

The total loss function $\mathcal{L}$ combines the reconstruction error $\mathcal{L}_{\text{recon}}$ with an auxiliary load-balancing loss $\mathcal{L}_{\text{aux}}$, weighted by a hyperparameter $\alpha$:

$$ $\mathcal{L}=\underbrace{\|\mathbf{x}-\mathbf{\widehat{x}}\|_{2}^{2}}_{\mathcal{L}_{\text{recon}}}+\alpha\cdot\underbrace{\left(N\cdot\sum_{i=1}^{N}f_{i}\cdot P_{i}\right)}_{\mathcal{L}_{\text{aux}}}.$ (2.2) $$

The auxiliary term encourages uniform utilization of the expert in a batch, where $f_{i}$ is the fraction of tokens routed to the expert $i$ and $P_{i}$ is its average routing probability.

#### 2.2 The Scale Sparse Autoencoder

Figure: Figure 1: Scale Sparse Autoencoder Architecture. An illustration of the three core mechanisms in the Scale SAE architecture. (a) Multiple Expert Activation. A router selects a subset of experts (e.g., 2 out of 3 shown) to process each input. (b) Global Top-K Activation. The activations from the selected experts are aggregated, and a global Top-K operation (K=3 shown) is applied to enforce sparsity. (c) Feature Scaling. The encoder weights of each expert are decomposed and scaled to dynamically amplify high-frequency components.
Refer to caption: tex/images/architecture.png

##### 2.2.1 Multiple Expert Activation

Although Switch SAE is the first model to incorporate the MoE architecture into a sparse autoencoder, its implementation did not fully leverage the benefits of the paradigm. A key factor limiting both the interpretability and performance of Switch SAE is its low degree of expert specialization, resulting in significant feature redundancy across experts. This high redundancy explains why the performance of Switch SAE is often comparable to that of a standard TopK SAE (Figure [3](https://arxiv.org/html/2511.05745v1#S3.F3)).

This challenge is addressed by modifying the routing mechanism to transition from activating a single expert to activating $e$ experts, where $e\geq 2$ (Figure [1](https://arxiv.org/html/2511.05745v1#S2.F1)). The set of selected experts, denoted by $\mathcal{T}$, is determined by $\mathcal{T}=\mathrm{argtopk}(\mathbf{x},e)$. At the same time, the routing probabilities are computed using a softmax function: $\mathbf{p}(\mathbf{x})=\mathrm{softmax}(\mathbf{W}_{\mathrm{router}}(\mathbf{x}-\mathbf{b}_{\mathrm{router}}))$.
For each selected expert $i\in\mathcal{T}$, initial feature activations are calculated as $f_{i}=\mathbf{W}^{\mathrm{enc}}_{i}\mathbf{x}$. A crucial distinction in our approach is that the final sparse activations are determined through a global Top-K selection process across all activated experts. The sparse activation for the $j$-th neuron of the $i$-th expert is given by:

$$ $z_{ij}=f_{ij}\cdot\mathbb{I}\left(f_{ij}\in\text{TopK}\left(\{f_{l,m}\mid l\in\mathcal{T}\}\}\right)\right)$ (2.3) $$

Subsequently, the output of each expert is calculated as $E_{i}(\mathbf{x})=\mathbf{W}^{\mathrm{dec}}_{i}\mathbf{z}_{i}$. The final reconstruction, $\widehat{\mathbf{x}}$, is a weighted sum of these individual expert outputs:

$$ $\widehat{\mathbf{x}}=\sum_{i}^{i\in\mathcal{T}}p_{i}(\mathbf{x})E_{i}(\mathbf{x}).$ (2.4) $$

The loss function for this model remains consistent with the one defined in Equation [2.2](https://arxiv.org/html/2511.05745v1#S2.E2).

A significant innovation of this approach is the global activation mechanism, formulated in Equation [2.3](https://arxiv.org/html/2511.05745v1#S2.E3). While its activation scope is more constrained than that of a dense SAE, it represents a fundamental departure from single-expert activation. This model can be viewed as an intermediate architecture between a traditional MoE SAE and a dense SAE, a concept explored further in Section [3.2.1](https://arxiv.org/html/2511.05745v1#S3.SS2.SSS1).

##### 2.2.2 Feature Scaling

Figure: Figure 2: The scaling law for the trained scale factor $\omega$.
Refer to caption: tex/images/scale_factor_omega.png

A significant challenge in MoE SAE architectures is activation collapse, where diverse inputs activate a recurring, limited subset of experts and features. This phenomenon impairs reconstruction by limiting the information available to the decoder, thereby increasing both the MSE and feature redundancy.
Our Feature Scaling mechanism is inspired by the principle of high-pass filtering from computer vision. The efficacy of such an approach for deep neural networks is supported by the work of wang2022antioversmoothingdeepvisiontransformers. In their research, they conclude that amplifying high-frequency features can counteract the "oversmoothing" phenomenon in Transformers. Adopting a similar philosophy, our method utilizes trainable parameters to amplify the high-frequency components of the encoder’s features, enabling the encoded representations to preserve more fine-grained information.

The Feature Scaling mechanism decomposes each expert’s encoder weight matrix, $\mathbf{W}^{\mathrm{enc}}_{i}$. The implementation, illustrated in Figure [1](https://arxiv.org/html/2511.05745v1#S2.F1), first defines a low-frequency component, ${\macc@depth\@ne\macc@set@skewchar\macc@nested@a 111{W}}^{\mathrm{enc}}_{i}$, as the average feature vector of the expert’s encoder weights: $\mathbf{w}_{i}=\frac{1}{n}\sum_{j=1}^{n}{\mathbf{W}^{\mathrm{enc}}_{ij}}$. Consequently, the high-frequency component, $\Delta\mathbf{W}^{\mathrm{enc}}_{i}$, is defined as the deviation from this low-frequency baseline: $\Delta\mathbf{W}^{\mathrm{enc}}_{i}=\mathbf{W}^{\mathrm{enc}}_{i}-{\macc@depth\@ne\macc@set@skewchar\macc@nested@a 111{W}}^{\mathrm{enc}}_{i}$. The rationale for selecting this mean-based decomposition over alternative strategies is validated in Appendix [C](https://arxiv.org/html/2511.05745v1#A3).

The final scaled weight matrix, $\widehat{W}^{\mathrm{enc}}_{i}$, is constructed by amplifying this high-frequency component:

$$ $\widehat{W}^{\mathrm{enc}}_{i}={\macc@depth\@ne\macc@set@skewchar\macc@nested@a 111{W}}^{\mathrm{enc}}_{i}+(1+\omega)\Delta\mathbf{W}^{\mathrm{enc}}_{i}$ (2.5) $$

The parameter $\omega$ is a trainable scaling factor that modulates the influence of the high-frequency component. Empirical analysis reveals two key properties of this parameter. First, it robustly converges to a positive value during training, indicating that the model learns to amplify high-frequency details, thereby enhancing its performance. Second, its magnitude exhibits a clear positive correlation with the number of activated experts, establishing a predictable scaling law (Figure [2](https://arxiv.org/html/2511.05745v1#S2.F2)).

### 3 Experiment

We train sparse autoencoders with a hidden dimension of 768 on the intermediate activations of the 8th layer of the GPT-2 (radford2019language). The neurons in this layer contain rich semantic information beyond simple lexical features, making it an ideal testbed for assessing the monosemanticity of learned representations. The model is trained for 100,000 steps using the OpenWebText (Gokaslan2019OpenWeb).

#### 3.1 End-to-end Performance

To ensure a fair comparison of computational efficiency, all models are evaluated under a strict FLOPS-matched paradigm. This approach normalizes the total number of floating-point operations across all models, allowing for a direct assessment of architectural benefits. We compare Scale SAE against three baseline models: Switch SAE (mudide2025efficientdictionarylearningswitch), TopK SAE (gao2024scalingevaluatingsparseautoencoders), and Gated SAE (rajamanoharan2024improvingdictionarylearninggated).
The hidden dimensions for each model are configured as follows to maintain this computational equivalence:

- •
Scale SAE: The total hidden dimension is set to 24,576. This is partitioned into settings of 256, 128, or 64 experts, with a corresponding activation of 8, 4, or 2 experts per forward pass.
- •
Dense SAEs (TopK & Gated): In order to match the computational load of activating a fixed number of experts, the hidden dimension is set to 768. This is derived from the total expert dimension divided by a typical expert count (e.g., 24576/32 = 768).
- •
Switch SAE: This model is configured with a total hidden dimension of 24,576 and 32 experts, consistent with the Scale SAE’s overall structure but limited to single-expert activation.

Figure: (a)
Refer to caption: tex/images/legend.png

Reconstruction MSE. We benchmark the Reconstruction MSE of our model against several baselines across two distinct data domains. The first is the general-domain OpenWebText corpus, and the second is a specialized biomedical question-answering dataset, the Biomedical subset of Humanity’s Last Exam (HLE-Biomedical) (phan2025humanitysexam).
On the OpenWebText dataset (Figure [3(a)](https://arxiv.org/html/2511.05745v1#S3.F3.sf1)), Scale SAE achieves a lower MSE than all baseline models without exception across all tested sparsity levels ($L_{0}$ norm). The performance advantage of Scale SAE becomes more pronounced as the target $L_{0}$ increases. At $L_{0}$ values of 32, 64, and 128, Scale SAE reduces the MSE by 37.21%, 41.99%, and 42.54%, respectively, compared to the next-best performing baseline. In contrast, the Switch SAE model fails to surpass the standard TopK SAE, despite possessing a 32-fold greater hidden dimension. This underperformance can be attributed to the high degree of feature redundancy among its experts.
The results on the specialized HLE-Biomedical dataset (Figure [3(b)](https://arxiv.org/html/2511.05745v1#S3.F3.sf2)) further underscore the robustness of our approach. In this cross-domain experiment, the performance degradation of the Switch SAE model is even more pronounced, particularly at higher activation densities ($L_{0}\geq 32$). In contrast, all three settings of Scale SAE maintain their superior performance, achieving the lowest reconstruction error in nearly all tested settings.

Loss Recovered. We next evaluate the Loss Recovered metric on the HLE-Biomedical dataset to assess model faithfulness as illustrated in the Figure [3(c)](https://arxiv.org/html/2511.05745v1#S3.F3.sf3). The performance of the various multi-expert settings of Scale SAE ($e\in\{2,4,8\}$) is highly competitive and comparable to that of the TopK SAE baseline. Specifically, while TopK SAE holds a slight advantage in the low-sparsity regime ($L_{0}\leq 4$), Scale SAE outperforms it as the number of activated features increases ($L_{0}\geq 8$).
The advantage of our model over other baselines is also evident, particularly when compared to Gated SAE, which underperforms all other models, exhibiting the lowest Loss Recovered scores. Similarly, the Switch SAE model again fails to match the performance of the simpler TopK SAE, particularly at higher sparsity levels ($L_{0}\geq 32$), which further suggests that its architecture suffers from significant feature redundancy. The substantial Loss Recovered scores of Scale SAE, combined with its low reconstruction MSE, validate that our proposed mechanisms achieve a superior balance between reconstruction fidelity and functional faithfulness.

Automated Interpretability. Interpretability is assessed using the automated scoring pipeline from juang2024open, following the methodology of mudide2025efficientdictionarylearningswitch. We test all models on the OpenWebText2 (pile).
Scale SAE’s superior interpretability is evident across all activation quantiles, as detailed in Figure [3(d)](https://arxiv.org/html/2511.05745v1#S3.F3.sf4). These quantiles ($\text{not},1,2,...$) represent the activation threshold required for a token to be associated with a feature. As expected, scores for all models improve at higher quantiles, as a stricter threshold isolates the most salient examples of a feature’s activation. Crucially, Scale SAE consistently outperforms all baselines, including the structurally similar Switch SAE, at every quantile.
Our proposed mechanisms enhance the monosemanticity and faithfulness of the learned features without compromising reconstruction accuracy or computational efficiency.
In the Appendix [A.3](https://arxiv.org/html/2511.05745v1#A1.SS3), we give a more detailed introduction to the Automated Interpretability pipeline.

#### 3.2 Ablation Study

##### 3.2.1 Multiple Expert Activation

Figure: (a)
Refer to caption: tex/images/legend_2.png

A key question is whether the performance improvements of Scale SAE stem primarily from its multi-expert architecture. A focused ablation study addresses this by varying the number of activated experts and the feature sparsity level.
We tested our model on the general-domain OpenWebText dataset. The results, depicted in Figures [4(a)](https://arxiv.org/html/2511.05745v1#S3.F4.sf1) and [4(c)](https://arxiv.org/html/2511.05745v1#S3.F4.sf3), reveal a stark performance dichotomy. The single-expert setup ($e=1$) always exhibits high reconstruction error and training instability. In contrast, all multi-expert setups ($e\geq 2$) achieve significantly improved accuracy and stability. Notably, the performance curves for these multi-expert models are tightly clustered, indicating that activating more than two experts yields diminishing returns. This principle is further underscored by the sharp decline in performance of the $e=16$ models at low sparsity levels.
To assess the generalizability of these findings, we replicate the experiment on the specialized HLE-Biomedical dataset (Figures [4(b)](https://arxiv.org/html/2511.05745v1#S3.F4.sf2) and [4(d)](https://arxiv.org/html/2511.05745v1#S3.F4.sf4)). The results were highly consistent with those on OpenWebText, confirming that the benefits of our Multiple Expert Activation strategy are also evident in a distinct, specialized domain.

##### 3.2.2 Feature Scaling

Figure: (a)
Refer to caption: tex/images/legend_3.png

While Multiple Expert Activation provides a robust architectural foundation, the second innovation, Feature Scaling, is designed to refine the learned representations. The specific contribution of this component was therefore isolated and quantified through a series of targeted experiments. The study directly compares the Reconstruction MSE and Loss Recovered for models with and without Feature Scaling across a range of expert and sparsity settings (Figures [5(a)](https://arxiv.org/html/2511.05745v1#S3.F5.sf1) and [5(b)](https://arxiv.org/html/2511.05745v1#S3.F5.sf2)).
The analysis confirms the efficacy of Feature Scaling, with the mechanism leading to lower MSE and higher Loss Recovered across nearly all scenarios. Most strikingly, Feature Scaling resolves the primary failure mode identified in the previous analysis: the sharp performance decline of the $e=16$ model at low sparsity levels. As shown in Figure [5(b)](https://arxiv.org/html/2511.05745v1#S3.F5.sf2), this intervention almost entirely mitigates the instability, reducing MSE by an average of 10.9% for this setup while improving the Loss Recovered by 1.75%. The benefits also extend to the high-sparsity regime ($L_{0}=2$), where Feature Scaling improves Loss Recovered by an average of 4.132%. These results underscore that the Feature Scaling mechanism is not merely an incremental improvement but a critical component for ensuring training stability, particularly in variants with a high number of activated experts.

#### 3.3 What Makes Scale SAE Work?

We hypothesize that the effectiveness of Scale SAE stems from two distinct mechanisms. The first hypothesis is that Multiple Expert Activation promotes a higher degree of expert specialization compared to single-expert routing. The second is that Feature Scaling enhances neuron activation diversity by amplifying the high-frequency components of the encoder’s weights. This section provides a detailed analysis to validate these hypotheses, arguing that both mechanisms derive their benefits from a fundamental reduction in feature redundancy within the learned dictionary.

##### 3.3.1 Expert specialization

Figure: (a)
Refer to caption: tex/images/analysis_multi_expert_ratio_bar.png

The first hypothesis that Multiple Expert Activation promotes specialization can be validated by analyzing the geometric properties of the learned feature dictionaries. A primary metric for this analysis is feature redundancy, which is quantified as the proportion of features with a maximum cosine similarity to any other feature exceeding 0.9. High similarity in this context indicates significant semantic overlap (NEURIPS2024_c212c1b8).
Figure [6(a)](https://arxiv.org/html/2511.05745v1#S3.F6.sf1) reveals a clear distinction between the two architectures. Multi-expert models demonstrate a strong signature of effective specialization. They exhibit high intra-expert feature similarity, indicating strong internal coherence. In contrast, their inter-expert similarity remains low; this metric is measured by comparing the feature sets of randomly sampled experts of the same size. Conversely, the single-expert model displays high similarity across all features, indicating both significant feature redundancy and a failure to achieve feature specialization.

To further validate these findings, we investigate the expert activation distribution using a specialized dataset of 1,500 past-tense verbs. The cumulative distribution function (CDF) of expert activation frequencies, plotted in Figure [6(b)](https://arxiv.org/html/2511.05745v1#S3.F6.sf2), provides strong evidence of specialized learning. A steep CDF is a hallmark of specialization, as it signifies that a small subset of experts handles the majority of activations. The results are unambiguous: the single-expert model ($e=1$) exhibits a flat CDF, confirming its inability to specialize for this targeted linguistic task. Conversely, all multi-expert models show much steeper CDFs, showing that specific experts have learned to represent distinct feature sets. The $e=16$ model has a comparatively flatter CDF than other multi-expert models, suggesting a less concentrated activation pattern, which may account for the slight decline in performance observed in our ablation study.

##### 3.3.2 Diversity of neuronal activation

Figure: (a)
Refer to caption: tex/images/analysis_scale_improve_heatmap.png

The mechanism by which Feature Scaling improves performance is elucidated through an analysis of its impact on the diversity of neuronal activation patterns. A curated set of 13,500 verb tokens, programmatically filtered from the OpenWebText, forms the basis for this analysis. We measure activation diversity by calculating the average pairwise similarity of activated neuron sets across these tokens. A higher average similarity score indicates a less diverse, more uniform activation pattern, signifying greater feature redundancy. A more formal description of this metric is given in the Appendix [A.4](https://arxiv.org/html/2511.05745v1#A1.SS4).

Figure [7(a)](https://arxiv.org/html/2511.05745v1#S3.F7.sf1) shows a significant difference when using Feature Scaling. In the multi-expert activation setting, the average similarity of neuron activations decreases significantly as the scale increases. The average similarity decreases by 6.19%. It is worth noting that in the two cases of $L_{0}=2,e=16$ and $L_{0}=2,e=8$, the similarity decreases by 31.25% and 25.42%, respectively, which correspond precisely to the two cases shown in Section [3.2.2](https://arxiv.org/html/2511.05745v1#S3.SS2.SSS2) where Feature Scaling has apparent optimization effects.
However, in direct contrast to multi-expert settings, which leverage the mechanism to increase feature diversity (i.e., lower similarity scores), the single-expert ones exhibit an increase in feature similarity when scaling is applied.
As shown in Figure [7(b)](https://arxiv.org/html/2511.05745v1#S3.F7.sf2), the baseline single-expert model already exhibits a low feature similarity score. This apparent diversity, however, is misleading; it represents a state of high redundancy, where numerous features capture similar semantic information, leading to uninformative and chaotic activation patterns. Applying Feature Scaling to this scenario does not simply reduce activation diversity; instead, it reduces feature redundancy, which remains helpful for improving interpretability.
Appendix [D](https://arxiv.org/html/2511.05745v1#A4) provides more details on this experiment.

#### 3.4 Feature similarity

Figure: (a)
Refer to caption: tex/images/analysis_multi_expert_similarity_heatmap.png

Prior work by mudide2025efficientdictionarylearningswitch identifies feature redundancy as a primary factor limiting the performance of Switch SAE. To demonstrate that our proposed mechanisms directly address this issue, we conduct an analysis of feature similarity. We quantify similarity as the proportion of features with a maximum cosine similarity to any other feature exceeding 0.9.
The impact of Multiple Expert Activation is shown in the heatmap in Figure [8(a)](https://arxiv.org/html/2511.05745v1#S3.F8.sf1), where lighter colors indicate higher similarity. The single-expert setup ($e=1$) exhibits high inherent redundancy, whereas all multi-expert setups ($e\geq 2$) achieve significantly lower similarity, with further reductions observed at lower values of $k$.
Feature Scaling substantially reduces feature similarity across most settings (Figure [8(b)](https://arxiv.org/html/2511.05745v1#S3.F8.sf2)). This reduction is particularly pronounced at an $L_{0}$ of 16, where the mechanism reduces the proportion of similar features by over 50%.

### 4 Related Works

Signal Processing on Deep Learning Architectures. Frequency-domain analysis, inspired by classical signal processing (daubechies1992ten; oppenheim2009discrete), is instrumental in addressing limitations in deep learning. For instance, leethorp2022fnetmixingtokensfourier replace self-attention with the Fourier Transform to improve computational efficiency, while others have used similar techniques to capture long-range dependencies (NEURIPS2021_07e87c2f) or enable efficient super-resolution (Xie_2021_ICCV). Critically, this analytical approach is also used to correct architectural flaws. wang2022antioversmoothingdeepvisiontransformers identify the self-attention mechanism as an intrinsic low-pass filter and proposed amplifying high-frequency features to counteract the resulting "oversmoothing" phenomenon.

Sparse Autoencoders. SAEs decompose polysemantic activations into an overcomplete feature dictionary; this approach is pioneered for LLMs by Anthropic(cunningham2023sparseautoencodershighlyinterpretable; bricken2023monosemanticity). Subsequent work introduces a series of architectural and training modifications to the original SAE framework. To enforce sparsity more directly, gao2024scalingevaluatingsparseautoencoders propose the TopK SAE, which selects only the strongest $K$ features per token. Another line of work focuses on the activation function itself, with ProLUNonlinearity and rajamanoharan2024jumpingaheadimprovingreconstruction introducing learnable, threshold-based activations that replace the standard ReLU. To mitigate bias in feature magnitude estimation, rajamanoharan2024improvingdictionarylearninggated develop the Gated SAE, which decouples feature selection from its magnitude by applying the sparsity penalty only to a dedicated selection gate.

Mixture of Experts Models. The core MoE concept of conditional computation is first proposed by 6797059. Over two decades later, shazeer2017outrageouslylargeneuralnetworks are the first to successfully apply the paradigm to large-scale deep learning. A series of subsequent studies further push the computational efficiency of MoE to its limits through innovations such as introducing the auxiliary load-balancing loss for stable training (lepikhin2020gshardscalinggiantmodels), simplifying the routing mechanism to a single expert (switchtransformer), and successfully scaling these architectures to over a trillion parameters (pmlr-v162-du22c). The versatility of this approach has since been demonstrated by its expansion into the computer vision domain (NEURIPS2021_48237d9f) and its adoption in state-of-the-art language models such as Mixtral 8x7B (jiang2024mixtralexperts).

### 5 Conclusion and Discussion

This paper fundamentally addresses the challenge of feature redundancy in the Mixture of Experts Sparse Autoencoder. We introduced Scale SAE, a framework that integrates two key insights to resolve the architectural and representational flaws of prior models. First, we demonstrate that the prevailing single-expert activation strategy is suboptimal, and that a minimal architectural shift to co-activating at least two experts is sufficient to unlock significant performance gains and enable true specialization. Second, our Feature Scaling mechanism provides a learnable method to enhance feature diversity by amplifying high-frequency components. By resolving these core issues, our work demonstrates that the MoE SAE paradigm is a promising approach for reconciling mechanistic interpretability with computational efficiency in LLM analysis.

### Acknowledgements

This research was partially funded by the National Institutes of Health (NIH) under award 1R01EB037101-
01. The views and conclusions contained in this document are those of the authors and should not be
interpreted as representing the official policies, either expressed or implied, of the NIH.

## Appendix

### Appendix A Detailed Evaluation Metrics

The performance of each Sparse Autoencoder (SAE) variant was assessed using a suite of three key metrics, each designed to evaluate a different aspect of the model’s quality: reconstruction fidelity, faithfulness to the original model, and the interpretability of its learned features.

#### A.1 Reconstruction Mean Squared Error (MSE)

This metric quantifies the fidelity of the SAE’s reconstruction by measuring the average squared difference between the original and reconstructed activation vectors. A lower MSE signifies a more accurate reconstruction and less information loss. It is formally defined as:

$$ $\mathcal{L}_{\text{MSE}}=\frac{1}{D}\sum_{i=1}^{D}(x_{i}-\widehat{x}_{i})^{2}$ (A.1) $$

where $x\in\mathbb{R}^{D}$ is the original activation vector from the Large Language Model (LLM), and $\widehat{x}$ is the corresponding vector reconstructed by the SAE. However, a low MSE alone is an insufficient measure of quality, as it can be achieved by learning polysemantic features. It must therefore be assessed in conjunction with the other metrics.

#### A.2 Loss Recovered

This metric assesses the SAE’s faithfulness to the original model by measuring the extent to which its sparse features can account for the LLM’s downstream performance. A higher Loss Recovered score shows that the SAE has successfully captured the features most salient to the LLM’s predictions. The calculation involves three forward passes through the LLM on a given dataset:

- •
First, the standard cross-entropy loss is calculated using the original, unmodified activations, yielding $\mathcal{L}_{\text{original}}$.
- •
Second, the loss is recalculated with the SAE’s reconstructed activations substituted at the target layer, yielding $\mathcal{L}_{\text{reconstructed}}$.
- •
Third, a baseline loss is established by zeroing out the activations at the target layer, yielding $\mathcal{L}_{\text{zero}}$.

The Loss Recovered score is the fraction of the performance drop caused by this zero-ablation that is "recovered" by the SAE’s reconstruction, formally defined as:

$$ $\text{Loss Recovered}=\frac{\mathcal{L}_{\text{zero}}-\mathcal{L}_{\text{reconstructed}}}{\mathcal{L}_{\text{zero}}-\mathcal{L}_{\text{original}}}$ (A.2) $$

#### A.3 Automated Interpretability Score

To objectively quantify the monosemanticity of individual features, we employ the Automated Interpretability method, a two-stage pipeline first proposed by juang2024open. This approach automatically assesses the conceptual consistency of the text passages that trigger a given feature. The pipeline consists of the following two stages:

- •
Stage 1: Max Activation Curation. The MaxAct (bricken2023monosemanticity) identifies a corpus of text passages from a large dataset that cause a feature to activate most strongly. These passages are assumed to represent the feature’s core semantic meaning.
- •
Stage 2: Automated Scoring. An external, powerful LLM (Llama-3 in this study) is prompted to act as an automated judge. It is presented with a held-out text passage that also activates the feature and performs a classification task: to determine if the new passage is conceptually consistent with the core meaning established by the MaxAct corpus.

The final interpretability score for a feature is the LLM’s classification accuracy on this task. A high score signifies that the feature represents a single, coherent concept, which is the definition of high monosemanticity.

#### A.4 Average Neuronal Activation Similarity

This metric is designed to provide a single, quantitative measure of the overall feature redundancy within the learned SAE dictionary. It is calculated by iterating through every unique pair of tokens in a dataset, computing the proportion of shared activated neurons for each pair, and then averaging these scores across all pairs. It is formally defined as:

$$ $\text{Similarity}=\frac{1}{N(N-1)}\sum_{i\neq j}\frac{k_{i,j}}{K_{\mathrm{total}}}$ (A.3) $$

where $N$ is the total number of tokens, $k_{i,j}$ is the number of common neurons activated by both token $i$ and token $j$ (i.e., $|S_{i}\cap S_{j}|$), and $K_{\mathrm{total}}$ is a normalization factor representing the total number of neurons activated per token (e.g., the value of $K$ in a TopK SAE).

The final score provides a global measure of the dictionary’s representational efficiency. A high score (closer to 1) indicates a high degree of feature redundancy, suggesting that different tokens activate very similar sets of neurons and that many features capture overlapping semantic concepts. Conversely, a low score (closer to 0) indicates a high degree of feature diversity, suggesting that individual features have learned to represent more specific, monosemantic concepts. Therefore, this metric serves as a crucial tool for quantifying the ability of different SAE architectures to learn a well-decomposed feature space.

### Appendix B Case Study: Interpreting the Token "apples"

Figure: (a) The intensity of different experts being activated.
Refer to caption: tex/images/appendix_expert_activation.png

**Table 1: The top-8 features with the highest activation intensity and their corresponding semantic explanations.**
| Expert ID | Feature ID | Activation Strength | Most Relevant Interpretation |
| --- | --- | --- | --- |
| 0 | 288 | 17.1770 | plural nouns |
| 0 | 121 | 15.5380 | plants |
| 0 | 86 | 14.4454 | countable nouns |
| 50 | 46 | 13.7324 | Apple/ios |
| 0 | 232 | 13.6768 | fruits |
| 53 | 9 | 12.9206 | Polysemantic Feature |
| 50 | 98 | 11.2872 | Apple/ios |
| 61 | 84 | 11.0593 | Ending with s |

To qualitatively assess the interpretability of Scale SAE, we analyzed the feature activations for the token "apples" in the sentence "I love apples but hate oranges completely." For each feature, its semantic meaning was derived by manually summarizing the concept familiar to the top five tokens that maximally activated it. Features for which a unified semantic concept could not be determined were labeled as "Polysemantic."

#### B.1 Multi-Faceted Interpretation

The Scale SAE router selected eight experts to represent the token, with the top 32 most active features drawn from across this expert subset (Figure [9](https://arxiv.org/html/2511.05745v1#A2.F9)). This global selection, involving multiple experts rather than a single one, is a key architectural advantage. An analysis of the most salient features (Table [1](https://arxiv.org/html/2511.05745v1#A2.T1)) reveals a multi-faceted decomposition of the token’s meaning. The model successfully identified several distinct conceptual layers:

- •
Lexical and Syntactic Features: Expert 0 contributed features corresponding to "plural nouns" (feature 228) and "countable nouns" (feature 86).
- •
Semantic Features: Other experts captured core semantic concepts related to "apples," such as "plants" and "fruits."
- •
Orthographic Features: The model also identified structural patterns, such as expert 61’s feature 84, which responds to tokens ending in "s."

Interestingly, the model also activated features in expert 50 related to the proper noun "Apple" (the company), demonstrating its ability to capture even contextually incorrect but related concepts.

#### B.2 Discussion and Limitations

Despite these positive results, this detailed analysis also highlights several limitations and avenues for future improvement. First, some polysemantic features persist. While Scale SAE significantly improves monosemanticity over baseline models, some features—such as feature 9 in expert 53—are still activated by a broad and diverse range of tokens, rendering their specific semantic interpretation difficult. Second, intra-expert semantic redundancy was not eliminated; for example, two distinct features in expert 50 were found to represent the same concept. Third, the model’s reliance on highly abstract grammatical features (e.g., "plural noun") suggests a lack of fine-grained semantic decomposition. Finally, expert specialization remains imperfect, as exemplified by expert zero activating features for disparate conceptual categories (lexical, syntactic, and semantic), which contradicts the goal of having each expert focus on a distinct domain.

### Appendix C Further Study on Feature Decomposition Methods

Our Feature Scaling mechanism is predicated on the decomposition of an expert’s encoder weight matrix, $M$, into a "low-frequency" component ($M_{lp}$) and a "high-frequency" component ($M_{hp}$), such that $\widehat{M}=(M-M_{lp})\times\text{Scale}+M_{lp}$. The primary implementation in this work defines the low-frequency component as the mean of the feature vectors, $M_{lp}=M_{\text{mean}}$. To validate this design choice, we conducted an ablation study comparing our approach against two alternative decomposition strategies.

#### C.1 Alternative Decomposition Strategies

We evaluated the following three methods for defining the low-frequency and high-frequency components:

- •
Method 0 (Our Proposed Method): Mean-based Decomposition. The low-frequency component is the average feature representation of the expert. This method hypothesizes that the mean captures the most common, low-frequency patterns, while deviations from the mean represent unique, high-frequency details.
$\widehat{M}=(M-M_{\text{mean}})\times\text{Scale}+M_{\text{mean}}$
(C.1)
- •
Method 1: Identity-based Decomposition. The low-frequency component is defined as the identity matrix, $I$. This strategy frames the problem as scaling the transformative part of the weight matrix ($M-I$) relative to the identity-preserving part ($I$).
$\widehat{M}=(M-I)\times\text{Scale}+I$
(C.2)
- •
Method 2: Learning-based Decomposition. The low-frequency component is a learnable matrix, $A_{LP}$, which is trained concurrently with the rest of the network. This method allows the model to dynamically learn the optimal separation of low- and high-frequency components.
$\widehat{M}=(M-A_{LP})\times\text{Scale}+A_{LP}$
(C.3)

Figure: Figure 10: Performance comparison of the three feature decomposition strategies across a range of sparsity levels.
Refer to caption: tex/images/appendix_mse.png

#### C.2 Discussion of Results

We compared the performance of these three decomposition strategies across a range of sparsity levels, with the results visualized in Figure [10](https://arxiv.org/html/2511.05745v1#A3.F10). The data reveal that Method 1 (Identity-based) always results in a significantly higher reconstruction error, establishing it as a suboptimal approach.
Conversely, our proposed Method 0 (Mean-based) and the more complex Method 2 (Learned-based) both achieve strong and highly competitive reconstruction performance. As the performance difference between these two methods is marginal and often within the range of typical training variance, there is no compelling evidence to justify the added complexity of the learning-based approach. The learning-based method introduces additional trainable parameters without offering a significant performance benefit over our simpler, more efficient mean-based strategy.
Therefore, given that the primary goal is to reconstruct activations faithfully, the superior performance-to-complexity ratio of our mean-based decomposition validates it as the most effective and well-balanced strategy.

### Appendix D Distribution of neuron activation

Figure: Figure 11: Distributions of the number of overlapping features for models with and without Feature Scaling across various expert activation and sparsity ($L_{0}$) models.
Refer to caption: tex/images/legend_4.png

To provide a more granular analysis of feature similarity, this section supplements Section [3.3.2](https://arxiv.org/html/2511.05745v1#S3.SS3.SSS2) by examining the distribution of the number of overlapping features, $k_{i,j}$, between token pairs. The experimental setup remains consistent, with a total of 64 experts, a varying number of activated experts ($e\in\{1,2,4,8,16\}$), and varying sparsity ($L_{0}\in\{2,4,\dots,128\}$).

The distributions of $k_{i,j}$ for models with and without Feature Scaling are presented in Figure [11](https://arxiv.org/html/2511.05745v1#A4.F11). A consistent bimodal pattern was observed across all scenarios. The primary mode, representing the highest proportion of token pairs, is centered at zero overlap ($k_{i,j}=0$). A smaller, secondary mode represents pairs with a non-zero number of overlapping features. We identified several key findings from these distributions.

First, in the single-expert settings, the distribution is heavily dominated by the zero-overlap mode. This seemingly high diversity is, in fact, an artifact of extreme feature redundancy; semantically similar tokens often activate entirely different sets of redundant features, resulting in low pairwise overlap.
Second, in multi-expert settings, the secondary mode becomes more prominent, particularly as activation density increases ($L_{0}>16$). This indicates that these models learn to represent similar tokens using shared, overlapping feature sets, which is a hallmark of effective specialization.
Finally, the application of Feature Scaling consistently shifts this secondary mode to the right. This shift demonstrates that for semantically related token pairs, our mechanism encourages the encoder to activate more diverse and less overlapping combinations of neurons, further enhancing feature diversity.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Could the authors clarify how the FLOPs-matching setup ensures a fair comparison between dense SAEs with 768 hidden units and Scale SAEs with a total of 24,576 dimensions? Is the compute matched per forward pass, per batch, or by total parameter count?

2. Did you observe any side effects of the Feature Scaling mechanism, such as instability, changes in sparsity dynamics, or degraded interpretability for certain expert configurations? Since this mechanism directly modifies encoder weights, a brief discussion of possible unintended effects would be helpful.
soundness: 3
strengths: The paper is well motivated, clearly written, and deeply engages with prior work. The authors correctly identify a central limitation of existing MoE-SAEs and propose two simple, conceptually coherent mechanisms to address it. Both techniques are well defined and integrated cleanly into the SAE framework. The experimental evaluation is thorough, including ablation studies that isolate the contribution of each innovation. The results convincingly demonstrate reductions in feature redundancy and improved interpretability metrics. Overall, the work provides a solid step toward making large-scale sparse autoencoders more computationally feasible for interpretability research.
summary: This paper addresses the problem of feature redundancy and poor expert specialization in Mixture-of-Experts Sparse Autoencoders (MoE-SAEs), which are used to make large language models more interpretable while reducing computational cost. Prior work such as Switch SAEs often suffers from redundant experts learning overlapping features, limiting interpretability and efficiency. The authors propose two key innovations: (1) Multiple Expert Activation, which activates several experts per input and applies a global Top-K sparsity constraint to encourage expert specialization, and (2) Feature Scaling, a learnable high-frequency amplification mechanism that promotes feature diversity and stabilizes training. Experiments on GPT-2 activations show that these methods improve reconstruction error, feature diversity, and automated interpretability scores relative to baseline SAEs. Ablation studies attribute these gains to the proposed mechanisms.
weaknesses: The experiments are limited to GPT-2, which is now a dated architecture. Including results on more recent models such as Gemma or LLaMA would strengthen the empirical claims and test generality.
The FLOPs-matching procedure is not clearly justified. The authors write that “to match the computational load of activating a fixed number of experts, the hidden dimension is set to 768” for dense SAEs, while Scale SAEs use a total hidden dimension of 24,576. It is unclear how this setup maintains computational parity, and a more detailed explanation of this comparison is needed.
Finally, the discussion of automated interpretability results is too brief. The paper would benefit from a few qualitative examples of discovered features or a quantitative measure of feature diversity across experts, to demonstrate that higher automated interpretability scores correspond to genuinely more distinct, human-understandable features.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: From Figure 6(a), adding more than two experts does not further improve specialization—in fact, performance worsens. This seems inconsistent with the stated property. Why is that?


I wonder if adding high-frequency components requires longer training to converge, since it perturbs the optimization direction. Is that correct? Could you also compare training iterations and time?
soundness: 2
strengths: The methodology is explained with precise mathematical notation, and the results are presented with effective visualizations.

The experimental evaluation is thorough and convincing. The use of FLOPS-matched comparisons, multiple datasets (in-domain and cross-domain), and a suite of complementary metrics leaves little doubt about the superiority of the proposed method. The ablation studies and mechanistic analysis are executed to a high standard.
summary: This paper identifies a critical limitation in Mixture-of-Experts Sparse Autoencoders: the failure of experts to specialize, leading to high feature redundancy that undermines both interpretability and performance. The authors propose Scale SAE, a novel framework with two core innovations:
- Instead of routing an input to a single expert, a subset of experts is activated. 
- Encoder weights are decomposed into low-frequency and high-frequency components, and the high-frequency parts are adaptively amplified.

Through extensive experiments on GPT-2, the paper demonstrates that Scale SAE significantly outperforms strong baselines (TopK SAE, Gated SAE, Switch SAE) under a FLOPS-matched paradigm.
weaknesses: > W1. The Mechanistic Rationale for Multiple Expert Activation Requires Deeper Justification. 

To be very honest, activating more than one expert in MoE is standard practice. The difference here is that we select the Top-K experts across all experts.

The paper shows that activating multiple smaller experts outperforms a single larger expert under a FLOPS-matched budget (e.g., 2 experts of size 128 vs. 1 expert of size 256). However, the fundamental reason for this performance boost is not sufficiently explained. A key question remains: is the benefit primarily due to the modularity and finer granularity of the experts, or is it the interaction and joint sparsification across experts that is crucial? For example, consider a 4-expert SAE (activating 2 at a time, each with 128 hidden units) versus a 2-expert SAE (activating 1 at a time, each with 256 hidden units). The computation is identical, so why should reconstruction quality, accuracy, and stability improve? The paper shows results in Figures 3 and 4, but does not clearly explain the underlying logic.

> W2. Insufficient Discussion and Comparison for Feature Scaling. 

Feature Scaling appears primarily as a load-balancing technique, both at the expert and neuron levels. Adding high-frequency components increases directional diversity, but alternative methods exist. 

At the expert level, how does the implicit balancing effect of Feature Scaling compare to more explicit load-balancing losses used in MoE literature?

At the neuron/feature level, orthogonal initialization or updates could achieve similar effects.

These alternatives are not discussed or compared. While the quantitative results in Section 3.3 make sense, the paper lacks a deeper analysis of why this approach is preferable.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - What is happening in Figure 5a)? The title of the charts suggest an L0 of 2 and 32 yet the bar and line charts show different values of L0 on the x axis. Is the x axis actually varying the number of experts?
- In 3.2.1, why would there be diminishing returns to having more than 2 experts active at once? Given that you're not actually using all of the features but are instead using another TopK filter afterwards (so the FLOPs are not increasing) it seems still to explain why 2 experts is markedly better than 1 but 4 isn't seemingly better than 2. 
    - This seems to be somewhat mitigated in 3.2.2 but I would like to know more of why this mitigation ought to work from the authors
- Does Feature Scaling ever amplify noise or cause training instability?
- What are the training and inference time memory implications for having the MoE layer? Is there a way to reduce the footprint of this compared to naive methods?
- Is the idea that the low frequency features are more coarse-grained and high-frequency ones flesh out more complex details? If so is there any evidence for this?
- The abstract states that the approach gives 99% less feature redundancy - where in the paper is this claim justified?
soundness: 3
strengths: - Clear writing style
- Generally readable figures
- Clearly notes the limits of the Switch SAE and how they overcome these
- Useful to see the performance on two different datasets across both the MSE and Loss Recovered metrics
- Interesting exposition when detailing how the two architecture changes help the overall performance of the SAE
- Valuable use of the signal processing literature which is a literature that is not always leveraged in interpretability research (and where interpretability research could likely learn a lot more from)
summary: The authors suggest an improvements to SAEs by using MoEs extending Mudide et al 2025. They suggest that the original approach to MoE SAEs was flawed and had polysemanticity which undermines the interpretability that the approach was intended to provide. Instead of choosing a single expert, they choose multiple experts with their Multiple Expert Activation scheme. They then have a DSP-inspired technique Feature Scaling which amplifies the component of the expert encoder's weights which is far from the mean of the encoder vectors. They refer to this as "amplifying the high-frequency component". With these changes their SAEs outperform TopK SAEs and the Switch SAE. The primary obstacle to performance from previous work that they solve is the problem of feature redundancy. The authors perform ablations to confirm that both of these changes are important in training more performant SAEs.

Overall this is a solid and well argued interpretability paper, though possibly incremental in terms of its impact to the interpretability (and broader ML) community.
weaknesses: - For interpretability researchers reading the authors might want to be careful about using the term "high frequency" without explaining what this means. In this case it seems to be mostly an analogy to the signal processing literature but within the SAE literature a high frequency feature is typically a feature which activates very often which is a quite different concept. Clarifying this would be useful.
  - It's also not totally clear why this analogy is a good analogy - exploring why "high-frequency" is the right term here (possibly with spectral plots or similar) could be valuable to help with readers' intuition. 
- Though the authors talk about computational efficiency being an advantage of their approach, they do not show any charts or tables tracking the computational efficiency of their approach relative to others. Seeing this would be useful to validate that claim. 
- The authors do not evaluate their results on downstream tasks. Doing so (possibly using SAE-Bench or a similar benchmark) would be an improvement to the work demonstrating the downstream usefulness across other metrics. 
- The work is somewhat incremental. Though I believe their results are an improvement over prior methods, it's not clear that the level of improvement that is presented will meaningfully impact the research community. This is not to take away from the interesting methodology, clear presentation and reasonable ablations. 
- All of the experiments are using GPT-2. Having at least one plot in the paper with results from a larger model would show that this method works well on larger models and that it scales well. in particular this is important because one of the main claims of the paper is about efficiency.

---

confidence: 3
contribution: 3
initial_rating: 2
presentation: 2
questions: Have you tried expanding evaluations to other models or families (even small ones like pythia 70m)?

Is fraction of loss recovered the LLM's loss or the SAE's EV?

Have you tried experiments linking the specialisation of features to their interpretability?
soundness: 2
strengths: The introduction of weighted MoE SAEs is novel, and feature scaling is effective as a way to enhance specialisation (which could be tied to interpretability, as an interesting future direction). 
There are many evaluations criteria, and improvements are shown across them. 
The problem of SAE feature interpretability, and the improvement of SAEs are well located problems within the literature.
summary: The paper introduces multi expert SAEs (as opposed to single expert SwitchSAEs). They further introduce feature scaling as a mean of encouraging specialisation and reducing feature redundancy. They run evaluations on gpt2 on fraction of loss recovered, mse, interpretability, and redundancy of features. They show improvements again some competing architectures of SAEs.
weaknesses: The main limitations are:
- evaluations restricted to gpt2
- the improvements on L0 against MSE/loss look incremental
- only.3 architectures (topk/switch/gated) are tested against (more baselines could help, such as matryoshka or jumprelu saes)

I would increase my score if evaluations were successfully (more than incremental improvements) run on more models/families.

Style:
- figure 3 looks cluttered (make all your saes one color and all the baselines another)


Decision: Reject