Title: Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoders
Venue: NeurIPS 2025
Accepted: yes
Keywords: interpretability

mixture of experts
Abstract: Multilayer perceptrons (MLPs) are an integral part of large language models, yet their dense representations render them difficult to understand, edit, and steer. Recent methods learn interpretable approximations via neuron-level sparsity, yet fail to faithfully reconstruct the original mapping--significantly increasing model's next-token cross-entropy loss. In this paper, we advocate for moving to layer-level sparsity to overcome the accuracy trade-off in sparse layer approximation. Under this paradigm, we introduce Mixture of Decoders (MxDs). MxDs generalize MLPs and Gated Linear Units, expanding pre-trained dense layers into tens of thousands of specialized sublayers. Through a flexible form of tensor factorization, each sparsely activating MxD sublayer implements a linear transformation with full-rank weights--preserving the original decoders' expressive capacity even under heavy sparsity. Experimentally, we show that MxDs significantly outperform state-of-the-art methods (e.g., Transcoders) on the sparsity-accuracy frontier in language models with up to 3B parameters. Further evaluations on sparse probing and feature steering demonstrate that MxDs learn similarly specialized features of natural language--opening up a promising new avenue for designing interpretable yet faithful decompositions. Our code is included at: https://github.com/james-oldfield/MxD.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the Mixture of Decoders (MxD) for post-hoc interpretation of MLPs in LLMs. The architecture achieves sparsity by approximating MLP with $N$ sparse transformations (or experts), which is unlike the existing sparse neuron approximations. The paper also proposes the Hadamard weight factorization, which allows for simplified implementation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The idea of sparse experts at the layer level is interesting and distinct from prior neuron-level sparsity formulations. It is nice that the paper also includes discussions about MxD's implementation and extension.\n2. MxD achieves a better sparsity-faithfulness trade-off across four models with sizes ranging from 124M to 3B. It also performs stably across different sparsity levels.\n3. The paper includes both faithfulness and interpretability evaluation of its approach, which is a fair experimental setup.\n\nWeaknesses:\n1. MxD does not show advantages over baselines on interpretability, based on the experimental results of probing and steering. Since the ultimate goal of this line of research is to enable better interpretation of MLPs, this approach does not particularly provide an advancement. On the other hand, the observation that MxD achieves higher faithfulness than existing approaches but only comparable interpretability also points out its technical weakness, i.e., its loss function does not particularly advocate better interpretability.\n2. The role of the \"shared expert\" in enabling the faithfulness gain needs more clarification. A key claim of the paper is that MxD improves reconstruction faithfulness under high sparsity via expert specialization. However, the presence of an emergent shared expert makes it unclear whether the gain stems from true specialization or this general-purpose expert. In an extreme case, it could happen that this shared expert replicates most of the original MLP and thus presents high faithfulness. However, relevant to Weakness 1, there's no particular architecture or loss function design that prevents this situation. \n3. Presentation: while the overall paper writing is clear, I find Fig 1 particularly hard to understand (or not intuitive enough without reading the technical details in the main text). For the neuron-level sparsity figure, the notation of $N$ corresponds to $H$ in Eq (2). It may be better to use the notations more consistently across the paper.\n\nQUESTIONS:\n1. Can the authors clarify how MxD can be used as a better interpretation tool by future research?\n2. Can the authors clarify the role of the shared expert?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an inherently interpretable method that complements existing neuro-sparsity based methods, e.g. sparse overcomplete autoencoders (SAEs) and Transcoders, by imposing sparsity constrains at the layer level.  \n\nMore specifically, proposed Mixtture of Decoders (MxD) method, replaces MLP components present in standard LLMs by an autoencoder where the decoder part is defined as a sparse mixture of decoders who match in number of parameters/capacity the original MLP it aims to replace. In doing so the proposed method is capable of learning specialized expert (decoder) branches that are interpretable.\n\nExperiments on 4 LLMs (~3B parameters), i.e. GPT2-124M [3], Pythia-410m, 187 Pythia-1.4b [41], and Llama-3.2-3B [1], show that the proposed method are comparable with the state of the art methods (SAEs and Transcoders) while having a performance that is significantly robust to varying levels of sparsity.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- To a good extent the manuscript has a very clear structure and the way content is presented, to a good extent, provides a very good and clear flow in the presentation of content.\n    \n- The proposed method seems to provide comparable if not superior performance to that of state of the art methods.\n    \n- The theoretical proofs put in place show, the proposed method has the capability to provide the desired sparsity properties without sacrificing information in the process, e.g. as is the case with SAEs which lose some information due to imperfect reconstruction processes.\n    \n- Several resources are released in order to ease the testing of the proposed method.\n    \n\n**Weaknesses**\n\n- The proposed evaluation seems to equate equal predictive performance to “faithfulness” to the original (standard MLP) model.\n    \n- Some of the adopted evaluation procedures (model faithfulness and feature steering) do not seem to be adequate.\n    \n- Some relevant parts of the manuscript have been delegated to the appendix.\n\nQUESTIONS:\n- As indicated in the manuscript the proposed methods introduces the capability of learning a representation among a mixture of sparse decoders. In this regard the training process of the proposed method is significantly more complex that of the considered competitors (SAEs and TCs). May you provide some details on how the training processes of these methods compare w.r.t. each other?\n    \n- In L. 196-198, it i stated that there is the expectation that the proposed MxDs are capable of scaling to significantly larger models. May you provide some theoretical insights or perhaps an intuition of why this would be the case? The manuscript would benefit from having this pointer further\n    \n    elaborated.\n    \n- In l.209-210, it is stated that [17,19] find sparse solutions to be more interpretable. Are there any guarantees, or perhaps clear indicators, that the findings from those articles are transferable to the proposed setting?\n    \n- In Sec. 3.2.2, two LLMs are used as gold standard for measuring properties of the output produced by feature steering. Since these are models that are not perfect, is there is pointer to existing work showing the validity of this practice/protocol?\n\nLIMITATIONS:\nIn addition to the points listed as weaknesses I would stress the following aspects:\n\n- In the related work section (Sec. 4), the aspect related to Sparse Decomposition, i.e. the core direction of this paper, limits itself to listing some existing efforts. This section would benefit with an explicit positioning on how the proposed method compares w.r.t. existing efforts.\n    \n    In addition, The proposed method seems to bear resemblance to the work from [Pearce et al, 2025], [Dooms et al., 2025] [Adler et al., 2025] (please see below for complete references). These works do not seem to be discussed in the manuscript.\n    \n- One of the metrics used for the evaluation of the proposed method is “faithfulness” to the base model composed by standard MLPs. In this regard, the evaluation seems to be equating comparable predictive performance with faithfulness. While there should be the requirement to meet (or improve) the performance already achieved by the original model, meeting this requirement does not guarantee that new model will learn the same representation as that learned by the original.\n    \n- Empirical evidence, e.g reconstruction result (Fig 10), output space similarity (Sec. B.1), etc., used to support statements made in the paper are actually  delegated to the Appendix. This gives the impression of the paper not being self-contained. A better balance needs to be achieved here as to ensure statements made are paired with proper evidence (or pointers to the literature).\n    \n- In Sec. 3.2.2, two LLMs are used as gold standard for measuring properties of the output produced by feature steering. Since these are models that are not perfect, this casts doubts on the observations made from this results.\n    \n\n**References**\n\n- Pearce et al. \"Bilinear MLPs Enable Weight-based Mechanistic Interpretability\" ICLR 2025.\n    \n- Dooms et al. \"Compositionality Unlocks Deep Interpretable Models\" AAAI 2025 ColoRAI Workshop\n    \n- Adler et al, \"Towards Combinatorial Interpretability of Neural Computation\" arXiv:2504.08842", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to learn an interpretable approximation of LLMs via sparse learning, but at the layer level rather than previously studied neuron level. The method replaces the computation in Transformer MLPs with a k-sparse sum of linear transformations on the hidden intermediate representation, and uses a Hadamard-product based parameterization to reduce the footprint of this sparse replacement. Through a special hadamard product trick, the mixture of linear transformations becomes much more computationally feasible, and experimental results show much closer performance to baseline models, while preserving interpretability capabilities.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. This paper presents clear experimental evidence for its claims, and presents numerous additional experiments for completeness in the appendix, like removing the shared expert or finding dead experts. \n2. The utility of the induced sparsity is well tested with probing experiments as well as steering experiments. Experimental results seem consistent with popular prior work on interpretability. \n3. The faithfulness results as measured by loss are far better than prior work in preserving original model performance. This helps address a very core issue in interpretability work where sparse approximations greatly underperform the corresponding LLM they intend to approximate. \n4. The Hadamard product trick is interesting and may be extensible to other applications outside of this paper, and seems to be a key insight to make this work feasible given the complexity of a naive implementation. \n\nWeaknesses\n1. The equivalence between sparse neurons from prior work and a sparse expert from this work is not clear to me as the same level of decomposition. While the steering experiments show that there is decomposition that can be steered, and probing experiments that show that different $a_n$ coefficients activate with different topics, the granularity seems quite different beyond these evaluations. In fact, even addressing potential limitations of this larger unit of sparsity would be helpful to situate this work in the context of prior work looking at the sparse neuron level. \n2. Some key details in the paper could be clarified for easier understanding; for example, more explanation of the Hadamard trick would be helpful as the explanation is quite dense and it takes some convincing of oneself as the reader that this works, some training details may be missing for how MxDs actually substituted and trained using OpenWebText, and some details about top-k activation training might also be necessary (despite it coming from prior work, it is important enough to mention its role in this work).\n\nQUESTIONS:\n1. Can you provide some high level justification for how we can think of the equivalence between a single expert in MxD and a single neuron in SAE as the core unit of decomposition? \n2. Despite efficiency of the sparse method not being the core importance in interpretability work, and despite the Hadamard trick improving feasibility, can you comment on the efficiency of this method versus SAE and transcoders?\n3. Lines 197-198 suggest that this work would scale well to larger models, but what is the justification for this claim based on the MxD construction? \n4. How are sparse layers trained? Is it via layer replacement and then training just the layers? Some training details seem glossed over perhaps as a result of being in line with prior work, but these details are important for reproducibility. \n5. The factorized forward pass in equation 5 is critical to understand how equations 3 and 4 relate to each other, but the rank preservation text appears between equations 4 and 5. I might suggest reordering sections 2.4 and 2.3 for reader understanding, but also additional clarity while introducing the parameterization could help.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Mixture of Decoders (MxDs), a novel method that overcomes the accuracy-interpretability trade-off in large language models by using layer-level sparsity to expand dense layers into thousands of specialized sublayers, faithfully preserving the model's performance while learning interpretable features.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1.  **Addresses a Critical, High-Impact Problem:** The paper tackles the core trade-off between performance and interpretability in large language models. This is a central challenge in AI safety, transparency, and alignment, making the research highly relevant and important.\n\n2.  **Clear Conceptual Innovation:** The proposed shift from \"neuron-level\" to \"layer-level\" sparsity is a powerful and intuitive conceptual leap. By arguing that individual neurons are not the right unit of analysis and proposing sublayers instead, the paper offers a compelling new way to think about model decomposition.\n\n3.  **Focus on \"Faithfulness\" and Practicality:** The authors rightly emphasize that an interpretable model is only useful if it faithfully reproduces the behavior of the original. This practical focus on maintaining low next-token loss ensures that the resulting model is not just a simplified, inaccurate curiosity but a potentially viable replacement for the original component.\n\n4.  **Strong Theoretical and Empirical Backing (as claimed):** The paper claims to provide a formal proof that its method preserves the expressive capacity (full-rank weights) of the original layers, lending it strong theoretical credibility. This is supported by claims of extensive and rigorous experiments (\"108 sparse layers in 4 LLMs,\" \"348 sparse probing and steering tasks\") that show it \"pareto-dominates\" existing techniques.\n\n### Potential Weaknesses\n\n1.  **Unaddressed Computational Overhead:** The paper expands a single layer into \"tens of thousands\" of sublayers. While it mentions parameter efficiency, it says nothing about the potential computational costs. This raises key questions about inference latency (how fast is the conditional routing?) and training complexity (how long and how much memory does it take to train these MxD layers?), which are critical for practical adoption.\n\n2.  **Scalability to Frontier Models:** The experiments are conducted on models \"up to 3B parameters.\" While respectable, this is significantly smaller than current state-of-the-art open models (e.g., 70B+ parameters). It's an open question whether the training and memory dynamics of this method will scale effectively to these much larger and more complex systems.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}