Sparse Autoencoders, Again?
Authors: Yin Lu
Xuening Zhu
Tong He
David Wipf
Venue: ICML 2025 poster (2025)
Keywords: autoencoders
sparse representations
variational autoencoders
low-dimensional manifolds
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Is there really much more to say about sparse autoencoders (SAEs)?  Autoencoders in general, and SAEs in particular, represent deep architectures that are capable of modeling low-dimensional latent structure in data. Such structure could reflect, among other things, correlation patterns in large language model activations, or complex natural image manifolds. And yet despite the wide-ranging applicability, there have been relatively few changes to SAEs beyond the original recipe from decades ago, namely, standard deep encoder/decoder layers trained with a classical/deterministic sparse regularizer applied within the latent space. One possible exception is the variational autoencoder (VAE), which adopts a stochastic encoder module capable of producing sparse representations when applied to manifold data. In this work we formalize underappreciated weaknesses with both canonical SAEs, as well as analogous VAEs applied to similar tasks, and propose a hybrid alternative model that circumvents these prior limitations.  In terms of theoretical support, we prove that global minima of our proposed model recover certain forms of structured data spread across a union of manifolds.  Meanwhile, empirical evaluations on synthetic and real-world datasets substantiate the efficacy of our approach in accurately estimating underlying manifold dimensions and producing sparser latent representations without compromising reconstruction error.  In general, we are able to exceed the performance of equivalent-capacity SAEs and VAEs, as well as recent diffusion models where applicable, within domains such as images and language model activation patterns.

Sparse Autoencoders, Again?
Yin Lu Xuening Zhu
Tong He? David Wipf?
Abstract
1. Introduction
Is there really much more to say about sparse au- toencoders (SAEs)? Autoencoders in general, and SAEs in particular, represent deep architectures that are capable of modeling low-dimensional la- tent structure in data. Such structure could re- flect, among other things, correlation patterns in large language model activations, or complex natural image manifolds. And yet despite the wide-ranging applicability, there have been rel- atively few changes to SAFs beyond the origi- nal recipe from decades ago, namely, standard deep encoder/decoder layers trained with classi- cal/deterministic sparse regularizer applied within the latent space. One possible exception the variational autoencoder (VAE), which adopts a stochastic encoder module capable of producing sparse representations when applied to manifold data. In this work we formalize underappreciated weaknesses with both canonical SAEs, as well as analogous VAEs applied to similar tasks, and pro- pose hybrid alternative model that cireumvents these prior limitations. In terms of theoretical sup- port, prove that global minima of our proposed model recover certain forms of structured data spread across a union of manifolds. Meanwhile, empirical evaluations on synthetic and real-world datasets substantiate the efficacy of our approach in accurately estimating underlying manifold di- mensions and producing sparser latent representa- tions without compromising reconstruction error. In general, we are able to exceed the performance of equivalent-capaal recent diffusion models where applicable, within domains such as images and language model acti- vation patterns. link to the code here.
Autoencoders represent a widely-applicable neural network model for obtaining useful low-dimensional representations of data, particularly when training labels are not available (Berahmand et al., 2024; Goodfellow et al., 2016). The ba- sic architecture is simple: An encoder network first projects each observable data sample to a latent representation (usu- ally low-dimensional), after which a decoder network at- tempts sample reconstruction based on this latent representa- tion. Sparse autoencoders (SAE) (Ng, 2011; Ranzato et al., 2006; 2007) adopt an additional degree of design flexibil- ity by accommodating but with the inclusion of an attendant regularization factor applied to the encoder output that encourages sparse repre- sentations (meaning most dimensions are pushed to zero). This capability facilitates adaptive latent representations reflecting complex underlying data structure, whereby the pattern of nonzeros can vary on sample-by-sample basis unlike with regular autoencoder.
digherrdi latent space,
Informally speaking, autoencoders (including SAEs) can be viewed as something like the horseshoe crabs of the deep learning kingdom, having existed for decades (or "cons") with more or less the same hardy conceptual de- sign (Bourlard & Kamp, 1988; Hinton Zemel, 1993; Le- Cun, 1987; Rumelhart al., 1986). For example, the SAE models used for learning interpretable representations of present-day large language model (LLM) activation patterns (Bricken et al., 2023; Cunningham et al., 2024; Chaudhary & Geiger, 2024; Gao et al., 2024) are nearly identical in form to original architectures from long ago; likewise for a variety of other modern use cases (Lan al., 2024; Movva et al., 2025; Pach al., 2025; Stevens al., 2025). One notable exception is the variational autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014), which substitutes the deterministic latent space of traditional autoencoders, with an alternative stochastic representation. Although seemingly distinct from the SAE, it has recently been shown that VAEs can also address sparse autoencoding tasks in certain set- tings (Dai al., 2021). Still, VAE models notwithstanding, given the long-lasting and durable usage of SAES with min- imal changes, it is natural to question the extent to which further innovation is possible on top of the original SAE design space, or if any such innovation is even needed at all.
SAEs and VAEs, as well as
School of Data Science, Fudan University 2 Amazon Web Services. Correspondence to: Yin Lu <yinlu23 mem.ffue wmm.udaa.cdu edu.en>, Xuening Zhu <xueningzhu@ fudan. codlan.ed..com edu.en>, Tong He aitong@amazon.ccom David Wipf <davidwipf@ wwa.:/@@mmaimcmmmmm.mm
Proceedings of the 12nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Sparse Autoencoders, Again?
We answer such questions in the affirmative based the ob- servation that both SAE and VAE designs have notable unre- solved. yet distinct, shortcomings. While details will be de- ferred to Section 2, we summarize here that SAEs generally rely on multiple hyperparameters and potentially-nonconver regularization factors that complicate both training and the interpretation of the latent representations produced by the encoder. In contrast, VAEs benefit from a hyperparameter- free and smoother loss surface, but are presently ill-equipped to produce adaptive sparse representations like the SAE. With this backdrop in mind, our contributions herein are encapsulated as follows:
2.1. Canonical SAE Formulation
Suppose we collect d-dimensional data points x X' C Fdd drawn from some probability measure such that [x w(dx) = 1. Like traditional autoencoders (Bourlard & Kamp, 1988; LeCun, 1987; Rumelhart et al., 1986), an SAE model is composed of an encoder that maps samples x to a latent representation z E R", and a decoder that sub- sequently attempts to reconstruct x from z. These modules are instantiated as functions given by (=(-:()) X - R and ja, :0) : I - Rd respectively, with the goal of learning parameters {6, o} such that x R pz [ (x: o); 0 for all x E X.
What differentiates SAEs from regular auto-encoders (AFs) is that, in addition to seeking an encoder-decoder pairing capable of accurately reconstructing data samples, we also favor latent representations z = [xx(x; Φ) that are sparse (Goodfellow et al., 2016; Ng, 2011; Ranzato et al., 2006; 2007). More formally, we ideally seek ||z||o < (or at least approximately so), where || llo denotes the fo-norm that counts the number of nonzero elements in a vector. To operationalize this preference, SAE models are generally trained with a loss in the form LSAR(d, 0) =
Algorithmie: Although both SAE and VAE models can reconstruct data using sparse encoder representations, we carefully detail their disjoint strengths and weak- nesses in Section 2; subsequently we propose a hybrid VAE-like approach in Section 3 that delivers the best of both worlds. We refer to our model as VAEase, for an easy-to-apply VAE with adaptive sparse estimation.
Theoretical: In Section 4 we rigorously prove that for data residing on a union of now-dimensional manifolds, globally-optimal VAEase solutions can provably recover the underlying manifold structure, adapting to multiple distinct manifolds, unlike existing VAEs. Likewise, we demonstrate simple illustrative settings whereby VA maintains fewer local minima relative to analogous deterministic SAE model.
J 111x- uu un ux: + Anh((e) Ju((x)) A2 12 (I)
variant thereof, subject to p=(x; o) In this ex- pression, h : E + R serves as a penalty designed to upproximate the lg norm while maintaining differentiability almost everywhere for training purposes. Candidates for h include the & norm i((z) = E, |2g| (the tightest convex relaxation to lo norm) or h(z) = Σ, log[|=ı] + +) where is a small constant (Candes et al., 2008; Fazel et al., 2003); there exist many other approximation possibilities as well (Chen et al., 2017; Fan & Li, 2001; Palmer et al., 2006). However, a consequence of all these approximation candidates is the inevitable introduction of a scaling ambi- guity, whereby elements of can be pushed arbitrarily close to zero, while a targeted subset of 0 can be proportionately increased towards infinity to compensate, such that the over- all decoder reconstruction unchanged. The penalty term |@1 in (1) is included to prevent this sort of degeneracy and ensure that h induces non-trivial sparsity; see Appendix A. for further details. A similar effect can be accomplished by adding explicit constraints on the norm of 0 during training (Cunningham et al., 2024) or allowing only the top & values of to be nonzero (Gao et al., 2024; Makhzani & Frey, 2013), although these approaches are effectively substitut- ing one form of hyperparameter for another. And finally, we note that \ and x2 in (1) are non-negative trade-off pa- rameters balancing regularization effects and reconstruction error.
Finally, we present complementary exper- iments in Section 5 that show VAEase outperforms ex- isting SAE and VAE architectures, as well as related diffusion models, across tasks such as estimating mani- fold dimensions or maximally sparse representations of large language model activations.
Empirical:
We conclude this section by emphasizing that, although our proposed approach explicitly relies on variational chassis, the work itself as a whole is not directly related to generative modeling per se, the original purpose for which VAEs were designed. Instead, we are repurposing specific VAE attributes for new sparse autoencoding tasks.
2. Basics of Sparse Autoencoding
In this section we will first present a broad SAE formulation covering typical use cases. Later, we will demonstrate how VAEs can be leveraged for similar purposes, before con- cluding with a brief contextualization w.r.t. related diffusion models, as have recently been applied to learning manifold dimensions.
Wee remark that the range of fay need not be X' exactly, and perfect reconstructions will not necessarily be feasible at all times.
2

---

Sparse Autoencoders, Again?
2.2. Alternative Variational Routes to SAE
M1
M2
Although originally proposed as form of deep genera- tive model for drawing new synthetic samples that approxi- mately follow the ground-truth measure w, it has recently been shown that VAEs (Kingma & Welling, 2014; Rezende et al., 2014) can actually be repurposed to handle sparse au- toencoding tasks (Dai et al., 2021). Broadly speaking, VAE models can be viewed as stochastic autoencoders, whereby the corresponding encoder and decoder modules now define distributions over and sample reconstructions respectively. For models of continuous data (our focus), these distribu- tions are commonly expressed as
X1 X9
41
7/2
Dark Aclptive Sparse Representation collos Frad Sparse
Figure 1: Fixed vs. adaptive sparsity. Depicted are two manifolds [M) with d1] = dim[M2]] = 2, and data points X1 E .M and X2 E 2 Adaptive sparse representations require only two informative/active latent dimensions for either x, X2 (the dark colored elements of {z1, which vary across samples). Meanwhile, fixed sparse representations involve shared active dimensions across all samples (the union of dark and light elements).
Me [x[x: 2) diag (x; {x(2)x) = N(z
o))²²
(x(x = N((x Ua [2: 0), TT)
(2)
where the functions μ4, and ptg now mean networks, while Ø= (upon squaring) defines corre- sponding variance network. In accordance with typical use cases, the decoder variance is defined by a single pa- rameter Y > O, which may be trained along with other parameters. The corresponding VAE objective given by LvAE(0,0)) :-
encode-decoder
SAE Advantages: After suitable training, the SAE for- mulation from (1) is quite powerful, capable of producing sparse representations of input samples drawn from w. In particular, a trained SAE can exhibit what is known as adap- tive or dynamic sparsity (Ponti & Martins, 2024), whereby the support pattern (meaning location of nonzero elements) of a given latent code z can vary for different input sam- ples x. See Figure for an illustration. This flexibility is critical because, as we will further detail in Section 4, it allows the SAE encoder to differentiate samples that lie on distinct manifold structures (as frequently encountered in practice (Brown et al., 2022)), expand and contract the number of nonzero latent dimensions in accordance with sample-specific manifold complexity. As a representative example, the images of MNIST hand-written digits '1' may generally require fewer degrees-of- freedom than an observably more complex digits '8',
/ { (Ego(ann)) [log po (x|2)] KLL [go (z)x)) (o((xx)
(3)
with p(z) N(zz00, I); the first term above represents a reconstruction factor analogous to the first term in (1), while the latter KL term enacts VAE-specific regularization. Notably, for any encoder distribution qo (z)x), it follows that L Φ) 2 SIx log pe (x)w(dz). with equality iff go (z|x) = pe (z) x) almost surely. Using simple reparam- eterization trick, it is possible to directly minimize (3) using SGD (Kingma & Welling, 2014; Rezende et al., 2014).
The stochastic VAE design obscures its ability to behave as a sparse autoencoder, and indeed, the mechanism for pruning unneeded dimensions of the latent space is dis- tinet from canonical SAEs. While we defer details to prior work, the crux of the idea is that superfluous dimensions of z are flooded with uninformative white noise that is sub- sequently filtered by the decoder to avoid compromising accurate reconstructions. Mathematically, we have that qo (0(2)) [x) = (2 |0, 1) for unneeded dimension j. which in the sequel we will refer to as an inactive dimension. This is in contrast with a so-called active dimension j' used to support reconstruction, which is characterized by qo(2y) /x) = (z.llu Φ) O())). Because under rele- vant circumstances the optimal decoder variance will satisfy 7 + 0 during training, there exists nearly deterministic pathway for producing arbitrarily accurate VAE reconstruc- tions using these active dimensions (Dai et al., 2021; Zheng et al., 2022).
offrres-o--freedo to reconstruct
SAE Disadvantages: On the downside, minimizers of (1) will typically be sensitive to the selection of a and A2, as well as the specific curvature of h. And for the latter, tighter approximations to the lo norm are relatively more difficult to optimize, with potentially numerous suboptimal local minimizers introduced in tandem with highly noncon- vex penalties. Meanwhile, looser approximations such as the& & norm, although easier to minimize, may not gener- ate optimally sparse latent representations. Hence for a given reconstruction error, ||z||o may be significantly larger compromising interpretability. Appendix A.3 provides an illustrative example of this contrast.
3

---

Sparse Autoencoders, Again?
SAE
VAE
VAEase
if inactivee active.
[W 1) (W O(+))
if inacttive if acttvve
[ O(r))
if innactive, if activve
(1))
(N O(y))
0.5
-03
-08
red & green cells are easy to distinguish by decoder
red green cells are hard to distinguish by decoder
red green cells are easy to distinguish by decoder
Figure 2: Comparison among methods. On the left SAE active (green) and inactive (red) dimensions are easily identifiable by the decoder, so accurate reconstructions are not disrupted when the locations of the nonzero elements adaptively shift from input sample-to-sampe sample.. Meanwhile in the middle figure, VAE active and inactive dimensions look similar for any given sample, which favors the VAE simply learning a fixed sparsity pattern such that the decoder knows which dimensions are active. And finally, on the right the -AEase-specific reparameterized representation ž mimics the original SAE behavior (as 7 + 0 near global optima), restoring the ability to adapt the sparsity pattern for each input sample. In this way, VAEase is more flexible in steering away from the fixed-sparsity representations favored by the vanilla VAE.
Since y can be learned along with other parameters as part of the overall probabilistic model, in principle sensitive hyperparameter tuning is not necessary. Additionally, if fixed sparsity is sufficient, then VAE models are potentially superior to SABs because they provably have fewer suboptimal local minima in certain settings (Wipf, 2023). This is a desirable artifact of the stochastic VAE encoder, which is uniquely capable of selectively smoothing away bad minima while simultaneously preserving sharp global optimal with maximal sparsity.
VAE Advantages:
(Kingma et al., 2021; Luo). Because of this, DMs are not capable of directly producing sparse latent representations as SAEs and VAEs intrinsically do; they are therefore not direct surrogate for sparse autoencoding tasks in general. However, it has nonetheless been recently demonstrated that using clever post-processing strategies, DMs are in fact capable of quantifying the dimensionality of data ly- ing on low-dimensional manifolds (Horvat & Pfister, 2024; Kamkari et al., 2024; Stanczuk et al., 2024; Tempozyk et al., 2022). So in this sense their ancillary capabilities overlap with SAEs/VAEs as we will briefly explore in Section 5.3.
VAE Disadvantages: The very stochastic smoothing that contributes to VAE advantages is also its Achilles heel. Specifically, whenever given latent dimension j is inactive for one or more data samples, this same dimension is more likely to be inactive for all data samples. This is because, unlike with SAE models, the first VAE decoder layer is generally forced to assign a zero-valued weight to incom- ing inactive dimensions, which are characterized by white noise that would otherwise contribute to high reconstruc- tion errors at the decoder output. Please see Figure 2 for an illustrative example. And once a zero-valued weight is assigned during training, this dimension will remain perma- nently blocked across all input samples. Hence a de facto fixed sparsity paradigm implicitly enforced (see Figure 1), in contradistinction to the more flexible adaptive/dynamic sparsity enjoyed by SAE models discussed previously. This limitation is especially problematic in real-world data comprised of complex manifold structure.
3. Sparse Autoencoding with (VA)
In the previous section we highlighted contrasting strengths of SAE and VAE models: the former enjoys dynamic, input- adaptive sparsity but at the expense of additional hyperpa- rameter and penalty tuning, meanwhile the latter requires no hyperparameters but can only reliably produce fixed sparsity patterns in practice. We now take steps towards achieving the best of both worlds via a novel, yet deceptively simple SAE/VAE hybrid loss formulation.
Our motivation comes from the fact that certain forms of conditional VAE models have in fact been shown to pro- duce adaptive sparsity patterns (Zheng et al., 2022). CVAE models augment the decoder (and possibly the encoder and prior well) to condition on some additional observable y leading to the revised distribution pa (x z, y). From here, peecially-desinned decoder self- attenttion mechanisms can exploit y to selectively turn on and off latent dimensions following an input-specific manner, at least to the extent that y varies in accordance with x.²
2.3. Diffusion Models
We conclude Section 2 by briefly mentioning less direct con- nections between SAEs and diffusion models (DMs) (Ho et al., 2020; Sohl-Dickstein et al., 2015). As powerful gen- erative architectures, DMs can be viewed as a hierarchical VAE, but with a parameter-free encoder that assumes x. = d
But how to proceed more generally in normal circumstances
While it may be possible to achieve something similar without conditioning variables, the learning process considerably more challenging. Please see further discussion in Appendix B.
4

---

Sparse Autoencoders, Again?
without access to helpful conditioning variables? Our in- sight here is two-fold:
Table I: Comparison of notable altributes.
adaptive sparsity
local min smoothing
hyperparm free loss
1. In VAE models, the o network contains information about which dimensions are active, but it does not di- rectly modulate reconstructions themselves as μtz does.
SAE
/
X
VAE
X
/
VAīase
/
/
2. But we cannot just arbitrarily introduce o₂ to the de- coder achieve the adaptive sparsity we are after; this would directly lead to overfitting (see Appendix B.3). Instead, we would like to specifically leverage o= only as a gating mechanism to clamp the noise introduced by inactive dimensions. In this way the decoder weights themselves need not permanently close any particular inactive dimension across all inputs.
of low dimensional manifolds. Hence we choose this set- ting as the starting point of our analysis of VAEase. After formalizing concrete assumptions, we move on to examine the degree which V/Eaaee global minimizers align with ground truth manifold structure. We then conclude by de- scribing a specific scenario whereby the VA objective has provably fewer local minima than its SAE counterpart.
With these considerations in mind, we propose the follow- ing apslicaapee modification to traditional VAE: We replace po [x 2) as defined in (2) with
4.1. Preliminaries
We now briefly lay out precursory definitions pertaining to our later data and modeling assumptions.
po (x x= +)) = N x Mr Pz 0).11.
Definition 4.1 (Data on union of manifolds). Let M == [M,) denote a set of n manifolds M, C Rd, where dim[M.] = 2 0 and there exists a diffeomorphic map- ping 1h M, + R". There is at most one manifold with r; = 0. We then assume data x E X S. . with prob- ability measure w such that a == JM, w(dx) > 0 and Σ a, 1. Finally, we stipulate that Js, w(dx) = 0 for any sub- mo-imolnf S, C M,, satisfying dim < ri.
where :: (1 o= xx Øl)
(4)
and @ denotes the Hadamard product. While seemingly minor change, the consequences are profound in terms of how inactive dimensions are now handled. In partic- ular, if (=]z]xx) x 10 1) for inactive dimension j. then žj Re O, and so the decoder now receives an uninfor- mative but still deterministic input that need not disrupt accurate reconstructions even for a decoder input layer with nonzero weights (analogous to the canonical SAE model). Meanwhile, for an active dimension j' with qo (zy (x) V (x= y), O(7),, we have Σy x (x(xx; Ø) such that transmissible information for reconstructing is not compromised. Figure 2 and Appendix B.3, convey the high- level picture of this phenomena.
Brown al. (2022) motivate the practical relevance of data following the spirit of Definition 4.1, along with the addi- tional assumption that there is no overlap among manifolds. In this sense our definition is actually more flexible and the analyses which follow later in Section 4.2 hold with or without overlapping manifolds. 3 We next turn to a precise definition of the VAEase model class we seek to analyze. Given resurgence of SAE use cases, including
We refer to the proposed VAE modification from above as VAEase and summarize the newly-enable capabilities in Table 1. Critically, VAEase preserves the original proper- ties that make VAE models attractive for sparse autoencod- ing (i.e.. no sensitive hyperparameters required, potential for local minima smoothing via a stochastic latent space), while removing the inflexible VAE favoritism towards fixed- sparsity solutions. Admittedly though, the motivations and claims thus far have been primarily supported by intuition at the expense of rigor; however, the next section we place VABase on sound theoretical footing, with a much closer examination of properties relevant to sparse autoencoding.
Definition 4.2 (Lipschitz VAEase objective). Assume that μ4= and A4r are Lipschitz continuous functions of x X and E E respectively. We then denote L (6, 0) as the resulting objective from (3), but with po (x/z) replaced by pe (x|z, a² [x ø)) as defined in (4).
Lipschitz continuity assumptions for neural network models are quite common (Neyshabur et al., 2017: Bartlett et al., 2017) and are naturally achievable in practice by deep ReLU networks with bounded model weights. For our purposes,
We are able to accommodate this greater flexibility because our results do not ultimately depend on the dimensionality of the tangent spaces at the boundaries of the support sets. Instead we rely on the optimization of integrals over the entire extent of each manifold such that individual manifold dimensions remain identifiable.
4. Comparative Analysis
The notion of adaptive sparsity is particularly relevant when the underlying data distribution is spread across a union
5

---

We now contrast with an analogous SAE model, assum- ing h E H. where H. denotes the set of functions that are Sparse Autoencoders, Again?
this assumption is merely included for mild technical rea- sons related to the adopted proof technique.
global minimizers of CvAE(ó, 0) in the limit y + 0 satisfy
Jma I[IA(x)] + re)w(dxx) = (1)
And finally, before proceeding to our main results we intro- duce two quantities (with origins stemming back to (Zheng et al., 2022)) that will collectively serve as a useful lens for differentiating VAE and VAEase capabilities.
(7)
for one or more constituent manifolds M,-
Critically, this result indicates that VAE models are always capable of adapting the sparsity of their latent rep- resentations to ground-trutie structure involving multiple manifolds. Instead, for reasons detailed in Appendix B.I, VAE models tend to favor solutions whereby
Definition 4.3 (Active dimension). For any fixed >, let {0., }. denote a minimizer of loss LVAEase 8) given by
Definition 4.2. We then specify that a latent dimension j E { {l. x} is active at sample x E X if of (x; Φn), O(7) as 7 - 0. Additionally, adopt A(x:) to denote the set of active dimensions associated with x.
J,, T((A(()) =
=[(AA())]=)))((>>)> rd)((x)) ~ 1,
(8)
The remaining so-called inactive dimensions will satisfy (x (x), =1 1 O(7); as shown in Appendix 1, other possibilitiies will not occur almost surely.
assuming r := E= S min(k, d) (we prove this claim for linear special cases in Appendix E.5), ). As such the VAE can readily estimate the aggregated manifold dimension, gener- ally with fixed set of active dimensions, but not necessarily that of the finer-graine individual manifolds themselves as our proposed VAEase model can. Complementary experi- ments in Section 5 will confirm this performance disparity.
Definition 4.4 (Reconstruction error). Borrowing notation and definitions from above, we define the VAEase recon- struction error as
/ Bus, (x/x))
R :=
[x Mex (z: B,,) w(dx).
(5)
4.3. Local Minima Smoothing
We remark that equivalent active dimension and reconstruc- tion error quantities can be defined for regular VAE models as well, the primary difference being (v 0) replacing L (c, 8) as the optimization objective used for obtain- ing {0., }.}.
Unlike the VAE, with an appropriate selection of h and {Ai, )2). the SAE model from (1) can in principle mimic the behavior of the VAEase global optima explored in Sec- tion 4.2. As a deterministic counterpart, this involves pro- ducing adaptive sparse solutions with negligible reconstruc- tion error and minimal number of nonzero latent dimen- sions aligned with each manifold. For example, we can accomplish this goal with (z) = ||z]|o and suitable choices for {^1, A2) (see Appendix A.2) But SAE adaptive sparsity comes at a cost: addition to the need for tuning (Ai, } and choosing some workable/smoothh approximation the lo norm that allows for SGD) training, the SAE loss poten- tially has much larger constellation of sub-optimal local minimizers. While difficult to explicitly quantify under broad conditions, in this section we introduce a simplified regime whereby an explicit, exponential gap in local minima counts can be established.
4.2. Properties of Global Solutions
We now analyze properties of VAEase global minimizers that are particularly relevant to sparse autoencoding.
Theorem 4.5. Assume data adhering to Definition 4.1 with E; Ts S K. Then as & + O, global solutions to LVALiaue (0, 8) achieve R = =(1) and
J. T((A(x)]=::)ddl wd)x = ai + o(1) Vi. (6)
= r3:))
Note that because a; = -M, (dx) and Σ. ar = 1, (6) indicates that within each data manifold M, the VAEase is almost surely using a number of active dimensions that matches dim [,]] Overall then, this result establishes fun- damental capabilities of the VAEase model: It can achieve arbitrarily good reconstruction error, all while relying ideal active dimension counts that perfectly align with each constitute data manifold. Meanwhile, a corresponding VAE cannot achieve the equivalent of Theorem 4.5.
Theorem 4.7. Assume K = d and decoder μtz pz(2;;0 0) Udiag (i'sglw'ss Z, where U E Bddd is a fixed/nnwwn orthonormal matrix and E IRd are learnable parameters. Moreover, assume arbitrary encoder functions μt: and σ Then the VAFase loss defined w.r.t. an arbitrary input data point x will always have a unique minimum (local or global).*
Corollary 4.6. Consider the VAE loss (vve( 0) defined as in (3) using arbitrary Lipschitz continuous encoder/decoder networks (i.e. equivalent to those adopted by the VAEase). Then there exist datasets adhering to Definition 4. such that
'Technically speaking, there are actually multiple equivalent global minima because the overall loss is invariant to randomly flipping the signs of and w. But this inconsequential ambiguity shared by all SAE and VAE models alike.
6

---

Sparse Autoencoders, Again?
SAE Loss
Heatmap of VAEase Loss
ters for explicitly quantifying model representations. To this end, extend setups inspired from Zheng et al. (2022); Stanczuk et al. (2024) to generate two datasets composed of multiple manifolds. The first involves 500,000 points ran- domly assigned to n = 3 linear subspaces of 4 dimensions each embedded an ambient space with d = 40. The second more challenging case involves n = 4 nonlinear manifolds with dimensions {5, 5, 10, 10} embedded in 100 dimen- sional space; 200,000 samples generated per manifold by passing random noise through different MLP networks.
077
04-
1055 value
0.3
0/1
02
01
00
3.5
1.0
3.5 0.3 0.2 04
066 0.6
1.4
14
Ha
Figure 3: Local minima smoothing example. On the left ID slice of the SAE loss (details in Appendix D.1) has two distinct minima appear (red dots). Mcanwhile, on the right VAEase under identical conditions has single (global) minimum because of the latent smoothing afforded by dz.
Baseline models. To compare against VAEase, we train equivalent VAE baseline well three SAE variants differentiated by the penalty function adopted for h in (1), Following prior work, we select SAF-C1 for the standard { norm as used by Cunningham et al. (2024), SAE-log whereby h(=) = log(I=| + E) ( e > is a small constant) as motivated in Candes al. (2008), and SAE-T; for the strategy of simply choosing the top/largest i elements of (per data sample) as advocated by Makhzani & Frey (2013); Gao et al. (2024). For the linear subspace dataset we adopt linear decoder for all models (as is also used in recent application to modeling the intermediate activation layers of LLMs (Chaudhary & Geiger, 2024)); for the more complex nonlinear dataset we instead stack multiple MLP decoder layers. Note that for both datasets set k, required by SAE-T;, the size of the largest individual manifold.
symmetric about zero, concave and non-decreasing in |z]. Commonly-un used into this category (Chen et al., 2017), which includes the & and lo norms, log-based penalties, and many other choices. Corollary 4.8. Under the same conditions Theorem 4.7, for any {>1, )2) IR2, there exists data samples x such that the SAF loss from (1) has 2 distinct local minimizers."
penalties typically fall
Please see Figure 3 for ID visualization of the contrast between Theorem 4.7 and Corollary 4.8. Collectively, these results suggest that the SAE loss is likely more prone to sub-optimal local minima than VAEase. The latter uniquely benefits from a selective smoothing effect introduced by the stochastic latent space, whereby desirable sparse global so- lutions can be preserved (Theorem 4.5) while at least some distracting local minimizers are smoothed away (Theorem 4.7), ultimately because of the expectation over qo (z|x). This phenomenon has been previously explored within the narrow context of fixed-sparsity VAE models (Wipf, 2023), but never generalized to the much more challenging adaptive sparsity scenario as we have done.
After training, evaluate models as fol- lows, For each manifold, we sample new test points not seen during training and feed them into model encoders. We then use the average og (for VAE models) or average absolute Hz (for SAE models), with the averages taken within each manifold, to estimate the active dimensions required. Ide- ally these averages should correspond with the ground-truth manifold dimension. For all experiments throughout this paper, we separate active from inactive dimensions using simple, intuitive variance criteria: A partitioning threshold is chosen such that the average variance above (active dims) and below the threshold (inactive dims) is minimum. This concept, adapted from Xia et al. (2015), avoids dependency on subjective thresholds that vary from experiment ex- periment. Results are displayed in Table 2, where VAīase clearly outperforms the other baselines. Note that all models produced negligible reconstruction errors (~ 10-3 or less). We also remark that SAE-T,, even when k = 4 on the linear dataset, does not correctly estimate individual manifold di- mensions; this is because the model fails to find a consistent set of active dimensions within each manifold.
Evaluations.
5. Empirical Validation
In this section we empirically compare VAEase against salient baselines on both synthetic and real-world datasets. Full experimental details, including data generation pro- cesses and model settings, are deferred to Appendix D.
5.1. Experiments with Known Ground-Trun -ruth
Dataset generation. Synthetic datasets are ideal in the sense that we have access to ground trrth manifold parame-
If an arbitrary loss C(u) is non increasing in u E R for all u > x' for some constant u', we also treat u = +o0 as a local minima as there is no trajectory leading back to another local minimum without increasing the loss. But this inconsequential caveat only applics when is bounded from above.
5.2. Real-World Data
Image data. As we turn to real-world datasets, it is no longer feasible to expect near-zero reconstruction errors
7

---

Sparse Autoencoders, Again?
Table 2: Synthetic dataset results. AD
# denotes active dims. for My. Closest result to ground (GT) in bold.
ject to a nominal reconstruction error, the idea being that fewer dimensions are inherently more explainable. 6 To this end, using code from (Cunningham et al., 2024) we pass the Pile- dataset (Gao et al., 2020) through a Transformer model from the Pythia Scaling Suite (Biderman et al., 2023) and extract 2,098,177 176 samples from an intermediate activa- tion layer, each with dimensionality of d = 512. And since the dataset is drawn from highly diverse domains including literature, academic papers, and web text, it is reasonable to assume complex underlying data structure. We then trained VAease and baseline models on these data and recorded results in Table 3, where we observe that VAEase simulta- neously achieves the lowest RE and AD values. We also note that the VAE performance is particularly poor w.r.t. AD, And yet in Appendix B.2 we show that even adding an arbi- trarily complex self-attention layer to the otherwise línear decoder cannot fix the core VAE difficulty of identifying high-sparsity codes for LLM activations.
Linear Subspaces
Nonlinear Manifolds
AD
AD₂
ADg
AD
AD2
ADg
AD₃
GT
4
4
4
5
5
10
10
SAB-1
10
11
8
20
26
39
29
SAB-log
8
8
15
28
39
31
SAE-Ia
10
10
7
10
12
18
19
VAE
13
13
17
20
19
VAEase
5
5
11
11
or known ground-truth manifold structure. In this under- determined regime, achieving a target reconstruction error using the fewest active dimensions is the core objective. To this end, we tune SAE hyperparameters such that the reconstruction error roughly matches VAE counterparts and then examine the required number of active dimensions. We apply this approach to MNIST (Deng, 2012) and Fashion- MNIST (Xiao et al., 2017) image datasets, both of which are likely to have rich underlying manifold structure. To better equip models for handling image data, we use ReLU convolutional layers for all encoder and decoder networks. Results are shown in Table 3, where now report the num- ber of active dimensions averaged across all test samples (AD) as well as the reconstruction error (RE) for reference. VAFase achieves strong performance on average as the only method to rank within the top two for all categories. The closest competitor is SAE-Tx; however this approach re- lies explicitly on access to data-dependent estimate of the manifold dimension applied through manual selection of k. In contrast, VAEase operates effectively without any such knowledge, a capability that is particularly valuable when obtaining such estimates are costly, or when there is wide variance across data samples.
Text embedding data. As a final complementary appli- cation domain for evaluating VAEase, we consider text em- bedding data as formed at the output layer of certain LLM architectures. We choose such data because recent work has shown that sparse representations thereof (as obtainable from SAEs) encode rich semantic and causal relationships that are readily applicable to various forms of hypothesis generation (Movva et al., 2025) as well as interpretability studies (O 'eileil et al., 2024). Following (Movva et al., 2025), we generate text embeddings for the Yelp dataset us- ing the ModemBBER Embed model (Nussbaum et al., 2024) resulting in 200,000 samples with dimensionality d = 768. We then compare the sparse representations learned from VAEase training against baseline models (SAE-base) and VAE) with results shown in Table 3. Similar to the LLM intermediate activation data from before, VAEase achieves significantly lower values for both AD and RE as desired. Mcanwhile the VAE exhibits the poorest performance, likely attributable to uniformity in active dimensions across sam- ples. We also remark that SAE-T; did not yield a smaller RE even when afforded higher AD for all L.ee experiments. We hypothesize that fixing the number of ac- tive dimensions (k=30) inherently constrains the model's 's capacity to adequately adapt across samples of varying com- plexity, hindering RE on average.
To further explore VAEase attributes, Figure 4 provides more fine-grained visualization of FashionMNIST results. From the displayed curves we observe that VAEase main- tains a stable reconstruction error as more and more latent dimensions are sequentially masked out. Results on other datasets (not shown) are similar. As an additional side point, Appendix C demonstrates how VAEase active dimensions correlate with MNIST/Fashion image labels.
Language model activation data. We further explore or- thogonal comparisons using a popular large language model (LLM) related use case. Specifically, SAE models have recently been applied to learning sparse reconstructions of the activation patterns produced within intermediate LLM layers for interpretability reasons (Bricken et al., 2023; Cun- ningham et al., 2024; Chaudhary& & Geiger, 2024). In these scenarios there is no explicit ground -tuuth; however, as be- fore the objective to minimize the active dimensions sub-
5.3. Comparison with Diffusion
As mentioned in Section 2.3, through suitable post- processing DMs are in principle capable of estimating man-
'Pursuing LLM explanations is well outside the scope of our work. Instead, we are merely adopting this use case to showcase dif- ferences between VAEase and existing approaches on complex, high-dimensional data. Hence we focus comparing active dimension counts, not downstream interpretations thereof.
8

---

Sparse Autoencoders, Again?
Table 3: Real-world dataset results. RE = reconstruction error; AD = average # active dimensions in test set. Note that SAE-T, R is given a dataset-dependent estimate of the manifold dimension through k, while VAEase requires no such access to prior knowledge and yet still maintains the best overall performance. Top result for each case in bold; second best is underlined.
Table 4: Pseudo-MNIST comparison with diffusion models.
GT
DM (NB)
DM (FLIPD)
VAEase (AD)
16
105.94
169.81
14.93
model. By construction, these new images will necessarily lie on manifold with no more than 16 dimensions, i.c. we now have access to a ground- truth upper bound. From here, we train VAEase and a DM on 500,000 samples and compare their respective estimates. For DM we used two recently-proposed post-processing techniques estimate the manifold dimension: NB (Stanczuk et al., 2024) and FLIPD (Kamkari et al., 2024). Results are shown in Table 4, where VAEase provides far more accurate estimate, being much closer to any possible value in the feasible range between I and 16 than either diffusion approach. We also remark that, because the pseudo-MNIST samples are visually quite similar the originals (see Appendix D.4), it would appear that 16 (or possibly somewhat smaller) is actually reasonable estimate for the true MNIST manifold dimension. This lends additional credence to the VAEase estimate of 16.2 from Table (the same is not true for SAE- Tg since the model was explicitly provided with 16).
MNIST
Fashion
LLM-Act
Text-Emb
RE
AD
RE
AD
RE
AD
RE
AD
SAE-1
10.4
19.1
8.69
19.0
46.2
38.2
0.224
61.9
SAE-log
9.84
18.5
8,95
17.8
45.1
58.2
0.230
56.0
SAE-Tx
10.1
16.0
8.11
15.0
45.1
30.0
0.217
30.0
VAE
9.81
18.0
8.42
18.1
48.0
87.9
0.277
90.0
VAEase
9.71
16.2
8,39
12.4
39.5
22.5
0.187
16.7
WAEase
WAE
Reconstruction Error
SAE ft
13
SAE- lAE-log
12
12-
10-
6. Conclusions
Despite a lengthy and stable existence, we have argued that there is indeed still justification for revisiting the original design space of sparse autoencoders, again. In doing so we proposed a simple alternative architecture called VAEase, based on a novel retooling of the VAE encoder variance as an adaptive sparsity selector. This refinement allows an otherwise rigid, fixed-sparsity VAE latent space, vary support patterns across input samples like canonical SAE, even while retaining a hyperparameter-free energy along with stochastic smoothing of moniily-min solutions. And no additional model parameters, beyond those of a vanilla VAE, are required for implementing VAEase. Given the contemporary resurgence of creative SAE use cases, particularly as related to understanding properties vision and language models (Lan et al., 2024: Pach et al., 2025; Stevens et al., 2025) and beyond (Movva et al., 2025), we anticipate that VAEasee may have widespread applicability moving forward.
12 26 18
20
22
AD mon--)
Figure 4: Reconstruction error (RE) curve as latent dimen- sions are sequentially masked out on FashionMNIST data. VAEase maintains stable/flat RE with the fewest (non- masked) active dimensions, followed by a sharper rise once key informative dimensions start being removed. Note that SAE-T; is not included as its active dimensions are effec- tively fixed by the selection of kt
ifold dimensions (although not a corresponding mapping onto low-dimensional representations as SAE/VAE-basel models do). Ideally, we would like to compare VAEase with diffusion in a real-world context, but cannot rely on joint RE/AD evaluations before, since DMs produce analogous encode-decoder bottleneck reconstruction. More- over, with real-world data we do not generally have ac- cess to ground-truth manifold dimensions that would other- wise facilitate head-to toad-to-hen -tad comparisons. In fact, even on MNIST there considerable ambiguity, with wide-ranging estimates from 10 to over 100 (Pope al., 2021; Tempozyk et al., 2022; Horvat & Pfister, 2024; Stanezuk al., 2024). So instead we develop the following setup that is novel in the context of manifold dimension estimation.
Impact Statement
This paper presents work designed to better understand and enhance the field of sparse autoencoders. While in principle machine learning models of this sort could be used for nefarious purposes, there is nothing specific to our work that stands out as a notable risk factor. And generally speaking, we would argue that by increasing transparency on model behaviors, we at least reduce the risk in inadvertent misuse or misapplication.
First, we train a GAN (Goodfellow et al., 2014) with 16- dimensional input noise on MNIST data. Once trained, we then generate a pseudo-MNNN MNIST dataset by sampling this

---

Sparse Autoencoders, Again?
References
Chen, Y., Ge, D., Wang, M., Wang, Z., Ye, Y., and Yin, H. Strong NP-hardness for sparse optimization with con- cave penalty functions. In International Confernece Machine Learning, 2017.
Ansuini, A., Laio, A., Macke, J. H., and Zoccolan, D. In- trinsic dimension of data representations in deep neural networks, Advances in Neural Information Processing Systems, 32, 2019.
Cunningham, H., Ewart, A., Riggs. L.. Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. International Conference on Learning Representations, 2024.
Bartlett, P. L.,, Foster, D. J., and Telgarsky, M. Spectrally- normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE Transac- lions on Pattern Analysis and Machine Intelligence, 35 (8):1798-1828, 2013.
Dai, B., Wang, Y., Aston, J., Hua, G., and Wipf, D. Connec- tions with robust PCA and the role emergent sparsity in variational autoencoder models. Journal of Machine Learning Research, 19(41): 11141):1-42.. 2018.
Berahmand, K., Daneshfar, F., Salchi, E. S., Li, Y., and Xu, Y. Autoencoders and their applications in machine learning: a survey, Artificial Intelligence Review, 57(2): 28, 2024.
Dai, B., Wenliang, L., and Wipf. D. On the value infinite gradients variational autoencoder models. Advances in Neural Information Processing Systems, 34:7180-7192, 2021.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O' Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., al. Pythia: suite for ana- lyzing large language models across training and scaling. In International Conference Machine Learning, pp. 2397-2430. PMLR, 2023.
Deng, L. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6):11-1142, 2012.
Fan, J. and Li, R. Variable selection via nonconcave penal- ized likelihood and its oracle properties. JASTA, 96(456); 1348-1360, 2001.
Bourlard, H. and Kamp, Y. Auto-association by multilayer perceptrons and singular value decomposition. Biological cybernetics, 59(4):291-29) 1988.
Fazel, M., Hindi, H., and Boyd, S. P. Log-det heuristic for matrix rank minimization with applications to hankel and euclidean distance matrices. Proceedings of the 2003 American Control Conference, 2003., volume 3, pp. 2156-2162. IEEE, 2003.
Brehmer, J. and Cranmer, K. Flows for simultancous mani- fold learning and density estimation. Advances in neural information processing systems, 33:442-455 4533, 2020.
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2, 2023.
Fefferman, C., Mitter, S., and Narayanan, H. Testing the manifold hypothesis. Journal of the American Mathemat- ical Society, 29(4):983 1049, 2016.
Gao, L.,, Biderman, S., Black, S., Golding, I.., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXh::2101.0002 2020.
Brown, B. C., Caterini, A. L.,, Ross, B. L., Cresswell, J. C., and Loaiza-Ganena G. Verifying the union of manifolds hypothesis for image data. arXiv preprint AXXi:2207.00886 2022.
Gao, L., Ia Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scal- ing and evaluating sparse autoencoders. arXiv preprint arXiv:2406.0444 04093, 2024.
Candes, E. J., Wakin, M. B., and Boyd, S. P. Enhancing spar- sity by reweighted & minimization. Journal of Fourier analysis and applications, 14:877-905 2008
Caterini, A. L., Loaiza-Ganem, G., Pleiss, G., and Cun- ningham, J. P. Rectangular flows for manifold learning. Advances in Neural Information Processing Systems, 34: 30228-30241, 2021.
Goodfellow, I., Pouget-Abadic, J., Mirza, M., Xu, B., Warde-Farley. D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. In arXiv preprint 2014.
Chaudhary, M. and Geiger, A. Evaluating open-source sparse autoencoders on disentangling factual knowledge in gpt-2 small. arXiv preprint arXiv:2409. 240.84447. 2024.
Goodfellow, L., Bengio, Y., and Courville, A. Deep Learning. Chapter 14. MIT Press, 2016.
10

---

Sparse Autoencoders, Again?
Hinton, G. and Zemel, R. Autoencoders, minimum de- scription length and helmholtz free energy. Advances in Neural Information Processing Systems, 6, 1993.
O'Neill, C., Ye, C., Iyer, K., and Wu, F. Disentan- gling dense embeddings with sparse autoencoders. arXiv preprint arXiv:24066 00657, 2024.
Ho, J., Jain, A., and Abbeel,
P. Denoising diffusion proba- bilistic models. Advances in Neural Information Process- ing Systems, 23:6840-6851, 2020.
Pach, M., Karthik, S., Bouniot, Q. Belongie, S., and
Akata, Z. Sparse autoencoders learn monosemantic features in vision-language models. arXiv preprint arXiv:2504.00 02821, 2025.
Horvat, C. and Pfister, J.-P. On gauge freedom, conserva- tivity and intrinsic dimensionality estimation in diffusion models. arXiv preprint arXiv:2402.03844 2024.
Palmer. J., Wipf, D., Kreutz- DeeuutzDeeaado, K., and Rao, B. Vari- ational EM algorithms for non- non-Gaussian latent variable models. Advances in Neural Information Processing Sys- tems. pp. 1059-1066, 2006.
Kamkari, H., Ross, B. L., Hosseinzadeh, R., Cresswell, C., and maaza- G. A geometric view of data complexity: Efficient local intrinsic dimension estimation with diffusion models. arXiv preprint arXiv:2406.03537, 2024.
Ponti, E. M. and Martins, A. Dynamic sparsity in machine learning. NeurIPS Tutorial, 2024.
Pope, P., Zhu, C., Abdelkader, A., Goldblum, M., and Gold- stein, T. The intrinsic dimension of images and its impact on learning. arXiv preprint arXiv:2 20i::2/0400884,,, 2021.
Kingma, D. and Welling, M. Auto-encoding variational Bayes. In International Conference on Learning Repre- sentations, 2014.
Ranzato, M., Poultney C., Chopra, S., and LeCun, Y. Effi- cient learning of sparse representations with an energy- based model. Advances in Neural Information Processing Systems, 19, 2006.
Kingma, D., Salimans, T., Poole, B., and Ho, J. Varia- tional diffusion models. Advances in Neural Information Processing Systems, 34:21696-21777 2021.
Ranzato, M., Boureau, Y.-.,, LeCun, Y., et al. Sparse feature learning for deep belief networks. Advances in Neural Information Processing Systems, 20, 2007.
Lan, M., Torr, P., Meek, A., Khakzar, A., Krueger, D., and Barez, F. Sparse autoencoders reveal universal feature spaces across large language models. arXiv preprint 1XXi::441000688.1 2024.
Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep gen- erative models. In International Conference on Machine Learning, 2014.
LeCun, Y. Modèles connexionnistes de I'apprentissage. PhD thesis, Université de Paris VI, 1987.
Luo, C. Understanding diffusion models: unified perspec- tive. arxiv 2022, arXiv preprint arXiv:2208. 19700.
Rumelhart, D., Hinton, G., and Williams, R. Learning in- ternal representations by error propagation. Parallel Dis- tributed Processing: Explorations the Microstructures of Cognition, MIT Press, Vol. 1:318 08--6,,,111 1986.
Makhzani, A. and Frey, B. K-sparse autoencoders. arXiv preprint arXiv: 2013.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International Conference on Machine Learning, pp- 2256-2265. 2015.
Mathieu, E. and Nickel, M. Riemannian continuous normal- izing flows. Advances in Neural Information Processing Systems, 33:2503-2515, 2020.
Movva, R., Peng, K., Garg, N., Kleinberg, J., and Pierson, E. Sparse autoencoders for hypothesis generation. arXiv preprint arXiv:2502.04388) 2025.
Stanezuk, J. P., Batzolis, G., Deveney, T., and Schönlieb, C.-B. Diffusson models encode the intrinsic dimension of data manifolds. In Forty-first International Conference on Machine Learning, 2024.
Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. Exploring generalization in deep learning. Advances in neural information processing systems, 30, 2017.
Stevens, S., Chao, W.-.., Belger T., and Su, Y. Sparse autoencoders for scientifically rigorous interpretation of vision models. arXiv preprint arXiv:25022066755, 2025.
Ng. A. Sparse autoencoder. CS294A Lecture notes, 72 (201 2011.
Tempezyk, P., Michaluk, R., Garncarek, L., Spurek, P., Tabor, J., and Golinski, A. Lidl: Local intrinsic dimen- sion estimation using approximate likelihood. In Inter- national Conference on Machine Learning, pp. 21205- 21231. PMLR, 2022.
Nussbaum, Z., Morris, J. X., Duderstadt, B., and Mulyar, A. Nomic embed: Training a reproducible long context text embedder, 2024.

---

Sparse Autoencoders, Again?
Wipf, D. Marginalization is not marginal: No bad VAE local minima when learning optimal sparse representations. In International Conference Machine Learning, pp. 37108-311 -31122, 2023.
Xia, Y., Cao, X., Wen, F., Hua, G., and Sun, Learning discriminative reconstructions for unsupervised outlier removal. In Proceedings of the IEEE international con- ference computer vision, pp. 1511-1519, 2015.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXhn:1708.07745 2017.
Zheng, Y., He, T., Qiu, Y., and Wipf, D. P. Learning manifold dimensions with conditional variational autoen- coders. Advances in Neural Information Processing Sys- tems, 01134709-3472111 2022.
12

---

Sparse Autoencoders, Again?
A. Further Analysis of Sparse Autoencoding (SAE) Models
In this section we further elaborate relevant SAE modeling details that were omitted from the main text for space considera- tions. Later in Appendix B we provide complementary information regarding VAE models.
A.1. Degeneracies in Deterministic SAE Models
We will now introduce a simplified scenario whereby minima of the SAE loss from (1) can have degenerate minima that undermine the central purpose of sparse autoencoding. Consider the case where the SAE decoder network is given by the linear function Mex zz 0] Wz, with 9 W e _d**, Furthermore, for some input sample x, define
z" == arg min -
(9)
In general x² will not have many or values equal or nearly zero; however, can always fabricate such solution by trivial transformation. Specifically, we can make all elements of z' arbitrarily small by the rescaling z' az" R 0, while maintaining equivalent reconstruction error of x via the revised weights W + a al In this way we can reduce any smooth penalty factor hz), with minimum at zero, to an arbitrarily small value just by rescaling. But of course this defeats the whole purpose of including a sparse penalty in the first place. is precisely because of this possibility that we must introduce ||0||3 into (I); by including this factor we limit the compensatory growth = such that can produce non non-trivial sparse solutions as desired.
While this particular scenario with linear decoder is chosen for transparency, it nonetheless illustrates broader complication with this type of model. Even if we were to deepen the decoder network by stacking additional nonlinear layers, the same scaling ambiguity will generally exist unless some form of restriction on 0 is included.
One exception to this rule is when h is replaced by top-k constraint (Gao et al., 2024; Makhzani & Frey, 2013) as mentioned in Section 2.1 I. Although this substitution mitigates scaling degeneracies, introduces new complications. First, the top-k constraint produces discontinuities in the loss surface that may interfere with timely convergence of the learning process. But secondly, and perhaps more importantly, forces the latent representation of every input x to have exactly the same number of degrees-of of-freedom (i.e., k),, regardless of varying levels of input complexity.
A.2. Maximal Sparsity Using an Infeasible SAE Model
As point of reference, is worthwhile to consider an SAE model capable of idcalized adaptive sparsity akin to VAEase.
To this end, consider the SAE loss from (1) actualized with h(z)) ||z||o and )a O. In this case have
{||x Mu [2:0]] 2 + Jallzloow(ass = J {211x Mrr [2: + 1| ju((xx)
CSAE (0,
(10)
subject (x;(=;;(0) Φ). In the limit as A becomes small, minimizing (SAE(d, 0) in this form is asymptotically equivalent solving the constrained problem
/ [|z|lo wxxx) s.t. x (z[z; z (x(x;)) Vx e X.
min 0.0
(11)
In this way, the SAE model can be designed to enforce perfect reconstructions using the minimal number of nonzero latent dimensions. Such representations will generally correspond with the ground-truth manifold dimensions for data drawn according to Definition 4.1 along with Lipschitz continuous encoder and decoder networks μæ and μ While conceptually notable, of course minimizing (11) is neither differentiable nor feasible to optimize, unlike our VAEase approach. We visualize SAE approximate solutions in Appendix A.3 below.
13

---

Sparse Autoencoders, Again?
A.3. Approximate Sparsity Using SAE-11 and SAE-log Models
INCcuate7
-00
-24
-22 10g01
-2
-152 -22.
-10.3
-3.5
-5.0
-2.5
3.0
2.5
knjlal
Figure 5: Because the ideal Eo norm cannot be optimized via SGD, some form of smooth approximation is required by SAE models in practice. Here we show histograms of 2|| values (on a log scale) for SAE-(1 (left) and SAE-log (right) MNIST data. Because the weaker convex & very loosely approximates the to norm, there is no clear-cut partitioning between active (large values) and inactive (very small) dimensions. In contrast, the non-convex log-based penalty creates a clear separation. This separation readily detectable by the variance criteria descrihed Section as adopted in our experiments (applied in log-space).
B. Further Analysis of VAE-based Models
This section provides more intuitive explanations for why VAEs struggle with adaptive sparsity, while VAEase does not.
B.1. Why Fixed Sparsity in Existing VAE models
Linear Decoder Case: We illustrate the VAE tendency towards fixed sparsity using linear decoder mean network given by Me z; = Wz, with f = W E Rxx analogous to the setup from Appendix A.1 although now reapplied to new context. Granting this assumption and an arbitrary input sample x, the VAE reconstruction loss will satisfy
0 (x: +)
-E Egg(a [log (og
||||-- Wz [2]]
-|| x Wx,
w w
(12)
7
where Wij denotes the j-th column of W and the first equivalence ignores constant factors independent of W. As mentioned in Section 2.2 and proven in prior work, for inactive dimensions we have qu(zylx) [[=z(0, 1) which implies (x: Φ)5 se 1. Hence the only way avoid divergent reconstruction loss y 0 is to have w.j for inactive dimensions. In contrast, for an active dimension j', recall that (0 (x) = (1z (x: o)) O(1)). And so from (12) the factor
(x; 6)
0(7)
W I
wl 11
(13)
7
will remain well-behaved even for Mrjr * O. And then of course since W is shared across all x E X, zero- and nonzero-valued channels (i.e., columns of W) will remain permanently so for all inputs leading to fixed sparsity solution.
Nonlinear Decoder Case: When we generalize to nonlinear decoders, the problem raised above largely persists, although the situation becomes more nuanced. The added wrinkle in the nonlinear case that a sufficiently complex decoder could in principle learn a form of partial adaptive sparsity pattern unlike in the linear setting, although with notable caveats that undermine practical realizations. Why is there such gap between principle and practice here? To answer this question, it helps to contrast the core challenge facing an SAE decoder versus VAE decoder.
Given a well-trained SAE model with suitable latent-space regularization, we may assume that 14, (x; Φ) is sparse vector in R= This implies that inactive dimensions can be trivially identified by simply checking which entries are equal to zero (or at least nearly so vis-à-vis a small threshold). Now consider a corresponding VAE model producing a latent representation with the exact same pz (x; Φ). now serving as posterior mean network, along with the posterior xx () variance satisfying
0, active dim
ax(x; 5) ~
(14)
1, inactive dim
14

---

Sparse Autoencoders, Again?
In this revised situation, what were once zero-valued inactive dimensions of are now filled with samples from ((0, 1). and are thus much less distinguishable from the active dimensions. Figure 2 in the main text provides toy illustration of this contrast. Because of this ambiguity, with limited capacity in the linear decoder setting from above, the only sure-fire way to differentiate active and inactive dimensions is to permanently encode a shared partition by zeroing out columns of W. But in the nonlinear case there is another subtle option: One or more dimensions of z can be permanently designated as specialized active dimensions (just as in the linear case) but with an expanded, two-fold role that accommodates adaptive sparsity across the remaining decoder dimensions.
Specifically,, to instantiate adaptive sparsity within a standard VAE architecture, there must exist a nonlinear decoder with p 2 1 permanently active dimensions; the remaining p dimensions can then reflect adaptive sparsity assuming suitable decoder capacity. Of these / permanently active dimensions, their role is two-fold:
1. Assist with reconstructions, í.e., the typical role of any ordinary active dimension, and
2. Designate which of the remaining x- p dimensions are active.
The latter is critical for instantiating adaptive spansity, since otherwise it is not possible for the decoder to know with certainty which dimensions are active on an input-by-input basis. But of course actually implementing a decoder capable of inducing specialized dimensions that simultaneously fulfill the dual roles from above is quite challenging, and requires significant additional complexity, even if the data manifold(s) upon which x E X' rest relatively simple, e.g., union of linear subspaces. And once such complexity is introduced, the risk of the VAE overfitting to finite-sample training data increases significantly. This is because, when granted sufficient decoder capacity, a VAE model applied to a finite sample training set can achieve perfect reconstructions using only a single active dimension, regardless of the true manifold structure (Dai et al., 2018) [Theorem 5]. Or stated differently, if VAE decoder capacity is expanded to instantiate adaptive sparsity, then this same capacity can hijacked to trivially overfit to finite sample datasets even when the underlying manifold structure relatively simple. We discuss how the VAEase architecture avoids this conundrum in Appendix B.3.
B.2. Can Self-Attention Boost Linear Cases?
In Appendix B.1
we demonstrated why VAEs with the linear decoder μtz [z: 0] Wz will necessarily lead to fixed-sparsity representations. This is the same decoder assumed by recent models of LLM activation patterns adopted for interpretability purposes (Cunningham et al., 2024). Therefore things presently stand, VAE is not suitable for handling LLM activation data where adaptive sparse representations are required (indeed this explains the poor VAE performance on the LLM data from Table 3. However, it is still reasonable to ask, what if we were to modify the VAE encoder to
P 2: 6 Wr(z),
(15)
where J 1 + R is an arbitrary parameterized function, e.g., a self-attention layer capable of learning adaptive sparse combinations of the columns of W. In this way we could presumably retain the interpretability provided by the original linear decoder basis vectors, while avoiding rigid, fixed-sparsity solutions. Unfortunately though, this approach will be ineffectual because there can exist an infinite space of VAE global minimizers whereby r(z) is not sparse. The reason is that VAE sparsity is induced by KL regularization as applied to the posterior mean and variance of z, but operates solely within the decoder where no regularization effect of any kind exists. Hence even if the informative dimensions of z are sparse, T r(z) can map them to arbitrary non-sparse representation that nonetheless minimizes the reconstruction error and globally minimizes the overall VAE loss. And provided that W is overcomplete, which it is by design for LLM interpretability applications, it will necessarily have a non-trivial null space such that infinite solutions with zero reconstruction error are possible.
B.3. How VAEase Circumvents Fixed Sparsity
To provide more intuition into how VAFase avoids producing fixed sparsity patterns, we extend our examination of the linear decoder setting from Appendix B.I. Under these circumstances, the VAEase version of (12) is modified to
(x 0)3 a2 (x; |
(xxx) [log Pe (x|2)] = -+|x-- -1|x-1/,, (x,
(16)
7
j=1
15

---

Sparse Autoencoders, Again?
From this expression we observe that the 0² o deeeendeent regularization factor can now be minimized when either o (x; +) 1 for an inactive dimension, or (x; o)} + for an active dimension. This allows all columns of W to potentially remain nonzero which then accommodates aaputtadattive sparse solutions. This same strategy naturaly transitions seamlessly to nonlinear decoders as well, since inactive dimensions will automatically compress noise to zero regardless of the decoder structure.
Before concluding this section, it is worth considering possible alternative way of achieving adaptive sparsity. Specifically, suppose instead of the VA technique of swapping 2 for z, we simply concatenate o, along with z to form the decoder :). As it turns out though, this strategy is not viable, since now the decoder can just learn to function fz [z; a= (x: 6)]
complete ignore z and use o, directly for input reconstruction. This will incur only a O(1) penalty cost from the KL loss term, while allowing for negligible reconstruction error as measured by (5), provided K sufficiently large. The associated log y normalization factor (as shared by VAE models) will then tend towards minus infinity, pushing the overall loss towards minus infinity. Moreover, the KL loss applied in this way has favoritism towards sparse solutions, and so the whole enterprise will not be successful in aligning with manifold structure.
C. Exploring Data Label Alignment with VAEase Active Dimensions
In following manifold hypothesis for high-dimensional data (Bengio et al., 2013), numerous studies have focused on estimating manifold dimensions (Ansuini al., 2019; Brehmer & Cranmer, 2020; Mathieu Nickel, 2020; Caterini et al., 2021). Along these lines it has been pointed out that the intrinsic dimension of dataset (akin to the manifold dimension under the manifold assumption), rather than the ambient dimension, plays a more crucial role in model fitting (Fefferman et al., 2016; Pope et al., 2021). Subsequently, a union of manifolds hypothesis has been introduced to extend the original manifold hypothesis (Brown et al., 2022); this extension aligns more closely with intuitions regarding real-world labeled datasets. Moreover, companion studies indicate that local intrinsic dimension of data points may vary across datasets, naturally so for points belonging to different classes.
Conceptually, data points with the same label likely exhibit inherent consistency in associated features, although assuming that their sample set forms single manifold is unlikely to hold in general. Thus, following (Brown al., 2022), we use the VAEase models trained in Section 5.2 to estimate the active dimensions of the MNIST and FashionMNIST datasets, and group the data by labels. We then compute the average differences in active dimensions within and between groups, with the results presented in Table 5. The difference is computed as |.4 U B| - | n B on two sets A and B.
Table Average differences in VAßase active dimensions within and between groups defined by class labels.
datasets
Intra-class
Inter-class
MNIST
0.3091
0.4861
FashionMNIST 0.6761
2.1143
The result in the Table 5 suggests that the labels in MNIST bear some significance w.r.t. partitioning of the manifolds; even more so for FashionMNIST. This is despite the fact that model training is completely independent of the class labels themselves.
D. Experimental Settings
D.1. 1D Local Minima Plots from Figure 3
For SAE, we instantiate the energy from ((1) using X = > = 0.1 and h(2) log[[=| + c) where < = 1e - 4. For VAFase we fix 7 = 0.1. We treat as a free parameter for both models; likewise for a₂ as required by VAEase. For the decoders we adopt wz, where w is a learnable parameter that has been optimized out of both SAE and VAEase models prior to plotting the figures.
D.2. Synthetic Datasets
Data generation. For the MLP dataset, the data in each manifold are mapped by a MLP containing 2 linear layers with activation leaky-ReLU. Specifically, the imput dimension of the MLPs are r; aligned to the respective manifold dimensions stated in main text. The dimensions of hidden layers follow the same r;. The output dimensions are d same as the ambient
16

---

Sparse Autoencoders, Again?
dimension. The slope of leaky-ReI.U is 0.2.
Model architectures and training. For the linear dataset, the encoders contain 4 linear layers connected by the activation function Swish, and the decoders are single linear layer. The hidden dimension in encoders is set as 2k. For the MLP dataset, the encoders contain 3 residual blocks and each block consists 3 linear layers, and the decoders are 2 layer MLPs activated by leakyReLU with slope 0.2. The hidden dimension of residual blocks is 8k. Models were trained for 1 150 epochs on linear dataset and 310 epochs on the MLP dataset. The batch size was set to 1024. We choose k. = 20 for the linear dataset and k = 60 for the MLP dataset. The learning rate for VAE models was 0.01 on linear dataset and 0.005 on the MLP dataset, while the learning rates for SAE models are 0.002 on linear dataset and 0.005 on the MLP dataset. The optimizer Adam and learning rate scheduler is CosineAnnealinggWrrmie WarmRestarts with To 10. The penalty weights for SAE models in (1) are (>1 le 3, X2 le- 4] on linear dataset. On the MI.P dataset, the weights are (A1 5e 4, 4, = 1e -5] and {A1 5e - A2 Ie- 5] for SAE-(, and SAE-log models. These hyperparameters are chosen to produce sufficiently small reconstruction errors 1e 3) while still enforcing sparsity as needed to facilitate head-to-head comparison with VAEs (which also produce negligible reconstruction error without any loss tuning parameters, i.e., 7 is learned during training).
D.3. Real-World Datasets
MNIST and FashionMNIIST. Model encoders for the MNIST dataset contain input convolution layer and 2 residual blocks, where each block has 2 convolution layers. A is 32. The decoders consist of I DenseNN, 1 upsampling layer, 2 residual blocks and a convolution layer. The residual block here also has 2 convolution layers. Models were trained for 300 epochs in the MNIST dataset using batch size 2048. Learning rates are 0.05. The optimizer is Adam and learning
rate scheduler is CosineAnnealing WarmRestarts with Tu = 20. The penalty weights are {A1 = 5e 1, 1e 1] and (>1 le 2, 5e- 2) for SAE-f; and SAE-log models on MNIST dataset, {>1 2 2, 2e - 2] and
{λi 1.3e- 2, A2 = 2e- 23 on FashionMNIST dataset correspondingly. Additionally, the estimated |a| values follow a heavy-tailed distribution as shown in Figure 5, so it is more reasonable to apply the variance criteria w.r.t. log |2| for dividing
active and inactive dimensions.
LLM intermediate-layer activations. As for data, online code is available for gathering the intermediate activation layers.7. 7, For model architectures, the encoders for LLM dataset consist of only one linear layer and a ReLU layer, and the decoders are simply linear transforms consistent with popular use cases (Chaudhary & Geiger, 2024). For VAE models, there is also linear layer as needed to output o. Models were trained for 35 epochs with batch-size 2048. The learning rate is 0.005 and is 300. The learning rate scheduler is CosineAnnealing! Wiw.amaaaainnannnettarrrr WarmRestarts with Tn = 5. The penalty weights on this dataset are {} 3.2e İ2 = 2e 1} and {Λ1 = 1e- 22 2 1} for SAB-(: and SAE-log models, respectively, tuned for similar reconstruction error.
Yelp text embedding dataset. The Yelp dataset was obtained from hugging face.* 8 The model architectures used for these experiments are the same as those from above as applied LLM intermediate activations. Models were trained for 150 epochs with a batch size of 512. The learning rate is 0.001 and λ is 512. The learning rate scheduler is CosineAnnealing WarmRestarts with To = 10. The penalty weights are {A₃ 3e 3, X = le - 2) and (A₁ = 2e - 4, = 5e - 44 for SAE-( and SAE-log models, respectively, tuned for similar reconstruction error as in previous experiments.
Pesudo-MNIST. We trained a GAN on MNIST for 500 epochs using publicly-available code." Subsequently we generated 500,000 new samples to form the Becudo-MIIST dataset. We adopted the same encode/decoder architecture for VAEase as used for the original MNIST data. We then trained VAEase model for 35 epochs on this dataset using batch size 512. is 100 and the learning rate is 0.005. The optimizer is Adam and learning rate scheduler is CosineAnnealingg WarmRestarts with To = 50. For the diffusion model, we adopted the "o associated with Stanczuk et al. (2024) and trained for 70 epochs. The NB poot-processing manifold dimension estimator is also implemented using the same repo. Meanwhile, the FLIPD estimator originates from separate repo.!! !!
" http://ggihubb.oo///ogggCCCC coding
1 hhip:://gghuuoooomm//aarrtosotn
http://///ffufooooooooooooooooon utils.py
17

---

Sparse Autoencoders, Again?
D.4. Representative Samples from Pseudo-MNIST
504
1
7 4
8 x
9
2
3 3
O 3
9
3
#
3
5
4
7
9
3
6
1
7
3
{
/
3
Figure 6: Samples from Pesudo-MNIST. On the left we show samples from the original MNIST, while on the right are samples from the proposed pesudo-MNIST: the latter are generated by GAN trained on MNIST with a known upper bound on the manifold dimension. Because visually the two sets of samples are quite similar, it increases credence that the Pseudo-MNIST data manifold dimension is a reasonable approximation to that of the original MNIST.
E. Technical Proofs
E.1. Proof Theorem 4.5
We first give a feasible solution and compute the loss rate, which could be the upper rate bound for the optimal loss. Then we prove that there is no solution to achieve a lower loss bound, and derive necessary conditions for the rate-optimal solution by analyzing the order of each term in the loss function.
feasible solution and upper bound. First, we will construct a solution to give a upper bound of optimal loss rate. For any manifold M, S EM, there exists an L-Lipschitz invertible function e),(-) that maps M, to I and eे." (0) = 0. Then we can construct pzz, E. def diag(o?) and μ. s follows:
μ. (1(x E Mi) /(1 /11-/1/3), ) 1.,,,(XEEEE (.. (LXEMM)))())/11-111-
Σ, = diag((c), - 1(x E M1) 71) cT, I(xE M.)(1 7)0 -.
Ps = Σ E (1(11 E! 1
where 0 is an all-zero vector of dimension x Σ rs and e, denotes an all-ones vector of dimension r. As the sum of multiple I.-Lipschitz functions, frr naturally also I.-Lipschitz on set M, satisfying Definition 4.2. Now compute the corresponding loss function check if reaches the lower bound we discussed.
/ M
Exolexx)]lIx -0000-111110 (()1 = Σ E E (1 -1 126))) (dx
+ (10xxx))((-21))))))1
Mr
Σ I L (1lv:(1)x) - vx(x)) + (1 1/2) 1/2 (
Σ / M. L² (2(1-21/2)3
12 Σ - --1-1//// 1123 2
18

---

Sparse Autoencoders, Again?
Then the KL term be computed
1 M (tr(Es) - log | + + ps= - (d)((d))
M. (mT + (N= rz)) ri log 7 1g kzz ((x)
M ((7- 11);- ri log + L2xx x))((x)
Σ aat log + L? 1,4 xJ xxx((dx)+ + (1))
Hence the upper bound of optimal loss rate is (d - E, log 0(1), obtained by perfect reconstruction function Pas (4g and r; instances o.j = O(7) for any x.
Note that we construct this solution without explicitly considering the intersection of the manifolds for the following reasons. First, if the region of intersection is in lower dimensional manifold, then according Definition 4.1 it has zero probability measure and therefore contributes nothing the overall loss. And secondly, if the intersection dimension matches the dimensions of the corresponding manifolds that are interesting, then these manifolds can be merged into single manifold. Either way, we are able to integrate the loss on well-defined manifolds one by one.
Optimal rate and associated necessary conditions. We will now show that the above loss rate is also optimal and the active dimension is equal to the local manifold dimension almost sure. Consider the loss function in VAEase model,
LvAtase 0x,7)
-Eg (a x) [log Po (x(2x) KL (z (z/x)//((z))
- log((2T) (xo(e
d
|x
pz
122
(tr(Ez)
log EE1| + u uz - x)
27
2
Assume there exists solution that can achieve rate smaller than (d E, aeri) log 7/2 0(1). Since terms tr( r(E.) + m__ M₂ and reconstruction error are (1) in the shown solution, to reduce the order of the loss, can only reduce the term log Ez|-
Optimal rate for active dimensions. Suppose there are a set X' S. MM; satisfying that, for any x E X', log Σ, r:0(- log and [x- w(dx) > Then the number of active dimensions for x must less than r;. Suppose that there are Ti - 1 active dimensions for which (x; Φy)5 = O()). and there one dimension for which (x dy), 0gg(7)) > ((r) + 0 as y 0 (if there is no other dimension j satisfying lim (x; Φb); O, we assert the reconstruction error would be @(1), resulting in infinite loss). We will show that the reconstruction error cannot remain at O(r) in this case.
Since X' is a subset of the manifold M, and its measure is not zero, at least r; dimensions of information are needed to reconstruct X' with Lipsehitz functions. Without loss of generality, we set the channels used for reconstruction to the first r; dimensions, i.c., (xx) = ).( To get the lower reconstruction loss, the function f must map R to X'. In addition, there exist compact set 2 C IR and positive constant I satisfying (z1) - fi(z))) 2 il|z1 zell for any 2 22 E Z.
19

---

Sparse Autoencoders, Again?
Then we have
BewNIODD1 - (I- + (I- - [1)
x
x Be-woonl|x--Heee E!! | |4: + [I Σ!
|| || & 1)w(dx)
2/ 7/x BEwNiOn (|xron|| E/! |b= + [I Σ. エリ 2
x
< 11 (I- M2: [I [I- - _ (d)x
2 /
(P
s n [Bi (I -!- M2, (I- -I--!/111 و!! S ())
N JR"
2 / ai a -
(I- (I- (I-E C ()d(x)
(17)
200(g(h)) [ w(xx),
where Bi (I-- -.---/-00 )=xx), (I- E is l norm ball and JM. I(B (II ,= 1 S ( could go to a, when is sufficiently small. The total loss on set X' should be in order (0(g()))/> + dllog (ra 1) log - log ((7) + +(11} fx. w(dx)/2. Since 0(g(7))/7 - log p(7) 2 O(1) + log there is no lower order for the feasible loss for any s(7) > O(7). However, the equation also shows that when Jx w(dx) = -70(((()) + O. the loss difference could be absorbed by O(1) In other words, the number of active dimensions in x will match the manifold dimensions, with a probability of almost I as y 0.
Optimal rate for inactive dimensions. First, from the above analysis, the rates of variance active dimensions converge to O. In contrast, the rate for inactive dimensions should be (1)) However, it is not sufficient to ensure the effectiveness of our revamping mechanism, which is expected as 1 σg Rs 0 for inactive dimensions. Formula17 17 implies the rate requirement of O()) reconstruction loss on inactive dimensions.
Suppose that the inactive dimension j" has no contribution in function pu-)) on set Z", which means ju(x) = ju(x)) for all (x, x') and x, x' E 3" are only different at dimension j". Then the reconstruction loss is independent with of and the left relative terms in loss function are of loo optimized at o; 1.
Otherwise, with some constant C, consider that there exists a subset 2, S Z. = fz|z = (I- 2//22/ + (I / ! e,z' = xxxx',x'' E MM,I|Ellee I C), where an inactive dimension jr contributes to function (-)()) also 0p, (2) # O. Combined with the continuity assumption, we could obtain a subset 2,' S 2,r that
(1zz(z) (z))l 2 1/ =z/=1 for n' 2. Let -j- g(()) O, we then have similar analysis in formula 17 as
J.L. EenNioullxx-aaae (I-I1// uz + (I- >!/ (8) 2)]
2 Sma. BEWN - 1x E//²) 14 + (I Σ زエ ||||e||| S Clw(dx)
.ma. Re-Nionnlx--utl 1x (1--1/12 mr + [I Σ! 22)/
fllEllo0 S SCnn((I 1//2) 11 + (I Σ! 1//² E Ex (1)
2/ =J I 10
1-11- - of 50 1(1 24/2) μr + (I - (-->!!-)!!
E (()))()(d))))((
I((I- + (I D/D////////e E (d)))))
20

---

Sparse Autoencoders, Again?
Denote the last integral terms by (C,,,,,, then is is held that A, - JM, 1(z E (dd when 1' is sufficiently
small. To make sure the reconstruction loos is O(7), we could obtain that if (c, 2 c as 7 - 0. g'()) - 0(2)/2)
must hold; if (C, + 0. 9 (7) (3) could be violated almost with probability of O.
Since JMM I(z E 2,) + JM, ((xx) with sufficiently large C' and 2 2x + 2 is the set where dimension j"
contributes nothing to un (.), we finalize the proof that oz- = 1 - -())) holds almost with a probability of 1.
E.2. Proof of Corollary 4.6
The proof is related to that of Theorem 4.5, but the upper bound of the loss differs. Without the additional (1 (1-og) to set inactive dimensions to zero, the feasible solution in VAEase model is invalid. Here we consider simple dataset combined with a point dataset and a line dataset, and 1 is the dimension of latent variables. Assume there exists solution satisfying
JM M:
1(IA(x)] =x))(dx))) = a; + ol1)
To avoid the loss function diverge to +o0, a necessary condition is R = O(7). For the point manifold, there should be no active dimensions. Since Mo only contains one point Xo, to make the reconstruction error O(7), it leads to (z)) - xoll 2 1) S R = O()) by Chebyshev's inequality. For the line manifold, it should be held that (x) + +(v)).
Assume the point manifold is on the line and consider the data point around Xo. For the a compact set X := [x|[x- > 1}. we know preimage of set X' under Lipschitz mapping 2s have probability no more than O(x) under the distribution N (xxz)xx). 1). This means that the preimage 2, as a compact set on R, must satisfy
exp(-(2 /xxxx)))//)d== = 0(7)-
(18)
Due to the Lipschitz continuity of Uz. consider xo and x' satisfying ||x' - Xoll = 1, then [1=(x') (xx)) should be bounded by some constant. Then compact set Z should be small than 0(7) under the Lagrangian measure to keep (E.2) fixed. Then there exist a x" E X satisfying ||X" - x'Il > c for some positive constant c. So we finally get [| [(()((()))))))))) - th (xx((((()))) & [x]xx(x")) - ps (x')ll because the decoder mean function is Lipschitz. As a result, we obtain L 2 (c - ( ((V)))((()) with probability almost 1, which means the decoder mean function is not Lipschitz almost surely. From this conflict, we can conclude that there is no solution that satisfies adaptive sparsity for VAE this dataset.
E.3. Proof of Theorem 4.7
Since the U is known, the optimization problem can be conveniently decoupled into d independent one-dimensional optimization problems (this stems from the fact that the Frobenius norm is invariant orthonormal transformations). So we just need to verify that there is a unique minima for each one one-dimensional problem. The loss function for single point in VAEase model is
LVAFase (I; W. P2, o2) = (dlog(2 + ||I w(1 -=)))= - -oop://===000-3////++- - -logo? + - 13/2
and the partial derivatives are
V2 (2; W. M22 o?)
= - 2(1-0,)) - -o)) - x3/7 + +w 1
(1)))
Ow
V2L (z) W. M2 0 = -2wps - oz) ue x)/7 + 2w2 oz (,,0,) 20 +)/7+20, 2//02
0o:
L (x; W. U1: 2
= 2ww1 - 0z)(w(1 - oz)u:= - x)/7 + 21
21

---

Sparse Autoencoders, Again?
From the KKT condition, we can obtain
1
(1 - (0) + 122)
w² x2)
(0x(1- - -20x) - (2)) +
2p x
7 0,² 02
w(1 - 02)2
P2 w (1 rz)3 +
Substituting the first equation into the last equation, we obtain
Mad
- 0z)
[1 +o?)) (1 - 02) +7
x²
(o? + m3)² +y = [0] + kg)
2 + + + 22 = 1²(0²² + 11))
10 1122² = 2²02-
Finally, we compute the second equation as
1 o. (o? + 222 (1 0=)(0=(1 - -202) (1-50) -11))=
(1 - 0 (o) + p2) M220 = 7(0==' -0z);
(a 01 -20=)= - m1 (a² + n2))
7/05'' -o;)(1 -0)),
(a) + 22
(o + 222 (0x(1 -20=)) 1 o + w²) 1(0= oz) -os),
22
100=² 1). (o + p22
Using +2)+ 2²02/7, we have
13222 220z/7 7 - 1),
2 (1 o2.--
Substituting this expression back into the above result, we get 7 = o² and o² 7//3?. At this point, we have calculated the unique minimum point (u), = (vI- / for the one-dimensional problem. The multi-dimensional conclusion then naturally follows.
E.4. Proof of Corollary 4.8
Following the same approach as in the previous proof above, we actually only need to demonstrate that a one-dimensional problem has two local minima. The loss function for single point in SAE model is
(SAE(a; z) (x -w)) + >11(())= A
and the partial derivatives for w is
ÖLsAe(x; (x;
2z(wz x)) + 2h2w.
(19)
Ow
From the KKT condition, we can solve = zz/(Ae + z²). Then the loss function without w is
d22²
())E(0;))= = d2 + zz
+ +11h(2).
22

---

Sparse Autoencoders, Again?
Then consider when z >> O, have
d[SAE(132)
222x²z dz (12 + 22)2 + (z)
The h=2) is concave and non-decreasing when 0, (2) is non-increasing and positive.
Consider the function of m(z) 2 + =²)² we have m0) = 0, m( m((22/3) 9 //3 (16Ag) is maxima and m²2)) 7 0 as + co. To obtain two local minima, we need the derivative of h has two intersection points with 2A2x* (2 2,
If (=) = 0 when O, then we know there is a local minima at = = oo since (=)) is decreasing when 2 > v 12/33. Because [[[=)) concave, we know that (0) S h(1), leading to another local minima = while = = 0 also the local minima for mz)..
If lim.-o (2) O, then A2/33, leading to a interval 122/3, /22/3 + c] where (SAE 2) is decreasing with some positive c. In other words, there must be local minima satisfying = > v /2/33.
sure that is local minima for any Then we just need 9 / /(102)116))))))))
E.5. Linear VAE on Composed Linear Dataset
For simplicity, we omit the subindex in LLVAE()) and write its loss function as,
((0,0,7)= = f (EEgstalx) po po (x/2)) + (L(g((()))l](()) w(dx),
(20)
where (z(2))) (22 Σx) with u4 Rdd, po(x(2) N(x TI) with Mh E R and p(z) NO. I). Here Σ, def (o2 ()) is diagonal matrix.
Recall that the decoder here is linear, i.e., 8 p2 Wz. Expand the formula in the integral as following
[(x; Φi 0,
E (xl())
(x(])((x(2)] + KL( (2/xx) (lp(z))
- (x/()//x)
d
log( (og(2
||x
Ma
12
+ (ir(Es) log Egl + ur ps k)
2
27
d log (2 + (x/xx
x M 11
+ 1
(Ir(E,) - log Σ + m |b= 1))
2
2%
2
d
Wz x|121
log(2
+ E (x(xx)
2 (tr(E)) log EEa + M ks x)
2
2y
d
1
log(20 (2mm) +
{ tr( WE, W + (WH: x) (WWx:
2
27
1
+ 2 (I(EE) log Σz1 #1_ p= 11-M--00 1 k).
The gradient of C(x) takes the form,
OC(x) = WT (Wu: - -x)+ + 4z
0p:
d[(x) WWw+ ) Σ
oΣ, 27
0[(x) 1 (E. t. W -,,xT}
OW 7
Note that μ.= and ///² are varying functions of x, then the KKT conditions provide the necessary conditions for an optimal
23

---

Sparse Autoencoders, Again?
solution as
(W W + =111== W
Σ. W/
/ [ p= WW w(dx) J p22 w(dx)..
(21)
Here, the first two equations are held for any x and the last equation is held on integral since W does not vary with x. For convenience, let (>I + W W)- A, then Σ. = and μs AWT Then (21) could be transformed
/ (2 + p. w w(dx) / 12 4=x" wdxx),
E + MnM W WAw(dx)) / ,xx WAw(dx),
x
(E + mm W Ww(dx) = S. MaM w(xx)(( W WW),
x'
Σ. N.TT. = 7 / pu. w(xx)
AW W = / AW xx WAw(dxx),
WT W WWWW W) = WT / xX w(dx)
W WW W WT / xxx w((w))
(22)
Denote the singular value decomposition of W as W = USVT with U Rdxx, S E Rx being diagonal matrix, and V e RE** Then (22) can be simplified as
VSUT / xx xxT USX
VSU (7I+ U JUVT USSST
Note that we have V = Ig and U = Id, then we have
S + = SUT / xx (dx)
(23)
Notice the left side is diagonal matrix, thus UT Sx xx TexTw(en w(dx a Σg should also be diagonal. Since x is lying in composed linear space, then the rank of ʃ xxTTw((ee (dxx must be the dimension of smallest linear space covering all samples.
Denote the dimension by r, and assume the first r diagonal element in Σ is nonzero. Finally, we obtain that S,; = =Vo? 25 when i and S when r, where recall Si the i-th diagonal element in S and of is the short of Σ² Now
the interim matrix is Vdiag(a -2 ..... 2xr ... -1 1... VT, and Σg Vdiag(no 21 the number of active dimensions is r.
-2 = I,... 1) V T So
To compute the loss function, we also need
W k= xd]
(WAW [dxx
Udiag
..., -.,0 -1, IJUT (
1/UT / xx xxT w(xx))
tr diag(12 -2 ...., 2 -2
24

---

Sparse Autoencoders, Again?
and
Jel || (dw
||AW x xl / x WW xw(dx)
x Udiag( - -7)/0211..... (o - -2:/02m: 0 O3U... xw(xx)
x
0e./o,,.......0 / xx w(((x)))
diag (o - 7)//0 1.....,,662,, - -7)/0=11....,00
tr
(a - 7)/
iS
With these results, the minimum of energy is
07,7) = /
d log(2 +
? Σ (o - 7//02/ + | ur - x112
2
27
x
isn
+
7/02/
+ k --
E(log log 22) + 11 w(xx)
ist
15"
d
1og(2
Σ [02 -7)/02/
1 ?
Σ 15"
2
2
2
1
1
Σ
0
2
r
r
rlog?
Σ
log 2 + 7
Σ ozi) 2
isr
d
1
+ 1
log(( (2mT)
log +
2 log
r.
(24)
2
2
2
2"
igr
As y + O, the rate of minimum energy is (d r)/2 2og + (1).
25


Reviews:
[{"review_id": "xXRsZpTduz", "reviewer": "Reviewer_JzHL", "summary": "This paper addresses the limitations of traditional Sparse Autoencoders (SAEs) and Variational Autoencoders (VAEs) in sparse representation learning, particularly their inability to adaptively adjust sparsity patterns and sensitivity to hyperparameters. The authors propose a novel model called VAEase, which combines the strengths of both SAEs and VAEs by introducing a new mechanism that dynamically adjusts sparsity based on input samples without requiring hyperparameter tuning. VAEase achieves this by modifying the VAE framework to use encoder variance as a gating mechanism, allowing it to selectively activate or deactivate latent dimensions for each input. Empirical evaluations on synthetic and real-world datasets demonstrate that VAEase outperforms existing SAE and VAE models in accurately estimating underlying manifold dimensions and producing sparser representations while maintaining low reconstruction error. Overall, VAEase provides a more flexible and efficient approach to sparse autoencoding tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for acknowledging the many positive aspects of our work, including the novel design, comprehensive proofs, solid theoretical foundation, well-designed experiments, and the robustness and generalizability of our proposed model.  We also appreciate the reviewer's statement that there is clear and convincing evidence of the claims made in our submission.\n\n**Comment:**\n*The proposed method make sense for the problem, but I do not find the advantages compared with traditional VAE.*\n\n**Response:**\nA traditional VAE model cannot readily learn adaptive sparse solutions (meaning solutions whereby the locations of informative latent dimensions vary from sample to sample).  In fact, we rigorously prove this VAE limitation via Corollary 4.6, and verify it empirically; see for example Table 2, where the VAE is incapable of learning the correct manifold structure *which requires adaptive sparsity to do so*.\n\n**Comment:**\n*The benchmark dataset and evaluation metrics in this paper are not popular in the field of autoencoder research.*\n\n**Response:**\nSAE models are commonly applied to a wide variety of different tasks, such that we are not aware that there is any particular most popular benchmark.  That being said, arguably the most high-profile recent application is to the analysis of LLM activation layers, and we have included such experimentation in our paper.  We have also added another new application example to the rebuttal; please see our response to Reviewer WZdZ. Moreover, the metrics we adopt closely mirror prior work as well.\n\n**Comment:**\n*1. ... Testing on more diverse image datasets like ImageNet or COCO.*\n\n**Response:**\nWhile we agree further testing can always be valuable, our focus is not specifically on computer vision tasks, which is why our submission also includes multiple non-image examples.  Moreover, we do not commonly see SAE models applied to ImageNet or COCO.  Still, application to broader image classes is a useful suggestion for future work.\n\n**Comment:**\n*2. Although the paper claims improvements over traditional VAEs, reported gains might not be substantial enough to justify the additional complexity introduced by the VAEase model ...*\n\n\n**Response:**\nWe respectfully disagree with this point for multiple key reasons.  First and most importantly, *our proposed VAEase approach does not introduce additional complexity over a regular VAE*.  Instead, it merely reuses the exact same encoder variances as a novel gating mechanism; hence no further justifications are needed with regard to complexity.  And secondly, for a fixed reconstruction error, VAEase does in fact produce much greater sparsity across a wide range of tasks as desired.  Of course the reported reconstruction errors are often similar, but this is specifically part of the experimental design to facilitate meaningful sparsity comparisons.  Along these lines, we also note that under listed strengths, the reviewer specifically mentioned that our experiments are \"well-designed and demonstrate clear improvements of VAEase over existing methods.\"\n\n**Comment:**\n*3. The paper emphasizes the theoretical and experimental aspects of VAEase but lacks a clear discussion on its practical applications ... benefits of adaptive sparsity and reduced reconstruction error are not directly linked to specific use cases.*\n\n**Response:**\nGiven the vast literature involving SAE models, we largely defer treatment of particular practical applications to prior work.  That being said, we do link the benefits of adaptive sparsity and reduced reconstruction errors to the specific use case of LLM activation interpretability.  In particular, to complement more subjective/qualitative analysis of SAE-learned representations, we follow common practice in the LLM literature that advocates specifically for comparing sparsity levels at a shared reconstruction error, with greater sparsity (meaning adaptive sparsity from sample to sample as our VAEase achieves) being associated with better interpretability.  Even so, we are happy to include further references to this effect (although space in our submission is tight, so there is quite limited room for much additional discussion).\n\n\n**Comment:**\n*4. ...The authors could benefit from a more concise summary of the key innovations and their implications, rather than focusing solely on detailed proofs.*\n\n**Response:**\nAs an important clarification, our work does not focus solely on detailed proofs.  In fact, all proofs are exclusively deferred to the very end of our paper deep in the appendices, and certainly not treated as a central focus.  And for reference, we concisely list our central innovations and their consequences on the bulleted paragraphs beginning on Line 60 as well as pointers to other sections therein.", "claims_and_evidence": "There is clear and convincing evidence of the claims made in the submission.", "methods_and_evaluation_criteria": "The proposed method make sense for the problem， but I do not find the advantages compared with traditional VAE.\n\nThe benchmark dataset and evaluation metrics in this paper are not popular in the field of autoencoder research.", "theoretical_claims": "The proofs in the article are very detailed. I didn’t fully understand them, but I think they are valid and correct.", "experimental_designs_or_analyses": "The benchmark dataset and evaluation metrics in this paper are not popular in the field of autoencoder research.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths：\n\n1. The proposed VAEase model introduces a novel yet straightforward modification to the traditional VAE framework by leveraging encoder variance as a gating mechanism. This simple change effectively enables adaptive sparsity without complicating the overall model architecture.\n\n2. The paper provides comprehensive theoretical proofs that support the effectiveness of VAEase. The authors rigorously demonstrate that VAEase can recover underlying manifold structures and prove that it has fewer local minima compared to traditional SAEs, thereby solidifying the theoretical foundation of the proposed method.\n\n3. The experiments are well-designed and demonstrate clear improvements of VAEase over existing methods. The model consistently outperforms SAEs, VAEs, and diffusion models across various synthetic and real-world datasets, validating its ability to achieve sparser representations with lower reconstruction errors.\n\n4.  The authors test VAEase on diverse datasets, including linear and nonlinear synthetic data, image datasets (MNIST and FashionMNIST), and high-dimensional language model activations. The consistent performance across these datasets highlights the robustness and generalizability of the proposed model.\n\n5.  The paper is well-written, with clear explanations of the model design, theoretical analyses, and experimental results. The authors effectively communicate complex ideas in an accessible manner, making it easy for readers to understand the innovations and contributions of their work.\n\nWeaknesses:\n\n1.  The experiments primarily focus on synthetic datasets, MNIST, FashionMNIST, and intermediate activations of language models. While these datasets are useful for initial validation, they lack the complexity and diversity of real-world image datasets like ImageNet or COCO. Testing on such mainstream datasets would provide stronger evidence of the model's practical applicability and robustness.\n\n2.  Although the paper claims improvements over traditional VAEs, the reported gains might not be substantial enough to justify the additional complexity introduced by the VAEase model. In some cases, the differences in performance metrics (e.g., reconstruction error, sparsity) may appear marginal, raising questions about the model's overall advantage.\n\n3.  The paper emphasizes the theoretical and experimental aspects of VAEase but lacks a clear discussion on its practical applications. The benefits of adaptive sparsity and reduced reconstruction error are not directly linked to specific use cases, making it difficult to assess the model's real-world impact and potential adoption in industry or other research areas.\n\n4.  While the paper provides extensive theoretical proofs, the innovation in the model design might not be immediately apparent to readers. The authors could benefit from a more concise summary of the key innovations and their implications, rather than focusing solely on detailed proofs. This would help highlight the model's novelty and contributions more clearly.", "other_comments_or_suggestions": "None", "questions_for_authors": "Please refer to the above weaknesses.\n\n\n## update after rebuttal\n\nBased on the author's responses and the comments from other reviewers, I still maintain my score of weak accept and lean towards accepting the paper. Although I am not an expert of sparse autoencoder, I still believe this work is very meaningful.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ed1t0UGpGU", "reviewer": "Reviewer_LJBm", "summary": "The paper introduces a method to explicitly model sparsity in variational autoencoders (VAEs) that leverages a simple (parameter-free) transformation of the latents before decoding. The method is introduced by intuitive construction and supported by theoretical arguments and results. Experimental comparisons show the effectiveness of the method in learning sparse representations of data.\n\n## Update after rebuttal\nAfter clarifications from the authors regarding my criticism of the active dimension removal experiment and additional experiments provided in response to another reviewer, I updated my score to 4 (Accept) to better represent my evaluation of the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are appreciative of the detailed, constructive comments, and for pointing out the high potential of our work being used by those interested in sparse autoencoders (which is our intended audience). We address main reviewer points in turn below.\n\n**Comment:**\n*As far as I understand, the masking of active dimensions in the experiment illustrated in Fig. 3 follows the partitioning criteria described in lines 367--384 (\"throughout this paper\" is stated therein). Would it not be fairer, when viewing this against reconstruction error, to mask dimensions in order of least to most disturbing w.r.t. the reconstruction error? In other words, sequentially remove the next active dimension that would perturb the average reconstruction loss the least (maybe different dimensions are picked per model).*\n\n**Response:**\nActually, the shape of Figure 3 does not appreciably change when enacting the reviewer's suggestion.  This is because the magnitude of $\\mu_z$ values for SAE models, or $\\sigma_z$ values for VAE models, already closely reflect their importance w.r.t. reconstruction error by design.  Moreover, using these magnitudes better reflects practical use cases, whereby we would ideally like to assess active dimensions without having to compute $O(\\kappa^2)$ separate reconstruction errors.\n\n\n**Comment:**\n*In lines 420--423: This sentence in the analysis is faulty in my view, since their own method estimates $r \\approx 15$ on Pseudo-MNIST. Since the setup upper bounds the ground-truth $r$, a simple (but very laborious) test could follow this setup from 16 dimension down to one dimension, stopping whenever the whole setup degenerates.*\n\n**Response:**\nOur intended point on Lines 420-423 and the surrounding text is the following:  By design of the experiment, we know that 16 is a strict upper bound on the ground-truth manifold dimension.  Hence any method that produces an estimate above 16 must be incorrect.  In particular, both diffusion approaches in Table 4 produce estimates vastly exceeding 16.  Meanwhile our VAEase approach produces an estimate below 16 despite no knowledge of the generative process.  Hence the VAEase estimate is at least plausible, even while conceding that the true manifold dimension might still be a bit smaller (which would also be even further from the diffusion estimates).  Of course intuition suggests it can't be dramatically smaller, as visual inspection of human-written MNIST digits indicates multiple non-trivial degrees of freedom.  \n\nOverall though, our main objective here is to show the superiority of VAEase relative to diffusion, not exactly pinpoint any exact manifold dimensions associated with MNIST.  The reviewer's suggestion would indeed help with the latter, but this is computationally intensive and involves subjectively deciding when GAN samples deviate too far from MNIST. Still, the reviewer's comment is well-noted, and we will adjust the writing to ensure our desired points are more clearly conveyed.\n\n\n**Comment:**\n*I reviewed most of the supplementary material. It was surprising to me how the quality of the writing is considerably worse in the appendix sections compared to the main paper.*\n\n**Response:**\nAdmittedly in the rush to submit, more attention was devoted to polishing the main text, which the reviewer agrees is well written and organized.  For the revision we can definitely remedy this imbalance.\n\n\n**Comment:**\n*A worthy mention is perhaps the applicability of sparse autoencoders in general, coupled with the size of the change proposed to VAEs. While properly mathematically substantiated and experimentally verified, the main contribution is still arguably quite straightforward and that could be viewed as its weakness.*\n\n**Response:**\nWith regard to SAEs, we believe their relevance is increasing along multiple fronts.  For example, as we discuss in Section 5 and elsewhere, there is rapidly growing usage w.r.t. learning interpretable representations of LLM activation layers. Moreover, other representative SAE examples spanning vision and language are frequently appearing in the literature, e.g., as in https://arxiv.org/pdf/2502.04382, https://arxiv.org/pdf/2502.06755v1, and https://arxiv.org/pdf/2410.06981.  And to better reflect this trend, we have added additional related experiments; please see our rebuttal response to Reviewer WZdZ for details.\n\nSecondly, although our VAEase proposal is admittedly simple, we view this as a fundamental strength as it is easy for anyone to apply, portending larger downstream impact.  Moreover, the underlying novelty is evidenced by 10 years of extensive VAE usage without prior work ever uncovering, let alone rigorously analyzing, the highly-effective modification we have proposed.\n\n\n**Comment:** \n*Some typos/mistakes ... Minor comments*\n\n**Response:**\nThanks for pointing these out, it is extremely helpful for improving the revision.  We will definitely correct each of them.", "claims_and_evidence": "The claims are overall reasonable and the evidence provided is mostly convincing.", "methods_and_evaluation_criteria": "The experimental setup is valid and the proposed method is sound for the problem of interest.", "theoretical_claims": "I checked the correctness of the proof for Theorem 4.5. Despite minor writing/formatting issues and convenient assumptions which I'm not sure about regarding applicability in pratical cenarios (Lipschitz continuity), I have no issues to mention.", "experimental_designs_or_analyses": "I think the experiments conducted are adequate overall, except for the few points below.\n\nAs far as I understand, the masking of active dimensions in the experiment illustrated in Fig. 3 follows the partitioning criteria described in lines 367--384 (\"throughout this paper\" is stated therein). Would it not be fairer, when viewing this against reconstruction error, to mask dimensions in order of least to most disturbing w.r.t. the reconstruction error? In other words, sequentially remove the next active dimension that would perturb the average reconstruction loss the least (maybe different dimensions are picked per model).\n\nIn lines 420--423: This sentence in the analysis is faulty in my view, since their own method estimates $r \\approx 15$ on Pseudo-MNIST. Since the setup upper bounds the ground-truth $r$, a simple (but very laborious) test could follow this setup from 16 dimension down to one dimension, stopping whenever the whole setup degenerates.", "relation_to_broader_scientific_literature": "The method introduced enable variational autoencoders to be used effectively as sparse autoencoders. This naturally builds upon both sparse autoencoders and variational autoencoders, the former now experiencing increased relevance due to their use in understanding the \"inner\" representations used by large language models.", "essential_references_not_discussed": "Not applicable.", "other_strengths_and_weaknesses": "**Strengths**\n\nThe paper is very well written and organised, which made it very easy to read. The idea pursued is simple, well-motivated, and with a straightforward implementation, leading to a high potential of being used by those interested in sparse autoencoders.\n\n**Weaknesses**\n\nA worthy mention is perhaps the applicability of sparse autoencoders in general, coupled with the size of the change proposed to VAEs. While properly mathematically substantiated and experimentally verified, the main contribution is still arguably quite straightforward and that could be viewed as its weakness.", "other_comments_or_suggestions": "Some typos/mistakes:\n- l. 051: hyperparaemters -> hyperparameters\n- l. 196: DGMs? Perhaps DMs is meant here?\n- l. 256: $\\mapsto$ should be $\\rightarrow$ (`\\rightarrow`)\n- l. 235-236: citation uses parenthesis\n- l. 318: \"large\" seems to no belong in the sentence\n- l. 391: \"projection\" is a somewhat overloaded term, but has a precise meaning as a linear function. Perhaps use \"transformation\" or \"mapping\"?\n- l. 847: \"encoders\" is mentioned twice\n- l. 854: wieghts -> weights\n- l. 980: isn't $\\mathcal{X}'$ meant in the integral?\n\nMinor comments:\n- The title of section C.1 struck me as quite colloquial\n- F.1: $\\mathbf{e}$ is often used as the canonical orthonormal basis vectors, perhaps a better name would be $\\mathbf{1}$?\n- l. 998: Could you indent the expression better to be aligned inside the integral?", "questions_for_authors": "None at this point.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "auCw2c30mq", "reviewer": "Reviewer_WZdZ", "summary": "After a discussion of the strengths and weaknesses of variational autoencoders (VAEs) and sparse autoencoder (SAEs), this paper proposes an adaptation of the VAE architecture to enhance sparsity and target the interpretability objective that motivates SAEs. The idea is conceptually simple: use the learned variance network $\\sigma_z$ as a mask to select the active latent dimensions of z by element-wise multiplying the latent representations z by 1 - $\\sigma_z$. The authors theoretically show that this adjustment allows to benefit from the local minima smoothing properties of VAEs while mimicking the behavior of SAEs. Then, the authors present empirical evidence supporting the effectiveness of the proposed VAEase architecture. In particular, they compare it to a vanilla VAE, and SAEs in which sparsity is enforced by L1 regularization, log and top-k activations. The methods are evaluated in a toy setting in which data are generated in a small number of low-dimensional manifolds and then embedded into a space with larger dimensionality. Additionally, the auto-encoding methods are evaluated on the reconstruction of image data (MNIST and Fashion-MNIST) and of the activations of a Pythia language model. Finally, the paper presents a comparison of VAEase to diffusion models on the reconstruction of pseudo-MNIST data generated using a GAN.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the quite comprehensive and accurate summary of our work.  Indeed this description captures the essence of what we hoped to convey.\n\n**Comment:**\n*...additional empirical validation of language model data (e.g., on a more recent, larger model) would make the paper stronger.*\n\n**Response:**\nWe agree that further investigation into LLM applications of our work is desirable.  However, because we are not domain specialists w.r.t. LLM interpretability in particular (which involves a number of qualitative investigations), we have deferred this application-specific dimension to future work.  We also remark that prior SAE papers devoted to LLMs have proposed the same metrics that we adopt herein for quantification purposes.  \n\nThat being said, as our emphasis is on diversity of application, especially very recent ones, we have added an additional LLM-driven experiment whereby the downstream goal is to generate hypotheses regarding relationships between text data (e.g., headlines) and a target variables (e.g., clicks); please see https://arxiv.org/pdf/2502.04382.  While we defer details to this prior paper (no affiliation to us), the key optimization task involves learning sparse representations of text embeddings from Yelp reviews.  Results are shown in the new table below, where we observe that VAEase again requires the fewest average active dimensions (AAD) even while maintaining the best reconstruction error (RE).  This further expands the diversity of scenarios whereby VAEase outperforms existing alternatives.  Thanks for the suggestion.\n\n\n| Model     | RE     | AAD         |\n|-----------|--------|------------|\n| SAE-$\\ell_1$    | 0.2236 | 61.92 |\n| SAE-$\\log$   | 0.2298 | 55.97 |\n| SAE-$T_k$  | 0.2168 | 30.00   |\n| VAE       | 0.2774 | 90.00   |\n| VAEase    | **0.1869** | **16.74** |\n\n\n**Comment:**\n*What does the reconstruction error/sparsity trade-off for language model data look like? A visualization similar to Figure 3 (possibly in the appendix) would be appreciated.*\n\n**Response:**\nGood question.  While Openreview does not allow us to upload new figures, we have run this experiment and the trend closely matches that from Figure 3.  This further solidifies the stable advantage of VAEase across quite distinct domains.  And such a figure can easily be added to a revised appendix as the reviewer suggests.", "claims_and_evidence": "The authors claim that the VAEase architecture is able to achieve a better sparsity-reconstruction trade-off than SAEs and vanilla VAEs. This claim is adequately supported by theoretical and empirical results.", "methods_and_evaluation_criteria": "The experimental setup in both the toy setting and on image and language model data is reasonable. The evaluation metrics for sparsity and reconstruction quality are appropriate.", "theoretical_claims": "I did not find any issues in the proofs of the theoretical claims. However, it’s possible that I might have missed something.", "experimental_designs_or_analyses": "The experimental analyses in the toy setting and on image and language model data are sound. However, given that one of the main applications of SAEs is in interpreting language model activations, additional empirical validation of language model data (e.g., on a more recent, larger model) would make the paper stronger.", "relation_to_broader_scientific_literature": "The contributions of the paper connect with the literature about VAEs and with recent works proposing the application of SAEs to interpret language model activations. This is adequately discussed by the authors in Section 2.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "- What does the reconstruction error/sparsity trade-off for language model data look like? A visualization similar to Figure 3 (possibly in the appendix) would be appreciated.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8ngc3kPNTD", "reviewer": "Reviewer_75rJ", "summary": "The paper proposes sparse variational autoencoders as an analog to sparse autoencoders by adding a sample wise sparsity mask.\nThe authors then consider low dimentional data and show that only this low dimensions are active for optimal parameters. They then compare this model on multiple real world data sets to validate that it can outperform other sparse autoencoder benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comprehensive review of our work, particularly with respect to checking proof details and pointing out multiple valuable corrections.\n\n**Comment:**\n*The benchmarks seems sensible ... but the only comparison made is against other sparse autoencoder models.*\n\n**Response:**\nActually, we also compare against *diffusion* in Section 5.3 which is definitively not an SAE.  Additionally, we compare extensively against vanilla VAEs, which have not historically been considered as SAEs per se.  Still, the predominant focus of our work is indeed on improving upon SAE models.\n\n\n**Comment:**\n*I did not check the experiments in detail ... Went through the proofs in appendix F...*\n\n**Response:**\nThanks for checking the proofs, *this is a great help*.  The diversity and complementary nature of our experiments also serve as a key part of our contribution.\n\n**Comment:**\n*(+)The idea for this paper is well motivated and the paper is well structured (+) The novel ideas in this paper warrant exploration and I believe this to be a good direction for research.*\n\n**Response:**\nWe appreciate acknowledgement of the novelty of our work, and indeed no prior paper has repurposed and analyzed VAE models in the way that we have.\n\n**Comment:**\n*(-) The overall contribution in this paper feels a bit shallow ... comparisons against other sparse autoencoder models only ...*\n\n**Response:**\nWe emphasize that VAEs represent one of the highest cited, influential ML frameworks of the last 10 years, and yet they have never been previously applied or analyzed in the context of SAE tasks as we have done.  Nor have the specific attributes of adaptive sparsity, local minima smoothing, and guaranteed maximally sparse global optima been cohesively addressed in the context of widely-used SAE models.  Still, as a contribution to an ML conference we do not match the mathematical depth or generality of a pure theoretical contribution on manifold geometry, etc.  Instead, we produce a brand new VAE use case, *that requires no additional model complexity*, and elucidate unique attributes shared by no prior approaches.  We also offer extensive supporting experimentation, including comparisons with diffusion models beyond SAEs alone.\n\n\n**Comment:**\n*(-) The theoretical contributions need clarification ...*\n\n**Response:**\nPlease see our specific responses elsewhere.  However, broadly speaking, our results establish that the proposed VAEase energy function uniquely and simultaneously achieves two desirable properties: (i)  The global minima exactly reflect union-of-manifold structure in data, and (ii) the loss exhibits a selective smoothing mechanism capable of reducing local minimizers.  *No prior SAE or VAE approach achieves both*; provably so in the sense described in Section 4 for the wide classes we consider as baselines. \n\n\n**Comment:**\n*Question about Definition 4.1 and and whether stated conditions can be relaxed, etc.*\n\n**Response:**\nGreat suggestion.  The issue here relates to an inadvertent typo in our submission, where we used $\\mapsto$ instead of $\\rightarrow$ in Definition 4.1.  Crucially, we do *not* require the image set in $\\mathbb{R}^r$ to be an open ball. This implies: (i) the manifold's image can be an arbitrary topology as long as a global diffeomorphism exists; (ii) manifolds with a boundary remain valid objects of study as the definition focuses on the dimensionality of image set rather than imposing openness constraints.  \n\n\n\n**Comment:**\n*Regarding Definition 4.3, do we really need $\\sigma_z(x;\\phi_\\gamma) = \\Theta (1)$?*\n\n**Response:**\nIn the proof from Section F.1 (Line 1003), the correct asymptotic requirement should be $\\sigma_z^2(x;\\phi_\\gamma) = O(\\gamma)$ for *active dims* or $1 - \\sigma_z^2(x;\\phi_\\gamma) = O(\\gamma)$ for *inactive dims* to eliminate constants in the inequality. We will clarify the revision accordingly; thanks for pointing out this oversight in our original draft.\n\n\n**Comment:**\n*The notation is not quite clear ... specifically brackets vs parentheses in equations (2) and (4).*\n\n**Response:**\nWe adopt the convention (shared by some ML works) that parentheses and brackets are interchangeable, and are chosen purely for making nested cases more readable, i.e., to avoid multiple parentheses or brackets in a row that can be visually difficult to parse.  But this is obviously a minor stylistic choice that can be changed if the reviewer prefers.\n\n\n**Comment:**\n*Why is 4.6 called a Corollary ...*\n\n**Response:**\nCorollary 4.6 involves analysis closely related to Theorem 4.5, and the final conclusions are complementary.  Still, we can easily change this designation if preferable.\n\n**Comment:**\n*The first paragraph in F.1 is not clear, etc...*\n\n**Response:**\nGreat catch.  Indeed, the original text was backwards; we will correct it to state that we first construct a feasible solution and then analyze to form a necessary condition for optimality.", "claims_and_evidence": "The claims are supported by evidence however I have some concerns about the theory part, see theoretical claims below.\n\nAfter the rebuttal the concerns have been addressed", "methods_and_evaluation_criteria": "The benchmarks seems sensible, it should be noted that the only comparison made is against other sparse autoencoder models.", "theoretical_claims": "I went through the proofs in appendix F, they seemed correct, comments are in questions.", "experimental_designs_or_analyses": "I did not check the experiments in detail.", "relation_to_broader_scientific_literature": "The results further the understanding of sparse autoencoders by considering variational autoencoders. It can also be of more general interest to representation and interpretability as the learned structure is quite explicit.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "(+)The idea for this paper is well motivated and the paper is well structured\n(+) The novel ideas in this paper warrant exploration and I believe this to be a good direction for research. \n(-) The overall contribution in this paper feels a bit shallow in the sense that the theoretical claims are quite surface level and the experiments are still on simpler (if real world) data and compared against other sparse autoencoder models only. There is not one particular thing wrong, but it seems to lack a strong point that really has an impact. I would be happy to have my mind changed on this by the authors rebuttal\n(-) The theoretical contribution is not very clear both the claims themselves as well as the proofs (see questions for below)", "other_comments_or_suggestions": "See questions below", "questions_for_authors": "- I am a bit confused about definition 4.1. Manifolds of dimension $r$ that are diffeomorphic to $\\mathbb{R}^r$, are in particular diffeomorphic to an open ball in $\\mathbb{R}^r$ (as $\\mathbb{R}^r$ is diffeomorphic to a ball). This seems very restrictive, as it means that up to differomorphic equivalence only open balls are considered. In particular this excludes any compact manifold (e.g. sphere) as well as manifolds with boundary. From an applied perspective this restriction seems strange, could the authors elaborate on why it is needed and whether it can be relaxed?\n- Regarding definition 4.3, why does $\\sigma_z(x;\\phi_\\gamma) = \\Theta(1)$ correspond to inactive dimensions, in principle we could still have $1 - \\sigma_z(x;\\phi_\\gamma) = \\Theta(1)$. Why do we not need that $ \\sigma_z(x;\\phi_\\gamma)  \\approx 1$?\n- The notation is not quite clear: in (4) $\\sigma_z[x;\\phi]$ is defined, is this different from $\\sigma_z(x;\\phi) defined in (2) ? For example in Definition 4.3 the () brackets are used.\n- For theorem 4.5 you need $\\sum_i r_i \\leq \\kappa$. Do you have any intuition what happens if this is not the case?\n- Why is 4.6 called a Corollary, how does it follow from Theorem 4.5 (it seemed that rather than being deduced it was proven in a similar manner)\n- The first paragraph in F.1 states that you first analyze and then construct a solution, but afterwards you seem to construct a solution and then analyze.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel model that combines the strengths of both sparse and variational autoencoders by introducing a mechanism that adjusts sparsity based on input samples without requiring hyperparameter tuning. This is a solid paper and the reviewers agree that the idea is well motivated, the paper is well structured and organised, and the experimental results are adequate.
Decision: Accept (poster)