Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Authors: Lucy Farnik
Tim Lawson
Conor Houghton
Laurence Aitchison
Venue: ICML 2025 poster (2025)
Keywords: Mechanistic Interpretability
Sparse Autoencoders
Superposition
Circuits
AI Safety
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: We propose JSAEs, an architecture aimed at discovering sparse computational graphs in a fully unsupervised way

Abstract: Sparse autoencoders (SAEs) have been successfully used to discover sparse and human-interpretable representations of the latent activations of language models (LLMs). However, we would ultimately like to understand the computations performed by LLMs and not just their representations. The extent to which SAEs can help us understand computations is unclear because they are not designed to “sparsify” computations in any sense, only latent activations. To solve this, we propose Jacobian sparse autoencoders (JSAEs), which yield not only sparsity in the input and output activations of a given model component but also sparsity in the computation (formally, the Jacobian) connecting them. With a naïve implementation, the Jacobians in LLMs would be computationally intractable due to their size. Our key technical contribution is thus finding an efficient way of computing Jacobians in this setup. We find that JSAEs extract a relatively large degree of computational sparsity while preserving downstream LLM performance approximately as well as traditional SAEs. We also show that JSAEs achieve a greater degree of computational sparsity on pre-trained LLMs than on the equivalent randomized LLM. This shows that the sparsity of the computational graph appears to be a property that LLMs learn through training, and suggests that JSAEs might be more suitable for understanding learned transformer computations than standard SAEs.

Jacobian Sparse Autoencoders:
Sparsify Computations, Not Just Activations
Lucy Farnik
Tim Lawson'
Conor Houghton' Laurence Aitchison
Abstract
1. Introduction
Sparse autoencoders (SAEs) have emerged as a power- ful tool for understanding the internal representations of large language models (Bricken al., 2023; Cunningham et al., 2023; Gao et al., 2024; Rajamanoharan et al., 2024b; Lieberum et al., 2024; Lawson et al., 2024; Braun et al., 2024; Kissane et al., 2024; Rajamanobaran et al., 2024a). By decomposing neural network activations into sparse, in- terpretable components, SAEs have helped researchers gain significant insights into how these models process informa- tion (Marks et al., 2024; Lieberum et al., 2024; Temple- ton et al., 2024b; O'Brien et al., 2024; Farrell et al., 2024; Paulo et al., 2024; Balcells et al., 2024; Lan et al., 2024; Brinkmann et al., 2025; Spies et al., 2024).
Sparse autoencoders (SAEs) have been suc- cessfully used to discover sparse and human- interpretable representations of the latent activa- tions of LLMs. However, we would ultimately like to understand the computations performed by LLMS and not just their representations. The extent which SAEs can help us understand com- putations is unclear because they are not designed to "sparsify" computations in any sense, only la- tent activations. To solve this, we propose Jaco- bian SAES (JSAEs), which yield not only spar- sity in the input and output activations of a given model component but also sparsity in the compu- tation (formally, the Jacobian) connecting them. With a naïve implementation, the Jacobians in LLMs would be computationally intractable due to their size. One key technical contribution is thus finding an efficient way of computing Jaco- bians in this setup. We find that ASAFS extract a relatively large degree of computational sparsity while preserving downstream LLM performance approximately as well traditional SAFs. We also show that Jacobians are reasonable proxy for computational sparsity because MLPs are ap- proximately linear when rewritten in the JSAE ba- sis. Lastly, we show that JSAFs achieve a greater degree of computational sparsity on pre-trained LLMS than on the equivalent randomized LLM. This shows that the sparsity of the computational graph appears be property that LLMs learn through training, and suggests that JSAEs might be more suitable for understanding learned trans- former computations than standard SAES.
When trained on the activation vectors from neural network layers, SAEs learn to reconstruct the inputs using a dic- tionary of sparse 'features', where there are many more features than basis dimensions of the inputs, and each fea- ture tends to capture specific, interpretable concept. How- ever, the goal of this paper is to improve understanding of computations in transformers. While SAEs are designed to disentangle the representations of concepts in the L.M,, they are not designed to help us understand the computations performed with those representations.
One approach understanding computation would be to train two SAEs, one at the input and one at the output of an MLP in transformer. We can then ask how the MLP maps sparse latent features at the inputs to sparse features in the outputs. For this mapping to be interpretable, it would be desirable that it is sparse, in the sense that each latent in the SAE trained on the output depends on a small number of latents of the SAE trained the input. These dependencies can be understood 'circuit' (Olah et al., 2020; Cammarata et al., 2020). SAES are not designed to encourage this computation graph to be sparse. To address this, we develop Jacobian SAES (JSAEs), where we include a term in the objective to encourage SAE bases with sparse computational graphs, not just sparse activations. Specifically, we treat the mapping between the latent activations of the input and output SAES function and encourage its Jacobian to be sparse by including an 1,1 penalty term in the loss function.
computation graph or
'School of Engineering Mathematics and Technology. Uni- versity Bristol, Bristol, UK. Correspondence to: Lucy Farnik <lucyfamik @ @ema@egmllloooo gomi.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
With a naive implementation, it is intractable to compute

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
SAET activations 5x
SAE2 activations By
Traditional SAES
JSAEs
= o
SAE1 activations ST
SAE2 activations By
SAE1 activations $x
SAE2 activations By
Decoder dx
Encoder by
MLP f
Dense input activations x
Dense output activations
Figure 1. A diagram illustrating setup. We have two SAEs: one trained on the MLP inputs and the other trained on the MLP outputs. We then consider the function fa. which takes the latent activations of the first SAE and returns the latent activations of the sccond SAE, ie.. fs.)s) sy. The function the decoder of the first SAE ds, the MLP f. and the encoder of the second (output) SAE ey. We note that the activation function Tı is included for computational efficiency only; see Section 4.2 for details. JSAEs optimize for fa having sparse Jacobian matrix, which we illustrate by reducing the number of edges in the computational graph that corresponds to fa. Traditional SAEs have sparse SAE latents on either side of the MLP a dense computational graph between them; JSAEs have both sparse SAE latents and sparse computational graph. Importantly, Jacobian sparsity approximates the computational graph notion, but, we discuss in Section 5,4 and Appendix B, this approximation highly accurate due the fact that fa is mostly lincar function.
described by the function composition of the TopK activation function of the first (imput) SAE 7k-
Jacobian matrices because each matrix would have on the or- der of trillion elements, even for modestly sized language models and SAEs. Therefore, one of our core contributions is to develop efficient means compute Jacobian ma- trices in this context. The approach we develop makes it possible to train pair of Jacobian SAEs with only approxi- mately double the computational requirements of training a single standard SAE (Section 4.2). These methods enabled us to make three downstream findings.
transformers, i.e., where the parameters have been reini- tialized. We find that the degree of Jacobian sparsity that can be achieved when JSAES are applied transformer is much greater than the sparsity achieved for random transformer (Section 5.3). This preliminary find- ing suggests that Jacobian sparsity may be useful tool for discovering learned computational structure.
pre-trained
Lastly, we find that Jacobians accurately approximate com- putational sparsity in this context because the function we are analyzing (i.e., the combination of JSAES and MLP) is approximately linear (Section 5.4).
First, we find that Jacobian SAEs successfully induce spar- sity in the Jacobian matrices between input and output SAE latents relative to standard SAES without a Jacobian term (Section 5.1). We find that JSAES achieve the desired in- crease in the sparsity of the Jacobian with only a slight decrease in reconstruction quality and model performance preservation, which remain roughly on par with standard SAEs. We also find that the input and output latents learned by Jacobian SAES approximately as interpretable standard SAEs, as quantified by auto-interpretability scores. importantly, we also find that the "computational units" discovered by JSAEs are often highly interpretable- for ex- ample, JSAES find an output latent corresponding to whether the text German, which computed using several input latents corresponding tokens frequently found in German text (Section 5.2).
can be found at https//ggihbbboommmlyyyaaa//bbbu
Our
source
code
2. Related work
2.1. Sparse autoencoders
SAEs have been widely applied 'disentangle' the repre- sentations learned by transformer language models into very large number of concepts, a.k.a. sparse latents, features, or dictionary elements (Sharkey et al., 2022: Cunningham et al., 2023; Bricken et al., 2023; Gao et al., 2024; Raja- manoharan et al., 2024b; Lieberum et al., 2024). Human experiments and quantitative proxies apparently confirm that SAE latents are much more likely correspond to human- interpretable concepts than raw language-model neurons,
Second, inspired by Heap et al. (2025), we investigated the behavior of Jacobian SAEs when applied to random
2

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
i... the basis dimensions of their activation vectors (Cun- ningham et al., 2023; Bricken et al., 2023; Rajamanoharan et al., 2024a). SAEs have been successfully applied to mod- ifying the behavior of LLMs by using a direction discovered by an SAE to "steer" the model towards a certain concept (Makelov, 2024; O' O'Brien et al., 2024; Templeton et al., 2024b).
2.3. Automated circuit discovery
In "automated circuit discovery", the goal is to isolate the causally relevant intermediate variables and connections between them necessary for neural network to perform given task (Olah et al., 2020). In this context, a circuit is defined as a computational subgraph with an interpretable function. The causal connections between elements are de- termined via activation patching, i.c., modifying or replacing the activations at a particular site of the model (Meng et al., 2022; Zhang & Nanda, 2023; Wang et al., 2022; Hanna et al., 2023). In some cases, researchers have identified sub- components of transformer language models with simple algorithmic roles that appear to generalize across models (Olsson et al., 2022).
Our work is based on SAEs but has a very different aim: standard SAEs only sparsify activations, while ISAEs also sparsify the computation graph between them (Figure 1).
2.2. Transcoders
In this paper, we focus on MLPs. Dunefsky al. (2024); Templeton et al. (2024a) developed transcoders, an alter- native SAE-like method to understand MLPs. However, ISAES and transcoders take radically different approaches and solve radically different problems. This is perhaps easi- est to see if we look at what transcoders and ISAES sparsify. ISAES are fundamentally an extension of standard SAEs: they train SAEs at the input and output of the MLP and add an extra term to the objective such that these sparse latents are also appropriate for interpreting the MLP (Fig- ure I). In contrast, transcoders do not sparsify the inputs and outputs; they work with dense inputs and outputs. Instead, transcoders, in essence, sparsify the MI.P hidden states. Specifically, a transcoder is an MLP that you train to match (using a mean squared error objective) the input-to-output mapping of the underlying MLP from the transformer. The key difference between the transcoder MLP and the under- lying MLP is that the transcoder MLP is much wider, and its hidden layer is trained to be sparse.
Conmy et al. (2023) proposed a means to automatically prune the connections between the sub-components of a neural network to the most relevant for a given task using ac- tivation patching. Given choice of task (i.e., dataset and evaluation metric), this approach to automated circuit dis- covery (ACDC) returns minimal computational subgraph needed to implement the task, e.g.. previously identified 'cireuits" like Hanna et al. (2023). Naturally, this is compu- tationally expensive, leading other authors to explore using linear approximations to activation patching (Nanda, 2023; Syed et al., 2024; Kramár et al., 2024). Marks et al. (2024) later improved on this technique by using SAE latents as the nodes in the computational graph.
In a sense, these methods are supervised because they re- quire the user to specify a task. Naturally, it not feasible to manually iterate over all tasks an LLM can perform, so fully unsupervised approach is desirable. With JSAEs, take a step towards resolving this problem, although the architecture introduced in this paper initially only applies to single MLP layer and not an entire model. Additionally, to the best of our knowledge, no automated circuit discovery algorithm sparsifies the computations inside of MLPs.
Thus, transcoders and ISAEs take fundamentally different approaches. Each transcoder latent tells us 'there is com- putation in the MLP related to [concept]." By comparison, ISAEs learn a pair of SAEs (which have mostly interpretable latents) and sparse connections between them. At a con- ceptual level, JSAES tell us that 'this feature in the MLP's output was computed using only these few input features'. Ultimately, we believe that the JSAE approach, grounded in understanding how the SAE basis at one layer is mapped to the SAE basis at another layer, is potentially powerful and worth thoroughly exploring.
There also other approaches which focus on locating relevant computation in MI. models by estimating the con- tribution of individual model components (Shah et al., 2024; Balasubramanian et al., 2024).
3. Background
Importantly, it is worth emphasizing that JSAES and transcoders are asking fundamentally different questions, as can be seen in terms of e.g., differences in what they sparsify. As such, it is not, to our knowledge, possible to design meaningful quantitative comparisons, least not without extensive future work to develop very general auto- interpretability methods for evaluating methods of under- standing MLP circuits.
3.1. Sparse autoencoders
In an SAE, we have input vectors, x E === We want to approximate each vector by a sparse linear combination of vectors, S, E S. = R.. The dimension of the sparse vector, nu is typically much larger than the dimension of the input vectors my (i.c. the basis is overcomplete).
In the case of SAEs, we treat the vectors as inputs to an autoencoder with an encoder 6x : X + S, and decoder
3

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
de: : Sxx + X' defined by,
of the sparse bases learned by the SAE pair for the original vectors x and y. Specifically, we define Ss by
5, =x(xx= = 6 W + (
(1)
8 = dh (sx) - W st + b
(2)
S. = o f o
(3)
Here, the parameters are the encoder weights Wene E R" decoder weights Wase E R,, encoder bias byte E "R".. and decoder bias by E RRm The non- linearity & can be, for instance, ReLUU These parameters are then optimized minimize the difference between x and &, typically measured in terms of the mean squared error (MSE), while imposing an 1,- penalty on the latent activations Sx to incentivize sparsity.
where o denotes function composition. Here, d2 : S, + X' maps the sparse latents given as input to Sa to "dense" inputs. Then, f f: X - y maps the dense inputs to dense outputs. Finally, ey : y + Sy maps the dense outputs to sparse out- puts. Note that fs first applies the TopK activation function Th to the sparse inputs, Sx- Critically, with k-sparse SAES, we produce the sparse inputs by 8, = ex(x), implying that 8, only has & non-zero elements. In that setting, Topk does not change the inputs, i.c. 9x = The (5x), but it does affect the Jacobian and, in particular, allows us to compute it much more efficiently (Section 4.2).
3.2. Automatic interpretability of SAE latents
In order to compare the quality of different SAEs, it is desirable to be able to quantify how interpretable its latents are. A popular approach to quantifying interpretability at scale is to collect the examples that maximally activate a given latent, prompt an LLM to generate an explanation of the concept the examples have in common, and then prompt an LLM to predict whether a given prompt activates the SAE latent given the generated explanation. We can then score the accuracy of the predicted activations relative to the ground truth. There are several variants of this approach (c.g., Bills et al., 2023; Choi et al., 2024); in this paper, we use "fuzzing" where the scoring model classifies whether the highlighted tokens in prompts activate an SAE latent given an explanation of that latent (Paulo al., 2024).
At a high level, we want the function fs to be "sparse". in the sense that each of its input dimensions (i.e. SAE latent activations) only affects a small number of its output dimensions, and each of its output dimensions only depends on a small number of its input dimensions. We quantify the sparsity of fs in terms of its Jacobian matrix. The Jacobian of fa is, in index notation:
J = (sx)(
(4)
0sx.j
Intuitively, we can consider maximizing the sparsity of the Jacobian as minimizing the number of edges in the compu- tational graph connecting the input and output nodes (Fig- ure 1), i.e. maximizing the number of near-zero elements in the Jacobian matrix. We note that the Jacobian is not perfect measure of the sparsity of the computational graph, but it is an accurate proxy (see Section 5.4 and Appendix B) while being computationally tractable.
4. Methods
The key idea with a Jacobian SAE is to train a pair of SAES on the inputs and outputs of a neural network layer while additionally optimizing the sparsity of the Jacobian of the function that relates the input and output SAE latent acti- vations (Figure 1). In this paper, apply Jacobian SAES to multi-layer perceptrons (MLPs) of the kind commonly found in transformer language models (Radford et al., 2019; Biderman et al., 2023).
We simultancously train two separate SAES on the input and output of a transformer MLP with the objectives of low re- construction error and sparse relations between the separate SAE latents (via the Jacobian). We do not need to optimize for the sparsity of the latent activations via penalty term in the loss function because we use k-sparse autoencoders, which keep only the largest latent activations per token position. Hence, our loss function is
4.1. Setup
Consider an MLP mapping from x E X' y E D, i.e., f X + y or y = f(x). We can then train two k-sparse SAEs, one on x and the other on y. The resulting SAES map from each of and y to corresponding sparse latents Sx E && and Sy E Sy. i.c., Sz ex(x) and s, ey (y), where e, is the encoder of the first SAE and ey is the encoder of the second SAE. Each of these SAES also has decoder that maps from the sparse latents back to an approximation of the original vector: &- (Sx) and y = dy (Sy).
Fly
C = MSE(X, x) + MSE(y, 8) +
Measl (5)
Here, l is the number of non-zero elements in the TopK activation function, 7x, ny are the dimensionalities of the latent spaces of the input and output SAEs, respectively, and X is the coefficient of the Jacobian loss term. We divide by k² because, as we will see later, there are at most 82 non-zero elements in the Jacobian. Finally, note that if we set À = 0, then our objective effectively trains traditional SAEs for each of x and y independently.
We may now consider the function f. : Sx + Sy, which intuitively represents the function, f. but written terms
4

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Proportion of elements above threshold
75%
JSAEs
L
Traditional SAEs
Common tokens
50%-
German text
25%
Text is in German
Place names German-speaking countries
0%
0.005
0.010
0.015
0.020
about Nazi Germany
Threshold
Figure 2. ISAFs induce much greater degree of sparsity in the elements of the Jacobian of fa than traditional SAEs. The bars show the average proportion of Jacobian elements with absolute values above certain thresholds. At most k x elements can be nonzero,
Figure 3. JSAEs allow us to locate the "input features" of each feature computed by the MLP. For instance, in Pythia-410m, the MLP at layer 15 is computing the feature "this text is in German". JSAEs discover the inputs which the MLP uses to decide whether this feature should be or off. These inputs correspond to tokens froquently found in German text, place names German -praain-ppakkng countries, and text about Nazi Germany. See Appendix C for details.
take 100% the y-axis to mean l x k. The average was taken across 10 million tokens. This example is from layer 15 of Pythia-410m. For layer 3 of Pythia-70m and layer of Pythia- 160m, see Figure 34, for more quantitative information on Jacobian sparsity across model sizes, layers, and hyperparameters see Figures 24, 25, and 26. We present further discussion of the sparsity the Jacobian Appendix F.
4.2. Making the Jacobian calculation tractable
fs wrt any of the inputs that are "off" is zero. Without Tke the Jacobian could be non-zero for any of the inputs, even if changing those inputs would not make sense, as it would give more than k elements being "on" in the input, and thus could not be produced by the k-sparse SAE.
Computing the Jacobian naively (e.g., using an automatic differentiation package) is computationally intractable, the full Jacobian has size B x ny x n, where B is the number of tokens in training batch nt is the number of SAE latents for the input, and ny is the number of SAE latents for the output. Unfortunately, typical values are around 1, 1,000 for B and around 32, 000 for n, and ny (taking as an example model dimension of 1, 0,000 and expansion factor of 32). Combined, this gives a Jacobian with around I trillion elements. This is obviously far too large to work with in practice, and our key technical contribution is to develop an efficient approach to working with this huge Jacobian.
Our second insight was that computing the Jacobian by automatic differentiation would still be relatively inefficient, e.g., requiring k backward passes. Instead, for standard GPT-2-style MLPs, we noticed that extremely efficient Jacobian formula can be derived by hand, requiring only three matrix multiplications and along with a few pointwise operations. We present this derivation in Appendix A.
With these optimizations in place, training a pair of JSAEs takes about twice as long as training a single standard SAE. We measured this by training ten of each model on Pythia- 70m with expansion factor of 32 for 100 million tokens on RTX 3090. The average training durations were 72mins for a pair of JSAEs and 33 mins for traditional SAE, with standard deviations below 30 seconds for both.
Our first insight that for each element of the batch, we have a ny n Jacobian, where n, and ny are around 32, 000. This is obviously far too large. However, remember that we are interested in the Jacobian of fa, so the input is the sparse SAE latent vector, s, and the output is the sparse SAE latent vector, sy. Importantly, as we are using k-sparse SAEs, only & elements of the input and output are "on" for any given token. As such, we really only care about the
5. Results
elements of the Jacobian of fs, corresponding to the inputs and outputs that are "on". This reduces the size of the Jacobian by around six orders of magnitude, and renders the computation tractable. However, to make this work formally, we need all elements of the Jacobian corresponding to "off" elements of the input and output to be zero. This is where the Tą in the definition of fa becomes important. Specifically, the T% ensures that the gradient of
Our experiments were performed on LLMs from the Pythia suite (Biderman et al., 2023), the figures in the main text contain results from Pythia-410m unless otherwise specified. We trained on 300 million tokens with k = 32 and an expan- sion factor of 64 for Pythia-4100 and 32 for smaller models. We reproduced all our experiments on multiple models and found the same qualitative results (see Appendix E).
5

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Cosine Similarity
Explained Variance
-10 2 Mean Squared Error
0.8
0.8
0.6
0.5
0.2
0.4
Cross- Pross-Entropy Loss Score
-10*
Num. Dead Features
Abs. Jacobian Values >> 0.01
0.8
600
0.6
400
0.4
200
0.2
0
11
11 m.
1000
1200
10
10
100
10
10
10
10
10
10
10
100
10'
102
10
10
10
10
10
10'
102
10
Jacobian Loss Coefficient
Jacobian Loss Coefficient
Jacobian Loss Coefficient
Input SAE
Output SAE
Jacobian
Figure 4. Reconstruction quality, model performance preservation, and sparsity metrics against the Jacobian loss coefficient. ISAEs trained layer of Pythia- 160m with expansion factor 64 and & 32; see Figure 26 for layer 3 of Pythia-70m. Recall that the maximum number of non-zero Jacobian values is k²a 1024. In accordance with Figure 5, all evaluation metries degrade for values of the coefficient above 1. See Appendix E for details of the evaluation metrics.
5.1. Jacobian sparsity, reconstruction quality, and auto-interpretability scores
reconstructions. Unsurprisingyy as À grows larger and the Jacobian loss term starts to dominate, our evaluation metrics degrade. Interestingly, this degradation happens almost entirely in the output SAE rather than the input SAE we leave it to future work to investigate this phenomenon further.
First, we compared the Jacobian sparsity for standard SAEs and JSAEs. Note that, unlike with SAE latent activations, there is no mechanism for producing exact zeros in the Jacobian elements corresponding to active latents. Hence, we consider the number of near-zero elements rather than the number of exact zeros. To quantify the differennce in sparsity between the two, we looked at the proportion of the elements of the Jacobian above particular threshold when aggregating over 10 million tokens (Figure 2). Here, we found that JSAEs dramatically reduced the number of large elements of the Jacobian relative to traditional SAES. We also note that the Jacobians are not only sparse on each individual token, but also when averaged across a large number of tokens (see Figure 36 in the appendix).
Critically, Figure 4 suggests there a 'sweet spot' of the À hyperparameter where the SAE quality metrics remain reasonable, but the Jacobian is much sparser than for stan- dard SAEs. To further investigate this trade-off, we plotted measure of Jacobian sparsity (the proportion of elements of the Jacobian above 0.01) against the average cross-entropy (Figures 4, 5, and 29). We found that there is indeed a sweet spot where the average cross-entropy is only slightly worse than traditional SAE, while the Jacobian is far sparser. For Pythia 410m (Figure 5) this value is around À 0.5, whereas for Pythia-70m, it is around À = 1 (Figure 29). We choose this value of the Jacobian coefficient (i.e. \ = 0.5 for Pythia-4100 10m in the main text, and À = 1 for Pythia-160m in the Appendix) in other experiments.
Importantly, the degree of sparsity depends on our choice of the coefficient \ of the Jacobian loss term. Therefore, we trained multiple JSAES with different values of this parameter. As we might expect, for small values of X, i.c., little incentive to sparsify the Jacobian, the input and output SAEs perform similarly to standard SAEs (Figure blue lines), including in terms of the variance explained by the reconstructed activation vectors and the increase in the cross- entropy loss when the input activations are replaced by their
We also measure the interpretability of ISAE latents using the automatic interpretability pipeline developed by Paulo et al. (2024) and compare this traditional SAEs. We find that JSAEs achieve similar interpretability scores (Figure 6).
6

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
10
Number of Jacobian elements above 0.01
Tradiiional SAEs (Randomized LLM)
Tradtiioal SAEs
ISAEs (Randomized LLM)
250
JSAEs
0.1
Proportion of elements above threshold
0.05
80%
500
0:01
60%
0.001
0.5
0.6
0.7
0.8
40%
Average cross-entropy score (1 = perfect reconstruction)
20%
Figure 5. The trade-off between reconstruction quality and Jaco- bian sparsity as we vary the Jacobian loss coefficient. Each dot represents pair of JSAEs trained with specific Jacobian coeffi- cient. The value of À is included for some points. We can see that coefficient of roughly
0%
0.005
0.010
0.015
0.020
Threshold
0.5 optimal for Pythia-410m with k = 32. Note that the CE loss score is the average of the CE loss scores of the pre-MI.P ISAF and the post-mip ISAF. Measured on layer 15 of Pythia-410m, similar charts with wider range of models metries can be found Figures 7, 28, 29,
Figure 7. Jacobians are substantially more sparse in pre-trained LLMs than randomly initialzied transformers. This holds both when you actively optimize for Jacobian sparsity with JSAEs, and when you oottimize for it use traditional SAEs. The figure shows the proportion of Jacobian elements with absolute values above certain thresholds. most k elements can nonzero, we therefore take k² be 100% the y-axis. Jacobians signifi- cantly more sparse in pre- pre-trained transformers than in randomly re- ee-initialized transformers. This shows that Jacobian sparsity is, least to some extent, connected to the structures that LLMs learn during training. This stands in contrast recent work by Heap ct al. (2025) showing that traditional SAEs achieve roughly equal auto- iooointteprrttbbllity scores on randomly initialized transformers as they do on pre-tres pre-trained LLMs. Measured on layer 15 of Pythia- 410m layer Pythia-70m see Figure Averaged across 10 million tokens.
Auto Auto-intery interpterp score
100%
50%
ISAEs (irpul SAE)
Traditional (ippt SAE)
(output SAE)
Traditional SALs (ourput SAB)
0%
15
20
Layer
Figure 6. Automatic interpretability scores of JSAEs very simi- lar to traditional SAEs. Measured on all odd numbered layers of Pythia-410m using the "fuzzing" from Paulo al. (2024). For all layers of Pythia-70m see Figure 37.
as "Pf", "sch", "Kle", and "von"
Names of places where people speak German, such as "Berlin" " or "Austria"
5.2. Max-activating examples of JSAEs
Words and phrases related to the Third Reich, such as "Nazi", "concentration camp", "Hitler", and "Holo- caust"
Next, we interpreted the amax-activatinng examples of ISAEs order to verify that ISAEs can locate semanti- cally meaningful computational units. Namely, we took latents of output SAE i which have large Jacobian values when averaging across a wide distribution of text. Then for each output SAE latent i, we found the 10 input SAE latents which have the largest average Jacobian el- ements Jf..ij. We find that these combinations are often highly interpretable. For example, as shown in Figure 3, the very first output latent layer 15 of Pythia-410m as sorted by average Jacobian value corresponds to "this text in German". We find that computed function of input latents corresponding to:
For few of handpicked examples, see Appendix C. A large number of examples which are handpicked is available at tinyuul.com/jsacs-uuuliiic
5.3. Performance re-initialized transformers
To confirm that JSAEs are extracting information about the complex learned computation, considered form of control analysis inspired by Heap et al. (2025). Specifically, we would expect that trained transformers have carefully learned specific, structured computations while randomly initialized transformers do not. Thus, a possible desider- atum for tools in mechanistic interpretability is that they
Tokens which frequently appear in German text, such

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
(a)
(b)
(c)
Linear
scalar functions in fs
80%
ISAEs
Traiiiinaal SACs
Change in downstream latent (abs. value)
0.6
Proportion of
JumpRean
60%
0.4
Sy.
40%
0.2
Other
20%
0.0
0%
5
Linear
JumpReLU
Other
0.0
0.2
i
Jacobian element (abs. value)
Figure 8. The function fs. which combines the decoder of the first SAF, the ML.,, and the encoder of the second SAF, is mostly linear. Specifically, the vast majority of scalar functions going from 8x.j to 8y,i are linear. (a) Examples of linear, JumpRel.U, U, and other functions relating individual input SAE latents and output SAE latents. See Figure 9 for more examples. (b) For the empirically observed s, and randomly selected j (of those corresponding active SAE latents), the vast majority of scalar functions from 8x,, sy, are linear. For details see Appendix B. The proportion of linear function also noticeably increases with JSAEs compared to traditional SAEs, meaning that ISAEs induce additional linearity in fa- (c) Because the vast majority of functions are linear, the Jacobian usually precisely predicts the change observed in the output SAE latent when we make large change to the input SAE latent's value (namely subtracting 1, note that the empirical median value of sx.j is 2.5). Each dot corresponds an (Sx.) 8y,s) pair. For 97.7% of pairs (across sample size of 10 million) their Jacobian value nearly exactly predicts the change we see in the output SAE latent when making large changes to the input SAE latent's activation, i.e. | = |. The seatter plot shows a randomly selected subset of 1,000 ($x,) 8y,x) pairs. For further details see Appendix B. Mcasured on layer 15 of Pythia-410m, for layer 3 of Pythia-70n 70m see Figure 39, for the linearity results on other models and hyperparameters see Figures 15, 16, and 17.
ought to work substantially better when analyzing the com-
5.4. fo is mostly linear
plex computations in trained LLMs than when applied to LLMs with randomly re-initialized weights. This is pre- cisely what we find. Specifically, we find that the Jacobians for trained networks are always substantially sparser than the corresponding random trained network, and this holds for both traditional SAEs and JSAEs (Figure 7). Further, the relative improvement in sparsity from the traditional SAE to the JSAE is much larger for trained than random LLMs, again indicating that ISAES are extracting structure that only exists in the trained network. Note that we also see that for traditional SAEs, there is somewhat more sparse Jaco- bian for the trained than randomly initialized transformer. This makes sense: we would hope that the traditional SAE basis is somewhat more aligned with the computation (as expressed by a sparse Jacobian) than we would expect by chance. However, it turns out that without a "helping hand" from the Jacobian sparsity term, the aligmment in a tradi- tional SAE is relatively small. Thus, Jacobian sparsity is property related to the complex computations LLMs learn during training, which should make it substantially useful for discovering the learned structures of LLMs.
Importantly, the Jacobian is a local measure. Thus, strictly speaking, a near-zero element of the Jacobian matrix implies only that a small change to the input SAE latent does not affect the corresponding output SAE latent. It may, however, still be the case that a large change to the input SAE latent would change the output SAE latent. We investigated this question and found that fa is usually approximately linear in a wide range and is often close to linear. Specifically, of the sealar functions relating individual input SAE latents $x,j to individual output SAE latents 8y.i, the vast majority are linear (Figure 8b). This is important because, for any linear function, its local slope is completely predictive of its global shape, and therefore, near-zero Jacobian element implies near-zero causal relationship. For the scalar functions which are not linear, we frequently observed they have a JumpReLU structure' (Erichson et al., 2019). Notably, a JumpReL.U is linear in subset of its input space, so even for these scalar functions the first derivative is still an accurate measure within some range of 8x. values. It is also worth
By JumpReLU, we mean any function of the form f(x) aJumpReLU + c). Recall that JumpReLU(x)) = if if > d and otherwise. a, b, d E are constants.
8

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
noting that with JSAEs, the proportion of linear functions is noticeably higher than with traditional SAEs, at least certain extent, JSAEs induce additional linearity in the MLP. To confirm these results, we plotted the Jacobian against the change of output SAE latent Sy,a as we change the input SAE latent 5x.j by subtracting (Figure 8c)?. We found that 97.7% of the time, | | Rs | For details see Appendix B. While these results are strongly suggestive, we would caution that it is difficult interpret them definitively as we are not evaluating the reconstruction enror for linear model fitted to the input-output relationship for the MLP latents.
Impact Statement
The work presented in this paper advances the field of mech- anistic interpretability. Our hope is that interpretability will prove beneficial in making LLMs safer and more robust in ways ranging from better detection of model misuse to editing LLMs to remove dangerous capabilities.
Author contribution statement
Conceptualization was done by L.F and LAA. Derivation of an efficient way to compute the Jacobian was done by LF and LA. Implementation of the training codebase was done by LF. The experiments in Jacobian sparsity, auto- interpretabilityy, reconstruction quality, and approximate lin- earity of Sa were done by LF, Qualitative examples of the computations found by JSAFS were done by TL.. LA and CH provided supervision and guidance throughout the project.
6. Conclusion
We introduced Jacobian sparse autoencoders (JSAEs), new approach for discovering sparse computation in LLMs in fully unsupervised way. We found that JSAEs induce spar- sity in the Jacobian matrix of the function that represents an MLP layer in the sparse basis found by JSAEs, with minimal degradation in the reconstruction quality and downstream performance of the underlying model and no degradation in the interpretability of latents. We demonstrated that the computation found by JSAFs is often highly interpretable, allowing to see not only the concepts computed by MLPs, but also the "input concepts" which are used to compute each "output concept". We also found that Jacobian sparsity is substantially greater in pre-trained LLMs than in ran- domly initialized ones suggesting that Jacobian sparsity is indeed a proxy for learned computational structure. Lastly, we found that Jacobians are highly accurate measure of computational sparsity due to the fact that the MLP in the JSAE basis consists mostly of linear functions relating input to output JSAE latents.
The text was written by LF, LA, TL, and CH. Figures were created by LF and TL with advice from LA and CH.
References
Balasubramanian, S., Basu, S., and Feizi, S. Decomposing and interpreting image representations via text in vits be- yond clip, 2024. URL 2406.01583 01583.
abs/
Balcells, D., Lerner, B., Oesterle, M., Ucar, E., and Heimersheim, S. Evolution of sae features across lay- ers in llms, 2024. URL https: httt:ps://axxi..rrrt abs/ 08869.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and Wal, O. V. D. Pythia: A Suite for Analyz- ing Large Language Models Across Training and Scal- ing. In Proceedings of the 40th International Confer- ence on Machine Learning, pp. 2397-2430. PMLR, July 2023. URL https::/p//...... /ttp::s///pccceeeeeee ngs. ml ..rees// v202 / wiiderran2a..atee..........
Acknowledgements
The authors wish to thank Callum McDougall and Euan Ong for helpful discussions. We also thank the contribu- tors to the open-source mechanistic interpretability tooling ecosystem, in particular the authors of SAELens (Bloom et al., 2024), which formed the backbone of our codebase. The authors wish to acknowledge and thank the financial support of the UK Research and Innovation (UKRI) [Grant ref EP/S02937711 and the University of Bristol. This work was carried out using the computational facilities of the Advanced Computing Research Centre, University of Bris- tol - http://www.. wwt.b.brstto.a... ac.k//ccc. We would like to thank Dr. Stewart for funding for GPU resources.
Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J. Wu, J. and Saunders, W. Language models can explain neurons in language models, May 2023. URL https: // //openaipublic..oo blo.. core windows, net/ neuron aeurron-explainne/ppprreee html. html.
Bloom, J., Tigges, C., and Chanin, D. SAELens. https:
Forr reference, the median value of $.jj without any interven-
//github. com htt.huub..co///blooouu.... /bl.:oooooommssssee..a. 2024.
tions is 2.5.
Braun, D., Taylor, J., Goldowsky N., and Sharkey, L. Identifying Functionally Important Features with End-to- End Sparse Dictionary Learning, May 2024.

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, TT, Turner, N., Anil, C., Denison, C., and Askell, A. Towards Monosemanticity: Decom- posing Language Models With Dictionary Learning, reuits.
Heap, T., Lawson, T., Farnik, L.,, and Aitchison, L. Sparse autoencoders interpret randomly initialized transform- ers, 2025. URL https:://arktt /ttp/aaaxvvv///////////////a.a org/ abs/ 17727.
2023. URL httpt::t///t/..... htttst:::://ttaaaee tttttss://trransformmeee pub/2023 aotonseeaattic-eeaaaaaee
Kingma, D. P. and Ba, J. Adam: A Method for Stochastic Optimization, January 2017. URL http:/ /arxiv. org/abs/1dde 6980. arXiv:1412.6980 [es].
Brinkmann, J., Wendler, C., Bartelt, C., and Mueller, A. Large language models share representations of latent grammatical concepts across typologically diverse lan- guages, 2025. URL https:/ /ttp///axvvvv//. org/ abs/ 2501.06346
Kissane, C.,, Krzyzanowski, R., Bloom, I., Conmy, A., and Nanda, N. Interpreting attention layer outputs with sparse autoencoders, 2024. URL sttps://arzs o org/ / abs / 2206 17759.
Cammarata, N., Carter, S., Goh, G., Olah, C., Petrov, M., Schubert, L., Voss, C., Egan, B., and Lim, S. K. Thread: Circuits. Distill, 5(3), March 2020. ISSN 2476-0757. doi: 10.23915/disiil.000022
Kramár, J., Lieberum, T., Shah, R., and Nanda, N. Atp An efficient and scalable method for localizing IIm behaviour components, 2024. URL https: http:///xxvvvorrgee abs /243340034.555
Choi, D., Huang, V., Meng, K., Johnson, D. D., Steinhardt, J., and Schwettmann, S. Scaling Automatic Neuron De- scription, Octoher 2024. URL ht.p::::t///s......m htt///aansslull luce. org/ org/neuuee neuron htte:/uuuonndescrieeee descriptions.
Lan, M., Torr, P., Meek, A., Khakzar, A., Krueger, D., and Barez, Sparse autoencoders reveal universal feature spaces across large language models, 2024. URL https:
Conmy, A., Mavor-Parker, A., Lynch, A.. Heimersheim, S., and Alrrioa-Alonso, A. Towards Automated Circuit Discovery for Mechanistic Interpretability. Advances in Neural Information Processing Systems, 36:16318- 16352, December 2023.
//arxiv. org/ /a0bb//41.......
Lawson, T., Farnik, L., Houghton, C., and Aitchison, L. Residual Stream Analysis with Multi-Layer SAEs, Octo- ber 2024.
Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse Autoencoders Find Highly Inter- pretable Features in Language Models, October 2023.
Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L., Sonnerat, N., Varma, V., Kramár, J., Dragan, A., Shah, R., and Nanda, N. Gemma Scope: Open Sparse Autoen- coders Everywhere All At Once on Gemma 2, August 2024.
Dunefsky, J., Chlenski, P., and Nanda, N. Transcoders Find Interpretable LLM Feature Circuits, June 2024.
Erichson, N. B., Yao, Z., and Mahoney, M. W. Jumprelu: A retrofit defense strategy for adversarial attacks, 2019.
Makelov, A. Sparse Autoencoders Match Supervised Fea- tures for Model Steering on the I0I Task. In ICML 2024 Workshop on Mechanistic Interpretabilitty June 2024. URL https:/ httpss////ppeeeeeeeeeeome wwt.:riwwww.eee..eeeeeeeeeee forum? id=JdrVuBoihs.
URL hwtps:tp///.. /wt./aivvv.oooo org. abs 904. 03750.
Farrell, E., Lau, Y.-T., and Conmy, A. Applying sparse autoencoders to unleam knowledge in language mod- els, 2024. URL ptttts:://ariiie orw. hatp///bb//.0./////////// 19278.
Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., and Mueller, A. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs Language Models, March 2024.
Gao, I., Biderman, S., Black, S., Golding, I.., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The Pile: An 800GB Dataset of Diverse Text for Language Modeling, December 2020.
Meng. K., Bau, D., Andonian, A., and Belinkov, Y. Locat- ing and Editing Factual Associations in GPT. Advances in Neural Information Processing Systems, 35:17359- 17372, December 2022.
Gao. L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scaling and evaluating sparse autoencoders, June 2024.
Hanna, M., Liu, O., and Variengien, A. How does GPT-2 compute greater-than?: than?: Interpreting mathematical abili- ties in a trenned language model. Advances in Neural Information Processing Systems, (6.76633376060, De- cember 2023.
Nanda, N. tion Patching At Industrial Scale, February 2023. actrribution-pptcchinn
Attribution Patching: Activa-
URL https: http:/wwwweeean.....
10

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
O'Brien, K., Majercak, D., Fernandes, X., Edgar, R., Chen, J., Nori, H., Carignan, D., Horvitz, E., and Poursabzi- Sangde, F. Steering Language Model Refusal with Sparse Autoencoders, November 2024.
Syed, A., Rager, C., and Conmy, A. Attribution Patching Outperforms Automated Circuit Discovery. In Belinkov, Y., Kim, N., Jumelet, J., Mohebbi, H., Mueller, A., and Chen, H. (eds.), Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 407-416, Miami, Florida, US, November 2024, Association for Computational Linguistics. doi: 10.18633/w1222244cccooooll
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. Zoom In: An Introduction to Circuits. Distill, 5(3), March 2020. ISSN 2476-0757. doi: 10. 2391 5/distill.00024..000
Templeton, A., Batson, J., Jermyn, A., and Olah, C. Pre- dicting Future Activations, Jamuary 2024a. URL https:
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A. Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Heraandez, D.. Johnston, S., Jones, A., Kernion, J., Lovitt, L.,, Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J.. McCandlish, S., and Olah, C. In-context Learning and Induction Heads, September 2022.
/ / jan-updatee/nndex.ttna http:///wwwe....mm.oorrrrete---.
Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDi- armid, M., Tamkin, A., Durmus, E., Hume, T., Mosconi, F. Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and Henighan, T. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet, May 2024b. URL https:
Paulo, G., Mallen, A., Juang, C., and Belrose, N. Automati- cally Interpreting Millions of Features Large Language Models, October 2024.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language Models are Unsupervised Multi- task Learners, 2019. URL
/ strrassorrmr-cctt www.t:ssu.uuuuttuuu.......o scaling-monoseeeen monosemanticitty//nee...
openai.......oii...o/aoaaa camm//etter--aaggage-ooeeeaaaapp learners.pds. pdf.
Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, November 2022.
Rajamanoharan, S., Conmy, A., Smith, L., Lieberum, T., Varma, V., Kramar, J., Shah, R., and Nanda, N. Im- proving Sparse Decomposition Language Model Ac- tivations with Gated Sparse Autoencoders. ICML 2024 Workshop Mechanistic Interpretability, June 2024a. URL httpt:tt//// https/////peeeeeeeeeeeeee www.:ewww.eeeeoeeeeeeeeeeeeeeeee forum? id-Ppj5Kv2U8Q.
Yun, 7., Chen, Y., Olshausen, B., and LeCun, Y. Trans- former visualization via dictionary learning: contextual- ized embedding as a linear superposition of transformer factors. In Agirre, E., Apidianaki, M., and Vulić, I. (eds.), Proceedings of Deep Learning Inside Out (DeeLJO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 1-10, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 deellio-.eeel111... 1.1.
Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy, A., Varma, V., Kramár, J., and Nanda, N. Jumping Ahead: Im- proving Reconstruction Fidelity with JumpReLU Sparse Autoencoders, July 2024b. URL http:// arxiv. org/abs/ 2407. 1....44533 14435. arXiv:2407.14433 [es].
Zhang, F. and Nanda, N. Towards Best Practices of Ac- tivation Patching in Language Models: Metrics and Methods. In The Twelfth International Conference Learning Representations, October 2023. URL https: / //openreview net www.oooidddoo.oo..oo..oo 7 6u9BC.
Shah, H., Ilyas, A., and Madry, Decomposing and editing predictions by modeling model computation, 2024. URL hittps://aran org/ abs////bb//aa /4444 11534.
Sharkey, I., Braun, D., and Millidge, B. Taking features out of superposition with sparse autoencoders, December 2022.
Spies, A. F., Edwards, W., Ivanitskiy, M. I., Skapars, A., Räuker, T., Inoue, K., Russo, A., and Shanahan, M. Transformers use causal world models in maze- solving tasks, 2024. URL https:://// htapp:://axxiv...// http///xxvvvvog///////.//// abs/2412. 218677

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
A. Efficiently computing the Jacobian
simple form for the Jacobian of the function f. - Ey o f dx o Tk, which describes the action of an MLP layer sparse input and output bases, follows from applying the chain rule. Note that here, the subscripts fax ey. etc. denote the function in question rather than vector or matrix indices. For the GPT-2-style MLPs that study, the components of fs are:
1. TopK. This function takes sparse latents S, and outputs sparse latents 8x. Importantly, 8, = 8.- This step makes the backward pass of the Jacobian computation more efficient but does not affect the forward pass.
Sx (sx)
(6)
2. Input SAE Decoder. This function takes sparse latents Sx and outputs dense MLP inputs &:
X d₂ (8.)
W sx + b
(7)
3. MLP. This function takes dense inputs & and outputs dense outputs y:
z = W - b
W20MLP + b2
(8)
where ÓMLP is the activation function of the MLP (e.g., GeLU the case of Pythia models).
4. Output SAE Encoder. This function takes dense outputs y and outputs sparse latents sy:
= ey( = (Wexy y + by
(9)
The Jacobian J,, X* for single input activation vector has the following elements, in index notation:
J = 08y.1
Dsy, Oyk Oze 0sm 08x, Oyk Oze dhm 0= Osn.j
Σ
(10)
Os..j
kemmn
We compute each term like so:
1. Output SAE Encoder derivative:
Osy. Oyk
The
Σ Woone yj Denc.i
Weno y,ak
Wenc y,sk
if IC2 otherwise
(11)
0
where K2 is the of indices selected by the TopK activation function тı of the second (output) SAE, Importantly, the subscript i does not indicate the je-th element of Tk. whereas it does indicate the k-th column of Wrene
2. MLP derivatives:
Oyk
Oze
Wa,e (x
W
(12)
0z
0i
3. Input SAE Decoder derivative:
Widee
(13)
08,m
x.m
4. TopK derivative:
i otherwise
(14)
0sx..
where XG, is the set of indices (corresponding to SAE latents) that were selected by the TopK activation function тy of the first (input) SAE, which we explicitly included in the definition of fa above.
12

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
When combine all the terms:
Exem
y,sk Ween W2, (2L( W1,em
Widee x,myf
i KejjE,
J.,55 = 0
(15)
otherwise
Let Wenclactive) R and Widcetactive) E m contain the active rows and columns, i.e., the rows and columns corresponding to the K2 or K indices respectively. The Jacobian then simplifies to:
J (activv)
Wenelactivo W₂
MLL (z)
W Wololactive))
(16)
R MULP
R Bh danp
where dMLP the hidden size of the MLP. Note that j(active) is of size x k, while the full Jacobian matrix J,, is of size By T2. However, j(actvo) contains all the nonzero elements f., so it is all need to compute the loss function train Jacobian SABs (Section 4.
A.1. JSAEs with GLUs
The equations above can be easily adapted work with gated linear units (GLUs), which are significantly more common in modern LLMs than GPyl-2-style MIPs.
To do this, we modify the MLP equations like so:
g Wg + bg
(17)
S @MLP(g)
(18)
h Wi + b
(19)
Z h ④
(20)
y = W22 + b2
(21)
where O is elementwise multiplication.
We then modify the derivatives accordingly:
Oyk W2.kt
(22)
Uzt
Ozt hi gam
Osi
The
+ 81 Dim
(23)
02m
The W1,m
(24)
Da
USE d <LL
(25)
0ge
0ge Wa g.fm
(26)
Fām
(27)
Combining the terms again:
E Entm ktm Wene Wa,e (he (hLP((E) Wg.t + 88 Wi.em) yl
Widse x.jj
if E 122 Aj E Kc
Ji.d =
(28)
otherwise
The Jacobian is then:
J (active))
Wencfactivve W2 (diag (h O (gr((g) Wg + diag(s) Wi) di,
W Wedectattvv))
(29)
fo
R gexdau
RAAML Xm..
Remxk Xk
13

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
B. I& is approximately linear
Consider the scalar function fم((,)))s, : R + R which takes as input the j-th latent activation of the first SAE (i.e. 8x,)) and returns as output the i-th latent activation of the second SAE (i.e., Sy,3), while keeping the other elements of the imput vector fixed at the same values as Sx. In other words, this function captures the relationship between the j-th input SAE latent and the i-th output SAE latent in the context of sx- Geometrically, start off at the point s,, and we move from it through the input spaces in parallel to the j-th basis vector, and then we observe how the output of Ss projects onto the i-th basis vector. Formally.
(a(t)))) = (1x(sx) 2, z)
(30)
fx if k
((s,,,,));
(31)
8x,5
otherwise
These are the functions shown in Figure 8a, of which the vast majority are linear (Figure 8b).
As we showed in Figure 8c, the absolute value of a Jacobian element nearly perfectly predicts the change we see in the output SAE latent activation value when we make large intervention on the input SAE latent activation. However, in the same figure, there is small cluster of approximately 2.5% of samples, where the Jacobian element is near zero, but the change observed in the downstream feature is quite large. We proceed by exploring the cause behind this phenomenon.
Note that each point in Figure 8 corresponds to a single scalar function fs,(o,))ss ls. (a pair of latent indices). An expanded version of Figure is presented in Figure 10. Importantly, we show the 'line", the top-left cluster, and outliers visible in Figure 8 in different colors, which we re-use in the following charts (Figures 11 and 12). It also includes 10K samples, compared 1K in Figure 8c: as above, most samples remain on the line, but the greater number of samples makes the behavior of the top-left cluster and outliers clearer.
Figure 11 illustrates some examples of functions fs.(t.))ss. taken from each category shown in Figure 8, i.c. the line, cluster, and outliers. The vast majority of functions belong to the line category and are typically either linear or akin to Jump activation functions (which include step functions as a special case). By contrast, the minority of functions belonging to the cluster or outliers are typically also JumpReLU-like, except where the unmodified input latent activation is close to the point where the function 'jumps', so when we subtract an activation value of from the input (as Figures 8c and 10), this moves to the flat region where the output latent activation value is zero.
As see, the vast majority of these functions are either linear JumpReLUs. Indeed, we verify this across the sample size of 10,000 functions and find that 88% are linear, 10% are JumpReLU (excl. linear, which is arguably special case of JumpReLU), and only 2% are neither?. This result is encouraging for a linear function, the first-order derivative is constant, so its value (i.e., the corresponding element of the Jacobian) completely expresses the relationship between the input and output values (up to a constant intercept). For the 88% of these scalar functions that are linear, the Jacobian thus accurately captures the notion of computational sparsity that interests us, rather than serving only as a proxy. And for the 10% of JumpReLUs, the Jacobians still perfectly measure the computational change we observe when changing the input latent within some subset of the input space.
While we expect the remaining 2% of scalar functions (Jacobian elements) to contribute only small fraction of the computational structure of the underlying model, we preliminarily investigated their behavior. Figure 12 shows 12 randomly selected non-linear, non-JumpReLU fa.(t.)) functions. Even though these functions are nonlinear, they are still reasonably close to being linear, i.e., their first derivative is still predictive of the change we see throughout the input space. Indeed, most of them are on the diagonal line Figure 10.
14

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
1.0
2
2
1.0
Linear
2
0.5
1
0.5
0.0
0
0
0.0
5
JumpReLU
0.5
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
0
0
5
5
1.0
Other
0.5
0.5
0.5
0.5
0.5
0.0
0.0
0.0
5
0
0
5
0
5
5
Figure 9. Additional examples of scalar functions between 8x. to 8y,i- The top row shows linear functions, the middle row shows JumpReLU functions, and the bottom shows other functions. Recall that linear functions constitute majority of the functions observe empirically and that using JSAEs instead of traditional SAEs further increases the proportion of lincar functions.
15

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Correlation between Jacobian value and change in downstream feature
1.0
Change in downstream feature (absolute value)
0.8
0.6
0.4
0.2
0.0
0.0
0.1
0.2 Jacobian element (absolute value)
0.3
0.4
Figure 10. An expanded version of Figure 8c, measured on layer of Pythia- 70m. scatter plot showing that values of Jacobian elements tend to be approximately equal to the change we see in the downstream feature when we modify the value of the upstream feature, namely when subtract 1 from it. Each dot corresponds the dots depending on which cluster they belong to blue for "on the line", green for "in the cluster", red for "outlier". Additionally, this figure contains 10,000 samples (rather than 1,000 Figure 8c), which allows us to see more the outliers and edge cases, though the cost of visually obfuscating the fact that 97.5% of the samples are on the diagonal line, 2. 1% are in the cluster, and 0.4% are outliers.
(input SAE latent, output SAE latent) pair. Unlike Figure 8c, this figure colors in
16

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
2
0.5
0.5
0.0
0.0
0
Downstream activation
5
0
0
5
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
5
5
0
5
2
0.5
0
0
0.0
5
0
5
0
5
Upstream activation
Upstream activation
U Upstream activation
Upstream activation
Figure 11. A handful of fa,(t)) functions corresponding the points in Figure 10.
The color matches the group (and therefore the
color) they were assigned Figure 10. The dashed vertical line denotes { i.c. the activation value of SAE latent before we intervened it. Note that the functions selected randomly but rather hand-selected to demonstrate the range of functions. We will quantitatively explore what proportion of fa,(i.) s. functions have which structure in other figures.
17

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
0.5
0.5
0
0.0
0.0
Downstream activation
2.5
0.5
0.0
0.0
5
5
5
0.5
2
2.5
0.0
0
0.0
0
10
5
Upstream activation
Upstream activation
Upstream activation
Upstream activation
Figure A random selection of non-linear, non-JumpReLU fa,tt,) functions. only constitute about 2% of fa.(i.) s, functions. Even though these functions clearly somewhat non-linear, their slope does still change quite slowly
non non-linear, non-JumpReLU functions
most part, which means that first-order derivative at any point in function still reasonably predictive the function's is behavior in at least some portion of the input space (though there are some rare exceptions). The color again matches the
group (and therefore the color) they were assigned in Figure 10; the red dashed vertical line denotes (!) i.e. the activation value of the SAE latent before we intervened
18

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
5000
Count
2500
0
-0.02
0.00
0.02
0.04
0.06
Mean second derivative the function
(b)
Count
5000
100
200
300
400
500
Mean absolute value of second derivative across the function
(c)
Count
5000
0
10000
20000
30000
40000
50000
60000
Maxmum absolute value of second derivative across the function
Figure 13. Distribution of second-order derivatives of functions fa. Includes all functions, regardless of whether they are linear, JumpReI.U, or neither. For a version that only includes non-linear, non-JumpReL.U functions, see Figure 14. (a) The mean of the second-order derivative region of the input space. (c) The maximum value the second-oue order derivative takes in the region of the input space. Note that are limit as
the region of the input space. (b) The mean of the absolute value of the second-order derivative
the
approximating the second derivative by looking changes over very small region (specifically 0.005), i.c., we the size of this small region goes to zero; this is important because derivatives which would otherwise be undefined or infinite become finite with this approximation and therefore can shown on histograms. Also, we note the means maxima are over the region of the input space in which SAE features exist; see the footnote on page 14.
19

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
(a)
40
Count
20
0.03
0.02
0.01
0.00
0.01
0.02
0.03
0.04
Mean second derivative across the function
(b)
200
Count
100
100
150
200
250
300
350
Mean absolute value of second derivative across the function
(c)
200
Count
100
0
5000
10000
15000
20000
25000
30000
35000
Maxmum absolute value second derivative across the function
Figure Distribution of second-order derivatives functions fs.(i.))aa la. Unlike Figure 13, figure only includes the subset of the functions that are neither linear nor JumpReLU-Iip (a) The mean of the second-orde derivative over the region of the input space. (b) second-order derivative over region of the input space. (c) maximum value second-order
mean the absolute value derivative takes in the region of the input space. Note that small region (specifically 0.005), i.c. we do not take the limit as the size of this small region goes to zero; this is important because derivatives which would otherwise be undefined infinite become finite with this approximation and therefore can be shown the histograms. Also, we note that the means and maxima are taken over the region of the input space in which SAE features exist; see the footnote page 14.
approximating the second derivative by looking at changes over very
20

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Pythia-70m
Pythia- Pythia-160m 160m
Pythia-410m
Fraction of Pairs
0.8
0.6
0.4
0.2
0
0
2
10
0
10
20
Layer
Layer
Layer
Linear
JumpReL.U
Other
Figure The fractions Jacobian elements that exhibit linear relationship between the input and output SAE latent activations, JumpReLU-like relationship. and uncategorized relationship. as described in Section 5.4. Here, we consider Jacobian SAEs trained on the feed-forward network at different layers of Pythia-70m, 160m, 410m with fixed expansion factors R 64 and 32. computed the fractions over I million samples.
Pythia-70m
Pythia- 160m
Fraction of Pairs
0.5
10
3
10
10
10
10
10a
10
10
10
2
10
10
10
10
10a
Jacobian Loss Coefficient
Jacobian Loss Coefficient
Lincar
JumpReLU
Other
Figure 16. The fractions of Jacobian elements that exhibit linear relationship between the input and output SAE latent activations, Ju-pReLU-like relationship. and an uncategorized relationship. described in Section 5.4. Here, we consider Jacobian SAEs trained on the beed-forward network layer Pythia-700 70m (left) and layer Pythia- 160m (right), with fixed expansion factors 64 and k = 32 and varying Jacobian loss coefficient (Section 4). We computed the fractions over I million samples.
Fraction of Pains
0.8
0.6
0.4
0.2
10
10a
10
10a
10
Number of Latents
Sparsity k Other
Linear
JumpReLU
Figure The fractions Jacobian elements that exhibit linear relationship between the input and output SAE latent activations, a JunpReLU-like relationship. and an uncategorized relationship. described in Section 5.4. Here, we consider Jacobian SAEs trained on the forddrr network at layer of Pythia-70m with varying expansion factors (and hence numbers of latents; left) but fixed sparsities k 32, and varying sparsities but fixed expansion factors R 64 (Section 4). We computed the fractions over I million samples.
21

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
We measure this more precisely by looking at the second-order derivative of (a.(t.)) s. A zero second-order derivative across the whole domain would imply linear function and, therefore, perfect predictive power of the Jacobian, while the larger the absolute value of the second-order derivative, the less predictive the Jacobian will be. This distribution is shown in Figure 13. The same distribution, which only includes the non non-linear, non-JumpReLU functions, is shown in Figure 14. On average, the second derivative is extremely small for all features and effectively zero for the vast majority.
C. Qualitative examples of the computations discovered by JSAEs
A common approach to interpreting LLM components like neurons or SAE latents is to collect token sequences and the corresponding activations over text dataset (e.g.. Yun et al., 2021; Bills et al., 2023). For example, the greatest latent activations may be retained, or activations from different quantiles of the distribution over the dataset (Bricken et al., 2023; Choi et al., 2024; Paulo et al., 2024).
We determined the set of 'top' output SAE latent indices by collecting the mean absolute values of non-zero Jacobian elements over a text dataset and sorting the output latents in descending order. Then, for each output latent, we found the input SAE latents that were most strongly connected to the output latent, again by sorting the input latents in descending order of the mean absolute value of non-zero Jacobian elements over the dataset. Finally, for both the output and input latents, we collected the individual latent activations over text samples with context length of 16 tokens, retaining samples where at least one token produced non-zero activation for the SAE latent. We chose short context length to conveniently display the examples in a table format, and display here the top eight examples for each latent index, sorting the examples in descending order of the maximum latent activation over its tokens.
Each of the following figures comprises table for single output SAE latent (in pink), and a series of tables for the input latents with the greatest influences on the output latent, as determined by the mean absolute value of non-zero Jacobian elements. Conceptually, one may consider each figure as describing single 'function', where the output and input latents represent the function output and imputs, respectively. Each table within the figure of examples displays list of at most 12 examples, each comprising 16 tokens; we exclude the end-of-sentence token for brevity. The values of non-zero Jacobian elements and the activations of the corresponding input and output SAE latent indices are indicated by the opacity of the background color for each token. We take the opacity to be the element or activation divided by the maximum value over the dataset, i.e., all the examples with a non-zero Jacobian element for a given pair of input and output SAE latent indices. For clarity, we report the maximum element or activation alongside the colored tokens.
NNote that we are testing whether functions are linear or JumpReLUs only in the region input space within which SAE activations exist. particular, this means that we are excluding negative numbers. More specifically, the domain within which we test the function's s structure is (0, max(5, 2, + 1)]. In 92% cases, [ + < 5; the median [ 2.5.
22

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
W enn Sie auf We iter lie ken ist Ih Bon us ver
5.596
A14ber fl A A1sss gew orden ist Lau f zeit 36 Mon ate
5.542
mussvon einer aus IA nd ischen Bank sein Log ischer we ise
5.522
Sie kAnnen sich eder zeit ab mommllee
Click Here
5.387
bei der Be re it st ell ung unse rer D ien ste \n
3.315
icher lich sch on ge A rt haben, kann es sehr ze
5.275
Geld \n ump No multiple accounts free bonuses in
5.254
we be borrowing food from China, next --hhhha aweh
5.247
k aufen en will kann ich tro tz dem das Gold isa be
5.225
bef A I 4r wort ete auch die vol stA nd ige ontrol le A A14ber
5.223
bonuscodes 2019, so dass ein We in ein W ett bA14
5.208
Immob il ie in Span ien auf ih N amen beh
5.207
(a) The top 12 examples that produce the maximum latent activations the output SAE latent index 34455.
Example tokens
Max. activation
dam it ein ver stand dass wir Cookics verww enden Die
1.292 x 10
we isen dass Sie nicht mehr als 11 Mon ate uss er hal
1.229 x 10
st super auf me ine 44n sche ing eg ang en Ins
1.173 x 101
Golden isa ), die gle ich zeit ig er la ub dass
1.167 x 10
TA14rdas Gold V isa in 2013 3 er lass Zu dies em
1.152 101
W enn Sie auf We iter k lic ken ist I hr Bon us ver
1.139 x 10
age W enn ich die Immob il ie durch eine Ges ells chaft
1.134 x 10
ung abs ch icken d auer ren ner zur 1 14ck le hnen
1.132 x 10
lich die ID arte ver Ang ert werden das sehr schn
1.126 x 101
te " 'n A uf alle W W sche ing eg ang en
1.125 10
keine oder an le icht f A14 Il bare B eding ungen gek
1.124 10
hat ap ien ein Ges etz er lass en dass einem icht
1.116 x 10
(b) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 39503, Paired with the output
SAE latent with index 34455, the Jacobian element is pon-zero zero for 15130 tokens, and has mean 3.966 10 (rank the output SAE latent) and standard deviation of 7.743 10-2 over its non-zero values.
23

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
ZA14 rich St utt gart Le ipzig
HH alle
6.834
CTCcad cts, students and staff' Ans bach Middle School
6.788
Out reach announced that twogroups of students from Ans bach Middle High School
6.693
ride from Zur ich to Rh inc Falls (witha section ofour
6.679
aud US5 Branden bur ger Tor 2us hr die
6.644
the Ans bach Middle High School track AnGrammyy awardwinning
6.566
Ans bach Germany on May 18 2015. Fryewas
6.546
parents to attend \n W ies baden students find matha little fish
6.480
into the Gi essen acht al valley OOn the trail along the edge
6.471
was the first case for the MCKoot len
office CCapt.
6.448
ol ie Follow the signs Branden bur ger Tor Brandenburg Gate
6.423
for the Kob len
office AnbMC' Captain Dennis Brand
6.391
(c) The top 12 examples that produce the maximum latent activations for the input SAF latent with index 3387. Paired with the output
SAE latent with index 34455, the Jacobian element is pon-zero for 10355 tokens, and has mean 3.437 10-1 (rank output
SAE latent) and standard deviation of 2.873 x 10 over its non-zero values.
Example tokens
Max. activation
DrAlex German his team found the main reason for that is the
1.149 10
for Nuclear Physics,
Rup recht Kar Is University the German Research Chris Kend rick, Sally Anne F itter Penny German More
1.142 10a
1.137 x 10
in Czech oslov akia 1 in the German Democratic
1.119 x 10
mind, intellect "), from Pro to- German ic mund iz
1.089 10
and ppo" Auuutbbahn articles from The German Way and
1.084 10a
and American soldiers both loved "L ili Mar lene the song that Germanand
1.075 10
Is Univers itAt the German Cancer Research Center ( DKFZZ),
1.070 10
In addition Dr German' team also confirmed that cats ate about
1.068 x 10
the German Constitution that addresses equality before the law.
1.064 10
under constant international external laboratory control of the Germanace reditation system In stand
1.052 10t
But the history of carn ival can be traced back to German ic tribes who
1.051 10a
(d) The top examples that produce the maximum latent activations
input SAE latent with index 41811, Paired with output SAE latent with index 34455, Jacobian element is non-zero for 764 tokens, has mean 1.619 x 10-1 (rank 2 output
SAE latent) and standard deviation of 8.654 x 10 over its non-zero values.
24

---

Jacobian Sparse Autoencoders:
Sparsify Computations, Not Just Activations
Example tokens
Max. activation
National Archives report
Hit ler
SShadow
Nazi War Crim inals
1.203 10a
of victims transported to Nazi concentration camps French railway company
1.195 x 10
Air CorCo after WW II (where he survived year as Nazi prisoner
1.170 x 10a
aname inspired by the chief Ukrainian Nazi leader Ste pan Band era
1.168 x 10
Powers refused cooperation with the company until connections with Nazi Germany were severed. An
1.155 x 10'
orrow, the World from1944,, about teen- age Nazi
1.145 10
000FrenchJews to Nazi concentration camps though experts disagree its degree of the dewishppoplee during the Nazi era it was the work of white Europeans
1.145 x 10²
1.118 x 10
computing pioneer and his of to crack Nazi codes in order it hadno effective control over operations during the Nazi occupation from 1940 1944
1.099 x 10"
1.090 x 10
it with water+ Nazichemiee chemicals, prayed god for assistance struck
1.079 x 10
symbol of Russian resistance to Nazi invasion... Russia ss Window on the West
1.067 x 10a
(e) The top examples that produce the maximum latent activations for the input SAE latent with index 32619, Paired with the output SAE latent with index 34455, the Jacobian element is non-zero 316 tokens, and has mean 1.518 x 10-1 (rank 3 output SAE latent) and standard deviation of 1.193 10-2 over its non-zero values.
Example tokens
Max. activation
ownedby the von W aken itz family in 14 34. The same family
1.605 10
World Industry Status Report2013 July 8n David von Hippel
1.526 10
as Marta, one of the singing von Trapp children Sheil2
1.509 10
knockdown kiss is administered when Von locks lips with GGway more enthusi astically than
1.437 10
Jassumee Gre Garbo here with Von and not
1.427 10a
HATETHAT IDON THATE Fel von
1.423 10a
was already published 150 years ago by the g german physician Herm von Hel
1.388 10
new life by Edward von LA ng us und reality technologies. exploring
1.377 10"
(19 28, Joseph von Stern berg silent AT ribute to
1.366 x 10a
ily Von has again been sight ed in that infamous suit Was Metroeconom
1.352 10"
artists and performers. known the Castle von Trapp, will immin
1.341 10
ung. Degen feld- F ests chr ift Vienna von Lag
1.320 10
(f) The top examples that produce the maximum latent activations for the input SAE latent index 63157. Paired with output SAE latent with index 34455, Jacobian element is non-zero 84 tokens, and has a mean 1.479 10 (rank the output SAE latent) and a standard deviation of 8.717 x 10 over its non-zero values.
25

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
Dum ont aG el rod, MSNBC al um now Berlin
1.040 10*
for determining adhesive and tensile strengths (w inner of the Berlin- Branden
1.026 10
That Dem on Within was selected for the 64 th Berlin International Film Festival Pan
1.025 x 10a
M irt 1, andM. Schmid, cds. Berlin Springer
1.017 x 10
processes in tropical forests (pp. 1-3- 172 ). Springer Berlin
1.011 10'
Twenty- five years after the fall of the Berlin wall in 1989. Coont
1.008 10*
the Berlin Wall legacy SA- 2, SA- 3, SA
9.963
emerged after the collapse of the Berlin Wall and the end of the cold war fall the Berlin Wall was great spur to Germany, though it took exhibitions museumsand galleries Cowe Town, Johannes burg, Berlinand of the following options for the Berlin to Sch ild ow route Michel in
9.932
9.898
9.885
9.885
have to get new one ( (with the Berlin citizen center,. it
9.847
(g) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 63657. Paired with the output SAE latent with index 34455, Jacobian element is non-zero tokens, and has mean 1.387 x 10 (rank 5 output SAE latent) and standard deviation of 6.091 x 10 over its non-zero values.
Example tokens
Max. activation
no zzles are known from the MercedesGLAA In addition Merreeess
3.361
anopenend- hole. Moreover, when cathe ters of this type
2.830
Nasm ith machine rollers for conveyor belt The invention relates inter alia
2.829
cathe ters which closed distal end. The principles of the invention therefore are also suitable An possible for you get your blood
2.757
2.487
analog matching devices can be used advantage ously for and monitoring and diagnostics of process
2.399
be the case that such support canbe secured, for example, in cases
2.249
ic acid Proper ventilation her ically sealed production apparatus are therefore of residence particular copyright data protection and competition law The provider products of this sopk great for increasing your store revvene,, as they action by the customer or from errors in the information byovided the latter
2.240
2.234
2.133
2.113
optionally rear axle differential lock An In contrast to the prospective main competitor
2.099
(h) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 7969, Paired with output SAE latent with index 34455, Jacobian element is non-zero for 331 tokens, and has mean 1.322 x 10-1 (rank output SAF latent) and standard deviation of 1.176 10 over its non-zero values.
26

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
anda private With public key. it A- possible
5.947
Cons olid ation 2592. In the last years of its active life
5.068
forward the realization of artistic works linked to this whole context
taking into account
4.883
test different runtimeeenviromment As consequence cloud- based tool ing currently release recommend to look at SF- Spring http :// customer satisfaction An What is more, with the app, it 's possible
4.585
4.558
4.501
of the hotel .oeel.Onnie the hotel land it is possible to camp in tents
4.438
introduce additional modifications not forget about re- publ ishing the whole site so
4.426
change requested, this tool enables to know which other requirements design and structures / steel constructions according the highest requirements 203, or, more precisely a mixture of NO
4.418
\n Dependingon
4.370
4.309
if you haveany questions (you can also havea look at our website
4.272
(i) The top 12 examples that produce the maximum latent activations for the input SAF latent with index 18964. Paired with the output SAE latent index 34455. Jacobian element non-zero for 3502 tokens. and mean 1.209 x 10-1 (rank output SAE latent) and standard deviation of 2.073 x 10 over its non-zero values.
Example tokens
Max. activation
will enjoy music inspired in or by America, France Austria
Scotland
1.170 x 10
in the world since its adoption by the Austrian army in 1977 Now available
1.116 x 10
Australia China, Austria England, Israel and Ireland \n AnTorough
1.112 x 10
teaching the basics ung Fu in Russia, Austria Spain, China
1.053 x 10"
occupied areas 1848 by the 1880 s, the Austrian governed
1.042 x 10
Russia, American Ambassador to Austria William Eacho,
wehad the which traces its roots to the work of the Card Austrian
1.033 x 10
American
1.016 x 10'
released in Germany, Austria and Switzerland. An Irish man in America was recorded
1.014 x 101
W ahl led to an invitation by the Austrian Ministry of Culture for myself
9.958
of State from Austria Poland, and Hungary, Min isters from Israel and
9.951
practice first abroad, in countries suchas Spain, Germany or Austria where
9.771
The Che came to Austria refugee,
9.769
(j) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 28112, Paired with output SAE latent index 34455, Jacobian element is non-zero for tokens, has mean 1.156 10-1 (rank 8 output SAE latent) and standard deviation of 8.429 10 over its non-zero values.
27

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
Reynolds & Gy Il en ha al Ar rives An Earlier tonight during
9.162
Gy en ha al Ryan Reynolds, and Ferguson star in the
8.938
Jim Mc El re ath Mario ret Gary Bet ten haus
8.771
M. Spikenn he uer A. agen A14
8.518
followed by Nick be in who finished fourth An If Friday night
8.185
aud U.55 Branden bur ger Tor Bus 20000 U hr die
8.145
the ritual iden acht en are not really a closed chapter of history
8.071
the husband to the late Mary Jane Hol tz cl aw Scoopwas
8.004
aG Chris Harlow Eric ink be iner Mike Deutsch, FAA
7.912
ding Music Producer and Artist LA EL., Jeff Schneewe is
7.688
team is only hitting 242 as ateam. Bob Stein
bock joined
7.678
Redd, TinaDen ise L oll is Timothy Ray Hol
7.668
(k) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 4287. Paired with the output SAE latent with index 34455. Jacobian element non-zero for 1820 tokens. and mean 1.131 x 10-1 (rank output SAE latent) and standard deviation of 2.500 x 10 over its non-zero values.
Example tokens
Max. activation
Mah avir Mer Dr hti Me One afternoon hewas resting
1. 1.030 10º
haben. Un rer Me ung nach wird es in dennA
9.290
waukee Magazine prof iled Le in bach the UEC last year. An
8.524
about Milwaukee s Urban Ecology Center \n KenLe bach, DOKOPY AR BHAREAL. me in bol new
7.583
7.505
and song writing team ofP ia Le onenandJon ial on April 27 2015.01... By. Dom Ein horn Cosm
7.337
6.740
growthtth sustainability Luc LuucasAnds And hel by. Smith.
6.319
top casino sites worldwide Das V ere
ig te K ig reich
6.226
zondheid /voinigg----ggooo met- we in ig- cal orie
6.008
inated A. C. Ke in ies as their candidate Anment
5.644
my heart live AnD ilon me in tumap ni bet abi
5.064
(I) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 62769. Paired with output SAE latent with index 34455, the Jacobian element is non-zero for tokens, and has a mean 1.121 10-1 (rank output SAF latent) and standard deviation of 7.225 x 10 over its non-zero values.
28

---

Jacobian Sparse Autoencoders:
Sparsify Computations, Not Just Activations
Example tokens
Max. activation
op316(1867 ). Vienna PO/ Carl os
Kle iber
\n
1.541 10t
Jape K rystal Kle
Korm ac" Big
1.460 10
.Jordan Kleeman, goes some of the upcoming features coming to the
1.445 x 10
Golden Wings Music Following his early beginnings on el kc Kle ijn
1.431 x 10
apenem resistant Pseudomonas aeruginosa (CRPA ), Klebs bs iella pneumoniae
1.373 10'
drink bring my Klean K ante with wieeeeeeeeeeeeeeeeeeeere
1.336 101
ina G arr ig ues ); Barceloaa (
K
pac
1.293 10
jin (w K le pac ): 2010- (:00--Budppet(W Med
1.263 x 10
AnDoopoison No. 2: In sect icides AnFlle
5.116
there and one was boarding yet.
We parked the car, sch
4.971
mark the perm al ink AnF le oreven we
4.956
left Gn le be RdSouth to Route Turn right onto
4.754
(m) The top examples that produce the maximum latent activations for the input SAE latent with index 14871. Paired with the output SAE latent with index 34455, Jacobian element is non-zero for tokens, and has a mean 1.095 10 (rank output SAE latent) and standard deviation of 9.772 x 10 over its non-zero values.
Example tokens
Max. activation
Rodiiggdr Com cast as Chair- Elect Emma
ister
1.913 10
another clever design from
ister but it really hard to set
1.897 10
cast Administration from Boston University AnEmma Pf ister (Tre asure
1.854 10'
ips / my college application essay of jobs at Pf izer since
1.817 x 10"
izer umed Limited, among others An Profile
1.781 10a
aann
O. Thomas eds
1.711 10
case in Pf le ide IV G Immob il ien The
1.685 10
am ore Ronald Bra ut ig am pf Leovan Does
7.684
Pen eenee Th wait es
AnMoz art W
6.898
). Yu ja Wang pf An in
6.714
ino pf. An ig is
6.594
Pre vin pf. An occe ini L. String
6.460
(n) The top examples that produce the maximum latent activations input SAE latent with index 32693, Paired with output SAE latent with index 34455, Jacobian element is non-zero for tokens, and has a mean 1.092 10-1 (rank output SAF latent) and standard deviation of 9.177 x 10 over its non-zero values.
29

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
A. A. planned, coordinated and sch moozed for four months
2.058 10
there aod one was boarding yet We parked the sch lc
2.045 x 10
intervention Fort us poor sch mucks it 's just something
2.004 x 10a
to- speech Now print out acopy of this page, sch
1.945 x 10
inky melting stage where you can just sch m ear on crust
1.934 10'
Paul VI for he resy y sch ism and scandal ten years
1.926 10
slate
sch ist quartz ite and limestone on the west; met
1.909 x 10²
he yesy. sch and scandal, it is in fact more grave
1.893 x 10
needed to sch irm ish ona3 area front USSR should
1.881 x 10
what the French call changes to the sshool astic rhythms.
1.877 x 10'
sch tick was new and the momentwas right and that we
1.876 10
ac ris ro bin sch u will i am,
1.849 x 10a
(o) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 30568. Paired with the output SAE latent with index 34455. Jacobian element non-zero for tokens, and has mean of 1.049 10-1 (rank output SAE latent) and standard deviation of 1.109 x 10 over its non-zero values.
Example tokens
Max. activation
31. \nRaich JW, Sch les inger WH, 1992.
1.313 10
and Care, the Sch les wig- Hol stein Ministry for Social Affairs
1,276 10t
Productions celebrated Justin Sch leg el birthday by presenting him with very special
1.274 x 10
from Politics and Society to Culture and Entertainment Ber Sch les inger
1.262 x 10
irling and mixingandd clear mixing lines visible it (Sch lie ren as
1.247 10
K yle Sch war ber may for the season, but the home
1.231 10
aSab ot age
Amold Sch war zeneg ger is backand looking
1.229 10'
21 ) but Sch les
inger 's pla te number suggests a earlier publication date
1.224 10
the one field target \n Just aboutone field by Sch war
1.222 x 10a
ville Texas, Sch iner University asmall four year private college
1.218 10
spoke at an event DO 10 Sch if ff ah
1.211 10a
Mike Sch re iber and countless more. She volunteers at Dade Correction
1.203 10
(p) The top examples that produce the maximum latent activations
input SAE latent with index 47756. Paired with output
SAE latent with index 34455, the Jacobian element non-zero for 451 tokens, and has mean of 1.040 10 (rank output SAE latent) and standard deviation of 1.167 x 10 over its non-zero values.
30

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
a- K34 al- alJi
alJI a121 alJI c12 al
1.484 10a
ei ciah! Loaa cle--
1.478 x 10
L aoa34Ial
e34 34.
1.422 x 10
a Nela al al eGl
to be, ei c-ckaa
1.419 x 10a
aHa
a IeL cl
a III
suspect
1.385 x 10
content cK- a121
alJi
cocLa12k
1.385 10a
1-c-e- a12 a
iG (aaa
Gel el
1.379 x 10
li aoi k)
AnaL
aa
Ig
lgegeL.
1.378 x 10
co
al
112eal e-
e--oo
1.374 x 10
La leao LH i141 2
eg 12
1.374 x 10a
the contents page a12 kc
1.368 10
alJi
aii
a121 Nal aHH aH
1.362 x 10
(q) The top examples that produce the maximum latent activations for input SAE latent with index 59459, Paired with the output SAE latent with index 34455, the Jacobian element is non-zero zero for 1602 tokens, and has a mean of 1.036 x 10 (rank 15 for the output SAE latent) and standard deviation of 1.771 10-2 over its non-zero values.
Figure The top examples that produce the maximum latent activations for the output SAE with index 34455, and the input SAE latents with which the mean values of the corresponding Jacobian elements greatest. The Jacobian SAE pair was trained on layer 15 of Pythia-410m with an expansion factor the English subset of the C4 text dataset with context length of 16 tokens.
64 and sparsity 32. The examples were collected over the first 10K records
31

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
kevAl retro f Ard fArden kA ns lig e.
Barn
4.636
lig ut ikt Aver Lip An Mys ig och barn
4.549
ochfArfAr st Ark upp till 20 tim mar ut
4.370
kar, mendu mAste vara in log ad som an st Al
4.337
Det fin nas er pro jek dAr Lars Bank vall med ver
4.165
Cancel E
ex expppe el pA fore x hand el att
4.149
Spes ial sp ill val g av omme
4.121
atal ogenunder Downloads menyn dArduenk elt bl
4.080
A iill A
Cong rat z Eva Ny strAm and Ad riel Young
4.037
kA pa Euro s ja US Dollar In Results per trade binary
3.989
An lig mos Ar och sk narum, dock kans
3.985
i ssmak ak AA det som ble ig jen An arn ed the
3.977
(a) The top 12 examples that produce the maximum latent activations the output SAE latent with index 64386.
Example tokens
Max. activation
Spes ial sp ill ut val av omm
1.218 x 10a
ass el og and en dans ke kon ge af den Gl 2522 cks i sm PA det som ble ig jen \n arn ed the
1.175 x 10
1.137 x 10
var kon ge af Dan mark 1906 til 1912 Han var det 230
1.136 x 10
lig ikt Aver Lipnos An
Mys ig och barn
1.018 10a
kevAl retro AArgade fArdenkA lig c. Barn
1.007 x 10
Detkan fin nas fl er jro dAr Lars Bank vall med ver
9.832
vAn lig at mos Ar sk A narum, dock kans
9.703
Detg der de este arer herunder
9.656
LA ks ind en 12 42 10
9.561
PA Chal mers fAr att kun na se Sometimes I like to add
9.513
jeg ings stock ings well anything that covers my legs is
9.509
(b) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 23581. Paired with the output SAE latent with index 64386, the Jacobian element non zero for 2755 tokens, and mean 2.578 x 10-1 (rank output SAE latent) and standard deviation of 2.428 10 over its non-zero values.
32

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
\n400 Pet ans Bj erg 2940 1200 2940
7.399
from Sig urd think we werere lied to Don tyouhave
7.343
d was invented by veter in arian Sig urd Ke il gaard in the
6.987
us Met ers en the film stars ver rir Gudn
6.918
beganworking as an artist and teacher An In ibg Arg
6.917
ALEV intage Metal Ch airs From B ali By Bj heim With
6.837
house, making Also In grid N ils en Bio D ating
6.760
film shot in Trin idad in 2013, work by Ay i
6.647
og oms dal
in Norway Anv84 ang st ind en
6.632
ell world aroundme. Cong ratulations Tor stein Horg mo, the latest
An intended the first piece basically bean introduction
6.463
6.459
modern technology AnAs Tar
N issen
M eyer writes
6.413
(c) The top examples that produce the maximum latent activations for the input SAE latent with index 48028. Paired with the output SAE latent with index 64386. the Jacobian element non-zero for 3567 tokens. and mean 2.417 x 10-1 (rank output SAE latent) and standard deviation of 1.944 x 10 over its non-zero values.
Example tokens
Max. activation
Netherlands. Italy Canada
Denmark Norway and Turkey and
9.613
BU, Germany Norway USand UK) ). PROMMNN has
9.009
aldskaand the Ale ut ian Islands Green land
Britain Norway
8.417
), Russia, Norway and Iceland which promotes dialogue, practical cooperation anddevelopment.
8.119
Sah ara human rights bodies from Norway and elsewhere \nThe
8.066
the matern ity leave in Norway (46 weeks Denmark (52weeks)
8.006
W oc schenende oderan an Fe iert agen. THE Norwegian Fisher Council
7.970
lands and islands and through the Norwegian f ord will continue to operate
7.870
single byrketby following Norway model by joining the European EconomicAAra.
7.853
can either subs ist without Norway or simplydead by stacking Ukr
7.850
is member the Norwegian Visual Artists Association and the Young Artists Soociety
7.838
Children Norway beasbeenwwarde the service contract for the Support the Education Sector
7.760
(d) The top examples that produce the maximum latent activations
input SAE latent with index 11698, Paired with output SAE latent index 64386, Jacobian element is non-zero 295 tokens, has mean 1.344 x 10-1 (rank 2 output SAF latent) and standard deviation of 6.924 x 10 over its non-zero values.
33

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
kar, men mAtt in log gad som an ld
1.023 10
kA pa AApaaEuro ouro s Al ja US Dollar
An Results per trade binary
9.895
add ning och lar att kopp las till slut st eg eller
9.721
kevAl retro f Ar g ade fAr den den ns lig c. Barn
9.363
PA mers fAr kun dem Sometimes! like to add
9.343
An lig mos or sk narum, dock kans
8.634
ochfArIAr st Ark are upp till 20 tim mar ut an
8.587
C Cancel E tt ex emp el pA fore x hand el Ar att
8.477
rar mell an oli ka prodduuk ter Dev ices systemsn
7.977
A till A Cong rat z Eva Ny AnrAmandd.... Ad riel Young
7.812
Det kan fin nas er pro dAr Lars Bank vall med ver
7.368
pool och le k pl ats Tre lig personal
An- ub
7.195
(e) The top examples that produce the maximum latent activations for the input SAE latent with index 22804. Paired with the output SAE latent with index 64386, the Jacobian element is non-zero 895 tokens, and has mean 1.342 10-1 (rank 3 output SAE latent) and standard deviation of 9.149 10-3 over its non-zero values.
Example tokens
Max. activation
s fixed it myself thats the first place you will get rust
5.811
Models for Rust ing T ail gate Str uts AnTak ata recalls at
5.395
because gave in easily Rust ic guys country awoman who is very to
5.063
bring to full resolution .Graphics are from a Rust re ath collection
4.752
Rust ic Pic nic Table Popular Outdoor Furn iture Hand made By App we present ust ic Pic nic Table Amazing All uring Ideas Tables
3.102
2.680
seeing Thve it in rust pots and here it is in an yields a Future gen cor out ine schedules the generator to be resumed
2.359
2.104
operations in order to ensure that the resulting LLVM IR canbe ing ested
1.849
with gen cor out ine \n In Python2, the sub from () The
1.805
has no meaning besides being
1.739
you will have to upload the
pub file andcopy its contents authorized
1.702
(f) The top examples that produce the maximum latent activations for the input SAE latent with index 12754. Paired with output SAE latent with index 64386, the Jacobian element non-zero for tokens, and has mean 1.168 10 (rank latent) and a standard deviation of 5.760 x 10 over its non-zero values.
output SAE
34

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
Turkey the League B victory An Sweden
making its Nations League
1.362 10
68 38 m 12 18 mHHPn on the Norwegian- Swed ish
1.330 10
for Vat ian independence, R iga basb ruled by Germans, Swed
1.286 x 10
bar. \nsummerwhen the Swed began to shine AnCup
1.180 x 10
en Un ser Swede Savage Booby Un ser Gordon
1.124 10'
25 ppm in Denmark, France, Spain, Snd An"
1.107 10*
An This study was supported the Swedish Research Council Grant K 20155
1.090 102
Canada, Finland, Italy Swedenand the UK In the next few
1.088 x 10
pa in English, French, Spanish, Swedish and Mand arin You
1.076 x 10
the Middle East and Southeast Asia. In reality the Swedishand this threat is from different countries such as Sweden, Malaysia, India, iron ore mine in the world. It ownedby the Swedish government
1.070 10'
1.040 10
1.020 10a
(g) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 30912. Paired with the output SAE latent with index 64386, Jacobian element is non-zero tokens, and has mean 1.133 x 10 (rank 5 output SAE latent) and standard deviation of 6.893
10 over its non-zero values.
Example tokens
Max. activation
ga 2012 CIA se A3
8.579
la ri Paul Bergag Walter Gilbert
An Dh ain mic
8.236
A ir agus us Spar An abbabbi fu lu achaH
8.205
a AA ar an su AN omh. TA
7.989
Ansan AN A3 don i om ath A3 ir
7.893
ire adh Chun Ce oil agus AN na G Is m And (" miidd, reason "). Related to Old English my nt
7.412
7.394
2013 nau irl is ANg g iol calgh mi ot ail
7.251
AnImporto k rom programo estas fac ila AnA
7.154
\nA lu event o lig A as al orig ina
7.080
St air na Ai ire ann will take on
7.046
b ild oj de event oj aj ild oj de ateg orio
7.012
(h) The top examples produce the maximum latent activations
input SAE latent with index 57769, Paired with output SAE latent index 64386, Jacobian element is non-zero for 321 tokens, has mean 8.286 10-2 (rank output SAF latent) and standard deviation of 1.343 10 over its non-zero values.
35

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens iGe likei,iill
Max. activation
2.700
iH iLD illii il
2.051
do that P\nl : :00 338KEVIN
you
1.881
C-e3 eielK ii iLliil
1.822
i:L-ik14e el iL ikl
1.763
Li eeli elll
1.637
II al aol !! Inalaal-c2e--
cli al
1.592
WJ ille
1.564
Bin is one pictures that are related with the picture before in the collection
1.488
iL.ei it. ikl e3eil.14 ikl
1.467
sealed bearing Now I feel kinda like a**** for asking in
1.458
eLik ik ii likillihh
1.416
(i) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 42113. Paired with the output SAE latent with index 64386, the Jacobian element non-zero for tokens, and has mean 7.971 10-2 (rank output SAE latent) and a standard deviation of 6.880 x 10 over its non-zero values.
Example tokens
Max. activation
and arthritis \nY og ar aj Gugg ulu Ing redients include
1.332 10
Europe Rest Rest World Ing redients of this market are Sol vents
1.315 10a
\n04. ng TOS so. io hn& Salv atore-1 Fl
1.298 10
an audio original en ing les El cost de adm isiA3n
1.298 x 10
000 calorie diet * Daily Value not established AnOthe Ing
1.245 10
Smooth ie Six Ingred ient Sp ir ul ina Snack ies
1.201 10a
100 Ing redients InA Pictures For Kitche Number Is Im possible !.
1.173 10'
Ind io Papago u ite Cream 20z Ing redients
1.153 10
Ma ing alls hadno cabinets or refrigerator andshe raised five children and
1.137 x 10a
inderstood Ing red ient With Ree ipes, Simply Rec ipes explanation
1.132 10
ropur ur Ing redients is global supplier of ingredients and services developed create
1.065 10
Bah asa Ing gr is Sing kat is just about the imagewe ascertained on
1.045 10
(j) The top examples produce the maximum latent activations for the input SAE latent with index 8827. Paired with output SAE latent with index 64386, Jacobian element is non-zero 21 tokens, and has a mean 7.330 10 1 (rank the output SAE latent) and a standard deviation of 5.439 x 10 over its non-zero values.
36

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens to scholars in Oriental Studies. Med ieval Literature and History The Gh ia world Be part of amedivvll revolution against the Word King and his trying to makea living in a amddieval town full of warriors sor ce
Max. activation
9.650
8.678
8.207
woo play medieval and Renaissance instruments. ssing. danceand celebrate at
8.198
Sem ikh was the rise of the medieval university wwicch began to issue
8.038
located in Med leval Spon Street Cov entry Andrew offers
8.015
armies of the iph ates through the action blooo medieval battles
7.999
Ngaw ang Namgy al,
father and un ifier of medieval Bh ut
7.911
nnedee to know what are they ate medieval northern England ? Lots
7.847
th Annual Award Win lough rea Med ieval Festival takes place from the
7.761
like the individual parts in cycle" of Med ieval
mystery plays
7.681
id yll ic picture for the season. Tnehe medieval charm of Vil
7.672
(k) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 21697. Paired with the output SAE latent with index 64386, Jacobian element is non-zero tokens, and has a mean 6.570 10 (rank output SAE latent) and standard deviation of 6.127 x 10-3 over its non-zero values.
37

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
football of his career, Russell Wilson finished Monday night 's over the Minnesota
1.082 104
.T. PAUL, Minn (AP) The Minnesota Department
1.068 x 10
the Minnesota Birth Center: ; and D ian na ol les
faculty at
1.060 x 10
the Minnesota Historical Society Begin ning with the 1929 volume there is
1.060 x 10
An'The Winnipeg Row ing Club, the Minnesota Boat Cluband the Saint
1.055 x 10'
Miami Marl ins Milwaukee Brew Minnesota' Tw ins Kentucky Wild cats Win
1.050 x 104
H. Allen. The lock out of the musicians at the Minnesota Orchestra has
1.042 x 10
Post started spec ulating this past week that the Minnesota str al Association might
1.032 x 10
The Minnesota have recalled defense man Ryan Murphy from HL lowapera
1.032 x 10
An index to the Minnesota period ical collection in the South Paul Library
1.010 x 10'
Marl ins Milwaukee Brew Minnesota Add the United States of Baseball to your
1.003 x 10
Paul line Row ing Club as the Minnesota and and Winnipeg Row ing Association
1.002 x 10a
(I) The top 12 examples that produce the maximum latent activations for the input SAF latent with index 13110. Paired with the output SAE latent with index 64386, Jacobian element is non-zero tokens, and has a mean 6,486 10 (rank for output SAE latent) and standard deviation of 6.027
10 over its non-zero values.
Example tokens
Max. activation
in Official News TawsTags::
utter programminggLavaacommmei
6.019
?An you re not familiar with the Fl utter programming languageand
5.044
Your Business with F BA? In Why You Should Master Fl utter Programming
4.472
mater the sought- after Google Fl
utter language Let begin now
3.056
Why You Should Master Fl utter Programming rub ber plants indoor rubber plant growing
2.706
ew Mah will be celebrated from8 th April 10 th
1.795
compatible with the industry An Widget your website Orderthem.
1.511
this place. Eleanor notices this upon first arriving thinking to herself that
1.489
illness and health out Swift Skin and W ound is
1.488
some patients feel st inging sensation from the injection & think isn You also the Contact Information widget from the Form W idget
1.473
1.358
Col oring Pages Image houzz. com. Free Online
1.283
(m) The top examples that produce the maximum latent activations the input SAE latent with index 26452. Paired output SAE latent with index 64386, Jacobian element is non-zero tokens, has mean 6.316 10 (rank 12 the output SAE latent) and a standard deviation of 6. 6.164 I x 10 over its non-zero values.
38

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
been made in extending eeveraide to American Indians and Alaska N atives the
1.224 10a
percent of American Indian and Alaska Native children were enrolled in Medicaid or CHIP
1.224 10²
uninsured rate for American Indian and Alaska Native children and families remain un accept ably
1.153 x 10
issues in the Park Service Former Alaska governor and advisory board chairman Tony
1.147 10
andmindset for the Alaska Con ce aled Hand gun Perm it Taught
1.106 10'
popular appet izers and AnLoc ally caught seafood
1.082 10
is Canada Alaska and Hawaii Your mattress will be refund ed, however
1.081 10²
for shipping quotes to other
destinations such as Alaska and Hawaii An This product
1.079 x 10²
Moose and Wolf hunt. An We hunt the Alaska Yuk on moose
1.066 10
row se listings Member users here at Alaska irt that are associated with
1.064 10
Cherry, BLM Alaska regional manager, said the agency was pleased
1.057 10
The paint appeared to still . The package includes Alaska Yuk on
1.045 10a
(n) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 32153. Paired with the output
SAE latent with index 64386. the Jacobian element non-zero for tokens, and has mean of 6.202 10 (rank output
SAE latent) and standard deviation of 5.347 x 10 over its non-zero values.
Example tokens
Max. activation
Han over, Wake, and Forsy th havemmore than half of their
1.788 10
he told SMSFAd vis AnMr Forsy th explained this
1.643 10a
arr ion Thanks to Anthony Fors of Absolute Clean for volunte cring
1.616 x 10
read more into AT OIDs than they should, Mr Forsy th
1.531 x 10
Soper Consult ing director Stuart Forsy th said shortly after the release of
1.517 10
webhomes fors ale com you find the Jacksonville Fo uit your id ols fors yke fond fond ings and the promised
1.051 10a
7.176
you, nor fors akeyou "oaun""HHer (Heb... 13: :):;
6.366
for comfort in my difficult situation not fors ake me, Good Jesus
6.248
ume already led many to fors ake the temple, and her ordin ances which needs to be confessed and fors aken \n2. We profit from be submitted b mail ( anned to fors ik ring @ lease
5.815
5.704
5.697
(0) The top examples that produce the maximum latent activations input SAE latent index 56394, Paired with output SAE latent with index 64386, Jacobian element is non-zero tokens, has mean of 5.939 10 (rank 14 the output SAE latent) and a standard deviation of 3.459 x 10 over its non-zero values.
39

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
Norwegian ords around ord Norway, An Read more our
1.093 10²
250m crossing undera ord between Krist ians undand
1.089 10'
coast in Turkey, eru ising the F ords of Norway, etc.
1.033 x 10
zonesand sc up to the top of the f ord. There
1.001 x 10a
60m 16 10 m ogn og ord ane Norway
9.988
lands and islands and through the Norwegian lords and will continue to operate
9.885
You will also experience a ord cruise on the mighty Sogne
9.601
f 80 stud is just slightly larger so all I had to do
9.451
steering boyanyways so I decided to convert to f 80 tie
9.137
$7.57kkg Sagne orden, N in
9.027
\n426Alks ord A- 4 m113
8.791
k vNoah: November Ethical dile mmas occurwhen situation
8.114
(p) The top examples that produce the maximum latent activations for input SAE latent with index 36481. Paired with the output SAE latent with index 64386, the Jacobian element is non-zero for tokens, and has a mean of 5.820 x 10 (rank for the output SAE latent) and standard deviation of 4.891 10-3 over its non-zero values.
Figure The top examples that produce the maximum latent activations for output SAE with index 64386, and the input SAE latents with which the mean values of the corresponding Jacobian elements greatest. The Jacobian SAE pair was trained on layer 15 of Pythia-410m with an expansion factor
64 and sparsity 32. The examples were collected over the first 10K records the English subset of the C4 text dataset with context length of 16 tokens.
40

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
surge into the cathode, producing current that does the all important work
6.720
so that he avoids distribution of designated product that apparently does not meet legal requirements
6.606
but also expansive alien environments that do their part to make the audience feel 2 (8 percent ) were opportun istic attacks that did not target a to make therapeutic claims, found to do doso. An Unfortumately the proposed
6.584
6.467
6.406
Children the most severely affected by poverty because they do not have the hospital admissions emergency visits that do not result in admission are excluded.
6.331
6.282
court order Proopos als relating to children often do not need to
6. 6.264
S uitable exercises for pregnant women are those that not strain the lower
6.255
?nn Unfortunately the traditional stinescaappoachh to training in the modern world does also request fat shaming to be made illegal because it noes haveean
6.240
6.232
select the 1 dB degradation to noise as the interference standard since it does
6.221
(a) The top 12 examples that produce the maximum latent activations for the output SAE latent with index 60542. Output latent 60542 responds a very specific use of word "do": its use pro-verb. In pro-verb a simple verb stands for another more complex one and here "do" is shorthand for an action that can only be understood from the context, for example, in "were found to do so" the "to do so" stands in for "to make therapeutic claims". Some of the inputs include very different uses of "do". one for example deals with the "Done" in "Donegal", auxiliary, modifying another verb, as in "[t]his does meet the requirements" Clearly this circuit is creating very fine distinction between different ways the word "do" can be used, a distinction we make language comprchension, but one we would have trouble identifying or describing.
Irish county. However, another input includes subtly different use of "do": cases where "do" is used as an
Example tokens
Max. activation
Your information will not be stored on HSB'C ssystems you do
2.098 10
Children are the most severely affected by poverty because they do not have the
2.077 10
surge into the cathode, producing current that does the all - important work .001) ) fnderworverr tri aged children who did not require inpatient management but also expansive alien environments that their part make the audience feel us at your own cost within days of delivery you not
2.056 10
2.043 x 10
2.032 10
2.027 10º
them post menopausal. They were compared with more than women who did not despite the risk When persecution came, they did not scatter .They remained
2.014 10
2.001 x 10²
levels protect public health and welfare if they do not exoteee dB. .. An
1.997 x 10a
so that he avoids distribution designated product that apparently does not meet legal requirements
1.996 10a
hospital admissions emergency room visits that do not result in admission are excluded.
1.984 10a
S uitable exercises for pregnant women are those that do noto strain the lower
1.976 10
(b) top 12 examples produce maximum latent activations for input SAE latent index 21465, Paired with output
SAE latent with index 60542, the Jacobian element is non-zero zero for 13107 tokens, and has a mean of 2.529 x 10 (rank for the output SAE latent) and standard deviation of 5.392 x 10-2 over its non-zero values.

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
We expect the market to do the opposite ofwhat the indicators are saying with the 2018 elections looming. Tney believe that Trump has done good
1.438 10
1.313 10!
only thing that wanted in the world, but she did bad thing
1.292 x 10
by its President Jim Cor cor an, have wonderful job
1.292 x 10
so well. And many state laws the same a"t ote: "I just buy the boxes These do can 't win games, you re in trouble An Saint did
1.264 10'
1.261 10
1.258 10²
beex orbit ant nou own research to find out the up for grabs it salwaysJJwssoing the grabbing An Here
1.257 x 10!
1,251 x 10
your muscles do the work going at asn ail 's pace
1.246 10'
worry about whether the Black haw ks are doing the right thing with defense
1.244 10!
You' re doing the same thing on Amazon
spddsoooooooddddnnso proving yourself
1.239 x 10
(c) The top examples that produce the maximum latent activations for the input SAE latent with index 61756. Paired with the output SAE latent with index 60542. the Jacobian element non-zero for 2683 tokens. and mean 1.076 x 10-1 (rank output SAE latent) and standard deviation of 2.253 x 10 over its non-zero values.
Example tokens
Max. activation
A du grand luxe, il fait cequ on lui dit
1.044 x 10
airamAAsinfommaciAnn haga cl ic en el la zoab
9.095
cl ase de ing A por
hacer la present aciA3n AE
8.587
qu aces 'n you want to go back to a previous
6.694
si vous
codes
NGNh12 NI1
5.713
co. donde se real İza la seg unda enay final mente.
5.181
into the software The K art ra interface is faire ly well designed for
4.361
which they draw drinking water. AnLeg isl ators must find the faire st
4.314
endoen An Living with Fran ", van af zondag Dekache Is maken've lawa ai We ver organis ieren tm als ih eigen en Games. Es tut uns le
4.312
4.253
4.225
cenp oner los hue v on el interior del huA
4.202
(d) The top examples that produce maximum latent activations input SAE latent index 51331. Paired with output SAE latent with index 60542, Jacobian element is non-zero 90 tokens, and has a mean 8.908 10 (rank the output SAE latent) and a standard deviation of 9.883 x 10 over its non-zero values.
42

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
the middle war again The Seventh Done gal took part in the all hurting tonight I' 1l share details later this week. Done! fit or there Woomethingwrong please contact us An Done with Multi
1.341 10a
1.216 x 10
1.211 x 10a
this point.. An Done. be interested to learn moreabout this myself
1.192 x 10
the goal of Universal Male- Female Liter acya done deal. \n
1,191 10'
feeling of being let down, overwhemmeddddd done Dis appoint ment is
1.191 10²
there ssomehhing wrong please contact us An Done with Egg- shaped
1.188 x 10*
first gin distilled in Co Done gal. Amm is the Cl ossure and re clamation amining property is acomplex process. they could take \n'The Done gal Guards, like no other arm were able to finish rebuilding efforts and the Done gal found in out his heart to the doneky ky Oneday, Kesh ava
1.179 x 10
Done
1.168 x 10
1.155 x 10
1.150 10
1.146 x 10
(e) The top examples that produce the maximum latent activations for the input SAE latent with index 11694, Paired with the output SAE latent with index 60542, Jacobian element is non-zero tokens, and has mean 7.039 x 10-2 (rank 3 output SAE latent) and standard deviation of 1.967 x 10-2 over its non-zero values.
Example tokens
Max. activation
relax mechanisms. On the other hand, between the classic models, it
1.057 10º
protein zirr prot oporph in on the other hand, compared to Cauc as ians On the other hand, the latter havea all change associated un ification An the first point see J
1.020 10a
9.974
9.717
the effect of class size varies across students An the first form disease An the say of hospitalization B its y's scope for development. On the other hand, there is muchscope for for whom is minimal use On the other hand, when hazards
9.667
9.630
9.605
9.602
regulates its collective temperature at different ambient air temperatures on the left side
9.537
.n the inside there are also lent icular films that diffuse the view associated with these ex libr ises On the one hand one part of states On the other hand, weee define destinations in
9.520
9.519
9.441
(f) The top examples that produce the maximum latent activations for the input SAE latent index 48418. Paired with output SAE latent with index 60542, Jacobian element is non-zero 91 tokens, and has a mean 5.1 10 (rank the output SAE latent) and a standard deviation of 9.043 x 10 over its non-zero values.
43

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens of the date of filing of the petition An A change of name
Max. activation
5.590
the jury. An An interruption the opposing counsel with withaspeech rather than of these statutory changes.. An new newform, called a Dis closure
5.370
5.359
Court will exlm marginally whether the principle fulfilled
detailed
5.215
filing of the petition merger is recorded by means a petition
5.214
hadccmmittedaccinee while on release obstruction of justice AnA
4.802
does not accept liability for failure to deliver within the stated time. AnA
4.802
igated sentencing An experienced Mar ic opa County Attormey will know what 169 of the Revised Statutes \n An te cedent Ref erring t
4.660
4.463
second discrimination trial against the University agrounds for dealing with the issue pharmacy is also nearbywhich sells
trial jjudge's availability limited
4.404
4.393
to complywith with its requirements The court noted that "[ dvance
4.323
(g) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 32517. Paired with the output SAE latent with index 60542, the Jacobian element non-zero for tokens, and has mean 5.015 10-2 (rank latent) and a standard deviation of 0.000 over its non-zero values.
output SAE
Example tokens
Max. activation
photos Import ant This product does not include the techniques used in the
1.195 10º
pumpyou are looking for An This product does not contain
1.182 10a
Furthermore expor ail org and web sites associated does not sell user
1.146 10"
ThThe name'T ee boeg not appear an of the top
1.130 x 10
suggestion for ASC Utt imate. Dou not need to worry about ube. com Limited does endorse any user submission, and expressly disclaim s ube.. com does not permit copyright infring ing activities or infringement of intellectual
1.123 10a
1.119 10²
1.119 10"
family, it doesn for Mark he told Ostrov
1.095 10"
ig ences com website or webpagee. Dil ig ences does not represent or market, does help no great the thought is
1.087 x 10
1.085 10
marketing, Blogging, internet marketing \t. world doesn
1.075 10a
daily basis AnHey, ddon know if I m posting
1.075 10a
(h) The top examples that produce maximum latent activations input SAE latent with index 23968. Paired with output SAE latent index 60542, the Jacobian element non-zero for 2561 tokens, and a mean 4.760 x 10-2 (rank output SAF latent) and standard deviation of 1.204 x 10 over its non-zero values.
44

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
youcanloweryour standard of living And a lower standard of living does cours not .A post hum diagnosis does change
9.428
9.376
holds equity a private business or has a large portfolio this does not that this a negative review, the above observations constitute qu bles Tax ote re and permanent hair loss those warnings do not appear on
9.172
9.046
8.894
hoveno responsibility control overthem. The existence of these links does not
8.834
and ambitious AnM ission success does not require these complex maneu vers,
8.667
Please note these restrictions do Jonot apply to exhib itors To help
8.648
Yellow stone if there are only that many sub species That does not include
8.556
the Tribunal explained, TTraining alone does not meet the requirements of due
8.529
AnSome schools prefer to be data controllers This doed make it easier for extreme heat It naturally darker colour doesn" t do itself any
8.483
8.473
(i) The top 12 examples that produce the maximum latent activations for the input SAF latent with index 19973. Paired with the output SAE latent index 60542. Jacobian element non-zero for 2691 tokens, and mean 4.681 x 10-2 (rank output SAE latent) and standard deviation of 9.146 x 10 over its non-zero values.
Example tokens
Max. activation
play revenve Well, that no longer appears to the case
8.329
thanhe believes or cares to say Poll now demonstrate this to bo
8.314
systems are often thought to bemuchmore securely protected than is actually the case
7.874
relations with residents Unfortunately .with some boards this not always the case
7.272
some cases the spam detection stopped working This is no longer the case
7.104
't)and if there are no clear winners netweenthem: which is usually
7.036
found the same to be true for them !). Courro Fried Ice Cream
6.854
AnHowever, even if that is not correct the Court erroneously uses
6.826
final However, that not always the case There are plenty
6.787
if that were bad bad enough for Thor pe membership. in the same the lease period has expired and oil has been produced. If this missing from the root of the domain added If this is the
6.773
6.688
6.680
()) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 13058, Paired with output SAE latent index 60542, Jacobian element is non-zero for 431 tokens, has mean 4.528 x 10-2 (rank 8 output SAE latent) and standard deviation of 1.961 x 10 over its non-zero values.
45

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
Compan ies must give workers the information they need to perform their jobs safely too short to drink bad coffee that makes perform poorly and feel awful.
1.448 10
1.448 x 10
your reserve fund. By performing a reserve study, you can determine how g maii account, perform my edit and then un install a matter worn fill at the tail pipe as permitted to perform every engine and every rate car just got from him and he performed lazy spin into the
1.426 x 10
1.410 x 10
1.409 10
1.400 10
from, modify, publish, edit translate distribute -perform display
1.400 x 10
The firm did not perform impurity testing during stability ontwo products since 2016. stop thinking about mess ing up. up. You forforrrrmeds so tent atively that hoped
1.397 x 10
1.389 x 10
like how black hat hackers are incent iv ized by money to perform malicious activities
1.388 x 10
payments encourage P sBMs actually perform to their contractual guarantees instead of
1.367 10
individual educational plans assists in inventory control and performs other duties related to
1.364 x 10
(k) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 56700. Paired with the output SAE latent with index 60542, Jacobian element is non-zero 574 tokens, and has mean 4.375 x 10 (rank 9 output SAE latent) and standard deviation of 1.275 x 10 over its non-zero values.
Example tokens
Max. activation
share the posts if your life dependedon it Which doesn
8.326
encounters an impossible reality that shouldn exist but does
AnA
8.273
ask ! Who knew las agnawas for dessert ? We sure didn"
7. 129
to make sure that works correctly and it does. An What
6.960
status and aside Not Notevery WordPress theme supports post formats and those that
6.757
you ever thanked me you actually did but I can count
6.578
if this would kick in but it didn An After installing
6.342
don need convincing. you know who does ? AnAn big one. It seems to work wrrk.Ordoes ?Then three fish
6.318
6.302
Ordoes ?Then three fish get the Spin You dont dont switch but your opponent does so sou can stack damageon
seems to work.
6.277
6.275
Somehow: envisioned our family a family of four I still do
6.228
(I) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 46097, Paired with output SAE latent with index 60542, the Jacobian element is non-zero for 3589 tokens, and a mean 3.995 x 10-2 (rank 10 the output SAE latent) and standard deviation of 2. 2.127 x 10 over its non-zero values.
46

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens its spark Employees inspired stakeholders are becoming color less andyou
Max. activation
5.325
are talking about are humans, and not robots and soa greater amount
4.681
nice not too fast not too slow "rhythman and settled in to endupbeing abottcommuiiicitiio not between professions but with clients and families
4.430
4.387
pre-s lic ed is easiest not too thin and nothing wet
4.313
did not know how the French health care worked and did not
4.230
move frequently not beon school school
district census rolls and are less likely
4.183
Total PM A way of life
.Notajob \n It was
4.141
the fire which does notburn the whtcr does not wet the hands do move do not breath close my eyes
4.113
4.075
It not for ameeting international
leader AnAnn it was
4.069
not for and your family and not such thing as safe plastic not
4.018
(m) The top examples that produce the maximum latent activations for the input SAE latent with index 4510. Paired with the output SAE latent with index 60542. the Jacobian element non-zero for tokens, and has mean of 3.978 10 (rank output
SAE latent) and standard deviation of 1.006 x 10 over its non-zero values.
Example tokens
Max. activation
witnessing before moving on to more difficult resistant areas Rather the opposite was
1.143 10
decision making authority Actually the opposite is implied Letme put it
1.059 x 10a
should come before focusing onone' sson,, and feel the reverse
1.048 x 10
t faith It s actually the opposite of hope. Our other options
1.017 x 10
tourist rich locale but rather the opposite is true good guy if anything the opposite was said when saying Zimmerman out of fear mode, you return to love mode The opposite of love
1.004 10a
9.978
9.799
in any way- just the opposite The fine balance between supp len ess process as perform argues the opposite
9.498
that learn on method must take heavily
9.484
says the right words to us, but acts just the opposite
He
9.464
saba thing in this case means the opposite
9.423
EGR- val ve Without voltage applied. it 's just the opposite
9.166
input SAE latent with index 28695, Paired with output SAE latent with index 60542, Jacobian element is non-zero for tokens, and has a mean 3.952 10-2 (rank output SAF latent) and standard deviation of 7.511 x 10 over its non-zero values.
(n) The top examples that produce the maximum latent activations
47

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
AnI ve got admit, heard some" dooz
8.284
silent film Google ever changing googledo od le "logo our annual celebrations can be. Dopp el lic keden Number SI
7.809
7.234
3. AnHere another do oz ie AothAv ig dor
7.154
doingafew od les of people suits with their arms crossed
6.835
the institutions tend first information in using the has hers print able do
6.828
ee aret This was long before the Doob ie Brothers ever thought
6.275
keepingwere part of our life and wanted to know about Doob an object ddoemmeberr is Doobee ee aret was
6.262
6.177
adoptme Nameon birth e irt ificate was baby do j john
6.137
found that dood lers performed29% better than non- dood lers
5.989
adoy ish Fedand institutional
instability in the US " said Kyle
5.951
(o) The top 12 examples that produce the maximum latent activations for the input SAE latent with index 26469. Paired with the output SAE latent with index 60542, Jacobian element is non-zero for tokens, and has a mean 3.831 10 (rank output SAE latent) and standard deviation of 1.040 x 10-2 over its non-zero values.
Example tokens
Max. activation
Span ien aufen, kAnnen Sie eine span ische Auf enthal
8.197
dabe auch auf Sport et ten die man auch acht eine Abzocke da herw A de ich r rA1 A4ck k auf en will kann ich tro tz dem das Gold isa be
8.086
8.018
7.818
wer de ich fter im Ar 14ck GIA 14ck Online Casino Turn
7.550
and und mehr \nFrage Kann ich ein span ches
7.512
sind An Frage Kann ich eine Finanz ierung hal
7.413
dass es ges etz. lich an erk ann te An An herreiiigee
7.390
K ann ich nur we ite remp fe h len An This
7.366
we isen dass Sie dire kt oder ind ire kt In hab der
7.351
isen dass Sie nicht mehr als 11 Mon ate hal
7.351
age: W enn ich die Imm il ie durch eine Ges ells chaft
7.225
(p) The top examples that produce the maximum latent activations input SAE latent with index 14813. Paired with output SAE latent with index 60542, Jacobian element is non-zero tokens, has mean of 3.811 x 10-2 (rank 14 the output SAE latent) and a standard deviation of 8.688 x 10 over its non-zero values.
48

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Example tokens
Max. activation
\n4 Does the Pale Diet E lim inate Healthy Foods Like Brown
3.101
for all for profit show tickets required by Louisiana law An Does
3.047
0334. An Doesmy infant / child needa ticket for the show
2.843
000. An CODE its dwn diligence ? AnnYes should
2.575
duties \n Does this mean we are moving to a ahere the Pope
1.695
Ronald Bra ig am ff: Leovan Does
1.543
Doesn' clog pores nor does cause any acne break out
1.511
DDoeshehaver lot of toys ? Yeah, can we bring
1.511
Does Ben Bishop fix their go alt ending Where do Alexander Rad ul
1.479
Does this mean that the leaders who engage in the methods are bad people
1.479
Does yours as pretty as msnee" 'nI' m ggoing to
1.479
day. \n your marketing team regularly grapp le problems' If so
1.306
(q) The top examples that produce the maximum latent activations for input SAE latent with index 41425, Paired with the output SAE latent with index 60542, the Jacobian element is non-zero for 17 tokens, and has a mean of 3.734 x 10-2 (rank for the output SAE latent) and standard deviation of 8. 8.725 10 over its non-zero values.
Figure The top examples that produce the maximum latent activations for output SAE with index 60542, and the input SAE latents with which the mean values of the corresponding Jacobian elements greatest. The Jacobian SAE pair was trained on layer 15 of Pythia-410m with an expansion factor the English subset of the C4 text dataset with context length of 16 tokens.
64 and sparsity 32. The examples were collected over the first 10K records
49

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
D. Training
Our training implementation is based on the open-source SAELens library (Bloom et al., 2024). We train each pair of SAEs on 300 million tokens from the Pile (Gao al., 2020), excluding the copyrighted Books3 dataset, for a single epoch. Except where noted, we use a batch size of 4096 sequences, each with context size of 2048. At a given time, we maintain 32 such batches of activation vectors in buffer that is shuffled before training, which reduces variance in the training signal.
We use the Adam optimizer (Kingma & Ba, 2017) with the default beta parameters and constant learning-rai schedule with 1% warm-up steps, 20% decay steps, and a maximum value of 5 x 10 Additionally, we use 5% warm-up steps for the coefficient of the Jacobian term in the training loss. We initialize the decoder weight matrix to the transpose of the encoder, and we scale the decoder weight vectors to unit norm at initialization and after each training step (Gao et al., 2024).
Except where noted, we choose an expansion factor R- 32, keep the k 32 largest latents in the TopK activation function of each of the input and output SAEs, and choose coefficient of x = 1 for the Jacobian term in the training loss.
D.1. Training signal stability
We initially considered the following setup:
Sx - (x)), 8 8= dd((ss),
y = [()), sy ey=)((r),
8 - ddy,s)
(32)
The problem with this arrangement is that the second SAE depends on an output from the first SAE. Since both SAEs are trained simultaneously, we found that this compromised training signal stability - whenever the first SAE changed, the training distribution of the second SAE changed with it. Additionally, at the start of training, when the first SAE was not yet capable of outputting anything meaningful, the second SAE had no meaningful training data at all, which not only made it impossible for the second SAE learn but also made the first SAE less stable via the Jacobian sparsity loss term.
To address this problem, we instead used this setup:
Sx xx)) 8=
dx)), y fx), Sy eyxxxx 8 = dy(ss)
(33)
Importantly, we pass the actual pre-MLP activations x rather than the reconstructed activations & into the MLP f. In addition to improving training stability, we believe this setup to be more faithful to the underlying model because both SAEs are trained on the unmodified activations that pass through the MLP.
E. Evaluation
We evaluated each of the input and output SAEs during training on ten batches of eight sequences, where each sequence has a context size of 2048, i.c., approximately 160K tokens. We computed the sparsity of the Jacobian, measured by the mean number of absolute values above 0.01 for single token, separately after training. In this case, we collected statistics over 10 million tokens from the validation subset of the C4 text dataset.
For reconstruction quality, we report the mean cosine similarity between input activation vectors and their autoencoder reconstructions, the explained variance (MSE reconstruction error divided by the variance of the input activation vectors), and the MSE reconstruction eиror.
For model performance preservation, we report the cross-entropy loss score, which is the increase in the cross-entropy loss when the input activations are replaced by their autoencoder reconstruction divided by the increase the loss when the input activations are ablated (set to zero).
For sparsity, report the number of 'dead' latents that have not been activated (i.e., appeared in the k largest latents of the TopK activation function) within the preceding 10 million tokens during training and the number of latents that have activated fewer than once per 1 million tokens during training on average.
Given an expansion factor of 64, = 32, and a Jacobian loss coefficient of 1, i.e., fixed hyperparameters, we find that the reconstruction error and cross-entropy loss score are consistently better for the input SAE than the output SAE, Additionally, we find that the performance is generally poorer for the intermediate layers than early and later layers.
50

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Pythia-700 70m
Pythia-1600 160m
Pythia-410m
Cosine Similarity
0.9
0.8
0.7
0.6
Explained Variance
0.8
0.6
0.4
-10-2
Mean Squared Eroo
0.8
0.6
0.4
0.2
2
3
4
5
0
2
10
12
16
20
Layer
Layer
Layer
Input SAE
Output SAE
Figure 21. Reconstruction quality metrics for Jacobian SAEs trained on the feed-forward networks at every layer (residual block) of Pythia transformers. The cosine similarity is taken between the input and reconstructed activation vectors, and the explained variance the MSE reconstruction error divided by the variance of the input activations. For each SAF, the expansion factor is R = 64 and k = 32; the Jacobian loss coefficient
Pythia-70m
Pythia- Pythia-160m 160m
Pythia-410m
Cross-Eias Eniropy Loss Score
0.8
0.6
0.4
0.2
2
3
5
0
2
10
12
16
20
Layer
Layer
Layer
Input SAE
Output SAE
Figure Model performance preservation metrics Jacobian SAEs trained on the feed-forward networks at every layer (residual block) of Pythia transformers. The cross-entropy loss score is the increase in the cross-entropy loss when the input activations replaced by their autoenender reconstruction divided by the increase when the input activations are ablated (set to zero). each SAE, expansion factor is R 64 and & 32; the Jacobian loss coefficient is 1.

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
-104
Pythia-708
Pythia-1660 160m
Pythia-410m
Num. Dead Features
4
2
10
Below 1 X 10
4
2
0
2
3
4
5
0
2
4
8
10
8
12 Layer
16
20
Layer
Layer
Input SAE
Output SAE
Figure 23. Sparsity metries per layer for Jacobian SAEs trained on the feed-forward networks at every layer (residual block) of Pythia transformers. Recall that L norm per token for each of input and output SAEs is fixed at by the TopK activation function. each SAE, the expansion factor is R 64 and k 32; the Jacobian loss coefficient is 1.
Cosine Similarity
Explained Variance
Mean Squared Error
-10
0.96
0.9
0.8
0.94
0.85
0.6
0.92
0.8
0.9
0.2
0.75
Cross-Entropy Loss Score
-104
Num. Dead Features
Abs. Jacobian Values >> 0.01
10
200
0.9
180
0.8
1-
160
0.7
140
10
10a
10a
10
10a
10a
10a
10a
Number of Latents
Number of Latents
Number of Latents
Input SAE
Output SAE
Jacobian
Figure 24, Reconstruction quality, model performance preservation, and sparsity metrics against the number of latents. Here, consider Jacobian SAEs trained on the feed-forward network at layer 3 of Pythia- 70m (model dimension 512) with 32. Recall that the maximum number of non-zero Jacobian values is A² 1024. The reconstruction quality and cross-entropy loss score improve as the number of latents increases, and the number of dead features grows more quickly for the output SAE than the input SAE See Appendix for details of the evaluation metries.
52

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Cosine Similarity
Explained Variance
-10
Mean Squared Error
3
0.9
0.8
2
0.8
0.6
0.7
1
0.4
0.6
0.2
0
Cross-Enin Pross-Entropy Loss Score
-10*
Num. Dead Features
Abs. Jacobian Values >> 0.01
1
800
0.8
600
0.6
100
0.4
200
0.2
0
10"
10a
10
10
10"
10a
Sparsity lc
Sparsity
Sparsity k
Input SAE
Output SAE
Jacobian
Figure 25, Reconstruction quality, model performance preservation, and sparsity metrics against the is largest latents to keep in the TopK activation function. Here, we consider Jacobian SAEs trained on feed-forward network layer of Pythia- with expansion factor R = 64. Recall that the maximum number of non-zero Jacobian values is k.. The reconstruction quality and cross-entropy loss score improve as increases, and the number of dead features decreases. See Appendix E for details of the evaluation metrics.
Cosine Similarity
Explained Variance
-10 Mean Squared Error
0.8
0.8
4
0.6
0.6
0.4
2
0.2
0.4
Cross- Cross-Entropy Loss Score
-10
Num. Dead Features
Abs. Jacobian Values >> 0.01
0.8
600
0.6
400
0.4
200
0.2
10
10
10
10
10
102
1
10
10
100
10a
10a
10a
10
10
10
10
10a
10a
10a
Jacobian Loss Coefficient
Jacobian Loss Coefficient
Jacobian Loss Coefficient
Input SAE
Output SAE
Jacobian
Figure 26. Reconstruction quality, model performance preservation, and sparsity metrics against the Jacobian loss coefficient. Here, we consider Jacobian SAEs trained on the feed-forward network at layer 3 Pythia-70m with expansion factor 64 and 32. Recall that the maximum number of non-zero Jacobian values is 62 = 1024. In accordance with Figure 5, all evaluation metrics degrade for values coefficient above See Appendix for details the evaluation metrics.
53

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Explained Variance
0.5
Cross- Entropy Loss Soore
0.5
0
20
10
0
800
600
400
200
600
400
200
Jacobian L' Norm
Abs. Jacobian Values
0.005
Abs. Jacobian Values > 0.01
Input SAE
Output SAE
Figure 27. Pareto frontiers of the explained variance and cross-entropy loss score against different sparsity measures when varying the Jacobian coefficient. Here, we consider Jacohian SAFs trained feed-forward network layer of Pythia-70m with expansion factor R = 64 and k 32. Recall that the maximum number of (dead) latents is 32768 (64 times the model dimension 51 212), and the maximum number of non-zero Jacobian values 2 1024. See Appendix E for details of the evaluation metrics.
Explained Variance
000
0.5
Cross- Entropy Loss Score
0.5
30
20
10
0
800
600
400
200
600
400
200
0
Jacobian T Norm
Abs. Jacobian Values 0.005
Abs. Jacobian Values > 0.01
Input SAE
Output SAE
Figure Pareto frontiers of the explained variance and cross-entropy loss score against different sparsity measures when varying the Jacobian loss coefficient. The coefficient has relatively small impact on the reconstruction quality and sparsity of the input SAE, whereas has a large effect the sparsity of output SAE elements the feed-forward network layer 7 of Pythia-160m with expansion factor R = 64 and & = 32. Recall that the maximum number of (dead) latents is 49152 (64 times the model dimension 768), and the maximum number of non-zero Jacobian values is A² = 1024. See Appendix for details of the evaluation metries.
Jacobian matrix. Here, we consider Jacobian SAEs trained on
54

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
Number of Jacobian
elements above 0.01
0.5
200
400
0.05
0.01
0.5
0.6
0.7
0.8
0.9
Average cross-entropy score (1 = perfect reconstruction)
Figure 29. The trade-of off between reconstruction quality and Jacobian sparsity as we vary the Jacobian loss coefficient. Each dot represents a pair of JSAEs trained with specific Jacobian cocfficient. Mcasured on layer of Pythia-70m with 32.
We speculate that is necessary to tune our hyperparameters for each layer individually to achieve improved performance; see, for example, Figures 26 and 4 for the variation of our evaluation metrics against the coefficient the Jacobian loss term for individual layers of Pythia-70m and 160m.
F. More data on Jacobian sparsity
In Figure 23 we showed that Jacobians are much more sparse with JSAEs than traditional SAEs. To this end, we provided a representative example of what the Jacobians look like with JSAES vs traditional SAEs. Some readers may object that this is not an opples-to-apples comparison since JSAEs are optimizing for lower LI on the Jacobian, so it may be the case that JSAEs merely induce Jacobians with smaller elements, but their distribution may still be the same. To address this criticism, the examples are L,2 normalized; provide un-normalized versions as well as LI normalized versions of the example Jacobians in Figure 30. We also provide histogram and CDF of the distribution of absolute values of Jacobian elements in Figure 32, which is taken across 10 million tokens.
F.1. Jacobian norms
In this section, we address objection expect some readers will have to measures of sparsity. Our main metric for sparsity is the percentage of elements with absolute values above certain small thresholds (e.g. Figure 2). However, one can imagine two distributions with the same degree of sparsity, but vastly different results on this metric due to a different standard deviation. For instance, imagine two Gaussian distributions, both with = O but with significantly different standard deviations, 01 >> 02. They would score very differently on our metric, but their degrees of sparsity would not be meaningfully different (since sparsity requires there to small handful of relatively large elements). Since our L1 penalty encourages the Jacobians to be smaller, it could be that they simply become more tightly clustered around 0. However, this is not the case. We measure this by looking the "norms" of the Jacobian, i.e. flatten the Jacobian, treat it as vector, and compute its Lp norms. If the Jacobian is merely becoming smaller, we would expect all of its Lp norms to decrease at roughly the same rate. On the other hand, if the Jacobian is becoming sparser, we would expect its L1, L2 norms to decrease while its l.,..., Tex norms, which depend more strongly on the presence or absence of a few large elements, should stay roughly the same. We present these results Figure 35, as we can see, the Jacobian does become slightly smaller, but most of the effect we see indeed the Jacobian becoming significantly more sparse.
55

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
a
(b)
12
24
12
24
12
24
0
12
24
ISAE Traditional SAE
JSAE Traditional SAE
10
10
0.4
0.4
20
20
30
30
0.2
0.2
10
10
20
20
30
0.0
30
0.0
Token
Token 2 Token Token 4
Token
Token 2 Token
Token 4
Figure Comparison Jacobians from traditional SAEs JSAEs, same as Figure with different normalization. (a) Not
normalized. (b) 1.2 normalized. Measured on layer 15 of Pythia-410m.
a
(b)
12
24
0
12
24
0
12
24
0 12 24
12
24
12 24
0
12 24
12 24
ISAE Traditiona SAE
ISAE Traditional SAE
0.075
10
0.6
20
20
0.050
0.4
30
30
10
0.025
10
0.2
20
20
0.000
0.0
30
30
Token Token 2 Token Token 4
Token 1
Token 2
Token 3
Token 4
Figure 31. Comparison of Jacobians from traditional SAEs vs JSAEs, same Figure 2 but with different normalization. (a) LI normalized.
(b) 1.2 normalized. Measured on layer 3 Pythia-70m.
(a)
(b)
1.00 -
40
Proportion of elements below thresholds
JSAEs
Frequency (%)
30
Traditional SAEs
0.75
0.50
Traditional SAEs
10
0.25
JSAEs
0.00
0.02
0.04
0.06
0.08
0.10
0.00
0.02
0.04
0.06
0.08
0.10
Absolute value of Jacobian element
Absolute value of Jacobian elements
Figure 32, Further data showing that ISAEs induce much greater Jacobian sparsity than traditional SAEs. (a) A histogram of the absolute values of Jacobian elements in JSAEs versus traditional SAEs. JSAEs induce significantly sparse Jacobians than standard SAEs. This means that there is relatively small number of onput-output feature pairs which explain very large fraction of the computation heing performed. Note that only the lo elements corresponding active latents included in the histogram the remaining (ny - k) x (n, k) elements are zero by definition both for JSAFs and standard TopK SAFs. The histogram was collected over 10 million tokens from validation subset the C4 dataset, which produced 10.24 billion feature pairs. (b) cumulative distribution function of the absolute values of Jacobian elements, again demonstrating that JSAEs induce significantly more computational sparsity than traditional SAEs. Measured layer 15 of P10m
56

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
(b)
0
10
20 30
0
10
20
30
JSAB Traditional SAF
Proportion of elements above threshold
Traditional SAEs JSAEs
10
75%
0.4
30
50%
0.2
25%
20
0.0
30
0%
Token 1
Token 2
Token 3
Token
0.005
0.010
0.015
0.020
Threshold
Figure ISAEs induce much greater degree sparsity in elements the Jacobian traditional Identical to Figure but measured on layer 3 of Pythia-70m.
-10
-109
6
0.001
0.005
0.001
0.005
0.01
0.05
0.01
0.05
0.1
0.5
0.1
0.5
4
5
Frequency
Fiequency
10
50
10
50
100
500
100
500
2
2
0
5 -2
0.1
0.15
0.2
0
5 10-2
0.1
0.15
0.2
Absolute value Jacobian element
Absolute value Jacobian element
(a) Pythia-70m Layer 3
(b) Pythia-1600 Layer 7
Figure Histograms that show frequency absolute values non-zero Jacobian elements for different values the coefficient the Jacobian loss term. As the coefficient increases, the frequency of larger values decreases, i.e., the Jacobian becomes sparser. We provide further details in Figure 32.
57

---

only taking the Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
L2 norm
L4 norm
Linf norm
0.80
0.08
0.15
0.60
Mean L2 nomm
Mean L4 norm
Mean Linf norm
0.06
0.10
0.40
0.04
0.05
0.20
0.02
0.00
0.00
0.00
Traditional
Traditional (Randomized,
JSAE
JSAE (Randomized)
Traditional
Traditional (Randomized)
JSAE
(Randomized)
Traditional
Traditional (Randomized)
JSAE
JSAE (Randomized)
JSAE
Figure Lp norms the Jacobians. measure these by flattening the Jacobians treating them as vector. These results imply that the Jacobians are in fact becoming sparse, as opposed to merely becoming smaller (see Section F.I). Averaged across I million tokens, measured on layer of P0thia-70m.
Proportion of elements above threshold
JSAEs
0.015%
Traditional
0.01%
0.005%
0.0%
0.005
0.010
0.015
0,020
Threshold
Figure The Jacobians aren't only sparse locally (i.e. on each token each prompt), also globally (i.e. when averaged across many tokens), much more than with traditional SAFs. In particular, here consider the full ny n, Jacobian (i.e. not slicing based the TopK), which we average across million tokens Σ promptassken J) before considering summary statistics. This an important measure it confirms that the connections found by JSAES are indeed sparse in global sense, just when conditioning model input. Measured on layer 15 Pytmia-410m. Note that the small numbers on the y-axis are due to the fact that, unlike in e.g. Figure 2, here average over the tokens on which the corresponding output SAE latent is selected by the TopK activation function (i.e. when least one element in the of the Jacobian is nonzero); this is important because otherwise this measure would significantly conflate the sparsity of the Jacohian itself with the sparsity of the activations of each individual latent.
specific
100% ny лı rather than & We also note that for each element in the Jacobian,
58

---

Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
100%
Auto-interp score
50%
ISAEs (ipul SAE)
Traditional SAEs (inpul
JSAEs (outpot SAE)
Traditional SAEs (output SAE)
0%
Layer
Figure Automatic interpretability scores JSAEs are very similar traditional SAEs. Measured on layers of Pythia 70m using the "fuzzing" scorer from Paulo et al. (2024).
Traditional SAEs (Randomized LLM)
Traditional SAES
ISAFs (Randomized LLM)
Proportion of elements above threshold
80%
60%
40%
20%
0%
0.005
0.010
0.015
0.020
Threshold
Figure 38. Jacobians are substantially more sparse in pre-trained LLMs than in randomly initialized transformers. This holds both when you actively optimize for Jacobian sparsity with JSAEs, and when you don't 't optimize for it and traditional SAEs. The proportion of Jacobian elements with absolute values above certain thresholds. The figure shows the proportion of Jacobian elements with absolute values above certain thresholds. Identical to Figure 7 but measured on layer of Pythia-70m.
a
(b)
(c)
Linear
Propertion of scalar functions in fs
80%
Traditional SAEs
ISAEs
Change in downstream
0.8
latent (abs. value)
JumpReLLU
60%
0.6
Syj
0.4
40%
0.2
Other
20%
0.0
0%
5
Linear
JumpReLU Other
0.0
0.2
0.4
0.6
Sx.
Jacobian element (abs, value)
Figure The function fa, which combines the decoder first SAF, the MLP, the encoder second SAE, mostly linear. Identical to Figure 8 but measured on layer 3 of Pythia-70m.
59


Reviews:
[{"review_id": "BcierlgyAv", "reviewer": "Reviewer_ZBhh", "summary": "This paper addresses the problem of better understanding computations in deep models, particularly LLMs. Recently sparse autoencoders (SAEs) have become popular as a tool to mechanistically understand a model, by decomposing features learnt at any layer into a sparse set of disentangled concepts. This work proposes Jacobian SAE (JSAE), which aims to additionally sparsify the computations by MLP layers in LLMs, by enforcing sparsity on connections between TopK latents at the input and output of the MLP, approximated using the Jacobian. Experimental evaluation is performed to show that JSAEs can learn to reconstruct features similar to traditional SAEs while having much sparser connections for the computations between the constituent input and output SAE. Sanity test experiments are also performed to show that what is learnt is meaningful, e.g. that the sparsity is better for JSAEs trained on a learnt MLP as opposed to a randomly initialized MLP.\n\n## Update after rebuttal\nThank you for your response, particularly for providing examples. They look quite interesting, and it would be very helpful to have more examples (or some latent exploration tool) in the final paper. Regarding the question about zeroing out connections: I actually meant something even simpler---one could pick a small threshold, e.g. 1e-4, and set any weight in the JSAE below that threshold to zero _post-hoc_, and then compute the performance, to see if the weights below the threshold are actually meaningfully important. This was intended to be more of a sanity check to verify results from Figure 3.\n\nOverall however I believe this is an interesting and useful paper, so I would like to change my score to accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review, noting \"(1) The paper looks into an interesting and potentially useful problem, of understanding computations made by large models, as opposed to just examining what features were learnt as is typically done with SAEs. (2) The proposed idea of learning sparse connections using the Jacobian is interesting and appears to be novel. (3) Experimental evaluation shows that the learnt JSAEs perform comparably to traditional SAEs while also being significantly sparser. (4) The paper is generally well written and easy to follow.\"\n\n### Qualitative results\n\nWe have updated the manuscript with qualitative [results](https://anonymous.4open.science/r/jacobian-saes-icml-D7BF/jacobian-saes-icml-examples.pdf). Specifically, we plot max-activating examples for an output feature, and the 16 input features that connect to it most strongly. The very first output latent (34455) in that document that activates fairly generally on German text. The input latents respond to: \n* Tokens that frequently appear in German text.  There is an input that mainly responds to \"Pf\", and another that responds to \"sch\", and another that responds to \"Kle\".\n* \"German\"\n* \"von\"\n* Place names such as \"Austria\" and \"Berlin\"\n\nThere are also far more subtle examples, such as output latent 52843 (easiest to Ctrl-F). This latent is activated by \"go\" and \"goes\" in different figurative contexts, including \"go on record\" and \"go behind his back\"; it does not appear to respond to the literal meaning of movement through space. This distinction may be part of what this circuit is designed to clarify. One of the corresponding inputs include measure of extent and distance, like \"a long way\" and \"right to the 6 th floor\" and another to the word \"back\" in \"go back\". Other input latents include \"go\" in a figurative context, or words that have a similar role to \"go\" such as \"proceed\", \"headed\" and \"come\" and even one responding to \"GO\" or \"GOTO\" in computer code. \n\nNote that the output features are not hand-picked, but are ordered starting with the output with the largest element of the Jacobian averaged across all inputs.\n\n### Zeroing-out connections with small Jacobians\n\nThere are two different kinds of zeros in our Jacobians.\n* First, there are zeros in the Jacobian that arise due to most of the inputs and outputs being ``off'', as they are zeroed out by the k-sparse SAEs.  This sparsity is already accounted for in the reconstruction errors in the main text.\n* Second, there are small elements of the Jacobian even where the input and output features are on.\n\nGiven our overall approach, this second kind of sparsity in the Jacobian is very difficult to zero-out.\nWe chose a particular approach which has many advantages, but sadly does not allow for an analysis such as this.\nSpecifically, we needed to choose between two potential approaches:\n* You could compute the Jacobian of the \\textit{actual} MLP in the underlying network. This has the critical advantage of having no approximation error, as it is working with the Jacobian of the actual MLP.\n* You could train a function with a simple/sparse Jacobian to approximate the input/output behavior of the MLP. This has the disadvantage that there is now approximation error.  But on the other hand, this function can designed to be manipulated more easily.\n\nWe chose the first approach, as it removes a source of approximation error (from the function you've trained to approximate the MLP).  But it does mean that you can't zero-out an element of the Jacobian, because this is the Jacobian of the underlying MLP, and we don't know how to zero-out elements of a Jacobian from a complex MLP. The second approach would allow for this, and we intend to investigate it in future work.\n\n> $L^1 \\rightarrow L_1$.  \n\nThanks!  Fixed.\n\n### Conclusions\n\nThank you for generously outlining the paper's strengths in your original review.\nWe hope this response (and especially the new qualitative results) has addressed your key concerns.\nIf so, we would greatly appreciate it if you would reconsider your score.", "claims_and_evidence": "Generally yes, except for concerns raised in the Weaknesses section below.", "methods_and_evaluation_criteria": "The method makes sense. The evaluations are reasonable but inadequate, as discussed in the Weaknesses section below.", "theoretical_claims": "I looked over the claims in Appendix A and B but did not go through them in full detail. To that extent, they appear reasonable.", "experimental_designs_or_analyses": "The experimental design and analyses appear sound to the extent they have been done. However, additional evaluation is needed, as discussed in the Weaknesses section below.", "relation_to_broader_scientific_literature": "This work builds upon existing literature that aims to disentangle concepts learnt by deep models such as LLMs using SAEs. However, different from typical SAEs, it also focuses on sparsifying and understanding the computations in the LLM and not just features at a given layer. This is similar to the idea of transcoders, but not identical, as discussed in Section 2.2.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "## Strengths\n1. The paper looks into an interesting and potentially useful problem, of understanding computations made by large models, as opposed to just examining what features were learnt as is typically done with SAEs.\n2. The proposed idea of learning sparse connections using the Jacobian is interesting and appears to be novel.\n3. Experimental evaluation shows that the learnt JSAEs perform comparably to traditional SAEs while also being significantly sparser.\n4. The paper is generally well written and easy to follow.\n\n\n## Weaknesses\n1. The biggest weakness is that there does not seem to be any example of a use case of such JSAEs. In typical SAEs, one can see examples of latents learning human interpretable features, along with evaluations, both qualitative and quantitative, to show that they are (for the most part) disentangled and meaningful (e.g. Bricken et al. 2023, Bills et al. 2023). Evaluations also often show downstream uses such as model steering. However, this work does not have any indication on what JSAEs could be used for at all. Since the stated motivation is to learn a sparse mapping between SAE concepts before and after an MLP, it would be critical to see what this learnt mapping actually encodes, particularly since the point of training such SAEs is to aid human interpretability. At the very least, one should have qualitative examples of concepts at the input, output, and weights connecting them, and ideally have a more thorough quantitative analysis.\n2. The sparsity of JSAEs is shown by thresholding weights and counting how many are above the threshold. However, it is possible that the weights below this threshold still contribute meaningfully—it would be make more sense to evaluate reconstruction quality by explicitly zeroing out everything below the threshold and then computing the metrics in Figure 3. This would after all be the likely practical use of such trained JSAEs.", "other_comments_or_suggestions": "- L048: $L^1 \\to L_1$", "questions_for_authors": "Please refer to the Weaknesses section. While the idea and approach appear interesting, in particular Weakness 1 is a critical omission and needs to be addressed. Any discussion on this in the rebuttal would be helpful.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zATTDwb6C6", "reviewer": "Reviewer_WxFv", "summary": "This work introduces Jacobian Sparse Autoencoders (JSAEs), an extension of sparse autoencoders (SAEs) designed to sparsify not only latent activations but also the computational graph (approximated via the Jacobian) of an MLP layer in LLMs models.\nAuthors also show how to compute efficiently the Jacobian thanks to the top-k SAE approach (namely only the non-zero parts are computed), encouraging the Jacobian to be sparse. Their findings include:\n- JSAEs significantly increase the sparsity of the Jacobian between input and output latent spaces compared to standard SAEs with minimal cost and small drop in reconstruction quality and model performance.\n- Input and output latents learned  by Jacobian SAEs are approximately as interpretable as  standard SAEs, as quantified by auto-interpretability scores.\n- When applied to randomly initialized transformers, JSAEs achieve much less Jacobian sparsity than when applied to pre-trained transformers. This suggests that Jacobian sparsity is a property of the trained model, indicating its potential as a tool to uncover learned computational structure, something latent sparsity alone may not capture.\n- The function learned by the JSAE-MLP-JSAE pipeline is shown to be mostly linear, meaning the Jacobian is a reliable proxy for true computational sparsity in this context.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive review, which notes \"The idea of sparsifying the computation (not just the representation) is new and compelling. It nicely bridges concepts from automated circuit discovery and dictionary learning.\nInsightful findings for discovering sparse computation in LLMs.\nThe papers is really well-written and organized\".\n\n### Qualitative results\n\nWe have updated the manuscript with qualitative [results](https://anonymous.4open.science/r/jacobian-saes-icml-D7BF/jacobian-saes-icml-examples.pdf). Specifically, we plot max-activating examples for an output feature, and the 16 input features that connect to it most strongly. The very first output latent (34455) in that document that activates fairly generally on German text. The input latents respond to: \n* Tokens that frequently appear in German text.  There is an input that mainly responds to \"Pf\", and another that responds to \"sch\", and another that responds to \"Kle\".\n* \"German\"\n* \"von\"\n* Place names such as \"Austria\" and \"Berlin\"\n\nThere are also far more subtle examples, such as output latent 52843 (easiest to Ctrl-F). This latent is activated by \"go\" and \"goes\" in different figurative contexts, including \"go on record\" and \"go behind his back\"; it does not appear to respond to the literal meaning of movement through space. This distinction may be part of what this circuit is designed to clarify. One of the corresponding inputs include measure of extent and distance, like \"a long way\" and \"right to the 6 th floor\" and another to the word \"back\" in \"go back\". Other input latents include \"go\" in a figurative context, or words that have a similar role to \"go\" such as \"proceed\", \"headed\" and \"come\" and even one responding to \"GO\" or \"GOTO\" in computer code. \n\nNote that the output features are not hand-picked, but are ordered starting with the output with the largest element of the Jacobian across all inputs.\n\n### Discussion of extending JSAEs to full multi-layer transformer analysis\n\nWe agree that we applied JSAEs only to locally (to individual MLPs). This mirrors many other interpretability approaches, such as: \n* SAEs, which allow you to interpret the activations at a single location in the network\n* transcoders, which allow you to interpret the activations again for an MLP.\n\nThe difference between JSAEs and these approaches is that there is a natural extension of JSAEs to \"global interpretability\".  In particular, you can train SAEs at many points in the network, and minimize the Jacobian for the mappings between latent activations in these SAEs.\nIndeed, we believe that this methodology could naturally be to further sparify the Jacobians between _transcoder latents_ in the fantastic work released by Anthropic only a few days ago [1].\n\n[1] https://transformer-circuits.pub/2025/attribution-graphs/methods.html\n\nWe have added this discussion to our working draft.\n\n> Figure 1, write in the figure (not only in the caption) what  is.\n\nThanks!  Fixed.\n\n> Are the MLP parameters updated when training JSAEs?\n\nNo.  The underlying MLP parameters are fixed to those in pre-training.\nThe only thing that is updated is the encoder/decoder of the SAEs.\nThe new thing is including a term in the loss for the encoder/decoder of the SAE which encourages sparse Jacobians when we consider the mapping from sparse inputs to sparse outputs.\n\n> Figure 2, what is the Jacobian of traditional SAEs?\n\nTraditional SAEs still have a Jacobian. You can still train a traditional SAE at the input and output of an MLP, then look at the Jacobian from sparse inputs to sparse outputs. The difference between an SAE and a JSAE is simply in the training objective. Specifically, the JSAE objective includes a term that encourages sparse Jacobians, while the SAE does not. As such, the SAE is also equivalent in e.g. Figure 3 to a Jacobian Loss Coefficient of zero.\n\n### Conclusions\n\nThank you for generously outlining the paper's strengths in your original review. We hope this response (especially the new qualitative results) has addressed your key concerns. If so, we would greatly appreciate it if you would reconsider your score.", "claims_and_evidence": "Yes, claims are well-supported by both theoretical derivations and experimental evidence.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes. The setups, loss, efficient computation of Jacobian and almost linearity of $f_s$ seems correct.", "experimental_designs_or_analyses": "Yes, the setting is legit. However, the paper would benefit from more qualitatevely results (i.e. the standard mechanistic approach to identify what concepts or features the model has learned).", "relation_to_broader_scientific_literature": "The paper builds directly on prior work on SAEs and contributes to sparsify not only the activations but also the computational graph.", "essential_references_not_discussed": "Essential references are discussed. However, in Section 2 the authors should mention why SAEs are introduces in general, so the superposition hypothesis and polysemantic neurons.", "other_strengths_and_weaknesses": "### Strengths\n\n- The idea of sparsifying the computation (not just the representation) is new and compelling. It nicely bridges concepts from automated circuit discovery and dictionary learning.\n\n- Insightful findings for discovering sparse computation in LLMs.\n\n- The papers is really well-written and organized (yes for me it's a strenght because it's not always the case, even for top conferences).\n\n### Weaknesses\n\n- Lack of qualitatevely results (i.e. the standard mechanistic approach to identify what concepts or features the (J)SAEs have learned).\n- JSAEs are only trained on individual MLPs. This is fine as a starting point, but the paper lacks a discussion on how to extend this to full multi-layer transformer analysis.", "other_comments_or_suggestions": "Figure 1, write in the figure (not only in the caption) what $\\tau_k$ is.", "questions_for_authors": "- Are the MLP $f$ parameters updated when training JSAEs?\n\n- Figure 2, what is the Jacobian of traditional SAEs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cLzqdeAzQG", "reviewer": "Reviewer_M6HE", "summary": "The authors introduce Jacobian SAEs, as a form of dictionary learning that also facilitates circuit analysis by creating sparse computational graphs. They jointly train SAEs on both input and output to an MLP layer, and make the Jacobian sparse by adding its L1 norm to the loss function.\n\nThough computing and differentiating the Jacobian is extremely complicated naïvely, the process is made tractable by analytically computing the Jacobian, leveraging the simple mathematical form of an MLP. Only standard MLPs studied, not the gated MLPs used in most common most modern models. Since only the k active latents matter, the operation only doubles the cost in the context, though the exact computational complexity as models scale isn't explored.\n\nThe most relevant prior work is transcoders which replace MLP layers with a wider sparser replacement. The authors argue that Jacobian SAEs are distinct as transcoders sparsify computation while their method sparsifies input and output.\n\nTo show JSAE effectiveness, they demonstrate:\n\n-   While Jacobian sparsity regularization degrades other metrics there's a sweet spot with reasonable performance and better sparsity\n-   Cross-entropy and explained variance are comparable to normal SAEs\n-   Auto-interpretability scores are comparable to normal SAEs\n-   They find sparser Jacobians with trained JSAEs on trained transformers than randomly initialised ones\n\nTo validate the approach of studying gradients to understand the computation of the MLP layer, on a given prompt they vary each SAE input latent to see effects on each output latents. They show that 88% of relationships are linear, many are ReLU like, and some are more complex", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and considered review!\n\n### 1. Transcoders vs JSAEs\n\nA few days ago, Anthropic published a huge new paper on circuit tracing, and which cites our work [1].\nThe key idea in our work and [1] is:\n* Take multiple layers in network.\n* Decompose each layer into sparse latents (we use SAEs at different layers, [1] uses transcoders at different layers).\n* Take the Jacobian for the mapping from sparse latents at one layer to sparse latents at later layers.\n* Identify strong connections using large elements in that Jacobian.\n* Interpret those connections (which is much _much_ easier if the connections are sparse).\n\nOur insight is that in any such approach, the Jacobians are denser than they could be because we don't have an efficient loss term that explicitly encourages Jacobian sparsity.\nWe introduce such a loss term.\nOf course, the most natural place to apply such methods is to SAEs at different points in the network, as we did in our paper.\nBut you could equally extend our idea of sparsifying Jacobians to the transcoders at different layers in [1] to get even sparser, more interpretable circuits.\n\n> The input vector of a transcoder latent can be projected onto a normal residual SAE's decoder to see which concepts are used, though perhaps that would have too much interference?\n\nWe agree you could do this, but based on our results, we would expect that the resulting projection would be dense, (or at least denser than it could be) and therefore difficult to interpret.  We will check for the camera-ready.\n\n[1] www.transformer-circuits.pub/2025/attribution-graphs/methods.html\n\n### 2. Toy tasks\n\nWe have not been able to run toy tasks during the short rebuttal period, but will explore this for the camera-ready.\n\n### 3. Qualitative results\n\nWe have updated the manuscript with qualitative [results](https://anonymous.4open.science/r/jacobian-saes-icml-D7BF/jacobian-saes-icml-examples.pdf). Specifically, we plot max-activating examples for an output feature, and the 16 input features that connect to it most strongly. The very first output latent (34455) in that document that activates fairly generally on German text. The input latents respond to: \n* Tokens that frequently appear in German text.  There is an input responding \"Pf\", another responding to \"sch\", and another responding to \"Kle\".\n* \"German\"\n* \"von\"\n* Place names such as \"Austria\" and \"Berlin\"\n\nThere are also far more subtle examples, such as output latent 52843 (easiest to Ctrl-F). This latent is activated by \"go\" and \"goes\" in different figurative contexts, including \"go on record\" and \"go behind his back\"; it does not appear to respond to the literal meaning of movement through space. This distinction may be part of what this circuit is designed to clarify. One of the corresponding inputs include measure of extent and distance, like \"a long way\" and \"right to the 6 th floor\" and another to the word \"back\" in \"go back\". Other input latents include \"go\" in a figurative context, or words that have a similar role to \"go\" such as \"proceed\", \"headed\" and \"come\" and even one responding to \"GO\" or \"GOTO\" in computer code. \n\nNote that the output features are not hand-picked, but are ordered starting with the output with the largest element of the Jacobian averaged across all inputs.\n\nIt's important to highlight that, as far as we can see, you could not get this kind of insight using transcoders of standard SAEs. Our qualitative results show that the MLP computes its output feature using _a specific set of input features_; transcoders and standard SAEs do not attempt to locate the input features used to compute a specific output feature.\n\n### 4. Pathologies \n\nThe qualitative examples above usually show one input feature that's similar, but a lot of other input features that are different. So the pathology you are suggesting does seem to occur, but it doesn't seem to obscure other interesting patterns.\n\n### 5. Skip transcoders\n\nThis is a great idea! This is definitely worth trying to see whether it makes things more interpretable. We will run analysis for the camera-ready deadline.\n\n### 6. Sparsity\n\nWe agree that our current manuscript could be clearer on this point and we will clarify it in the camera-ready! We did check the global sparsity levels and your intuition is correct, it is indeed quite sparse globally. We will include data on this in the camera-ready.\n\n### Gated MLPs\n\nWe've already worked out the math for this and will add an Appendix with experiments for the camera-ready\n\n### Feature absorption\n\nWe have cut the claim\n\n### Linearity\n\nAgreed, we will add this nuance\n\n### Conclusions\n\nThank you for your extensive and excellent comments! Given the number of great suggestions you made, we hope you will forgive the need to explore several points in the camera-ready deadline. If this response (especially the new qualitative results) has nonetheless addressed your key concerns, we would greatly appreciate it if you would reconsider your score.", "claims_and_evidence": "See below", "methods_and_evaluation_criteria": "See below", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See below", "relation_to_broader_scientific_literature": "See below esp re transcoders", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "See below", "other_comments_or_suggestions": "## Major Comments\n*The following are things that, if adequately addressed, would increase my score*\n\nSummary; I overall consider this paper to be a weak accept. I think it provides some innovative new ideas, in particular using Jacobians as a way to measure the sparsity of a computational graph and tricks for how to compute and train this well. However, I don't feel that the authors provide a compelling enough argument for why one would prefer these over prior techniques like transcoders. If I were convinced there were compelling use cases where one would prefer JSAEs, I would be a lot more excited about this paper.\n\n1. The paper doesn't clearly explain why I should prefer this method over transcoders. Transcoders already provide sparsity in their output, as the output is essentially a sparse linear combination of the decoder vectors from transcoder units. The main difference seems to be that transcoders don't explicitly sparsify the input, only the output. However, it’s unclear why sparsifying the input is especially valuable or significant. The input vector of a transcoder latent can be projected onto a normal residual SAE's decoder to see which concepts are used, though perhaps that would have too much interference?\nOne potential advantage of JSAEs could be clearer representation in cases where multiple transcoder latents share identical decoders. JSAEs would represent this scenario more cleanly with a single output latent. \nIf any of these differences matter, the paper should explicitly explain why and ideally provide a concrete example. \n2. More broadly, the paper doesn't clearly motivate why Jacobian SAEs are interesting or practically useful. I'm sympathetic to the idea that high-quality circuit analysis with interpretable nodes is valuable, and am excited about improved techniques there. However, if you're trying to argue that JSAEs improve interpretability compared to methods like sparse feature circuits on normal SAEs or transcoders, then clear empirical evidence is needed. For example, analyzing tasks or contexts studied in [Dunefsky et al.](https://arxiv.org/abs/2406.11944) or [Marks et al.](https://arxiv.org/abs/2403.19647) and explicitly showing that JSAEs give better or clearer insights would significantly strengthen your argument. Or, pick a task where MLPs matter like the greater-than task from [Hanna et al.](https://arxiv.org/abs/2305.00586) and concretely show how JSAEs provide new insights or clearer interpretations.\n3. Also, the paper would greatly benefit from more qualitative analysis. Right now, the argument primarily relies on summary statistics, which can obscure what's actually happening underneath. It would help if you showed the correlation or cosine similarity between transcoder latents and output JSAE latents, or between JSAE and normal SAE latents. Are these methods capturing similar concepts but with slightly different directions, or is the JSAE method genuinely capturing fundamentally different computations to achieve sparse Jacobians?\n4. Another concern is about potential pathologies: [Lindsey et al.](https://transformer-circuits.pub/2024/crosscoders/index.html) demonstrated substantial redundancy and showed concept directions in the residual stream shifting pre- and post-MLP. My hypothesis is that this occurs because MLPs have a significant first order linear component that explains a fair amount of the variance in the output. Consequently, if this is true, achieving a sparse Jacobian might just reflect input and output SAEs capturing essentially the same concepts, with the MLP performing a simple linear transformation between them. This would result in a clean and sparse Jacobian, especially if the input features are nearly orthogonal. However, this scenario would not be particularly interesting or insightful.\n5. A suggestion for improving this approach: inspired by [skip transcoders](https://arxiv.org/abs/2501.18823), if part of the issue is the MLP's output having a large linear component, why not explicitly train a linear approximation, subtract it off, and then apply your JSAE to the nonlinear remainder? This way, your Jacobian sparsity would more meaningfully represent non-linear computations rather than simple linear mappings. This might also simplify interpretation, as a purely linear output essentially extends the input SAE, which isn't particularly interesting and wastes capacity in the output SAEs.\n6. Another concern: the paper implicitly mixes two distinct notions of sparsity—local sparsity (how each concept affects each output on a single prompt) and global sparsity (connections existing across all prompts). You might have an output latent activated by many different input latents but on any given prompt only fires due to a single input latent as they're all anti-correlated. Locally, that's sparse, but globally, it's dense. Currently, the paper only looks at sparsity on individual prompts. It would be helpful to see global statistics—for instance, looking at multiple prompts and measuring the probability that a given edge exists conditional on the output latent activating, then assessing sparsity at this global scale. My intuition is that it might still be somewhat sparse globally, but clarifying and verifying this explicitly is worthwhile.\n\n\n\n## Minor Comments\n*The following are unlikely to change my score, but are comments and suggestions that I hope will improve the paper, and I leave it up to the authors whether to implement them. No need to reply to all of them in the rebuttal*\n\n1. I do like the trick used for efficiently computing Jacobians as a proxy for sparse computational graphs. It's clever and seems to be executed effectively.\n2. Only studying GELU MLPs is somewhat limiting, as gated MLPs are now the standard, so this work is not directly applicable to modern LLMs. While your methods should transfer fine (albeit with more complicated derivative calculations), adding an appendix discussing this extension would help make the paper feel more complete.\n3. Regarding your mention of feature absorption in the intro, it's not clear at all that JSAEs address this issue effectively. When concepts consistently co-occur, like \"elephant\" and \"starts with E,\" they typically merge into something like \"starts with E but isn't elephant.\" I don’t see why this would differ significantly in JSAEs. If this isn't a key claim, it might be better either to justify it clearly or just cut it.\n4. The summary statistics about relationships between input/output latents have an unclear relationship to linearity. Since near-zero Jacobian values imply trivial linearity, it would help to filter first for active edges to better highlight meaningful relationships. The paper mentions that 88% of relationships are linear, but it isn't clear whether this percentage is notably high or low since I don't have good context for these numbers. Clarifying whether this is good or bad and providing context for this number by comparing to eg the fraction of trivial entries in the Jacobian, would significantly help interpretation.", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SMJrV9lAA8", "reviewer": "Reviewer_PVaJ", "summary": "**Summary**\n\nSparse Autoencoders (SAEs) help interpret latent activations in LLMs but do not explicitly reveal how computations are performed. This paper extends SAEs to study the sparsity of the computational transformations within MLP layers of transformer-based LMs. The authors train two SAEs—one before and one after an MLP transformation—and enforce sparsity in the Jacobian of the input-output mapping. This is done via an $\\ell_1$ penalty on the Jacobian, encouraging learned features to be sparse linear functions of each other. To make this tractable, the paper introduces optimizations leveraging the inherent sparsity of SAE latents. Main results: (a) JSAEs induce Jacobian sparsity without significantly degrading reconstruction quality, (b) Trained LMs exhibit greater computational sparsity than randomly initialized ones, indicating that sparsity is a learned property, (c) Jacobian sparsity is a reasonable proxy for computational sparsity since the JSAE+MLP mapping is approximately linear.\n\n**Strengths**\n- JSAEs extend recent SAE-based interpretability techniques from analyzing activations to analyzing computations, providing a more structured view of how LLMs process information. The problem is formulated in a clean and well-motivated way. The paper also does a great job at differentiating JSAEs from standard SAEs and transcoders, clearly explaining the conceptual differences in what each method sparsifies.\n- The proposed optimizations make Jacobian regularization computationally feasible for large models, where naive Jacobian computation would be intractable. JSAEs only increase compute by ~2x compared to standard SAEs.\n- Evaluation metrics demonstrate minimal trade-offs between sparsity and reconstruction fidelity, suggesting that the Jacobian penalty does not significantly degrade representation quality.\n- Strong empirical results show that JSAEs extract meaningful structure rather than fitting noise, as demonstrated in the random initialization ablation. Trained LMs exhibit far greater Jacobian sparsity than randomly initialized ones, suggesting that JSAEs capture properties learned during training.\n\n**Weaknesses**\n- The paper provides a tool to study MLP computaion but does not use it to provide new insights into MLP computation. For example, the work would be stronger with case studies showing what JSAEs reveal about computation: Can JSAEs recover known mechanistic circuits in toy models? Do they expose meaningful functional structures in large LMs? How would an interpretability researcher use JSAEs to answer practical questions about LLM internals?\n- The study focuses on reconstruction and sparsity metrics, but does not evaluate whether JSAE features are useful for real-world interpretability tasks. The study should use recent benchmarks for SAEs to incorporate additional metrics. AxBench (https://arxiv.org/abs/2501.17148) evaluate SAE features for concept detection and model steering and SAEBench (https://www.neuronpedia.org/sae-bench/info) provides structured metrics for assessing interpretability and feature usefulness. Comparing JSAEs to these benchmarks would make the empirical results more convincing.\n- The method is specialized for MLP modules and does not consider attention mechanisms, residual streams, or full transformer computations. It focuses on single MLP layers, making it unclear how JSAEs scale to studying entire transformer models with hundreds of MLPs. This limitation suggests that JSAEs may be more suited for local rather than global interpretability.\n- Missing related work on interpretability techniques that focus on model computation. Shah et al. (https://arxiv.org/abs/2404.11534) uses linear surrogate models to analyze how model components (e.g., conv filters in vision models, MLPs) in a trained neural network contribute to final model outputs. Similarly, Balasubramanian et al. (https://arxiv.org/abs/2406.01583) uses attribution techniques to quantify the role of individual components in shaping vision model representations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments and useful recommendations.\n\n\n### Qualitative results and insights\n\nWe have updated the manuscript with qualitative [results](https://anonymous.4open.science/r/jacobian-saes-icml-D7BF/jacobian-saes-icml-examples.pdf). Specifically, we plot max-activating examples for an output feature, and the 16 input features that connect to it most strongly. The very first output latent (34455) in that document that activates fairly generally on German text. The input latents respond to: \n* Tokens that frequently appear in German text.  There is an input that mainly responds to \"Pf\", and another that responds to \"sch\", and another that responds to \"Kle\".\n* \"German\"\n* \"von\"\n* Place names such as \"Austria\" and \"Berlin\"\n\nThere are also far more subtle examples, such as output latent 52843 (easiest to Ctrl-F). This latent is activated by \"go\" and \"goes\" in different figurative contexts, including \"go on record\" and \"go behind his back\"; it does not appear to respond to the literal meaning of movement through space. This distinction may be part of what this circuit is designed to clarify. One of the corresponding inputs include measure of extent and distance, like \"a long way\" and \"right to the 6 th floor\" and another to the word \"back\" in \"go back\". Other input latents include \"go\" in a figurative context, or words that have a similar role to \"go\" such as \"proceed\", \"headed\" and \"come\" and even one responding to \"GO\" or \"GOTO\" in computer code. \n\nNote that the output features are not hand-picked, but are ordered starting with the output with the largest element of the Jacobian averaged across all inputs.\n\n### Evaluation\n\nThese benchmarks were not available on the ICML submission date ([1] was published in March, and [2] was published in very late January).\nAs such, we plan to run them for the camera ready.\n\n[1] https://arxiv.org/abs/2503.09532\n\n[2] https://arxiv.org/abs/2501.17148\n\n### Other model components\n\nWe agree that we used JSAEs only on MLPs, i.e. in the local interpretability setting, as it was a useful starting point to introduce the JSAE methodology.\nWe note that other interpretability methods, e.g. SAEs and transcoders, were also initially developed only on a single MLP at a time.\n\nTo understand out approach to moving towards using JSAEs to obtain a global understanding of LLMs, it is worth stepping back and thinking an overall approach in the interpretability literature, of which the JSAE forms a part. Perhaps the most exciting recent work following this approach is the Anthropic circuit tracing paper, which was published only a few days ago, and cites our work.\nThe key idea in our work and [3] is:\n* Take multiple layers in network.\n* Decompose each layer into sparse latents (we use SAEs at different layers, Anthropic use transcoders at different layers).\n* Take the Jacobian for the mapping from sparse latents at one layer to sparse latents at later layers.\n* Identify strong connections using large elements in that Jacobian.\n* Interpret those connections (which is much _much_ easier if the connections are sparse).\n\nIn any such approach, whether local (our current work) or global ([3]), the Jacobians are denser than they could be, simply because we don't have a term in the loss that explicitly encourages them to be sparse.\nOur key insight was that it is actually possible to develop an efficient loss term that sparsifies Jacobians.\nWhile we initially only applied this loss term to MLPs, the overall approach certainly could be applied globally, indeed, we are super-excited about further sparsifying the Jacobians between transcoders latents studied in [3] to get even sparser, more interpretable global circuits.\n\n[3] https://transformer-circuits.pub/2025/attribution-graphs/methods.html\n\n### Additional Related work\n\nThanks!  We have added this to the Related Work section in our working manuscript.\n\n### Conclusion\n\nThank you for generously outlining the paper's strengths in your original review.\nWe hope this response (and especially the new qualitative results) have addressed your key concerns.\nIf so, we would greatly appreciate it if you would reconsider your score.", "claims_and_evidence": "Yes. Please see my review above", "methods_and_evaluation_criteria": "Please see my review above", "theoretical_claims": "Yes, the appendix on efficiently computing MLP Jacobians.", "experimental_designs_or_analyses": "Please see my review above", "relation_to_broader_scientific_literature": "Please see my review above", "essential_references_not_discussed": "Please see my review above", "other_strengths_and_weaknesses": "Please see my review above", "other_comments_or_suggestions": "Please see my review above", "questions_for_authors": "Please see my review above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies sparse autoencoders and proposes a new methodology to sparsify also the computations and not just the activations, while preserving reasonable performance. The discussion between the reviewers and the authors was insightful, and the paper's scores significantly improved after the rebuttal, arriving to unanimous acceptance. Therefore, in consultation with the reviewers, the AC suggests acceptance and recommends the authors to include all rebuttal discussions and new qualitative results in the final version of the paper.
Decision: Accept (poster)