Sparse Autoencoders Find Highly Interpretable Features in Language Models
Authors: Robert Huben
Hoagy Cunningham
Logan Riggs Smith
Aidan Ewart
Lee Sharkey
Venue: ICLR 2024 poster (2024)
Keywords: language model
interpretability
representation learning
sparsity
dictionary learning
unsupervised learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We use a scalable and unsupervised method called Sparse Autoencoders to find interpretable, monosemantic features in the residual streams of real LLMs (Pythia-70M/410M).

Abstract: One of the roadblocks to a better understanding of neural networks' internals is \textit{polysemanticity}, where neurons appear to activate in multiple, semantically distinct contexts. Polysemanticity prevents us from identifying concise, human-understandable explanations for what neural networks are doing internally. One hypothesised cause of polysemanticity is \textit{superposition}, where neural networks represent more features than they have neurons by assigning features to an overcomplete set of directions in activation space, rather than to individual neurons. Here, we attempt to identify those directions, using sparse autoencoders to reconstruct the internal activations of a language model. These autoencoders learn sets of sparsely activating features that are more interpretable and monosemantic than directions identified by alternative approaches, where interpretability is measured by automated methods. Moreover, we show that with our learned set of features, we can pinpoint the features that are causally responsible for counterfactual behaviour on the indirect object identification task \citep{wang2022interpretability} to a finer degree than previous decompositions. This work indicates that it is possible to resolve superposition in language models using a scalable, unsupervised method. Our method may serve as a foundation for future mechanistic interpretability work, which we hope will enable greater model transparency and steerability.

Published as a conference paper at ICLR 2024
SPARSE AUTOENCODERS FIND HIGHLY INTER- PRETABLE FEATURES IN LANGUAGE MODELS
Hoagy Cunningham" 122 Aidan Ewart Logan Riggs"1, Robert Huben, Lee Sharkey'
EleuthhrAA, "MATS, 3 Uniivessity of Bristol, 4 "Apollo Research
(hoagycunningham, aidenprattewart, logansmi th gma com
ABSTRACT
One of the roadblocks to better understanding of neural networks' internals is polysemanticity, where neurons appear activate in multiple, semantically dis- tinct contexts. Polysemanticity prevents us from identifying concise, human- understandable explanations for what neural networks are doing internally. One hypothesised cause of polysemanticity is superposition, where neural networks represent more features than they have neurons by assigning features to an over- complete set of directions in activation space, rather than to individual neurons. Here, we attempt to identify those directions, using sparse autoencoders to re- construct the internal activations of language model. These autoencoders learn sets of sparsely activating features that are more interpretable and monoseman- tic than directions identified by alternative approaches, where interpretability is measured by automated methods. Moreover, we show that with our learned set of features, we can pinpoint the features that are causally responsible for coun- terfactual behaviour on the indirect object identification task (Wang et al.) 2022) finer degree than previous decompositions. This work indicates that possible to resolve superposition in language models using scalable, unsuper- vised method. Our method may serve as a foundation for future mechanistie in- terpretability work, which we hope will enable greater model transparency and steerability.
INTRODUCTION
Advances in artificial intelligence (AI) have resulted in the development of highly capable Al sys- tems that make decisions for reasons we do not understand. This has caused concern that Al systems that we cannot trust being widely deployed in the economy and in our lives, introducing num- ber of novel risks (Hendrycks et al. 2023), including potential future risks that Als might deceive humans in order accomplish undesirable goals (Ngo et al. 2022 Mechanistic interpretability seeks to mitigate such risks through understanding how neural networks calculate their outputs, al- lowing reverse engineer parts of their internal processes and make targeted changes them
(Cammarata et al. 2021 Wang et al. 2022 Elhage et al. 2021).
To reverse engineer neural network,
necessary to break down into smaller units (features) that can be analysed in isolation. Using individual neurons as these units has had some success (Olah al. 2020, Bills al.) 2023), but key challenge has been that neurons are often polysemantic, activating for several unrelated types of feature (Olah et al., (020)) Also, for some types of network activations, such as the residual stream of transformer, there little reason to expect features to align with the neuron basis (Elhage et al. 2023).
Elhage et al. (2026) investigate why polysemanticity might arise and hypothesise that may result from models learning more distinct features than there are dimensions in the layer. They call this phenomenon superposition. Since a vector space can only have as many orthogonal vectors as it has dimensions, this means the network would learn overcomplete basis of non-orthogonal features. Features must sufficiently sparsely activating for superposition to arise because, without
"Equal contribution
Code replicate experiments can he found http:///ggttbbb.oo////eee/////s/

---

Published as a conference paper at ICLR 2024
Language
Sparse Autoencoder
Festure Dictionary
Embedding
Activation Vector
Peature
Meaning
Integretabiity
Encoder matrix
Score
D Transtormer Blocks
(tied with decoder)
Words
apply
Sparse feature coefficients
5-3333
Decoder matrix (dictionary)
k-2048
Chemsstry
Unembedding
B 0
Reconstructed activation vector
Interpret the resulting dictionary festures
Sample activations from language model
Learn dictionary using autoencoder that represent activation vectors as sparse linerr combination of feature vectors.
Figure 1: An overview of our method. We a) sample the internal activations of language model, either the residual stream, MLP sublayer, or attention head sublayer, b) use these activations train neural network, sparse autoencoder whose weights form a feature dictionary; c) interpret the resulting features with techniques such as OpenAl's autointerpretabiliiy scores (meanings here are illustrative only).
high sparsity, interference between non-orthogonal features prevents any performance gain from superposition. This suggests that we may be able to recover the network's features by finding a set of directions in activation space such that each activation vector can be reconstructed from a sparse linear combinations of these directions. This is equivalent to the well-known problem of sparse dictionary learning (Olshausen & Field 1997p.
Building on Sharkey et al. 2023)), ) we train sparse autoencoders to leam these sets of directions. Our approach also similar to Yun
(202I). who apply sparse dictionary learning all residual stream layers in language model simultaneously. Our method is summarised in Figure and described in Section 2
We then use several techniques to verify that our learned features represent semantically mean- ingful decomposition of the activation space. First, we show that our features are on average more interpretable than neurons and other matrix decomposition techniques, as measured by autointer- pretability scores (Section 3 (Bills et al. 2023). Next, we show that we are able to pinpoint the features used for set task studies on a small number of features, showing that they are not only monosemantic but also have predictable effects the model outputs, and can be used for fine-grained circuit detection. (Section 5).
precisely than other methods (Section 4. Finally,
2
TAKING FEATURES OUT OF SUPERPOSITION WITH SPARSE DICTIONARY LEARNING
To take network features out of superposition, we employ techniques from sparse dictionary learn- ing (Olshausen & Field 1997: Lee al.) (006). Suppose that each of a given set of vectors {x; Fi= C Rat is composed of sparse linear combination of unknown vectors {g,}"* 1y: C Rdd
Le. X, Σ, di,yg where a; is sparse vector. In case, the data vectors {x4 [x are internal activations
language model, such as Pythia-70M (Biderman et al. 2023), and {g/} are un- known, ground truth network features. We would like lear dictionary of vectors, called dictionary features, (fr Aiai that gj fg.
jk=1 C ddd where for any network feature gj there exists a dictionary feature fş such
To learn the dictionary, we train an autoencoder with a sparsity penalty term on its hidden activations. The autoencoder is neural network with single hidden layer of size daia Rd,,. where dn is the dimension of the language model internal activation vector and R is hyperparameter that controls the ratio of the feature dictionary size the model dimension. We use the ReLU activation function in the hidden layer (Fukushima, 1975). We also use tied weights for our neural network, meaning the weight matrices of the encoder and decoder are transposes of each other Thus, on
We mainly study residual streams in Pythia-70M and Pythia 410-M, for which the residual streams of size du = 512 and de = 1024, respectively (Biderman et al. (2023)
We use tied weights because (a) they encode our expectation that the directions which detect and define the feature should be the
highly similar, (b) they halve the memory cost of the model, and (c) they remove
2

---

Published as a conference paper at ICLR 2024
input vector {x;), network produces the output &, given by
ReLU(Mx + b)
(I)
x
M
(2)
du
Σ cffi ==
(3)
where M Rdaad,, and b Rdu are our learned parameters, and M is normalised row-wisd Our parameter matrix M is our feature dictionary, consisting of daid rows of dictionary features f. The output & is meant be reconstruction of the original vector x, and the hidden layer consists of the coefficients we use in our reconstruction of x.
Our autoencoder is trained minimise the loss function
X 8 dim X
((x)
a||c|| Sparsity loss
Reconstruction loss.
where is hyperparameter controlling the sparsity of the reconstruction, 4m is the width of the original activation. The é loss term on с encourages our reconstruction be a sparse linear combi- nation of the dictionary features. It can be shown empirically (Sharkey et al.) 2023) and theoretically (Wright & Ma 2022) that reconstruction with an e penalty can recover the ground-truth features that generated the data.
0.60
Layer
Layer
Layer
Ratio
Fraction variance unexplained
Ratio
0.10
Rabio Rx52
0.80
Layer
Layer 4
Layer
0.10
0.01
Average number of active features
Figure 2: The tradeoff between the average number of active features and the fraction of the variance that is unexplained, as the e coefficient is varied. Model is Pythia70M. Black dot represents the R = 2, 0.00086 point used for autointerpretation.
ambiguity about whether the learned direction should be interpreted the encoder decoder direction. They do not reduce performance when training on residual stream data but have observed some reductions performance when using MI.P data.
NNormalisation of the rows (dictionary features) prevents the model from reducing the sparsity loss term ||c|]] by increasing the size of the feature vectors in M.

---

Published as a conference paper at ICLR 2024
Feature
Description (Generated by GPT-4)
Interpretability Score
1-0000
parts of individual names, especially last names.
0.33
1-0001
actions performed by a subject or object.
-0.11
1-0002
instances of the letter "W" and words beginning with 'w'.
0.55
1-0003
the number '5' and also records moderate to low activa- tion for personal names and some nouns.
0.57
1-0004
legal terms and court case references.
0.19
Table 1: Results of autointerpretation on the first five features found in the layer residual stream, with R = 2, = 0.00086 on Pythia70m. Autointerpretation produces a description of what the feature means and score for how well that description predicts other activations.
INTERPRETING DICTIONARY FEATURES
3.1 INTERPRETABILIE AT SCALE
Having learned set of dictionary features, we want to understand whether our learned features dis- play reduced polysemanticity, and are therefore more interpretable. To do this in scalable manner, we require a metric to measure how interpretable a dictionary feature is. We use the automated approach introduced in Bills et al. 2023) because it scales well to measuring interpretability on the thousands of dictionary features our autoencoders learn. In summary. the autointerprctability proce- dure takes samples of text where the dictionary feature activates, asks language model to write heman-readable interpretation of the dictionary feature, and then prompts the language model to use this description to predict the dictionary feature's activation on other samples of text. The correlation between the model's predicted activations and the actual activations is that feature's 's interpretability score. See Appendix A and Bills al. (2023) for further details.
We show descriptions and top-and-random scores for five dictionary features from the layer resid- ual stream Table [ dictionary.
The features shown are the first five under the (arbitrary) ordering in the
3.2 SPARSE DICTIONARY FEATURES ARE MORE INTERPRETABLE THAN BASELINES
We assess our interpretability scores against variety of alternative methods for finding dictionaries of features in language models. In particular, we compare interpretability scores on our dictionary features to those produced by a) the default basis, b) random directions, с) Principal Component Analysis (PCA), and d) Independent Component Analysis (ICA). For the random directions and for the default basis in the residual stream, we replace negative activations with zeros so that all feature activations are nonnegative
Figure Bsowss shows that our dictionary features are far more interpretable by this measure than dictionary features found by comparable techniques. We find that the strength of this effect declines as move through the model, being comparable to ICA in layer 4 and showing minimal improvement in the final layer.
This could be result of our use of a consistent a = 0.00086, R = 2 in our automatic interpretation results, which seen in Figure2led to higher number of average active features in the later layers. However, it may also indicate that sparse autoencoders work less well later layers but also may be connected the difficulties of automatic interpretation, both because by building on earlier layers, later features may be more complex, and because they are often best explained by their effect on the output Bills al. (2023) showed that GPT-4 is able to generate explanations that are very close the average quality of the human- human-generann generated explanations given similar data. However, they also showed that current LLMS are limited in the kinds of patterns that they can find, sometimes struggling to find patterns that center around next previous tokens rather than the current token, and in the current protocol are unable to verify outputs by looking at changes in output or other data.
'For PCA we use an online estimation approach and run the decomposition on the same quantity of data we used for training the autoencoders. For ICA, due to the slower convergence times, approximately million activations for the residual stream and Im activations for the ML.ss
only 2GB of data,
4

---

Published as a conference paper at ICLR 2024
Sparse Coding
0.3
Identity ReLU
PCA
Random
automated interpretabilyy
0.1
0.0
1
3
Layer
Figure 3: Average top-and-randon autointerpretabiliiy score of our learned directions the residual stream, compared to number of baselines, using 150 features each. Error bars show 95% confi- dence intervals around means. The feature dictionaries used here were trained for 10 epochs using .00086 and R on Pythia 70M.
We do show, in Section 5 method to see feature's 's causal effect on the output logits by hand, but we currently do not send this information to the language model for hypothesis generation. The case studies section also demonstrates closing parenthesis dictionary feature, showing that these final layer features can give insight into the model's workings.
See Appendix C for fuller exploration of different learned dictionaries through the lens of auto- matic interpretability, looking at both the MLPs and the residual stream.
4 IDENTIFYING CAUSALLY-MMPORTANN DICTIONARY FEATURES FOR INDIRECT OBJECT IDENTIFICATION
In this section, we quantify whether our learned dictionary features localise specific model be- haviour more tightly than the PCA decomposition of the model's s activations. We do this via acti- vation patching. a form of causal mediation analysis (Vig et al.) (2020). through which we edit the model's internal activations along the directions indicated by our dictionary features and measure the changes to the model's outputs. We find that our dictionary features require fewer patches to reach a given level of KL divergence on the task studied than comparable decompositions (Figure
Specifically, we study model behaviour on the Indirect Object Identification (IOI) task (Wang al, 2022). in which the model completes sentences like "Then, Alice and Bob went to the store. Alice gave a snack to behaviour, which in particular has been widely explored through causal mediation analysis (Wang et al. 2022) (Conmy et al., 2023) and it captures simple model behaviour. Recall that the training of our feature dictionaries does emphasize any particular task.
This task was chosen because captures a simple, previously-studee
model
4.1 ADAPTING ACTIVATION PATCHING TO DICTIONARY FEATURES
In our experiment, we run the model on counterfactual target sentence, which a variant of the base IOI sentence with the indirect object changed (e.g., with "Bob" replaced by "Vanessa") "); save the encoded activations of our dictionary features; and use the saved activations to edit the model's residual stream when the base sentence.
5

---

Published as a conference paper at ICLR 2024
In particular, perform the following procedure. Fix layer of the model intervene Run the model on the target sentence, saving the model output logits y and the encoded features Ci,..., Cj of that layer at each of the k tokens. Then, the model on the base sentence up through the intervention layer, compute the encoded features cı...., C2 at each token, and at each position replace the residual stream vector x; with the patched vector
x = x Σ (cis Cc.)-f
-
JEF
where is the subset of the features which we intervene on (we describe the selection process for F later in this section). Let z denote the output logits of the model when you finish applying the patched residual stream xi.... x Finally, compute the KL divergence [ (zll), which measures how close the patched model's predictions are the target's. We compare these interventions to equivalent interventions using principal components found as in Section 3.2
To select the feature subset F, we use the Automated Circuit Discovery (ACDC) algorithm of Conmy et al.] (2023). In particular, we use their Algorithm 4.1 on our features, treating them as a flat computational graph in which every feature contributes an independent change to DrL. output metric, described above and averaged over a test set of 50 IOI data points. The result is an ordering on the features so that patching the next feature usually results smaller D k loss than each previous feature. Then our feature subsets F are the first k features under this ordering. We applied ACDC separately on each decomposition.
4.2 PRECISE LOCALISATION OF IOI DICTIONARY FEATURES
We show in Figure 4 that sparse feature dictionaries allow the amount of model editing, as measured by KL divergence from the target, in fewer patches (Left) and with smaller edit magnitude (Right) than the PCA decomposition. We also show that this does not happen if we train sparse dictionary (a 0). However, dictionaries with a larger sparsity coefficient a have lower overall reconstruction accuracy which appears in Figure 4 as a larger minimum KL divergence. In Figure 4 we consider interventions on layer 11 of the residual stream, and we plot interventions on other layers Appendix F
Dict, d- Rod
Ro-d
Dict. 4=0, Rodd
Nonnog, PCA
1.5
Lovergenn
0.5
300
0.5
2.5
Magnitade
Figure 4: (Left) Number of features patched vs KL divergence from target, using various residual stream decompositions. We find that patching relatively small number of dictionary features is more effective than patching PCA components and features from the non-sparse = 0 dictionary. (Right) Mean edit magnitude vs KI. divergence from target we increase the number of patched features. We find that our sparse dictionaries improve the Pareto frontier of edit magnitude vs thoroughness of editing. In both figures, the feature dictionaries were trained the first 10,000 elements of the Pile (Gao et al. 2020) (approximately 7 million activations) using the indicated a values and R 4, layer 11 of 1ytthia410MM (see Appendix for results other layers).

---

Published as a conference paper at ICLR 2024
5 CASE STUDIES
In this section, we investigate individual dictionary features, highlighting several that appear to cor- respond to a single auman-understandbbl explanation (i.e., that are monosemantic). We perform three analyses of our dictionary features to determine their semantic meanings: (1) Input: We iden- tify which tokens activate the dictionary feature and in which contexts, (2) Outpul: We determine how ablating the feature changes the output logits of the model, and (3) Intermediate features: We identify the dictionary features in previous layers that cause the analysed feature to activate.
5.1 INPUT: DICTIONARY FEATURES ARE HIGHLY MONOSEMANTIC
We first analyse our dictionary directions by checking what text causes them to activate. An ide- alised monosemantic dictionary feature will only activate on text corresponding single human- understandable concept, whereas polysemantic dictionary feature might activate in unrelated con- texts.
Input Token Activations
Output Logits Difference
700
340
600
120
100
100
0.00
0.61
123
184
2.46
5.07
2.09
4.30
492
533
6.5
0.00 0.50
1.72
2.50
3.44
4.30
5.10
602
685 7.4 360
Activations
Logit Difference
Figure 5: Histogram of token counts for dictionary feature 556 in layer 4 of Pythia-70M-dedupp (Left) For all datapoints that activate the feature, we show the count of each token in each activation range. The majority of activations are apostrophes, particularly for higher activations. Notably the lower activating tokens are conceptually similar to apostrophes, such as other punctuation. (Right) We show which token predictions are suppressed by ablating the feature, as measured by the differ- ence in logits between the ablated and unablated model. We find that the token whose prediction decreases the most is the "s" token. Note that there are 12k logits negatively effected, but set threshold of 0.1 for visual clarity. The autoencoder hyperparameters used were R = 4, a = 0.0014.
To better illustrate the monosemanticity of certain dictionary features, we plot the histogram of activations across token activations. This technique only works for dictionary features that activate for a small set of tokens. We find dictionary features that only activate on apostrophes (Figure F) periods; the token the"; and newline characters. The apostrophe feature in Figure 5 stands in contrast to the default basis for residual stream, where the dimension that most represents an apostrophe is displayed in Figure TO in Appendix D.I this dimension is polysemantic since it represents different information at different activation ranges.
Although the dictionary feature discussed in the previous section activates only for apostrophes, does not activate on all apostrophes. This can be seen in Figures 13 and [4 in Appendix D.2, showing two other apostrophe-accivaaing dictionary features, but for different contexts (such as "[I/We/They]'I" and "/don/won/wouldn)) tionary features be found in Appendix D.3
Details for how we searched and selected for dic-
5.2 OUTPUT: DICTIONARY FEATURES HAVE INTUITIVE EFFECTS ON THE LOGITS
In addition looking which tokens activate the dictionary feature, we investigate how dictionary features affect the model's output predictions for the next token by ablating the feature from the residual stream If our dictionary feature is interpretable, subtracting its value from the residual
SSpecifically we less-than-rank-one ahlation, where lower the activation vector in the direction of the feature only up the point where the feature is no longer active.

---

Published as a conference paper at ICLR 2024
stream should have logical effect the predictions of the next token. We see in Figure (Right) that the effect of removing the apostrophe feature mainly reduces the logit for the following "s". This matches what would expect from dictionary feature that detects apostrophes and is used by the model to predict the "s" token that would appear immediately after the apostrophe in possessives and contractions like "let's".
5.3 INTERMEDIATE FEATURES: DICTIONARY FEATURES ALLOW AUTOMATIC CIRCUIT DETECTION
We can also understand dictionary features in relation to the upstream and downstream dictionary features: given a dictionary feature, which dictionary features in previous layers cause it to activate, and which dictionary features later layers does cause to activate?
To automatically detect the relevant dictionary features, choose target dictionary feature such as layer 5's feature for tokens parentheses which predicts closing parentheses (Figure (6. For this target dictionary feature, we find its maximum activation M across our dataset, then sample 20 contexts that cause the target feature to activate the range [M//2, M.. For each dictionary feature in the previous layer, rerun the model while ablating this feature and sort the previous-layer features by how much their ablation decreased the target feature. desired, we can then recursively apply this technique the dictionary features in the previous layer with large impact. The results of this process form causal tree, such as Figure
Being the last layer, layer 5's role is output directions that directly correspond to tokens in the embedding matrix. In fact, when we unembed feature 52027, the top-tokens are all closing parenthe- ses variations. Intuitively, previous layers will detect situations that precede closing parentheses, such as dates, acronyms, and phrases.
O.
0. 0,28114
D_1697
II 11
0_141 Arrmnymm
e) 2.24 Autonyms
1.197 Singln
' 1906 coiltog
4:5 Open brackers or vatherca tolooss Espocially first position
1,736 Appontmos after Pradhcts
L9 Acronyms to wllas (
++
partscularly aiter mom bers. paresilbesia
AJ words la paronthocce. Especielly in first position Predicts
Fredicts tokeus to pasculhenis
2501
2628 aaafee dirense namess acroryym
Acronyms aiter E. acronymm
280 Vario clasing pareallessa. ospocaally 1I
Takem & yranctuacion
3. 1244
1086 Preducta
1,1025 Acronym for Prodicts Y
2.297 of
Procicts
2688 Varioue after Predictt "y
Prwdiche 'y
Varfour Precicts "I'
Figure 6: Circuit for the closing parenthesis dictionary feature, with human interpretations of each feature shown. Edge thickness indicates the strength of the causal effect between dictionary features successive residual stream layers, as measured by ablations. Many dictionary features across lay- ers have similar interpretations and often point in similar directions in activation space, as measured by cosine similarity. Model used was Prthia-7MM-dedupel with autoencoder hyperparameters R = 4, 0.0014.
DISCUSSION
6.1 RELATED WORK
Several previous works have attempted to decompose language representations into sparsely- activating features, varying both the representation studied and the technique used. Our approach,
8

---

Published as a conference paper at ICLR 2024
training neural network with a sparsity term in the loss function, is similar to the approaches in Faruqui et al. (2013): Subramanian et al.] (2018); Sharkey et al. (2023). In other works, such as Yun et al. (2021}: Zhang al. (211)). the decomposition found via the FISTA algorithm, and Murphy et al. 2012) uses the Non-Negative Sparse Embeddings method. Of these works, |aruquui al. (2015); Subramanian et al. (20T8); /Zhang al. 22019;; Murphy et al. (2012) applied these lechniques to word embeddings, while only (Sharkey et al. (2023): Yun et al. (202I) found sparse decompositions of the activations of a language model. Many of these works, including Murphy iet al. (2012); Subramanian et al. (2018); ; Yun et al. (202T) also find improved interpretability of their features, as measured by techniques such as -suured judgements, the word intrusion detection test, and word-level polysemy disanbiguation, respectively.
The works most similar to ours are (Sharkey et al. ) which inspired this work, and SSbramanian et a (018). The latter use sparse autoencoders to learn their decomposition of word embeddings, though for their main results they use losses which train the learned features to approximate a sparse
binary unit, finding in preliminary experiments that this outperformed the use of an l penalty.
Other previous works have tried to encourage sparsity in neural networks via changes to the ar- chitecture training process. These approaches include altering the attention mechanism (Correia ct al., 2019). adding e' penalties to neuron activations (Kasioumis et al. 2021 Georgiadis 2019). pruning neurons (Frankle & Carbin 2018), and using the softmax function as the non- non-linearity in the MLP layers (Elhage et al. (202aa). However, training a state-of-the-art foundation model with these additional constraints is difficult (Elhage et al., 2022a), and improvements to interpretability are not always realized (Meister et al. 2021).
6.2 LIMITATIONS AND FUTURE WORK
The approach we present in this paper found interpretable directions, but depending on the choice of hyperparameters leaves a significant fraction of the model's 's variance unexplained (Figure Future work could seek to improve the Pareto frontier of sparsity and reconstruction accuracy by exploring alternative architectures for the autoencoder or incorporating information about the weights of the model or dictionary features found in adjacent layers into the training process. This approach could also be applied other components of transformer, such as the output of the MLP or attention sublayers, as our attempt to find sparse directions in the MLP layer met only mixed success (see Appendix
In Section we show that for the 1OT task, behaviour is dependent on relatively small number of features. We expect that, because our dictionary is trained in a task-agnostic way. these result will generalize to similar tasks and behaviours, but more work is needed to confirm this suspicion. If this property generalizes, we would have a set of features which allow for understanding many model behaviours using just a few features per behaviour. We would also like to trace the causal dependencies between features in different layers, with the overarching goal of providing a lens for viewing language models under which causal dependencies are sparse. This would hopefully be step towards the eventual goal of building an end-to-end understanding of how a model computes its outputs.
6.3 CONCLUSION
Sparse autoencoders are scalable, unsupervised approach to disentangling language model network features from superposition. Our approach requires only unlabelled model activations and uses or- ders of magnitude less compute than the training of the original models. We have demonstrated that the dictionary features we learn are more interpretable by autointerpretation. letting us pinpoint the features responsible for a given behaviour more finely, and are more monosemantic than comparable methods. This approach could facilitate the mapping of model circuits, targeted model editing, and a better understanding of model representations.
An ambitious dream in the field of interpretability is enumerative safety (Elhage et al. 2022b producing a human- understandable explanation of a model's computations in terms of a complete list of the model's 's features and thereby providing a guarantee that the model will not perform dangerous behaviours such as deception. We hope that the techniques we presented in this paper also provide step towards achieving this ambition.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We would like thank the OpenAl Researcher Access Program for their grant of model credits for the autointerpretation and CoreWeave for providing EleutherAI with the computing resources for this project. We also thank Nora Belrose, Arthur Conmy, Jake Mendel, and the OpenAl Automated Interpretability Team (Jeff Wu, Williiam Saunders, Steven Bills, Henk Tillman, and Daniel Mossing) for valuable discussions regarding the design various experiments. We thank Wes Gurnee, Adam Jermyn, Stella Biderman, Leo Gao, Curtis Huebner, Scott Emmons, and William Saunders for their feedback on earlier versions of this paper. Thanks to Delta Hessler for proofreading. AE, and LR are supported by the Long Term Future Fund. RH is supported by Open Philanthropy grant. HC was greatly helped by the MATS program, funded by Al Safety Support.
REFERENCES
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, llya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in lan- guage models. URL http:://opeaiipublcc blob. core. windows, net/neuron-expiee www.ammeecpiiiieepppprnnneee himl.(Dute uccessed: 14.05. 2023), 2023.
Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. Curve circuits. Distill, 2021. doi: 10.23915/distill.000444 circuits.
https//diisillllubb00///cceiei//
Arthur Conmy, Augustine N Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretability. arXiv preprint arXiv:2304. 14997, 2023.
Gonçalo M Correia, Vlad Niculae, and André FT Martins. Adaptively sparse transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language P Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP,, pp. 2174-2184, 2019.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer: Llm. int8 O): 8-bit matrix multiplication for transformers at scale. arXiv preprint 2rXiv:2208.07339 2022.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1, 2021.
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer EIShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hermandez, Amanda Askell, Ka- mal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli. Liane Lovitt, Zac Hatfield-Dodds, Jackson Kermion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Ka- davath, Josh Jacobson, Eli Tran-Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCan- dlish, Dario Amodei, and Christopher Olah. Softmax linear units. Transformer Circuits Thread, 2022a.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Haddield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposi- tion. arXiv preprint arXiv:2209. 10652, 2022b.
Nelson Elhage, Robert Lasenby, and Chris Olah. Privileged bases in the transformer residual stream, 2023, URL
http::/wwwaaa.a.t....o/////////a Accessed: 2023-08-07.
basis/indes html
Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah Smith. Sparse overcomplete word vector representations. arXiv preprint arXiv:1506.000 1500. 02004, 2015.
10

---

Published as a conference paper at ICLR 2024
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:180.033633 2018.
Kunihiko Fukushima. Cognitron: A self-organizing multilayered neural network. Biol. Cybern., 20 (3-4):2111336, sep 1975. ISSN 13440-200, doi: 10.1007//F00342633 URL httpp:////oi
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.
Georgios Georgiadis. Accelerating convolutional neural networks via activation map compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7085-7095, 2019.
Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks. arXiv preprint arXiv:230.. 12001, 2023.
Theodoros Kasioumis, Joe Townsend, and Hiroya Inakoshi. Elite backprop: Training sparse inter- pretable neurons. In NeSy, pp. 82 92-93,220 2021.
Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Ng. Efficient sparse coding algorithms. Advances in neural information processing systems, 19, 2006.
Clara Meister, Stefan Lazov, Isabelle Augenstein, and Ryan Cotterell. Is sparse attention more interpretable? In Annual Meeting of the Association for Computational Linguistics, 2021. URL
Brian Murphy, Partha Talukdar, and Tom Mitchell. Learning effective and interpretable semantic models using non-negative sparse embedding. Proceedings of COLING 20/2, pp. 1933-1950, 2012.
Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from deep learning perspective. arXiv preprint arXiv:2209.00626, 2022.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 503):000024000,, 2020.
Bruno A Olshausen and David J Field. Sparse coding with overcomplete basis set: A strategy employed by v1? Vision research, (1123)33311-3255 1997.
Qing Qu, Yuexiang Zhai, Xiao Li, Yuqian Zhang, and Zhihui Zhu. Analysis of the optimization landscapes for overcomplete representation learning. arXiv preprint 19X2v:112..02227. 2019.
Lee Sharkey, Dan Braun, and Beren Millidge. Taking features out of superposition with sparse autoencoders, 2023. URL http://wwww..aalonneoooooooo.ooo
(oggobbtplexxxxoo//inttee of st-suurppooiiiion Accessed: 2023-05-10.
Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick. and Eduard Hovy. Spine: Sparse interpretable neural embeddings. In Proceedings of the AAAI conference artifi- cial intelligence, volume 32, 2018.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 18:12888-12011 2020.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022.
John Wright and Yi Ma. High-dimensions data analysis with div-diimnsional models: Principles, computation, and applications. Cambridge University Press, 2022.
11

---

Published as a conference paper at ICLR 2024
Zeyu Yun, Yubei Chen, Bruno A Olshausen, and Yann LeCun. Transformer visualization via dictio- nary learning: contextualized embedding as a linear superposition of transformer factors. arXiv preprint ar arXiv:2103.15949, 2021.
Juexiao Zhang, Yubei Chen, Brian Cheung, and Bruno A Olshausen. Word embedding visualization via dictionary learning, arXiv preprint arXiv:1910.038. 2019.
A AUTOINTERPRETATION PROTOCOL
The autointerpretabbiiiyy process consists of five steps and yields both an interpretation and an au- tointerpretability score:
1. On each of the first 50,000 lines of measure the feature's activation on each token of this fragment. Feature activations are rescaled integer values between and 10.
take 64-token fragmen,, and
2. Take the 20 fragments with the top activation scores and pass 5 of these GPT-4, along with the rescaled per-token activations. Instruct GPT-4 to suggest an explanation for when the feature (or neuron) fires, resulting in an interpretation.
3. Use G to simulate the feature across another 5 highly activating fragments and randomly selected fragments (with non non-zero variation) by asking it to provide the per-token activations.
4. Compute the correlation of the simulated activations and the actual activations. This corre- lation is the autointerpretabiliiy score the feature. The texts chosen for scoring a feature can be random text fragments, fragments chosen for containing particularly high activa- tion that feature, or an even mixture of the two. We use a mixture of the two unless otherwise noted, also called 'top-random' scoring.
5. If, amongst the 50,000 fragments, there are fewer than 20 which contain non-zero variation in activation, then the feature skipped entirely.
Although the use of random fragments Step is ultimately preferable given large enough sample size, the small sample sizes of a total of 640 tokens used for analysis mean that random sample will likely not contain any highly activating examples for all but the most common features, making top-random scoring desirable alternative.
B SPARSE AUTOENCODER TRAINING
To train the sparse autoencoder described in Section 2 we use data from the Pile (Gao et al. 2020). large, public webtext corpus. We run the model that we want to interpret over this text while caching and saving the activations at particular layer. These activations then form dataset, which we use train the autoencoders. The autoencoders are trained with the Adam optimiser with a learning rate of 1e-3 and are trained on 5-50M activation vectors for 1-3 epochs, with larger dictionaries taking longer to converge. single training run using this quantity of data completes in under an hour on single A40 GPU.
When varying the hyperparameter a which controls the importance of sparsity loss term, we consistently find smooth tradeoff between the sparsity and accuracy of our autoencoder, as shown in Figure 2 The lack of 'bump' or 'knee' in these plots provides some evidence that there is not a single correct way to decompose activation spaces into sparse basis, though to confirm this would require many additional experiments.
While the process described in Bills et al (2033) uses GPT-4 for the simulation step. we use GPT-3.5. This is because the simulation protocol requires the model's logprohs for scoring, and OpenAl's public API for GPT-3.5 (but not GPT-4) supports returning logprohs.
12

---

Published as a conference paper at ICLR 2024
0.3
Mean automated interpretabligg score
0.2
1: sparse coding ratio = 0.5
2: sparse coding
3: sparse coding ratio 2.0
4: sparse coding ratio = 4.0
X 5: sparse coding ratio
6: sparse coding ratio 16.0
7: sparse coding ratio 32.0
2
4
5
Layer
Figure 7: Comparison of average interpretability scores across dictionary sizes. All dictionaries were trained on 20M activation vectors obtained by running Pythia-70M over the Pile with a = .00086.
C FURTHER AUTOINTERPRETATION RESULTS
C.1 INTERPRETARILIN CONSISTENT ACROSS DICTIONARY SIZES
We find that larger interpretability scores of our feature dictionaries are not limited to overcomplete dictionaries (where the ratio, R, of dictionary features to model dimensions even in dictionaries that are smaller than the underlying basis, as shown in Figure 7 These small dictionaries are able reconstruct the activation vectors less accurately, so with each feature being similarly interpretable, the larger dictionaries will be able to explain more of the overall variance.
1). but occurs
C.2 HIGH INTERPRETABILIH SCORES ARE NOT AN ARTEFACT OF TOP SCORING
possible concern is that the autointerpretabiliiy method described in Section 3 combines top acti- vating fragments (which are usually large) with random activations (which are usually small), mak- ing it relatively easy to identify activations. Following the lead of Bills et al. (2023), we control for this by recomputing the autointerpretation scores by modifying Step 3 using only randomly selected fragments. With large sample sizes, using random fragments should be the true test of our ability interpret potential feature. However, the features we are considering are heavy-tailed, so with limited sample sizes, we should expect random samples to underestimate the true correlation.
In Figure g we show autointerpretabiliiyy scores for fragments using only random fragments. Match- ing Bills et al. (2033), we find that random-only scores are significantly smaller than top- rapaddo scores, but also that our learned features still consistently outperform early layers. Since our learned features are more sparse than the baselines and thus, activate less given fragment, this is likely to underestimate the performance of sparse coding relative to baselines.
baselines, especially
An additional potential concern is that the structure of the autoencoders allows them to be sensitive to less than a full direction in the activation space, resulting in an unfair comparison. We show in Appendix G that this is not the source of the improved performance of sparse coding.
While the residual stream can usually be treated as a vector space with no privileged basis (a basis in which we would expect changes to be unusually meaningful, such the standard basis after linearity in an MLP), it has been noted that there is a tendency for transformers to store information in the residual stream basis et al. 2022), which is believed be caused by the Adam
13

---

Published as a conference paper at ICLR 2024
Sparse Coding
ICA
Identity ReLU
Mean automated interpretabligg score
PCA
Random
0.1
3
5
Layer
Figure 8: Random-only interpretability scores across each layer, a measure of how well the interpre- tation of the top activating cluster is able to explain the entire range of activations, a = 0.00086, R 2, Pythia70M.
optimiser saving gradients with finite precision in the residual basis Elhage et al. 2023). We do not find residual stream basis directions be any more interpretable than random directions.
C.3 INTERPRETING THE MLP SUBLAYER
Our approach of learning feature dictionary and interpreting the resulting features in principle, be applied to any set of internal activations of a language model, not just the residual stream. Ap- plying our approach the MLP sublayer of transformer resulted in mixed success. Our approach still finds many features that are more interpretable than the neurons. However, our architecture also learns many dead features, which never activate across the entire corpus. In some cases, there many dead features that the set of living features does not form an overcomplete basis. For example, in dictionary with twice many features neurons, less than half might be active enough to perform automatic interpretability. The exceptions to this are the early layers, where a large fraction of them are active.
For learning features in MLP layers, we find that we retain a larger number of features if we use different matrix for the encoder and decoder, so that Equations andd] become
ReLU
(5)
x
Mfc
(6)
We are currently working on methods to overcome this and find truly overcomplete bases in the middle and later MLP layers.
C.4 INTERPRETABILIT SCORES CORRELATE WITH KURTOSIS AND SKEW OF ACTIVATION
has been shown that the search for sparse, overcomplete dictionaries can be reformulated in terms of the search for directions that maximise the ft-norm (Qu et al. 2019)
We offer a test of the utility of this by analysing the correlation between interpretability and a number of properties of learned directions. We find that there correlation of 0.19 and 0.24 between the
14

---

Published as a conference paper at ICLR 2024
Top and annd-radddm scoring
Random-ork scoring
Sporse Coding
Sparso Coding
CCA
Iduntity
Identity PeLL
Mean anformtedt interretatiitty
Hoan 82
0.0
0.
Figure 9: Top-and-random and sondomm-only interpretability scores for across cach MLP layer, using an l coefficient a 3.2e 4 and dictionary size ratio R. = Pythia70M.
degree of positive skew and kurtosis respectively that feature activations have and their top-and- random interpretability scores, as shown Table 2
Moment
Correlation with top-random interpretability score
Mean
-0.09
Variance
0.02
Skew
0.20
Kurtosis
0.15
Table 2: Correlation of interpretability score with feature moments across residual stream results, all layers of Pythia70M, with dictionary size ratios R E (0.5, 1,2,48).
This also accords with the intuitive explanation that the degree of interference due to other active features will be roughly normally distributed by the central limit theorem. If this is the case, then features will be notable for their heavy-tailedness.
This also explains why Independent Component Analysis (ICA), which maximises the non- Gaussianity of the found components, is the best performing of the alternatives that we considered.
D QUALITATIVE FEATURE ANALYSIS
D.1 RESIDUAL STREAM BASIS
Figure 10 gives token activation histogram of the residual stream basis. Connecting this residual stream dimension to the apostrophe feature from Figure 5 this residual dimension was the 10th highest dimension read from the residual stream by feature")
D.2 EXAMPLES OF LEARNED FEATURES
Other features from Pythia-70M-dedupe (layer 4, R = 4, a 0.0014) are shown in Figures II and 14
D.3 FEATURE SEARCH DETAILS
We searched for the apostrophe feature using the sentence " don't know about that. It is now up to Dave"", and seeing which feature (or residual stream dimension) activates the most for the last apos- trophe token. The top activating feature in dictionary was outlier dimension feature (i.e., feature direction that mainly reads from an outlier dimension of the residual stream), the apostrophes after O (and predicted O' Brien, O' Donnell, O'Connor, O' d'clock, etc), then the apostrophe-precedin- s feature.
Thhe first 9 did have apostrophes in their top-activations like dimension 21.
15

---

Published as a conference paper at ICLR 2024
Input Token Activations: Residual Stream Basis
200000
and
150000
part
100000
number
[All Other Tokens]
50000
[AlI Tokens For Scale]
2500
2000
1500
1000
500
0.00
1.28
2.55
3.83
5.11
6.38
7.66
8.94
10.21 11.49 12.77
Activations
Figure 10: Histogram of token counts in the neuron basis (dimension 21 of layer 4 of Pythia-70M- deduped, positive direction). Although there large fraction of apostrophes in the upper acti- vation range, this only explains very small fraction of the variance for middle-to-lower activation ranges.
Input Token Activations
Output Logits Difference
40
you then
Indst cofine
wincc
20
tuless]
10-
0.00
0.77
1.53
2.30
3.07
3.84
4.50
5.37
6.14
5.0
7.67
0.00
0.27
0.54
1.08
1.35
1.52
1.99
2.16
243
2.0
Activations
Logit Difference
Figure 11: 'If" feature in coding contexts (Feature #1087 from Pythia-70M-dedupeer layer 4, with
autoencoder hyperparameters = 4, a = 0.0014).
For the residual basis dimension, searched for and min activating dimensions (since the residual stream can be both positive and negative), where the top two most positive dimensions were outlier dimensions, the top two negative dimensions were our displayed one and another outlier dimension, respectively.
D.4 FAILED INTERPRETABILITY METHODS
We attempted weight-based method going from the dictionary in layer 4 to the dictionary in layer 5 by multiplying a feature by the MLP and checking the cosine similarity with features in layer 5. There were meaningful connections. Additionally, it's unclear how to apply this to the Attention sublayer since we'd d need to see which position dimension the feature is in. expected this failed by going out of distribution.
16

---

Published as a conference paper at ICLR 2024
Input Token Activations
Output Logits Difference
1200
8C0
pesse
plaps
mn
pos too
600
pat
rekers]
takens]
13-
10
200
0.00
4.85 Activations
117
4.1:
Logit Difference
Figure 12: 'Dis' token-level feature showing bigrams, such as 'diSCLAIM', 'disclosed', 'disor- dered', etc. (Feature #1 128 from Pythia-70M-deduped layer 4, with autoencoder hyperparameters R = 4, a 0.0014).
Input Token Activations
Output Logits Difference
600
500
400
Cther Tokara]
[A] Other Toaars]
500
200
10
100
1:22
Activations
Logit Difference
Apostrophe feature in "I'll-like contexts (Feature #1243 from Pythia-70M-deduped layer 4, with autoencoder hyperparameters R = 4, a 0.0014).
Figure 13:
E NUMBER OF ACTIVE FEATURES
In Figure 15 we see that, for residual streams, we consistently learn dictionaries that least 4x overcomplete before some features start to drop out completely, with the correct hyperparameters. For MLP layers you see large numbers of dead features even with hyperparameter figures informed the selection of a 8.6c - 4 and
0. These 3.2e 4 that went into the graphs in Section 3 for the residual stream and MLP respectively. Due to the large part of the input space that never used due to the non-linearity, it is much easier for MLP dictionary features to become stuck at position where they hardly ever activate. In future we plan to reinitialise such 'dead fcatures' to ensure that we learn many useful dictionary features possible.
F EDITING 1OI BEHAVIOUR ON OTHER LAYERS
In Figure 16 we show results of the procedure in Section 4 across range of layers in Pythia-410M.
G TOP K COMPARISONS
As mentioned in Section the comparison directions learnt by sparse coding and those in the baselines are not perfectly even. This is because, for example, a PCA direction active to an entire half-space on one side of a hyperplane through the origin, whereas sparse coding feature activates on less than a full direction, being only on the far side origin. This is due to the bias applied before the activation, which is, in practice, always negative. To test whether this difference is responsible for the higher scores, we run a variant PCA and ICA in which have fixed number of directions, K, which can be active for any single datapoint. We
hyperplane that does not intersect the
17

---

Published as a conference paper at ICLR 2024
Input Token Activations
Output Logits Difference
700
50-
600
[el other Takars)
200
200
8.00
2.28
3.04
0.90
271
3.6.
Activations
Logit Difference
Figure 14: Apostrophe feature "don't" contexts. (Feature #101 from Pythia-77M-ddedupe layer 4, with autoencoder hyperparameters 4, a 0.0014).
Number of actlue features for each byer
The
Tostal
Aggho
11.00
Abfh
LL.ggM
L1 ABA
Apt
LL/DPs
Figure 15: The number of features that active, defined activating more than 10 times across 10M datapoints, changes with sparsity hyperparamter a and dictionary size ratio R.
set this K to be equal to the average number of active features for a sparse coding dictionary with ratio R = 1 and = 8.6e 4 trained on the layer in question. We compare the results in Figure 17
showing that this change does not explain more than a small fraction of the improvement scores.
18

---

Published as a conference paper at ICLR 2024
Layer 11
Layer 12
Layer
250
355
Figure 16: Divergence from target output against number of features patched and magnitude of edits for layers 3, 7, 11, 15, 19 and 23 of the residual stream of Pythia-410M. Pythia-410M has 24 layers, which we index 0, 1, ..., 23. Once again, we use with varying a parameter.
Top-and-random scoring
Sparse Coding
ICA
ICA Top-k
Identity ReLU
Mcan automated interpretabiiity
0.3
X PCA
PCA Top-k
0.2
0.1
0.0
2
5
Layer
Figure 17: Autointerpretation scores across layers for the residual stream, including top-K baselines
for ICA and PCA. As with Figure these use = 0.00086, R = 2 and Pythia70M
19


Reviews:
[{"review_id": "CLgs4HlIkg", "reviewer": "Reviewer_V2wZ", "summary": "Superposition refers to the hypothesis that neural model representations are in fact linear compositions of sparse features. This paper attempts to identify these sparse features using dictionary learning. More specifically, the authors train an autoencoder with sparsity penalties on language model representations of interest (as both input and output), and use the learned sparse encoding as a proxy to understand the original language model. The resulting sparse encodings are passed through an auto-interpretation model to identify meaningful features. Under the auto-interpretation evaluation, the learned sparse encoding by the proposed method beat several baselines.", "questions": "Any comments on how the proposed method scales with complexity of the underlying task / number of possible features? Is the approach bottlenecked by auto-interpretation method?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and easy to follow. The idea of using sparse autoencoder to tackle superposition makes intuitive sense, and it turns out that it works surprisingly well. The activation patching experiments validates the effectiveness of the proposed method beyond the auto-interpretation metrics, which emphasizes the method's practical value.", "weaknesses": "Using dictionary learning for superposition isn't particularly novel.\nAuto-interpretation score hasn't been shown to correlate with actual interpretability use cases yet.\nActivation patching experiment was only conducted on one relatively synthetic dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pUhTTQRaPx", "reviewer": "Reviewer_FVSC", "summary": "This paper proposes using sparse autoencoders to learn interpretable and monosemantic features from the internal activations of language models. The key idea is to reconstruct model activations as sparse linear combinations of latent directions, which aims to disentangle the model's features from superposition. The authors train autoencoders with an L1 sparsity penalty on various model layers and analyze the resulting feature dictionaries.\n* The learned dictionary features are shown to be more interpretable than baseline methods like PCA/ICA according to automated scores.\n* The features localize target model behaviors more precisely for a counterfactual evaluation task.\n* Case studies demonstrate some highly monosemantic features that influence model outputs in predictable ways.\n* The approach is scalable and unsupervised, requiring only unlabeled activations.", "questions": "How well do the features transfer across different models and architectures?\nCould you incorporate model weights or adjacent layer features to improve dictionary learning?\nWhat changes allow learning overcomplete MLP bases?\nHow well do the features generalize to unseen tasks? More thorough evaluation would be useful.\nThe authors claim the method is scalable, but have not demonstrated it on very large models. Experiments on models with billions of parameters could better support the scalability claims.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Applies sparse coding in a novel way for model interpretability. The approach is intuitive and theoretically motivated.\nProvides solid evidence that the learned features are more interpretable and monosemantic.\nDemonstrates the technique can pinpoint features for analyzing model behaviors.\nCase studies show highly intuitive individual features.\nThe method is scalable and unsupervised.", "weaknesses": "The reconstruction loss is not fully minimized, indicating the dictionaries do not capture all information.\nThe method works less well for later model layers and MLPs compared to early residual layers.\nMore analysis needed on:\n1) generalizability of behaviors across tasks,\n2) sparsity of dependencies between features.\n\nLimited comparison to other interpretability and disentanglement techniques.  For example, beta-VAE, infoGAN, FactorVAE had similar disenanglement goals.  And Michaud 2023 \"The Quantization Model of Neural Scaling\" suggested a spectral clustering approach for identifying \"monogenic\" signals in large models. The paper would be strengthened if it provided a more complete comparison to previous disentanglement approaches.\n\nThe authors could mention the previous works in the field of sparse autoencoding and dictionary learning for word embeddings. Subramanian et al. [1] similarly found linear factors for word embeddings, in this case using a sparse autoencoder. Zhang et al. [2] solved a similar problem using methods from dictionary learning. Their method discovers elementary structures beyond existing word vectors.\n\n[1] Spine: Sparse interpretable neural embeddings Subramanian, A., Pruthi, D., Jhamtani, H., Berg-Kirkpatrick, T. and Hovy, E., 2018. Proceedings of the AAAI Conference on Artificial Intelligence, Vol 32(1).\n\n[2] Word embedding visualization via dictionary learning Zhang, J., Chen, Y., Cheung, B. and Olshausen, B.A., 2019. arXiv preprint arXiv:1910.03833.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XDBQaVVF3J", "reviewer": "Reviewer_oRDV", "summary": "This paper is addressing the challenge of polysemanticity in DNNs. In DNNs context neurons appear to activate in multiple, semantically distinct contexts. This paper is trying to address the challenge of disambiguating neurons. They suggest sparse auto-encoders for this challenge.", "questions": "N/A", "limitations": "", "rating": 1, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "+ Very interesting research question", "weaknesses": "- Overly complex approach, where the auto-encoder will likely add noise to the results\n- No real baseline comparison \n- Experimentation is weak \n- Results are difficult to interpreter and would need to be better presented or colored in the context", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lC2LAtV2HG", "reviewer": "Reviewer_k1em", "summary": "This paper presents a way to make the individual features of Large Language Models more interpretable by learning simple autoencoders with activation sparsity. They demonstrate that the learned features are more interpretable than original or simple methods like PCA via several ways, such as automated interpretability, example neurons and helpfulness in detecting circuit components.", "questions": "- Figure 2: Is identity ReLU the default basis?\n- Section 3.2: Why is ReLU applied to the default and random basis? It is said its applied to make activations nonnegative, but why is this necessary? The actual model also uses negative activations right?\n- What is the intended way to use these learned features? Adding the autoencoder changes the network behavior, and increases computational cost. Are you supposed to use the expanded network in practice to make explainable decisions, or is this better thought as an approximation to explain decisions made by the unmodified network.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "Tackles a very important issues with mostly sound methodology and original results. The improvement in interpretability is quite noticeable, and evaluation is diverse.\nFigure 4 is a good clear explanation of 1 feature, and the comparison to the relevant feature in standard residual stream is very useful. \nFigure 5 is also interesting and a good example of the potential usecases of this method.", "weaknesses": "Missing a lot of important basic information.\n - What is R? It is an important part of many text captions such as Figure 2 and Figure 3 but never explained in main text. Only explained in Appendix C.\n- Almost all the results do not mention the details of how they were achieved. In particular, table 1, and figures 2, 4, 5 don't mention the model used at all, and most figures do not mention R and alpha used to train the in question.\n\nIn general the main text needs more discussion of the basic methods and results like reconstructions loss, some of the content from Appendix B and C should be moved to main text.\n\nThe high reconstruction loss seems like a problem for the usefulness/faithfulness of this method, and I would like more discussion on it, such as the i.e. large increase in perplexity discussed in section 6.2. What R and Alpha were used for this result? How does this change as a function of different parameters like R and alpha or different layers?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1k43kwisPy", "reviewer": "Reviewer_apJf", "summary": "The authors train a sparse autoencoder to find interpretable hidden units in an LLM. The sparse autoencoder linearly transforms the learned representation in a layer of the LLM and then applies a rectifying nonlinearity. The resulting code is regularized by L1 penalty so as to prefer sparse representations and it is trained to do well at reconstructing the LLM representation through a linear map. Using previously published methods to quantify the level of interpretability of a feature with a score,  the authors show that some of the resulting sparse features (those with the highest interpretability score) are more interpretable than those obtained with ICA, PCA and random projections. Other experiments study the effect of perturbing one of these sparse features on the output as well as whether some features are associated with a single word (monosemantic features).", "questions": "Please try to address the questions I raised in the previous section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Interpretability is an important subject for improving the safety of frontier AI systems, so this work belongs in a socially important research niche. The main contribution of showing greater interpretability with sparse features derived from the learned LLM layers is useful, although similar sparse representations have already been shown to be useful from an interpretability point of view for neural networks.", "weaknesses": "The paper claims in the abstract and conclusion that the results show *greater* monosemanticity than other methods but I do not see such comparisons in the paper (it should be in section 5.1). Either I missed something or one of the main claims is not supported.\n\nThe authors used linear encoders (followed by a ReLU) while it is well known that this is suboptimal (the optimal encoder needs to be much more nonlinear). A useful contribution would have been to evaluate and compare different types of encoding mechanisms for obtain a sparse representation. A simple variant is to introduce hidden layers (and depth) in the encoder. There are also optimization-based encoders that work very well and are well-known for sparse coding.\n\nMost of the pieces in this paper (the sparse autoencoder and the interpretability score) are pre-existing, and so is the idea and demonstration that sparse features yield greater interpretability. This makes the overall level of originality and significance of the paper pretty low.\n\nIn the first paragraph of sec. 5.1, the authors talk about 'real-word features'. Please clarify what that means. If it is a \"human-understable concept\", one crucial problem is that such a concept (except for single-token cases studied in the paper) can be represented in many ways as a sequence of words, making any practical extension of the proposed idea (in sec 5.1) not obvious at all (or maybe not even feasible).\n\nIn section 5.3, it would be good to add evaluations using not just the top hidden layer (which has a very special semantics because it lives in the space of token embeddings) but also lower hidden layers\n\nIn section 6 about limitations, I would venture that two reasons for the poor reconstruction could be that (a) the proposed linear encoder does not have enough expressive power or (b) there exists no sparse linear basis to explain the LLM layers or (c) there is no compact natural language description for the sparse features. \n\nFinally, the whole study is only looking at the *most interpretable* features in the lot, but in order to obtain any kind of safety guarantee, one will need to make sense of *all* the information in the LLM layers. We seem very far from that, which makes me doubt that the whole approach will ever be providing sufficient safety reassurances one day.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)