Finding Neurons in a Haystack: Case Studies with Sparse Probing
Authors: Wes Gurnee
Neel Nanda
Matthew Pauly
Katherine Harvey
Dmitrii Troitskii
Dimitris Bertsimas
Venue: Accepted by TMLR ()

Abstract: Despite rapid adoption and deployment of large language models (LLMs), the internal computations of these models remain opaque and poorly understood. In this work, we seek to understand how high-level human-interpretable features are represented within the internal neuron activations of LLMs. We train $k$-sparse linear classifiers (probes) on these internal activations to predict the presence of features in the input; by varying the value of $k$ we study the sparsity of learned representations and how this varies with model scale. With $k=1$, we localize individual neurons that are highly relevant for a particular feature and perform a number of case studies to illustrate general properties of LLMs.  In particular, we show that early layers make use of sparse combinations of neurons to represent many features in superposition, that middle layers have seemingly dedicated neurons to represent higher-level contextual features, and that increasing scale causes representational sparsity to increase on average, but there are multiple types of scaling dynamics. 
In all, we probe for over 100 unique features comprising 10 different categories in 7 different models spanning 70 million to 6.9 billion parameters.

Published in Transactions on Machine Learning Research (11/2023)
Finding Neurons in a Haystack: Case Studies with Sparse Probing
Wes Gurnee
wesg edu
Massachusetts Institute of Technology
Neel Nanda Independent
nechnanda27@gmaii. com
Matthew Pauly
Horvard University
Katherine Harvey
Abarvey@ccoleege.aarorr
Horvard University
Dmitrii Troitskii Northeastern University
Dimitris Bertsimas Massachusetts Institute Technology
dbertsim@@mie edu
Reviewed on OpenReview: https: openrevvee.. net/ forum? id= JRRJMM
Abstract
Despite rapid adoption and deployment of large language models (LLMs), the internal computations of these models remain opaque and poorly understood. In this work, seek to understand how high-level human-interppettbble features are represented within the internal neuron activations of LLMs. We train k-sparse linear classifiers (probes) on these internal activations to predict the presence of features in the input; by varying the value of we study the sparsity of learned representations and how this varies with model scale. With k 1, we Jocalize individual neurons that are highly relevant for a particular feature and perform number of case studies to illustrate general properties of LLMs. In particular, we show that early layers make use of sparse combinations of neurons to represent many features in superposition, that middle layers have seemingly dedicated neurons to represent higher-level contextual features, and that increasing scale causes representational sparsity to increase on average, but there are multiple types of scaling dynamics. In all, we probe for over 100 unique features comprising 10 different categories in 7 different models spanning 70 million 6.9 billion paramcters.
1 Introduction
Neural networks are often conceptualized being flexible "feature extractors" that learn iteratively develop and refine suitable representations from raw inputs (Bengio et al. 2013 Donahue et al. 2014). This begs the question: what features are being represented, and how? Probing is standard technique used to study if and where a neural network represents a specific feature by training a simple classifier (a probe) on the internal activations of a model to predict a property of the input (Belinkov) 2022 (e.g., classifying the tense of a verb based on the activations of a specific layer). Bocause models are paramcterized as scries of dense matrix multiplications and elementwise nonlinearities, natural intuition is that features are represented as linear directions in activation space (Elhage et. al. 2022b) and are iteratively combined to synthesize increasingly

---

Published in Transactions on Machine Learning Research (11/2023)
taken Fronch not
0.16-
0.12
0.10
0.00
pic octvattonn
0.02
prmme
0.00
rescarc
Unkted Sutio
Mile
pro wnttraai
Tuctors
Vigrama
Dastnct'
Vardf
(a) A monosemantic French ncuron
(b) A polyscmantic neuron activating on six unrelated r-grams
Neurons that respond to single features (left) can be understood independently, in contrast to polysemantic neurons (right) that activate for many unrelated features (in this case, the occurrence of specific multi-token spans).
Figure 1:
abstract features using linear combinations of previously computed features (Olah et al. 2020b). To zoom in on these dynamics in modern transformer language models, we propose sparse probing, where we constrain the probing classifier to use at most k neurons in its prediction; we probe for over 100 features to precisely localize relevant neurons and elucidate broader principles of how to study and interpret the rich structure within LLMs.
In the simplest case, one might hope there exists a 1:1 correspondence between features of the input and neurons in network-thie for the correct. feature definition, 1-sparse probe would be sufficient. Although the literature contains many examples of such seemingly monosemantic neurons (Radford et al. 2017: Bau
ct al. 2020 Olah et al.] 2020b Goh et al. 2021 Cammarata et al.) 2021 Elhage et al.) 2022a). an obvious problem arises when a network has to represent more features than it has neurons. To accomplish this, a model must employ some form of compression to embed n features in d < n dimensions. While this superposition of features enables more representational power, it also causes loss-increasing "interference" between non-orthogonal features (Elhage et al., 2022b)). Recent works in toy models demonstrate that this tension manifests in spectrum of representations: dedicated dimensions neurons for the most prevalent. and important features with incrcasing levels of superposition -anddttioonnnnnn hence decrcasing levels of sparsity for the long (ail of rarer or less important features (Elhage el al. 2022b Scherlis el al. 2022).
As with any conceptual insight, these results raise more questions than they answer: To what extent do these observations transfer to full scale language modcls? What kinds of fcatures do or do not appear in superposition? AL what scale? How do we reliably find and verify such (feature, neuron) pairs? We leverage sparse probing systematically study such questions, finding clean examples of both monosemantic neurons and superposition "in the wild." Better understanding- and potentially resolving superposition is the critical path to ambitious full-model interpretability because logical consequence of superposition is the presence of polysemantic neurons that activate for a large collection of seemingly unrelated stimuli (Figure) D (Elhage et al. 20022a] Mu & Andreas (020)). Such tangled representations undermine our ability to decompose networks into independently meaningful and composable components, thwarting existing approaches to reverse-engineer networks (Olah et al. 2020b).
In addition to being especially well suited studying superposition, our sparse probing methodology addresses several shortcomings identified in the probing literature (Belinkov) 2022 Antverg & Belinkov 2021 Hewitt &
Liang 2019
Ravichander et al. 2020). By leveraging recent advances in optimal sparse prediction Bertsimas
ct al. 2020 2021), we are able to prove optimality of the k-sparse feature selection subproblem (for small
k), addressing the conflation of ranking quality and classification quality raised in (Antverg & Belinkov 2021). Second, by employing sparsity as an inductive bias, our probes maintain a strong simplicity prior and are more capable of precise localization of important neurons. This precision enables a more granular level of analysis illustrated throughout our case studies. Finally, the lack of capacity inhibits the probe from memorizing correlation patterns associated with the feature of interest. (Hewitt & Liang 2019), offering
2

---

Published in Transactions on Machine Learning Research (11/2023)
more reliable signal of whether this feature is explicitly represented and used downstream Ravichander et al. 2020
In the first part of the paper, we outline several variants of sparse probing, discuss the various subtleties of applying sparse probing, and run a large number of probing experiments. In particular, we probe for over 100 unique features comprising 10 different categories in 7 different models spanning 2 orders of magnit in parameter count (up 6.9 billion). The majority of the paper then focuses on zooming in (Olah al., 2020bb on specific examples of general phenomena in series of more detailed case studies to demonstrate:
There is tremendous amount of interpretable structure within the neurons of LLMs, and sparse probing is an effective methodology to locate such neurons (even in superposition), but requires careful use and follow-up analysis to draw rigorous conclusions.
Many early layer neurons in superposition, where features are represented as sparse linear combinations of polysemantic neurons, each of which activates for a large collection of unrelated n-grams and local patterns. Moreover, weight statistics combined with insights from toy models suggest that the first 25% of fully connected layers employ substantially more superposition than the rest.
Higher-level contextual and linguistic features (e.g., is w-pptho__coodeepp are seemingly encoded by monosemantic neurons, predominantly in middle layers, though conclusive statements about monose- manticity remain methodooogically out of reach.
As models increase in size, representation sparsity increases on average, but different features obey different dynamies: some features with dedicated neurons emerge with scale, others split into finer grained features with scale, and many remain unchanged or appear somewhat randomly.
2
Related Work
Probing Originally introduced by (Alain & Bengio 2016), probing is a standard technique used to determine
whether models represent specific features or concepts (Belinkov] 2022). For language models in particular, there exist many probing studies showing the rich linguistic representations learned by models (Conneau)
et al. 2018 Tenney et al. 201 9a b). contributing to the broader field of "BERTology" (Rogers et al. 2020
Particularly relevant to our work are investigations on the role of individual neurons (Bau et al.) 2020 Suau
et al. 2020
Sajjad et al. 2022
Wang et al. 2022b) and the identification of sparsely represented features
Durrani et al. 2020 Dalvi et al.
2019h
While probing in general has number of limitations (Hewitt &
Liang 2019
Donnelly & Roogiest 2019
Ravichander et al. 2020 Maudslay & Cotterell 2021 Elazar et al.
2021), our work seeks to address, at least partially, the shortcomings most associated with probing individual
neurons (Antverg & Belinkov 2021). Methodologically, sparse probing is situated within the broader category
of probing methods (Hennigen et al.] 2020 Voita & Titov 2020
Pimentel al.
2020
Cao al. 2021]
which itself is just one paradigın among other localization techniques (Bau al.
2018
Dhamdhere al.,
2018
Amjad et al. 2021 De Cao et al. 2021
Durrani et. al. 2022
Goldowsky-Dill et al.
2023
Mechanistic Interpretability Philosophically, our work is motivated by the growing field mechanistic
interpretability (MT) (Olah et al. 2020bb Elhage et al. 2021 Casper et al.p. MI concerns itself with rigorously understanding the learned algorithms (circuits) utilized by neural networks (Olsson al. 2022 Wang et al.,
2022a Nanda et al. 2023 Chughtai et al.
2023) in the hope of maintaining oversight and diagnosing failures
of increasingly capable models (Ngo al.
2023). Although research reverse-engineerin specific neurons
in LLMs is limited, (Geva et al. 2020) proposed-and later refined (Geva et al. 2022)-the hypothesis that feed-forward layers function as key-value memories, responding to specific input fealures (keys) and updating the output vocabulary distribution accordingly (Dar et al. 2022). These results are further inspired by earlier work in vision models (Bau et al. 2017 Kim et al. 2018). However, given the differences in scale, training objective, modality, and architecture between vision models and LLMs, unclear which of these results we should expect to transfer.
3

---

Published in Transactions on Machine Learning Research (11/2023)
Superposition Perhaps the most significant obstacle to interpreting neurons in LLMs, and consequently, the success of MI as a whole, is the phenomenon of superposition. As first observed by (Arora et al. 2018 superposition involves compressing multiple features into a smaller number of dimensions (Elhage et al. 2022b Recent research has investigated when superposition occurs (Elhage ct al. 2022b Scherlis et al. 2022), how to design models to have less of it. (Jermyn et al. 2022 Elhage el. al.] 2022a and how to extract features spite of (Sharkey et al.) 2022). Many of the underlying mathematical intuitions rely on prior work on compressed sensing (Donoho, 2006). Moreover, similar questions have been studied elsewhere in
machine learning within the disentanglement literature (Bengio et al., 2013 Chen et al. 2016 Kim & Mnih
2018
Connections to Neuroscience. In a promising demonstration of consilience, our previous discussion of superposition has striking analogues with coding theory from biological neuroscience the study of how neurons in the brain map to sensory stimulus (Rieke et al., 1999 Olshausen & Field 1997 Barlow] 2001). On one extreme, local coding theory posits the existence of "monosemantic" biological neurons which respond to very specific stimulus (e-g-, pictures of Jennifer Aniston (Quiroga al.] 2005 Superposition is then analogous sparse coding where subset of neurons encode some feature about the input (Olshauscn & Field 1997 Vinje & Gallant (2000). Finally population coding theorizes the responses of a whole brain region are relevant, analogous to the computations of a full layer being required to compute or represent feature PPugget et al.
2000 Panzeri et al. 2015
3
Sparse Probing
3.1
Preliminaries
Transformers We restrict our scope to transformer-based generative pre-trained (GPT) language models RRaford et al. 2018 that currently power the most capable AT systems (Bubeck et al. 2023). Given an input sequence of tokens x [Z1...., distribution over the token vocabulary V to prodict the next token in the sequence. M is parameterized by interleaving L multi-head atlention (MHA) layers and multi-layer perceptron (MLP) layers. The central object of the transformer is the residual stream, the sum of the output of all previous layers. Each MHA and MLP layer reads its input from the residual stream and writes its output by adding it to the stream.
X from the vocabulary, model M: X' + + outputs probability
MHA layers are responsible for moving information between token positions, MLP layers apply pointwise nonlinearities to each token independently and therefore perform the majority of the feature extraction, and the residual stream acts the communication channel between layers. In this study, we are primarily concerned with the MLP layers (accounting for & of total parameters) and investigate models with parallel attention (i.e., where the MHA layer and MLP layer are applied concurrently). For such models the value of the residual stream (also referred to as the hidden state) for token & at layer e is
1 ( = h1 (c-1) MMA (9) (7 (z( + MIP(()) (T(" (M)--)
(1)
where y is a layer normalization nonlincarity Ba et al. (2016). Our focus is the MLP layers given by
MLP(0(s)
w(( proj
(w/O0 0/22 162
(2)
where Wproj- ; Wfes Uproj , bie are learned weight matrices and biases, and a is a GeLU Hendrycks & Gimpel (0166). For a more detailed mathematical explanation of the transformer architecture we refer the reader to Elhage et al. 2021).
Probing is technique to localize where specific information resides in model by training simple classifier (a probe) to predict labeled feature of the input using the internal activations of the model (Belinkov) 2022p. More formally, we require lokenized text dataset. X E ynxT (where is the number of sequences and T' is the context length) and an associated labeled dataset Dprobe = [Iz, 230} which provides labels for a subset of the tokens (c.g., the tense of every verb). In our setting, we focus on the MLP neuron activations (!)) (w!!!) fc ( (((((-1))) the set of representations immediately after the elementwise nonlinearity and
4

---

Published in Transactions on Machine Learning Research (11/2023)
(8)) Ejz to minimize a classification loss C(ze, Age) for each layer of the train linear binary classifier ge(a)e) network.
Key Concepts We make frequent reference to the concept of feature. The field has not yet settled on a consensus definition (Elhage et al. 2022b), but for our purposes, we mean an interpretable property of the input that would be recognizable to most humans. A monosemantic neuron is a neuron which activates for exactly one feature, though this can also be subtle depending on what one is willing to count as one feature versus a composition of related features. In contrast, a polysemantic neuron is a neuron which activates for multiple unrelated features. Polysemanticity is a consequence of superposition, the phenomenon of representing n. features with d < dimensions. We focus on applying sparse probing to the activations of the MLP layers because these activations form a privileged basis (Elhage et al. 2021). That is, because applying an elementwise nonlinearity breaks a rotational invariance of the representations, it is more likely that the basis dimensions (cach ncuron) are independently meaningful. Without a privileged basis, as in the residual stream, there is no reason for features to be basis aligned, and therefore no reason to expect sparsity (modulo optimization quirks (Elhage et al (023)). ). Finally, superposition manifests differently in the residual stream than in the neuron activations because of this privileged basis; discuss this further in A.9
3.2 Sparse Feature Selection Methods
Rather than just localize a feature to particular layer, we attempt to identify a single neuron or sparse subset of neurons which fire if (and ideally only if) the feature is present in the input. This more gramular localization can be accomplished by training a k-sparse probe, linear classifier with at most & non-zero cocfficients. The problem becomes, which of the (c) possible neuron subscts are most predictive? While this is an N'P-hard combinalorial oplimization problem, there exist a nunber of fast heuristics and tractable optimal algorithms [Tilmaan et al. 2021).
In probing, this is typically formalized as a ranking problem (Antverg & Belinkov 2021 where ncurons are individually scored by some importance measure and then rank ordered to include the top k. Given binary classification dataset, natural scoring rules that have been explored are the absolute difference between class means for each neuron (Antverg& & Belinkov 2021), the mutual information between each neuron and the labels (Ross] 2014 Voita & Titov 2020; Pimentel et al.) 2020). or the absolute magnitude of the coefficients of a dense probe trained with 4 regularization (Dalvi et al. 2019).
We propose two additional techniques: adaptive thresholding and optimal sparse probing (OSP). OSP leverages recent advantages in sparse prediction solution techniques to train a k-sparse classifier to provable optimality using tractable cutting plane algorithm (Bertsimas et al. 2021), ; though this is only feasible for smaller values of k. For larger rangos of k, we adaptive thresholding to train series of classifiers that iteratively decrease the value of k, where at each step I we retrain a probe to only use the top ki neurons with highest coefficient magnitude from the previous ks--spe probe. For the sake of computational efficiency, in both methods we perform an initial filtering step to take the top neurons with maximum mean difference.
For all methods, we retrain a logistic regression probe for the k neurons selected. A comparison of these different methods is given in B.5) Additionally, while we focus on classification, all of these methods have a straightforward generalization to continuous targets.
3.3 Probing in Practice
Constructing Probe Datasets An informative probing experiment begins with designing a suitable probing dataset. To illustrate a common issue, take the case of probing for the is ss-poiitiiian feature. If the dataset just includes the names of people labeled as politicians or not, the trained probe will not be able to distinguish the model's is pssppoiiiiiian fcature from the is sss_pooiticall feature which fires for all political content (e.g,, "The Democratic Party"). On the other extreme, if the dataset contains just the names of politicians and random tokens, the probe won't distinguish is js_politician from is is_person as very few random tokens concern non-politician people. In other words, there general tension in how to shape the negative examples in the dataset to create the most conceptual seporation between the true feature of interest. and all possible correlates.
5

---

Published in Transactions on Machine Learning Research (11/2023)
Another issue arises when the feature of interest is property of multiple tokens full sequence of tokens (e.g,, is _s_python_ comment or is french). If the relevant neuron fires on every token in the feature sequence, one could just sample tokens randomly, but there might be more specific conditions on when a neuron fires that cannot be known a prior!" A solution to this is to perform an elementwise aggregation (o.g. mean or max) of the activations over the token span of each occurrence of the feature. For example, for a dataset. of sequences many different languages, the input to our probe could be the average activation vector for each sequence, with the target being the language of the sequence. Unfortunately, this gives a somewhat weaker result, as this process is not able to distinguish the is frrnch_novue noun feature from the is french verb feature and thus requires further analysis to interpret correctly.
There are additional subtleties having to do with feature granularity, rare features, and overlapping features Ravichander et al.) 2020) which we explore in Section 5.4 For all of these issues, the most appropriate recourse will depend on the researcher's intent and the nature of the feature being probed for.
Evaluation and Interpretation To evaluate the performance of a probe, we compute the number of true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN) of our binary classifier on an out-of-sample test set. We then calculate the precision (PR), recall (RE), and F1 score (F1):
TP
TP
2PR X RE PR + RE
PR
RE
F1
FP
TF+
We use the F1 score as our primary evaluation metrics to determine which neurons are most likely associated with the target feature given the asymmetric importance of the positive class] Precision and recall give insight into how the selected neurons' implicit feature granularity compares to the feature being probed for. Low precision and high recall indicates either selected neurons represents a more general feature than is being probed for. High precision and low recall of the probing classifier may indicate that the identified submodule represents a more specific feature than the feature being probed for (e.g. is is_french_ _ou instead of is fs_frenc)). We can adjust our preference for classifiers with high precision or high recall by modifying the threshold of the classifier or by tuning the class weights the probe loss function; a higher class weight for the positive class (assuming has fewer samples than the negative class) will prioritize recall over precision.
highly polysemantic or the model
4 Empirical Overview
Models We study EleutherAl's Pythia suite of autoregressive transformer language models (Biderman) ct al. 2023 These models are fairly standard GPT variants that use parallel attention and rotary positional encodings and were trained on The Pile (Gao al. 2020 In particular, we run experiments on the 7 models ranging from 70M to 6.9B parameters; full model hyperparameters are given in Table
Data We study ten different feature collections: the natural language of Europarl documents, the program- ming language of Github source files, the data source of documents from The Pile, the part-of-speech and grammatical dependency of individual tokens, morphological features of tokens (c.g., verb tensc), plain text features of tokens (e.g,, whitespace or capitalization), the presence of specific compound words, 1АTE features in ArXiv documents, and a number of factual features associated with people (e.g., gender, occupation). Full descriptions of datascts, their construction, and summary statistics are available in [B.2 Categorical features are converted into separate one-versus-all binary features for over 100 total binary classification tasks.
Experiments For each combination of model, feature, layer, and sparse feature selection methods, we train probes for range of values for h and report classification performance on a held-out test set. We compare the feature selection methods in B.5 and illustrate the relationship between model size and sparsity
As example, in the of factual recall, the results from Meng et al. 2022 suggest that MLP neurons are especially important for the last token of a multi-token subject. We also mcasure Matthew's Correlation Cocfficient (MCC), a balanced accuracy metric, and the results largely the same. /https://github. con/Elee//aeee Buthar///terr unfortunately, our experiments were performed with the suite of models which have recently been updated.

---

Published in Transactions on Machine Learning Research (11/2023)
For neurons identified being especially relevant, we save the activations over larger text data sets and perform further analyses described throughout Section 5 All code and data are available at httpss//ggtabbb.ccc////ss///pppp
in Figure 7
5
Case Studies
We conduct a series of more detailed case studies to carefully study the behavior of individual neurons, while also illustrating the challenges that pose barriers to further progress. Although we zoom in on very narrow examples, we found many neurons of the same calegory in our probing experiments and believe these examples are representative of broader neuron families (Olah et al. 2020b) that exist in all LLMs.
5.1 Superposition in the Wild: Compound Word Neurons
The token vocabulary is a fairly unnatural symbolic language to perform all of the linguistic and conceptual processing required of a language model. For instance, compound words like "social security" are treated two separate tokens, despite meaning something very different when the tokens appear together versus apart. Moreover, quirks of spacing and capitalization frequently cause words to be broken up (e.g., despite Harvard" being a single token, when not preceded by a space "Harvard" gets tokenized into "Har" and "vard"). Based on many such examples, (Elhage et al. 2022a) hypothesize that primary function of the early layer neurons is to "de-tokenize" the raw tokens into a sequence of more useful abstractions. However, this pseudo-wocubulary might be extremely large (e.g., all common n-grams) making it a natural candidate for being represented in superposition. To investigate this, we probe for neurons which respond to 21 different compound words, where the first and second words mean something quite different when appearing separately versus together (c.g., "prime factors").
After probing for newrons which activate for specific compound words xY while not firing on any bigrams of the form XZ or WY Y We V, Z4Y, W for each compound word we found many individual neurons which were almost perfectly discriminating. However, after inspecting the activations across a much wider text corpus, we observe these neurons activate for a huge variety of unrelated n-grams (see Figure 2(a)) a classic example of the well known phenomena of polysemanticity (Olah et al. 2020b) Mu & Andreas 2020 Elhage et al.) (022a)). Superposition implics polyscmanticity by the pigconhole principle-if layer of neurons reacts to a set of m > features, then neurons must (on average) respond to more than one feature. This example also underscores the dangers of "interpretability illusions" caused by interpreting neurons using just the maximum activating dataset examples (Bolukbasi et al. 2021). A researcher who just looked at the top 20 activating examples would be blind to all of the additional complexity of neuron 70M. N111, with Figure (a) still only scratching the surface] While inconvenient for interpretability rescarchers, polyscmanticity also problematic for the model, as it causes interference between different features (Elhage al., 2022b). That is, if 70M.L1 M1.N11 fires, the model gets mixed signals that both the "prime factors" feature and the "International Coven" feature are present.
However, detecting n-grams in particular (and hence constructing the hypothesized pscuuo-vocabulary), turns out to be a lask particularly well suited for superposition (see A. In short, the model can take advantage of the fact that (for fixed n) exactly one possible n-gram out of [V]"" can occur at any time that is, n-grams are mutcualy-- binary features with respect to the current token of the input, despite coming from a massive set of possible features. While it would be impossible to dedicate a unique neuron to all possible n-grams, by leveraging the activations of multiple polysemantic neurons, each of which react to a bigram XY but with no other overlapping stimuli, the magnitude of the "true" feature gets boosted above all of the possible interfering features. As an example, in Figure 2(b) we show three neurons from the same layer which activate for the "social security" bigram while (mostly) not activating for bigrams with just one of the words (see the blue histograms being significantly to the right of the red and green histograms). However, because these neurons are polysemantic, there are many other inputs which cause them to ictivate potentially even more so than the "social security" feature (see the orange histogram having a longer tail than
*This figure was generated with 300 million tokens from the Pile Test set but when run 10 billion tokens of the Pile train of the top 20 dataset examples show the neuron activating on the "he" of German words starting with "Schönhe". These maximum activating dataset examples can be viewed on Neuroscope Nanda (022)
7

---

Published in Transactions on Machine Learning Research (11/2023)
Activation distribution TOM.11.N111 by token colored by prefix
Teclass
(a) Activations of a single polysemantic neuron on different tokens when preceded by spocific stimuli,
Superposition of sociel-security festure ecross neurons of same layer
security
security social
other
0.4
www.anlllliiiiimmmmmmmm 20,
022
0.0
-02
102
102
sytlia
1.0
0.0
02
0.1
0.G
0.8
1o
pro uttivaaion 2..011
pro activction N4081
pre astivatioon 12,00819
pool activation sum
re attve layer depth
(b) Example superposition representing the compound word "social security. (c) Evidence of superposition in early lay-
crs
Figure 2: Summary of our key results of superposition. Polysemantic neurons firing on many diverse itmulli (a) are necessary to implement superposition (b) to adequately represent many more possible n-grams than dimensions. The natural mechanism for implementing superposition toy models leverages large weight norms and negative biases et al. 2022b) (see Section C for additional motivation and discussion), which observe to be most associated with early layers (c).
the blue histograms). Despite this polysemanticity, by summing the activations across the three neurons, we achieve nearly perfect separation between the total activation magnitude of the "social security" feature and all other observed token combinations! We include examples for all 21 compounds words from layers 1-3 of Pythia 1B in Figures g TO and [T] and further results on basis alignment. A. and Figures [3 and We believe this be first example of neuron superposition exhibited "in the wild' in real L.LMs.
We believe that this computational motif-mergiin merging individual lokens into more semantically meaningful r-grams in the pseudo vocabullyy via a linear combination of massively polysemantic neurons -is one of the primary function of carly layers. As supporting line of evidence, we observe that natural idcalized model of n-gram recovery in superposition predicts a mechanistic fingerprint of superposition: the presence of large input weight norms and large negative input biases. We provide an idealised construction in Section C for how an arbitrary number features can be compressed into two linear dimensions, similar to the construction in (Elhage al. 2022b). When we measure the product of the input weight norm and bias for each neuron in every layer and every model, we observe a striking difference in the early layers, exactly in line with our conceptual argument (see Figure [[c] for summary and Figure g for the full distributions, and Section C for additional motivation of the metric and discussion).
We conclude by noting that, in another encouraging display of consilience, the phenomenon of different regions of the model employing different coding strategies for different functions bears resemblance to the diverse coding strategies employed by biological neural networks in different brain regions with different functional roles OOshhusen & Field 1996 Hromádka al. 20088: Johnssn & Leon 2000 Leutgeb et al.) (2005). We expect further research connecting local mechanisms to macroscopic structure to be particularly fruitful.
Spythia models parallel attention, layer only function the current. token and not subject the n-gram analysis.
8

---

Published in Transactions on Machine Learning Research (11/2023)
Neuron activation by context
Neuron activation by context split by mosi differentisted tokens
Loxs increase with ablation by context
tnben
30.0
200 sogue 000 Horeace
666
70H 70412.000
80
0.00
0.05
distrttttion
0.1
0.15
1012
Yand
brber
"anco'
TH
Targ
hadh
Oatran
Bock
10. 10L0G-E31C setivation
500
-.0
-24
DC
cantodt
String
C4+
Jeha
Tython
391
usplo
244
82
00
T-
Arother
00. Iaateed 18710
Thand
potant
proposed
vecpeds
Figure 3: Analysis of individual context neurons that activate tokens in French (top), in Go code (middle), and US Patent office documents (bottom). We show the distribution of activation when in and not in the target context (left) in addition to the tokens with average activation most distinguished by the neuron (middle). Finally we report the increase in sequence loss across contexts when the neuron is ablated (right).
5.2 Context Neurons: Monosemantic Neuron Family
Given the potential benefits superposition, is reasonable to ask any features are represented monose- mantically, and if so, why? We hypothesized that likely candidate would be contezt features featwres-highhleeeee descriptions of all (or most) tokens within sequence (e-g. is french is is ass_python_ccodeee quite important, to the point that avoiding interference is worth full neuron, while also being higher-level property that may or may not be mutually exclusive or binary, making it harder to represent in superposition.
Such features
In particular, we probe on the language of natural language sequences from the Europarl dataset (oohn 2005), the programming language of different Github source files, and the data subset of The Pile from which a sequence originated. We probe for these features using mean aggregation training classifier on the averaged activations over cach sequence to predict the sequence label (c.g., is french is is_python). Figure g depicts sample of our results, illustrating the existence of highly specialized context neurons that activate approximately only when token is in specific context (with 21 additional examples in Figures 15
16 and 17.. To better understand the function of these neurons, for each token, we study the distribution of activations broken down by context (c.g., the activation of "return" in every programming language). Sorting by the largest differences (middle panel), we observe that one explanation for these neurons' roles is token disambiguation. For example, many programming languages have a return keyword, but neuron LB. 108 activates return and only if it is in the context of Go code.
To further support interpretation and gain more insight into function, conduct ablation experiments comparing the language modeling loss of the base model to the loss of the model with the identified neuron fixed
By recording the activation every token, eliminate the concern of finding french noun neuron instead of french

---

Published in Transactions on Machine Learning Research (11/2023)
part-of-speech
plain text features
programming languages
factual features
mean F1 score for best layer
0.85
0.90
0.90
0.95
0.85
0.75
0.90
0.80
0.70
0.85
0.75
0.65
0.75
0.80
0.70
0.65
100
10
10a
10a
10a
10
10a
10
10a
10a
10
10
probe sparsity
probe sparsity
probe sparsity
probe sparsity
model
70M
160M
1.48
6.98
(a) Probe spansity clussification performance for different feature types and model sizes.
emergent features
constant features
split features
stochastic features
score for best single neuron
1.0
0.9
0.8
0.7
0.6
feature
feature 6 Python
feature
feature
0.5
EnErron
absttrcct
asci
VeraFormFin
0.4
sipha,
2.88
160M
410M
160M
18
160M
2.88
model size
model size
model size
model size
(b) Classification performance of most alagur neuron by model size for different feature classess.
Figure 4: Summary of results on model scale. Representational sparsity increases on average with scale (top)
but individual features obey different scaling dynamics (bottom).
to 0 for all tokens in every sequence (right panel). As anticipated, ablating the context neurons significantly degrades language modeling performance within the relevant context, while leaving other contexts nearly unaffected. The impact, however, heavily depends on the model size- in the 70M parameter model (> 12k neurons), ablating a single neuron causes an average loss increase of 8% per French sequence, while in the 6.9B model (s5524k neurons), ablating one neuron results in only 0.2% increase in loss.
While these neurons appear to be gemuinely monosemantie, emphasize that it is extremely difficult to prove this. Doing so requires answering thorny ontological questions (what is French?) and then efficiently searching through a dataset of billions of tokens to verify the neuron implements the answer. Moreover, the "true" feature the neuron responds to might be quite subtle. For instance, many of the code neurons do not fire when in a code comment, whereas the French neuron will fire in a non-French context on a French name or other token strongly associated to Francd In some sense, these exceptions prove the rule of the primary role of the neuron, but highlight. the difficulty of mapping human features the ontology of the network. Perhaps most challenging though is ruling out the existence of very rare features the neuron also responds to. Sufficiently rare features will likely be undetectable from random samples or summary statistics even when broken down by token and therefore require manually examining max activating dataset examples in subdistributions where we don't expect to find much French text.
5.3 Effects of Scale: Quantization and Splitting
Given the importance of scale in LLMs, we turn our attention to studying the relationship between model size and the sparsity of representations, and the dynamics that drive this relationship. For all of our feature datasets described in B.2 train series of probes sweeping the value of k from 256 to using adaptive thresholding For sake of summarization, we report the maximum out-of-sample F1 value over the layers for
Too further explore this, we encourage the reader explore max activating dataset examples Neuroscope Nanda 2022 8This is an area where we expect AT assisted interpretability be particularly useful.
10

---

Published in Transactions on Machine Learning Research (11/2023)
each model, while averaging over the features within the collection (see Figure ((a) for sample and Figure for full results with random baselines).
While some features appear to be more sparsely represented with scale (e.g., programming languages and factual features), we were surprised by how consistent others were (e.g., part-of-speech and compound words). In fact, for some of the simplest plain-text features, the smallest models seem to actually implement greater sparsity. When analyzing individual features on a per neuron basis, several general patterns become more apparent (Figure [(b)). We believe there are two main dynamics driving these results: the quantization model of scaling MMchaaud et al. 2023 and neuron splitting (Elhage et al. 2022a
In particular, the quantization hypothesis posits that there exists a natural ordering of features learned by a model with increasing scale based on how loss reducing they are, with larger models able to learn a longer tail of increasingly rare features (Michaud et al.) 2023). In our context, this suggests that features like part-of-speech or compound words are within the feature set learned by even small models, and are represented with very similar sparsity patterns. In contrast, factual features and some (but not all) contextual features only get represented by single neurons at sufficient scales. As a countervailing force, with increasing scale the model can dedicate multiple neurons to represent more granular features that previously comprised one coarser feature. Consider the AAL_ feature; while it would likely be advantageous to have dedicated circuitry for representing this feature in all models, a larger model might have dedicated neurons for all of the particular reasons a token might be in all capital letters (e.g, an abbreviation, a constant in python, shouting on the internet, etc.), eliminating the need to have just one more coarse grained representation. The result, as viewed from a probing experiment, is less sparsity, not more.
5.4 Refining and Verifying Interpretations
The result of sparse probing experiment is a set of is neurons with a collection of classification performance metries, usually for range of values of k. We illustrate what conclusions might follow, simple techniques to refine and corroborate such conclusions, and confounding factors to be wary of.
One confounding factor is an ambiguily between superposition and composition (Mu & Andreas 2020 Olah 2023)- If a (layer, feature) pair has a 1-sparse probe with poor accuracy and a k-sparse probe with high accuracy, it is temping to take this as evidence of superposition, but it is also consistent with a feature being coarse or compositionallbert
either the union or intersection of multiple independent features. As an example, consider an is st_hhlette feature. Such feature could either be represented by a single neuron, a superposition of polyscmantic neurons, as the union of neurons for specific types of athletes, as those shown in Figure 5(a) While we can distinguish a feature union from superposition by analyzing the activation co-occurrence (only one athlete type neuron activates for each athlete), this reasoning does not work for a truc compositional feature. For example, we also found individual neurons which coded for a person's 's gender and whether they are alive or not (Figure 20). It is likely more natural for a model to represent the is diivingffemall deemale_scccer__. soccer_player feature as simply a composition of three different neuron aligned-featuree is s_living, is is_ femaem and 3-sparse accuracy. Of course, this also assumes that these neurons are all in the same layer. However, models can, and almost certainly do, leverage superposition and composition across multiple layers, a subtlety which we blissfully ignore and leave for future work.
which would nevertheless have low 1-sparse accuracy but high
For the remaining discussion, we consider individual neurons identified by 1-sparse probes how do we interpret such neurons? The most basic analysis is to simply inspect the maximum activating dataset examples (Nanda (022),, though this is subject to interprctability illusions (Bolukbasi et al., 2021). A slightly more robust analysis involves computing the average activation for each token in the vocabulary. Doing so for our athlete neurons in Figure 5(a) reveals these neurons are perhaps better thought of as generalized sport neurons which activate for all tokens generally having to do with a particular sport, including the names of athletes (with the notable exception of the hockey neuron which appears to be Canadian neuron).
In addition to analyzing a neuron via the input, one can also gain insight by analyzing the output; in particular, one can compute a neuron's effect on the output logits by simply considering the product of the unembedding matrix and the neuron output weight Dar et al. (202)). Doing so for neurons which always activate on punctuation ending a sentence, we find that the output probability of most uppercase tokens gets
11

---

Published in Transactions on Machine Learning Research (11/2023)
Top athlete neurons in Pychio-6.98
atficte sport
Quddecs
6
besbelite
scccer
Rottsll
Ican
Sruak
1eborcch
Kiler
fandded
fumderd
Utlane
artacking
pinch
touchdoon
Edmonttnn
ahooc
Pa Am
+truns
bathing
rasting
Calgery
Rro
Iratnderr
172rg
p0:
01
Steen
hoekans
Maple
L19.N00761
L193549
1203333335
120.N10306
120.02307
L10.N3520
N10761
N549
N13139
N10106
422167
h0520
(a) Neurons which activate for the names of specific types of athletes (left)) turn out be more general sport neurans when
analyzing the top per-token average activations (right).
0.6
Whether
Cov
lzpph
Parhaps
Parc
annumbero
Contents
F
contents
Whether
Ub
oggt diforence MMgaaal
0.4
Liv
Discussion
Tube
Rnaly
@coolopp
(PC. actvotonnn
02-
Through
26
plst
Othenitte
Bur
31
Nakes
However
Math
Thel
0.0
0.0
Rann
0.2
Conclusion
Tbx
intumbered
Instead
Ore
Joml
-0.2
end sent
next tollon lpppent wot
lirensue
stnigg
There
Hegepet
-0.4
Attemativcly
Constants
-0444
bvercase
JOM 14.188 300M X22
4EOM 71
100M LB M2
4LOCC 71
70M
160M.L9.N2730
(b) End of sentence neurons: analyzing the effect on the output vocabulary can corroborate and refine neuron interpretations. token heatmaps (right), the columns correspond to capital tokens with max increased logits, capital tokens with max decrease, and lowercase tokens with max increase, rospectively.
Low precision passive verb nouron
Low rocall verb neuron
10a
vet (est pariiceee (801 eiterverb ether owr
0.14
Ather webl
IC
540 untiiaatiioonn
berticeee
count (og)
104
0.08
101
0.66
0.04
10²
0.02
101
6.99118
69919977566
2.86.10.N7612 pre-octvation
2.88.110.N7612 rn-accivahion
(c) Precision and recall metrics can guide further analysis.
(d) Distributional statistics can hide rare features.
Figure 5: Examples of analyses used to refine and support neuron interpretations.
increased while the probability of most lowercase tokens gets decreased (Figure 5(b)). Inspecting specific examples is often constructive; for instance, the highest increasing lowercase tokens are words like "amazon" or "wordpress," " indicating such neurons are also useful in constructing URLs. For uppercase tokens, those with the highest increase in probability correspond words like "Finally" or "However" while the most decreased tokens those representing atomic elements, proper nouns, CamelCase strings.
Most of the neurons discussed thus far have had excellent overall classification performance; what happens if there does not exist such a crisp fit? As discussed in Section 3.3 classification performance can be further decomposed into precision (sensitive to false positives) and recall (sensitive false negatives). In our context, high recall and low precision with respect to a specific feature polentially suggests that a neuron represents less granular feature than the feature manifest in the probe dataset low recall and high precision suggest a neuron represents a more specific feature. As an example, Figure 5(c) depicts the activations for a www.eellll----eesiitttiiee
neuron identified when probing for is ss__passive___e werb and a low-recall-hine www---iihh-hhhiiiii neuron identified when probing for is, verb in Pythia 6.9B. When analyzed these neurons in more detail, we find that in addition to activating on all occurrences of passive verbs, N8915 also activates on adjacent adposition tokens, and past participles when in an adnominal clause. Almost symmetrically, L9. L9.N756 57566 activates on most verbs, especially infinitive verbs, but not passive verbs, adjectival modifiers, plural root
9I, also consistent with superposition. Indend, all the compound word neurans had higher recall than precision.
12

---

Published in Transactions on Machine Learning Research (11/2023)
verbs, past participles in certain dependency roles. These examples illustrate the more general point that it should not be assumed that a model will learn to represent features in an ontology convenient or familiar to humans.
Of course, precision and recall are only defined with respect to the labels of a probing dataset, which may themselves contain imbalances or spurious correlations that confound the results. Perhaps most common, especially when constructing a dataset from scratch, is the problem of asymmetric sampling and rare features. Consider probing dataset for an is_ Satpddv_apee__rr_e _tille feature (Figure 5(d).. Without more specific hypothesis the relevant negative examples, one is likely to simply sample random non-title tokens (or at least weight all non-title examples the same). However, when the space of negatives is large (e.g., all non-title tokens), at a distribution level, all rare features get drowned out. Hence, when looking at summary or distributional statistics, the results can look impressive (Figure [[d] the neuron had F1 > 0.95 for is ss_title), but when looking at raw counts (Figure 5[d] right) one observes that the is is_title feature actually explains
less than half of the activations. Though, upon inspection, we find that the rest of the activations are for section titles!
6 Discussion
6.1 Strengths and Weaknesses of Sparse Probing
The primary use case of sparse probing is to quickly and precisely localize neurons relevant to a specific feature or concept, while naturally accounting for superposition and composition. The speed and precision of recovery is in contrast to gradient-based (De Cao et al. 2021) and causal-intervention based methods (Meng ct al. 2022), which are too slow or coarse-grained to be used for features requiring precise localization within large models. The feature specificity contrasts with sparse autoencoding based methods which aim to recover all features stored in the model, but in an unsupervised manner and without attributing semantic meaning to each feature (Sharkey et al.) 2022). Having probes with optimality guarantees further addresses the pitfall raised by (Antverg & Belinkov 2021 regarding the conflation of classification quality and ranking quality when analyzing individual neurons with probes. Moreover, sparse probes are designed to be of minimum capacity, mitigating the concern that the probe is powerful enough to construct its own representation to learn the task (Hewitt & Liang, 2019). While probing requires supervised dataset, once constructed, it can be reused for interpreting any model (modulo features regarding tokenization), enabling investigation into the universality of learned circuits (Olah et al. 2020b Chughtai et al.) 2023) and the natural abstractions hypothesis (Chan et al. 2022). By design, spanse probing is particularly well suited for studying superposition, and can be used to automatically test the effect of architectural changes on the frequency of polysemanticty and superposition, as opposed to relying human cvaluations as in (Elhage et al. 2022a
However, sparse probing also inherits many of the weaknesses of the general probing paradigm (Hewitt
& Liang. 2019 Donnelly & Roegiest 2019 Ravichander et al.) 2020 Maudslay & Cotterel] 2021 Elazar
et al. 2021.. In short, the results from a probing experiment do not allow for drawing especially strong conclusions without more detailed secondary analysis on the identified neurons. Probing offers limited insight into causation, and is highly sensitive to implementation details, anomalies, misspecificationn ions, and spurious correlations within the probing dataset. For interpretability specifically, sparse probes cannot detect features built up over multiple layers easily distinguish between features in superposition versus features represented as the union of multiple independent and more granular features (Mu & Andreas 2020). In seeking the sparsest classifier, sparse probing may also fail to select important neurons that are redundant within the probing dataset, requiring the use of iterative pruning to enumerate all important neurons. Multi-token features require special processing, often in the form of aggregations that can further weaken the specificity of the result. Finally, depending on how much representations change with scale (Michaud et al. (202)), larger models may utilize more specific features (Elhage et al. 2022a), hampering the transferability of datasets between different model scales.
13

---

Published in Transactions on Machine Learning Research (11/2023)
6.2 Strengths and Weaknesses of Empirical Findings
In light of the limitations of probing, we attempted to corroborate every case study with independent evidence, including theoretical predictions, ablations, and vocabulary analyses. We find our evidence compelling, although not always conclusive. In particular, we believe we have provided the clearest evidence to date of superposition, monosemantic neurons, and polysemantic neurons in full scale language models. Furthermore, by demonstrating this behavior in seven differrnt models spanning two orders of magnitude in size while exploring over a hundred features, we believe our basic insights are likely to be general and to transfer to current frontier models like GPT-4.
However, much of our analysis ad hoc, tailored to the specific feature being investigated, and requires substantial rescarcher effort to draw conclusions. While we explored models of varying size, they were all from the same model family and trained with the same data. We think it is unlikely our results are specific to the implementation details of the Pythia model suite, but we do not rule this out. Additionally, the largest model we studied is 6.9 billion parameters which is still more than order-of-maguitudee off the frontier. Given the emergent abilities of LLMs with scale (Wei et al. 2022), it is possible our analysis misses a key dynamic underlying the success of the largest models. Moreover, our results are restricted to binary features and categorical features converted to binary features, and we are not confident that our insights will cleanly transfer to continuous features or cleverly encoded categorical features.
6.3 Implications
For interpretability researchers, our results support the conclusion that superposition is important for the success of models. Hence, attempts to remove it (Elhage et al., (202aa) are likely either hiding it or are unlikely to be competitive, but that the highest leverage interventions are perhaps best aimed at the early layers. For AI cthicists and legal scholars, further inquiry-udeerslann understanding how neurons encoding protected altribules compute this feature and affect downstream predictions of the model. For AI alignment scholars, our work highlights the potential of identifying safety-critical features and perhaps even manually intervening in computations to enhance our ability to steer models. While none of the features we probed for were especially critical, the Pile subset context neurons could be considered a minimal precursor for situational awareness (Ngo al. 2023 Carlsmith 2022). as these could be used to detect whether an input is from the training or test distribution.
study of factual neurons point to an important avenue of
6.4 Future Directions
We only scratched the surface of possible applications and experiments involving sparse probing. Scientifically, further understanding superposition ambitious version of mechanistic interpretability. As outlined in (Elhage el al. 2022b), this requires either designing architectures which do not use superposition or to take features out of superposition using a sparse coding like technique- both of which can be assisted with automatic sparse probing While not explored here, it would also be possible to apply sparse probing to predicting properties of the output (e-g.
how to cope with seems contral to making progress on the
form of validation.
is s_ebb,, as opposed to properties of the input, to better understand the neurons most implicated in making specific types of predictions. With neurons identified in each experiment, it would be potentially insightful to also track how these neurons develop and change through time (Liu et al.) 2021 as well as more carefully analyze how the set of neurons change with scale (Michaud et al. (202)), which is naturally enabled by the Pythia suitc's (Biderman et al. 2023) checkpoints. In particular, sparse probing (or similar) is well suited to study neuron splitting in more detail by training probes to predict the value of less gramular neuron from the sum of sparse set of more granular neurons.
Our primary motivation however, is in ensuring the development of safe AI systems. To further this goal, we envision the development of a large library of probing dalasets potentially with Al assistance that capture features of particular relevance to bias, fairness, safety, and high-stakes decision making. In addition to automating cvaluations of new models, having large and diverse supervised datasots will enable better evaluations of the next generation of unsupervised interpretability techniques (Sharkey et al. 2022 Burns
et al. 2022 that will be needed to keep pace with AI progress.
14

---

Published in Transactions on Machine Learning Research (11/2023)
7 Conclusion
Guided by sparse probing, we have found some of the cleanest examples of monosemanticty, polysemanticity and superposition in language models "in the wild," and contributed practical guidance and conceptual clarification for how to interpret neurons in greater detail. More than any specific technical contribution, we hope to contribute to the general sense that ambitious interpretability is possible- that. LLMs have a tremendous amount of rich structure that can and should be understood by humans. We believe this is most productively accomplished with empirical approach more reminiscent of the natural sciences, such biology neuroscience, than the traditional experimental loop of ML. While such research can be hard to finish, it is casy to start, so we encourage the curious rescarcher to just start looking!
References
Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in nearal information processing systems, 31, 2018.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using lincar classifier probes. arXiv preprinl ar arX:::6610.01644 2016.
Uri Alon. Design principles of biological circuits. Biophysical Journal, 96(3):15a-115 15a, 2009.
Rana Ali Amjad, Kairen Liu, and Bernhard C Geiger. Understanding neural networks and individual neuron importance via information-orderee cumulative ablation. IEEE Transactions on Neurol Networks and Learning Systems, 2021.
Omer Antverg and Yonatan Belinkov. On the pitfalls of analyzing individual neurons in language models. arXiv preprinl ar arXim:21 (tt:2100.7783,, 2021.
Sanjoev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Lincar algebraic structure of word senses, with applications to polysemy. Transactions of the Associalion for Compulational Linguislies, 6:483-495, 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. 2016.
arXiv preprint
Horace Barlow. Redundancy reduction revisited. Network: computation in neurol systems, 12(3):241, 2001.
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass, Identifying and controlling important neurons in neural machine translation. ar preprint arXi::1811.01157, 2018.
David Bau, Bolci Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissoction: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9541-6549, 2017.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Under- standing the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 2020. ISSN 0027-8424. doi: http://ass.1903751111 URL hnttps://ww/ w/tp://wwwww.. pnas. org/content/ ear1y//200//8/31//90773555
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48
(1):207-219, 2022.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-11 -1888, 2013.
Dimitris Bertsimas, Jean Pauphilet, and Bart Van Parys. Sparse regression: Scalable algorithms and empirical performance. 2020.
Dimitris Bertsimas, Jean Pauphilet, and Bart Van Parys. Sparse classification: a scalable discrete optimization perspective. Machine Learning, 110:3177-3200 2021.
15

---

Published in Transactions on Machine Learning Research (11/2023)
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Blair Bilodcau, Natasha Jaques, Pang Wei Koh, and Been Kim. Impossibility theorems for feature attribution. arXiv preprinl arXiv:2212.11870, 2022.
Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Mcrizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, and Connor Leahy. Interpreting neural networks through the polytope lens, 2022.
Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wattenberg. An interpretability illusion for bert. arvio preprint orXin:2104.071 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXi::2303.122 12712, 2023.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. arXiv preprinl Xrx:::2212, 033027777 2022.
Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah, Curve circuits. Distill, 2021. doi: 22239155disttill00022
Steven Cao, Victor Sanh, and Alexander M Rush. Low-complexity probing via finding subnetworks, arv preprint arXiv:2104. 08514, 2021.
Joseph Carlsmith. Is power-socking ai an existential risk? arXiv preprint arXh::2206.1338 2022.
Stephen Casper, Tilman Rauker, Anson Ho, and Dylan Madffeld-Menelll Sok: Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. In First IEEE Conference Secure and Trustworthy Machine Learning.
Lawrence Chan, Leon Lang, and Erik Jenner. Natural abstractions: Key claims, theorems, and critiques, 2022. http://wwwwlgimmeettrrn...mmmm /ttpaoo/tto/////000aaAAAAAAAA..
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Picter Abboel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. ar preprint arxiv:2204 NXX::2204000311,, 2022.
Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. arXie preprint XxXiv:2302.03025, 2023.
Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wattenberg. Visualizing and mcasuring the geometry of bert, 2019.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into single vector: Probing sentence embeddings for linguistic properties, 2018.
Fahim Dalvi, Nadir Durrani, Hassan Sajaad, Yonatan Belinkov, Anthony Bau, and James Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8809-6317, 2019.
16

---

Published in Transactions on Machine Learning Research (11/2023)
Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. Analyzing redundancy in pretrained transformer models. arXiv preprint arXi::2004.044: 04000, 2020.
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. arXiv preprint Xrv: 02535, 2022.
Nicola De Cao, Leon Schmid, Dieuwke Hupkes, and Ivan Titov. Sparse interventions in language models with differentiable masking. arXiv preprint arXiu:2112.06837, 2021.
Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How important is a neuron? Xiv preprinl arXiv:1805. (0tp:1805.122333 12233, 2018.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pp. 647-655. PMLR, 2014.
Jonathan Donnelly and Adam Roegiest. On interpretability and feature representations: an analysis of the sentiment neuron. In Advances in Information Retrieval: 41st Europeon Conference on IR Research, ECIR 2019, Cologne, Germany, April 14 18, 2019, Proceedings, Port 41, pp. 795 802. Springer, 2019.
David L Donoho. Compressed sensing. IEEE Transactions information theory, 52(4):1289-1306 2006.
Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. Analyzing individual neurons in pre-trained language models. arXiv preprint arXiv:2010.02695 2020.
Nadir Durrani, Fahim Dalvi, and Hassan Sajjad. Linguistic correlation analysis: Discovering salient neurons in deepulp models. arxiv preprint arXiv:2206.1328 2022.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Compulational Linguistics, 9:160-175, 2021.
N Elhage, N Nanda, C Olsson, T Henighan, N Joseph, B Mann, A Askell, Y Bai, A Chen, T Conerly, et al. A mathematical framework for transformer circuits. Transformer Cirewits Thread, 2021.
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodds, Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Kadavath, Josh Jacobson, Eli Tran-Johuson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCandlish, Dario Amodei, and Christopher Olah. Softmax linear units. Transformer Circuits Thread, 2022a. https:/ http:////taassomm-rcrriiiitss huclsssuuubbuuuu hst////ooo//ii.tmmm
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Slaffeed-Doddss Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint ar Xix::2200. 10652, 2022b.
Nelson Elhage, Robert Lasenby, and Christopher Olah. Privileged bases in the transformer residual stream. Transformer Circuits Thread, 2023. httpss//taansforeeerrreeiitt pub//0233 2023///ivilego/dll... privilegoodd-aasis index.html.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXie:2101 00022, 2020.
Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. Transformer food-forward layers are key-value memories. arXio preprint 2tt::0012.49111 2020.
Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer ford-forward layers build predictions by promoting concepts in the vocabulary space. arXiu preprint arXio:2209. 14680, 2022.
17

---

Published in Transactions on Machine Learning Research (11/2023)
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6(3):30,, 2021.
Nicholas Goldowsky-Die Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. Xivivv preprint arXiv:2304.0500 2023.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandcharioun. Does localization inform oditing? surprising differences in causality-based localization vs. knowledge editing in language models. arv preprinl WwXi:2231.04213, 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.0844 2016.
Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, memorization, and double descent. Transformer Circuits Thread, 2023. https:/ pttpss//taansfomme--iiiiite cireuits..rrr...rrpe..e.. wwt.uits.puub/ putp://3/oo---duube-deseeet
Lucas Torroba Hennigen, Adina Williams, and Ryan Colterell. Intrinsic probing through dimension selection. Xr preprint arXix:2010.0088 2020.
J Hewitt and P Liang. Designing and interpreting probes with control tasks. Proceedings of the 2019 Con, 2019.
Tomás Hromádka, Michael DeWeese, and Anthony M Zador. Sparse representation of sounds in the unanesthetized auditory cortex. PLoS biology, 6(1):e16, 2008.
Adam Jermyn, Chris Olah, and Tom Henighan. Circuits updates may 2023: Attention head superposition, 2023. http:://rransoomrreccrciiii.pppp http:/rriiiiiaaaa////////aaa/aaa
Adam S Jermyn, Nicholas Schiefer, and Evan Hubinger. Engineering monosemanticity in toy models. arXiv preprint 2xf::2211009160 2022.
Brett A Johnson and Michael Leon. Modular representations of odorants in the glomcrular layer of the rat olfactory bulb and the effects of stimulus concentration. Journal of Comparatice Newrology, 422(4):496 509, 2000.
Amir-Hossein Karimi, Krikamol Muandet, Simon Kornblith, Bernhard Schölkopf, and Been Kim. On the relationship between explanation and prediction: causal view. arXiv preprint arXiv:2212. 0602225 2022.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Inter- pretability beyond fcature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-28 2678. PMLR, 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649 2658. PMLR, 2018.
Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of machine translation summit x: papers, pp. 79-86, 2005.
Stefan Leutgeb, Jill K Leutgeb, Carol A Barnes, Edvard I Moser, Bruce L McNaughton, and May-Brit Moser. Independent codes for spatial and episodic memory in hippocampal neuronal ensembles. Science, 309(5734):619 623, 2005.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. arXiv preprint ar 2rX1v:2210.1888 18382, 2022.
Leo Z Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A Smith. Probing across time: What does roberta know and when? arXiv preprint ar arXiv:2104.024 07885, 2021.
18

---

Chris Olah. Distributed representations: Composition & superposition, 2023. https://transformee http:///w/..././spppssssoooooops Published in Transactions on Machine Learning Research (11/2023)
Rowan Hall Maudslay and Ryan Cotterell. Do syntactic probes probe syntax? experiments with jabberwocky probing- arXiv preprint orXii:2106. 02559, 2021.
Rajiv Mehrotra, Kameswara Rao Nanuduri, and Nagarajan Ranganathan. Gabor filter-based edge detection. Pottern recognition, 25(12):1479 1494, 1992.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Newral Information Processing Systems, 35:17359-17372, 2022.
Eric J Michand, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. ar preprint ar Xrv: 13506, 2023.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Information Processing Systems, 1117153-171663 2020.
Neel Nanda. Neuroscope: A website for mechanistic interpretability of language models, 2022. URL
https://heuroocoop.et
Neel Nanda. A comprehensive mechanistic interpretability explainer & glossary, 2023a.
URL
https
//neelnanda.1o/ io/glossary
Neel Nanda. Actually othello-gpt has a linear emergent world model, Mar 2023b. URL. <https://neelnanda
Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint ar xrXi::231.002117, 2023.
Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning perspective, 2023.
Anh Nguyen, Jason Yosinski, and Jeff Clune, Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks, 2016.
Chris Olah. Interpretability neuroscience, 2021.
URL http::///olaaa..gtuuu.ooooo
interp-v-neuro/
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
An overview of early vision in inceptionvl. Distill, 2020a. doi: 10.23915/disttl1.0002 tistiil..00024.022
(012D//ircuits/earlyviiii
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in:
An introduction to circuits. Distill, 5(3):e00024 001, 2020b.
Bruno A Olshausen and David Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):660 609, 1996.
Bruno A Olshausen and David J Field. Sparse coding with overcomplete basis set: A strategy employed by v1? Vision research, (7(23)33311-233
1997.
Catherine Olsson, Nelson Elhage, Ncol Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hadfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context lcarning and induction heads. Transformer Circuits Thread, 2022. http://aaanfoorrrrrrrrrr leariiggand-inducttin---nnnn weweetiionneadddddeeee....e /tt/diddddhmmtt html.
2022/in-contess conttxtext-
Op Gpt-4 technical report, 2023.
19

---

Published in Transactions on Machine Learning Research (11/2023)
Stefano Panzeri, Jakob H. Macke, Joachim Gross, and Christoph Kayser. Neural population coding: combining insights from microscopic and mass signals. Trends in cognilive sciences, 19(3):162-177 172, 2015.
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. information-theoreee probing for linguistic structure. arXiv preprint arXit:2044.03061, 2020.
Alexandre Pouget, Peter Dayan, and Richard Zemel. Information processing with population codes. Nature
Reviews Neuroscience, 1(2):125-132, 2000.
R Quian Quiroga, Leila Reddy, Gabriel Kreiman, Christof Koch, and Itzhak Fried. Invariant visual represen- tation by single neurons in the human brain. Nature, 435(7045):1102 1107, 2005.
Alec Radford, Rafal Jozefowicz, and Nya Sutskever. Learning to generale reviews and discovering sentiment. arXiv preprint arXiv: 1101170444144. 2017.
Alec Radford, Karthik Narasimhan, Tim Salimans, Llya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. Probing the probing paradigm: Does probing accuracy entail task relevance? arXiv preprint arXiv:2005.00119 2020.
Fred Rieke, David Warland, Rob de Ruyter Van Steveninck, and William Bialek. Spikes: exploring the neural code. MIT press, 1999.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works, 2020.
Brian C Ross. Mutual information between discrete and continuous data sets. PloS one, 922) 2014.
Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-level interpretation of deep nlp models: A survey. Transactions of the Association for Computational Linguistics, 10:1285 2022.
Adam Scherlis, Kshitij Sachan, Adam S Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks. arXiv preprint arXiv:2210. 01892, 2022.
Lee Sharkey, Dan Braun, and Beren Millidge. Taking features out of superposition with sparse autoencoders, 2022. http://www.lgmmettoorou...m /ssp://ss////QQQhhAAAAA33...
Xavier Suau, Luca. Zappella, and Nicholas Apostoloff. Finding experts in transformer models. arXiv preprint arXiv:2005.07647, 2020.
Ian Temney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. arXie preprint 2019a.
Tan Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint arXiv:1905.06316 2019b.
Andreas M Tillmann, Daniel Bienstock, Andrea Lodi, and Alexandra Schwartz. Cardinality minimization, constraints, and regularization: a survey. arXiv preprinl XrXi::2106. 096000 2021.
William E Vinje and Jack L Gallant. Sparse coding and decorrelation in primary visual cortex during natural -1776, 2000.
vision. Science,
Elena Voita and Ivan Titov. Information- probing with minimum description length. arXiv preprint arxiv:2003. 12298, 2020.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211005593, 2022a.
Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. Finding skill neurons in pre-trained transformer-basee language models. arXio preprint arXiv:2211.00 1.07349, 2022b.
20

---

Published in Transactions on Machine Learning Research (11/2023)
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint ar Xic:22000 07682, 2022.
A Frequently Asked Questions
A.1 Why expect ambitious interpretability to be possible or worthwhile?
We divide our answer into two parts: why it should be possible even in theory, and how we interpret the progress so far.
Although gradient descent has no reason or incentive to learn representations scrutable to humans, the same could be said of all the biological structures "learned" by natural selection. Yet biological structures are constrained by the laws of nature- an organism must make efficient use of limited energy and space, its genetic material is encoded in RNA and DNA, its functionality is encoded in proteins, and these proteins fit together into biological circuits. Although these biological circuits are complex, often idiosyncratic organisms, and times seem inscrutable, these constraints give us insight into their functioning and create common motifs and principles that can be engieee-e (Alon 2009]. In similar way, neural networks are trying to achieve complex tasks and have significant constraints, namely, the number of parameters, non-lincarities and weight norm. And more broadly, the algorithms their architecture allows them to represent. By studying analogous structures of neural nets both at a metaataiiaal level (Elhage et al. 2021 I and in the wild (Wang et al. 2022a) is possible to uncover some of these principles and motifs, and to start gaining a foothold in reverse-ongineering these artificial circuits.
A potential source of pessimism for ambitious interpretability is the progress made thus far. Though we have gained some insights about model internals, most works in mechanistic interpretability have focused on small or toy models (Elhage et al., 2021 Nanda et al. 2023 Chughtai et al. 2023 Wang et al. 2022a Cammarata et al. 2021), while frontier models become larger at far faster pace (Brown et al. 2020
Chowdhery et al.,
2022 OPenAT 2023). While more scalable approaches to interpretability have often been shown to have results
that are easy to misinterpret, or sometimes actively misleading (Adebayo et al. 2018 Bilodeau et al. 2022
Hase al. 2023 Karimi al. 2022). It natural to look at this confusion and seeming incomprehensiblitty and feel discouraged. Yet, natural structures were no doubt similarly incomprehensible to early pioneers in molccular biology, genetics, and neuroscience, exhibiting emergent complexity that scemed irroducible. However, rather than claiming cells, genes, brains were "uninterpretable" " entire scientific disciplines emerged which have made great strides in understanding the core principles in sufficient detail to enable intervening, engineering, and controlling biological systems. We believe artificial neural networks can be fully interpreted-
doing so requires a comparable amount of effort as interpreting biological ones. In many ways, this emerging artificial neuroscience is unusually amenable to the scientific method (Olah] 2021): it possible run arbitrary counterfactual experiments, iteration times are rapid, resource requirements are minimal, all with full observability and measurement precision equal to floating point machine precision. While ambitious, we think such effort is paramount for addressing issues in the alignment and control of increasingly capable AI systems.
evenn even reverse-enging penrorse-nngnneeed--eee
A.2 What do you mean by neuron?
The term neuron is sometimes used to refer to elements of any activation tensor, in the standard basis. Here, we instcad reserve neuron to refer to the clements of the model activations immediately after an clementwise non-linearity: the post GELU activation in the hidden state of a transformer's MLP layer. We do nol use neuron to refer to elements of a transformer's residual stream, a layer's output, or the key, query or value within an attention head. These are the output of lincar map, and so do not have privileged basis (Elhage et al. (221)): we expect the model function the same under an arbitrary rotation (modulo optimiser quirks (Elhage al. 2023
21

---

Published in Transactions on Machine Learning Research (11/2023)
A.3 What is the difference between polysemanticity, superposition, and distributed representations?
These are similar concepts with crucial differences, that are commonly confused. We propose the following schema:
Superposition is the phenomena when an activation represents more feat than it has dimensions. Each feature is a direction in space, and as a consequence they canmot all be orthogonal.
Polysomanticity is when a ncuron seems to represent multiple, unrelated concepts. This is in contrast to a monosemantic neuron that activates if and only if a cerlain feature present.
Distributed representations are where a feature is represented as a linear combination of neurons, ie., direction that does not correspond to single basis element. Also known as non-basis aligned features. Notably, this could be a fairly sparse distributed representation (e.g., using 2-5 neurons), fairly dense (e.g., using 10-20% of all neurons in the layer), to completely dense/not at all basis aligned.
Notably, polysemanticity and distributed representations are local notions; they can be demonstrated by studying individual neurons or features respectively. Superposition is a global phenomena, and requires identifying more fcatures than neurons to conclusively show.
A.4 What is the relationship between polysemanticity, superposition, and distributed representations?
A linear representation may be distributed because of rotation/skew (e.g., when lacking a privledged basis),
composition, or superposition (Olah 2023
Superposition necessarily implies polysemantic neurons (and likely distributed representations), because there are more features than neurons! However, it is possible to have distributed representations and polysemanticity without superposition. There may be many features as dimensions, but in some rotated basis not aligned with the neuron basis.
Indeed, without a privileged basis, this is what we would expect to observe even without superposition. The model has no incentive to align features with basis elements, and as we would expect, prior work [Bolukbasi et al. 2021 has found polysemanticity of residual stream basis elements. However, it is surprising that this should occur in the presence of an elementwise non-linearity. If model acting as a feature extractor, it is useful to represent features such that they can independently vary. Individual neurons have a separate GELU, but if multiple features share neuron then they may significantly interfere with each other.
Morcover, we can have polysemantic neurons and superposition without a distributed representation if a neuron represents multiple features, but each of those features is only represented by that neuron. This is a form of superposition. The model will then need to have circuitry to disambiguate which feature activated the neuron, but we can imagine ways this could be efficiently implemented. A toy example: a model could have 100 neurons, each of which represents both some feature of Python code and some fcature of romance novels. Python code and romance novels are mutually exclusive contextual features that could each already be computed and represented in the residual stream, which can then be used to disambiguate which feature each neuron activated for. Though this could be considered certain kind of distributed representation where feature is represented as linear combination of its neuron and the romance/ Pythnn feature.
Our interpretation of our results that this is further evidence that, at least in carly layers, models engage in superposition involving both polysemanticity and distributed representations.
A.5 What is the difference between faceted and polysemantic neurons?
A basic approach to detecting polyscmanticity is to apply clustering to the activations of texts that strongly activate a neuron (e.g,, the residual stream immedialely before the MLP layer (Black et al., 2022) ): if each cluster has some shared semantic meaning, then there are multiple clusters the neuron is seemingly polyscmantic! And indcod, there is a single clear cluster, that is evidence of monosemanticity. However, there are two distinct phenomena here: related and unrelated clusters. neuron faceted (aceyed(Nguyen al. 2016) if it activates for multiple different things with shared meaning, e.g., cutlery neuron activating for
22

---

Published in Transactions on Machine Learning Research (11/2023)
knives, forks and spoons. While polysemantic neuron activates for clusters without shared meaning, e-g., dice and poetry. Unfortunately, this is currently a subjective definition: what does it mean to have shared meaning? We see formalizing these intuitive concepts as a promising area of future work.
A.6 What are the two different kinds of interference, and how do they affect superposition?
The phenomena of superposition arises because models can decrease loss by representing more features, yet having non-orthogonal features introduces interference which increases loss. This creates pareto-frontier trading off the two for each feature and each model, and this determines the representations a model learns. Crucially, interference between two features A and decomposes into two conceptually different forms of interference (Nanda) 2023a); alternating interference where A is present, is not present (or vice versa), and the model needs tell that A is present and B is not, and simultaneous interference where and B are present, but the model needs to tell that A and B are both present, but that, for example A is not present at twice strength. To understand the difference, let's consider the case where A and B are binary features corresponding to different directions of unit norm.
Alternating interference is fundamentally about distinguishing high activations (feature A is on) from low activations (feature B is on, feature A is off, but its direction not the same as feature A's direction), which is fairly straightforwards with GELUs and softmaxes. But simultaneous interference is fundamentally about treating medium activations (feature A is on, feature B is off) as the same as high activations (feature A is on and feature B is on) but different from low activations (feature A is off and feature B is on, both are off). This is much harder to do with GELUs or softmaxes.
In toy models, models seem to avoid superposition with high simultancous interference (e.g., correlated features), but tolerate high alternating interference (e.g,, anti-correlated features). Notably, (Elhage et al., 2022b) observed that independent features occurring with probability p engage in more superposition when p is lower the probability of alternating interference lincar in p while simultancous is quadratic in p. The expected loss reduction from representing a feature is also linear in p; suggesting that whether or not a model engages in superposition is predominantly driven by the costs of simultaneous interference, not alternating. For sufficiently rare and uncorrelated features simultaneous interference is a non-issue.
Mutually exclusive features like n- n-grams are an extreme case where simultaneous interference can never occur, and so are well suited to superposition.
A.7 How does the type of feature affect how it might be expressed in superposition?
Three notable categories of features are continuous (e.g., the height of an object), binary (e.g., whether 'social socurity' is present) and categorical (c.g., whether an object is blue, red or ycllow). Categorical features can be represented as one-hot encoded binary features, so it is most instructive lo compare binary features and continuous features. We argue that interference is significantly greater for continuous features than binary Seatures-intuitivee the true feature and large values of an interfering feature.
alternating interference will prevent us from distinguishing small values of
This effect has been observed in the literature when studying toy models. (Elhage al. 2022b) studied contimuuus variables and found superposition tending not to compress more than 5 features into two dimensions (and with significant interference), while (Henighan et al. 2023) studied binary variables (memorization) and found that models could compress significantly more features into two dimensions with minimal performance lost (though this is confounded, as memorization is also mutually exclusive).
A.8 Why might we expect n-gram detection in particular to use superposition?
n-gram detection is unusually well suited to superposition for several reasons. n-grams are mutually exclusive ('social security' cannot co-occur with 'prime factors', for example), and are binary variables.
Morcover, n-grams are an example of a simple feature: detect 'social security'; a neuron need only compose with a previous token head detect social' and with the current token embedding to detect security', which will create sharp decision boundary: significant gap between the smallest neuron activation when
23

---

Published in Transactions on Machine Learning Research (11/2023)
the feature is present and the largest neuron activation when the feature is not present. This means that for downstream computation the social security' feature can be used with minimal noise from false positives. In contrast, complex or subtle features like whether a text is in French may require compiling many tiny pieces of evidence, creating an unclear decision boundary between true and false. In this way, despite the ground truth being a binary feature, models may think of complex and nuanced features as continuous variables associated with the probability of being present, making these features harder to store in superposition.
A.9 What is the difference between residual stream and neuron superposition?
Residual stream superposition is an example of representational superposition (sometimes called bottle- neck superposition) and is fundamentally about compression. That is, mapping a high dimensional space to a low dimension space such that the model can recover the compressed features at later point in the network. This also implies that the model has already computed the feature, and the superposition is just for the purposes of storage. In addition to occurring in the residual stream, we also expect the keys, queries and value vectors of attention heads to employ representational superposition to represent more feat than there are dimensions.
In contrast, neuron superposition is an instantiation of computational superposition, where a model can extract or compute with more features than it has dimensions (in this case neurons). If there exist more features than neurons that must be potentially computed, then each individual neuron must (on average) be involved in the computation of many features. Prior work on superposition (Elhage et al. 2022b predominantly (but not entirely) focused on linear superposition, and we understand neuron superposition less well. As another example of computational superposition, preliminary findings suggest attention layers also employ superposition between heads to implement a larger number of skip-trigram circuits than would naively be possible (Jermyn et al. 2023
Conceptually, computational superposition is more complex phenomena. To perform superposition, the model must tradc-off the benefit of representing more features against the costs of interference. With representational superposition the interference between features is just given by their dot product, and by embedding features as almost orthogonal vectors we can achieve low interference while representing many more features. But computational superposition involves nonlincarities (elementwise GELUs in the neuron case), which introduce significantly more interference in ways we do not fully understand.
A.10 How can you conclude you found superposition as opposed to simply non-basis aligned features?
We think that our findings of superposition rest on two sub-claims: (1) that there cxist more features than neurons to represent them and (2) that the neuron basis is meaningful, such that models are incentivized to align features with neurons if there are fewer features than neurons, and that superposition is implemented as a sparse combination of neurons.
We think claim (1) is evidently true, given the extremely long tail of features in internet text in addition to all the possible meaningful n-grams and other repeated patterns that model would want to memorize. In practice, models can detect and respond appropriately to a wide array of compound words and facts, seemingly far more so than the size of their vocabulary or number of neurons. As an example, just take the case of names of people. There are likely more people that GPT-3 knows about than it has neurons (5 million). A first and last name gets split into at Icast two tokens, but with most surnames being tokenized further. We think it would be impossible to fit all of the possible features of these people as just linear combinations of the token embeddings, therefore requiring some nonlinearity to detect the presence of a sequence of tokens corresponding to name, and adding back the appropriate information about this person.
To further support claim (2), we run an experiment on the compound words datasel comparing the classification performance of the top 50 individual neurons in a layer to the top 50 dimensions in a random basis. In particular, for both the standard neuron basis, and the activation dataset when multiplied by a random d d Gaussian matrix, we train 1-sparse probes for the 100 neurons with the largest class mean difference, and report the out of sample F1-score for the top 50 (averaged over 5 such random bases). Our results
24

---

Published in Transactions on Machine Learning Research (11/2023)
in Figure 13 demonstrate that individual neurons are more predictive than random linear combinations of neurons, which implies that the neuron basis is meaningful.
Although there are usually 1-5 neurons with notably higher Fl score than the rest, the full top 50 neurons still maintain high F1 scores. This seems consistent with results from Section 8 of (Elhage et al., 2022b), where there exists a main neuron per feature, with a longer tail of less important neurons. Intuitively, just. achieving linear separability is unlikely to be sufficient for overcoming interference. To minimize interference models likely want to maximize the separation between potentially interfering features, implying that the potential number of neurons in superposition may be much higher than the minimal number required to achieve near perfoct classification performance. If the largest activation negative examples is close to the smallest activation on positive examples, then using the represented feature for downstream computation will introduce significant noise. For a rough approximation, in Figure [4 we measure the logistic test loss, rather than F1 score, of k-sparse probes for a range of values of k. Again, we see that probes trained in the neuron basis generally have lower loss than those trained in a random basis. Perhaps more interestingly, there appears to be two regimes governing the loss with respect to sparsity, one with power law scaling, and then a kink with no returns to using additional neurons. It is plausible this kink gives an indication of the "truc" sparsity used to implement features in superposition.
Finally, we note that these results likely understate the basis aligmment, because they are with respect to an "easy" classification task: just distinguishing bigrams XY from bigrams XZ or WY rather than all possible n-grams, and that there likely exist many confounding contextual correlates that improve the performance of the random basis dimensions. Designing more careful experiments to untangle these different effects is an important area. for future work.
A.11 Should we ever expect to find monosemantic neurons?
Monosemantic neurons have been convincingly demonstrated in image classification models (Cammarata et al. 2021), but we are not aware of work rigorously showing monosemanticity of transformer language model neurons. So on an empirical front, this is an open question. It has been shown in toy models (Elhage el al. 2022b) that in the presence of significant differences in feature importance (i.e., expected reduction in loss from representing that feature) and for contimuous features that the most important features will have dedicated monoscmantic neurons. Textual fcatures vary significantly in importance (e.g., detecting Python code is far more important than knowing some niche fact) suggesting that crucial features may get their own neuron (e.g. our context neurons). However, for important binary features this is less clear. Further, even important and highly prevalent features likely have rare features that they are anti-correlated with, and may be represented in superposition with these. Rare features are a case that may be particularly hard to detect.
A.12 Which sparse probing method should use?
It depends. If your experiments must be extremely fast or scalable, then simply doing maximum mean difference is likely best. If you are less runtime sensitive and want to sweep over a large range of k, then adaptive thresholding is most appropriate. Finally, if you require any sort of formal guarantees then you should use optimal sparse probing, with the caveat that this requires potentially substantial compute time.
A.13 Why think of features directions?
This is known as a linear, decomposable representation (Elhage al. 2022b)
the model's activations may be decomposed into independently varying features, and these features correspond to directions in activation space. The intuition behind this is that the primary capability of models is doing linear algebra-addittion addition and matrix multiplication, which further breaks down into addition, scalar multiplication, and projecting onto specific directions. Given these capabilities, it is especially natural for a model to represent features as directions: if later layer wants to access a feature it can project onto that feature's direction, a neuron can casily access and combine multiple fcatures, features can vary independently, and the component in that feature direction represents the strength of that feature. If a model used some more complex and non-linear representation then it would need to dedicate downstream parameters to decoding these, costly
25

---

Published in Transactions on Machine Learning Research (11/2023)
endeavour. Previous findings of monosemantic neurons in image models Camamaraaa et al. 2021 and reverse-engineering toy models on mathematical Iasks(Nanda et al. 2023 Chughtai et al. 2023 have found linear features, however the empirical evidence base here is slim. Mechanistic interpretability is still a young and pre-paradigmatic field, and we see gaining empirical evidence for foundational assumptions like this as a key area of future work. If the fealures-asdirections hypothesis were false, the lask of reverse-engineeriing models would seem far more daunting: models are extremely high dimensional objects, and we must be able to decompose them somehow into independently meaningful units in order evade the curse of dimensionality and understand them.
A naive way to test this hypothesis is by using linear and non-lincar probes (e.g., one hidden layer MLP) to extract features: if non-linear probes work and linear probes do not, then this is evidence that those features are represented non-linearly. However, this can be illusory, as the model may instead linearly represent simpler features and the non-linear probe itself does the computation to find the more complex feature. In the extreme, we could imagine having our 'probe' be GPT-3 trained on the token embeddings- this can surely probe for any feature, represented not! This is concern even with linear probes, eg a model may represent whether shape is red and whether it is a triangle as directions in space, and linear probe on the sum of these directions may seem to indicate an 'is a red triangle' feature.
There was a recent. natural experiment in the literature supporting the features as direction hypothesis. Li et al. 2022) trained a model to predict the next move in the board game Othello, and were able to probe for an emergent representation of the board state. They could only find this representation with non-linear probes, not linear probes, yet could validate this representation with causal interventions. However, follow-up work (Nanda 2023b) showed that there was in fact a linear representation of the board state, but that the model predicted both black and white moves, it was in terms of whether a cell had color equal to the current player or to the current opponent. We take results like this as promising additional evidence that the features-ss-directions hypothesis has real predictive power.
A.14 What are possible conceptual models for what MLP layers are actually doing?
Perhaps the main model, as hypothesized and studied in (Geva et al. 20201, is that of a key-value store (note that the key and value terms here are unrelated to those used in the attention layers). A key-value store can be used in tasks such as memorization and factual recall, where the model wants to look up a fact, such as the location of the Fiffel Tower. If the residual stream contains both the features "Fiffel" and 'Tower," then a neuron's key could be the sum of both of those features, and the value could be the feature indicating that the current token is in Paris.
The non-linearity of the MLP might serve several possible roles, such as thresholding, where only if the 'Liffel' and 'Tower' features are present, do we conclude that it is in Paris. It may also perform saturation: for features where the model must accunulate many small pieces of evidence but the underlying ground truth is bimodal, such as "is Python code', the model may want to have the same thresholding Additionally, some form of translation memory management could be necessary. If features like "is in Paris' are only used in middle layers, it would not be in the model's interests to produce the "is in Paris" feature earlier on for fear of causing interference. Another possible role is that of activation range management. More broadly, neurons seem well-suited to representing Boolean functions such as AND, which is inherent in studying simple structures such as bigrams and trigrams. An AND can be implemented by stating that if the features A and B are both present. with size 1, then only if their sum is present with size does the neuron fire, and otherwise it set to zero (c.g., x, RcLU(x + y y-yll 1)).
Another role that neurons might play is that of disambignation. There are certain tokens which occur in very different contexts, but where the same token arises, such as "Die" in English corresponding to 'death," / 'dice,' or 'Dic' as common token in languages such as German or Dutch or Afrikaans. Neurons that disambiguate "Die" have been observed in the literature (Elhage et al. 2022a Coenen et al. 2019). In some sense, this can be thought of as having a more general "this is in Dutch' neuron producing an "is in Dutch' feature, and having the neuron having the 'Die in Dutch' feature be the lincar combination of the "is in Dutch' feature and the "Die" token is present feature. Depending on perspective, this could be considered a superposed, or distributed representation of the 'Die in Dutch' feature.
26

---

Published in Transactions on Machine Learning Research (11/2023)
One useful model for early layer MLPs is detokenization (Elhage et al. 2022a]: taking the raw imput of tokens and converting it to more useful semantic concepts, la the n-gram detectors we study. These are analogous to sensory newrons in biological systems, or gabor filters and high-low frequency detectors of carly vision layers (Mehrotra et al., 1992: Olah et al. 2020a): neurons that take the raw sensory input and convert it to a more useful form. A non-obvious consequence of this that detokenization neurons may simultaneously serve as key-value stores. An "Eiffel Tower' n-gram detector may have an 'is Paris' feature that it outputs, in addition to the "is Eiffel Tower' feature. From another perspective, assuming the representations are linear, it could be the case that the 'is in Paris" feature is nothing more than the average of features like "is Eiffel Tower", 'is the Louvre', etc.
The converse model for late layer MLPs is retokenization (Elhage et al. 2022a):: converting the features represented in some semantic space into the concrete output tokens, analogous to motor neurons. This is likely to be particularly important with multi-token words: if the model completes a prompt like "A famous landmark in Paris is' with Eiffel Tower', must in fact output 4 separate tokens (" E", """"" "'i'"'''''""" 'ell' Tower"), and this is natural to be performed by late layer MLPs on the current and subsequent tokens. Though, a notable difficulty in detecting these neurons is distinguishing between neurons converting an 'output Eiffel Tower' feature on the is" token to concrete outputs, from neurons that detect and continue the n-gram of Eiffel Tower", but only figure it out after the E' and 'iff'. This is a similar to difficulties in evaluating model performance when predicting multi-token outputs.
One interesting application of neurons in transformers that does not occur with simpler models such as convolutional networks neurons engaging with and enhancing the function of attention heads. Models with single attention-only layer exhibit functions such as skip trigrams, but where this has bugs (Flhage et al. 202Ip, a neuron could be used to fix this. They could also compose with important attention head circuits, such as induction heads, to help clarify edge cases. Induction heads detect and continue repeated text, but if a token arises in multiple different contexts in the prior text with different subsequent tokens, we may want the induction heads to not activate at all. This may be casier with clarifying neurons than with an allenseon-focuued circuit.
Many of our intuitions here involving thinking of GELU as essentially the same activation as a ReLU, while know that GELUs perform better in practic e e(Hendrycks & Gimpel 2016). Though this may be down to ease of optimisation, we speculate that the "bowl" of the GELU, in particular the fact that it is close to a quadratic when close to the original, may allow the model to express more complex non non-linear functions. We would be particularly excited to see future work exhibiting such case studies.
A.15 Did you train your probes on pre-GELU or post-GELU activations?
We train our probes on the post-GELU neuron activations (i.e., CELU(W...)) because there's a linear map from these to residual stream, so linear combinations of them are meaningful to the network, in a way that pre-GELU linear combinations are not. We often choose to plot histograms of pre-GELU activations for individual neurons for clarity, to better show the negative range, and avoid an enormous spike around zero from all highly negative activations.
A.16 Did you have any negative results?
Yes. We had two types of feature datasets which did not appear to be spansely represented: falsehoods in the counterfact datasct or the occurrence of particular suffixes and prefixes, though both attained fairly high accuracy with a dense probe. For prefixes and sulfixes, since these are purely a properly of single tokens, we believe these aren't likely to get dedicated neuron representation since the feature already exists within the token embedding.
For counterfact, a dataset of true and false factual completions, we hypothesized there might exist some dedicated "falsehood" neuron. However, sparse probes performed quite poorly, while increasing continued to yield performance gains until about k = 1024 where we achieved 90% accuracy on the best layer. Our interpretation of this is that the probe is picking up some pattern of confusion or dissonance, associated with unexpected completion, rather than true falsehood or truth feature.
27

---

Published in Transactions on Machine Learning Research (11/2023)
Our last negative result was in attempting to causally implicate the factual neurons. That is, run ablations that degrade the performance of few shot prompting based classification of people's gender, occupation, and whether they are alive or not. However, ablating individual neurons seemed to have very limited effect, indicating that either the "fact" was computed previously, and these neurons are simply responding to this lact, or that there is some amount of ensembling or redundancy (Dalvi al. 2020) that is robust. to the deletion of single neuron.
A.17 What do you mean by the model "wants"?
Throughout this FAQ, sometimes use anthropomorphie language, such that the model 'wants' to minimise interference. We do not, of course, think that these small models are capable of having wants intentionality in the sense of a human! In fact, it would be more accurate to say that stochastic gradient descent selects for models with low expected loss, and that model weights that minimise interference seem likely to have lower expected loss. But we believe that notions like 'want" can be valuable intuition pumps to reason about what algorithms may be expressed in a model's weights, so long as the reader careful keep in mind the limits and potential illusions of anthropomorphism.
B Experimental Details
B.1
Models
We study the Pythia model suited (Biderman et al. 2023) which was trained on The Pile (Gao el al. 2020 with each model trained for the same number of steps with an identical data ordering. These models follow a fairly standard architecture, but parallel attention and rotary positional embeddings. See Table for the architectural parameters of each model studied.
Model
Maayers
dmodel
Theads
dhead
Total Number of Neurons
Pythia 70M
6
512
8
64
12288
Pythia 160M
12
768
12
64
36864
Pythia 410M
24
1024
16
64
98304
Pythia 1B
16
2048
8
256
131072
Pythia 1.4B
24
2048
16
128
196608
Pythia 2.8B
32
2560
32
80
327680
Pythia 6.9B
32
4096
32
128
524288
Table 1: Hyperparameters of Pythia models studied,
B.2 Datasets
Table contains summary statistics of all of our probing datasets in addition to the full list of features each. We briefly describe the design, preprocessing, and motivation of each one.
Natural Language We used both the raw text and the labels from the EuroParl subset of the pile, which contain large number of parliamentary proceedings in many different languages. Many documents were quite a bit longer than context length, for each, took random contiguous sequence from the document. This choice was also made to minimize any context clues from the beginning of the document about the country speaking. For probing, we performed mean-aggregation. That is, computing the activation for all non-padding tokens the sequence and using the average neuron activations as the representation for the sequence to be used for probing- One problem with this datasct that all of the sequences are similarly styled, and about similar topics, hence our probes could be picking up on a particular style or topic feature rather than true language feature.
28

---

Published in Transactions on Machine Learning Research (11/2023)
Dataset
Sequences
neta
Non-pad tokens
Source
Pos ratio
Total Features
part-of-speech
1438
512
281044
EWT
0.20
16
dependencies
1438
512
281044
EWT
0.20
29
morphology
1438
512
281044
FWT
0.20
22
code language
5397
512
2757867
pile-githul
0.11
natural language
28084
512
14350924
pilc-curoparl
0.11
text. feat ures data subset
10000
256
2248714
pile-test-all
0.26
8413
512
4299043
pile-test-all
0.11
compound words
167959
24
4031016
pile-test-all
0.20
21
latex
4486
1024
1889178
pile-arxiv
0.26
14
wikidata sex or gender
6000
128
688085
pile-test-all
0.50
2
wikidata is alive
6000
128
688179
pile-test-all
0.50
2
wikidata occupation
6000
128
677512
pile-test-all
0.17
wikidata athlete
5000
128
575350
pile-test-all
0.20
wikidata political party
3000
128
337755
pilc-test-all
0.50
Table 2: Summary statistics of probing datasets studied.
Data Subset Similar to the above, we randomly selected several thousand sequences from cach data subset of the Pile lest set, where the labels were implied. We also used the same random sub-sequencing and mean aggregation.
Programming Languages For our programming language dataset, we took all of the github subset from the Pile test set. We then used a code recognition package classify the type of code, and only include the source files with prediction of over 90% confidence. A challenge with code is that it includes many tokens which aren't code (licenses, copywrite, comments, etc.). As very coarse cleaning attempt, ignore the first 50 tokens in the sequence when applying mean aggregation, to avoid beginning of file boilerplate. However, for some languages like HTML or XML that likely contain substantial plain text, this does little lo help (and hence we see the lowest accuracy on HTML and XML).
Compound Words For our compound words dataset, we computed the top thousand alphabetical bigrams XY over the pile test set. We then filtered for bigrams where P(X z y) and P(Y y/ = x) were below 0.3 to make the prediction task less trivial. We then manually filtered to bigrams with the property that the first and second word together means something quite distinct from either word separately. For the probing dataset, we then included 2000 short (24 tokens) sequences from The pile ending with the bigram XY, 4000 examples ending with XW, W and 4000 examples ending with ZY, Z We then probe on the activations of the last token of every sequence.
Latex Features For our latex feature dataset, we included the first 1024 tokens of all documents from the ArXiV subset of the Pile lest set which contained at least 1024 tokens. We mostly relied on regular expressions to extract features, though used simple finite state machine to parse the math text. For each of the features involving any sort "containment," (which is most them), we used the policy that all tokens within the containment counted as being part of the feature, but any tokens defining the containment for not. For instance for is_ ss_subscript, the " "_(" ( token is not included, but anything within the braces is. For our positive class, randomly sample tokens from the set of all positive tokens. For features with natural make the negative class be occupied by half of
complement (c.g., i numerator and is denominator) tokens within the complement, and half random tokens.
Text Features For our plain-text feature dataset, we similarly extracted features using regular expressions, but just applied to the raw token strings. Therefore, for our text features, fcature could be correctly predicted simply by partitioning the tokens into two classes, with no additional context. We sampled tokens randomly from subset of the Pile test set.
29

---

Published in Transactions on Machine Learning Research (11/2023)
Dataset
Features
part of speech
AUX, ADP, VERB, ADJ, X, CCONJ, PROPN, NOUN, INTJ, SYM, PRON, DET, SCONJ, ADV, PUNC, NUM
dependencies
aux:pass, acl:relel, nsubj, xcomp, flat, ec, mark, acl, ccomp, appos, root, nood:poss, aux, amod, nsubj:pass, obj, obl, det, advmod, punct, parataxis, conj, case, list, advel, cop, compound, nummod, umod eos _True Person 2, Gender Fem, VerbForm Inf, PronTType Dem, Gen- der Masc, first eos True, Gender Neut, VerbForm Part, NumType Card, PronType Intp,__n,,___ PronType_ Prs, Person 3, Past, Number Plur, Pron- Type_Are Voice Pass, PronType. Rel, VerbForm_Goon Mood Imp, Person_]. VerbForm Fin
morphology
code language
Python, XML, Java, C++, HTML, C, Go, PHP, JavaScript
natural language text features
Swedish, Portuguese, German, English, French, Spanish, Greek, Italian, Dutch leading leading_capital, no le_ddladnggsppp space space_and and lode__oooappppa all digits, io_not___ _ot_ascii, has leading space, contains all whitespace, all capitals, is not alphanumeric, contains _whitsspcce,, contains nantaiss_appitll, contains digit github, pubmed abstracts, stack exchange, wikipedia, freelaw, hackernews, arxiv, enron, uspto mental-health, magnetic-fiedd, trial-court, control-group, human-rights, north- america, clinical-trial high-school, third-party, public-health, cell-lines, living- room, sccond-derivative, credit-card, social-media, primo-factors, foderal- government, pocial-securityy blood-pressure, gene-expressee side-effects is superscript, is innline math, is s__ttlle, is subscript, is reference, is denominator, is author, is numerator, is display math, is math, is abstract, is frac is female, is male truc, false
data subset
compound words
latex
wikidata sex or gender wikidata is alive wikidata occupation
is actor, athlete, is journalist, is politician, researcher, is singer is association footballl player, is bascball pasceall_player, is basketball playyer is icc hec__hocky__ppp player is democratic party, is republican party
wikidata athlete
is american football player,
wikidata. political party
Table 3: All feature within the feature collections.
Linguistic Features For our linguistic features, we use the text and labels from the well known Penn Treebank Corpus. The dataset is provided at a sentence level, so we first merge sentences from the same documents, and then perform a token alignment and character mapping to adapt the dataset labels to the specific tokenization of the Pythia models. For features that apply to all tokens (like part-of-specch or dependency relations), we have ristrin on the tokens sampled for the negative class. For features restricted to tokens of specific types (e.g. verb tense), restrict the negative examples to be from the same token class (c.g. only verbs).
Wikidata Features To create a Wikidata feature dataset for a specific property (e.g. gender, occupation), search through JSON dump of all Wikidata entities and compile table containing the names and relevant property value for each person with data on the chosen property. When there multiple Wikidata entities with the same name, we choose the entity with more total Wikidata propertics defined as hcuristic for relevance. We then search the text of the test set of The Pile for instances of these names and filter out all examples with names that occur only once in the dataset. We tokenize each matching string and truncate to 128 tokens such that the tokenized name is the end (if there are sufficient preceding tokens, otherwise we pad the tokenized string length 128). We further filter the tokenized examples ensure that each name occurs more than three times in each feature dataset. This helps us find general knowledge neurons that.
30

---

Published in Transactions on Machine Learning Research (11/2023)
aren't overly impacted by individual people. Then when probing, consider only the activations for the tokens of the surname for each example to reduce the effect of correlations between first names and certain properties (such as gender).
B.3 Feature Selection Methods
Throughout, assuine we have dataset of token activations for tokens and d neurons X E Rx and a vector of labels y 1-1, 1)". We refer to the set of tokens in the positive class P and the set of tokens in the negative class N. Below we describe the different feature selection methods in more detail. With the exception of optimal sparse probing and adaptive thresholding, we can think of these approaches as scoring algorithms that can be used to rank the different neurons by importance. Then, to get a h sparse classifier, in all cases one just selects the k neurons with highest score
Mean Difference The score given by the average mean difference between classes for each neuron. That is, for neuron j we have 8) Eiep - INT EiEN Xij-
Mutual Information We use the algorithm presented by Ross 2014), to compute the mutual information between continuous data (in this case each neuron independently) and a discrete target relying on a nearest- neighbor approximation. The score for each neuron then just the estimated mutual information.
L1 regularized We train a dense logistic regression probe on the activations with L₁ regularization. The score for each neuron is the absolute value of the corresponding coefficient of the dense probe.
F-statistic We utilize a one-way ANOVA test to evaluate the relationship between the continuous data of each neuron independently and a discrete larget. The F-statistic is calculated by partitioning the total variability into between-grow group and within-group components, represented by sum of squares between groups (SSB) and sum of squares within groups (SSW), respectivcly. The ratio of the mcan square between groups (MSB) to the mean square within groups (MSW) is computed as the F-statistic. A higher F-statistic indicates a larger discrepancy between group means and is used as the score for each neuron.
Optimal Sparse Probing We train a cardinality- cow.tiriiio support vector machine (SVM) with hinge- loss trained to provable optimality using the cutting planes technique described in Bertsimas al. 2021 The selected coefficients are simply those selected by the classifier.
Adaptive Thresholding Starting from k = dmip (or some large k on filtered set of neurons from a different hcuristic), we train series of logistic regression probes with clasticnct regularization (that is combined l and 12 regularization). Given a schedule for reducing k, at each step 4, we lake the kN < h neurons with highest absolute coefficient magnitude and retrain a logistic regression probe on these neurons. This process enables us to sweep through many values of l while leveraging past computation to achieve better feature selection.
B.4 Experimental Procedure
Here we provide additional details on the experimental procedure regarding both the feature selection experiments and the sparsity with scale experiments within Section 4.
Perhaps most significantly, for all experiments we preceded probe training with a heuristic filtering step, where only trained probes on the top 1024 neurons, as judged by mean difference, for each combination of layer, feature, and model. This was done for computational reasons, as computing the mutual information, training dense probes, and optimal sparse probing methods are very expensive to run on hundreds of features for many layers containing in excess of 10,000 neurons. For optimal sparse probing, we only used the top 50, and set a timeout of 60 seconds.
All classifiers were trained with balanced class weights account for class imbalances in the dataset. Finally, selected hyperparameters using small subset of feature and model combinations, and then used the best.
31

---

Published in Transactions on Machine Learning Research (11/2023)
Method
Sparsity
Runtime (s)
1
3
4
7
8
Random
0.573
0.629
0.663
0.690
0.712
0.729
0.743
0.755
0.257
MMD
0.832
0.857
0.867
0.874
0.880
0.884
0.887
0.890
0.292
FS
0.832
0.856
0.867
0.874
0.880
0.884
0.887
0.890
0.303
LR 0.818
0.853
0.871
0.882
0.890
0.897
0.902
0.906
7.796
MI
0.831
0.851
0.863
0.869
0.876
0.880
0.884
0.886
32.241
OSP
0.831
0.861
0.876
0.884
0.891
0.896
0.899
0.902
53.028
Table 4: Comparison of sparse feature selection methods. Results are for the out-of-sample F1 score averaged for the best scoring layer for each combination of model and feature.
performing hyperparameters for all other experiments, as doing seperate tuning for every trial would be too computationally expensive.
B.5 Feature Selection Results
For cach combination of model, feature, and layer described above, we train a sparse probe for values of k == 1,...., 8 using the features selected by the aforement ioned subset selection methods. In Table J we report the out-of-sample F1 score of each method averaged across models and features, and for the maximum across layers since features tend to be only represented in specific layers.
While our results are averaging over many trials, several points stand out. The top methods are always within 1% of cach other, with method strictly dominating the others. Although OSP comes with optimality guarantees, we set one minute timeout, which is frequently reached especially for larger values of k. Moreover, this provable optimality is with respect to the specific level of l2 regularization which often needs to be turned fairly high to speed up convergence (Bertsimas et al. 2021 Nevertheless, it is impressive that such fast and simple heuristics as the mean difference are comparable to far more sophisticated approaches, with even random neurons containing substantial information of the target task. This suggests that perhaps the best. design is two-stage approach where very fast heuristics are used to find relevant neurons and layers, and optimal methods are used to improve and verify the solution.
32

---

Published in Transactions on Machine Learning Research (11/2023)
os
1 cos
Figure 6: Intuition for representing 5 features in 2 dimensions in superposition.
C
Superposition Construction
Consider a set of binary and mutually exclusive features (i.e., one-hot vectors like n-grams),, that we want to embed in two dimensions, and later losslessly recover. Specifically, given a one hot vector x R", we seek W e gax2 such that
== ReLLL(WW**+
where ReLU(x) max(x, 0). By embedding each point to be equally spaced around a circle of radius a (i.e., w; a cos(2ri/n), sin(2ri/n)) we get that
(WWT ex); = a² cos 2n - J
where a = ||/: To recover the initial one-hot representation, we require
12TT
2n
a² 1
(1-co
1
and b= -a cos
12
Analyzing our proxy metric, bl[w[|2
(cos(2n/n)-3 monotonically decreases for n with n > 2.
cos /n)
See (Elhage et al. 2022b) for a far more extensive set of experiments and results on related topics.
Note that this construction refers to residual stream superposition and not neuron superposition since we are assuming we have exactly as many neurons as features. However, we expect the same basic motif to clean up interference to hold for the case with fewer neurons than features. Morcover, it is plansible incrcased levels of residual stream superposition associated with increased levels of neuron superposition. Hence believe this construction, and therefore the presence of large weight norms and negative biases, to be suggestive evidence of neuron superposition.
Of course, it is possible factors other than superposition explain this observed weight pattern. Additionally, there might multiple mechanisms for computing features in superposition (likely those that are not as binary or mutually exclusive as n-grams) that do not strongly affect the distributions of weights and biases. We would be very excited about future work that more carefully explained the source of these weight statistics, that studied other concrete mechanisms motifs enabling effective computation in superposition.
33

---

Published in Transactions on Machine Learning Research (11/2023)
D
Additional Results
compound words
data subset
latex features
morphological features
mean score for best layer
0.9
0.9
0.8
0.8
0.8
0.7
0.7
0.6
0.7
0.6
0.6
0.4
0.5
0.5
0.4
0.5
0.4
100
101
10a
100
101
102
10a
101
102
10
101
10a
programming languages
plain text features
part-of-speech
factual features
mean F1 score for best layer
1.0
0.9
0.8
0.7
0.0
0.6
0.6
0.5
0.4
0.5
0.4
0.4
10a
10a
10a
10a
10a
10a
10a
10a
10a
10a
10a
10a
probe sparsity
probe sparsity
probe sparsity
probe sparsity
model (baseline)
410M
70M
--- 160M
410M
--- 18
1.48
2.88
6.98
Figure 7: Difference between a k-sparse classifier using sparse features selection versus random bascline. Specifically, the F1 score of training a classifier with & coefficients on the full n x d activation dataset when multiplied by a random d x orthogonal matrix.
34

---

Published in Transactions on Machine Learning Research (11/2023)
Distribution of || by layer
0.2
||Wal|200
1
[|Wall,0p
-0.2
-0.4
layer (pythia-70m)
layer (pythia-160m)
0.0
-0.2
10
12
13
14
15
16
17
18
19
20
21
22
23
layer (pythie-410m)
0.00
[[Wallabp
-0.75
layer (pythia-1b)
[Wallbbb
layer (pythia 1.44b
0.2
0.0
[[aalibe
-0.4
11
12
14
15
17
18
20
21
23
24
25
26
27
28
29
31
layer (pythia-2.80)
0.0
[Walllbe
-1.0
-1.5
12
14
15
17
18
20
21
23
26
28
29
(pythia-6.9b)
Figure 8: Distribution of input weight norms and biases for all Pythia models studied.
35

---

Published in Transactions on Machine Learning Research (11/2023)
Superposition prime-facten feature across neurens of same layer
Sigram
prime factors
(log scalo)
other
prime
pre activation L2,N2950 SSperpooiiioon human-rights feature across neurons saime layer
pro actvation N2.N199
pro octivatioon L2.N352
pre activstion L2. 22.00000
pre activation Lz NS717
post: activation sum
10
Bigram
human rights
X rights
10"-
(log
pre-activation L1.N4408 Superposition security across neurons layer
prevac ple-activation
pre-activation N4830
p00 mwwacciivatin 11.N7445
posi-activation sum
10
bigram
social security
X security
count (og scate)
10
10*
107-
101
-5
10
pre-ectivation Superposition sederal-govermment
pre-actiaatio
pre-activetion layer
pre
L2
post-activation sum
10a
bigram
federal govermmet
gevernment federal
10
101
(log
100
10a
10a
-5.0
-2.5
0.0
2.5
-500
-2.5
0.0
2.5
-5.0
-2.
0.0
2.5
-5.0
-2.
0.0
2.5
0
5
rre-aacivation
pre-activation N3.N2886
L.3N3451
pre-activstin L3.N4728
artivaciiation L3.07947
acttaactivtion
Superposit second-derivaaivve feature across neurons layer
second derivative other
derivative second Y
10a-
10*
count (log
10*
10a
10'
-5
-5
0
pre-ectivsiion L2.N1-27
pre-ectivation N2.10
presectivation L2.N3030
pre-ectivetion N3969
pre-ectivation N6646
post-aciivation
Snperposition of social-media feature across neurons of same layer
bigram
10a
other
social Y
20
(og
10a
107
101-
-5
-5.0
-2.5
0.0
25
-3 actvvcctiooion L3.N2471
-5
10 cost-activation sum
15
eel--cctiaaion L3 N1D42
pre-activation L3 N1230
pre-activetion L3.N1463
pre activation L3 N4656
10
Superposition of blood-pressure feature across neurons of same layer
bigram
other
blood Y
(bgg
-5 Ine-activation 12.177
-5
-5
pre-activation N663
pre-activation 2,092
pre-activation NS639
pre-activation AS820
post-artivation sum
Figure 9: More examples of superposition of compound words in Pythia- 1B
36

---

Published in Transactions on Machine Learning Research (11/2023)
Superposition north annerica feature across neurons of same layer
bigram
america
Xamerica north
scalo)
other
pre activation L1 N2124
pro art-vcctvation L1 Na567
pre setisccttaion L1.N5457 same layer
pre activvatoon L. N6675
pro activation L1.N9092
post activation sum bigram
Superposition living-room feature across neurons
10
living
X
105
(log
0.0 2.5
pre-aclivation N166
peeaccivation pre-actication L3.N3558
pre-activation NS671
prevccivattio L3. L3.N7053
posi-activaaion sum
Superposition gene expression acress neurens layer
10
bigram
gene expression
expression gene Y
count (og scate)
10%
10*
107-
101-
-5.0
-25
-3.0
-2.
2.5
0.0
pre-activtton N901 pre-activtton
pre-activetion layer
pre-activation
post ecsivecctivtion sum
Superppoitiin mental-health
10a
bigram mental healh other
health mental
10
201
(log
100
10a
10a
-5
-5
-3.0
-25
0.0
25
-5
-5.0 -2.5
0.0
2.5
0.0
2.5
5.0
activatitaan N3652
pre-activation LL.N4559
pro-activation L1N5996
pre-activation L1.88663
post-activation
Superpasition trial-court feature ecross neurons
same
blgram
thia court other
x court brai
(log scale!
10F
10*
30"
10"
101
-5.0
-2.5
0.0
2.5
-5.0
-2.5
0.0
2.5
-5.0 -2.5
00
2.5
0.0
2.5
5.0
7.5
pre-sctivation N1.00
pre-ectivation 12 N2296
pre-sctivation 12.N3927
presectivation L2 N4006
pre-ectivation L2,N6312
posteccivvtionn
SSpeerposition of public-healtn feature across neurons of same layer
bigrem
publi other
10a
public
20*
10a
107
101
-5
-3
-5 pre pre-activation N2.84392
-5
10
pre pre-actvation L2 M293
pre-activation L2 N2289
pre-activation L2.N3329
pre ant--ctivaion L2.N3879
post activatttoooon sum
Superposition of highh-school feature across neurons of same layer
bigram
scale!
other
high Y
-5 pre-acrivation 2.N7085
5.0 post-artivation sum
2.5
pre-ac Inveactivation 12 462
pre-activation N4793
pre-activation 12.N5886
pre-activation MK580
Figure 10: More examples of superposition of compound words in Pythia-
37

---

Published in Transactions on Machine Learning Research (11/2023)
Superposition cell-lines feature across neurons of same layer
Bigram
call lines
count (log scate)
other
0.0
-2.5
-2.5
0.0
pre-activation L3,100
pre-activatian
pre-activation layer
pre-activation
pre-activation
post-activation
10
Superpositoon credit-card
acress neurons
bipram
credit card
x card
10
30*
(log
10a
10
10a
-5
-5
-5.0 -2.5
0.0 2.5
-5 activation NS967
-5
0
activation L2.N1039
activation N1177
activation L2.N2537
activation 12.N7954
activation
10
Superpositoon side-effects
feature across neurons
same layer
bigram
10x
other
side
20*
307
-5.0
-25
2.5
-5.0
-2.5
0.0
2.5
-5.0
-2.5
2.5
-3 pre-activetion LL NB04/
-5
pre-activetion L1.N5/
pre-activetion LLN22/6
prevactivation L1.M4330
pre pre-activetion L1.N6266
post cost-activvtion sum
Superpositan on of control-group feature across neurons of same layer
bigram
control other
scate
control
-5
pre 12.N3455
pre-activation L2.M4434
pre-activation 12.N5234
pre-activation 12.N5709
pre pre-activation L2.N6175
post-activne nost-activation
Superpositoon clinical-trials ecross neurons layer
10
bigrom
clinicl trials
x triels clinical
(log scate!
10%
201
2.0
10
10
-5
0
-5.0
-2.5
0.0
2.5
-5
-5 activation 12.N7094
-5
0.0
2.5
5.0
actsvatiiionno Superposit magnetic-field feature across neurons of same layer
pre-activation L2.N3502
L2,N3712
activation L2.N7889
cost-activatio
10
bigram
magnetic other
10"
magnetic
30
llg
10
10
-5.0
-2.5
2.5
-5.0
-2.5
0.0
2.5
-2
-3.0
-2.
-3.0
-2.3
0.0
2.5
0.0
2.5 post-activation sum
pre-ectivetion 1.895
pre-activerion L1 116188
pre-ectivation L1.M6053
pre-activetion L1 L1156135
pre-acbvetion L1.66647
Superpasit.an third third-party feature across neurons
same layer
bigram
(lgg scaee
other
2.5
5.0
pre-octivation pre-activarion
pre-activation
pre-activation 12.45794
pre-activation 12.N7629
post-activation
Figure 11: More examples superposition of compound words in Pythia-
38

---

Published in Transactions on Machine Learning Research (11/2023)
Pythia 70m Layer Neuron
'stionally
bayens
Incing"
Pythia Layer
'ine'
Tablees
linvel
'ine
'iinn"
'Line'
Yeerne!
'ordf
'ounded
Pythia 410m
Layer
Neuron
presstivation
Isttrs
cycle
ledy
letee
potential
"number'
numbers
Pythia 1b Layer 1- Neuron
'NT'
pasitioring'
'place'
lomatic!
10e
Pythia 1.4b Layer
Neuron
other
'oug'
'East
causes
'ast'
Tune"
'ay
Tale'
'Imaa
CCpp'
HHigh
Pythia 2.8b Layer 1 Neuron 0
propi!
habileation
inttenssted
defect
Commussoo' "chrrccerized
"chambers
"olated"
Pythia Layer
pio aetaattttoonn
'sync'
'utf'
'trensmmte
'0'
'panaliel'
'Color'
'emiitingg
'agnet'
'aaching'
Figure 12: More examples of n-gram polysemanticity in layer neuron 0 of all Pythia models studied.
39

---

Published in Transactions on Machine Learning Research (11/2023)
byer
(reuren basei
2 preuron bas6)) prentsmm heeig
innurrn basis (tarsine twesn)
crearen basial Erenatoo hwags)
(rwndem hargs)
Bood pressure
cell- ires
clinical-trials
0.55
0.8
0.00
030
0.7
0.0
0.65
tonzrolgroup
credit-card
federal govermment
0.95
0.9
0.90
0.8
6.70
gene on
High school
human-rights
0.0
ost-chameete
6.9
0.6
0.5
lving-room
magneticfiild
mental-health
1.0
0.0
0.9
0.3
08
0.75
0.7
0.70
0.6
north-america
prime-"actors
public-health
1.3
1.00
1.0
6.9
0.95
0.9
oubotaamee
0.5
0.90
0.0
0.7
0.5
0.7
6.5
0.80
second derivative
side effects
3.3
1.00
1.00
0.9
0.4
0.00
0.3
8.85
0.80
0.7
0.5
075
0.70
005
social security
third
trial-court
1.00
1.0
100
0.95
0.9
0.80
out-ifamalle
0.6
10
70
30
40
50
30 70
30
30
70
30
nzuron rank
neuro's rank
nouron rank
Figure 13: Demonstration of neuron basis alignment in early layers of Pythia- IB, Compares the compound- word classification performance of the top neurons for layers 1-4 the standard ncuron basis, as opposod random basis (the post-activation dataset multiplied by random Gaussian d x d matrix).
40

---

Published in Transactions on Machine Learning Research (11/2023)
bye.
I Insuron base
neuron 1305 random base
heupon bedl
neunon beas randon Basta
1 rundom 2866
3 rundam basu
eide-effiects
prime-factirs
third-partt
sdde effects
prime ficte-factrrs
third third-pertyy
10
300
12
60
60
gene expression
second derivative
fede el government
gene expression
second derivative
rederel-govermeen
220
10-1
12-1
223
220
60
of
10
10a
12-2
10
cinica -trils
cell-ines
iving-rom
dinicalttrils
call coll-linez
lkinggroom
50
10
10
arromm
200
80-
out
60
10
40
20
12nh
-20
human-rights
pressure
rights
pressure
10
122
60-
12-4
20-
10
25
10a4
magnetic-fiedd
socialsecurity
high-school
mag sotic-fffcccoo
social-security
high-school
30-
10
40-
10
12nd
10
40
20
rrbil-court
mental-health
north-america
trial-court
mental-health
north-americs
335
62
10
325-
10
4
20
25
fecrosss campeettteeeo Sitrreee
12n9
credit-card
contral.group
public-health
credit-card
contrci-group
lopean
3
15
10-2
10n4
101
102
10a
101
10
30
10
11P
101
107
10F
307
317
10F
127
20'
probo sparsiyy
prbbe sparsts
probe spartit
p700 rpo
probo sporsiyy
probe sponchy
Logistic loss probe sparsity
(b) Logistic loss
versus probe sparsity
Figure 14: Logistic loss for classifying the occurrence of particular compound word in layers 1-4 of Pythia-1B (in both the neuron basis and the average of random bases) undergo power law scaling until hitting break point (left). In general, probes in the neuron basis achieve lower overall loss, further suggesting the neuron basis is meaningful (right).
41

---

Published in Transactions on Machine Learning Research (11/2023)
Nouron activation context
Neuron activation by co nost splir by mase di iffrrntiated takens
Lass increase aith abation
A3-43 pre-srtivtion
22.5
75
0.00
empirical distiibtion
0.04
Dutch
Fexe
Greek Hriggeee swedsl
66
3.0
4.0
70NL N622
-30
0.00
dbttnbution
10ere
37267
3rdds
100"
Dutch
Hese
romann
creek
Pottiquee seeds
7OM.LS NU-L5 scevvaion
4.0
-10
22
0.00
-2
000
Gommon
Spanes
amginical
7OMLL 92965
Tast
Ven
Diait
Cemman
Spanss
hene
Melagase
kclen
TORLL3 setratinnn
Bulch
Vher
Derman
emgliical disiibutionn
Joken
Gendech
06
23
20H
0.00
0.02
0.00
0.00
188
Coner
1100"
100
Degish
Gertan
Spanes
Figure 15: Monosemantic language context neurons in Pythia-70M.
42

---

Published in Transactions on Machine Learning Research (11/2023)
context
by split by toxens
takon PiP
1.0
So0uchce N0S menesse
0.5
0.0
003
emgitiad cistibation
0.9
c.d
6.10
6.1
0.25
Yee
vioiiee
"oon
Utiow
C1-
HTHL
Jana
Pyiton
tokenn
38 4080 presariittion
3.0
not
100
20
0.02
5.00
0.05
0.05
3:0
0.12 Tale
chice
'iteboo "Srtag"
sejor
100
Python
omginiaal clantbution
63323 re-acttrccoon
tocon
secucnce MSS incresse (M
not
-10
000
0.2
0.04
oce
ods 0.10
c.12
17
mare
rsturr
56"
conteet
HthL
Jana
Pyston
empinical distribution
10.110 4B855 arttacttttin
ticn
Pyttom
1.
0.0 022 abs 0.0 0.19 012 cis
Inpoor
Pymion
ompirical ciantibtionn
18LE.N7372 ree-actvotenn
tkkn
ryshon not
1.
0.00
0.03
5:88
0.10
0.12
cmppinical distributioo
1306,44070 pre-activsoon
Toker JALA
1.
ConboRnce Iis rerease 19
E
0.0
002
001
0.6
008
020
0.22 Makie
Jeea
Python
ongirial cismibbtin
Toker
JoWa
2.0
cesuence DoS FCReSse ENC
000
0.03
empiiiccal cistnbution
0.05
008
0i5
0.12
0.5
brotoched
Throee
100""
Theicle
Ch
J0na
Figure 16: Monosemantic code context neurons in Pythia- 1B. 43

---

Published in Transactions on Machine Learning Research (11/2023)
by split by
Dion
Seguorc 022 Ricresse 080
4,4
0.10
Toiter
Faceiring
li condheer Tososime
seammss reopectivll
Sehu
Aaittd UEPTO
empinicall
Cepersde
Sector
Txtll
2007
Hackiisnn StadEs NE
6-987777700 presstivation
boich
Lapio
00
000 1orr000 0%)
not
0.4
There"
how londbaatin ccop bey
The
Dva:
NoMted USTTD
ampirical
thore
recrborred tactnoloyy
Redditiono StadEs Neppol
6.90 h12000 aetarrtttion
Teslow
socuince MS incress
-07
Jamuugrr BUST
mist
mvion
wibttd upfro
setivation
02
aKVV
9.7
0.00
0.66
"below yoare happootntttthmmnd
"chine Movooaer
Strub Abbitd DEPTD
taken
2.0
seovence 1006 Foresse
0.5
-1.0
0.00
0.04
10 6:12
Thankk
Doest
sowwer Trgarecty
contect
Othub
Adbited
USPTB
EXRDCRCe 955 Indreste (4)
6.98 M155122
0.60
0.00
0.10
A15
W000
1000
Adsftct us/iD
empiriall cisnibution
01
Inr0ose (36)
22
-2-
N5223
0.0
0.02
cmmiiiical cistnibtton
0.04
0.56
9.08
3:10
product
There
"rreeee
sspeert
approodyy
Vileer
ReoLzw Harkfooo RoadE podo
Mditeed USPiD
Figure 17: Monosemantic distribution identification context neurons in Pythia-6.09
44

---

Published in Transactions on Machine Learning Research (11/2023)
compound words
latex features
1.00
EOOd pressut
anLnreth
sJneth
ILaberam
F1 3core for bost sinle neurcs
0.
0.90
sit.inglee
0.85
rghrzchee
medi
Wiee nath
CAT matt
Taran ngant
0.00
0.7
0.75
0.70
0.65
0.5
model size
mode size
programming languages
part-of-spaach
0.9
Inttoon
f1 score for bost single nouren
0E00 PROM
upa MUX
upec PINC
uSB5 500M
0.6
0.6
0.4
0.2
2014
103M
<10N
181488
2.88
6.98
70M
100M
GL0M
18.14
2.88
0.99
model wiaw
made kide
subeet
morphological features
CRY CIton
DIStadd SCHTTNCT SECC CHcrange
1.0
Gender Fan
Pentlp Int
0.9
tesle: Nox
Prontype
Mumber Ins
TaCHSTICNS
Fl corre for best sirle
0.
Number Par
Wertoomm
Watroam Nur
0.6
0.
0.6
0.3
02
model size
modde size
plain text features
factual faatures
1.0
1.0
Rcsubliar Pery
pamelios
athanumerc
Forre for bost oingle nouron
0.9
allx
mede
In nSt.
pkyy
dgt
0.8
LANcon plager
0.7
0.6
0.6
0.5
0.1
20M
162M
<10N
181.48
2.88
6.98
70M
160M
410M
181.48
2.88
6.98
model
Figure 18: The single neuron with highest F1 classification performance for each feature by model size.
45

---

Published in Transactions on Machine Learning Research (11/2023)
Property: gender
max pre-activation of surmamn
famale male
L12.N4043
119.916039
L16.N740
L12.N13198
L19.N6101
L9.N996
Property: is alive
max pre-activat of surmame
tue
fas
L17 N5653
N11.61
L16.N3376
L14.N205
L16.N4634
L17.N18 117.N16123
Property: occupetion
max pre-activation of sumaae
jommalitt
coalician
resparcher
sttlett
Property: occupation athlete
max pre activabion of Surnene
12.0
soccer
baslethball player
football player
3:0
haseball plaarr
Ice hickley player
-2.5
-1.5
NL.761
L19.N2490
L19.N545
L20.N2367
Property: political_party
max pre-activation of surname
republica party
-2
L20.N14647
L9.N12483
L11.N9111
121.1635
L12.N17
L11.N926
Figure 19: Top neurons in Pythia-6.9B for each category of factual neuron.
46

---

Published in Transactions on Machine Learning Research (11/2023)
Properly: sex or gender
mex pre activation of surname
female
L10 L11 L12
L13
L14
L15
L16
L18
L19 120
L21 L22 L23 L24 L25 L26 L27 L28 129 L31
Property: is_alive
max pre-activation of sumname
true
Figure 20: Top neuron for each layer in Arthia-6.92 for the sex_or, _ox_or_gended gender and is alive feature categories.
47


Reviews:
[{"review_id": "SsKapPpb3y", "reviewer": "Reviewer_XC3T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. This paper aims to contribute to the general sense LLMs have a tremendous number of correlations between neurons and specific features that humans can understand. The authors propose a k-sparse probing method that could locate neurons related to these interpretable structures. \n\n2. Understanding the role of neurons is an important perspective on the interpretability of neural networks.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a k-sparse probing method that could locate neurons that are highly relevant for a specific meaning/property of samples (so-called a feature). In this way, potential correlations between the neurons of LLMs and specific features could be found, and the authors perform a number of case studies to illustrate the general properties of large language models (LLMs).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tUnderstanding the role of each neuron is not a novel idea. This paper locates highly relevant neurons for a particular feature. It is not novel to propose a method to find the meaning represented by each neuron [1].\n2.\tMore crucially, the proposed explanation conflicts with existing findings in explainable AI.\n3.\tMany papers argue that neural networks encode features/properties into a few sparse individual neurons. Instead, Kim et al. [2] argued that neural networks used each feature direction for the classification of a certain category. In words, different neurons were densely involved in the classification. A network did not use sparse neurons for the classification. Thus, please conduct experiments to examine whether a network uses sparse neurons for classification. Without the experiments, this study is built upon an unexamined assumption.\n4.\tI cannot see any solid conclusion or new insights, which guide the learning of DNNs, from the explanation.\n\n[1] David Bau*, Bolei Zhou*, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network Dissection: Quantifying Interpretability of Deep Visual Representations. Computer Vision and Pattern Recognition (CVPR), 2017.\n\n[2] Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., & Viegas, F. (2018, July). Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In International conference on machine learning (pp. 2668-2677). PMLR.\n\n5.\tLearning the meaning of each neuron relies on a problematic assumption, i.e., a certain neuron always represents the same feature/property/meaning given different sentences. Given different input sentences, different neurons in the transformer represent different tokens. Therefore, for each neuron in the transformer model, there is not theoretical guarantee to assume that the same neuron always represents the same feature/property/meaning in different sentences.\n\n6.\tThe definition of “features” should be further clarified. The target feature used to explain the properties of neurons usually represents a too broad concept, e.g., the “is political” feature. Moreover, the **is_political** feature usually depends on the complex patterns of the input. In this way, it is unrealistic to localize certain neurons that correspond to complex patterns of the “is political” feature.\n\n7.\tIn the first equation in Section 3.1, the structure of the transformer network needs to be represented more clearly. The equation is incorrect. Besides, the “attention” layer, the two “Layer Norm” operations are also ignored in this equation. BTW, why not give each equation an index number? E.g., Equation (1) and Equation (2).\n\n8.\tFurthermore, we do not think the authors can faithfully explain the transformer model by ignoring its multi-head attention (MHA) layer.\n\n9.\tThere are no quantitative metrics and baselines to verify the method. I cannot find any competing methods in experiments. The correctness of the explanation is not evaluated. I do not believe several case studies can justify the correctness of the proposed method.\n", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u0ZeT8wIEd", "reviewer": "Reviewer_WTD2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "** Strengths\n\n* The paper discusses interpretability in the representations of LLM. Two topics that are extremely relevant and important in today's discussions in ML.\n\n* The authors look at a variety of ablations to verify their findings.\n\n\n** Weaknesses\n\n* Many of the findings are not surprising and as the authors admit follow from simple coding arguments.\n\n* Some statements can use further plots/demonstrations to be convincing. For example, on page 6, the authors claim that \"by summing the activations across the three neurons, we achieve nearly perfect separation between the total activation magnitude of the “social security” feature and all other observed token combinations!\". However, it is not clear if this perfect separation happens in cases other than \"social security\" or if all the neurons are required to make the separation.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission looks at the representations learned in LLMs by using a probing method. They find interesting, if not surprising, features in this representation. They also look at how scale affects their findings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "Discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gen7AOuLlw", "reviewer": "Reviewer_JZkJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I find the paper to be a very careful and thorough study of the interpretability of Large Language Models. The proposed investigation techniques are promising and the observations of the authors should be interesting to the community. As such, I think this paper is well suited for this venue.\n\nStrengths:\n- The paper aims to tackle the ambitious topic of studying the interpretability of LLMs from the perspective of distributed representations, polysemanticity, and superposition, which were previously well-studied in toy models, but not necessarily in large language models. \n- Sparse probing seems to be a promising technique for investigating the interpretability of neurons in LLMs.\n- Through their studies, the authors discover multiple novel findings about the nature of neurons in LLMs, including the changes in superposition patterns depending on the layer, the existence of monosemantic neurons, and the relation of the model size and neuron splitting or quantization.\n- The authors approach the studied problem rigorously and explicitly list the limitations of their study.\n- The paper is very clearly written and is easy to read. In particular, the FAQ section of the Appendix is especially detailed and explains many important concepts and definitions, which is especially important in this very rapidly emerging field.\n\nWeaknesses: \\\nNote: in general, I find the paper to be quite high quality and the comments below are not necessarily grave errors that should be improved, but rather certain points that I would like to see in e.g. future work.\n- Many of the results in this paper are somewhat subjective (how granular should a feature be? How to differentiate between is_french and is_french_noun or is_french_verb?) and as such it is difficult to make rigorous statements. At the same time, the authors discuss these risks and limitations in detail, and I don't see a clear way of bypassing these difficulties.\n- The paper focuses only on the interpretability of neurons in the MLPs of LLMs. Studying other aspects, such as the attention mechanism, could provide some additional insights.\n- It would be interesting to see if similar conclusions can be drawn for other models than the one from the Pythia suite.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of interpreting the behaviors of Large Language Models (LLMs) through their internal mechanisms. In particular, the authors use sparse probing to find sets of neurons that can be used to predict particular features (e.g. whether the current token is in French, or if it represents a part of an n-gram). With this toolset, the authors conduct a thorough study of models of varying sizes from the Pythia suite. The authors note multiple interesting phenomena, such as the occurrence of superposition, neurons changing meaning depending on the context and the effects of scale on neuron splitting. The authors conclude their work with a discussion of the results and a careful consideration of the limitations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The quality of the paper is high and as of now, I don't have any direct actionable suggestions. I mentioned some of the problems in the \"Weaknesses\" section above, but I don't think these problems can be solved in the TMLR submission process timeframe.", "broader_impact_concerns": "I don't think this paper needs a Broader Impact Statement. There are numerous potential dangers of developing and LLMs, but understanding them does not directly lead to bad outcomes from my perspective. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Among the three reviewers, two reviewers are positive. Reviewer JZkJ thinks highly of this paper, noting the meticulous study and careful and extensive discussion. Reviewer XC3T remains concerned with novelty, conclusion and the method. However, I think the paper is novel by applying sparse probing to LLMs. Linear probing is a valid method for understanding LLMs. It is possible that individual neurons may predict features. Therefore, I would like to override Reviewer XC3T's recommendation. 

Overall I think the paper makes a useful contribution to a timely topic, and I recommend acceptance. But I do hope that the authors will revise their paper to address the concerns of Reviewer XC3T.
Decision: Accept (poster)