Universal Neurons in GPT2 Language Models
Authors: Wes Gurnee
Theo Horsley
Zifan Carl Guo
Tara Rezaei Kheirkhah
Qinyi Sun
Will Hathaway
Neel Nanda
Dimitris Bertsimas
Venue: Accepted by TMLR ()

Abstract: A basic question within the emerging field of mechanistic interpretability is the degree to which neural networks learn the same underlying mechanisms. In other words, are neural mechanisms universal across different models? 
In this work, we study the universality of individual neurons across GPT2 models trained from different initial random seeds, motivated by the hypothesis that universal neurons are likely to be interpretable. In particular, we compute pairwise correlations of neuron activations over 100 million tokens for every neuron pair across five different seeds and find that 1-5\% of neurons are universal, that is, pairs of neurons which consistently activate on the same inputs. We then study these universal neurons in detail, finding that they usually have clear interpretations and taxonomize them into a small number of neuron families. We conclude by studying patterns in neuron weights to establish several universal functional roles of neurons in simple circuits: deactivating attention heads, changing the entropy of the next token distribution, and predicting the next token to (not) be within a particular set.

Published in Transactions on Machine Learning Research (06/2024)
Universal Neurons in GPT2 Language Models
Wes Gurnce'
wesg edu
Massachusetts Institute of Technology
Theo Horsley University of Cambridge
tjhe03@cam..cc uk
Zifan Carl Guo Massachusetts Institute of Technology
carlguo@mil. edu
Tara Rezaei Kheirkhah Massachusetts Institute of Technology
tarark@mitt.cm edu
Qinyi Sun
wendysum @ettstsummmmmmmmm eduu
Massachusetts Institute of Technology
Will Hathaway
willhath Ceciiath@@mmm.cmmm edu
Massachusetts Institute of Tecknology
Neel Nanda
nachnanda270@mmai.ccoo com
Dimitris Bertsimast
dbertsim Cerssimmmmmtt..m edu
Massachusetts Institute of Technology
Reviewed on OpenReview: https: // opeerevee......iee.. net/ Sttomm/ccazz220g0222000et1eeeeae
Abstract
A basic question within the emerging field of mechanistic interpretability is the degree to which neural networks learn the same underlying mechanisms. In other words, neural mechanisms universal across different models? In this work, we study the universality of individual neurons across GPT2 models trained from different initial random seeds, motivated by the hypothesis that universal neurons are likely to be interpretable. In particular, we compute pairwise correlations of neuron activations over 100 million tokens for every neuron pair across five different seeds and find that 1-5% of neurons are universal, that is, pairs of neurons which consistently activate the neurons in detail, finding that they usually have clear interpretations and taxonomize them into small number of neuron families. We conclude by studying patterns in neuron weights to establish several universal functional roles of neurons in simple circuits: deactivating attention heads, changing the entropy of the next token distribution, and predicting the next token to (not) be within particular set.
inputs. We then study these universal
1 Introduction
As large language models (LLMs) become more widely deployed in high-stakes settings, lack of under- standing of why or how models make decisions creates many potential vulnerabilities and risks (Bommasani
let al. 2021 Hendrycks et. al. 2023 Bengio et al. 2023). While some claim deep learning based systems are fundamentally inscrutable, artificial neural networks seem unusually amenable to empirical science compared to other complex systems: they are fully observable, (mostly) deterministic, created by processes we control,
*Corresponding Author; Senior Author

---

Published in Transactions on Machine Learning Research (06/2024)
[a] Alphabet Neurons
(b) Previous Token Neuron
Position Neurans
X/x token
previous token
beqin Xix
empirical distribution
0.15
has
does not
mian pre activaton
0.0
L1.1190
1.1205
ure-activeo on
003
1.2515
0.10
-1.0
-15
0.05
-2.0
-1
-25
-2
0.00
-3.0
-2
100
200
300
400
500
L0.3825
L1.3970
10.1761
L4 3859 pre-activation
sequercc position
(d) Context Neuron
(e) Suppression Neuron
Entropy Neurons
cimiitical didriburtion
0.15
distribution
medical non-medical
10
vocab token
year
mean next tolen entrony
3,6
3.4
Neuror
L23.945
L22.2882
randor
count
10a
3,0
0.05
10
2.8
2.6
10
2.4
-2
-0.8
-0.6
-0.4
-0.2
0.0
0.2
L1.1716 pre-activation
W.W.o L23.2260
fixed act Vation value
Figure 1: Universal neurons in GPT2 models, interpreted via their activations (a-d), weights (e), and causal interventions (f). (a) Neurons which activate primarily on a specific individual letter and secondarily on tokens which begin with the letter; (b) Neuron which activates approximately and only if the previous token contains a comma; (c) Neurons which activate as function of absolute loken position in the context. (shaded area denotes standard deviation around the mean); (d) A neuron which activates in medical contexts (e.g. pubmed abstracts) but not in non-medical distributions; (e) neuron which decreases the probability of predicting any integer tokens between 1700 and 2050 (i.e., years); (I) Neurons which change the entropy of the next token distribution when causally intervened.
admit complete mathernatical descriptions of their form and function, can be run any input with arbitrary modifications made to their internals, all at low cost and on computational timescales (Olah) 2021). An advanced science of interpretability enables a more informed discussion of the risks posed by advancod AI systems and lays firmer ground to engineer systerss less likely to cause harm (Doshi- Velez and Kim 2017
Bender et al. 2021 Weidinger et al. 2022 Ngo et al. 2023 Carlsmith 2023
Olah et al. 2020b propose three spcculative claims regarding the interpretation of artificial neural networks: that features directions in activation space representing properites of the input are the fundamental unit of analysis, that features are connected into circuits via network weights, and that features and circuits are universal across models. That is, analogous features and circuits form in diverse array of models and that different training trajectories converge on similar solutions (Li et al. 2015). Taken seriously, these hypotheses suggest a strategy for discovering important features and circuits: look for that which is universal. This line of reasoning motivates our work, where we leverage different notions of universality identify and study individual neurous that represent features or underlie circuits.
Beyond discovery, the degree to which neural mechanisms universal is basic open question that informs what kinds of interpretability rescarch are most likely to be tractable and important. If the universality hypothesis is largely true in practice, we would expect detailed mechanistic analyses (Cammarata et al., 2021 Wang et al. 2022a Olsson et al., 2022 Nanda et al. 2023 McDougall et al. 2023) to generalize across models such that it might be possible develop a periodic table of neural circuits which can be automatically referenced when interpreting new models (Olah al. (020b). Conversely, it bocomes less sensible dedicate substantial manual labor understand low-level details of circuits if they completely different. in every
2

---

Published in Transactions on Machine Learning Research (06/2024)
model, and instead more efficient to allocate effort to engineering scalable and automated methods that can aid in understanding and monitoring higher-level representations of particular interest (Burns et al. 2022; Conmy et al. 2023 Bills et al. 2023 Zou et al.) 2023 Bricken et al. 2023). However, even in the case that not all features or circuits are universal, those which are common across models are likely to be more fundamental (Bau et al.
2018 Olsson et al., 2022 and studying them should be prioritized accordingly.
In this work, we study the universality of individual neurons across GPT2 language models (Radford et al,, 2019) trained from five different random initializations (Karamcheti et al., 2021]. While it is well known that individual neurons are often polysemantic (Nguyen et al. 2016 Olah et al. 2020b Elhage et al. 2022b Gurnec et al., 2023) i.c., represent multiple unrelated concepts, we hypothesized that universal neurons were more likely to be monosemantic (see (A.1). potentially giving an approximation on the number of independently meaningful neurons. We choose to study models of the same architecture trained on the same data to have the most. favorable experimental conditions for measuring universality to establish a rough bound for the universality over larger changes. We begin by opcrationalizing neuron universality in terms of activation correlations, that is, whether there exist pairs of neurons across different models which consistently activate the same imputs. We compute pairwise correlations of neuron activations over 100 million tokens for every ncuron pair across the different seeds and find that only 1-5% of neurons pass a target threshold of universality compared to random baselines ($ 4.1 We then study these universal neurons in detail, analyzing various statistical properties of both weights and activations ($ 4.2 and find that they usually have clear interpretations and taxonomize them into a small number of neuron families (S 4.3).
In Section 5 we study more abstract form of universality in terms of neuron weights rather than activations. That is, rather than understand neuron in terms of the inputs which cause it to activate, understand neuron in terms of the effects the neuron has on later model components or directly on the final prediction. Specifically, we analyze patterns in the compositional structure of the weights and find consistent outliers in how neurons affect other network components, constituting very simple circuits. In Section 5.1 we show there cxists a large family of late layer ncurons which have clear roles in predicting or suppressing a coherent set of tokens (e.g., second-person pronouns or single digit numbers), where the suppression neurons lypically come in later layers than the prediction neurons. We then investigate a small set of neurons that leverage the final layer-norm operation to modulate the overall entropy of the next token prediction distribution ($ (.22 We conclude with an analysis of neurons which control the extent to which an attention head attends to the first token, which empirically controls the output norm of the head, effectively turning a head on or off ($ 5.3)
2 Related Work
Universal Neural Mechanisms Features and circuits like high-low frequency detectors (Schubert et al.) 2021a) and curve circuits (Cammarata et al. 2021 have boen found to reoccur in vision models, with some Teatures even reappearing in biological neural networks (Goh et al. 2021). In language models, recent research has found similarly universal circuits and components like induction heads (Olsson et al. 2022) and successor heads (Gould al.] 2023) and that models reuse certain circuit components to implement different tasks
(Merullo et al. 2023 There has also been a flurry of recent work on studying more abstract universal mechanisms in language models like function vectors (Todd et al. 2023 Hendel el al. 2023). variable binding
mechanisms (Feng and Steinhardt) 2023), and long context retrieval (Variengien and Winsor 2023). Studying
universality in toy models has provided "mixed evidence" on the universality hypothesis (Chughtai et al.
2023) and shown that multiple algorithms exist to implement the same tasks [Zhong et al. 2023 Liao et al,
2023
Representational Similarity Preceding the statement of the universality hypothesis in mechanistic interpretability, there has been substantial work measuring representational similarity (Klabunde et al. 2023 Common methods include canonical correlation saalyiis-bsse mcasures (Raghu et al. 2017 Morcos et al.
2018), alignment-based measures (Hamilton et. al.
2018
Ding el al. 2021 Williams et al. 2022 Duong el al.
2023), matrix based measures (Kornblihh et al.,
2019 Tang et al. 2020
Shahbazi et al.) 2021
2022
Boix-Adscra et al.] 2022 Godfrey et al.) (2023). neighborhood-bascd mcasures (Hryniowski and Wong 2020
Gwilliam and Shrivastava 2022), topology-base measures (Khrulkov and Oscledets 2018 Barannikov et al,
2022), and descriptive statistics (Wang and Isola
2022
Lu et al. 2022
Lange et al. 2022 Previous work,
3

---

Published in Transactions on Machine Learning Research (06/2024)
mostly in vision models, has yielded mixed conclusions on whether networks with the same architecture learn similar representations. Some studies have found that networks with different initializations "exhibit very low similarity" (Wang et al. 2018)) and "do not converge to a unique basis" (Brown et al. 2023), while others have shown that networks learn the same low-dimensional subspaces but not identical basis vectors (Li et al.) 2016) and that different. models can be linearly stitched logether with minimal loss suggesting they learn similar representations (Bansal et al. 2021
Analyzing Individual Neurons Many prior interpretability studies have analyzed individual neurons. Tn vision models, researchers have found neurons which activate for specific objects (Bau et al. 2020 curves at spocific orientations (Cammarata et al., 2021). high frequency boundarics (Schubert et al. 2021b). multimodal concepts (Goh al. (2021), as well as for facets (Nguyen et al., 2016) and compositions (Mu and Andreas
2020) thereof. Morcover, many of these neurons seem universal across models Dravid et al. (2023). In
language models, neurons have been found to correspond to sentiment (Radford et al. 2017 Donnelly and
2022b), de- de-/re-tokenizee tre-ooeeiizttion Ellhage et al. Roegiest 2019), knowledge (Dai et al. 2021 skills (Wang et al.)
2022a), contexts Gurnee et al. 2023 Bills et al. 2023). position (Voita et al. 2023) , space and time (Gurnee
and Tegmark 2023 and many other linguistic and grammatical fcatures (Bau ct al. 2018 Xin ct al. 2019
Dalvi el. al. 2019: 2020 Durrani et al. 2022 Sajjad et al. 2022). More generally, it is hypothesized that
neurons in language models form key-value stores (Geva al. 2020) that facilitate next token prediction by promoting concepts in the vocabulary space (Geva et al.) 2022) However, many challenges exist in studying
individual neurons, especially in drawing causal conclusions (Antverg and Belinkov
2021 Huang et al. 2023
3 Conceptual and Empirical Preliminaries
3.1 Universality
Notions of Universality Universality can refer to many different notions of similarity, each at a different level of abstraction and with differing measures and methodologies. Similar to Marr's levels of analysis in neuroscience (Hamrick and Mohamed 2020 Marr 2010), relevant notions of universality are: compulational or functional universality regarding whether a (sub) network implements a particular input-output- suuu--uupput-eehvvoo behavior (e.g., whether the next token prodictions for two different networks are the same); algorithmic universality regarding whether or not a particular function is implemented using the same computational steps (e.g., whether a transformer trained to sort strings always learns the same sorting algorithm): representational universality, the degree of similarity of the information contained within different representations (Kornblith et al. 2019) (c.g., whether every network represents absolute position in the context); and finally implementation universality, ie., whether individual model components learned by different models implement the same specialized computations (e.g., induction heads (Olsson et al., 2022). successor heads (Gould et al.) 2023). French neurous (Gurnee et al. 2023), inter alia). None of these notions universality are usually binary, and the universality between components or computations can range from being formally isomorphic to simply sharing a common high-level conceptual statistical motif.
In this work, are primarily concerned with implementation universality in the form of whether individual neurons learn to specialize and activate for the same imputs across models. If such universal neurons do exist, then this is also a simple form of functional universality, as the distinct neurons constitute the final node of distinct subnetworks which compute the same output.
Dimensions of Variations Universality must be measured over some independent dimension of variation, i.e,, some change in the model, data or, training. For example, model variables include random seed, model size, hyperparameters, and architectural changes; data variables include the data size, ordering, and distribution contents; training variables include loss function, optimizer, regularization, finctuning, and hyperparameters thereof. Assuming that changing random seed is the smallest change, this work primarily focuses initialization universality in an attempt to bound the expected similarity of larger changes.
4

---

Published in Transactions on Machine Learning Research (06/2024)
3.2 Models
We restrict scope to transformer-basee aulo-regressive language models (Radford et al. 2018) that currently power the most capable AI systems (Bubeck et al. 2023). Given input sequence of tokens = [7, E X S y from the vocabulary V, a language model .M X y outputs probability distribution over the vocabulary to predict the next token in the sequence.
We focus on replication of the GPT2 scrics of models (Radford et al. 2019) with some supporting experiments on the Pythia family (Biderman et al., 2023 For GPT2-small and GPT2-medium architecture (see B.3 for hyperparameters) we study five models trained from different random seeds, referred GPT2 (small, medium))(a--) (Karamcheti et al. 2021
Anatomy of a Neuron Of particular importance to this investigation is the functional form of the neurons in the feed forward (also known as multi-layer perceptron (MLP)) layers in the transformer. The out of an MLP layer given a normalized hidden state x E Rdmodel is
MLP(x)= W Wonto (Win bin) + bout
(1)
where WT out. Win Rd Xdmodel are learned weight matrices, bin and bout are learned biases, and is an elementwise nonlinear activation function. For all models we study, is the GeLU activation function (x) = (x) (Hendrycks and Gimpel 2016). One can analyze an individual neuron j in terms of its activation a (w x + bkn)) for different inputs x, or its weights row j of Win or W which respectively dictate for what inputs a neuron activates and what effects has downstream.
We refer the reader to (Elhage al. 2021 for full description of the transformer architecture. We employ standard weight preprocessing techniques described further in B.1
4
The Search for Universal Neurons
4.1 How Universal are Individual Neurons?
Experiment Inspired by prior work studying common neurons in neural networks (Li et al. 2015 Bau
et al., 2018 Dravid al. 2023), we compule maximum pairwise correlations neuron activations across five different models GPT22 c, d, e} to find pairs of neurons across models which activate on the same inputs. Let (aa) be the set of neurons in model a. For cach neuron i N(a), compute the Pcarson correlation
E [(v'= pi)(v' p5)]
PCj
010,
with all neurons j E (mm) in every model m E(b, c, d, e), where ir and a; are the mean and standard deviation of a vector of neuron activations v' computed across a dataset of 100 million tokens from the Pile test set (Gao et al. 2020). For a baseline, we also compute Pij a.m where instead of taking the correlation of p(v*, v²), we compute p(v', (RV);) for random dmip dmip Gaussian matrix R and the matrix of activations V for all neurons in particular layer Ne(m). In other words, we compute the correlation between neurons and elements within a random (approximate) rotation of a layer of neurons to establish a bascline correlation
for the case where there does not exist privileged basis (Elhage et al. 2021 Brown et al. 2023 to verify the importance of the neuron basis.
For set of models M we define the ezcess correlation of neuron as the difference between the mean maximum correlation across models and the mean maximum baseline correlation in the rotated basis:
L:
M
Σ MEM
Pij (EN(m)
max
a.m max
(e)
m Pi
(3)
5

---

Published in Transactions on Machine Learning Research (06/2024)
(a) neuron correlation distributions
(b) max correlation range
(c) most similar neuron layer
10
max [b max neuron correlation
50
correlation
10a
max
baseline
layer gpt2-mentt 1122--eduuma
40
most similar neurons layer
neuron count
excess
0.6
10
30
10
0.4
15
20
neuron
20
10
mean y|x
0.0
0.2
0.4 0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
5
10
15
20
correlation
min [b el max neuron correlation
layer in gpt2-medium-lb-el
Figure 2: Summary of neuron correlation experiments in GPT2-medium-a. (a) Distribution of the mean (over models b-e) max (over neurons) correlation, the mean baseline correlation, and the difference (excess). (b) The max (over models) max (over neurons) correlation compared to the min (over models) max (over neuron) correlation for each neuron. (c) Percentage of layer pairs with most similar neuron pairs.
Results Figure summaarizes our results. Figure 2a, we depict the average of the maximum neuron correlations across models [b-c], the average of the baseline correlations, and the excess correlation i..., the left term, the right term, and the difference in (3). While there is no principled threshold at which neuron should be deemed universal, only 1253 out of the 98304 neurons in GPT2-mcdium-a have excess correlation greater than 0.5. In Figure 14] we report the (complement) cumulative distribution of these correlation metrics to show how the number of universal neurons changes with threshold.
To understand if high (low) corrolation in one model implics high (low) correlation in all the models, in Figure 2 we report maxm max)eN(()) a.m compared min,, (a)) P.] for every neuron E Na).. Figure 2p suggests there relatively little variation in the correlations, as the mean difference between the max-max and min-max correlation 0.049 for all neurons and 0.105 for neurons with e > 0.5. Another within the network (as
natural hypothesis is that neurons specialize into roles based on how deep they suggested by (Olah et al. 2020b Flhage et al. 2022a))- In 2 for each layer I of model a, compute the fraction of neurons in layer I that have their most correlated neuron in layer l' for all e' in models [b-e]. Averaging across the different models, we obscrve significant depth. specialization, suggesting that neurons do perform depth specific computations, which we explore further $ 4.3
We repeat these experiments on GPT22small and Pythia-160m displayed in Figures and 1 respectively. A rather surprising finding that while the percentage of universal neurons (e: > 0.5) within GPT2-medium and Pothia-160M) are quite consistent (1.23% and 1.26% respectively), the number in GPT2-small-a is far higher at 4.16%. We offer additional results and speculations in $ C.3
4.2 Properties of Universal Neurons
We now seek to understand whether there are statistical proprieties associated with whether a neuron is universal or not, defined as having an excess correlation 2 > 0.5. For all neurons in GPT2-medium-a, GPT2-smallaa, and Pythia-160m, we compute various summary statistics of their weights and activations. For activations, we compute the mean, skew, and kurtosis of the pre-activation distribution over 100 million tokens, as well as the fraction of activations greater than zero, termed activation sparsity. For weights, we record the input bias bin the cosine similarity between the input and output weight cos(wi.. Wout); the weight decay penalty ||wiall? + /o and the kurtosis of the neuron output weights with the unembedding kurt(cos(w.o.. (w.tt Wo)) to measure the composition with the unembedding (Geva et al. 2022
Dar et al. 2022
In Figure 3 we report these statistics for universal neurons as a percentile compared to neurons within the same layer; we choose this normalization to enable comparison across different layers, models, and metrics (a breakdown per metric and layer for GPT2-medium-a is given in Figure 15 Our results show that universal neurons do stand out compared to non-universal neurons. Specifically, universal neurons typically have large

---

Published in Transactions on Machine Learning Research (06/2024)
100
model
percentile (normalizee lized by layer)
pythia-160m
80
gpt2 small-a
gptz -
60
40
20
trequency
mean
kurtosis
input
cos(Wan Weurh
La penalty
Wy kurtosis
neuran metric
Figure 3: Properties of activations and weights of universal neurons for three different models, plotted as percentile compared to neurons in the same layer
weight norm (implying they are important because the model was trained with weight decay) and have large negative input bias, resulting in large negative pre-activation mean and hence lower activation frequency. Furthermore, universal neurons have very high pre-activation skew and kurtosis, implying they usually have negative activation, but occasionally have very positive activation, proprieties we would expect of monosemantic neurons (Olah et al. 2020b Elhage et al., 2022b Gurnee et al. 2023 which only activate when a specific feature is present in the imput. In contrast, non-universal neurons usually have skew approximately and kurtosis approximately 3, identical to Gaussian distribution. We will discuss the meaning of high Wy kurlosis in 5.1 and high (oin Wout) in $ C
4.3 Universal Neuron Families
Motivated by the observation that universal neurons have distributional statistics suggestive of monoscmantic- ity, zoom-in individual neurons with > 0.5 and attempt to group them into a partial (axoninization of neuron families (Olah et al. 2020a Cammarata et al., 2021). After manually inspecting many such neurons, we developed several hundred automated tests to classify neurons using algorithmically generated labels dcrived from clements of the vocabulary (c.g., whether a token is_ as_all_capt or contains_ (ettaiinnndiiit)) and from the NLP package spaCy (Honnibal al. (020).. Specifically, for each neuron with activation vector v, and each test explanation which binary vector y over all tokens in the input, we compute the reduction in variance when conditioned on the explanation:
(1 (1- 0) + Bo 1) (²(v)
(4)
where 3 the fraction of positive labels and (-) is the variance of vector or subset thereof. In words, Eq measures the change in variance between the original activation distribution, and the weighted (by 3) variance of the distribution slices where y; = and = 0. As useful intuition, this is the same metric used to decide how to split in a regression tree, where the goal is to find split which most reduces the variance in the prediction target. Below, we qualitatively describe the most common families, and find our results replicate many findings previously documented the literature.
Unigram Neurons The most common type of neuron we found were unigram neurons, which simply activate approximately if and only if the current token is particular word or part of a word. These neurons often have many near duplicate neurons activating for the same unigram (Figure 16 and appear predominately in the first two layers (Figure 17). We breakdown activations of neurons responding to alphabetical unigrams based on the unigram's position in word, as common words often have four tokenizations, and find that duplicate neurons can respond differently to unigram variations (Figures 4a and 16 Such neurons illustrate
7

---

Published in Transactions on Machine Learning Research (06/2024)
1.00
token infinitive verb
Calgary
Fukushima
empiniaal dicrtbattion
0.17
not
Edmonton
Guatemala
pre-activation
0.10
Canadian
japanese
Nicaragua
0.90
Toronto
Miy
Chile
0.85
Quebec
Kyoto
Handuras
0.80 3
Winnipeg
Okinawa
Rica
0.04
0.75
[e=<]] on
Ottawaa
Fuk
Venezuela
other
0.02
laranto
Anime
Peru
0.70
0,2008
10.590
0.00
-2
Vancouver
Mits
Dominican
neuron
L5.983 pre-eclivetion
L17.2896
L17.944
L14.3870
(a) Similar "on" unigram neurons
(b) Syntax neuron
(c) Place Neurons
Figure 4: Additional examples of universal neuron families in GPT2-medium.
that the token (un)embeddings may not contain all of the relevant token-level information, and that the model uses neurons to create an "extended" embedding of higher capacity.
Alphabet Neurons A particularly fun subclass of unigram neurons are alphabet neurons (Figure la which activate most strongly on tokens corresponding to an individual letter, and secondarily on tokens which begin with the respective letter. For 18 of 26 English letters there exist alphabet neurons with @ > 0.5 (Figure 18), with some letters also having several near duplicate neurons.
Previous Token Neurons After finding an example of one neuron which seemed to activate purely as a function of the previous token (e.g., if contains a comma; Figure [b), we decided to rerun our unigram tests with the labels shifted by one-that is, with the label given by the previous token. These tosts surfaced many more previous token neurons occurring most often in layers 4-6 (see Figure 19 for additional 25 universal previous token neurons). Such neurons illustrate the many potentially redundant paths of computat ions that, can occur which complicates ablation based interpretability studies.
Position Neurons Inspired by the recent work of (Voita et al. 2023), we also run evaluations for posilion neurons, neurons which activate as a function absolute position rather than token or context (Figure Ich. We follow the procedure of |Voia et al. 2023) (who run their experiments on ÓPT models with ReLU activation (Zhang al.] 2022)) by computing the mutual information between activation and context position, and find similar results, with neurons that have a variety of positional patterns concentrated in layers 0-2 (see Figure 20 for 20 more neurons). Similar to the unigram neurons, the presence of these neurons is potentially unexpected given their outputs could be learned directly by the positional embedding at the beginning of the model with less variance in activation.
Syntax Neurons Using the NLP package spaCy (Honnibal et al. 2020), we label our input data with part- of-speoch, dependoncy role, and morphological data. We find many individual neurons that selectively activate for basic linguistic features like negation, plurals, and verb forms (Figure 4b) which are not concentrated to any part of the network and resemble past findings on linguistic properties (Dalvi al. 2019 Durrani et al., 2022
Figure 21 includes 25 more examples.
Semantic Neurons Finally, we found large number of neurons which activate for semantic features corresponding to coherent topics (Lim and Lauw 2023). concepts (Elhage et al. 2022a). contexts (Gurnee
ct al. 2023). Such features are naturally much harder to algorithmically supervise. We use the subdistribution label from the Pile dataset (Gao el al. (2220) and manually labeled topics from an SVD based topic model as a best attempt, but this leaves many interpretable neurons undiscovered and uncategorized. In Figure 4c] we show three rogions neurons which activate most strongly on tokens corresponding places Canada, Japan, or Latin America respectively. Figure 22 depicts 30 additional context neurons which activate on specific subdistributions, with many neurons which always activate for non-english text.
8

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Prediction neuron (L19 1169)
(b) Supression neuron (L23.2042)
(c) Partition neuron (L18.3483)
104
vocab
vocab token
vocab token
2000
10a
year
contains open paren
starts with space
vocabulary count
is not
10a
does not
does not
1500
10a
102
10
10a
10
10
-0.1
0.0
0.1
0.2
0.3
0.4
-0.8
-0.6
-0.4
-0.2
0.0
0.2
-0.4
-0.2
0.0
0.2
WyWout
WyWout
WoWout
Figure 5: Example prediction neurons in HPP2-mediume Depicts the distribution of logit effects on the output vocabulary Wot)) split by token property for different neurons. (a) Prediction neuron increasing logits of integer tokens between 1700 and 2050 (i.e. years; high kurtosis), (b) Suppression neuron decreasing logits for tokens containing open parenthesis (high kurtosis and negative skew), and (c) Partition neuron boosting tokens beginning with space and suppressing tokens which do not (high variance).
5 Universal Functional Roles of Neurons
While the previous discussion was primarily focused on analyzing the activalions of neurons, and by extension the features they represent, this section is dedicated to studying the weights of neurons to better understand their downstream elfects. The neurons in this section are examples of action mechanisms (Authropic) 2023) model components that are better thought of as implementing an action rather than purely extracting or representing a fcature, analogous to motor neurons in neuroscience.
5.1 Prediction Neurons
A simple but effective method to understand weights is through logit attribution techniques (Nostaleebraist, 2020; Geva al.] 2022 Dar al.) 2022). Because the final residual stream is the sum of all previous layers, can approximate neuron's 's effect the final prediction logits by simply computing the product between the unembedding matrix and a neuron output weight Wout and hence interpret the neuron based on how it promotes concepts in the vocabulary space (Geva et al. 2022).
When we apply our automated tests from $ |.3 on Wow rather than the activations for our universal neurons, we found several general patterns (Figure 5). many individual neurons with extremely clear interpretations (Figure 24j. and clusters of neurons which all affect the same tokens (Figure 25). Specifically, we find many examples of prediction neurons that positively increase the predicted probability of coherent set of tokens while leaving most others approximately unchanged (Fig 5p); suppression neurons that are similar, except decrease the probability of group of related tokens (Fig 5p); and partition neurons that partition the vocabulary into two groups, increasing the probability of one while decreasing the probability of the other (Fig 5p). The prediction, suppression, and partition motifs can be automatically detected by studying the moments of the distribution vocabulary effects given by Wo In particular, both prediction and suppression neurons will have high kurtosis (the fourth moment- a measure of how much mass in the tails of a distribution), but prediction neurons will have positive skew and suppression neurons will have negative skew. Partition neurons will shift the probability of most tokens and have high variance in overall logit effect. From this, we see almost all universal neurons (e 0.5) in later layers are one of these prediction neuron variants (Figure 15).
To better understand the number and location of these prediction neurons, we compute the moment metrics of cos Wu, Wout) for all neurons in all five GPT2-medium models, and show how these statistics vary over model depth Figure G We find striking pattern which is quite consistent across the different sceds: after about the halfway point in the model, prediction neurons become inereasingly prevalent until the very end of the network where there is sudden shift towards suppression neurons. To ensure this is not just an artifact.
9

---

Published in Transactions on Machine Learning Research (06/2024)
(a)
gpt2-medium-(a- kurtosis aut Wu)
(b) gpt2-medium-la skew (Wout. Wu)
(c) gpt2-medium(la-e pred neuron skew
mean layer percentile
0.35
layer
pred
16
5%
0.30
142
cos War) kurtosis
14
10%
50%
cos(Wu: War) skew
0.5
21
L22: 207
339
12
90%
0.0
95% %
density
0.00
10
-0.5
mean layer percentile
0.15
10%
0.10
6-
-1.0
50%
90%
4-
95%
0.00
5
10
20
5
10
15
20
-5.0
-2.5
0.0
2.5
5.0
layer
layer
cos(Wu. skew
Figure 6: Summary statistics cosine similarity between neuron output weights (Wont and token unembed- ding (We) for GPTT2 (aaetuuum-aae-eaaaa-e (a,b) Percentiles of kurtosis and skew by layer averaged over [a-e]. (c) Distribution of skews for neurons with kurtosis greater than 10 in last four layers. Shaded area denotes range across all five models.
of the tied embeddings (We- w in the GPT2 models, we also run this analysis on five Pythia models ranging from 410M to 6.9 parameters and find the results are largely the same (Figure 23
We observed an interesting pattern when examining the activations of suppression neurons: they activate much more frequently when the next token actually belongs to the set of tokens they suppress the probability of predicting. In other words, neurons which decrease the probability that the next token is a year (e.g. "1970"), activate much more often when the next token actually a year compared to not. We intuit that these suppression nourons fire when it is plausible but not cortain that the next token is from the relevant set. Combined with the observation that there exist many suppression and prediction neurons for the same loken class (Figure 25), we take this as evidence of an ensemble hypothesis where the model uses multiple neurons with some independent error that combine to form more robust and calibrated estimate of whether the next token is in fact a year.
In addition to being a clean example of an action mechanism AAtthroiic 2023), these results are interesting as they refine a conjecture made by (Geva et al., 2022). Specifically, rather than "feed-forward layers build predictions by promoting concepts in the vocabulary space," " we claim late feed-forward (MLP) layers build predictions by both promoting and suppressing concepts in the vocabulary space. Moreover, it suggests there different stages in the iterative inference pipeline (Belrose al. 2023 Jastrzębski al. 2017). where first affirmative predictions are made, and then the distribution is sharpened or made more calibrated by suppression neurons at the very end. The existence of suppression neurons also sheds light on recent observations of individual neurons (Bills et al.) 2023) and MLP layers (McGrath et al. 2023) suppressing the maximum likelihood token and being a mechanism for self-repair.
5.2 Entropy Neurons
Because models are trained with weight decay (lg regularization) we hypothesized that neurons with large weight norms would be more interesting important because they come at. a higher cost. While most turned out to be relatively uninteresting (mostly neurons which activate for the beginning of sequence token), the 15th largest norm neuron in GP2-medium-a (L23.945) had an especially interesting property: it had the lowest variance logit effect Wout of any neuron in the model; i.e., it only has a tiny effect on the logits. To understand why a final layer neuron, which can only affect the final logit distribution, has high weight norm while performing an approximate no-op on the logits, recall the final decoding formula for the probability of the next token given final residual stream vector x
X E[x
p(y[x) = Boftmax(W,,LaerNooon
LayerNorm(x)
(5)
/ /Var(x] +E
10

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Neuron weight norms
(b) Neuron composition w/ unembed
(c) Final layer norm scale
10a
10
Neuron
23.945
0.80
neuron count
10a
22.2882
vocab count
10a
mean scale
random
0.75
102
10a
0.70
10
10a
0.65
10P
100
-0.15
-0.10
0.00
0.15
IWo.1++ + !Word?
cos(Wu, Wann)
fixed activation value
Next token prediction recip rank
(e) Next token prediction entropy
(f) Next token prediction loss
mean reciprocal rank
3.6
3.03
0.580
mean entropy
3.02
0.578
3.2
mean loss
3.01
intervention
3.0
3.00
0.576
2.8
2.99
2.6
2.98
2.4
2.97
0
0
fixed activation value
fixed activation value
fixed activation value
Figure 7: Summary of (anti-)entropy neurons in GPT2-medium-a compared to 20 random neurons from final two layers. Entropy neurons have high weight norm (a) with output weights mostly orthogonal to the unembedding matrix (b). Fixing the activation to larger values causes the final layer norm scale to increase dramatically (c) while leaving the ranking of the truc next token prediction mostly unchangod (d). Inereasod layer norm scale squeezes the logit distribution, causing a large increase in the prediction entropy (e; or decrease for anti-entropy neuron) and increase or decrease in the loss depending on the model's baseline level of under- over-confidence (f). Legend applies to all subplots.
We hypothesize that the function of this neuron is to modulate the model's uncertainly over the next token by using the layer norm to squeeze the logit distribution, in manner quite similar to manually increasing the temperature when performing inference. To support this hypothesis, we perform a cansal intervention, fixing the neuron in question to a particular value and studying the effect compared to 20 random neurons from the last two layers that are not the top decile of norm or in the bottom decile of logit variance (Figure 7 We find that intervening on this entropy neuron indeed causes the layer norm scale to increase dramatically (because of the large weight norm) while largely not affecting the relative ordering of the vocabulary (because of the low composition), having the effect of increasing overall entropy by dampening the post-layer norm component of x in the row space of We.
Additionally, we observed a neuron (L22.2882) with cos ( 23.945 w 02228882 = 00886 (i.c., a neuron that writes nut
in the opposite direction forming an antipodal pair (Elhage et al. 2022b)) that also has high weight norm. Repeating the intervention experiment, we find this neuron decreases the layer norm scale and decrcases the mean next token entropy, forming an anti-entropy neuron. These results suggest there may be one more global uncertainty directions that the model maintains to modulate its overall confidence in its prediction. However, our experiments with fixed activation value do not necessarily imply the model uses these neurons to increase the entropy as a general uncertainty mechanism, and we did notice cases in which increasing the activation of the entropy neuron decreased entropy, suggesting the true mechanism may be more complicated.
We repeat these experiments on GPT2-small-a and find an even more dramatic antipodal pair of (anti- Jentropy neurons in Figure 26] To our knowledge, this is the first documented mechanism for uncertainty quantification in language models and the second example of mechanism involving layer norm (Brody et al. 2023
11

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Weight composition distribution
(b) BOS attention with ablation
(c) Output norm with ablation
0.125
10"
Heuristic
P(BOS) orig P(BOS) ablated
best fit (p=0.94)
0.1
best fit (p--0.92)
count (neurons x heads)
Wann WTKkos
0.100
ablation result
ablation result
10
Raok
0.075
0.0
0.050
-0.1
10a
0.025
-0.2
0.000
10
-0.025
-0.3
-0.050
-0.4
-10
-5
0.2
0.6
-0.05
0.05
0.10
heuristic score (hn)
original neuron activation
P(80S) orgg P(BOS) ablated
Figure 8: Summary of attention (de- ade-vactivation neuron results in
(a) Distribution of
heuristic score hn for every pair of neurons and heads compared to random neuron directions R. (b:c) path ablations effect of neuron L4.3594 on head L5.HO: ablating positive activation reduces attention to BOS (b) causing the norm to increase (c).
5.3 Attention Deactivation Neurons
In autoregressive models, attention heads frequently place all of their attention on the beginning of sequence (BOS) token (Xiao et al., 2023). . We hypothesise that the model uses the attention to the BOS token as a kind of (de-)activation for the head, where fully attending to BOS implies the head deactivated and has minimal effect. Moreover, we hypothesize that there are individual neurons which control the extent to which heads allend to BOS.
Recall the output of an attention head og for destination token d from source tokens s is given by
M(Sas), Vdn
F. Word, k,- WKTs, Sa. - Ad. softmax.
V. = WVTT: Fo Wo Σ Add
where rs/d is the residual stream at the source / destination token, dn is the bottleneck dimension of the head, and M()) applics the causal attention mask to the attention scores. The calculation of the attention pattern Ads via softmax across the source positions means that the attention given to the source tokens by given destination token sums to one.
Assuming the BOS token is always the first token in the context, the vector WOVBOS is constant for all prompls and contains no semantic information. it has a low norm, altending to BOS scales down the outputs of attending to other source positions while maintaining their relative attention because the attention scores must sum one. If the BOS output norm is near zero, the head can effectively turn off by only attending to the BOS token. In practice, the median head in GPT-2-medium-a has WoVBOS with norm 19.4 times smaller than the average for other tokens.
We can identify neurons which may use this mechanism by a heuristic score hn = W normalized W Wout- Positive scores suggest activation of the neuron will increase the attention placed on BOS, decreasing the output norm of the head, and the opposite for negative scores. Figure X shows the distribution of the scores for all heads in GPT2-mcdium-a compared a unit normalized Gaussian matrix R.
out W WoBs for unit
For given neuron, can measure the effect of activation the attention to BOS and output norm of a given head by path ablation (Wang et all 2022a) of the neuron at a particular destination token. Specifically, we can mcasure the difference in BOS attention and norm of the output of the head between the original run and a forward pass where the contribution of a neuron is deleted (i.e, zero path ablated) from the input of a particular head at the current token position. We perform this procedure over a random subset of tokens in the second half of the context to avoid spurious effects stemming from short contexts. Figure P and R depict the results of these path ablations for the highest scoring neuron in layer for head in attention layer 5. This is an example of an attention deactivation neuron-increasio increasing the activation of the neuron
12

---

Published in Transactions on Machine Learning Research (06/2024)
increases the attention to BOS reducing the output norm of the head ||ou||- See Figure 27 for 5 additional examples of attention (de-)activating neurons.
6 Discussion and Conclusion
Findings In this work, we explore the universality of individual neurons in GPT2 language models, and find that only about 1-5% of neurons pass a certain threshold universality across models. We have shown that leveraging universality is an effective unsupervised approach identify interpretable model components and important motifs. In particular, those few neurons which are universal are often interpretable, can be grouped into smaller number of neuron families, and often develop with near duplicate neurons in the same model. Some universal neurons also have clear functional roles, like modulating the next token prediction entropy, controlling the output norm of an attention head, and predicting or suppressing elements of the vocabulary in the prediction. Moreover, these functional neurons oflen form antipodal pairs, potentially enabling collections of neurons to ensemble to improve robustness and calibration. These findings raise useful lessons and motifs for further interpretability rescarch ($..2)
Limitations Compared to frontier LLMs, we study small models of only hundreds of million parameters and tens of thousands neurons due to the expense of training multiple large scale language models from different random initializations. We also study a relatively narrow form of universality: neuron universality over random seeds within the same model family. Studying universality across different model families is made difficult by tokenization discrepancies, and studying models across larger sizes is difficult due to the expense of computing all pairwise neuron correlations over sufficiently sized text corpus. Additionally, many of our interpretations rely on manual analysis or algorithmic supervision which restricts the scope and generality of our methods. Moreover, our narrow focus on a subset of individual elements of the neuron basis potentially obscures important details and ignores the vast majority of overall network computation.
Future Work Each of these limitations suggest avenues for future work. Instead of studying the neuron basis, our experiments could be replicated on an overcomplete dictionary basis that is more likely to contain the true model fcatures (Cunningham et al.) 2023 Bricken et al. 2023). Motivated by the finding that the most correlated neurons occur in similar network depths, our experiments could be rerun on larger models where pairwise correlations are only computed between adjacent layers to improve scalability. Additionally, the interpretation of common units could be further automated using LLMs provide explanations (Bills et al. )023). Finally, by uncovering interpretable footholds within the internals of the network, our findings can form the basis of deeper investigations into how these components respond to stimulus perturbation, develop over training (Quirke et al. (023), and affect downstream components to further clucidate general motifs and specific circuits within language models.
References
Authropic (2023). Circuits updates - july 2023.
http://oneerriccuuutts////////// htmll
Antverg, O. and Belinkov, Y. (2021). On the pitfalls of analyzing individual neurons in language models. arXiv preprint arXio:2110007483
Bansal, Y., Nakkiran, P., and Barak, B. (2021). Revisiting Model Stitching to Compare Neural Representations.
Barannikov, S., Trofimov, I., Balabin, N., and Burnaev, E. (2022). Representation Topology Divergence: A Method for Comparing Neural Network Representations.
Bau, A., Belinkov, Y., Sajjad, H., Durrani, N., Dalvi, F., and Glass, J. (2018). Tdentifying and controlling important neurons in neural machine translation. arXiv preprint arx::11811.01155
Bau, D., Zhu, J.-Y., Strobelt, H., Lapedriza, A., Zhou, B., and Torralba, A. (2020). Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences.
13

---

Published in Transactions on Machine Learning Research (06/2024)
Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., and Steinhardt, J. (2023). Eliciting latent predictions from transformers with the tuned lens. orXie preprind or arXim:2303.081 12.
Bender, E. M., Gebru, T., McMiilan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference fairness, accountability, and transperency, pages 610 623.
Bengio, Y., Hinton, G., Yao, A., Song, D., Abbeel, P., Harari, Y. N., Zhang, Y.QQ.: Xue, L.., Shalev-Shwartz, S., Hadfield, G., et al. (2023). Managing ai risks in an era of rapid progress. arXio preprint ar orXic:2310.1788.
Biderman, S., Schoclkopf, H., Authony, Q., Bradley, H., O'Brion, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, S., Raff, E., Skowron, A.; Sutawika, L., and van der Wal, O. (2023). Pythia: A suite for analyzing large language models across training and scaling.
Bills, S., Cammarata, N., Mossing, D.,
Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J., Wu, J., and Saunders, W. (2023). Language models can explain neurons in language models. https://openaaipullic blob. core. windows. hh00...oo....oodeeeeeaarrarrrrr http://ooo..paaaaaepaaaereeeeeee indexxiiixx..... htmml
Boix-Adsera, E., Lawrence, H., Stepaniants, G., and Rigollet, P. (2022). GULP: a prdictionnbbsed metric between representations.
Bommasani, R.,, Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bossclut, A., Brunskill, E., et al. (2021). On the opportunities and risks of foundation models. arXiv preprinl ar Stt:2208.07255
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., HoddieldDDodss, Z., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. (2023). Towards monosernnntcctty, Decomposing language models with dictionary learning. Transformer http://taannffmmr-riinniie putittssspu/uuuu habb//22//aaa /tt///200mmmmmmmmm// httmm/moommatti-fftttreeee
Circuits Thread.
Brody, S., Alon, U., and Yahav, E. (2023). On the expressivity role of layernorm in transformers' attention. arXiv preprint arxic:2305. 02582.
Brown, D., Vyas, N., and Bansal, Y. (2023). On privileged and convergent bases in neural network representations. arXiv preprint arXiv:2307.129/1.
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., al. (2023). Sparks of artificial general intelligence: Early experiments with gpt-4. ar preprint ar XrX:::2303.122 12712.
Burns, C., Ye, H., Klein, D., and Steinhardt, J. (2022). Discovering latent knowledge in language models without supervision. arXiv preprint arXio:221..08827.
Cammarata, N., Goh, G., Carter, S., Voss, C., Schubert, L., and Olah, C. (2021). Curve circuits. Distill. https://distilllpubb/ 222/22222/2/2 cercccitcccs/uuueeeee cireuits/curve-cireuiti curve-cireuusss.
Carlsmith, J. (2023). Scheming ais: Will ais fake alignment during training in order to get power? arXiv preprint arXie:2311.088
Casper, S., Boix, X., D'Amario, V., Guo, L., Schrimpf, M., Vinken, K., and Kreiman, G. (2021). Frivolous units: Wider networks are not really that wide. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 6921-6929.
Chughtai, B., Chan, L., and Nanda, N. (2023). A toy model of universality: Reverse engineering how networks learn group operations. In Proceedings of the 40th International Conference Machine Learning, ICML/23. JMLR.org.
Conny, A., Mavor-Parker, A. N., Lynch, A., Heimersheim, S., and Garriga-Alonso, A. (2023). Towards automated circuit. discovery for mechanistic interpretability. arxis preprint Xrv::2234 149977
14

---

Published in Transactions on Machine Learning Research (06/2024)
Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. (2023). Sparse autoencoders find highly interpretable features in language models. arXiv preprint w:Xiv2339.086000
Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., and Wei, F. (2021). Knowledge neurons in pretrained transformers. arXiv preprint or Xrx:21104 0...2166..
Dalvi, F., Durrani, N., Sajjad, H., Belinkov, Y., Bau, A., and Glass, J. (2019). What is one grain sand in the desert? analyzing individual neurons in deep ulp models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages
Dalvi, F., Sajjad, H., Durrani, N., and Belinkov, Y. (2020). Analyzing redundancy in pretrained transformer models. arviv preprint oriiv2004.04010.
Dar, G., Geva, M., Gupta, A., and Berant, J. (2022). Analyzing transformers in embedding space. arXiv preprinl ar Xiv:2200.. 02535.
Ding, F., Denain, J.-S., and Steinhardt, J. (2021). Grounding Representation Similarity with Statistical Testing.
Donnelly, J. and Roegiest, A. (2019). On interpretability and feature representations: an analysis of the sentiment neuron. Advances in Information Retrieval: 41st European Conference on IR Research, ECIR 2019, Cologne, Germany,, April 14 18, 2019, Proceedings, Purt 41, pages 795 802. Springer.
Doshi-Velez, F. and Kim, B. (2017). Towards rigorous science of interpretable machine lcarning. arXiv preprint arXiv:1702. 08608.
Dravid, A., Gandelsman, Y., Efros, A. A., and Shocher, A. (2023). Rosetta neurons: Mining the common units in a model zoo. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1934-1943.
Duong, L. R., Zhou, J., Nassar, J., Berman, J., Olieslagers, J., and Williams, A. H. (2023). Representational dissimilarity metric spaces for stochastic neural networks.
Durrani, N., Dalvi, F., and Sajjad, H. (2022). Linguistic correlation analysis: Discovering salient neurons in doepnlp models. arXiv preprint arXiv:2206. 13288.
Elhage, N., Hume, T., Olsson, C., Nanda, N., Henighan, T., Johnston, S., ElShowk, S., Joseph, N., DasSarma, N., Mann, B., Hernandez, D., Askell, A., Ndousse, K., Drain, D., Chen, A., Bai, Y., Ganguli, D., Lovitt, L.,
Z., Kernion, J., Conerly, T., Kravec, S., Fort, S., Kadavath, S., Jacobson, J., Tran-Johnson, E., Kaplan, J., Clark, J., Brown, T., McCandlish, S., Amodci, D., and Olah, C. (2022a). Softmax lincar units. Transformer Circuils Thread. https:/ http:///rraooeeerrrrriitttssses
http//22////a// 2022/solu/imdex.hii
Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds Z., Lasenby, R., Drain, D., Chen, C., et al. (2022b). Toy models of superposition. arXiv preprint orXiv:2209.1063
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., et al. (2021). A mathematical framework for transformer circuits. Transformer Circuits Thread.
Feng, J. and Steinhardt, J. (2023). How do language models bind entities in context? arXiv preprint 2rX1v:2310.11111.
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2020). The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027
Geva, M., Caciularu, A., Wang, K. R., and Goldberg, Y. (2022). Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv: 2201:2203..1680.
Geva, M., Schuster, R., Berant, J., and Levy, O. (2020). Transformer memories. arXio preprint ar 2rxi::2012.1400
layers are key-value
15

---

Published in Transactions on Machine Learning Research (06/2024)
Godfrey, C., Brown, D., Emerson, T., and Kvinge, H. (2023). On the Symmetries of Deep Learning Models and their Internal Representations.
Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., and Olah, C. (2021). Multimodal neurons in artificial neural networks. Distill, 6(3):330.
Gould, R., Ong, E.., Ogden, G., and Commy, A. (2023). Successor heads: Recurring, interpretable attention heads in the wild. arXi preprint arXiv:2312.0922 09230.
Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troitskii, D., and Bertsimas, D. (2023). Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint [xXi:2305.01610
Gurnee, W. and Tegmark, M. (2023). Language models represent space and time. arXiv preprint arXiv:2310. 02207.
Gwilliam, M. and Shrivastava, A. (2022). Beyond Supervised vs. Unsupervised: Represcntative Benchmarking and Analysis Image Representation Learning.
Hamilton, W. L., Leskovec, J., and Jurafsky, D. (2018). Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change.
Hamrick, J. and Mohamed, S. (2020). Levels of analysis for machine learning. arXiv preprint 055112004.001007.
Hendel, R., Geva, M., and Globerson, A. (2023). In-context learning creates task vectors. arXiv preprint arXiv:2310. 1.111..........
Hendrycks, D. and Gimpel, K. (2016). Gaussian error linear units (gelus). arXiv preprinl ar 084 15.
Hendrycks, D., Mazeika, M., and Woodside, T. (2023). An overview of catastrophic ai risks. arXio preprint arX1v:230.11901.
Honnibal, M., Montani, I., Van Landeghem, S., and Boyd, A. (2020). spacy: Industrial-sstennthh natural language processing in python.
Hryniowski, A. and Wong, A. (2020). Inter-layer Information Similarity Assessment of Deep Neural Networks Via Topological Similarity and Persistence Analysis of Data Neighbour Dynamics.
Huang, J., Geiger, A., D'Oosterlinck, K., Wu, Z., and Potts, C. (2023). Rigorously assessing natural language explanations of neurons. arXiv preprint or 2tt22300.10322
Jastrzębski, S., Arpit, D., Ballas, N., Verma, V., Che, T., and Bengio, Y. (2017). Residual connections encourage iterative inference. arXiv preprint (i..:i:11000....
Karamcheti, S., Orr, L., Bolton, J., Zhang, T., Goel, K., Narayan, A., Bommasani, R., Narayanan, D., Hashimoto, T., Jurafsky, D., Manning, C. D., Potts, C., Ré, C., and Liang, P. (2021). Mistral journey towards reproducible language model training.
Khrulkov, V. and Oseledets, 1. (2018). Geometry Score: A Method For Comparing Generative Adversarial Networks.
Klabunde, M., Schumacher, T., Strohmaier, M., and Lemmerich, F. (2023). Similarity of Neural Network Models: A Survey of Functional and Representational Measures.
Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. (2019). Similarity of neural network representations revisited. In International conference on machine learning, pages $519-3529, PMLR.
Lange, R. D., Rolnick, D. S., and Kording, K. P. (2022). Clustering units in neural networks: upstream downstream information.
Li, Y., Yosinski, J., Clune, J., Lipson, H., and Hopcroft, J. (2015). Convergent learning: Do different neural networks learn the same representations? arXiv preprint arxiv 51 1.07543.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Li, Y., Yosinski, J., Clune, J., Lipson, H., and Hoperoft, J. (2016). Convergent Learning: Do different neural networks learn the same representations?
Liao, I., Liu, Z., and Tegmark, M. (2023). Generating interpretable networks using hypernetworks. arXiv preprint ar 21::22312.000511
Lim, J. and Lauw, H. (2023). Disentangling transformer language models as superposed lopic models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8646-8666.
Lin, B. (2022). Geometric and Topological Inference for Deep Representations of Complex Networks. In Companion Proceedings of the Web Conference 2022, pages 334 338.
Lu, Y., Yang, W., Zhang, Y., Chen, Z., Chen, J., Xuan, Q., Wang, Z., and Yang. X. (2022). Understanding the Dynamics of DNNs Using Graph Modularity.
Marr, D. (2010). Vision: computational investigation into the human representation and processing of visual information. MIT press.
McDougall, C., Commy, A., Rushing, C., McGrath, T., and Nanda, N. (2023). Copy suppression: Comprehen-
sively understanding an attention head. arXiv preprint orXiv:2310.0400
McGrath, T., Rahtz, M., Kramar, J., Mikulik, V., and Legg, S. (2023). The hydra effect: Emergent self-repair in language model computations. arXiv preprint Xix:2290.157711
Merullo, J., Fickhoff, C., and Pavlick, F. (2023). Circuit component reuse across tasks in transformer language models. arXiv preprint arXbv:2310.087/4.
Morcos, A. S., Raghu, M., and Bengio, S. (2018). Insights representational similarity in neural networks with canonical correlation.
MU, J. and Andreas, J. (2020). Compositional explanations of neurons. Advances in Neural Information Processing Systems, 33:17153 17163.
Nanda, N. (2022). Transformerlens.
Nanda, N., Chan, L., Liberum, T., Smith, J., and Steinhardt, J. (2023). Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiw:2301.022r1
Ngo, R., Chan, L.., and Mindermann, S. (2023). The alignment problem from a deep learning perspective.
Nguyen, A., Yosinski, J., and Clune, J. (2016). Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks.
Nostalgebraist (2020). Interpreting gpt: The logit lens.
httpe://www.aliigmeesssfrrrrrrrp
Olah, C. (2021). Interpretability vs neuroscience.
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. (2020a). An overview of carly vision in inceptionvl. Diställ http://ssilllpup/////////reeeiis
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. (2020b). Zoom in: An introduction to circuits. Distill, 5(3):e00024 001.
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., HatfieldDDodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousso, K., Amodci, D.,, Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. (2022). In-context learning and induction heads. Transformer Circuits Thread. https:/ /toss///ransfommrr Strcuits.pub/2000 /tt///222//nnnonteee weattexx--lerrinng -aaaaninnaan---aa and-aanddeeen induction- heads/ midadddiiixx..t....
Quirke, L., Heindrich, L., Gurnee, W., and Nanda, N. (2023). Training dynamics of contextual n-grams in language models. arXiv preprint ar Xiv: 1.00863.
17

---

Published in Transactions on Machine Learning Research (06/2024)
Radford, A., Jozefowicz, R., and Sutskever, I. (2017). Learning to generate reviews and discovering sentiment. arviv preprinl ar Xrxi::1100 2704.01444.
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I,, et al. (2018). Improving language understanding by generative pre-training-
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, (2017). SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability.
Sajjad, H., Durrani, N., Dalvi, F., Alam, F., Khan, A. R.,, and Xu, J. (2022). Analyzing encoded concepts in transformer language models. arXiv preprinl Xiv: 13289.
Schubert, L., Voss, C. Cammarata, N., Goh, G., and Olah, C. (2021a). High-low frequency detectors. Distill. wwtp/p/0///rruiiiiiiee
Schubert, L.,, Voss, C., Cammarata, N., Goh, G., and Olah, C. (2021b). High-low frequency detectors. Distall, 6(1):e00024 005.
Shahbazi, M., Shirali, A., Aghajan, H., and Nili, H. (2021). Using distance the Riemannian manifold to compare representations in brain and in models. Newrolmage, 239:118271.
Tang, S., Maddox, W. J., Dickens, C., Dicthe, T., and Damianou, A. (2020). Similarity of Neural Networks with Gradients.
Todd, E., Li, M. L., Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. (2023). Function vectors in large language models. arxiv preprint arXin:2310.15213
Variengien, A. and Winsor, F. (2023). Look before you leap: A universal emergent decomposition of retrieval tasks in language models.
Voita, E., Ferrando, J., and Nalmpantis, C. (2023). Neurons in large language models: Dead, n-gram, positional. arXiv preprinl Xix::22309. 04827.
Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. (2022a). Interpretability in the wild: circuit for indirect object. identification in gpt-2 small. arXiv preprint arXin:2211.00598.
Wang, L., Hu, L., Gu, J., WI, Y., Hu, Z., He, K., and Hoperoft, J. (2018). Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation.
Wang, T. and Isola, P. (2022). Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere.
Wang, X., Wen, K., Zhang, Z., Hon, L., Liu, Z., and Li, J. (2022b). Finding skill neurons pre-trained transformer- baseed language models. arXiv preprint 221.::221..0784.
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022). Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 214-229.
Williams, H., Kunz, E.,, Kornblith, S., and Linderman, S. W. (2022). Generalized Shape Metrics on Neural Representations.
Xia, M., Gao, T., Zeng, Z., and Chen, D. (2023). Sheared llama: Accelcrating language model pro-training via utructured pruning. arXiv preprinl ar Xiv:22300. 2.x::2310..0694.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. (2023). Efficient streaming language models with attention sinks. Xiv preprint 2rX.::309. 17453.
18

---

Published in Transactions on Machine Learning Research (06/2024)
Xin, J., Lin, J., and Yu, Y. (2019). What part of the neural network does this? understanding lstms by measuring and dissecting neurons. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5830.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., al. (2022). Opt: Open pro-trained transformer language models. arXiv preprint arXiv:2205. 01068.
Zhong, Z., Liu, Z., Tegmark, M., and Andreas, J. (2023). The clock and the pizza: Two stories in mechanistic explanation of neural networks. arXiv preprint ar 2xXx:23066.78444
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazcika, M., Dombrowski, A.-K., et al. (2023). Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiu:2310001405
A Additional Discussion
A.1 Why expect universal neurons to be more monosemantic?
The intuition for the connection comes from Elhage et al. (2022b) which showed that important features (those that are constructed to have maximum effect on the loss) are the features which get their own dedicated neuron (making the neuron monosemantic). Since these GPT2 models are training on the same data, the important features should be similar across the networks. So monosemantic neurons primarily represent important features, and the important features are shared (i.e., universal) across the networks, then we would expect the neurons which are universal across the networks to be representing the same set of important features.
Another line of argument is to consider the probability of polysemantic neuron being universal. It is extremely unlikely that there would exist a pair of neurons which activates for the same k > 0 unrelated features, assuming unrelated fcatures have roughly similar probabilitics of being assigned to ncurons. Morcover, this simple model suggests that as k + is much more likely for there to exist a neuron which represents the same set of features.
A.2 What are these observations useful for?
Our motivation here is primarily to improve our understanding of models for interpretability sake, rather than make models more performant. Specifically, we think interpretability is largely an immature field, without much grounding thcory. Therefore, we think it's inherently valuable gain lot of "empirical surface area" real worksrks constrain the hypothesis space and develop the theory and practice of interpretability.
Example insights from our paper that might help future interpretability researchers
Single neuron ablations can be misleading if there is an ensemble of similar neurons there are neurons which consistently cancel each other out.
The depth specialization results and the prediction followed by suppression neuron transition emphasize the sequential and residual nature or model processing.
Our results on entropy neurons exemplify a concrete case where it would not be valid to lincarize layer norm.
Our results on attention deactivation neurons show how certain features might be difficult to interpret, because they features which take internal actions rather than represent external inputs.
19

---

Published in Transactions on Machine Learning Research (06/2024)
B
Additional Empirical Details
All of our code and data is available at httppe///gtaauu.coo///////////ie -nurooss
Most of our plots in the main text (and therefore neuron indices) correspond to the HuggingPace model stt
with our additional correlation experiments being conducted on - -122a-l1l12- and EleutherA//pythaa-00m
B.1 Weight Preprocessing
We employ several standard weight preprocessing techniques to simplify calculations (Nanda 2022
Folding in Layer Norm Most layer norm implementations also include trainable parameters 7 E IR" and
b E R
Exxx * * + b.
LayerNorm(x) v /Var(x)
(6)
To account for these, we can "fold" the layer norm parameters in to Win by observing that the layer norm parameters are equivalent to a linear layer, and then combine the adjacent linear layers. In particular, we can create effective weights
Wer Win diag(7) bell b + Winb
Finally, we can center the reading weights because the preceding layer norm projects out the all ones vector. Thus can center the weights Wat becomes
W Wen(i.;)= Wir((i Wir((i;)
Writing Weight Centering Every time the model interacts with the residual stream it applies a LayerNorm first. Thus the components of W Won and bout that lic along the all-ones direction of the residual stream have effect the model's calculation. So, again mean-center Wout and bout by subtracting the means of the columns of W out
w Wonn (:, Wour(; W (:.
Unembed Centering Additionally, since softmax is translation invariant, we modily Wo into i) =) w;
For both of theses, see the transformer lens documentation for more details.
The purpose of all of these translations is remove irrelevant components and other parameterization degrees of freedom so that cosine similarities and other weight computations have mean
B.2 Correlation Computations
We compute our correlations over a 100 million token subset of the Pile test set (Gao et al. 2020 to context length of 512 tokens. We compute correlations over all tokens that are not padding, beginning- of-sequence, or new-line tokens.
tokenized
Efficient Computation Because storing neuron activations for two models over 100M tokens would be 36 petabyles of data, require streaming algorithm. To do so, observe that given a pair of neuron activations (x1 v1) ..... (In, yn)} consisting of pairs, the correlation be computed
Di (a: 2) (v. 5)
Σ, Telh ndy
Pay
E (x) -x)² E" (v. -7)
VE, x.
n
/ E.
V. nij
20

---

Published in Transactions on Machine Learning Research (06/2024)
where 2, y the sample mean. Therefore, instead of saving all neuron activations, can maintain four neuron dimensional vectors and one n_ neuron x neuron matrix corresponding to the running neuron activation means in model A and model B, a running sum of each neurons squared activation, and a running sum of pairwise products. At the end of the dataset, we perform the appropriate arithmetic to combine the results into pairwise correlations for all models.
B.3 Model Hyperparameters
Property
GPT-2 Small
GPT-2 Medium
Pythia 160M
layers
12
24
12
heads
16
dmodel
768
1024
768
dvocab
50257
50257
50304
dMLP
3072
4096
3072
parameters
160M
410M
160M
context
1024
1024
2048
activation function
gelu gelu_new new
gelu new
gelu_new
gelu
pos embeddings
absolute
absolute
RoPE
rotary percentage
N/A
N/A
25
precision
Float-32
Float-32
Float-16
dataset
OpenwebText
Openwebleet
Pile
Pdropout
0.1
0.1
0
Table 1: Hyperparameters of models
C Additional Mysteries
We conclude our investigation by commenting on several miscellaneous results that we think are worth reporting but that we do not fully understand.
C.1 Cosine and Activation Frequency
An unexpectedly strong relationship we observed is the correlation between activation frequency of neuron and the cosine similarity between its input and output weight vectors cos cos(Win, Wout) shown in Figure g Almost all neurons with very high activation frequency have input and output weights in almost opposite directions. These neurons are predominantly in the first quarter of network depth and have small excess correlation, i.c., they are not universal as measured by activation. We also find it noteworthy that there appears to an approximate ceiling and floor on the cosine similarity of approximately +0.8.
C.2 Duplication and Universality
While neuron redundancy has been observed in models before (Casper et al. 2021 Dalvi et al. 2020 and large models can be effectively pruned (Xia al., 2023). we were surprised by the number seemingly duplicate universal neurons we observed (c.g., Figure 16 or the 105 BOS neurons we observed). Naively, this is surprising, as it seems wasteful to dedicate multiple neurons the same feature. Larger models have more capacity and are empirically much more effective why have redundant neurons when you could instead have one neuron with twice the output weight norm?
A few patential explanations are (1) these models were trained with weight decay, creating an incentive to spread out the computation. (2) Dropout-howwvee, however, in these models dropout is applied to the output of the MLP layer, rather than the MLP activations themselves. (3) These neurons are vestigial remnants that were useful earlier in training (Quirke et al., 2003), but are potentially stuck in local minima and are no longer useful. (4) The duplicated neurons are only activating the same common features, but polysemantic
21

---

Published in Transactions on Machine Learning Research (06/2024)
2pl2 2a
gpl mediimm ectivation frequency 0.4
pythia-160m 160m
ectivation frequency
activation frequency 0.6
1.0
0.0
1.0
corr: -0.69
cor:: -0.63
con: -0.58
0.75
0.50
cos Wan)
0.25
0.00
-0.25
-0.50
0075
50
40
561
neurrnn cOunT
meuron countL
neuron count
Figure 9: Activation frequency of neuron (fraction of activation values greater than zero) versus cosine similarity of neuron input and output weights for GPT2-small-a (left), GPT22medium-a (center), and Pythia- 160M (right).
gpt2-small-a
gpt2 med--medium-a
pythia 160m
neuron universal
10
neuron universal non-universal
neuron universal
10.0
density
5.0
0.0
0.2
0.4
0.6
0.8
0.2
0.6
0.8
0.2
0.4
max,costws w
max,cos(w w(.)
max,cos(wt w(,)
neuron universal non- non-uriversal
neuron universal non- universal
neuron universal
12.5
10.0
non- universal
7.5
2.5
0.2
max,cosiV W(..)
max W
max,cos W W
Figure 10: Distribution cosine similarities of most similar neurons measured by input weights (top) and output weights (bottom) for GPT2-small-a (left), GPT2-madium-a (middle), and Pythia-160M (right) colored by universality (e 0.5).
22

---

Published in Transactions on Machine Learning Research (06/2024)
model
model
10
model
gpt2-small-a
apt2-small--
gpt2 gpt2-small-a
neuron density
gpt2-me a
gpt2 medium-a
8
get2-medium-a
pythia-160m
pythia-160mm
pythia-160m
4-
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.8
correlation
[b-e] baseline correlation
excess correlation
Figure 11: Empirical distribution of max neuron correlation averaged across models (left), max baseline correlation averaged across models (middle), and the difference denoted as the excess correlation (right)..
with different sets of rarer features. (5) Ensembling, where each neuron computes the same feature but with some independent error, and together form ensemble with lower average
By measuring redundancy in terms of similarity in weights (Figure 10p, we find very few neurons which are literal duplicates, providing more evidence for (4) and (5). Based on the much higher level similarity for universal neurons, possible this effect is relatively small in general.
C.3 Scale and Universality
As mentioned in S GPT2-medium and Pythia-160M have a consistent number of universal neurons (1.23% and 1.26% respectively), while GPT2-small-a has many more 4.16%. In Figure 11 we show the distribution of max, baseline, and excess correlations for all models, where we see that GPT2-medium and Pythia-1600 160M have almost identical distributions while GPT2-small is an outlier. GPT2-small also has correspondingly greater weight redundancy shown in Figure 1O One explanation for this is the number of universal neurons decreases in larger models. This is potentially implied by results from (Bills al. 2023) who observe larger models have fewer neurons which admit high quality natural language interpretations. However, without additional experiments on larger models trained from random seeds, this remains an open question.
D Additional Results
(a) neuron correlation distributions
(b) max correlation range
(c) most similar neuron layer
max [b-e] max neuron correlaiion
50
1
correlation
% most similar neurons in layer
max
baseline
40
neuron count
excess
10
0.6
30
0.4
10
layer
20
neuron
mean ylx
10
10
0.0
0.0
0.2
0.4
0.6
0.8
100
0.0
0.2
0.4
0.6
0.8
10
4
10
correlation
min [b-e] max neuron correlation
layer gpt2 (malll[b-el]
Figure 12: Summary of neuron correlation experiments in GPT2-small-a. (a) Distribution of the mean (over models b-e) max (over neurons) correlation, the mean baseline correlation, and the difference (excess). (b) The max (over models) max (over neurons) correlation compared to the min (over models) max (over neuron) correlation for each neuron. (c) Percentage of layer pairs with most similar neuron pairs.
23

---

Published in Transactions on Machine Learning Research (06/2024)
(a) correlation distributions
(b) max correlation range
(c) most similar neuron layer
correlation
[1-3] neuron correlation
50
10a
max]
most similar neurons in layer
baseline
0.8
layer in pythia- 160m
40
neuron count
excess
10a
0.6
30
0.4
20
10
neuron
mean yix
10
10
0.0
0.00
0.25
0.50
0.75
1.00
0.0
0.2
0.4
0.6
0.8
10
10
correlation
min [1-3] max neuron correlation
layer pythia 160m seed[1-3]
Figure 13: Summary of neuron correlation experiments in Pythia-160m. (a) Distribution of the mean (over models b-e) max (over neurons) correlation, the mean baseline correlation, and the difference (excess). (b) The max (over models) max (over neurons) correlation compared to the min (over models) max (over neuron) correlation for each neuron. (c) Percentage of layer pairs with most similar neuron pairs.
100
109
109
fraction neurons
10-1
10-1
10-1
10.2
10-2
10-2
10-3
10-3
model
model
model
gpt2-small-a
gpt2-small-a
gpt2-small-a
10-4
gpt2-medium-a
10-4
gpt2-medium-a
10-4
gpt2-medium-a
-5
1ythia-160m
1
pythia-160m
10-5
pythia-160m
0.2
0.6
1.0
0.2
1.0
0.0
10
mean [b-e] max neuron correlation
mean [b-e] max baseline correlation
excess correlation
Figure 14: Complement cumulative distribution function for correlation metrics across all model families.
24

---

Published in Transactions on Machine Learning Research (06/2024)
gpt2 gpt2-medium-a neuron metrics by layer for universal and non-universal neurons
is universal
25
True
We kurtosis
20
15
10
1.5
1.0
We skew
0.0
-0.5
-15
0.50
costero. Mod!
0.25
0.00
-0.5
1.0
12 penaal
0.8
0.6
0.4
0.2
activatin Sparsity
0.6
0.4
activation Skew
-3
activatin kurtosis
10
10
11
12 13
14
16 17
18
19
20
21
22
23
layer
Figure 15: Distribution of neuron metrics for universal and non-universal neurons in GPT2-medium-a by layer. From top bottom: the kurtosis of cos(Wu, W. Wout), the skew of cos (We, Wous), cosine similarity between inpul and output weight cos(we Win Wout), weight decay penalty [[ 12 + ||W ll activation frequency (percentage of activations greater than 0), the pre-activation skew, and the pre-activation kurtosis.
25

---

Published in Transactions on Machine Learning Research (06/2024)
unigram token role (see subplot title for unigram) beginning of word
word middle word
other token
the unigram neurons
of unigram neurons
at unigram neurons
pre actaactttttonn
011001
0.1581
1.545
0.197
0.891
1.3522
0.1802
0.2496
1.3566
that unigram neurons
by unigram neurons
unigram neurons
pre actavaatttoonn
016118
0.1908
0239
11281
01138
0.2936
1933
0.3223
0.3423
12165
3716
on unigram neurons
for unigram neurons
with unigram neurons
0.2088
0.818
1,2009
0.3954
0.3963
0.771
in unigram neurons
unigram neurons
unigram neurons
pre eee-artivation
0.4027
1.3722
unigram
unigram
unigram
pre eee-activatinn
0.1882
0.3238
2800
0.185
13344
12.1160
0.509
0.671
1.3727
unigram neurons
unigram neurons
7.5
5.0
pre activaattion
25
-5.0
01115 0.8825 16227 14.2879 15. 15.3217 17. 744 19.4 22.1558 23
1.0081
13:2394
16.683
17.199
18 18.2583
21.398
23.3857
Figure 16: Duplicate unigram neurons in GPT2-medium-a. Each subplot depicts several neurons which activate particular token, broken down by whether this token exists standalone word, is the first token in a multi-token word, or is non-first token in multi-token word, versus all other tokens (e.g., "an," "an aanlagram," "Gigaan tism").
26

---

Published in Transactions on Machine Learning Research (06/2024)
to wunguam
contains_commy
to wnigram
onee _eediiiit
contains period
10a
10a
count (log scale)
10a
false
false
10a
false
false
107
false
10
10a
107
10
10
10
107
10
10a
10
101
10
10
-2
-2
2
-2
10.32/0 pre ree-activation on
pre-activation
111116666 pre-activation
2504 pre-activation
LO pre-activation
i__ possessive_pronoun
contains comma
and unigram
is_ neutral_pronoun
from unigrram
true false
10
tuu faise
true talse
true false
true faise
count (1og scale)
10a
10a
10
10
107
107
10
10a
10a
101
101
10
10
L1.1729 pre-activation
L1. 1304 pre-activation
LO 919 pre-activation
LO 2234 pre-activation
L1 1813813 pre-activation
contains_ colon
contains_ caatis_opee__areee
is c_.ooraatiieeeecc _contrastive_onnuutton is second Baanndd_ersoo__eeee prrson_proonon
this unigram
true
10
true
true
true
true
count (log scale)
107
10
10a
10a
107
10
10
10a
10
10
10a
10
10a
10
10
-2
-2.5
-7
L1.1031
activation
LO. 1686 pre activation
L0.3325 pre activation
LO 2424 pre activatioon
LD 2946 pre activation
is_year
male pronoun
end w_ing
all numeric
contains_ caaoasscoooo__piaeeeee
true
true false
true false
count (log scale)
10
false
10
false
10
10
false
107
10a
10
10a
10a
10
10a
10
10a
10
-2
0
-2
0
-2
-2
3875 pre-activation
pre-activation
553 pre-activation
LO pre-activation
second person pronoun
contains colon
and unigram
contains open paren
interrogative pronoun
true
10a
true false
10a
10a
true
count (log scale)
107
false
10
false
10a
false
10a
false
10a
10
10
10
10
10a
10a-
10a
10a
10
102
10a
101
10
103
10
-2
-2
-2
-2
-2
1413 pre-activation
pre-activation
L1 1347 pre-activation
2676 pre-activation
pre-activation
85
second person pronoun
w. ing
unigram
demonstrative pronoun
there unigram
10a
true tase
10
true faise
true false
10a
true faise
true false
count (log scale)
103
10a
107
10
10
10
102
10
10
10a
10a
10a
10a
10
101
101
10
-2
L1 11:377 pre-activation
L2:2130 pre-activation
10.2510 pre-activation
L10.1695 pre-activation
L1 603 pre-activation
Figure 17: Universal unigram neurons in GPT2-mediun-a.
27

---

Published in Transactions on Machine Learning Research (06/2024)
(LO.1761)
"d" 3758)
3046)
token
token
taken
taken
token
detritution
0.25
begin A/a
Als
8/b
begin R/b other
begin Ctc other
begin Didd
DMd
begin Pyt other
0.0
other
other
0.15
0.10
0.05
2
22
"9" neuron (L1 131)
"th neuron (L1 47)
* neuron (L1 3349)
" neuron [LO 2825)
"i"" neuron (LO 2081)
0.25
decributioon
begin
begin
0.20
0.15
0.10
0.05
0.00
neuron (LO-2517)
"m" neuron (LO 35265
"n" neuron (L1. 2803)
"p" neuron (LO.2043)
"r" neuron (LO. (.8))
Inken
token
taken
toben
tokem
detribttion
begin
Lt
M/m
begin Mim
Kmn
begin Min other
begin Pyp
P/p
begin RMr
R/y
other
other
other
-1
-1
-2
0
(LO. 20230233
1629)
(L1. (.404)
0.25
WBw begin
begin
begin other
0.00
other
other
0.15
0.10
0.05
0.00
0
2
-2
pre-ectrvation
Zne waltinn
Figure 18: Universal alphabet neurons in GPT2-medium-a
28

---

Published in Transactions on Machine Learning Research (06/2024)
L5.551
L6. 16.443
L4.3859
L5.1174
L4.2642
previoun token article
previous token coordinating conjunction other
previous taken has comma other
previous token coordinating conjunction other
previous token
envoncc detribution
015
"In" other
other
0.10
0.05
2
-2
-5
-2
L3.155
L4.1705
L4.1055
L4.3578
L4.2386
previous taken "the"
previous token
previous taken article other
previous token auxillary passive other
previous token has comma other
empircall diterbutooo
other
other
0.10
0.05
0.00
L10. 38.3
L3. 3939
L4 3454556
L3.1069
L4 44.3623
emperial destrbuution
0.15
auxillary passive
other
other
other
other
0.00
2
-25
67.
L6.4023
L3.2838
14.1446
L3 13.2817
previous taken
previous token suxillary other
previous taken
previous taken
previous taken quxilaary
empercal detribuutios
0.15
adposition
0
-2.5
0.0
2.5
L14.316
15.1351
L4.3364
L719968
previous taken "In"
previous token
previous taken "there" other
previous taken article
previous token auxillary passive
empincal detributton
0.15
0:10
-2
0
2
-4
-2
-2
0
2
-6
-4
-2
0
2
-4
-2
activation
pre-act iee-activation
pre ontiactivatin
pre activattiionn
pre-activation
Figure 19: Universal previous token neurons in GPPT2-ediumm
29

---

Published in Transactions on Machine Learning Research (06/2024)
Layer O,
Layer
Layer O, Neuron 2542
Layer 1064
pre mean
1.5
0.5
post mean
+/- post
0.5
0.5
0.0
0.5
-10
-0.5
-15
-1.0
-1.0
-2.0
-1.5
-1.5
-2.5
200
400
200
200
400
200
400
Layer 2, Neuron 2989
Layer 1. Neuron 2813
Layer 2. Neuron 2618
Layer 1. Neuron 322
0.5
0.5
0.0
0.0
0.0
-0.5
-0.5
activation
-1.0
-0.5
-1.5
-2.0
-2.0
-2.5
-2.
-3.0
-3.0
200
400
200
400
200
400
200
400
Layer 2, Neuron 1172
Layer 1. Neuron 290
Layer 1. Neuron 2076
Layer Neuron 1190
M
&civation
0.0
-0.5
0.5
-1.0
-1.0
-0.5
-10
-1.5
-15
-15
-20
-2.0
-1.0
-2.0
-2.5
-2.5
Layer Neuron 1699
Layer Neuron 1158
Layer Neuron 2593
Layer Neuron 1205
0.5
V
0.0
0.0
actvvttoo
-0.5
-055
-0.
-0.5
-1.0
-10
-10
-1.0
-15
-1.5
-1.5
-1.5
2.0
-2.0
-2.5
-2.5
-2.0
200
400
200
400
200
400
200
400
Layer 1, Neuron 1369
Layer 1, Neuron 2613
Layer 1. Neuron 590
Layer 1, Neuron 2883
0.5
0.
0.0
-0.5
-0.5
-0.5
xtivation
-1.0
00.5
-15
-1.0
-20
-.0
-2.0
-25
-25
-1.5
-3.0
-300
200
400
200
400
200
400
200
400
position
position
position
position
Figure 20: Universal position neurons in GPT2-small-a.
30

---

Published in Transactions on Machine Learning Research (06/2024)
L1.3259
10.715
L0.1951
L3 2817
L15.3504
taken
taken
Ialann
taken
token
1ispaay_nng
spacyy __g
0.12
0.10
0.00
0.06
0.04
0.02
0.00
L4.3623
115.1184
L12 3268
L14.164
L14.4012
0.16
0.14
15kmn
takke
talken
baken pssect
talinn 5.5pacy_net
0.12
0.10
0.08
0.06
0.04
0.02
0.00
LB -5-4
L13 113.1854
L12. 32.30
L7-1968
L18.2157
0.16
0.14
tilken
p.ppurall
talken
J_specy_pron
taken 1s_spacy_neg
talien
tollen
0.12
other
0.10
0.08
0.06
0.04
0.02
0.00
983
121 2932
L11 3285
17 1044
252250
0.16
0.14
tiken
t5ken
taken
taken Huplural
taken
verb
h_spacy__ux other
empiical detne
0.12
other
0.10
0.08
0.06
0.04
0.02
0.00
L17.2608
3791
L15 1030
1274
0.16
dxtreution
0.12
0.14
1s, s_.saay__ng nog
other
taken
is first prrson other
Taken
is J.5pacy puxpass other
taken
other
Taken
$J_ppoyy_oe
taken suxpass diher
0.10
0.08
0.06
0.04
0.02
activttion
actaccivaaion
activiettion
Figure 21: Universal syntax neurons in
31

---

Published in Transactions on Machine Learning Research (06/2024)
12,3305
19.3200
L34.3535
L4.2
L3.5433
0.27
0.08
D00
0.05-
0.56
Fall
0.00
0.04
5.00
0.00
0,04
0.54
0.04-
0.0
0.33
0.02
0.21
0.72
0.52
0:01
2067
123.2349
L16. 1683
L1B 3733732
L1. L1,903
dutibuttinn
0.27
5.0
Sitribation
0.00
0.27
grweterg
Buter
0.50-
1
0.55
0.04
0.33
coe Bis
0:03
0.04
00:22
6.00
0.22
0.07
0.51
0:01
0.23
0.20
0.00
0.52
0.00
-2
-2
112.
132.1919
115.200
Esstroutio
sittrbution gutantarg
deiibel oY
4 Butte jadentery
0.00
0.33
0.26-
6:05-
0.55
0::34
0:32-
0:37
0.2
0.00
0:01
0.52
0.00
0.59
0:00
0.51
0.90
0.00
0.20
5.00
0.28
vedan
0.50
erpatea pontaatio
0.23
0.00
Tull
poweberg
0.02
0.4
0/42-
0.00
0.22
0.32
00:00
0.23
0.00
0.42
0.00
L.7.515
0:00
L2
0.07
L72.1855
exvvouation 00en9022011
1000 cal 0ar13224
0.54
0.00
0.54
0.55
0.00-
0.05
6.00
0..4
0.22
0.82
00:22
0.52
0.02
0/02
0.53
0.20
0.22
0.00
L35.50
123 969
120 300
12 4059
13.2015
0.37
3,07
024 dari4tee
eurpprt
0.06
euopprt
tedia
0.23
5:04-
10.35
0.44
8.00
0.22
0.22
5/00
0.22
D.02
0:21
0/01
Figure 22: Universal context neurons in GPT--medium-..
32

---

Published in Transactions on Machine Learning Research (06/2024)
pythia-410m
pythia-410m
percentile
cost Wael kurtosis
1-99%
5-95%
10-90%
cost W.r)
10a
15-85%
-2
median
4
layer percentile blue: kurtosis
10 red: kurtosis 10
mean
10-90%
1.0%
5.55%
10 10-90%
15-85%
25-75%
median
0
10
15
20
10
15
20
layer
layer
pythia-1b
pythia-1b
layer percentlle
coslWo. Wad) kurtoss
1-99%
cos/W.. Waz) skew
5-95%
2.5
0.0
15-85%
10
25-75%
-2.5
-5.0
percentile blue: kurtosis
10 red. kurtosis 10
mean
1-99%
5-95%
10-90%
15-85%
25-75%
median
-7.5
1-99%
5-95%
10 10-90%
15 35-85%
25-75%
median
10
12
14
10
12
14
layer
layer
pythia-1.44 -1.4b
pythia-1.44 -..ii--..4bb
10a
layer percentile
costWu. Wad) kurtosis
1-99%
cost Waz) skew
5-95%
2.5
10.90%
0.0
15-85%
25-75%
-2.5
median
layer percentile blue: kurtosis
10
red: kurtosis 10
1.9%
5.95%
10 90%
15 15-85%
25-75%
median
1.0%
10-90%
15-85%
25-75%
-7.5
20
layer
layer
pythia-2.88 8b
pythia-2.8 8b
10a
layer percentile
costW..WW.. Waz) kurtoss
1.99%
10-90%
15-85%
10a
25-75%
median
layer percentile | blue: kurtosis > 10
red: kurtosis < 10
mean
1-99%
5955%
10-90%
15-85%
25-75%
median
20
30
20
30
layer
pythia-6.98
pythia-6.9b
cos/W.. (Wa) kurtosis
percentile
10
1-99%
5-95%
5
10-90%
15-85%
cos/
10a
median
-5
layer percentile blue: kurtosis >= 10
red: kurtosis 10
mean
1-00%
15-85%
25-75%
-10
1-99%
5955%
10-90%
15-85%
25-75%
median
10
15 layer
20
25 30
10
15
20
30
layer
Figure 23: Distribution of vocabulary composition statistics for five different Pythia models mcasured over layers. Left shows percentiles of cos( (os, W out) kurtosis. Right shows percentiles of cos (Wr, Wr, W oux) skew broken down by whether neuron has cos( Wu, W Wout) kurtosis greater than or less than 10.
33

---

Published in Transactions on Machine Learning Research (06/2024)
L18.8 836
L25 2205
L23 25.23300
122.73
L25 2774
10'
2..econd,
10
101
1,male,
Tue
101
10'
10
1a02n court
100
10
107
10
10.
30"
100
30"
300
10"
L20. 1601
123.2110
121.1920
L23.390 223.3803
L22 1603
10a
contains questtio
2_one_1 digit
concains exclamation
15. __.ontrastive conguction Yru
d_relative pronoun
Yos
Frue
True
Fuu
10
10
10
False
10'
False
10
False
108ch count
207-
10
10'
10
100
104
10'
100
100
107
100
10a
121.
332
123 2657
2965
7800
300
contaims sppottrppp
[t_ynar Tue False
contains dige
30"
containe, close parmn
contains, quictation
300
Palier
Tue
10
10a
Fabse
Tue
Tue
10'
Falue
True
tokkn
202
10
30
107
107
107
10-
300
100
10
10
0.0
0.4
0.0
0.2
0.4
0.0
0.4
-0.2
02
3151
10'
month Tiee False
contains semicolon
contains open bracket
containns open paren
d..emnstrratvee pronoun
TRue
Tue
Tue
Tue
107
10'
Paise
Talse
30
Faise
10a
Faise
20
10
30
100ent
300
100
301
10
10
10
12
0.2
0.4
-0.6
-0.4
-0.2
0.0
-0.6
-0.4
-0.2
0.0
-0.6
-0.4
-0.2
-0.6
-0.4
0.0
L23. 2333
L23 1033
L19.3794
123.2182
L23.219
contains currency
contains_ colon
11_taae
1__possesivee puronou
containo, exclamation Tue
30
False
Tue
10
False
Tue
70a
Tue Fasee
10
Fulse
Tue
Fasse
202
10
102
10a
300
10
10'
102
100
10P
-0.4
-0.
0.0
-0.6
-0.4
-0.2
0.0
0.0
02
0.0
0.2
0.4
0.0
0.2
0.4
0.6
123.2772
120.13
L10 3483
L21 59.993
L22.
1585
allcaps
end_weed
end_wwing
Paton
10'
Tue Palse
10
Palse
Tuu
301
Falue
Iree
10
False
Bue
20
107
107
30
10'
10
105
100
107
-
0.1
0.0
0.1
0.0
02
0.3
-0.2
-0.1
0.0
0.1
0.0
0.2
0.0
0.2
044
COsSWW Mar)
CONDWL Waa)
CosEW Wes?
Figure 24: Universal prediction neurons in GPT2-medium-a.
34

---

Published in Transactions on Machine Learning Research (06/2024)
Prediction Neuron Logit Attribution
Prediction Neuron Activation
pre activaatioo
WWW
-0.4
-0.6
ends with "ing"
does not
ends with "ing"
does not
14.2046
15 1699
16.3122
16.3346
19.1647
19.2871
20 1818677
21.118
22.1585
22 00.
23.3844
14.4048
15 1690
36.3222
16.3346
18.1984
19.1
19 09.3
20 1867
21
22.1505
22 22.3534
22 904
18
22
0.50
025
0.00
pre activation -2
W
-0.25
-0500
-0.75
vocab token pronoun
next token neutral pronoun
-1.00
not
15.1552
16.2229
16 4099
17.797
18 3690
19.2820
20 603
21.1274
22.1732
22.73
1625299
16 4092
37,799
1922220
22.1732
22.73
20.60
18
0.50
0.00
pre activation
WWW
-0
-0.50
-1.00
vocab
-1.25
2nd person pronoun
not
2nd person pronoun
Is not
27.774
18 1532
18 1631
18 1932
18 633
20
23,2205
14.2719
77.774
18 1532
1631
18 3930
18 638
20 2583
23.2205
pre activation
WwW ox
-0.2
0
-0.4
2
vocab token
next token
-0.8
Is all caps
not
all caps
Is not
15.591
15.84
17.559
18.3336
20.13
21.2148
15.591
11221
21.2148
733440
15
0.50
0.5
pre activation
0.00
0
W.W
3
-0.75
vocab token
b
next token
open paren
open
-1.00
37.398
18.138 337
20 1561
23 1131
1325
22.811 811
2042
23.5
398
1561
2042
23.5
23
Figure Prodiction neurons for the same feature GPT2-modium-a. Left column depicts logit effect
broken down by vocabulary item per neuron and right column shows activation value broken down by true next token per neuron.
35

---

Published in Transactions on Machine Learning Research (06/2024)
(a) Neuron weight norms
(b) Neuron composition w/ unembed
(c) Final layer norm scale
10a
Neuron
10a
11.3030
neuron count
10a
vocab count
10a
mean scale
random
202
10a
101
10a
-0.2
-0.1
0.0
0.1
0.2
|W..1 1...
Cos(Wu. Wan)
fixed activation value
(d) Next token prediction recip rank
(e) Next token prediction entropy
(f) Next token prediction loss
mean reciprocal rank
0.554
4.5
3.40
0.552
mean entropy
4.0
3.35
0.550
no intervention
3.5
0.548
3.30
3.0
0.546
0.544
2.5
fixed activation value
fixed activation value
fixed activation value
Figure 26: Summary of (anti-)entropy neurons in GPT2-small-a compared to 20 random neurons from final two layers. Entropy neurons have high weight norm (a) with output weights mostly orthogonal to the unembedding matrix (b). When activated, this causes the final layer norm scale to increase dramatically (c) while leaving the relative ordering over the next token prediction mostly unchanged (d). Increased layer norm scale squeezes the logit distribution, causing a large increase in the prediction entropy (e; or decrease for anti-entropy neuron) and an increase or decrease in the loss depending on the model's baseline level of under- cer-confidence (f). Legend applies all subplots.
36

---

Published in Transactions on Machine Learning Research (06/2024)
PIBOS orig p(BOS) ablated
80
(1050 - 11000) (%)
0.0
60
(10g' -oto
-0.1
40
40
20
20
-0.2
best fit (p=-0.97)
best fit (p=-0.92)
0-
best fit (p=0.999 0.95)
-0.3
ablation result
-20
ablation result
-20
ablation result
0.0
0.5
1.0
1.5
-0.
-0.2
-0.1
0.0
0.0
0.5
1.0
1.5
neuron activation pre-ablation
P(BOS) orig PBBOS) ablated
neuron activation pre-ablation
P(BO5) orig p|BOS) ablated
10
10-
0.100
best fit (a=0.95)
1 - 1o2"1//00" (%)
best fit (am-0.94)
(10g - 102" (9)
ablation result
0-
ablation result
0-
0.075
0.050
-10
-10-
0.025
-20
-20
0.000
-30
best fit (p--0.94)
ablation result
-0.025
0.0
0.5
1.0
1.5
-0.0250.000 0.025 0.050 0.075 0.100
0.0
0.5
1.0
1.5
neuron activation pre-ablation
P(BOS) orig P(BOS) ablated
neuron activation pre-ablation
P[8OS) orig P(805) ablated
0.06
best fit (o= 0.97)
(1 | lod Doto (%)
best fit (p=-0.96)
(1 1og"D>/oo" ( (%)
best fit (p-60.96)
0.04
ablation result
ablation result
0-
ablation result
-5
0.02
-10-
-10
0.00
-15
-15
0.25
0.50
0.75
0.00
0.02
0.04
0.06
0.75
neuron activation pre-ablation
P(BOS) orig P(BOS) ablated
neuron activation pre-ablation
PIBOS) orig PPBB5) ablated
best fit (am-0.91)
(%)
( (%)
best fit (p- (.-0
0.00
ablation result
20
1.00 jo Llloaa
20
ablation result
-0.02
-0.04
0-
0
best fit (p--0.89)
ablation result
-0.06
-10
-10
0.0
0.5
10
-0.06
-0.04
-0.02
0.00
0.0
0.5
1.0
neuron activation pre-ablation
P(BOS) orig- P(BOS) ablated
neuron activation pre-ablation
P(8OS) orig PIBOS) ablated
0.03
(000 199 Moooo 1 196)
5-
(1 1M (%)
5
0-
0.02
-5
-5
0.01
-10
-10-
0.00
-15
-15
best fit (p=0.79)
best fit (p=-0.84)
best fit (p=-0.81)
-0.01
ablation result
-20
ablation result
-20-
ablation result
0.0
1.0 neuron activation pre-ablation
0.5
-0.01
0.00
0.01
0.02
0.03
0.0
0.5
1.0
P(BOS) orig P(BOS) ablated
neuron activation pre-ablation
Figure 27: Further examples of attention activation and deactivation neurons. Row 1: A15H8 with L14N411,
Row 2: A15H8 with L14N2235, Row 3: A15H8 with L14N1625, Row 4: A20H4 with L1 L199250,, Row 5:
A22H7 with L20N2114
37


Reviews:
[{"review_id": "WGR6UwEr3m", "reviewer": "Reviewer_v8MV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n 1. Addresses an important question: at what level can we \"understand\" a neural network?\n 1. Reports a number of interesting observations about the inner workings of Transformers\n 1. Tests two different models (GPT2 and Pythia), observing similar results\n\n\n\n### Weaknesses\n\n 1. Reasoning is not always easy to follow\n 1. Observations are largely descriptive; not clear how to derive insights for future improvements\n\n\n### Detailed comments\n\n\nOverall the paper is well organized and mostly well written. It reports a number of (not necessarily novel) observations about the inner workings of Transformers that researchers interested in interpretability methods would most likely find interesting and could inspire future work. I am therefore overall fairly supportive of the paper, although I believe it could be improved by more clarity in the exposition of the results and by more clearly stating if there are any implications for future neural network design that can be derived from this work.\n\n\n### 1. Clarity\n\nWhile I find the overall organization of the paper easy to follow, I got lost a few times in the individual sections. In particular, the following sections' clarity could be improved:\n\n 1. Section 4.1: While I can generally follow the reasoning and analysis, I am a bit puzzled about the results of Fig. 2b: Doesn't the fact that the difference between max-max and min-max is larger for \"universal\" neurons imply that such neurons are more likely to be missing in at least one of the models (the min-max is smaller than expected by the fluctuations across most neurons), i.e. they're not that universal?\n\n 1. Section 4.2: It is not obvious to me what cos(w_out, w_U) measures. As far as I understand W_U is used only in the last layer. Why would it be interesting to look at this direction in earlier layers? The authors could provide a better rationale for this analysis.\n\n 1. Section 4.3: I did not understand the significance of Eq. (4). The variables y and beta are unclear to me and I'm not sure what the quantity actually measures and how it exactly relates to the authors \"finding\" neurons of the types described below. Perhaps this part can be unpacked a bit more?\n\n 1. Section 5.1: Again W_U shows up in an analysis applied to all layers, but -- unless I'm confused -- it never gets applied to earlier layers. Thus, I am not sure what's the meaning of or the rationale behind this analysis. Also, I did not directly understand why the product of w_out and W_U? Isn't there also the residual path that gets added and layer norm in between? What does that product tell us? I think this could be explained better and perhaps supported by an illustration (illustrations may help also for earlier points).\n\n 1. Section 5.1: I could not follow the paragraph starting with \"When studying the activations of suppression neurons...\" at all. It states that they \"activate far more often when the next token is in fact from the set of tokes they express\". Far more often than what? And if they indeed activate more often, why are they called suppression neurons?\n\n 1. Section 5.3: It wasn't immediately clear to me why \"the vector W_O v_BOS is constant for all prompts\". Is it because the causal attention mask prevents this token from attending to any other token? \n\n\n\n### 2. What do we learn?\n\nAlthough I find the observations overall interesting, I am not really sure what we learn from this type of \"neuroscience\" experiments on deep learning models. They describe what some neurons do, but what follows from such knowledge? Can we use it to design better models? If so, how? Can we use it to understand why they fail if they fail? If so, how? I think it would be great if the authors could discuss a bit more how they picture the path forward.\n\n\n\n### 3. Invalid conclusion\n\nI don't think the following conclusion in the discussion is valid: \"... find that only about 1-5% of neurons are universal across models, constituting another piece of evidence that individual neurons are not the appropriate unit of analysis for most network behaviours.\" -- The number of universal neurons is completely arbitrary. As you state yourself, there is no objective threshold and you arbitrarily chose 0.5. Had you chosen 0.2, most of the neurons would be \"universal\" and the conclusion would have been the opposite. In fact, I do not see evidence in this paper for the claim that \"individual neurons are not the appropriate unit of analysis for most network behaviours\", so this claim should be removed or rephrased.\n\n\n\n### Typos:\n\n - P. 4, line 5 from bottom: \"in terms of [the] its\"\n - P. 11, last sentence in 5.2 \"... models [something missing] the second ...\"\n - P. 12, line 5: \"Positive scores suggest[s] activation...\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the question whether there are universal neurons in Transformer-based language models. In the authors' definition of universality, a neuron is universal if in a second model of the same architecture but trained with different initial conditions there exists another neuron that behaves largely in the same way, i.e. has highly correlated activation patterns over a large number of input sequences. The authors find a relatively small subset of such \"universal\" neurons and investigate their properties.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like the authors to improve the clarity on the points listed above, include a discussion on how we use the insights from this analysis and fix the invalid conclusion.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ypqgh4m6no", "reviewer": "Reviewer_xxZ1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper is clearly written and related work is adequately covered.\n- The primary analysis of 'universality' is straightforward and easy to understand, the results are presented in an intuitive manner. \n- A particularly compelling finding is that similar neurons emerge in similar layers across models.\n- A wide range of functional properties are identified and extensively characterized, including many neuron types that are new discoveries of this work (to my knowledge). This body of findings will be very useful for informing future work on interpretability.\n\n## Weakness\n- It is not clear why there should be any relationship between universality and monosemanticity. No justification is provided to motivate this hypothesis.\n- It is also unclear why universality should be defined at the level of individual neurons. We can imagine a scenario where a particular layer in two separate networks represents the input in a similar manner, but with representations that are rotated with respect to one another. I don't see why in particular the dimensions of such a representation should align with individual neurons, but it still seems like it would be reasonable to characterize such a scenario as involving a universal solution.\n- Many neuron types are considered, with lots of data illustrating examples of these neurons. It is not always clear to what extent these identified neurons are universal though. It would be helpful if, for each neuron type, there was some statement as to whether that neuron type appears in all five models, and in what proportion.\n- It would also be helpful to quantify the percentage of the neurons defined as universal end up being interpretable according to the various categories identified in the paper. In the end, there does not seem to be a systematic conclusion regarding the hypothesized relationship between universality and monosemanticity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents an analysis of the prevalence of 'universal neurons' (neurons with similar response properties) across different instances of GPT-2 trained from different random initializations, together with an analysis of the interpretability of these neurons. The results suggest that a small number of neurons are universal, but that many of these neurons have interpretable functions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would appreciate some further insight on the issues raised in the 'weaknesses' section above.", "broader_impact_concerns": "There are no discernible negative ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6tSnbFEou7", "reviewer": "Reviewer_wZeq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. This paper attempts to analyze neurons of GPT-2 models and reveal the properties of neurons to understand whether they are universal or not. It provides various statistical properties of weights and activations of neurons.\n\n\n\n**Weaknesses**\n1. Some observations seem weird, why these observations can be attributed to the universality of neurons. The selection of hyperparameters seems specific. Are these hyperparamters (e.g., $\\varrho > 0.5$) chosen based on experiences?\n2. In these analyses of this paper, authors have summarized many different roles of neurons (e.g., unigram neurons, alphabet neurons, and etc.). But are there any specific rules to identify these neurons and can these obsevations be generalize to other language models?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores mechanistic interpretability to understand the universality of individual neurons acrorss GPT-2 model. More specifically, authors compute the pairwise correlations of neuro activations over 100 million tokens. The observations indicate that 1-5% of neurons are universal. By analyzing these patterns in these neuron weights, this paper attributes these neurons to several universal functional roles.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I suggest authors to provide more explainations to describle the connections between the observations and the conclusions (i.e., the universality of different neurons).", "broader_impact_concerns": "I think this paper does not need any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers felt that the clarity of the paper could still be greatly improved. The AE strongly encourages the authors to try to address all of the points regarding clarity provided by the reviewers. As well, there was still a request from one reviewer to include a quantitative analysis comparing universality and monosemanticity (using an approach that doesn't require setting an arbitrary threshold, such as ROC/AUC analysis). The AE would recommend that the authors also provide this.
Decision: Accept (poster)