Neurons in Large Language Models: Dead, N-gram, Positional (2024)

Abstract: We analyze a family of large language models in such a lightweight manner that can be done on a single GPU. Specifically, we focus on the OPT family of models ranging from 125m to 66b parameters and rely only on whether an FFN neuron is activated or not. First, we find that the early part of the network is sparse and represents many discrete features. Here, many neurons (more than 70% in some layers of the 66b model) are “dead”, i.e. they never activate on a large collection of diverse data. At the same time, many of the alive neurons are reserved for discrete features and act as token and n-gram detectors. Interestingly, their corresponding FFN updates not only promote next token candidates as could be expected, but also explicitly focus on removing the information about triggering them tokens, i.e., current input. To the best of our knowledge, this is the first example of mechanisms specialized at removing (rather than adding) information from the residual stream. With scale, models become more sparse in a sense that they have more dead neurons and token detectors. Finally, some neurons are positional: them being activated or not depends largely (or solely) on position and less so (or not at all) on textual data. We find that smaller models have sets of neurons acting as position range indicators while larger models operate in a less explicit manner.

heading: 1 Introduction
text: The range of capabilities of language models expands with scale and at larger scales models become so strong and versatile that a single model can be integrated into various applications and decisionmaking processes (Brown et al., 2020; Kaplan et al., 2020; Wei et al., 2022; Ouyang et al., 2022; OpenAI, 2023; Anil et al., 2023). This increases interest and importance of understanding the internal
∗Work done as part of internship at Meta AI.
workings of these large language models (LLMs) and, specifically, their evolution with scale. Unfortunately, scaling also increases the entry threshold for interpretability researchers since dealing with large models requires a lot of computational resources. In this work, we analyze a family of OPT models up to 66b parameters and deliberately keep our analysis very lightweight so that it could be done using a single GPU.
We focus on neurons inside FFNs, i.e. individual activations in the representation between the two linear layers of the Transformer feedforward blocks (FFNs). Differently from e.g. neurons in the residual stream, FFN neurons are more likely to represent meaningful features: the elementwise nonlinearity breaks the rotational invariance of this representation and encourages features to align with the basis dimensions (Elhage et al., 2021). When such a neuron is activated, it updates the residual stream by pulling out the corresponding row of the second FFN layer; when it is not activated, it does not update the residual stream (Figure 6).1 Therefore, we can interpret functions of these FFN neurons in two ways: (i) by understanding when they are activated, and (ii) by interpreting the corresponding updates coming to the residual stream.
First, we find that in the first half of the network, many neurons are “dead”, i.e. they never activate on a large collection of diverse data. Larger models are more sparse in this sense: for example, in the 66b model more that 70% of the neurons in some layers are dead. At the same time, many of the alive neurons in this early network part are reserved for discrete features and act as indicator functions for tokens and n-grams: they activate if and only if the input is a certain token or an n-gram. The role of the updates coming from these token detectors to the residual stream is also surprising: at the same
1Since OPT models have the ReLU activation function, the notion of “activated” or “not activated” is trivial and means non-zero vs zero.
1288
time as they promote next token candidates (which is to be expected according to Geva et al. (2021, 2022)), they are explicitly targeted at removing information about current input, i.e. their triggers. This means that in the bottom-up processing where a representation of the current input token gets gradually transformed into a representation for the next token, current token identity is removed by the model explicitly (rather than ends up implicitly “buried” as a result of additive updates useful for the next token). To the best of our knowledge, this is the first example of mechanisms specialized at removing (rather than adding) information from the residual stream.
Finally, we find that some neurons are responsible for encoding positional information regardless of textual patterns. Similarly to token and n-gram detectors, many of these neurons act as indicator functions of position ranges, i.e. activate for positions within certain ranges and do not activate otherwise. Interestingly, these neurons often collaborate. For example, the second layer of the 125m model has 10 positional neurons whose indicated positional ranges are in agreement: together, they efficiently cover all possible positions and no neuron is redundant. In a broader picture, positional neurons question the key-value memory view of the FFN layers stating that “each key correlates with textual patterns in the training data and each value induces a distribution over the output vocabulary” (Geva et al., 2021, 2022). Neurons that rely on position regardless of textual pattern indicate that FFN layers can be used by the model in ways that do not fit the key-value memory view. Overall, we argue that the roles played by these layers are still poorly understood.
To sum up, we find neurons that:
• are “dead”, i.e. never activate on a large diverse collection of data;
• act as token- and n-gram detectors that, in addition to promoting next token candidates, explicitly remove current token information;
• encode position regardless of textual content which indicates that the role of FFN layers extends beyond the key-value memory view.
Larger models have more dead neurons and token detectors and are less focused on absolute position.

heading: 2 Data and Setting
text: Models. We use OPT (Zhang et al., 2022), a suite of decoder-only pre-trained transformers that are publicly available. We use model sizes ranging from 125M to 66B parameters and take model weights from the HuggingFace model hub.2
Data. We use data from diverse sources containing development splits of the datasets used in OPT training as well as several additional datasets. Overall, we used (i) subsets of the validation and test part of the Pile (Gao et al., 2020) including Wikipedia, DM Mathematics, HackerNews, (ii) Reddit (Baumgartner et al., 2020; Roller et al., 2021), (iii) code data from Codeparrot3.
For the experiments in Section 3 when talking about dead neurons, we use several times more data. Specifically, we add more data from Wikipedia, DM Mathematics and Codeparrot, as well as new domains from the Pile4: EuroParl, FreeLaw, PubMed abstracts, Stackexchange.
Overall, the data used in Section 3 has over 20M tokens, in the rest of the paper – over 5M tokens.
Single-GPU processing. We use only sets of neuron values for some data, i.e. we run only forward passes of the full model or its several first layers. Since large models do not fit in a single GPU, we load one layer at a time keeping the rest of the layers on CPU. This allows us to record neuron activations for large models: all the main experiments in this paper were done on a single GPU.

heading: 3 Dead Neurons
text: Let us start from simple statistics such as neuron activation frequency (Figure 1).
Many neurons are “dead”. First, we find that many neurons never activate on our diverse data, i.e. they can be seen as “dead”. Figure 1a shows that the proportion of dead neurons is very substantial: e.g., for the 66b model, the proportion of dead neurons in some layers is above 70%. We also see that larger models are more sparse because (i) they have more dead neurons and (ii) the ones that are alive activate less frequently (Figure 1b).
2https://huggingface.co/models 3https://huggingface.co/datasets/codeparrot/ codeparrot-clean 4https://huggingface.co/datasets/EleutherAI/ pile
Only first half of the model is sparse. Next, we notice that this kind of sparsity is specific only to early layers. This leads to a clear distinction between the first and the second halves of the network: while the first half contains a solid proportion of dead neurons, the second half is fully “alive”. Additionally, layers with most dead neurons are the ones where alive neurons activate most rarely.
Packing concepts into neurons. This difference in sparsity across layers might be explained by “concept-to-neuron” ratio being much smaller in the early layers than in the higher layers. Intuitively, the model has to represent sets of encoded in a layer concepts by “spreading” them across available neurons. In the early layers, encoded concepts are largely shallow and are likely to be discrete (e.g., lexical) while at the higher layers, networks learn high-level semantics and reasoning (Peters et al., 2018; Liu et al., 2019; Jawahar et al., 2019; Tenney et al., 2019; Geva et al., 2021). Since the number of possible shallow patterns is not large and, potentially, enumerable, in the early layers the model can (and, as we will see later, does) assign dedicated neurons to some features. The more neurons are available to the model, the easier it is to do so – this agrees with the results in Figure 1. Differently, the space of fine-grained semantic concepts is too large compared to the number of available neurons which makes it hard to reserve many dedicated neuron-concept pairs.5
Are dead neurons completely dead? Note that the results in Figure 1a can mean one of the two things: (i) these neurons can never be activated (i.e. they are “completely dead”) or (ii) they correspond to patterns so rare that we never encountered them in our large diverse collection of data. While the
5There can, however, be a few specialized neurons in the higher layers. For example, BERT has neurons responsible for relational facts (Dai et al., 2022).
latter is possible, note that this does not change the above discussion about sparsity and types of encoded concepts. On the contrary: it further supports the hypothesis that models assign dedicated neurons to specific concepts.

heading: 4 N-gram-Detecting Neurons
text: Now, let us look more closely into the patterns encoded in the lower model halves and try to understand the nature of the observed above sparsity. Specifically, we analyze how neuron activations depend on an input n-gram. For each input text with tokens x1, x2, ..., xS , we record neuron activations at each position and if a neuron is activated (i.e., non-zero) at position k, we say that the n-gram (xk−n+1, . . . , xk) triggered this neuron.
In Sections 4.1-4.4 we talk about unigrams (i.e., tokens) and come to larger n-grams in Section 4.5.

heading: 4.1 Number of N-grams Triggering a Neuron
text: First, let us see how many n-grams are able to trigger each neuron. For each neuron, we evaluate the number of n-grams that cover at least 95% of its activations. For the bottom model half, Figure 2 shows how neurons in each layer are categorized by the number of covering them n-grams (we show unigrams here and larger n-grams in Appendix A).
As anticipated, neurons in larger models are covered by fewer n-grams. Also, the largest models have a substantial proportion of neurons covered by as few as 1 to 5 tokens. This agrees with our
hypothesis: the model spreads discreet shallow patterns across specifically dedicated neurons.6

heading: 4.2 Token-Detecting Neurons
text: Presence of neurons that can be triggered by only a few (e.g., 1-5) tokens point to the possibility that some neurons act as token detectors, i.e. activate if and only if the input is one of the corresponding tokens, regardless of the previous context. To find such neurons, we (1) pick neurons that can be triggered by only 1-5 tokens, (2) gather tokens that are covered by this neuron (if the neuron activates at least 95% of the time the token is present), (3) if altogether, these covered tokens are responsible for at least 95% of neuron activations.7
Figure 3a shows that there are indeed a lot of token-detecting neurons. As expected, larger models have more of them and the 66b model has overall 5351 token detectors. Note that each token detector is responsible for a group of tokens that, in most of the cases, are variants of the same word (e.g., with differences only in capitalization, the space-before-word special symbol, morphological form, etc.). Figure 5 (top) shows examples of token groups detected by these neurons.
Interestingly, the behavior of the largest models (starting from 13b of parameters) differs from that of the rest. While for smaller models the number of token detectors increases then goes down, larger models operate in three monotonic stages and start having many token-detecting neurons from the very first layer (Figures 3). This already shows qualitative differences between the models: with more capacity, larger models perform more complicated reasoning with more distinct stages.
6Note that the 350m model does not follow the same pattern as all the rest: we will discuss this model in Section 6.
7We exclude the begin-of-sentence token from these computations because for many neurons, this token is responsible for the majority of the activations.

heading: 4.3 Ensemble-Like Behaviour of the Layers
text: Now, let us look at “detected” tokens, i.e. tokens that have a specialized detecting them neuron. Figure 3b shows the number of detected tokens in each layer and cumulatively over layers. We see that, e.g., the 66b model focuses on no more than 1.5k tokens in each layer but covers over 10k tokens overall. Thus, across layers, token-detecting neurons are responsible for largely differing tokens. Indeed, Figure 4 shows that in each following layer, detected tokens mostly differ from all the tokens covered below. This points to an ensemble-like (as opposed to sequential) behavior of the layers: layers collaborate in a divide-and-conquer-style manner which allows larger models to cover many tokens and use their capacity more effectively.
Originally, such an ensemble-like behavior of deep residual networks was found in computer vision (Veit et al., 2016). For transformers, previous evidence includes simple experiments showing that dropping or reordering layers does not hurt performance much (Fan et al., 2020; Zhao et al., 2021).

heading: 4.4 Token Detectors Suppress Their Triggers
text: Finally, let us explain the role of token-detecting neurons by interpreting how they update the residual stream. Throughout the layers, token representation in the residual stream gets transformed from the token embedding for the current input token8 to the representation that encodes the next token. This happens via additive updates coming from attention and FFN blocks in each layer. Whenever an FFN neuron is activated, the corresponding row of the second FFN layer (multiplied by this neuron’s value) is added to the residual stream (Figure 6). By projecting this FFN row onto vocabulary, we can interpret this update (and, thus, the role of this
8For OPT, along with an absolute positional embedding.
neuron) in terms of its influence on the output distribution encoded in the residual stream.
Current token suppression: implicit or explicit? Previously, this influence was understood only in terms of the top projections, i.e. tokens that are promoted (Geva et al., 2021, 2022). This reflects an existing view supporting implicit rather than explicit loss of the current token identity over the course of layers. Namely, the view that the current identity gets “buried” as a result of updates useful for the next token as opposed to being removed by the model explicitly. In contrast, we look not only at the top projections but also at the bottom: if these projections are negative, the corresponding tokens are suppressed by the model (Figure 6).
Explicit token suppression in the model. We find that often token-detecting neurons deliberately suppress the tokens they detect. Figure 5 shows examples of these neurons along with the top promoted and suppressed concepts. While the top promoted concepts are in line with previous work (they are potential next token candidates as in Geva et al. (2021, 2022)), the top suppressed concepts are rather unexpected: they are exactly the tokens triggering this neuron. This means that vector updates coming from neurons play two different roles at the same time: (i) point in the direction of the next token candidates and (ii) point away from the tokens triggering the neuron. In total, for over 80% of token-detecting neurons their updates point in the negative direction from the triggering them tokens (although, the triggering tokens are not always at the very top suppressed concepts as in Figure 6).
To sum up, we show that models can have mechanisms targeted at removing information; future work can explore this further.

heading: 4.5 Beyond Unigrams
text: In Appendix A, we show results for bigrams and trigrams that mirror our observations for unigrams: (i) larger models have more specialized neurons, (ii) in each layer, models cover mostly new n-grams. Interestingly, for larger n-grams we see a more drastic gap between larger and smaller models.

heading: 5 Positional Neurons
text: When analyzing dead neurons (Section 3), we also noticed some neurons that, consistently across diverse data, never activate except for a few first token positions. This motivates us to look further into
how position is encoded in the model and, specifically, whether some neurons are responsible for encoding positional information.

heading: 5.1 Identifying Positional Neurons
text: Intuitively, we want to find neurons whose activation patterns are defined by or, at least, strongly depend on token position. Formally, we identify neurons whose activations have high mutual information with position. For each neuron, we evaluate mutual information between two random variables:
• act – neuron is activated or not ({Y,N}),
• pos – token position ({1, 2, . . . , T}).
Formal setting. We gather neuron activations for full-length data (i.e., T = 2048 tokens) for Wikipedia, DM Mathematics and Codeparrot. Let fr (pos) n be activation frequency of neuron n at position pos and frn be the total activation frequency of this neuron. Then the desired mutual information is as follows:9
I(act, pos) = 1
T ·
T∑
pos=1
[ fr(pos)n · log fr (pos) n
frn +
(1− fr(pos)n ) · log 1− fr(pos)n 1− frn
] .
Choosing the neurons. We pick neurons with I(act, pos) > 0.05, i.e. high mutual information with position – this gives neurons whose activation frequency depends on position rather than content. Indeed, if e.g. a neuron is always activated within certain position range regardless of data domain, we can treat this neuron as responsible for position; at least, to a certain extent.
9For more details, see appendix B.1.

heading: 5.2 Types of Positional Neurons
text: After selecting positional neurons, we categorize them according to their activation pattern, i.e. activation frequency depending on position (Figure 7).
Oscillatory. These neurons are shown in purple in Figure 7. When such a pattern is strong (top row), the activation pattern is an indicator function of position ranges: a neuron is activated if and only if the position falls into a certain set. Since the activation pattern does not change across data domains, it is defined solely by position and not the presence of some lexical or semantic information.
Both types of activation extremes. These are the neurons whose activation pattern is not oscillatory but still has intervals where activation frequency reaches both “activation extremes”: 0 (never activated) and 1 (always activated). Most frequently, such a neuron is activated only for positions less than or greater than some value and not activated otherwise. Similarly to oscillatory neurons, when such a pattern is strong (Figure 7, top row), it is also (almost) an indicator function.
Only one type of activation extremes. Differently from the previous two types, activation patterns for these neurons can reach only one of the extreme values 0 or 1 (Figure 7, green). While this means that they never behave as indicator functions, there are position ranges where a neuron being activated or not depends solely on token position.
Other. Finally, these are the neurons whose activation patterns strongly depend on position but do not have intervals where activation frequency stays 0 or 1 (Figure 7, yellow). Typically, these activation patterns have lower mutual information with position than the previous three types.
Strong vs weak pattern. We distinguish “strong” and “weak” versions of each type and show this with color intensity (Figure 7, top vs bottom rows). For the first three neuron types, the difference between strong and weak patterns lies in whether on the corresponding position ranges activation frequency equals 0 (or 1) or close, but not equals, 0 (or 1). For the last type, this difference lies in how well we can predict activation frequency at some position knowing this value at the neighboring positions (informally, “thin” vs “thick” graph).

heading: 5.3 Positional Neurons Across the Models
text: For each of the models, Figure 8 illustrates the positional neurons across layers.
Small models encode position more explicitly. First, we notice that smaller models rely substantially on oscillatory neurons: this is the most frequent type of positional neurons for models smaller than 6.7b of parameters. In combination with many “red” neurons acting as indicator functions for wider position ranges, the model is able to derive token’s absolute position rather accurately. In contrast, larger models do not have oscillatory neurons and rely on more generic patterns shown with red- and green-colored circles. We can also see that from 13b to 66b, the model loses two-sided red neurons and uses the one-sided green ones more. This hints at one of the qualitative differences between smaller and larger models: while the former encode absolute position more accurately, the latter ones are likely to rely on something more meaningful than absolute position. This complements re-
cent work showing that absolute position encoding is harmful for length generalization in reasoning tasks (Kazemnejad et al., 2023). Differently from their experiments with same model size but various positional encodings, we track changes with scale. We see that, despite all models being trained with absolute positional encodings, stronger models tend to abstract away from absolute position.
Positional neurons collaborate. Interestingly, positional neurons seem to collaborate to cover the full set of positions. For example, let us look more closely at the 10 strongly oscillatory neurons in the second layer of the 125m model (Figure 8, dark purple circles). Since they act as indicator functions, we can plot position ranges they indicate. Figure 9 shows that (i) indicated position ranges are similar up to a shift, (ii) the shifts are organized “perfectly”: together, these ten neurons cover all positions and none of these neurons is redundant.
The two stages within the model. Finally, Figure 8 reveals two stages of up-and-downs of positional information within the model: roughly, the first third of the model and the rest. Interestingly,
preferences in positional patterns also change between the stages: e.g., preference for “red” neurons changes to oscillatory purple patterns for the 1.3b and 2.7b models, and “red” patterns become less important in the upper stage for the 13b and 30b models. Note that the first third of the model corresponds to the sparse stage with the dead neurons and n-gram detectors (Sections 3, 4). Therefore, we can hypothesize that in these two stages, positional information is first used locally to detect shallow patterns and then more globally to use longer contexts and help encode semantic information.
Previously, the distinct bottom-up stages inside language models were observed in Voita et al. (2019a). The authors explained how the way representations gain and lose information across the layers is defined by the training objective and why, among other things, positional information should (and does) get lost. This agrees with our results: while there are many positional patterns in the second stage, they are weaker than in the first stage.

heading: 5.4 Positional Neurons are Learned Even Without Positional Encoding
text: Recently, it turned out that even without positional encoding, autoregressive LMs still learn positional information (Haviv et al., 2022). We hypothesize that these “NoPos” models encode position via positional neurons. To confirm this, we train 125m models with and without positional encodings and compare the types of their positional neurons.
Setup. We trained 125m models with the OPT setup but smaller training dataset: OpenWebText corpus (Gokaslan and Cohen, 2019), an open clone of the GPT-2 training data (Radford et al., 2019). This dataset contains 3B tokens (vs 180B for OPT).
Positional neurons without positional encoding. We see that, indeed, the model without positional encoding also has many strong positional patterns (Figure 10). Note, however, that the NoPos model does not have oscillatory neurons which, in combination with other positional neurons, allow encoding absolute position rather accurately. This means that the NoPos model relies on more generic
patterns, e.g. “red” neurons encoding whether a position is greater/less than some value.
Oscillatory neurons require longer training. Finally, we found that oscillatory patterns appear only with long training. Figure 11 shows positional patterns learned by the baseline 125m model trained for 50k, 150k and 300k training batches. We see that all models have very strong positional patterns, but only the last of them has oscillatory neurons. Apparently, learning absolute position requires longer training time.

heading: 5.5 Doubting FFNs as Key-Value Memories
text: Currently, it is widely believed that feed-forward layers in transformer-based language models operate as key-value memories. Specifically, “each key correlates with textual patterns in the training examples, and each value induces a distribution over the output vocabulary” (Geva et al. (2021, 2022); Dai et al. (2022); Meng et al. (2022), etc.). While in Section 4.4 we confirmed that this is true for some of the neurons, results in this section reveal that FFN layers can be used by the model in ways that do not fit the key-value memory view. Namely, activations of strong positional neurons are defined by position regardless of textual content, and the corresponding values do not seem to encode meaningful distributions over vocabulary. This means that the role of these neurons is different from matching textual patterns to sets of the next token candidates. Therefore, the roles played by Transformer feedforward layers are still poorly understood.

heading: 6 The 350m Model: The Odd One Out
text: As we already mentioned above, the 350m model does not follow the same pattern as the rest of the models. Specifically, it does not have dead neurons (Section 3) and its neuron activations do not seem to be sparse with respect to triggering them n-grams as we saw for all the other models in Figure 2.10
Modeling bits affect interpretability. This becomes less surprizing when noticing that the 350m model is implemented differently from all the rest: it applies LayerNorm after attention and feedforward blocks, while all the other models – before.11 Apparently, such seemingly minor implementation details can affect interpretability of model components rather significantly. Indeed, previous work also tried choosing certain modeling aspects to encourage interpretability. Examples of such work include choosing an activation function to increase the number of interpretable neurons (Elhage et al., 2022), large body of work on sparse softmax variants to make output distributions or attention more interpretable (Martins and Astudillo (2016); Niculae and Blondel (2017); Peters et al. (2019); Correia et al. (2019); Martins et al. (2020), among others), or more extreme approaches with explicit modular structure that is aimed to be interpretable by construction (Andreas et al. (2016); Hu et al. (2018); Kirsch et al. (2018); Khot et al. (2021), to name a few). Intuitively, choosing ReLU activation function as done in the OPT models can be seen as having the same motivation as developing sparse softmax variants: exact zeros in the model are inherently interpretable.

heading: 7 Additional Related Work
text: Historically, neurons have been a basic unit of analysis. Early works started from convolutional networks first for images (Krizhevsky et al., 2012) and later for text classification (Jacovi et al., 2018). Similar to our work, Jacovi et al. (2018) also find n-gram detectors; although, for small convolutional text classifiers this is an almost trivial observation compared to Transformer-based LLMs. For recurrent networks, interpretable neurons include simple patterns such as line lengths, brackets
10There are, however, positional neurons; see Figure 16 in Appendix B.2).
11https://github.com/huggingface/transformers/ blob/main/src/transformers/models/opt/modeling_ opt.py
and quotes (Karpathy et al., 2015), sentiment neuron (Radford et al., 2017), and others (Bau et al., 2019). For BERT, Dai et al. (2022) find that some FFN neurons store factual knowledge. Larger units of analysis include attention heads (Voita et al. (2018, 2019b); Clark et al. (2019); Kovaleva et al. (2019); Baan et al. (2019); Correia et al. (2019), etc), feed-forward layers (Geva et al., 2021, 2022) and circuits responsible for certain tasks (Wang et al., 2022; Geva et al., 2023; Hanna et al., 2023).

heading: 8 Implications and Conclusions
text: Overall, neurons in LLMs can (i) be dead (neveractivating), (ii) act as token- and n-gram detectors, (iii) encode position regardless of textual content. Note that differently from most of the previous influential mechanistic interpretability work, we experiment with an entire family of models (instead of a single model as in Geva et al. (2021); Wang et al. (2022) among others) and consider very large models up to 66b. We also provide a way to analyze large models with very limited resources – we believe this is of high value to academic community. Finally, our main findings are not only about the presence of certain neurons in the OPT models but also about high-level conclusions regarding current beliefs in the community. Specifically, (1) information can be explicitly removed (rather than added) from the residual stream, (2) positional neurons question the key-value memory view of FFNs, (3) we explain how LMs trained without positional information still encode position, (4) we show that minor architecture changes can significantly influence interpretability, among others. On top of that, our analysis can easily be extended to other models: e.g., for other models later work validated our findings regarding positional neurons and suppressed concepts (Gurnee et al., 2024).

heading: Acknowledgements
text: The authors thank Nicola Cancedda, Yihong Chen, Igor Tufanov and FAIR London team for fruitful discussions and helpful feedback.
Limitations
The experiments in this paper are restricted to the OPT model family, which incorporate the ReLU activation function. As a result, the results may not be generalizable to alternative architectures and activation functions. The analysis itself, however, can be extended to other models in a way we explained above.
In our study, we define the term “neurons” to refer to the activations within the intermediate representation inside the feed-forward layers. While we provide rationale for focusing on these particular representations, it is worth noting that we don’t explore other representations within the model.
Additionally, it is important to emphasize that the largest model used in this work is a 66B parameters model. Nevertheless, current Large Language Models exceed that parameter count, and may potentially exhibit different properties.

heading: A N-gram-Detecting Neurons
text: A.1 Number of N-grams Triggering a Neuron
Figure 12 shows how neurons in each layer are categorized by the number of covering them bigrams, Figure 13 – trigrams. As expected, neurons in larger models are covered by less n-grams.
A.2 Trigram-Detecting Neurons
Similarly to token-detecting neurons in Section 4.2, we also find neurons that are specialized on 3- grams. Specifically, we (1) pick neurons that are covered by only 1-50 trigrams, (2) gather trigrams that are covered by this neuron (if the neuron activated at least 95% of the time the trigram is present), (3) if altogether, these covered trigrams are responsible for at least 95% of neuron activa-
tions. Figure 14 shows the results. Overall, the results further support our main observations: larger models have more neurons responsible for n-grams. Interestingly, when looking at trigrams rather than tokens, at 30b of parameters we see a drastic jump in the number of covered n-grams. This indicates that one of the qualitative differences between larger and smaller models lies in the expansion of the families of features they are able to represent.
A.3 Ensemble-Like Layer Behavior
Figure 15 shows the number of covered trigrams in each layer. We see that in each layer, models cover largely new trigrams.

heading: B Positional Neurons
text: B.1 Mutual Information
For each neuron, we evaluate mutual information between two random variables:
• act – neuron is activated or not ({Y,N}),
• pos – token position ({1, 2, . . . , T}).
Formal setting. We gather neuron activations for full-length data (i.e., T = 2048 tokens) for Wikipedia, DM Mathematics and Codeparrot. Let
fr (pos) n be activation frequency of neuron n at position pos and frn be the total activation frequency of this neuron.
Then the desired mutual information is as follows:
I(act, pos) =
= ∑
act
T∑
pos=1
p(pos)p(act|pos) · log p(act|pos) p(act) =
Since we only feed full-length texts, all positions appear with the same frequency: p(pos) = 1/T .
= 1
T ·
∑
act∈{Y,N}
T∑
pos=1
p(act|pos)·log p(act|pos) p(act) =
= 1
T ·
T∑
pos=1
p(act = Y |pos)·log p(act = Y |pos) p(act = Y ) +
1 T · T∑
pos=1
(1−p(act = Y |pos))·log 1−p(act=Y |pos) 1− p(act = Y ) =
= 1
T ·
T∑
pos=1
[ fr(pos)n · log fr (pos) n
frn +
(1− fr(pos)n ) · log 1− fr(pos)n 1− frn
] .
B.2 Positional Neurons for the 350m Model The results are shown in Figure 16.

heading: Future Work
text: To sum up, we show that models can have mechanisms targeted at removing information; future work can explore this further