Confidence Regulation Neurons in Language Models
Authors: Alessandro Stolfo
Ben Peng Wu
Wes Gurnee
Yonatan Belinkov
Xingyi Song
Mrinmaya Sachan
Neel Nanda
Venue: NeurIPS 2024 poster (2024)
Keywords: LLMs
Interpretability
Mechanistic Interpretability
Primary area: interpretability_and_explainability
TL;DR: We study how LLMs regulate uncertainty using entropy neurons, which modulate entropy via an unembedding null space, and token frequency neurons, which adjust logits based on token frequency.

Abstract: Despite their widespread use, the mechanisms by which large language models (LLMs) represent and regulate uncertainty in next-token predictions remain largely unexplored. This study investigates two critical components believed to influence this uncertainty: the recently discovered entropy neurons and a new set of components that we term token frequency neurons. Entropy neurons are characterized by an unusually high weight norm and influence the final layer normalization (LayerNorm) scale to effectively scale down the logits. Our work shows that entropy neurons operate by writing onto an \textit{unembedding null space}, allowing them to impact the residual stream norm with minimal direct effect on the logits themselves. We observe the presence of entropy neurons across a range of models, up to 7 billion parameters. On the other hand, token frequency neurons, which we discover and describe here for the first time, boost or suppress each token’s logit proportionally to its log frequency, thereby shifting the output distribution towards or away from the unigram distribution. Finally, we present a detailed case study where entropy neurons actively manage confidence: the setting of induction, i.e. detecting and continuing repeated subsequences.

Confidence Regulation Neurons in Language Models
Alessandro Stolfo'
Ben Wu" University of Sheffield
Wes Gurnee MIT
ETH Züricch
Yonatan Belinkoy Technion
Xingyi Song University of Sheffield
Mrinmaya Sachan ETH Zürich
Neel Nanda
Abstract
Despite their widespread use, the mechanisms by which large language models (LLMs) represent and regulate uncertainty next-token predictions remain largely unexplored. This study investigates two critical components believed to influence this uncertainty: the recently discovered entropy neurons and a new set of compo- characterized
nents that we term token frequency neurons. Entropy neurons normalization by an unusually high weight norm and influence the final layer (LayerNorm) scale effectively scale down the logits. Our work shows that entropy neurons operate by writing onto an unembedding null space, allowing them to impact the residual stream norm with minimal direct effect on the logits themselves. We observe the presence of entropy neurons across range of models, up to 7 billion parameters. discover and describe here for the first time, boost suppress each token's 's logit proportionally to its log frequency, thereby shifting the output distribution towards away from the unigram distribution. Finally, we present detailed case study where entropy neurons actively manage confidence in the setting induction, i.c. detecting and continuing repeated subsequences.
the other hand, token frequency neurons, which we
Introduction
As large language models (LLMs) increasingly permeate high-stakes applications, the lack of trans- parency in their decision-making processes poses significant vulnerabilities and risks (5). Understand- ing the basis of these models' decisions, especially how they regulate confidence in their predictions, crucial not only for advancing model development but also for ensuring their safe deployment tasks, token-level probabilities of model prediction generally match the probability of the model being
[3, LLMs have been empirically shown to be fairly well calibrated: on question-answertee
correct 40 58) This raises the question of whether L.MSs possess mechanisms for general-purpon calibration to mitigate the risks associated with overconfident predictions.
Significant research has been conducted on estimating model uncertainty in neural networks 19 21 and specifically in LLMs 122]
While many studies focus quantifying and calibrating model inter alia), there is little research into the internal mechanisms LLMs might use to calibrate their predictions.
confidence [49 40 47
45 65
In this work, we explore two types of components in Transformer- based language models that we calibration function: the recently identified entropy neurons and components that we term token frequency neurons. Entropy neurons have been brought to light by recent work 42 28 and characterized by their high weight norm and their low compo- sition with the unembedding matrix despite being in the final layer. The low composition with the unembedding matrix would suggest that they play minor role in the next-token prediction.
believe
class of
Equal contribution. Correspondence stolfoa@ethz. ch, bpwut@sheffield..ee uk.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, their high weight norm, despite LLMs being trained with weight decay 48]. indicates that these neurons must be important for performance. This leads us to infer that these neurons may play calibration role, as hypothesized by Gurnee et al. (88. We show that entropy neurons work by writing to an effective null space of the unembedding matrix and leverage the layer normalization 4 that is applied to the residual stream. This way, these components effectively modulate the entropy of the model's output distribution in a manner that only minimally affects the actual model prediction (Figure T. We observe the presence of entropy neurons in GPT-2 [62]. Pythia [6], Phi-2 (38), Gemma 2B [68] and LLaMA2 7B [70], demonstrating for the first time their role across different model families and scales [
Final-layer Neurons in LLaMA2 7B
DE
0.8
0.03
|aEntropy|
0.6
0.4
0.01
0
0
0.5
1
1.5
Prediction Change on Ablation (%)
Entropy and Prediction. We mean ablate final-layer neurons across 4000 tokens and measure the variation in the entropy of the model's output Pmodel against average change of model's predic- tion (argmax,/'model) set of neurons whose effect depends on LayerNorm (red points; metric described in $3.2), and which affect the model's ' con- fidence (quantified as entropy of Pmodde) with minimal impact the prediction.
Figure I:
Motivated by the importance the token frequency distribution (i.e., the unigram distribution) in language modeling [1. we identify a novel class of neurons that boost or suppress each token's 's logit proportionally to its frequency. We provide evidence showing that these neurons, which term token frequency neurons, modulate the distance between the model's 's output dis- tribution and the unigram distribution, which the model defaults to in settings of high uncertainty.
We identify
As case study, analyze the activation of these neurons in scenarios involving induction (i.e., the repetition of subsequences the input; 57)). Our results show that entropy neurons increase the output distribution's s entropy, thus decreasing the model's confidence in its predictions for repeated sequences. This represents a hedging mechanism, aimed at mitigating loss spikes on confidently wrong predictions.
Our contributions are as follows: (1) We show that models have dedicated circuitry to calibrate their confidence, with dedicated neurons in the final layer: the previously discovered entropy neurons and novel token frequency neurons. (2) We explore the mechanism by which each neuron family affects confidence ( and $ Notably, find that the model learns low-rank effective null space for the unembedding that entropy neurons write to, and the final LayerNorm scale is used to calibrate the output logits. This striking phenomenon would have been missed by the assumptions made in standard methods such as direct logit attribution [T6] 24 46). (3) We study how these neurons are used in practice regulate confidence ($5). As expected, neurons that lower confidence worsen performance when models are correct and improve it when models are incorrect. We close with a mechanistic case study of how induction heads (57) use entropy neurons to control confidence when
detecting and continuing repeated text ( ($6)
2 Background
In this section, we provide an overview of the Transformer architecture 71.. focusing on the components relevant our analysis. Given vocabulary V, we denote autoregressive language model as M : X' + y, where y is the space of probability distributions over V. .M takes as input a token sequence x (Fi,...,TT, ...
E X C ymm and outputs a probability distribution Pmodel V + [0, 1] to predict the next token in the sequence. In this work, we focus on decoder-only Transformer-basee models, which represent the backbone of the most capable current AI systems [58 26). The Transformer architecture consists of two core components: the multi-head self-attention and multi-layer perceptron (MLP). These two components read information in from and write out to the residual stream (i.e., the per-token hidden state consisting of the sum of all previous component outputs) [16].
In the main paper, we report detailed results for GPT-2 Small and LLaMA2 7B. Results for other models are provided in the Appendix. Our code and data are available htt.t//ss/ttttoocooooooooottttte

---

MLP. Central to this study is the structure of the MLP layers of the transformer" Given a normalized residual stream hidden state X E Rudddd the output of an MLP layer is
MLP(x) Wont (Win + (in)) + Bout
(I)
where W Win E R Xdmodel are learned weight matrices, B:,. and Bont are learned biases. The function a denotes an element-wise nonlinear activation function, typically GeLU [32]. In this paper, the term neuron refers an entry in the MLP hidden state. We denote specific neurons in the
model using the format <layer> <index>. We use to refer to the activation values of the neurons
(i.e.. the output of the activation function σ). Furthermore, we use w (8) output weights of neuron i (i.c., the i-th column of Wout).
out E Rdmodel to indicate the
LayerNorm. Layer normalization (LayerNorm) is a commonly employed technique to improve the stability of the training process in deep neural networks [4] Given hidden state R goodel LayerNorm applies the transformation
X m(x)
LN(x)
VVar(x) + 07+
(2)
In words, LayerNorm centers a hidden state by subtracting its mean (m(x) Jmodel Σ3 a
x)).
re-scales it by its e-adjusted standard deviation (v (vVar(x) + where RR, and applies an element- wise affine transformation determined by the learned parameters y, B E Rdmoool
Unembedding, After the final transformer block, the final state of the residual stream is passed through a final LayerNorm and then projected onto the vocabulary space through an unembedding matrix Wu TR. V[Xdmadel, The logits obtained by mapping the residual stream final state onto the vocabulary space are then passed through softmax function to convert them into probability distribution moddel E Y. Finally, the loss is computed using a function L y x V + E, (typically cross entropy for autoregressive language models). Following previous work [66, we employ standard weight pre-pree pre-procsssing techniques [53]. In particular, we "fold" LayerNorm trainable parameters into the W in and Wu matrices. We report the details about weight pre-processing in Appendix C
3 Entropy neurons
Prior research examining individual neurons in GPT-2 language models [62] identified a category of neurons notable for their high norm and limited interaction with the unembedding matrix. Katz and Belinkov 42 speculate that these neurons may act as regularizers, contributing a constant bias to the residual stream. Gurnee et al. 28 independently rediscovered these neurons, termed them entropy neurons, and suggest that they play role in regulating the entropy of the model's 's output distribution. They show that interventions increasing the activation value of these neurons affect the entropy of the output distribution to a larger extent compared to other neurons. Despite these insights, the natural activation effects of these neurons on standard language inputs have not been examined, function remains unclear. In this section, explore
and how they carry out this mntropy-modulane the mechanism of action of these neurons within the model.
3.1 Identifying entropy neurons
Our initial step is to identify which neurons are entropy neurons. We do this by searching for neurons with a high weight norm and a minimal impact on the logits. To detect minimal effect on the logits, we follow the heuristic used by Gurnee al. (88 analyzing the variance in the effect of neurons on the logits. In particular, we project the last- neuron weights in GPT-2 Small onto the vocabulary space. This projection, sometimes referred to logit attribution, represents an approximation of the neuron's 's effect on the final prediction logits [55] 15]. Then, we compute the variance of the normalized projection. That is, given neuron with output weights Wout, we compute
Wu [
Logit
Var
Ildim=ill Wout
'In particular, our study focuses on the final-layer MLPs due to their direct path to the final LayerNorm and unembedding, which excludes possible interactions with other transformer layers.
3

---

(a) Norm vs. LogitVar
(b) Machanism of Action
(c) Total & Direct Effects
0.04
Direct Effect
[bLoss]
Model Output
Lagitvertw.
0.00
Neuron
0.001
200g
2910
1611
2/23
506
Total Effect
Layer Norm
41.001
M.2233
21.2ng
Mi3910
12,500
Ranston
5
10
15
Neuron Name
Neuron Type:
Normal
Entropy
Effect Type
Total
Direct
(d) Proj. of Wiee onto W. Sing. Vectors
(e) LN-mediated Effect & W., Null Space
(f) LN-mediated Effect
LLaMAZ 7B
0.5
0.5
[4H|
Value
0.5
DE
0.6
2910
2013
1611
1 -EDa/TTE
0.6
0.03
750
760
0.4
22375
0.02
200
600
600
0.2
Singular Vector Index
11.384
11.2375
13.1611 Wexr
11,2910 West
D1.
-0.2
0.5
5ing. Values
Frection of Norm In Null Spece (p)
Fraction of Norm in Null Space (p)
Figure 2: Identifying and Analyzing Entropy Neurons. (a) Neurons in GPT-2 Small displayed by their weight norm and variance in logit attribution. Entropy neurons (red) have high norm and low logit variance. (b) Causal graph showing the total effect and direct effect (bypassing LayerNorm) of a neuron on the model's s output. (c) Comparison of total and direct effects on model loss for entropy neurons and randomly selected neurons. (d) Singular values and cosine similarity between neuron output weights and singular vectors of Wu. (e) Entropy neurons (red) show significant LayerNorm-men -erenm--mdddited effects and high projection onto the null space (p). (f) Relationship between and the SayerNorm-median effect in LLaMA2 7B. is computed with k 40 x 0.01 * dmodel- Color represents absolute change in entropy upon ablation (AH).
where || ||dim=1 indicates column-wise norm.
This measure allows us quantify how specific the direct effect of neuron on the output logits, with the underlying intuition that diffused contribution (i.e., close to a constant added to all logits before applying the softmax) results in small effect on the output probabilities. In the last layer of GPT-2 Small, we identify subset of neurons with particularly low variance but substantial norm (FigureZhb). Since the optimization of the model's weights preserves these high-norm low-composition neurons (that are expensive in terms of weight decay penalty 48)). it is likely that these neurons are important and exert their effect through a different mechanism than directly modifying the logits.
3.2 Mechanism of action
Given the low composition with the unembedding matrix, we hypothesize that most of the effect of entropy neurons on the model's output is mediated by the re-scaling of the residual stream performed by the final LayerNorm. We test this hypothesis using a causal mediation analysis formulation [60 64), illustrated in Figure Zp. We distinguish the total effect that a neuron has the model's s output (represented by the two causal paths stemming from the neuron node in Figure 2p) from its direct effect (the green arrow in Figure 2b), which is not mediated by the change in the LayerNorm scale. Our hypothesis posits that the difference between the former and the latter is significantly larger for entropy neurons than for normal neurons.
We carry out an ablation experiment where we intervene on the activation value specific neuron by fixing it to its mean value across reference distribution while constraining the LayerNorm scaling coefficient to remain constant. More formally, consider a neuron with index E {1, ..... dmip) and denote by n, IR its activation value. Given input x E X, denote by the last hidden state in the model (i.e., the output of the last transformer block), and denote by x the last hidden state in the mean-ablating model obtained after mean- ablating neuron i:
x
x (n.
n. n.Jw outs
(4)
where is the mean activation value computed over subset of X.

---

We quantify the total effect of neuron upon mean ablation by measuring the absolute variation in the model's loss C specifically:
TE(i) = Es | (W)LLN())),) < <WWLLN(( ),x²)
(5)
where the expectation is taken over uniformly sampled subset of a corpus X. Similarly, quantify the direct effect of neuron i by preventing the LayerNorm denominator from varying:
DELN (i) = E. (W) - <<wwo( X m(x ')
Var(x) + о 07+ 2 ) (6)
For each neuron, we measure the total and direct effects when its activation value is set to the mean across a dataset of 25600 tokens from the C4 Corpus [3 (additional experimental details are provided in Appendix D) We compare these metrics for six selected entropy neurons against those from 100 randomly selected neurons. The results (Figure 2 show that in the randomly selected neurons there is no significant difference between their total and direct effects. On the other hand, the difference between these two quantities is substantial for entropy neurons, and in some cases the direct effect represents only a small fraction of the total effect.
These findings represent an interesting and novel example of how language models can use Layer- Norm to indirectly manipulate the logit values. Such a mechanism could easily be overlooked by conventional analyses, which typically focus on direct logit attribution and fail to account for the normalization effects imposed by LayerNorm [6
3.3 The unembedding has effective null space
The unembedding is linear map to a significantly higher-dimension space (e.g., from 768 to 50,257 dimensions in GPT-2). Therefore, is surprising that the output of high-norm neuron would have little effect on the logits. We hypothesize that there is subspace of the residual stream with an unusually low impact on the output that entropy neurons write onto. To investigate this, we compute the singular value decomposition (SVD) of the unembedding matrix Wt: = UΣVT.
Analyzing the singular values in E (thick blue line in Figure I), we observe that the bottom values are extremely small. In particular, we notice a sharp drop to values close to O around index 755. This observation indicates a remarkable phenomenon within the model: the training procedure optimizes the weights to preserve what effectively functions as a null space within Wu. This finding is striking, as it suggests that the model deliberately limits its representational power by projecting a set of residual stream directions onto small neighborhood of the O-vector.
Next, study the directions within the residual stream that entropy neurons write to. We do this by computing the cosine similarity between the output weights (Wout) for each neuron and singular vectors V E I dmoddal Xdmodal The results, illustrated by the colored lines in Figure 2 show that all the entropy neurons write almost exclusively to directions within the effective null space. These findings illustrate how entropy neurons leverage this null space to add norm the residual stream without directly affecting the logit values.
3.4 Universality across model families
The relationship observed between entropy neurons and the unembedding null space suggests that entropy neurons may be identified by analyzing the proportion of their norm that is projected onto such a null space. To test this, given a neuron we compute the fraction p; of the neuron's ''s norm projected onto the effective null space Vy In particular, we define the effective null space of Wu VT /
as the space spanned by its bottom singular vectors, i.e., VT := (VT dd
and we compute Ai VVVT (i) /// (/ l (-..1- Then, we measure the quantity which represents the fraction of the neuron's 's effect that is mediated by the final LayerNorm We
DE( DE(1)/E( TE)).
For readability, here we omit the final softmax to normalize the logits before computing the loss. 'We opt for representing the LayerNorm mediated indirect effect of the LayerNorm scale because a large indirect effect might highlight neurons with high norm and large direct effect, such as neuron that boosts a specific logit. In our setting, we are interested in neurons that actually have low direct effect, which not implied by large indirect effect.
mediattee effect by 1 DE(1)/ (E((()))) rather than computing the
5

---

(a) KL Div. From Token Freq. & Entropy
(b) Token Frequency Neurons
(c) Total & Direct Effects
Con: -0.63
mosdll
0.05
0.05
[ALassi
15
0.04
0.05
10
2007
2412
0.02
23,2017
23,3412
23,417
23,730
Random
1- DE TTE
10
15
20
Nouron Type
Normal
Entropy
Nearon
Dyg. (Pree (rood)
Token Prequency
Efrect Type
Total
Direct
( and Entropy are correlated negatively. (b) Scatter plot of neurons highlighting token frequency neurons (in green), with high effect on DEL (Pteegl moodel), significantly mediated by the token frequency direction. (c) Box plots showing substantial difference in total vs. direct effect in token frequency neurons.
Figure 3: Token Frequency Neurons in Pythia 410M. (a)
compare these two quantities for each neuron at the last layer in GPT-2 Small (Figure 2 with ==12 The results show that neurons with the largest projections onto the null space have the most significant soyeerorm--mdiittee effects on the model's output. These findings connect with recent work highlighting the importance the bottom Wu singular vectors [[O] and represent further evidence supporting our hypothesis about the entropy neurons' mechanism of action.
We extend our investigation of entropy neurons to models beyond the GPT-2 family. In Figure we analyze the proportion of norm projected onto the bottom singular vectors of Wu (i.e., p) and the influence of LayerNorm on the output for neurons the last layer of LLaMA2 7B our observations with GPT-2 Small, the results indicate the existence of distinct group of neurons that predominantly write onto the effective null space of Wu- These neurons significantly influence the model's 's output via LayerNorm, confirming the presence of entropy neurons across different model families. As entropy neurons act by re-scaling the residual stream and have no effect that is specific to subset of the tokens, their activation has large effect on the entropy of the output distribution but little impact on the relative ranking of the tokens. That is, they affect the model's confidence in the prediction without affecting the actual token being predicted. This is illustrated in Figure I We repeat these analyses on GPT-2 Medium, Pythia 410M and [ Phi 2 38), and Gemma 2B [68), obtaining overall consistent results (Appendix Fb.
(70] Similar to
4 Token frequency neurons
The token frequency distribution (i.e., the distribution of unigrams over a corpus) can serve as a reliable baseline for next-token prediction, especially in scenarios of high uncertainty [1] [5]. Thus, we hypothesize that a useful ceneral-purpose mechanism for regulating confidence in language models could involve modulating the distance between the output distribution and the token frequency distribution. We explore this hypothesis using the 410M- -aM-mmaaeettrr Pythia model, for which the train- ing corpus has been publicly released (The Pile: 20). For this corpus, we compute the empirical token frequency distribution Pireq- We observe that the entropy of the model's output distribution in fact negatively correlated with its Kullback-Leible (KL) divergence from the empirical token frequency distribution: as the confidence of the model decreases (higher entropy), its output distribution tends to be closer the empirical token frequency distribution (Figure 3).
As for entropy neurons, we aim identify neurons whose operational mechanism relies on a particular subspace- in this case, the direction corresponding the token frequency distribution. We obtain such a direction by computing the logit vector Vireq R by centering the log- log-probability values for each token in V. That is, Virega log log(Ptreq.)) m (log (ogg((r))))),,....))))))ggg))))) where pireq E IR [V] the vector of the token frequency probabilities values, and m H V + R indicates the mean function.
To identify the neurons that rely on this direction modulate the model's output, we conduct an ablation experiment similar to the one described in $ In this experiment, we mean-ablate neurons and assess the absolute change in model loss while preserving the value of the logits along the
'LLaMA actually uses RMSNorm [8] which differs from LayerNorm in the absence of re-centering and bias term. However, experimental procedure is not affected by this difference.

---

(a) 11.2378: Change Loss Upon Abl.
(b) 11.2378: Example of Activation
(c) Token Freq. Neuron: 23.417
2378 Act.
0.2
Abletion
10
0.8
AEntropy
02
0.5
0,4
-0.2
-0.4
-02
Sakin ceeetten
10 20 30
0.1
0.5
0.2
0.5
Lass
AD Free (Proas)
Figure 4: Examples of Neuron Activity in Language Models. (a) Change in loss after ablation of entropy neuron 11.2378 in GPT-2 Small. Color indicates reciprocal rank (RR) of the correct token prediction. (b) Activation of neuron 11.2378 on an example from the C4 Corpus. The neuron mitigates a loss spike at the token "Mes," after which the model predicts "otherapy." (c) Change in entropy and KL divergence on correct tokens (RR 1) post ablation neuron 23.417 in Pythia 410M. The neuron increases entropy and aligns the model's output with the token frequency distribution.
direction of Vfreq as constant. Denote by == (xLN(x) the logits produced by the model on input a E X. and denote by 1 WuLN((x>) the value of the logits obtained on the same input after the ablation of neuron i (where x is defined in Eq. 44).. To measure the effect of neuron i that is not mediated by the token frequency direction (i.e., its direct effect), we obtain the adjusted value of the post-ablation logits 1- as
(I" (ireg)
(1 iT Vireq) Vireq
-
V 2
Then compute the total and direct effects of neuron i as
TE(i) ExNx - (1+).)) ) and DErreq (i) = Exwx (((x)))(((++))) (8)
x
In Figure Bb, we report the results comparing the token frequency-mediated effect against the average absolute change in the KL divergence between Pmodel and Prreg for neurons at the final layer of Pythia 410M. We also highlight the entropy neurons with the highest LayerNorm-mediated effect (see Appendix F for comprehensive results on entropy neurons in Pythia 410M). We observe that there are neurons whose effect on Pmodel is substantially mediated by the token frequency direction (i.e., positive value along the x-axis in Figure Pp). These neurons, upon ablation, lead to significant variation in DKL (Prreg (PrreellPPoodd model) (large positive value along the y-axis in Figure 3b), comparable to the variations caused by some entropy neurons.
These results validate the presence of components (which we term token frequency neurons) that affect the model's s output distribution by bringing it closer or away from the token frequency distribution by directly modulating it. Figure 3 focuses on 5 selected token frequency neurons and shows their impact on the loss and decrease upon the inhibition of the token frequency direction (Eq. (7)), which parallels the LayerNorm mediation for entropy neurons discussed in $3 We additionally show the presence of token frequency neurons GPT-2 Small and Pythia 1B Appendix G
5 Examples of neuron activity
We have shown that confidence-regulltion neurons improve model performance by calibrating its output across the entire distribution. However, it remains unclear what this looks like in practice. To better illustrate the role that these neurons play in language models, we provide additional analyses about the changes in the model output induced by particular entropy and token frequency neurons.
Entropy neurons. In GPT-2 Small, we examine the change in entropy induced by one of the strongest entropy neurons identified in $3 (11.2378). To study this, we conduct an experiment in which we clip the activation value the neuron to its mean and measure the resulting change in the model output g We analyze the difference in loss caused by the ablation of the neuron on
"Clipping" refers to setting the activation value of the neuron to its mean only when it exceeds it, unlike mean ablation, which sets the activation value its mean regardless.
7

---

(a) Induction: Activations, Entropy, Loss
(b) Change in Entropy Upon Ablation
(c) 11.2378 Activation after BOS Ablation
Entropy Post Ablation/ Entropy
Entropy
....
11,584
10
Attn heads
10
11,2378
ind hesd
11.594
11,2870
13:29 www.ciiiioo
11.2578
11.2125
Ind head
11,2570
11.1611
Ind heacs 1-2
11.2123
11.2910
11.1611
Baselines
Oreginal
11,2910
20
1.00
150
50
100
250
50
200
Positton in Sequence
Position In Sequance
Position Sequeric
Figure 5: Entropy Neurons on Induction. (a) Activations, entropy, and loss across duplicated 200-token input sequences. (b) The effect of clip mean-ablation of specific entropy neurons. Neuron 11 11.2378 shows the most significant impact, with up to a 70% reduction in entropy. (c) BOS ablation of induction heads: Upon the ablation of three induction heads in GPT-2 Small, the activation of entropy neuron 11 11.2378 decreases substantially.
individual tokens and compare it to the initial loss value. This analysis is depicted in Figure 4h, which also illustrates the reciprocal rank of the correct token prediction the model's output distribution (RR). These observations suggest that the antropy-increasing
function of neuron 11. 1..2378 acts as hedging mechanism: marginally increases the loss on correct model predictions (low initial loss) but prevents the loss from spiking when the model confidently predicts a wrong token (high initial loss). In Figure we provide an example of input, taken from the C4 corpus [63] on which the model confidently predicts wrong token (" ("otherapy"), "), and neuron 11.2378 activates to mitigate the loss spike. We perform the same analysis for the strongest entropy neuron in LLaMA2 7B in Appendix F.1
Token frequency neurons. We next focus on neuron 23.417 in Pythia 410M (identified in $4). This is token frequency neuron: when its activation is increased, it pushes the model's output Pmodel model towards the token frequency distribution Pieeq. This is illustrated by Figure fe. showing the KL divergence between the two distributions pre- and post-ablation. As with increased entropy, bringing Pmodee closer to Piveq decreases the model's confidence (positive values along the x-axis correlate with negative y values in Figure 4 This leads an increase in loss on correct predictions, reflected in the negative change in loss upon ablation (darker points in Figure 4c). Interestingly, we find that the projection of this neurons' Wout onto the frequency direction Vireg is negative, suggesting that this neuron is suppressing common tokens and promoting rare ones. This indicates that the model on average biased predict common tokens even more frequently than their frequency would dictate. This way, lowering confidence by pushing Pmoddel closer to Pireg requires suppressing the token frequency direction, which is the opposite of what one would have naively expected.
Case study: Induction
As more detailed case study to examine the active role and celiddence-rggllatinn function of entropy neurons, we focus on the setting of induction. Induction occurs when there is a repeated subsequence of tokens in the model's input, and the model must detect the earlier repeat and continue it. Mechanistically. this is implemented by specialized attention heads, called induction heads 57), which attend from some token A to the token B that immediately follows A in the early occurrence and predict B (AB... > B). Repeated text frequently occurs in natural language (e.g., someone's 's name) and, during repeated subsequence, the next token can often be predicted with very high confidence.
6.1 Entropy neurons
To analyze this phenomenon, we create input sequences by selecting 100 tokens from C4 63 and duplicating them to form a 200-token input sequence. Across 100 such sequences, we measure GPT-2 Small's performance and observe a significant decrease in both average loss and entropy during the second occurrence of the sequence (solid lines in Figure 54)). Additionally, we track the activation val- ues of the six entropy neurons in GPT-2 Small analyzed (dotted lines Figure (p)). For four of these neurons (1 (11.584, 11.2378, 11.2123, and 11.2910). we note substantial change in average activa- tion values, suggesting that they may play an important role. Furthermore, among final-layer neurons, entropy neurons exhibit the largest change in activation during induction (Appendix H. Figure II).
8

---

To further explore the specific effects of entropy neurons on the model's 's output entropy and loss, we conduct an experiment where, during the second occurrence of the sequence, we clip the activation values of the neurons to their mean values from the first occurrence. The results (Figure (5)) reveal that while the ablation of some neurons such as 11.584 and 11.2123 leads to a slight increase in output entropy, the most significant impact was observed with neuron 11.2378. This results in up to a 70% reduction in entropy, suggesting its role in boosting entropy to counterbalance the model's high confidence during induction. We additionally study the effect of token frequency neurons during induction and we report the results in Appendix H..2, revealing that these neurons also lead to a signifi- cant change in entropy compared to randomly selected neurons. We repeat these analyses in naturally occurring induction cases identified in the C4 corpus, obtaining consistent results (Appendix H.I.
6.2 Induction head-entropy neuron interaction
In order to verify that induction heads have a causal effect on the activation of our entropy neurons, we perform BO$ ablations [31] That is, motivated by the observation that attention heads frequently attend to BOS when inactive (e.g., when induction is not occurring)
[72]. we set the attention pattern of an attention head to always attend to the first token of the sequence (i.c., BOS) and record the neuron's 's activation. We perform this procedure on the top 3 induction heads, selected according to the prefix matching score introduced by Olsson et al. [57]. In GPT-2 Small, these heads are L5H1, L5H5, and L6H9, respectively. As baselines, we randomly select heads in layers 5 and that, like our induction heads, have mean attention to BOS > 0.6 but have prefix matching score < 0.1. For both entropy neurons and baselines, we ablate heads individually and in combinations of up to 3 heads.
Figure 5p shows that ablating 1,51| (ind. head 1) leads to a significant decrease in 11 11.2378's s mean activation. Ablating L,H5 (ind. head 2) or L6H9 (ind. head 3) individually does not make substantial difference, but ablating them alongside I.5H1 further decreases activation, bringing 1.23378 close to its activation on the first 100 tokens. We perform additional BOS ablations for other entropy neurons in Appendix H and further study the induction head-entropy neuron interaction in Appendix [ providing prelminary evidence that entropy neurons respond to an internal "induction-bas-occuree signal that is produced by induction heads.
Related work
Uncertainty in language models. Uncertainty quantification for machine learning models has been extensively studied to assess the reliability of model predictions 19 2T. With the increasing adoption of LLMs, accurately determining their confidence has drawn significant attention [22]. Existing research has explored various strategies to calibrate model output, including ensembling model predictions [76] 34], fine-tuning [39]4]] 40], and prompting models to explicitly express their confidence
47
65
69 Additionally, efforts have been made to evaluate the truthfulness 9 3 50) and
35 of model outputs by inspecting internal representations. However, the investigation of internal, general-purpal mechanisms determining model confidence remains underexplored.
uncertainty
Interpretability of LLMs' components. The analysis of language models' internal components and mechanisms is an increasingly popular area of research (we refer to Ferrando et al. [8] for an
overview). Prior work has explored how to attribute and localize model behavior 73 23 52 and to identify specific algorithms that models implement to perform tasks [54] [7
29
67
12 Additionally,
many studies have focused on investigating individual neurons 41 61 14
13
74 27 77. However,
neurons are not always the correct unit of analysis, as they can be polysemantic (56 17 Recent
work has employed sparse autoencoders (SAEs) to find more interpretable linear combinations of
neurons [79 g 36). Despite this, in the context of entropy modulation, prior findings of relevant neurons motivated us to focus on neurons.
Closest related work. An anomalous class of neurons characterized by high norm and low compo- sition with the unembedding matrix was observed by Katz and Belinkov [2], though this was not linked the regulation of output distribution entropy. This class of neurons was independently redis- covered by Gurnee et al. [28] while investigating the universality of neurons in GPT-2 models. Our work connects the concept of entropy neurons to the presence of an effective null space represented by the bottom singular vectors of WT. Our findings highlight the significance of this seemingly unimportant subspace within the residual stream and align with the observations of Cancedda [10), who associated the bottom Wt u singular vectors with the phenomenon of the attention sink (78).

---

8 Conclusion
This paper presents analysis of the mechanisms by which LLMs manage and express uncertainty, focusing on two specific components: entropy neurons and token frequency neurons. We show that entropy neurons, with their high weight norm and low direct interaction with the unembedding matrix, affect the model's output via the final LayerNorm. We also introduce and investigate token frequency neurons, which adjust the model's output by aligning it closer to further from the token frequency distribution. In a case study on induction, we demonstrate the practical implications of these neurons. We show that one example role of entropy neurons is acting confidence in repetitive sequence scenarios. Some limitations of our work include focusing only on two types of components in the neuron basis, relying on proxies for confidence, and observing varying effects across models. We thoroughly discuss the limitations of our work in Appendix [A] This study represents the first thorough investigation into the mechanisms that LLMs might use for confidence calibration, providing insights that can guide further research in this area.
hedging mechanism to manage
Acknowledgments
This work was conducted as part of the ML Alignment & Theory Scholars (MATS) Program. Throughout the project, we received valuable input from fellow MATS participants. In particular, we would like to thank Andy Arditi, Joseph Bloom, Connor Kissane, and Robert Krzyzanowski. We also extend our gratitude to McKenna Fitzgerald for organizational support. Additionally, we would like to thank Arthur Conmy, Giulia Lanzillotta, Mazda Moayeri, Kumar Shridhar, and Vilém Zouhar for useful discussions and comments on our work. Yonatan was supported by the Israel Science Foundation (grant no. 448/20), an Azrieli Foundation Early Career Faculty Fellowship, and an AT Alignment grant from Open Philanthropy. Alessandro and Ben were supported by Long-term Future Fund. Alessandro is further supported by armasuisse Science and Technology through CYD Doctoral Fellowship. Ben supported though an EPSRC Doctoral Training Partnership Grant. The project was also supported by UK's ' innovation agency (Innovate UK) grant 100981 1 12 (project name ASIMOV: Al-as-a- -asvvcerr and by the European Union under the Horizon Europe vera.ai project (grant 101070093).
Author Contribution
Alessandro and Ben led the project, carried out the experiments, and wrote the paper. Alessandro proposed the causal formulation and conducted most of the experiments validating the entropy neurons' mechanism of action. He also performed experiments to identify entropy neurons and token frequency neurons across multiple models and wrote the majority of $$ [ to 5 Ben carried out numerous validation experiments ensure the robustness of our findings and obtained preliminary results on the universality of entropy neurons. Ben also led the experiments for the induction case study and wrote most of aand Apps. [Fpss[H]and[]]] Wes gave frequent and detailed feedback on experiment design and paper write-up. Yonatan, Xingyì, and Mrinmaya provided valuable input on experiment design and presentation of the results. Neel supervised the project and provided guidance at all stages on experiment design and analysis, in addition to editing the paper.
References
[1] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606. 06565, 2016. URL https:// arxiv. hwwp://gg//bbbbtbeeeeeo 1606. 06565
[2] Omer Antverg and Yonatan Belinkov. On the pitfalls of analyzing individual neurons in language models. In International Conference on Learning Representations, 2022. URL
https://openrevies: http:/pponrviiee.neeefforrrrm www./eeeoofoiiooo -u
[3] Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it's lying. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967-976, Singapore, dec 2023. Association for Computational Linguistics.. doi: 10.1 1/.11653//1/223... 100.116655511222222nnnnngeeem findingg--ennl...... URIhtttss: / /aaec://aacalaai aclanthology.ooo org//
2023. f. wwndiinggs-ii. emnlp. 68
10

---

1/20://111/2222.loooooogg [4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint org/ abs/1607 06450
167.06660 2016. URL httptt:t///arti
[5] Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shward Gillian Hadfield, Jeff Clune, Tegan Maharaj, Frank Hutter, Atilim Güneş Baydin, Sheila Mcllraith, Qiqi Gao, Ashwin Acharya, David Krueger, Anca Dragan, Philip Tor,, Stuart Russell, Daniel Kahneman, Jan Brauner, and Sören Mindermann. Managing extreme Al risks amid rapid progress. Science, 384 (6698))842-845, 2024. doi: 10.112//sciccc aan.ie..dn01117 URL https: doi/abs/10 www./cccicccccccceeeeee//ee/eccc adn0117
org/
[6]
Stella Biderman, Hailey Schoelkopf. Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023. URL httpp:://rroeeedings mlr
html html
19
[7] Steven Bills, Nick Cammarata, Dan Mossing. Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. OpenAl Blog, 2023. URL http::::///peaa..co///// language-moder models- can catess-cnneeppppppaa explain
neurons explain-neurons-- in- mongaage-models..
[8] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac HatfeeldDoddss, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. URL https //transformer cwwccitttt....e wtwpeuitsssuuuueeeeeeeeeeee put//222//ooooommmmntccc htwpraantiiiefeeeteeeeeeeeeeeeee featurrssiidde:
html
[9] Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 2023. URL htttt::/opppeeeerrrit
net/forumm teiimmmidd--iie ietiiiid-eEtubhhyhee
[10] Nicola Cancedda. Spectral filters, dark signals. and attention sinks. arXiv:2402.092221 2024. URL https://arxiv wwtpzivv.orggaaa wtt//bs////aeeeeee abs/ 09221
arXiv preprint
[11] Tyler A. Chang and Benjamin K. Bergen. Word acquisition in neural language models. Trans- actions of the Association for Computational Lingusttcs,, 10:1-16, 01 2022. ISSN 2307-387X. doi: 10.1 (622ttcl_a_000444 URL/https:///o// /ttp::::////iooo//1////// wtt.:o/gg///..........oo 11wp::111a1eeeaa1e01111110 a00044 6
[12] Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonoo. Towards automated circuit discovery for mechanistic inter- pretability. Advances in Neural Information Processing Systems, 36:16318-16352.
2023. URL https: lhai::://proceedingee www.pps.ccpppppperrrrroc html
[13] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villav- icencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers). pages 1493-8502, Dublin, Ireland, may 2022. Association for Computational Linguistics. doi:
URL
org/
wwg./022...eeeeoeeoo. 2022. ael-long 581
[14] Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nip models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6309-6318
2019. URL htRl.httts:/// aaai. org/ ojs/index. php/ hhlp//AAA/aann hww.A//artiieeeiiiiooo 4592 9
[15] Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers),
11

---

pages (6124-16170, Toronto, Canada, jul 2023. Association for Computational Linguistics. doi: 10. 10.18655//1/2233.acaa acl--onn...99, URL httpp:://aclaaaooollggooooooo/oe org 2023. acl acl-long. 893
[16] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Doddee Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. URL btttps://transformer.coo http://eeeeoeoooooooooooooo///// wwtpee/wwwooiine.x..o.ee index..ddex.e//.. html
5
5
[17] Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer EiShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Haffield-Dodds, Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Kadavath, Josh Jacobson, Eli Tran-Johnson Jared Kaplan, Jack Clark, Tom Brown, Sam McCandlish, Dario Amodei, and Christopher Olah. Softmax linear units. Transformer Circuits Thread, 2022. URL htttps://trrnssformess circuits. pubppppb//// 2022/ htt/:/soll/nnnee html 9
[18] Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-jussà. A primer on the inner workings of transformer- based language models. arXiv preprint arXiv:2405.0022 400000008000088 2024. URL.https wrwaarivvvv....//ee.aeaaaaeelaa. www.iv..rgg/bbbe 224p///...//./...././ 2405 00208
[19] Yarin Gal et al. Uncertainty in deep learning, 2016. URL httptttpp.......//2222 74/22199 20190729-1uuzy pdf
[20] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint 2xXiv:210100022: 2020. URL https
wwtp:/ogg//bb/////e/ //arxiv.org/ abs 2ttp/2101.000 abt/aaaa/////.../ 00027
6 19
[21] Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe. Rudolph Triebel, Peter Jung, Ribana Roscher, al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56 (Suppl 1):1513 15899 2023. URL https://link. springer com/ wttpa/rriice..11i1 1007/ s10462 023 10562-9
9
[22] Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. sur- vey of language model confidence estimation and calibration. arXiv preprint arXiv:2311.08200 2023. URL ftt.lthttss///// http:///aaaaiwwoooaa///////
org/
08298
[23] Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. Transformer are key-value memories.
layers Mtarie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic, nov 2021. Association for Computational Linguistics. doi: 10.18653/w1/2021..mms 1ww.221eemmlpppmmme. m4i..4-- URL htttt:::s/cccaaaaaaat
org 2021. www.eennniii.eeeee main. 446
9
[24] Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30-45, Abu Dhabi, United Arab Emirates, dec 2022. Association for Computational Linguistics. doi: 10. 18633////2022.emmlppppp main.3. URL https://aclanthoo.
22022 emnlp-main.3 3
2 3.5
[25] Aaron Gokaslan and Vanya Cohen. Openwebtext corpus, 2019. URL http://Skyyiiooo07
A.thub.io//pem@eelll iow.opppeeeeeeeeetorrrpsps 21
[26] Gemini Team Google. Gemini: A family of highly capable multimodal models, 2024. URL
https: hopps://azze //arxivv..ivv/// org/ abs/2312. 11805
2
12

---

[27] Wes Gurnee. Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL htttth::://perreiiieeeee net/
forum2id=JYs1RRIMJ. 9
[28] Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaci Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in GPT2 language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856, URL https://openrevieet net/
forumiid-2o1104223)
9
[29] Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater- than?: Interpreting mathematical abilities pre-trained language model. In Shirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: net/forum?id Peierrummddp4PPPccu/e ckNQR8k 9
[30] Charles R. Harris, K. Jarrod Millman, Stéfan van der Walt, Ralf Gommers, Pauli Vir- tanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Hal- dane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Srrard-Marccant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis Oliphant. Array programming with NumPy. Nature, 585(7825):357-3300 sep 2020. doi: 0000444156-000-6644
/tLLsttps://// http::::://oi..og/1....11 00111138/441 0010888/41886-0000 020-2-9-22--49999
T9
[31] Stefan Heimersheim and Jett Janiak. A circuit for python doc- Aligmment Forum, 2023.
strings in 4-layer attention-onl transformer. URL
http:://wwww.llliinmettffoor..m org http://www....oooo/oo///////a/aa
a- corircuit-for--
[32] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
1rXv:16666008155 2016. http://t///a wtwp://///aba////eeeeeee///////e 08415
[33] Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprint arXiv:2109. 13916, 2021. URL
13916
[34] Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. Decom- posing uncertainty for large language models through input clarification ensembling. arXiv preprint arXiv:231 1.087/8 18, 2023. (tRtss :tappss////xiii./ org/ httpaabb///1///////////
9
[35] Brian RY Huang and Joe Kwon. Does it know?: Probing for uncertainty in language model latent beliefs. In NeurlPS Workshop on Anributing Model Behavior at Scale, 2023. URL
9
[36] Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference Learning Representations, 2024. URL
id=F76bwRSLeK g
[37] Plotly Technologies Inc. Collaborative data science, 2015. URL hhttps://plll 1y
19
[38]
Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power small language models. Microsoft Research Blog, 2023. URL httpp:///wwttt http://wwww.ccoooft.oo/ www.soottttoocmmmmmmmmmm coweeefftccmeeeeeeeeeeeeeeeee http//ennusrrreeee htt///nuusrrrrercc//blgeg
phi thii-2thhe-see setneeesurrissingpposeoeee
smal www.ooofsaaaalaaaaaaaaa waagmmmllaggggeeeeanoome models/
2
[39] Zhengbao Jiang. Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language
models know? on the calibration of language models for question answering. Transactions of
the Association for Computational Linguistics, 9:962-977, 2021. doi: 10.1 162/tacl a a_00407.
URL httphttps/// //ap:s:///ccla http:///wlaaattoooorrrr///////// wwwpey...g/.2.../. tac1-111111
13

---

het//oruum2dd-CoJEtWWWS.. 9 [40] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan. Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac
Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint urXiv:2207.05221, 2022. URL https
//arxiv. abs/ /207 05221
[41] Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.0200 2015. URL https///aarr www.://.///aaaa.....oo..o...saaa /106 02078
[42] Shahar Katz and Yonatan Belinkov. VISIT: Visualizing and interpreting the semantic informa- tion flow of transformers. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the
Association for Computational Linguistics: EMNLP 2023, pages 14094-14113, Singapore, dec
2023. Association for Computational Linguistics. doi: 10. 1865/11112222 http:////222...nnngs-nnmnem findingg-emmlp.9999
URL https:t/accaataaaaneee
ptg/2023. findings emn 939
2
[43] Connor Kissane, Robert Krzyzanowski, Arthur Conmy, and Neel Nanda. Sparse autoen- coders work on attention layer outputs. Aligmment Forum, 2024. URL https://wwi
alignmentforum.cmm org/ www.:ooggggg/o.oieeeeeeee possts/DiaaluPPssdttsse...
25
[44] Robert Krzyzanowski, Connor Kissane, Arthur Conmy, and Neel Nanda. We in- spected every head GPT-2 small using SAES so you don't have to. Alignment Fo- rum, 2024. URL https: //www/wwwwwwwoo alignmentforum org http:ppotss/mmmee//nnningaaa Baveee//
we we-inspected-eree every- head-in in gpt smmalllussing saes so you- don. 23
[45] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023. URL http////pp/eeeeeee/////e/e wwtprrewwwwe//fouun
AdtPD-AYtPOdve T
[46] Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretabillty scale? evidence from multiple choice capabiliitess chinchilla. arXiv preprint arXiv:2307.00 09458, 2023. URL httpp::////xii http:::://xxxvv.rrp abs/
09458
2
[47] Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty words. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL
https://openrevaee
[48] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam, 2018. URL
https://openrevies
[49] Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. predittiion...... In International Conference on Learning Representations, 2021. URL /tppp:/t/////oo
met/forumiid=jll
[50] Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets, 2024. URI. ltttps://openrevien
[51] Clara Meister, Wojeiech Stokowiec, Tiago Pimentel, Lei Yu, Laura Rimell, and Adhiguna Kuncoro. natural bias for language generation models. In Anna Rogers, Jordan Boyd-Graber, Graber, and Naoaki Okazaki, editors, Proceedings of the 6t Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pages 243-255, Toronto, Canada, jul 2023. Association for Computational Linguistics. doi: 10.18655/v1/22233.cc- waw.:22r...l-s-o.222.2 URL https://aclanttol..g. org. 2023. acl-shorts 22
2,6
[52] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations GPT. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Newral Information Processing Systems, 2022. URL https::/ openreview www..neeeoormmidd
[53] Neel Nanda and Joseph Bloom. Transformerlens, 2022. URL.
19 24
14

---

[54] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress mea- sures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023. URL.
9
[55]
Nostalgebraist. Interpreting gpt: The logit lens, 2020. URL wtm.e//trr..llgggeteffrrrmmmm org htt.:/oott/ccccaB////aaaa/b/tttt
gpttsg-ptttheeee 1ese-logittt- logitllenn lens
[56]
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi:
9
[57]
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-oddds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. URL http:://trasssfrmerr- cwr.ciiicctcttiii ptt.uiits.p.b//22221 pub/2022/ in- Ann-oonext--eent learning and innuuiiduininnnn www.shheddddddheeeeeeeeee
[58]
OpenAl. GPT-4 technical report, 2024. URL https://aarxii org/ awt.aa/...00200 08774
[59]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL httLitttt//pprcooddinng http://wddninnnnnnnnnneeeee fi
wwt.:ee......p..pppppeeee httccc/pppp ccppaper
pdf
19
[60] Judea Pearl. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAr01, page 41 12-00, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001. URL https://dll acm. org//oi/o0
5555/2074022 2074073
[61] Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to generate reviews and discover- ing sentiment, 2018. URL httphttt:///oeeeeeeieee wwtp:/wwww....eooorrr.rd//////
g
[62] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019. URL attps://ansiihhcciivil
us- east-ast-11 1. amazonaws amaazonaas.comm/lagguug www.s//oogmaaaagaggnooeee... pdf
[63] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020. URL
https://www.
http://wwwww.....o..apaaeeaappep wttpppppes/v21///// wwtp:////2///000//// html
5 8 19
[64] James M Robins and Sander Greenland.
Identifiability and exchangeabil-
ity for direct and indirect effects. Epidemiology, 3(2) 1992. URL https:
1ww. wtt.coo////./// httpp//aw..aaa....oooooo//////aa and_ aod_eecchaggabiittyffoooooooomo httpp//cceeeaeeeeeee///////////p
and. aspx
[65] Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd- Graber, and Lijuan Wang. Prompting GPT-3 to be reliable. In The Eleventh International
Conference on Learning Representations, 2023. URL URLlattps: http://ppperrvee..eeeeee id=98p6x51L5af
[66] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine
learning research, 19(1):199--1988 2014. URL https: h//p://///ww..// jmlr org/ papers/
Doluuae///rrriisssssuu////rrvvvr pdf
wwwwstm.maaaaaaeeeeeeeeeeeeeeeee bfue 19
15

---

[67] Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpreta- tion of arithmetic reasoning language models using causal mediation analysis. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing, pages 7035-7052, Singapore, dec 2023. Association for Computational Linguistics. doi: 10. 10.118631111202.emmll www.:0223.mmmmmmmmmm3333333 mainn.35... URL https://aclanthollog org/ 222/./223.... emnlp-main. 435
[68] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre. Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08220 2024. URL https://arxs www.orgg////aoo...
08295
[69] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafacl Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5442, Singapore, dec 2023. Association for Computational Linguistics. doi: 10. 1/053//112023.mmm///// www.:emmmmlmmmmmmmm...m main.330. URL https:// cellanthology.org/22000 wttp:////00.....mmmm....... 330
g
[70] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint Stt.:2337700288, 2023. URL https
//arxiv. org /200 09288
2
[71] Ashish Vaswani, Noam Shazeer, Niki Parmar. Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural infor- mation processing systems, 30, 2017. URL https::/pproceedinngg neurips. cwturis..c/pappeeee cc/paper/ 7181 1181-attention-221 is-allli---llll--
[72] Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in a transformer language model. In Tal Linzen, Grzegorz Chrupata, Yonatan Belinkov, and Dieuwke Hupkes, editors, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63-76, Florence, Italy, aug 2019. Association for Computational Linguistics. doi: 10. 10.18853/v1// 19--9988..
URL attps://aclannthlogg..oo org./19 1etg/w19-4980 4808
[73] Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12401. Curran Associates, Inc., 2020. URL URt newr.ps......................... cc/ /acc//aper..oe filest http:/pappppp///// 2265082992217111ff332222aaa22332
[74] Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional. arXiv preprint arXiv:2309.04827, 2023. URL https://arxiv org/ org/abs/ htt//bs/23300 04827
[75] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference Learning Representations, 2023. URL https
//openreview.net.. wwt..//oruuumiii---ssssseee....
[76] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL
https://openrevier..ee wtt.://www.....eooradaad-a.a 1P-.LILI
9
[77] Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman, editors, Proceedings of the 9th Python in Science Conference, pages 56
61, 2010, doi: 10.25 10.25080/Majora- 9ttiiiorrr2211122--000 URL https:::: ptatts::/ssciie /ssippy....../////////.ppp/../ wecip..orrgg.........o......ee.e org 19
16

---

[78] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2023. URI. https:://
9
[79] Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann LeCun. Transformer visualization via dic- tionary learning: Contextualized embedding as linear superposition transformer factors. In Eneko Agirr,, Marianna Apidianaki, and Ivan Vulić, editors, Proceedings of Deep Learning In- side Out (DeeLiO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learn- ing Architectures, pages 1-10, Online, jun 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.d66 deelloo....l
URL https: httpppt http:///cccannttoooooor/////////
9
[80] Biao Zhang and Rico Sennrich. Root mean square layer normalization. In H. Wal- lach, H. Larochelle, A. Beygelzimer. F. d'Alché-Buc, E. Fox, and R. Garnett, edi- tors, Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc.. 2019. / newrips. http:/ggww..n.iieeppppppreeere.e httpp//seeeppppeffeeeeeeeeeeeeee
1e2119900228008334447777755555p9 http:rrejj.coopppppeee/////////p saaper_filee//ppprr///////11
pdf
[81] Vilém Zouhar. Ryanize bib, 2023. URI. httt:://gthaab..cc///ooberr////
T9
17

---

A Limitations
Our study focuses exclusively on two particular types of final-layer components: entropy neurons and token frequency neurons. However, likely that there are multiple other components and mechanisms within language models that contribute to confidence regulation. Exploring these additional components could provide a more comprchensive understanding how language models manage uncertainty.
We also acknowledge the lack of a precise definition of confidence. Instead, we rely on rough proxies such as entropy and the distance from the token frequency distribution. These measures, while useful, may not fully capture the concept of confidence in language models.
Although we observed the presence of entropy neurons in most of the models we studied, the fraction of their effect explained by LayerNorm re-scaling varies. For instance, in Pythia and Gemma, this fraction is around 40%, which is significantly less than the 80% observed in GPT-2, LLaMA2 7B, and Phi-2. We believe that variations in model architecture and training hyper-parameters may explain these inconsistencies. Although we do not explore these factors in detail, we discovered that the application of dropout during training leads to the emergence of a larger null space (Appendix E which we hypothesize may encourage the formation of entropy neurons. Future research should rigorously examine the factors that determine the emergence of entropy neurons, as well as other mechanisms that regulate model confidence.
Lastly, study is primarily focused how entropy and token frequency neurons contribute confidence regulation in next-token prediction. An interesting direction for future research would be to investigate oafidence-regulating neurons in broader contexts and real-world tasks, such as question-answering or reasoning. This could provide deeper insights into the practical applications and limitations of these neurons diverse scenarios.
B Broader Impacts
This research enhances the understanding of how LLMs regulate confidence in their predictions through entropy neurons and token frequency neurons. While our study primarily advances our technical understanding of language models, some broader impacts should be considered. Enhanced confidence calibration might lead to biased or discriminatory decisions if not adequately mitigated, perpetuating amplifying existing biases in training data. Additionally, insights from this research could facilitate extraction attacks, compromising data privacy, and be exploited for adversarial attacks, undermining model security. To mitigate these risks, it is essential to integrate robust fairness checks and bias mitigation strategies during model training and deployment. Implementing techniques, such as differential privacy, help protect against data leakage. Additionally, developing adversarial defense mechanisms and deploying real-time monitoring systems can counteract potential misuse.
C Weight Pre-processing
With the purpose of removing irrelevant components and other parameterization degrees of freedom, we employ a set of standard weight pre-processing techniques [53]. This way, cosine similarities and other weight computations have mean of 0.
Folding in Layer Norm. Most layer norm implementations include trainable parameters E IR" and B R (see Eq. (2). To account for these, we "fold" the layer norm parameters into W. by treating the layer norm parameters as equivalent to linear layer and then combining the adjacent linear layers. We create effective weights follows:
Wer Win diag(y), Ben = fm + Wi. B.
Finally, we center the reading weights because the preceding layer norm projects out the all-ones vector. Thus, we center the weights W... att as follows:
W' Win(i;)) Wer(i,;) W :.)(i.:).
Writing Weight Centering. Every time the model interacts with the residual stream, it applies LayerNorm first. Therefore, the components of W out and Gour that lie along the all-ones direction of
18

---

the residual stream have no effect on the model's calculations. Consequently, we mean-center W and ou by subtracting the means of the columns of W out:
W
out (:. 2) W Woul - W (o.r..
Unembedding Centering.
Since softmax translation invariant, we also center Wy:
Wc(:,) = -W(())-We(t)).
D Experimental details
We carry out all experiments using data from the C4 Corpus 663] which is publicly available through a ODC-BY license | solely to compute token frequency statistics, as Pythia was trained
We use the Pile (20). which is no longer distributed by its creators To
it and we required the original training data for the model. The experiments to measure the total and direct effects, both for entropy and token frequency neurons, were carried out on 256-token input sequences. We used 100 sequences for GPT-2 and Pythia 410M, 50 for Phi-2 and GPT-2 Medium, and 30 for LLaMA2 7B and Gemma 2B. The scatter plot in Figure 3a was obtained on 10000 tokens. The results on the induction case study presented in I were obtained on 500 input sequences, the shaded area around each line represents the standard error. For the sake of visualization clarity, the box plots in Figure 2 and Figure 3 do not include outliers: the whiskers represent the first and third quartiles. For naturally occurring induction, we selected 1000 sequences from C4 that contained repeated n-gram (n=6) and met our filtering criteria (each n-gram must be unique and contain no duplicate tokens). The BOS and mean attention output ablation experiments were each carried out on one hundred 100-token input sequences. Baselines for BOS ablation consisted of 6 single-head ablations, 3 two-head ablations, and 3 three-head ablations. Mean attention to BOS (used for baseline selection) was calculated using 1000 sequences from C4. To calculate prefix matching scores for attention heads, use 1024 sequences of duplicated random tokens.
Computing resources. The static analyses of the model weights were conducted on a MacBook Pro with 32GB of memory. The experiments carried out to quantify the neurons' effects were carried out on single 80GB Nvidia A100. The longest experimental run took approximately 12 hours. Our experiments were carried out using PyTorch [59] and the Transf ormersLens library [53] We performed our data analysis using NumPy 30 and Pandas [77]. Our figures were made using Plotly (37] The paper's bibliography was curated using Ryanize-bi [8]].
E Unembedding null space & dropout
Singular Values of Wy in Pythia 160m
1000
We hypothesize that the presence of a null space in the model's unembedding matrix Wo is signifi- cantly affected by the application of dropout [66 during training. Dropout represents an incentive for the model to encode information redundantly, align- ing it with a specific basis in the residual stream. We expect this redundancy to be reflected in some linear dependencies among the rows of We, effectively creating a null space. To empirically validate this hypothesis, we compare the singular values of Wu between two versions of Pythia 160M 6 identical except in their application of dropout to the residual stream (Figure 6
100
Volue
745
750
760
10
Singular Vector Index
Without Dropout With Dropout
Figure 6: Effect of Dropout on Wu. Com- parison of singular values for the unembed- ding matrix between two versions of Pythia 160M-one trained with dropout (red) and one without dropout (blue).
Our observations reveal that the singular values of the model trained with dropout are generally lower, sug- gesting lower stability and greater linear dependence among the rows of Wu. More notably, the difference in singular values between the two model versions
httppp///ppaaaatccccoooooo////// hatpss//theeegg...a////blll/LLLp txt
19

---

GPT-2 Medium
Pythia 410M
Pythia 1B
0.6
0.4
1-DEw/T
0.4
0.015
D-DEWTTE
DEDEWTE
0.5
0.2
0.02
0.01
0.005
-0.1
0.5
0.2
0.4
0.5
0.8
Fraction of Norm in Null Space (p)
Fraction of Norm in Null Space (p)
Fraction Norm Null Spacce (p)
Phi-2
Gemma 25
(a) LLaMA2 31.3228: ALoss Upon Abl.
0.4
[AH]
IAHI
1/RC
DE//TE
0.0
0.1
Nloss Post Ablation
0.4
0.8
0.4
0.015
0.6
0.01
Fraction of Null Space (p)
Fraction Null Space
AEntrroy Post Ablation
Figure 7: Entropy Neurons Across Different Models. Relationship between the fraction p of neuron norm in the Wu u null space and the LayerNorm- mediiaatmeddd effect on model output for neurons the last layer of various models. (a) Change in loss after ablation of entropy neuron 31.3228 in LLLMA2 7B. Ablation decreases entropy and increases loss on incorrect predictions.
larger for the 10 smallest values, indicating the existence of a subspace in the dropout-trained model that has substantially reduced impact on the logits. Such findings suggest that entropy neurons are more prevalent in models trained with dropout, as the subspace they leverage within the residual stream is more marked. However, this does not rule out the existence of entropy neurons in models trained without dropout.
Entropy neurons in different models
In Figure 7 we report the results for the final-layer neurons of GPT-2 Medium, Pythia 410M and 1B, earlier findings from GPT-2 and LLaMA2 7B discussed in $3 is computed by considering the the bottom 1% Wt singular vectors as null space. We observe consistent pattern: there exists subset of neurons that predominantly write the Wu null space. However, the extent to which entropy neurons appear in models appears to vary: we observe that entropy neurons in Pythia and Gemma exhibit weaker effect compared to those in other models. In particular, the fraction of their total effect explained by the LayerNorm scale is around 30-40%. which is notably lower than the approximately 80% observed in GPT-2 Small, LLaMA2 7B, and Phi-2. This suggests that while entropy neurons are common feature, their strength and the extent of their influence can vary significantly between different models.
Phi-2, and Gemma 2B, complementing
F.1 Additional example of neuron activity
As for neuron 11.2378 GPT-2 Small in $5 we analyze the strongest entropy neuron in LLaMA2 7B (31.3228). We perform similar analysis as in $5] where we clip the activation value of the neuron back to its mean when it exceeds it, and we measure the changes in loss and entropy after the ablation (Figure 7b). The ablation of this neuron leads to significant decrease in entropy, which results in loss increase on incorrect predictions and a loss decrease on correct predictions.
G Token frequency neurons across different models
we report the results comparing the token frequency effect against the average absolute change in the KL divergence between P Pmodel and Pireq for neurons the final layer of GPT-2
In Figure 3
20

---

(a) Taken Freq. Neurons in GPT-2 Small
(5) Taken Freq. Neurons Pythia 18
18 Dude Palle))
18 DuuP vell PPllll
0,20
To x
-0.5
0.5
Neuron Type
Normal
Entropy
Neuron Type
Normal
Entrogy
Token Prequercy
Token Frequeray
Figure 8: Token Frequency Neurons Across Different Models. Relationship between the token frequency-mediate effect and the average absolute change in the KL divergence from Preq for final-layer neurons in (a) GPT-2 Small and (b) Pythia IB. Neurons are categorized as normal, entropy, token frequency neurons.
(4) Induction in the wild Activations
(b) Change in Entropy Upon Ablation
(6) 11.2378: Change in Loss Upon
skat mauchan
11.304
11 584
URC
11.2276
2372
Detaa Loss Post Abtatnn
11,2870
Eroopp Prost Abbiton
11.0811
11,2300
0.
11 2012
Suadires
0A
02
7. 76 870848
4 80 6-
10
Relative Postioon
Rolative Posttion
loss
Figure 9: GPT2-Small Entropy Neurons on Natural Induction: (a) Change in activation values. (b) Change in entropy upon clipped mean ablation. (c) 11.3889 change in post-ablation loss. Color indicates reciprocal rank of the correct token prediction (1/RC). We use A0-A5 refer to the positions of the repeated n-gram; B and X indicates the token position that follows the first and second occurrences of the n-gram
Small and Pythia IB We also highlight the entropy neurons with the highest LayerNorm--mediated effect, represented by the points with the largest y-axis values in Figure 7
In both models, we observe a set of neurons with substantial token frequency-mediee mediated effect (~~0.6). In Pythia IB, the ablation of these neurons leads to a significant variation in the KL divergence from Ptreg. comparable to the variations caused by entropy neurons. On the other hand, in GPT-2 Small, the effect of the entropy neurons DKL (Preegl Pl mmudee is significantly larger, suggesting that these neurons might play a more important role on the regulation of the model's 's output compared to token frequency neurons.
Furthermore, both models, we observe neurons with significant negative DErreg/ TE values, indicating that their effect increases when the impact of the token frequency direction is suppressed. This suggests that the effect through the token frequency direction actually counteracts their effect that bypasses the frequency direction.
H Additional Results on Induction
H.1 Naturally occurring induction
To verify that these neurons behave similarly outside of our artificial setting, we perform the same experiments on examples of induction natural language data. We select 1000 sequences from the C4 corpus that contain a repeated n-gram of length n=6. In order to remove redundant and trivial examples, we also filter sequences such that the n-gram is unique and no individual n- n-grams contain duplicate tokens. We perform the same clipped mean ablation procedure as the synthetic setting, mean-ablating to the neuron's 's activation value on random text.
"We do not have access to the GPT-2 training corpus. However, since the model's training data included a substantial amount of internet-scrane scraped text, we use randomly sampled 500M-token subset of the Open WebText corpus (25 proxy for the original distribution.
21

---

(a) Induction: Activations, Entropy, Loss
(b) Change in Entropy Upon Ablation
(c) 23.417: Change in Loss Upon Abl.
Sart
... 23.57
Entropy Post Atletion Entropy
1.1
.... 23.87
... 23.417
23.730
22.730
23.427
ALoss Post Ablation
23,2017
0.8
23.2852
1.05
23.2952
23.3412
23:3412
-0.2
Baselines
0.5
0.2
-0.6
100
1.50
1.5
20
20
Position in Sequence
Position Sequencee
Loss
Figure 10: Token Frequency Neurons on Induction. (a) Activations, entropy, and loss across duplicated 200-token input sequences. (b) The effect of clip mean-ablation of specific token frequency neurons. (c) Scatter plot of loss changes per token post-ablation neuron 23, 23.417, colored by reciprocal rank (RR). Ablation tends to increase loss for low initial loss tokens and decrease it for high initial loss tokens.
Figure [ shows a similar trend to the synthetic setting shown in Vigure[5p. though the mean change in activations (and thus change in entropy) is less significant. The most impactful neuron remains 11.2378, the 'hedging' neuron that we believe counter-balange with this hypothesis, when the neuron's 's effect is removed via (clip) mean ablation, the model's predictions become more confident: entropy decreases by up to 20% of its original value.
overconfident predictions. Consistent
The other cutropy-increasigg neuron shown in Figure 99 is 1 1..2870. Unlike the other 4 entropy neurons, fires strongly only on the first few tokens of induction and then decreases in activation (Figure 12). In both synthetic and natural induction settings, 11.2870 has (Figure 5b and Dp), suggesting that it increases uncertainty at the start of induction where the model should be less certain that copying previous tokens will lead to the correct next-token prediction. (As an example, for the phrase 'Mr. Smith... ... Mr., the model should not be overly confident in predicting that the next token will be 'Smith', since different name could follow instead.) Since we choose a relatively short prefix length for our natural induction setting (n=6), the neuron is active across our entire repeated sequence and increases entropy.
effect
H.2 Token frequency neurons
To analyze the activity of token frequency neurons in the context of induction, we repeat the analysis carried out for entropy neurons. Using 100 input sequences created by concatenating two repetitions of a 100-token sequence, we study the activation values (averaged across token positions) for six token frequency neurons identified in $ within the last-layer MLP in Pythia 410M. We observe that, in this setting, four neurons (412, 2017, and 87) display a substantial increase in activation value (Figure [Oa). Upon clipped mean-ablation, neurons 417, 3412, and 87 lead significant increase in entropy compared to other neurons and baseline of 10 randomly selected neurons (Figure (Ob)..
Focusing more closely neuron 417, we find that reversing its state change leads to somewhat positive delta in loss for tokens with initially low loss, and larger, negative changes in loss for tokens with high initial loss (Figure (Fig This trend, which is inverse to that observed with entropy neuron 2378 in GPT-2, demonstrates how different mechanisms of entropy modulation can operate at different levels to balance the model's output distribution.
H.3 Change in activation of final-layer neurons
Figure II shows that entropy neurons exhibit among the largest changes in activation for final-layer neurons GPT2 Small and LlaMa2 7B, suggesting that they are performing an important function in induction scenarios. When comparing synthetic and natural induction, we note that the set of most-changing entropy neurons different (but overlapping). This explained by the fact that most entropy neurons increase in activation over the course of induction (e.g. 1 11.2378), whereas others, such as 11. 1.2870 only fire strongly at the start of induction. As discussed in Appendix [H.]] our natural induction setting highlights start-of-induction neurons, since it consists of short repeated n-grams (n=6). By contrast, our synthetic setting consists of long 100-token repeated sequences and so favors neurons that continuously increase in activation.
22

---

Mean AcL. Diff. for synthetic induction
Mean Act. Diff. for synthetic induction
Activaatoon Difference
Null Space Frac. (P)
Mean Activation Difference
31.2664
31.3717
Null Space
(F)
11. 11.594
0.8
11,2123
0.6
0.6
0.4
0.2
31.3228
0.2
1000
2000
3000
Sk
10k
GPT-2 Small Neuron Index
LLaMA2 7B Neuron Index Mean Act. Diff. natural induction
Mean Act, Diff. natural induction
Mean Activatioon Difference
11.2878
Null Space Frac. (o)
Mean Activation Difference
31,482
31.2725
Null Space
Frac. (p)
1.5
11.584
0.0
0.8
11 2970
0.5
0.4
-2
0.4
31.301
11. 16.1611
3220
-0.5
GPT-2 Small Neuron Index
LLaMA2 7B Neuron Index
Figure 11: Neuron activation differences for synthetic and natural induction: (left) GPT2-Small final-layer neurons (right) LLaMa2 7B final-layer neurons. Among final-layer neurons, entropy neurons exhibit the largest activation change for both synthetic and natural induction settings. Entropy neurons with especially high fraction of neuron norm in the null space (p) and large change in activation are labelled.
I Induction head-entropy neuron interaction
In this section, we study the interactions between induction heads and entropy neurons in more detail. In Appendix we use BOS ablations to verify that induction heads have significant causal effect for 5/6 of our entropy neurons. Next, in Appendix [.2 we provide preliminary evidence that induction heads produce an internal signal that indicates when induction has occurred, and that our entropy neurons respond to this signal. Finally, in Appendix [.3 we check the direct composition between entropy neurons and induction heads, showing that induction heads do have especially strong direct effect, and identifying set of components that may mediate the 'induction-har -ow-ooccr-edddddd occurred' signal.
L1 BOS Ablations for GPT2 Small neurons
We present the full results BOS ablations (..2) for all six of the entropy neurons under investigation in GPT2 Small (Figure (2). Neurons 11.584, 11.2123 123, and 11.2910 display similar trend to 11.2378: intervening on the top induction head, L5HI, leads to a substantial decrease in activation, suggesting has important causal effect. By comparison, ablating baseline heads in layers 5 and 6 have little impact on activation.
We note that for 1128870, our start-of-induction neuron (Appendix H.I. BOS ablation of LSHI has strong causal effect, but that the direction of this effect changes: on the first few tokens of the repeated sequence, the head boosts activation (activation decreases upon BOS ablation) but it suppresses activation across the remainder of the sequence (activation increases upon ablation). Prior work has found that attention head L.51 is specialized for long-prefia induction (induction over long sequences) [4.. Thus, we believe 11. 11.2870 is a concrete example of where such specialization is useful: the output of L.H1 enables 11. 1.28870 to fire strongly (and increase entropy) only at the start of induction.
1.2 Mean attention output ablations
Because entropy neurons respond strongly to induction and are causally influenced by induction heads, hypothesize that induction heads produce a generic 'induction has occurred' signal that
23

---

11. 11,594 Activvatio after BCS Abiation
11:1611 Activation after BOS Ablation
11,2870 Activation after BOS Abiitton
Adm heads
lown
Arm heads
Adn heads
Ind head
vnd head
head
11.584 Actvatcn
Ind head
11.1E11 autraion
hwad
11.2870 Actvationn
heed
and heads 1/2
Ind heads 1/2
0.5
heads 3/2
ind heads 1-3
Ind hwads 1-3
heods 1-8
Original Arz.
Original Ace
Original
My
M
ch
300
Prsdttio Sieq aence
Posilion Seqqarnce
Posilion in Siqquentz
11. 11.2123 Activation after BOS Abristion
11. 11.2910 Activetion after BOS Abiesion
11, 2378 Activetion after BOS Abxistion
Amm headss
of
Arm heads
Azn heads
0222 Actaaion
twed
11.2 2512810 Acteationn
hwad?
11.2379 Actvation
heed
Ind head 3
Ind Ieaad 3
ind tauad
Ind heads 1-2
hoads 2.2
headu 3-2
Ind heads
heads
Origital
Original
Original
Bacelinws
Raoe iilesiee
Secoel
c
Positoo
Postioo Sequence
Poction Sequence
Figure 12: GPT2-Small Entropy Neurons after BOS Ablations: Ablation of induction heads leads
to substantial decrease in activation for most entropy neurons.
11.584 1..544 Activation after Ablation
11.1611 Activation Mean Ablation
11,2870 Activation Mean Ablation
heads
oflnc.
11.504 Actiasion
ind heads 1-2
11.1611 Autraaton
Ind hwads 1-2
11,2870 Acdivation
0.6
heods 1-2
Ind heads 1.9
heads 1.5
heads 3/5
Original Act.
0.4
Orginall
0.2
-0.2
200
Sequence
Sequance
Sequence
11.2123 Activetion alter Mean Altn. Ablation
11. 2910 Activelion after Meen Atnn. Ablation
11. 2L.2378 Activetion after Mean Arn. Ablation
Spangl
Atn heads
of
heads
ind heod
and hwad 1
ind head
11,2123 Aztaiinnn
Ind heads 1-2
111200 11 Actvatton
hwate 1.2
11.2378 Acovatton
hoeds 1-2
Original
Original Act.
Original Act.
50
300
50
3C0
300
Positoo Sequence
Posttoon Sequence
Postion Sequence
Figure 13: GPT2 Small Entropy Neurons after Mean Attention Ablations: Replacing induction head outputs with their mean induction output leads to similar activations.
indicates to later components that the model is an induction context. To test this hypothesis, in our synthetic induction setting, we perform mean attention output ablations: replace each induction head's contribution to the residual stream with its mean contribution averaged over the second half of the sequence (i.e. when induction is occurring) 12 In this way, we hope to retain the generic "induction has occurred" signal, while removing -seenfi behavior, such as directly boosting the logits of attended-to token.
Figure 1 shows that mean attention ablations are able to produce activations that are close to their original values. For example, for 11. 11.2378, mean attention ablation of the top induction head results in small activation decrease from ~9 ~8. For comparison, recall that BOS ablation of this same head causes mean activation to fall to roughly 4. This suggests that a substantial portion of the
In practice, this is implemented using attention Z vectors ('attention output pre- pre-linear') rather than attention output. However, the two are equivalent for our purposes since Z vectors are converted to attention outputs by linear map [53]
24

---

11.594: DFA differncce
11. 1611: DFA difference
11. 1872870: DFA difference
Induction
induction
Induction score
DFA diffrencc
DFA diternss
score
2.
24
24
04
92
02
-0.4
Components
Components
Components 11. 11,2378: DFA difference
2123: DFA difference
2910: DFA difference
LI31B
Induction score
L13Mi
Induction soor
Induction Sooee
DFA diforenno
CFA difernnce
offerencce
05
D6
06
24
32
GATT
Figure 14: GPT2 Small Direct Effect of attention heads and mips on induction: DFA difference between induction and -oninuuctionn sequences. Attention heads are the left of the dotted line. Induction score refers to the prefix matching score.
entropy neurons' activation is derived from this generie induction signal, rather than token-specific information.
We note that though mean attention ablation is sufficient to raise the activations of our entropy neurons close to their original activations on the second half of the sequence, they do not have much impact over the first half of the sequence. (We would expect activation to increase, since adding an 'induction is happening' signal.) We speculate that this is because there are other important components that we have not factored into our analysis. For example, the signal from induction heads to entropy neurons is likely mediated by later-layer MI Ml.Ps/Attenton Heads. These could perform an AND operation over our induction heads' signals and duplicate token heads, such that both signals are necessary in order to produce the trigger for our entropy neurons. We leave a more thorough investigation of this potential circuit to future work.
1.3 Direct effect of components
In this section, we study whether the output of induction heads directly contributes to the activation of our entropy neurons, or whether their effect is mediated by other components. To do this, we produce a corrupted version of our induction sequences by replacing the first 100 tokens of each sequence with another random 100-token sequence sampled from C4. Then, for both our original and corrupted sequences, we use direct feature attribution (DFA) [43 to calculate the direct contribution of each preceding component in the transformer our neuron's 's activation across the last 100 tokens. This is done by taking the dot product of the output of each component (attention heads and MLPs) with the neuron's 's W vector, after applying LayerNorm.
Figure 14 shows the difference in DFA between the original (induction) sequence and the corrupted (non-induction) sequence, indicating which components were responsible for the change in activation. For entropy neuron 1 11.2870 induction heads have a strong direct contribution to the neuron's 's activation. However, this is not the case for our other entropy neurons. In particular, LII as well as MLPs in layers 8-10 have a strong direct effect, suggesting that the 'induction has occurred' signal effect may be mediated by these other components. Further evidence for this is suggested by the observation that the direct effect of induction heads is often opposite to the expected change in activation. That is to say, for entropy neurons that increase in activation upon induction, the direct effect of the induction heads is often negative (i.e. suppressing activation). and becomes even more negative upon induction. Since our BOS ablation experiments show that turning off these heads leads to decrease in activation, this suggests the presence of intermediate components that convert the negative direct effect signal from induction heads into a positive one.
25

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification:
and $4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
We do not provide any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification: We uploaded our code through the submission platform and we provide experimental details in Appendix D
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli//uuuu guwde//ddee/ee/ Cot//ooodeeesiseeeeeee hwtpbbbmsssiooooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: and Appendix D
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: $ and $4 we include box plots, which provide comprehensive information about the distribution of the effects measured, For the line plots in section So the standard error as shaded area around each line.
we indicate
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
28

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Appendix B
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix B
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any data or model with high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Appendix D
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
30

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "d1tx5FEzEK", "reviewer": "Reviewer_C4Wm", "summary": "This work studies a small but potentially important subset of neurons in the final layer for a trained transformer model that appear to regulate the confidence of a model (proxied by the variation in entropy of the model's output). Specifically, the paper builds upon entropy neurons found in previous work, extending the analysis further to show that these entropy neurons (which have high norm but low composition with the unembedding matrix) essential modify the null space of the unembedding matrix. This effectively scales the logits without changing the relative ranks of the tokens, thus regulating confidence. The paper continues this exploration by finding \"token frequency\" neurons, which are neurons that move the model's output distribution towards unigram token frequency distribution. The work claims that this is another form of confidence regulation, essentially \"moving\" the distribution towards unigram frequencies when the model is not confident about its prediction.\n\nThe paper provides the methodology for finding these neurons, as well as performs ablation/intervention experiments on these neurons to measure the difference in \"confidence\". Experiments are conducted on several pre-trained models, including GPT2, Pythia, LLaMA-2, Gemma and Phi-2. The paper also ends with a specific analysis of \"induction\"-behavior depicted in recent mechanistic interpretability works, studying how entropy/token-frequency neurons affect the output distribution when sequences are repeated in the output.", "questions": "- How are the token frequencies computed for the second half of the paper? Are these sub-word frequencies? On what dataset are these computed, and do you expect the training data distribution to have any effect on the results?\n- How many entropy neurons usually exist? From the plots, it seems this number may be somewhere in the single digit range? Explicitly stating this will still be useful information for the reader\n- Can you hypothesize why Gemma 2B has few to no entropy neurons?\n- Relatedly, what is the effect of model size on the number of these entropy/token-frequency neurons? Should one assume that model size has no effect? What about the vocabulary size?\n- Token freq is confusing; by changing distribution differently for every token, aren’t we changing the logits necessarily?\n- In Figure 4, what is the reciprocal rank exactly? Is a higher value better? \n- One thing that is unclear is how token-frequency neurons are regulating confidence specifically? I understand that they are modulating the output distribution towards unigram frequencies, but this affects both the logit ranks as well as the variance, so the effect on \"confidence\" specifically is unclear. I would appreciate some clarity on this.\n\n### General comments\n- It will be useful to state somewhere that your neuron indexing strategy is `<layer>.<neuron>`, which may be unclear to a new reader.", "limitations": "Authors have addressed limitations adequately.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The overall paper is well written, describing its motivation well, as well as the methodology. The figures presented do a good job at compressing the useful information in an accessible form.\n- The experiments conducted as convincing, and improve upon the findings of the previous works on which this paper is built upon. The experiments are also conducted on several models, which further showcases the universality of the findings.\n- The paper focuses on an important problem of understanding transformer models (although the immediate practical applicability of the findings is limited, it still answers important questions)", "weaknesses": "- While the results presented in the paper are indeed exciting, they raise a lot of questions (see Questions section) that are easy to answer but have been ignored by the paper. For instance, while the paper has run experiments on many models, there is very little discussion of these results. The paper would be much stronger if some additional discussion revolving around model and vocabulary sizes was included.\n- The paper mentions a few existing works on confidence regulation and calibration, but does not tie the work done in this paper within the context of this broader field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing that our work “focuses on an important problem”, with experiments that are “convincing, and improve upon the findings of the previous works on which this paper is built upon.”\nWe appreciate the many insightful questions you pose in your review.\nWe address the weaknesses and the reviewer’s primary questions (Q3, Q4, Q5, and Q7) in this rebuttal message. However, due to the numerous questions posed by the reviewer, many of which require detailed responses, and the character limit for the rebuttal, we address three clarification questions (Q1, Q2, and Q6) in a separate comment.\n\n> **Weakness 1:** The paper would be much stronger if some additional discussion revolving around model and vocabulary sizes was included.\n\nThank you for your input, we agree that further discussion on model and vocabulary sizes would enhance the paper. We address specific questions related to this issue (Q3 and Q4) below and will incorporate the corresponding answers and additional considerations into the final version of the paper.\n\n> **Weakness 2:** The paper mentions a few existing works on confidence regulation and calibration, but does not tie the work done in this paper within the context of this broader field.\n\nRecent work has explored various strategies to calibrate model output, including ensembling model predictions [1, 2], fine-tuning [3, 4], and prompting models to explicitly express their confidence [4, 5, 6, 7]. While our findings do not immediately suggest a strategy for improving models' confidence calibration, they provide valuable insights into the internal mechanisms that LLMs might use for this purpose. In connection with Burns et al. [8] and Azaria and Mitchell [9], our results suggest that it could be possible to estimate models' confidence states (e.g., overconfidence in specific settings) by tracking the activation values of a small set of neurons. This could potentially lead to new methods for dynamically adjusting model behavior based on real-time confidence estimates.\n\n> **Q3:** Can you hypothesize why Gemma 2B has few to no entropy neurons? & **Q4**: Relatedly, what is the effect of model size [...]? What about the vocabulary size?\n\nThese are good questions. We believe that the small presence of entropy neurons in Gemma might be due to two characteristics that differentiate it from the other models considered: the large MLP dimensionality and the large vocabulary size. \nIn Gemma, the MLP layers have \\~32k dimensions, which is roughly 10x larger than in GPT-2 Small and 3x larger than in LLaMA 2 7B. Given the very large number of neurons, the entropy-regulating function might be implemented in a different way within the final MLP layer.\nThe vocabulary used for the Gemma models is also significantly larger than for other models (\\~256k tokens, 8x larger than LLaMA’s vocabulary). Such a high dimensionality in the unembedding projection might be in contrast with the presence of a null space in the projection matrix.\n\nIn general, a larger vocabulary size implies a higher-dimensionality mapping performed by the unembedding matrix. The projection from a low-dimensional space to a higher-dimensional space makes the presence of a null space in the projection matrix more costly. In other words, dedicating the same number of dimensions to a null space results in a greater loss of representational capacity, making the presence of an unembedding null space (and therefore of entropy neurons) less likely.\n\nOne non-architectural but training-related factor that we studied, in relation to the emergence of entropy neurons, is the application of dropout. Even though dropout cannot be the sole factor influencing this phenomenon, as we observe entropy neurons in LLaMA, which was trained without dropout, we observed that it has an effect on the size of the unembedding effective null space (the results are reported in Appendix G).\n\nIn conclusion, studying the architectural and training factors that determine the emergence of entropy neurons is an interesting direction for future research that warrants further investigation.\n\n> **Q5:** By changing distribution differently for every token, aren’t we changing the logits necessarily? & **Q7:** One thing that is unclear is how token-frequency neurons are regulating confidence specifically? \n\nToken frequency neurons, as opposed to entropy neurons, do affect the output logits directly, and you are correct in noting that they impact logit ranks. The confidence-regulation function of these neurons is achieved by shifting the output distribution closer to or further away from the token frequency distribution. This token frequency distribution is what the model can default to in cases of high uncertainty.\nThis phenomenon is supported by the observed anti-correlation between the entropy of the model's output distribution and its KL divergence from the token frequency distribution. Intuitively, the token frequency distribution represents an educated guess for the next-token prediction (e.g., when little or no contextual information is available), thus providing a baseline confidence level for the model. Token frequency neurons regulate the model's confidence in its predictions by adjusting how closely the output distribution aligns with this baseline.\n\n---\nIn conclusion, we would like to thank the reviewer for their input. We will incorporate their comments and the corresponding answers in the final version of the paper. In particular, we will include the details about the token frequency computation (Q1) in Appendix D and reference it in Section 4. We will add the considerations about the effect of model and vocabulary size on the presence of entropy neurons (Q3 & Q4) in Section 3.4, the definition of reciprocal rank (Q6) in Section 5, and we will correct the neuron notation. Thank you again for the thorough feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cIph5v45TD", "reviewer": "Reviewer_BDF8", "summary": "This paper investigates two kinds of neurons by which transformer language models calibrate their predictions. These are (1) “entropy neurons”, which can affect logit values, but do not promote specific tokens, and (2) “token frequency neurons”, which influence a model’s likelihood of outputting bigram word statistics.", "questions": "- The total effect is much larger than the direct effect for both sets of neurons. For the “token frequency neurons”, does this mean they are influencing other components that actually promote bigram statistics? Or what exactly is their effect mediated by? It is unclear to me how the same direction could promote the most common bigrams for every distinct token in the vocabulary.\n\n- Are entropy neurons and token frequency neurons only studied at the last layer of the model? Do they appear elsewhere, and are just strongest at the last layer?\n\n- Does the discovery of these neurons influence how we should think about using direct logic attribution as a way to understand components and their interactions? Lines 70-71 briefly mention this, but could you can briefly expand on what is meant here?", "limitations": "The authors do point out a number of limitations to their study, which are valid and could strengthen the paper when/if addressed. Though many may not be in scope.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Calibration of model confidence is an important area of study for trustworthy deployment of ML systems.\n- The experiments attempt to show that both kinds of neurons for model calibration arise across various model sizes and architectures up to LlaMa-2-7B, suggesting this description is general to an extent (varying success across models).\n- The results have nice synergy with existing work, providing a clearer understanding of how these previously discovered neurons can influence model behavior.", "weaknesses": "- It is unclear what the connection is between “token frequency neurons” and “entropy neurons”. Do these two kinds of neurons interact (if so, how?) or are they separate mechanisms by which models calibrate logits? As this is a major part of the paper, it should be made clear.\n\n- The case studies provided are pretty simple settings. Studying the effect of calibration neurons on more real-world tasks like question answering - where there is a small set of potential answers and seeing if and how the same mechanisms apply would help strengthen the generality of the claims made. (This is noted by the authors in the limitations section, and I agree with them).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing that our work addresses “an important area of study for trustworthy deployment of ML systems”, providing “a clearer understanding of how these previously discovered neurons can influence model behavior.”\n\n> **Weakness 1:** Do these two kinds of neurons interact (if so, how?) or are they separate mechanisms by which models calibrate logits?\n\nThe two types of neurons represent separate mechanisms that language models can implement to calibrate their logit values. Their activation might be triggered by different features of the input, and they affect the output distribution differently. While they can be active simultaneously (e.g., during induction, where entropy neurons play the most significant role), they exhibit distinct activation patterns. Token frequency neurons tend to activate densely, with activation values significantly larger than 0 occurring frequently. In contrast, entropy neurons generally activate sparsely, often in response to specific sequences like repetitions, structured strings (e.g., email addresses and links), and common phrases or n-grams that are likely repeated in the training data.\n\n> **Weakness 2:** Studying the effect of calibration neurons on more real-world tasks like question answering [...] would help strengthen the generality of the claims made. (This is noted by the authors in the limitations section, and I agree with them).\n\nWe agree with the reviewer that showing the effect of entropy and token frequency neurons on more real-world tasks like question answering would provide strong evidence for the generality of the mechanisms we study. However, we would like to highlight that our research is the first to investigate these specific mechanisms within language models. By identifying and characterizing entropy neurons and token frequency neurons, we provide a foundational understanding that future work can build upon. Our findings offer an important starting point for exploring these mechanisms in more complex and varied settings, and we are optimistic that subsequent research will extend our work to a broader range of tasks, including real-world applications like question answering.\n\n> **Q1:** The total effect is much larger than the direct effect for both sets of neurons. For the “token frequency neurons”, does this mean they are influencing other components that actually promote bigram statistics? Or what exactly is their effect mediated by? It is unclear to me how the same direction could promote the most common bigrams for every distinct token in the vocabulary.\n\nToken frequency neurons work by promoting or suppressing the *unigram* distribution component in the model's output. This means they adjust the model's predictions based on the individual token frequencies rather than sequences of two tokens (bigrams). We demonstrate this mechanism by identifying the residual-stream direction representing the unigram/token frequency distribution. (To provide some intuition: increasing the value of the residual stream along this direction promotes common tokens and suppresses uncommon ones.) We show that this direction mediates a significant portion of these neurons’ effect on the output: when we prevent the residual stream from varying along this direction (isolating the direct path that \"bypasses\" the token frequency direction), the effect of these neurons decreases significantly. In other words, the total effect is larger than their direct effect (which excludes the contribution along the token frequency direction), indicating that these neurons significantly influence the model's output distribution by modifying the residual stream along the unigram/token frequency direction.\n\n> **Q2:** Are entropy neurons and token frequency neurons only studied at the last layer of the model? Do they appear elsewhere, and are just strongest at the last layer?\n\nWe focused our study on final-layer neurons, as we expect entropy neurons to be strongest at the final layer, where their effect on the output cannot be mediated by other intermediate model components. However, these neurons might also be present in previous layers. An exhaustive search and comparison of entropy neurons across layers represent an interesting direction for future analysis.\n\n\n> **Q3:** Does the discovery of these neurons influence how we should think about using direct logic attribution as a way to understand components and their interactions?\n\nThank you for bringing up this point. We believe this is an interesting insight from our study. Interpretability analyses based on direct logit attribution typically involve projecting an internal model representation or weight vector onto the vocabulary space to determine whether the set of tokens being promoted share a specific feature or concept [1, 2]. Our work shows that entropy neurons write onto a residual-stream effective null space, which gets mapped onto a neighborhood of the zero vector in the vocabulary space. If we were to interpret these neurons via direct logit attribution, we would mistakenly conclude that their impact on the model prediction is minimal and uninterpretable, missing their interaction with the final LayerNorm.\nThese observations suggest that, while performing direct logit attribution, we must consider that some subspaces in the model residual stream might not influence the next token prediction directly. Thus, we might be trying to understand the *direct* contribution of a model component to the final prediction when its main effect is *indirect* (e.g., mediated by LayerNorm) and its direct effect is minimal, even when the component studied is at the final layer of the model.\n\n---\n\n[1] Elhage, N.,et al., A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.     \n[2] Geva, M., et al., Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. EMNLP 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "niFIVSUmpC", "reviewer": "Reviewer_Thsu", "summary": "The paper investigates specific neurons in LLMs  (termed \"confidence regulation neurons\")  that modulate the uncertainty of the next token prediction by modulating the output distribution. First, entropy neurons modulate the overall entropy of the output distribution by writing to an effective null space of the unembedding matrix, thereby influencing the residual stream norm with a minimal direct effect on the logits themselves. Second, token frequency neurons modulate the predictive distribution proportionally to the token frequency in the training data.", "questions": "- Why did the authors analyze 6 entropy neurons but only 5 token frequency neurons?\n- In Figure 5(a), are entropy, loss, and neuron activations really all on the same scale (single y-axis)?\n- Why does the induction case study not consider the novel token frequency neurons but only the known entropy neurons?", "limitations": "The authors adequately addressed limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper clearly written and well-organized, making complex concepts accessible.\n- It provides deeper insight into the role of entropy neurons in regulating the confidence of LLMs through the unembedding null space.\n- It introduces token frequency neurons, a type of neurons that have not been discussed in prior work.\n- It demonstrates the presence of these entropy and token frequency neurons across various models.", "weaknesses": "- **Novelty**: The authors claim that they provide a \"novel example of how language models can use LayerNorm to indirectly manipulate the logit values\" and that prior studies \"fail to account for the normalization effects imposed by LayerNorm\" (lines 160 to 163).\nHowever, the mechanisms of entropy neurons have already been discovered in previous work [1]. For instance, Gurnee et al. (2023) show that entropy neurons \"modulate the model’s uncertainty over the next token by using the layer norm to squeeze the logit distribution, in a manner quite similar to manually increasing the temperature when performing inference.\". Although prior studies do not differentiate between the total and direct effect of an entropy neuron on the output distribution, the novelty of the analysis is not clear-cut. \n\n- **Clarity**: Individual neurons are referred to as *layer.position* and simply *position* interchangebly (e.g. 11.2378 and 2378). Since only neurons in the final layer were investigated, referring to the layer is redundant. Also, the authors should consider further simplifying the name of the neurons, as the exact *position* does not add immediate value to the reader.\n\n- **Interpretability**: The interpretation of the results is sometimes unclear. For instance, the authors observe that token frequency neurons suppress common tokens and promote rare ones. They suggest this indicates the model is biased to predict common tokens more frequently than their actual occurrence. However, an alternative interpretation could be considered. The model might also assign high probability to a single rare token while assigning almost no probability to other rare tokens. Promoting all other rare tokens to match their token frequency would bring the output distribution closer to the token frequency distribution, while increasing entropy and lowering confidence. In other words, the model's bias might not be just towards common tokens but also towards certain rare tokens. Such alternative interpretations would also resolve the counterintuitive explanation. The authors should provide more insights to support their interpretations.\n\n---\n[1] Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing that our work “provides deeper insight into the role of entropy neurons” and that our paper is “clearly written and well-organized, making complex concepts accessible”.\n> **Novelty**: The novelty of the analysis is not clear-cut.\n\nGurnee et al. identified \"entropy neurons\" with large norms and low compositionality with the unembedding matrix across GPT-2 models trained with different seeds. They hypothesized that these neurons modulate the model's uncertainty through LayerNorm. Their experiments involved artificially increasing these neurons' activation values, leading to significant changes in LayerNorm scale and entropy. While insightful, these observations do not conclusively establish LayerNorm as the primary mechanism.\nOur work extends these findings in several ways:\n1. We identify and quantify the presence of an unembedding null space.\n1. We differentiate and quantify the total and direct effects of entropy neurons, establishing LayerNorm as the key mechanism mediating their impact.\n1. We analyze examples where entropy neurons naturally achieve maximum activation, providing insights into their behavior.\n1. We detect entropy neurons across multiple models, reinforcing the generality of our findings.\n1. We demonstrate the practical implications of entropy neurons in the induction setting, highlighting their role in modulating model confidence.\n\nIn conclusion, our study offers a substantially deeper and more rigorous investigation into the mechanisms of entropy neurons, building on the work of Gurnee et al. while introducing novel analyses and findings. Combined with our discovery of token frequency neurons, we believe our work provides a substantial degree of novelty and contributes meaningfully to the understanding of confidence regulation in language models.\n> **Clarity:** Individual neurons are referred to as layer.position and simply position interchangeably\n\nThank you for pointing this out. We will add a sentence to clarify the notation and ensure consistency across the entire paper.\n> **Interpretability:** The model might also assign high probability to a single rare token while assigning almost no probability to other rare tokens. \n\nThank you for helping us consider alternative hypotheses for our observations. It is important to explore different interpretations, and we appreciate your feedback.\n\nTo clarify this point, we dug deeper into the effect of token frequency neurons on the model output. We know that token frequency neurons suppress common tokens and promote rare ones, and the fact that we observe an increase in entropy upon such contribution leads us to conclude that (1) the model's output distribution is typically skewed towards assigning probability mass to common tokens that is higher than the token frequency dictates. The reviewer suggests that the same change in entropy could be observed if (2) the model assigns high probability to a single rare token while assigning almost no probability to other rare tokens.\n\nTo test these interpretations, we studied the change in the KL divergence between the model's output distribution (averaged over 15M tokens) and the token frequency distribution, while adding to the model’s output logits the logits representing the unigram distribution (multiplied by a factor $k$ that we vary). Adding this vector promotes common tokens and suppresses rare ones, while subtracting it achieves the opposite effect, simulating the contribution of token frequency neurons.\n\nIn the scenario suggested by Interpretation 2, adding the token frequency vector to the output logits would remove probability mass from the rare tokens (and particularly from the rare token that received a high probability score) and increase the probability assigned to common tokens. If this were the case, we would observe a final output distribution more aligned with the token frequency distribution, indicated by a decrease in the KL divergence between the two distributions.\n\nAccording to Interpretation 1, when adding the token frequency vector to the output logits, the already high probability assigned to common tokens should become even larger, making the output distribution even more skewed. Therefore, the KL divergence between the model's output and the token frequency distribution should grow even larger.\n\nThe results of our analysis are illustrated in the attached PDF. We observed that the KL divergence between the model's output and the token frequency distribution increases as we add the token frequency vector to the logits, which is consistent with Interpretation 1. We will include these results in the final version of the paper.\n> **Q1:** Why did the authors analyze 6 entropy neurons but only 5 token frequency neurons?\n\nThere was no particular reason for analyzing 6 entropy neurons but only 5 token frequency neurons. Our results (Figs. 2e, 2f, and 3b) show that entropy neurons and token frequency neurons exist on a continuous spectrum rather than distinct clusters. For our in-depth analyses, we focused on the most pronounced outliers to represent these mechanisms effectively, rather than investigating every neuron that exhibited these characteristics. \n> **Q2:** In Figure 5(a), are entropy, loss, and neuron activations really all on the same scale?\n\nCorrect, they are all on the same scale. \n> **Q3:** Why does the induction case study not consider the novel token frequency neurons but only the known entropy neurons?\n\nWe do provide results for token frequency neurons on induction in Appendix H. Similar to entropy neurons, the activations of token frequency neurons change substantially upon induction. \n\n---\nIn conclusion, we would like to thank the reviewer for their feedback, particularly regarding the interpretation of our results. We will include the additional analysis in the paper to provide further clarity. We hope that, in light of these clarifications and additional analysis, the reviewer will consider increasing the overall rating.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iGRoVfZI6c", "reviewer": "Reviewer_S9nW", "summary": "This paper studies how large language models (LLMs) regulate uncertainty in next-token predictions through specific components: entropy neurons and token frequency neurons. Entropy neurons, identified by their high weight norm and minimal direct impact on logits, influence model confidence by operating within an unembedding null space. Token frequency neurons adjust logits based on token frequency, modulating the output distribution toward or away from the unigram distribution. The study includes a detailed examination of these neurons in various models, demonstrating their role in managing prediction confidence, particularly in repeated sequence scenarios.", "questions": "Please, read my questions and suggestion as a starting point for a follow-up paper.\n- Could you provide a more precise definition of confidence in the context of LLMs?\n- While LayerNorm plays a crucial role in the functioning of entropy neurons, what might be the impact of other normalization techniques (for example, BatchNorm or RMSNorm) on these neurons?\n- I'd suggest also to perform an analysis of how specific features of the models and training hyperparameters influence the effectiveness of entropy and token frequency neurons.", "limitations": "The authors properly addressed all the possible limitations of the paper.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The extensive experimental validation, including ablation studies and cross-model analysis, gives empirical support to the theoretical claims, enhancing the overall robustness and reliability of the study.\n- The paper is very well written, and it was easy to follow through even though the topic is rather complex.\n- I think that the paper provides an good mechanistic explanation of how entropy neurons operate through the unembedding null space and LayerNorm, improving our understanding of their indirect impact on model predictions.\n\nOverall, I find the paper's findings very valuable, the methodology is rouboust, and the authors adressed themselves the few limitations of the paper. I honestly enojoyed reviewing this paper.", "weaknesses": "- As the authors say in the \"limitations\" section, they use entropy and distance from the token frequency distribution as proxies for confidence, which may not fully capture the complexity of confidence in language models.\n- The extent to which entropy neurons influence model output via LayerNorm seems to change across models. This variability suggests that model architecture and training parameters could play a important role.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing that our work provides a “good mechanistic explanation of how entropy neurons operate [...], improving our understanding of their indirect impact on model predictions.” We are encouraged by your recognition of our findings as \"very valuable\" and appreciate that you enjoyed reviewing our paper. Your positive feedback is greatly appreciated.\n\n> **Weakness 1** & **Q1:** Could you provide a more precise definition of confidence in the context of LLMs?\n\nThis is a good and challenging question. Defining and quantifying uncertainty in machine learning models, including LLMs, is a complex challenge. In general, the uncertainty of a machine learning model can be divided into two types: aleatoric, which arises from inherent noise in the data, and epistemic, which stems from the model’s lack of knowledge [1].\nIn the context of LLMs, which predict a probability distribution over a set of possible next tokens, possible candidates for gauging the model's confidence are, for instance, the largest predicted probability value, the gap between the two highest probabilities, or other, more global properties of the output distribution, such as entropy or divergence from a baseline distribution [2, 3]. We opt for the latter measures as single probability scores neglect broader characteristics of the output distribution.\nIn particular, the token frequency distribution represents a natural baseline for next-token prediction, which models resort to at early stages of training [4, 5], and serves as an educated guess based on the previously seen tokens.\nAlthough we believe that our chosen proxies are reasonable and practical measures for assessing a model’s confidence in its next-token prediction, we acknowledge the challenges in uncertainty estimation. We will add a note to the paper to highlight this point.\n\n\n> **Q2:** While LayerNorm plays a crucial role in the functioning of entropy neurons, what might be the impact of other normalization techniques (for example, BatchNorm or RMSNorm) on these neurons?\n\nRMSNorm is essentially identical to LayerNorm, except it does not center the residual stream by subtracting its mean. Since the impact of entropy neurons is mediated through the re-scaling operation of LayerNorm rather than the centering, the function and presence of entropy neurons should not be affected by using RMSNorm instead of LayerNorm. Empirical validation of this comes from LLaMA 2, which uses RMSNorm and still exhibits the presence of entropy neurons.\nBatchNorm, on the other hand, normalizes along the batch dimension. This operation involves normalizing across a dimension external to the model's architecture because the model, during a forward pass on input $x$, has no access to information about activations computed on other inputs $x'$ in the same batch. Therefore, we do not expect BatchNorm to impact the presence or function of entropy neurons.\n\n\n> **Q3:** I'd suggest also to perform an analysis of how specific features of the models and training hyperparameters influence the effectiveness of entropy and token frequency neurons.\n\nThis is definitely an interesting question. In Appendix E, we analyze the effect of one training hyperparameter: the application of dropout during training. We study this in two versions of Pythia 160M that differ only in the application of dropout. We observe that dropout leads to lower unembedding singular values and, interestingly, to a more pronounced effective null space, making the presence of entropy neurons more pronounced. We agree that a more systematic analysis of the impact of architectural choices on confidence regulation neurons is an interesting avenue for future research and could provide valuable insights into how these mechanisms emerge.\n\n\n---\n\n[1] Kendall, A. and Gal, Y., What uncertainties do we need in bayesian deep learning for computer vision?. NeurIPS 2017.    \n[2] Huang, Y., et al., Look before you leap: An exploratory study of uncertainty measurement for large language models. arXiv 2023.    \n[3] Yoshikawa, H. and Okazaki, N., Selective-LAMA: Selective Prediction for Confidence-Aware Evaluation of Language Models. In Findings of EACL 2023.    \n[4] Meister, C., et al., A Natural Bias for Language Generation Models. ACL 2023.    \n[5] Chang, T.A. and Bergen, B.K., Word acquisition in neural language models. TACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their thorough feedback. We are glad they found our work well-motivated, clearly presented, and insightful. We address each reviewer’s points in the respective rebuttal sections. Additionally, we attach a PDF file with the additional results referenced in the response to Reviewer Thsu.

Meta-review: The work studies two types of neurons in the last layer of the transformer models that are responsible for calibrating their predictions. The paper makes a small solid contribution to the field of interpretability. The study would be of more value if conducted or extended on real-world tasks. At the current form of the result, it is unclear how generalizable are the findings presented in the paper.
Decision: Accept (poster)