Confidence Regulation Neurons in Language Models

Authors: alessandro  stolfo
ben  wu
wes  gurnee
xingyi  song
mrinmaya  sachan
neel  nanda
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 10:51:32

Abstract: Despite their widespread use, the mechanisms by which large language models (LLMs) represent and regulate uncertainty in next-token predictions remain largely unexplored. This study investigates two critical components believed to influence this uncertainty: the recently discovered entropy neurons and a new set of components that we term token frequency neurons. Entropy neurons are characterized by an unusually high weight norm and influence the final layer normalization (LayerNorm) scale to effectively scale down the logits. Our work shows that entropy neurons operate by writing onto an \textit{unembedding null space}, allowing them to impact the residual stream norm with minimal direct effect on the logits themselves. We observe the presence of entropy neurons across a range of models, up to 7 billion parameters. On the other hand, token frequency neurons, which we discover and describe here for the first time, boost or suppress each token’s logit proportionally to its log frequency, thereby shifting the output distribution towards or away from the unigram distribution. Finally, we present a detailed case study where entropy neurons actively manage confidence: the setting of induction, i.e. detecting and continuing repeated subsequences.

Main paper: 
 introduction: 
 As large language models (LLMs) increasingly permeate high-stakes applications, the lack of transparency in their decision-making processes poses significant vulnerabilities and risks [5] . Understanding the basis of these models' decisions, especially how they regulate confidence in their predictions, is crucial not only for advancing model development but also for ensuring their safe deployment [1, 33] . LLMs have been empirically shown to be fairly well calibrated: on question-answering tasks, token-level probabilities of a model prediction generally match the probability of the model being correct [40, 58] . This raises the question of whether LLMs possess mechanisms for general-purpose calibration to mitigate the risks associated with overconfident predictions.Significant research has been conducted on estimating model uncertainty in neural networks [19, 21] and specifically in LLMs [22] . While many studies focus on quantifying and calibrating model confidence [49, 40, 47, 45, 65, inter alia], there is little research into the internal mechanisms LLMs might use to calibrate their predictions.In this work, we explore two types of components in Transformer-based language models that we believe serve a calibration function: the recently identified entropy neurons and a new class of components that we term token frequency neurons. Entropy neurons have been brought to light by recent work [42, 28] and are characterized by their high weight norm and their low composition with the unembedding matrix despite being in the final layer. The low composition with the unembedding matrix would suggest that they play a minor role in the next-token prediction. ). We identify a set of neurons whose effect depends on LayerNorm (red points; metric described in §3.2), and which affect the model's confidence (quantified as entropy of P model ) with minimal impact on the prediction.However, their high weight norm, despite LLMs being trained with weight decay [48] , indicates that these neurons must be important for performance. This leads us to infer that these neurons may play a calibration role, as hypothesized by Gurnee et al. [28] . We show that entropy neurons work by writing to an effective null space of the unembedding matrix and leverage the layer normalization [4] that is applied to the residual stream. This way, these components effectively modulate the entropy of the model's output distribution in a manner that only minimally affects the actual model prediction (Figure 1 ). We observe the presence of entropy neurons in GPT-2 [62] , Pythia [6] , Phi-2 [38] , Gemma 2B [68] , and LLaMA2 7B [70] , demonstrating for the first time their role across different model families and scales. 2Motivated by the importance of the token frequency distribution (i.e., the unigram distribution) in language modeling [11, 51] , we identify a novel class of neurons that boost or suppress each token's logit proportionally to its frequency. We provide evidence showing that these neurons, which we term token frequency neurons, modulate the distance between the model's output distribution and the unigram distribution, which the model defaults to in settings of high uncertainty.As a case study, we analyze the activation of these neurons in scenarios involving induction (i.e., the repetition of subsequences in the input; 57). Our results show that entropy neurons increase the output distribution's entropy, thus decreasing the model's confidence in its predictions for repeated sequences. This represents a hedging mechanism, aimed at mitigating loss spikes on confidently wrong predictions.Our contributions are as follows: (1) We show that models have dedicated circuitry to calibrate their confidence, with dedicated neurons in the final layer: the previously discovered entropy neurons and novel token frequency neurons. (2) We explore the mechanism by which each neuron family affects confidence ( §3 and §4). Notably, we find that the model learns a low-rank effective null space for the unembedding that entropy neurons write to, and the final LayerNorm scale is used to calibrate the output logits. This striking phenomenon would have been missed by the assumptions made in standard methods such as direct logit attribution [16, 24, 46] . (3) We study how these neurons are used in practice to regulate confidence ( §5). As expected, neurons that lower confidence worsen performance when models are correct and improve it when models are incorrect. We close with a mechanistic case study of how induction heads [57] use entropy neurons to control confidence when detecting and continuing repeated text ( §6). 32 BackgroundIn this section, we provide an overview of the Transformer architecture [71] , focusing on the components relevant to our analysis. Given a vocabulary V, we denote an autoregressive language model as M : X → Y, where Y is the space of probability distributions over V. M takes as input a token sequence x = [x 1 , ..., x m ] ∈ X ⊆ V m , and outputs a probability distribution P model : V → [0, 1] to predict the next token in the sequence. In this work, we focus on decoder-only Transformer-based models, which represent the backbone of the most capable current AI systems [58, 26] . The Transformer architecture consists of two core components: the multi-head self-attention and a multi-layer perceptron (MLP). These two components read information in from and write out to the residual stream (i.e., the per-token hidden state consisting of the sum of all previous component outputs) [16] .Central to this study is the structure of the MLP layers of the transformer. 4 Given a normalized residual stream hidden state x ∈ R d model , the output of an MLP layer isEQUATIONwhere W T out , W in ∈ R d mlp ×d model are learned weight matrices, β in and β out are learned biases. The function σ denotes an element-wise nonlinear activation function, typically a GeLU [32] . In this paper, the term neuron refers to an entry in the MLP hidden state. We denote specific neurons in the model using the format <layer>.<index>. We use n to refer to the activation values of the neurons (i.e., the output of the activation function σ). Furthermore, we use w(i)out ∈ R d model to indicate the output weights of neuron i (i.e., the i-th column of W out ).LayerNorm. Layer normalization (LayerNorm) is a commonly employed technique to improve the stability of the training process in deep neural networks [4] . Given a hidden state x ∈ R d model , LayerNorm applies the transformationLN(x) =x -m(x)EQUATIONIn words, LayerNorm centers a hidden state by subtracting its mean m(x) = 1 d model j x j , re-scales it by its ϵ-adjusted standard deviation ( Var(x) + ϵ, where ϵ ∈ R), and applies an elementwise affine transformation determined by the learned parameters γ, β ∈ R d model .Unembedding. After the final transformer block, the final state of the residual stream is passed through a final LayerNorm and then projected onto the vocabulary space through an unembedding matrix W U ∈ R |V|×d model . The logits obtained by mapping the residual stream final state onto the vocabulary space are then passed through a softmax function to convert them into a probability distribution P model ∈ Y. Finally, the loss is computed using a function L : Y × V → R + (typically cross entropy for autoregressive language models). Following previous work [16] , we employ standard weight pre-processing techniques [53] . In particular, we "fold" LayerNorm trainable parameters into the W in and W U matrices. We report the details about weight pre-processing in Appendix C. 
entropy neurons: 
 Prior research examining individual neurons in GPT-2 language models [62] identified a category of neurons notable for their high norm and limited interaction with the unembedding matrix. Katz and Belinkov [42] speculate that these neurons may act as regularizers, contributing a constant bias to the residual stream. Gurnee et al. [28] independently rediscovered these neurons, termed them entropy neurons, and suggest that they play a role in regulating the entropy of the model's output distribution. They show that interventions increasing the activation value of these neurons affect the entropy of the output distribution to a larger extent compared to other neurons. Despite these insights, the natural activation effects of these neurons on standard language inputs have not been examined, and how they carry out this entropy-modulating function remains unclear. In this section, we explore the mechanism of action of these neurons within the model.Our initial step is to identify which neurons are entropy neurons. We do this by searching for neurons with a high weight norm and a minimal impact on the logits. To detect minimal effect on the logits, we follow the heuristic used by Gurnee et al. [28] , analyzing the variance in the effect of neurons on the logits. In particular, we project the last-layer neuron weights in GPT-2 Small onto the vocabulary space. This projection, sometimes referred to as logit attribution, represents an approximation of the neuron's effect on the final prediction logits [55, 24, 15] . Then, we compute the variance of the normalized projection. That is, given a neuron with output weights w out , we compute where ∥ • ∥ dim=1 indicates column-wise norm.EQUATIONThis measure allows us to quantify how specific is the direct effect of a neuron on the output logits, with the underlying intuition that a diffused contribution (i.e., close to a constant added to all logits before applying the softmax) results in a small effect on the output probabilities. In the last layer of GPT-2 Small, we identify a subset of neurons with particularly low variance but substantial norm (Figure 2a ). Since the optimization of the model's weights preserves these high-norm low-composition neurons (that are expensive in terms of weight decay penalty [48] ), it is likely that these neurons are important and exert their effect through a different mechanism than directly modifying the logits.Given the low composition with the unembedding matrix, we hypothesize that most of the effect of entropy neurons on the model's output is mediated by the re-scaling of the residual stream performed by the final LayerNorm. We test this hypothesis using a causal mediation analysis formulation [60, 64] , illustrated in Figure 2b . We distinguish the total effect that a neuron has on the model's output (represented by the two causal paths stemming from the neuron node in Figure 2b ) from its direct effect (the green arrow in Figure 2b ), which is not mediated by the change in the LayerNorm scale. Our hypothesis posits that the difference between the former and the latter is significantly larger for entropy neurons than for normal neurons.We carry out an ablation experiment where we intervene on the activation value of a specific neuron by fixing it to its mean value across a reference distribution while constraining the LayerNorm scaling coefficient to remain constant. More formally, consider a neuron with index i ∈ {1, 2, . . . , d mlp } and denote by n i ∈ R its activation value. Given an input x ∈ X , denote by x the last hidden state in the model (i.e., the output of the last transformer block), and denote by x -i the last hidden state in the model obtained after mean-ablating neuron i:EQUATIONwhere n i is the mean activation value computed over a subset of X .We quantify the total effect of neuron i upon mean ablation by measuring the absolute variation in the model's loss L,5 specifically:EQUATIONwhere the expectation is taken over a uniformly sampled subset of a corpus X . Similarly, we quantify the direct effect of neuron i by preventing the LayerNorm denominator from varying:EQUATIONFor each neuron, we measure the total and direct effects when its activation value is set to the mean across a dataset of 25600 tokens from the C4 Corpus [63] (additional experimental details are provided in Appendix D). We compare these metrics for six selected entropy neurons against those from 100 randomly selected neurons. The results (Figure 2c ) show that in the randomly selected neurons there is no significant difference between their total and direct effects. On the other hand, the difference between these two quantities is substantial for entropy neurons, and in some cases the direct effect represents only a small fraction of the total effect.These findings represent an interesting and novel example of how language models can use Layer-Norm to indirectly manipulate the logit values. Such a mechanism could easily be overlooked by conventional analyses, which typically focus on direct logit attribution and fail to account for the normalization effects imposed by LayerNorm [16, 24, 46] .The unembedding is a linear map to a significantly higher-dimensional space (e.g., from 768 to 50,257 dimensions in GPT-2). Therefore, it is surprising that the output of a high-norm neuron would have little effect on the logits. We hypothesize that there is a subspace of the residual stream with an unusually low impact on the output that entropy neurons write onto. To investigate this, we compute the singular value decomposition (SVD) of the unembedding matrix W U = UΣV T .Analyzing the singular values in Σ (thick blue line in Figure 2d ), we observe that the bottom values are extremely small. In particular, we notice a sharp drop to values close to 0 around index 755. This observation indicates a remarkable phenomenon within the model: the training procedure optimizes the weights to preserve what effectively functions as a null space within W U . This finding is striking, as it suggests that the model deliberately limits its representational power by projecting a set of residual stream directions onto a small neighborhood of the 0-vector.Next, we study the directions within the residual stream that entropy neurons write to. We do this by computing the cosine similarity between the output weights (w out ) for each neuron and singular vectors V ∈ R d model ×d model . The results, illustrated by the colored lines in Figure 2d , show that all the entropy neurons write almost exclusively to directions within the effective null space. These findings illustrate how entropy neurons leverage this null space to add norm to the residual stream without directly affecting the logit values.The relationship observed between entropy neurons and the unembedding null space suggests that entropy neurons may be identified by analyzing the proportion of their norm that is projected onto such a null space. To test this, given a neuron i, we compute the fraction ρ i of the neuron's norm projected onto the effective null space V T 0 . In particular, we define the effective null space of W U as the space spanned by its bottom k singular vectors, i.e., V T 0 := ⟨V T :,d model -k , . . . , V T :,d model ⟩, and we compute ρ i = ∥V T 0 w(i) out ∥/∥w (i)out ∥. Then, we measure the quantity 1 -DE(i)/TE(i), which represents the fraction of the neuron's effect that is mediated by the final LayerNorm. 6 We compare these two quantities for each neuron at the last layer in GPT-2 Small (Figure 2e ), with k = 12. The results show that neurons with the largest projections onto the null space have the most significant LayerNorm-mediated effects on the model's output. These findings connect with recent work highlighting the importance of the bottom W U singular vectors [10] and represent further evidence supporting our hypothesis about the entropy neurons' mechanism of action.We extend our investigation of entropy neurons to models beyond the GPT-2 family. In Figure 2f , we analyze the proportion of norm projected onto the bottom singular vectors of W U (i.e., ρ) and the influence of LayerNorm on the output for neurons in the last layer of LLaMA2 7B [70] . 7 Similar to our observations with GPT-2 Small, the results indicate the existence of a distinct group of neurons that predominantly write onto the effective null space of W U . These neurons significantly influence the model's output via LayerNorm, confirming the presence of entropy neurons across different model families. As entropy neurons act by re-scaling the residual stream and have no effect that is specific to a subset of the tokens, their activation has a large effect on the entropy of the output distribution but little impact on the relative ranking of the tokens. That is, they affect the model's confidence in the prediction without affecting the actual token being predicted. This is illustrated in Figure 1 . We repeat these analyses on GPT-2 Medium, Pythia 410M and 1B [6] , Phi 2 [38] , and Gemma 2B [68] , obtaining overall consistent results (Appendix F). 
token frequency neurons: 
 The token frequency distribution (i.e., the distribution of unigrams over a corpus) can serve as a reliable baseline for next-token prediction, especially in scenarios of high uncertainty [11, 51] . Thus, we hypothesize that a useful general-purpose mechanism for regulating confidence in language models could involve modulating the distance between the output distribution and the token frequency distribution. We explore this hypothesis using the 410M-parameter Pythia model, for which the training corpus has been publicly released (The Pile; 20). For this corpus, we compute the empirical token frequency distribution P freq . We observe that the entropy of the model's output distribution is in fact negatively correlated with its Kullback-Leibler (KL) divergence from the empirical token frequency distribution: as the confidence of the model decreases (higher entropy), its output distribution tends to be closer to the empirical token frequency distribution (Figure 3a ).As for entropy neurons, we aim to identify neurons whose operational mechanism relies on a particular subspace-in this case, the direction corresponding to the token frequency distribution. We obtain such a direction by computing the logit vector v freq ∈ R |V| by centering the log-probability values for each token in V. That is, v freq,i = log(p freq,i ) -m(log(p freq )), where p freq ∈ R |V| the vector of the token frequency probabilities values, and m : R |V| → R indicates the mean function.To identify the neurons that rely on this direction to modulate the model's output, we conduct an ablation experiment similar to the one described in §3.2. In this experiment, we mean-ablate neurons and assess the absolute change in model loss while preserving the value of the logits along the direction of v freq as constant. Denote by l = W U LN(x) the logits produced by the model on input x ∈ X , and denote by l -i = W U LN(x -i ) the value of the logits obtained on the same input after the ablation of a neuron i (where x -i is defined in Eq. ( 4)). To measure the effect of neuron i that is not mediated by the token frequency direction (i.e., its direct effect), we obtain the adjusted value of the post-ablation logits l -i asEQUATIONThen we compute the total and direct effects of neuron i asEQUATIONIn Figure 3b , we report the results comparing the token frequency-mediated effect against the average absolute change in the KL divergence between P model and P freq for neurons at the final layer of Pythia 410M. We also highlight the entropy neurons with the highest LayerNorm-mediated effect (see Appendix F for comprehensive results on entropy neurons in Pythia 410M). We observe that there are neurons whose effect on P model is substantially mediated by the token frequency direction (i.e., positive value along the x-axis in Figure 3b ). These neurons, upon ablation, lead to significant variation in D KL (P freq ∥P model ) (large positive value along the y-axis in Figure 3b ), comparable to the variations caused by some entropy neurons.These results validate the presence of components (which we term token frequency neurons) that affect the model's output distribution by bringing it closer or away from the token frequency distribution by directly modulating it. Figure 3c focuses on 5 selected token frequency neurons and shows their impact on the loss and its decrease upon the inhibition of the token frequency direction (Eq. ( 7)), which parallels the LayerNorm mediation for entropy neurons discussed in §3. We additionally show the presence of token frequency neurons in GPT-2 Small and Pythia 1B in Appendix G. 
examples of neuron activity: 
 We have shown that confidence-regulation neurons improve model performance by calibrating its output across the entire distribution. However, it remains unclear what this looks like in practice. To better illustrate the role that these neurons play in language models, we provide additional analyses about the changes in the model output induced by particular entropy and token frequency neurons.Entropy neurons. In GPT-2 Small, we examine the change in entropy induced by one of the strongest entropy neurons identified in §3 (11.2378 ). To study this, we conduct an experiment in which we clip the activation value of the neuron to its mean and measure the resulting change in the model output. 8 We analyze the difference in loss caused by the ablation of the neuron on individual tokens and compare it to the initial loss value. This analysis is depicted in Figure 4a , which also illustrates the reciprocal rank of the correct token prediction in the model's output distribution (RR). These observations suggest that the entropy-increasing function of neuron 11.2378 acts as a hedging mechanism: it marginally increases the loss on correct model predictions (low initial loss) but prevents the loss from spiking when the model confidently predicts a wrong token (high initial loss).In Figure 4b , we provide an example of input, taken from the C4 corpus [63] , on which the model confidently predicts a wrong token ("otherapy"), and neuron 11.2378 activates to mitigate the loss spike. We perform the same analysis for the strongest entropy neuron in LLaMA2 7B in Appendix F.1.Token frequency neurons. We next focus on neuron 23.417 in Pythia 410M (identified in §4). This is a token frequency neuron: when its activation is increased, it pushes the model's output P model towards the token frequency distribution P freq . This is illustrated by Figure 4c , showing the KL divergence between the two distributions pre-and post-ablation. As with increased entropy, bringing P model closer to P freq decreases the model's confidence (positive values along the x-axis correlate with negative y values in Figure 4c ). This leads to an increase in loss on correct predictions, reflected in the negative change in loss upon ablation (darker points in Figure 4c ). Interestingly, we find that the projection of this neurons' w out onto the frequency direction v freq is negative, suggesting that this neuron is suppressing common tokens and promoting rare ones. This indicates that the model is on average biased to predict common tokens even more frequently than their frequency would dictate. This way, lowering confidence by pushing P model closer to P freq requires suppressing the token frequency direction, which is the opposite of what one would have naively expected. 
case study: induction: 
 As a more detailed case study to examine the active role and confidence-regulating function of entropy neurons, we focus on the setting of induction. Induction occurs when there is a repeated subsequence of tokens in the model's input, and the model must detect the earlier repeat and continue it. Mechanistically, this is implemented by specialized attention heads, called induction heads [16, 57] , which attend from some token A to the token B that immediately follows A in the early occurrence and predict B (AB...A → B). Repeated text frequently occurs in natural language (e.g., someone's name) and, during a repeated subsequence, the next token can often be predicted with very high confidence.To analyze this phenomenon, we create input sequences by selecting 100 tokens from C4 [63] and duplicating them to form a 200-token input sequence. Across 100 such sequences, we measure GPT-2 Small's performance and observe a significant decrease in both average loss and entropy during the second occurrence of the sequence (solid lines in Figure 5a ). Additionally, we track the activation values of the six entropy neurons in GPT-2 Small analyzed in §3 (dotted lines in Figure 5a ). For four of these neurons (11.584, 11.2378, 11. 2123, and 11.2910), we note substantial change in average activation values, suggesting that they may play an important role. Furthermore, among final-layer neurons, entropy neurons exhibit the largest change in activation during induction (Appendix H.3, Figure 11 ).To further explore the specific effects of entropy neurons on the model's output entropy and loss, we conduct an experiment where, during the second occurrence of the sequence, we clip the activation values of the neurons to their mean values from the first occurrence. The results (Figure 5b ) reveal that while the ablation of some neurons such as 11.584 and 11.2123 leads to a slight increase in output entropy, the most significant impact was observed with neuron 11.2378. This results in up to a 70% reduction in entropy, suggesting its role in boosting entropy to counterbalance the model's high confidence during induction. We additionally study the effect of token frequency neurons during induction and we report the results in Appendix H.2, revealing that these neurons also lead to a significant change in entropy compared to randomly selected neurons. We repeat these analyses in naturally occurring induction cases identified in the C4 corpus, obtaining consistent results (Appendix H.1).In order to verify that induction heads have a causal effect on the activation of our entropy neurons, we perform BOS ablations [31] . That is, motivated by the observation that attention heads frequently attend to BOS when inactive (e.g., when induction is not occurring) [72] , we set the attention pattern of an attention head to always attend to the first token of the sequence (i.e., BOS) and record the neuron's activation. We perform this procedure on the top 3 induction heads, selected according to the prefix matching score introduced by Olsson et al. [57] . In GPT-2 Small, these heads are L5H1, L5H5, and L6H9, respectively. As baselines, we randomly select 6 heads in layers 5 and 6 that, like our induction heads, have mean attention to BOS > 0.6 but have prefix matching score < 0.1. For both entropy neurons and baselines, we ablate heads individually and in combinations of up to 3 heads.Figure 5c shows that ablating L5H1 (ind. head 1) leads to a significant decrease in 11.2378's mean activation. Ablating L5H5 (ind. head 2) or L6H9 (ind. head 3) individually does not make a substantial difference, but ablating them alongside L5H1 further decreases activation, bringing 11.2378 close to its activation on the first 100 tokens. We perform additional BOS ablations for other entropy neurons in Appendix H and further study the induction head-entropy neuron interaction in Appendix I, providing preliminary evidence that entropy neurons respond to an internal 'induction-has-occurred' signal that is produced by induction heads.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: §3, §4, and Appendix D Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
related work: 
 Uncertainty in language models. Uncertainty quantification for machine learning models has been extensively studied to assess the reliability of model predictions [19, 21] . With the increasing adoption of LLMs, accurately determining their confidence has drawn significant attention [22] . Existing research has explored various strategies to calibrate model output, including ensembling model predictions [76, 34] , fine-tuning [39, 40] , and prompting models to explicitly express their confidence [47, 40, 65, 69] . Additionally, efforts have been made to evaluate the truthfulness [9, 3, 50] and uncertainty [35] of model outputs by inspecting internal representations. However, the investigation of internal, general-purpose mechanisms determining model confidence remains underexplored.Interpretability of LLMs' components. The analysis of language models' internal components and mechanisms is an increasingly popular area of research (we refer to Ferrando et al. [18] for an overview). Prior work has explored how to attribute and localize model behavior [73, 23, 52] , and to identify specific algorithms that models implement to perform tasks [54, 75, 29, 67, 12] . Additionally, many studies have focused on investigating individual neurons [41, 61, 14, 2, 13, 74, 27, 7] . However, neurons are not always the correct unit of analysis, as they can be polysemantic [56, 17] . Recent work has employed sparse autoencoders (SAEs) to find more interpretable linear combinations of neurons [79, 8, 36] . Despite this, in the context of entropy modulation, prior findings of relevant neurons motivated us to focus on neurons.Closest related work. An anomalous class of neurons characterized by high norm and low composition with the unembedding matrix was observed by Katz and Belinkov [42] , though this was not linked to the regulation of output distribution entropy. This class of neurons was independently rediscovered by Gurnee et al. [28] while investigating the universality of neurons in GPT-2 models. Our work connects the concept of entropy neurons to the presence of an effective null space represented by the bottom singular vectors of W U . Our findings highlight the significance of this seemingly unimportant subspace within the residual stream and align with the observations of Cancedda [10] , who associated the bottom W U singular vectors with the phenomenon of the attention sink [78] .Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: In §3 and §4 we include box plots, which provide comprehensive information about the distribution of the effects measured. For the line plots in section §6, we indicate the standard error as a shaded area around each line. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) 
conclusion: 
 This paper presents an analysis of the mechanisms by which LLMs manage and express uncertainty, focusing on two specific components: entropy neurons and token frequency neurons. We show that entropy neurons, with their high weight norm and low direct interaction with the unembedding matrix, affect the model's output via the final LayerNorm. We also introduce and investigate token frequency neurons, which adjust the model's output by aligning it closer to or further from the token frequency distribution. In a case study on induction, we demonstrate the practical implications of these neurons. We show that one example role of entropy neurons is acting as a hedging mechanism to manage confidence in repetitive sequence scenarios. Some limitations of our work include focusing only on two types of components in the neuron basis, relying on proxies for confidence, and observing varying effects across models. We thoroughly discuss the limitations of our work in Appendix A. This study represents the first thorough investigation into the mechanisms that LLMs might use for confidence calibration, providing insights that can guide further research in this area. 
 
 Appendix: Our study focuses exclusively on two particular types of final-layer components: entropy neurons and token frequency neurons. However, it is likely that there are multiple other components and mechanisms within language models that contribute to confidence regulation. Exploring these additional components could provide a more comprehensive understanding of how language models manage uncertainty.We also acknowledge the lack of a precise definition of confidence. Instead, we rely on rough proxies such as entropy and the distance from the token frequency distribution. These measures, while useful, may not fully capture the concept of confidence in language models.Although we observed the presence of entropy neurons in most of the models we studied, the fraction of their effect explained by LayerNorm re-scaling varies. For instance, in Pythia and Gemma, this fraction is around 40%, which is significantly less than the 80% observed in GPT-2, LLaMA2 7B, and Phi-2. We believe that variations in model architecture and training hyper-parameters may explain these inconsistencies. Although we do not explore these factors in detail, we discovered that the application of dropout during training leads to the emergence of a larger null space (Appendix E), which we hypothesize may encourage the formation of entropy neurons. Future research should rigorously examine the factors that determine the emergence of entropy neurons, as well as other mechanisms that regulate model confidence.Lastly, our study is primarily focused on how entropy and token frequency neurons contribute to confidence regulation in next-token prediction. An interesting direction for future research would be to investigate confidence-regulating neurons in broader contexts and real-world tasks, such as question-answering or reasoning. This could provide deeper insights into the practical applications and limitations of these neurons in diverse scenarios.This research enhances the understanding of how LLMs regulate confidence in their predictions through entropy neurons and token frequency neurons. While our study primarily advances our technical understanding of language models, some broader impacts should be considered. With the purpose of removing irrelevant components and other parameterization degrees of freedom, we employ a set of standard weight pre-processing techniques [53] . This way, cosine similarities and other weight computations have a mean of 0.Folding in Layer Norm. Most layer norm implementations include trainable parameters γ ∈ R n and β ∈ R n (see Eq. ( 2)). To account for these, we "fold" the layer norm parameters into W in by treating the layer norm parameters as equivalent to a linear layer and then combining the adjacent linear layers. We create effective weights as follows:W eff = W in • diag(γ), β eff = β in + W in • β.Finally, we center the reading weights because the preceding layer norm projects out the all-ones vector. Thus, we center the weights W eff as follows:W ′ eff (i, :) = W eff (i, :) -W eff (i, :).Writing Weight Centering. Every time the model interacts with the residual stream, it applies a LayerNorm first. Therefore, the components of W out and β out that lie along the all-ones direction of the residual stream have no effect on the model's calculations. Consequently, we mean-center W out and β out by subtracting the means of the columns of W out :W ′ out (:, i) = W out (:, i) -W out (:, i).Unembedding Centering. Since softmax is translation invariant, we also center W U :W ′ U (:, i) = W U (:, i) -WU (:, i).We carry out all our experiments using data from the C4 Corpus [63] , which is publicly available through a ODC-BY license. 9 We use the Pile [20] , which is no longer distributed by its creators, 10solely to compute token frequency statistics, as Pythia was trained on it and we required the original training data for the model. The experiments to measure the total and direct effects, both for entropy and token frequency neurons, were carried out on 256-token input sequences. We used 100 sequences for GPT-2 and Pythia 410M, 50 for Phi-2 and GPT-2 Medium, and 30 for LLaMA2 7B and Gemma 2B. The scatter plot in Figure 3a was obtained on 10000 tokens. The results on the induction case study presented in §6.1 were obtained on 500 input sequences, the shaded area around each line represents the standard error. For the sake of visualization clarity, the box plots in Figure 2 and Figure 3 do not include outliers: the whiskers represent the first and third quartiles. For naturally occurring induction, we selected 1000 sequences from C4 that contained a repeated n-gram (n=6) and met our filtering criteria (each n-gram must be unique and contain no duplicate tokens). The BOS and mean attention output ablation experiments were each carried out on one hundred 100-token input sequences. Baselines for BOS ablation consisted of 6 single-head ablations, 3 two-head ablations, and 3 three-head ablations. Mean attention to BOS (used for baseline selection) was calculated using 1000 sequences from C4. To calculate prefix matching scores for attention heads, we use 1024 sequences of duplicated random tokens.Computing resources. The static analyses of the model weights were conducted on a MacBook Pro with 32GB of memory. The experiments carried out to quantify the neurons' effects were carried out on a single 80GB Nvidia A100. The longest experimental run took approximately 12 hours. Our experiments were carried out using PyTorch [59] and the TransformersLens library [53] . We performed our data analysis using NumPy [30] and Pandas [77] . Our figures were made using Plotly [37] . The paper's bibliography was curated using Ryanize-bib [81] . We hypothesize that the presence of a null space in the model's unembedding matrix W U is significantly affected by the application of dropout [66] during training. Dropout represents an incentive for the model to encode information redundantly, aligning it with a specific basis in the residual stream. We expect this redundancy to be reflected in some linear dependencies among the rows of W U , effectively creating a null space. To empirically validate this hypothesis, we compare the singular values of W U between two versions of Pythia 160M [6] -identical except in their application of dropout to the residual stream (Figure 6 ).Our observations reveal that the singular values of the model trained with dropout are generally lower, suggesting lower stability and greater linear dependence among the rows of W U . More notably, the difference in singular values between the two model versions is larger for the ∼10 smallest values, indicating the existence of a subspace in the dropout-trained model that has a substantially reduced impact on the logits. Such findings suggest that entropy neurons are more prevalent in models trained with dropout, as the subspace they leverage within the residual stream is more marked. However, this does not rule out the existence of entropy neurons in models trained without dropout.In Figure 7 , we report the results for the final-layer neurons of GPT-2 Medium, Pythia 410M and 1B, Phi-2, and Gemma 2B, complementing our earlier findings from GPT-2 and LLaMA2 7B discussed in §3. ρ is computed by considering the the bottom 1% W U singular vectors as null space. We observe a consistent pattern: there exists a subset of neurons that predominantly write to the W U null space. However, the extent to which entropy neurons appear in models appears to vary: we observe that entropy neurons in Pythia and Gemma exhibit a weaker effect compared to those in other models. In particular, the fraction of their total effect explained by the LayerNorm scale is around 30-40%, which is notably lower than the approximately 80% observed in GPT-2 Small, LLaMA2 7B, and Phi-2. This suggests that while entropy neurons are a common feature, their strength and the extent of their influence can vary significantly between different models.As for neuron 11.2378 in GPT-2 Small in §5, we analyze the strongest entropy neuron in LLaMA2 7B (31.3228 ). We perform a similar analysis as in §5, where we clip the activation value of the neuron back to its mean when it exceeds it, and we measure the changes in loss and entropy after the ablation (Figure 7a ). The ablation of this neuron leads to a significant decrease in entropy, which results in a loss increase on incorrect predictions and a loss decrease on correct predictions.In Figure 8 , we report the results comparing the token frequency-mediated effect against the average absolute change in the KL divergence between P model and P freq for neurons at the final layer of GPT-2 Small and Pythia 1B. 11 We also highlight the entropy neurons with the highest LayerNorm-mediated effect, represented by the points with the largest y-axis values in Figure 7 .In both models, we observe a set of neurons with substantial token frequency-mediated effect (∼0.6). In Pythia 1B, the ablation of these neurons leads to a significant variation in the KL divergence from P freq , comparable to the variations caused by entropy neurons. On the other hand, in GPT-2 Small, the effect of the entropy neurons on D KL (P freq ||P model ) is significantly larger, suggesting that these neurons might play a more important role on the regulation of the model's output compared to token frequency neurons. Furthermore, in both models, we observe neurons with significant negative 1 -DE freq /TE values, indicating that their effect increases when the impact of the token frequency direction is suppressed. This suggests that the effect through the token frequency direction actually counteracts their effect that bypasses the frequency direction.To verify that these neurons behave similarly outside of our artificial setting, we perform the same experiments on examples of induction in natural language data. We select 1000 sequences from the C4 corpus that contain a repeated n-gram of length n=6. In order to remove redundant and trivial examples, we also filter sequences such that the n-gram is unique and no individual n-grams contain duplicate tokens. We perform the same clipped mean ablation procedure as the synthetic setting, mean-ablating to the neuron's activation value on random text. Figure 9 shows a similar trend to the synthetic setting shown in Figure 5a , though the mean change in activations (and thus change in entropy) is less significant. The most impactful neuron remains 11.2378, the 'hedging' neuron that we believe counter-balances overconfident predictions. Consistent with this hypothesis, when the neuron's effect is removed via (clip) mean ablation, the model's predictions become more confident: entropy decreases by up to 20% of its original value.The other entropy-increasing neuron shown in Figure 9b is 11.2870. Unlike the other 4 entropy neurons, it fires strongly only on the first few tokens of induction and then decreases in activation (Figure 12 ). In both synthetic and natural induction settings, 11.2870 has an entropy-increasing effect (Figure 5b and Figure 9b ), suggesting that it increases uncertainty at the start of induction where the model should be less certain that copying previous tokens will lead to the correct next-token prediction. (As an example, for the phrase 'Mr. Smith ... Mr.', the model should not be overly confident in predicting that the next token will be 'Smith', since a different name could follow instead.) Since we choose a relatively short prefix length for our natural induction setting (n=6), the neuron is active across our entire repeated sequence and increases entropy.To analyze the activity of token frequency neurons in the context of induction, we repeat the analysis carried out for entropy neurons. Using 100 input sequences created by concatenating two repetitions of a 100-token sequence, we study the activation values (averaged across token positions) for six token frequency neurons identified in §4 within the last-layer MLP in Pythia 410M. We observe that, in this setting, four neurons (417, 3412, 2017, and 87) display a substantial increase in activation value (Figure 10a ). Upon clipped mean-ablation, neurons 417, 3412, and 87 lead to a significant increase in entropy compared to other neurons and a baseline of 10 randomly selected neurons (Figure 10b ).Focusing more closely on neuron 417, we find that reversing its state change leads to a somewhat positive delta in loss for tokens with initially low loss, and larger, negative changes in loss for tokens with high initial loss (Figure 10c ). This trend, which is inverse to that observed with entropy neuron 2378 in GPT-2, demonstrates how different mechanisms of entropy modulation can operate at different levels to balance the model's output distribution.Figure 11 shows that entropy neurons exhibit among the largest changes in activation for final-layer neurons in GPT2 Small and LlaMa2 7B, suggesting that they are performing an important function in induction scenarios. When comparing synthetic and natural induction, we note that the set of most-changing entropy neurons is different (but overlapping). This is explained by the fact that most entropy neurons increase in activation over the course of induction (e.g. 11.2378), whereas others, such as 11.2870 only fire strongly at the start of induction. As discussed in Appendix H.1, our natural induction setting highlights start-of-induction neurons, since it consists of short repeated n-grams (n=6). By contrast, our synthetic setting consists of long 100-token repeated sequences and so favors neurons that continuously increase in activation. In this section, we study the interactions between induction heads and entropy neurons in more detail. In Appendix I.1 we use BOS ablations to verify that induction heads have a significant causal effect for 5/6 of our entropy neurons. Next, in Appendix I.2 we provide preliminary evidence that induction heads produce an internal signal that indicates when induction has occurred, and that our entropy neurons respond to this signal. Finally, in Appendix I.3 we check the direct composition between entropy neurons and induction heads, showing that induction heads do not have especially strong direct effect, and identifying a set of components that may mediate the 'induction-has-occurred' signal.We present the full results of BOS ablations ( §6.2) for all six of the entropy neurons under investigation in GPT2 Small (Figure 12 ). Neurons 11.584, 11.2123, and 11.2910 display a similar trend to 11.2378: intervening on the top induction head, L5H1, leads to a substantial decrease in activation, suggesting it has an important causal effect. By comparison, ablating baseline heads in layers 5 and 6 have little impact on activation.We note that for 11.2870, our start-of-induction neuron (Appendix H.1), BOS ablation of L5H1 has a strong causal effect, but that the direction of this effect changes: on the first few tokens of the repeated sequence, the head boosts activation (activation decreases upon BOS ablation) but it suppresses activation across the remainder of the sequence (activation increases upon ablation). Prior work has found that attention head L5H1 is specialized for long-prefix induction (induction over long sequences) [44] . Thus, we believe 11.2870 is a concrete example of where such specialization is useful: the output of L5H1 enables 11.2870 to fire strongly (and increase entropy) only at the start of induction.Because entropy neurons respond strongly to induction and are causally influenced by induction heads, we hypothesize that induction heads produce a generic 'induction has occurred' signal that indicates to later components that the model is in an induction context. To test this hypothesis, in our synthetic induction setting, we perform mean attention output ablations: we replace each induction head's contribution to the residual stream with its mean contribution averaged over the second half of the sequence (i.e. when induction is occurring). 12 In this way, we hope to retain the generic "induction has occurred" signal, while removing token-specific behavior, such as directly boosting the logits of an attended-to token.Figure 13 shows that mean attention ablations are able to produce activations that are close to their original values. For example, for 11.2378, mean attention ablation of the top induction head results in small activation decrease from ∼9 to ∼8. For comparison, recall that BOS ablation of this same head causes mean activation to fall to roughly 4. This suggests that a substantial portion of the entropy neurons' activation is derived from this generic induction signal, rather than token-specific information.We note that though mean attention ablation is sufficient to raise the activations of our entropy neurons close to their original activations on the second half of the sequence, they do not have much impact over the first half of the sequence. (We would expect activation to increase, since we are adding an 'induction is happening' signal.) We speculate that this is because there are other important components that we have not factored into our analysis. For example, the signal from induction heads to entropy neurons is likely mediated by later-layer MLPs/Attention Heads. These could perform an AND operation over our induction heads' signals and duplicate token heads, such that both signals are necessary in order to produce the trigger for our entropy neurons. We leave a more thorough investigation of this potential circuit to future work.In this section, we study whether the output of induction heads directly contributes to the activation of our entropy neurons, or whether their effect is mediated by other components. To do this, we produce a corrupted version of our induction sequences by replacing the first 100 tokens of each sequence with another random 100-token sequence sampled from C4. Then, for both our original and corrupted sequences, we use direct feature attribution (DFA) [43] to calculate the direct contribution of each preceding component in the transformer to our neuron's activation across the last 100 tokens. This is done by taking the dot product of the output of each component (attention heads and MLPs) with the neuron's W in vector, after applying LayerNorm.Figure 14 shows the difference in DFA between the original (induction) sequence and the corrupted (non-induction) sequence, indicating which components were responsible for the change in activation.For entropy neuron 11.2870 induction heads have a strong direct contribution to the neuron's activation. However, this is not the case for our other 5 entropy neurons. In particular, L11H8 as well as MLPs in layers 8-10 have a strong direct effect, suggesting that the 'induction has occurred' signal effect may be mediated by these other components. Further evidence for this is suggested by the observation that the direct effect of induction heads is often opposite to the expected change in activation. That is to say, for entropy neurons that increase in activation upon induction, the direct effect of the induction heads is often negative (i.e. suppressing activation), and becomes even more negative upon induction. Since our BOS ablation experiments show that turning off these heads leads to a decrease in activation, this suggests the presence of intermediate components that convert the negative direct effect signal from induction heads into a positive one.Answer: [Yes] Justification: We uploaded our code through the submission platform and we provide experimental details in Appendix D. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.In the main paper, we report detailed results for GPT-2 Small and LLaMA2 7B. Results for other models are provided in the Appendix.Our code and data are available at https://github.com/bpwu1/confidence-regulation-neurons.In particular, our study focuses on the final-layer MLPs due to their direct path to the final LayerNorm and unembedding, which excludes possible interactions with other transformer layers.For readability, here we omit the final softmax to normalize the logits before computing the loss.We opt for representing the LayerNorm-mediated effect by 1 -DE(i)/TE(i), rather than computing the indirect effect of the LayerNorm scale because a large indirect effect might highlight neurons with high norm and large direct effect, such as a neuron that boosts a specific logit. In our setting, we are interested in neurons that actually have low direct effect, which is not implied by a large indirect effect.LLaMA actually uses RMSNorm[80] which differs from LayerNorm in the absence of re-centering and bias term. However, our experimental procedure is not affected by this difference."Clipping" refers to setting the activation value of the neuron to its mean only when it exceeds it, unlike mean ablation, which sets the activation value to its mean regardless.https://opendatacommons.org/licenses/by/1-0/https://the-eye.eu/public/AI/pile/readme.txtWe do not have access to the GPT-2 training corpus. However, since the model's training data included a substantial amount of internet-scraped text, we use a randomly sampled 500M-token subset of the OpenWebText corpus[25] as a proxy for the original distribution.In practice, this is implemented using attention Z vectors ('attention output pre-linear') rather than attention output. However, the two are equivalent for our purposes since Z vectors are converted to attention outputs by a linear map[53] 


Decision: The work studies two types of neurons in the last layer of the transformer models that are responsible for calibrating their predictions. The paper makes a small solid contribution to the field of interpretability. The study would be of more value if conducted or extended on real-world tasks. At the current form of the result, it is unclear how generalizable are the findings presented in the paper.

Reviews: review: paper_summary: paper_summary: This work studies a small but potentially important subset of neurons in the final layer for a trained transformer model that appear to regulate the confidence of a model (proxied by the variation in entropy of the model's output). Specifically, the paper builds upon entropy neurons found in previous work, extending the analysis further to show that these entropy neurons (which have high norm but low composition with the unembedding matrix) essential modify the null space of the unembedding matrix. This effectively scales the logits without changing the relative ranks of the tokens, thus regulating confidence. The paper continues this exploration by finding "token frequency" neurons, which are neurons that move the model's output distribution towards unigram token frequency distribution. The work claims that this is another form of confidence regulation, essentially "moving" the distribution towards unigram frequencies when the model is not confident about its prediction.

The paper provides the methodology for finding these neurons, as well as performs ablation/intervention experiments on these neurons to measure the difference in "confidence". Experiments are conducted on several pre-trained models, including GPT2, Pythia, LLaMA-2, Gemma and Phi-2. The paper also ends with a specific analysis of "induction"-behavior depicted in recent mechanistic interpretability works, studying how entropy/token-frequency neurons affect the output distribution when sequences are repeated in the output.
strength_weakness: strength_weakness: - While the results presented in the paper are indeed exciting, they raise a lot of questions (see Questions section) that are easy to answer but have been ignored by the paper. For instance, while the paper has run experiments on many models, there is very little discussion of these results. The paper would be much stronger if some additional discussion revolving around model and vocabulary sizes was included.
- The paper mentions a few existing works on confidence regulation and calibration, but does not tie the work done in this paper within the context of this broader field.
questions: questions: - How are the token frequencies computed for the second half of the paper? Are these sub-word frequencies? On what dataset are these computed, and do you expect the training data distribution to have any effect on the results?
- How many entropy neurons usually exist? From the plots, it seems this number may be somewhere in the single digit range? Explicitly stating this will still be useful information for the reader
- Can you hypothesize why Gemma 2B has few to no entropy neurons?
- Relatedly, what is the effect of model size on the number of these entropy/token-frequency neurons? Should one assume that model size has no effect? What about the vocabulary size?
- Token freq is confusing; by changing distribution differently for every token, aren’t we changing the logits necessarily?
- In Figure 4, what is the reciprocal rank exactly? Is a higher value better? 
- One thing that is unclear is how token-frequency neurons are regulating confidence specifically? I understand that they are modulating the output distribution towards unigram frequencies, but this affects both the logit ranks as well as the variance, so the effect on "confidence" specifically is unclear. I would appreciate some clarity on this.

### General comments
- It will be useful to state somewhere that your neuron indexing strategy is `<layer>.<neuron>`, which may be unclear to a new reader.
limitations: limitations: Authors have addressed limitations adequately.
paper_summary: paper_summary: This paper investigates two kinds of neurons by which transformer language models calibrate their predictions. These are (1) “entropy neurons”, which can affect logit values, but do not promote specific tokens, and (2) “token frequency neurons”, which influence a model’s likelihood of outputting bigram word statistics.
strength_weakness: strength_weakness: - It is unclear what the connection is between “token frequency neurons” and “entropy neurons”. Do these two kinds of neurons interact (if so, how?) or are they separate mechanisms by which models calibrate logits? As this is a major part of the paper, it should be made clear.

- The case studies provided are pretty simple settings. Studying the effect of calibration neurons on more real-world tasks like question answering - where there is a small set of potential answers and seeing if and how the same mechanisms apply would help strengthen the generality of the claims made. (This is noted by the authors in the limitations section, and I agree with them).
questions: questions: - The total effect is much larger than the direct effect for both sets of neurons. For the “token frequency neurons”, does this mean they are influencing other components that actually promote bigram statistics? Or what exactly is their effect mediated by? It is unclear to me how the same direction could promote the most common bigrams for every distinct token in the vocabulary.

- Are entropy neurons and token frequency neurons only studied at the last layer of the model? Do they appear elsewhere, and are just strongest at the last layer?

- Does the discovery of these neurons influence how we should think about using direct logic attribution as a way to understand components and their interactions? Lines 70-71 briefly mention this, but could you can briefly expand on what is meant here?
limitations: limitations: The authors do point out a number of limitations to their study, which are valid and could strengthen the paper when/if addressed. Though many may not be in scope.
paper_summary: paper_summary: The paper investigates specific neurons in LLMs  (termed "confidence regulation neurons")  that modulate the uncertainty of the next token prediction by modulating the output distribution. First, entropy neurons modulate the overall entropy of the output distribution by writing to an effective null space of the unembedding matrix, thereby influencing the residual stream norm with a minimal direct effect on the logits themselves. Second, token frequency neurons modulate the predictive distribution proportionally to the token frequency in the training data.
strength_weakness: strength_weakness: - **Novelty**: The authors claim that they provide a "novel example of how language models can use LayerNorm to indirectly manipulate the logit values" and that prior studies "fail to account for the normalization effects imposed by LayerNorm" (lines 160 to 163).
However, the mechanisms of entropy neurons have already been discovered in previous work [1]. For instance, Gurnee et al. (2023) show that entropy neurons "modulate the model’s uncertainty over the next token by using the layer norm to squeeze the logit distribution, in a manner quite similar to manually increasing the temperature when performing inference.". Although prior studies do not differentiate between the total and direct effect of an entropy neuron on the output distribution, the novelty of the analysis is not clear-cut. 

- **Clarity**: Individual neurons are referred to as *layer.position* and simply *position* interchangebly (e.g. 11.2378 and 2378). Since only neurons in the final layer were investigated, referring to the layer is redundant. Also, the authors should consider further simplifying the name of the neurons, as the exact *position* does not add immediate value to the reader.

- **Interpretability**: The interpretation of the results is sometimes unclear. For instance, the authors observe that token frequency neurons suppress common tokens and promote rare ones. They suggest this indicates the model is biased to predict common tokens more frequently than their actual occurrence. However, an alternative interpretation could be considered. The model might also assign high probability to a single rare token while assigning almost no probability to other rare tokens. Promoting all other rare tokens to match their token frequency would bring the output distribution closer to the token frequency distribution, while increasing entropy and lowering confidence. In other words, the model's bias might not be just towards common tokens but also towards certain rare tokens. Such alternative interpretations would also resolve the counterintuitive explanation. The authors should provide more insights to support their interpretations.

---
[1] Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models.
questions: questions: - Why did the authors analyze 6 entropy neurons but only 5 token frequency neurons?
- In Figure 5(a), are entropy, loss, and neuron activations really all on the same scale (single y-axis)?
- Why does the induction case study not consider the novel token frequency neurons but only the known entropy neurons?
limitations: limitations: The authors adequately addressed limitations.
paper_summary: paper_summary: This paper studies how large language models (LLMs) regulate uncertainty in next-token predictions through specific components: entropy neurons and token frequency neurons. Entropy neurons, identified by their high weight norm and minimal direct impact on logits, influence model confidence by operating within an unembedding null space. Token frequency neurons adjust logits based on token frequency, modulating the output distribution toward or away from the unigram distribution. The study includes a detailed examination of these neurons in various models, demonstrating their role in managing prediction confidence, particularly in repeated sequence scenarios.
strength_weakness: strength_weakness: - As the authors say in the "limitations" section, they use entropy and distance from the token frequency distribution as proxies for confidence, which may not fully capture the complexity of confidence in language models.
- The extent to which entropy neurons influence model output via LayerNorm seems to change across models. This variability suggests that model architecture and training parameters could play a important role.
questions: questions: Please, read my questions and suggestion as a starting point for a follow-up paper.
- Could you provide a more precise definition of confidence in the context of LLMs?
- While LayerNorm plays a crucial role in the functioning of entropy neurons, what might be the impact of other normalization techniques (for example, BatchNorm or RMSNorm) on these neurons?
- I'd suggest also to perform an analysis of how specific features of the models and training hyperparameters influence the effectiveness of entropy and token frequency neurons.
limitations: limitations: The authors properly addressed all the possible limitations of the paper.

Comments: title: Thank you for the reply
comment: Thank you for the rebuttal; it addressed the majority of my concerns and questions. My current score reflects my overall assessment of this work pretty well, so I have decided to maintain the score.
Thank you for your response, some of my queries have been answered. I have gone through the other reviewers' comments and will maintain my score.
Before this phase of the discussion period ends, we wanted to check in with the reviewer on whether we have addressed your concerns with our work?
Thank you for your detailed responses, I appreciate the clarifications provided, particularly regarding the definition of confidence in LLMs and the impact of different normalization techniques on entropy neurons.
We sincerely appreciate your positive feedback. Thank you!
Thank you for the detailed reply. After reading the authors' rebuttals and other reviews, all of my questions/concerns have now been addressed and I think the paper would be a solid contribution to the conference. As such, I have raised my score to a 7.
We believe the key parts of the review were addressed in the main rebuttal, but we include this comment for completeness.

> **Q1:** How are the token frequencies computed for the second half of the paper? 

The empirical token frequency distribution that we compute is the unigram distribution: the distribution of tokens (i.e., entries in the vocabulary) over the whole training corpus. More specifically, the frequency of token $t \in \mathcal{V}$ over corpus $\mathcal{C}$ is computed as  \# of occurrences of $t$ in $\mathcal{C}$ / $|\mathcal{C}|$. This distribution depends on the training data, and ideally, it should be computed for a specific model using the exact training corpus that the model was trained on. We could achieve this for the Pythia models as we had access to the data statistics of The Pile. For GPT-2, since the exact training data is not available, we used a randomly sampled 500M-token subset of the OpenWebText corpus as a proxy for the original distribution (as mentioned in Appendix G). Different training data distributions might lead to different token frequency distributions. However, we expect a model to align its output distribution with the token frequency distribution it was exposed to during training.

> **Q2:** How many entropy neurons usually exist?

The extent to which a neuron is considered entropy-regulating depends on the fraction of its effect that is mediated by the LayerNorm. This quantity varies on a continuous scale, and defining a specific number of entropy neurons would require setting a hard threshold on this measure, which is somewhat arbitrary and difficult to determine precisely. However, in our analyses, we observed that the number of neurons for which the LayerNorm-mediated effect is substantial typically accounts for around 0.1-0.3% of the MLP dimensionality.

> **Q6:** In Figure 4, what is the reciprocal rank exactly? Is a higher value better?

The reciprocal rank is computed as $ \frac{1}{r}$, where $r \in \{1, \dots, |\mathcal{V}| \}$ is the position of the correct next token in the list of all vocabulary tokens sorted by their predicted probability mass in descending order. For example, if the correct next token is ranked 1st in the predicted probability distribution $P_{\text{model}}$ (i.e., the next token prediction is correct), the reciprocal rank is 1. Lower values indicate that the correct token is ranked lower in the predicted probability distribution (i.e., that the prediction is worse).

> It will be useful to state somewhere that your neuron indexing strategy is <layer>.<neuron>, which may be unclear to a new reader.

Thank you for pointing this out. We will add a sentence to clarify the notation.

---

[1] Wang, X., et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.    
[2] Hou, B.,et al., Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling. ICML 2024.    
[3] Jiang, Z., et al.,. How can we know what language models know?. TACL     
[4] Kadavath, S., et al.,  Language models (mostly) know what they know. arXiv.    
[5] Lin, S., et al., Teaching models to express their uncertainty in words. arXiv 2022.    
[6] Si, C., et al., Prompting GPT-3 To Be Reliable. ICLR 2023.    
[7] Tian, K., et al., Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. EMNLP 2023.    
[8] Burns, C., et al,. Discovering latent knowledge in language models without supervision. ICLR 2023.    
[9] Azaria, A. and Mitchell, T., The internal state of an LLM knows when it's lying. arXiv 2023.