Identifying and Evaluating Inactive Heads in Pretrained LLMs
Authors: Pedro Sandoval-Segura
Xijun Wang
Ashwinee Panda
Micah Goldblum
Ronen Basri
Tom Goldstein
David Jacobs
Venue: ICLR 2026 Poster (2026)
Keywords: dormant attention
multi-head attention
attention heads
attention sinks
Primary area: interpretability and explainable AI
TL;DR: Although attention sinks and sink tokens are synonymous with inactive heads in transformer LLMs, we demonstrate that looking beyond attention weights can identify a different and larger set of model-agnostic inactive heads.

Abstract: Attention is foundational to large language models (LLMs), enabling different heads to have diverse focus on relevant input tokens. However, learned behaviors like attention sinks, where the first token receives the most attention despite limited semantic importance, suggest some heads may be inactive, and point to a significant source of computational redundancy. To analyze this phenomenon, we evaluate 12 score functions that measure different ways a head can be inactive. Thresholding these scores allows us to analyze different sets of potentially inactive attention heads. We evaluate whether identified heads are inactive through model interventions, finding that more than 12% of attention heads are inactive on average, and can be ablated in specific contexts while maintaining MMLU accuracy to within 1% of the pretrained LLM. Across 3 model families, our score functions that measure the average norm of a head's output consistently identify inactive heads that would not have been found by score functions that rely solely on attention weights. We establish that relying on a score function that measures a first token attention sink would underestimate the prevalence of inactive heads, failing to identify more than 7\% of inactive heads on average. We also show how measuring score distributions can provide insights into attention behavior. For instance, we find evidence that finetuning causes little to no change in attention behavior, and that even within the same model family, large model scales present different attention behaviors.

Published as a conference paper at ICLR 2026
IDENTIFYING AND EVALUATING INACTIVE HEADS IN PRETRAINED LLMS
Pedro Sandoval-Segural Xijun Wang'
Ashwinee Panda
Micah Goldblum²
Ronen Basri Tom Goldstein David Jacobs
University of Maryland
2Columbia University
#Wezmann Institute of Science
ABSTRACT
Attention is foundational to large language models (LLMs), enabling different heads to have diverse focus on relevant input tokens. However, learned behav- iors like attention sinks, where the first token receives the most attention despite limited semantic importance, suggest some heads may be inactive, and point to a significant source of computational redundancy. To analyze this phenomenon, we evaluate 12 score functions that measure different ways a head can be inactive. Thresholding these scores allows to analyze different sets of potentially inac- tive attention heads. We evaluate whether identified heads are inactive through model interventions, finding that more than 12% of attention heads are inactive on average, and can be ablated in specific contexts while maintaining MMLU ac- curacy to within 1 1% of the pretrained LLM. Across model families, our score functions that measure the average norm of a head's output consistently identify inactive heads that would not have been found by score functions that rely solely on attention weights. We establish that relying on a score function that measures a first token attention sink would underestimate the prevalence of inactive heads, failing to identify more than 7% of inactive heads on average. We also show how measuring score distributions can provide insights into attention behavior. For instance, we find evidence that finetuning causes little to no change in attention behavior, and that even within the same model family, large model scales present different attention behaviors.
INTRODUCTION
Attention is a key component of the transformer architecture, which has led to breakthroughs in language modeling (Radford et al. 2019 Touvron et al. [0033] Dubey al. 2024 OLMo et al., 2024| Yang et al. 2024). The attention mechanism allows tokens to incorporate information from relevant tokens, with multiple heads of attention capturing different types of relevance (Vaswani 2017). But several works have found that attention can become "dormant" and concentrate on initial tokens, which are semantically irrelevant (Yu al. 20244) Chen al. 2025). The question we seek to answer is: How prevalent are inactive attention heads?
Depending how one defines the word inactive, different answers possible. Past work has focused exclusively on the attention weights (Guo et al.] (024a), labeling heads as "dormant" using a maeshold-basel score function. This is motivated by the idea that if head attends primarily to the first token, and the first token has a near-zero value state, then the head output will be near-zero. However, this reasoning ignores other possibilities. Because an attention head's output is a convex combination of value vectors, head could attend to multiple tokens with near-zero value states to produce a near-zero output. In this work, we explore multiple ways an attention head can be considered inactive by evaluating a range of score functions and determine the most robust method for identifying inactive heads.
While classifying heads using score functions offers insights into learned attention behaviors, simply identifying potentially inactive heads not enough. We must also evaluate which set of identified heads best preserves model performance when such heads are zeroed out. Without zeroing out atten- tion head outputs to erase their contribution to the hidden state, it is impossible to verify identified
Code available http://///gggtuu.b.cc////aaaapap segura/ naatt ve heada

---

Published as a conference paper at ICLR 2026
Attention Weights
Weights
Avg Weight First Token
Avg Entropy of Query Distributions
First Token Value Vector Norm
(Gu al., 2023)
Value Vector Norm
Outpud NorntNoor Norm
Output
Avg Value Vector Norm
Last Token Head Output Norm
Avg Head Output Norm
Vector Norm
Figure 1: Score Functions for Inactive Heads. Our simple score functions measure all three com- ponents of attention: attention weights, value vectors, and head output vectors. In each cell, we give an example of the type of attention head that would identified by each score function once scores thresholded. For attention weights, the colorbar displays weights. For vectors, color represents direction, and length of the blue bar represents magnitude. For example, the Avg Weight of First Token score (Gu al.) 2025 Guo al., 2024a) is calculated by computing average weight to the first token, and heads that exhibit high attention to the first token are identified if their their score exceeds threshold. Value vectors and head outputs do not play role, which illustrate with fade. Including normalized versions of these score functions, there are 12 in total.
heads are actually inactive. To address this, we perform model interventions while evaluating on the MMLU benchmark, and measure the effect of removing identified heads on model accuracy. Using 14 models across 3 model families, this experimental setup also allows us to analyze how atten- tion head behaviors change as models scale and as models are finetuned. We make the following contributions:
We evaluate 12 score functions that measure distinct properties of attention heads, and we threshold scores to classify attention heads, based on different definitions of inactive. We evaluate whether identified heads are truly inactive through model interventions which show that, on average, more than 12% of attention heads are inactive pretrained LLMs we consider. Using prior characterizations of "dormant heads" would underestimate inactive heads, and fail to identify 7% of heads that are inactive.
We find that, across model families and across model scale, inactive heads consistently identified through the same score functions. In particular, measuring the average head out- put norm most indicative, By finding more model-agnostic score function, we provide evidence that inactive heads across transformer L.LMs should be understood through head outputs rather than attention weights.
We demonstrate that analyzing score distributions provides useful insights into attention: they reveal that finetuning induces minimal changes in attention behavior, and that scal- ing has little effect until models reach large sizes. Together, this suggests attention head behavior is more invariant to common training modifications than one might expect.
While our work has the potential to be used for efficient inference, our focus is strictly on under- standing inactive attention heads. Specifically, how can identify them and evaluate whether they are truly inactive?
BACKGROUND AND RELATED WORK
The idea that transformers may not be effectively utilizing all attention heads first came up in the context of machine translation. The work of Voita al. (019) Michel et (2019) demonstrated
2

---

Published as a conference paper at ICLR 2026
Avg Predision
Arg Weight Pirst Token
1.0
Aug Weight First Toienn (LN)
Aog Entropy of Quecy Destributios
0.8
0.0
Entropy of Distributions (LN)
First Thken Value Vector Norm
12
0.6
0.
First Token Value Vectio Norm (LN)
Aog Value Vector Norm
Avg Value Vestic Norm (LN)
0.4
0.4
Last Token Hoad Octput Norm
Last Token Head Output Norm (LN)
0.2
0.2
Jog Heed Output Norm
Head Output Norm (LN)
0.0
ARPY LAA
HRPT
abgo
Cort (LN)
SrAS (LR)
LETION
EEhON (LA)
ARRO
ANOON (LA)
AECO PTTA
(1:0)
(44)) SATTI
(LM) Hoon
MOHLET
Score functions identify different sets of heads. Identifying different sets of heads ensures we capture broad range of head characteristics, rather than focusing solely on the attention sink pattern of Avg Weight of First Token (Gu et al.] 2025). Using 100 FeneWeb-Edu training samples, we measure IoU of classifications between each score function on Llama-3. 1-8BB We also measure Precision of one score function's s classifications relative to another's, s, using the column score function as ground truth. Column score functions are abbreviated using the first letter of each word, and are in the same order as rows. For each score function, we dynamically choose thresholds such that ~10% of heads are identified as potentially inactive. Even head scores normalized by scores of other heads in the layer, denoted by "(LN)", do not show significant IoU or Precision with their unnormalized counterpart.
Figure 2:
that attention heads transformer models could be removed with minor performance degradations. Methods for determining which heads can be removed have involved optimization of different kinds of objectives based on: stochastic gates (Voita et al. [2019), importance scores (Michel et al.) [2019), iterative pruning (Behnke & Hcafield 2020). and subset pruning (Li et al., (021). In this work, however, we do not use static pruning, where a head is permanently removed. Rather, our model intervention study (Section 443) is form of dynamic pruning. where different heads are dropped at every forward pass. Our objective is to measure how much head computation is wasted in each forward pass by inactive heads that can be zeroed out, We assume inactive heads are those that can be zeroed out, prior to concatenation and output projection of the multi-head attention mechanism.
Multi-head attention. The multi-head self-attention mechanism (Vaswani 2017 allows multiple attention heads to operate in parallel, each focusing different aspects of the imput sequence. For an input of N tokens with dimensionality dmm learned linear projections transform queries, keys, and values (Q, K, V E R xd ) into lower-dimensional representations: dą dd-dimensional queries/keys and dy dimeennnia values, specific to each of the h heads. The self-attention operation within each head computes attention weights, defined as A- softmax( QK", ). where A E [0, 1) NNN and each Vdzy
row sums to 1, which are then applied to the value vectors. Another way to view the output of the 7th attention head head' E Nxd. is that, for every position in the sequence, we compute a convex combination of value vectors, weighted by the corresponding row of A. The outputs of all heads are concatenated and passed through a final linear projection to produce the module's 's output. The attention weights, A, are one of the few features LLMs work. For a token sequence, high attention weights (close to 1) reveal which tokens are considered most relevant. Thus, it is surprising that attention concentrate initial tokens, called "attention sinks", as shown in App. Figure 19
visualize to get a sense of how pretrained
Attention sinks. An attention sink or sink token is a token that, despite limited semantic impor- tance. disproportionally receives high attention weight from other tokens in sequence. They tend to occur either the first position of the input sequence (Xiao et al. 2024 Guo et al. 2024a), at certain
word tokens (e.g.. "and" and "of"), or delimiter tokens (Sun et al., 2004 Yu et al. 2024).. The first
token can be an attention sink even when it is not a BOS token (Xiao et al. 2024 Gu et al. 2025).
To understand the prevalence of attention heads with sinks, Gu et al. (0255) propose to measure the
average weight assigned to the first token and check if it exceeds threshold 7: N Ei= Ago T.
3

---

Published as a conference paper at ICLR 2026
Table 1: Equations of Score Functions. We define how each score function used to identify inactive heads with a threshold-base based rule. Sample attention heads are in Appendix A.3 We sweep a range of thresholds 7 for each function.
Figure
Score Function
Definition
Avg Weight of First Token (AWFT)
# E.
Aro >T
Avg Entropy of Query Distributions (AEQD)
Ei= Ent(Ar,) <T
First Token Value Vector Norm (FTVVN)
Voo <T
Avg Value Vector Norm (AVVN)
Ei= VV <T
Last Token Head Output Norm (LTHON)
Iheadyy-1. <T
Avg Head Output Norm (AHON)
N Ei= |
<T
In our work, we refer to this metric as "Avg Weight of First Token" and use it as starting point, but ultimately find that different threshold-based score functions are needed for identifying inactive heads in model families outside of Ilama (Dubey et al. 2024) and GPT-2 (Radford et al.) 2019).
As can be seen in App. Figure 19 attention sinks also exhibit value-state drains (Guo et al., 2024b Gu et al. 2025 Kobayashi et al. 2020). where the norm of the value state is near zero. Note that the sink token's s value state is zero but the attention weight to the sink token one, the output of this head (prior to the output projection) is zero, leading Guo et al. (2024a) to call this a dormant attention head.
2024a propose the idea that attention heads can be either active or dormant. They perform model intervention study on 3 attention heads of Llama-2-7B Base, where specific attention head output is zeroed out, and the difference in loss as result of the intervention measured. They find that the difference in loss can depend on whether input text from Wikipedia or GitHub; head that is dormant on Wikipedia samples does not change the loss when the head output is zeroed. our work, we go beyond analyzing one attention head at a time and propose to zero out all "dormant heads" while evaluating models on real-world benchmark task. If "dormant heads" are truly inactive, then zeroing them out should have little effect on model performance. We also propose
Dormant attention. Guo et al.
score functions that measure different definitions of inactive. Unlike Guo et al. 2024a) and Gu et al. (025) who measure only attention weights, our score functions also consider value vectors and head outputs. By considering the range of ways head can be inactive, present a more accurate picture of inactive attention heads in pretrained models. A better understanding of inactive attention heads has the potential to be used for efficiency (Li al. 2021), KV cache reduction (Liu et al. 2023), and compression (Ge et al. 2024)).
EVALUATING SCORE FUNCTIONS FOR INACTIVE HEADS
We outline a number of simple score functions, each capturing different ways a head could be considered inactive. Each score function assigns score every attention head in transformer LLM. By thresholding the scores, we classify and study different sets heads. We summarize each score function in Table We use the f-rrm wherever applicable.
Attention patterns. An attention head could appear inactive due to the patterns in its attention weights. Both overly focused and overly diffuse attention can be considered "inactive," depending on the input. Prior work has recognized the case where attention can concentrate on a single token (Gu et al. 2025 Guo et al. 2024a), where attention is distributed too little, and quantified it by calculating the Average Weight of the First Token (AWFT). However, this does not capture the case where there multiple sink tokens (See App. Figure 19.. To measure when attention concentrates on a few tokens, we choose to measure the Entropy of each Query's 's Distribution over keys, averaged over all queries. Calculating this amounts measuring entropy of each row of the attention weights,
4

---

Published as a conference paper at ICLR 2026
then taking average. If average entropy of the head is low, it implies attention concentrates few tokens. If a head's average entropy is under a threshold т, we can classify it as potentially in classifying heads that exhibit this pattem.
inactive. The smaller we make т, the more strict Changing the direction of the inequality in Avg Entropy of Query Distributions could capture cases where attention is too uniform (i.e., much attention all keys), but chose not to consider this case because heads in early layers tend to exhibit this pattern, and early layers tend to be important (Sun al., 2025).
Value vectors. An attention head could appear active with its attention patterns, but could be inactive if all the value vectors are near-zero. To measure this, we compute Avg Value Vector Norm: the average 22-nrrm of value vectors in the head. It is also possible that while attention patterns may not primarily concentrate on the first token, some tokens could still use the first token as a "value- state drain" (Guo et al., 2024a). To capture heads that do this, we compute the First Token Value Vector Norm, i.e., the morm of the value vector corresponding to the first position. As with other score functions, consider normalizing by the average score of other heads in the layer.
Head outputs. Finally, an attention head could be inactive if its output is small. This key assump- tion is most closely captured by the score functions that measure the head output: when head produces small outputs, its contribution to the residual stream will be minimal, and these heads can be zeroed with little to no accuracy drop. Because multiple choice (MC) benchmark evaluations re- trieve the probability assigned by the model to the correct answer from the probability distribution of the last position, we also propose to measure the Last Token Head Output Norm. To capture cases where most positions have head outputs that are small, we measure the Avg Head Ouiput Norm. More information on why we do not use the circuit-based definition of a head output can be found in Appendix/A.1 A.1
Normalization. We classify heads using the score functions Table [ but we also consider nor- malization by the score of other heads in the same layer. Abusing terminology, we refer to this as layer normalization, or "(LN)" throughout. We do this because raw scores can vary dramati- cally across layers and models. For example, in Llama-3.2-3B. be largest for a head in layer index O, but smallest in layer index 27. Using a relative score al- lows us make meaningful comparisons across layers and models. As example, the Avg Head Output Norm (AHON) score function for the 7th attention head a layer is computed as follows: AvgNorm(heat hhead'), where AvgNorm(T) = 1 Σ |TTi,lla computes the average fg-norm of the rows of input matrix T' and where head' E xxde is the head output. Then, the normalized ver- sion, Avg Head Output Norm (LN), is simply normalized by the average AHON score of heads in the same layer:
value vector norm of 1.0 may
N
AvgNorm(head"
Naxyrr AvgNorm((ead')
Nuyez
==
Thresholding the score of Equation I whenever it is < r controls how strictly we classify a head as inactive. For example, when = 0.1, heads with output norms less than 10% of the layer average are considered inactive. Code implementations are provided in Appendix(A.I) A.12
4 EXPERIMENTS
First, we study which score function is best identifying inactive heads. Then, we use head scores to study how attention behaviors change as models scale and are finetuned.
4.1 SETUP
We download 14 pretrained models using Hugging Face transformers (Wolf et al. 2020
Llama-3. 3-B. Llama-3. -.8---Istttt (Dubey al. 2024), Llama-3.2-3B, Llama-3.2-3D Instrrcct Meta 2024), 1-M0-2-1124-7B OLMo-2-1124
OLM.--2-11244---PDPO OLMo- 1224 Instructt (OLMo et al. 2024). Qwen2.5-0.5B, Owen2.5-1.5B. Qwen2.5-3B, Qwen2.5-7B, 2024). We refer to models from the same We evaluate models on MMLU (Hendrycks)
Dwwe2.55-BBIInstruee and Qwen2.5-14B (Yang et al.
organization as belonging to the same "model family."
5

---

Published as a conference paper at ICLR 2026
Rank of Score Functions
Lamm-3.2-3B
11
12
10
1aama-32-3B--mssrric
10
11
Llama-3.1-88-
10
13
13
12
1.ama-3. 1-88-Instruct
11
12
5
10
10
OLMo-2-1124-7B
11
12
01Mo-2-1124-7B-SFT
10
12
11
88
0LMo-2-1124-7B-DDB)
10
12
11
OLMo-2-1124-7B-nsttuid
10
14
11
Qwen2.5-0.58-
10
12
11
-6
(wee22.51.355
10
13
5
Qwen2.5-3B
12
Qwen2,5-7B
12
11
Qwen2
10
5w112,5-14B
Avg Weight of First Token
Aug Weight of First Tokon (LN))
veg Entropy Quaryy Distributions
Ang Entropy of Query Distribations (LN)
Raudom
First Tokon Valuo Vector Norm
First Taken Value Vactor Norm (LN)
Avg Value Vectar Norm
Avg Vilne Vector Norm (LN)
Last Token Head Output Norm
Last Token Head Output Norm (LN)
Avg Head Outpur Norm
Avg Head Output Norm (LN)
Figure 3: Measuring Avg Head Output Norm is best at identifying inactive heads for most models. For every model, we rank the 12 score functions by normalized AUC, which captures their ability to select inactive heads. Avg Head Output Norm (LN) ranks 1st for out of 14 models, and ranks in top-3 for 13 out of 14 models. Top score functions are consistent across model families.
et al. 2021) (5-shot) in our model intervention experiments. For datasets that do not use an LLM- as-a-Judge, MMLU is the multiple-choice (MC) benchmark with the highest Spearman correla- tion with ChatBot Arena (Li et al. [023] Chiang et al. 2024), and thus makes it the most ap- propriate choice for aligning with real-world chatbot preferences while keeping evaluation consis- PIQA (0-shot) (Bisk et al., 2020) and WinoGrande (5- shot) (Sakaguchi et al. (022)) can be found in Appendix A.4 All evaluations are performed using (Gao et al. (002)). We also use FineWeb-Edd (Lozhkov et al., (024), which is a pretraining dataset of educational webpages. We randomly truncate FineWeb-EEa train- ing sequences to between 10 and 3000 tokens exhibit variety of sequence lengths. Additional model and dataset details are in Appendix A.13
tent and tractable. Additional results
1m-eva www.aalaanoonaaaaa...aa
Metries and Thresholds. In transformer with Nhedds attention heads per Niayers layers, and a dataset D of token sequences, forward pass on input sequence x E 'D will produce attention weights, value states, and head outputs for every head. We measure these components, and assign score to every head, arranged in matrix S IR & Thresholding S results in boolean matrix B of the same size. our model intervention experiments, different boolean matrix is constructed for every new forward pass and "% of Model Heads Zeroed" refers to the percent of True values in B, averaged over all token sequences in D. The proportion of heads each score function identifies inactive can be controlled by the threshold т, which vary for each function. The thresholds for each score function are chosen by measuring head scores across MMLU inputs, constructing the CDF of scores, and calculating the p-th quantile" for p [0, 5, 10, 15, 20, 25, 30)- This way, we can estimate the proportion of heads that will be selected for each threshold, allowing us focus on zeroing out at most 30% of heads. All score distributions in App. Figure 16
4.2 SCORE FUNCTIONS IDENTIFY DIFFERENT SETS OF HEADS
To ensure that each score function measures a different aspect of attention heads, we measure the agreement between predictions ppost-threshlldigg using loU. We also measure precision, taking one of the score functions as ground truth. We use Llama-3.1-88 and 100 FineWeb-Edu sequences
In the case of Avg Weight of the First Token, calculate the (1 p)-th quantile because the threshold incquality reversed.

---

Published as a conference paper at ICLR 2026
Iama--3. 3.1-an...11---mmm Insttuctttt
0lMo-2-1124-97 Ins-ntrutt
Qwen2.5-3B
65
60-
53-
NMLU Acqurecy
MMJI Accuracy
30
MMLU Accuresy
53
40
Easeline Accuracy
Daseltnn Accurace
Random
Suseline Accuracy
Bandom
Bood Norm (LS)
Randon
Aug
Dutput 0.50
Aug Numn
Jog Weggtt of First
Tokas Vector Norn (LN)
Ang Weight Flrst Takes
Taken Hind Outpput Norm (120
Ang Wenght of Pirst Toker
Auy Head Outputt Norm (LN)
20
25
20
25
10
20
25
Model Zerued
Model Heads
Modil Heads
Figure 4: Inactive heads can be identified and zeroed with minor performance degradation. For each model, we plot the top-3 score functions. Average Weight of First Token is not in the top-3 for
but we include it for comparison to prior work. The gray dotted line represents baseline accuracy of the pretrained model. The black line represents zeroing out heads uniformly at random. The Avg Head Output Norm (LN) score function is best at identifying the most heads, that when zeroed, maintain accuracy. Complete results for all models and score functions can be found in App. Figure 10
to collect scores, and we threshold them dynamically to make predictions. In Figure we see that the max IoU 0.58, while the max Precision is 0.73. This suggests no score function is effectively the same another. High agreement tends to occur between unnormalized and normalized pairs of score functions (i.e., AWFT and AWFT (LN)). but not always. For example, Last Token Head Output Norm has IoU of 0.58 with Average Head Output Norm. In similar vein, First Token Value Vector Norm has an loU of 0.47 with Average Value Vector Norm. This effect may occur because the presence of small-magnitude vectors drives down the overall average magnitude. The score function that best predicts the same heads as AWFT is AWFT (LN) with a precision of 0.61. Notably, normalizing by the average score of other heads in the layer significantly reorders heads. All other score functions have Precision under 0.19 with AWFT. Because each score function iden- tifies a distinct subset of heads, we can verify their predictions knowing that each captures different underlying properties of attention heads.
4.3 VERIFYING INACTIVE HEADS THROUGH MODEL INTERVENTIONS
To verify inactive attention heads, we conduct model interventions by zeroing out the outputs of identified heads from each score function, and measuring the resulting impact on model accuracy. For every model, we also compare random baseline where attention heads are zeroed uniformly at random. An effective inactive head score function should identify a substantial fraction of heads that can be zeroed out without compromising the performance of the pretrained model. What score function is best at identifying inactive attention heads?
In Figure 4 we measure how MMLU accuracy changes as we zero out heads selected by different score functions. For brevity, we present one model from each model family: Llama-3. 1.-aaa....-BB--- I-8t-Inntruct, OLMo--2-1124-7B-nnsttno
and Qwen2.5-3B. To do this, we vary the threshold of each score func- tion such that identified heads are at most 30% of the model's 's heads. We plot the top-3 score functions for every model in Figure 4 but complete results for all models and score functions can be found in App. Figure TO In Table we quantify the percent of heads that can be zeroed while keeping accuracy to within 1% of baseline. We find that, for out of 14 models, the Avg Head Output Norm (LN) score function is best at identifying the most attention heads that when zeroed, maintain accuracy to within 1% of baseline. For 13 out of 14 models, AHON (LN) ranks in the top-3 score functions. Avg Weight of First Token underestimates the number of inactive heads.
The fact that a single, simple threshold-basee function like Avg Head Output (LN) can identify the most inactive heads for multiple models. suggests that looking at attention weights is misleading signal. While heads with attention sink patterns seem to occur in all models we consider (See App Figure 19), our results suggest these patterns are not solely indicative of inactivity. Moreover,
7

---

Published as a conference paper at ICLR 2026
Table 2: On average, more than 12% of attention heads can be zeroed while keeping average accuracy within 1% of the original model. For every model, we calculate the highest percentage of heads that can be zeroed using Average Weight of First Token (AWFT), and compare to our score functions (i.e., all score functions excluding AWFT). Higher is better. The overall best score function also shown. Our score functions only improve or are within less than 0.3% of AWFT. We are able to identify and verify more inactive heads than previously possible with AWFT. Specifically, using AWFT would classify less than 5% of heads as inactive, which misses 7% of heads, on average.
Model
% of Heads Zeroed
Best Score Function
AWFT
Ours
Llama-3.2-3B
8.34
8.05 (-0.29)
Avg Weight of First Token
1amma-,,,2B-Insttteee
6.87
13.04 (+6.17)
Avg Head Output Norm (LN)
Llama-3.1-8B
8.56
17.11 (+8.55)
Avg Head Output Norm (LNN)
Llama-3.1-8B-Inttruuc
1.01
10.97 (+9.95)
Avg Head Output Norm (LN)
OLMo-2-1124-7B
0.42
(47.93)
Avg Head Output Norm (LN)
OLMo-2-I124-7BSFFT
1.70
18.95 (+17.25)
Avg Head Output Norm (LN)
0LMo-2-112447B-DPOO
2.14
20.60 (+18.46)
Avg Head Output Norm (LN)
1111-22--112--7-Innstte
1.46
19.54 (+18.07)
Avg Head Output Norm (LN)
Qwen2.5-0.5B
7.43
14.42 (+6.99)
Last Token Head Output Norm (LN)
Qwen2.5-1.55
7.55
7.49 (-0.07)
Avg Weight of First Token
Qwen2.5-3B
5.67
8.78 (43.11)
Avg Head Output Norm
Qwen2.5-7B
1.25
7.54 (+6.29)
Avg Head Output Norm (LN)
Qwen2.5-7B ---nl.5----Instrrc
1.13
5,76 (44.63)
Avg Value Vector Norm (LN)
Qwen2.5-14B
11.04
9.88 (-1.17)
Avg Weight of First Token
Average
4.61
12.18 (+7.56)
our results confirm that while attention head patterns may appear "dormant" (Guo al. 2024a). their outputs are not. Removing attention heads using Avg Weight of First Token as the score is particularly ineffective for OLMo-2 models. While is appealing look only at attention patterns as consistent feature across models of varying organizations and architectures, it appears that small head outputs are consistently more indicative of inactivity.
Despite looking for negligible head outputs, the complexity and non-linearity of LLMs make it difficult to know whether heads with "small" outputs can be zeroed while maintaining accuracy. To the best of our knowledge, we are the first to systematically evaluate the degree of head inactivity and investigate whether these heads are unnecessary. Takeaway: Inactive attention heads are best identified by measuring Avg Head Output Norm (LN), Measuring attention weight patterns like Avg Weight of First Token are not model-agnostic.
Ranking Score Functions. Each of 12 score functions a different set of 7 thresholds. Aceu- racy curves that terminate at different x-axis values make it difficult to assess which method is best. So, we choose measure performance by Area Under the Curve (AUC) normalized by the x-axis span of each accuracy curve. For every model, the ranking of each score function is displayed in Figure 3 Notably, Avg Head Output Norm and its normalized version are more aggossic,, ranking 1s for 10 out of 14 models. In contrast, Avg Weight of First Token ranks 1st for only Llama-3.2-3B and Qwen2.5-14B,, and performs most poorly for the OLMo-2 family of models.
4.4 INACTIVE HEADS BY LAYER
To better understand the poor performance of Average Weight of First Token (AWFT) on the OLMo- 2 family, we investigate the stability of two score functions on different data distributions. Using
OLMo-2- 1-24-11 7 we evaluate 3 datasets: MMLU (Hendrycks et al. 2021), PIQA (Bisk et al. 2020), and WinoGrande (Sakaguchi et al. 2021), and plot the layer-wise percentage of iden-
tified inctiive heads in Figure 5 We compare Average Weight of First Token (AWFT) (Gu et al.
2025
against Average Head Output Norm (LN), the best performing method in Section 4.3
8

---

Published as a conference paper at ICLR 2026
Avg Weight of First. Token
Avg. Head Output Norm (LN)
MMUU
MMUI PIQA WinxCrunde
56 of Inoictive Heads in Layer
80-
PIQA
WinoGrunde
50-
40-
30-
20-
5
10
1.5
20
25
30
10 15
20
25
30
Layer Index
Lager Index
Figure 5: Average Head Output Norm is stable measure layer-wise inactivity. Layer- percentage of inactive heads for OLMo-2-1 1-2-11124-BB-Inttreee across MMLU, PIQA, and WinoGrande. (Left) Identifying inactive heads via Average Weight First Token (AWFT) shows high sensitivity to the dataset. (Right) In contrast, using Average Head Output Norm (LN) yields more stable, dataset-agnostic percentages of layer inactivity.
To ensure fair comparison, we calibrate thresholds for both score functions such that 15% of heads are identified as potentially inactive. Notably, achieving this target classification with AWFT requires highly disparate thresholds depending on the task (7MMLU
0.077, TPiQA = 0.265, TWinoGrunde 0.109). Conversely, the thresholds required for AHON (LN) are more consistent across datasets (TмMLT 0.457, TPiQA 0.435, TinnGrande = 0.473), indicating a higher degree of generalizability.
The instability of AWFT is further evidenced by the layer-wise percentage of inactive heads. When using AWFT (Figure 5 Left), the per-layer inactive percentages exhibit high sensitivity to the dataset. For instance, on MMLU, inactive heads concentrated in later layers (peaking above 60% in layer 23). On PIQA, however, large fractions of inactive heads are in early layers (above 50% in layer 3). In contrast, identifying inactive heads via AHON (Figure 5 Right) mitigates this variance and shows stable inactive percentages across all three datasets. Measuring Avg Head Out- put Norm (LN) provides dataset-agnostie measure of head inactivity.
4.5 SCORE DISTRIBUTIONS FOR STUDYING ATTENTION BEHAVIORS
As described in Section 3 each score quantifies characteristic about attention head: how much attention paid to the first token, how small is the first value vector, how small is the average head output, etc. Analyzing the scores of heads can give Every model and score function combination produces a distribution of scores. By measuring the similarity of these score distributions, different. With the exception of the AWFT score function, most score distributions have similar modes (See App. 16g Thus, we chose the Wasserstein distance measure similarity between head score distributions. In Figure (6] we see that AWFT score distributions are similar within model family, illustrated by dark block diagonal pattern. The only outlier is Qwen2.5-14DD which has distributions more similar to Llama models than Qwen2.5 models. For AHON (LN) score distributions. Llama and OLMo-2 models are more similar to each other than to Qwen2.5 models.
broad idea of attention head behavior.
determine whether attention behavior is similar
We use the Qwen2.5 models (Yang et al. 2024) to understand how model scale affects attention head behaviors. There are model types ranging from 0.5B parameters to 14B parameters, but while the models share the same pre-training data, the exact number of training tokens each model was exposed to is not publicly documented. Assuming model scale is the only variable, it appears that model scale impacts attention head behaviors. As Qwen2.5 models grow in size, they are quite similar to one another, until the 14B scale. For example, on most score functions, Wasserstein distance is near-zero, indicated by clear dark region that corresponds to inter-model similarity. But at the 14B scale, there appears to be only score function distribution where Qwen2.5- 14B is similar to other models in the Qwen2.5 family: Avg Value Vector Norm and its layer normalized

---

Published as a conference paper at ICLR 2026
Wasserstein Distance Models' Score Distributions
Avg Weight of First Token
Heed Output Norm (LN)
Weight of First Token (LN)
-
3..38
0.14
0.16
Liama
0.4
0.12
0.14
0LM0-2-1124-00
0.12
2..Me-21124-50000
033
0.10
GLMo
2-122-770--nmmmm
0.00
Qwan2.5-.a5
0..08
0.06
0.06
Owen2
0.04
0.04
0wea2.5
0.1
5-78-hnirrcl-
0.02
0.02
0wen2.5-140
0.0
0.00
0.00
Clatee aud
Chess
Figure 6: Head score distributions can be similar within model family, Smaller Wasserstein distances indicate similar score distributions. Full results for all 12 score functions can be found in App. Figure 17
version. This may suggest that larger models learn to specialize their heads different ways. The similarity of score distributions between the Llama-3 models we consider may have more to do with their training objective as opposed to training data. In particular, because Llama-3.2 was created from pruning Llama-3.1-88 then performing knowledge distillation, these models are trained to be similar Meta 2024).
To study finetuning, primarily rely the OLMo-2 models which have checkpoints after SFT, after DPO (Rafailov et al. 2023), and after RLHF (Lambert et al.) 2024). We also have the Instruct models corresponding to the Llama-3.2-3B. Llama-3. and Qwen2.5-7B base models. 6 and 17 illustrate that all finetuning strategies we consider (SFT, DPO, RLHF) have nearly no effect on score distributions, suggesting little change in attention behaviors. Across all score functions, finetuned models have the smallest Wasserstein distances with their base model. In the case of all the OLMo-2 models, the Wasserstein distances of their score distributions are more similar each other than any other model. These results indicate that finetuning primarily preserves the underlying attention head behaviors of the base model.
CONCLUSION
Understanding when attention heads are inactive requires looking beyond attention weights. Simple score functions, based on previously ignored components like value vectors and head outputs, can capture a wider array of inactivity. By assigning different scores to every head a transformer LLM,,
classify different subsets of attention heads inactive using threshold. While there is some overlap in the subsets of inactive heads we classify using different score functions, find that most score functions capture diverse characteristics. By zeroing out identified inactive heads during MMLU evaluations, we find that more than 12% of heads can be zeroed out while maintaining accuracy to within 1% of the baseline model, on average. Using score function based on first token attention sink would classify less than 5% of heads as inactive on average, which underestimates the true prevalence of inactive heads. For the majority of models we consider, the score function that measures average head output norm produces the best signal for identifying inactive heads across model families. While attention sinks and sink tokens are almost synonymous with inactive head, our work demonstrates that there other sets of heads that more inactive, and that we can identify them in a more model-agnostic way. Additionally, we show how measuring score distributions can be informative: change to attention behavior, and how model scale does very little to attention behavior until very large scale. Future work could consider how the MLP module, which proceeds the attention module, could be inactive per-token if converged to an optimal hidden state.
them to explore how finetuning causes little to no
ACKNOWLEDGMENTTS
This work was made possible by National Science Foundation (NSF) grant #2213335. Pedro is supported by National Defense Science and Engineering Graduate Fellowship (NDSEG) and ac-
10

---

Published as a conference paper at ICLR 2026
knowledges the support and encouragement of John D. Moriarty. RB was supported in part by the Israeli Council for Higher Education (CHE) via the Weizmann Data Science Research Center, by the MBZUAI-WE - Joint Program for Artificial Intelligence Research, and by research grants from the Estates of Tully and Michele Plesser and the Anita James Rosen and Harry Schutzman Foundations. TG was supported in part by the NSF TRAILS Institute (2229885), DARPA TIAMAT, and grant from Coefficient Giving.
REFERENCES
Joshua Ainslie, James Leorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head check- points. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023.
URL https: /tmpps://ppa http:///ppppeeieeeeeeeeeeeeeeeee /tL.ttpps///ppeeeiiiis
net forum? forum??ddhnnooo2W22
Maximiliana Behnke and Kenneth Heafield. Losing heads in the lottery: Pruning transformer. In
The 2020 Conference on Empirical Methods Natural Language Processing, pp. 2664-2674. Association for Computational Linguisties (ACL), 2020.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on arlificial intelligence, volume 34, pp. 7432-7439, 2020.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer Plug-ani and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer, 2025.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li. Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating Iims by human preference, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.22 173,, 2024.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfiel-Dodds, Danny Hernande7, Andy Jones, Jackson Kemion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformerr citpraaannooommmmrrrrttttsteeste circuits. pww.so22 hwwp://///...aamoooo/nddde....e
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL httttttt///eoooee orw.eooooooorreeeeeeeeeeeeeeoeee rwwordd/rrerrr////oooooooooo.. 12608602
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In The Twelfth International Conference Learning Representations, 2024. URLhtttss:/oppereeeeee
net fforum?
id-uNrFpDPMyo
Xiangming Gu, Tianyu Pang, Chao Du Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. In The Thirteenth International Conference Learning Representations, 2025. Uttphttss://opem wpwp//veeereeeeeeeeeeeeeee
Tianyu Guo, Druv Pai. Yu Bai, Jiantao Jiao, Michael Jordan, and Song Mei. Active-dormant attention heads: Mechanistically demystifying extreme-token phenomena in llms, 2024a. URL
https://arrn wttp:vo..gg..////.//o// 13835

---

Published as a conference paper at ICLR 2026
Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. Attention score is not all you need for to- ken importance indicator in KV cache reduction: Value also matters. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference
Empirical Methods in Natural Language Processing, pp. 21 21158-211 21166, Miami, Florida, USA, November www.:/24..mmmmmmmmmm...m. 2024b. Association for Computational Linguistics. doi: 10. 10.18653/1/2224.cmm
URL pettps://aclantholooe wwwpeehhoooooooooooooooooo 2axg/22224 emnip wommnip-maii....
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only weight: Analyzing transformers with vector norms. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), P) pp- 7057-7075, Online, November 2020. Association for Computational Linguistics. doi: 10. 2020. www.emnnppppmm main. 5747
URL https::/acll
ogy org/
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint urXiv:2411.1512,, 2024.
Jiaoda Li, Ryan Cotterell, and Mrinmaya Sachan. Differentiable subset pruning of transformer heads. Transactions of the Association for Computational Linguistics, 9: 9:1442-1459, 2021.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of Instruction-follwiin models. https:: //t::////ittt /ggtttb......... www.tttttttaaaaaaaaaaa. aalaacaaaaaaalllla_aaaaaala_a_aa 5 2023.
Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. Advances in neural information processing systems, 30, 2017.
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for 11m kv cache compression test time, Advances in Neural Information Processing Systems, 36:52342-517 -2334444,, 2023.
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL. htttps:://huginnggfaccc co..aaaseaa.aaaaaaaaaaaaaa
HuuggingFaceTW/EI wwweee/www/wwwwweeeeeee eduu
Meta.
Llama 3.2: Revolutionizing edge ai and vision with open, tomizable models, Sep 2024.
URL atttp::/ai.mmr meta. com/blog/ Ilama 3-2 connect 2024 vision edge mobile devices/
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
Yongyu Mu, Yuzhang Wu, Yuchun Fan, Chenglong Wang. Hengyu Li, Qiaozhi He, Murun Yang, Tong Xiao, and Jingbo Zhu. layer attention sharing for large language models, 2024. URL https: arxiv. wwwp/os.////oooooooooto.o/o..trt 2408 01890
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656. 2024.
Paszke. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint 19110:191,.07703, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
12

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-537777 2023.
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based lan- guage models. arXiv preprint arXiv:2404.00 02258,, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adver- sarial winograd schema challenge at scale. Commun. ACM, 64(9)999-106 August 2021. ISSN 0001-0782. doi: 10.1145/347438. /Rthttts:///do/ orgeiiorgg/iee 1145/3474381
Noam Shazeer. Fast transformer decoding: One write-head all you need. arXiv preprint
arXiv:1911.02/500 2019.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. In First Conference on Language Modeling, 2024. URL nett/Forum?ld=777ahhht
Qi Sun, Mare Pickett, Aakash Kumar Nain, and Llion Jones. Transformer layers as painters. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 25219-25222 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Anna Korhonen, David Traum, and Lluís Märquez (eds.), Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 5797-5808, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.188633v1/119-110 1580. URL petpss//aclanttiee ogy.
org/P19-11 15807
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick vOn Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, 'evve Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. As- sociation for Computational Linguistics. htth::/ /wwww www.wb..wwbb........ wwwp:ew..rrg/ataooog.......
1020.emnlp-demcz
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Rep- resentations, 2024. StLhsttss://// http::::////oeeeeveeeeee net forumm NGG--G775511
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint 1111::2412.15115. 2024.
Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Un- veiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. In Forty-first International Conference Machine Learning, 2024. URL BtLLhttp::// hbttps://openreviiee net forum? (FF.DDDIFFFIII
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 DEFINING ATTENTION HEAD OUTPUT
Vaswani 2017 define attention head output the convex combination of value states, resulting in d diyddimeniional vector where de hs Omodel is the hidden size, and h is the number of heads. Other works multiply the dy-dimensional vector by IR Xdmas slice of the output projection matrix (Elhage et al. (02T). This is often called the "circuit- based" definition. In the following, we explain why this choice does not affect our intervention results.
Given N input tokens, an attention head will multiply attention weights A, E R-xx with value h, where the head index. In
states V, E R.d. so that Z; A,V; R. xdxx for multi-head attention, we first concatenate H [Zi;...: Zn. E R (xdde)) Then, do an output projection
y= HWo with Wo E (hdd..) Xxdum
Elhage et al. (202I) partition Wo row-wise into h blocks of size de x dmodee
w w
Wo
RRd. mpdexdam ddada
(2)
w
Then they note that the output can be written exactly as the sum of h terms:
Y Σ Z.W
(3)
Elhage al.) (202T) use the above argue that attention heads are independent saying that this is "equivalent to running heads independently. multiplying each by its own output matrix, and adding them into the residual stream." " But erucially, before we add the output of the attention module to the residual stream, This means that Wo can implement cancellations (if, for example, two head contributions sum to zero) other special linear combinations of heads.
actually add up all head contributions (i.e., the summation in Equation (3).
Our choice and why it does not affect results in Section 4.3 Because Wo mixes all heads' outputs into the shared model space, did not want to integrate these parameters into mea- surement. Thus, we define the pre-output-projection Z; as the attention head output. Importantly, whether choose to define Z, or Z, Z.i as the ith head's output, when "zero out a head" Ze both will be zero.
Table 3: Circuit-based definition of head output. We measure the percent of model heads that be removed while maintaining accuracy within 1% of baseline (higher is better). For 3 different models evaluated on MMLU, using the circuit-based definition of head output is comparable to using the original definition.
Model
AWFT
AHON (LN)
Circuit-based AHON (LN)
Llama-3, 1-lam--.11-8B-mmee I-strunstttutt
1.01
10.97
9.88
Qwen2.5-3B Qwen2.5-7B
5.67
7.29
5,95
1.25
7.54
9.04
Average
2.64
8.60
8.29
14

---

Published as a conference paper at ICLR 2026
Liama www.aaS--------teiee -iB--tnttuutt
Qwin25-35
Qwendd 2wim2.5-75
70-
02
KOHUL Accura
B0HLL A33m4
KHHU
40
Jurdea
Jap00e
Jasken
Hand
Sead Neer
Jeg tiad Chapal nome (U:)
Creaal Heat DiAppr Norr
Caveet Hord Dhaper Norra (150
Cheen Had Ourpet (120)
10
25
20
25
30
10
75
20
25
30
10
15
20
25
30
Figure 7: Circuit-based definition of head output. For 3 different models evaluated on MMLU, using the normalized circuit-based definition of head output is comparable to using the original definition. Dashed lines denote layer normalized (LN) score functions.
Implementing the circuit-based definition. We implement the circuit-based definition of Av- erage Head Output Norm (LN), and evaluate it on MMLU using models in Figure 7 More specifically, this means that for every head of index i we measure the norms of the rows of (, E Nx then average them produce a score. Table quantitatively shows that the percent of heads that can be zeroed using both definitions is comparable.
A.2 ADDITIONAL NORMALIZATION STRATEGIES
In Section describe normalization by the score of other heads in the same layer. For score functions that measure a particular token in the sequence, like Last Token Head Output Norm, it also possible normalize by other vectors within the same head. We refer to normalization by other vectors in the same head as head normalization, or "(HN)" within this Appendix. We explored this normalization strategy with the Last Token Head Output Norm score function, but found it was always worse than the layer normalization. "(LNY.. HN is only included in App. Figures [00 to I2 In this Appendix, when we refer to 13 score functions, we mean the 12 score function from the main body plus Last Token Head Output Norm (HN).
A.3 SAMPLE INACTIVE HEADS IDENTIFIED BY SCORE FUNCTIONS
Figure we give examples attention heads that are identified by each unnormalized score function, by using threshold-based rules of Ta To keep each attention weights matrix small, we use a random, short 50 token sequence from FineWeb-Eat and execute a forward pass on Qwen2.5- 7B and OLMo-2-11 2LMo-2-1124-7B 124-7B. In Figure 9 show how the normalized AHON (LN) score function can capture distinct kinds of heads. For example, the first head has a first token sink, the second head has two sinks, and the third head has second token sink. AHON (LN) is able to identify all three as inactive.
As summary of the kind of head that each score function is meant capture, we provide short explanation for each:
Avg Weight of First Token (AWFT): When attention concentrates on first token
Avg Entropy of Query Distributions (AEQD): When attention concentrates on a few tokens
First Token Value Vector Norm (FTVVN): When first value vector is near-zero
Avg Value Vector Norm (AVVN): When most value vectors are near-zero
Last Token Head Output Norm (LTHON): When last head output token near-zero
Avg Head Output Norm (AHON); When most head output tokens are near-zero
15

---

Published as a conference paper at ICLR 2026
Heods idencifeed by Ang Weight of Finst Thben
CAMT JOHHOO
PARKA
Poidaca
Telee Posaise
PULEAS
Entrapy
M Scoa Duttut
12
Booa Cuppur
Hoads Alentiied by: Parst Token Value Voctor Norm
Audia
Poides
Beeds Identiffed Aug Valae Vector Norm
Notion
Sobent
Itaad alunstiadd
Tokon Head Dutpptt
New
Zhive Foetso
Heads Mennited by Ang Head Output Norn
12 FOC MOOMMORDDAA
Figure& 8: Sample Attention Heads for Unnormalized Score Functions. In each row, we plot 3 sample attention heads for each score function in Table To the right of each attention weights matrix, we plot the norm of Value Vectors and Head Outputs. We also include the layer index and head index of each attention head above each plot. Lower scores are better for all score functions other than AWFT. Note the variety of heads that are identified by each score function.
A.4 FULL RESULTS OF SECTION
In Section Figure
we present intervention results for 3 models. Results for all 14 models are
presented in Figure 10
A.4.1 ADDITIONAL DATASETS
We also run the same experiments and analysis on the PIQA (0-shot) (Bisk et al. 2020) and Wino- Grande (5-shot) Sakaguchi et al. (2021) datasets. Plots how accuracy degrades with different percent of model heads zeroed can be found in Figures and [2] The quantitative analysis, where measure the maximum proportion of heads that can zero while maintaining accurcay within
16

---

Published as a conference paper at ICLR 2026
Teeos identifted by Ang Tood Ourpot Norn (UN)
N M
Day Neiiice
Z0o0 Foatiio
Day Neiiice
Zden Foetion
Sky Pocken
Soler Fontaan
Figure 9: Sample Attention Heads using Avg Head Output Norm (LN). We plot sample atten- tion heads for AHON (LN), the best score function by AUC (See Figure B). To the right of each attention weights matrix, we plot the é2 norm of Value Vectors and Head Outputs. Note that the first head has first token sink, the second head has two sinks, and the third head has second token sink. AHON (LN) is able capture all three distinct heads.
1% of the original model can be found in Tables 4 and 5 On PIQA, our score functions show that more than 21% of attention heads can be zeroed while keeping average accuracy within 1% of the original model. On WinoGrande, more than 14% of attention heads can be zeroed while keeping average accuracy within 1% of the original model. Thus, our claim in the main body of the paper conservative estimate of inactive heads. In all cases, our score functions identify more heads than Avg Weight of First Token (AWFT).
Table 4: PIQA Benchmark: On average, more than 21% of attention heads can zeroed while keeping average accuracy within 1 % of the original model. For every model, we calculate the highest percentage of heads that can be zeroed using Average Weight First Token (AWFT), and compare to overall best score function is also shown. We are able to identify and verify more inactive heads than previously possible with AWFT.
functions (i.e., all score functions excluding AWFT). Higher is better. The
Model
of Heads Zeroed
Best Score Function
AWFT
Ours
Llama-3.2-31
21.76
17.37 (-4.39)
Avg Weight of First Token
Llama-3.2-3B-
3.2ttB-Insttt
15.32
25.78 (+10.46)
Avg Head Output Norm (LN)
Llama-3.1-8B
0.00
17.85 (+17.85)
First Token Value Vector Norm (LN)
-lama-3.1-88--nnsiiee
31.30
25.39 (-5.91)
Avg Weight of First Token
OLMM-2-1124-7B
20.74
22.70 (+1.97)
Avg Head Output Norm (LN)
11MMo-2-112477-933-
1.87
20.12 (+18.26)
Last Token Head Output Norm (LN)
OLMo-2-11124-7BDDR-
2.93
15.87 (+12.94)
First Token Value Vector Norm (LN)
OLMo--211224-78-nnttuu
2.54
26.83 (+24.30)
Avg Head Output Norm (LN)
Qwen2.5-0.5B
0.00
18.36 (+18.36)
Avg Head Output Norm (LN)
Qwen2.5-1.5B
24.33
21.32 (-3.01)
Avg Weight of First Token
Qwen2.5-3B
16.51
21.42 (+4.91)
Avg Head Output Norm
Qwen2.5-7B
0.00
28.66 (+28.66)
First Token Value Vector Norm
1ann2.557B-Inttruus
2.84
11.45 (+8.61)
Avg Head Output Norm (LN)
Qwen2.5-14B
0.00
26.12 (+26.12)
First Token Value Vector Norm (LN)
Average
10.01
21.38 (+11.37)
All models and score function evaluations be found in Figure 10
A.5 MULTIPLE FORWARD PASS ANALYSIS
The main body of the paper focuses on analyzing a single forward pass because that how log- likelihood evaluations of MMLU, PIQA, WinoGrande and other MC benchmarks performed. To explore how heads may change from inactive to active, based on increasing context, we choose Fine Edu train sequences of least 512 OLMo-2 tokens in length:
17

---

Published as a conference paper at ICLR 2026
Table 5: WinoGrande Benchmark: On average, more than 14% of attention heads can be zeroed while keeping average accuracy within 1% of the original model. For every model, we calculate the highest percentage of heads that can be zeroed using Average Weight of First Token (AWFT), and compare to our score functions (i.e,, all score functions excluding AWFT), Higher is better. The overall best score function is also shown. We are able to identify and verify more inactive heads than previously possible with AWFT.
Model
% of Heads Zeroed
Best Score Function
AWFT
Ours
Llama-3.2-3B
9.11
11.64 (+2.53)
Avg Head Output Norm (LN)
5.57
20.37 (+14.79)
Avg Head Output Norm (LN)
Llama-3.1-8B
6,86
17.08 (+10.22)
Avg Head Output Norm (LN)
Jlama-3.l-8B-Instruutt
12.44
11.89 (-0.55)
Avg Weight of First Token
OLMo-2-1124-77
15.39
16..2
Avg Head Output Norm (LNN
0LMo-2-1124-7B-SFT
2.32
19.00 (+16.68)
Avg Head Output Norm (LN)
OLMo-2-1112477--DPO
2.79
16.94 (+14.15)
Avg Head Output Norm (LN)
OLMo---11244-BB-nnstruu
3.02
20.91 (+17.89)
Last Token Head Output Norm (LN)
Qwen2.5-0.5B
6.49
11.35 (+4.86)
Avg Head Output Norm (LN)
Qwen2.5-1.5B
7.49
12.68 (+5.18)
First Token Value Vector Norm
Qwen2.5-3B
5.73
10.32 (+4.60)
First Token Value Vector Norm
Qwen2.5-7B
6.60
15.69 (+9.10)
First Token Value Vector Norm (LN)
Brem2.5-7B-Instrree
2.00
8.03 (+6.03)
Avg Value Vector Norm (LN)
Qwen2.5-14B
10.80
10.25 (-0.55)
Avg Weight of First Token
Average
6.90
14.45 (+7.55)
Seq 0 ID: um..iicess..00222554445.all11112 Category: Literary com- mentary.
Seq ID: ummmutt:1ccnn5544444444077777773 Category: Popular Science or Journalism.
Seq ID: umnuidc3777ccccaaaa4444444444443 Category: Educational arti- cle on software licensing.
as each of these sequences grows in Figure 13. We analyze AWFT and AHON (LN) score functions because AWFT is prior work, and AHON (LN) is the best score function discover in Section As expected, AWFT presents decaying curves where the model has fewer inactive heads as sequence length increases. As se- quences increase in length, each query has
We choose to evaluate inactive heads of
potential tokens it can attend to, making it less likely that the first token receives disproportionate attention. Additionally, as can be seen in Fig- ure 20 and Appendix A.3 OLMo-2 models tend to additional sink tokens (not just the first token). In contrast, measuring AHON (LN) inactive heads as sequence length increases, we observe noisy measurements of inactive head proportions in short sequences that converge as each sequence gets sufficiently long. In measuring both AWFT and AHON (LN) inactive heads, it is clear that inactivity context-dependent.
We are also interested in where changing attention heads are located. We examine inactive head masks for AWFT and AHON (LN) in Figures [4 and [5] Every matrix cell indicates particular attention head, and we color yellow any head that is identified as inactive. Unlike AWFT score function measurements, AHON (LN) inactive heads are more spread out throughout the model. We
also observe the same phenomenon in Section 4.4 Figure
A.6 SCORE DISTRIBUTIONS ON MMLU
All score function distributions, from MMLU, can be found in Figure 16
18

---

Published as a conference paper at ICLR 2026
A.7
FULL RESULTS OF SECTION 4.5 WASSERSTEIN DISTANCES
All Wasserstein distance matrices, for all score functions, can be found in Figure 17
A.8 SCORE DISTRIBUTIONS COMPARISON; FINEWEB-EDU VS. MMLU
In Figure 18 show how the distribution of Avg Weight of First Token scores change when the
input dataset changes. Qwen2.5-7B has small fraction of heads that exceed 0.8 average attention the first token on FineWeb-Edu, but on MMLU there are essentially none. One explanation is the token sequence length which, for 5-shot MMLU, tend to be approximately 3000 tokens long. In contrast, the FineWeb-EGe Edu sequences we use are randomly truncated to between 30 and 3000 tokens.
A.9 ATTENTION SINKS AND VALUE-STATE DRAINS
Sample attention sink patterns and value- state drains can be found in Figure
A.10 REDUNDANCY OF ATTENTION PATTERNS
Numerous works have noted the prevalence of redundant attention patterns across attention heads, even from different layers Xiao et al. (2024); Mu al. (2024): Guo et al. (2024a): Liu et al.] (2023); Ge et al. (2024), and attention sinks make up part of those patterns. In Figure 20 we observe the same phenomena recent pretrained LLMs. We plot attention matrices from the last layers five models. The chosen head indices are simply ordered sequentially. Not only are attention patterns similar among heads every model, different models have different sink token behaviors. Llama- 2-7b-hf uses two sinks and divides weight between them. Llama-3.1-8 sink. OLMo-2-1 124-7B, like Llama-2-7b-hff uses two sink tokens, but the second intermediate sink different position than that of Llama-2-7b-hf Qwen2.5-7B tends to the second token sink, while (wee22.514DP uses the first token. We exclude Llama-3.24 332-BB from Figure because it presents similar attention patterns to Llama-3.1-8B.
uses the first token
For every model in Figure 20 we also show the top 2 principal components across all attention surprising see few principal components
matrices (of which there are Nayer x Neeads)) capturing more than half of the observed variance of attention matrices, for all models.
The input to all models in Figure 20 from MMLU's high.school. split, index 2. The input text is in the same format used by im-evaluationn-aanee (Gao al. 2024):
school wcwpnneeeeeeieeeeeeeeeeeeeeeeeee dev
The following are multiple choice questions (with answers) about high school computer science.
What is the output of "abc" [::-1] in Python 3?
A. Error
B. abe
C. cba
D. c
Answer:
A.11 ON AM-EVALLAAION-HARNAA EVALUATIONS
For reliable and complete evaluations, use size=s/zzzaato Im-evaluatiion-hanee not only finds an optimal batch size for your hardware to prevent memory errors but also mitigates the issue where an entire batch may be skipped because one sample cannot fit on the GPU.
A.12 PYTORCH IMPLEMENTATION OF AvG HEAD OUTPUT NORM (LN) AND AVG WEIGHT TO FIRST TOKEN
We provide sample Py Torch code (Paszke 2019), for Avg Head Output Norm (LN) and Avg Weight to First Token, that can be integrated into self-attention module's forward pass. When using
19

---

Published as a conference paper at ICLR 2026
Lm-evaluation-hare (Gao et al. 2024) to evaluate pretrained models, additional steps must be taken to ignore padding tokens during log likelihood evaluations (on MC datasets).
All score functions implemented within the self-attention module and take as input the following tensors: attention weights size (B, Neead, S, S), value states of size (B, Meadd S, d,), and a float threshold. B denotes the batch size, S denotes the sequence length, Nhead denotes the number of attention heads, and d, is the dimension of the value states. As output, both implementations return boolean mask of dormant heads of size (B, Nhead) and attention outputs of size (B, Meead, S, d.).. In the dormant mask, an entry is True if the head is declared dormant, and False otherwise. Note that models that use grouped-quers attention (GQA) (Ainslie et 2023) or multi-query attention (MQA) (Shazeer (2019) still construct tensors of these sizes, so the specific kind of multi-head attention relevant.
Listing Avg Weight to First Token in PyTorch
def avft_dormant_mask (attannteeeasktt___eeeeeeeeeeeee (attn_weights. value_states, threshold)
avg_weight attn_weights mean (dim=-2)
(B, N_head, 5)
first_token_ar__eighh avg_weight ; ;; formant_mase = toter_avoke_eg_c_eo
[B, N_head)
threshold (B, N_head)
Model intervention: set dormant head outputs to zero at attn_output = torch marchl (attn_weights, value_states)
at atttouttput [dormant_mask]
return attn_output, dormant_mask
Listing 2: Avg Head Output Norm (LN) in PyTorch, following Equation
def ahon_in_dormant_me (attn_weightss value_states, threshold)
attn_cutput - torsh.matmul matmul weitht_ value_states) norm_per_token attn_output. norm (dim=-1) avg peg head norm per per_ tcken. mean (dim=-1) # (B, head)
(B, h_aad, 5)
compute average across all heads in layer layer_content awg_norm_per_head (dan (dim=1) (B,
@e__avg_nerm_per_r # (B, (haadd
(avg_nerm_per_head / layer_context I:, Nonel)
dormant_mask = rel_avg_norm__r wat_aagggo____aeeee head < threshold # (B, N [_hadd)
Model intervention: dormant head outputs to zero
12
attr_output [dormant_mask]
13
return attn_output, dormant_mas
A.13 ADDITIONAL. MODEL AND DATASET INFORMATION
Model details. Model inference done in the original data type of the saved weights using Aut SeatddeeForCausallM from-pretrained
torch..t.ye--auutte
ex-
cept for OLMo-2 models, where we use float16. All models are downloaded using Hugging Face (HF) trans formers (Wolf al. 2020 Parameter counts and release dates shown in Table
Model Name
Params
Heads Per Layer
Num Layers
Total Heads
Release Date
Llama-3.1 8B
8.03B
32
32
1024
Jul 2024
Llama-3,2-3
3.21B
24
28
672
Sep 2024
OLMo-2-1122
1-Mo-22-1244-BB 124-7B
7.3B
32
1024
Nov 2024
Qwen2.5-7B
7.62B
28
28
784
Feb 2025
Qwen2.5-14B
14.8B
40
1920
Feb 2025
Table 6: Additional model information of pretrained LLMs used this work.
20

---

Published as a conference paper at ICLR 2026
Dataset details. The test split of the MMLU benchmark contains 14,042 multiple-choice (MC) questions spanning 57 tasks including mathematics, US history, computer science, law, and more. Each question has 4 answer choices.
We also use PIQA (Bisk et al,, 2020) and WinoGrande (Sakauuchi et al. 2021, but only in Ap- pendix to reduce clutter in the main body. PIQA is MC benchmark dataset of physical com- monsense questions where each question has 2 answer choices. WinoGrande a MC benchmark dataset of pronoun resolution problems where each problem has answer choices.
A.14 PRACTICAL IMPLICATIONS AND LIMITATIONS
Architectures. We think there is evidence that different problems require different fractions of model parameters to solve (Raposo et al. 2024 Lin et al. (017). Our experiments provide concrete evidence that this occurs in self-attention, and we think sparsity can be beneficial for designing architectures like MoEs but for attention modules. We may not need to have all heads in memory only subset will be used. A router could potentially be developed select the appropriate heads. Our work provides necessary background on inactive attention heads so that future work can develop new architectures for this problem.
Efficiency. It should be possible to dynamically prune different heads on-the-fly whenever we identify head as inactive. But it is unknown whether the overhead in identifying these heads (by computing scores) will make inference so slow that the compute savings are not worth it. In particular, our best score function (AHON (LN)) has the issue that requires computing the head output itself, making it impossible to save compute when "skipping" head. However, future work may discover a more proper score function that would allow for compute saving. For example, Lin et al. (2017) showed that image classifiers could be dynamically pruned at runtime, for each individual input. Another potential direction is KV cache compression: specifically, not storing K/V for heads that are inactive. Our work also opens up directions for investigation like finetuning inactive heads.
Limitations. We focus on analyzing a single forward pass because that is how log- -ikeliidl evaluations of MMLU, PIQA, WinoGrande and other MC benchmarks are performed. Analyzing generation is complex because zeroing out a single incorrectly identified head can ruin the rest of the generated sequence (i.e., more opportunities for failure). For example, if we zero out too many heads, we could sample token representing a different language. Analyzing generation a natural next step, as it is used for more practical use-cases.
A.15 LLM USAGE
LLMs were used to aid or polish writing, Gemini 2.5 Pro and ChatGPT were used with prompts like "revise this sentence".
21

---

Published as a conference paper at ICLR 2026
Ulmma 3.338
Dame-322 Instrve
Damm 3.188
70
0: Madi Hoall Servec
MooME
SRAAhh Zereed
56.11 Mocel Hoeds Zwrod
3.1 Instrucc
0LMo 2-14--1114-70
2124-21 SFT
20
25
70
30
Model
4effetall Deade Zerted
Quen7 5-1.5
Qu002.5031
Qwer1.5520
9.00 MEdid Zande
NE Daaut Z4roed
Nocal Iouda Zerood
Qwam2.5-340
Baseliee Ausurat
Rendom
Arg Weggtt First Tolec
Acg Winght Pins Takes (LN)
Untrupy Dntributions
Deteibations (IN)
Ang Curput
Cuipat
Last Tikken Curput Norm (HNI
Ang Curpott Norm
Ang Head Ourgut Necm (LN)
15
3f Bouth
Figure 10: Complete MMLU Results of Section 4.3 For 14 models, consider 13 score func-
tions, and evaluate each at 7 distinct thresholds for each.
22

---

Published as a conference paper at ICLR 2026
3. 5:38
Damee 22.33 Instrvet
9-00 MadE Hoall Served
MOORE PRNAA Zereed
Mocel Hoeds dwrodd
1Jtno 3.1 InstRNCC
0 2-144-7
0L.M 1124 SFT
10
20
20
50
15
30
70.0
07.5
79
60.0
60.3
57.5
55.0
Mode
Modil Thedk
Medaa Deadk
5-1.51
Qun02.5.00 5:31
Qwer1255570
75
33
0/00 3ldd ZArai
N0 Saauh 24roed
Nocal Ieada Zarood
Qwee22 785-nnatrutt
Qwam2.5-340
Baselie Ausuracy
Rendom
Arg Weggtt First Toles
Acg Winght Pins Takes (LN)
Intrupy Dntiibutions
Deteibations (IN)
Ang Curput
Cuipat
Last Tokken Curput Norm (HNI
Ang Curpott Norm
Ang Head Ourgut Necm (LN)
15
Figure 11: Additional PIQA Results for Section 4.3 For 14 models, consider 13 score func-
tions, and evaluate each at 7 distinct thresholds for each.
23

---

Published as a conference paper at ICLR 2026
03.0
510
Modd Shouch Zeroed
55 Mode Haads Zarred
36 Node Heads Zureed
Model
Mada
D1Me-2-1 124-77-1100
Ma-2-1 126-778-tnddmut
(wen22554135 541.3
60
30
00.0
62.5
00
57.5
10.0
47.
50-
Modil Thauch Zerrod
55 Model Heads Zarree
27 Model Hende:
Owam2255 5-1.-mttrttttttt Irstrrctt
Queen2 5-148
Buselize Accunacy
60.0
00
Bandom
Ang Waight at Pirnt Toienn
Ang Waight of Pimst Token (1.4
Arg Entrogy Ouers Distribttooss
Avg Estrogy Quety Distibutious (LN)
Fiss Thlen Velue Victue Norm
First Thleen Velue Vecte Norm
50.0
Value Norm
Vilne Vettor Norm
50.0
Auy Cargen
Movil Seeale Seraad
Figure 12: Additional WinoGrande Results for Section 4.3, For 14 models, we consider 13 score functions, and evaluate each distinct thresholds for each.
24

---

Published as a conference paper at ICLR 2026
Model: 0LM6-2-112247B-nsttrutt
Avg Weight of First Token
Avg Head Output Norm (LN)
100
24
Sog
Seq 1
Seq
Seq 2
Seq
%6 of Inartive Hoads
90-
% of Inactive Heads
20-
16
70-
12
100
200
300
400
500
100
200
300
400
500
Sequence Length
Sequence Length
Figure 13: Inactive Heads Sequence Length Changes. For 1-1------11--77------t-e consider 3 FeneWeb-Edu sequences, and evaluate the percent of inactive heads in the model as the sequence length increases. We compare AWFT (r 0.077) and AHON (LN) 0.457) score functions.
Inaotive Heads ssing Ang Weight of Flist Tokic th- 6.077 for OLMo 2.24 Instrett
Figure 14: Inactive Heads as Sequence Length Changes. For OLMMo-21112-77B InIttruct, we plot an inactive head mask for increasing sequence lengths of "Seq 0" from Figure [3] Yellow indicates an inactive head, as identified by AWFT score function.
Inactive Hoads using Ang Hoad Ourpet Noem (LN) te d-tt----55mn toe OLMe- 11226//Biizttrrt
Intes
10
bate
Figure 15: Inactive Heads as Sequence Length Changes. For
we plot
an inactive head mask for increasing sequence lengths of "Seq 0" from Figure [3 Yellow indicates an inactive head, as identified by AHON (LN) score function.
25

---

Published as a conference paper at ICLR 2026
Disriletios of Scorns
CDP of Sccene
Diertvatios of Sooms Weighe
CDT of Soores
1224
20
1/3
1143
Useed 140
02
04
06
00
00
Aug Extrgyy of Queey Ditriyttioss
Jng Entrope of Quiry Datriburtons
Ang Setrope of Quary Dattrbatizee (120
Aog Estrogy Query Seerbbtiize (136)
Lse Usead 211
64
1.5
0.5
0006-113377
Heed Seate
Psad Foor
Hind Scere
Head Soore
Desrilettton of Sccees
CDR of Screns
Deardvviion of Soores
CS2 Shoces
2.0
0010-1110 1104
20
1.5
1.5
1/3
2143
0.5
Us 140
Queca
0.0
0.
Haad Stame Avg Voltin Ventie Nomr
Aug Valle Vecuor Koem
Ard Value Secter Noen (.N)
Ang Wiltes Sootee Norn (120
0.4
3.2
Uneid a/1
0.5
3.5
3.4
0:2
0.5
2.0
Hard Scate
Isad Score
Haad
Head Store
Disridertenn of Scort5 Oripet
CDE of Scores
Dizcibutton Stores Outpat
CBC Sooces
Chatpen
1.2
Unaon 1:06
Mi (154
1.3
0.-
0/
03
1143
02
11a Swend
2.5
2/3
Hand Scane
Iead
Haau. Scara
Loct Tokee Head Oveget Noen (HNN)
Last Tolom desd Octpon Noms (HN)
Rug Food Outpet Nare
Xay Heed Dinguat Neen
Lol
0.0
Dewid
3.1
E.C
3.5
0.5
0.0
044
3.4
0.4
3.2
0.2
02
0.1
001
15
20
Head Seate
Ised Four
Score
Datroleziann Sourca
CDP of Bececss
Aug Outgor Norm
Head Deggut
1.4
1.2
1.0
0.0
Hand Scats
Clad Score
Figure 16: Head Score Distributions on MMLU. For 13 score functions, we plot a density distri- bution and a CDF of head scores. We only show scores for Llama-3.1-8B, OLMo-2-1124-7B, and Qwen2.5-7B for brevity.
26

---

Published as a conference paper at ICLR 2026
Wozde Seower
Score Disonbutions Eatray Quayy Bairtorass
Wuagat
Ary Exteayy Quary Satakdiar
Tokes Vahan Mertor Noms
Takon Viluo Shster Koma
Vikee Victer Kora
Vectar Norn 1U20
12:00
Loxt Taless Isxrd Orgaa Noca
Laxt Tikee Ioad Output Norra (LN0
Lost Thkca thou Datgat Necn (th)
Ang tbod Oalpat Narm
Nr Head Ourgur Kena C.I
Figure 17: Wasserstein Distance of Head Score Distributions between Models. For each of 13 score functions, we plot matrix of Wasserstein distances between score distributions of different models.
Socres FFincWcb Weight
Scorcs Edu) Tukknn
(MMUU)
Scores (MMLL)
Weicht Tirst
Weoght Tubws
Slarad
04.99//12281
Owend
134
Oveed 5-70
123
0.5
0.6
01N6331126-7
01302-112477
68
0.8
0.2
64
0.5
0.6
0.
0d6
0.4
64
0.6
1/
Rood Boxes
Head Scaro
Food Soors
Head Scoro
Figure 18: Distribution of Avg Weight of First Token Scores changes when the input distribu- tion changes. Using the same 3 models, we find that the distribution of Avg Weight of First Token scores varies significantly when using input sequences from MMLU (right) instead FineWeb-Edu (left).
27

---

Published as a conference paper at ICLR 2026
Llama-2-7b-hf
Llama-3.1-80
OLMo-2-1124-7B
Qwen2.5-7B
Qwen2.5-14B
Layer 26 Head 13
Layer 26
Layer 26
Layer 26 Head 13
Layer Head 13
Head 13
Head 13
In yrr
L2 Norm
8-
M NA
WWw
WWW
6
2-
15
40
40 Value Position
Value Position
Value Position
Value Position
Value Position
Figure 19: Attention sink tokens have the smallest value vector norms. For different models,
input the same question from MMLU. In the first row, we plot the attention weights for an arbitrary head (Layer 26, Head 13) across all models. the second row, we plot the f--norm of the value vector at every position.
Attention Matrices
PCA Components
Laper 24 Heud
Eapar 25 Head
Layer Houdl 2
Tayyer 27 Headl
Laper 28 Haad
Laywrr 20 Houd
Eayer 3n Heud
Loper 21 Haad
Camp 41.8%
Gamp 8.8%
Dlvna
40.0%
Dwned 01M6 Dems
Lager 24
Laser 25 llaad
Layor 26 lined
Lagee 27 Daxd
Laper 28 lleed
Layer 20 Dledd
Laper 30 limed in
Layer Bt ileed
Conp 41.7%
Comp 8.0%
Dead i
Layer
Layer
Layer 20
Layer
Loyer Hind
Layer 26
Laser
Layer Head
Comp 45.2%
Comp
Weight
0.1
Figure 20: PCA of Attention Matrices. Attention patterns from different heads appear homoge- neous and dominated by attention sinks. We input the same multiple-choice question from MMLU into Llama-2-7b-htt Llama-3.1-8B. OLMo-2-1124- alize the attention matrices. Each model row displays a sample eight attention matrices from the last eight layers, followed by the top-2 principal components of all attention matrices, with the explained variance of each component displayed. The top-2 principal components display bright columns of attention sinks while capturing 50% of the variance. "L26 H2" denotes the attention head at index 2 of layer 26.
Qwen2.5-7B, and Qwen2.5-14p then visu-
28


Reviews:
[{"review_id": "fE68klpFWY", "reviewer": "Reviewer_j42Y", "summary": "This paper studies the phenomenon of \"inactive heads\" in pretrained LLMs. To comprehensively identify inactive heads, the authors propose 13 score functions related to attention heads for evaluating the importance of heads and assess their effectiveness in identifying inactive ones. Extensive experiments on MMLU and pretraining corpora show that the proposed \"Average Head Output Norm\" metric performs best in identifying inactive heads. It can remove as many attention heads as possible while maintaining the original performance, and it detects more inactive heads than the traditional method based on the first-token attention sink. The paper also uses these metrics to briefly analyze attention behaviors across models of different sizes and training settings, showing that finetuning hardly changes attention behavior, while larger models exhibit distinct attention patterns.", "questions": "1. In Section 4.2, the paper states that different score functions identify different groups of inactive attention heads, with low overlap among them. Could the authors explain what kinds of attention heads are identified by each metric? Do these metrics capture different aspects of the model's behavior?\n\n2. Head scores computed by functions in Table 1 require computation to obtain during the actual inference. I am curious whether the model's static parameters might already contain some clues about inactive heads (e.g., the value projection matrix of each head, or the corresponding output projection matrix).\n\n3. In Section 4.4, Line 453, the authors suggest that larger models learn to specialize their heads in different ways. However, the score distribution change only appears from 7B to 14B, and there is no larger model to support this claim. I hope the authors can validate this hypothesis on larger models (e.g., 32B).\n\n4. Do the findings in this paper have some pratical contributions, such as guiding model performance improvement? I know that this work focuses on interpretability, but it would be better if it also had potential for practical guidance.\n\n5. Minor formatting and typo issues: several elements exceed the page boundary (e.g., Table 1, Figures 6–7, and the example text in Section A.6); the opening quotation mark in Line 148 is incorrect.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed scoring functions comprehensively covers various aspects of the attention layers, and are extensively evaluated on various LLMs across different model families, showing strong generality.\n\n2. The proposed AHON metric effectively identifies a more complete range of inactive heads and provides useful guidance for studying inactive heads.\n\n3. The paper is clearly written, includes complete experimental results, and is easy to follow.", "weaknesses": "1. The main contribution of the paper is limited to identifying inactive heads in LLMs. Beyond that, the discussion on the mechanisms and behaviors of inactive heads is quite restricted. I would like to see why AHON can better identify inactive heads from an internal, model-based behavior perspective.\n\n2. In the experiment section, the paper only uses the pretraining dataset and the 5-shot MMLU dataset. As MMLU is indeed commonly used to evaluate general model capability, using only one dataset is rather limited. Moreover, the 5-shot setting may interfere with the evaluation of inactive heads related to in-context learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8OANsHpEt1", "reviewer": "Reviewer_8PXa", "summary": "This paper investigates the phenomenon of \"inactive\" attention heads in large language models (LLMs). The authors argue that prior work, which defined inactivity solely through attention patterns (like \"attention sinks\" on the first token), provides an incomplete picture. They propose that a head can be inactive in multiple ways, not just through its attention weights, but also through its value vectors or its final output vectors.", "questions": "Did the authors observe any cases where a head identified as inactive on MMLU became critical for performance on a different type of task? How task-specific do you believe the phenomenon of head inactivity is?\n\nThe analysis provides a \"snapshot\" of head activity. Did the authors investigate if the inactivity of a head is static, or can a head dynamically switch between active and inactive states depending on the specific input token or the evolving context within a sequence?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper moves beyond a single definition of inactivity to propose a multi-faceted taxonomy. The 13 score functions provide a much richer analysis.\n\nTesting across 14 models from 3 different families provides robust, generalizable results that are not specific to a single architecture or training corpus.\n\nThe analysis of score distributions to study the effects of fine-tuning and model scaling provides valuable insights into model behavior.", "weaknesses": "**Limited Task Evaluation**: The primary evaluation is on MMLU, a multiple-choice knowledge-based benchmark. The generality of the findings to other tasks (e.g., code generation, long-context reasoning, creative writing) is not verified. A head inactive on MMLU might be active on a different task.\n\n**Static, Single-Pass Analysis**: The analysis is performed on a single forward pass. The role of a head might be context-dependent or dynamically change across layers or tokens in a way that this static analysis misses. The paper acknowledges this by noting that inactivity is verified \"in specific contexts.\"\n\n**Threshold Reliance**: The method relies on choosing thresholds, which, while systematically swept, is still a heuristic. The \"inactive\" label is not absolute but relative to a chosen threshold.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4TbWZZ0O1C", "reviewer": "Reviewer_92AY", "summary": "This paper investigates inactive attention heads in pre-trained LLMs by proposing a taxonomy of 13 score functions that measure attention weights, value vectors, and head outputs. The authors verify head inactivity through model interventions, zeroing out identified heads and measuring MMLU accuracy across 14 models from 3 families. Their main finding is that more than 12% of attention heads can be zeroed while maintaining accuracy. These results are significantly better than prior methods. The paper also analyzes score distributions to show that finetuning causes minimal changes to attention behavior and that model scaling has little effect until very large scales.", "questions": "1. Do you have a speculation on Why the model does not leverage 12% of its heads?\n2. Is it clear how to design better architecture to take advantage of that?\n3. Can we prune these heads in production system to improve latency and cost?\n4. Do you have any hypothesis on why these head emerge?\n5. Can we change the training data or algorithm to incentivize the model to use all attention heads?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Clarity and readability\nThe writing style of this paper significantly enhances the quality of the work.\n\n2. Depth\nThe paper discusses a wide-range of scores, to identify and evaluate inactive attention heads in a robust manner.\n\n3. Novelty\nThe paper shares a new surprising finding about LLMs that challanges prior work.\n\n3. Rigorous Experimental Design\nThe paper tests across 14 models from 3 families (Llama, OLMo, Qwen), it uses model interventions to verify inactive heads rather than just classification.", "weaknesses": "1. Single Benchmark Evaluation\nThe current evaluation is limited to MMLU, but inactive heads might become active in very specific context that are not triggered in MMLU. \n\n2. Lack of Practical Implications\nThe fact that 12% of heads are inactive is astounding, however, it is unclear what kind of actionable steps can be taken to improve modern LLMs. Does this insight imply a different architecture? Could we remove heads in deployment and achieve better performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ywjvwstG8f", "reviewer": "Reviewer_WQ2Q", "summary": "The paper aims to reform the existing attention-sink-based characterization of inactive heads by focusing on the distribution of attention weights in LLMs. It introduces a taxonomy of 13 score functions to identify inactive heads, based on either the attention weight distribution or the norm of an attention head’s value vector or output at different positions, under various normalization strategies. Through ablation-based studies across 14 models, it shows that up to 12% of heads can be ablated without accuracy loss when the proper score metric is chosen, and that output-norm-based metrics outperform traditional attention-weight heuristics in many cases. The study also finds that fine-tuning strategies and moderate scaling barely affect attention behavior, highlighting strong structural invariance across models within the same family.", "questions": "1, In the first row of Table 2, why is there a difference percentage between AWFT and your method (which is also AWFT)? The same applies to the case of Qwen2.5-1.5B and Qwen 2.5-14B as well.\n\n2, Line 407-408: \"Each of 13 score functions use a different set of 7 thresholds, which may not identify 30% of model heads (the maximum we consider)\". I am unsure of what this means. Don't you ablate the same percentage of heads (whether it is 30% or not) identified using different criteria?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper represents a meaningful step toward moving beyond the current landscape of inactive head identification and attention pruning, which is predominantly attention-sink-based and focuses heavily on surface-level attention weight characterization. The score metrics proposed in the paper—based on the norms of different aspects of attention head outputs across token positions—are indeed novel. They open the possibility for a new paradigm in this subfield that diverges from the traditional attention-sink-centered narrative and instead evaluates inactivity through more direct and mechanistically grounded measures of an attention head’s functional output.\n\n2. The discussion of intercorrelations among inactive heads defined under different criteria across model sizes, families, and training stages (SFT/RLHF/DPO) is both novel and compelling. This comparative perspective is the paper’s key advantage over prior studies, which typically characterize head inactivity within each model in isolation. By analyzing how these relationships generalize across architectures and alignment methods, the paper contributes a broader, more systematic understanding of attention head behavior.", "weaknesses": "1. Despite its novelty, the proposed taxonomoy of inactivity criteria seem highly ad-hoc and disorganized, due to following reasons:\n\n   **Problematic definition**: the author uses a rather unorthodox definition of the attention head output, i.e. as the convex combination of value vectors weighted by attention coefficients prior to multiplication with the output projection matrix. This would result in a $d_v$ (dimension of value embedding) dimensional vector, which diverges from the more widely-accepted and -used circuit-based definition in [1] which multiplies the $d_v$ dimensional vector futher with the corresponding $R^{d_v \\times d}$ dimensional slice of the projection matrix.\n\n   **Significant conceptual overlap between criteria**: As a direct consequence of the problematic definition, there is significant conceptual overlap and redundancy between the value-vector-based criteria and head-output-based criteria, since the norm head output at a token position $i$ could be trivially bounded by the maximum (and minimum) norm of value vectors from position 1 to $i$. Thus it is very likely that a head with low average value vector norm will have low average head output norm/last token head output norm and vice versa. Indeed the paper's observation of a 0.55 IOU value between average head output norm and average value vector norm demonstrates this redundancy.\n  \n   **No justification for the inclusion and exclusion standards**: The authors made no justifications for why they include/exlucde certain criteria in their taxonomy. For instance, since they included the last token head output norm, they could have easily included the last token value vector norm as well. But instead they only included the first token value vector norm, without explaining the rationale.\n\n   **Disorder related to normalization strategies**: The authors included layer-normalized version of their proposed 6 criteria into the taxonomy, without explaining the reason for this. They further included a head-normalized version specifically for the last token head output norm, but not for the other metrics.\n\n2. The paper spent little efforts in analyzing the properties of the identified inactive heads, which could include the heads distribution across layers, their activation/attention weights patterns for different kinds of inputs, etc. Moreover, the paper only discussed the accuracy consequences of the ablation of the identified heads, without any qualitative analysis of how the ablation affects model outputs, or any fine-grained quantitative analysis of how the ablation affects the model's full forward computation process (hidden states etc). These shortcomings limit the mechanistic depth of the paper's analysis.\n\n3. The writing of certain parts of the paper appears perplexing. For instance, lines 400-403 appear completely irrelevant to the context. There are also grammatical issues here and there. For instance line 420: \"Every model and score function combination produce (should be produces)...\"\n\n[1] Elhage, Nelson, et al. \"A mathematical framework for transformer circuits.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q94pxN87an", "reviewer": "Reviewer_DGdn", "summary": "This paper proposes a comprehensive perspective for detecting inactive heads in Transformers and identifies Average Head Output Norm (AHON) as the most effective indicator, thereby enhancing our understanding of inactive attention heads.", "questions": "1.\tCould the authors further clarify the significance of dynamically detecting inactive heads?\nBased on my understanding, this dynamic pruning is not directly applicable for inference efficiency, since the pruning is not static.\nIf the main goal of the paper is to enhance our understanding of inactive attention heads in Transformers, what is the broader significance of detecting them dynamically? What insights or implications can we gain from this beyond efficiency?\n2. I am also curious whether the set of dynamically inactive heads changes significantly with different levels of context or question difficulty？\n3. Could the authors comment on whether the presence of “inactive” heads should be viewed as a normal characteristic of the model rather than a deficiency to be mitigated? For instance, in simpler contexts, certain heads might naturally remain unused, whereas in more complex inputs, more heads could become activated. Do the authors expect such adaptive activation behavior, and would they consider it a potentially desirable property?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is well organized and easy to follow.\n2.\tThe conclusion is clear and direct, effectively demonstrating that the Average Head Output Norm (AHON) serves as a reliable indicator for identifying inactive heads.", "weaknesses": "1.\tWhile the paper arrives at a clear and well-supported conclusion, it lacks further discussion on how the proposed detection method could be effectively utilized in practice. This limitation raises questions about the extent of the paper’s contribution beyond.\n2.\tThe experiments are conducted on a limited set of datasets, which may restrict the generalizability of the findings to broader domains or tasks.\n3.\tAlthough the proposed set of metrics is comprehensive, the paper lacks deeper analysis and interpretation of the observed results. The integration of these indicators might not yet be sufficiently thorough, and the underlying causal explanations for the findings remain somewhat unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)