Which Attention Heads Matter for In-Context Learning?
Authors: Kayo Yin
Jacob Steinhardt
Venue: ICML 2025 poster (2025)
Keywords: interpretability
in-context learning
large language models
mechanistic interpretability
induction heads
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: We compare the role of induction heads and function vector (FV) heads in in-context learning (ICL), we find that FV heads drive most of few-shot ICL which challenges the popular belief that induction implements ICL.

Abstract: Large language models (LLMs) exhibit impressive in-context learning (ICL) capability, enabling them to generate relevant responses from a handful of task demonstrations in the prompt. 
Prior studies have suggested two different explanations for the mechanisms behind ICL:
induction heads that find and copy relevant tokens, and function vector (FV) heads whose activations compute a latent encoding of the ICL task.
To better understand which of the two distinct mechanisms drives ICL, we study and compare induction heads and FV heads in 12 language models. Through detailed ablations, we find that few-shot ICL is driven primarily by FV heads, especially in larger models. We also find that FV and induction heads are connected: many FV heads
start as induction heads during training before transitioning to the FV mechanism. This leads us to speculate that induction facilitates learning the more complex FV mechanism for ICL.

Which Attention Heads Matter for In-Context Learning?
Kayo Yin Jacob Steinhardt UC Berkeley
Abstract
Pythia 6.9B
Large language models (LLMs) exhibit impres- sive in-context learning (ICL) capability, enabling them to perform new tasks using only few demonstrations in the prompt. Two different mechanisms have been proposed explain ICL: induction heads that find and copy relevant to- kens, and function vector (FV) heads whose acti- vations compute a latent encoding of the ICL task. To better understand which of the two distinet mechanisms drives ICL, we study and compare induction heads and FV heads in 12 language models. Through detailed ablations, we discover that few-shot ICL performance depends primarily FV heads, especially in larger models. In ad- dition, uncover that FV and induction heads are connected: many FV heads start as induction heads during training before transitioning to the FV mechanism. This leads us to speculate that in- duction facilitates learning the more complex FV mechanism that ultimately drives few-shot ICI
ICL. Accuracy
0.4
Clean
Random
Induction
0.2
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Ablation (%)
(a) ICL accuracy of Pythia 6.9B across different percentages of heads ablated.
Head 8.17 in Pythia 6.BB
0.02
Induction score
0.4
FV score
0.01
0.2
-0.00
2
23
2
2
214
2
Training steps
(b) Induction score (bluc) and FV score (pink) of an FV head during training.
1. Introduction
Figure 1. (a) Ablating function vector (FV) heads significantly degrades few-shot in-context learning (ICL) accuracy, while ablat- ing induction heads has minimal impact beyond ablating random heads. (b) Evolution of an FV head during training, demonstrating high induction scores earlier in training that decrease as FV score emerges. This pattern suggests induction may serve as precursor for FV mechanism.
One of the most remarkable features of large language mod- els (LLM) is their ability to perform in-context learning (ICL), where they can adapt to various new tasks using only context given in the prompt at inference time. This capability has become crucial to adapt pre-trained LLMs to specific tasks, sparking significant research into its underly- ing mechanisms (Olsson et al. 2022; Akyürek et al., 2023 von Oswald al. 2023).
to be the primary mechanism behind ICL in LLMs OOssson ct al. 2022 Singh et al. [0224] Crosbie & Shutova 2024
To date, two key mechanisms have been associated with ICL, each supported by different lines of evidence. First, induction circuits (Elhage et al., 2021) were hypothesized
Dong et al. 2023 Induction circuits operate by identifying previous occurrences of the current token in the prompt and copying subsequent token. More recently, (odd et al.) (024g) and Hendel et al. (2023) propose the existence of function vectors (FV). FVs are compact representation of task extracted from specific attention heads, and they can be added to model's 's computation to recover ICL. behavior without in-context demonstrations.
Correspondence to: Kayo Yin <kayoyin@ berphine@eeeeeeeedd....... Jacob Steinhardt <jsteinhardt@@ wewernddeeeeeeeeeed..d......
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
CCode and data: httpp:://github..cmm/aaa///
To resolve whether one or both mechanisms drive ICL in transformer LLMs, we conduct comprehensive study of
ieadheads

---

Which Attention Heads Matter for In-Context Learning?
Table Summary of findings in this work, where represents findings with evidence directly shown by our experiments and represents conjectures that our results suggest.
Findings
Evidence
Section
Contribution
Induction heads and FV heads are distinct.
Context-setting
Induction scores and FV scores correlated.
ContexI-setting
Ablating FV heads hurts few-shot ICL accuracy more than ablating induction heads. Some FV heads evolve from induction heads during training.
Main finding
Main finding
FV heads implement more complex abstract computations than induction heads. Induction heads serve as a stepping stone for models to develop FV heads
Speculation
Speculation
the attention heads implementing these mechanisms (termed induction heads and FV heads) across 12 decoder-only trans- former models ranging from 70M to 7B parameters (Table 2) and 45 natural language ICL tasks (listed in Appendix A.8). Our analysis reveals several key findings.
ings offer broader lessons for model interpretability re- search. They highlight how correlations between related mechanisms can lead illusory explanations (e.g. the con- founding effect of the correlation between induction and FV heads), and the choice of definitions may lead to different conclusions (e.g. different metrics to measure ICL), Addi- tionally, our results challenge strong versions of universality the difference between the importance of FV heads and induction heads shifts significantly with model scale ($4)
First, we verify that there a difference to explain, i.e. that induction and FV heads are indeed distinet ($)). Across models, there is low or zero overlap between induction and FV heads. These heads also have distinct characteristics: induction heads generally appear in slightly earlier layers than FV heads, and emerge significantly earlier during train- ing, On the other hand, there are correlations in behavior: FV heads behave more similarly to induction heads than a random head from the same network, and vice versa.
Table 2. Models studied this work. We use huggingface imple- mentations (Wolf et al. (220) for all models. We report the number of parameters, number of layers [T], and number of attention heads for each model.
Second, through ablation studies ($4). we demonstrate that FV heads are the primary drivers of ICL performance. Re- moving FV heads substantially degrades ICI. task accuracy, while removing induction heads has limited effect (Figure Ia). This effect is consistent across all 12 models we studied, and is more pronounced in larger models ($4). Interestingly, this challenges the prevailing view of induction heads as the key mechanism for few-shot ICL (Olsson et al. 2022
Model
Param.
|
[a|
Pythia (Biderman et al. 2023
70M
48
160M
12
144
410M
24
384
IB
16
128
1.4B
24
384
Crosbie Shutova 2024: Dong et al. 2023).
2.8B
32
1024
Third, reconcile findings with previous work by iden- tifying three key methodological differences ($7): earlier studies used a different metric for ICL that does not strongly track few-shot performance, did not account for correlations between FV and induction heads, and sometimes focused on small models. We find the choice metric is the most significant factor, discussed $7 and demonstrated by experiments in $4
6.9B
32
1024
GPT-2 (Rafford et al. 2019
117M
12
144
345M
24
384
774M
36
720
1.6B
48
1200
Llama 2 Touvron et al. 2023
7B
32
1024
Finally, by analyzing training dynamics ($5). we uncover surprising developmental relationship: many induction heads evolve into FV heads during training, but the reverse never occurs (Figure [b). This leads us to speculate that induction heads facilitate learning the more complex FV heads for ICL- the FV mechanism more effective per- forming ICI., and therefore eventually replaces the simpler induction mechanism |6
2. Background
We present comparative analysis of two mechanisms pro- posed to explain ICL: induction heads (Elhage et al. 2021 Olsson et al. (022) and FV heads (Todd et al. 2024 Hendel et al. 2023). In this section, we first present the different conceptualizations of ICL. Then, we provide the definitions of induction heads and FV heads.
Aside from clarifying the drivers of few-shot ICL, our find-
2

---

Which Attention Heads Matter for In-Context Learning?
Pythia 160M
Pythia 1B
Pythia 6.9B
0.45
Induction score
0.04
0.10
FV score (X)
0.45
0.04
0.42
0.40
0.03
0.08
0.02
0.35
0.06
0.40
0.02
0.41
0.04
5
10
0
10
15
0
5
10
15 20 25 30
Layer
Layer
Layer
Figure 2. Location of induction heads (blue) and FV heads (pink) in model layers. The average layer of induction and FV heads are shown in blue and pink dotted lines respectively. Most induction heads appear in carly-middle layers, FV heads appear at layers slightly deeper than induction heads.
2.1. In-context learning
ICL accuracy while showing reduced token-loss difference, and vice versa. This distinction helps explain conflicting conclusions about ICL mechanisms in prior studies.
ICL broadly refers the ability of a machine learning model, especially LLMs,, adapt its behavior and perform new tasks by leveraging contextual information provided with the input prompt, without any changes to its underlying parameters. This allows the model to dynamically interpret and respond to a wide range of tasks solely based on the information given at inference time, such as instructions or examples, making it highly adaptable. Two distinct concep- walizations of ICL have emerged the literature, reflecting different operationalizations of the phenomenon:
2.2. Induction heads
Induction heads were first identified by Elhage al. (2021 and extensively studied by Olsson et al. (022) the mech- anism behind ICL. They are attention heads that identify repeated patterns in the input: when processing token, they attend to the token that followed a previous occurrence of the same token, predicting will appear next.
ICI as few-shot task accuracy. This conception, popular- ized broow et al. 2020). defines ICL through few-shot performance - the model's accuracy on novel tasks when provided with a few demonstrations in the prompt. The majority of existing works studying ICI, focuses on this conception of ICL (Dong et al. 2023 Akyürek al. 2023 Wei et al., 2023 Bansal al. 2023 Crosbie & Shutova
The initial evidence for induction heads' role ICL came from Olsson al. (2022). who studied small attention-only models (1-3 layers). They observed that the emergence of in- duction heads during training coincided with improvements in ICL ability measured as the difference between the loss at the 500th versus 50th token in the context. Their ablation studies showed that removing induction heads impaired this metric.
2024), and ICI. is often used synonymously with this few- shot ability at inference time. We adopt this definition of ICL in this paper as well, as it currently the most standard in the literature. Unless specified otherwise, "in-context learning" this paper will refer to this definition.
To identify and analyze induction heads, we measure their induction scores using the TransformerLens framework (Nanda & Bloom| 2022). For each attention head a, we compute its induction score on a synthetic sequence con- structed by repeating a uniformly sampled random token sequence: r = "102 The induction score is defined as:
ICL context-dependert loss reduction. On the other hand, few-shot learning could be interpreted as subset of ICL, where ICL. is defined more broadly as the model's 's ability to reduce loss at later tokens by using the context of earlier observations (Kaplan et al. 2020) Olsson et al. 2022 Lampinen et al. 2024). In this conception, ICL is measured by the difference between the model loss on earlier tokens and later tokens in the sequence. This difference was pre- viously called "ICL score" in Olsson al. (2022), , to avoid confusion we call this "token-loss difference".
50
S1(a, Σ
arg-
where dy/-r144 represents the attention weight that head places on token ri+1 when processing token r/- For each attention head each model, we take the mean induction score over 1000 samples random sequences r, normalized by total attention mass to obtain value between 0 and
While these two conceptualizations of ICL have often been conflated, our experiments demonstrate these metrics can di- verge significantly ($4). Models may maintain high few-shot
3

---

Which Attention Heads Matter for In-Context Learning?
15
96
Induction %ile
95
Overlap (%)
10-
FV %ile
94
90
92
85
90
10
109 Model size
10
10 Model size
10a
109
Model size
Figure 3. Percentage of head overlap between induction and FV heads (left) in green, and between induction and randomly sampled heads in gray. Percentile of induction of FV heads (center). Percentile of FV score of induction heads (right). There is little overlap hetween induction and FV heads, but FV heads have relatively high induction scores and vice versa.
2.3. FV heads
represent distinct mechanisms, while noting important cor- relations between them.
Function vectors (FV) were concurrently discovered by Todd et al. (2024) and Hendel et al.] (202)). They repre- sent a different mechanism for ICL: FVs are compact vector representations of ICL tasks that can be extracted from specific attention heads and added back into the language model's computations reproduce ICL behavior. We re- fer to the attention heads that encode and transport these function vectors FV heads.
3.1. Head locations
We begin by examining the location of the top induction and FV heads within the models. Figure shows the layers where the top 2% induction heads and FV heads appear in three representative Pythia models (see Appendix (A. for all 12 models). certain experiments such as this one, we need to decide threshold differentiate between mean- ingful induction or FV heads and the long tail of attention heads that perform neither induction nor the FV mechanism. The 2% threshold was carefully chosen following previous work by (Todd et al. (2024), and verified with the shape of the distribution of induction and FV scores (Appendix A.2) that this threshold indeed meaningfully distinguish the important heads from others.
To identify FV heads, we employ the casual mediation anal- ysis framework from [odd et al. (2024). For each ICL task t in our task set T, where is defined by a dataset Ps of in-context prompts pt E consisting of input-output pairs (ara, y:), we:
1. Compute the mean activation of an attention head a over prompts in P,: af P. Extr,, a(p{)
In general, induction heads seem appear in early-middle layers and FV heads appear in slightly deeper layers than induction heads. although this trend is not statistically sig- nificant in all models (Appendix[A.I) induction and FV heads not fully overlap and are indeed distinct mechanisms. Moreover, the deeper locations of FY heads may indicate they implement more abstract com- putations than induction heads, though this interpretation remains speculative.
2. Create corrupted ICL prompts i E P. by randomly shuffling the output labels yi while maintaining the same inputs 2;
This suggests that
3. Measure each head's function vector score (FV score) as its causal contribution to recovering the correct out- put for the input given corrupted examples (æ; Ji) when its activation pattern is replaced with the mean task-conditioned activation ā':
Syv (alpt) f a') [w) (V)
3.2. Overlap between induction and FV heads
For each attention head, we take the mean FV score across 37 natural language ICI. tasks from (Todd et al.] 2024) (Appendix[A) A.8)) using 100 prompts per task. Each prompt contains 10 input-output demonstration pairs followed by a single test instance.
To further examine how distinct induction and FV heads are, we analyze the extent of the overlap between the two types of heads in two ways.
First, we measure direct overlap- the percentage of heads that rank in the top 2% for both mechanisms: 100x [LHPP where IH and FV represent the sets of top induction and FV heads respectively. The results show minimal overlap: seven of our twelve models show zero overlap, with the remaining models showing only 5-15% overlap (Figure (3] left). This leads us to conclude that induction heads and
| [7
3. Induction heads and function vector heads are distinct but correlated
Before analyzing the relative contributions of induction and FV heads to ICI. performance, we first establish that these

---

Which Attention Heads Matter for In-Context Learning?
FV heads are mostly distinct. This motivates our subse- quent experiments that study induction and FV heads two separate phenomena.
output 1/g. We summarize the full set of ICL tasks we study in Appendix A.8
To avoid leakage between ICL tasks used to identify FV heads and those used to evaluate FV head ablations, we randomly split the 37 ICL tasks from Todd et al. (0024) into 26 tasks used to measure FV scores of heads, and 11 tasks to evaluate ICL performance. We also add new tasks for ICL evaluation: 4 tasks are variations of tasks in Todd et al. and 4 are binding tasks from |Feng & Steinhardt 2024 In total, we evaluate ICL accuracy on 19 natural
However, more nuanced pattern emerges when we com- pute the percentile of the induction score of the top 2% FV heads (Figure center) and the percentile of the FV score of the top 2% induction heads (Figure right).. In most models, FV heads are at around the 90-95th percentile of induction scores, and vice versa. Therefore, although there is little overlap between the sets of induction and FV heads, induction and FV scores are correlated: FV heads have high induction scores relative to other attention heads, and induction heads have relatively high FV scores
2024
language tasks, with 100 prompts per task.
Token-loss difference. In Olsson et al. (2022), ICL perfor- mance is measured by the difference between the model loss of a token appearing early in the context (e.g., the 50th to- ken) and later in the context (e.g., the 500th token). We also report results using this metric to compare with previous work.
4. Function vector heads drive in-context learning
Having established that induction and FV heads represent distinct mechanisms, we now investigate their relative causal importance for ICL through systematic ablation studies. Our analysis focuses primarily few-shot ICL accuracy while also examining effects on metrics used in previous work for comparison.
We measure token-loss difference by taking the loss of the 50th token in the input prompt minus the loss of the 500th to- ken in the promp averaged over 10,000 randomly sampled examples from the Pile dataset (Gao et al. 202T.
4.2. Results
4.1. Method
We evaluate the impact of ablating different proportions (1-20%) of the top attention heads based on induction FV score, across all models. We compare against two baselines: model performance with no ablation ("clean") and with ablations of randomly sampled heads ("random")) Figure shows results for three representative models, where few- shot ICL accuracy is averaged over the 19 evaluation tasks. We provide comprehensive results across all models and ICL accuracy broken down by task in Appendix (A.9. We plot the average induction/FV scores and percentiles of the heads preserved during ablations in Appendix A.11
Ablation. To assess the causal contribution of different at- tention heads, we measure how ICL performance changes when specific heads are disabled. We use mean ablation, where we replace each target head's output with its average output across our task dataset (described later sections). This approach avoids the out-of-distribbtion effects associ- ated with zero ablation (Hase et al., 2 Wang et al. 2023 Zhang& & Nanda 2024). though our findings remain robust across different ablation methods (Appendix A.4)
To control for the correlation between induction and FV heads identified in Section 3 we introduce ablation with "exclusion": when ablating n FV heads, we select the top n heads by FV score that are not in the top 2% by induc- tion score, and vice versa. This helps isolate the unique contributions of each mechanism.
Our initial ablation experiments, shown in the top row of Figure removed heads based on their scores without con- sidering the potential overlap between induction and FV heads. These results revealed that ablating FV heads caused greater degradation in few-shot ICL performance compared to ablating induction heads, with this disparity becoming more pronounced in larger models. We also find that ablat- ing induction heads has more effect on ICL performance than random. The effect of ablating induction heads con- verges to the effect of ablating FV heads as we increase the number of heads ablated.
Few-shot ICL accuracy. We primarily evaluate ICL perfor- mance on a series of few-shot ICL. tasks. Each ICL task is defined by a set of input-output pairs (z,, y:). The model is prompted with 10 input-output exemplar pairs that demon- strate this task, and one query input xy that corresponds to target output Vg that is not part of the model's 's prompt. We compute the model's accuracy in predicting the correct
However, the convergence noted above may be due to an increasing overlap in the set of heads ablated in the induction head and FV head ablations (Appendix A. To address
'In our main analysis, do not nely the correlation between the distribution of induction scores and FV scores across the full set of attention heads because there is a long tail of attention heads with low scores on both induction and FV. For completeness, we plot the induction and FV scores of all heads in Appendix A.2
We invert the difference used in Olsson al. 2022 so higher scores indicate better ICL performance.
5

---

Which Attention Heads Matter for In-Context Learning?
Pythia 160M
Pythia 1B
Pythia 6.9B
ICI. Accuracy
0.4
0.2
0.4
0.2
0.1
0.2
10
15
20
5
10
15
20
5
10
15
20
ICL. Accuracy
0.25
0.4
0.4
0.20
0.2
0.15
0.2
5
10
15
20
5
10
15
20
5
10
15
20
TI. difference
0.6
0.5
0.4
0.5
0.4
0.2
0.4
0.3
0.0
0.3
5
15
20
5
15
20
5
10
15
20
Ablation (%)
Ablation (%)
Ablation (%)
Clean
Random
Induction
FV
Ablation with exclusion
Figure 4. Top: Few-shot ICL accuracy after ablating induction and FV heads. Center: Few-shot ICL accuracy after ablating non-FV induction and non-induction FV heads. Bottom: Token-loss difference after ablating non-FV induction and non-induction FV heads. Ablating FV heads lead a bigger drop in ICL accuracy, especially in larger models. Ablating induction heads with low FV scores does not significantly affect ICL accuracy. ICL accuracy and token-loss difference behave differently.
this, we conduct a second set of experiments using ablation with exclusion, shown in the center row of Figure
affect token-loss difference more than FV head ablations. This experiment primarily demonstrates that few-shot ICL accuracy and token-loss difference measure two very dif- ferent things. particular, we find instances where ICL accuracy and token-loss difference diverge: when ablate random heads and induction heads, the model has high ICL accuracy but low token-loss difference. We speculate that this is due to the model preserving its ICL abilities (thus the high ICL accuracy), but loses other abilities that are also associated with learning important signals from con- text (thus the low token-loss difference). These contrasting results between the two metries help reconcile apparently contradictory findings in existing literature.
When ablating induction heads while preserving the top 2% FV heads, we observe minimal impact on few-shot ICL performance comparable to random ablations in models exceeding IB parameters. Conversely, ablating FV heads while preserving induction heads continues to significantly impair ICL performance. The performance gap between FV and induction head ablations widens with model scale, sug- gesting that the observed effects of induction head ablations without exclusion are primarily due to heads exhibiting both induction and FV properties.
These ablations suggest that the contributions of induction heads to ICL in the top row of Figure 4 mostly come from heads that are both induction and FV heads, and that FV heads matter the most for few-shot ICL: as long as the model preserves its top 2% FV heads, can perform ICL. with reasonable accuracy even if we ablate induction heads.
5. FV heads evolve from induction heads
Finally. to further understand how these two families of attention heads develop, we analyze their evolution dur- ing model training. We examine attention heads across 8 intermediate training checkpoints in 7 Pythia models.
The bottom of Figure presents the effects of abla- tions with exclusion on token-loss difference. In smaller models (below 160M parameters), neither ablating induc- tion nor FV heads shows significant impact to token-loss difference compared to random ablations. However, in mod- els with over 345M parameters, induction head ablations
5.1. Induction and FV strength during training
To measure the general strength of induction and FV mech- anisms during training, we plot the mean induction and FV scores of the top 2% induction and FV heads at each model
6

---

Which Attention Heads Matter for In-Context Learning?
Induction score
Pythia 160M
Pythia 1B
Pythia 6.9B
0.01
FV score
0.25
0.02
0.25
0.05
0.25-
0.00
0.00
0.00
0.00
0.00
0.00
10
103
105
10
10
10s
10
10
10a
Training steps
Training steps
Training steps
Induction score
FV score
ICL accuracy
Figure Evolution of induction and score averaged over top 2% heads across training. Induction score rises sharply, then plateaus. FV score rises slightly later and gradually increases. ICL accuracy rises around the same time as induction and gradually increases.
checkpoint, along with few-shot ICL accuracy (Figure 5 We include plots for all Pythia models in Appendix A.13
6. Interpretation and discussion
Our investigation revealed several key insights about the relationship between induction and FV heads in transformer models and their effect on ICL. While these mechanisms are distinct, they show notable correlation ($3) FV heads con- sistently appear in deeper layers and emerge later in training compared to induction heads ($3..I). Our ablation studies demonstrated that FV heads are crucial for few-shot ICL performance, particularly in larger models, while induction heads have comparatively minimal impact ( 4 Further- more, we observed multiple instances of heads transitioning from induction to FV functionality during training, but never the reverse 15.1. We propose two working conjectures to explain these empirical findings more broadly, and consider arguments for and against them.
Our analysis reveals a consistent pattern across all Pythia models: induction heads emerge early in training, at around step 1,000 out of 143,000, while FV heads appear sub- stantially later at around step 16,000. The development of these heads shows distinct characteristies as well induction scores exhibit sharp initial rise followed by plateau or slight decline, whereas FV scores demonstrate gradual but sustained increase from step 16,000 through the end of train- ing. This temporal asymmetry suggests that induction heads represent a simpler mechanism that models can acquire ear- lier, while FV heads embody a more complex mechanism that requires extended training. In addition, we observe that in all models, few-shot ICL accuracy begins to improve around the same time as when induction heads appear, and continues to gradually increase throughout training.
Our first conjecture (C1) posits that induction heads serve as precursots to FV heads. Under this interpretation, induc- tion heads serve as a stepping stone for models to develop the more sophisticated FV mechanism. The mechanism un- derlying induction heads is simpler and easier to learn, but this method does not fully solve more complex ICL, prob- lems. As FV heads emerge and prove to be more effective at difficult ICL tasks, they gradually supersede the simpler induction mechanism.
5.2. Evolution of individual heads during training
To gain more granular insights into head development, we in- vestigate the evolution of individual attention heads through- out training. Figure 6 the induction scores (top row) and FV scores (bottom row) the top 2% induction and FV heads across training steps. Individual heads are represented by continuous lines, with line opacity corresponding to their final induction or FV scores.
Several lines of evidence support this conjecture. First, we observed multiple FV heads that initially displayed strong induction behavior before transitioning to FV functionality. The subsequent decline in induction scores suggests these heads abandon the simpler mechanism once the more ef- fective FV capability develops. The unidirectional nature of this transition we never observe induction heads with initially high FV scores further supports this interpretation.
A striking pattern emerges across all models: many heads that ultimately become strong FV heads initially exhibit high induction scores, emerging around the same time as dedicated induction heads. These proto-FV heads initially achieve induction scores comparable to those of special- ized induction heads. However, training progresses, their induction scores gradually decline while their FV scores increase. Importantly, this pattern is unidirectional; we found no instances of induction heads that develop signifi- cant FV capabilities during training, as evidenced by their consistently low FV scores throughout training. This sug- gests many FV heads evolve from induction heads during training, but not vice versa.
Additionally, the minimal effect of ablating pure induction heads (those with low FV scores) on few-shot ICL perfor- mance suggests their role becomes less critical once FV heads develop. To further verify this, future work could ex- plore how removing induction heads during training could impact the development of FV heads. However, CI does not fully explain the existence of FV heads with low induction
7

---

Which Attention Heads Matter for In-Context Learning?
Pythia 160M
Pythia 1B
Pythia 6.9B
Induction score
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
24
2
2 24
24
2
21
2
29
2
FV score
0.10
Induction heads
0.04
0.04
FV heads
0.02
0.05
0.02
0.00
0.00
0.00
24
29
24
24
2
2
24
2
2
Training steps
Training steps
Training steps
Figure 6. Evolution of induction scores (top) and FV (hottom) of individual induction and FV heads training. Certain FV heads have high induction score earlier in training; the reverse is not true for induction heads.
scores throughout training.
7. Related work
Cl also aligns with our observations about architectural and training dynamics. FV heads consistently emerge later in training and appear in deeper layers, consistent with them implementing a more complex computation. Furthermore, we observe that in all models, few-shot ICL accuracy begins to rise around the same time when induction heads appear, and continues to gradually increase until the end of training. Since ICI. accuracy continues to improve even after the formation of induction heads, we speculate that the sharp emergence of induction heads contributes to an initial rise ICI. performance, but the gradual formation of FV heads drives the further improvements in ICL. This reinforces our conjecture that induction heads may serve a stepping stone for FV
In this paper, we compared mechanisms that have been previously proposed to explain ICL: induction heads (EP
hage et al. 202 Olsson et al., 2022) and FV heads (Todd
et al. 2024 Hendel et al. (023). ). A line of previous work have attributed the mechanism behind ICL to induction heads. Olsson et al. (2022) studied small attention-onle models with 1-3 layers and showed that the emergence of induction heads co-occur with the emergence of ICL perfor- mance. We observe similar co-occurence in $5.1 They also showed that ablating induction heads decreases token- loss difference, similarly to our results in $4.2 Singh et al. (2024) expanded on this by introducing causal framework to manipulate activations in toy models, and identified three subcircuits that drives induction head formation, and conse- quently, the phase transition in ICL.
An alternative conjecture (C2) suggests that FV heads are combination of induction and another mechanism. This conjecture proposes that heads that appear to "transition" from induction FV functionality may actually be polyse- manticheads, implementing both mechanisms and possibly others. Their measured induction scores decline their attention patterns diversify to support multiple mechanisms.
Crosbie & Shutova 2024) performed ablation studies LLMs of size 8B and 20B parameters, and Bansal al. 2023 on an LLM of size 66B, to demonstrate that induction heads are important for ICL in large models as well. We observe similar results in the top row of Figure 4 where we performed ablations without accounting for the overlap between induction and FV heads.
While C2 explains the correlation between induction and FV mechanisms as arising from shared underlying mecha- nisms, faces significant challenge: our ablation studies show that removing monosemantic FV heads (those without significant induction scores) substantially hurts ICL perfor- mance. This is difficult reconeile with C2's prediction that pure FV heads should be less critical if the key functionality depends on combined induction FV mechanisms.
Further works explored the ICL capabilities and internal mechanisms of transformer models through multiple com- plementary lenses. Edelman et al. (2024) demonstrated how models develop statistical induction heads for Markov chain prediction. Xie et al. 2022) frames ICL as implicit Bayesian inference when pretrainng data exhibits latent coherence. Garg al. (2022) empirically proved transformer models can learn linear functions and complex classes in-context.
8

---

Which Attention Heads Matter for In-Context Learning?
Guo et al. 2024) extended this to compositional tasks and observed mechanisms such copying behaviors within transformers.
driver of ICL, our evidence demonstrates that function vec- tor (FV) heads play a more crucial causal role in few-shot ICL performance. We attribute previous misconceptions to two key factors: the conflation of few-shot ICL with token- loss difference, and not accounting for the overlap between induction and FV heads.
Reconciling divergent findings
While induction heads and FV heads have been proposed by their respective works as the mechanism behind ICL, our side-by-side analysis of induction and FV heads reveals that FV heads seem to primarily contribute to ICL performance. We believe that the main reason for the divergence between our result and previous work lies in several intuitively related concepts in the literature that are assumed be the same.
Remarkably, although induction and FV mechanisms ap- pear implement two distinct processes, we also observe interesting interplay between the two types of heads: in- duction and FV scores are correlated, and many FV heads are "former" induction heads with high induction scores earlier in training. This observation supports the conjecture that induction heads serve as precursors to FV heads: the simpler induction mechanism provides an initial foundation for ICL, from which the more sophisticated FV mechanism eventually emerges.
First, ICL is used synonymously with few-shot learning in most of the literature, while other works conceptualize ICL as loss reduction in tokens appearing later in a sequence. The two distinct definitions of ICL have often been conflated to refer to the same phenomenon. However, our experiments reveal that the two ICL metrics capture different phenomena: FV heads strongly influence few-shot ICL accuracy but not token-loss difference, while induction heads show the oppo- site pattern 4..) This divergence of token-loss difference from few-shot ICL performance accounts for much of the apparent contradiction with previous findings.
Our investigation also yields important methodological in- sights for the broader field of model interpretability. First, seemingly equivalent definitions of model capabilities (such as few-shot accuracy versus token-loss difference) can lead to substantially different conclusions. Second, studying mechanistic components in isolation may produce mislead- ing results when these components share overlapping behav- iors, as demonstrated by the confounding effects of ablating heads that exhibit both high induction and FV scores. We thereby recommend future works to carefully define the type of ICL studied, and consider the interactions between different circuits.
Second, we find that induction heads and FV heads are correlated ($3). a confound not previously controlled for. Initial ablation studies, including ours 4 showed that re- moving induction heads significantly degrades few-shot ICL accuracy (Crosbie & Shutova, 2024; Bansal et al.) (023). However, when we control for this correlation by only ab- lating induction heads with low FV scores, their impact becomes comparable to random ablation. In contrast, ablat- ing FV heads with low induction scores still significantly degrades ICL performance. This suggests that previous studies may have attributed to induction heads effects that actually stemmed from FV-like behavior in subset of in- duction heads.
Furthermore, our results challenge strong versions of the universality hypothesis in interpretability. While both in- duction and FV heads contribute meaningfully to few-shot ICL in smaller models, their relative importance diverges with scale - FV heads become increasingly crucial while in- duction heads' impact approaches that of random ablations. This scale-dependent behavior suggests that mechanisms may vary across model architectures.
Third, scale matters. Previous work establishing induction heads as the key ICL mechanism (Olsson et al. 2022 Singh le al., 2024) focused on small models enable detailed mechanistic analysis. However, we find that the relative importance of FV heads increases with model scale. In our smallest model (70M parameters), induction and FV heads have similar causal effects on few-shot ICL ($A.3). but this does not hold true in larger models, which highlights the importance of studying these phenomena across different model scales.
These findings prompt several important questions for fu- ture research. If induction heads indeed serve as precursors to FV heads, what makes this necessary? What role do the remaining induction heads serve in fully trained mod- els? Are there additional mechanisms that provide an even more complete explanation of ICL capabilities? We also leave practical applications of our findings to future work evidence of the higher importance of FV suggests that ICL could be optimized in smaller models by using training methods or architectures that promote the formation of FV heads.
8. Conclusion
Acknovledgements
Our research challenges the prevailing understanding of in- context learning mechanisms in transformer models. While induction heads have been widely considered the primary
We thank Jiahai Feng, Neel Nanda, Robert Kirk, and Anish Kachinthaya for their helpful feedback, and anonymous
9

---

Which Attention Heads Matter for In-Context Learning?
reviewers for useful comments. KY is supported by the Vitalik Buterin Ph.D. Fellowship in Al Existential Safety.
context learning markov chains. In Advances in Neural In- formation Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver. BC, Canada, December 10 15, 2024, 2024.
Impact statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
References
Feng, J. and Steinhardt, J. How do language models bind entities context? In The Twelfth International Confer- ence on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.
Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh Inter- national Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
Gao, I.., Biderman, S., Black, S., Golding, I.., Hoppe, T, Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N.. Presser, S., and Leahy, C. The Pile: An 800gb dataset of diverse text for language modeling. ArXiv preprint, 2021.
Bansal, H., Gopalakrishnan, K., Dingliwal, S., Bodapati, S., Kirchhoff, K., and Roth, D. Rethinking the role of scale for in-context learning: An interpretability-bass case study at 66 billion scale. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume! I: Long Papers), 2023.
Garg, S., Tsipras, D., Liang, P., and Valiant, G. What can transformers learn in-context? A case study of simple function classes. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022. New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O' Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L.., and van der Wal, O. Pythia: A suite for analyzing large language models across training and scaling. In Inter- national Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, Proceedings of Machine Learning Research, 2023.
Guo, T., Hu, W., Mci, S., Wang, H., Xiong, C., Savarese, S., and Bai, How do transformers learn in- in-context beyond simple functions? A case study on learning with representations. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,
Hase, P., Xie, H., and Bansal, M. The out-of-distribution problem in explainability and search methods for feature importance explanations. In Advances in Neural Infor- mation Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021. NeurlPS 2021, December 6-14 14. 2021, virtual, 2021.
J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Heerbrrt-Vsss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J. Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances in Neural In- formation Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020.
Hendel, R., Geva, M., and Globerson, A. In-context learning creates task vectors. In Findings of the Association for Computational Linguisties: EMNLP 2023, 2023.
Hernandez, E., Sharma, A. S., Haklay, T., Meng. K., Watten- berg. M., Andreas, J., Belinkov, Y., and Bau, D. Linearity of relation decoding in transformer language models. In The Twelfth International Conference on Learning Repre- sentations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.
Crosbie, J. and Shutova, E. Induction heads as an essential mechanism for pattern matching in-context learning. ArXiv preprint, 2024.
Dong. Q. Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., and Sui, Z. A survey on in-context learning. ArXiv preprint, 2023.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. ArXiv preprint, 2020.
Edelman, E., Tsilivis, N., Edelman, B. L., Malach, E., and Goel, S. The evolution of statistical induction heads: In-
10

---

Which Attention Heads Matter for In-Context Learning?
Lampinen, A. K., Chan, S. C. Y., Singh, A. K., and Shana- han, M. The broader spectrum of in-context learning. 2024.
Todd, E.,, Li, M. L.., Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. Function vectors in large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 1-1,, 2024, 2024.
Lample, G., Conneau, A., Ranzato, M., Denoyer, L.,, and Jégou, H. Word translation without parallel data. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- MMay 3, 2018, Conference Track Proceedings, 2018.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. ArXiv preprint. 2023.
McGrath, T., Rahtz, M., Kramar, J., Mikulik, V., and Legg, S. The hydra effect: Emergent self-repair in language model computations, 2023.
von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov,
M. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, Proceed- ings of Machine Learning Research, 2023.
Nanda, N. and Bloom, J. Transformerlens. https
//qithub. www.e//eeaattoteeeeeee.emeemeeee TransformerLens 2022.
Nguyen, K. A., Schulte im Walde, S., and Vu, N. T. Dis- tinguishing antonyms and synonyms in a pattern- based neural network. In Proceedings of the 15th Conference of the European Chapter of the Association for Computa- tional Linguistics: Volume 1, Long Papers, 2017.
Wang. K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: circuit for indi- rect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., et al. In-context learning and induction heads. ArXiv preprint, 2022.
Wei, J., Wei, J.. Tay, Y., Tran, D., Webson, A., Lu, Y., Chen, X., Liu, H., Huang, D., Zhou, D., et al. Larger language models do in-context learning differently. ArXiv preprint, 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C.,, Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q. and Rush, A. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020.
Singh, A. K., Moskovitz, T., Hill, F., Chan, S. C. Y., and Saxe, A. M. What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An ex- planation of in-context learning as implicit bayesian infer- ence. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
Socher, R., Perelygin, A., WU, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 2013.
Zhang, F. and Nanda, N. Towards best practices of activation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Repre- sentations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.
Talmor, A., Herzig, J., Lourie, N., and Berant, J. Com- monsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.
Zhang, X., Zhao, J. J., and LeCun. Y. Character-level con- volutional networks for text classification. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-22, 2015, Montreal, Quebec, Canada, 2015.
Tjong Kim Sang, E. F. Introduction to the CoNLL-2002 shared task: maaguugge-nnependeen named entity recog- nition. In COLING-02 The 6th Conference on Natural Language Learning 2002 (CoNL.L-2002), 2002.

---

Which Attention Heads Matter for In-Context Learning?
A. Appendix
and we sample A L heads from other layers ablate.
A.1. Limitations of our work
A.6. Induction and function vector scores across models
Our findings are limited to models under 7B parameters, and we leave scaling our analysis to larger models to future work. The range of models we study also limits the complexity of the ICL tasks we studied, as we selected tasks where models have reasonable few-shot performance. To ensure that our findings apply to real-world tasks, we included realistic tasks such as translation, commonsense question answering, news topic classification. Our ablation studies also do not account for the hydra effect (McGrath et al., [003)), as measuring the extent of self-repair when ablating specific heads remains an open problem (the method in McGrath et al. (2023) applies to ablating all heads in a single layer). If the same magnitude of the hydra effect is present in our ablations of different head types, our findings on the relative importance of heads would still hold, however this is not yet proven. We leave to future work to measure the extent of second-order effects in ablations.
Our ablation studies reveal a consistent trend where FV heads are increasingly important relative induction heads for ICL performance model scale increases. To further explore this trend, we examine how induction scores and FV scores vary with model scale, and whether these scores follow similar trends to our ablation experiments.
In Figure 12 we plot the maximum and mean induction and FV scores across all heads, and mean scores of top 2% heads, for each model. The left plot in Figure [2 shows that induction scores are relatively similar across model size, with small increase in maximum induction score and a decrease in the top 2% mean induction score with model scale.
In the right plot of Figure 12, there is no clear trend between FV score and model scale, however, Pythia IB and 1.4B models have markedly higher maximum FV scores. One possible explanation is that models with high head dimen- sionality relative to total parameter count have stronger FV heads: Pythia IB and 1.4B have head dimensionality of 256 and 128 respectively (Table whereas other models with similar parameter count have only 64-80 attention head dimensions.
A.2. Induction scores vs. FV scores
In Figure 7 we plot the induction score and FV score of each attention head. We also color the heads that are determined as induction head, an FV heads, or both, using the top 2% for each score the threshold. The plots show that the 2% threshold meaningfully select important heads from the cluster of other low-scoring heads across models. Taking a higher percentage for the threshold would select heads with
We also find very low FV scores in Pythia 70M and Llama 2 models. FV scores may be low in Pythia 70M because it is too small in parameter size for FV heads to emerge. Low scores in Llama 2 compared to other models may be due to differences in architecture, and additional experiments can help confirm this. Overall, we do not recover the same trend in induction/FV scores as the trend in our ablation studies.
FV scores close to O.
A.3. Ablations
In Figure g we plot model accuracy averaged over ICL tasks across different quantities of heads ablated in each head type. In Figure 9 we plot the token-loss difference of models across different quantities of heads ablated.
For reference, also provide box plots of the full distribu- tion of induction and FV scores in Figure 13
A.7. Evaluating function vectors task execution
A.4. Random and zero ablations
To further inspect the prevalence of the FV mechanism in different models, evaluate the efficacy of FVs for ICL task execution. A successful FV triggers the model to execute the particular task the FV encodes, even when the model sees no useful in-context demonstrations of the task. First, to extract FVs, for each model we gather the top 2% attention heads with highest FV scores as the set A. Then, for each ICL task / E T, we sum the average outputs of heads in A over prompts from t and obtain the FV for the task t: FV, = EnEAd. a'.
In Figure I0 we plot model accuracy averaged over ICL tasks across different quantities of heads ablated with ran- dom ablation or zero ablation. For random ablations, we replace the head's output vector with the output vector of a randomly sampled different head. For zero ablations. we replace the head's output vector with a zero vector.
A.5. Ablating random heads at specific layers
we ablate heads randomly sampled from spe- cific layers of the model. Let I. be the number of heads in each layer, A be the number of heads we'r re ablating, and be the layer we're targeting. Then, if A < L, we sample A heads from layer& f. If A > L, we ablate all L heads in layer
In Figure II
In Figure 14 we report model accuracy averaged over 40 ICL tasks where the model performs inference on uncor- rupted prompts (clean), prompts with shuffled labels (shuf- fled), shuffled prompts with FV, added to hidden states at
12

---

Which Attention Heads Matter for In-Context Learning?
Pythia 70M
GPT-2
Pythia 160M
0.04
0.004
0.04
PV scoree
0.002
FV score
FV score
0.03
0.02
0.000
0.02
0.01
0.01
0.002
0.00
0.00
-0.004
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
Induction score
Induction score
Induction score
GPT-2 Medium
Pythia 410M
GPT-2 Large
0.08
0.03
0.06
0.02
FV score
0.03
FV score
0.04
0.01
0.02
0.02
0.01
0.00
0.00
0.00
-0.01
0.0
0.2
0.4
0.4
0.4
Induction score
Induction score
Induction score
Pythia 1B
Pythia 1.4B
GPT-2 XL
0.12
0.12
0.05
0.10
0.10-
0.04
0.08
0.03
0.06
FV scare
FV score
0.06
0.04
0.02
FV
0.04
0.02
0.02
-0.0
0.00
0.00
0.0
0.1
0.2
0.3
0.4
0.0
0.2
0.4
0.0
0.2
0.4
Induction score
Induction score
Induction score
Pythia 2.8B
Pythia 6.9B
Llama 2 7B
0.05
0.05
0.015
0.04
0.04
FV scote
0.03
score
0.03
score
0.010
0.02
0.02
0.005
0.01
0.00
Sur ob
0.000
0.00
-0.01
-0.005
0.0
0.1
0.3
0.4
0.2
0.4
0.2
0.4
Induction score
Induction score
Induction score
Induction beads
FV heads
Induction-FV heads
Figure Induction and FV scores of attention heads.
13

---

Which Attention Heads Matter for In-Context Learning?
Pythia 70M
Pythia 70M w/ exclusion
GPT-2
GPT-2 w/ exclusion
ICL Accuracy
ICI. Accuracy
ICL Accuracy
0.3
ICL. Accuracy
0.3
0.10
0.10
0.2
0.2
0.05
0.05
0.1
0.1
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Pythia 160M
Pythia 160M w/ exclusion
GPT-2 Medium
GPT-2 Medium w/ exclusion
ICL. Accuracy
TCL. Accuracy
0.25
ICL Accuracy
0.3
ICL. Accuracy
0.3
0.2
0.2
0.2
0.1
0.15
0.1
0.1
10
20
10
20
20
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Pythia 410M
Pythia 410M w/ exclusion
GPT-2 Large
GPT-2 Large w/ exclusion
ICI. Accuracy
ICL Accuracy
0.4
ICL Accuracy
ICL Accuracy
0.3
0.3
0.3
0.3
0.2
0.2
0.2
0.2
0.1
0.1
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Pythia 1B
Pythia 1B w/ exclusion
Pythia 1.4B
0.0
Pythia 1.4B w/ exclusion
ICL Ancuracy
ICL Accuracy
ICL Accuracy
Ancuracy
0.4
0.4
0.4
0.4
0.3
0.2
0.2
0.2
10
20
10
20
10
20
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
GPT-2 XL
GPT-2 XL w/ exclusion
Pythia 2.8B
Pythia 2.8B w/ exclusion
0.4
0.4
ICI, ICI, Accuracy
ICL ICL Accuracy
ICL Accuracy
ICL. Accuracy
0.3
0.3
0.4
0.4
0.2
0.2
0.2
0.1
0.2
10
20
10
20
20
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Pythia 6.9B
Pythia 6.9B w/ exclusion
Llama 7B
Llama 7B w/ exclusion
IC1. Accuracy
ICL Accuracy
ICL Accuracy
0.6
ICL Accuracy
0.6
0.4
0.4
0.5
0.4
0.2
0.2
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Clean
Random
Induction
PV
Ablation with exclusion
Figure 8. ICL accuracy after ablating induction and FV heads.
14

---

Which Attention Heads Matter for In-Context Learning?
Pythia 70M
GPT-2
Pythia 160M
difference
0.50
TL difference
0.50
difference
0.6
0.4
0.25
0.25
0.2
0.00
0.00
-0.25
15
15
5
15
Ablation (%)
Ablation (%)
Ablation (%)
GPT-2 Medium
Pythia 410M
GPT-2 Large
TL difference
0.6
TL difference
TL difference
0.5
0.4
0.4
0.2
15
15
5
15
Ablation (%)
Ablation (%)
Ablation (%)
Pythia 1B
Pythia 1.4B
GPT-2 XL
TL. difference
difference
TL difference
0.6
0.5
0.5
0.5
0.4
0.4
0.4
0.3
0.3
15
15
15
Ablation (%)
Ablation (%)
Ablation (%)
Pythia 2.8B
Pythia 6.9B
Llama 2 7B
TL difference
difference
0.5
TL difference
0.4
-2
0.4
0.3
15
15
5
15
Ablation (%)
Ablation (%)
Ablation (%)
Clean
Random
Induction
FV
Figure Token-loss difference after ablating induction heads with low FV scores heads with low induction scores.
15

---

Which Attention Heads Matter for In-Context Learning?
Pythia 160M
Pythia 1B
Pythia 6.9
0.3
0.5
Randomn abhaion
0.3
0.1
02-
0.1
5
10
15
20
10
15
20
10
0.5
0.25
Zaro ablation
0.4
0.20
0.15
0.10
0.05
0.1
0.00
0.30
0.3
Revita ablatoo
0.25
04-
0.20
0.4
0.15
0.3
15
10
0.5
Zato aNobon
0.4
0.15
0.05
10
20
15
20
Abdetion (%6)
Abiation (%)
Ablation (%)
Clean
Induction
Random
Ablation wilh excluusion
Figure Few-shot ICL accuracy after ablating induction heads, using either random ablation method (rows and 3) or zero ablation method (rows 2 and 4). Overall, the observation that ablating FV heads decreases ICI. accuracy more than induction heads, is robust against different methods of ablation.
16

---

Which Attention Heads Matter for In-Context Learning?
All layers
Layer
Layer
0.50
0.50
0.50
0.25
0.25
0.25
5
10
15
20
10
15
20
5
10
15
20
Layer 7
Layer 8
Layer 9
0.50
0.50
0.50
0.25
0.25
0.25
10
10
10
Layer 10
Layer 11
Layer 12
0.50
0.50
0.50
0.25
0.25
0.25
10
15
20
10
20
10
20
Layer 13
Layer 14
0.50
0.50
0.25
0.25
10
15
20
10
15
20
Clean
Random
Induction
PV
Figure ICL accuracy after ablating randomly sampled heads from specific layers. The clean ICL accuracy, induction ablations and FV ablations are also plotted for comparison but only the random ablations (green curve) are affected by the choice of target layer.
Induction score
0.10
FV score
max
mean top 2%
0.2
mean
0.05
6p
0.0
0.00
70M
117M
160M
345M
410M
774M
B
4B
CB
2. 2B
6.98
70M
117M
160M
345M
774M
1,8
108
2.8B
6.9
B
6.
Model size
Model size
Figure 12. Induction score (left) and FV score (right) of attention heads across model size. We plot the maximum score of all heads, mean
top scores. and mean score heads. Overall, induction scores are similar across models. Pythia 70M Llama have relatively low FV scores, Pythia 1B and 1.4B have relatively high FV scores.
17

---

Which Attention Heads Matter for In-Context Learning?
0.6
0.5
lnduction Booze
0.4
00
Accurscy
0.4
0.3
clean
shuffed
0.2
shuffed-JP
shuffed
704
117M
1604
343
410M
774M
18
1.48
L.08
2.88
6.08
7B
0.0
Sibe
705
117M
160M
345M
MAMM
778M
10
1.40
2-80
0.98
70
Size
Figure 14. Model ICL accuracy on prompts with 10 in-context examples (clean), uninformative shuffled prompts, shuffled prompts with FV, and on shuffled prompts with random head outputs. Adding FV recovers most of the model accuracy clean run, with the exception of Pythia 2.8B.
0.88
IVV score
0.06
where randomly sample outputs seen during training and compare these random outputs to the ground truth. The random baselines are shown in red horizontal lines.
0.00
-0.02
70M
1174
T604
345M
410M
774M
LGB
2,88
6,08
78
A.10. Head locations
Size
In Figure 19 we plot the locations of induction heads and FV heads across model layers.
Figure Distribution induction scores (top) and FV scores (bottom) across model size.
A.11. Induction and FV scores of heads after ablation
we plot the induction scores, percentile of induction scores, FV scores, and percentile of FV scores of the heads ablated in $ for both ablation and ablation with exclusion.
In Figures 20
layer [L//3, and shuffled prompts with FV extracted from random heads added to hidden states layer LI/3. We take 1000 examples per task that are previously unseen during FV score computation.
A.12. Overlap between ablated induction and FV heads
In most models, adding the FV recovers model performance on uncorrupted prompts, with the exception of Pythia 2.8B. One possible explanation for this again due head di- mensionality: Pythia 2.8B has head dimension 80, which significantly smaller than other models with similar pa- rameter size that have head dimensions of 128. Together with our experiments in ŞA.6 results provide preliminary evidence that high head dimensionality relative to model size is a predictor of FV strength (H6).
In Figure 28 we plot the percentage of attention heads that overlap between the set of induction heads and heads we ablate. We find that as the number of ablated heads increases, the overlap between the two sets of ablated heads also increases. This demonstrates the importance of performing ablations with exclusion to control for overlap.
A.13. Scores across training
plot the evolution of induction and FV scores averaged over top 2% heads across model training, along with the few-shot accuracy of the model checkpoints averaged over the evaluation tasks. In Figure we plot the evolution induction and FV scores individual heads across training.
In Figure 29
A.8. ICL tasks
In Table we list the ICL tasks used in this study. We refer to Todd et al. (2024) and [Fng & Steinhardi (2024) for a detailed description of each task.
A.9. Ablations by task
In Figures 15
we plot the ICL accuracy after ablating induction heads and FV heads for each task the evaluation set. We also compute the random baseline for each task,
18
18

---

Which Attention Heads Matter for In-Context Learning?
Table 3. Summary of ICL tasks used in our study. Tasks in bold are new tasks that were not used in (Todd et al. 2024
Abstractive Tasks
Extractive Tasks
Abstract elf (Ours.)
Adjective verb
Todd al.
2024
Antonym (Nguyen et al.. 2017
Animal vs. object Todd et al. 2024
Binding capital (Feng & Steinhardt, 2024
Choose first of list
(Todd al. 2024
Binding capital parallel (Feng & Steinhardt Binding fruit (Fen & Steinhardi
2024
Choose middle of list (Todd al..
2024
2024
Choose last of list (Tddd ct al.
2024
Binding shape (Feng & Steinhardt
2024
Color vs. animal Todd et al.
2024)
Capitalize first letter (Nguyen et al.
2017
Concept vs. object (Todd et al.
20244
Capitalize index (Ours.)
Fruit vs. animal (Todd et al. 20244)
Capitalize second letter (Ours.)
Object vs. concept (Todd et al.
2024
Capitalize (Nguyen et al. 2017
Verb vs. adjective (Todd et al.
2024
Country-capital (Todd al.
2024
CoNLL-2003, NER-person (Tjong Kim Sang
2002
Country-currency (Todd al.
2024
CoNLL-2003, NER-location (Tjong Kim Sang
2002
English-French (ample al.
2018
CoNLL-2003, NER-organizztton (Tjong Kim Sang
2002
English-German Lample ot
2018
English-Spanish Lample et
2018
EnglihhEnglish Lample et al..
2018
Landmark-Counne (Hernandez et al.
2024
Lowercase first letter (Todd et al.
2024)
National parks (Tod ct al. 2024
Next-item (Todd et al. 2024
Previous-ine -ttious-iteem Todd et al.
2024
Park-country (odd et al.
2024
Person-instrumen (Hernandez et al.
2024
Person occupaticn Hernandez el al.
2024
Person-sport (Hernandez et al. 2024
Present-pass (Tod et al. 2024
Product-company (Hernandez et al.
2024
Singular-plural (Todd et al. (0244)
Synonym (Nguyen et al. 2017
CommonsenseQA (MC-QA) Talmor et al. 2019
Sentiment analysis (SST-2) (Socher et al."
2013
AG News Zhang et al. 2015
19

---

Which Attention Heads Matter for In-Context Learning?
oapitalion first letter
coa122003 organization
national parks
parkecunntry
0.076
0.2
0.2
3.000
011
0 3,025
19
0.0
10
10
16
20
10
20
person- ooccoooiccnttionn
animal object 3
choose first of 5
e gemmmannnem
person--nstrummnt
0.2
0.05
0.-
0.1
0.1
0.00
0.1
0.2
10
10
10
20
10
20
10
20
commonsensee qa
capital_indes Index
cap pppitalizeeeoo second letter
abstract clf
french-english
0.0L0
6.1
0.1
0.10
0.05
0.2
0.005
0.0
8.000
10
16
20
20
bind fruit
bind capital par
bind capital
bind shape
Clean Rindom
0.2
0.81
0.1
Induction
0.13
0.00
0.00
13
10
20
Pytbla
capitalize first letter
conl12003 organization
national
park-coustry
0.4
0.1
0.10
0.04
0.2
0.05
0.02
0.0
10
20
10
20
20
occupation
animal object
choose first of 5
english german
personn-intrummmn
0.2
0.05
0.4
0.25
0.00
0.2
0.00
10
10
10
20
10
20
commonsense qa
capital Index
capitalize second letter
abstract clf
frunch-english
0.2
0.2
0.2
0.1
0.1
0.0
0.00
10
10
20
10
20
10
20
10
20
bind fruit
bind capital par
bind capital
bind shape
0.2
0.4
0.05
0.2
Clean
Induction
Readom
10
10
10
20
10
20
GPT2
3g news
capitalize first letter
2003 organization
national parks
park country
0.25
0.4
0.20
0.2
0.1-
0.05
10
20
10
20
10
20
20
20
20
person--ocmpation
animal v. object 3
choose first, of
eng hnggrsha
personn-nstrummm
0.24
0.5
0,865
025
0.2-
n.n
0.000
20
commossense ga
capital Index
capitalize second letter
abstract elt
french- english
0.2
0.1
0.1-
0.01-
0.0
0.0-
0.0-
0.0
0.00
Wind_fruit
bind_ capital par
bin [and__ce capital
bind shape
0.4
0.4-
0.65
0,2-
0.2-
0.04
0.00
0.00
0.0-
Pythin 160M
Figure 15. ICL accuracy after ablations by task. The red horizontal line represents the random baseline.
20

---

Which Attention Heads Matter for In-Context Learning?
capitalize first Jutter
coa1120033 organization
nationaal parks
corntc
0.4
0.4
0.5-
0.1-
0.2
0.2
10
20
10
20
30
20
10
20
petson-occupation ation
animal v objoct 3
choose Erst of
ongliib-eermaa german
person--sitrummnt
0.2
0.50
0.010
0.3
0.25-
0.5-
0.005-
0.2
0.1-
10
10
20
10
20
30
20
10
20
commossennse qa
capital.inee Lidex
capitalize second letter
abstract, clf
french-english
0.2
0.1
0.2
0.0-
0.0-
0.0-
10
20
10
20
10
20
bind fruit
bind capital par
bind capital
bind shape
0.50
0.2
0.2
0.4-
Clean Rundoms
0.25
0.1-
0.1-
Induction
0.00
0.0
0.0
0.0
capitalize first lettar
coc312003 organization
national
park--wunnry
0.5
02
0.3
0.0
0.0
10
20
10
10
10
20
animal object 3
choose first
english english-german
instrument
1.3
0.2
0.3
0.1
0.5
0.3
0.2
0.3
0.0
10
20
0.00
10
10
20
10
20
commonselse.co
capital index
capitalize sacond lotter
abstract clf
french-english
0.2
0.4
0.1
01
02
0.6
10
20
10
20
10
10
20
10
20
bind fruit
bind capital par
bind capital
bind shape
Clean
0.3-
Random
0.25
0.2
Leductioo
FV
10
20
10
20
Pythia 4L0M
3g news
capitalize first letter
2003 organization
national parks
park country
0.4
050
0.5-
0.2
0.25
0.1-
10
10
20
10
20
20
30
20
person-ocmuallio Wiom
animal v object 3
choose first, of
umglishh-gormas
personn-nstrummm
0.2-
0.010
0.3
0.5-
0.5
0,905
0.2-
n.n
0.00n
26
commossense gD
capital index
capitalize second letter
abstract dlt
french-english
0.2-
0.4-
0.10
0.1-
0.2
0.0
0.0
0.0-
0.0
0.00
fruit
bind capital_patr
bind cind_capital
shnd_shape
0.50
0.2-
0.50
0.00
0.0-
0.0-
0.00
Zarge
Figure 16. ICL accuracy after ablations by task. The red horizontal line represents the random baseline.
21

---

Which Attention Heads Matter for In-Contest Learning?
capitalia first letter
cma122033 organization
national parks
cortecountry
0.6
0.6
05
0.5
0.2
0.2
10
20
10
20
10
20
10
20
10
20
pe/50n-occuatioon
animal objoct
choose first of 5
encilis gemtan
pertoon-instrumment
0.2
0.4
0.1
0.5
0.5
0.2
0.2
10
20
10
20
10
20
10
20
30
20
commozseenee
capital_inde
cap papitalize second letter
abstract, cll
french-english
0.2
0.50
0.1-
8.L-
0.2
0.00
10
20
20
10
20
10
20
10
20
bind fruit.
bind capital par
bind capital
bind shape
Cleun
0.5
0.2
0.5
Rundots
0.2
Inductin
0.13
0.0
2.0
0.0
20
10
20
10
20
10
20
Pytho
capitalize first lettar
co2312003 organization
national
park-cwuntry
0.5
0.50
2.1
0.4
0.5
D25-
0.5
0.00
0.0
10
10
10
10
20
animal object 3
choose first
engliss engliss-gerrman
instrument
1.0
0.5-
0.2
0.0
10
20
10
10
20
10
20
commonselse.co
capital Index
capitalize sacond letter
abstract clf
french-english
0.50
0.1
011
0.2
0.25
8.9
10
20
10
20
10
10
10
20
bind fruit
bind capital par
bind capital
bind shape
0.50
059
Clean Random
D25-
Leductioo
FV
0.00
10
20
10
20
Pythu
3g news
capitalize first letter
2003 organization
national parks
park country
0.50
0.5-
0.1-
10
10
20
10
20
20
20
20
person-ocmiaali Whom
animal v object 3
choose first, of
omggishh-grrmmn
parson--nstrummm
050
0.050
0.5
0.25
a.
0.00n
0.n-
26
commossense ga
capital index
capitalize second letter
abstract dit
french-english
0,2
0.2-
0.1
0.1-
0.1-
0.0
0.0-
0.0-
0.0
0.0
fruit
bind candd_apptal_paa ppar
bind cind_capital
shape
0.50
0.4
0.2
0.2
0.00
0.0-
0.0-
0.0-
GPT2 XL
Figure 17. ICL accuracy after ablations by task. The red horizontal line represents the random baseline.
22

---

Which Attention Heads Matter for In- Learning?
capitalian first letter
con122003 organization
national parks
partecountry
0.6
0.6
0.4
0.5
0.5
0.25
0.2
10
20
10
20
10
20
10
20
10
20
oecson-occuaation
animal objoct
choose first of 5
encilis gezminn
person-instrument
2.0
0.2
0.4
0.1
0.5
255
0.2
0.2
10
20
10
20
10
20
30
20
10
20
commonsense
capital_inder
cap papitalize second letter
abstract clf
french-english
0.2
0.0
0.5
0.1
8.1-
0.2
0.0
10
20
10
20
10
20
10
20
10
20
bind fruit.
bind capital par
bind capital
bind shape
Clsun
0.5
Randots
0.2
0.26
Inductin
0.13
0.00
2.0
0.00
20
10
20
pythis
capiallio first letter
coo1/2003 organization
national
parfecountry
0.6
0.5
0.5
0.50
0.5-
0.5
0.2
0.00
10
20
10
20
10
20
10
20
10
20
animal object
choose first of 5
english
instrmmestttttt
0.2
0.5
0.50
0.6
0.1
0.5
0.25
0.2
10
20
10
20
10
20
10
20
10
20
commorsense
capital Index
capilalize second lotber
abstract elf
french-english
0.2
0.5
0.1
0.1-
0.1-
0.2
10
20
10
20
10
20
10
20
10
20
bind fruit
bind capital par
bind capital
bind shape
0.50
0.50
0.5
Clean Bardom
0.25-
0.25
Indactice
FV
10
20
10
20
10
20
15
20
Pythi
ag news
1.0
cap italtazze first letter
cool/2003 organization
national parks
paris country
0.5
0.5
5.5
0.5
10
20
10
20
10
20
10
20
10
20
pecson-ccrupatiion
animal object
choose first of 5
anglish genmmnn
person-ins/rummt
0.50
0.
9.5
0.25
0.6
20
n.6
10
20
5.
26
19
20
commezsense q3
capital index
capitalize second letter
abstract cif
french- french-english
0.2
0.2
0.1
0.2
0.0
0.0
2.0
0.0
0.0
20
bind fruit
bind_ capital_patt par
hind_cappita
Minel_ve shape
0.5
05
0.5
Induction
0.0
0.0
0.0
0.0
12ams
Figure 18. ICL accuracy after ablations by task. The red horizontal line represents the random baseline.
23

---

Which Attention Heads Matter for In-Context Learning?
Pythia 70M
GPT-2
Pythia 160M
Induction score Induction score
0.42
0.005
score (X)
Induction score
0.4525
Induction score
x
0.04
FV score (X)
0.41
0.004
0.4500
0.03
FV score
0.42
0.03
0.40
0.003
0.4475
0.02
0.02
0.39
0.002
0.41
0.4450
10
10
t-stat: nan, p-value: nan
t-stat: -2.83, p-value: 0.05
t-stat: -4.00, p-value: 0.02
GPT-2 Medium
Pythia 410M
GPT-2 Large
Induction score (o)
0.03
FV FV score (X)
Induction score
0.45
0.075
0.450
0.04
0.45
0.050
FV SCOLS
0..2
FV score
0.425
0.40
0.02
0.025
0.40
0.01
0.400
0.35
20
20
20
I-stat: -0.57, p-value: 0.58
t-stat: 22100,
p-value: 0.05
t-stat: 0.19, p-value: 0.85
Pythia 1B
Pythia 1.4B
GPT-2 XL
Induction score Induction score (+)
0.45
0.10
Induction score
0.45
0.10
Induction score
0.48
0.04
0.40
FV FV score
0.46
FV score
0.08
TV 0000
0.40
0.05
0.44
0.02
0.35
0.06
0.35
0.42
0.04
10
20
25
t-stat: -0.85,
p-value: 0.44
t-stat: -1.90,
p-value: 0.08
t-stat: -0.42, p-value: 0.68
Pythia 2.8B
Pythia 6.9B
Llama 7B
Induction score (c)
0.45
FV score (x)
Induction score
FV score (x)
A/ Score ( (x)
0.04
0.45
0.04
0.4
0.015
0.40
0.35
0.02
0.40
0.02
0.3
0.010
0.005
20
20
20
t-stat: -1.69, p-value: 0.11
t-stat: -0.42, p-value: 0.68
t-stat: 0.22, p-value: 0.83
Figure 19, Location of induction heads (bluc) and FV heads (pink) in model layers
24

---

Which Attention Heads Matter for In-Context Learning?
Pythia 70M
GPT-2
Pythia 160M
GPT-2 Medium
Induction score
0.04
Induction score
Induction score
0.04
0.04
0.03
Induction
0.02
0.02
0.01
0.01
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Pythia 410M
GPT-2 Large
Pythia 1B
Pythia 1.4B
Induction score
Induction score
0.03
Induction score
Induction score
0.03
0.02
0.02
0.02
0.02
0.01
0.01
0.01
0.01
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
GPT-2 XL
Pythia 2.8B
Pythia 6.9B
Llama 2 7B
Induction score
0.03
Induction score
Induction score
Induction score
0.0150
0.020
0.0125
0.02
0.015
0.02
0.0100
0.010
0.0075
0.01
0.01
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Induction
FV
Figure 20, Induction scores of the remaining heads after ablation.
25

---

Which Attention Heads Matter for In-Context Learning?
Pythia 70M
GPT-2
Pythia 160M
GPT-2 Medium
Induction %ole
Induction %ile
80
Induction %ile
Induction %ile
80
75
70
70
60
60
60
70
50
50
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Pythia 410M
GPT-2 Large
Pythia 1B
Pythia 1.4B
Induction %ile
80
Induction %ile
80
Induction %lle
Induction %ile
80
70
70
60
60
60
60
50
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
GPT-2 XL.
Pythia 2.8B
Pythia 6.9B
Llama 2 7B
Induction %olle
80
Induction %ile
80
Induction %ale
Induction %ile
80
70:
70
70
60
60
60
60
50
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Induction
FV
Figure 21. Percentile of induction scores the remaining heads after ablation.
26

---

Which Attention Heads Matter for In-Context Learning?
Pythia 70M w/ exclusion
GPT-2 w/ exclusion
Pythia 160M w/ exclusion GPT-2 Medium w/ exclusion
Induction score.
Induction score
0.04
Induction score
Induction score
0.04
0.03
0.04
0.03
0.02
0.02
0.02
0.02
0.01
0.01
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Pythia 410M w/ exclusion
GPT-2
Large w/ exclusion
Pythia 1B w/ exclusion
Pythia 1.4B w/ exclusion
Induction scare
Induction score
0.03
Induction score
Induction score
0.025
0.03
0.02
0.02
0.020
0.02
0.015
0.01
0.01
0.010
10
20
10
20
10
20
10
20
Ablation (%)
Ablation
Ablation (%)
Ablation
GPT-2 XL. w/ exclusion
Pythia 2.8B w/ exclusion
Pythia 6.9B w/ exclusion
Llama 7B w/ exclusion
Induction score
0.025
Induction score
0.03
Induction spore
0.0150
0.025
0.020
Induction
0.0125
0.020
0.015
0.02
0.0100
0.015
0.01
0.0075
10
20
10
20
20
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Induction
FV
Figure 22. Induction scores of the remaining heads after ablation with exclusion.
27

---

Which Attention Heads Matter for In-Context Learning?
Pythia 70M w/ exclusion
GPT-2 w/ exclusion
Pythia 160M w/ exclusion GPT-2
Medium w/ exclusion
Induction %olle
Induction %ole
80
Induction %ole.
80
Induction %ile
80
75-
70
75
70
60
70
70
50
60
10
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Pythia 410M w/ exclusion
GPT-2 Large w/ exclusion
Pythia w/ exclusion
Pythia 1.4B w/ exclusion
Induction %ile
Induction %ile
Induction %ile
Induction %ile
80
80
80
75
70
70
70
60
60
65
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
GPT-2 XL w/ exclusion
Pythia 2.8B w/ exclusion
Pythia 6.9B w/ exclusion
Llama 2 7B w/ exclusion
Induction %ile
Induction %ile
Induction %ile
80
Induction Kile
80
80
70
70
70
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Induction
FV
Figure 23, Percentile of induction scores of the remaining heads after ablation with exclusion.
28

---

Which Attention Heads Matter for In-Context Learning?
Pythia 70M
GPT-2
Pythia 160M
GPT-2 Medium
0.0005
0.0002
FV score
-0.0002
FV score
FV score
0.0005
FV score
0.0000
0.0000
-0,0004
0.0000
-0.0002
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Pythia 410M
GPT-2 Large
Pythia 1B
Pythia 1.4B
0.002
0.0005
0.0002
0.0001
FV score
FV score
FV score
FV score
0.001
0.0000
0.0000
0.0000
0.000
-0.0002-
-0.0001
00.011
-0.0005
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
GPT-2 XL
Pythia 2.8B
Pythia 6.9B
Llama 2 7B
0.0001
0.0002
0.0002
0.0001
FV score
FV score
FV score
FV score
0.0000
0.0000
0.0000
0.0000
-0.0001
-0.0001
-0.0002
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Induction
FV
Figure 24. FV scores of the remaining heads after ablation.
29

---

Which Attention Heads Matter for In-Context Learning?
Pythia 70M
GPT-2
Pythia 160M
GPT-2 Medium
40
80
80
80
FV %ile
FV %ile
FV %ole
FV %ile
60
60
60
30
40
40
40
20
20
20
20:
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Pythia 410M
GPT-2 Large
Pythia 1B
Pythia 1.4B
80
75
80
EV %ile
EV %ile
%ile
60
60
50
40
40
40
25
20
20
20
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
GPT-2 XL.
Pythia 2.8B
Pythia 6.9B
Llama 2 7B
80
80
EV %lle
FV %ile
EV %lle
FV %ile.
60
60
40
40
40
20
20
20
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Induction
FV
Figure 25, Percentile FV scores of the remaining heads after ablation.
30

---

Which Attention Heads Matter for In-Context Learning?
Pythia 70M w/ exclusion
GPT-2 w/ exclusion
Pythia 160M w/ exclusion GPT-2 Medium w/ exclusion
-0.0002
0.00050
0.0002
IV soorr
0.0005
FV score
0.00025
0.0000
-0.0004
FV
0.00000
IV
0.0000
0.00025
-0.0002
10
20
10
20
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Pythia 410M w/ exclusion
GPT-2
Large w/ exclusion
Pythia 1B w/ exclusion
Pythia 1.4B w/ exclusion
0.0002
0.002
0.0005
0.0002
FV score
0.0001
FV score
0.001
FV score
PV
0.0000
0.0000
0.000
0.0000
-0.0002
-0.0001
-0.001
20
10
20
10
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
GPT-2 w/ exclusion
Pythia 2.8B w/ exclusion
Pythia 6.9B w/ exclusion
Llama w/ exclusion
0.0002
0.0001
FV scort
0.0002
FV soore
0.0001
FV
0.0000
0.0000
IV
0.0000
0.0000
0.0001
-0.0001
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Induction
Figure 26. scores the remaining heads after ablation with exclusion.
31

---

Which Attention Heads Matter for In-Context Learning?
Pythia 70M w/ exclusion
GPT-2 w/ exclusion
Pythia 160M w/ exclusion GPT-2 Medium w/ exclusion
80
80
FV Villee
EV %ill
%illeee
EV Vr%le
60
30
40
40-
40
20
20
20
20
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Pythia 410M w/ exclusion
GPT-2 Large w/ exclusion
Pythia 1B w/ exclusion
Pythia 1.4B w/ exclusion
80
80
FV %6lle
E "%ill
60
%ile
60
EV (%ill
60
40
40
10
25
20
20
20
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
GPT-2 XL. w/ exclusion
Pythia 2.8B w/ exclusion
Pythia 6.9B w/ exclusion
Llama 2 7B w/ exclusion
80
80
80
80
FV %ile
60
TV %ile
60
FV %ille
60
FV %ile
60
40
40
440
20
20
20
20
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Induction
FV
Figure 27. Percentile of FV scores of the remaining heads after ablation with exclusion.
Pythia 70M
GPT-2
Pythia 160M
GPT-2 Medium
75
Overlap (%)
40
Overlap (%)
40
Overlap (%)
30
50
Overlap
20
20
25
20
10
0-
0
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Pythia 410M
GPT-2 Large
Pythia 1B
Pythia 1.4B
40
30
30
30
Overlap (%)
Overlap (%)
Overlap
20
averlap
20
20
20
10
10
10
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
GPT-2 XL
Pythia 2.8B
Pythia 6.9B
Llama 2 7B
25
25
30
Overlap (96)
Overlap (%))
20
Overlap
20
Overlap
20
20
10
15
15.
10
10:
20
10
20
10
20
10
20
Ablation (%)
Ablation (%)
Ablation (%)
Ablation (%)
Figure 28, Overlap between set of induction heads and FV heads ablated.
32

---

Which Attention Heads Matter for In-Context Learning?
Pythia 70M
Pythia 160M
Pythia 410M
Induction score
0.2
Induction score
Induction
0.4
0.0050
0.1
PV score
0.11
8.92
FV score
0.1
0.02
0.2
PV score
0.2
0.0025
0.01
0.0
24
2
2
8.8000
0.0
24
8.80
0.0
24
2
214
0.00
Steps
Steps
Steps
Pythia 1B
Pythia 1.4B
Pythia 2.8B
0.4
Induction score
0.4
Induction score
0.10
PV score
FV score
0.010
FV score
Induction
0.1
8.95
0.1
0.02
0.2
0.05
0.005
0.0
24
2
2
8.00
24
2
24
0.00
0.00
24
2
21
8.000
Steps
Steps
Steps
Pythia 6.9B
0.010
0.4
Induction
0.1
8.905
0.0
0.000
29 Steps
244
Figure 29, Evolution of induction score and FV score averaged over top 2% heads across training.
33

---

Which Attention Heads Matter for In-Context Learning?
Pythia 70M
Pythia 70M
0.5
0.015
0.010
0.005
0.0
21
2'
27
215
0.000
21
2'
2
Stegss
Stops
Pythia 160M
Pythia 160M
0.04
0.2
0.02
0.1
0.0
21
2
23
27
2
211
2
215
217
0.00
21
2
2
21
2
211
277
Stegas
Stops
Pythia 41 OM
Pythia 4 410M
0.4
0.10
0.08
Induction
0.04
0.1
0.0
21
2
2
211
2
2
21
0.00
21
2
21
2
2
21
Stops
Stops
Pythia
Pythia
0.5
0.4
0.3
0.2
0.050
0.1
0.025
0.0
21
20
20
211
2,a
245
0.000
21
2
241
2.5
235
Steps
Steps
Pythia
Pythia
0.5
scoe
0.4
0.3
0.075
0.2
0.050
0.025
25
2.3
Stegs
Staps
Pythla 2.88
Pythia 2.8B
0.4
0.3
0.04
0.02
21
23
27
211
2.3
25
217
21
2.5
213
217
Stego
Stops
Pythia 6,98
Pythia 6.9B
Induction
0.4
0.04
0.03-
0.2
0.1
0.01
0.0
21
25
23
27
29 Stops
211
25
2
237
0.00
21
2"
23
27
2 Stops
21
2
21
237
Figure 30. Evolution of induction scores (left) and FV scores (right) of individual induction and FV heads across training
34


Reviews:
[{"review_id": "GPjL5GpFUu", "reviewer": "Reviewer_X9CA", "summary": "This paper explores the mechanisms behind in-context learning (ICL) in large language models (LLMs), specifically examining two types of attention heads: induction heads and function vector (FV) heads. The authors conduct experiments to determine which of these mechanisms is primarily responsible for ICL. Through various ablations, the paper argues that FV heads play a more significant role than induction heads, especially in larger models, while also showing that induction heads evolve into FV heads during training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback! \n\nTo address your weaknesses:\n* On the shift from induction to FV: we will elaborate on why this shift might be necessary for larger models. We hypothesize that induction is a more lightweight mechanism (based on just attending to the subsequent token of a previous copy of the current token) that offers a crude way for models to retrieve information from context, however this method does not sufficiently solve more complex ICL problems. FV heads, on the other hand, offer a compact representation of the ICL task given the examples - the exact mechanism of FV heads is still unknown, and plausibly more complex, but allows larger models to achieve higher training accuracy. We also discussed the possibility of other mechanisms that may play a role in ICL, we leave this investigation for future work.\n* On the conjecture between induction and FV heads relationship: we performed an additional experiment measuring the evolution of ICL accuracy during training. We observe that in all models, few-shot ICL accuracy begins to improve around the same time as when induction heads appear, and continues to gradually increase throughout training until the end. Since ICL accuracy continues to improve even after the formation of induction heads, we speculate that this suggests the sharp emergence of induction heads contributes to an initial rise in ICL performance, but the emergence of the FV mechanisms contributes to further improvements in ICL. This reinforces our conjecture that induction heads may serve as a stepping stone for FV, and we leave further exploration to rigorously verify this conjecture to future work.\n* We will also add a further discussion on the application of our findings: our evidence of the higher influence of FV heads suggests that this is a more effective mechanism for ICL than induction heads. This suggests that ICL can be optimized in small models by using training methods that promote the formation of FV heads.\n\nThank you for suggesting the relevant work by Wu et al.! We also extensively discussed and compared with the Anthropic work suggested (which we cite as Olsson et al. 2022). \n\nThank you again for your feedback, please let us know if any questions or concerns remain unsolved!", "claims_and_evidence": "1. Induction and FV heads are distinct but correlated. - Ablations and training dynamics support this claim, showing minimal overlap but some correlation in the functionality of these heads.\n2. FV heads are primarily responsible for ICL in LLMs. - Evidence: Ablating FV heads results in a significant drop in ICL performance, whereas ablating induction heads has a smaller impact.\n3. Induction heads evolve into FV heads during training. - Evidence: The authors observe that some heads that initially exhibit induction-like behavior transition to performing more abstract FV functions later in training.", "methods_and_evaluation_criteria": "The study uses ablation experiments to measure the effect of removing induction and FV heads on ICL performance, focusing on few-shot learning tasks. The paper also tracks the evolution of these heads during training to understand their transition from simpler induction-based operations to more complex FV-based mechanisms. Additionally, they explore the overlap between induction and FV heads and provide a detailed breakdown of the results for models ranging in size from 70M to 6.9B parameters.", "theoretical_claims": "The paper introduces a key theoretical claim that induction heads may serve as an initial mechanism that helps the model learn the more complex FV heads. This hypothesis is supported by the observation that FV heads often start with high induction scores during training but diverge as training progresses.", "experimental_designs_or_analyses": "Ablation experiments were conducted on 12 different language models of varying sizes. Causal mediation analysis was used to assess the contribution of specific heads to ICL performance. Training dynamics analysis helped track the evolution of attention heads during the learning process.", "relation_to_broader_scientific_literature": "The paper contributes to ongoing research in head interpretability and in-context learning. This may be related to https://arxiv.org/abs/2404.15574 and https://www.anthropic.com/research/in-context-learning-and-induction-heads. Especially the second one, I think authors may need to discuss a bit with the second one.", "essential_references_not_discussed": "While the paper cites foundational works, it could benefit from incorporating studies that investigate the broader implications of these findings on neural architecture optimization and future model scaling.", "other_strengths_and_weaknesses": "Strengths: Comprehensive ablation studies across models of various sizes. Clear presentation of results and robust statistical analysis. Thought-provoking theoretical contribution regarding the evolution of induction heads.\n\nWeakness: The paper could expand on why the shift from induction to FV heads is necessary for larger models and whether other mechanisms could play a role. The conjectures about the relationship between induction and FV heads could benefit from further validation with real-time model adjustments. The application of FV heads could be further discussed.", "other_comments_or_suggestions": "The findings present a compelling case for rethinking the role of induction heads in ICL, but further exploration of the interaction between FV heads and other components of the model might provide a more holistic view of in-context learning dynamics.", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UmKjXA9u7Y", "reviewer": "Reviewer_joQW", "summary": "The authors study Induction Heads and Function Vector Heads in relation to In-Context Learning in a variety of small and medium-sized models.\nTheir experiments suggest that induction heads and FV heads are distinct, though there is some overlap in their behaviours.\nThey also show that removing FV heads greatly impacts accuracy, while it is not the case for induction heads, while removing induction heads has a greater impact on token-loss difference (the difference in loss between the 50th and the 500th token of the prompt) than removing FV heads.\nFinally, they show that FV heads tend to be found slightly deeper within the network, and that during training some heads turn into induction heads before later involving into FV heads.\n\nThe authors attribute some differences between their findings and earlier literature to their carefully distinguishing between the metrics of interest (token-loss difference and ICL accuracy), as well as to their controlling for the behavioural overlap between FV heads and induction heads.\n\n## update after rebuttal\nThe authors have answered many of my concerns, and have even brought forth new interesting experimental results. Consequently, I have raised my recommendation to a 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback! We address concerns below:\n1. Our task suite does include dictionary-style tasks as you mentioned, and we plotted the task-specific ablations in Figure 18 of the appendix - for the dictionary tasks (e.g. national_parks, park-country, person-occupation, person-instrument, english-german, french-english) we do still consistently find that FV heads contribute more than induction heads significantly.\n2. In many cases, we had to decide on a threshold to differentiate meaningful FV/induction heads from the long tail of other heads. The 2% threshold was carefully chosen following previous work by Todd et al. (2024), and after we verified this threshold does seem to meaningfully separate important heads (Figure 7 in appendix shows the 2% threshold does select heads outside the cluster of other low-scoring heads - taking a higher percentage would select heads with FV scores close to 0). If we take the top 5% of heads, there would be between a 20-40% overlap (except for our smallest model with a 67% overlap). \n3. While we used a held-out set of tasks to identify FV heads, the FV heads are task-agnostic (as shown in our ablations where they contribute to tasks that were not used to identify them) - so regardless of the choice of the ICL task, with enough tasks the same set of heads should be identified as FV heads. In addition, the tasks we used to compute FV heads are ones that were previously often associated with induction heads, but demonstrated in our paper as being less influenced by induction heads than FV heads. Therefore, the lack of overlap is a general phenomenon. \nAlso, the main goal of the claim on overlap is to establish that the two sets of heads we're studying are not identical, to motivate subsequent experiments comparing the two sets. Our ablation and training experiments later that shows different trends for FV vs. induction heads further reinforces that the two are distinct types.\n4. We agree token-loss difference trends are noisy and we will remove the claim about scale-dependence. The key finding we'd like to emphasize here is that token-loss difference and ICL accuracy measure distinct phenomena.\n5. We agree that the observation on layer depths is not statistically backed up - we will clarify that this observation is speculative.\n6. Great point! We used a different set of tasks to compute FV scores than the tasks used in our ablation experiments, to specifically ensure that our main claim is not tautological. What's interesting about FV heads is that their contribution to ICL seems to generalize across different ICL tasks. \n7. Interesting question! The difference between the two metrics was first described in Olsson et al. (2022), but we will include a discussion in our revision as well. In the prompt you described, it would be a normalized version of few-shot ICL, but in practice, TL difference is not computed using such prompts (in previous work and ours, it is taken over natural sentences). And yes, high ICL accuracy with low TL difference (which is what we observe when we ablate random heads and induction heads for example) means the model preserves its ICL abilities but loses other abilities associated with learning important signals from context. These other signals could capture non-ICL signals such as remembering entities, tracking style from context, etc. (it would be great for future work to analyze what is being lost when we see these drops in TL difference!)\n\nThank you for providing the additional references, suggesting better clarity in 2.2, and pointing out the typos - we have incorporated these in our revision!\n\n> “However, C2 would predict that ablating monosemantic FV heads would not hurt ICL performance, […]” Why is that ?\n\nC2 posits that FV heads are polysemantic heads that implement both induction and the FV mechanism. Then, if we ablate monosemantic FV heads but preserve the polysemantic FV heads, models should still be able to perform ICL - we will clarify this in our revision!\n\nThank you again for your feedback, please let us know if any questions or concerns remain unsolved!", "claims_and_evidence": "I am currently not convinced of the soundness of the following claims:\n\n1) One of the paper’s main claims is that induction heads contribute less to ICL accuracy than FV heads. I suspect that this is very task dependent: e.g., if the context describes a dictionary of tokens of sorts, i.e. is composed of pairs “$t_{1,1} t_{1,2}, t_{2,1} t_{2,2}, \\ldots$” where $t_{i,2}$ is the output token associated with the input token $t_{i,1}$, then I would expect induction heads to contribute greatly to the accuracy. I did not read in detail the references where the various tasks studied are described to see whether such tasks are in fact included.\n\n2) The authors claim that there is little overlap between FV heads and induction heads. However, they also state that “In most models, FV heads are at around the 90-95th percentile of induction scores, and vice versa.” Unless I misunderstood that sentence, does it not mean that there is little overlap mainly because the authors arbitrarily defined FV heads as the heads with the top 2% of FV scores (and same for induction heads), but that there would be a lot of overlap if FV heads were defined as the top 5% of heads for FV scores (and same for induction heads)?\n\n3) In relation to the same claim: FV heads are by definition heads such that their activation can have a large impact on ICL accuracy for the tasks considered (since the score measures the difference in accuracy for two distinct activations). If the tasks are such that induction heads are not needed to solve them (which the authors demonstrate themselves), then won’t that choice of tasks necessarily reduce the overlap between induction heads and FV heads? In other words, isn’t this potential lack of overlap an almost direct consequence of the choice of tasks, rather than a general phenomenon?\n\n4) The authors claim that “Here, we find that in smaller models (less than 160M parameters), ablating induction or FV heads does not influence token-loss difference more than random. In models with over 345M parameters, ablating induction heads has a larger effect token-loss difference than ablating FV heads. The gap between the effect of ablating induction and FV heads decreases with model scale.” However, the behaviours of the “FV heads ablated”, “Induction heads ablated” and “random heads ablated” curves with respect to each other seem to vary so much between models (see Figure 4 and Figure 9 - note in particular that removing induction heads is sometimes less impactful than removing random heads) that I find it difficult to conclude with so few data points that there really is a trend with respect to model size, rather than simply a lot of model-dependent variance. In particular, the results for the largest model in Figure 9 in the Appendix (Llama 2 7B) go strongly against the claim.\n\n5) The authors claim that “In general, induction heads appear in early-middle layers and FV heads appear in slightly deeper layers than induction heads.” However, looking at the results in Figure 19 in the Appendix, we find that the p-values are often quite large and that the phenomenon, if it exists, might again be model-dependent.\n\nNote: I recognise that it is very hard to draw rigorous conclusions regarding such empirical matters, where confounding factors abound. Though I do not fully agree with many of the authors’ claims, I do not question the seriousness of their work or their scientific rigor.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Experiments are sound overall.", "relation_to_broader_scientific_literature": "I feel like the findings fit well within a series of articles investigating similar topics and phenomena.\nOne of the findings (induction heads are not that important for ICL accuracy) contradicts conventional wisdom, and the authors offer an explanation for it (which does not entirely convince me, see above).", "essential_references_not_discussed": "As far as I can tell, the relevant literature is adequately cited by the authors. A few additional references on either ICL for transformers or the mechanistic study of subcircuits that might further enrich the Related Works section:\n\nThe evolution of statistical induction heads: In-context learning Markov chains (Edelman et al.)\n\nIteration Head: A Mechanistic Study of Chain-of-Thought (Cabannes et al.)\n\nAn explanation of in-context learning as implicit bayesian inference (Xie et al.)\n\nWhat can transformers learn in-context? A case study of simple function classes (Garg et al.)\n\nHow do transformers learn in-context beyond simple functions? A case study on learning with representations (Guo et al.)\n\nDissecting recall of factual associations in auto-regressive language models (Geva et al.)", "other_strengths_and_weaknesses": "The paper is well-written and clear. The authors ask relevant questions and investigate them thoroughly (though I am not entirely convinced by some of their conclusions). They clearly distinguish between what their experiments prove (insofar as anything can be proven regarding such empirical phenomena) and what they merely suggest.", "other_comments_or_suggestions": "The following sentence is unclear to me, and additional explanations might be needed:\n“However, C2 would predict that ablating monosemantic FV heads would not hurt ICL performance, […]” Why is that ?\n\nSubsection 2.2 is perhaps too concise; a slightly more detailed exposition (as in Todd et al.) would increase clarity.\n\nA few typos:\n\n\"two distinct sets of heads To do so, we\" (missing \".\")\n\"task $t\\in T$ defined by a dataset $P_t$\" (missing \"is\")", "questions_for_authors": "I would appreciate additional details regarding the claims I have mentioned in “Claims and evidence” (points 1) to 5)). I would be happy to improve my evaluation of the paper if presented with satisfactory answers.\n\nOther questions:\n\n6) By the very definition of the function vector score, heads whose FV score is high are necessarily heads such that their activation can have a large impact on ICL accuracy (since the score measures the difference in accuracy for two distinct activations). As such, is it not somewhat tautological, or at least not very surprising, that ablating FV heads greatly hurts accuracy (one of the authors’ main contributions)?\n7) The authors emphasise the importance of distinguishing between few-shot ICL accuracy and token-loss difference, but make no attempt at describing the difference between the two metrics. Consider a prompt composed of input-output pairs $(x_i,y_i)$ where the 50th and 500th tokens are the correct answers $y_l$ and $y_r$ to some inputs $x_l$ and $x_r$; in that case, isn’t token-loss difference essentially a renormalized version of few-shot ICL accuracy (renormalised by the accuracy at $y_l$ given the shorter context of the first 50 tokens) ? In general, if few-shot ICL accuracy is high but token-loss difference is low, doesn’t that mean that there is no improvement in the models predictions as the context increases in length, hence that the model is not actually using the context?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wSAsoXUrnH", "reviewer": "Reviewer_wNzV", "summary": "This paper investigates the mechanisms behind ICL. Specifically, it focuses on analyzing two popular explanations for the mechanism of ICL: induction heads (token-copying) and function vector (FV) heads (task-encoding). Through extensive experiments on 12 LLMs and 45 tasks, multiple interesting findings are presented:\n- Induction heads and function vector heads are distinct but correlated.\n- FV heads matter the most for few-shot ICL, especially in larger LLMs. Particularly, the effect of ablating induction heads while preserving top FV heads is close to random, which offers new insight into the effect of induction heads.\n- Many induction heads transition to FV heads during training. Induction heads are simpler and learned earlier, whereas FV heads are more complex.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive assessment! We also address minor concerns:\n\n* Larger models: we will consider experiments on models in the 13B range for our camera-ready. However, computational constraints on our end prevent testing on larger models due to memory limits, since the FV score computation and ablation experiments require several caching of model activations. We would love to see future work extend our studies to bigger models! The models we study are also much bigger than earlier mechanistic analyses of ICL (e.g. Olsson et al. 2022 on toy models with 1-6 attention-only layers or MLPs) which shows promise of scaling our methods to bigger models given enough compute.\n* Task difficulty: while small models limit complex reasoning tasks, our findings hold on tasks such as translation, binding, named entity recognition, commonsense QA... It would be a great idea to also include the more challenging tasks you suggest for future work studying bigger models that have a reasonable base accuracy for those tasks!\n* Thank you for the suggestion! We have repeated the ablation experiments and training dynamics analysis using 5% as the threshold, and results confirm the robustness of the observed trends. \n* Practical applications of findings: we will include a discussion on how evidence of the higher influence of FV heads may suggest that this is a more effective mechanism than induction heads. This suggests that ICL can be optimized in small models by using training methods that promote the formation of FV heads.", "claims_and_evidence": "The claims are well supported by the extensive experiments presented in this work.", "methods_and_evaluation_criteria": "The evaluation criteria for the three major findings are valid. The measures of token-loss difference and ICL accuracy effectively support the discrepancy between the findings of this work and those of Olsson et al. (2022), demonstrating that FV heads significantly contribute to ICL accuracy but not to token-loss difference.", "theoretical_claims": "There is no proofs involved in this paper.", "experimental_designs_or_analyses": "In general, the experiments are rigorous and well-designed, thoroughly supporting the findings of this paper.\n\nSome minor issues include:\n- ICL is usually conducted on larger LLMs, ranging from a few billion to hundreds of billions of parameters. The largest LLM evaluated in this paper is 7B. Scaling evaluations to include larger LLMs in the range of 13B to 70B could enhance the robustness of the findings.\n- The tasks evaluated focus on simple classification tasks. Extending this to more challenging tasks (e.g., math reasoning, graduate-level MCQs) and diverse prompt formats (ICL with intermediate CoT explanations) would enhance the generalizability of the findings.\n- Most experiments are based on the top 2% of attention heads. Exploring different ratios can reveal whether the observed trends are robust. A sensitivity analysis would strengthen the findings.", "relation_to_broader_scientific_literature": "This work builds on two previous explanations of the ICL mechanism: induction heads and FV heads. It presents multiple novel and interesting insights into these two distinct explanations.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Additional Strengths**:\n- This work provides valuable insights into the mechanism of ICL, and can potentially inspire future works on ICL mechanism.\n- The paper resolves contradictions of findings in this work and prior studies on induction heads by highlighting the metric difference. It convincingly demonstrates that ICL is largely driven by FV heads rather than induction heads.\n\n**Additional Weaknesses**:\n- It would be better to discuss potential practical applications of the findings, such as how these insights could be leveraged for model training and ICL optimization.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wzeg09QdV7", "reviewer": "Reviewer_Eo6g", "summary": "This paper presents a comprehensive study on attention heads and their impact on in-context learning. Specifically, it examines two types of attention heads: induction heads and function vector heads. Through ablation studies, the authors found that function vector heads play a more significant role in in-context learning performance, while induction heads are more crucial to the token-loss difference. Interestingly, the two types of heads also seem to be correlated.\n\n## update after rebuttal\n\nGiven that the authors provided additional results to address my concerns, I increase my score to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! \nWe will first respond to the questions about claims and evidence:\n* Choice of percentile: In many cases, we had to decide on a threshold to differentiate meaningful FV/induction heads from the long tail of other heads. The 2% threshold was carefully chosen following previous work by Todd et al. (2024), and after we verified this threshold does seem to meaningfully separate important heads (Figure 7 in appendix shows the 2% threshold does select heads outside the cluster of other low-scoring heads - taking a higher percentage would select heads with FV scores close to 0). \n* Ablations with exclusion: we repeated our ablation with exclusion experiments on Pythia-6.9B taking 5% and 10% as the threshold. Overall, we observe the same trends as with 2%, where ablating induction heads does not hurt ICL much more than random, while ablating FV heads quickly significantly degrades ICL performance. Our ablations are robust to the threshold chosen for exclusion. We will repeat these experiments with other thresholds for our other models, and add them to the Appendix.\n\nWe will also address weaknesses:\n* Newer models: we are currently reproducing all experiments except section 5 on training dynamics on Llama 3 8b and Qwen 2.5 7B. Since our conclusions have applied to three families (GPT-2, Pythia, Llama 2) that were released in different years (2019, 2022, 2023), we would expect our conclusions to generalize to the models released in 2024 as well. If surprisingly they don't, we would still like to share the interesting negative results!\n* Threshold selection: we will elaborate in our revision the rationale behind choosing 2% - that in addition to following prior work, we also empirically verified in Figure 7 that selecting a higher threshold such as top 5% or 10% would end up selecting attention heads with FV and induction scores that are close to 0, which would prevent meaningful analysis. We have also previously computed the correlation between the two scores, but the correlations are not informative due to the long tail of heads that are neither high on FV or induction.\nAs for the question about contradiction, we would like to clarify that we first established a lack of significant overlap to motivate our subsequent experiments that study FV and induction as two separate phenomena (if the two heads significantly overlap and correspond to the same set of heads, we no longer have a reason to compare these two heads). However, while the two sets of heads are mostly distinct, there are some heads that have both induction and FV properties which may have confounded earlier studies (e.g. heads in purple on Figure 7). Thank you for raising this confusion, we will clarify this in our revision!\n* For our ablations, we have also computed and plotted the average scores and percentils of the heads preserved. We have added these plots to our appendix.\n\nAs for your last suggestion on separating the performance, we have plotted the ablations for each task in Figures 15-18 of the Appendix. Following your suggestion, we have added markers for the 8 new tasks to better differentiate them from the other 11 tasks.\n\nThank you again for your feedback, please let us know if any questions or concerns remain unsolved!", "claims_and_evidence": "Most of the claims are well supported by the evidence. However, I have some concerns regarding the choice of percentile for identifying different types of attention heads. It appears that you use the top 2% as a threshold to classify heads as induction heads, function vector heads, or neither. How is this percentile determined? As shown in Figure 3, while most of the top 2% induction heads do not have top 2% FV scores, they still exhibit relatively high FV scores. Consequently, the ablation study in Section 4.1 is not entirely convincing to me. The heads being ablated, though not in the top 2% of the other category, may still be in the top 5% or 10%, which is still very high to me. The experiments with “exclusion” seems not so exclusive.", "methods_and_evaluation_criteria": "They make sense.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "I check all the experiments. One issue that I have pointed out in \"Claims And Evidence\" section.", "relation_to_broader_scientific_literature": "It helps us to understand the underlying rationale of large language models.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths\n- Comprehensive empirical studies on induction heads and function vector heads and their impact on in-context learning.\n- Interesting finding that function vector heads contribute more to performance than induction heads.\n- Several models including different sizes are studied.\n\nWeaknesses\n- The selected model are relatively old. I understand part of the reason is the need to train the models. However, for those experiments do not involve training, I would like to see some more advanced models to know if the conclusion can be applied to them as well.\n- As mentioned in the \"Claims And Evidence\" section, the way to decide the threshold needs more discussion. For example, the current conclusion is there is no significant overlap among the top 2% heads. How about top 5% or top 10%? How strong is the correlation between the two types of heads? Especially, you argue that previous work wrongly interprets the impact of induction heads because some induction heads that also behaved like FV heads. However, in the following section, you argue that there is no significant overlap, which sounds a bit contradictory to me. I suggest more discussions, perhaps along with more analysis, are needed.\n- Similarly, when you ablate one type of head, it would be much helpful if the average score/percentile of the other type can be reported as well. I do not think the current design well separates the effect of the two types of heads.", "other_comments_or_suggestions": "- It would be great if the performance of 11 out of 37 tasks and the new 8 tasks can be separately displayed. The performance gap helps to understand if the conclusion can be really generalized to new tasks.", "questions_for_authors": "Please see the weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XU2BGTiPde", "reviewer": "Reviewer_2akh", "summary": "This paper studies the functionality of attention heads in in-context learning. Specifically, two functionalities are investigated, namely induction heads and function vector heads. By using the established metrics to locate induction heads and function vector heads across layers, several observations have been made. For example, induction heads and FV heads are distinct but correlated; FV heads are more influential for ICL performance compared to induction heads; FV heads appear in higher layers while induction heads appear in lower layers; FV heads evolve from induction heads. Experiments on language models of different scales across a number of ICL tasks reveal these patterns, together with detailed analysis.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! We will first address the concerns around claims and evidence:\n\n* On the layer distribution, we will clarify that the claim on the layer depth difference is more of a speculative observation than a claim sufficiently backed up statistically.\n* On the selection of 2%: the 2% threshold was carefully chosen following previous work by Todd et al. (2024), and after we verified this threshold does seem to meaningfully separate important heads. Figure 7 in appendix shows the 2% threshold does select heads outside the cluster of other low-scoring heads - taking a higher percentage would select heads with FV and induction scores close to 0, which would prevent meaningful analysis. We also repeated ablation experiments taking 5% and 10% as thresholds, and find that our claims are robust to the thresholds.\n* On head overlap: the main goal of the claim on overlap is to establish that the two sets of heads we're studying are not identical, to motivate subsequent experiments comparing the two sets. Our ablation and training experiments later that shows different trends for FV vs. induction heads further reinforces that the two are distinct types. If we take the top 5% or top 10% of heads, there would be between a 20-40% overlap (except for our smallest model with a 67% overlap for top 5% and 80% overlap for top 10%) - while taking 5% and 10% as the threshold does not make as much sense since it includes heads with FV and induction scores close to 0, it still shows that the overlap is low enough to justify treating the two sets of heads as distinct heads for study.\n* On the rigor of conjectures: we would like to clarify that these are mere conjectures, inspired from observations in our study, but that remain to be rigorously proven and we are not positing them as empirically verified claims. We leave these conjectures as discussions and interpretations of our empirical findings, and to inspire future analyses.\n\nWe also address weaknesses raised by the reviewer:\n* On ICL tasks: our study combines ICL tasks that are synthetic but require reasoning that is achievable by the models we study, with realistic tasks (such as translation, commonsense question answering, news topic classification, etc.) so we believe that the selection of ICL tasks is sufficient for our analysis. We do appreciate the suggestion for realistic tasks, and will consider adding further tasks that is feasible by our models (such as sentiment classification).\n* On token loss: the difference between TL difference and few-shot ICL accuracy was first described in Olsson et al. (2022), but we will include a discussion in our revision as well. In short, TL difference approximates how much the model is using context by measuring how much the model's loss decreases on tokens further in context than earlier. When we observe high ICL accuracy with low TL difference (which is what we observe when we ablate random heads and induction heads for example), this means the model preserves its ICL abilities but loses other abilities associated with learning important signals from context. These other signals could capture non-ICL signals such as remembering entities, tracking style from context, etc. It would be great for future work to analyze what is being lost when we see these drops in TL difference!\n* On newer models: we are currently reproducing all experiments except section 5 on training dynamics on Llama 3 8b and Qwen 2.5 7B. Since our conclusions have applied to three families (GPT-2, Pythia, Llama 2) that were released in different years (2019, 2022, 2023), we would expect our conclusions to generalize to the models released in 2024 as well. If surprisingly they don't, we would still like to share the interesting negative results!\n* On suggestions for future development: while papers in the interpretability literature are often not expected to provide this (e.g. the Olsson et al. (2022) and Todd et al. (2024) papers that we compare the mechanisms for do not) as the interpretability contributions are standalone from model development, we will include a discussion on how evidence of the higher influence of FV heads may suggest that this is a more effective mechanism than induction heads. This suggests that ICL can be optimized in small models by using training methods that promote the formation of FV heads.\n\nThank you again for your feedback, please let us know if any questions or concerns remain unsolved!", "claims_and_evidence": "The paper proposes several claims. Most of them are revealed from empirical results, but some of them are not strongly supported. \n- The authors claim that \"induction heads appear in early-middle leayers and FV heads appear in slightly deeper layers\". However, it is not so evident from Figure 2. The average layer may not sufficiently inform the distribution of heads. It is also not clear to me why the authors choose 2% as the selection criteria.\n- The authors claim \"The induction heads and FV heads are distinct\", but \"they are correlated\". Figure 3 shows that FV (induction) heads contain 90-95 percentile of induction (FV) heads. That means, whether induction heads and FV heads are overlapping or not depends on how you set the threshold (in this paper, the authors set it as top 2%). In that sense, the above claim seems not rigorous to me.\n- The conjecture that \"FV heads implement more complex or abstract computations than induction heads\" is not empirically validated or analyzed.\n- The conjecture about the evolution of induction heads to FV heads does not seem rigorous to me.", "methods_and_evaluation_criteria": "The method and evaluation criteria make sense.", "theoretical_claims": "There is no theoretical claim.", "experimental_designs_or_analyses": "I have checked the validity of the experimental designs and analyses.", "relation_to_broader_scientific_literature": "The relationship between induction heads and FV heads and their distributions are relevant to LLM interpretability and the broad area of ICL behavior analysis. The findings could be useful for future research in this area.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n- The investigation between induction heads and FV heads is interesting and meaningful to future research.\n- Extensive controlled experiments are made to uncover the characteristics of the two heads and their relationships.\n- A number of interesting findings are provided which could potentially guide ICL design and LLM training. \n\nWeaknesses:\n- The ICL tasks being tested are a bit synthetic, such as \"Capitalize first letter\", \"choose first of list\". Incorporating more realistic tasks could strengthen the impact of the analysis.\n- Some claims are not very strongly supported. Please refer to the \"Claims And Evidence\" section for more details.\n- The discussion of token loss is not clear to me. More explanations on why token loss can be used as the metric and why it presents opposite pattern compared to ICL accuracy should be given.\n- The evaluated LLMs are mostly Pythia and GPT-2. More advanced and update-to-date models could be incorporated to make the conclusion general and appropriate in the current development.\n- The paper lacks insightful suggestions for future development of ICL or LLMs. Given the findings in this paper, what are critical takeaways when developing future ICL or LLM training techniques?", "other_comments_or_suggestions": "NA", "questions_for_authors": "Refer to the above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HWpyx9urft", "reviewer": "Reviewer_jdwn", "summary": "The authors investigate the nechanisms behind in-context learning (ICL). Specifically, they study about two special types of attention heads called \"induction heads\" and \"function vector heads\" (FV heads). By detecting these two heads, they find that: 1) The induction heads and FV heads are distinct; 2) FV heads are mainly response to the ICL; 3) Some FV heads evolve from induction heads.\n\n\n## Update after rebuttal\n\nI think the most important concerns have been solved, if all the mentioned modifications are finally applied. Now I think the paper is OK to be accepted but I feel a litter unconfident because the large difference between these two version (and I cannot see the complete changing log due to the ICML policy --- Honestly, I feel somewhat disappointed about restricting the depth and format of rebuttals, especially when both authors and reviewers are willing to engage in more thorough discussions.) I will raise my score to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback, and we especially appreciate the clear questions to resolve reviewer concerns!\n\nWe first address reviewer concerns on the claims:\n1. We used Pythia models mostly to facilitate the analysis over training dynamics, which other open-source models do not enable since they do not release intermediate checkpoints. We will add Llama 3 8B and Qwen 2.5 7B to our paper - as these models were not yet released during the development of this paper. \n2. In many cases, we had to decide on a threshold to differentiate meaningful FV/induction heads from the long tail of other heads. The 2% threshold was carefully chosen following previous work by Todd et al. (2024), and after we verified this threshold does seem to meaningfully separate important heads (Figure 7 in appendix shows the 2% threshold does select heads outside the cluster of other low-scoring heads - taking a higher percentage would select heads with FV scores close to 0). If we take the top 5% of heads, there would be between a 20-40% overlap (except for our smallest model with a 67% overlap) - which is still a small enough overlap to motivate our subsequent experiments in comparing the two sets of heads. The main goal of the claim on overlap is to establish that the two sets of heads we're studying are not identical, and furthermore, our ablation and training experiments later that shows different trends for FV vs. induction heads reinforces that the two are distinct types. Beyond a 5% threshold, we would be selecting too many heads with very low FV and induction scores that it would not be meaningful to consider them FV/induction heads.\n3. We agree that the observation on layer depths is not statistically backed up - we will clarify that this observation is speculative.\n4. Great suggestion! We have computed and added the evolution of ICL accuracy to Figure 5 of section 5: we observe that in all models, few-shot ICL accuracy begins to improve around the same time as when induction heads appear, and continues to gradually increase throughout training until the end. Since ICL accuracy continues to improve even after the formation of induction heads, we speculate that this suggests the sharp emergence of induction heads contributes to an initial rise in ICL performance, but the emergence of the FV mechanisms contributes to further improvements in ICL (which reinforces our conjecture 1).\n\nWe also address the questions at the end:\n1. Thank you very much for pointing this out and for the thorough, actionable feedback under \"Other comments or suggestions\"! We have incorporated your feedback to remove repetitions; reorganize the structure of related work / background / methods; added suggested references to related work; edited Table 2 in the main body to show the model, parameters and |L| only (while putting the rest in the Appendix); moved figures in page 7 to be closer to their text; added a limitation section (where we discuss limitations on the complexity of ICL tasks studied and the scale of the models in our analysis); added additional markers, grids, subcaptions to figures; removed unnecessary borders and empty images. \n2. We are currently repeating our experiments on Llama 3 8B and Qwen 2.5 7B. We will also try to include models in the 12-13B parameter range that might fit in A100s. However, we cannot perform experiments on larger models since, to compute the FV score, we need to store several caches of all the attention head activations - we will suggest experiments on larger models for future work.\n3. Our conclusion mainly serves to clarify misunderstandings in the current interpretability literature, that attributes few-shot ICL to induction heads, as well as providing general lessons for interpretability such as how interpretability conclusions change with model scale, and the definition of the metric used to measure ICL can also affect conclusions. Since few-shot ICL is still an active field of study in interpretability, we believe our conclusions are very important to share with the field, and we hope that our general lessons for interpretability will also apply to future interpretabiltiy research on other methods!\n\nThank you again for your feedback, please let us know if any questions or concerns remain unsolved!", "claims_and_evidence": "These paper focus on the roles and relations between FV heads and induction heads. The claims are well expressed in Table 1 and these claims are interesting. However, I am not sure whether these claims are well supported:\n\n1. Although the experiments are operated on 14 models, the selection of these models are strange. 1) The Pythia models (released in 2023) and GPT-2 models (released in 2019) are somehow not most newest models and there is only one model in Llama family. 2) There are only Pythia-6.9B and Llama-2-7b models are large models. A common problem in interpretability is that phenomena that originally seemed correct may disappear as the model size increases. \n2. The conclusion that \"induction heads and FV heads are mostly distinct\" are based on the chosen threshold 2%. I wonder if the results are robust to the threshold. I hope the authors can provide an analysis on the threshold.\n3. In figure 19, the locations of FV heads and induction heads are significantly distinct only for the small models. For, GPT-2 Large, GPT-2 XL, Pythia 6.9B, and Llama 2-7B, the p-value is large and cannot get the conclusion that the FV heads and induction heads are distinct. The authors should explain why they choose these models and how the results can be generalized to other models. \n4. Not a drawback but a suggestion: The current section 5 is only about the heads. I believe the conclusions can be more clear if the ICL accuracy is also tested during the training. I wonder if the ICL accuracy is also related to the occurrence of FV heads. These experiments may be also helpful to verify some of your conjecture in section 6.\n\nAdditionally, In section 4.2, the authors state that the token-loss difference and few-shot ICL are different things. I am curious about what in fact the token-loss difference is, and why the induction heads are most responsible for the token-loss difference. Even an intuitive explanation would be helpful.", "methods_and_evaluation_criteria": "The main methods in the paper is locating the FV heads and induction heads, where the probing metrics are proposed in previous works. They also use mean ablation of heads, which is also a typical method to test the function of heads. These methods are not innovative but reliable. One of their contributions is to find that the difference between the previous metrics \"token-loss difference\" and \"ICL accuracy\". I believe this is a good contribution but I still hope the authors can provide more explanation about what the token-loss difference actually reflect.", "theoretical_claims": "No theoretical claims", "experimental_designs_or_analyses": "The experimental is well designed and the logic is clear. First, the authors detect these two heads and then analyze their roles and finally talk about the evolution. Some concerns about the analysis are mentioned in the \"Claims and Evidence\" section.", "relation_to_broader_scientific_literature": "I have no idea about the relation to broader scientific literature. It is a work about interpretability of ICL ability of LLMs. The explanatory work may shed some light on related research, but I am not currently aware of any direct applications of its conclusions.\n\nI think that work about explaining mechanism has its own value, and it may not be directly applied. But some people may worry about the application value, especially considering that the capabilities of few-shot ICL have been replaced to some extent by instruction tuning, zero-shot ICL, or RLHF. For me, the lack of direct application value is not a disadvantage, but if some more practical applications can be pointed out (out of the interpretability field), it may be more inspiring for other readers.", "essential_references_not_discussed": "Essential references are discussed. But the discussion is highly mixed with their own methods. It is the most shortcoming of this article.\n\nThere could be some relatively related works that can be included in the related work. For example, there are some works explaining the ICL ability from the perspective of Bayesian inference [1] or gradient descent [2,3]. These works are better to be included in the related work.\n\n[1] S. Xie, et. al., An Explanation of In-context Learning as Implicit Bayesian Inference\n\n[2] R. Ren and Y. Liu, Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens\n\n[3] H. Sun, et. al., In-Context Learning of Polynomial Kernel Regression in Transformers with GLU Layers", "other_strengths_and_weaknesses": "1. These paper further investigate the training dynamics of the models. The conclusion about how the functions of heads evolve (or change) during training is very interesting and important. \n2. Their discussion in section 6 is very interesting and inspiring. I hope the authors can further investigate them in the future.", "other_comments_or_suggestions": "I believe the organization of the paper needs to be improved. The current article is poorly readable.\n\n1. There are many repeated expressions:\n    1. In the beginning of section 4, the authors have state that \"we control for the correlation between induction and FV heads by only ablating induction heads that do not have high FV scores, and vice versa\". However they repeat this point in line 263-267: \"we take the top n heads by FV score that do not appear in the top 2% heads\". If the specific statement has been appear, the upper one can be removed.\n    2. The figure 1 is copied from the figure 3 and figure 4. Although the author may think that this helps the reader understand the content of the article more quickly, it is still a form of repetition that should be avoided.\n    3. The figure 5 is just the average value of figure 6. The figure 6 is enough to express the conclusion.\n2. I recommend the section 2 background & related work should be splitted into two sections. The \"related work\" should introduce the ideas of related work and also the relation between theirs and the work. Too specific discussion should not be included in the section. On the other hand, some preliminary like how they define their detectors can be stated in a preliminary subsection. Further, all the statement about your methods (even if these methods are not proposed in these paper) should be placed into a method section. For example, the statement in line 125-150 about how the authors use the causal mediation analysis to identify FV heads, the statement in line 100-114 about how to compute the induction score should be moved to your method section. As for the section 2.3, it could be in introduction or after your experiments.\n3. The table 2 can be moved to the appendix. The authors can keep only the \"parameters\" (and maybe the \"|L|\" column) in the main text.\n4. Now, the page 7 only have three figures and the text explaining them are far from them. Please arrange the size and distribution of the images reasonably so that they appear in the most reasonable position possible to ensure readability. For example, you can together the three models in one figure in Figure 5 and use different color or marker to distinguish them.\n5. Please add a limitation section.\n6. The figures could be further improved. For example, the markers should be added in line chart. The upper and right border of the figures can be removed. Some grids can be added. In figure 4, some sub caption could be helpful to understand the figures in each row.\n7. The figure 10 should add more explanation in its caption.\n8. The figure 11 has an empty image.", "questions_for_authors": "1. My main concern is the poor structure of the article, especially the repetitive expressions, the confusion of related work, background introduction and methods, which seriously affected my understanding of the content of the article. I approve of the content of the article but the organization it is currently presented makes it not ready for publication.\n2. How can the results be generalized to newer and larger models? At least some Llama-3 and qwen-2.5 models should be included. It is better to include some MoE models or models with GQA. As for the param size, I believe the location of heads can be operated on a 70B model on one A100 GPU. (If I am wrong, please correct me.)\n3. How does the conclusion can be applied, especially considering the few-shot ICL has been replaced by some other methods?\n\nIf the question 1 has been solved, I will raise my score to 2 or 3. And if the question 2 has been solved, I will raise my score to 3 or 4. If the question 3 can be explained well, I will raise my score to 4 or 5.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers unaninmously appreciate that the paper tackles an interesting and important problem concerning the mechanism behind in-context learning in language models. All reviewers praise the paper's results as insightful, compelling and as having the potential to inspire interesting future work. All reviewers also note that there are comprehensive, controlled experiments across a large number of models. 

There were multiple concerns brought up in the initial round of reviews, which the reviewers and I agree, have been addressed well. 

**Summary of concerns and how they were addressed**: 

- Reviewer `joQW ` makes an interesting point: the qualitative results here intuitively must be task-dependent while the claims suggest that they will generalize to all tasks. The reviewer suggested some alternative tasks to test.
     - The authors have clarified this well with new results.
- Some sub-claims (like the layers where some heads appear) seem to be model- or model-size- dependent (`2akh`, `joQW`). 
     - The authors acknowledge this is speculative. I encourage the authors to edit these sub-claims accordingly.
- The seemingly-arbitrary choice of a 2\% threshold to determine the FV heads (`joQW`, `wNzV`, `Eo6g`, `jdwn`).  
     - The authors clarify how this is based off of the score distributions, and how a 5\% threshold would imply greater overlap between FV and Induction heads. Multiple reviewers agreed later that the choice is not arbitrary 

Onto other concerns that I believe were reasonably addressed to begin with:
- Lack of practical applications (`2akh`, `wNzV`). 
     - However as Reviewer `jdwn` nicely puts it, this should not be penalized. Work that understands model behavior is insightful in its own right.
- Model choices are old (`Eo6g`, `2akh`) or strange (`jdwn`). 
     - Although an understandable concern, I agree with the authors these are model families across three different years. If the findings that replicate within these families do not generalize to other families, that would in itself make interesting future work.

I hope the authors found the reviews helpful in polishing their paper.
Decision: Accept (poster)