ICML 2025

Which Attention Heads Matter for In-Context Learning?
Kayo Yin Jacob Steinhardt
UC Berkeley
Abstract
Large language models (LLMs) exhibit impres-
sive in-context learning (ICL) capability, enabling
them to perform new tasks using only a few
demonstrations in the prompt.
Two different
mechanisms have been proposed to explain ICL:
induction heads that find and copy relevant to-
kens, and function vector (FV) heads whose acti-
vations compute a latent encoding of the ICL task.
To better understand which of the two distinct
mechanisms drives ICL, we study and compare
induction heads and FV heads in 12 language
models. Through detailed ablations, we discover
that few-shot ICL performance depends primarily
on FV heads, especially in larger models. In ad-
dition, we uncover that FV and induction heads
are connected: many FV heads start as induction
heads during training before transitioning to the
FV mechanism. This leads us to speculate that in-
duction facilitates learning the more complex FV
mechanism that ultimately drives few-shot ICL1.
1. Introduction
One of the most remarkable features of large language mod-
els (LLM) is their ability to perform in-context learning
(ICL), where they can adapt to various new tasks using
only context given in the prompt at inference time. This
capability has become crucial to adapt pre-trained LLMs to
specific tasks, sparking significant research into its underly-
ing mechanisms (Olsson et al., 2022; Aky¨urek et al., 2023;
von Oswald et al., 2023).
To date, two key mechanisms have been associated with
ICL, each supported by different lines of evidence. First,
induction circuits (Elhage et al., 2021) were hypothesized
. Correspondence to: Kayo Yin <kayoyin@berkeley.edu>,
Jacob Steinhardt <jsteinhardt@berkeley.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
1Code and data:
https://github.com/kayoyin/
icl-heads.
2.5
5.0
7.5
10.0
12.5
15.0
17....