ICML 2025

Beyond Induction Heads:
In-Context Meta Learning Induces Multi-Phase Circuit Emergence
Gouki Minegishi 1 Hiroki Furuta 1 Shohei Taniguchi 1 Yusuke Iwasawa 1 Yutaka Matsuo 1
Abstract
Transformer-based language models exhibit In-
Context Learning (ICL), where predictions are
made adaptively based on context. While prior
work links induction heads to ICL through a sud-
den jump in accuracy, this can only account for
ICL when the answer is included within the con-
text. However, an important property of practical
ICL in large language models is the ability to
meta-learn how to solve tasks from context, rather
than just copying answers from context; how such
an ability is obtained during training is largely un-
explored. In this paper, we experimentally clarify
how such meta-learning ability is acquired by ana-
lyzing the dynamics of the model’s circuit during
training. Specifically, we extend the copy task
from previous research into an In-Context Meta
Learning setting, where models must infer a task
from examples to answer queries. Interestingly, in
this setting, we find that there are multiple phases
in the process of acquiring such abilities, and that
a unique circuit emerges in each phase, contrast-
ing with the single-phases change in induction
heads. The emergence of such circuits can be
related to several phenomena known in large lan-
guage models, and our analysis lead to a deeper
understanding of the source of the transformer’s
ICL ability.
1. Introduction
Transformer-based language models (Vaswani et al., 2017)
show an intriguing ability to perform In-Context Learning
(ICL) (Brown et al., 2020; Xie et al., 2021; Garg et al., 2022;
Dong et al., 2024). ICL is the ability to predict the response
to a query based on context without any additional weight
updates. A widely adopted application of ICL is few-shot
*Equal contribution 1The University of Tokyo. Correspondence
to: Gouki Minegishi <minegishi@weblab.t.u-tokyo.ac.jp>.
Proceedings of the 42 nd International C...