Revisiting In-context Learning Inference Circuit in Large Language Models
Authors: Hakaze Cho
Mariko Kato
Yoshihiro Sakai
Naoya Inoue
Venue: ICLR 2025 Poster (2025)
Keywords: In-context Learning; Induction Circuit; Mechanistic Interpretability
Primary area: interpretability and explainable AI
TL;DR: We decompose In-context Learning into 3 operations and measure their operating dynamics to catch many inference phenomenon of ICL in Large Langauge Models.

Abstract: In-context Learning (ICL) is an emerging few-shot learning paradigm on Language Models (LMs) with inner mechanisms un-explored. There are already existing works describing the inner processing of ICL, while they struggle to capture all the inference phenomena in large language models. Therefore, this paper proposes a comprehensive circuit to model the inference dynamics and try to explain the observed phenomena of ICL. In detail, we divide ICL inference into 3 major operations: (1) Input Text Encode: LMs encode every input text (in the demonstrations and queries) into linear representation in the hidden states with sufficient information to solve ICL tasks. (2) Semantics Merge: LMs merge the encoded representations of demonstrations with their corresponding label tokens to produce joint representations of labels and demonstrations. (3) Feature Retrieval and Copy: LMs search the joint representations of demonstrations similar to the query representation on a task subspace, and copy the searched representations into the query. Then, language model heads capture these copied label representations to a certain extent and decode them into predicted labels. Through careful measurements, the proposed inference circuit successfully captures and unifies many fragmented phenomena observed during the ICL process, making it a comprehensive and practical explanation of the ICL inference process. Moreover, ablation analysis by disabling the proposed steps seriously damages the ICL performance, suggesting the proposed inference circuit is a dominating mechanism. Additionally, we confirm and list some bypass mechanisms that solve ICL tasks in parallel with the proposed circuit.

Published as a conference paper at ICLR 2025
REVISITING IN-CONTEXT LEARNING INFERENCE CIRCUIT IN LARGE LANGUAGE MODELS
Hakaze Cho
Mariko Kato"
Yoshihiro Sakai
Naoya Inouc
JJapan Advanced Institute of Science and Technology 2RIKEN
Primary Contributor, Correspondence to: : yfwhaaojjast......
ABSTRACT
In-context Learning (ICL) is an emerging few-shot learning paradigm on Language Models (LMs) with inner mechanisms un-explored. There are already existing works describing the inner processing of ICL, while they struggle to cap- ture all the inference phenomena large language models. Therefore, this paper proposes comprehensive circuit to model the inference dynamics and try to ex- plain the observed phenomena of ICL. In detail, we divide ICL inference into 3 major operations: (1) Input Text Encode: LMs encode every input text (in the demonstrations and queries) into linear representation in the hidden states with sufficient information to solve ICL tasks. (2) Semantics Merge: LMs merge the encoded representations of demonstrations with their corresponding label tokens produce joint representations of labels and demonstrations.
(3) Feature Re- trieval and Copy: LMs search the joint representations of demonstrations similar the query representation on task subspace, and copy the searched represen- tations into the query. Then, language model heads capture these copied label representations certain extent and decode them into predicted labels. Through careful measurements, the proposed inference circuit successfully captures and unifies many fragmented phenomena observed during the ICL process, making it a comprehensive and practical explanation of the ICL. inference process. More- over, ablation analysis by disabling the proposed steps seriously damages the ICL performance, suggesting the proposed inference circuit is dominating mecha- nism. Additionally, we confirm and list some bypass mechanisms that solve ICL tasks in parallel with the proposed circuit.
INTRODUCTION
In-Context Learning (ICL.) (Radford et al., 2019; Dong al., 2022) is an emerging few-shot learning
paradigm: given the demonstrations {(r,, 31) ]= consisting of [input text]- token] pairs and
a query Zg., Language Models (LMs) take the sequence [x1 [r.][s1[[v]]- [ [mx]] [s [yve] [x.][s (Fig. 1) as input and then predicts the label for x.g by causal language modeling operation. Typically, the label tokens y; are preceded by and also predicted on forerunner tokens 8; (e.g., the colon in "Label: "). ICL has aroused widespread interest, but its underlying mechanism is still unclear.
There have been theoretical or empirical trials to characterize and explain the inference process of ICI. (Xie et al., 2021; Dai et al., 2023; Wang et al., 2023; Han et al., 2023a; Jeon al., 2024; Zheng et al., 2024). However, to capture all the operating dynamics and many fragmented observed interesting phenomena of ICL in Large Language Models (LLMs), a more comprehensive char- acterization is still necessary. Therefore, this paper tries to propose a unified inference circuit and measures various properties LLMs for conformation the observed ICL phenomena.
In detail, as shown in Fig. 1, we decompose ICL dynamics into 3 atomic operations on Transformer layers. Step INPUT TEXT ENCODE: LMS encode each input text x; into linear representations in the hidden state of its corresponding forerunner token Step 2: SEMANTICS MERGE: For
In this paper, we denote tokenization as and token concatenating as [-[[-.
2 Large refers to scaled LMs trained by natural language data, such as Ilama 3 (AI Meta, 2024), contrast simplified work that uses simple models trained and test well-embeddee input in toy models.

---

Published as a conference paper at ICLR 2025
demonstrations, LMs merge the encoded representations of 8; with the hidden state of the corre- sponding label tokens y/- Step 3: FEATURE RETRIEVAL AND COPY: LMs retrieve the similar memand label representations Vı:k (from Step 2) to the query representation sg in task- relevant subspace, then copy them into the query's ' forerunner token representation. Finally, LM heads predict the label for Zi using the label-attached query representation 8y. Steps and 3 form typical induction circuit, which is a key mechanism of ICL but only examined in synthetic scenar- ios (Elhage et al., 2021; Singh et al., 2024b; Reddy, 2024).
We empirically find evidence for the existence of each proposed step in LLMs, and conduct more fine-grained measurements to gain in- sights into some phenomena observed ICI. scenarios, such as (1) positional bias: the pre- diction is more influenced by the latter demon- stration (Zhao al., 2021), (2) noise robust- ness: the prediction is not easy to be affected by demonstrations with wrong (noisy) labels (Min et al., 2022), while larger models are less ro- bust to label noise (Wei al., 2023); and (3) demonstration saturation; the accuracy im- provements plateau when sufficient demonstra- tions are given (Agarwal et al., 2024; Bertsch al., 2024). etc. (discussed in $5.3). More- over, we find multiple bypass mechanisms for ICL conducted by residual connections, while the 3-phase dynamics remain dominant.
LM Head
Vainn inkermation Floe
Derparitz
Step Pesture Retrieval & Copyy
Transformer Layers
Step Semantics Merge
Step
Imput Enoode
Input
Test
Irput Text Xg
Fmmally How Uris morile, Labal
No: good, Labeb Sulisfactory. Labal:
Figure 1: The 3-phase inference diagram of ICL. Step 1: LMs encode every input text into repre- sentations, Step 2: LMs merge the encoded text representations of demonstrations with their cor- responding label semantics. Step 3: LMs retrieve merged label-text representations similar to the encoded query, and copy the retrieved represen- tations into the query representation.
Our contributions can be summarized as: (1) We propose a comprehensive 3-step inference circuit to characterize the inference process of ICL, and find empirical evidence of their existence in LLMs. (2) We conduct careful mcasurements for each inference step and successfully capture large number of interesting phenomena observed in ICL, which enhances the practicality of the proposed circuit. (3) Our ablation analysis suggests that the proposed circuit dominates, but some bypass mechanisms exist in parallel to perform ICL, and we introduce some of these bypasses along with their empirical existence evidence.
PREPARATION
2.1 BACKGROUND & RELATED WORKS
In-context Learning. Discovered by Radford et al. (2019), ICL is an emerging few-shot learning
paradigm with only feed-forward calculation in LMs. Given demonstrations {(z,, v:) , composed of lnbut-label pairs and a query Z... typical ICL creates input [ [821 ... with some structural connectors (c.g. "Label: "" including forerunner token s; (c.g. ": "), as shown in Fig. 1. LMs receive such inputs and return the next token distribution, where the label token with the highest likelihood is chosen as the prediction. Explaining the principle of ICL is an open question, although there have been some efforts on the relationship between ICL capacity and pre-training data (Li & Qiu, 2023; Singh et al., 2024b;a; Gu et al., 2023; Han et al., 2023b; Chan et al., 2022), the feature attribution of inputs (Min et al., 2022: Yoo al., 2022; Pan, 2023; Kossen et al., 2024), and reduction to simpler processes (Zhang et al., 2023; Dai et al., 2023; Xie et al., 2021: Han et al., 2023a). However, a comprehensive and unified explanation on real-world LMs is still needed to capture the operating dynamics of ICI..
[=
Induction Circuit. Introduced by Elhage et al. (2021), an induction eircuit a pair of two cooper- ating attention heads from two transformer layers, where the "previous token head" writes informa- tion about the previous token to each token, and the "induction head" uses the wroten information to identify a token that should follow each token. Concretely, such function is implemented by two atomic operations in attention calculations: (1) copy the representation of the previous token [A] to the next token [B], and (2) retrieve and copy similar representations on [B] (copied from [A]) to the current token A'.. Concisely, it performs inference in the form of [A][B] ... [A'] + [B], which is similar to the ICL diagram. Therefore, the induction circuit has been widely used to explain the in-
2

---

Published as a conference paper at ICLR 2025
ference dynamics of ICL (Wang et al., 2023) and the emergence of ICL during pre- pre-training (Olsson et al., 2022: Reddy, 2024: Singh et al., 2024b). Despite their valuable insights, these experiments rely synthetic setting: using simplified models and well-embedded (linearly separable) inputs, which differs from the practical ICL scenario using real-world LMs with many layers and compli- cated inputs: believe these synthetic works indicate the potential of Transformers optimal behavior, and can not demonstrate that their observation exists in scaled LLMs trained on wild data. So, in this paper, we try to bridge the gap between the synthetic and real-world settings with more detailed observations for the inference of ICL on real-world LLMs.
locally
2.2 EXPERIMENT SETTINGS
Models. We mainly conduct experiments on 4 modern LLMs: Llama 3 (8B, 70B) (AI@Meta, 2024),
and Falcon (7B, 40B) (Almazrouei al., 2023). Unless specified, we report the results on Llama 70B, since its deep and narrow structure (80 layers, 64 heads) makes it easier to show hierarchical inference dynamics (discussed in $5.2). The results of other models can be found in Appendix H.2.
Datasets. We build ICL-formed test inputs from 6 real-world sentence classification datasets, and unless specified, we report the average results on them: SST-2 (Socher et al., 2013), MR (Pang & Lee, 2005), Financial Phrasebank (Malo et al., 2014), SST-5 (Socher et al., 2013), TREC (Li & Roth, 2002; Hovy et al., 2001), and AGNews (Zhang et al., 2015).
use k 4 demonstrations in ICL inputs. For each dataset, randomly sample 512 test data points and assign one fixed demonstration sequence for each test sample to form test input. About the prompt templates, etc., please refer Appendix A.I.
Others. Unless specified,
3. STEP 1, INPUT TEXT ENCODE: SEMANTICS ENCODING AS LINEAR REPRESENTATIONS IN HIDDEN STATES
This section mainly confirms that LMs construct task-relevant and linearly separable semantic rep- resentations for every input text (demonstrations and queries) in the hidden states. Such linear representations are an important foundation for explaining the dynamics of ICL based on induction heads, since ceanttoonbbsedd feature retrieval, a key mechanism of induction heads, can be easily done linear representations. Current successful studies on simplified models and inputs (Chan et al., 2022; Reddy, 2024; Singh et al., 2024b) also assume the existence of such linear representa- tions. Moreover, we confirm some interesting properties of the encoded input text representations: (1) It is based on the capacity in the model weights and can be enhanced by demonstrations in context. (2) The similarity of representations is biased towards the encoding target's position.
3.1 LLMS ENCODE INPUT TEXT ON FORERUNNER TOKENS HIDDEN STATES
We first study the existence of input text encoding in hidden states and then explain their linear separability and task relevance in $3.2. For each text-label pair (zt, yt) (encoding target) sampled from the datasets, we prepend them with & demonstrations, resulting in augmentated (by label (yc)) [<>]][s]]]] [[<<][ [s]]
[w)] These inputs are then fed into an LM to extract the hidden states of specific token in [re] [ [+] from each layer, serving as the ICL in- sentence representations, x, encoded by BGE M3 (Chen et al., 2024), a SotA encoder-only Transformer, reference representation and then calculate the mutual nearest-neighbor kernel alignment' (Huh et al., 2024) between these representations. See Appendix A. and A.2 for details.
ICL-styled inputs [r]](s][[и ... [r [sx
ner representations. To assess the quality of these representations
use the sentence embedding
Forerunner Tokens Encode Input Text Representations. We plot the kernel alignment the types of tokens in Fig. 2 (Left), where the forerunner token, while often overlooked in previous work, produces the best input text encoding, emerging in the early phase (layers 0-28) of the inference process, and keeping a high level to the end of inference. Interestingly. hidden states of label words are not satisfactory input text representations even with high background value (the result at layer
Intuitively, kernel alignment measures similarity hetween representations toward the same datasets, and according Huh et al. (2024), higher cross-model kernel alignment usually means better representation.
3

---

Published as a conference paper at ICLR 2025
#lop0-00.010 k=
Kornel Alimmeent
Kornel Aliymzent
0.30
# Demn Rand
CorL
Aligusent
0.25
0.20-
Tokos Type Randen Banillaa Exprrizeet
0.26
Hloge +0,008 K=8
0.35
0.35
Tr Tine Token Laaeel
0.30
0.5
Transformer Number
Transformer Number
Language Modeling Loss
Figure 2: Input text encoding magnitudes (metricized by kernel alignment with feature encoded by an encoder-structure model) of hidden states in various layers in ICL scenario (The controlled experiments are results between current 6 datasets and THE (Mohammad et al., 2018)). Left: En- coding magnitudes on hidden states from various types of token. Middle": Encoding magnitudes with different & on the forerunner tokens. Right: Encoding magnitudes in layer 24 of Llama 70B against the causal language modeling loss of the input text with (upper) k = 0 and (lower) k = 8.
O, refer to Appendix A.2.1), which is a critical supplement to previous work which suggests the label tokens are pivots for collecting the information of demontrations (Wang et al., 2023).
Input Text Encoding is Enhanced by Demonstrations. We investigate the influence of contextual information on input text encoding by repeating the experiments with different k. As shown in Fig. (Middle), when the demonstrations increase, kernel alignment is enhanced, which is counterintuitive since longer preceding texts are more likely confuse encoding targets. Such findings indicate that LMs (1) utilize contextual information to enhance the input text encoding and (2) correctly segment different demonstrations (detailed operation discussed in Appendix C).
Perplexed Texts are Encoded Worse. the perplexity of encoding targets with various k. Fig. 2 (Right, upper) shows negative correlation for k = 0, that is, LMs generate poorer encodings for more perplexed input text when no demon- strations are given, which can be identified as an In- in-weight Learning (IWL) property of the inner text encoding. While, when demonstrations are given in context (Fig. 2 (Right, lower)), the nega- tive correlation disappears, which suggests that LMs effectively encode more complex samples with the help of demonstrations in context. More discussion about the correlation between classification performance and perplexity is in Appendix F.
We investigate the correlation between kernel alignment and
The above findings suggest that the inner text encoding is hybrid process of ICL and IWL: Basic encoding capability presents from LMs weights, and is enhanced by demonstrations in context, which can be clue to how demonstrations help ICL. Moreover, we are about to illustrate that these encodings are sufficiently informative for ICL tasks and linearly separable, which meets the presumption of simplified models on the linear and well-embedded input features.
3.2 INPUT TEXT ENCODING IS LINEAR AND TASK-RELEVAR BUT POSITION-BIASES
Input Text Encoding is Linear Separable and Task-relevant. We train a centroid classifier on hold-out 256 input samples (Cho et al., 2024), using the hidden states of specific token in [2r][s,]s from each layer and then predict the label ye (see Appendix A.3
0.8
Kornll Al Mignment
0.7
Aroureny
0.6
for details). The results are shown in Fig. 3, where considerably high classification accuracy of the forerunner token suggests the high linear separabilities of the hidden states in the task-semantic- relevant subspaces since the centroid classifier is linear. In addi- tion, a similar emerging trend in accuracy and kernel alignment confirms the reliability of the kernel alignment measurement.
0.s
0,29
0.4
Tolen Type
0.5
Poreuzaar Tolen Lfba
Last Tokun cf
0.35
60
Transformer Block Number
Figure 3: Test results of cen- troid classifier trained on ICL. hidden states. Solid: Centroid classification accuracy, Dotted: Kernel alignment.
Input Text Encoding is Biased towards Position. Ideally, the inner representations of similar queries should be highly simi- lar regardless of their position in ICL. inputs to support attention-
'Experiments of Fig. 2 (Middle) on Ilama 3 70B do not involve results on AGNews.
We skip the experiments label tokens because of the Icakage of ground-truth lahel information.
4

---

Published as a conference paper at ICLR 2025
based operations for classification. To verify this, for each encoding target, we extract the hidden states of forerunner tokens with various numbers of preceding demonstrations, and then calculate the cosine similarity between all possible pairs of the hidden states for the same target or different targets. As shown in Fig. 4, although the overall similarities on the same target are higher than on the different targets, they are both especially higher when their positions to be discussed in $5.3, such positional similarity bias may lead to one flaw: Demonstrations closer to the query have stronger impacts on ICL (Zhao et al., 2021; Lu al., 2022; Chang & Jía, 2023; Guo et al., 2024). The principle of such bias is discussed in Appendix C.
close to each other. As
INDUCTION CIRCUITS IN LARGE LANGUAGE MODELS
This section mainly shows how LMs utilize the encoded linear text representations in induction circuits with typ- ical 2-step form (Singh et al., 2024b): Forerunner To- ken Heads merge the demonstration text representations in the forerunner tokens into their corresponding label tokens with a selectivity regarding the compatibility of demonstrations and label semantics. Induction Heads copy the information in the label representations similar to the query representation back to the query, which is conducted on task-specific subspaces, enabling LMs to solve multiple tasks by multiplexing hidden spaces.
0.72
2:05
0.5
0.5 2.76 0.::
1.00
0.35
0.5
Query Location
Quacy Location
Figure 4: states in different positions on layer 24 between Left: the same queries. Right: two different queries (on SST-2).
The similarities of ICL hidden
4.1 STEP 2, FORERUNNER TOKEN HEAD: COPY FROM TEXT FEATURE TO LABEL TOKEN
This subsection mainly examines and measures the forerunner token heads, which copy the informa- tion in the forerunner token into label tokens. We focus on how the representations in these tokens are merged, especially when the semantics of demonstrations and corresponding labels are disjoint, to explain the robustness of ICL against wrong labels present in the demonstrations.
Text Representations are Copied to Label Tokens. To confirm the existence of the representation copy process, we start by calculating the kernel alignment between the hidden state of forerunner token [st] at layer/ / (the copy source) and that of label token [yr] at layer (1+ + 1) (the copy target). To suppress the high background values caused by the semantics of labels (refer to Appendix A.2.1), we use abstract label tokens ("A", "B", "C" ..... instead of the original label tokens. The results are shown in Fig. 5 (Left), where the kernel alignment between the hidden states of the label token and the forerunner token gradually increases and then bumps up after the encoding magnitude (described in $3) in the forerunner token finished ascending. Such phenomenon indicates that the hidden states of the input text representation encoded in the forerunner tokens are merged into their label tokens, suggesting the existence of copy processing from the forerunner token to the label token.
Text Representations are Copied without Selectivity. For each attention head, we extract the attention score" Os,,-,, from the forerunner token [sæ] (as attention key) to the corresponding label token [yi] (as attention query). We then mark the head with as.-tz. > 5/n, (ny: the length of tokens before [u:]) as a Forerunner Token Head and count them in each layer. The results are shown in Fig. (Middle, "Correct Label"),, where the peak matches the copy period in Fig. (Left). Moreover, investigate the influence of the correctness of label tokens, we replace [yd] with a wrong label token?, where the results in Fig. 5 (Middle, "Wrong Label") are almost identical to the correct- label setting, suggesting that the forerunner token heads don't show selectivity toward the semantic consistency between input text and labels, and simply merge the input text representations into the label tokens. Furthermore, we find (Appendix D) that the copy processing is inherent: During the copy processing, the preceding tokens are copied to the subsequent tokens, regardless of whether these copied tokens are forerunner tokens, which indicates that such copy processing is a universal inference behavior established during pre- pre-tinin rather than being evoked by the special tokens in the in-context learning (ICL) input, while still aiding ICL process (discussed in 15.1).
This paper use notation œg-Q to denote attention score with attention query Q and attention key K, which is same with the flow of information. For example, for lahel space of "positive" and "negative", if [ys] is "positive", we replace it to "negative".
5

---

Published as a conference paper at ICLR 2025
Copy Magnibude
Kocnel Alignmentt
0.40
Fororunmer Tokan Hoad
30-
Coge To:
Carnent Labo
Wiong Label
11.
0.0
20-
Accuracy
0.25
35-
10-
0.6
0.20
Rendee Bankze
Cogrect Lakel
Abslaed Labe: Derocanon
Paterenant
0.15
Brroding
Coctret Labet upj Content
0.0
30
20
40
60
80
20 40
60
Transformer Block Number
Transformer
Block Number
Transforeer Block Number
Figure 5: Hidden states copy magnitude from forerunner tokens to label tokens against layers. Left: Kernel alignment between the forerunner token (the copy source) and the abstract label token of the next layer (the copy target). Middle: Curves: The count of marked forerunner token heads with correct and wrong labels; Colored Areas: The maximum attention scores from forerunner token to query (copy magnitude) with correct and wrong labels (detailed attention head statistical data is in Appendix H.1). Right: Centroid classifier results predicted on the hidden states of correct and wrong label tokens, on SST-2 and MR. Solid: Predicted by classifiers C, trained on hidden states of forerunner tokens. Dotted: Predicted by classifiers Cy trained on hidden states of label tokens.
Hidden States of Label Tokens are Joint Representations of Text Encodings and Label Seman- tics. Given the findings above, we probe the content of hidden states of label tokens [w.]. i.e., how the copied text representation interacts with the original label semantics. We first train two centroid classifiers to predict the corresponding label y:: (1) C. trained on the hidden states of forerunner tokens [st] and (2) C,, trained on the hidden states of label tokens [y.]. To check whether the label tokens include the information of forerumner tokens, we use C. to predict the label on the hidden state of label token [ye] in Fig. 5 (Right, solid). It shows that, during the copy processing, high classification accuracies can be achieved both on the correct label tokens and wrong label tokens, suggesting that the text features in the forerunner tokens can be partly and linearly detected in the label tokens. Moreover, results using Cy (dotted line) show extreme results, suggesting the label information remains in the label token. So, we can conclude that: Hidden states of label tokens are joint representations of label semantics and text representations. Moreover, interestingly, the accu- racies from C, shown in Fig. 5 (Right) decline after layer 35, suggesting that the information sharing between the forerunner tokens and the label tokens ends in later layers, which aligns with the results in Fig. (Middle).
Label Denoising is Conducted on the Overlap of Label Semantics and Text Representations. Notice that in Fig. 5 (Right, solid), compared to the typical results predicted on the forerunner tokens, accuracies are improved on the correct label and suppressed on the wrong label, which suggests that information consistent with the label semantics is casier to be enhanced by the label tokens and vice versa, showing feature selectivity the consistency between the text representations and the label semantics. Given the observation that the information on label semantics and text features can be extracted separately and linearly, we can confirm that these two kinds of information located in different sub-spaces of the hidden states, and linearly merged by the attention operation of forerunner token heads. Moreover, given the fact that there is no selectivity observed in the copy behavior of the forerunner token heads (Fig. 5 (Middle)), it is intuitive that the feature selectivity shown in Fig. (Right) comes from the arithmetical interaction of feature vectors on the overlap of sub-spaces between the label semantics and text features, making ICL stable against label noise (Min et al., 2022). Moreover, as mentioned by Wei et al. (2023), large models show poorer stability against label noise, and we infer that the larger hidden dimensions in larger models lower the overlap between the sub-spaces of label semantics and text representations to reduce the interaction.
4.2 STEP 3, INDUCTION HEAD: FEATURE RETRIEVAL ON TASK SUBSPACE
This subsection examines the existence of the induction heads, which retrieve label token features similar to the queries' forerunner token feature, and copy the retrieved features back to the query. We claim the necessity of multi-head attention in this process: Correct feature retrieval can only be conducted on the subspace of the hidden space, which is captured by some attention heads.
6

---

Published as a conference paper at ICLR 2025
30-
Indurtion Correct Induction Copy (Steg 21
Correct Label Assignment
0.7
M nw Mmm Mmm
25-
Induction Head
0.6
0.05
20-
0.5
Dittatatt
0:67
15-
6STG
0.59
0.58 0.50
0.4
20-
TRBIC
0.6
0.57
Attention Dest lnd. Hond
Average
AGN
0.59
0.62
0.06
0.68
Handom
557
FF
TRBC
AON
60
Transformer Block Number
Transformer Blook Number
Datasel
Figure 6: Mcasurements for induction heads. Left: The count of marked induction heads and
correct induction heads against layers. Middle: The correctness of attention assignment (the sum of attention scores from query's forerunner token towards correct label tokens normalized by attention scores towards all the label tokens) extracted from the following attention calculations: Vanilla Attention: attention scores directly calculated on full dimensionality. Head Average: the averaged attention scores among all the heads. Best Ind. Head: the scores of attention head with the most correctness. Right: The correct induction heads overlap of all dataset pairs.
Induction is Correct in Minority Subspaces. Similar to Fig. (Middle), we mark (1) the attention heads with the sum of attention scores from all the label tokens in the demonstrations [y.]..... [M]] (as attention keys) to the query's forerunner token [sg] (as attention query) of more than 5k/ng n as Induction Heads, and (2) attention heads with the sum of scores from all the correct label tokens more than 5k//y mr as Correct Induction Heads (Y' is the label space). We show the number of both kinds of induction heads in Fig. 6 (Left, detailed head statisties in Appendix H.1), where an unimodal pattern is observed later than the copy processing of Step 2. Moreover, more than half of the induction heads are not correct ones, suggesting that task-specific feature similarity can only be caught on some induction subspaces (defined by low-rank transition matrix W Wh of correct induction head h). We enhance this claim in Fig. 6 (Middle) (details in Appendix A)4), where both vanilla attention (without transformation and head split) and attention scores averaged among all heads show low assignment on correct label tokens, while some heads show considerable correct- ness. Considering the average value, the majority of attention heads almost randomly copy label token information to the query, causing the final prediction biased to the frequency of labels present the demonstrations (Zhao al., 2021). 1). As the reason, we infer that the hidden states are sufficient (Fig. 3) but not minimum for ICL, where redundant information interferes with the operation of induction heads.
Layar 31, Hoad 32, Correct Rate: 0.95 mm
Layer 31, Head 9, Connect Rate: 0.00 k
x
Some Induction Subspaces are Task-specific. We check if different tasks share the same induction subspaces based on the overlap of the correct induction heads across different datasets. Given nD(hh the number of times h marked as correct induction head on dataset D. the overlap rate is defined as:
Is 15
k- 16
15
k 16
XX
*
*
S(Di, D2) 22mh min(nD, (h), (h)) Eva NDD (h) + nDD (h)
Figure 7: Label representations of k demonstra- tions visualized on Left 4: correct and Right 4: wrong induction head, on one sample of SST-2 (see Appendix H.3). o: "positive" label, x: "neg- ative" label, A: zero vector. Color: Radial den- sity of each categories, to estimate the attention assigned to query, negative to positive (cartogra- phy: Appendix A.5).
The results shown in Fig. 6 (Right), where: (1) A significant overlap of induction heads in- dicates that a part of correct induction heads is inherent in the model, built by the pre-training process (Reddy, 2024; Singh et al., 2024b). (2) Such overlap is not fully observed, suggesting that some induction subspaces are task-specific: Input texts evoke task-specific induction heads, enabling the anisotropy multiplex of different sub- spaces in the hidden spaces to transmit relevant information for various tasks. Therefore, we can analogize ICL multi-task learning also utilizes various task heads on common bottom network layers and informa- tive hidden states (Zhang& & Yang, 2021).
implicit end-to-end multi-task learning with hidden state multiplexing since
7

---

Published as a conference paper at ICLR 2025
Demonstrations Saturate Induction Subspace. We visualize the demonstrations' label token representations mapped on the induction subspaces by the transition matrix Wa W K and princi- pal component analysis in Fig. 7, indicating: (1) Compared to correct induction heads (Left 4), wrong induction heads (Right 4) are easier to map label representations linear-inseparably. (2) In the early stage of demonstration feeding (k - 2), when new demonstration is given, the mor- phology of attention assignment towards query updates significantly (shown as background color in Fig. 7, estimated by the radial density of various labels; see $A.5 for details), while the late stage (k = 15 + 16), attention assignment morphology is stable. This can explain the demonstration saturation (Agarwal et al., 2024; Bertsch al., 2024): ICL performance is submodular against the number of demonstrations. Intuitively, since demonstrations follow prior distribution, represen- tation of a new demonstration is likely to be located within the closure of existing demonstrations, making it less diverse to contribute to the attention assignment in the induction subspace.
PUTTING THINGS TOGETHER
So far, we have revealed the existence of the circuit with steps, organized by the sequential in- ference process among Transformer layers. In this section, we find that the circuit is dominant in the ICL inference. while some bypass mechanisms activated by residual connection assist ICL in- ference. Moreover, series of phenomena observed in ICL is successfully explained by the eircuit.
Table 1:
Accuracy variation (%) with each inference step ab- lated on Llama 3 8B. Small numbers are controlled results (mean + std) of randomly ablating equivalent amounts of con- nections. Ablations are applied from the bottom to the top layers, and results with various ablated layers are reported (detailed set- tings: Appendix A.7).
5.1 ABLATION ANALYSIS
To (1) examine the causality be- tween the attention connection specified by the 3 steps and ICL inference, and (2) demonstrate that our 3-phase circuit domi- nates or least participates in ICL process, we disconnect the related attention connection of each step in the proposed circuit (see Appendix A.7 for details), and test the accuracies without such connections, as shown in Table I. The results indicate that, compared to the controlled results, where trivial connec- tions are removed, the accuracy of ICL significantly decreases when the non-trivial connections designated by the proposed circuit are ablated. This supports the existence of our circuit. However, the result doesn't fully match expectations, for example, the result without induction (line 5) should be consistent with the zero-shot inference result (line 6), since all the expected communication from demonstration to query is intercepted, but the real result is better; and the contribution of Step I in later layers are unexpectedly high, indicating the existence of some bypass mechanisms parallelly contributing to ICL accuracies.
Attention Disconnected
Affected Layers Ratio (from layer I)
Key - Query
25%
50%
75%
100%
None (4-shot baseline)
10 (Acc. 68.55)
Stepl: Input Test Encode
2 Demo. Texts Forerunner
-198
-15.82
-2343
300000
-0.0 tom
-1.5 2 0.0
-3.2 1.5
-1 -166:20000
Query Texts 2, Forerunner 9g
-3.87
21,10
24.74
-28.38
Step2: Semantics Merge- 2224
Demo. Forerunner Lubel
-un 1.0
3.45
3.30
-342
Step3: Feature Retrieval & Copy
Label y Query Forerunner
-0.6 0.00
+0.00 1 0.20
00.00 2 0.00
10,22
Reforence Value
Zero-shol
17.90 (Ace. 50.65)
Random Prediction
36.05 (Ace. 32.50)
5.2 BYPASS MECHANISM
Step
Step
Llama.3 708
Motivated by the ablation results, we believe that several mechanisms including our circuit parallelly for ICL, since the residual con- nection supports complex paths among layers and attention heads. We list some possible by- passes and plan a complete enumeration as fu- ture work.
64H
Palcoa 403
60L. 12011
Lama 3 93
32L. 3211
Falcon
32t., 713
0%
20%
40%
60%
80%
100%
Nomalize Laye: Number
Figure 8: Dynamics and deserialization of magni- tudes of proposed 3 inference steps (cartography details: Appendix A.6).
Parallel Circuits. Multiple 3 3-step circuits can execute in parallel, and one layer assign
8

---

Published as a conference paper at ICLR 2025
multiple inference functions to different heads, causing dispersion and deserialization shown in Fig. 8, where a narrow (fewer heads per layer) and deep (more layers) model is more likely to generate localized inference, and vice versa.
Direct Decoding, Residual connection to the LM heads allows intermediate hidden states to be decoded directly. Intuitively, shortcut from Step encodings the LM head enables ICL with zero-shot capacities, since we have confirmed that encoded representations are informative for ICL tasks ($3.2), while the decoding methods should be selected carefully (Cho al., 2024) (e.g. with essential calibration). On the other hand, shortcuts from insufficiently encoded features may lead meaningless information decoded by language model heads, causing prediction bias, i.c. even no query is given, ICL still returns unbalanced results (Zhao et al., 2021) decoded from tokens of prompt template (see Appendix E for details).
Table 2: Accuracy drop with shorteut ablated.
Shorteut Induction. Note that k-shot ICL input sequence always contains (k 1)- shot sequence, where the k-th forerunner token (served as the query of (k- 1)-shot sequence) is previously processed by the (A - -h- inference. So every forerunner can directly retrieve previously processed forerunner tokens in the demonstra- tions to copy their induction results directly. The non-trivial result (Table 2) with all forerunner tokens disconnected from each other confirms such infer.
Attention Disconnected (71)
25%
50%
75%
100%
Forerumner An
Forerunner Ptg
-1300
0.75
0.78
1.56
5.3 EXPLAINATION TOWARDS OBSERVED ICL PHENOMENA
Difficulty-based Demonstration Selection. In $3.1, we find that in the zero-shot scenario, per- plexed texts are harder encode, explaining the observation of PPL-ICL (Gonen al., 2023): Selecting demonstrations with lower perplexity can improve ICL. performance. Moreover, while the demonstrations increase, LMs can encode more complex inputs with diverse information to update the attention assignment shown in Fig. 7, making it beneficial to input harder demonstrations later, which explains the ICCL (Liu et al., 2024), which build demonstrations sequence from easy to hard.
Prediction Bias. (1) Contextual Bias: As mentioned in $5.2 and shown in Appendix E, direct decoding insufficiently encoded information adds meanless logits into LM's output, causing a back- ground prediction value even if no queries
given (named bias). (2) Positional Bias: As shown in $3.2, closer demonstrations are encoded more similarly, so label tokens near the query have more similar information the query, causing more attention assignment in the induction processing, so that more influences on the prediction. (3) Frequency Bias: As shown in $4.2, in the induction, some attention heads are without correct selectivity towards labels, causing an averaged copy pro- cessing from label tokens to the query, triggering a prediction bias towards the label frequency in the demonstration, even if their contribution (absolute value of attention score on label tokens) is small. All three biases are observed by Zhao et al. (2021), and can be removed by ICL calibration methods.
The Roles and Saturates of Demonstrations. performance of ICL. We decompose such performance improvement into 2 parts: (1) demonstra- tions help early layers encode better ($3.1), 1), and (2) more demonstrations provide larger label token closure, enabling more accurate attention assignment ($4.2), while the volume of such closure is submodular to demonstrations, causing the saturates of ICL performance towards demonstrations.
is well known that demonstrations improve the
The Effect of Wrong Label. It is well-known that the label noise is less harmful in ICL (Min et al., 2022) than in gradient-based learning (Zhang et al., 2021). We have explained in $4.1 that ICL implies labels denoise to stabilize ICL against label noise, while weakened by dimensionality.
6 CONCLUSION AND DISCUSSION
In summary, this paper restores ICL inference into 3 basic operations and confirms their existence. Fine-grained measurements are conducted to capture and explain various phenomena successfully. Moreover, ablation studies show the proposed inference eircuit dominates and reveals the existence of bypass mechanisms. We hope this paper can bring new insight into ICL practice.
Conclusion.
The Role of Early and Later Layers. Our framework shows: The encoding result of Step I can be directly used for classification with reliable decoding, and later transformer layers are not con- tributing centroid classification accuracies (Fig. 3), leading to taxonomy of Input Encoding for

---

Published as a conference paper at ICLR 2025
Step 1 and Output Preparation for Step and 3: LMs complete multi-task classification implicitly in early layers, and verbalize it by merging task-specific label semantics in later layers. Therefore, suggest an early-exiting inference: removing some top layers and using centroid classifier (Cho et al., 2024) to accelerate ICI, as shown in Table 3 and Appendix E.
Pre-training Possibility from Natural Language Data. A large gap can be considered between such a deli- cate circuit and gradient descent pre-training on the wild data. However, we believe the wild training target con- tains the ICL circuit functionally: Based on the previ- ous works finding trainability of ICL on simplified lin-
Table Performance of full and layer- pruned ICL inference.
Inference
Param.
Speed
Full + LM Head
66.19%
70.6B
1x
Full Centroid
83.24%
69.5B
1.00x
Layer34 + LM Head
49.29%
32.7B
2.16x
ear representation-label pairs (Chan et al., 2022; Reddy,
Layer34 Centroid
84.27%
31.2B
2,38x
2024; Singh et al., 2024b), we speculate that in early training step, Transformers learn to extract linear representations shown $3 from wild data (detailed in Appendix B), serving as the training input of later layers to evoke the emergence of induction heads with the same mechanism shown in aforementioned previous works. Moreover, our conclusion of Step 3 highlights the input data requirements for the later layers: These data should activate the multiplex of hidden space, i.e,, it should implicate multi-task classification with a wide distribution, which is consistent with the aforementioned previous works.
Comparison with Previous Works. Several prior studies have sought to interpret ICL as known processes, including implicit gradient descent (Dai al., 2023), kernel regression (Han et al., 2023a), and implicit Bayesian inference (Xie et al., 2022), etc. However, as stated in $2.1, these approaches fall short of fully explaining the phenomena observed during the ICL inference process. For instance, gradient descent is known to be fragile against label noise (Zhang et al., 2021), leading to misalignment when analogies are drawn to ICL, which is robust against label noise (Min al., 2022: Liu et al., 2020). Similarly, attempts to explain ICL as kernel regression fail to account for po- sitional bias, making them disconnected from empirical studies that have observed various inference phenomena in real-world LMs. Also, other works have employed induction circuits to explain ICI. dynamics (Wang et al., 2023; Elhage et al., 2021; Olsson et al., 2022), yet significant gaps remain in aligning these explanations with empirical observations. Our work is the first to unify the frag- mented conclusions from prior empirical studies (discussed in $5.3) through detailed experimental measurements. By demonstrating the alignment of these observations, we emphasize the novelty and primary contribution of our paper.
I Limitations & Open-questions. (1) These 3 basic operations not functionally indivisible. Ide- ally, mechanistic interpretability aims to reduce every operation in ICL inference to the interconnec- tion of special attention heads to ulteriorly examine how the operating subspaces interact between steps, and reconstruct ICL behavior from a minimal set of attention heads. Also, although we show the significance of the inference circuit in the ablation analysis ($5.1), measuring the connectivity of these attention heads can also be beneficial to get more insights into the circuit. (2) The con- clusions may not align with scenarios where ground-truth truth labels are not provided in the context in some cases, which are often referred to as in- in-weight learning, where significant differences or even antagonism with standard ICL. have been highlighted by previous works" (Chan et al., 2022; Reddy, 2024), reasonably and necessarily warranting separate discussion (discussed in Appendix G). This paper explains the inference behavior of the model under ICL conditions, leaving the in-weight learning scenario for future works. (3) We only focus classification tasks, while we believe that our findings can be applied to non-classification tasks, efforts are still needed to fill the gap.
AUTHOR CONTRIBUTIONS
Hakaze Cho, also known Yufeng Zhao, handled the entire workload in this paper. He provided ideas, designed/ / conducted experiments, collected / described data, and wrote / revised the paper.
Naoya Inoue, as the supervisor, provided valuable feedback, revisions, and financial support for this paper. M.K. and Y.S. made minor contributions that placed them on the borderline of authorship criteria: they participated in discussions and offered comments, some of which were incorporated.
In these works, toy models even struggle to work on both inputs simultancously, so that it is natural to consider different inference mechanism for the so-called wn-wetght learning.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
The official code implementation of this paper by the author can be found com/ho4955 ICI__Circuu Please follow the instructions in this GitHub repository to repro- duce the experiments. If any final data or intermediate results are required, please get touch with the first author by contact information shown in O
ACKNOWLEDGMENTS
This work supported by The Nakajima Foundation Start-Up Support Grant.
The authors would like to thank Mr. Lihao Liu at the Beijing Institute of Technology, and Ms. Yunpin Li at the Capital Normal University for proofreading.
The authors would like to sincerely thank the chairs and reviewers of ICLR 2025 for their thoughtful and insightful feedback on this paper, their perceptive reviews and discussions have greatly improved this work.
REFERENCES
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie CY Chan, Biao Zhang, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning. In ICML 2024 Workshop on In-Contexi Learning, 2024. URL https://arrxxv.orrr/bb////
11018.
AI@ Meta. Llama model card. 2024. URL /:0b///aaaaa hataaaaaia//MaaaMMMMAA md.
http://...cmcccmceooeeeeeoe alama/
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Mérouane Debbah, fitenne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311 1..-27711..666,, 2023. URL attpps:/aazee
org/ alss 2311. 16867.
Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neu- big. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.000200 2024. URL https:
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scal- ing. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023. URL https: /ttp//ixvvvv../// org/ abs / 223./21333..
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learn- 2022.
ing in transformers. Advances in Neural Information Processing Systems, URL https:: arxivv. org/ abs 2205. 05055.
Ting- Yun Chang and Robin Jia. Data curation alone can stabilize in- learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long abs /0lss/2... 2.212. 1 10378.
Pupers), pp. 8123-8144, 2023. URL https: / arxiv. wrg/abs/////.ooooooo.o....o.....
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, cultiifunctionaaity. multi-granularity text embeddings through self anflllnoweddg dis- tillation. arXiv preprint arXiv:2402.032. 2024. URL /tttts://arxivv org/ abs / 2402. 03216.
Hakaze Cho, Yoshihiro Sakai, Mariko Kato, Kenshiro Tanaka, Akira Ishii, and Naoya Inoue. Token- based decision criteria are suboptimal in in-context learning. arXiv preprins arXiv:2406. 16535, 2024. URL https: arxiv woo.oo///aao... abb / 2406...0 16535.
Damai Dai, Yutao Sun. Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent meta-optimizers. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023. URL https: //assi////xivv//// org/ abs 221 10559.

---

Published as a conference paper at ICLR 2025
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. survey on in-context learning. arXiv preprint AXiv:22301.02234. 2022. URL https://arxixl org/abggsss.. https//31.....22
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anma Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021. URL https:// trans www.erorrrrrriitttt..eeeoo-----e wtt.rrrtttttttpp.e/eeeeooeeoaeee pttpp////..//raaewo//////t//e
Hila Gonen, Srini Iyer. Terra Blevins, Noah A Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. In Findings of the Association for Computa- tional Linguistics: EMNLP 2023, pp. 10148, 2023. URLhttps: /ttp/v/arvvv//v.//ei//eeee...ava org/ abs/ 2212. 04037.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Pre-training to learn in context. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4849-4870, 2023. https: // //axivv...v///////////aa/////av. org/ abs 2tt.//35.0000
Qi Guo, Leiyu Wang. Yidong Wang, Wei Ye, and Shikun Zhang. What makes good order of examples in-context learning. In Findings of the Association for Computational Lin- guistics ACL 2024, pp. 14892-14904, 2024. URL MtLL:t://///llatt http///aaaaaaaaaaaeeeeeee org/2024 findings acc.a 884/.
Chi Han, Ziqi Wang, Han Zhao, and Heng Ji. Explaining emergent in-context learning as ker- nel regression. arXiv preprint arXiv:2305. 2023a. URL https:///eee http:ss///axvvooo org//bs/
Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz. and Tianlu Wang. Understanding in-context learning via supportive pretraining data. In Proceedings of the sl Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers),
pp. 12660-12673, 2023b. URL https:ttsst........tt / arxiv, orgoor/ggooo abs .t../........00.1 15091.
Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran. ward semantics-basee answer pinpointing. In Proceedings of the First International Confer- ence on Human Language Technology Research, 2001. URL https: http://wwwww...r....c// orgg
To-
anthology 1069.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hy- pothesis. arXiv preprint 240.:2405 2024. URL / 07987.
Hong Jun Jeon, Jason Lee. Qi Lei, and Benjamin Van Roy. An information-thhorete analysis of in-context learning. In Forty-first International Conference on Machine Learning, 2024. URL https://arxiv org/ alss 1.401.15533
Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of neural network models: A survey of functional and representational measures. arXiv preprint SEIV:22305.0332, 2023. URL https: http//awwwi.aaaaaaaaaaaaaaooeaea abs/ 06329.
Jannik Kossen, Yarin Gal, and Tom Rainforth. In-context learning learns label relationships but is not conventional learning. In The Twelfth International Conference on Learning Representations, 2024. URL https: arxiv org/ abs / 2207 12375.
Xiaonan Li and Xipeng Qiu. Finding support examples for in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 6219-6235, 2023. URL https:
// //arxivv.ot www.iii.rrg/as////..e
13539.
Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th Interna- tional Conference on Computational Linguistics, 2002. URL https: peattp:://www.e. www..acleel.... org/
anthology/CO2 1150.
Jinlong Liu, Guoqing Jiang, Yunzhi Bai. Ting Chen, and Huayan Wang. Understanding why neural networks generalize well through gsnr of parameters. arXiv preprint arXiv:2001.07384, 2020.
URL https: ///xx/i//// .ogg abt0//00s/00011/ 07384.
12

---

Published as a conference paper at ICLR 2025
Yinpeng Liu, Jiawei Liu, Xiang Shi, Qikai Cheng, and Wei Lu. Let's learn step by step: Enhancing in-context learning ability with curriculum learning. arXiv preprint arXiv:2402.10738, 2024.
URL https: http:///axivvvooooo org/ abs 220..22022.22 10738.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long
Pupers), pp. 8086-8098, 2022. URL https: a///iiivivv.////..///aa////aaavv www.oorg/////.oooo 2tbps/2114.000 .07866.
Malo, A. Sinha, P. Korhonen, Wallenius, and P. Takala. Good debt bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014. URL https: / /arxiv...././/////a///aa/////aaa .ww.iii..rr.....o org/ abs / 1t//1111..3333
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zeulemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048-11064, 2022. URL https: arxiv. axggv.org//. abs 2202.22222222 12837.
Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. Semeval- 2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop semantic evaluation, pp. 1-17, 2018. URL https: /tt///allllc wocelanohoooooooo loyy org. wwtg///-////00///// 1-1000//
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 11895, 2022. 11895.
URL https: /ttpps://axxii /ttp//axivvv///////////////..av. wtw///r////aaaaao.oo/.o/oooe
Jane Pan. What in-context learning "learns" in-context: Disentangling task recognition and task learning. Master's thesis, Princeton University, 2023. URL https://axxiv.cr wtrp abs/ 23C5.09733
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL pp. 115-124, 2005. URL https: //aps///aaaattoo...... oxg/PO5-1015/.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask leamnrs. OpenAl blog, 1(8):9, 2019. URL. httpt:tt//// wwt.e/nnn....e... models_ aeeellsssse__uuurreeeeee sepppeervieeddddeeee mu wea.ttsssklaarrs.pppp. language-mode s anguage_
Gautam Reddy. The mechanistic basis of data dependence and abrupt learning an in-context classification task. In The Twelfth International Conference Learning Representations, 2024.
URL https: //axiii
abs 2312. 03002.
Aaditya Singh, Stephanie Chan, Ted Moskovitz, Erin Grant, Andrew Saxe, and Felix Hill. The transient nature of emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 36, 2024a. URL https: http///rxvvv////////////////a org/orrg///.... abs//ass/ll1// 2211. 08360.
Aaditya K Singh, Ted Moskovitz, Felix Hill, Stephanie CY Chan, and Andrew M Saxe. What needs to go right for an induction head? a mechanistic study of in-context learning circuits and their formation. In Forty-first International Conference on Machine Learning, 2024b. URL https: //tp://aaivvv.//////aa/eeaarawaa wwtp/ass////oooooooooooo..taoasa
abs / 2404 07129.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, sentiment
and Christopher Potts. Recursive deep models for semantic compositionality treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL https://// /www.www......mm acwweb. org/ hnt 1170.
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng. Jie Zhou, and Xu Sun. Label words anchors: An information flow perspective for understanding in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp-
9840-9855, 2023. URL https:/: /tapps/////xxvvi .rr.oo///aao........ 2305 141 60.
13

---

Published as a conference paper at ICLR 2025
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint AXXv:22300.08846 2023. URL https: /ttp:///aaavvv//ia org/ abs///b///////// / 2303. 03846.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Rep- resentations, 2024. URL https: / //axivvvv..//////// www.iv.ogg//...a...ooo.o..oooeee abs / 3009. 17453.
Sang Michael Xie, Aditi Raghunathan, Percy Liang. and Tengyu Ma. An explanation of in-context learning implicit bayesian inference. arXiv preprint arXiv:2111.02080, 2021. URL https:
//arxiv. org/ abs /aabs///1ll. 02080.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning implicit bayesian inference. In International Conference Learning Representations, 2022. URL https: / arxiv. org/ abs 211//11111111 02080.
Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang- Woo Lec, Sang- goo Lee, and Taeuk Kim. Ground-truth labels matter: deeper look into input-label demon- strations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2422-2437, 2022. URL https:tpp///omm httttps://axxii.ortt wtt.://///.........////// 12685.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021. URL https: arxiv. o abs/ / 1611.03530.
09927. Ruiqi Zhang, Spencer Frei, and Peter L Bartlett, Trained transformers learn linear models in-context. arXiv preprint arXiv:2306.099) 2023. URL httpt::t/axaiiiiiie org. abs //..a///.666. wttp:///g/aaabaaaa/abeeb/e
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015. URL https:
//aaxiiv.org org abs 1509. 01626.
Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE transactions on knowledge and data engineering, 34(12):5586-5609 2021. URL httppppp//wxwi..orgg/ooooo 08114.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pp- 12697-12706. PMLR, 2021. URL. https: / //t:///axat arxiv org abs / 22.2..0000000 09690.
Bowen Zheng, Ming Ma. Zhongqiao Lin, and Tianming Yang. Distributed rule vectors is a key mechanism in large language models' in-context learning. arXiv preprint arXiv:2406.1600) 2024. URL https: arxiv. wwwpxiiv....g...oooooooooooooeee owg/aogg///....oooooooo........e abw//bbs//455 /000b./246..... 16007.
14

---

Published as a conference paper at ICLR 2025
Revisiting In-context Learning Inference Circuit in Large Language Models
Appendices
A EXPERIMENT DETAILS AND SETTINGS
A.1 DETAILED OVERALL EXPERIMENTAL. SETTINGS
Table 4: Prompt templates used in this paper.
Dataset
Promat Template (Unit)
Label Tokens
SST-2
sentence:
(input santence] senciment: [label tokan] \r
negative, positive negative, positive negative, noutral, positive poor, had, neutral, good, great sbort, entity, description, petson, location, number world, sports, business, science
MK
review:
(input sentence] seetinent: [label token]
\n
FP
[input sentence] sentiment; [label token]
SST-5
sentence: Linput sentence] sentiment: [Label tokan]
TREC
question:
(inpuz sentence] target: [label token]
AGNews
(Anput sentence] topic: [label token]
Prompt Template. We conduct experiments on a specific prompt template for each dataset as shown in Table 4. Morcover, similar to typical ICL practices, we reduce the label into token to simplify the prediction decoding. The reduced label tokens are also shown in Table 4.
Quantization. In our experiments, we use Bit sAndyyess to quantize Llama 3 70B and Falcon 40B to INT4. For the other models, full-precision inference is conducted.
Other. All the experiment materials (models and datasets) downloaded from huggingface. For the BGE M3, we use its pooler output as the output feature.
A.2 CALCULATION OF MUTUAL NEAREST-NEIGHBOR KERNEL ALIGNMENT
In this paper, we need to measure the similarity between features from two different models or model layers. There many approaches (Klabunde al., 2023), and neighbor kernel alignment (Huh et al., 2024), which is relatively concise and accurate, calculated follows measure the similarity of representation from the same object set X' {x in different feature spaces.
mutual nearest-
Calculation of Mutual Nearest-neighbor Kernel Alignment. Given representation mapping 5 X' + Hid from the objects to space where similarity measurement (. : Hdd x Hd R is defined, we can calculate the similarity map from dataset X' as Sg Ænx*, where the elements are Saki, = (8 (x.) (x,)), especially, we axiomatic define (I, = 1, so we set the diagonal element S3p.1 = since they are trivial values that disturb the following calculation with values
Given two representation functions ō1 and 62, two similarity map can be calculated as S;, and Ss, on the same object set X. For each line vector index i 1....... n in St,, we select the index of top-K elements from greater lower as top; (8), Similarly, get top (Sa21t) from Szy.
Then, we calculate the kernel alignment for sample as:
(0 82); |lop (Sa.k) n topx (St21t)
k
The kernel alignment for dataset Y' is the average on each KAx(61, 02);
choose cosine similarity the (:.), and K = 64. According to experiment settings in $2.2, n = 512 is defined, and a randomlized matrix S have KA 64/512 0.125 as the random baseline.
Implementation. In our experiments,
A.2.1 BACKGROUND VALUES OF KERNEL ALIGNMENT: LABEL TOKEN TEXT ENCODING.
In a specific layer of decoder Transformer, the representations of a token x can be written as 8(x) +e(), where e(x) is the embedding vector of the token x, and enp) are the residual side- flow w.r.t. the context p. Given two specific tokens 2; and r, where kernel alignment calculated
index
15

---

Published as a conference paper at ICLR 2025
Representations from Encoder Model
Nepresentations from ICL: Porerunner
Representations from ICL: Label Token
Representations Space
Represeutations Space
Representations Space
el Soeettvve BA ( F "PSILIMY
Stop 1: Enecading Prasee Pruneis
BGE(x) "pesitive'
H +(U) y. "posltve"
l'pocinee
BGE(x) "nogative"
nogtitve DertiiseeDeee D ft paganio Toggatie
RTE oig "eggttig"
Figure 9: Distributions (clusters) of representations generated by: Left: encoder model (BGE), clustering w.r.. the label. Middle: ICL on the forerunner token, where representations gather into one point (e([": "I)) when no Transformer operation is conducted. Right: ICL on the label token, where representations gather into points w.n.. label (using a 2-way example) when no Transformer operation is conducted, causing high background value.
from different ICL-styled input sequences P; and Pj. we discuss the background value of kernel alignment as below.
Intuition. As shown in Fig. 9, the hidden states on the label token have prior clustering, making them naturally similar to the representation generated by the encoder model, even if the ICL process does not encode it sufficiently. So, at layer O, since the model is not able to perform any contextual encoding in this layer, the kernel alignment on the forerunner is on a random baseline value of 0.125, but the value on the label token will be greater (estimated below). In the case where (1) BGE generates fully linearly separable clusters against in each class, with sufficient inter-cluster distance, and (2) the number of samples for each label is not less than 64, such upper bias can be expected to be 0. 0.122(( -1) (proof omitted). Intuitively, such bias can propagate along the residual network from the embedding of [y] to every layer, making the results of every layer upper-biased. biased.
Similarity on Forerunner Tokens. The cosine similarity between 8(x;) and d(zz), where the x; and zj are forerunner tokens of the input sequence can be written as:
(((x)),(zz)) (e(x;) + c(ps), + c(p)))
(2)
c((,)) + (e(x:)).(p))) + (e(x,), c(pi)) + (((p)))(ppp))
[e((x;) + <(p))) |le(x,) + e(p,)ll2
Denote Br, = (e(x;), e(py)) + (e(x,), e(pi)), Ct, = [|e(,;) + <(pi) |e(,,) + e(p)) l and no- tice that Iz, = Iy since they forerunner tokens which is kept consistent in experiments, we have:
((xz),((z))) + Ba,j + (c(pt).((;)))
(4)
Ci.
Similarity on Label Tokens. Similarly, the cosine similarity between 5(y:) and o(y)) label tokens can be written as:
((.,),((55)) (e(vi),e(us)) Bis + (e(p)), Ct.5
(5)
Encoding on Label Tokens Enhances the Similarity with Same Labels. Given K = 1 for sim- plicity, the probability top; (Sajs) selects sample with the similar label with i-th sample on the forerunner token can be written as:
E El (((2)),6(2))
PF yi = Mtop, (Sore)
Tws
(6)
E, (((x))) 5(x5))]
+ E, 25 [Ba,] + E v (e(p))).(p;)))
Ty.
E, B E, ((((p)),e(p)))))
16

---

Published as a conference paper at ICLR 2025
where the ryy is the label ratio of ys. Similarly, the probability the label token can be written
Pr. yi Vrop, (S6.))
Ty. (1 + E Bolycuu [Bij] + Bo (((()))
(1 + E j/y: [Ba,]] + E, 1o [<(p;)) c(;
+ (1 Ty (((())) c e(v)))
+ E, [Bag] + B, c(p)))).
PF = Mrop, (S:
(8)
That is, the inputs with the same labels with vi easier to be selected into the top, (Sas). Notice that we make approximations here: (1) consider the Bil,==; [Ct,] E, [Cty], i... the 2-norm of two encoding vectors are considered equal granted by normalization used in Transformer. (2) We consider the context term in the label token scenario the same as the forerunner scenario since the difference is only label token, which usually occupies quite small part of the input sequence.
Background Values of Kernel Alignment. According to the explanation above, as shown in Fig. 9, it is intuitive to conclude that lop (Sajz) from label tokens is easier to cluster samples with the same label as yı- Moreover, a well-pre-trained encoder can catch the prior distribution of the input texts determined by their labels, and also cluster samples with the same label, causing a high similarity of similarity map, so that a high but unfaithful kernel alignment as the background value from the similarity on e(v) but not c(p). An intuitive verification of such background value is shown in the Fig. 2 (Left), where the "Label Token" curve has a high value in layer O with c(p) = However, the background value also indicates that the representation generated by BGE correctly clusters the samples, confirming its reliability.
A.3 TRAINING AND INFERENCE OF CENTROID CLASSIFIER
In this paper, we follow Cho et al. (2024) to train centroid classifiers as a probe toward hidden states of LMs. In detail, given the LM's hidden states set {k]] m experimental setting, the last label token or forerunner token) in layer I from an [ICL, input)- lqqueyy label] set Z (pi, yt M= where the labels limited in label space W, in the training phase, we calculate the centroid of the hidden state n, for each label respectively:
i=1 of the selected tokens (according the
h = Exj. [M]]
In the inference phase, we extract the equitant 10 hidden state hn as the training phase from the test input, and calculate the similarity between hy and the centroids calculated above. Then, we choose the label of the most similar centroids as the prediction:
C(h)) = argmax (Ri, (4)
(10)
y
Implementation. In our experiments, we set training sample number m = 256, similarity function (a, b) -1aa b/|2-
A.4 CARTOGRAPHY DETAILS OF FIG. 6 (MIDDLE)
In Fig. 6 (Middle), we define a Correct Label Assignment, here we introduce how this measurement is calculated. Suppose we have an attention score Aw, , Q) calculated as (also written as ak-Q in brief):
Aw,r (K = (KKK))
(1I)
with hidden dimensionality of d, give a certain layer, K E Rdx. is the hidden state matrix of full context, Q E dx is the hidden state of query's 's forerunner token (Q = K,,). f RT + 2 is normalization mapping from n, ne-dieensional real vector to n4 -ydiimnnsional probability vector (usually softmax function), the W is linear kernel, usually WA W for multi-head attention or == diag (1t) for vanilla attention.
K
Equitantt refers hidden states from the same layer and token type. While, in experiments shown in Fig. 5 (Right), we don't 't keep the token type consistent.
17

---

Published as a conference paper at ICLR 2025
Trivial Connections
Demn. Texts x Forerunner
(4) Demo. Forerunner Lahel
(3) Query Texts Xg
Forerunner's
(5) Label y;
Query Forerunner &4
atention Connection
really
like
Label:
Not
sc
good.
Label:
Good.
Label:
Figure 10: Visualization of non non-trivial attention connection defined and disconnected ablation analysis in $5.1 and Table 1. Notations and line numbers same as Table 1.
For one input sample, given the loken-indes set of label tokens as L, the token index set of label tokens which is the same as the query's ground truth label as L+, we define the Correct Label Assignment (CLA)) of one sample as:
CLAw. (K, Q,C, Eiect Aw. (K,
(12)
Eiec Aw.s f (K,Q);
Intuitively, CLA reflects the accuracy of attention computation Aw.r towards label tokens on one input. For input set built from dataset, we calculate the averaged CLA these inputs, and repeat in every layer to plot curve of Averaged CLA against layer numbers. Specifically:
(1) Vanilla attention. We assign W I, to linear normlization.
(2) Best Induction Head. For each attention head h, we assign Wh = W WWTTW = softmax. For each input, we calculate max CLAw*,J (K, Q, L, C+) as the result for single input.
(3) Head Average. For each attention head h, we assign Wh = W W f = softmax. For each input, we calculate >, CLAw",, (K, Q, C, C+) /(H) where the [H] is the amount of heads in current layer, as the result for single input.
Note that we do not consider the absolute value of attention assignment on label tokens in this experiment, and most of the heads have little scores assigned to the label (Fig. 6 (Left)), therefore, although the average assignments tend to be average, this result shown in Fig. (Middle) does not contradict the phenomenon that ICL can achieve high accuracy.
A.5 CARTOGRAPHY DETAILS OF FIG. 7
For Fig. 7, we input one sample from SST-2 into Llama 3 70B, take the output of layer 30 on the label tokens to span a matrix Kg, and map them by W W of head (the best induction heads in this layer) and 9 (the worst induction heads) of layer 31 (the layer with the most correct induction heads), respectively. We visualize the distribution of these mapped W W Kg, we conduct principal component analysis them, and plot them on the plane of the first two components.
For each point E I2 on the principal component plane, we calculate the attention assignment follows. Give the index set of "positive" label token in Ke as L+, the index set of "negative" label token as _-, we calculate the attention assignment, which can an estimate of ICL predic- tion (Wang al., 2023),
Aut.Assig(())
IT T Kh T
WTT Wk
(13)
SEC+
SEC-
We map this value to the degree of blue color of each pixel. The larger the positive value, the bluer it is, and the smaller the negative value, the redder it is.
A.6 CARTOGRAPHY DETAILS OF FIG. 8
For Fig. 8, we calculate the magnitude of Step 1 as the finite differences of kernel alignment in Fig. 2 (Left, "Forerunner Token of Label"). We directly use the head counting of Fig. 5 (Middle)
18

---

Published as a conference paper at ICLR 2025
Porerumaer Tokon Head
Komet Alganeat
12.5
0.20
0.34
10.0
Induction
512
1k
0.36
5k
2.5
0.0
Transformer Block Number
Transformer Block Number
Transformer Block Number
Figure 12: The 3 operating magnitudes on Pythia 6.9B with various pre-training steps. Left: Step 1, Input Text Encode. Middle: Step 2, Semantics Merge. Right: Step 3, Feature Retrieval and Copy, measured by the correct induction head numbers.
and Fig. 6 (Left) the magnitude of Steps 2 and 3. These data are regularized and converted into transparencies.
A.7 EXPERIMENT SETTING OF ABLATION EXPERIMENTS IN TABLE
In Table 1, attribute each step of the inference process to specific attention connections, also shown in Fig. 10. When we aim to remove this step from the inference, we eliminate (i.c. zeroing) all corresponding attention connections from layer 0 to layer (25%, 50%, 75%, 100%} x TotalLayers.
For reference, for each experiment, we also conduct controlled experiments where the same amount of randomly selected attention connections removed from the same layers the controlled values. The controlled results are shown as smaller numbers under the experimental results.
B LM PRE-TRAINING DYNAMICS MEASURED BY ICL CIRCUIT
We extend the discussion of pre-training dynamics in $6 here.
One can divide a self-regression model into an early part and a later part in depth of layers, where the early part encodes the input into a hidden representation, and the later part decodes the hidden representation back to the input. So, the training ob- ject can also be divided into an encoding loss and a decoding loss. According to the discussion in $6, the operation of Step
Normlized Value Accuracy
0.8
0.6
can be classified as encoding, and the other two steps of the in- duction circuit can be classified decoding. Intuitively, since the decoding operations require the encoding results as input,
0.2
Layar Copy
Layur: 30, Come. Ind. x
Accurey
10
unless the encoding operation converges a stable output, the decoding can not be trained since the input-output mapping is noised, causing unstable gradients interfere with the train- ing (Liu et al., 2020).
Pretraining Steps 10000
Figure 11: Operating magnitude for each inference step (normal- ized) and ICL accuracy against pre-training steps on Pythia 6.9B and SST-2.
We confirm such a hypothesis by measurements in Pythia 6.9B (Biderman et al., 2023) as shown in Fig 11, where: (1)
The magnitude the operations emerge in the early phase pre-training (less than 10k steps), and is monotonically increasing, while the encoding operation has the fastest growth rate. (2) ICL capacity appearance after three operations reaches a high level (around 50k steps, notice that the random accuracy is 0.5), while the curve morphologies of the operating magnitudes against the layer numbers (shown in Fig. 12) are convergence to the last training step. Such results suggest that: LMs start to produce the inner encoding in the very early steps of the pre-training, and can be important fundamental in building the subsequent induction circuits, as explained in the previous works mentioned in 1.
19

---

Published as a conference paper at ICLR 2025
Layer 21
Layer
0.07
10-
10
0.06
0.05
20
0.94
30
0.04
0.03
0.03
40-
0.02
0.02
50-
50-
60
60
0.00
10
20
30
60
10
60
Input Tokens:
Sentiment
Tokeno
Figure 13: Attention score visualized with the forerunner token of the ICL query as the attention query, from (Left) layer 21 and (Right) layer 2 in Llama 3 70B from input case.
B. DATA DISTRIBUTION REQUIREMENT EXPLAINED BY HIDDEN STATE MULTIPLEX
Moreover, the hidden space multiplexing in the induction operation observed in this paper can give a prototypical and phenomenological conjecture for the data distribution requirement found in the previous works (Olsson et al., 2022: Reddy, 2024; Singh al., 2024b), where data with large label space and various tasks can promote ICL and suppress In-weight Learning (IWL), and vice versa. Intuitively, suppose the encoding inputted into the later layers is clustered by their labels (similar to the well-embedded input styles in the works above, confirmed in Fig. 3). that case, we can say cluster center is the eigen-subspace of the corresponding label. Since attention only conducts dot-multiplication operations, let us assume that these eigen-subspaces are radially distributed.
During the training, (1) When the label space is small, the trained attention heads only need to extract the projected length of the query on each label's eigen-subspace. For each label, such op- eration has parameters' analytical solution with (encoding kernel) W I and (decoding transformation) Wy = og Cy. where 0y is the label token's 's output embedding, and e, is the la- bel's eigen-subspace. From such an operation, theoretically, one layer can handle at most [+]] labels, where [7| is the head amounts. While, considering the sparsity of these eigen-subspaces, such upper bound can be increased to [7[ by multiplex one head to decode an orthogonal group of la- bels with orthogonal eigen-subspaces of & [ey i.gi.... eyarl and orthogonal output embedding O = [o ogii..... iyrl where d' is the inner dimension of attention head, with decoding transfor- mation WoWy OT E. (2) When the label space expands ! the decoding transformation can not distinguish all the clusters since the Wo low-rank. Driven by the training loss, the model can choose to transform the encoding kernel to focus catching the most similar label tokens with the query, and copy the label tokens' information back to the query. As a result, one attention head can catch most d' groups of label tokens mapped collinearly by the encoding kernel (note that this set of labels may not appear simultaneously in the context, so confusion can be avoided), and the common space of these label words become the induction subspace shown in $4.2.
Our other conjecture is that the ICI. training endpoint is thermodynamically stable (with a lower loss), and in contrast, the IWL training endpoint kinetically stable (with more accessible training trajectory). Moreover, the IWL training object can be a precursor number of labels fed into the model gradually increases with the data. So, can hypothesize that: When the metastable state is disturbed by some condition, such as the appearance of rare or noisy labels, the training can show phase transition toward thermodynamically stable state.
ICL training, since the total
Notice that such a situation can also occur when the orthogonality between rigen-subspace, output embeddings is lost. A common situation is that the variance of the cluster increases, creating confusion within the decoding space of the attention head.
20

---

Published as a conference paper at ICLR 2025
Notice that this section is our hypothesis based the results of this paper, the detailed dynamics still unknown, which should be empirically validated in future work. One can start by decomposing pre-training targets into implicit tasks, and examine how these tasks can evoke the occurrence of the 3-step inference operations. A possible beginning is: finding implicit input-label tuples in wild data.
Layer 2, Head 53
C CAN LMS SEGMENT ICL INPUTS?
0.16
10
The experiments in $3.1 imply that LMs conduct effective segmentation on ICI.-styled inputs, en- abling LMs to block the interference of preceding demonstrations in the input text encoding opera- tion. Here, as a prototypical discussion, we con- firm the existence of the segmentation operation and then reveal that such segmentation can be done in very early layers from an attention operation fo- cusing on some specific segmentation tokens in the inputs.
20
Query Token
30
40-
0.04
50-
0.02
60-
0.00
40
50
60
As preliminary observation, we visualize the at- tention scores with the last forerunner token as the attention queue at layer 21 (a layer with high en- coding magnitude, refer to Fig. 2) from an input case, as shown in Fig. 13 (Left), where most of the attention heads are focused the query text. The visualization suggests that encoding operations are localized into the query tokens.
Key Tolen 8
Figure 14: Attention matrix visualized on layer
2. head 53 in Llama 70B from an input case.
Although position embedding inserts sufficient positional information to hidden states and enables attention heads to identify nearby tokens, we believe that position embedding is insufficient to ac- curately segment the various input parts of uncertain lengths. We hypothesize that LM focused on natural delimiters (e.g. "\n", ": ") in the input during the early stages of inference, and visualiza- tion in Fig. 13 (Right) supports such hypothesis: In layer 2, most of the attention heads focus on the natural delimiters, and another visualization in Fig. 14 shows that all attention queries (not only the forerunner token) exhibit similar separator- focuusing behavior, suggesting that: Some attention heads merge all the preceding delimiters' representation into every token as delimiter-based positional en- coding, making the representations of tokens with the same number of preceding delimiters similar, while differentiating the representations of tokens with different numbers of preceding delimiters. In the subsequent inference process, LM can utilize these delimiter-based positional encodings for localization operations. Such observation is also consistent with Fig. 4.
Furthermore, we empirically demonstrate that de- limiters have significant saliency towards ICL accu- racies in Table 5 (upper), experimented by remov- ing them from prompt templates. Interestingly, the
Table 5: Accuracy drop with delimiters re- moved/modified from prompts SST-2,
Template Modification
Acc. (S)
None (Table 4) w/o "(n"
91.60
trial to completely remove these delimiters from the
93.36
inputs yielded almost random results, even though these inputs still conform to the primary form of
w/o ":
85.74
"sentence: ", iment:
79.10
wlo all above
50.98
ICL. A reliable reason can be that: The hiatus of the delimiter interferes with the encoding operation (Step 1) on both demonstrations and queries, which completely disrupts the ICL. process.
"hello"
78.91
91.41
"positive"
71.48
(Random)
50.00
The scale of such a segmentation operation can surprise one, since more than half of the heads focus on the segmenting operation as shown in Fig. 13 (Right). However, as an assumption, we want to argue that dividing the input text into local segmentation is a crucial step in language modeling, so, functionally, LM has sufficient "motivation" to focus on segmenting by the pre-training objective. Moreover, based on the above principles, as long as the delimiter appears periodically at appropriate positions and can be captured by attention heads (only in the structured parts of the prompt template), as shown in Table 5 (lower), the delimiter can be designed to any token. While we still recommend natural delimiters without semantics in the template design.
21

---

Published as a conference paper at ICLR 2025
D SEMANTICS MERGE IS NON-SELECTIVE ON LOCATION
To investigate whether the copy processing described in $4.1 has selectivity on the forerunner token, on every attention head of each layer, for a given token position i, we measure the Normalized Copy Magnitude shown below:
NCMM, (a) = (i
(14)
where the (ii-))-i is the attention score with i-th token serving as the attention query and (i 1)-th 1--h)-th token serving as the attention key. For each layer, we export the NCM at all positions and on all attention heads, and separately statistics the cases where the i-th token is label token or non-label token. The results for 4 models on SST-2 are shown in Fig. 15.
From the results, no significant statistical differences between these types of tokens can be observed, suggesting that the semantics copy process, which is identified as Step of our circuit, is not selective on the token types. However, even if we demonstrate that the model cannot exhibit selectivity in the copy positions within the current input, a potential direction for future research is to investigate whether any forerunner tokens in the inputs can enhance weaken this copy process. Given that this copying is known to be related to label denoising ($4.1), exploring and carefully designing these forerunner tokens could significantly benefit the control of ICL behavior.
E MEASUREMENT ON DIRECT DECODING
This section measures the direct decoding bypass, suggesting that: (1) Direct decoding on well- processed hidden states with some later layer skipped can get satisfactory accuracy even better than the full inference process. (2) Direct decoding on insufficient processed hidden states adds bias towards the predicting distribution.
Direct Devoding Accuracy
We examine the first claim by applying the language model head on each layer's ' hidden state on SST-2, Llama 3 8B, and conduct a standard ICL process on the decoded token predic- tion distribution. The results are shown in Fig. 16, where di- rect decoding accuracy emerges from random to near 1 around layer 18. Refer Fig. and results in Appendix H.2, we can confirm: Accuracy emerges after all three steps are executed. Moreover, the accuracies on the intermediate hidden states are even higher than the last hidden states, which is aligned with the discussion in Table 3. So, we can conclude: Direct decod- ing on well-processed hidden states can classify well.
0.9
0.6
0.5
10
20
30
Transformer Block Number
Figure 16: Direct decoding accura- cies various layers.
Moreover, we infer that direct decoding from lower layers, where hidden states are not sufficiently processed, causes prediction bias. We investigate the influence of the direct decoding result of layer 0, by the relationship between direct decoded distribution and final prediction distribution. detail, on SST-2 and Llama 3 8B, butions on the label tokens "positive" and "negative", and calculate their ICL prediction probability distributions respectively, shown in Fig. 17 (Upper), where forerunner tokens with biased direct decoding distribution produce prediction biases with the same tendencies. While, when we apply contextual calibration (Zhao et al., 2021), which removes the background value without query from the prediction, such similar tendencies disappear (Fig. 17 (Lower)).
various forerunner tokens with different direct decoding distri-
F DEMONSTRATIONS ENHANCE THE INFERENCE OF PERPLEXED QUERIES
We investigate the correlation between the queries' perplexities and the classification accuracies with and without demonstrations, as a supplement of results in Fig. 2 (Right). We divide the queries into 10 bins w.r.t. the language modeling loss, and calculate the prediction accuracy in each bin, shown in Fig. 18. In these results, although a unified correlation can not be observed, we can confirm that: Compared to the 0-shot results, the 4-shot inference shows better accuracies, especially queries with high language modeling loss. So, we can conclude that: Demonstrations enhance the inference accuracy of perplexed queries, consistent with the results in Fig. 2 (Right).
22

---

Published as a conference paper at ICLR 2025
Nagsttute
Label Thies
Necstand Cree Maguitm
Norslaed Dopy Nagahnda Narmal Thiens and Takene
Nomize c1p3 Nagniat
Lager
Nervitavl Ovey Mayshutta to Narmol Thiees and Labell Yelren Pricen
Toole.
Menskee Coyy Mequiaas
Layer lades
Xamslaan Maysltude Nammal Thione label Telwes Pulces
Cripe Labbl Toloe
Nonsind Cip Neploat
Figure 15: Copy magnitude normalized by the sequence length from the previous token to the non- label token and label token, for every token and head in each layer. Significant statistical differences cannot be observed.
G DEGRADATION ANALYSIS: IN-WEIGHT LEARNING WITHOUT GROUND-TRUTH LABEL IN CONTEXT
Given the circuit proposed in this paper, it is intuitive that some label tokens, especially the ground- truth label token of the query should be presented in the demonstrations, which is the typical in- context learning setting compared to the In- Weight Learning (IWL) setting (Chan et al., 2022; Reddy, 2024) where the ground-truth label is not offered in the demonstration. When the ground- truth label is missing, the behavior of Steps 1 and 2 remains constant as they are independent of the query (due to the causal attention mask), while the behavior of Step 3 should be re-discussed.
In this section, we demonstrate that our circuit, especially the induction processing of Step 3, can still explain the inference behavior in the IWL scenario, but is not as robust as in the ICL scenario. Moreover, our experiments demonstrate that in the IWL scenario, induction heads are detrimental to the model's predictions, which is consistent with the conclusion drawn from the results in the main text. Also, some inference phenomena in the IWL scenario can be explained by our conclusions. While, we also illustrate through a counterexample that under the condition of IWL, even if the induction heads work negatively, the demonstrations can still enhance the inference.
23

---

Published as a conference paper at ICLR 2025
Vanilla ICL
0.8
negative
Avg. Prodiction
0.6
0.2
0.0
Ground-truth
'label" 40,034
"label : -0,0007
'label lax -0.0003
"label it" "label posttive" +0.0005
0U02
Forerunner Tokens/ Direct Decoding Logits Difference (N -P, Layer 0)
Contextual Calibration
0.8
negative
Avg. Prediction
0.6
0.2
0.0
Ground truth
'label" 0,0034
"labll 2 -0,0077
label lat 0.0003
"label V "label positive"
+0.0005 +0.0025
Forerunner Tokens/ Direct Decoding Logits Difference (N Layer 0)
Figure 17: Predicting distributions on different forerunner tokens with various direct decoding logits. Inference process used: Upper: Vanilla ICL, Lower: Biased removed inference by Contextual Calibration proposed by Zhao et al. (2021).
2
THDC 6-
ACNosss 2
Ianokege Vadeliie Lone
Iezexage Stdeiie Locn
Ieackepe Vadeking lows
Ixnokeee Madekie Lowe
Izzowee Vadelig Lone
Eeaorge Sodeeiig Loin
Laageage
Figure 18: The correlations between language modeling loss and ICL prediction accuracies. Upper: 0-shot results; Lower: 4-shot results.
Experimental results in this section require us to consider new inference dynamics for IWL scenar- ios, although we believe the discussion for ICL scenarios in this paper is robust. We demonstrate that such separate consideration of ICL and IWL is reasonable and responsible by highlighting the essential differences between ICI. and IWL settings, as also shown previous works (Chan al., 2022; Reddy, 2024).
G.1 How ACCURATE CAN INDUCTION HEADS EXPLAIN IWL INFERENCE?
Our main concern is: Whether the induction heads in our circuit for ICL can explain predict the inference behavior the IWL scenario. We design a metric to investigate the accuracy of such an explanation: We calculate the divergence between the real output probability of the model and the predicted output probability through the attention assignment of the induction heads. detail, given a label space Y, for each label / Y. we denote the loken-index set of Li. Then, we calculate the predicted output probability ò as:
ه softmax (101, ([o1,02....,99)) (xil)
(15)
24

---

Published as a conference paper at ICLR 2025
Diwepetze,
Jense Stanaes
Predicted
Real Outpet Paicen 41D
Jreae Shtrroo Divergozee Predicte Ouipen Rodl Outgest on lieno DOTTS
Diverposss
Laver TiaGiox
Figure 19: The IS divergence between the predicted output by the induction attention score (Eq. 15) and real output on ICL and IWL settings. Since not all attention heads are induction heads, the lowest data each layer selected.
= The Σ al-gg VEC:
(16)
where the ai- is the attention score of head h where the query's forerunner token serves as the attention query, and the i-th token as the attention key. Given the real output distribution ICL as we calculate the Jensen-Shannon divergence Dos o) as the metric for the accuracy of predicting from ô. For each Transformer layer, we select the results of the attention heads with the lowest divergence, and the results on SST-5 are shown in Fig. 19.
In the results, statistically, compared to the ICL setting, induction heads provide poorer explanations of model predictions in the IWL setting. However, these divergences consistently remain below the random baseline, indicating that model outputs in the IWL setting can still be weakly explained by induction heads. This indicates that the circuit utilized in this paper aligns with the IWL scenario, albeit slightly weaker.
G.2 Do INDUCTION HEADS CONTRIBUTE TO IWL PROCESSING?
To test the generalizability of our conclusion towards IWL scenarios, we conduct more ablation experiments similar to 1, but with filtered inputs with no ground-truth query label presented in the context. The results are shown in Table 6. Generally speaking, these experiments confirm that: Our theoretical framework still able explain inference phenomena IWL scenarios, thus strengthening our framework.
In detail, compared to the results of unfiltered inputs (Table 1), we highlight major phenomena, and explain them within our framework: (1) The baseline accuracies (line 1) in IWL settings are lower than in normal settings, even worse than random prediction. This easy to understand given
We select more-way task reduce the influence of frequency hias of ICI (introduced in $5.3).
25

---

Published as a conference paper at ICLR 2025
Table 6: Accuracy variation (%) with each inference step ablated on Left: Llama 3 8B, Right: Falcon 7B in IWL scenarios. Notations are the same with Table 1, significant comparisons are highlighted with arrows.
Attention Discomected
Affected Layers Ratie (frmm lager
Attention Divconmettdd
Affected Layers Ratio (from layer I)
Key Query
100%
Kay
50%
75%
None bescline)
+0 (Acc.
beaclinc)
(Acc,
Suep: Iopue Encode
Suppi Inpur Teat Eocode
Denoa Tests - Forerumuer A
26.08
2. Demn, Texls is - Forerunner 17.64
-2641
Query Texls X, Forerunner Y
17,87
Query Texls F2 Forerannrr
-2.50
Step: Semuniic Merge-
Sep2: Seowniis Merge
Label y;
Query Forerunner T4
-1:20
+0.181
+0.52
+0,85
Demo, Forerunner A
Label y
+2.95
+1.101
+0.35
+6.9%
Sixyd: Frature Retrienal & Cups
Steyd: Foururs Retrienal & Cupy
Label y.
Query Forerunner 74
$2.08
12,03 1
+5.40
222255
y. Query Fererannrr 74
$5,20
+5,031
+9.25
220.25
Beferene Volar
Beferenc Volur
Zaro-shut Randon Prodiction
+1 22 (Ac. 60,65) 0.98 (Ace: 32.50)
6
Zero-she Random Prodiction
+83. (Ace +5.00 (Ace. 32.50)
(Acs 60.00)
the frequency bias introduced in $5.3. (2) As we gradually suppress the induction heads (line 5), the accuracies increase, which is consistent with the conclusion in the main text of this paper: In IWL setting, induction heads can not find and copy any correct label-related information in the context, and only copy the label information presented in the context noisily. Therefore, after removing it from the inference process, the accuracies increase and approach O-shot level. These results confirm our main claim of this section: Induction heads are detrimental to IWL inference. (3) Interestingly, removing the copying processing (line 4) unexpectedly enhances the accuracy. We infer that the input texts of different labels under the same task still have certain background similarity of the encoding the forerunner token. Such similarities are propagated into the label tokens by Step 2, enhancing the attention flow from labels to the query's s forerunner token in Step 3. causing a decrease in accuracy. (4) Removing the demonstration text encoding (line 2) decreases the accuracy to around O. According to discussion about Step 1, in this case, forerunner tokens cannot receive information from the demonstration text, leading an equivalent sequence for the later layers without demonstration text like [ ... [sx] [ [ Given that the forerunner token [s.] is consistent throughout the input, later layers are likely to conduct induction only the forerunner token, whose subsequent tokens do not contain the correct label, leading to 0 accuracy. This is also a critical rebuttal to previous work (Wang et al., 2023), which believes the label tokens directly collect the information of input text: If that the case, then disconnecting the links from demonstration texts to the forerunner tokens will not have such a significant disturbance on the accuracy.
In summary, these results indicate: Even in the IWL setting, this paper's conclusions can explain the inference phenomena, and the findings in our paper are also significantly enhanced. This extends the applicability of our conclusions, while, given the existence of some counterexamples shown below, we still hope that future work can provide more detailed discussion of the inference dynamics under the IWL scenario.
G.3 THE COUNTEREXAMPLE
Given the demonstration shown in Fig. 20, we use the query "Geoffrey Hinton", whose ground- truth label is presented in the context as "Researcher" as the ICL example, and the query "Michael Jordan", whose ground-truth label "Athlete" is NOT presented in the context as an IWL example. Here, to investigate the behavior of the induction heads, we input both examples to Llama 3 70B, and visualize the attention scores from the query's forerunner token (which serves as the attention query) on layer 31, which is identified as the layer with the highest induction magnitude, as shown in Fig. 20. In the right part of the figure for the IWI. scenario, the attention magnitude directed toward the labeled tokens is significantly weak, with most of the attention scores being absorbed by the Attention Sink (Xiao et al., 2024) of the first token. A comparison with the left part, where a ground-truth label is given in the context can particularly highlight such an observation. Such an observation is currently aligned with our expectations since no label features similar to the query's 's forerunner token can be accessed in the IWL input.
In other words, the induction heads are almost not writing demonstration relevvan information to the query in the IWL scenario. It is intuitive to infer that the model cannot predict the label for "Michael
26

---

Published as a conference paper at ICLR 2025
Demonstration: Taylor Swift: Singer An Donald Trump: Politician \n Ilya Sutskever: Researcher \n
Query: Geoftrey Hinton (Researcher)
Query: Michael Jordan (Athlete)
"Singer" "Politician" "Restarch# #or*
"Singer" "Politician" "Rescarch#
Aneroon Sorres Visallnd from the Queey's bevnuster Loyer
Alteeticn Scoees Vasalord froe the Query's Reenuparr Laver
0.30
4-
0.30
10-
12
12
18-
18
20
Attention Head Index
20
24
Attention Head
25
Altemban Scores
34
0.15
22
0.15
0.10
05-
02
64
80
M0
00.00
-000
S-R22 22 28R
Token Index
Token Index
Figure 20: A counterexample when the induction head's behavior cannot predict the LM's inference behavior. Given the demonstration shown in the figure, the attention scores from the label's forerun- ner token are visualized layer 31 of Llama 70B. The left part is standard ICL scenario where ground-truth label the query can be accessed the demonstrations. The right part is the IWL scenario where the ground-truth label of the query is not presented in the demonstrations. A clear induction pattern can not be observed in the IWL scenario.
Jordan" well, and the demonstrations cannot help the prediction either. However, as shown in Ta- ble 7, the model produces good predictions and benefits from the demonstration, which contradicts our expectations. Such contradiction indicates that even if our inference circuit can robustly explain the inference behavior in the ICI. scenario, it can not generalize to the IWL scenario generalizatively.
G.4 DISCUSSIONS: THE POSSIBILITY OF DIFFERENT INFERENCE DYNAMICS IN IWL SCENARIOS
Our explanation for this counterex- ample is: It be considered that in the IWL. settings, some different inference dynamics contribute the prediction. Such difference is natural, since even if the ICL and IWL input data share a consistent format, they are fundamentally distinct, and some- times even antagonistic, as shown in previous works (Chan et al., 2022;
Table 7: Label probabilities from the model predictions of the ICL (query: "Geoffrey Hinton" (Researcher) and IWL (query: "Michael Jordan" (Athlete)) scenario shown in Fig. 20. In this case, IWL predictions also benefit from demonstrations.
Label Token "Ath!"
"Research#"
Singer"
"Pliiician"
3-shot
1.00
0.00
0.00
0.00
IWL
O-shot
0.01
ICL 3-shot
0.00
1.00
0.00
Reddy, 2024), which find that toy Transformers are difficult to perform well both types of data with a same set of parameters. While, large models may allow for the coexistence of multiple in- ference dynamics, as discussed in $5.2, making LLMs able yield better performance both ICL and IWL inference scenarios.
Therefore, although claim that circuit explain inference behavior under some of the IWL scenarios, it is reasonable to consider ICL data and IWL data separately. and this paper conducts a robust analysis under ICI. conditions, leaving the IWL scenario for future works.
27

---

Published as a conference paper at ICLR 2025
Loyer
Hoad Contect k
Lopee 9.00
k=1
k=
x=
kw1
k-2
og
Co
Layse 31, Hoad 52, Oornuct Bute: 0.05
lager Dand Correct Rate, 0.00
Layse $1, Hoad 52, Connect Rate: 0.d5
Layer Daad Correct Rate, 3.00
X
Figure 21: Supplemental experiment result on more samples for Fig. 7.
H AUGMENTATED EXPERIMENT RESULTS
H.1 ATTENTION HEAD STATISTICS
We count the marked times of each attention head as Forerunner Token Head (Fig. 22, 23, 24, 25) Correct Induction Head (Fig. 26, 27, 28, 29) by all data samples from each dataset on each model.
Forerunner Token Head Statistics. We plot the distributions of the marked Forerunner Token Heads towards correct and wrong labels: There are observable morphological differences figures across different datasets, while the forerunner token heads marked on the correct and incorrect labels of the same dataset are almost identical. The detailed data confirms our conclusion in $4.1.
Induction Head Statistics. We plot the distributions of the marked Correct Induction Heads, where there are observable morphological differences figures across different datasets, but also signifi- cant overlaps, which confirms our conclusion in $4.2.
H.2 OTHER LMS' EXPERIMENT RESULTS
The results of most experiments in the main text on Llama 8B are
Table 8: Results of Table I on Falcon 7B.
Atlention Disconnected
Layers layer
shown in Fig. 30, 33, 36, and 39;
Key + Query
25%
50%
75%
100%
The results of most experiments in
None (4-shot buseline)
10 (Aec. 66.27)
the main text on Falcon 40B
Stepl: Inpur Text Encode
shown in Fig. 31, 34, 37, and 40; The results of most experiments in the main text on Falcon 7B are shown in Fig. 32, 35, 38, 41, and Table 8.
2 Demo. Texts Forerunner
no 24
win w
Query Texts x,
S
-8. 30
-n -21.1
-28 81
-
-11.4
Semantics
Demo. Foreruaner Label
w0.00 2.33
-0.e 1.000
+8.06 0Lc0
0000 L.
Step3: Feature Retrieval & Capy
From these results, we can conclude consistently with the main text. How- ever, as discussed in $5.2, inference dynamics these models are delo- calized, thus clear serialization of the 3 steps can not be observed in these results.
5 Label - Query Forerunner
+3
-3 01
7.91
- 02
Reference
6
Zero-shot
4.28 (Acc. 60.90)
Random Prodiction
W2.77 (Ace. 32.30)
H.3 MORE RESULTS OF FIG.
To enhance the persuasiveness, we additionally and randomly try input samples as supplements to Fig. 7 SST-2 and Llama 70B as shown in Fig. 21. From these results, can observe similar phenomena to Fig. 21 and conclude consistently.
28

---

Published as a conference paper at ICLR 2025
Pereuazar: Ddea Head Comaliid
Parmreearr Stam tteed Wenay
Pormsnar Token Bead Carrel
Pererusner Taken Whong
330
-38
300
-34
Houdd
Head
SST-2
MR
Reveusman Tike Hand Comactaao
Pommenee Teken Read Cameet Label
Decerumer Takes Haad Whong Lakel
-28
10
FP
SST-5
Revensase Thkee Hoad Cam
Panmrnnee Bead
Darvenmet Tekoo Road
Dorouamer Hosd
189
-14
100
3laed
Iladd
TREC
AGNews
Figure 22: Forerunner Token Head marked on Llama 3 70B.
SST-2
MR
(HNE
F.P
SST-5
29

---

Published as a conference paper at ICLR 2025
TREC
AGNews
Figure 23: Forerunner Token Head marked on Llama 8B.
Forerunner Token Head to Correct Label
Forerunner Taken Head to Wrong Label
500
Transformer Block
Transformer Blocke
21
21
300
43
41
51
200
200
120
100
Head
Head #
SST-2
Foreranner Token Head to Correct Label
Forerunner Token Head to Wrong Label
Transformer Block
Transformer Block
31
300
300
41
51
200
200
20
40
60
80
1.00
120
20
40
8.0
1.00
120
MR
Forerunner Token Head to Correct Label
Forerunner Token Head to Wrong Label
Transformer Block
Transformer Block
500
21
440
21
400
31
300
31
300
41
41
200
200
00
80
100
120
20
60
100
120
Head
Head
FP
Head Correct Label
Forerunner Head Wrong
Transformer Block
500
Transformer Block
500
211
400
21
400
31.
51
20
1.00
40
Head
Head
SST-5
30

---

Published as a conference paper at ICLR 2025
Forerunner Token Head to Correct Label
Forerunner Taken Head to Wrong Label
500
Transformer Blook
Transformer Block
400
400
31
300
21
300
01
01
200
200
214
20
40
60
80
100
120
20
60
60
80
100
120
Head
Head
TREC
Forerunner Token Head Correct Label
Forerunner Token Head Wrong Label
Transformr Block
500
Transformer Blook
500
21
400
21
400
31
31
300
20
1.00
40
8.0
1.00
Head
Head
AGNews
Figure 24: Forerunner Token Head marked on Falcon 40B.
Porerunner Token Head to Correct Label
Forerunner Token Head to Wrong Label
Transformer Block
500
Transformar Block
500
11
-100
100
300
300
21
21
40
50
20
30
60
Head
Head
SST-2
Foreranner Token Head to Correct Label
Forerunner Token Head to Wrong Label
500
500
Transformer Blook
Transforeer Blook
11
400
400
300
300
21.
20
30
60
70
10
20
0.0
500
Head
Head
MR
Forerunner Token Head to Correct Label
Forerunner Token Head to Wrong Label
Transformer Block
500
Transformer Block
500
21
400
11
400
10
20
30
40
50
60
70
10
20
30
40
50
60
70
Head
FPP
31

---

Published as a conference paper at ICLR 2025
Forerunner Token Head to Correct Label
Forerunner Token Head to Wrong Label
500
500
Transforme Block
Transforrner Blook
11
4.00
400
300
300
21.
10
20
30
40
60
70
1.0
20
30
40
50
60
Head
SST-5
Forerunner Token Head to Correct Label
Forerunner Token Head to Wrong Label
Transformet Block
500
Transformer Block
500
400
200
100
20
30
60
50
60
70
10
20
30
60
50
60
70
Head
TREC
Foreranner Token Head Correct Label
Forerunner Token Head Wrong Label
500
500
Transformet Block
Transformer Block
21
2000
200
31
10
20
30
40
50
60
70
10
20
30
40
50
60
Head
Head
AGNews
Figure 25: Forerunner Token Head marked on Falcon 7B.
SST-2
MR
FP
Cerrel Suuudis Hsed
Carnel Setddis thrd
Cornel Inbuctis tsed
12
SST-5
TREC
AGNews
Figure 26: Correct Induction Head marked on Llama 3 70B.
32

---

Published as a conference paper at ICLR 2025
SST-2
MR
FP
SST-5
TREC
AGNews
Figure 27: Correct Induction Head marked on Llama 3 8B.
Correct Induction Head
Correct Induction Head
280
250
Transformer Block
Trensformer Block
21
150
150
31
31
100
100
41
41
00
80
100
120
20
60
100
120
Head
Head
SST-2
MR
Induction
Correct
Transformer Blook
250
Transformer Block
250
011
200
200
211
21
31
31
100
100
51
Head #
Head
FP
SST-5
Correct Induction Head
Correct Induction Head
Transforuee Block
Transformer Blook
200
200
21
100
41
41
51
-50
51
50
120
100
Head
Head #
TREC
AGNews
Figure 28: Correct Induction Head marked on Falcon 40B.
33

---

Published as a conference paper at ICLR 2025
Correct Induction Head
Correct Induction Head
250
250
Transformer Block
Transformer Block
11
150
150
21
21
100
Head #
Head #
SST-2
MR
Correct Induction Head
Correct Induction Head
250
250
Transform Block
Transformer Block
21
150
150
21-
100
21
100
20
60
Head #
Head #
FP
SST-5
Correct Induction Head
Correct Induction Head
250
250
Transforman Block
Transforman Block
200
01
150
130
100
21
100
40
20
30
Head
Head
TREC
AGNews
Figure 29: Correct Induction Head marked on Falcon 7B.
0.30
0.35
slope- 0.014
Korcal Aligmment
Kornel Alimmmntt
Band
0.20
Thiwn Type Barder Basdow
0.20
slope-- 0.006
k-
Conniidd Depermentt
0.30
0.10
0.0
25
Transformer Block Number
Transformer Block Number
Language Modeling Loss
Figure 30: Augmentated results towards Fig. on Llama
alope:
0,275
0.30
Demo
Xortsll Algament
0.500
Allguaent
0325-
0.25
Cooz
Token Type
0.20
Bandonn
slop--o 008 km
Selum lapai Stal
Deeunaase Taken tftibbl
1.0
60
10
Transformer Block Number
Transformer Block Numbor
Language Modeling Lous
Figure 31: Augmentated results towards Fig. 2 on Falcon 40B.
34

---

Published as a conference paper at ICLR 2025
skoe: 0.023 k=
Komel Alqument
0,275
0.200
Karmel Allgnment
Denoo Rand
9.5
0.235
0.25
CouL
6200
0.175
Token Iype Bandon Noxalio
0.20
skope: 0.017 K-
0.100
Takon Teat
Deesnonee Tokeo sfEabal
10
20
25
25 Number
Modiling
Figure 32: Augmentated results towards Fig. 2 Falcon 7B.
0.30
0.76
Alignment
0.25
Query Location
0.62
Query Location
0.55
80
2w 0.54 0 02
0.85
0.5
0.20
0.4
Token Type
0.48
0.8
0.87
0.92
-0.46
0.74
0.8
0.84
0.85
0.3
Token of Last Telewn Ingut Twas
0.15
0.46
0.44
8.44
0:85
10
20
30
12
2
12
Transformer Block Number
Query Location
Query
Figure 33: Augmentated results towards Fig. 3 and 4 (on Layer 16) Llama 3 8B.
0.9
0.25
Alignmmnt
Query Location
Query Location
0.6
0.82
0.95
0.8
0.94
0.96
0.79
0.92
0.94
0.96
0.4
Token
Porseunter Trienn
0.15
Lasi Token of Input Teat
0.77
0.91
0.95
0.97
96
12
0.77
0.9
0.93
0.95
0.
0.96
0.96
20
40
60
12
12
Transformer Block Number
Query Location
Query Location
Figure 34: Augmentated results towards Fig. 3 and 4 (on Layer 24) on Falcon 40B.
0.7
0.25
Alggmeet
Query Location
0.65
Query Loration
0.57
7778
0.5
0.20
0.51
0.78
0.85
-0.47
0.71
0.78
0.82
Type
0.47
0.73
0.82
0.09
0.44
0.67
0.76
0.02
0.83
Forermnne Tabon of Label
0.15
Last Tolown Twet
0.45
0.88
0.74
0.05
30
Transformer Block Number
Query Location
Query Location
Figure 35; Augmentated results towards Fig. 3 and 4 (on Layer 16) on Falcon 7B.
35

---

Published as a conference paper at ICLR 2025
0.45
Forerumnerr Token Hoad
20.0
Copy Tou
Connect Lubel
1.0
Copy Magnitude
1.0
Korrnll Alignment
0.35
15.0
Whing Label
6.8
Accuracy
0.30
12.5
0,4
Predac
0.20
Bassine Label
7.5
0.15
Bsooding
0.2
Wineg
Gurrent Lakel with Creteet
50
23
10
1.3
20
25
30
Transformer Block Number
Transformr Block Number
Transformer Block Number
Figure 36: Augmentated results towards Fig. 5 on I.lama 3 8B.
Fommunneer #
1.0
Magnltade
Kornel Aligameat
Woung Labal
0.35
Accuracy
0.30
0.6
2.0
40
Handom
Dereett Label
Abotrust Pocurannar
0.2
0.15
Foreruneer Encoding
20
50
30
50
Transformer
Block
Transformer
Figure 37: Augmentated results towards Fig. Falcon 40B.
Copgy Magninude
Kornel Aligument
0.45
Head
Cops To:
Commet Jhwa
1.0
Wibnng Label
0.8
2s
Accursey
Anm
0.30
Pororunner
35
0.4
0.1
0.20
Aloizact Label Poncunee
Rancom Sancins
10
0.2
Cortect Label
0.15
Pomernmner Bnosding
0.0
Transfomeer Blnck Number
Transformer Block Number
Transformar Block Number
Figure 38: Augmentated results towards Fig. 5 on Falcon 7B.
20
Induction Contect Induction
Correct Label Ansignment
S5T2
Induction Heud
0.04
0.5
Dataood
FP
0.74 0.70
10.
0.4
SSTS
0.6
0.55
0.58
TREC
0.6
0.54
0.55
0.81
0.3
Vanilla Altention
Head Average Random
AGN
-0.6
0.65
0.59
0.72
0
0.2
Best Ind. Head
55T2
10
15
20
25
30
10
15
20
25
30
Transformer Block Number
Transformer Block Number
Figure 39: Augmentated results towards Fig. 6 Llama 8B.
36

---

Published as a conference paper at ICLR 2025
5S12
Inducioo Heud
MR 0013
30-
0.5
Dataset:
P7 2P-00.3
0.05
20-
Correct Label
RSTS 0.58 0.58
0.53
TABIC 0.52 0-40
0.44
0.69
0.3
Venllis Attention
Head
AGN 00.00
0.50
0.53
9.67
0.63
0.2
S5T2
MI
EP-
TRBC
AGX
10
sl
40
so
60
40
50
60
Transformer Number
Transformer Block Number
Dataset
Figure 40: Augmentated results towards Fig. Falcon 40B.
25
Induction Correct Indoction
SST2
Induction Head
0.6
MR-0.83
Datawit
2P 0.71
0.72
1.5
Correct Label
GSTO 0.59
0.58
0.53
10
0.4
VRBC 9948 0.45 0.43
0.69
Venilla Attention
Head Aveen
AGN -0.54
0.54
0.52
0.67
0.64
0.2
SST2
MIK-
E2
TRSC
AGN
20
25
Transformer Block Number
Transformer Block Number
Dataset
Figure 41: Augmentated results towards Fig. 6 on Falcon 7B.
37


Reviews:
[{"review_id": "fzb5M5P4xB", "reviewer": "Reviewer_keS9", "summary": "This paper aims to explain the mechanisms behind in-context learning (ICL) using the inference circuit framework.\n\nAccording to the authors, the ICL process consists of three internal steps:\n1. Summarize: Large language models (LLMs) encode each demonstration within its corresponding forerunner token,  $s_i$ .\n2. Semantics Merge: The semantics of each demonstration and its label are combined into the representation of the label  $y_i$ .\n3. Feature Retrieval and Copy: LLMs rely on the accumulated labels  $y_{1:k}$  to respond accurately to the query  $s_q$ , yielding the most appropriate answer.\n\nEach step is empirically validated using methods such as kernel alignment and embedding comparisons. The authors also seek to align their findings with those of prior research, reinforcing the credibility of the arguments presented in this work.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Attempts to explain the inner workings of ICL, based on reasonable assumptions and investigative tools.\n- The findings align with previous work, encouraging readers to accept the claims presented in the paper.\n- Visualized results help readers quickly grasp the core concepts and findings of the paper.", "weaknesses": "- While the proposed framework is logical and reasonable, it remains challenging to argue definitively that the core mechanism of ICL follows the assumptions presented in the paper. As noted in Section 5.2, there are exceptions that do not align well with the proposed framework, raising concerns that the explanations may be superficial and fail to capture the essence of ICL. This is understandable, as fully explaining the inner workings of neural networks is inherently difficult, if not nearly impossible.\n- I am somewhat unclear about the core novelty of this paper. As I understand it, the primary contribution seems to be the attempt to apply the existing inference circuit framework to the ICL of specific LLMs, including LLaMA 3. In Section 2.1, I did not find explanations that clarify why the procedure conducted in this paper is particularly innovative, compelling, or novel. More comprehensive comparisons with prior work employing induction or inference circuits to illustrate the inner workings of ICL would be helpful to underscore the merits and uniqueness of this study.\n- In Section 3.1, sentence embeddings generated by an external encoder (BGE M3) are compared to hidden representations computed by an LLM. Since these two representations come from different models, without any modification or fine-tuning, there is a risk that their vector spaces are not aligned or compatible. This raises concerns about whether this experiment is sufficiently reasonable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R92J6BarjR", "reviewer": "Reviewer_Z26B", "summary": "The authors propose a three-step inference circuit to capture the in-context learning (ICL) process in large language models (LLMs):\n\n1. Summarize: Each input (both demonstration and query) is encoded into linear representations in the model's hidden states.\n\n2. Semantics Merge: The encoded representation of each demonstration is merged with its label, creating a joint representation for the label and demonstration.\n\n3. Feature Retrieval and Copy: The model retrieves and copies the label representation most similar to the query's representation, using this merged representation to predict the query's label.\n\nThis circuit explains various ICL phenomena, such as position bias, robustness to noisy labels, and demonstration saturation. Ablation studies show that removing steps in this process significantly reduces performance, supporting the dominance of this circuit in ICL. The paper also identifies some bypass mechanisms that operate in parallel, assisting the inference circuit through residual connections.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors proposed to use the mutual nearest-neighbor kernel alignment of the intermediate representations of LLMs and sentence embeddings produced by another pre-training model to assess the quality of these representations. This method is novel.\n\n2. Extensive analysis has been performed on all three steps of the proposed framework. Possible explanations have also been provided for many phenomena.\n\n3. The experiments are performed with real-world LLMs and datasets, which makes the insights more likely to be useful in practice.\n\n4. The paper is well-written and easy to follow.", "weaknesses": "1. The majority of the analysis is based on associations without verifying their strength and whether those effects are causal. For example, Figure 2 right does not look significant enough for me. The peaks highlighted in Figure 5 also look pretty noisy to me.\n\n2. The causal evidence that the authors provided in the ablation study only shows the effect of deleting the hypothesized important components in ICL. What if unimportant components are deleted? Would they have a similar effect? Only if the unimportant components have a significantly weaker effect on ICL performance, can we draw a causal conclusion that the proposed three-step process dominates ICL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gzBaVto45N", "reviewer": "Reviewer_gt4r", "summary": "This paper investigates the mechanisms within large language models (LLMs) that enable in-context learning (ICL) tasks, breaking down the process into three distinct stages: summarization, semantic merging, and feature retrieval/copying. The study employs a variety of experiments across multiple LLMs to validate its findings. Overall, this paper presents valuable insights that can contribute to the field of LLM research, particularly within the ICL community.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The findings in this paper are clearly explained. Experimental results and visualizations enhance readability and help the audience follow the study's progression easily.\n\n2. This work is well-connected to existing ICL research, with discussions that compare its findings to prior studies on ICL explainability and demonstration selection.\n\n3. The study uses multiple LLMs, strengthening the generalizability of the findings across different model architectures.\n\n4. The insights provided are thought-provoking and have potential practical implications for ICL applications.", "weaknesses": "1. [Section 3.1] The authors used mutual nearest-neighbor kernel alignment to evaluate LLMs' summarization abilities. However, the term “summarize” lacks clarity. Does it refer to encoding capabilities similar to those in BGE?\n\n2. [Section 3.1] Additionally, the kernel alignment metric may not be sufficiently robust, as alignment scores in Figure 2 range only from 0.25 to 0.35, which is not significant enough. Consequently, the finding on “summarization” may hold only to a limited extent.\n\n3. [Section 4.1 – Copying from Text Feature to Label Token] It is unclear whether the copying mechanism is applied solely to label tokens or if it extends to other tokens within the input. Using results from other tokens as a baseline could provide a more nuanced understanding of the copying process.\n\n4. [Figure 5, Right] After layer 40, the classification accuracy drops significantly. The authors did not investigate potential reasons for this decline. Could it be due to the gradual degradation of copied information?\n\n5. The experimental setup in Section 5.1 is insufficiently detailed. For instance, how many attention heads are disconnected at each layer? Additionally, the experiments lack certain baselines, such as randomly disconnecting some attention heads to observe the impact on model performance.\n\nDespite these questions and weaknesses, I believe this paper still offers meaningful insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bfcRvjr0vE", "reviewer": "Reviewer_yFjP", "summary": "This paper proposes a three-stage ICL circuit hypothesis and provides thorough empirical examinations of the existence and significance of these stages. Within this circuit framework, many phenomena are explained, such as how Forerunner Tokens encode input text representations and the bias in input text encoding towards position. These findings present intriguing insights.", "questions": "The three-stage framework proposed in the paper is quite interesting. The questions I have raised are mentioned in the weaknesses section. Here, I would like to know what inspired you to propose this framework. Each part of the framework consists of very specific ideas—were they derived from repeated trial and error, or were they inspired by something else? Alternatively, are they improvements on a significant prior work?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **Originality:** To my knowledge, the three-stage circuit proposed by the authors is a novel contribution.\n- **Quality:** The hypothesis put forward is reasonable, and the experiments are thorough with a well-crafted methodology.\n- **Clarity:** The arguments and evidence presented in the paper are clear, and the experimental descriptions are appropriately detailed.\n- **Significance:** Currently, ICL is one of the most important applications in the LLM field, and understanding the mechanisms behind ICL will greatly aid in enhancing its performance.", "weaknesses": "The three-stage ICL framework appears to have implicit applicability conditions, which I believe should be clarified.\n\nFor example, in Fig. 1 on page 2, a few-shot scenario with $ k=2 $ is presented, which indeed fits the three-stage ICL circuit framework. However, in a zero-shot scenario ($ k=0 $), step 1 may still exist, but steps 2 and 3 would not be applicable. In a few-shot scenario with $ k=1 $, steps 1 and 2 might still apply, but step 3 cannot exist.\n\nTherefore, the framework proposed in this paper should be limited to discussions of scenarios where $ k \\geq 2 $. A related question arises: if the focus is restricted to this scenario, what potential issues might emerge?\n\nFurthermore, if we condition on $ k \\geq C $ (where $ C $ is a fixed value), could this value vary depending on the problem type? For instance, in tasks like SST-2 and SST-5, which have different label set sizes, might the value of $ C $ differ across these scenarios?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)