Identifying Semantic Induction Heads to Understand In-Context Learning (2024)

Abstract: Although large language models (LLMs) have demonstrated remarkable performance, the lack of transparency in their inference logic raises concerns about their trustworthiness. To gain a better understanding of LLMs, we conduct a detailed analysis of the operations of attention heads and aim to better understand the in-context learning of LLMs. Specifically, we investigate whether attention heads encode two types of relationships between tokens in natural languages: the syntactic dependency parsed from sentences and the relation within knowledge graphs. We find that certain attention heads exhibit a pattern where, when attending to head tokens, they recall tail tokens and increase the output logits of those tail tokens. More crucially, the formulation of such semantic induction heads has a close correlation with the emergence of the in-context learning ability of language models. The study of semantic attention heads advances our understanding of the intricate operations of attention heads in transformers, and further provides new insights into the in-context learning of LLMs.

heading: 1 Introduction
text: In recent years, the transformer-based large language models (LLMs) (Kaplan et al., 2020; Brown et al., 2020; Touvron et al., 2023; Bubeck et al., 2023) have rapidly emerged as one of the mainstreams in the field of natural language processing (NLP). While these models demonstrate emergent abilities as they scale (Brown et al., 2020; Wei et al., 2022), they become less interpretable due to the vast number of parameters and complex architectures, which emphasizes LLMs’ safety and trustworthiness (Carlini et al., 2021; Manakul et al., 2023; Ren et al., 2024). Thus, beyond classical gradient-based explanations (Simonyan et al.,
*This work is done during internship at Shanghai Artificial Intelligence Laboratory.
†Corresponding author.
2013; Li et al., 2015), and perturbation-based explanations (Ribeiro et al., 2016; Lundberg and Lee, 2017; Sundararajan et al., 2017), recent studies in mechanistic interpretability (Cammarata et al., 2020; Elhage et al., 2021) attempt to reverse engineer the computations in transformers (particularly attention layers).
The mechanistic interpretability on transformer language models was first performed by Elhage et al. (2021). They disentangle two circuits from the operation of each attention head in transformers: Query-Key circuit (determines which token the head prefers to attend to) and Output-Value circuit (determines how the head affects the output logits of the next token). Then, Elhage et al. (2021) discover that some attention heads prefer to search for a previous occurrence of the current token in context and copy the next token associated with that occurrence, as shown in Figure 1. The attention heads performing such operations are termed induction heads. Taking a step further, Olsson et al. (2022); Bansal et al. (2023) have discovered that the presence of induction heads has a close correlation with the in-context learning (ICL) ability of LLMs. This finding highlights the importance of understanding the behavior of attention heads to the overall learning capabilities of LLMs.
On the other hand, semantic relationships have a vital importance on natural language understanding and processing. However, Elhage et al. (2021) only focus on whether the attention heads copy the attended token, without studying semantic relationships between tokens. Another major limitation of previous studies is that Olsson et al. (2022) does not explain the popular few-shot in-context learning schema. Instead, they study the loss decreasing along with the increase of token indices. This setting does not fully capture the complete ability of LLMs to learn from the context.
In this work, beyond simple copying, we delve deeper into high-level relationships encoded in at-
6916
tention heads. We focus on two types of relationships: (1) syntactic dependencies in the sentence and (2) semantic relationships between entities. Please refer to Figure 1 for examples. Each relation is represented as a triplet: (head, relation, tail). We find that when attending to head tokens, some attention heads prefer to raise the output logits of tail tokens associated with specific relations. Such attention heads encoding semantic relationships are termed semantic induction heads. Unlike conventional induction heads, semantic induction heads learn and leverage the semantic relationships between words to infer the output, thereby providing a better understanding of the behavior of networks.
Inspired by the study of induction heads and incontext learning, we further explore the correlation between semantic induction heads and in-context learning. We first categorize the in-context learning ability into three basic levels: loss reduction, format compliance, and pattern discovery. These three levels progressively increase in difficulty, with each subsequent level building upon the achievements of the previous one. The experimental results are consistent with our hypothesis, demonstrating the emergence of three levels of ICL in a sequential manner. Specifically, we observe the emergence of loss reduction from the beginning of the training, followed by the emergence of format compliance at around 1.6B tokens, and finally, the emergence of pattern discovery after training on approximately 4B tokens. Moreover, we find semantic induction heads mainly emerge around the same time as pattern discovery. Based on this finding, we infer that the emergence of semantic induction heads plays a
crucial role in facilitating the ICL of LLMs. Our contributions can be summarized as follows. • We unveil the existence of semantic induction heads in LLMs that extract semantic relationships within the context. This discovery deepens the study of mechanistic interpretability and enhances our understanding of transformer-based models.
• To study the ICL in LLMs, we categorize it into three different levels and observe the gradual emergence of different levels of ICL during the early training stage of LLMs.
• Through a meticulous analysis of early checkpoints in the training of LLMs, we establish a close correlation between semantic induction heads and the occurrence of ICL.

heading: 2 Related Works
text: In this section, we provide an overview of recent advancements in the interpretability of neural networks, particularly mechanistic interpretability. On the other hand, previous studies (Petroni et al., 2019; Zhang et al., 2022) have also explored the topic of semantic relationships in models. However, our study distinguishes itself by focusing on mechanistic interpretability.
Previous studies in interpretability can be roughly categorized into the following four types: estimating the attribution of input features to the network output (Ribeiro et al., 2016; Sundararajan et al., 2017; Lundberg and Lee, 2017; Yang et al., 2023; Modarressi et al., 2023), discovering interaction patterns between input features (Ren et al., 2021, 2023; Liu et al., 2024; Zhou et al., 2024), extracting concepts from intermediate-layer
features (Kim et al., 2018; Thomas et al., 2023; Qian et al., 2024), and designing self-explainable architectures (Li et al., 2018; Das et al., 2022). As transformer-based models become mainstream, recent works focus on understanding the attention mechanism. The most direct approach is to visualize the attention using bipartite graphs (Vig, 2019; Yeh et al., 2024) or heatmaps (Park et al., 2019). Another line of research aims to reverse engineer the operation of attention heads, called mechanistic interpretability (Cammarata et al., 2020; Elhage et al., 2021).
Elhage et al. (2021) proposed the circuit analysis (introduced in Section 3) to examine the operation of attention heads, and they found induction heads in attention-only models. Olsson et al. (2022) further investigated the correlation between the formation of induction heads and ICL. Bansal et al. (2023) observed an overlap between the set of induction heads and the set of important attention heads for ICL. Using circuit analysis, Wang et al. (2023) also found some attention heads performing the function of identifying/removing names in the indirect object identification task. Other studies (Lieberum et al., 2023; Geva et al., 2023; Mohebbi et al., 2023) intervened the attention or FFN layers to study their functions. In this paper, we leverage the circuit analysis to investigate semantic relationships in attention heads.

heading: 3 Semantic Induction Head
text: Preliminary. Elhage et al. (2021) rewrite the operation of a multi-head attention (MHA) layer containing h attention heads as follows. ∑H
h=1 softmax
( xW hq (xW h k ) T / √ dh ) xW hv W h o
= ∑H
h=1 softmax
( xW hQKx T / √ dh ) xW hOV
(1) where x = [xT1 , x T 2 , . . . , x T N ]
T ∈ RN×d denotes the embedding sequence, and xi = tiWe ∈ R1×d is the embedding of the i-th input token ti. We ∈ R|V|×d denotes the embedding layer over a vocabulary V . W hq ,W hk ,W hv ∈ Rd×dh denote the query, key, and value transformations in the h-th attention head. The output transformation Wo can be decomposed as Wo=[(W 1o ) T (W 2o ) T . . . (WHo )
T ]T , where W ho ∈ Rdh×d.
In Equation (1), W hQK = W h q (W h k ) T , termed the Query-Key (QK) circuit, is responsible for computing the attention pattern of the head, thus determining the head prefers to attend to which token.
On the other hand, the matrix W hOV = W h v W h o , termed the Output-Value (OV) circuit, computes the independent output of each head at the current token regardless of the attention pattern. The output of the OV circuit can be projected back to the vocabulary as xW hOV Wu by the unembedding transformation Wu ∈ Rd×|V|. The projected vector represents the influence of the attention head on the output. Importantly, according to (Elhage et al., 2021), both the QK circuit and OV circuit are directly performed on input embeddings, facilitating the understanding of operations in attention heads.
Based on the above decomposition, Elhage et al. (2021) identify a specific behavior in attention heads, which they refer to as induction heads. They observe this behavior in attention heads when presented with sequences like “[A] [B] · · · [A]”. In these induction heads, the QK circuit causes the attention head to attend to the token [B], which appears next to the previous occurrence of the current token [A]. This behavior is termed prefix matching. Then, the OV circuit increases the output logit of the attended token [B], termed copying. This mechanism is shown in Figure 1.
Main experimental setup. We use the opensourced InternLM2-1.8B1, which contains 24 layers and each layer consists of 16 attention heads. We use the Abstract GENeration DAtaset (AGENDA)2 (Koncel-Kedziorski et al., 2019) for testing because it contains well-annotated relations between entities. The test set of the AGENDA dataset has a total of 1,000 samples, each consisting of a knowledge graph and a corresponding paragraph that describes relations in the knowledge graph. For syntactic dependencies, we split paragraphs in the AGENDA dataset into individual sentences, and use spaCy (Honnibal et al., 2020) to extract dependencies between tokens.

heading: 3.1 Syntactic Dependency in Attention Heads
text: In this section, we explore whether attention heads encode more complex knowledge beyond copying. We first study a basic pairwise relation inherent in natural language, syntactic dependency, representing the grammatical structure of a sentence.
We focus on three frequent types of syntactic dependencies: subject-predicate (subj), predicateobject (obj), and modifier-noun/verb (mod). Each relation is represented as a triplet: T =
1https://huggingface.co/internlm 2https://github.com/rikdz/GraphWriter
(ts, relation, to), relation ∈ {subj, obj, mod}. ts denotes the token in the head node, termed the head token, where s denotes its index in the input sequence. to represents the token in the child node, termed the tail token, where o represents its index in the sequence.
To examine whether attention heads encode dependency relationships between tokens, we use the OV circuit to analyze the influence of attention heads on output logits of tail tokens when attending to head tokens. Given an input sequence [t1, . . . , tn] and the triplet T = (ts, relation, to), we measure how much each head h raises the tail token to when attending to the head token ts.
First, we look for attention heads that attend to the head token via the QK circuit. Given the current token tj , let Ahj = softmax(xjW h QKx
T ) denote the attention probability of the h-th attention head over all tokens. If the head h attends from tj to the head token ts with a high probability, i.e., s = argmax1≤k≤j Ahj,k and A h j,s/maxk ̸=s{Ahj,k} > τ , we consider this head as a potential candidate for representing the triplet associated with the head token ts. Otherwise, we skip this head on this triplet. We set τ = 2.2 in experiments3.
Second, we examine whether these heads raise the tail token to by computing the projection of the OV output on the vocabulary, xjW hOV Wu ∈ R|V|. Similar to (Bansal et al., 2023), we first compute the output probabilities of tokens as ph,j = softmax(xjW hOV Wu) ∈ R|V|. Then, we extract the probability ph,jtk (k ≤ j) of each token tk before the token tj . Here we suppose all tokens tk before tj are unique for simplification, and if several positions share the same token (e.g., tk1 = tk2), we only consider it once. These probabilities are further transformed by subtracting their mean value and ruling out values smaller than zero, as follows.
qh,jtk = max(0, p h,j tk − E1≤k′≤j [ph,jtk′ ]) (2)
3Please refer to Appendix A for discussions about the setting of τ .
This transformation helps to focus on tokens whose output probabilities are raised. Then, we compute the following ratio ah,jT to measure the significance of raising the tail token to relative to all tokens before tj .
ah,jT = q h,j to / ∑j k=1 qh,jtk (3)
Finally, for each head h, we average the relation index ah,jT across all current tokens tj and across all triplets T . Note that we only consider current tokens after the head and tail tokens, i.e., j ≥ max(s, o). Model’s ability in understanding dependencies. Before examining whether attention heads encode dependencies between tokens, we first test the model’s overall proficiency in learning and understanding dependency relationships. We follow Clark et al. (2019) to train an attention-andwords probing classifier, which takes the word embeddings and the attention weights extracted from InternLM2-1.8B as input and fits the probability of each token being the syntactic head of another token. We train the classifier on 200 sequences from the AGENDA dataset, each with a length of less than 32. Then, we evaluate the accuracy of the predicted head positions on another 100 sequences. For 52% tokens in the input sequence, the classifier can identify the position of their head tokens based on attention weights extracted from InternLM21.8B. This accuracy is significantly higher than the random guess, indicating that InternLM2-1.8B can well understand syntactic dependencies. Therefore, we are motivated to further study dependencies in its attention heads.
The subj and obj dependencies are encoded in attention heads. Figure 2 shows heatmaps of the relation index of each attention head w.r.t. three types of dependency relationships. As a baseline, we also compute the relation index when setting tail tokens in all triplets to the 10th token. Such triplets do not represent any relationships, and the relation
indexes of attention heads are lower than 0.1. In comparison, for the subj and obj dependencies, the attention heads exhibit relation indexes close to 0.3, and the relation index w.r.t. the mod dependency is a bit lower. This suggests the model may have better learned the subj and obj dependencies than mod. In Section 4.1, it is also observed that the model exhibits better performance on the justification of the subj and obj dependencies than the mod dependency, coinciding with this discovery. Furthermore, relation indexes for the obj dependency are more sparsely distributed than those for subj dependency. There are about ten attention heads that exhibit salient values for the obj dependency. This indicates that the model may store the obj dependency using a few attention heads, while the subj dependency is widely encoded in more attention heads. These observations highlight the varying degrees of the model’s understanding and representation of different dependency relationships.

heading: 3.2 Semantic Relationship in Attention Heads
text: In addition to syntactic dependencies, we also study semantic relationships between entities in knowledge graphs. There are seven types of relations between entities in the AGENDA dataset: Partof, Compare, Used-for, Feature-of, Hyponym-of, Evaluate-for, and Conjunction.
Similar to dependencies in Section 3.1, we also expect to represent each relation between entities as a triplet T = (ts, relation, to). Because the entities in sentences often consist of multiple tokens, we replace them with capital English letters4 as shown in Figure 3. By representing each entity with a single letter, we can directly adopt the metric in Equation (3). Besides, we remove the most frequent function words annotated by spaCy in the sentence. This step helps to reduce noise and focus on the more informative content words.
Various semantic relationships are encoded in attention heads. Figure 4 illustrates relation in-
4We exclude special letters like A,I,N,S,W, and E, which often appear alone and are meaningful alone.
dexes of attention heads w.r.t. seven types of semantic relationships. In contrast to syntactic dependencies, semantic relationships exhibit clearer patterns within attention heads. Each type of relationship is represented by a range of 5 to 15 attention heads. Interestingly, certain relationships, such as "Usedfor," "Hyponym-of," and "Conjunction," appear to be more clustered in specific attention heads. These findings suggest that the model possesses a capacity to represent various semantic relationships in attention heads. Furthermore, considering these semantic relationships are bidirectional, we also analyze the reverse relation triplet (T̃ = (to, relation, ts)) in attention heads in Appendix B. Results in Figure 16 show that some attention heads store both directions of the relationship, reflecting the model’s ability to understand the reciprocal nature of these semantic relationships.

heading: 4 In-Context Learning and Semantic Induction Heads
text: In this section, we investigate the correlation between in-context learning and semantic induction heads. We first categorize the ICL ability into three levels and observe the gradual emergence of different levels of ICL. Then, we investigate the formation of semantic induction heads in the training process to understand the emergence of ICL.

heading: 4.1 In-Context Learning of Different Levels
text: In-context learning refers to the ability to learn from the context to perform an unseen task. However, there is no standard measurement for the ICL ability of LLMs. Kaplan et al. (2020); Olsson et al. (2022) consider ICL as the ability to better predict later tokens in the context than earlier tokens. Another more widely adopted definition of ICL follows a few-shot setting. In this setting, language models are provided with a few examples within the context of the prompt, and the model can better perform the task with more examples given. This definition emphasizes the model’s ability to extract and generalize the information in the context.
We rethink the ICL ability from the perspective of what the model has learned from the context. The loss reduction considered in (Olsson et al., 2022) only demonstrates that the context does help models make predictions, but it is unclear what the model has learned. Chomsky (1957) has proposed that when humans learn new languages, they initially grasp the surface structure of the language
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs Part-of
0.0
0.1
0.2
0.3
0.4
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Compare
0.0
0.2
0.4
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Used-for
0.0
0.1
0.2
0.3
0.4
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Feature-of
0.0
0.2
0.4
0.6
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Hyponym-of
0.0
0.2
0.4
0.6
0.8
1.0
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Evaluate-for
0.0
0.1
0.2
0.3
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Conjunction
0.0
0.1
0.2
0.3
Figure 4: Heatmaps of the average relation index of attention heads for semantic relationships in knowledge graphs.
Task Entity set Template
Binary classification (fruit, month), (furniture, profession) <E1, E2>: 0; <E2, E1>: 1
Four-class classification (fruit, month) <E1, E1>: 0; <E1, E2>: 1; <E2, E1>: 2; <E2, E2>: 3
Nine-class classification (fruit, animal, month) <E1, E1>: 0; <E1, E2>: 1; <E1, E3>: 2; <E2, E1>: 3; · · · Relation justification (subj, verb), (verb, obj), (mod, obj), (part, whole) <E1, E2>: true ; <animal, month>: false
Table 1: We construct toy tasks for evaluating the ICL ability of models. E1, E2, and E3 in the template refer to instances belonging to the 1st, 2nd, and 3rd categories in each pair of entity sets. For example, binary classification contains inputs like “apple, January: 0” and “April, orange:1”.
before delving into the deep structure. Inspired by this, we hypothesize that LLMs also first learn the surface format of the context, and then gradually comprehend the deep patterns or rules within the context. Based on this hypothesis, we categorize the ICL ability into three levels:
• Loss reduction: This level of ICL is characterized by a reduction in the loss of tokens as the model predicts later tokens in the context. Olsson et al. (2022) demonstrates ICL at this initial level.
• Format compliance (few-shot): At this level, the model learns the format of examples in the prompt (e.g., numbers and symbols), and generates outputs following the same format. Although the outputs have the correct format, the predictions may be incorrect.
• Pattern discovery (few-shot): This level expects the model to recognize and comprehend the underlying pattern within the examples, and apply it consistently to generate the correct prediction.
By categorizing ICL into these levels, we can systematically assess the progression and development of the model’s ICL abilities.
Model. To study the ICL ability of the model during the training process, we train a model from scratch using the InternLM framework (Team, 2023). The model contains 20 transformer layers, and each layer consists of H=16 attention heads.
The hidden size of the model is d = 2048, thus each head has a dimension of dh = 128. The model is trained using the SlimPajama dataset (Soboleva et al., 2023) on 32 GPUs, and the batch size on each GPU is 128K tokens. We train the model for 40k steps, with checkpoints saved every 200 steps to monitor the model’s progress in relationship representing and ICL during training.
Measurements and results. We assess the ICL ability at each level using the following methods.
For loss reduction, we follow Olsson et al. (2022) but with a minor modification that improves the statistical stability. Specifically, we adjust the formula from the loss at the j-th token minus the loss at the i-th token (i ≤ j), to the averaged loss over the interval from the j ∼ (i+ j)-th tokens minus the averaged loss over the 0 ∼ i-th tokens.
We sample 100 sentences from the SlimPajama
dataset and we set (i, j) ∈ {(50, 50), (50, 450)} to measure the loss reduction at different training checkpoints. Figure 5 shows that the loss difference between later and early tokens decreases quickly from the very beginning of the training. This suggests that from the beginning of the training, the model progressively improves its ability to leverage longer contexts for better predictions.
For format compliance, we construct classification tasks in Table 1. We adopt the few-shot setting of ICL, and the prompt is designed to include several examples followed by a query. Here we ensure that when the number of examples exceeds the number of classes, there is at least one example of each class in the prompt. To test the format compliance ability of the model, we force the model to generate only one token. If the generated token is also a number, matching the format presented in the examples, we consider it to have a correct format. We compute the accuracy of the format to measure the format compliance ability of models.
Figure 6 reports the format accuracy given different numbers of shots at different training checkpoints. For two binary classification tasks, we observe that the model’s format accuracy progres-
sively improves as the number of shots increases, starting from the 400th step. This suggests that the model’s format compliance ability emerges at the early training stage, and it is independent of the entities involved in the task. For the four-class and nine-class classifications, the model gains improvement with an increasing number of shots at later stages of training (the 600th step and the 800th step, respectively). This indicates that the format compliance to more difficult tasks tends to appear at later training stages. Despite that, the model consistently achieves around 100% format accuracy when using 20 shots at the 1k-th step, and achieves a high accuracy with only one shot after 3k steps. We also measure the format compliance ability from the perspective of the minimum number of shots required to achieve a format accuracy of over 80%. Please refer to Appendix C for details.
For pattern discovery, we still use the above classification tasks, but the difference is that we compute the accuracy of the predicted label. If the model can generate correct labels, we consider it to have successfully discovered and applied the underlying pattern in the prompt. Besides, we also construct four relation justification tasks in Table 1,
which are related to the syntactic dependency and semantic relationship studied in this paper.
Figure 7 reports the prediction accuracy of the model at different checkpoints. For classification tasks in the first row, the model achieves considerable accuracy with 20 shots at around 1400 steps, indicating that the pattern discovery ability is mastered later than the format compliance. Furthermore, simple binary classification tasks are learned earlier than complex four-class and nine-class classification tasks. The relation justification tasks in the second row, which are more difficult than classification tasks, are learned at later stages, typically starting from around the 2k-th step. After approximately 10k steps, the prediction accuracy tends to saturate. Figure 13 in Appendix E shows that till the end of the training, even with 100 examples, the model cannot fully learn the pattern in the prompt.
Progressive learning of ICL of different levels. From the above results, we can observe a progressive learning process for the different levels of ICL. The loss reduction happens from the beginning of the training, followed by the emergence of format compliance (after 400 steps), and pattern discovery is mastered in the last (after 1k or 2k steps). This discovery aligns with our hypothesis that three levels of ICL have increasing difficulties. Moreover, within the format compliance and pattern discovery, we observe that the model typically learns more
challenging tasks at later training stages.

heading: 4.2 Correlation Between Semantic Induction Heads and ICL
text: In this section, we investigate the formation of semantic induction heads during the training process and discover their correlation with ICL. We compute the average relation index of attention heads over all triplets for each syntactic dependency and each semantic relationship in different checkpoints. Here we only ensure s = argmax1≤k≤N Ahj,k, and do not require Ahj,s/maxk ̸=s{Ahj,k} > τ any more, because in the early stage of the training, it is too challenging to find attention heads having an extremely high attention probability on head tokens.
We find that the relation index of some attention heads increases during the same stage as the emergence of the ICL ability. Specifically, Figure 8 shows the change in the relation index for syntactic dependencies of attention heads. We sampled attention heads with an increasing relation index for visualization. It can be observed that relation indexes of some attention heads begin increasing from the beginning of the training, aligning with the emergence of loss reduction. On the other hand, relation indexes of other attention heads begin to increase after around 1k or 2k steps, which coincides with the emergence of the pattern discovery ability. Thus, we infer that the formation of semantic induction heads plays a crucial role in the development of the ICL ability. These semantic induction heads likely contribute to capturing and representing relationships between tokens, which are essential for the ICL ability.
On the other hand, Figure 9 shows the change in copying scores (Olsson et al., 2022; Bansal et al., 2023) of attention heads. Copying scores of some attention heads start increasing from 200 steps, so we infer the copying mechanism is responsible for the loss reduction and the format compliance. It is reasonable because the task used for evaluating
Figure 10: Occurrence of each attention head having the largest value of Ej [ah,jT ] with Ej [ah,jT ] > 0 over all triplets T of each type of dependency.
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
subj
0
100
200
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
obj
0 100 200 300 400 500 600
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
mod
0 100 200 300 400
the format compliance can be simply achieved by copying the token (“0” or “1”) after the colon in the preceding context to the output. More interestingly, copying scores of other heads begin to drop from the 1K step, where the pattern discovery ability emerges. Therefore, we hypothesize that the copying behavior is not always good for ICL, because sometimes direct copying may cause incorrect predictions.

heading: 5 More Discussions about Relationships Encoded in Attention Heads
text: A notable distinction between copying and relationship is that the relationship between tokens depends on the input context, while copying is context-agnostic. Therefore, different inputs may utilize different attention heads. Thus, we propose to find a common group of attention heads that represent specific relationships in different inputs. For each triplet T , we identify the attention head that has the largest value of Ej [ah,jT ] (larger than 0). Then, we count the occurrence of each head having the largest value among all triplets T for each dependency relationship.
Figure 10 shows the number of occurrences of each attention head having the largest relation index for syntactic dependencies. Please refer to Appendix G for results on semantic relationships. We find that for each type of dependency/relationship, there are around 5∼15 attention heads frequently activated by different inputs. Besides, different relationships tend to share some common attention heads (e.g., layer2, head4 for syntactic dependencies). This may indicate that these human-defined relationships are not mutually exclusive from the LLMs’ point of view. In other words, there exists a many-to-many mapping between human-defined relationships and attention heads in LLMs.

heading: 6 Conclusion
text: Previous studies (Elhage et al., 2021; Olsson et al., 2022) in mechanistic interpretability only studied the simple functions in very specific tasks (Wang et al., 2023; Lieberum et al., 2023). In this study, we extended the conventional induction
heads to analyze high-level relationships between words/entities in natural languages. Our experiments revealed that specific attention heads encode syntactic dependencies and semantic relationships in natural languages. Furthermore, we identified three levels of the in-context learning ability of LLMs, and experimental results showed they are progressively learned during the training process. Finally, we observed a close correlation between the formation of semantic induction heads and incontext learning ability, strengthening our understanding of in-context learning.
Limitations
Limitations of this paper lie in the following three perspectives. (1) While the proposed relation index has the potential to be adapted to different relationships in various languages, this paper only focuses on syntactic dependency and semantic relations in English. We think it is a promising direction to examine the representation of relationships in different languages and leave it to future work. (2) Although we have extended the simple copying operation to complex semantic relationships, the proposed method is limited to relationships between two tokens/entities. (3) Due to limitations in computational resources, we only conduct experiments on ∼1B models.

heading: Acknowledgement
text: This work is supported by Shanghai Artificial Intelligence Laboratory.

heading: A Setting of τ in the relation index.
text: The setting of the threshold τ = 2.2 for the value of Ahj,s
maxk ̸=s{Ahj,k} is based on our observation in
the distribution of values of Ahj,s
maxk ̸=s{Ahj,k} . Us-
ing input sentences and corresponding triplets in the AGENDA test set, we computed the value of Ahj,s maxk ̸=s{Ahj,k} at all heads h and all current tokens tj that satisfy s = argmaxk{Ahj,k}. The distribution of this value is shown in Figure 11. The frequency of values larger than 2.2 dropped significantly to less than 5%. Thus, we empirically set the threshold τ = 2.2 to only focus on attention heads that exclusively attended to the head token.
tion of Ahj,s
maxk ̸=s{Ahj,k} within the range of 0 10.
Moreover, we also conduct ablation experiments with different values of τ . Specifically, we compute the relation index for syntactic dependencies on InternLM2-1.8B with a smaller value τ = 2.0 and a larger value τ = 2.5, respectively. Heatmaps in Figure 15 show that the setting of τ does not significantly affect the distribution of the relation index. Different settings of τ yield a similar set of semantic induction heads that have a high relation index.

heading: B Relation index for the reverse semantic relationships
text: The semantic relationships in knowledge graphs are bidirectional, thus we also compute the relation index of attention heads for the reverse semantic relationships. Figure 16 shows the results in InternLM2-1.8B. Comparing Figure 16 and Figure 4, we find that some attention heads represent both directions of the relation.

heading: C Format compliance ability
text: Besides the format accuracy in Figure 6, we also measure the format compliance ability from another perspective: the minimum number of shots required to achieve a format accuracy of over 80%. A lower minimum number of shots indicates a better format compliance ability. We set a maximum limit of 20 shots. If the model fails to achieve an accuracy of 80% even with 20 shots, we record the result as 20. Figure 12 consistently shows that the model learns format compliance on simple tasks earlier than on complex tasks, but all achieve a good performance at 1000 steps.
Moreover, considering that the format of generating "true" or "false" in relation justification tasks is different from the simpler format in classification tasks in Figure 6, we additionally examine the format accuracy on relation justification tasks in Figure 17. The format compliance in these tasks emerges from about 2K steps, later than that in simpler tasks in Figure 6, and achieves about 50%-70% at 15K steps.

heading: D Pattern discovery tasks are not binary classification tasks
text: Unlike binary classification tasks, pattern discovery tasks are actually more difficult for generative models.
First, the model generates the next token as the prediction, which is different from the classic binary task. When generating the next token, there are a total of |V| = 92544 candidates in the vocabulary V . If the model could perfectly comply with the format, the problem is simplified as a binary classification task. However, models in the early stages of training do not have such ideal capabilities yet. For example, Figure 17 shows that the format compliance in relation justification tasks
emerges from about 2K steps, later than that in simpler tasks in Figure 6, and achieves about 50%-70% at 15K steps. Thus, the exact prediction accuracy of the model will be lower. Second, the model might inherit certain biases from the training data. Thus, the probability of generating either the token “true” or “false” is not 0.5 vs 0.5.

heading: E Pattern discovery ability of a well-trained model
text: Although we have observed the development of the pattern discovery ability in Figure 7, we find that it is hard to be fully mastered by the model. Figure 13 reports the prediction accuracy of the well-trained InternLM2-1.8B on classification tasks. Even with 100 examples in the prompt, the model still cannot perfectly recognize and utilize the pattern in the prompt. On the other hand, Figure 14 shows that InternLM2-1.8B achieves higher accuracy on relation justification tasks.

heading: F Change of relation index for semantic relationships over training time
text: This section provides results of the change of relation indexes of attention heads for semantic relationships in knowledge graphs. We consider both directions of the semantic relationship, and results are shown in Figure 18 and Figure 19.

heading: G Grouping attention heads for relations
text: As discussed in Section 5, the semantic relationships are dependent on the context, so they may be stored in different attention heads given different contexts. In this section, we perform the grouping analysis on semantic relations in knowledge graphs.
Figure 20 and Figure 21 show the occurrence times of each attention head having the largest value of Ej [ah,jT ] for triplets in semantic relations and the reverse triplets. We observe that some attention heads are commonly highlighted in different types of relationships.
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Part-of
0.0
0.2
0.4
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Compare
0.0
0.2
0.4
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Used-for
0.00
0.05
0.10
0.15
0.20
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Feature-of
0.0
0.2
0.4
0.6
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Hyponym-of
0.0
0.2
0.4
0.6
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Evaluate-for
0.0
0.2
0.4
0.6
0.8
1.0
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Conjunction
0.0
0.2
0.4
0.6
Figure 16: Heatmaps of the average relation index of attention heads for the reverse semantic relationships in knowledge graphs.
1. 0K 2. 0K 3. 0K 4. 0K 5. 0K
10 .0
K
15 .0
K
checkpoints
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
subj shot number 20 10 5 4 3 2 1 0
1. 0K 2. 0K 3. 0K 4. 0K 5. 0K
10 .0
K
15 .0
K
checkpoints
0.0
0.1
0.2
0.3
0.4
0.5
obj shot number 20 10 5 4 3 2 1 0
1. 0K 2. 0K 3. 0K 4. 0K 5. 0K
10 .0
K
15 .0
K
checkpoints
0.0
0.1
0.2
0.3
0.4
0.5
mod shot number 20 10 5 4 3 2 1 0
1. 0K 2. 0K 3. 0K 4. 0K 5. 0K
10 .0
K
15 .0
K
checkpoints
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
Part-of shot number 20 10 5 4 3 2 1 0
Figure 17: Format accuracy of different relation justification tasks at different checkpoints. Each line represents the format accuracy with different numbers of shots (examples) in the prompt.
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Part-of
0
5
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Compare
0
5
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Used-for
0 20 40 60
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Feature-of
0
5
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Hyponym-of
0
5
10
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Evaluate-for
0
5
0 4 8 12 Heads
0 4
8 12
16 20
La ye
rs
Conjunction
0
5
10
15
Figure 21: Heatmaps of occurrence of attention heads having the largest value of Ej [ah,jT ] for each triplet in reverse semantic relations.

heading: Future Work
text: We think it is a promising direction to examine the representation of relationships in different languages and leave it to future work. (2) Although we have extended the simple copying operation to complex semantic relationships, the proposed method is limited to relationships between two tokens/entities. (3) Due to limitations in computational resources, we only conduct experiments on ∼1B models