Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning
Authors: Lean Wang
Lei Li
Damai Dai
Deli Chen
Hao Zhou
Fandong Meng
Jie Zhou
Xu Sun
Venue: EMNLP 2023 Main (2023)
Keywords: in-context learning
label words
anchors
large language models
TL;DR: We prove that label words play an anchor role in the information flow of in-context learning, and design several methods to improve in-context learning based on this.

Abstract: In-context learning (ICL) emerges as a promising capability of large language models (LLMs) by providing them with demonstration examples to perform diverse tasks. However, the underlying mechanism of how LLMs learn from the provided context remains under-explored. In this paper, we investigate the working mechanism of ICL through an information flow lens. Our findings reveal that label words in the demonstration examples function as anchors: (1) semantic information aggregates into label word representations during the shallow computation layers' processing; (2) the consolidated information in label words serves as a reference for LLMs' final predictions. Based on these insights, we introduce an anchor re-weighting method to improve ICL performance, a demonstration compression technique to expedite inference, and an analysis framework for diagnosing ICL errors in GPT2-XL. The promising applications of our findings again validate the uncovered ICL working mechanism and pave the way for future studies.

Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning
Lean Wang Lei Lit, Damai Dai", Deli Chen
Hao Zhou,, Fandong Meng Jie Zhou,, Xu Sun
Nattooal Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University Pattern Recognition Center, WeChat AI, Tencent Inc., China [lean, daidamai, xusun) edu. cn nlp. lilei lileigmmaii com victorchen@deepseek. com (tuxzhou, fandongmeng, wi wwwoomoooueetteeeeetttt com
Abstract
Aeeiiww
Review
Review
In-context learning (ICL) emerges as a promis- ing capability of large language models (I.LMs) by providing them with demonstration exam- ples to perform diverse tasks. However, the un- derlying mechanism of how LLMS learn from the provided context remains under-explored. In this paper, we investigate the working mech- anism ICL through an information flow lens. Our findings reveal that label words in the demonstration examples function as anchors: (I) semantic information aggregates into label word representations during the shallow compu- tation layers' processing; (2) the consolidated information in label words serves reference for I.LMs* final predictions. Based on these in- sights, we introduce an anchor weighting method to improve ICI. performance, demon- stration compression technique to expedite in- ference, and an analysis framework for diag- nosing ICL errors in GPT2-XL. The promising applications of our findings again validate the uncovered ICL working mechanism and pave the way for future studies.'
good
good
movil
movie
movie
movie
Senl
Sant
iment
imant
imser!
imant
Posrive
Poative
Postive
Postive
Reviow
Revkew
Reviow
Raviow
waste
wasto
waste
widto
money
money
money
money
Sent
Sent
Sant
Sent
imont
iment
Imont
Nogazive
Negative
Nogative
Megative
Reviow
Reviow
Reviow
fantastic
fantaztte
tantastic
fantastic
Sent
Sent
Iment
Imont
Imont
imett
Shallow layer
Deep layer
Figure 1: Visualization of the information flow a GPT model performing ICL. The line depth reflects the sig- nificance the information flow from the right word to the left. The flows involving label words highlighted. Label words gather information from demonstrations in shallow layers, which then extracted in deep layers for final prediction.
1 Introduction
In-context Learning (ICL) has emerged power- ful capability alongside the development of scaled- up large language models (LLMs) (Brown et al., 2020). By instructing LLMs using few-shot demon- stration examples, ICL enables them to perform a wide range of tasks, such as text classification (Min et al., 2022a) and mathematical reasoning (Wei et al., 2022). Since ICL does not require updates millions trillions of model parameters and relies on sumaan-understandaabe natural language instructions (Dong et al., 2023), it has become a promising approach for harnessing the full poten- tiality of LLMs. Despite its significance, the inner working mechanism of ICL remains an open ques- tion, gamering considerable interest from research
communities (Xie et al., 2022; Dai et al., 2022; Akyürek et al., 2022; Li et al., 2023b).
In this paper, we find that the label words serve as anchors that aggregate and distribute information in ICL. We first visualize the attention interactive pattern between tokens with GPT model (Brown et al., 2020) on sentiment analysis (Figure 1). Ini- tial observations suggest that label words aggregate information in shallow layers and distribute in deep layers. 2 To draw a clearer picture of this phe- nomenon, we design two metrics based on saliency
'In this paper, "shallow" or "first" layers refer to those closer to the input, while "deep" or "last" layers are closer to the output. Here, "deep layers" include those around the midpoint, e.g., layers 25-48 in a 48-layer GPT2-XI..
'tttps:/ //gbttttttuuub com/anncoku// label-wol worddwword----o are- anchors

---

Information aggregation
Information aggregation
Shallow Layers
Review: dislike
Sentiment: Negative
Review: good
Sentiment: Positive
Review: Sentiment:
Label Prediction
Deep Layers
Review:
dislike
Sentiment: Negative
Review: good
Sentiment: Positive
Review:
Sentiment:
Illustration our hypothesis. In shallow layers, label words gather information from demonstrations to form semantic representations for deeper processing, while deep layers extract and utilize this information from label words to formulate the final prediction.
Figure 2:
scores portray the information flow in ICL and further propose the following hypothesis:
into pre-calculated anchor representations since model predictions primarily rely on label word acti- vations. Testing shows a 1.8 speedup in inference with only minimal performance trade-off. (3) An error analysis of ICL on GPT2-XL demonstrates that the label confusion matrix aligns closely with the distance distribution of anchor key vectors, im- plying that errors might result from similar anchor representations. These promising applications fur- ther validate our hypothesis and shed light on future ICL studies for better transparency of LLMs.
Information Flow with Labels as Anchors :1 In shallow layers, label words gather the information of demonstrations to form seman- tic representations for deeper layers.
H2: In deep layers, the model extracts the information from label words to form the final prediction.
Two experiments are designed to validate the hy- pothesis using GPT2-XL (Radford et al., 2019) and GPT-J (Wang and Komatsuzaki, 2021) across sev- eral text classification benchmarks. (1) By blocking the information aggregation path to label words in certain layers. we find that such isolation in shal- low layers significantly impairs model performance. This indicates that label words collect useful in- formation during forward propagation in shallow layers. (2) We investigate the relationship between the attention distributions on the label words of the target position and the model's final prediction. Our results illustrate a strong positive correlation, where candidate label's probability increases with more attention weight its corresponding label token. In summary, these experimental findings suggest that our hypothesis holds well with large language models on real-world datasets.
Label Words are Anchors
This section confirms the intuitive findings using two saliency score-basee metrics as discussed $ 2.1. The quantitative results lead to a proposed hypothesis for the ICI. working mechanism: H1: In shallow layers, label words aggregate information from demonstration examples to form semantic representations for later computations. H2: In deep layers, the model makes predictions by extracting information from label words. The validation for these hypotheses is presented in 2.2 and 2.3, respectively.
2.1 Hypothesis Motivated by Saliency Scores
This section aims discover the inherent patterns in the attention interaction between tokens for a GPT model. The saliency technique (Simonyan et al., 2013), a common interpretation tool, is em- ployed for highlighting critical token interactions. Following common practice, we use the Taylor expansion (Michel et al., 2019) to calculate the saliency score for each element of the attention matrix:
Drawing on insights from the information flow perspective, we explore three approaches to en- hance ICL's effectiveness, efficiency, and inter- pretability. (1) An anchor re-weighting method is introduced, which employs learnable vector to adjust the significance of different label words in demonstrations, leading to 16.7% average accu- racy boost compared to standard ICL baselines. (2) For quicker ICL inference, inputs are compressed
And 0C(x)
I
(1)
d

---

Here, An is the value of the attention matrix of the h-th attention head in the 1-th layer, a is the input, and ((x) is the loss function of the task, e.g. the eross-entropy objective for a classification problem. We average all attention heads to obtain the saliency matrix 1 for the 1-th layer. ((i,j) represents the significance of the information flow from the j-th word to the i-th word for ICL. By observing Jt we can get an intuitive impression that as the layer goes deeper, demonstration label words will become more dominant for the prediction, as depicted in Figure 1.
Swe assesses average information flow among words, serving as a benchmark gauge the in- tensity of the patterns identified by Sup and Spg-
Experimental Settings We choose GPT2-XL from the GPT series (Radford et al., 2019) as our primary model for investigation, due to its moder- ate model size (of 1.5B parameters) that is suitable for our hardware resource and its decent ICL perfor- mance (Dai et al., 2022). For datasets, we use Stan- ford Sentiment Treebank Binary (SST-2) (Socher et al., 2013) for sentiment analysis, Text REtrieval Conference Question Classification (TREC) (Li and Roth, 2002; Hovy et al., 2001) for question type classification, AG's news topic classification dataset (AGNews) (Zhang et al., 2015) for topic classification, and EmoContext (EmoC) (Chatterjee et al., 2019) for emotion classification. Templates for constructing demonstrations are provided in Appendix A. 1000 examples are sampled from the test set for evaluation, with one demonstration per class sampled from the training set. Experiments with more demonstrations yield similar outcomes (refer to Appendix F.1 for details). Results reflect averages from five random seeds.
To draw a clearer picture of this phenomenon, we propose three quantitative metrics based on 4, Our focus lies in three components: (i) the label words, such as "Negative" and "Positive" in Fig- ure 2, denoted
P1....,,.. where C' represents the total number of label words; (ii) the target po- sition, where the model generates prediction labels (i.e., the final token in the input), which we denote as q and (iii) the text part, i.e., the tokens before label words in the demonstration.
The definitions of the three quantitative metrics follow below.
S Sup the mean significance of information flow from the text part to label words:
Results and Analysis Figure 3 reveals that: (1) in shallow layers, Spg. the significance of the infor- mation flow from label words targeted positions, is low, while Swp. the information flow from the text part to label words is high; (2) in deep layers, Spy the importance of information flow from label words to the targeted position becomes the dom- inant one. Notably, Spn and Suep usually surpass Swe. suggesting that interactions involving label words outweigh others.
Eisrecu ()(
Swp =
(Cupl
(2)
Cup = ()pk))) e C1, < pc).
S ppy the mean significance of information flow from label words to the target position:
Spq Emwecc J((
Cpol
(3)
Cpp = ((g.px): : E [].
Proposed Hypothesis Based on this, we propose the hypothesis that label words function as anchors in the ICL information flow. In shallow layers, label words gather information from demonstra- tion examples to form semantic representations for (4) deeper layers, while in deep layers, the model ex- tracts the information from label words to form the final prediction. Figure 2 gives an illustration for our hypothesis.
Sww, the mean significance of the information flow amongst all words, excluding influences represented by Sup and Spg:
Sum Eaiheoww 1((.5) 1Cavl
Own =((i,) < Cup Cpg-
Swp Spy. and Sww help assess different informa- tion flows in the model. Sup indicates the intensity of information aggregation onto label words. A high Spg demonstrates a strong information extrac- tion from label words for final decision-making
2.2 Shallow Layers: Information Aggregation In this part, we validate our hypothesis' first compo- nent. We assume that the information aggregation in ICL relies on the information flow from the text part to label tokens, which is facilitated by the trans- former's 's attention mechanism. By manipulating
'In this study, the term 'label words' is approximately equal to 'label tokens'. The only deviation is the Abbrevia-- tion' in the TREC dataset, where we use the first subword in experiments, following Zhao et al. (2021).

---

No locletton
Labe Words (Laed)
Randsm (hant)
Lobel Words (Fink)
Randon (Tinl)
10-
100
0.8
Sup
Loyelt
60
5
5pa
0.4
Suw
0.2
Label Leyalty (GPT2-XL)
Word Lovelty (GPT2-0L))
Label Coattiity (GPTT-))
Word Lovally (GPTS)
10
20
30
40
Layer
(a) Results on the SST-2 datasct
Figure 4: The impact of isolating label words versus randomly isolating non-label words within the first or last 5 layers. Isolating label words within the first 5 lay- exerts the most substantial impact, highlighting the importance of shallow-layer information aggregation via label words.
0.8
Swp
5pa
information from the prior demonstration text.
5uw
0.2
Metrics We use the following metrics to assess the impact of blocking information flow from the text part to label tokens: (1) Label Loyalty: mea- sures the consistency of output labels with and without isolation. (2) Word Loyalty: employs the Jaccard similarity compare the top-5 predicted words with and without isolation, capturing more subtle model output alterations (See Appendix C for details). Low loyalty indicates a profound im- pact of isolation on model predictions.
10
20
40
Layer
(b) Results on the AGNews dataset
Figure 3: Relative sizes of Swp. Suq. and Sww in dif- ferent layers on SST-2 and AGNews. Results of other datasets can be found in Apendix Initially, Sup oc- cupies significant proportion, but it gradually decays over layers, while Syn becomes the dominant one.
Results and Analysis Figure illustrates no- table influence on the model's s behavior when label words are isolated within the first 5 layers, Yet, this influence becomes inconsequential within the last layers, or when random non-label words are used. This observation underlines the fundamental impor- tance of shallow- information aggregation via label words in ICI. It also emphasizes the superi- ority of label words over non-label words. Further tests with variable numbers of layers reaffirm these findings (Appendix D). Moreover, similar results were obtained when testing ICL with semantically unrelated labels (refer to Appendix F.2).
the attention layer in the model block this flow and examining the model behavior change, we val- idate the existence of the information aggregation process and its contribution to the final prediction.
Experimental Settings We retain the same test sample size 1000 inputs as 2.1. We use the same demonstration for single random seed. To further validate our findings on larger models, we incorporate GPT-J (6B) (Wang and Komatsuzaki, 2021) in experiments, which exceeds GPT2-XL in model size and capacity.
Implementation Details To block the informa- tion flow to label words, isolate label words by manipulating the attention matrix A. Specifi- cally, we set Ap(p, i)(i < p) to 0 in the attention matrix A, of the 1-th layer, where p represents label words and i represents preceding words. Conse- quently, in the /-th layer, label words cannot access
2.3 Deep Layers: Information Extraction
We proceed validate the latter part of our hy- pothesis that the model extracts information from label words to form the final prediction. We de- note the sum of the attention matrices in the I-th

---

layer as A, In deeper layers, we find a strong correlation between the attention distributions on the label words of the target position, represented as (A)(q,P1),... .i),..,, Ar(g, Ac)). and the model's s final prediction, affirming our hypothesis. The experi- mental setup mirrors that discussed in $ 2.2.
0.85
AUCROC
R.
0.80
0.8
0.75
AUGROC
0.70
0.65
0.4
0.35
2.3.1 Experiments
0.50
0.45
0.0
We utilize the AUC-ROC score to quantify the cor- relation between Ar(g; Pp) and model prediction, which we denote as AUCROC, for the I-th layer. We prefer the AUC-ROC metric due two pri- mary reasons: (1) Ai pi) might differ from the probability of the model outputting label by a constant factor. As Kobayashi et al. (2020) points out, attention should be multiplied by the norm of the key vector to yield 'more interpretable atten- tion'. The AUC-ROC metric can implicitly account for these factors, thus allowing correlation more effectively. (2) The proportion different labels output by the model may be un- balanced. Using the AUC-ROC metric can help mitigate this issue, reducing disturbances caused by class imbalance.
Layers
(a) GPT2-XI. (total 48 layers).
0.9
AUCROC
0.8
AUCAOC.
8
0.
0.6-
0.
uncover the
15
20
25
(b) GPTJ (total layers).
Figure 5: AUCROC, and RS of cach layer in GPT mod- els. The result is averaged over SST-2, TREC, AGNews, and Emoc. AUCROC, reaches 0.8 in deep layers, and R; increases mainly in the middle and later layers.
Considering the residual mechanism of trans- formers, we can view each layer's hidden state as the cumulative effect of all prior layer calculations. To quantify the accumulated contribution of the first layers to model prediction, we introduce Ri:
2.4 Discussion of Our Hypothesis
In $ 2.2, we have affirmed that the model's shallow layers assemble information from demonstrations via label words form semantic representations. In $ 2.3, we verify that the aforementioned aggre- gated information on label words then extracted to form the final prediction in the deep layers. Rec- ognizing the crucial function of label words in this process, we have introduced the term "Anchors" to denote them. Given the considerable role these "anchors" fulfill, find it intuitive to design ICL improvements based on them, elaborated in $ 3.
Ri (AUCROC, 0.5)
(5)
(AUCROC: 0.5)
This measure tracks the positive contribution above baseline AUC-ROC threshold of 0.5. The value of R signifies the proportional contribution of the first layers to the model prediction.
2.3.2 Results and Analysis
Figures 5a and 5b delineate correlation metrics for GPT2-XL. and GPT-J, averaged across four datasets. The AUCROC; for deep layers approaches 0.8, il- lustrating a strong correlation between the attention distributions label words of the target position and the model's final prediction. Moreover, shal- low layers show negligible cumulative contribu- tions (R)), with significant increase in middle and deep layers. These results signify the crucial role deep layers for final prediction, validating that the model extracts information from label words in deep layers to form the final prediction.
Applications of Our Anchor-Based Understanding
With insights from the validated hypothesis, we propose strategies to boost ICL's accuracy and in- ference speed. We propose an anchor re- reweeigtingg method in $ 3.1 adjust the demonstrations' contri- butions and improve accuracy. In 3.2, explore context compression technique that reduces origi- nal demonstrations to anchor hidden states speed up ICL inference. Besides, in $ 3.3, we utilize an- chor distances to perform an analysis to understand
'Here we sum up the attention matrices of all attention heads in the ith layer for convenience of analysis.

---

the errors ICL made in real-world scenarios. These approaches corroborate our hypothesis, pointing to potential paths for future ICL enhancements.
as Anchor Re-weighting, It can also be viewed modification of the demonstration contributions since demonstration information has been incorpo- rated into the anchors as suggested by our prior analysis in $ 2.2. Additionally, it can be interpreted as a unique adapter variant, introducing minimal parameters while preserving most of the original model. However, it is specifically designed based anchor hypothesis and requires fewer pa- rameters than traditional adapters.
3.1 Anehor Re-weighting
Based on our analysis in $ 2. we draw parallels between ICL and logistic regression and propose an approach to improve ICL's accuracy by re- weighting label anchors.
3.1.1 Method
$ 2.3 illustrates a strong correlation between the model's output category and the attention distri- bution (A (1) ,..,..., A (q, Pc)) label words Pl; PC of the target position q in deep layers. We can view the attention module as a classifier f.
3.1.2 Experiments
We choose one sample per class as normal demon- strations and choose four extra samples per class to form the auxiliary training set (X Xtain train). The setup follows 2.2, with results averaged over five random seeds. Owing to computational con- straints, we employ GPT2-XL for evaluation, ex- cluding GPT-J. The parameters {+6 are trained using gradient descent. More details can be found in Appendix H.
PY = i|X = x)
p(),p() explaak.. /Vd)
(6)
E1=1 exp(q,k] /vd)
We compare Anchoring Re-weighting with baselines: (1) Vanilla ICL. with the same demon- stration (1-shot per class) (2) Vanilla ICL, where the auxiliary training set of B is included as demon- strations (5-shot per class) for fair comparison.
By setting 9n /dd = & and kpp kpc = B we
deduce:
log PY = ||x 13
(7)
PY C|X =
This approximates logistic regression model where:
3.1.3 Results
As Table 1 shows, the proposed anchor re- weighting significantly enhances ICL performance, particularly on the SST-2 and EmoC datasets. Be- sides, adding more demonstrations for vanilla ICL may not bring a stable accuracy boost due the po- tential noise introduced. discussed in Zhao al. (2021). Different from vanilla ICL which utilizes the extra examples to form a demonstration, we train a re- re-weighting vector / to modulate label an- chor contributions. This shortens the input context and thus brings (almost) extra cost to the infer- ence speed. The consistent improvements of our method suggest that the re-weighting mechanism could be better alternative utilize demonstra- tion examples. Furthermore, it reiterates the crucial role that anchors play in ICL.
PY iX = = 56 37 lug Pry(Y = =pX=x)
(8)
In this equation, 8/ and ( are parameters that can be learned, while x the input feature. Inspired by the similarity between ICL and lo- gistic regression, we ve incorporated a learnable 8) into Eq. (7), which is equivalent to adjusting the attention weights A(g.p)):
A(g. cxp(P))A(g.P4)
(9)
Each 8 is Iearnable parameter, set uniquely for different attention heads and layers. Refer Ap- pendix G for more details.
To train the re-weighting vector f {. we utilize an auxiliary training set (X train Y train). Here, we perform ICL with normal demonstrations and optimize / with respect to the classification loss L on X train, Yirain):
3.2 Anchor-Only Context Compression
We further explore a context compression tech- nique that reduces the full demonstration to anchor hidden states for accelerating ICL inference.
x arg min L X train: Y Irain)- (10)
8
3.2.1 Method
This approuch can be metaphorically described as "re-weightin the anchors," leading us to term
In $ 2.3, we find that the model output heavily re- lies on the label words, which collect information

---

Method
SST-2
TREC
AGNews
EmoC
Average
Vanilla In-Context Learning 1-shot per class
61.28
57.56
73.32
15.44
51.90
Vanilla In-Context Learning 5-shot per class
6.1.55
60.10
52.52
9.80
16.87
Anchor Re-weighting (1-shot per class)
90.07
60.92
81.94
41.64
68.64
Table 1: The effect after adding parameter Sj- For AGNews, due to the length limit, we only use three demonstrations per class. Our Anchor beightgg method achieves best performance overall tasks.
from the demonstrations. Given the nuto--regeesivee nature of GPT-like models, where hidden states of tokens depend solely on preceding ones, label words' information aggregation process inde- pendent of subsequent words. This allows for the calculation and caching of the label word hidden
Method
Label Loyalty
Word I Loyalty
Acc.
ICL (GPT2-XL)
100.00
100.00
51.90
Texfactor
51.05
36.65
38.77
Hiddenenndo
48.96
5.59
39.96
Hidden
57.52
4.49
41.72
Hiddenggehher
79.47
62.17
45.04
states = {( 1,N (hj is the 1-th layer's hidden state of the i-th label word in the demon- stration). By concatenating h/....,hi hf at the front in each layer during inference, instead of using the full demonstration, we speed up inference.
ICL (GPT-J)
100.00
100.00
56.82
TeXlashor
53.45
43.85
40.83
Hiddennansom
49.03
2.16
31.51
Hiddeenennom-itn
71.10
11.36
52.34
Hicden,,rchur
89.06
75.04
55.59
In our preliminary experiments, concatenating hidden states label words alone was inadequate for completing the ICL task. $ This might be due to the critical role of formatting information in helping the model to determine the output space at the target position, 6 as highlighted in Min et al. (2022h). As solution, states of both the formatting and the label words, method we've termed Hiddenanchor-
Table 2: Results different compression methods on GPT22XL and GPT-J (averaged over SST-2, TREC, AG- News, and EmoC). Acc. denotes accuracy. The best results are shown in bold. Our method achieves best compression performance.
amalgamate the hidden
words, rather than the mere text of label words. If we find that Hiddenanchor surpasses Textanchor in performance, it solidifies the notion that the ag- gregated information within label words carries significant importance. The Hiddenrandom method introduced to illustrate that anchor hidden states encapsulate most of the demonstration information among all hidden states.
3.2.2 Experiments
We follow the same experimental settings as $ 2.2. We compare our Hiddenanehor input compression method with two equally efficient baselines.
aextor: This method concatenates the format- ting and label text with the input, as opposed to concatenating the hidden states at each layer. Hidden This approach concatenates the hid- den states of formatting and randomly selected non- label words (equal in number to Hiddenanchor). Hidden,zadommtto To establish a stronger baseline, we randomly select 20 sets of non-label words in Hiddenrandom and report the one with the highest label loyalty.
We assess all compression methods using the label loyalty and word loyalty introduced in 2.2. in addition to classification accuracy.
3.2.3 Results
We can see from Table 2 that the proposed com- pression method Hiddenanchor achieves the best results among all three compression methods on all metrics and for both models. For example, with the GPT-J model, the compression method with anchor states only leads to 1.5 accuracy drop compared to the uncompressed situation, indicating that the compression introduces negligible information loss. Further, we estimate the efficiency improvements over the original ICL. As shown in Table 3, the speed-up ratio ranges from 1.1x to 2.9x, as the efficiency gain is influenced by the length of the demonstrations. We refer readers to Appendix for
The Textanchor method is included to demon- strate that the effectiveness of Hiddenanchor is at- tributed to the aggregation of information label
'Omitting formatting significantly reduces accuracy, the model will favor common tokens like "the" over label words, indicating confusion about expected output type.
'Here, "formatting" refers to elements like "Review:" and "Sentiment:" in Figure 2.

---

Model
SST-2
TREC
AGNews
EmoC
1.0
Abbreviat on
0.49
0.45
0.9
0.83
GPT2-XL
1.1x
1.5x
2.5x
14xx
0.9
GPTJ
1.5x
2.2x
2.9x
1.9x
0.8
Description
0.45
0.34
0.68
0.75
0.72
0.7
Table Acceleration ratios of the Hncher method.
Person
0.9
0.75
0.
0.75
0.55
0.72
more elaborated analysis of the speed-up ratios. Besides, we observe that the acceleration effect is more pronounced in the GPT-J model compared to GPT2-XL, demonstrating its great potential to apply to larger language models.
Number
0.83
0.92
0.79
0.95
0.79
0.4
Entty
Person
Locatonn
Numbbr
Aabrewatt
Descrppoo
(a) Confusion matrix Confusion reed
3.3 Anchor Distances for Error Diagnosis
1.0
Lastly, we perform an error analysis for ICL by examining the distances between the key vectors in the attention module that correspond to the label
Abbresint on
0.84
0.76
0.9
Dascription
0.76
0.58
0.98
0.00
0.97
words.
0.8
Person
0.95
0..90
3.3.1 Method
0.39
Our previous analysis in 2.3 shows strong cor- relation between the model output and A(g, Pt). which is determined by qykj as per Eq. 7. Should the key vectors for label words pt and pk be similar, A(q, P.) and A(q, Pt)) will also likely be similar, leading to potential label confusion. Fur- thermore, considering the distribution of query vec- tors 4y. we employ PCA-like method to extract the components of the key vectors along the direc- tions with significant variations in qy. denoted as k (sce Appendix J for details). We anticipate that the distances between these ks can correspond to the category confusion of the model, thus revealing one possible origin of ICL errors. Here, we normal- ize the distances to scale of 0-1, with 0 indicating the highest degree of category confusion:
Number
0.87
0.97
0.99
-0.5
Astreviatoon
Enthy
Descritton
Person
Location
Number
(b) Confusion matrix of Confusionay.
Figure 6: Predicted and real confusion matrix on TREC. We set undefined diagonals I for better visualization. The heatmaps display similarity in confusing category pairs, particularly in lighter-colored blocks.
3.3.3 Results
Figure shows that the proposed approximation metric, Confusion pred can identify the most confus- ing case (Description-Entigy and performs reason- ably well for highly confusing categories (Entity- Abbreviation, Description Description-Abbreiiiie Abbreviation). This high correlation indicates that ICL makes errors in categories with similar label anchors. Overall, this result demonstrates that our anchor-based anal- ysis framework could serve as an interpretation tool better understanding ICL's errors.
kkp.
kpy
Confusion pred
(11)
max,41 |kpp_
kp.
3.3.2 Experiments
We utilize the GPT2-XL model and TREC dataset, as the model displays varying confusion levels be- tween categories on this dataset. We use 500 samples of the TREC test set and use demonstra- tion per class for convenience of analysis.
Related Work
The existing literature on in-context learning analy- sis can be broadly divided into two streams, each focusing different aspects. The first stream plores the influencing factors of ICL based on input perturbation, such the order (Min al., 2022b), the formatting (Yoo et al., 2022; Wei al., 2022), and the selection of the demonstration (Liu et al., 2022). Designing proper demonstration construc-
We calculate the actual model confusion score, Confusion,j, between category and category i us- ing the AUC-ROC metric (detailed in Appendix K), We then compare the predicted confusion score, pred Confusion and the actual confusion score, Confusion,y, via heatmaps.

---

tion strategies (Ye al., 2023; Li al., 2023a) and calibration techniques (Zhao al., 2021; Min et al., 2022a) could bring clear boosts to the ICL performance. The second stream investigates the inner working mechanism of ICL through different conceptual lenses, such as making an analogy of ICL to gradient descent (von Oswald et al., 2022; Dai et al., 2022) and viewing the process of ICL as Bayesian inference (Xie al., 2022).
Acknowledgement
We thank all reviewers for their thoughtful and in- sightful suggestions. This work is supported in part by Tencent Research Grant and National Natural Science Foundation of China (No. 62176002). Xu Sun is the corresponding author.
References
In this paper, we provide novel perspective by examining the information flow in language mod- els to gain an understanding of ICL. Our approach offers new insights and demonstrates the potential for leveraging this understanding to improve the ef- fectiveness, efficiency, and interpretability of ICL.
Ekin Akyürek, Dale Schuurmans. Jacob Andreas, Tengyu Ma, and Denny Zhou. 2022. What learn- ing algorithm in-context learning? investigations with linear models. ArXiv preprint, abs/2211.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners In Ad- vances in Neural Information Processing Systems 33: Annual Conference Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
5 Conclusion
this paper, we propose a hypothesis that label words serve as anchors in in-context learning for aggregating and distributing the task-relevant infor- mation flow. Experimental results with attention manipulation and analysis of predictions correla- tion consolidate the hypothesis holds well in GPT2- XL and GPT-J models. Inspired by the new under- standing perspective, we propose three practical applications. First, an anchor re-weighting method is proposed to improve ICL accuracy. Second, we explore a demonstration compression technique to accelerate ICL inference. Lastly, we showcase an analysis framework to diagnose ICL errors on real-world dataset. These promising applications again verify the hypothesis and open up new direc- tions for future investigations ICL.
Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi, and Puneet Agrawal. 2019. SemEval-2019 task 3: EmoContext contextual emotion detection in text. In Proceedings of the 13th International Workshop Semantic Evaluation, pages 39-48, Minneapo- lis, Minnesota, USA. Association for Computational Linguistics.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. ArXiv preprint, abs/2212.
Limitations
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2023. A survey for in-context learning. ArXiv preprint, abs/2301.00233
Our study, while providing valuable insights into in-context learning (ICL), has several limitations. Firstly, our research scope was limited to classi- fication tasks and did not delve into the realm of generative tasks. Additionally, our hypothesis was only examined within conventional ICL paradigms, leaving other ICL paradigms such as the chain of thought prompting (CoT) (Wei et al., 2022) unex- plored. Secondly, due to hardware constraints, mainly investigated models up scale of 6 bil- lion parameters. Further research that replicates our study using larger-scale models would be bene- ficial in corroborating our findings and refining the hypotheses set forth in our investigation.
Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin- Yew Lin, and Deepak Ravichandran. 2001. Toward semantics-based answer pinpointing. In Proceedings of the First International Conference on Human Lan- guage Technology Research.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight:

---

Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057-7075, Online. Association for Computa- tional Linguistics.
semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023a. Unified demonstration retriever for in- context learning, ArXiv preprint, abs/2305.0433 04320..
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971.
Xin Li and Dan Roth. 2002. Learning question clas- sifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
Yingcong Li. Muhammed Emrullah Ildiz, Dimitris Pa- pailiopoulos, and Samet Oymak, 2023b. Transform- ers as algorithms: Generalization and stability in in-context learning.
Johannes von Oswald, Eyvind Niklasson, E. Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. 2022. Trans- formers learn in-context by gradient descent, ArXiv preprint, abs/2212.07677.
Jiachang Liu, Dinghan Shen. Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-114 114, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A Billion Parameter Autoregressive Lan- guage Model. https: wwcchhuu..../kiiiiee com/kingoflolz/ aush-transformert
Jason Wei, Xuezhi Wang, Dale Schuurmans. Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. ArXiv preprint, abs/2201. 1903.
Paul Michel, Omer Levy, and Graham Neubig, 2019. Are sixteen heads really better than one? In Ad- vances in Neural Information Processing Systems 32:
Annual Conference on Neural Information Process- ing Systems 2019, NeurlPS 2019. December 8-14, 2019, Vancouver, BC, Canada, pages 14014-14024.
Jerry W. Wei, Jason Wei, Yi Tay, Dustin Tran, Al- bert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023. Larger language models do in-context learning dif- ferently. ArXiv, abs/2303.038.
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022a. Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), pages 5316-5330, Dublin, Ireland. As- sociation for Computational Linguistics.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. An explanation of in-context learning implicit bayesian inference. In The Tenth International Conference on Learning Representa- rions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe. Mike Lewis, Hannanch Hajishirzi, and Luke Zettle- moyer. 2022b. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference Empirical Methods in Natural anguage Processing, pages 1048-1 I 1064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. ArXiv preprint, abs/2302.0566 05.698.8....
Kang Min Yoo, Junycob Kim, Hyuhng Joon Kim, Hyun- soo Cho, Hwiyeol Jo, Sang- Woo Lee, Sang-goo Lee, and Taeuk Kim. 2022. Ground- truth labels matter: A deeper look into input-label demonstrations. In Pro- ceedings of the 2022 Conference Empirical Meth- ods Natural Language Processing, pages 2422- 2437, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Suiskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 18):9.
Karen Simonyan, Andrea Vedaldi, and Andrew Zis- serman. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312. 312.6604..
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in Neural Information Pro- cessing Systems 28: Annual Conference on Neural In- formation Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. 2013. Recursive deep models for

---

Table 4: Demonstration templates and label words. Here <SI> represents the demonstration, <S> represents the input to be predicted, and <L> represents the label word corresponding the demonstration. To save space, we only show demonstration for each task.
Task
Template
Label Words
SST-2
Review: <$1>
Positive, Negative
Sentiment: <>>
Review: <S>
Sentiment:
TREC
Question: <SI>
Abbreviation, Entity
Answer Type: <L>
Description, Person
(a) Results on TREC dataset
Question: <S>
Location, Number
Answer Type:
AGNews
Article: <S1>
World, Sports
10-
Answer: <l.>
Business, Technology
Article: <S>
Spe
Answer:
EmoC
Dialogue: <S1>
Others, Happy Sad, Angry
Emotion: <L>
Dialogue: <S>
Emotion:
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Vir- tual Event, volume 139 of Proceedings of Machine Learning Research, pages 12697-12706. PMLR.
Lxyer
(h) Results on the EmoC dataset
Figure 7: Relative size Sups Spy. and Swn on TREC and EmoC, which is similar to that on SST-2 and AG- News.
Appendix
prominent, while Spo (the information flow from label words to targeted positions) is less signifi- cant. However, in deeper layers, Spg dominates. Importantly, Sup and Spy generally exceed Sww. indicating that interactions involving label words predominant.
Experimental Settings
For models, we use GPT2-XL (L.SB) (Radford al., 2019) and GPT-J (6B) (Wang and Komat- suzaki, 2021) in this paper.
For datasets, we use sentiment analysis task, Stanford Sentiment Treebank Binary (SST- 2) (Socher et al., 2013), question type classifi- cation task, Text REtrieval Conference Question Classification (TREC) (Li and Roth, 2002; Hovy al., 2001), a topic classification task, AG's news topic classification dataset (AGNews) (Zhang et al., 2015), and an emotion classification task, Emo- Context (EmoC) (Chatterjee et al., 2019). The ICL templates of these tasks are shown in Table 4.
C Reason for Using Word Loyalty Besides Label Loyalty
Label loyalty alone may not capture changes in the probability distribution of non-label words the relative ratio of the probability of the label words within the entire vocabulary. Word loyalty helps address this limitation, which shown in Table
D Isolating Different Numbers of Layers
B Results of Sup Spas and Sao on TREC and EmoC
We study the impact of the numbers of isolated layers, as shown in Figures 8a and 8b. It can be found that isolating shallow layers cause signifi- cant impact, isolating deep layers has a negligible impact on the model, even when the number isolation layers increases. This further illustrates
Figure 7 illustrates the relative sizes of Sup. Spq- and Syw on TREC and EmoC, mirroring results on SST-2 and AGNews. In shallow layers, Sup (the information flow from the text part to label words)

---

Isolation Layer
Output Label
V/ (sorted by probability)
First layers
World
"n"" ; Thh", " Google", Google","<lende"e"" "<lendoftcxi>>" A" "Wold"", "Tehhnlogy"", Politics", Israell" Human"
No isolation
World
Table 5: Results on a test sample with the label "World" from AGNews.
trices of all attention heads at the I layer is A We postulate that there's a strong correlation be- tween the attention distributions on the label words of the target position (Ar(q:
1.0
Ar(a, Pc)) and the model's final prediction. We use the AUC- ROC score to quantify this correlation. We re- gard (1), (4 Pc)) as classifier's pre- diction for the model output label (that is, A(g. Pt) is equivalent to the probability of model outputting label i), and compute the AUC-ROC value of this prediction relative to the actual model output. We denote this AUCROC,. For the case with more demonstrations (Appendix F.I), we simply sum up all A,(g, P) of the same class.
Loyalty
0.6
0.4
Label Ry (Fisc
Loyalty (Last)
Word Loyelly (Fiitt)
Loyalty
0.0
Isolation Layer Num
Effect different numbers isolated layers on GPT2-
XI.
F Additional Experimental Results
Loyaly
0.6
Label Loyalty (Last)
Label Loyalty (Fistt)
F1 Results with More Demonstrations
Loyalty (Fist)
We implement our experimental analysis utilizing two demonstrations per class, resulting in a total of 4. 12, 8, and demonstrations respectively for SST-2, TREC, AGNews, and EmoC. Our findings, as depicted in Figure 9, Figure 10, and Figure 11, exhibit high degree of similarity to the results obtained from experiments that employ one demon- stration per class.
Word Loyalty (Lazt)
0.4
Isolation Layer Num
(b) Effect of different numbers of isolated layers on GPT-I
Figure The chart demonstrates variations in label loyalty and word loyalty, dependent whether label or non-label words are isolated in various layers. 'First' refers to the first several layers, while 'Last' the last ones. Deep-colored lines represent label word isolation, whereas light colors denote non-label words. Remark- ably, isolating label words in the shallow layers signif- icantly influences the outcome, regardless of whether this compared isolation deep layers or to non- label word isolation.
F.2 Results for In-Context Learning with labels
semantically-unrelee
The applicability of our analytical conclusions to ICL variants, such as the semantically unrelated la- bel ICL (Wei et al., 2023), is an intriguing subject. Given that both GPT2-XL and GPT-J-6B perform at levels akin random guessing in this ICL set- ting, we chose LLaMA-33B (Touvron et al., 2023) and SST-2 for our experiment. We substituted la- bels with A''' B', and adhered to a similar experi- mental setup as in sections 2.2 and $ 2.3. How- ever, we applied eight shots per class to facilitate the model in achieving accuracy of 83.0% on SST-2. The outcomes align with those derived in 2.2 and 2.3. Figure 12 shows the more pro- nounced impact of isolating labels in the shallow layers compared to their isolation the deep layers or the isolation non-label tokens. Figure 13 con-
the important role of information aggregation via label words the shallow layers.
Details for the Calculation of E AUCROC,
Suppose the positions of the label words in the input x are PI....PC (without loss of generality, we suppose p: corresponds to the ith class), the targeted position is q. the sum of the attention ma-

---

1.0
Loyaty
0.6
0.4
Label (First)
Label Loyalty (Lasi)
Word Loyalty (Rrst)
Loyalt
0.0
Isolation Layer Num
Dayer
(a) Effect of different numbers of isolated layers on GPT2-
(a) Results the SST-2 datasct
2.0
0.5
0.5
Loyalty
0.6
6.4
0.4
Label Loyalty (first)
622
Label Loyalty
Word Loyalty (First)
Wond Loyally (Last)
0.0
20
20
Jayer
Isolation Layer
(b) Results on the TREC dataset
(b) Effect of different numbers of isolated layers GPT-J
Figure Variations label loyalty and word loyalty when more demonstrations are employed.
firmed that the model leverages information from anchors in the deeper layers to perform classifica- tion.
G Implementation of Anchor Re-weighting
Results on the AGNews datasct
In order to implement anchor re-weighting, spe- cific adjustments are made the model's compu- tational process. After calculating the attention matrix A of the hth head in the Ith layer, we mul- tiply each A P:) by exp(3),n) before proceed- ing with further computations. This means that for each attention head, we introduce the following modifications:
2.2
0.8
0.6
64
Attention! (Q, (,K A/V.
20
Jayor
QKT
4 = softmax
(d) Results the EmoC datuset
vd
Figure Relative sizes Swp, Spm. and Swe when more demonstrations are employed.
A (k.
(o (k)
q. = p
otherwise
(12)

---

AUCROC
1.0
1.0
AUCROC
0.8
0.6
ALCRCC
0.7
1.6
0.7
3.4
0.6
0.4
0.2
0.3
0.0
Layers
(a) GPT2-XL (total 48 layers).
Figure 13: AUCROC, and Ri of each layer of LLaMA- 33B on SST-2. Still, deep layers display higher rele- vance model prediction, reinforcing the idea that the model extracts information from deep-layer anchors for classification.
0.8
AUCAGC
3.6
ter vector /3 is carried out using gradient descent, specifically with the Adam optimizer (Kingma and Ba, 2015). The learning rate is set at 0.01, with B1 0.9 and f2 = 0.999. Due to memory con- straints, use batch size of 1. This optimization process is repeated for 10 epochs. Owing to limi- tations in computational resources, restrict evaluation the GPT2-XL model and exclude the GPT-J model from our assessment.
3.0
(b) GPT-J (total layers).
Figure II: AUCROC, and Ry of each layer in GPT models when more demonstrations are employed.
Io Ioclation
Label Wonds dast Random (Raak)
Rendom (Lavt]
I
The Factor of T-demo and L.x
Label Words (Tii))
100
SST-2
TREC
AGNews
EmoC
GPT2-XL GPT-J
1.1x
1.5x
2.5x
1.4x
Loyalt
60-
1.5x
2.2x
2.9x
1.9x
Lieao
61
151
53
Lx
19
7
37
12
Table 6: Acceleration ratios, Laemo and Lx--
Loyalty (LLaMA-30D)
ILLaMAN- 300)
From Table 6, we observe correlation between the acceleration ratios and the ratio of the total demonstration length (Ldcmo) to the length of the text predicted (Ix)). It suggests that greater ratio of total length predicted text length may yield higher acceleration ratio.
Figure 12: The impact of isolating label words versus randomly isolating non-label words within the first last 5 layers. Isolating label words within the first 5 layers exerts more pronounced effect, highlighting the importance of shallow-layer information aggregation via label words.
In addition, the table illustrates that datasets with longer demonstration lengths tend to exhibit higher acceleration ratios. For instance, the AGNews dataset, which has the longest Ldtemn. presents the highest acceleration ratio among the datasets ana- lyzed. These findings could indicate an increased efficiency of the Hiddenanchor method in contexts involving longer demonstration lengths.
H Training Settings of Anchor Re-weighting
For each random seed, we fix the demonstration and sample 1000 test samples from the test datasets as described $ 2.2. The optimization of parame-

---

J Calculation of k
the Receiver Operating Characteristic Curve (AUC- ROC) value of this classifier f. we get the degrec of confusion between category and k, termed as Confusion.j-
For the sampled sequence #1, "1,..,;7 to be pre- dicted, we denote the query vectors of the target po- sitions as 91, ...,, qr- We then compute the matrix Q = (q) - q q q) by subtracting the mean vector, q. from each query vector. Subsequently, we determine the M directions, Vi...., MM. that correspond to the M largest variation directions for the centralized query vectors Q .1..... qr- The 7t direction, V;, is chosen to maximize the variance of the projection of the centralized query vectors onto it, while also being orthogonal to the previously chosen directions, VI,-.., Vi-1- This process can be formalized as follows:
The computed Confusionij is a value that never exceeds 1. The closer Confusionij upproximates I, the less pronounced the confusion, and vice versa.
We use the above metric instead of directly ana- lyzing the output labels of the model because pre- vious work has indicated the issue of insufficient output probability calibration in ICL (Zhao et al., 2021), which is greatly affected by factors such as sample ordering and model preferences for specific label words. By leveraging our defined degree of confusion, Confusionay, we can implicitly alleviate the disturbances arising from insufficient probabil- ity calibration the output labels. This allows for a more accurate representation of the model's degree of confusion for different categories, miti- gating the impact of randomness.
arg max Var {T]@..
|v|==
{ [
arg max Var I
(13)
L Reproducibility
Var { @T]...
VM
arg max | V Mar-1
In the supplementary material, we have provided codes that allow for the faithful replication of our experiments and subsequent result analysis. To ensure consistency and reproducibility across dif- ferent devices, we have fixed the five random seeds to the values of 42, 43, 44, 45, and 46. We invite readers to delve into the code for additional imple- mentation details that may arouse their interest.
We define or as the square root of the variance of the projection of Q onto the ith direction, i.e.,
Var V. V
To derive features ks, we project the key vector k onto the directions Vi...,, VM and scale the pro- jections by the corresponding standard deviations I..... TM. Each feature, k,, thus calculated as ovv k.
We further examine the influence of M on the prediction confusion matrix, Confusioniypred, as depicted in Figure 14. Given the similarity in outcomes for various M, settle value of M 10 for computation of po
K Calculation of Confusion.
To gauge the true degree of confusion between categories and for a given model, we suggest utilizing the Confusion; metric:
First, we procure all test samples bearing true labels or k. We then obtain the probabilities pf and pt yielded by the model for categories and k. respectively, on these samples. These probabilities are normalized a classifier f that delivers the probabilities y and p/ for the categories and & respectively, on the test samples r,. By calculating the Area Under
total of 1, Essentially, we derive

---

Aboreviation
0.44
0:31
0.9
0.61
Abbrevistion
1
0.49
0.45
0.9
0.83
0.9
Faiity
Farnyy
0.8
Description
Description
0.7
0.74
0.6
-.5
Location
0.9
0.74
0.74
0.74
Location
0.89
0.75
0.73
0.72
-0.4
Number
0.81
0.91
6.69
0.51
0.74
1
Number
0.83
0.92
0.72
0.55
0.72
0.4
Abbreviation
Entry
Descrittion
Locatinn
Number
Aboreviation
Entry
Desociption
(a) M
(b)
10
Aboreviation
Abbreviation
0.85
0.9
0.9
Entiy
Entiyy
0.51
0.8
0.8
Description
0.49
0.35
0.68
0.76
0.72
Description
0.51
0.37
0.68
0.76
0.72
0.7
Person
0.89
0.73
0.60
1
0.72
0.55
0.5
Person
0.89
0.73
0.68
0.72
0.55
Location
0.89
/b/b
0.72
02/2
Location
0.89
0. 0.76
/2/
01
0.5
0.72
-04
0..4
Apbreviation
Entry
Descritioon
Location
Number
Abbreviction
Entry
Dosciittion
Location
(c) M
20
(d)
50
1.
1.0
Aboreviation
0.85
Abareviazion
0.9
0.9
Entity
0.51
0.38
0.73
0.88
0.9
Entiyy
0.52
0.38
0.73
0.88
0.9
0.8
Description
0.52
0.38
0.88
0.75
0.77
Destription
0.52
0.38
68
0.75
0.
0.7
0.7
Person
0.89
0.73
68
0.72
0.55
Person
0.99
0.73
8.88
0.72
0.55
-0.
0.5
Number
0.85
0.71
Numbor
0:85
0.4
0.1
Abbrevisiio
Entry
Locadon
Number
Abbreviation
Entiy
Doser poon
Person
(c) M
100
(f) M 200
Figure 14: Predicted confusion matrices under M 5, 10, 20, 50, 100, 200.


Reviews:
[{"review_id": "WvGoiVJ0gb", "reviewer": "Reviewer_5cZV", "summary": "", "questions": "A. I am curious how the information flow from the test-time text to the target position across layers looks like? This would indicate the information contribution of actual text w.r.t. demonstrations.\nB. Similarly, how would the information flow into test-time text tokens look like?\nC. Anchor reweighting looks very much similar to adapter in principle. Can you expand on the similarities?\nD. In lines 410-416, I didn't quite understand why formatting is important? Please explain it\nE. I didn't understand Text_{anchor} in 3.2.2 experiment.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **For questions:**\n\n**Question(A):** How does the information flow from the test-time text to the target position across layers look like? This would indicate the information contribution of actual text compared to demonstrations.\n\n**Answer(A):** \nTo picture the flow, we resorted to saliency scores. We utilized two metrics to assess the nature of the flow:\n\n1. **Coefficient of Variation (std/mean)**: We calculate the ratio std/mean of saliency scores corresponding to the test-time text tokens within a specific layer. This illuminates the variance in the influence of different tokens.\n2. **Relative influence of layers**: We compare the saliency scores of different layers. This indicates the importance of this flow in different layers.\n\nOur results are listed below:\n\n- For the **Coefficient of Variation**, as layers go deeper, the disparity in token influence tends to widen. Generally speaking, the initial layers (approximately the first 15 layers among the total 48 layers in GPT2-XL) show a modest coefficient of variation (around 1). In contrast, this value increases in subsequent layers (to about 1.5). However, the results may differ across tasks. For instance, AGNews exhibits a consistently high coefficient of variation (higher than other datasets by about 0.5).\n- For **Relative influence of layers**,  the influence is small for the initial layers, then peaks around the 20th-30th layer, and decreases in the last several layers.\n\nAdditionally, the relative contribution of the test-time text v.s. demonstrations were quantified via saliency. They are 0.65:1 (SST2), 3.46:1 (TREC), 2.39:1 (AGNews) and 2.09:1 (EmoC). So the relative contribution may be task-dependent.\n\n\n**Question(B):** How does the information flow into test-time text tokens look like?\n\n**Answer(B):** \nWe use the metrics outlined in Answer(A) to picture the flow. The result is:\n\n1. The **Coefficient of Variation** for the flow is generally above 2, higher than that in Question(A).\n2. As for the **Relative influence of layers**, the initial 20 layers seem to be dominant, and there's a pronounced decline in influence in subsequent layers.\n\n**For (A) and (B):** In brief, the dynamics and characteristics of the information flow about test-time text are intricate. We look forward to delving deeper into this realm in future research.\n\n**Question(C):** Anchor re-weighting's similarity to adapters\n\n**Answer(C):** Actually, anchor reweighting can be seen as a special type of adapter, since it introduces a small number of parameters and keeps most parts of the model as original, but it is designed based on our anchor hypothesis and owns fewer parameters than normal adapters. We will include a discussion in the related work section in the revised version.\n\n**Question(D):** Why formatting is important as discussed in lines 410-416\n\n**Answer(D):** As emphasized in Rethinking the role of demonstrations: What makes in-context learning work (Min et al., 2022b), formatting plays an important role in ICL. **Formatting ensures that the model is oriented correctly to perform ICL** (or the subsequent compression). In experiments, when formatting is omitted, the accuracy significantly drops, and the most likely tokens predicted by the language model tend to be frequently occurring tokens like ‘the’, rather than the label words. This phenomenon indicates that the model is confused when formatting is omitted and does not know what kind of output is expected.\n\n**Question(E):** Explain the reason for introducing Text_{anchor} in 3.2.2 experiment.\n\n**Answer(E):** The primary aim behind introducing Text_{anchor} is to show that Hidden_{anchor} works due to the aggregated information in label words, rather than the pure text of label words. When we observed that Hidden_{anchor} outperforms Text_{anchor}, it reinforces the idea that the aggregated information in label words is important.\n\n**About the term ‘Deep':** Thanks for pointing out this. We will clarify the meaning of 'deep' and mention a number for what \"deep\" indicates in the revision.\n\n**For typos and presentation suggestions:** Thanks for your kind suggestions. We will revise our paper based on your suggestions 1, 2, and 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Great analysis\n- Neatly presented\n- The further enhancements of ICL proposed that are inspired by the analysis would have broader impact.", "reasons_to_reject": "I must admit that I liked the whole paper and I am being very stringent while mentioning the following points. \n\n1. In figure 5, when we look into the rate of increase in cumulative function, it was the highest after layer 10 for GPT J and layer 20 for GPT2-XL. From this, we can conclude that, not just the deepest layers such as (40-50) contribute, even the middle ones such as layer 20-25 also contribute well. This shows that anything after the initial layers start extracting information from label words. Maybe this would require a partial rewording of the usage \"deep\" layers in the hypothesis-2 or mentioning a number for what \"deep\" indicates should be sufficient.", "questions_to_authors": "A. I am curious how the information flow from the test-time text to the target position across layers looks like? This would indicate the information contribution of actual text w.r.t. demonstrations.\nB. Similarly, how would the information flow into test-time text tokens look like?\nC. Anchor reweighting looks very much similar to adapter in principle. Can you expand on the similarities?\nD. In lines 410-416, I didn't quite understand why formatting is important? Please explain it\nE. I didn't understand Text_{anchor} in 3.2.2 experiment.", "paper_topic_and_main_contributions": "This paper analyses in-context learning and reveal that the label words in the demonstration examples function as anchors where semantic information is aggregated into the label word representations during initial shallow layers and the information from label words serve as reference for LLM's predictions in deep layers. This is well supported by experiments. The enhancements to ICL proposed are great.", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. \"Demonstration tokens in the demonstrations\" is confusing in Lines 134-135. Better to use some other phrase.\n2. No need of repeating the hypotheses again in lines 96-100.\n3. I recommend moving Figure 2 to the pages-1 or 2 if possible. Very informative."}}, {"review_id": "iH4qniiQ67", "reviewer": "Reviewer_XFQJ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1:** Analysis of different ICL formats on the final prediction like random labels, reversed labels (e.g.,True->False, False->True), and label agencies (replace labels with meaningless characters) may be helpful.\n\n**Answer 1:**\n\nIn experiments, we find that GPT2-XL and GPT-J-6B perform similarly to random guessing in these different ICL formats, so we do not analyze them. Even for llama-30b, we find that only label-agency ICL works. We then conducted label-agency ICL with llama-30b on SST-2 (in this case, the model can achieve an accuracy of 0.83). **The results are similar to those in Sections 2.2 and 2.3 in our paper, thereby reinforcing our initial conclusions.**\n\nThe detailed results are listed below: \n\n1. The graph of AUCROC_{l} and R_l is similar to Figure 5 in the paper. For all 60 layers of llama-30b, AUCROC_{l} is about 0.5 for layers 0-20, about 0.9 for layers 20-50, and fluctuates for layers 50-60 as in Figure 5. \n2. The results for isolating label words in the first and last layers are similar to Section 2.2. Isolating labels in the first several layers has a greater impact than isolating in the last several layers or isolating non-labels. There is only one minor difference: label loyalty of isolating labels in the first several layers is slightly higher than that of isolating non-labels in the first several layers. \n\n   |               | Word Loyalty | Label Loyalty |\n   | ---------------------------- | ------------ | ------------- |\n   | Isolating labels (first)   | 40.8     | 41.7     |\n   | Isolating labels (last)   | 100     | 99.0     |\n   | Isolating non-labels (first) | 60.0     | 39.3     |\n   | Isolating non-labels (last) | 100     | 99.3    |\n\nGiven the consistency of these results with our earlier findings, we can conclude that our observations and insights remain valid. We hope this detailed response addresses your concerns.\n\n**Question 2:** I note that anchor re-weighting uses an auxiliary training set. So can you add fine-tuning the model on the auxiliary training set as a baseline to justify the superiority of anchor re-weighting?\n\n**Answer 2:** Here, our primary intention behind proposing anchor-reweighting was to validate our anchor hypothesis and **show the significance of anchors in ICL**. We do not aim to claim it as a new state-of-the-art technique in a much border range. We have tested finetuning, and the result is 83.2 (SST-2), 67.6 (TREC), 81.2 (AGNews), and 55.7 (EmoC), lower than anchor-reweighting in SST-2 and AGNews, but higher in TREC, EmoC, and the average. However, it is worth noting that finetuning trains all parameters of the model. In contrast, the number of parameters trained in our anchor re-weighting is **less than 0.002% of the total model parameters**. Our anchor re-weighting method is more parameter-efficient while maintaining competitive performance.\n\n**Question 3:** In anchor-only context compression, for hidden_random, can you develop a stronger baseline by selecting non-label words with significant influence?\n\n**Answer 3:** To decide which combination of non-label words from demonstrations has the most significant influence, a direct way is to check the loyalty metric of all possible combinations of non-label words, but it is too time-consuming and thus computationally intractable. As a practical compromise, we devised a surrogate approach. We randomly generated 20 combinations of non-label words and subsequently selected the one exhibiting the highest label loyalty among them. The results for this approach (denoted as **Hidden_{top-20}**) are shown below (averaged over SST-2, AGNews, TREC, EmoC). **This approach's performance is better than Hidden_{random} while still worse than Hidden_{anchor}, which further validates our conclusion:** \n\n| GPT2-XL     | Word Loyalty | Label loyalty | Accuracy |\n| --------------- | ------------ | ------------- | -------- |\n| ICL (No compression) | 100     | 100      | 51.90  |\n| Hidden_{anchor} | 62.17    | 79.47     | 45.04  |\n| Hidden_{random} | 5.59     | 48.96     | 39.96  |\n| Hidden_{top-20} | 4.49     | 57.52     | 41.72  |\n\n| GPT-J-6B    | Word Loyalty | Label loyalty | Accuracy |\n| --------------- | ------------ | ------------- | -------- |\n| ICL (No compression) | 100     | 100      | 56.82  |\n| Hidden_{anchor} | 75.04    | 89.06     | 55.59  |\n| Hidden_{random} | 2.26     | 49.03     | 31.51  |\n| Hidden_{top-20} | 11.36    | 71.10     | 52.34  |\n\n**Question 4:** I am confused about the advantages of using anchor distances for error diagnosis. There might be other tools for error diagnosis.\n\n**Answer 4:**  Here, our primary intention behind introducing the 'Anchor Distances for Error Diagnosis' approach was to **investigate and illustrate the potential correlation between anchor distances and model misclassification.** This method offers a novel perspective to interpret model misclassification through the lens of anchor distances. For now, we use it mainly as **an interpretation method**. We will try to explore the possibility of developing it into a mature tool for error diagnosis in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Clear analysis of the information flow with label words as anchors to explain the ICL mechanism.\n2. Proposed anchor re-weighting is simple and effective to make significant improvements over vanilla ICL.\n3. Proposed anchor-only context compression has faster inference speed than the original ICL while remain comparable performance.", "reasons_to_reject": "1. More analysis should be done since recent studies explore the influence of different ICL formats on the final prediction, such as random labels, reversed labels (e.g.,True->False, False->True), label agencies (replace labels with meaningless characters) and so on. I think the label words anchors analysis on these setups will make deeper insights and more robust conclusions. The current analysis is clear but seems to be trivial.\n2. I note that anchor re-weighting uses auxiliary training set, which seems to be a unfair comparison with the vanilla ICL, since one core advantage of ICL is no extra fine-tuning when transferred to new tasks. For a fair comparison, I think fine-tuning the model on the auxiliary training set is a necessary baseline.\n3. In anchor-only context compression, for hidden_random, I think a more persuasive choice is to select non-label words that show significant influence on label words in shallow layers. Since previous analysis has demonstrated the information aggregation. The current random non-label words selections also seems to be too trivial and inadequate.\n4. I am confused about the advantages of using anchor distances for error diagnosis as no other tools provided to detect similar label words.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Paper topic and main contributions: This paper introduces an analysis perspective by using label words as anchors to understand ICL among different layers. This analysis derives two facts: (1) in shallow layers, label words gather information from demonstrations; (2) in deep layers, label words leads to the final prediction. Based on these facts, the authors propose three methods (anchor re-weighting, anchor-only context compression and anchor distances for error diagnosis) improve the effectiveness, efficiency, and interpretability of ICL.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PlKw1jdLFY", "reviewer": "Reviewer_ghKz", "summary": "", "questions": "1. Both the \"Label words\" and \"label tokens\" are used across the paper. Are they with the same meaning? Which token will the information flow in and gather from if the label word contains more than one tokens (subwords). The answer to this question will help extending the observation to generation tasks, where the label usually contains more than one tokens.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1:** Both the \"Label words\" and \"label tokens\" are used across the paper. Are they with the same meaning?\n\n**Answer 1:** Sorry for causing confusion. In our experimental settings, ‘label words’ are roughly equal to ‘label tokens’. The only exception is the ‘Abbreviation’ in the TREC dataset, which has more than one subword. For this exception, we simply use the first subword in our paper, as Calibrate Before Use: Improving Few-Shot Performance of Language Models (Zhao et al., 2021) does.\n\n**Question 2:** Which token will the information flow in and gather from if the label word contains more than one token (subword)? \n\n**Answer 2:** We explore the importance of the subwords {‘Ab’, ‘be’,’vation’} in ‘Abbreviation’ via saliency metrics (we use the sum of all saliency scores corresponding to a specific subword as its importance). In this case, the relative importance of {‘Ab’, ‘be’,’vation’} is about 4: 1: 3.5. This suggests that, at least in this instance, the first subword holds the most significance, followed by the last subword, with the middle word being the least important.\n\n**Question 3:** What about generation tasks?\n\n**Answer 3:** For generation tasks, the current token-level anchor might need to be expanded to span-level ones (or the weighted combination of multiple tokens), and their function might differ due to the difference between classification and generation. We look forward to exploring this in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "How ICL works is an important question and this paper provides insights of it on a specific setting, the classification task.\n\nThe correctness of the observation is verified through thorough experiments.\n\nAlthough the observation is only on the classification task where the label words contain few (1?) tokens, it is possible to extend to a more general situation including generation task. This work encourages relevant research on interpretable LLM.", "reasons_to_reject": "Refer to question 1.", "questions_to_authors": "1. Both the \"Label words\" and \"label tokens\" are used across the paper. Are they with the same meaning? Which token will the information flow in and gather from if the label word contains more than one tokens (subwords). The answer to this question will help extending the observation to generation tasks, where the label usually contains more than one tokens.", "paper_topic_and_main_contributions": "This paper explores the underlying mechanism of how in-context learning works for classification tasks.\n\nThrough the saliency score of the attention matrix between tokens (context tokens, label words, target position token) in shallow and deep layers, they observe that label words function as anchors. \n\nParticularly, in the shallow layers information aggregates into label word representations and in the deep layers information of label words largely contribute to the label prediction of the input.\n\nThree experiments are designed to leverage this observation: 1) re-weighting the attention of the label words to the target position by a few learnable parameters; 2) compressing each demonstration into only label words; 3) utilizing the distance between key vectors of the label words to predict the confusion matrix. These experiments show performance improvement thus verify the correctness of the observation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a deep dive into the mechanics of in-context learning (ICL) for classification tasks by analyzing how label words function as anchors.

In general, the paper is well-structured and written. Proposed anchor re-weighting and anchor-only context compression that improved the effectiveness and inference speed over vanilla ICL. All reviewer agreed the paper could have a wider impact and encourages relevant research on interpretable LLM.

Some concerns were also pointed out by several reviewers that require additional experiments and clarification.
Decision: Accept-Main