Not All Demonstration Examples are Equally Beneficial: Reweighting Demonstration Examples for In-Context Learning
Authors: Zhe Yang
Damai Dai
Peiyi Wang
Zhifang Sui
Venue: EMNLP 2023 Findings (2023)
Keywords: In-Context Learning
Training Example Reweighting

Abstract: Large Language Models (LLMs) have recently gained the In-Context Learning (ICL) ability with the models scaling up, allowing them to quickly adapt to downstream tasks with only a few demonstration examples prepended in the input sequence. 
Nonetheless, the current practice of ICL treats all demonstration examples equally, which still warrants improvement, as the quality of examples is usually uneven. 
In this paper, we investigate how to determine approximately optimal weights for demonstration examples and how to apply them during ICL. 
To assess the quality of weights in the absence of additional validation data, we design a masked self-prediction (MSP) score that exhibits a strong correlation with the final ICL performance. 
To expedite the weight-searching process, we discretize the continuous weight space and adopt beam search. 
With approximately optimal weights obtained, we further propose two strategies to apply them to demonstrations at different model positions.
Experimental results on 8 text classification tasks show that our approach outperforms conventional ICL by a large margin.
Our code are publicly available at https:github.com/Zhe-Young/WICL.

Not All Demonstration Examples are Equally Beneficial:
Reweighting Demonstration Examples for In-Context Learning
Zhe Yang, Damai Dai, Peiyi Wang, Zhifang Sui
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University {yz_young, daidamai, szkkk edu. Srrppeiy19979@gmme com
Abstract
Large Language Models (LLMs) have recently gained the In- In-Context Learning (ICL) ability with the models scaling up, allowing them to quickly adapt to downstream tasks with only a few demonstration examples prepended in the input sequence. Nonetheless, the current practice of ICL treats all demonstration ex- amples equally, which still warrants improve- ment, the quality of examples is usually uneven. In this paper, we investigate how determine approximately optimal weights for demonstration examples and how ap- ply them during ICI. To assess the quality of weights in the absence additional vali- dation data, design masked self-prediction (MSP) score that exhibits a strong correlation with the final ICL performance. To expedite the weight-searching process, we discretize the continuous weight space and adopt beam search. With approximately optimal weights obtained, we further propose two strategies apply them to demonstrations different model positions. Experimental results 8 text classification tasks show that our approach outperforms conventional ICL by large mar- gin. Our code are publicly available at https: gi gith.b.or com/ com/Che- Youh-Youn//IIL.
90
SST2 Accuracy (%)
80
70-
60
355M
1.38
2.78
6.78
13B
Model Size
Figure 4-shot ICL. performance on SST2 with differ- ent sets of weights assigned to demonstration examples. Gray points represent the accuracy of different weight sets, and the red point denotes the accuracy of the weighting strategy. The performance varies significantly with different weights.
The performance of ICL is closely related to the quality of demonstration examples (Wu al., 2022; Gonen et al., 2022). Therefore, various methods have been proposed to select high-quality exam- ples. For example, Liu et al. (2022) selects nearest examples based on semantic similarity, Zhang et al. (2022) employs reinforcement learning for exam- ple selection, and Sorensen et al. (2022) maximizes mutual information on an unlabeled validation set for choosing templates. However, these methods assume the availability of a large number of exam- ples for selection, which is not the case in real few- shot settings. From another aspect, previous ICL methods treat all demonstration examples equally, which can be further improved, as the quality of demonstrations is not even. As shown in Figure 1, assigning different sets of weights demonstration examples has significant impact on ICL perfor- mance. For a 13B GPT model, in terms of the accuracy on the SST2 dataset, the best weights surpass the worst by 40 points and surpass the non-
1 Introduction
With the increase in model size and training cor- pus, Large Language Models (LLMs) have demon- strated in-context learning (ICL) capabilities (Rad- ford et al., 2019; Brown et al., 2020; Wei et al., 2022: Dong et al., 2022). Unlike traditional fine- tuning, which requires updating model parameters, ICL allows LLMs to adapt to downstream tasks while keeping the parameters fixed. To enable ICL, small number of examples will be prepended be- fore the query to form prompt. The prompt is then fed into the LLM for prediction. Numerous studies have shown the effectiveness ICL,, demonstrat- ing that can achieve, and sometimes even surpass, the performance of fine-tuned models.

---

bers and templates. As for the approximately opti- mal weights discovered by our approach, we find that they demonstrate close performance to the global optimal weights. Furthermore, we discover that our example reweighting approach mainly works at middle layers, and reweighting only in few middle layers even outperforms full layers. We summarize our contributions follows:
100
Weight Vector
90
Regression Line
Accuracy (%)
80
0.73
70
60
50
1.
We propose the Weighted In-context Learn- ing (WICL) method, which enhances ICL by determining and applying approximately opti- mal weights demonstration examples.
40
-0.8
-0.7
-0.6
-0.5
-0.
MSP
Figure 2: Regression line of MSP score and accuracy on MR dataset. Each point denotes the performance of weight vector, the Pearson correlation coefficient 0.73, indicating a strong correlation between MSP score and accuracy.
2. We introduce a proxy metric called Masked Self-Predices Prff-Predicton (MSP) score to foretell the final ICL performance without additional vali- dation data.
3. We evaluate WICL on 8 NLP tasks and show that WICL can significantly outperform the conventional ICL method.
weighting strategy by 15 points. This highlights the necessity of assigning appropriate weights to demonstration examples.
In this paper, we propose the Weighted In- context Learning (WICL) method enhance ICL
4.
We perform elaborate quantitive analysis reveal the robustness WICL, and the qual- ity of the discovered approximately optimal weights.
by determining and applying approximately op- timal weights to demonstration examples. There are two key challenges for determining weights: how to evaluate the quality of set of weights without additional validation data (weight evalua- tion), and how to efficiently search the best set of weights in the continuous space (weight search). For weight evaluation, we introduce Masked Self Prediction (MSP) score as a proxy metric, which can be computed solely on demonstration examples but is strongly correlated with the final ICL perfor- mance, as shown in Figure 2. For weight search, we discretize the weight space and employ beam search to efficiently find the weights that yield the highest MSP score. With approximately optimal weights discovered, we propose two strategies to apply them to demonstrations. Scaling Key Ma- trix (SKM) applies the weights to the attention key matrices of demonstration examples, while Scal- ing Attention Weights (SAW) directly adjusts the attention weights. Both strategies adjust the impor- tance of demonstrations by influencing the atten- tion weights according to set of weights.
Problem Formulation
For text classification task, given set of k demonstration examples denoted by S { m1), (I22, 92).... (#k, Wx)), where 2; and yi denote the input text and its corresponding label, respectively. We define a transformation function T that maps each sample (æ, to an ICL example T(x, y) (c.g. T(x, v) "Sentence: x Sentiment: y" for the SST2 dataset). The concatenation of these transformed examples is then defined the final demonstration C:
C - C T @T..eTTTAA @TTK, Me), (1)
where @ denotes concatenation. The goal of ICL is identify suitable demonstration enable the LLM to generate probability distribution P that closely approximates the actual distribution, and subsequently select the label Spredict with the highest probability as the predicted output:
We evaluate WICL on text classification tasks and show that achieves substantial accuracy im- provement over the conventional ICL method that follows a nei-htigh strategy. In addition, our approach exhibits robustness to varying shot num-
(2) In the above equation, 9 and D represent the parameters of LLM and the candidate label set,
Ypreddctt argmax Ply|C x;;

---

Discovering Optimal Weights
ICL Example Reweighting
0ption1:
Or
Option2:
LLM
Weight vector
Scaling Key Matrix
Scaling Attention Weights
Masked Self- prediction Score
Weight
AttentionWeight
Self Attention
FFN
Indicator
Beam search
1.2
1.0
1.0
vector
1.2
1.0
example2
example
Key
Query
new
exampio
weights
at12
att1
O
vector
1.0
1.2
Narm
example2
example1
Value
1.0
example3
at13
1.0
example3
0.8
0.8
example4
at14
0.8
example4
Self-Attention Layer
Kuy=Query
Softmax
Attention weights
Values
Figure 3: An illustration of weighted in-context learning. Reweighting at the self-attention layer could be scaling key matrix or scaling attention weights. The example weights can be obtained by beam search with masked self-prediction score as an indicator, which shows a strong correlation with final performance.
respectively. Conventional ICL does not take the weight of each example into account, and LLM almost treats each example equally. To address this limitation, we introduce a weight vector w (W1, u..,.., wk)) for the demonstration, where wi denotes the weight of the ith example T(x;, W1). A higher value of wi indicates that the example T( y4) is more important, and thus LLM should pay more attention it. When given a specific demonstration, the objective of WICL is to fully use demonstration examples and identify an appro- priate weight vector w that enables the model to achieve performant performance.
we propose two simple yet effective approaches assigning weights to examples by Scaling Key Ma- trix (SKM) or Scaling Attention Weights (SAW) respectively.
Reweighting by Scaling Key Matrix In self- attention layer, demonstration examples represent key, and new example to be predicted can be viewed as query. So we can add weights to demon- stration examples by scaling vectors in the key ma- trix
WK- [WK,1, WK22 WK..).
(3)
where WK. E Rdxl, denotes the key matrix for the 8th example, and d and l, denote the hidden dimension and length of the i-th example, respec- tively. After scaling by w, the weighted key matrix can be represented as:
3 Methodology
Demonstration example reweighting faces two chal- lenges: (1) How to add weights to demonstration examples when given a weight vector. (2) How to find performant weight vector in the absence of a held-out validation set. Section 3.1 aims to solve (1) by reweighting the self-attention layer, and Section 3.2 aims to address (2) through weight quantization and beam search with an indicator. Our approach is illustrated Figure 3.
(4) and the weighted self-attention is calculated by the following equation:
w O WK
[w]] WK.1..., Wi WKA..
Attention Wy, wWK, WQ)
3.1 Reweighting Demonstration Examples
Wy softmax( [n C WK] Wk|TWo
(5)
Vd
Almost all of the current L.MSs are based on Trans- former decoder architecture, comprising multiple repeated blocks, each of which contains a multi- head self-attention layer and a feed-forward net- work layer. Sclf-attention layer is responsible for token interaction, and we can modify the weights for each example at this layer. Intuitively, increas- ing the attention weights that examples receive in the self-an atteentnnion layer can be viewed as assigning more weights to the examples. Inspired by this,
where Wy denotes value matrix; We denotes
query matrix; vd denotes scaling factor.
Reweighting by Scaling Attention Weights Af- ter the softmax layer, attention weights (the prod- uct query and key) are mapped into normalized probability distribution space. Adding weights examples could be scaling attention weights of ex- amples while maintaining the sum of them to be

---

Stepl:
-0.50
SST2
MR
weight for examplel
MSP. 0.69
Beam state 1.0.8
0.76
R 0.73
-0.38
Beam state 1.2
Acourad (N0
70
60
Step2:
0.8
MSP: 40.45
80
80
weight for example2
0,8
MSP:-0,50
50
1,2
MSD. 0.49
40
0.8
MSP:-0-47
0.5
08
Beam state I: (0.8,088)
Subj
CR
Beam state 2: (1.2.100)
MSP: -0.38
40
80
0.88
R 0.79
MSP:-0.47
Step3:
MSP:-0.47
Accumcy
80
40
weight for example3
MSS 0.45
40
Weight Viector Regression
MSP:
MSP: -0.40
0.2
state (:(1.2,1.0,0.8) Beam state (1.2,1100)1
MSP: -0.38
1 MSP: 0.45
Figure 4: Correlation MSP and accuracy under dif- ferent example weights. For each task, we randomly sample 50 legal weights under 8-shot setting and test accuracy on GPT-1.3B, showing scatter plots and regres- sion lines.
Step4:
0.8 MSR;-0.41
weight for example4
MSP
1.0
MSP: 40.29
Bearn state
1.0 MSP 9.38
(12.110.10...8)
1. attention weight for 7th example denoted as att, we have:
12MSP:
Figure 5: An illustration of beam search for exam- ple weights. We take 4-shot setting, beam size 2 as an example, and legal weight set for each example 1.00.., In each step, we extend beam states and preserve the 2 states with max MSP score.
W
[atti, ... atty] softmax(
(6)
Vd
We scale original attention weights att by weight vector w and normalize them. After scaling, new attention weight for T example is:
called self-prediction since we are predicting the demonstration itself. Label yı in the demonstration C is masked when predicting sample (xi, w: prevent the model from copying the answers from the demonstration. MSP is defined the average log probability of the predictions on S, and is calculated as follows:
waatt
att new
f Wyally
3.2 Discovering Optimal Weights
Indicator for Weight Selection How to select the optimal weight is challenging problem. One naive approach is test the model's performance validation set under range of weights and se- lect the weight that yields the best result. However, this approach may not be feasible when an extra validation set is unavailable. Fortunately, we have discovered a new metric called average Masked Self-Prediction score (MSP), which can be ob- tained without the need for an additional validation set yet still has strong correlation with the model's performance (as shown in Figure 4).
Cmask ...D T(:, (mask))). Vk.
(8)
Pi(w) P(y Cmnaskn + 23; w. w.
(9)
MSP(w)= Ei hog.:...llgpppppp log(p:(w)
(10)
k
To find a performant weight, we can simply choose weight with high MSP score, instead of testing and comparing the performance of all weights on validation set.
We consider the demonstration set S as a special "validation set" and predict the example labels in conditioned on weight vector w. This process is
Example Weight Searching To select proper weight vector w R² we employ weight quan- tization strategy that restricts each dimension w;

---

Model
Method
SST2
CR
AGN
RTE
TREC
Subj
Avg
ICL
72.1
65.7
47.0
53.7
34.5
75.0
65.7
49.7
57.9
GPT 335M
WICL-SAW
77.1
76.4
54.6
54.1
36.1
76.3
71.1
54.5
62.5
WICL-SKM
80.9
81.1
55.4
54.3
34.7
75.0
75.5
59.9
64.6
ICL
64,9
51.0
74,8
53.2
32.2
61.3
71.2
64.0
59.1
GPT 1.3B
WICL-SAW
68.7
55.4
76.3
53.2
33.3
64.1
71.1
69.3
61.4
WICL-SKM
76,5
74.9
78.5
53.3
33.7
68.0
73.4
69.0
65.9
ICL
68.1
63.1
79.1
53.0
27.1
86.1
83.8
66.4
65.8
8 shot
GPT 2.7B
WICL-SAW
71.6
72.2
80.1
53.9
28.4
86.1
86.4
71.8
68.8
WICL-SKM
79.1
80.2
79.7
53.7
29.4
86.7
87.8
75.6
71.5
ICL
87.4
73.0
78.7
57.1
82.8
52.5
68.1
GPT 6.7B
WICL-SAW
88.5
74.8
79.7
$7.5
84.6
53.3
69.1
WICL-SKM
88.5
82.0
82.1
57.3
85.2
85.2
57.3
70.9
ICL
89.5
87.0
85.3
51.9
34.0
87.9
88.2
58.8
72.8
GPT 13B
WICL-SAW
90.6
88.1
85,4
53.2
34.3
88.1
89.5
60.6
73.7
WICL-SKM
89.4
88.9
85.6
57.8
33.8
88.2
88.8
68.2
75.1
ICL
64.4
66.9
44.3
53.3
38.9
72.7
68.8
49.2
57.3
GPT 335M
WICL-SAW
78.5
77.3
56.7
53.2
41.4
74.8
71.4
56.2
63.7
WICL-SKM
84.0
79.9
55.0
54.3
40.6
75.1
75.9
61.6
65.8
ICL
65.0
58.1
75.0
53.6
35.2
64.5
72.3
64.6
61.0
GPT 1.3B
WICL-SAW
72.1
66.0
77.2
53.2
37.6
68.3
74.0
75.1
65.4
WICL-SKM
82.6
76.0
78.7
53.5
36.6
73.9
73.5
74.2
68.6
shot
ICL
69.4
60.7
79.9
87.0
82.6
62.2
65.3
GPT 2.7B
WICL-SAW
78.1
76.5
80.1
53.6
29.8
87.2
87.2
74.6
70.9
WICL-SKM
83.8
83.9
79.5
53.5
81.9
73.7
ICL WICL-SAW
88.4
60.9
78.9
57.4
28.9
86.6
86.7
$2.0
67.5
GPT 6.7B
89.9
65.0
80.1
$7.9
31.4
86.6
87.7
52.4
68.9
WICL-SKM
89.7
81.9
81.2
57.8
31.7
85.8
87.1
64.7
72.5
ICL WICL-SAW
90.7
83.0
85,4
51.1
34.6
89.3
87.8
61.9
73.0
GPT 13B
92.0
85.9
85.7
53.2
37.0
89.4
89.7
64.5
74.7
WICL-SKM
91.2
89.7
85.7
58.7
37.0
89.2
89.8
77.2
77.3
Table 1: Main experiment results. Taking conventional ICL as baseline, we compare the performance of WICL two reweighting methods datasets with different models under 8-shot 16-shot settings. For simplicity. DBPedia and AGNews written DBP and AGN, respectively.
w to values in a candidate weight set Q = { (weight1, weight2..... w This strategy compresses weights from continuous infinite space to discrete finite space and the number of legal weights drops from infinite to n²t, making the weight selection problem computationally tractable. However, brute force enumeration of all of the pos- sible weights and calculation of MSP score still requires Okk n²) time complexity, which un- scalable for large k. To address this problem, we apply beam search strategy which is commonly used in language model decoding process. From example I to example k, in each beam search step we search for weight for one example, illus- trated in Figure 5. The pseudo-code of our weight selection approach is presented in Appendix A.
Experiments
Similar to previous work (Lu et al., 2022), we use 8 classical text classification datasets, respectively SST2 (Socher et al., 2013), CR (Hu and Liu, 2004), MR (Pang and Lee, 2005), Subj (Pang and Lee, 2004), TREC (Voorhees and Tice, 2000), DBPedia (Zhang et al., 2015), AGNews (Zhang al., 2015) and RTE (Dagan et al., 2006), to evaluate the effi- cacy of our demonstration examples reweighting approach. our experiments, we utilize differ- ent GPT-like causal language models released by fairseq (Artetxe et al., 2022), and the number of parameters of these models is 355M, 1.3B, 2.7B, 6.7B, 13B, respectively!
Previous works (Wu et al., 2022; Gonen et al., 2022; Liu et al., 2022: Zhang et al., 2022) have
'https: ttt.///ttttttttt//e////////eeett htt..//aaeeooooreeeerr///aaaaeea tree//main/example///ee

---

355M
1.3B
2.7B
Avg
355M
1.3B
Avg
ICL Baseline
72.1
64.9
68.1
68.4
ICL Baseline
72.1
64.9
68.1
68.4
WICL-Dual
80.8
76.1
77.7
78.2
Label-Only Masking
80.8
76.5
79.1
78,8
WICL-SKM 80.9
76.5
79.1
78.8
Whole-Example Masking
75.0
74.8
81.2
77.0
Whole-Fxample Removing
59.0
70.0
63.4
64.1
Table SST2 accuracy (%) for different methods and models. WICL-Dual denotes WICL by combining scal- ing key matrix and scaling attention weights. WICL- SKM outperforms WICL-Dual.
Table 3: Comparing different approaches to calculate MSP SST2. Label-only Masking strategy outper- forms other strategies.
reported ICL. performance depends heavily on the example selection, which is important but not the focus of our work. In order to reduce the possible effects of example selection on our experiments, repeat each group of experiments 100 times der different seeds and report average results. For each group of experiments, we randomly sample demonstration examples from the training set with random sceds of 0,1,2, 99 respectively, and use balanced sampling strategy to ensure that labels of examples are as balanced as possible (e.g., 3,3,2 examples for different classes for a 3 3-classification task in 8-shot setting). To reduce computational cost, we limit the number of data of validation set no more than 2000; for validation with more than 2000 data, we only randomly sample 2000 data for testing. The experiments are conducted under 8-shot and 66-sho settings. The templates used in the experiments and more details are shown in Appendix B.
strategies when calculating MSP (Section 5.1), and compare MSP with held-out validation set (Sec- tion 5.2). Nextly, we explore the robustness of our approach to different templates, shot numbers (Section 5.3, 5.4). Then, we find that our approach can obtain near-optimal weights (Section 5.5). In Section 5.6, we discover that example reweighting mainly works at middle layers, and reweighting on only few middle layers can even outperform full-layer reweighting. Section 5.7 shows some empirical results on relationship between example Weight and example quality/position.
5.1 Label-Only Masking Outperforms Whole-Example Masking/Removing
Our MSP score calculation process is similar to Leave-One-Out
(LOOCV) be- cause both test example based on the other --1 examples when given k examples. How- ever, unlike LOOCV, we only mask the label of the example to tested rather than removing it entirely. Inspired by LOOCV, we also compare the performance of label-only making, whole-example masking and whole-example removing. As shown in Table 3, label-only masking outperforms other methods. Whole-example masking/removing sults in performance drop due the demonstra- tion sensitivity of ICL. When the entire example is masked/removed MSP calculation, the cor- rupted demonstration differs significantly from the original one used in final testing, leading to in- consistency that harms ICL performance. Label- only masking can strike a balance between preserv- ing demonstration information and preventing the model from copying the answers from the demon- stration.
As the experiment results shown in Table 1, the average accuracy is reported under the following three methods: (1) conventional ICL, (2) WICL by scaling key matrix and (3) WICL by scaling attention weights. our approaches outperform con- ventional ICL on almost all 8 tasks, 5 different models. Besides. SKM also shows more power in performance improvement than SAW. We also try to combine these two reweighting strategies, which means reweighting by scaling key matrix and atten- tion weights simultaneously. As shown in Table 2, combining SAW and SKM does not improve per- formance; instead, it does harm to the performance and the final performance little weaker than SKM.
5 Analysis
5.2 MSP is Approximation of a Held-out Validation Set
We also further analyze our experiments and in- vestigate properties of our approach. Since SKM outperforms SAW, we do our analysis experiments only with SKM reweighting strategy. Firstly, we compare different example masking/removing
held-out validation set can be viewed as a "strong supervision", while our MSP more like an ap- proximation this supervision under true few-shot

---

WICL- Validation
Template
WICL
Size
Method
ICL
-MSP
10
20
50
100
2
4
8-shot
64.9
76.5
76.
82.4
83.2
84.9
ICI.
72.1
69.6
54.6
71.8
16-shot
65.0
82.6
79.6
83.0
83,4
85.3
355M
WICL
80.9
71.5
69.2
80.0
ICI.
64.9
63.1
55.6
68.1
Table 4: Comparing MSP with held-out validation set of 10, 20, 50, 100 examples on SST2 using GPT 1.3B. Though validation set outperforms MSP, MSP can improve ICL significantly under true fow-shot setting.
1.3B
WICL
76.5
79.3
73.8
80.6
ICL
68.1
68.9
2.7B
WICI.
79.1
85.3
83.1
85.5
Table 5: Performance of different models on SST2 un- der 4 different templates. WICL is robust different templates.
85
Method
ICL
SSt Accuracy (%)
80
WICL
with the increase of shot number, which demon- strates that example reweighting can enhance the model's ability to make full use of examples. We also observe that for SST2 (a binary classification task), the ICL performance fluctuates between odd and even shot numbers as the number of shots in- creases. Differently, this oscillation on WICL is relatively small, indicating that WICL is more bust odd even shot numbers.
75
10
16
Shot
Figure 6: Performance of WICL on SST2 under dif- ferent shot settings, WICL is robust different shot numbers.
5.4 Example Reweighting is Robust to Different Templates
In our main experiment, only one template is used for each task, and we would like to further explore the robustness of our method to different templates. As shown in Table 5, we use 4 different templates the SST2 dataset, and for each template, method can achieve significant improvement over the conventional ICL method. Details of these templates are shown in Appendix C. Moreover, conventional ICL is sensitive to the choice of tem- plates, and the performance can vary significantly under different templates (e.g. 17.2% difference in accuracy under template3 and template4 for GPT 355M). On the contrary, approach significantly reduces the model's sensitivity templates and performs well on different templates.
setting. The comparison of MSP and held-out validation of 10, 20, 50, 100 examples are shown in Table 4. Though MSP is slightly weaker than vali- dation set, it outperforms ICL baseline by a large margin. In true few-shot setting where resource limited to only a few examples, MSP is an efficient indicator for weight search.
5.3 Example Reweighting is Robust under Different Shot Settings
Our main experiment is conducted on 8 and 16 shot settings, and we aim to explore the effects of example reweighting under different shot settings, the experimental results presented in Figure Our findings indicate that our approach does not significantly improve model performance when the number of shots less than 4; however, when the number of shots is greater than 4, there is a sig- nificant improvement over the baseline. Moreover, observe that the larger the number of shots, the model's ICL performance is necessarily better, and the ICL performance may gradually decline with the increase of shots. This could be due to in- formation distraction caused by too many examples. On the contrary, our approach can steadily improve
5.5 MSP and Beam Search Yields Near-optimal Weights
Finding the optimal example weights can be very expensive as it requires enumerating all legal weights and validating them. In contrast, our ap- proach requires no extra data and has relatively low computational complexity. Therefore, it is worth exploring whether our simplification for obtaining weights significantly impacts performance and how far our example weights are from optimal weights. To investigate this, we randomly sample 50 weights

---

1.0
Cumulative Distribution
Acc of WICL
0.8
1.03
Average Weight
1.02
Proportion
0.6
1.01
1.00
0.4
0.98
0.97
0.2
5
6
Position
Figure 9: Average weight for examples in different positions for 8-shot WICL. Examples positioned in the middle tend be assigned less weight, while those at the beginning are assigned more.
0.0
40
50
60
70
80
90
100
SST2 Accuracy (%)
Figure Cumulative distribution of accuracy under different weights on SST2. Our approach outperforms about 80% proportion of weights.
full-layer WICL as two baselines and compare the performance of reweighting only on few consecu- tive layers, as shown in Figure 8. Our findings sug- gest that example reweighting has almost no effect at the top and bottom layers, but has significant impact the middle layers. Moreover, reweighting only a few intermediate layers can even outperform reweighting all layers.
Number
layers
80
Norie layer
Full layers
75-
70-
5.7 Relationship between Example Weight and Example Quality/Position
As shown in Figure 9, we empirically find that weights of examples at different positions are not evenly distributed; examples positioned in the mid- dle tend to be assigned less weight, while those at the beginning are assigned more. Besides, has been observed that examples with higher 1-shot per- formance tend to maintain their original weights, while examples worse 1-shot performance tend to be rescaled. For examples of weight 0.9, 1.0, and 1.1, their average 1-shot performance on SST2 is 61 1144 65.2% and 58.3%, respectively. Exam- ple reweighting makes the demonstration strike intrinsic balance by scaling these examples, and 1-shot performance indicates the intrinsic balance degree of the example.
12
166
24
Layer
Figure 8: WICL performance GPT 1.3B when only reweighting few consecutive layers. Reweighting mainly work at middle layers.
from the weight space under the 8-shot setting, test the accuracy the validation set, and plot the mulative distribution function (as shown in Figure 7). Our findings indicate that our approach outper- forms approximately 80% the weights, demon- strating that the MSP indicator and beam search can yield near-optimal weights.
5.6 Example Reweighting Mainly Works at Middle Layers
Related Work
In main experiment, example reweighting is applied at all layers, but it still unclear which layers the reweighting strategy most effective. To investigate this, we use partial reweighting ap- proach, where examples are reweighted only on cer- tain layers of the model. we adopt direct ICL and
In-Context Learning The emergence of LLMs' in-context learning ability has drawn fair amount of attention to the study of few-shot learning with LLMss. However, ICL. is still faced with the problem of instability and vulnerability (i.e. huge differences in performance under different

---

prompts). Many existing works have focused on addressing this problem and improving ICL performance: Sorensen et al. (2022) attributes the poor performance to bad prompt section and proposes an cnrrmaationthhoreetie approach to select prompt based on mutual information, which can be obtained with an external unlabeled set. Lu et al. (2022) considers the poor performance is from poor order of demonstration examples and proposes an order selection algorithm based on entropy, which can be obtained with the help of probing set produced by LLM itself. Zhao et al. (2021) proposes that the failure of ICL is mainly caused by all kinds of biases in demonstration examples or LLM itself, and designs a simple debias method. Different from previous work, we propose new perspective that poor performance of ICL comes from lack of proper weights for demonstration examples and de- sign an effective reweighting methodology for ICL.
under this setting, Our approach for ICL examples reweighting is under true few-shot setting.
Conclusion
In this paper, we find that treating demonstration examples equally can be bottleneck for ICL, and assigning proper weights to examples can signif- icantly improve performance. Based on this ob- servation, we attempt to reweight demonstration examples and propose weighted in-context learning.
To explicitly assign weights, we reweight examples in the self-attention layer by scaling key matrix scaling attention weights; determine the weight vector for examples, we adopt weight quantization beam search by maximizing masked self peef-rredittion score, which can be obtained with any held-out val- idation set yet shows a strong correlation with the final performance. Our approach can significantly improve ICL performance in true few-shot setting, and experiments on 8 NLP tasks also demonstrate the efficacy of our approach.
Reweighting Training Examples in Machine Learning Machine learning can easily overfit to training set biases, especially when there is a de- viation between training set distribution and true distribution. To address this problem, reweight- ing training examples is a commonly used strategy. To reduce the variance of estimation, Importance Sampling (Kahn and Marshall, 1953) equalizes the two distributions by assigning weights to examples. AdaBoost (Freund and Schapire, 1997) iteratively assigns more weights to misclassified examples to obtain a stronger classifier; Focal Loss (Lin et al., 2017) assigns more weights to harder examples with a soft weighting scheme; Ren et al. (2018) learns to assign weights to training examples based on their gradient directions. However, previous example reweighting work mainly focuses on ma- chine learning with abundant training data and few- shot learning is not included. Different from previ- ous work, we try to explore example reweighting of ICL under few-shot setting.
Limitations
Although our approach can significantly improve the performance of ICL by reweighting the demon- stration examples, there are still many limitations: (1) Our approach only has a noticeable effect when the number of examples is greater than 4. (2) Un- der our approach, performance variance still exists and the quality of examples still matters to the performance. Our approach can improve perfor- mance but can not totally solve ICL'S vulnerability to demonstration examples. (3) Our approach per- forms reweighting the self-attention layer, which requires access to the model's parameters; for some large models whose parameters are unavailable (e.g. GPT4 (OpenAI, 2023)), our approach can hardly (4) We simply add the same weights to all layers or a few consecutive layers, without per- forming more fine fine-grained reweighting strategies to different layers (e.g. different example weights for different layers), and the mechanism of exam- ple reweighting on the improvement of ICL is still unexplored.
be applied.
True Few-shot Learning In true few-shot set- ting, held-out validation set for better perfor- mance is unavailable, and resource is limited to only few examples. Previous work (Perez al., 2021; Schick and Schütze, 2022; Bragg al., 2021) have fully discussed the importance of true few- shot setting for ICL. Min et al. (2022a); Lu et al. (2022); Lyu et al. (2023) also try to improve ICL
Acknowledgements
This paper is supported by the National Key Research and Development Program of China 2020AAA0106700 and NSFC project U19A2065.

---

References
Xiaonan Li and Xipeng Qiu. 2023. Finding support- ing examples for in-context learning, arXiv preprint arXiv:2302.1333
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ont Sam Shleifer, Xi Victoria Lin, Jingfei Du Srinivasan Iyer. Ramakanth Pa- sunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Pumit Singh Koura, Brian O'Horo, Jef- frey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Veselin Stoyanov. 2022. Effficient large scale language modeling with mixtures of ex- perts. In Proceedings of the 2022 Conference on Empirical Methods in Naturul Langwage Processing, pages I 11699-11753 11732, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming Hc, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEF international conference on computer vision, pages 2980-22 -9888.
Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan,
Lawrence Carin, and Weizhu Chen. 2022. What makes good in in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeelJ0 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures pages 100-114, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Jonathan Bragg, Arman Cohan, Kyle Lo, and Iz Beltagy. 2021. Flex: Unifying evaluation for few-shot nlp. Advances in Neural Information Processing Systems, 34:15787-15800.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel. and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086 8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Tom Brown Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. 7-ICI: Zero-shot in-context learning with pseudo-demonstrations, In Proceedings of the st Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 2304-2317, Toronto, Canada. Association for Computational Linguisties.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment chal- lenge. In Machine Learning Challenges. Evaluat- ing Predictive Uncertainty, Visual Object Classifi- cation, and Recognising Tectual Entailment: First PASCAI. Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 1-13, 2005, Revised Selected Papers, pages 197- Springer.
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022a. Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 5316-5330, Dublin, Ireland. As- sociation for Computational Linguistics.
Damai Dai, Yutao Sun, Li Dong. Yaru Hao, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.1055:
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. survey for in-context learning. arXiv preprint arXiv:2301.002.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis. Hannanch Hajishirzi, and Luke Zettle- moyer. 2022b. Rethinking the role demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048-1 1064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yoav Freund and Robert E Schapire. 1997. A decision- theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):119-139
Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. 2022. Demystifying prompts language models via perplexity estimation. arXiv preprint arXiv:2212.04037
OpenAl. 2023. Gpt-4 technical report.
Bo Pang and Lillian Lee. 2004. A sentimental educa- tion: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 271-278, Barcelona, Spain.
Minqing Hu and Bing Liu. 2004. Mining and summa- rizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 168-177.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploit- ing class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Compu- tational Linguistics (ACL 05), pages 115-124, Ann
Herman Kahn and Andy W Marshall. 1953. Methods of reducing sample size in monte carlo computations. Journal of the Operations Research Society of Amer- ica, 155):263-278.

---

Arbor, Michigan. Association for Computational Lin- guistics.
deeper look into input-label demonstrations. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2422- 2437, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. Ad- vances neural information processing systems, 34:11054-11070.
Xiang Zhang. Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances neural information processing systems, 28.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188)9
Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. Ac- tive example selection for in-context learning. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 9134- 91 9148, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun, 2018. Learning to reweight examples for robust deep learning. In International conference on machine learning, pages 4334 4343. PMLR.
Timo Schick and Hinrich Schüitze. 2022. True few-shot learning with Prompts-A real-world perspective. Transactions of the Association for Computational Linguistics, 10:716-731.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- rernational Conference on Machine Learning, pages 12697-12706. PMLR.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference Empiri- cal Methods in Natural Language Processing, pages 1631-1642. Seattle, Washington, USA. Association for Computational Linguistics.
Taylor Sorensen, Joshua Robinson, Christopher Ryt- ting, Alexander Shaw, Kyle Rogers, Alexia Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. 2022. An iioommttion-thooette approach to prompt engineering without ground truth labels. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers). pages 819 862, Dublin, Ireland, Association for Computational Linguistics.
Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR confer- ence on Research and development in information retrieval, pages 200-207.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.0766 07662.
Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Ling- peng Kong. 2022. Self-adaptive in-context learning. arXiv preprint arXiv:2212.105:
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context leam- ing as implicit bayesian inference. arXiv preprint arXiv:2111.02080
Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyun- soo Cho, Hwiyeol Jo, Sang- Woo Lee, Sang-goo Lee, and Taeuk Kim. 2022. Ground-truth labels matter:

---

Appendix
A
Algorithm for Weight Selection with MSP Score
Algorithm 1: Weight Selection with MSP
Input:
Q: candidate weight set for each example;
beam size; shot number
f: function to calculate MSP score;
Output: weight vector w Initialize Wo < {(1.0, 1.0, 1.0)}
for i {1, 2, k) do
W, + 0
for in - do
for candi in Q do
w[i] s candi score + f(w)
aapennd (score.w)) )
end
10
end
W;+ < (ttbbpb
11
end
Return 1o)l ()
B Experiment Details
For GPT-355M, GPT-1.3B, GPT-2.7B and GPT- 6.7B, we the experiments on a single NVIDIA 3090 24G GPU. For GPT-13B, we do the experi- ments on a single NVIDIA A40 48G GPU, because this model is too large to load on an NVIDIA 3090 24G GPU. For scaling key matrix approach, the weight candidate set is (0.9, 1.0, 1. and for scal- ing attention weight approach, the weight candidate set is {0.8, 1.0, 1.2). In beam search, we beam size to 1 (i.e. greedy search). The template and label mapping in our experiments is shown in Table 6.
4 Templates for SST2
To explore the robustness of WICL different templates, do the experiments on SST2 with different templates. The templates and label map- ping are shown Table

---

Tasks
Template
Label Mapping
SST2
Sentence: {text) Sentiment: (answer)
negative, positive
CR
Sentence: (text) Sentiment: (answer)
negative, positive
AGNews
Article: {text) Category: (answer)
RTE
[sentencel] | Question: [sentence2] True or False?
True,False
Answer: (answer)
TREC
Question: [text) Type: (answer)
Description, Entity, Expression, Person. Number, Location
DBPedia
Input: [content] Type: (answer)
company, school, artist, sport, politics, transportation, building, nature, village, animal, plant, album, film, book
MR
Review: (text) Sentiment: (answer)
Negative, Positive
Subj
Input: (text) Type: (answer)
objective, subjective
Table 6: Templates and label mapping for different tasks.
ID
Template
Label Mapping
1
Sentence: (text) Sentiment: (answer)
negative, positive
2
Input: (text) Prediction: (answer)
negative, positive
3
Review: (text) It was (answer)
bad, good
4
Review: [text) Sentiment: (answer)
bad, good
Table 7: 4 different templates and label mapping for SST2.


Reviews:
[{"review_id": "0qbzz6N93j", "reviewer": "Reviewer_FRfS", "summary": "", "questions": "1. Rewighting demonstrations is orthogonal to demonstration selection.  And this paper conduct experiments on randomly selected demonstrations. Can reweighting demonstrations that are selected by existing SOTA selecting methods such as KATE and EPR further improve performance? Whether these well-selected demonstrations keep similar weights?\n2. Existing studies show that ICL is sensitivity to the order. Does reweighting correlate with order? If we change the order, does the weight should be modified? If not, does reweighting reduce the sensitivity of order?\n3. MSP does not require an extra validation set. An intuitive method that evaluates the quality of an example is using this example to conduct ICL and compare it with the zero-shot performance. With a random validation set, MSP, and this method, which is better?\n4. Experiments are conducted on 8-shot and 16-shot settings, smaller shots, such as 2,3 are the proposed method still useful? Intuitively, reweighting seems to not work for smaller shots.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Responses to Reviewer FRfS\nThank you for taking the time to thoroughly review our paper, and I sincerely appreciate your feedback. However, I want to clarify some of reasons for rejection that you've provided. The following are the responses to your reasons for rejection and questions.\n\n## Q1: \n\"Lack of more baselines.”\n\n## Answer:\nThe two baseline approaches you suggested for comparison, namely \"1) assigning random weights\" and \"2) selecting the best weights from k randomly assigned trials,\" serve primarily to answer “Are the weights obtained using the proposed method better than other weights?”. Actually, we have answered this question in Section 5.4. \nIn this section, we describe our process of randomly selecting 50 sets of weights, evaluating their accuracy on the validation set, and subsequently constructing a cumulative distribution function that maps accuracy distribution across the weight space. Notably, our analysis reveals that the weights generated by our method outperform over 80% of the weights.\nIt's worth noting that this achievement is particularly impressive given that our method doesn't rely on any additional data.\n\n***\n\n## Q2:\n\"Lack of some heuristic conclusions.”\n\n## Answer:\nThere is a certain correlation between weights and the positioning of examples, as well as the quality of examples. For instance, examples positioned in the middle tend to be assigned less weight, while higher-quality examples tend to be assigned more weight. That’s a good question, and we will add this part to section 5. Actually, even without these heuristic conclusions, our approach can automatically compute the weights for demonstrations. This eliminates the need for manual observation of demonstration features and further human intervention.\n\n***\n\n## Q3:\n\"Reweighting demonstrations is orthogonal to demonstration selection”. Combining example reweighting with example selection. \n\n## Answer: \n\nIt's improper to combine our approach with example selection, because our approach is conducted under true few-shot setting while example selection is not.  \n\nIn **true few-shot setting**, a held-out validation set for better performance is unavailable, and resource is limited to only a few examples. Previous work like [1],[2],[3] have fully discussed the importance of true few-shot setting for ICL. [4],[5],[6] also try to improve ICL under this setting. However, SOTA selecting methods such as KATE and EPR need a dataset to select examples from, which is contradictory to the true few-shot setting. Combining our example reweighting with example selection ruins our true few-shot setting. \n\nThe strength of our method lies in its ability to enhance performance for any given demonstration without the need for additional data. This randomness in example selection is more practical in true few-shot setting. Therefore, combining our method with example selection is not quite suitable. If forcefully applied, it would result in an all-ones vector as the weight for performant demonstration, which is already balanced in weight.\n\n***\n\n## Q4: \n\" Does reweighting correlate with order?\"\n\n## Answer:\nIn our main experiment, each result represents the average result of 100 repeated trials with randomly sampled examples. This is sufficient to demonstrate the robustness of our method to order. \nWhen the order of examples is changed, the weight for example needs to be modified. We also observed that examples in the middle tend to be assigned less weight, while those at the beginning and end are assigned more.\n\n***\n\n## Q5:\n\"An intuitive method that evaluates the quality of an example is using this example to conduct ICL”. “With a random validation set, MSP, and this method, which is better?\"\n\n## Answer:\n\nIn our preliminary experiments, we explored using 1-shot performance to gauge the quality of examples. However, the 1-shot performance does not necessarily reflect the significance of that particular example within the entire demonstration. The result is that our MSP outperforms this method.\n\nComparing our MSP with an additional random validation dataset is unfair, as the validation set disrupts true few-shot setting.\n\n***\n\n## Q6:\n\"Intuitively, reweighting seems to not work for smaller shots.\"\n\n## Answer:\nOur method has a weak effect when the shot number is less than 4, which has been shown in Figure 6. We have also pointed it out in the Limitations. The effect of example weighting at low shot number is worth further exploration.\n\n## References\n[1]. True Few-Shot Learning with Language Models (Perez et al., NeurIPS 2021)  \n[2]. True Few-Shot Learning with Prompts—A Real-World Perspective (Schick & Schütze, TACL 2022)  \n[3]. FLEX: Unifying Evaluation for Few-Shot NLP (Bragg et al., NeurIPS 2021)  \n[4]. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity (Lu et al., ACL 2022)  \n[5]. Noisy Channel Language Model Prompting for Few-Shot Text Classification (Min et al., ACL 2022)  \n[6]. Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations (Lyu et al., ACL 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper finds a new problem of in-context learning: existing ICL methods treat demonstrations equally. And this paper argues that reweighting the demonstrations could further improve ICL and conducts experiments to show that the ICL performance varies significantly with different weights.\n2. To find optimal weights, this paper proposes masked self-prediction score (MSP), which is correlated with the model's performance. And MSP does not require an additional validation set. And this paper employs beam search in the weight selection process. Then, this paper applies the weights by scaling key matrix and scaling attention weights. Section 5.4 also verifies that the proposed method outpeforms approximately 80% weights, which show its effectiveness.\n3. Extensive experiments on 8 datasets across 5 LMs show the effectiveness of the proposed method.", "reasons_to_reject": "1. Lack of more baselines. In Table 1, this paper only compares the proposed method with the original ICL. Only one baseline could not show the effectiveness of the proposed method. It is necessary to design other baselines which assign weights to different weights in ICL. For example, 1) assign random weights, 2) select the best weights from randomly assign k times. In specific, this paper should answer two main questions: 1) Why do we need to assign weights? and 2) Are the weights obtained using the proposed method better than other weights?\nThe experiments neglect the second question.\n2. Lack of some heuristic conclusions. Such as, what types of demonstration are assigned higher/lower weights? Correlate this with label distributions, word distributions would benefit the total community.", "questions_to_authors": "1. Rewighting demonstrations is orthogonal to demonstration selection.  And this paper conduct experiments on randomly selected demonstrations. Can reweighting demonstrations that are selected by existing SOTA selecting methods such as KATE and EPR further improve performance? Whether these well-selected demonstrations keep similar weights?\n2. Existing studies show that ICL is sensitivity to the order. Does reweighting correlate with order? If we change the order, does the weight should be modified? If not, does reweighting reduce the sensitivity of order?\n3. MSP does not require an extra validation set. An intuitive method that evaluates the quality of an example is using this example to conduct ICL and compare it with the zero-shot performance. With a random validation set, MSP, and this method, which is better?\n4. Experiments are conducted on 8-shot and 16-shot settings, smaller shots, such as 2,3 are the proposed method still useful? Intuitively, reweighting seems to not work for smaller shots.", "paper_topic_and_main_contributions": "This paper studies an interesting topic, which assigns different weights for different demonstrations in in-context learning. This paper argues that existing ICL methods treat all demonstrations equally and this could be further improved. This paper employs experiments to verify the importance of the weight of demonstrations. To obtain optimal weights, this paper proposed a masked self-prediction (MSP) score and employs beam search in the weight-searching process. And this paper further proposes two methods for applying these weights: scaling key matrix and scaling attention weights. Extensive experimental results show the effectiveness of the proposed method.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kWfBofK2op", "reviewer": "Reviewer_Mxfy", "summary": "", "questions": "A. line 051 posits that prior example selection strategies need a lot of data to work. However, this is not empirically validated. I would suggest still trying them out and showing the empirical performance. Also, I don’t see why the leave-one-out approach used in the proposed method cannot be used with these baselines. \n\nB. Is the search algorithm proposed sensitive to example order? \n\nC. How is the discretized weight space chosen?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Resopnses to Reviewer Mxfy\n\nThank you for taking the time to review our paper. However, it seems there is a great misunderstanding about core idea of our work. With regard to \"this work proposes an example selection strategy for in-context learning\", that’s not what our work actually does. Our work aims to find proper weight for examples in a given demonstration, and our work does not involve example selection, as selecting examples from a dataset contradicts true few-shot setting. We do experiments with randomly sampled examples, and our approach can improve ICL performance for almost any given demonstration under true few-shot setting.  \n\nIn **true few-shot setting**, a held-out validation set for better performance is unavailable, and resource is limited to only a few examples. Previous work like [1],[2],[3] have fully discussed the importance of true few-shot setting for ICL. [4],[5],[6] also try to improve ICL under this setting. This setting is meaningful in practice.\n\n***\n\n## Q1:\n\"The paper does not consider strong baselines.”\n\n## Answer:\n“retrieval/ranking based on query similarity” is not a proper baseline because example selection or example order selection is not our work actually does. Besides, retrieval/ranking methods are orthogonal to our approach.\n\nRetrieval method needs a dataset to choose examples from, which contradicts our **true few-shot setting**.  \n\n\n***\n\n## Q2:\n“The search algorithm proposed is computationally expensive”\n\n## Answer:\nThe computational complexity of our approach is not high at all.  \n1.The additional computation introduced by our method is a one-time cost, and we only need to search for proper weights one time before inference.   \n2.Beam search is linear in time complexity for the number of shots. Even for a 16-shot setting, searching for weights only needs 3*16 forward passes through language model.  \nSo the computational cost of our approach is totally acceptable.  \n\n\n***\n\n## Q3:\n\"In fig 6 a monotonic decrease is observed for the baseline ICL when number of demonstrations are increased\"\n\n## Answer:\n\nWe have thoroughly examined our code and confirmed that there are no issues. In order to better align with the true few-shot setting, our experimental setup involves randomly selecting demonstrations, repeating the process 100 times and reporting the average performance. The best performance may increase with the number of shots, but it is not necessarily the same case when reporting average performance. Similar data of worse performance for larger shot numbers have also been observed in other ICL studies, such as [7].\n\n***\n\n## Q4:\n\"line 051 posits that prior example selection strategies need a lot of data to work. However, this is not empirically validated. I would suggest still trying them out and showing the empirical performance. \"\n\n\n## Answer:\nPrevious example selection strategies (e.g. [8],[9],[10]) need the whole training set or subset of original training to select examples from. For instance, [8] select examples from a candidate set of 100 or 1000 examples. However, this is contradictory to the true few-shot setting. \nShowing previous example selection performance as baselines is not suitable, because example selection is not the topic of our work. our work improves ICL performance by reweighting examples in a given demonstration. Besides, our work is under true few-shot setting, which also differs from example selection. \n\n***\n\n## Q5:\n\"Is the search algorithm proposed sensitive to example order?\"\n\n## Answer:\nIn our main experiment, each result represents the average result of 100 repeated trials with randomly sampled examples. This is sufficient to demonstrate the robustness of our method to order.\nBesides, we also observed that examples in the middle tend to be assigned less weight, while those at the beginning and end are assigned more.\n\n***\n\n## Q6:\n\"How is the discretized weight space chosen?\"\n\n## Answer:\nDiscretized weight space can be viewed as a hyper-parameter. We first try [0.8, 1.0, 1.2] and [0.9, 1.0, 1.1] on SST2, finding that [0.9, 1.0, 1.1] performs well and each weight is relatively balanced. So we choose [0.9, 1.0, 1.1] as the weight space.\n\n***\n\n## References\n[1]. True Few-Shot Learning with Language Models (Perez et al., NeurIPS 2021)  \n[2]. True Few-Shot Learning with Prompts—A Real-World Perspective (Schick & Schütze, TACL 2022)  \n[3]. FLEX: Unifying Evaluation for Few-Shot NLP (Bragg et al., NeurIPS 2021)  \n[4]. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity (Lu et al., ACL 2022)  \n[5]. Noisy Channel Language Model Prompting for Few-Shot Text Classification (Min et al., ACL 2022)  \n[6]. Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations (Lyu et al., ACL 2023)  \n[7]. Calibrate Before Use: Improving Few-shot Performance of Language Models (Zhao et al, ICML 2021)  \n[8]. Active Example Selection for In-Context Learning (Zhang et al., EMNLP 2022)  \n[9]. Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering (Wu et al., ACL 2023)  \n[10]. What Makes Good In-Context Examples for GPT-3? (Liu et al., DeeLIO 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Re-scaling the attention based on learned weights is an interesting idea.\n* Ablations presented are somewhat useful.", "reasons_to_reject": "The paper does not consider strong baselines. For instance, a simple baseline that could have been considered is retrieval/ranking based on query similarity. \n\nThis also makes me question whether the complexity of the method proposed is justified by the experimental results. The search algorithm proposed is computationally expensive and involves multiple forward passes through the language model. \n\nIn fig 6 a monotonic decrease is observed for the baseline ICL when number of demonstrations are increased, which is unusual. I suggest analyzing this behavior with more models and debugging the baseline implementation.", "questions_to_authors": "A. line 051 posits that prior example selection strategies need a lot of data to work. However, this is not empirically validated. I would suggest still trying them out and showing the empirical performance. Also, I don’t see why the leave-one-out approach used in the proposed method cannot be used with these baselines. \n\nB. Is the search algorithm proposed sensitive to example order? \n\nC. How is the discretized weight space chosen?", "paper_topic_and_main_contributions": "This work proposes an example selection strategy for in-context learning. The authors propose to ‘weight’ the examples so that more relevant examples receive higher weight. The actual weighting is performed by rescaling the attention weights of the pre-trained language model. In order to find the optimal weights, a scoring mechanism is proposed to quantify the quality of a given weight vector which is based on the likelihood prediction of the label for a heldout example given demonstration examples. The optimal weights are chosen using a beam-search decoding procedure, where the weights are limited to a pre-defined set of discrete options. Experiments on 8 classification tasks show that the method performs better than baselines.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In section 3, the paper directly talks about query, key, value matrices without providing any background or motivation. As a result, I wasn’t able to get a complete understanding of the technical details. \n\nI couldn’t get the main intuitions from Fig 3. The figure needs to be improved. \n\nline 284: the description about random seeds isn’t very meaningful as random seeds are generally platform specific."}}, {"review_id": "1XgNaQalKK", "reviewer": "Reviewer_zns4", "summary": "", "questions": "1. What are the weights look like found by WICL? Are there any patterns that exist in the examples if the method assigns high/low weights?\n2. How does the order of the demonstrations influence the assigned weights?\n3. The method seems more effective for small models than large models: for 8-shot, +6.7 on 335M, but +2.3 on 13B; for 16-shot, +8.5 on 335M but +4.3 on 13B. Following this tendency, will the improvement vanish on 70B models? (Of course, the improvement on the 13B model is also meaningful in practice. You can only share your thoughts if the resource is limited for larger experiments.)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Responses to Reviewer zns4\nThank you for taking the time to thoroughly review our paper, and I sincerely appreciate your feedback. Some of your questions are helpful to further improvement our paper. I’d like to answer your questions.\n\n***\n\n## Q1:\n“How the method works on the text generation tasks”\n\n## Answer:\nText classification is a special case of text generation when only one token/label is generated. Our approach can be easily applied to text generation task by replacing the likelihood of labels by PPL.\n\n***\n\n## Q2:\n\"The method introduces extra inference costs on LLMs when searching for the weights.\"\n\n## Answer:\nComputational complexity of our approach is not high at all.  \n1.The additional computation introduced by our method is a one-time cost, and we only need to search for proper weights one time before inference.  \n2.\tBeam search is linear in time complexity for the number of shots. Even for 16-shot setting, searching for weights only needs 3*16 forward passes through language model.  \nSo the computational cost of our approach is totally acceptable. \n\n***\n\n## Q3:\n \"Are there any patterns that exist in the examples if the method assigns high/low weights?”, “How does the order of the demonstrations influence the assigned weights?”\n\n\n## Answer:\nThere is a certain correlation between weights and the positioning of examples, as well as the quality of examples. Examples positioned in the middle tend to be assigned less weight, while those at the beginning and end are assigned more. Besides, experiment result also shows higher-quality examples tend to be assigned more weight, which is consistent with intuition. That’s a good question, and we will add this part to section 5. \n\n***\n\n## Q4:\n“Will the improvement vanish on 70B models? ”\n\n## Answer:\n\nIn my view, the current research on enhancing ICL stems from the weaknesses inherent in the models themselves and their lack of robustness (e.g. sensitivity to examples and order). Compared to smaller models, larger models exhibit superior performance and greater robustness. Therefore, applying our method to a more powerful 70B model will result in a smaller improvement. However, this improvement won't vanish, as even the 70B model's ICL capabilities are not yet perfect. From another perspective, our approach is an economical and straightforward method that to some extent narrows the ICL performance gap between large and small models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method proposed in the paper is simple and easy to follow.\n2. The MSP score seems to be a good metric to evaluate the in-context learning performance under more circumstances.\n3. The improvement of the weight searching method is constant across different datasets and model sizes.\n4. The analysis provides valuable insights on the reweighting approach.", "reasons_to_reject": "1. The experiments are mostly conducted on text classification tasks. How the method works on the text generation tasks, which are more used in current LLMs, is not clear.\n2. The method introduces extra inference costs on LLMs when searching for the weights.\n", "questions_to_authors": "1. What are the weights look like found by WICL? Are there any patterns that exist in the examples if the method assigns high/low weights?\n2. How does the order of the demonstrations influence the assigned weights?\n3. The method seems more effective for small models than large models: for 8-shot, +6.7 on 335M, but +2.3 on 13B; for 16-shot, +8.5 on 335M but +4.3 on 13B. Following this tendency, will the improvement vanish on 70B models? (Of course, the improvement on the 13B model is also meaningful in practice. You can only share your thoughts if the resource is limited for larger experiments.)", "paper_topic_and_main_contributions": "This paper studies the weights of different demonstrations in in-context learning. They design a score to evaluate the quality of demonstration weights and search for approximately optimal weights to improve the final in-context learning performance. The contribution is ``approaches for data and compute efficiency''.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the authors present a simple yet effective algorithm to reassess possibly disparate importance of few-shot examples. To determine optimal weights for better In-Context Learning, the proposed algorithm seamlessly utilizes Masked-Self Prediction (MSP) scores and beam search, subsequently differentiating the importance by rescaling attention weights. To elaborate the current version, nevertheless, take into consider the following valuable points from our reviewers.

1)	Clarify the setting and relationship with demonstration selection: The current paper presumes the absence of any held-out validation data. Such “true few-shot setting” must be better clarified on the main draft.
2)	Stronger baselines: Although using held-out data could overwhelm the effect of using MSP and may not be congruent with a true few-shot setting, it remains pertinent to enrich the paper with additional experimental results. This could involve reweighting with an additional validation set or with example retrieval.
3)	Additional analysis on ordering sensitivity and inference costs: It would be advantageous to illuminate the interplay between the weights and the orders of few-shot example. In addition, considering the method predominantly operates with a higher number of shots, real-time inference cost (beyond asymptotic complexity) becomes relevant. Kindly incorporate some analysis.
4)	Potential implications for generative tasks: It would be beneficial to include some insights or preliminary thoughts when we apply the proposed method to generative tasks.
Decision: Accept-Findings