Representative Demonstration Selection for In-Context Learning with Two-Stage Determinantal Point Process
Authors: Zhao Yang
Yuanzhe Zhang
Dianbo Sui
Cao Liu
Jun Zhao
Kang Liu
Venue: EMNLP 2023 Main (2023)
Keywords: In-Context Learning
Representative Subset Selection
Determinantal Point Process
TL;DR: This paper invertigates how to select representative demonstrations for in-context learning and propose a two-stage DPP selection frame to solve this problem.

Abstract: Although In-Context Learning has proven effective across a broad array of tasks, its efﬁciency is noticeably inﬂuenced by the selection of demonstrations. Existing methods tend to select different demonstrations for each test instance, which is time-consuming and poses limitations in practical scenarios. Therefore, this study aims to address the challenge of selecting a representative subset of in-context demonstrations that can effectively prompt different test instances in a speciﬁc task. We propose that this representative subset should be of high quality and diversity. Our empirical analyses conﬁrm that demonstrations that meet these criteria can indeed bolster model performance. To satisfy these criteria, this paper further introduces a two-stage Determinantal Point Process (DPP) method designed to incorporate both quality and diversity in the process of demonstration selection, thereby obtaining representative in-context demonstrations. Through comprehensive experimentation, we have conﬁrmed the efﬁcacy of our proposed method, paving the way for more practical and effective In-Context Learning.

Representative Demonstration Selection for In-Context Learning with Two-Stage Determinantal Point Process
Zhao Yang!., Yuanzhe Zhangl? Dianho Sui', Cao Liu", Jun Zhao! Kang Liu 12,55
'School of Artificial Intelligence, University of Chinese Academy of Sciences
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, 3 "Meituan, Beijing Haabii Institute of Technology, Weihai, 'Shanghai Artificial Intelligence Laboratory (zhao. yang, yzzhang, jzhao, kliu)@nlpr: ia an suidianbo@hit t. edu. cn, liucaoêmei tuan. com
Abstract
Train Set
Test
Train Set
Although In-Context Learning has proven ef- fective across a broad array tasks, its effi- ciency is noticeably influenced by the selection demonstrations. Existing methods tend to select different demonstrations for each test in- stance, which is time- consuminn and poses lim- itations in practical scenarios. Therefore, this study aims address the challenge of selecting a representative subset of in-context demon- strations that can effectively prompt different test instances in a specific task. We propose that this representative subset should be of high quality and diversity. Our empirical analyses confirm that demonstrations that meet these criteria can indeed bolster model performance. To satisfy these criteria, this paper further in- troduces a two-stage Determinantal Point Pro- cess (DPP) method designed to incorporate both quality and diversity in the process of demonstration selection, thereby obtaining rep- resentative in-context demonstrations. Through comprehensive experimentation, have con- firmed the efficacy our proposed method, paving the way for more practical and effective In In-Context Learning.
Irstane
Fraance
Irstares 2
Freance
Instarcs
Retrieval-based In-context
Representative In-context Demonetration Selection
Demonetration Selection
Figure 1: Comparison between existing retrieval- braseval-based in-context demonstration selection and our representa- tive in-context demonstration selection.
2022; Rubin al., 2022), demonstration formats (Min et al., 2022), demonstration labels (Zhao al., 2021; Yoo et al., 2022), and demonstration order (Lu et al., 2022). Among these factors, demon- stration selection is the most significant, as they provide only information about the task (Liu et al., 2022). Therefore, selecting suitable demonstra- tions is necessary.
Existing demonstration selection methods usu- ally adopt retrieval based solutions, which aim to select different demonstrations for each test in- stance (Liu et al., 2022; Sorensen al., 2022; Rubin al., 2022: Zhang al., 2022a). How- ever, these retrieval-based methods can be costly in practice. In detail, these methods can only per- form inference on one sample at a time, which incurs high costs in terms of token usage, particu- larly when considering leveraging batch prompting (Cheng et al., 2023). Besides, performing demon- stration selection for each testing example is obvi- ously time-costing ($4.3). Based on this, shown in Figure 1, this paper aims select a representa- tive demonstration subset from a pool of training examples for all test instances in a specific task rather than each test instance.
1 Introduction
The emergence of Large Language Models (LLMs) has swept various Natural Language Processing (NLP) tasks and brought new research paradigms (Bommasani al., 2021; Wei et al., 2022: Ope- nAI, 2023). In-Context Learning (ICL) (Brown et al., 2020) the primary paradigm for leverag- ing LLMs, which enables the model to generalize rapidly from a few examples without parameter update. Recent studies have shown that ICL has significantly improved over the zero/few-shot set- ting and even surpassed the performance of full data fine-tuning (Dong al., 2022).
However, ICL is insufficiently stable and robust in performance when compared to traditional fine- tuming, and exhibits sensitivity to numerous fac- tors, including demonstration selection (Liu et al.,
To select representative demonstration subset for a task, we argue that there are two criteria that should be satisfied: quality and diversity. Firstly, in ICL, the quality of an instance is defined as the

---

degree that it can help the LLM make correct predictions. Intuitively, high-quality instances can improve the LLM's performance on as many test examples as possible. Secondly, diversity means that the elements in the selected demonstration sub- set should be mutually dissimilar and represent the training set as comprehensively as possible. More- over, in this paper, we argue that diversity can be refined into semantic diversity and influence diver- sity. Semantic diversity means that the selected examples should be diverse enough to cover more semantics or expressions in the given training set, which is consistent with existing representative sub- set selection in traditional deep learning (Coleman et al., 2020; Guo et al., 2022). Influence diversity means that each selected example should help the model correctly classify diverse set of testing sam- ples, enabling the total demonstration set to support the correct classification of more lest instances for the given LLM. Pilot experiments ($2) also ver- ify that the demonstration subsets satisfying these criteria could improve ICL performance.
the quality scores ($2.2) of these examples to this DPP. As result, both quality and influence diver- sity could be optimally introduced in the selection process. This two-stage DPP selection method en- ables the selected demonstration subset could not only cover more semantics in the training set but also help the given LLM classify more examples with better performance. Our contributions can be summarized as follows:
This paper aims to select representative in- context demonstrations and propose two cri- teria (quality and diversity) that such demon- strations should satisfy.
To satisfy the proposed criteria, this paper pro- poses a two-stage DPP selection method. Ac- cording to our method, the selected demon- strations could not only be representative of the large training set but also facilitate the model's 's generalization to new instances.
Extensive experimental results demonstrate the effectiveness of our proposed method. And we also show the significant advantage in inference time and token usage compared to retrieval-based methods.
In this paper, we make the following efforts to meet the above criteria. To achieve the quality criterion, we resort to recent instance-based ex- planation methods (Koh and Liang, 2017; Pruthi et al., 2020). Different from them, this paper tends existing instance- based explanation methods ICL and defines the influence score to measure the contribution of training example to the correct classification of another example. To achieve the diversity criterion, this paper seeks the help of the determinantal point process (DPP), which is prob- abilistic model that could maintain high diversity among different instances by pairwise repulsion between instances (Kulesza et al., 2012).
2
Pilot Experiments
This section explores whether the proposed criteria could improve the performance of ICL. In specific. we measure the impact of three factors: semantic diversity, instance quality, and influence diversity. We choose SST-2 (Socher et al., 2013) as the test dataset and conduct 4-shot experiments on GPT2- xl (1.5B) (Radford al., 2019), GPT-J (6B) (Wang, 2021) and GPT-NeoX (20B) (Black et al., 2022).
2.1 Impact of Semantic Diversity
Moreover, to incorporate both high quality and diversity for demonstration selection, this paper fur- ther proposes a two-stage DPP selection method. In specific, in the first stage ($3.2.1), we utilize the se- mantic information measure instance similarity and then leverage DPP to obtain a subset of instances with high semantic diversity. In the second stage ($3.2.2), we first compute the quality score of each instance in this subset based on our defined influ- ence scores ($2.2), which could reduce significant amount of meaningless inference time compared computing the quality for each training example. Then we utilize the obtained influence scores to measure influence similarity between instances and construct another DPP. Besides, we also introduce
To explore the impact of semantic diversity, we firstly utilize sentence-BERT (Reimers and Gurevych, 2019) to encode all train instances. Then, with these sentence representations, we use K-means (Hartigan and Wong, 1979) to par- tition all train instances into 4 clusters. Finally, we design three strategies with different seman- tic diversity: 1) single-cluster: we sample 4 examples from each cluster. 2) random: ran- dom sampling 4 examples from the whole train- ing set. 3) multi-cluster, we sample one ex- ample from each of these clusters. Obviously, multi-cluse has the best semantic diversity and single-clusted has the worst diversity.

---

andleciate Clatt"
high auuly runders
Audras Arutgs Accanco 1h
13
40
UPT 1900
WWw. 2065
48
Wai 1000
4872
1000
(a) Impact of semantic diversity
(b) Impact of instance quality
Impact of influence diversity
Figure 2: Comparisons of the three factors: semantic diversity, instance quality, and influence diversity. We list the 4 shot performance on SST-2 for three LLMs: GPT2-xl (1.5B), GPT-J (6B), and GPT-NeoX (20B).
Figure 2(a) presents the corresponding results of the five sample strategies. From the figure, we observe that mul tilcuulu is always superior to the other strategies across these three LLMs. According these results, we could find that better semantic diversity could lead higher accuracy across models of different scales.
To improve efficiency, we randomly choose 100 examples and measure their quality by randomly sampling additional 200 samples as the score set. Then design three strategies with different quality: 1) high-quality: we choose the top 4 examples with the highest quality as demonstra- tions. 2) low-quality. ty: we choose the bottom 4 examples with the lowest quality as demonstrations. 3) random: we randomly choose 4 examples from these 100 examples as demonstrations.
2.2 Impact of Instance Quality
To measure the quality of an instance, we need to define the influence score of an instance at first. Existing example-basee explanation methods (Koh and Liang, 2017: Pruthi et al., 2020) define the influence of an example diction difference of the test set between the model trained with and without this example. Inspired by this, we define the influence of the demonstration e on another example = (zi, ya) in ICL as follows:
Figure 2(b) shows the experimental results. From these results, find that demonstrations with higher quality could get better performance.
training set as the pre-
2.3 Impact of Influence Diversity
With the same setting in $2.2. we have obtained the influence scores on 200 score instances, which could be seen as a 200-dimension vector. Based on this, we can get the influence vector for each instance and then we utilize the similar cluster method in $2.1 to partition the 100 examples into 4 clusters. We construct the following sam- pling strategies with different influence diversity: 1) simi from each cluster and average their performance. 2) diivers-innfleneee
Inf(e, e,) (uule 2.) (x. (1)
where is the golden label of example Ii. PLM(yile, and (3,(a)) refers to the output probability of yı conditioned on (e, v) and for given language model LM, respectively.
sample 4 examples
Inf(e, ej) could measure the contribution e to the correctness of the example e,. Based on this, we define the quality metric of the demonstration as follows:
we sample one example from each cluster as demonstrations. 3) random randomly sample 4 examples as demonstrations. Figure 2(c) shows the corresponding results. Ac- cording to these results, we observe that demonstra- tions with better influence diversity could improve ICL performance for LLMs of different scales.
Q(e) DeseDao Inf(e, ei)
(2)
T
where Decore denotes the set used for measur- ing quality and T refers its size. Q(e) character- izes the contribution of the model performance on the score set. The higher the value of this metric, the more e can help the language model making correct predictions on this score set.
2.4 The Conclusion on Pilot Experiments
Our three experiments show that semantic diver- sity, instance quality, and influence diversity all can improve ICL performance, which verifies the ef- fectiveness of the proposed criteria. Therefore, our

---

method would try to introduce these three factors in the selection process.
Obviously, the maximum a posteriori inference in DPP has high time complexity (Kulesza and Taskar, 201 1). Following Chen et al. (2018), the time com- plexity of selecting a subset of size is O(K² M).
3 Method
In $3.1, we illustrate the formulation of in-context learning and the background of the determinantal point process. Then, introduce the implementa- tion of our method in $3.2, which could introduce the three factors in the whole selection process.
The core of DPP is how to obtain the correspond- ing PSD matrix. Existing studies usually incorpo- rate task-relevant information into this matrix, in order to utilize DPP to achieve diversity.
3.2 Two-Stage DPP Selection Method
3.1 Preliminary
In this section, we introduce novel two-stage DPP selection method, which could introduce semantic diversity, high quality, and influence diversity in demonstration selection.
3.1.1
In-context Learning
In-context learning the core emergent ability of LLMs and only requires few examples and corre- sponding labels solve a task. Formally, demon- stration E = (xi, 94) consists of an instance It and its label yt- For a new test instance Itest: K- shot in-context learning generate its label Upredict as follows:
3.2.1 First Stage: Selecting Candidate Subset with Semantic Diversity
According to Equation 1, computing the influence score requires inference time. Thus, computing the influence score for each instance in the training set is infeasible for its high costs. Intuitively, we need to reduce the size of the that requires calculating influence scores.
PLM (Upredict C1, C2, ... CKK Fiest)
(3)
where e; is sampled from the whole train dataset and (er, "2 ,Ek) is usually called context.
Based on this, this paper selects a small sub- set from the original training set by introducing semantic diversity. Formally, the training set D {e1, 62, ,NN) contains N instances where ej = (a;, V:). We firstly utilize sentence-BERT (Reimers and Gurevych, 2019) to encode these in- stances and obtain their semantic representations f21,22, NN). Then we could easily get the dataset representation
3.1.2 Determinantal Point Process
Determinantal Point Process (DPP) is an ele- gant probabilistic model with the ability to ex- press negative interactions (Kulesza et al., 2012). DPP could help us find a representative subset while keeping high diversity among different items (Chen et al., 2018). Formally, for an index set A (1, .M and its corresponding item set T {a1, .2.... .ar) DPP provides prob- ability mcasurement for N item subsets. Given the feature representation a; for the item a, DPP computes a positive semi-definite (PSD) matrix L E M where Lij- k(a,,a,) for an ker- nel function k(.,)) and Lij usually measures the correlation between and aj. Based on this, if a subset is determined by the index set Y C A, the probability of selecting Y could be defined as follows:
Nxd by stack, where the dimension of each sentence. Finally, we could get the PSD matrix Lg in DPP as follows:
Ls = ss
(6)
Obviously, Ls NxN is real symmetric matrix. With Lg, could obtain candidate subset Dsem of size Nsem according to Equation 5.
Selecting Demonstrations via High Quality and Influence Diversity In previous parts, we have introduced semantic di- versity into D In the following parts, we would illustrate how to select high- instances while maintaining influence diversity to form the final demonstration set.
3.2.2 Second Stage:
det(Ly) P(Y) det(L II)
(4)
where Ly refers to a submatrix of L and consists of Lij for i.j E Y. 1 is an identity matrix and det(-) is the determinant of a matrix.
Firstly, we need select the score to measure the quality of an instance in Psem according Equation 2. Recent probing studies (Min et al., 2022; Yoo et al., 2022) show the importance (5) the label for ICL. Therefore, we only require the
Under this distribution, resentative subset Yoest of size follows:
could select the rep-
Yoest argmax dyy(Ly)

---

Algorithm Two-Stage DPP Selection
this paper, only focus on demonstration selec- tion and just determine the order in ascending order of their quality scores D sccore.
Input:
{er y language model LM, didate subset size Nem. set size T. demonstration size sentence encoder sbert.
Training
Output: Demonstration Suhset Dass for e; do
4 Experiments
2, shert(a))
3: end for
4.1 Experimental Setups
22, NN)
4.1.1 Datasets
5: Lg = ss"
6: Deom = arg M dedl Le
deet Ly)
Following previous studies (Zhao et al., 2021; Liu et al., 2022), we conduct experiments on six text classification tasks: SST-2 (Socher et al., 2013), TREC (Voorhees and Tice, 2000), CB (De Marn- effe et al., 2019), AGNews (Zhang et al., 2015), DBPedia (Zhang et al., 2015) and RTE (Dagan et al., 2006). And we utilize accuracy as the evalu- ation metric. Statistics and the prompt format for each dataset could be found in Appendix A.
random sampling Dave size? T from D/
for es E "Dam do
9: for é. E Drom do
10: Inf(es, e,) (an(usle, PLM(Usl
11: end for
12: Q(es) Le inf
13: 6 =IInf(es, 61); enf(er,e)).. Inf((,,
14: end for
15: stack(e1, e2, eNaa)
16: Q = (Qay Qry
Qex.n
17: L, = Q -1" Q
18: Daa arg maxyg |Y dell
K
Lx)
4.1.2 Large Language Models
19: return dem
This paper conducts experiments three large lan- guage models of different scales: GPT2-xl (1,5B) (Radford et al., 2019), GPT-J (6B) (Wang, 2021) and GPT-NeoX (20B) (Black al., 2022).
examples in the scoring set to include all of the labels. With this premise satisfied, obtained the set Soore of size T through a random sample from D/ D/Dssem. With the score set, we could easily compute the influence score and quality of each instance in Deem according to Equation and 2. For item Ij E Deem= we could obtain T- dimension influence embedding Ij where each di- mension denotes its influence on each instance in the score set and its quality score Q(e,). Then, can obtain the total influence representation ma- trix I E R XT and the quality representation Q T by stacking and contacting operation. Thus we can compute another PSD matrix L as follows:
4.1.3 Baselines
In this paper, compare method with the fol- lowing baseline methods: Representative Subset Selection in Deep Learning: Least Confidence (LC) (Coleman et al., 2020) seleets representa- tive examples according to the max probability on all labels. Cal (Margatina et al., 2021) se- lects representative examples with the biggest logits divergence from neighbors. Recent Rep- resentative Demonstration Selection Methods: Random selects demonstrations by randomly sam- pling. Cluster (Zhang et al., 2022b; Gao et al., 2023) selects demonstratinns by choose cluster centers of K-means cluster by semantic. DPP (Levy et al., 2023) utilize the semantic information select k demonstrations according to k-DPP (Kulesza and Taskar, 2011; Chen et al., 2018).
L, -1 .Q
(7)
Actually, IIs could construct basic PSD ma- trix to help introduce influence diversity. In- spired by Chen al. (2018), we further introduce Q to help us select high-quality instances. In this way, DPP which is based on L, could help us select high-quality and diverse examples. Concretely, for k-shot ICI,, we can select a demonstration subset Ddem of size k according to Equation 5.
4.2 Main Results
Table I presents the corresponding results. Tra- ditional representative subset selection methods get poor performance in ICL settings. Least Conf fonfidence even gets worse performance than random selection and Cal just gets comparable performance with random selection. Our designed baselines Cluster and DPP, which both introduce semantic diversity, achieve some improvements compared to random selection. And DPP could
In summary, Algorithm presents the whole process of our two-stage DPP selection method. Figure 9 shows more direct illustration of our method. Besides, recent study (Lu al., 2022) shows the ordering of demonstrations also has a significant influence on the performance. But in

---

SST-2
Tree
C8
AGNews
D8podin
RTE
AVG
Model
Method
2/4/8
2/4/8
2/4/8
2/4/8
2/4/8
2/418
2/4/8
LC
52.14/ 53.72/55.7
18.20/2244036.22 20.0000000000 43.78/44.85466 75 37, 37.62/38.4844
39.45446.741522 08
48.95/51.02/50.21
40.0224.886477.0
Cal
53 33.88866044499111
19.10/24 50138,
86/46.55466
41 02/42222 S845.12 12
36/61, $2899.56
50.22/49.355500
89
44.07146. 7015 58
GPT2xx
Random
22.33/55.27/56.99
07.36/3376/37..1
46.43/46.47/46
1..0041886/33.4
55.80/61.2488 24168,006 06
51.84/52.20/52
56
44.04/46.80
80/50.87
1.5B
Cluster
53.08/55,43660.77
76/52.15 63.57/64
59,25/67
51.33/50.96/5L.D
DPP
13.02/56.12/63.59
18.21V27 60050.80
47 07//55000055..
36 63 63.8866 72097 86 58. 38.48/69.44778777
$1.52/11011/334
48.70/53.4865
Ours
50 3177744317722440
52.68/38 60099866087722
91/76.2
53.07/53.99955
54.49799.07655
LC
07784,7698802
10/0.60
A5/55.22//55
53.87/54
00-96676.66//800 50.07/51 881/51888//2222 24 50.99/59
27/63
Cal
84.2589.04/99.55
37.02/38.6/44
55/58 04 56.65/58 7209927
64.43/1 .6983.58
52.02/52,96/5 9.499494
53,74162,
66866 36
GPT- 68
Randon
01120/88.57/899
36.80/38. 08/44.00
56/53.57/55.36
55,74/56 80V67.19 1 62. 2..23880.666 46/8144
55.02/56.31/53.88
53.23362.30M65 22
Chuster
85, 34/88 96990.00
57.22/39 20/46.6
24/52 87/58 15 8867 8371 00 68 44/13, 52/84.95
11.88/22445/54.00
83164 19667 55
DPP
86. (6.33289.66999 022899667/9088
38.40/40.60/53.40 40
47 47156. 10/62 74/26/76 30/80.54
28/88668//8895 95
52.65/54, 22.65/54.90063.11
18
62. 62.36/67.31/73
58.02/67 60/72.77
66/64
79 66/82
80.45/89 26992.47
53.02/55.44/56.67
(48.1474,160 16/77/
L.C.
82, 02.55/85.4411 .47/89
42.18/81 53/60.88 89 46.99188, 74 43/22
70.08/18, 24/33.4
21..82886666 67/57 88
99/71.99/11
Cal
14140/90.01/9..00
74.72/54488/6.999
50.05/61..7/62 18 73 23.62775.4777 47/79.15
20/83.9//44.22
$6.44/57.05/577 05/57.5777.99 .292
63.9 63.91/70.31/73.
GPT--NoX Random
83.57/89, 25/92.89
81163,
22.1366.,716 OU
74, 1373
63.55/69, 73/22
20B
Chasser
84.48/90.39/9 3.002
44 44.88/55.
12/6..77
S1. 27/62.5 75. 75.89/76 78/90.42 74 74.05822.56884
57,40/58
02/55///////88... 34
64.65/70.69/73
DPP
86. 19..999999999411
26.00/57.20065..0 4D 50.00/62. 93/63 29 77.36/80 34/3344 80.76/89.3277 32991.988..08 58.48/56.48853
87
53/73,24/75.91
Ours
88. 88.76/93.8094445
61.12/71.0879996 54.18/65.56/9922 31.57/84776/86.688
84.14/92.27/93.66 38.43/58.87/59.2
71.37/77.2/88005
Table 2-shot/4 shoot/4-shot/8-eee shot performance comparison on six datasets across three LLMs. AVG denotes the average performance of the whole six datasets.
dings (Liu et al., 2019). Figure 3 shows the average 8-shot performance of the six datasets. From these results, it is not surprising that these two methods could achieve better performance. This is because they both introduce information about the test set while our method does not.
BM25
0UF5+8N25
as
80-
75-
Bahet
70
65-
56
To make a fair comparison, we also try to incor- porate test set information into our method. Specifi- cally, we used a retriever to find the most similar ex- amples from the training set to construct the score set Dscore for using the BM25 retriever in BM25 and the Roberta retriever in Kate as Our5+BM25 and Ours+Emb, re- spectively. According to the results, these two vari- ants achieve further performance improvements and the performance gaps with retrieval methods decrease as the model size increases.
50
GPT2 (1.58)
GPT-J (6B)
GPT-NeoX (208)
Figure 3: Comparsions average 8-shot performance of the six datasets across the three LLMs.
method. We name the variants
achieve better performance compared to Cluster, which indicates that DPP is the better choice to in- troduce diversity. Compared to these baselines, our methods significantly improve the performances among all datasets in different settings across these three LLMs. In specific, compared to random, our methods bring more than 10% improvements across the three shots for GPT2-xl and GPT-J and 8% improvements across the three shots for GPT- NeoX. Compared to DPP (only stage 1, $3.2.1), which is the strongest baseline, our method also achieves an improvement ranging from 4% to 7% under different settings, which verifies the effec- tiveness of the second stage ($3.2.2).
Benefits of Our Method Actually, it is inevitable that the performance of representative demonstra- tions is inferior to that of retrieval methods, as they provide different demonstrations for each test instance. However, providing the same demon- stration subset (our method) can save significant amount of resources in real-world scenarios (Cheng et al., 2023). In specifie, if the demonstration sub- set is kept the same, they can be pre-encoded', thus saving significant amount of time spent on demon- stration encoding when prediction. Figure 4 shows a comparison of the time spent on inferring 5000 instances on AGNews. With the support of pre-
4.3 Comparison with Retrieval Methods
Performance Comparison We compare our method to the retrieval-based methods BM25 (Wang et al., 2022) and KATE (Liu et al., 2022), which re- trieve the demonstrations according to the BM25 scores (Liu et al., 2009) and the RoBERTa embed-
'This paradigm has been widely implemented in Huggingfacc and these pre-encoded sentenees are called past_key_vall values in current available LLMs.

---

BM25
KATE
Ours+batch prompting
variant
SST-2
Trec
AGNews
2-shot
15357
15349
4093
random
54,86
26.11
42.07
4-shot
26653
26677
4160
sem_div
57.58
32.20
66.82
8-shot
49241
46274
4160
ins_qua
61,93
37.08
68. 11
inf div
54.26
29.47
65.97
Table 2: Comparisons of token costs when testing Trec with text-davinci-022 API, whose maximum input
sem sem__iv div__div- + qus_uaa
63.86
39.92
69.88
sem div + inf div
59.29
34.87
68.08
ins qua inf div
62.47
37.76
68.75
length 4097. We list the results of three shots.
sem div + ins qua div (ours) 65.52
42.83
72.51
1750
BM22
Table 3: Effect of the three factors: semantic diversity (sem_div), instance quality (ins_qua), and influence diversity (inf_div) We list the average performance among the three shots for GPT2-xl.
Ours
1000
750
500
1000
2000
3000
4000
5000
Number of Instances
6.53
6.46
Figure 4: Comparsions of inference time of 5000 in- stances from AGNews for GPT-J.
635
6:22
Randow D/p Oum
Randor C49
Durt
Randow DP0
Cur
Kander Oue
Figure 5: Order sensitivity GPT2-xl. We conduct 4-shot experiments on SST2, Trec, CB, and AGNews and show the results of the total 24 orders.
encode, only about 1/3 of the inference time is re- quired. API calling scenario, the cost calcu- lated based on tokens. Cheng et al. (2023) proposes batch prompting, which could generate responses for multiple examples in one batch using the same demonstration set. We calculate the token costs Ta-
the influence diversity would have relatively posi- tive effect when appears with instance quality.
when testing Tree with text-davinci-Oee ble 2 presents the comparisons. With batch prompt- ing, ICL with the same demonstration subset shows a significant advantage token usage, requiring only 1/10 of the token costs of the retrieval-based method for 8-shot learning.
5
Discussions
5.1 Order Sensitivity of Representative In-Context Demonstrations
Therefore, we believe that representative demon- stration selection is the more practical paradigm, which could save significant amount of inference time and token costs.
Lu et al. (2022) shows that ICL performance is highly sensitive to the demonstration order. There- fore, we also test whether the carefully selected representative demonstrations suffer from the order problem. In specific, we compare our method with random selection and the strongest baseline DPP.
4.4 Effect of Factors: Semantic Diversity, Instance Quality, and Influence Diversity
this part, we explore the effect of the proposed three key factors in our selection method: semantic diversity, instance quality, and influence diversity.
Figure presents the corresponding results on GPT2-xl. From these results, we could observe that DPP could reduce the order sensitivity of ICL and method further reduce the order sensitivity. According to this, conjecture that the selected representative in-context demonstrations could ef- fectively characterize the corresponding task, lead- ing to reduction in order sensitivity of ICL, which is also consistent Chen et al. (2022).
Table 3 presents the corresponding results and Appendix B shows the detail of cach baseline. From these results, find that instance quality is the most important factor, which contributes sig- nificantly to performance improvement. Semantic diversity also plays a significant role, which is con- sistent with the results of Cluster and DPP. And

---

random
GPT2-XI (1.5B)
GPT-J (68)
GPT-NeeX (208)
Aserge Acourcc IN
60
40-
20
OP72 (782M)
GPT2 (M
1981
OPF MoUS dawinc (3798) davino (1798)
Figure 6: Transferability of representative demonstrations. We test the representative demonstrations selected based on GPT2-xI, GPT-J, and GPT-NeoX models on six different models GPT2-large, GPT2-xI, GPT-J, GPT-NeoX, text-davinei-e02 and text-davinci We show the average performance on Trec 2-/4-/8- shots.
5.2 Transferability of Representative In-Context Demonstrations across LLMs
Cumulative Distribution
1.0
Curulativn Probabitty
0.8
In this section, we investigate the transferability of the selected representative demonstrations.
In specific, we utilize the demonstrations se- lected based on the original model to test its per- formance on the new model. We choose GPT2- xl, GPT-J, and GPT-NeoX which were employed in previous experiments as the original models, and GPT2-large, GPT2-xl, GPT-J, GPT-NeoX, text-davinci-002, and text-davinci-009 as the new models, whose scale varies from 762M to 175B. Figure shows the corresponding results. From these results, observe that the selected representative demonstrations all achieve better per- formance compared to random selection, which verifies their transferability. Besides, we also find that the transferability will be better when the size the original model and the new model is closer.
0.53
0.55
0.65
6.65
0.70
0.75
0.66
385
SST-2 Accuracy
Figure Cumulative distribution of the C(10,4) 210 4-shot combinations on SST-2. And we sign our method with the red line, which outperforms more than 90% proportion of combinations.
As shown in Figure tive distribution of the 210 combinations. And our method is signed with the red line. From the results, we can find that our method outperforms 90% combinations, which reveals the effectiveness of our method.
show the cumula-
5.3 Assessing the Quality of the Selected In-Context Demonstrations
5.4 Effect of Subet Size Dxem
In this part, we explore the quality of the selected in-context demonstrations. Considering that it is not feasible to enumerate all possible combinations in training dataset, conduct simulations under the following conditions.
In Algorithm 1, get subset of size Dsem after the first stage DPP and select high-quality instances from this subset. However, is inevitable filter out some high-quality instances in the first stage. Therefore, we explore the effect of the subset size in this section. In our main experiments, we set 'Dseml 200, and we show the performances of more value here and show the results of SST-2 on GPT2-xl in Figure 8.
Specifically, we sample 10 instances from SST- 2, yielding C(10,4) = 210 possible combinations for 4-shot experiments. Then we test each of these 210 combinations the corresponding test accuracy for each combination. Finally, we examine the ranking of the combina- tions selected by our method among all possible combinations.
and obtained the
We could observe that our method achieves bet- ter performance with the bigger subset size across different shots. This indicates bigger subset size

---

demonstrations, which could prompt each test ex- ample with the same demonstrations.
70
6.2 Determinantal Point Process
OCT-3
Determinantal Point Process (DPP) is an elegant probabilistic model that could select representative subsets while maintaining high diversity among each instance (Kulesza et al., 2012). Such effi- cient method has been applied to introduce diver- sity in various tasks: objection detection (Azadi et al., 2017), recommendation (Chen et al., 2018), summary (Cho et al., 2019), and parsing (Shi al., 2021). More recently, Levy et al. (2023) using DPP in composition tasks, sampling a diverse subset of in-context examples to cover more sub-phrases, which is task-specific and dependent on the specific model. Ye et al. (2023) improve the retrieval based methods by introducing diversity into the retrieved examples via DPP. However, this paper focuses on totally different paradigm.
50 100
150
200
250
300
Figure 8: Effect of the subset size of SST-2 for GPT2-xl.
could introduce more high-quality instances in the candidate set. Based on this, the second stage could select better demonstrations. However, could also observe that the performance improvements diminish significantly once the size exceeds 200. Considering that the bigger size would introduce much more inference costs when computing qual- ity scores, this paper just selects 200 to conduct experiments for efficiency.
7 Conclusion
6 Related Work
This paper aims to address the challenge of select- ing representative demonstration subset and pro- poses two criteria (quality and diversity) that such demonstrations should satisfy. And we further pro- pose two-stage DPP method to incorporate both high quality and diversity in the selection process. Extensive experimental results show the effective- ness of our method and the significant advantages compared to sarieval-basee methods.
6.1 In-Context Learning
Existing studies have shown ICL sensitive to demonstration selection (Liu et al., 2022), demon- stration formats (Dong et al., 2022), demonstration labels (Min et al., 2022; Yoo al., 2022), and demonstration order (Lu et al., 2022). This paper focuses the selection of demonstrations.
Existing methods aim find different demon- strations for each test instance by retrieval-ban based methods. These methods can be divided into two categories based on whether the retriever requires training (Dong et al., 2022). The training-free meth- ods utilize the sentence representations (Liu et al., 2022), BM25 (Wang al., 2022), mutual informa- tion (Sorensen al., 2022), and perplexity (Go- nen et al., 2022) select demonstrations. As for methods that require training, Zhang al. (2022a) utilized reinforcement learning to train retriever. Rubin et al. (2022) trained an example scorer using contrastive learning with signals from LLM. More recently. Li et al. (2023) integrated different tasks to train a universal retriever, further enhancing the performance of retrieval methods.
Ackonwledgements
This work was supported by the National Key R&D Program of China (20227D0160503), the National Natural Science Foundation of China (No.62276264 and No.62306087), and the Natural Science Foundation of Shandong Province (Grant No. ZR2023QF154), This research was also sup- ported by Meituan. And thank the reviewers for their valuable feedback.
Limitations
The main limitation of this paper is the proposed method could not be transferred into the black-box scenario such the GPT-3.5 family. Existing Ope- nai APIs only provide the log probability of the top-5 tokens, which leads to inaccurate calculation of the influence score. With such an inaccurate influence score, the accuracy of the values the
However, these methods have certain limitations in real-world applications, especially in more in- ference time and high token costs ($4.3). Based on this, this paper tries to solve a more challeng- ing problem: selecting representative in-context

---

two important factors, instance quality and influ- ence diversity, will also be affected. Therefore, our method is difficult to be transferred to these black-box models.
Laming Chen, Guoxin Zhang, and Eric Zhou. 2018. Fast greedy map inference for determinantal point process to improve recommendation diversity. Ad- vances Neural Information Processing Systems, 31.
Besides, we have realized the defined influence score (Equation 1) is not the optimal choice. We also design more accurate influence score and Equation 1 is just special case. These more ac- curate scores would bring more performance im- provements. However, the computational cost of this more accurate method is several tens of times higher than Equation used in this paper. For large language models, this cost is unacceptable (Yang et al., 2023). Therefore, this paper made trade-offss between efficiency and performance. And we leave the challenge of how to efficiently incorporate the more accurate influence scores into our method as our future work.
Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. 2022. On the relation between sensitivity and accuracy in in-context learning. arXiv preprint arXiv:2209.0766)
Zhoujun Cheng, Jungo Kasai, and Tao Yu. 2023. Batch prompting: Efficient inference with large language model apis. arXiv preprint arXiv:2301.00 0.7721.
Sangwoo Cho. Logan Lebanoff. Hassan Foroosh, and Fei Liu. 2019. Improving the similarity measure of determinantal point processes for extractive multi- documeut summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1027-1038.
Cody Coleman, Christopher Ych, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. 2020. Selection via proxy: Efficient data selection for deep learning. In International Conference Learning Representa- tions.
Ethics Statement
This paper aims to select representative demonstra- tions for in-context learning, and the experiments are conducted publicly available datasets with available LLMs. As result, there is no data pri- vacy concern. Meanwhile, this paper does not in- volve human annotations, and there are no related ethical concerns.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment chal- lenge. In Machine Learning Challenges. Evaluat- ing Predictive Uncertainty, Visual Object Classifi- cation, and Recognising Tectual Entailment: First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers, pages 177-190. Springer.
References
De Marneffe, Mandy Simons, and Ju- dith Tonhauser: 2019. The comnitmentbank: Inves- tigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124.
Samanch Azadi, Jiashi Feng, and Trevor Darrell. 2017. Learning detection with diverse proposals. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7149-7157.
Sidney Black, Stella Biderman, Fric Hallahan, Quentin Anthony, Leo Gao. Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. -pt-neox-20b: An open-source autoregres- sive language model. In Proceedings of BigScience Episode # 5-Workshop on Challenges & Perspec- rives in Creating Large Language Models, pages 95- 136.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.00 10034..
Shuzheng Gao, Xin- -h-ehhnn Wen, Cuiyun Gao. Wenx- uan Wang, and Michael R Lyu. 2023. Constructing effective in-context demonstration for code intelli- gence tasks: An empirical study, arXiv preprint urXiv:2304.07577
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosse- lut, Emma Brunskill, et 2021. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07255
Hila Gonen, Srini Iyer. Terra Blevins, Noah A Smith, and Luke Zeulemoyer. 2022. Demystifying prompts in language models via perplexity estimation. arXiv preprint 22X1:2212.040::3
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 1877-91-1
Chengeheng Guo, Bo Zhao, and Yanbing Bai. 2022. Deepcore: A comprehensive library for coreset se- lection in deep learning. In Database and Expert Systems Applications: 33rd International Conference, DEXA 2022, Vienna, Austria, August 22-24, 2022, Proceedings, Part I, pages 181-195. Springer.

---

John A Hartigan and Manchek Wong, 1979. Algo- rithm as 136: A k-means clustering algorithm. Jour- nal of the royal statistical society. series (applied statistics), 2801):100-108
2021 Conference on Empirical Methods in Natural Language Processing, pages 650-663.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe,
Mike Lewis, Hanmanch Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048- 1064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894. PMLR.
Alex Kulesza and Ben Taskar. 2011. k-dpps: Fixed-size determinantal point processes. In Proceedings of the 28th International Conference on Machine Learning (ICMI-11), pages 1290-1200.
OpenAL. 2023. Gpt-4 technical report.
Garima Pruthi. Frederick Liu, Satyen Kale, and Mukund Sundararajan. 2020. Estimating training data influ- ence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920-1992
Alex Kulesza, Ben Taskar, et al. 2012. Determinantal point processes for machine learning. Foundations and Trends@ in Machine Learning, 5(2-3):123-286.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8:9
Itay Levy. Ben Bogin, and Jonathan Berant. 2023. Di- verse demonstrations improve in-context composi- tional generalization. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1401- 1422, Toronto, Canada, Association for Computa- tional Linguistics.
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert Sentence embeddings using siamese nett-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992.
Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni. Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023. Unified demonstration retriever for in- context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 4644 4668, Toronto, Canada. Association for Computational Lin- guistics.
Ohad Rubin. Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655-2671.
Jaachang Liu, Dinghan Shen, Yizhe Zhang, William B
Dolan, Lawrence Carin, and Weizhu Chen. 2022.
Tianze Shi, Adrian Benton, Igor Malioutov, and Ozan Irsoy. 2021. Diversity-aware batch active learning for dependency parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2616-2626.
What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (Dee- LIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures. pages 100-114.
Tie- Yan Liu et al. 2009. Learning to rank for informa- tion retrieval. Foundations and Trends in Informa- tion Retrieval, 333)233--31.
Richard Socher, Alex Perelygin, Jean Wu. Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 1631-1643
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: arXiv:1907.11 9999.11.9...
Taylor Sorensen, Joshua Robinson, Christopher Rytting, Alexander Shaw, Kyle Rogers, Alexia Delorey, Mah- moud Khalil, Nancy Fulda, and David Wingate. 2022. An information -eaematton-thhoreiicc theoretic approach to prompt engi- neering without ground truth labels. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 819-862.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Pupers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Ellen M Voothees and Dawn Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR confer- ence on Research and development in information retrieval, pages 200-207.
Katerina Margatina, Giorgos Vemiikss, Loic Barrault, and Nikolaos Aletras. 2021. Active learning by ac- quiring contrastive examples. In Proceedings of the

---

Ben Wang. 2021. Mesh-Tranen Transformer-AAX: Model- Parallel Implementation of Transformer Lan- guage Model with JAX. https://githe https://github.com thub.com/ trrero///mmms-ttrttaarreeee---
Shuohang Wang. Yichong Xu, Yuwei Fang. Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. 2022. Training data is more valuable than you think: simple and effective method by retrieving from training data. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3170- 3179.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thougle prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824 24837. Curran Associates, Inc.
Chenghao Yang, Fan Yin, He Hc, Kai-Wei Chang, Xi- aofei Ma, and Bing Xiang. 2023. Efficient shapley values estimation by amortization for text classifica- tion. arXiv preprint 23055 19998.
Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional emplars for in-context learning. arXiv preprint arXiv:2302.05698
Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyun- Cho, Hwiyeol Jo, Sang- Woo Lee, Sang-goo Lee, Taeuk Kim. 2022. Ground-truth labels matter: deeper look into input-label demonstrations. In Pro- ceedings of the 2022 Conference Empirical Meth- ods Natural Language Processing, pages 2422- 2437, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 20155 Character-level convolutional networks for classi- fication. Advances in neural information processing systems, 28.
Yiming Zhang, Shi Feng, and Chenhao Tan. 2022a. Ac- tive example selection for in-context learning. arXiv preprint arXiv:2211 11.4462.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022b. Automatic chain of thought prompt- ing in large language models. arXiv preprint arXiv:2210.03-93..
Zihao Zhao, Fric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In- ternational Conference Machine Learning, pages 12697-12706. PMLR

---

A Details of Datasets
each instance. Following the second stage ($3.2.2). utilize DPP to incorporate both quality and in- fluence diveristy choose demonstrations.
Table shows the data statistics our used datasets the experiments. Considering the test set of DBPedia is too large, we just sample 3000 exam- ples for testing in retrieval-based comparisons (F'ig- ure 3).
C Frame of Our Method
To provide more direct illustration of our method, present the overall workflow in Figure 9, which aids in better understanding Algorithm 1.
Dataset
Class
Train
Test
SST-2
2
6921
1821
Trec
6
5452
500
CB
3
250
250
AGNews
4
120000
7600
DBPedia
14
50000
70000
RTE
2 2490
3000
Table 4: Stastics of the datasets.
Table 5 shows the prompt formats for the six datasets. Following Zhao et al. (2021), we let the model predict the label for the given sentence for SST-2 and AGNews. For TREC and DBPedia, add the instruction sentence to illustrate the label space. For CB and RTE, which is sentence match- ing task, we take the first sentence as background and combine the second sentence with question prompt.
B Details of the Ablation Baseline
Table 3 presents the effect of the three factors. In this part, illustrate the details of each baseline. sem_div introduces semantic diversity via DPP and chooses demonstrations, which the same as the DPP bascline and similar to the first stage 1$3.2.1). ins_qua randomly samples the candi- date subset and computes the influence score of each instance. Then we choose the top-k sam- ples with the highest quality scores demonstra- tions. ins_qua also randomly samples the can- didate subset and computes the influence score of each instance. Then we introduce influence diversity via DPP and choose demonstrations. sem sem_di wilddiviis_ooo obtains the candidate subset by semantic DPP and computes the influence score of each instance. Then we choose the top-k samples with the highest quality scores as demonstrations. sem_ iin_dii=e diifiiffifi..... _dv obtains the candidate subset by semantic DPP and computes the inffuence score of each instance. Then we introduce influence diversity via DPP and choose demonstrations. mns_quatinf_dee randomly samples the candi- date subset and computes the influence score of

---

Task
Prompt
Label Names
SST-2
Review: [sentence]
Positive, Negative
Sentiment:
AGNews
Article:
[sentence]
World, Sports, Business, Technology
Answer:
TREC
Classify the questions based on whether their answer type is Number, Location, Person, Description, En- tity, Abbreviation.
Number, Location, Person, Description, Entity, Ab- breviation
Question: [sentence]
Answer Type:
DBPedia
Classify the documents based on whether they are about Company, School, Artist, Athlete, Politician, Transportation, Building, Nature, Village, Animal, Plant, Album, Film, or Book.
Company, School, Artist, Athlete, Politician, Trans- portation, Building, Nature, Village, Animal, Plant, Album, Film, Book
Article: [sentence]
Answer:
CB
[sentence1]
True, False, Neither
question: [sentence2] True, False, Neither?
answer:
RTE
[sentence1]
True, False
question: [sentence2] True or False?
answer:
Table 5; The prompt formats used for the six datasets.
Candidate Set Deem
((s))
samands sirilaity
average
OPP
Inf
Qled)
sport
L
Inf
slack
Demonstration jcm
Score Set D Daome
Figure Total frame of our method (Algorithm


Reviews:
[{"review_id": "8sLBLAi3gm", "reviewer": "Reviewer_3tF3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and constructive review! We address your questions and concerns below:\n\n1. Weakness 1: Intuitively, the \"diversity\" may average or smooth the distribution of the selected samples. Q1: How can we guarantee that the useful demonstration will not be filtered out? Q2: How can we guarantee that the obtained subset is the optimal distribution in terms of the whole dataset?\n* Reply For Q1: We agree that some useful demonstrations may be filtered out in the first stage. The first stage only employs semantic diversity to select a candidate subset and it is inevitable to filter out some useful demonstrations in this process. However, in our experiments, we also observe that this will have minimal effects on the final performance. In specific, we conducted an experiment on the CB dataset, whose training set is small. Therefore, we can exclude the first stage that introduces semantic diversity and instead compute quality scores for all training instances. However, the results indicated a minimal improvement of less than 1%. \n\n  Actually, for k-shot ICL, we just need k demonstrations and the number of useful demonstrations may be far more than k for a big training set such as AGNews (12000 training instances). Therefore, it is inevitable to filter out some useful demonstrations in the first stage. Besides, the second stage ensures that the selected k demonstrations are all of high quality. And the influence diversity factor further ensures that the composition of these demonstrations would support more test instances. Therefore, incorporating more useful demonstrations may not necessarily lead to a significant performance improvement. Another thing to consider is that the cost of determining the usefulness of each example in the entire training set is enormous. Therefore, this paper just takes a trade-off and introduces the influence diversity factor to alleviate the problem. And we believe this would be a crucial problem for representative demonstration selection and we would take this as our future work. Thanks for your valuable suggestion.\n\n* Reply For Q2: As for the second problem, the obtained subset indeed may be not the optimal distribution of the whole dataset. Actually, this problem could be seen as coreset selection problem, which has been explored in the past two years in machine learning and deep learning research areas [1]. And coreset selection has been regarded as NP-hard problem [2]. In this paper, we also take classical coreset selection methods Least Confidence and Cal as baselines. The experimental results indicate that these effective methods in deep learning do not perform well in ICL. For comparison, DPP could achieve good performance. Based on these empirical results, we think DPP could help obtain a good distribution of the whole dataset for ICL.  Furthermore, our method employs DPP to incorporate the three factors into the selection process. \n\n  To further alleviate your concerns, we will add the following experiments in the next version: Suppose the training set has only 10 instances, there are C(10,4)=210 4-shot subsets. Then we can inference 210 times to obtain the test performance of each 4-shot subset. In this way, we can know the position of the subset obtained from our method among all 210 subsets. We will add these experimental results in the next version to make our paper more complete. And we apologize that we cannot show more theoretical support because the mechanism of ICL is still unexplored now. And we believe that how to obtain optimal subset distribution of the whole dataset for ICL is indeed a problem worth long-term research like coreset selection in deep learning and machine learning.\n\n  [1] DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning, DEXA2022\n\n  [2] Coresets for data-efficient training of machine learning models, ICML2020\n\n2. Weakness 2: The proportion of the selected subset should be explored, and for the unselected samples it needs to further investigate that whether there are useful samples that are dropped falsely.\n* Reply: Thanks for your valuable suggestion. We agree that the proportion of the selected subset should be explored. In specific, we conducted experiments under GPT2-xl and tested the subset size of 50, 100, 150, 200, 250, and 300 on SST-2. The performance increases with bigger size but the improvements diminish significantly once the size exceeds 200. Considering that the bigger size would introduce much more inference costs when computing quality scores, this paper takes 200 for the experiments. We would add the effects of the subset size in the next version to make our paper more complete.\n\n | Subset Size |  50 | 100 | 150 | 200 | 250 | 300 |\n | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n | 2-shot SST-2| 54.92 | 57.10 | 58.67 | 59.86 | 60.47 | 60.82 |\n | 4-shot SST-2| 57.86 | 60.72 | 62.97 | 64.31 | 64.90 | 65.30 |\n | 8-shot SST-2| 66.03 | 68.94 | 71.02 | 72.40 | 72.92 | 73.40 |\n\n\n\n3. Weakness 3: The motivation of Representative-based method, compared with Retrieval-based approaches, is not very intuitive, since all the testing samples share the same subset although \"influence diversity\" could somewhat solve the issue.\n* Reply: The motivation of comparing Representative-based method with Retrieval-based approaches is to show the inference efficiency in our setting. Selecting the same demonstration set in Representative-based method would decrease inference time and API costs compared with Retrieval-based approaches that select different demonstrations for each testing instance (Line 446-474). Figure 4 shows that Representative-based method only requires 1/3 inference time and Table 2 shows that Representative-based method only requires 1/10 API costs, which is more applicable. And the performance of Retrieval-based methods could be intuitively regarded as the upper bound of Representative-based method and we also try to narrow this gap (Figure 3).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Influence diversity is proposed which could support the paradigm sharing the same subset.\n2. Pilot experiments are given to explore whether the proposed criteria could improve the performance of ICL.\n3. The method can obviously reduce the time-costs in the selection stage for each testing example.", "reasons_to_reject": "1. Intuitively, the \"diversity\" may average or smooth the distribution of the selected samples. \nQ1: How can we guarantee that the useful demonstration will not be filtered out? \nQ2: How can we guarantee that the obtained subset is the optimal distribution in terms of the whole dataset? \n2. The proportion of the selected subset should be explored, and for the unselected samples it needs to further investigate that whether there are useful samples that are dropped falsely.\n3. The motivation of Representative-based method, compared with Retrieval-based approaches, is not very intuitive, since all the testing samples share the same subset although \"influence diversity\" could somewhat solve the issue. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper aims to select a representative demonstration subset for a task in ICL, to satisfy high quality and diversity at the same time. The two criteria are explicitly defined in the paper, followed by the empirical analyses, demonstrating that meet these criteria can indeed bolster model performance. To achieve the goal, a method based on two-stage Determinantal Point Process (DPP) is proposed. Experimental results show the effectiveness of the method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ALq33mRyvY", "reviewer": "Reviewer_4SPy", "summary": "", "questions": "* Could you clarify the meaning of the gray background in Table 1? It appears to denote the best result, but in the case of RTE 2-shot under GPT-NeoX 20B, the paper's method seems to perform worse than DPP. How is this reconciled?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and constructive review! We address your questions and concerns below:\n\n1. Weakness 1: Although the paper mentions some related work, such as retrieval-based methods, it misses the opportunity to discuss recent similar works like [1]. Including such references would enrich the context and comparison.\n* Reply: Thanks for your advice. We would add the discussion about more recent retrieval-based methods including [1] in our next version to make our paper more complete. \n  \n  [1]  Unified Demonstration Retriever for In-Context Learning. ACL 2023\n\n2. Weakness 2: The experiments could benefit from incorporating newer baselines such as (Zhang et al., 2022a). Including more contemporary comparisons would strengthen the credibility and persuasiveness of the results.\n* Reply: Thanks for your valuable suggestion. Due to our oversight, we indeed failed to compare our method with the newer baseline ([2]) in 2023. Actually, this method primarily focuses on semantic diversity, sharing similar motivations with the cluster baseline in our paper. In the next version, we will add experimental comparison results of these updated baseline methods to enhance the persuasiveness of our paper. \n\n  [2] What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs? ASE2023\n\n3. Question 1: Could you clarify the meaning of the gray background in Table 1? It appears to denote the best result, but in the case of RTE 2-shot under GPT-NeoX 20B, the paper's method seems to perform worse than DPP. How is this reconciled?\n* Reply: We added a gray background just to highlight the results of our method in Table 1. We will add bold numbers to denote the best result and make the table clearer in our next version. In the case of 2-shot RTE under GPT-NeoX 20B, we review the results and find that DPP achieves much higher results (may be an outlier) in seed 2, which leads to the average performance even surpassing our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper addresses a crucial and timely issue in the field of large language models, making it highly relevant. \n* The paper is well-written, clear, and easy to follow. The persuasive analysis in section 2, for instance, enhances the overall comprehensibility and presents strong arguments supporting the proposed method.\n* The proposed two-stage DPP method is simple and effective. The solid experimental design and execution validate the method's efficacy.", "reasons_to_reject": "* Although the paper mentions some related work, such as retrieval-based methods, it misses the opportunity to discuss recent similar works like [1]. Including such references would enrich the context and comparison.\n* The experiments could benefit from incorporating newer baselines such as (Zhang et al., 2022a). Including more contemporary comparisons would strengthen the credibility and persuasiveness of the results.\n\n[1] Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, Xipeng Qiu. Unified Demonstration Retriever for In-Context Learning. ACL 2023", "questions_to_authors": "* Could you clarify the meaning of the gray background in Table 1? It appears to denote the best result, but in the case of RTE 2-shot under GPT-NeoX 20B, the paper's method seems to perform worse than DPP. How is this reconciled?", "paper_topic_and_main_contributions": "The paper focuses on the key problem of selecting a representative subset of in-context demonstrations. It emphasizes two criteria for demonstration selection: quality and diversity. To achieve this, the paper introduces a two-stage Determinantal Point Process (DPP) method, aiming to find a representative subset that bolsters model performance. Comprehensive experiments affirm the effectiveness of this method, presenting a robust comparison to retrieval-based methods.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, Xipeng Qiu. Unified Demonstration Retriever for In-Context Learning. ACL 2023"}, "extra_text": {}}, {"review_id": "HcEVGFWElK", "reviewer": "Reviewer_twH8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and constructive review! We address your questions and concerns below:\n\n1. Weakness 1: The baseline method compared in the article is somewhat outdated, currently in 2023, with only one baseline method in 2022.\n* Reply: Thanks for your suggestion. This paper aims to find representative demonstrations for the whole testing dataset in ICL, where this setting seems to be the firstly addressed. More recently, we indeed found some newer baselines that are also suitable for this problem ([1]). In summary, their methods focus on semantic diversity, sharing similar motivations with the cluster and DPP baselines in our paper. Considering their core idea focuses on only semantic diversity, we believe our method could get better performance.  We will add the comparisons with these newer baselines in the next version. \n  \n  [1] What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs? ASE2023\n\n2. Weakness 2: The author should open source code to enhance the persuasiveness of the article and increase its contribution to the research community.\n* Reply: Thanks for your advice. We promise to release our source code in github.\n\n3. Weakness 3: The author may consider adding a model frame diagram to better understand the entire process.\n* Reply: We will add a frame diagram to describe the entire process of our method (Algorithm 1) in the next version. We believe that it could indeed make our paper easier to understand. Thanks for your valuable suggestion.\n\n4. Weakness 4: Is it appropriate for the authors to use the DDP process to achieve semantic diversity, high quality, and influence diversity, as these features are considered in stages.\n* Reply: We admit that if we incorporate these three factors into a single DPP process would solve your concerns. However, the latter two factors, high quality and influence diversity both require the computation of the quality score of each instance. It is infeasible to compute the quality score for each instance in the total training set because of the computation costs. Therefore, this paper employs a two-stage method and we believe the two-stage method could also incorporate these three factors. After the first stage DPP, the obtained demonstration set satisfies the semantic diversity. With this set, we could compute the quality score for each instance and incorporate high quality and influence diversity into the second stage DPP. Therefore, we think our two-stage method could also incorporate these three factors into the selection process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1)\tThe paper is easy to follow, and the logical structure of the article is clear. Furthermore, the authors propose two criteria (quality and diversity) that in-context demonstrations should satisfy.\n2)\tThe experimental effects achieve great improvements, and the results show a significant advantage in inference time and token usage compared to the retrieval-based method. ", "reasons_to_reject": "1)\tThe baseline method compared in the article is somewhat outdated, currently in 2023, with only one baseline method in 2022.\n2)\tThe author should open source code to enhance the persuasiveness of the article and increase its contribution to the research community.\n3)\tThe author may consider adding a model frame diagram to better understand the entire process.\n4)\tIs it appropriate for the authors to use the DDP process to achieve semantic diversity, high quality, and influence diversity, as these features are considered in stages.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The research paper focuses on the concept of In-Context Learning (ICL) with Large Language Models (LLMs) and its limitations, particularly in relation to demonstration selection. ICL has shown significant improvements over zero/few-shot settings, yet, it is sensitive to factors like demonstration selection, format, labels, and order. The authors emphasize the importance of choosing suitable demonstrations.\nTo alleviate the problem of low efficiency of existing methods, the authors offer a two-stage solution to meet these criteria. The first stage involves using semantic information to measure instance similarity and leveraging a DPP to obtain a subset with high semantic diversity. The second stage involves computing the quality score of each instance in the subset based on influence scores. Then another DPP is constructed using these scores to incorporate quality and influence diversity in the selection process. This method aims to optimize the selection process, ensuring it covers more semantics in the training set and improves classification performance.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Strengths**:

1. Critical problem of selecting a representative subset of in-context demonstrations that can effectively prompt different test instances in a speciﬁc task. 

2. high quality and diversity are stressed together for good quality examples. 

3. two-stage Determinantal Point Process (DPP) method is simple and effective. results show a significant advantage in inference time and token usage compared to the retrieval-based method.


**Weaknesses**:


1. The proportion of the selected subset should be explored, and for the unselected samples it needs to further investigate that whether there are useful samples that are dropped falsely. Although the authors show results for GPT2-XL in rebuttal, they should include such analysis for other models also.

2. Missing latest baselines: comparisons with Xiaonan Li et al, ACL 2023; Zhang et al., 2022a.

3. Code is not made public.

**Suggestions**:


1. C(10,4)=210 4-shot subset quality evaluation and checking rank of selected subset should be done at least for a few samples.

2. Good to add a frame diagram to describe the entire process of our method (Algorithm 1).

3. Please address above weaknesses also in the revised draft.
Decision: Accept-Main