Active Learning Principles for In-Context Learning with Large Language Models
Authors: Katerina Margatina
Timo Schick
Nikolaos Aletras
Jane Dwivedi-Yu
Venue: EMNLP 2023 Findings (2023)
Keywords: active learning
in-context learning
few-shot learning
large language models
TL;DR: We explore data selection algorithms through the lense of active learning for in-context learning with large language models.

Abstract: The remarkable advancements in large language models (LLMs) have significantly enhanced predictive performance in few-shot learning settings. By using only a small number of labeled examples, referred to as demonstrations, LLMs can effectively perform the task at hand through in-context learning. However, the process of selecting demonstrations for maximizing performance has received limited attention in prior work. This paper addresses the issue of identifying the most informative demonstrations for few-shot learning by approaching it as a pool-based Active Learning (AL) problem over a single iteration. We compare standard AL algorithms based on uncertainty, diversity, and similarity, and consistently observe that the latter outperforms all other methods, including random sampling. Our extensive experimentation involving a diverse range of GPT and OPT models across $24$ classification and multi-choice tasks, coupled with thorough analysis, unambiguously demonstrates the importance of using demonstrations that are semantically similar to the domain of the test examples. In fact, we show higher average classification performance using ``similar'' demonstrations with GPT-2 ($124$M) than random demonstrations with GPT-Neox ($20$B). Notably, while diversity sampling shows promise, uncertainty sampling, despite its success in conventional supervised learning AL scenarios, performs poorly in in-context learning.

Active Learning Principles for In-Context Learning with Large Language Models
Katerina Margatina
Timo Schickt
Nikolaos Aletras
Jane Dwivedi- Yut
University of Sheffield TFAIR, Meta [k. margatina, n. y.aletras)@sheffiiee ac janeyuêmeta.co com
Abstract
Clessification
examples
The remarkable advancements in large lan- guage models (LLMs) have significantly en- hanced predictive performance in few-shot learning settings. By using only a small number of labeled examples, referred to as demonstra- tions, LLMs can effectively perform the task at hand through in-context learning. However, the process of selecting demonstrations for maxi- mizing performance has received limited atten- tion in prior work. This paper addresses the is- of identifying the most informative demon- strations for few-shot learning by approaching it as a pool-based Active Learning (AL) prob- lem over single iteration. We compare stan- dard AL algorithms based on uncertainty, di- versity, and similarity, and consistently observe that the latter outperforms all other methods, including random sampling. Our extensive ex- perimentation involving diverse range of GPT and OPT models across 24 classification and multi-choice tasks, coupled with thorough anal- ysis, unambiguously demonstrates the impor- tance of using demonstrations that are semanti- cally similar to the domain of the test examples. In fact, we show higher average classification performance using "similar" demonstrations with GPT-2 (124M) than random demonstra- tions with GPT-Ncox (20B). Notably, while diversity sampling shows promise, uncertainty sampling, despite its success in conventional supervised learning AI scenarios, performs poorly in in-context learning.
23
Multi-choice
26
35
314
G3F--42
S2T-0R2x-006
Figure 1: Performance of different in in-context selection algorithms classification and multi-choice tasks.
predictive performance. However, in-context learn- ing has changed this status-quo by enabling LLMs to learn from limited, context-specific examples and adapt to new tasks and domains with remark- able proficiency (Zhao et al., 2021; Chowdhery et al., 2022; García et al., 2023; Wei et al., 2023b; Touvron et al., 2023: Bubeck et al., 2023). Unlike more traditional approaches, which require exten- sive retraining fine-tuning for every new task, in-context learning empowers LLMs to general- ize from few examples that are fed the model through prompting to learn new task at hand, without any weight updates.
The data efficiency of few-shot in-context learn- ing of LLMs is indeed remarkable with only small number of demonstrations. Still, such demonstra- tions constitute labeled data examples, raising two key questions: (1) When faced with tasks where there is only unlabeled data available, how can we select the most appropriate samples to label and then use as in-context demonstrations? (2) When we have labeled data for given task, how can
1 Introduction
The field of Natural Language Processing (NLP) has recently witnessed a remarkable paradigm shift with the emergence of in-context learning with large language models (LLMss, also referred to few-shot learning (Brown et al., 2020). Tradi- tionally, NLP systems heavily relied on supervised learning approaches, where large amounts of la- beled training data were necessary achieve high Work done during an internship at FAIR, Meta.
"We use the terms in-context examples,,ewwaee few-shot examples. demonstrations, descriptors and exemplars interchangeably throughout the paper.

---

we efffiiintly identify the most informative combi- nation of demonstrations for in-context learning? Answering these questions is essential to ensure ef- fective and efficient few-shot learning using LLMs. A growing line of work has investigated how in- context learning works (Reynolds and McDonell, 2021: Razeghi et al., 2022; Xie et al., 2022; Ye al., 2023b), which demonstrations to (Liu al., 2022; Zhang et al., 2022b; Wu et al., 2022: Kim et al., 2022), how to form the prompt (Zhao et al., 2021; Lu et al., 2022: Yang et al., 2023) and whether ground truth labels matter (Webson and Pavlick, 2022; Min et al., 2022; Yoo et al., 2022; Wang et al., 2022; Wei et al., 2023b). Still, to the best of our knowledge, prior work has explored the problem of in-context demonstration selection explicitly through the lens of active learning (AL). Based on the core principle that not all data points are equally useful, AL (Cohn et al., 1996; Settles, 2009) aims to identify the most informa- tive instances from a pool of unlabeled data for annotation. lterating through model training, data acquisition and human annotation, the goal is to achieve data efficiency. A data-efficient AL al- gorithm ensures that model achieves satisfactory performance on a withheld test set by selecting only a small fraction of the unlabeled data for annotation that typically is better than randomly selecting and annotating data of equal size.
We evaluate 15 models, between 125M and 30B parameters, from the GPT (Radford al., 2019; Brown et al., 2020; Black et al., 2022) and OPT (Zhang et al., 2022a) families 15 classification and 9 multi-choice tasks, using different AL sampling techniques to select demonstrations for few-shot learning.
We demonstrate that while diversity and uncer- tainty sampling perform slightly better than random sampling, choosing in-context exam- ples that are semantically similar to the in- put test examples outperforms consistently all other methods by large margin across model families and sizes in all tasks.
We show that while uncertainty sampling is one of the strongest AL approaches in super- vised learning, this does not generalize to in- context learning, where interestingly it under- performs. Our analysis, however, shows that larger models might perform better with uncer- tain demonstrations, hinting that uncertainty might be emerging LLM ability.
2 Active In-context Learning
2.1 Problem Formulation
To build our in-context learning framework with actively acquired demonstrations, depicted in Fig- ure borrow the formulation from the standard pool-based active learning paradigm. We consider an AL setting where we have a large pool of unla- beled data from which we want to sample a batch of k data points using a data acquisition algorithm. We assume that these l are subsequently labeled by humans (Figure 2, top). Instead of following the standard approach that involves multiple iterations only per-
In this paper, our main aim is to redefine the concept of data efficiency within the framework of in-context learning inspired by conventional active learning settings. For this purpose, we assume that given a pool of labeled or unlabeled data, the objective is to identify a set of & examples that will serve as demonstrations to an L.M,, resulting in optimal performance on a held-out test set. Given this formulation of data efficiency, we explore the effectiveness of the most prevalent AL approaches based on uncertainty (Lewis and Gale, 1994; Cohn al., 1996; Gal al., 2017), diversity (Brinker, 2003; Bodó al., 2011: 1; Sener and Savarese, 2018) and similarity (Margatina et al., 2021; Kirsch et al., 2021; Liu et al., 2022), as demonstration selection methods for in-context learning (Figure 1). Our key contributions are as follows:
of data selection and model training, form single iteration (Longpre et al., 2022), since we do not train or perform any model-in modeelin-the--ove -hop- updates. We use the acquired set of k examples as demonstrations for in-context learning with an LLM (i.e., as part of the prompt). We assume the existing datasets as the pool from which to select these & examples. The goal is to find the most informative examples from the pool, which pected yield improved performance on the test when employed as few-shot prompt, com- pared to demonstrations randomly sampled from the same pool. The resulting prompt consists the concatenation of the & acquired examples (text
We formulate the selection of in-context ex- amples single iteration AL problem and explore the effectiveness of four standard ap- proaches: uncertainty, diversity, similarity and random sampling.

---

poal of unlabeled data
data acquisition algorithm
human annotation
Ke Ni
Mylh's Kayli Kaghia
Kig
x,
x
Kip
Kya
X-p
Myp
In-context learning with actively acquired demonstrations
prompt
LLM
prediction
Circulstion ion revenue has wexed by 5% Finland. Panostaja dieciose purchase price.
In pos neutral nogathiv
demonstrations
positive
Paylnng off the national debt will extromely painful
test example
The acquisition will have immeddiate posit impect. In
Figure 2: Top: Active data collection (single iteration). Bottom: Prompt construction and model inference.
inputs and labels with standard verbalizers), along- side the test example, repeated for all data instances in the test set (Figure 2, bottom).
to Yu et al. (2022). Specifically, we first encode all data points in the pool of unlabeled data with Sentence-Berr
(Reimers and Gurevych, 2019) embeddings and then we perform k-means cluster- ing. 2 We choose the number of clusters to be k and select one data point from each cluster. The under- lying principle of this approach is that leveraging diverse of in-context examples can offer greater advantages compared random sampling. This selection strategy ensures that the chosen demon- strations are likely to encompass a broad range information, enhancing the overall effectiveness of the learning process.
2.2 Few-shot Data Acquisition Algorithms
We build few-shot data acquisition algorithms in- spired by the most prevalent AL algorithmic fami- lies that are uncertainty sampling, diversity ty sampling and similarity (also known test- aware sampling) (Zhang al., 2022c). We ac- knowledge that there are more elaborate demon- stration selection methods for in-context learning that are not considered in our experiments, such as Q-learning (Zhang et al., 2022b), Self Adaptive (Wu et al.. 2022), SG-ICL (Kim et al.. 2022), MI (Sorensen et al., 2022), inter alia. These methods fall beyond the scope of our analysis, our ob- jective is to gain insights into AL principles for in-context learning, rather than benchmarking all available demonstration sampling algorithms. Ad- ditionally, there are techniques, complementary to the aforementioned fhw-shot data selection meth-
Uncertainty The second approach is uncertainty-based sampling algorithm that is based SPELL, proposed by Gonen al. (2022). Since we use an off-the-shhele LLM that does have a fine-tuned classification layer, we cannot compute the model probabilities associated with cach class (for classification or multi-choice task). This essentially means that we cannot use standard AL uncertainty baselines such as maximum entropy or least confidence. Instead, loss, i.e., perplexity, of the LLM to score each candidate example from the pool. Gonen al. (2022) define perplexity of the prompt as the perplexity of the full prompt sequence, including the input itself, and without the label, averaged over 1, 000 examples. Our approach is different since we want to evaluate the perplexity of each in-context example individually. We also do not do the averaging over thousand examples we wanted to make the method more general, without
ods, such as calibration (Zhao et al., 2021) and prompt re-ordering (Lu et al., 2022), which can fur- ther enhance few-shot learning performance, while also being out of the scope of our work.
use the
The overarching objective of any data se- lection method, like AL algorithms, is to identify data points that, however used, yield superior mod- els compared to randomly sampled data from the same pool which we consider baseline method.
Random
The first data selection method that we use as representative for the diversity family methods is a simple clustering technique, similar
Diversity
We use the implementation from https: hoos:://wwww http://wwwwttteteeeee///e////.ee net/exane wwwppeelll/////eeeeeeeeeee http:///aapaaaiios//////// wtw.tiionncsssssnngg...

---

the need assume access to that many examples. The underlying principle guiding this approach is the belief that a high perplexity set of in-context examples can yield greater advantages compared to randomly sampling from the dataset (or at least for data efficiency in a supervised learning setting this is proven to enhance the learning process),
4 Results
Figure 3 shows the results on few-shot in- context learning across all data acquisition meth- ods (random, diversity, uncertainty and simi similarity), model families (GPT and OPT) and tasks (classification and multi-choice question an- swering). Overall, we observe the anticipated trend of performance enhancement with increas- ing scale, particularly notable in the multi-choice tasks for both OPT and GPT models.
Similarity Finally, the third AL algorithm consider is based KATE KNN-augmented in- context example selection method proposed by Liu et al. (2022). This method retrieves examples from the pool that are remantically-simille to a test query sample. We use Sentence-BERT (Reimers and Gurevych, 2019) representations of both the pool and the test set to find the k-nearest neighbours. The rationale behind this approach is that the most similar demonstrations to the test example will best help the model answer the query. We have to high- light, however, that by definition each test example will have a different prompt, as the k most similar demonstrations will be different. This is a crucial limitation of this approach compared to the others, as it assumes that we are able to acquire labels for any in-context example selected from the pool.
Still, the most remarkable finding is the sub- stantial performance improvement achieved by lecting simi lar in-context examples for few-shot learning, particularly in classification tasks. This observation aligns with the findings reported by Liu et al. (2022). who demonstrated similar patterns in sentiment analysis tasks with GPT-3. Our results indicate that the selection of appropriate demonstra- tions can hold greater significance than the number of model parameters, at least within the scope of the models evaluated in this study. In multi-choice tasks, simi lari ty is also the top-performing ac- quisition method, while the other three approaches exhibit closely competitive performance.
The data selection method based on diversity is consistently the second best approach after simi larity (with very few exceptions in the multi- choice tasks for OPT models). Even though it is not the top performing method, we can consider that consistently outperforming random sampling is strong signal that diversity in the demonstra- tions is characteristic of effective demonstrations. Levy et al. (2022) explore the setting of composi- tional generalization, where models are tested on outputs with structures that are absent from the training set and thus selecting similar demonstra- tions is insufficient. They show that combining diverse demonstrations with in-context learning substantially improves performance for the task of compositional generalization semantic parsing. Remarkably, uncer sampling, typically regarded as one of the best approaches for tradi- tional supervised AL (Shen et al., 2017: Margatina et al., 2022; Schröder et al., 2023), exhibits the lowest performance. This finding contradicts the conventional AL principles that suggest selecting few highly uncertain labeled data points for data efficiency. Similar to our findings, Gonen et al. (2022) explore the performance variabilty of dif-
3 Experimental Setup
Models We evaluate 15 LLMS in total, 8 mod- els from the GPT (Radford et al., 2019; Brown al., 2020: Black al., 2022) and from the OPT (Zhang et al., 2022a) family. We choose our models to span from few million to tens of billions parameters, as we want to study how the model size affects the effectiveness of in-context example se- lection methods. All models considered in this work are publicly available.
Tasks & Datasets Following Min et al. (2022), we evaluate all LLMS in 15 classification and multi-choice tasks taken from the Crossfit (Ye et al., 2021) benchmark. We provide details for all tasks and datasets considered in the Appendix A.I.
In-context Learning Prompting Unless speci- fied otherwise, we sample k=16 demonstrations, i.e., labeled data, from the pool with each AL method. After collecting the k input-label pairs, we concatenate them all together with the test ex- ample that we want to make a prediction for to form the L.LM prompt (Figure 2). Our inplementation, including prompt verbalizers, is based on those by Min et al. (2022) and Yoo et al. (2022).
*We provide the results per dataset and model in the Ap- pendix A.2, including the majority vote baseline.

---

Classification
Multi-choice
0.7
0.6
0.6
0.5-
0.5-
AL methods
0.4
0.4
random
0.3
0.3
diversity
uncertainty
0.2
0.2-
similarity
0.1
0.0
0012
gatz mee--medium
0912 Aarge
m922 ml
get reep- .B
gnt-nerx-2200 20h
GRY2
get2 miz--meiium
ma Large
0mi2-xt
got 38
oat-nes-22 78
got-neoe 20b
apt-nes--2. -
got-
got.
Classification
Multi-choice
0.7
0.7
AL methods
0.4
0.4
random
diversity uncentainiyy
0.2-
0.2
similarity
opt-125m
opt-350m
opt-1 3b
opt-6.r 7B
33
opt-30b
125m
opt-35tm
opt-1 36
opt-2.78
opt6.7b
opt 130
opt. 300
opt
Figure 3: Results for various GPT (top) and OPT (bottom) models and AL methods averaged over 15 classification and 9 multi-choice tasks. Similarity is consistently the best performing approach overall, followed by diversity and random. Interestingly, we observe that uncertainty sampling underperforms this setting of in-context learning.
ferent prompts (consisting of randomly sampled demonstrations) for in-context learning using un- certainty, and find that the lower the perplexity of the prompt is, the better the prompt is able per- form the task. Still, in a later analysis we show that larger models might able to handle high uncer- tain prompts better than the smaller ones ($5.4).
especially models grow and, most likely, will continue to grow exponentially in terms of model parameters. Our findings show that the in-context learning ability of models from few millions a few billions of parameters follows similar patterns. However, this might not be the case when studying even larger models, as primary results hint (Rae et al., 2022: Wei et al., 2023b; Chowdhery et al., 2022; Touvron et al., 2023).
5 Analysis
5.1 Effect of Model Size
5.2 Ground Truth Demonstrations
In order to gain some intuition the effect of scale, we group together GPT and OPT models with similar number of parameters. We provide the results in Figure 4. Even after aggregating the results from both model families, we do not see any specific pattern as the model parameters increase. We wanted to explore whether the largest models of our collection would behave differently under the varying in-context learning settings, thus perhaps attributing such behaviour potential emergent abilities of the bigger LLMs, but we observe the same patterns (in terms of ranking between the considered data selection methods). We believe that this an interesting avenue of future research,
We next delve into the debate of whether ground truth demonstrations, i.e., providing the correct la- bel to the in-context examples, is crucial for high performing in-context learning. Various findings have shown mixed results for randomly sampled data, which essentially means that the benefit of ground truth labels depends on the label space the distribution of inputs specified by the demon- strations (Min et al., 2022: Yoo et al., 2022). In our analysis, we differentiate from prior work by exploring the importance of ground truth demon- strations the case of leveraging similar in contett examples ($2.2). The rationale is that if the find-

---

Classification
Multi-choice
In-context data selection
In-context selection
0.7
random
duersty
urcetariyy
simleity
0.7
random
dieestty
untertoity
simterty
0.6
0.5
accuracy
0.5
0.4
0.4
11
0.3
0.2
0.2
0.1
0.0
0.0
125M
350M
750M
1.58
38
68
108
208
125M
350M
750M
1.58
38
68
108
208
Figure 4: Results per model size.
In-context learning with similerity acquisition
and choose the least similar examples from the pool to form the prompt. This investigation aims to ascertain whether the remarkable performance gains can be attributed solely to the semantic simi- larity between the demonstrations and the test input. hy-
0.7
rardom
9.5
0.4
The results depicted in Figure 6 substantiate pothesis, demonstrating significant performance drop when employing opposite examples from the pool as in-context exemplars. While this pattern is particularly pronounced the classification tasks, consistently emerges across different model sizes and task types. Hence, we can assert that maximiz- ing semantic similarity between the demonstations and the inpul test sample is an unequivocally vital attribute for achieving successful in-context learn- ing outcomes with L.LMs. Future endeavors. in the field of building effective in-context learning frame- works should incorporate this principle to enable data-efficient algorithms that can fully harness the potential of LLMs.
0.2
4.0
poem pentiment
glue-ite
Nick
ghue-mpp
teest eval-nato
Figure 5: Effect of ground truth labels in-context
learning with with the similarity AL selection method.
ings of Min et al. (2022) ubiquitously hold, then the performance should only marginally drop if replace ground truth labels with random ones. If the high performance of the simi similarity acquisi- tion method can be retained, we would be able to construct an efficient and effective in-context se- lection algorithm that would be agnostic to correct labels. However, we find that this is not the case. We show in Figure 5 that for almost all datasets considered in this part of analysis, the performance with random labels drops significantly as expected. There are cases where replacing the original labels with random ones as Min et al. (2022) retains the same performance (e.g., in the glue-rte dataset), but this is certainly a finding that does not general- ize overall. In summary, we find that ground truth demonstrations are crucial for high performing. ro- bust in-context learning (Yoo et al., 2022).
5.4 Most vs. Least Uncertain Demonstrations
Along these lines, also opt to examine the dual- ity between selecting the most the least uncertain in-context examples from the pool. We show the results of these experiments for the GPT models in Figure 7. Interestingly, we observe that while the smaller language models (gpt2, 1opt2-medium, gpt-large) perform better with the least uncertain prompts, the larger models seem to start benefit- ing from the demonstrations with high uncertainty. This is particularly clear in the largest model of our collection, GPT-NeoX (20B parameters). This inter- esting finding shows that even larger models will most likely perform better with high entropy in- context examples, similar to their supervised learn-
5.3 Most Least Similar Demonstrations
To investigate the striking effectiveness of the sim- ilarity-based acquisition strategy, we conduct ad- ditional experiments where we invert the approach

---

Clasatication
Mullschoice
in the pool that can be used in-context exemplars. We can see that the uncertainty baseline, which is usually top performing in supervised AL, gen- erally underperforms in the few-shot setting. Still, there is some evidence that this could change with even larger and better models ($5.4). Finally, delv- ing into the debate on whether ground truth labels matter not (Min et al., 2022; Yoo et al., 2022), show that replacing original with random in- context labels hurt sigmificantly the performance of simi larity, the best data selection method ($5.2).
Duta octio
of get ger
andh
Figure 6: Most vs, least similar in-context examples.
Classification
Multicchoice
We further emphasize the significance of em- ploying a meticulous evaluation framework, partic- ularly in the selection of appropriate metrics. In Figure 8, we illustrate the same classification ex- periments, but with the Fi score plotted on the left and accuracy on the right. The use of Fi, the conventional metric for classification tasks, reveals a distinct ranking among the various AL methods, with similarity exhibiting the best per- formance, followed by diversity. Conversely, when employing accuracy to compare the methods, di versitty ty emerges the top approach, followed by similaritt ty and random selection. This dispar- ity highlights the potential for misconceptions obscured findings, underscoring the need for cau- tion when evaluating and comparing different meth- ods across various models within the in-context learning framework (Dehghani et al., 2021; Min et al., 2022; Yoo et al.. 2022; Tedeschi et al., 2023).
Doca splection
Figure 7: Most least uncertain in-context examples.
ing counterparts. Such findings open plethora of research questions regarding understanding how in- context learning works (Reynolds and McDonell, 2021: Ruzeghi et al., 2022; Xie et al., 2022; Min et al., 2022), how AL and data acquisition methods reshape with larger language models or whether we can properly investigate potential emergent abili- ties of LLMs acquired by model scaling (Wei et al., 2022:
Schaeffer et al., 2023).
Related Work
5.5 Evaluation with Different Metries
6.1 Understanding In-Context Learning
Finally, we want to provide a clear overview of our experiments and summary of our findings, while making some clarifications regarding how we evalu- ate and compare different approaches to in-context learning. Figure& 8 shows the results for in-context learning with random sampling, three data selec- tion techniques inspired by AL ($2.2), namely diversity, uncertainty and similaritt and a zero-shot baseline where no labeled examples are included in the prompt (no_ (no_) We show that in- context learning with k=16 demonstrations consis- tently outperform zero-shot learning for an average of 15 classification tasks for gpt2-large, gpt-j and gpt-neox. Next, we observe that the best performing in-context example selection method is by clear margin similarity, followed by diversity. This finding corroborates the origi- nal hypothesis of AL that, indeed, not all data is equal and there exist more informative data subsets
Few-shot in-context learning with LLMs has gar- nered significant attention in recent NI.P research. Simply concatenating a few labeled examples to form the prompt for the LLM results in high perfor- mance gains, even outperforming fine-tuned mod- els (Brown et al., 2020; Chung et al., 2022; Ouyang et al., 2022: Dong et al., 2022). This has naturally lead to study its effectiveness with multiple few- shot learning benchmarks such as Crossfit (Ye et al., 2021) and BigBench (Srivastava et al., 2022). Another active area of research is on understand- ing how in-context learning works (Xie et al., 2022; Garg et al., 2022; Akyürek et al., 2022; Xie et al., 2022; Pan et al., 2023), and what are its strengths and limitations (Webson and Pavlick, 2022: Jang et al., 2022; Levy et al., 2022; Shi et al., 2022; Agrawal et al., 2022; Wei et al., 2023b; Ye et al., 2023b). Previous work has explored the effec-

---

Classification
Classification
0.-
0.5
AL methods
041
0.4
random diversity
0.3
0.3
uncertainty
similority
gpt2-arge
spt.-68
set neox-200
uat2 Jarger
spa--68
got notx--0x200
Figure 8: The ranking of data selection methods is different depending on the metric used.
tiveness of the chain-of-thought prompting tech- nique (Wei et al., 2023a; Wang al., 2022; Madaan and Yazdanbakhsh, 2022), while other studies try to determine the importance of in-context ground truth labels, with Min et al. (2022) showing that random labels do not hurt performance consider- ably and Yoo et al. (2022) providing a rebuttal. Wei et al. (2023b) explain that model size plays an role in the effect of ground truth labels, showing that small LMs ignore flipped labels, while LLMs can override semantic priors learned during pretraining. Interestingly, Razeghi et al. (2022) demonstrates that in-context learning performance is highly cor- related with the prevalence of each instance in the pretraining corpus, showing that models are more accurate on few-shot numerical reasoning on in- stances whose terms are more frequent.
fulness of in-context examples and find that the best performing prompts have low perplexity. Zhang et al. (2022b) formulate example selection for in- context learning as a scquential decision problem and show modest performance improvements by acquiring data with their proposed method based on reinforcement learning. Other previous work, instead of focusing the part of acquiring data for in-context learning, show that demonstration order- ing (Lu et al., 2022) and model calibration (Zhao et al., 2021) are additional properties that influence the few-shot learning performance.
6.3 Active Learning for NLP
AL has been extensively studied in various NLP tasks, including machine translation (Miura et al.. 2016; Zhao et al., 2020), natural language infer- ence (Snijders al., 2023), named entity recog- nition (Shen et al., 2017: Wei et al., 2019), and text classification (Ein-Dor et al., 2020; Margatina et al., 2022; Schröder et al., 2023), among others.
6.2 Selecting Informative Demonstrations
Typically, work on evaluating LLMs in few-shot settings commonly uses randomly sampled exam- ples to compose the in-context prompt (Brown al., 2020; Zhang et al., 2022a; Chowdhery et al., 2022; Chung et al., 2022; Touvron et al., 2023). Nonetheless, has been demonstrated that the effectiveness of few-shot performance signif- icantly depends on the selection of in-context ex- amples (Kocielnik et al., 2022; Ye et al., 2023a; Diao et al., 2023; Xu et al., 2023). Consequently, there is ongoing research generating or select- ing the most informative demonstrations, aiming to maximize the downstream few-shot performance.
Still, its importance and potential value is the rise (Zhang et al., 2022c; Rauch et al., 2023), as the current language model pretraining paradigm tinues advance the state-of-the-art (Tamkin al., 2022). Given the fundamental premise that"not all data is equal" it is reasonable to expect researchers to actively seek the "most informative" data for pretraining or adapting their large language models (LLMs), as well as identifying the most valuable in-context examples for few-shot learning scenar- ios. Previous work has explored AL for prompt- based finetuning (Köksal al., 2022), proposing method based in inter-prompt uncertainty sam- pling with diversity coupled with the PET archi- tecture (Schick and Schütze, 2021a,b) that outper- forms all AL baselines.
Some approaches are based on retrieval compo- nent that sources the most relevant examples from a pool. The prompt retriever can be trainable (Ru- bin et al., 2022) or based on pretrained embed- dings (Liu et al., 2022; Agrawal et al., 2022). Go- nen et al. (2022) use uncertainty to evaluate the use-

---

7 Conclusion
and OPT models that span 125M to 30B param- eters. Even though our experimentation thor- ough, our findings might not generalize to larger or smaller transformer- based models, or models based in a different architecture.
In this study, we have examined the selection of demonstrations, i.e., labeled data that provide ex- amples of solving task, for in-context learning with LLMs. We formulated the selection process as a single iteration active learning problem and evaluated four standard approaches: uncertainty. diversity, simi larity, and random sampling. Our evaluation involved 15 models of varying size from the GPT and OPT families, encom- passing 15 classification tasks and 9 multi-choice tasks. Through extensive experimentation, we have demonstrated that selecting demonstrations that semantically similar the test input examples con- sistently outperforms all other methods by a signif- icant margin across all model families, sizes, and tasks. This corroborates findings of several previ- ous and concurrent studies that explore the proper- ties of "good" in-context examples (Liu et al., 2022; Shi et al., 2022). Interestingly, our findings reveal that uncertainty sampling, although effective in su- pervised learning, underperforms in the in-context learning paradigm. This highlights the importance of our work in exploring the principles of active learning in the context of sho-shot lcarning.
Active learning considerations We explicitly note in the paper that we do a single active learning iteration, which is different than the common AL loop that consists of multiple iterations. As we ex- plained, because the moodel-in-the-oon (the LLM) is not updated (no fine-tuning) with new data, per- forming multiple iterations does not make sense in this context (Figure 2). Still, it would be inter- esting for future work to explore how we can per- form multiple AL iterations while constructing the prompt (i.e., acquiring the demonstrationss) The upper bound would be to try all the combinations of a set of labeled data and find the best performing prompt. However, doing this with unlabeled data, in efficient way, is far from trivial. We refer to Zhang al. (2022c); Treviso et al. (2023); Mar- gatina and Aletras (2023) for in-depth suggestions for future work in this area.
References
A Acknowledgements
Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2022. In- context examples selection for machine translation.
We would like to thank the anonymous reviewers for their suggestions to improve work. We also thank Louis Martin, Patrick Lewis, Fabio Petroni and other members of FAIR for their constructive feedback on previous versions of the paper.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2022. What learn- ing algorithm in-context learning? investigations with linear models. ArXiv, 0bb//2111.1566
Limitations
Francesco Barbieri, Jose Camacho-Colladon Luis Es- pinosa Anke, and Leonardo Neves. 2020. Tweeteval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644-1650.
Tasks & Datasets We acknowledge that even though we experimented with a well established benchmark, the Crossfit (Ye et al., 2021) benchmark consisting of 15 classification and 9 multi-choice question answering datasets (Ap- pendix A.1). it might still not be sufficient to ensure that our findings will generalize any NLP clas- sification or multi-choice application of in-context learning.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Authony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Puro- hit, Laria Reynolds, Jonathan Tow, Ben Wang. and Samuel Weinbach. 2022. GPT-NeoX-20B: An open- source autoregressive language model. In Proceed- ings of BigScience Episode #5 Workshop on Chal- lenges & Perspectives in Creating Large Language Models, pages 95-136, virtual+ Association for Computational Linguistics.
Language We also acknowledge that all the datasets and models considered in this work are based on the English language alone. This limits gencralizability of our findings to other languages.
Zalán Bodó, Zsolt Minier, and Lehel Csató, 2011. Ac- tive learning with clustering. In Proceedings of the Active Learning and Experimental Design workshop
Model scale We investigated in-context learning with actively acquired demonstrations with 15 GPT

---

conjunction with AISTATS 2010, volume 16, pages 127-139.
Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction finetiun-- language mod- els.
Klaus Brinker. 2003. Incorporating diversity in active learning with support vector machines. In Proceed- ings of the International Conference on Machine Learning, pages 59-66.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.0557.
David A. Cohn, Zoubin Ghahramani, and Michael Jordan. 1996. Active learning with statistical mod- els. Journal of Artificial Intelligence Research, 1(1):129-145.
Ona de Gibert, Naiara Pérez. Aitor Gareia-Pablos, and Montse Cuadros. 2018. Hate speech dataset from white supremacy forum. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2). pages 11-20.
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Pe- ter Lee. Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general in- telligence: Early experiments with gpt-4.
Catre-Catherine de Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Investi- gating projection in naturally occurring discourse.
Michael Chen, Mike D' Arcy, Alisa Liu, Jared Fer- nandez, and Doug Downey. 2019. CODAH: An question answering dataset for common sense. In Proceedings of the 3rd Work- shop on Evaluating Vector Space Representations for NLP. pages 63-69, Minneapolis, USA. Association for Computational Linguistics.
Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. 2021. The benchmark lottery.
adversarially-authorr
Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of- thought for large language models.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov. Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-HHellstenn, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bu- lian, Massimiliano Ciaramita, and Markus Leippold. 2020. Climate- fever: A dataset for verification of real-world climate claims.
William B Dolan and Chris Brockett. 2005. Automati- cally constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. ArXiv,
Liat Fin-Dor, Alon Halfon, Ariel Gera, Ryal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2020. Active Learning for BERT: An Empirical Study. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 7949-7962, Online. Association for Computa- tional Linguisties.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. 2017. Deep Bayesian active learning with image data, Proceedings of the 34th International Conference
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang,

---

on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1183-1192 PMLR.
Itay Levy, Ben Bogin, and Jonathan Berant, 2022. Di- verse demonstrations improve in-context composi- tional generalization.
Xavier Gareía, Yamini Bansal, Colin Cherry, George F. Foster, Maxim Krikun, Fan Feng, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effective- ness of few-shot learning for machine translation.
David D. Lewis and William A. Gale. 1994. se- quential algorithm for training text classifiers. In In Proceedings of the Annual International ACM SI- GIR Conference on Research and Development in Information Retrieval.
Shivam Garg. Dimitris Tsipras, Percy Liang, and Gre- gory Valiant. 2022. What can transformers learn in-context? a case study of simple function classes. ArXiv, 0as/22008000666
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan,
Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-114, Dublin, Ireland and Online, Associa- tion for Computational Linguistics.
Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, and Luke Zeulemoyer. 2022. Demystifying prompts in language models via perplexity estimation. ArXiv,
abs/2212.04037.
S. Longpre, Julia Reisler, Edward Greg Huang. Yi Lu, Andrew J. Frank, Nikhil Ramesh, and Chris DuBois. 2022. Active learning language tasks. ArXiv, abs/2202.00255
Andrew Gordon, Zornitsa Kozareva, and Melissa Roem- mele. 2012. SemEval-2012 task 7: Choice plau- sible alternatives: An evaluation of commonsense causal reasoning. In #SEM 2012: The First Joint Conference on Lexical and Computational Seman- tics- - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Eval- uation (SemEval 2012), pages 394-398, Montréal, Canada. Association for Computational Linguisties.
multiple domains in natu-
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel. and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Joel Jang, Seonghyeon Ye, and Minjoon Sco, 2022. Can large language models truly understand prompts? case study with negated prompts. ArXiv, abs/2209.12711.
Aman Madaan and Amir Yazdanbakhsh. 2022. Text and patterns: For effective chain of thought, it takes two to tango. ArXiv, 0abs/2209007686
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qase: A dataset for question answering via sentence compo- sition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8082-8090.
Katerina Margatina and Nikolaos Aletras. 2023. On the limitations of simulating active learning. In Find- ings of the Association for Computational Linguisties: ACL 2023, pages 4402-4419.
Katerina Margatina, Loic Barrault, and Nikolaos Ale- tras. 2022. On the importance of effectively adapting pretrained language models for active learning. In Proceedings of the 60th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 825-836, Dublin, Ireland. As- sociation for Computational Linguistics.
Hyuhng Joon Kim, Hyunsoo Cho, Junyeob Kim, Taeuk Kim, Kang Min Yoo, and Sang goo Lee. 2022.
Self-generater in-context learning: Leveraging auto- regressive language models as a demonstration gen- erator. ArXiv, abs/2206.08000 08082.
Andreas Kirsch, Tom Rainforth, and Yarin Gal. 2021. Test distribution-aaarte active learning: A principled approach against distribution shift and outliers.
Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. 2021. Active learning by ac- quiring contrastive examples. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 650-663, Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Rafal Kocielnik, Sara Kangaslahti, Shrimai Prabhu- moye, M Hari, R. Michael Alvarez, and Anima Anandkumar. 2022. Can you label less by using out-of-domain data? active & transfer learning with few-shot instructions. ArXiv, abs/2211.11798.
Clara H. McCreery, Namit Katariya, Anitha Kannan, Manish Chablani, and Xavier Amatriain. 2020. Effec- tive transfer learning for identifying similar questions: Matching user questions covid-19 19 faqs.
Abdullatif Köksal, Timo Schick, and Hinrich Schutze. 2022. Meal: Stable and active learning for few-shot prompting. ArXiv, abs/2211 1.08358.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work?

---

Shervin Minace, Nal Kalchbrenner, Erik Cambria, Nar- jes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2021. Deep learning-based text classification; com- prehensive review, ACM computing surveys (CSUR), 14(3):1-40.
language models: Methods, analysis & insights from training gopher.
Lukas Rauch, Matthias Aßenmacher, Denis Huseljic, Moritz Wirth, Bernd Bischl, and Bernhard Sick. 2023, Activeglae: A benchmark for deep active learning with transformers.
Akiva Miura, Graham Neubig, Michael Paul, and Satoshi Nakamura. 2016. Selecting syntactic, non- redundant segments in active learning for machine translation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. pages 20-29, Stroudsburg, PA, USA. Association for Computational Linguistics.
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 840- 054, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
loannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. 2022. ETHOS: a multi- label hate speech detection dataset. Complex Intelli- gent Systems, 8(6):4663-465 4678.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNN) 3982-3992, Hong Kong, China, Association for Com- putational Linguistics.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
pages
Laria Reynolds and Kyle McDonell. 2021. Prompt pro- gramming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Com- puting Systems, CHI EA '21, New York, NY, USA. Association for Computing Machinery.
Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. 2023. What in-context learning "learns" in-context: Disentangling task recognition and task learning.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. pages 2655-2671, Scattle, United States. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Nya Sutskever. 2019. Language models are unsupervised multitask learners.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young. Eliza Rutherford, Tom Hennigan, Ja- cob Menick, Albin Cassirer, Richard Powell, George van den Driessche. Lisa Anne Hendricks, Mari- beth Rauh, Po-Sen Huang, Amelia Glaese, Jo- hannes Welbl, Sumannh Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Anto- nia Creswell, Nat McAleese. Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Bud- den, Esme Sutherland, Karen Simonyan, Michela Pa- ganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Yean-Baptise Lespiau. Maria Tsim- poukelli, Nikolai Grigorev, Doug Fritz, Thibault Sot- tiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama. Cyprien de Masson d' Autume, Yujia Li. Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel. William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Benneti, Demis Hassabis, Ko- ray Kavukcuoglu, and Geoffrey Irving. 2022. Scaling
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. Are emergent abilities of large language mod- els mirage?
Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 255-269, Online. Association for Computa- tional Linguistics.
Timo Schick and Hinrich Schütze. 202 It's not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339-2352, Online. Association for Computational Linguisties.
Christopher Schröder, Lydia Müller, Andreas Niekler, and Martin Potthast. 2023. Small-text: Active learn ing for text classification in python. In Proceedings of the 17th Conference of the European Chapter of

---

the Association for Computational Linguistics: Sys- tem Demonstrations, pages 14-95, Dubrovnik, Croa- tia. Association for Computational Linguistics.
Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski. Batuhan Özyurt, Behnam Hedayatmia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ek- mekci, Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Ar- gueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng. Chitta Baral. Chiyu Wu, Chris Callison-Buree Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raf- fel. Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy. Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Ju- rgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dicuwke Hupkes, Diganta Misra, Dilyar Buzam, Dim- itri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Ekate- rina Shutova, Ekkn Dogus Cubuk, Flad Segal, Bleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, El- lie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu. Eric Tang, Erkut Erdem, Ernie Chang. Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice En- gefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinez-Plumeel Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Ger- mán Kruszewski, Giambattista Parascandolo, Gior- gio Mariani, Gloria Wang, Gonzalo Jaimovitch- López. Gregor Betz. Guy Gur-Ari, Hana Galijase- vic, Hannah Kim, Hannah Rashkin, Hannaneh Ha- jishirzi. Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng. Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jae- hoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Ko- coñ, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Ja- son Yosinski, Jekaterina Novikova, Jelle Bosscher. Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse En- gel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jil- lian Tang. Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez Orallo,, Joseph Boudeman. Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan. Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gim- pel, Kevin Omondi, Kory Mathewson, Kristen Chi- afullo, Ksenia Shkaruta, Kumar Shridhar, Kyle Mc- Donell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang. Liam Dugan, Lianhui Qin, Lidia Contreras- Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui. Mantas Mazeika, Marco Bauran, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn,
Ozan Sener and Silvio Savarese. 2018. Active learn- ing for convolutional neural networks: A core-set approach. In International Conference on Learning Representations.
Chris Waites, Christian Voigt,
Burr Settles. 2009. Active learning literature survey, Computer Sciences Technical Report 1648, Univer- sity of Wisconsin-Madison.
Yanyao Shen, Hyokun Yun, Zachary Lipton, Yakov Kronrod, and Animashree Anandkumar. 2017. Deep active learning for named entity recognition. In Pro- ceedings of the Workshop on Representation Learn- ing.orr for NLP. pages 252-256.
Emily Sheng and David C Uthus. 2020. Investigating societal biases in poetry composition system. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 93-106.
Weijia Shi. Xiaochuang Han. Hila Gonen. Ari Holtzman, Yulia Tsvetkov, and Luke Zettlemoyer. 2022. Toward human readable prompt tuning. Kubrick's 's the shining is a good movie, and a good prompt too? ArXiv, abs/2212.105399
Ard Snijders, Douwe Kiela, and Katerina Margatina. 2023. Investigating multi-source active learning for natural language inference. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 2 2187- 2209, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
Taylor Sorensen, Joshua Robinson, Christopher Ryt- ting, Alexander Shaw, Kyle Rogers, Alexia Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. 2022. An information- -ereottionthhooettic approach to prompt engineering without ground truth labels. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pa- pers). pages 192-966221166 Dublin, Ireland. Association for Computational Linguistics.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowyez, Akshat Agarwal, Alethca Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Par- rish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane. Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, An- drew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang. Angelica Chen. Anh Vuong. Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabas- sum, Arul Menezes, Arun Kirubarajan, Asher Mul- lokandov, Ashish Sabharwal, Austin Herrick, Avia

---

Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schu- bert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Co- hen, Michael Gu, Michael Ivanitskiy, Michael Star- rit,, Michael Strube, Michal Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo Tiwari, Mo- hit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T. Nanyun Peng, Nathan Chi. Nayeon Lee. Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Ni- tish Shirish Keskar, Niveditha S. Iyer. Noah Con- stant, Noah Fiedel, Nuan Wen, Oliver Zhang. Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang. Paul Vicol. Pegah Alipoormo- labashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eekersley, Phu Mon Htut, Pinyu Hwang. Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramón Risco Delgado. Raphaël Millière, Rhythm Garg. Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymackers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le- Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Rus- lan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Sto- vall, Ryan Teehan, Rylan Yang. Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebas- tian Gehrmann. Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixi- ang Shane Gu, Shubh Pachchigar, Shubham Tosh- niwal, Shyam Upadhyay. Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee. Spencer Torene, Sriharsha Hatwar, Stanislas De- haene, Stefan Divic, Stefano Ermon, Stella Bider- man, Stephanie Lin, Stephen Prasad, Steven T. Pi- antadosi, Stwart M. Shieber, Summer Misherghi, Svet- lana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali. Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes. Theodore Rothschild, Thomas Phan, Tianle Wang. Tiberius Nkinyíli, Timo Schick. Timofei Kornev, Timothy Telleen- Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj. Tushar Khot, Tyler Shultz, Uri Sha- ham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fe- dus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov. Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui
Wang, and Ziyi Wu. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2019a. Quarel: A dataset and models for answering questions about qualitative relationships. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7063- 7071.
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019b. Quartz: An open-domain dataset of qualitative relationship questions. In Proceedings of the 2019 Conference on Empirical Methods in Natu- rul Language Processing and the 9th International Joint Conference on Natural Language Processing JEMNLP-IJCNLLS pages 5941-5946.
Alex Tamkin, Dat Pham Nguyen. Salil Deshpande, Jesse Mu, and Noah Goodman. 2022. Active learning helps pretrained models learn the intended task. In Advances in Neural Information Processing Systems.
Simone Tedeschi, Johan Bos, Thierry Declerek, Jan Ha- jic, Daniel Hershcovich, Eduard H. Hovy, Alexander Koller, Simon Krek, Steven Schockaert, Rico Sen- nrich, Ekaterina Shutova, and Roberto Navigli. 2023. What's 's the meaning of superhuman performance in today's 's nlu? ArXiv, abs/2305.08411
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Marcos Treviso, Ji- Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Hcafield, Sara Hooker, Colin Raffel, Pe- dro H. Martins, André F. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubra- manian, Leon Derezynski, Iryna Gurevych, and Roy Schwartz. 2023. Efficient Methods for Natural Lan- guage Processing: A Survey. Transactions of the Association for Computational Linguisties, 11:826- 860.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations.
Boshi Wang. Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2022. Towards understanding ohthn-of- prompt- ing: An empirical study of what matters. ArXiv, abs/2212.10001.
Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning their prompts? In Proceedings of the 2022 Conference of

---

the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2300-2344, Seattle, United States. Association for Computational Linguisties.
Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyun- soo Cho, Hwiyeol Jo. Sang- Woo Lee, Sang-goo Lee, and Taeuk Kim. 2022. Ground-truth labels matter: A deeper look into input-label demonstrations.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud. Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emer- gent abilities of large language models.
W. Yu, Dan Iter. Shuohang Wang. Yichong Xu. Mingx- uan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022. Generate rather than retrieve: Large language models are strong context generators. ArXiv, abs/2209.10063.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zenteeoyer.. 2022a. Opt: Open pre-trained transformer language models.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023a. Chain-of-thoughe prompting elicits reasoning in large language models.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023b. Larger language models do in-context learning dif- ferently.
Yiming Zhang, Shi Feng, and Chenhao Tan. 2022b. Ac- tive example selection for in-context learning. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 9134- 9148, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Qiang Wei, Yukun Chen, Mandana Salimi, Joshua C Denny, Qiaozhu Mei, Thomas Lasko, Qingxia Chen, Stephen Wu, Amy Franklin, Trevor Cohen, and Hua Xu. 2019. Cost-aware active learning for named entity recognition in clinical text. Journal of the American Medical Informatics Association, 26(11):131--332.
Zhisong Zhang. Emma Strubell, and Eduard Hovy. 2022c. A survey of active learning for natural lan- guage processing. In Proceedings of the Conference on Empirical Methods Natural Language Process- ing.
Zhiyong Wu, Yaoxiang Wang, Jiacheng Yc, and Ling- peng Kong. 2022. Self-adaptive in-context learning. ArXiv, abs/2212.10375.
Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Im- proving few-shot performance of language models. ICML, abs/2
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. An explanation of in-context learning as implicit bayesian inference. In Interna- tional Conference on Learning Representations.
Yuekai Zhao, Haoran Zhang, Shuchang Zhou, and Zhi- hua Zhang. 2020. Active learning approaches enhamcing neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1796-1806, Online, Association for Computational Linguistics.
Canwen Xu, Yichong Xu, Shuohang Wang. Yang Liu, Chenguang Zhu, and Julian McAuley. 2023. Small models are valuable plug-ins ins for large language mod- els.
Sohee Yang, Jonghyeon Kim, Joel Jang, Seonghyeon Ye, Hyunji Lee, and Minjoon Seo. 2023. Improving probability- based prompt selection through unified evaluation and analysis.
Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023a. Compositional exemplars for in-context learning.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. CrossFit: A few-shot learning challenge for cross- task generalization in NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7 7163-7189, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xi Ye, Srinivasan Iyer, Asli Celikyilmaz, Ves Stoy- anow, Greg Durrett, and Ramakanth Pasunuru. 2023b. Complementary explanations for effective in-context learning. Findings of the Conference of the Asso- ciation for Computational Linguistics.

---

Clessification
Muiti-choice
0.8
64
03
AL methads random dicersiy
0.4
urierssiity
similanlyy
Figure Results per model family.
A Experimental Details
A.1 Tasks & Datasets
Following Min et al. (2022), we evaluate our models in 15 classification and 9 multi-choice tasks taken from the Crossfit (Ye et al., 2021) benchmark. Specifically the tasks we evaluate are poem_see serm__sntimen/ (Sheng and Uthus, 2020), glue- wnli (Wang al., 2019; Levesque al., 2012), climate_ (Diggelmann et al., 2020), glue- rte (Wang al., 2019), superglue-cb (de Marn- effe et al., 2019). sick (Minaee et al., 2021), medi- cal LoL_qquettoos___pe pairs (McCreery et al., 2020), glue- mrpc (Wang et al., 2019; Dolan and Brockett, 2005), hate ctatesppecchl (de Gibert et al., 2018), ethos-national_origee _origin (Mollas et al., 2022), ethos- race (Mollas al., 2022), ethos-religion (Mollas et al., 2022), sweet_eval-stane aatheism (Barbieri al., 2020), arrei__evl-stance_eee feminiit (Barbi- eri et al., 2020) and quarel (Tafjord al., 2019a), openbookqa.qass (Khot et al., 2020), common- sense_gu ai2_arc (Clark et al., 2018), codah (Chen et al., 2019), superglue-copa (Gordon et al., 2012), muaroe-with_knowwedgee (Tafjord et al., 2019b), mourtz-no_knowleddn (Tafjord et al., 2019b), for classification and multi-choice respectively.
A.2 Full results
We provide below the full set of results, for each dataset, model and active learning acquisition strat- egy considered. The dashed line depicts the major- ity vote baseline.
A.3 Model Family
We provide the results on few-shot learning with k=16 demonstrations per prompt per model family and task type in Figure 9. We observe the same patterns for both GPT and OPT models.

---

gpt2 (124M)
Classification
AL methods random diversity
0.5
uncertainty
similarity
0.0
reggon
doe
Sie
mvsicol detioons pabr
prem tentmeed
brstt avt aatisaacce atheres
dinate
Multi-choice
1.0
accuracy
0.5
AL methods
random diversity
0.0
uncertainty
bard
wortd dg Jonetodge
Besto Lncelaogg
similarity
gpt2-medium (355M) Classification
AL methods random diversity uncertainty
0.5
0.0
similarity
cliost fover
maps
storte
late Soeecild
mestral proatoor centouts
eneegieeet 4D
Grent dabaanaa athesm
Buent tis formived
Wbit retional
Multi-choice
1.0
accuracy
0.5
AL methods random diversity uncertainty similarity
0.0
with

---

gpt2-large (774M) Classification
AL methods
0.5
random diversity
uncertainty
0.0
similarity
reigon
doe
Sie
mvsicol petioons 2005
poem tentmned
brstt avt avdiasance, atheres
dinate
Multi-choice
1.0
accuracy
0.5
AL methods
random diversity
0.0
uncertainty
bard
morte dg Jonetogg
similarity
gpt2-xl (1.5B)
Classification
AL methods random diversity uncertainty
0.5
0.0
similarity
diiast Ioser
maps
storte
lale Soeecild
mestral prnators centouts
Grent sabaanaa athesmm
Buentt tes formived
Wbot rntional
Multi-choice
1.0
accuracy
0.5
AL methods random diversity uncertainty similarity
0.0
with

---

gpt-neo-1.38 Classification
AL methods random diversity
0.5
uncertainty
0.0
similarity
reigon
doe
mvsicol petioons, pabr
prem centmeed
bestt avd atheres
dinate
Multi-choice
1.0
accuracy
0.5
AL methods
random diversity
0.0
uncertainty
worte dg Jonetogge
similarity
gpt-neo-2.7B
Classification
AL methods random diversity uncertainty
0.5
similarity
0.0
clinst Ioser
maps
storre
lale Sooccild
Grent aataaaaa athesm
Buent tis fortive
Wbit rntional
mestral proatoor
Multi-choice
1.0
accuracy
0.5
AL methods random diversity uncertainty similarity
0.0
with

---

gpt-j-6B Classification
AL methods random diversity
0.5
uncertainty
0.0
similarity
reigon
doe
evsicol petioons 2007
brstt avd atheres
dinate
Multi-choice
1.0
accuracy
0.5
AL methods
random diversity
0.0
uncertainty
bar
dp Jonetogg
similarity
gpt-neox-20b
Classification
AL methods
0.5
random diversity uncertainty
0.0
similarity
cliast foser
maps
storre
late Gomctid
brent aabaanaa athesm
Beent mes stance formive
Wbit ontional
mestral proatoor
Multi-choice
1.0
accuracy
0.5
AL methods random diversity uncertainty similarity
0.0
with

---

opt-125m Classification
AL methods random diversity
0.5
uncertainty
0.0
similarity
reigon
doe
Sie
mvsicol detioons 2005
prem tentmeed
benot avdl atheres
dinate
Multi-choice
1.0
accuracy
0.5
AL methods
random diversity
0.0
uncertainty
dp Jonetogg
Besto Incetaigg
similarity
opt-350m Classification
AL methods random diversity uncertainty
0.5
0.0
similarity
clinst Iover
maps
storre
lale Seoecild
brent salaanaae athesm
Buent mes formived
Wbot retional
mestral proatorr
Multi-choice
1.0
accuracy
0.5
AL methods random diversity uncertainty similarity
0.0
with

---

opt-1.3b
Classification
AL methods random diversity
0.5
uncertainty
similarity
0.0
reggon
doe
mvsicol petioons 20b7
prem centmned
brsot avd atheres
dinate
Multi-choice
1.0
accuracy
0.5
AL methods
random diversity
0.0
uncertainty
MC
dg Jonetogge
Basto Incetaogg
similarity
opt-2.77 7b
Classification
AL methods random diversity uncertainty
0.5
similarity
0.0
clindt Ioser
maps
storre
lale Soeccild
mestral proatoor centouts
eneegieeet 4
brent aalaanaa, athesm
Buent tes stance formive
oetional
Multi-choice
1.0
accuracy
0.5
AL methods random diversity uncertainty similarity
0.0
with

---

opt-6.70 7b
Classification
AL methods random diversity
0.5
uncertainty
0.0
similarity
reggon
Boe
Sie
mvsicol detioons, 20b7
prem centmved
bestt avd atheres
dinate
Multi-choice
1.0
accuracy
0.5
AL methods
random diversity
0.0
uncertainty
dp Jonetogg
similarity
opt-13b
Classification
AL methods random diversity uncertainty similarity
0.5
0.0
clinst Iover
storre
late Soeecild
eneengeeet 4
Grent salaanaae athesm
Beent mes stance formive
Wbit retional
mestral proatoor
Multi-choice
1.0
accuracy
0.5
AL methods random diversity uncertainty similarity
0.0
with

---

opt-30b Classification
AL methods random diversity
0.5
uncertainty
0.0
similarity
(Bimat Atier
medical pwsstion Shacy
breet eos ortesin
hets
tres shance
Multi-choice
1.0
accuracy
0.5
AL methods random diversity uncertainty similarity
0.0
Besto Inqulcade


Reviews:
[{"review_id": "EfOAPov7OM", "reviewer": "Reviewer_RZBu", "summary": "", "questions": "- How would you expect the ranking of the different methods to change based on the choice of $k$? \n- Overall, I find it unclear what are the actual contributions of the work since most findings have already been observed in previous work. Can you please clarify this?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer RZBu for their thoughtful and constructive review of our paper. We appreciate that the reviewer considers our paper clear and the subject of our study to be of high importance. We address each of their comments below.\n\n- Choice of Clustering Methods:  \nWe acknowledge the comment on using different clustering methods for diversity and similarity methods, potentially clouding their proper assessment. Still, we believe that the fair comparison between diversity and similarity methods  is justified by the use of the same representations (SBERT embeddings). Furthermore, given that k-means is an unsupervised clustering technique, it is non-trivial how it could be applied to the similarity selection method. We opted to use algorithms proposed in prior work off-the-shelf for a fair comparison and to avoid confusion. We will clarify this point in the revised version of the paper.\n\n- Contradictory Conclusions:  \nWe appreciate the reviewer's observation regarding contradictory conclusions based on different evaluation metrics. However, this is a common issue with in-context learning examples that pertains across the literature [1,2]. Our intention was to better highlight this issue rather than hiding it by including the results for both evaluation metrics in the main body of the paper.  We will make sure to clarify this part more in the camera ready version of the paper. We agree that combining both similar and diverse examples in a prompt is insightful and an interesting avenue for future work. In this paper we opted to focus on an extensive empirical study  of general active learning (AL) inspired data selection methods instead of trying to propose a specific algorithm. We hope that the findings and analysis of our paper will inspire future work on the impact of similarity, diversity and uncertainty, and explore the synergies between them. We will make sure to propose explicit avenues for further research in the updated version of the paper.\n\n- Impact of Number of Demonstrations & Method Ranking:  \nThe reviewer's point about the impact of the number of demonstrations on the studied AL methods and their relative ranking is well-taken. We had explored different numbers of k (1,4,8,16) and did not find any different patterns (i.e., different selection methods performed similarly, independently of the choice of k). This is why we chose to perform the bulk of the experiments with k=16, as in prior work [1,2]. We will include a short analysis that explores different numbers of demonstrations and their effects on method performance in the final version of the paper.\n\n- Clarification of Contributions & Soundness:  \nWe regret any ambiguity in conveying the contributions of our work. While some findings have been observed in previous studies, our paper offers a systematic comparison of different AL-inspired methods specifically in the context of prompting pre-trained language models. We have conducted a thorough study of 15 language models and 26 datasets, comparing 4 representative data selection methods. We believe that the soundness of our experimental work is solid, which is also mentioned by reviewers 87v7 and BaZZ (who both gave us a score of 4). Furthermore, our experiments illustrate that prioritizing the application of the similarity method in demonstration selection can outweigh the significance of model size in classification tasks. For instance, we observe in Figure 4 that OPT models from 125M to 30B parameters perform almost equally when using the similarity method. This surprising finding suggests that investing in superior-quality demonstrations could yield more favorable outcomes compared to the utilization of larger, cost-intensive models, which leads to favorable implications for model efficiency in general. Additionally, our investigation into the interplay between the top and bottom candidate examples from the pool in terms of uncertainty and similarity (Section 5), across different model families and number of parameters, contribute novel insights to the field. We will revise the paper upon acceptance to better emphasize these contributions.\n\nWe are committed to addressing all concerns and suggestions raised by the reviewer in order to improve our paper. We believe that the revisions proposed will strengthen the overall contribution and merit of the work. \n\n\n[1] Rethinking the role of demonstrations: What makes in-context learning work? Min et al., 2022.  \n[2] Ground-truth labels matter: A deeper look into input-label demonstrations. Yoo et al., 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is clear and well-written.\n- The paper tackles the important and timely problem of improving the prompting of pre-trained language models such that no fine-tuning is needed.", "reasons_to_reject": "- The diversity and similarity measures use different clustering methods (k-means vs knn) which might cloud the proper assessment of each attribute. Instead of using $k$NN for the similarity measure, you can use SBERT embedding clustering and pick $k$ random points in the same cluster as the test point (or the closest cluster if there aren't $k$ points in the test point cluster).\n- The contradictory conclusions when using different evaluation metrics (F1 vs accuracy) are rather alarming and suggest that more work needs to be done to reach a solid conclusion. Particularly, this might suggest that a combination of similar and diverse examples in a given prompt might achieve the best overall performance.\n- The effect of the number of demonstrations $k$ is also unclear; exploring how different $k$ values impact the studied AL methods (and their relative ranking) seems pertinent.", "questions_to_authors": "- How would you expect the ranking of the different methods to change based on the choice of $k$? \n- Overall, I find it unclear what are the actual contributions of the work since most findings have already been observed in previous work. Can you please clarify this?", "paper_topic_and_main_contributions": "This paper studies several active learning methods that aim at selecting a subset of demonstrations from a pool to then use them as prompts to large language models. The authors analyze four simple approaches based on uncertainty, diversity, similarity, and random sampling. The experimental results on GPT and OPT models with several sizes suggest that the similarity between the selected demonstrations and the test example makes for the best-performing prompt.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "I recommend writing Srivastava et al instead of listing all the author names.", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 032: inin-context"}, "extra_text": {}}, {"review_id": "H5OciF6oEv", "reviewer": "Reviewer_87v7", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 87v7 for their detailed and insightful review of our paper. We appreciate the reviewer’s comments that our findings will likely impact future studies on this area, while our extensive analysis improves the reader's confidence in the generalizability of the findings.\n\n We respond to each of their comments and suggestions below.\n\n- Clarification on Novelty:  \nWe appreciate the reviewer's comment regarding the novelty of treating the in-context learning (ICL) example selection as an active learning (AL) problem. Throughout our paper, we have refrained from claiming the introduction of a novel AL algorithm specifically tailored for the ICL example selection task. Instead, we recognize the parallels between data selection in ICL and the AL paradigm. Our aim was to conduct an extensive empirical study that explores how established algorithms from the AL domain could be adapted and applied to the ICL framework. We have incorporated a comprehensive compilation of related papers from both the AL and ICL fields, as expected, aiming to offer researchers a go-to guide as they address the problem of few-shot learning data selection via the AL lens. Our intention is to present an empirical study that offers a comprehensive analysis of the strengths and limitations of existing data selection methodologies inspired by AL algorithms (we chose the relevant track “Language Modeling and Analysis of Language Models” for submission accordingly). Conducting an exhaustive series of experiments across diverse models, ranging in size and family, as well as multiple tasks and datasets, constituted a pivotal facet of our study. We acknowledge that the results we obtained corroborate in part previous findings, and we have clearly acknowledged this alignment in our paper. We do not consider this problematic, given that we had a different starting point and a more thorough experimental setup. Surprisingly, we showed that smaller models with demonstrations drawn with the similarity method can outperform larger ones that use different in-context selection methods (Figure 4, OPT models in classification tasks). This is an impactful finding that has not been presented before in previous work. Nevertheless, we believe that the true novelty of our paper resides predominantly within the thorough analysis in Section 5. Our analysis explores the properties of \"good\" in-context examples and lays the foundation for future research to devise algorithms grounded in these properties. Sections 5.3 and 5.4 distinctly encapsulate our exploration of this pivotal research question. The implications we unveil, particularly those pertaining to a model's capacity to leverage uncertainty and the role of model scale (Section 5.4), resonate as intriguing, influential, and indeed novel insights. We deeply value the reviewer's feedback, and we are committed to elucidating our contributions more explicitly in the revised manuscript. \n\n- Addressing Known Phenomena:  \nAs previously outlined, we acknowledge that while some of our findings may align with previous work, the application of these principles to the specific context of analysing AL inspired methods for few-shot learning and the systematic evaluation across models and tasks still provide valuable insights. Specifically regarding the highly cited KATE paper, our investigation surpasses its scope by exploring the impact of selecting in-context examples semantically similar to the test input across a comprehensive array of 15 models and 24 datasets encompassing multiple tasks. In contrast, KATE's purview was confined to GPT-3, limited to three datasets spanning sentiment analysis, table-to-text generation, and question answering. Our extended experimental setup not only reaffirms the utility of selecting such demonstrations but also substantiates the generalizability of this conclusion across diverse tasks and models. Furthermore, the importance of our work lies in our exploration of the similarity method across a vast spectrum of model sizes, spanning from a few million parameters to tens of billions. This robust spectrum substantiates the rather surprising finding that prioritizing the similarity-based example selection methodology can outweigh the relative influence of model size in the classification. This hints the prospect that investing in high-quality demonstrations could be more beneficial than deploying resource-intensive large-scale models, which often entail considerable operational costs. To deepen our contribution, we will include a more extensive discussion on the implications of these findings, separating us from prior work. We will better highlight all these points given the extra space in the camera ready upon acceptance.\n\n- Combining Criteria and Further Directions:  \nWe appreciate the reviewer's suggestions for exploring the combination of similarity, diversity, and uncertainty in a novel selection procedure. We agree that this avenue holds promise for advancing the data selection problem we explore. In the revised version upon acceptance, we will discuss the potential benefits and challenges of such combinations, acknowledging that a balanced approach might yield even better results. We will provide further discussion on how this finding could shape future research directions, potentially leading to strategies that, for instance, exploit uncertainty more effectively for larger models or specific tasks.\n\nWe are grateful for the reviewer's suggestions to enhance the clarity, significance, and applicability of our work. We believe that their concerns can be easily addressed in the camera ready version of the paper  by providing a more clear presentation of our contributions. We acknowledge that the revisions proposed will elevate the quality of our paper and its potential impact on the community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Given the popularity of few-shot learning and in-context learning in LLMs, selecting the ICL examples has become a hot area of research. This paper presents a timely and systematic analysis of this topic, and its findings will likely impact future works working on similar areas.\n\n- The analysis is largely thorough and extensive, considering the number of tasks and models used. This improves the reader's confidence in the generalizability of the findings.", "reasons_to_reject": "While the authors have tried to argue the novelty of their paper in treating the ICL example selection problem as an active learning example and claimed that they are the first to do so, I have not fully grasped any substantive difference from previous works in terms of execution of the algorithm, and the claimed difference seems to lie fully in narrative and rhetoric -- I am therefore requesting the authors to clarify this crucial point in the author rebuttal in how the proposed casting of the problem into an active learning one *substantively* differentiate the paper from the previous works on similar topics. Specifically, my concerns are:\n  - Finding 1, as I listed in *Paper Topic And Main Contributions*, is largely known from the widely-cited KATE paper the authors also cited in their paper. While it is nice that authors show that retrieving similar queries as ICL examples works the best across a wide number of model and task combinations, I wonder in what ways this paper furthers or deepens understanding of this phenomenon apart from confirming the KATE paper?\n  - While there might not be previous work explicitly presenting Finding 2 in the exactly same setup, many papers have hinted at this point, and an example is AutoCoT which concludes that retrieving most similar examples does not work when the demonstrations are imperfect and maximising diversity improves performance. Again, while this paper confirms that such benefit exists even in the few-shot setup, which is expected, I have not fully grasped the implication beyond that, similar to point 1.\n  - Finding 3 is perhaps the most surprising and most related to the active learning paradigm the authors have referred to. However, the paper neither provides convincing explanations nor practical next steps on how to act upon this (although I agree this is an interesting finding).\n\nOverall, while this paper presents very solid empirical evidence and can be potentially useful as a technical report, it seems to me that it is mostly confirming known phenomena and has provided limited further insights. A possible way forward, in my opinion, is given that the authors have confirmed the superiority of similarity and diversity (and the general inferiority of uncertainty). Is there an optimal way to combine these criteria into a novel selection procedure that balances these principles, or, alternatively, is the learning-based method such as [1] implicitly performing retrieval based on one or more criteria mentioned in the paper? These are just some non-exhaustive suggestions that the authors may possibly consider.\n\n\n### References\n\n[1] Rubin, O., Herzig, J., & Berant, J. (2021). Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633.\n\n\n***\n\nAfter rebuttal: I thank the authors for their detailed feedback. I agree with the authors' claim on the extensiveness and soundness of the experiments, and I remain convinced that this paper is sound, can potentially be an influential source for future references, and therefore spur further research in this direction -- I have therefore kept my high soundness score.\n\nRegarding the excitement assessment, I have the impression that the author feedback largely confirmed my initial assessment. While the experiments are indeed thorough across a wide range of tasks and model choices, since the excitement score specifically asks for aspects of novelty and interestingness, I tend to keep my original score given that most analyses are re-confirmations and extensions of known results. There are indeed some new results, such as the effect of model size and the role uncertainty plays, but like I mentioned in the original review, I still think the arguments, especially regarding their practical uses, are rather underdeveloped. Although the authors made several promises of future exploration in response to my raised points, unfortunately, such extensions are yet to be done. Please be aware that I am not as negative as the score description would otherwise suggest literally, but I still regard the novelty aspect as a major, pending concern.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper systematically studies various principles in selecting in-context learning (ICL) examples for few-shot learning, tested on several LLMs of varying sizes and across many classification and multi-choice tasks. The key findings of the paper are 1) semantically similar samples serve as better ICL examples, 2) sampling diverse samples is promising, and 3) perhaps most surprisingly, selecting uncertain samples, while widely used in active learning literature, does not lead to performance improvement for smaller models while the larger models may take advantage of the uncertain demonstrations to a certain extent.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EwR2w3nvZb", "reviewer": "Reviewer_BaZZ", "summary": "", "questions": "line 340: you claim you do not see any specific patterns, but I think there are patterns, at least in the right panel in Figure 4. The left panel is really interesting: it seems that the similarity approach that you study boosts classification performance a lot, and then model size stops improving the performance, relatively speaking. Could you discuss that in a bit more depth? I guess you could say that there is no pattern for the other methods too, i.e. that the other methods (uncertainty, diversity and random) seem to not improve with model size. Interesting, what is the difference between classification and in-context learning here? Maybe the larger the models get the association between the exemplar and its label is better, and it is thus well utilized to make inference about the test sample.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer BaZZ for their comprehensive review of our paper. We are grateful for the valuable feedback and insightful suggestions they have provided. We address each of their points and questions below.\n\n- Role of Sentence Embeddings:  \nWe thank the reviewer for highlighting the potential influence of SentenceBERT embeddings in driving the strong results obtained with the similarity method. We agree that exploring the role of these embeddings bears profound significance. Within the updated version of our paper, we intend to explicitly present this aspect as a plausible trajectory for subsequent investigation. Such follow-up work could focus on discerning the embeddings that yield optimal advantages for the similarity-based data selection approach and articulating the intrinsic attributes that these embeddings should have. Previous studies and our own research, undoubtedly show that the underpinning factor for high performing in-context learning  lies in the semantic similarity between in-context examples and the test input of the prompt. Consequently, future work could investigate in detail what is the most beneficial and efficient way to perform data selection.\n\n- Analysis of Patterns in Figure 4:  \nWe appreciate the reviewer's observation of the patterns in Figure 4. Our comment that we do not observe any patterns in L340 was to convey that as the models’ size is increasing, there is not a data selection method that would benefit more than any other method. In other words, the ranking between the data election methods we explored was more or less the same across all model sizes. We will certainly provide a more in-depth discussion in the camera ready version of the paper to address these findings. Furthermore, the reviewer correctly mentioned that few-shot learning with demonstrations selected with the ‘similarity’ method outperforms larger models that use few-shot examples drawn with different approaches in the classification tasks. We believe that this is one of the most interesting findings in our study, showing that it is perhaps better to invest in finding the most useful demonstrations than using larger models that are expensive for inference. We do observe a difference between classification and multi-choice tasks. In the former, we hypothesize that smaller models can harness well-chosen demonstrations to aptly differentiate between classes due to the relatively 'easier' nature of the task. In contrast, multi-choice tasks necessitate more substantial models to achieve satisfactory outcomes. This discrepancy can potentially be attributed to the distinct complexities inherent in each task category. We will delve into this hypothesis and analyze the interplay between model size, demonstration utilization, and the nature of the tasks in greater detail given the extra space in the camera ready upon acceptance.\n\n\nWe are thankful for the insightful suggestions and questions that Reviewer BaZZ has brought to our attention. We will incorporate these suggestions into the revised version of the paper to enhance its quality, clarity, and impact.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is very clearly written, and it attacks an important practical problem in the era of LLMs. \n\n2. The experiments and analysis are well-designed. \n\n3. The results are interesting and could be used as reference to future work.", "reasons_to_reject": "1. I was quite surprised that given the success of the similarity method, the authors didn't study the role of SentenceBERT embeddings in extracting that similarity. Is it possible that something about SentenceBERT makes the results so good for that method? I would expect the authors to compare SentenceBERT with more modern sentence embedding models, such as SimCSE and DiffCSE. It is particularly important to look at such an ablation study, in order to understand the role of the nature of the embeddings that you use. ", "questions_to_authors": "line 340: you claim you do not see any specific patterns, but I think there are patterns, at least in the right panel in Figure 4. The left panel is really interesting: it seems that the similarity approach that you study boosts classification performance a lot, and then model size stops improving the performance, relatively speaking. Could you discuss that in a bit more depth? I guess you could say that there is no pattern for the other methods too, i.e. that the other methods (uncertainty, diversity and random) seem to not improve with model size. Interesting, what is the difference between classification and in-context learning here? Maybe the larger the models get the association between the exemplar and its label is better, and it is thus well utilized to make inference about the test sample.", "paper_topic_and_main_contributions": "The paper studies the effect of using different methods to select exemplars for in-context learning as a function of model family, size, type of task (classification, multiple choice), metric (F1, accuracy), etc. It is a thorough study and it relates to a few works from the literature. The main methods considered are based on active learning (uncertainty-, diversity-, similarity-, and random-based). The general finding is that 1) similarity is very strong; 2) uncertainty is surprisingly not very strong given the norm (albeit for larger model sizes the most uncertain exemplars are quite useful); 3) the results for the order of the methods are metric specific; 4) ground truth labels are actually needed.\n\nThe study is thorough, which is a contribution on its own. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "032: inin <- in"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies active learning methods for selecting demonstrations to prompt large language models. Four simple approaches are analyzed: uncertainty, diversity, similarity, and random sampling. Results on GPT and OPT models suggest that similarity to the test example is best.

Pros:
- The problem that the paper is solving is important and relevant to the community.
- Thorough and extensive analysis of the topic
- The findings of the evaluations in the paper are relevant to the practitioners.

Cons:
- Novelty of the findings in the paper. Some of the findings are confirming already known results.
- Some of the findings are contradictory across different metrics.
Decision: Accept-Findings