IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models
Authors: Shaokun Zhang
Xiaobo Xia
Zhaoqing Wang
Ling-Hao Chen
Jiale Liu
Qingyun Wu
Tongliang Liu
Venue: ICLR 2024 poster (2024)
Keywords: in-context learning
large language models
active learning
Primary area: general machine learning (i.e., none of the above)

Abstract: In-context learning is a promising paradigm that utilizes in-context examples as prompts for the predictions of large language models. These prompts are crucial for achieving strong performance. However, since the prompts need to be sampled from a large volume of annotated examples, finding the right prompt may result in high annotation costs. To address this challenge, this paper introduces an influence-driven selective annotation method that aims to minimize annotation costs while improving the quality of in-context examples. The essence of our method is to select a pivotal subset from a large-scale unlabeled data pool to annotate for the subsequent sampling of prompts. Specifically, a directed graph is first constructed to represent unlabeled data. Afterward, the influence of candidate unlabeled subsets is quantified with a diffusion process. A simple yet effective greedy algorithm for unlabeled data selection is lastly introduced. It iteratively selects the data if it provides a maximum marginal gain with respect to quantified influence. Compared with previous efforts on selective annotations, our influence-driven method works in an end-to-end manner, avoids an intractable explicit balance between data diversity and representativeness, and enjoys theoretical support. Experiments confirm the superiority of the proposed method on various benchmarks, achieving better performance under lower time consumption during subset selection.

Published as a conference paper at ICLR 2024
IDEAL: INFLUENCE-DRIVER SELECTIVE ANNOTA- TIONS EMPOWER IN-CONTEXT LEARNERS IN LARGE LANGUAGE MODELS
Shaokun Zhang**
Xiaobo Xia
Zhaoqing Wang"
Ling-Hao Chen
Jiale Liu
Qingyun Wu-1
Tongliang Liu
Pennsylvania State University
"The University of Sydney
Tsinghua University
4 Xidiaa University
shaokun. shaokun.zhaneeeeen wwwng@psu.geeecccooommm edu
x aoboxi un @gma i com
ABSTRACT
In-context learning is a promising paradigm that utilizes in-context examples as prompts for the predictions of large language models. These prompts are crucial for achieving strong performance. However, since the prompts need to be sam- pled from large volume of annotated examples, finding the right prompt may result in high annotation costs. To address this challenge, this paper introduces an influencedrriven selective annotation method that aims to minimize annota- tion costs while improving the quality of in-context examples. The essence of our method select a pivotal subset from large-scale unlabeled data pool anno- tate for the subsequent sampling of prompts. Specifically, a directed graph is first constructed to represent unlabeled data. Afterward, the influence of candidate unlabeled subsets is quantified with diffusion process. A simple yet effective greedy algorithm for unlabeled data selection is lastly introduced. It iteratively selects the data if it provides maximum marginal gain with respect to quantified influence. Compared with previous efforts on selective annotations, our influence- driven method works in end-to-end manner, avoids intractable explicit bal- ance between data diversity and representativeness, and enjoys theoretical support. Experiments confirm the superiority of the proposed method on various bench- marks, achieving better performance under lower time consumption during subset selection. The project page is available at https://skzhang/ giitubbio//DEAA..
INTRODUCTION
In-context learning (ICL) entails presenting a small set of examples with demonstrations as prompts (called in-context examples) to large language models (LLMs), before making predictions test inputs (Wei et al., 2022a; Min et al., 2022; Akyürek et al., 2023). This emerging few-shot learn- ing paradigm is appealing alternative to supervised fine-tuning it can avoid heavy parameter updates of language models while improving accuracy (Liu et al., 2021; Yoo et al., 2022).
Recent studies indicate that obtaining prompts from cial to achieving strong performance (Rubin et al., 2022). Notably, these studies have illuminated the substantial performance improvements when retrieving analogous examples (under specific em- bedding criteria) as in-context examples tailored for each individual test input. Since different test scenarios need distinet in-context examples, and each of them is equipped with its pertinent an- notations, the necessity of large volume of annotated examples is emphasized (Su et al., 2023). However, obtaining large-scale annotated examples for ICL requires substantial manpower and fi- nancial resources (Baldridge & Osborne, 2004; Engelson & Dagan, 1996; Snow et al., 2008). This because humans not only need to annotate the true label for each example but also need to provide the example demonstration in the annotation process (Wei et al., 2022b).
collection of annotated examples is cru-
"Equal contributions. Corresponning authors.

---

Published as a conference paper at ICLR 2024
Accuracy 43.7%
Accuracy 48.8%
Normalized diffusion steps 1
Normal diffusion steps
(a) Low-influence subset unlabeled data.
(b) High-influence subset unlabeled data.
Figure I: Visualization of the information diffusion process (Goldenberg et al., 2001) of two subsets with equal sizes. Experiments are conducted using the SST-5 training set (Socher al., 2013). To avoid the denseness, we randomly sample 100 examples in total. In this visualization, black nodes present the initial subset without information diffusion. White nodes correspond the examples that are not influenced by diffusion. For other nodes, darker nodes represent earlier influenced examples. We can observe that the subset with high influence (b) can achieve better performance by influencing a larger group of examples in the unlabeled data pool compared to the subset with low influence (a).
To reduce the annotation cost, the previous effort Vote-k (Su et al., 2023) made attempts by propos- ing to select a diverse and representative subset from large-scale unlabeled data pool annotate. Particularly, Vote-k initially selects a small portion of data for diversity and annotates them man- ually. Then, these annotated data act as prompts for predictions on all other unlabeled data, and choose the remaining ones that need to be annotated, based on diverse confidence scores. However, despite its strong performance in empirical evaluations, Vote-k is still unsatisfactory in practice. We detail the issues from three aspects. (1) The data selection procedure of Vote-k: is not end-to-end. This results in inconvenience, increased processing complexity, and added inference costs due the predictions on unlabeled data. (2) Diversity and representativennes need to be balanced carefully (Su al., 2023). Highlighting diversity in data selection is crucial for comprehensive coverage, but may sacrifice representativeness by overlooking exemplary data. Besides, the excessive emphasis on di- versity of Vote-k causes the selection of outliers (see evidence in Appendix C.2). (3) Vote-k lacks theoretical guarantees, making it challenging to assess the algorithm's reliability in realistic tasks and constraining its practical utility.
In this paper, to minimize annotation costs for ICL and address the issues of existing work, an inno- vative data selection method introduced, where we utilize jnfluence-driven selective annotations to empower in-context Jearners (IDEAL). In essence, IDEAL aims identify subset of data that acts as a proxy and closely approximates the vast unlabeled dataset. Once annotated, these selected data can be considered a viable substitute for the large annotated examples in subsequent ICL tasks. In further detail, our method works in an unsupervised and end-to-end manner. We first construct a directed graph, where its vertices represent unlabeled data and its edges bridge different data based on their similarities. Inspired by influence maximization that aims to select a vertex set at key posi- tions in social graphs (Li et al., 2018), we then propose quantify the influence of each candidate unlabeled subset in our constructed graph, through classic independent cascade diffusion model il- lustrated in Figure 2. To find the subset with high influence, a simple greedy algorithm for unlabeled data selection is introduced. The algorithm does not need a delicate trade-off between diversity and representativeness. Instead, it iteratively selects vertex if provides a maximum marginal gain to the influence metric, until the selection is completed based on the annotation budget.
cessadde
Theoretically, under the influence-driven selective paradigm, we provide the lower bound for the subset influence selected by our method, demonstrating it is at least as large as a certain propor- tion of the influence of the optimal solution. Empirically, we conduct comprehensive experiments over 9 datasets across diverse tasks (covering classification, commonsense reasoning, dialogue, and text/code generation). Various LLMs and prompt retrieval technologies are included in evaluations. Experimental results demonstrate that our IDEAL achieve better performance than Vote-k in 17 out of 18 cases in the experiments, with only 13% time consumption during subset selection. This creates strong baseline of selective annotations for follow-up research. Source codes have been attached for the reproducibility of results.
2

---

Published as a conference paper at ICLR 2024
2 METHODOLOGY
In this section, to reduce the annotation cost of ICL, framework of influence-driven selective annotations is formulated. We discuss how examples should be selected to annotate, leading to better in-context learners for LLMs.
2.1 PROBLEM SETUP
We begin by defining notations and setting up the research problem. Specifically, L.MMs perform in-context learning tasks based on task-specific prompt Z = [z....., ze], where each z; represents one example (x., 34) consisting of the instance x; and label y/, with e examples in total. LLMs generate the prediction for one test input Xtest conditioned on the prompt Z followed by Xtest i.e. Veest = arg maxyec P(yl Z Xtest), where C. denotes the label space. As each prompt needs dis- tinct annotations, the importance of having a substantial number of annotated examples stressed, resulting in huge annotation costs. This motivates us to investigate selective annotations.
Given a pool of unlabeled instances D. = fx: ) where is the number of unlabeled instances, the aim of selective annotations is to select subset S, C DD to make manual annotations, such that performing ICL. using prompts retrieved from the selected subset can yield good performance on an unseen test set Deest- The size of S, is controlled by the annotation budget m, i.e., Sul m.
2.2 INFLUENCE-DRIVED SELECTIVE ANNOTATIONS
Overview. For selective annotations in ICL, we need to identify a subset that approximates vast unlabeled data. Therefore, quantifying the coverage of each candidate subset is critical. To achieve this, we construct directed graph using the embeddings of unlabeled data and portray their rela- tionships using the edges in the graph. We then quantify the influence of each candidate subset in the constructed graph. An information diffusion model is used for this purpose. Through the infor- mation diffusion model to quantifying the influence of each candidate subset, we avoid the delicate trade-off between diversity and representativeness. After the quantification, we can search the subset with maximum influence, which most closely approximates the unlabeled data. Below we detail the above procedure step by step.
Constructing the directed graph. We first compute vector embedding for each unlabeled instance using Sentence- BERT (Reimers & Gurevych, 2019)'. The obtained embeddings are employed to build directed graph g (V, E, P), where the vertices V {v: represent the embeddings of the unlabeled instances, E denotes the set of edges in the graph, and P denotes the set of weights assigned edges. In more detail, for each vertex v E V, connect it to its i nearest successors in terms of the cosine similarity between the embeddings and then get E. For the edge (v, u) E E that connects v and its successor u, we assign the weight p(v, u) cos(v v, /) EZZA((yA)) cos(v, z) with p E P. where ((v, k) represents the set including k nearest successors of v. and cos(; -) is a function that calculates the cosine similarity of two embeddings. The constructed graph depicts the relationships between unlabeled examples in terms of the embedding similarity.
Quantifying subset influence. Here we propose to quantify each candidate subset within the structed graph, which detailed in Algorithm Specifically, given the constructed graph 9 and a candidate subset S, the quantification algorithm simulates the progression of information diffusion originating from S. The number of influenced vertices can be considered as a measure of the in- fluence of the candidate subset. In other words, the subset that influences more vertices within the graph can provide a better approximation of the vast unlabeled data. The diffusion process unfolds discretely, progressing through multiple steps. At the beginning, the subset S is activated. Then at each step, each vertex activates its successors that remained inactive in the last step with proba- bility defined by p(v, u). The activation can be conceptualized as a coin toss where the outcome determined by the head probability p(v, u). . If the result is the head, the vertex becomes activated; otherwise, remains inactive. Starting from S, the diffusion terminates when no further vertex can be activated in the graph. Lastly, we quantify the influence of the set with the number of activated vertices, where a larger number corresponds to greater influence. In order to get a stable result, we
htts://uuginggaccc.mmmm http:/innnacccccc/eetnnneeeeeeee colkentencc-transformressee wwwprr/aalllmmtee-----eeee In graph theory (Harary, 2018), a vertex is the successor of a vertex if it is at the end of an outgoing directed edge (v
3

---

Published as a conference paper at ICLR 2024
10
90
Successful
Falled
examples
champles
net tortion
artiven ion
Figure 2: The procedure aims to quantify the influence of each subset of in-context examples. In this procedure, we start with a subset of examples (the red points in (a)). Gradually, the successors of this subset are activated based on the weight and random number sampled from 0 to 1. From (a) to (d). The influence of the subset is determined by the number of points that have been activated.
Algorithm 1: Subset influence quantification.
Input Directed graph G (V, E, P), subset S.
Output: Number of influenced vertices by S in g. Sactive S, Snew <- 0), L = 0;
while Suctive # do for each node in Sactive do for each successor of'v in & do if u not in S then Generate random number E [0, 1]; if < p(v. u) then L SUu; Snew f Snew U u;
Sective Snew: + T. SSew Snew <-
return L.
Algorithm 2: Searching the subset with maximum influence.
Input: The directed graph G (V, E, the annotation budget Result: The set S,, that includes m examples to annotate. Initialize Sy 0, t O:
while < m do
Vi < arg vax,e,
U
S - S, Uv:
++
Obtain S. with S,n using the correspondence between embeddings and instances; return S,,
repeat this process ten times and take the average influence. To help understand the procedure of Algorithm 1, provide illustration as shown in Figure 2. For convenience, express Algo- rithm I as an influence function fS(S) for the graph G that takes example set as inputs, and returns the number of activated vertices L.
Searching the subset with maximum influence. We exploit simple yet effective greedy algo- rithm (Kempe et al., 2003) search the subset with maximum influence, which illustrated in Algorithm 2. Specifically, the algorithm is initialized with an empty set, and iteratively involves an instance if it can provide the maximum marginal gain to the influence function. The search algorithm terminates when the selected subset meets the annotation budget. Finally, we achieve the set S, that includes m examples annotate, using correspondence between embeddings and instances. worth mentioning that this searching process aims to maximize the influence of the whole selected subset rather than considering each example separately. This is because combining all the individual high-ing high-impact examples together does not necessarily achieve the highest-immn subset.

---

Published as a conference paper at ICLR 2024
2.3 PROMPT RETRIEVAL
After the above influence-driven selective annotations, the subset S, is achieved. By making man- ual annotations on S., a set of annotated examples is obtained. We can then retrieve examples from the annotated
in-context examples for each test input. Following previous studies (Liu et al., 2021: Su et al., 2023), we will calculate embeddings for all annotated examples using Sentence- BERT (Reimers & Gurevych, 2019) and identify the most similar instances each test input based on the cosine similarity. Notice that, the proposed method is agnostic to prompt retrieval methods. As demonstrated in $4.3.3, our method can be combined with any other prompt retrieval technolo- gies. Better prompt retrieval technologies can further boost final performance.
3 THEORETICAL ANALYSIS
In this section, we perform theoretical analysis on the influence of the subset searched by our algorithm and provide the corresponding lower bound. For any constructed graph G. we ex- ploit (S) to denote the influence improvement of the subset S after adding v into S, i.e., fg(S (t 2 1) to
(S) = ( - fS) For convenience, we use th = fg())) denote the incremental value of the influence function fg after adding v, into St-1. Also, em- ploy S, to represent the subset with the optimal influence value in the graph G with annotation budget m. Afterward, the optimal solution we expect to search in Algorithm 2 can be regarded
arg max fg(S), s.t. S = m.
(I)
m
SCV
In the following, we present the submodular condition to facilitate theoretical analysis of our method. Condition 1 (submodular condition). In the problem of selective annotations, given any graph g constructed by our procedure, the influence funnttion fg is submodular function which satisfies, for Vv E V, VS, C Sp CV,
(G(S.vv) - 2 So(S, Uv) - fs(S)).
(2)
Remark 1. Intuitively speaking, given any graph G. we say the influence function fg satisfies the submodular condition if adding one data point to a smaller subset provides more influence than adding the same data point to a larger subset. In other words, it reflects the principle of diminishing returns: the marginal gain of including data point in set decreases as the size of the set increases. This condition can hold within the influence function (Li et al., 2019). Considering extreme case, when subset S = V, the influence improvement of adding any data point to S will be zero Proposition 1. In Algorithm 2, if the influence function fg satisfies Condition I, then for fg(())).. 1L [0, m 1), (gS S (S.) + -mbb411
Remark 2. Proposition proposes an upper bound for (5())) in the form of the influence fc( and its improvement at next step 1. + when Algorithm 2 is applied to selective annotations.
Theorem 1. In Algorithm& 2, if influence function Ig satisfies Condition 1, when the algorithm terminates at the step m - 1, (s(Sm.) has a lower bound:
fg((m)) 2 21 (1-
(4)
Remark 3. Theorem 1 provides an approximation guarantee for the influence of the selected subset returned by our method. The influence of the selected subset is at least as large as a certain proportion of the influence of the optimal solution, i.c., 1 1-c.,1111111 (1-1/m)". With the annotation budget m increases, this fraction gets closer to 1 1/e.
For the proofs of Proposition and Theorem 1. readers can refer Appendix
4 EXPERIMENTS
In this section, we evaluate our method (IDEAL) on multiple datasets that have different categorics of tasks. Experimental setups are first introduced (84.1). We then demonstrate that the proposed method can find better selective annotation subset in a more efficient way compared with base- lines ($4.2). Moreover, we perform in-depth investigations provide better understanding of the superiority of the proposed method (64.3). Finally, a case study is also provided to further evaluate the selected subset from our method in automatic annotation scenario ($4.4).
5

---

Published as a conference paper at ICLR 2024
4.1 EXPERIMENTAL SETUPS
Datasets and tasks. Following previous work (Su et al., 2023), we employ 9 datasets for the eval- uations, which can be categorized into 4 different tasks, including classification, multi-choice, dia- logue, and generation. The details of the datasets are provided in Appendix D.I. each dataset, the original "train/dev/tsst split from the Transformer library (Wolf et al., 2019) is utilized. We use test data for evaluation if they are available publicly (SST-5 (Socher et al., 2013), DBpedia(Lchmam et al.. 2015), MWoZ (Budzianowski et al.. 2018), and Xsum (Narayan et al., 2018)). Otherwise, we follow the same setting in (Su et al., 2023) and use the development set. We use accuracy as metric for all classifications and multiple choices tasks, joint accuracy (Budzianowski et al., 2018) for MWoZ, test suite accuracy (Zhong et al., 2020) for GeoQuery (Zelle & Mooney, 1996), and ROUGE-L (Lin, 2004) for Xsum.
Models. If not otherwise specified, we run all experiments on the GPT-J 6B model (Wang & Ko- matsuzaki, 2021) except the GeoQuery and MWoZ datasets where we use Text-devinci-002 (Chen et al., 2021). We also provide experiments on other models including GPT-Neo 2.7B (Black et al., 2021) and more advanced models GPT-3.5-Turbo (Openai, 2023) in $4.3.4. Our implementation is detailed in Appendix D.2.
Baselines. In the main experiments, we perform a comprehensive evaluation of our method that is compared with previous state-of-the-ar selective annotation baselines, i.e., Vote-k (Su et al., 2023) and random selection (abbreviated as "Random" below). Note that, in $4.3.2, also compare method with alternative methods that can select a coreset from large-scale unlabeled data on typical datasets. For the baseline Vote-k; we conduct experiments by running its official code?.
4.2
MAIN RESULTS
Classification
Multi-Choice Dialogue Generation
Method
MRPC SST-5 MNLI DBpedia RTE
HellaSwag
MWoZ
GeoQ Xsum
100
Random
64.3
49.6
38.2
89.8
55.3
66.7
39.9
55.3
15.3
100
Vote-k
64.6
46.6
38.9
89.2
57.6
67.9
48.3
58.8
17.2
100
IDEAL
66.4
51.4
41.0
90.6
58.9
68.6
52.2
58.2
19.9
18
Random
57.4
42.9
37.8
85.2
57.9
66.0
37.0
47.5
13.6
18 Vote-k
61.1
41.7
39.1
89.9
58.2
66.5
37.7
50.9
15.2
18 IDEAL
63.0
43.2
40.0
90.1
59.4
67.1
38.5
52.0
19.6
Table 1: The performance of our method and baselines 9 different datasets with annotation budget of 100 and 18. We use similar- prompt retrieval for all methods and report the average results with different runs for each method. We observe that method works better than Random and Vote-k in almost all cases (17/18) under two annotation budgets. The best result in each case is bolded. We also provide the maximum and minimum values of the results in Appendix C.3.
Measurement on performance. We first perform the evaluations for Ran- dom, Vote-k, and our method. The annotation budget is set to 18 and 100 respectively following the same set- ting as Vote-k. Note that we include 18 as the annotation budget consider- ing all annotated examples can be fit to the prompt of the large language models within context limits. There- fore, the prompt retrieve stage can be ignored and the evaluation results can naturally represent the quality of the selected examples. We provide ex- perimental results in Table I. As can be seen, our method achieves better performance than basclines
Verhode
Log Time
Figure 3: Comparison of our method and Vote-k with re- spect to time consumption during subset selection under the same hardware condition. Here the annotation budget 18. The y-axis represents the time consumption with a log scale. We can observe that our method largely reduces the time cost compared with Vote-k.
hhans://gituab.co////UUIIII wtwp://niii.iiicssselttiiii..ie

---

Published as a conference paper at ICLR 2024
in most of the evaluation scenarios (17 out of 18). Interestingly, we find that random selection outper- forms Vote-k in out of 18 cases. We conjecture that, under some ideal circumstances, the selected subset by random selection approximate the distribution of full data. If test data follows the same distribution, good performance can be achieved. Note that we also illustrate selected examples and label distributions in selective annotations in Appendix C.1 and Appendix C.4 respectively.
Measurement on time cost. Previous work Vote-k (Su et al., 2023) encompasses generating predic- tion for most unlabeled data with set of selected examples prompts and performs data selection according to the confidence scores of the prediction. However, this process results in large unnec- essary costs at inference time. Meanwhile, LLMs are often used as a service and an extra charge will appear with the usage of the token in both the input and output. In Figure 3, we compare the time cost of subset selection in our method against Vote-k on all tasks with the same hardware. The annotation budget is set to 18. We can observe that our method saves a tremendous amount of cost compared to Vote-k. Specifically, under the same hardware conditions, IDEAL achieves a 7.8x x lead on average over Vote-k. The speed improvement benefits from the fact that the proposed method does not need to perform example selection by generating predictions on a large number of unlabeled examples and is completely unsupervised.
4.3
MORE ANALYSIS
4.3.1
LARGER INFLUENCE BRINGS BETTER PERFORMANCE
We conduct experiments investigate the correlation between subset influence and its corresponding in-context learning perfor- mance. Specifically, we randomly select a collection of example subsets from a large unlabeled data pool. We then evaluate each subset as a prompt and record its performance and influence in the constructed graph, result- ing in a set of influence-pp influence-performanc pairs. Our goal is analyze the correlation between these two metrics. To achieve this, we per- SST-5 and MNLI. We
6.3
626
624
(a) SST-5
(b) MNLI
Figure 4: Influence vs. Performance. The illustra- tion of the positive correlation between the influence achieved by Algorithm and final performance.
form experiments sample 30 subsets and order them accord- ing their influences, where each subset in- cludes 5 examples. We divide this sorted subset sequence equally into three influence levels, with each level containing 10 subsets. We visualize the performance of subsets each influence level in Figure 4. Our analysis reveals that subsets with larger influence levels achieve better average, me- dian, and worst-case performance. This finding further demonstrates that quantifying the influence of each potential subset is an effective metric in the selective annotation problem.
4.3.2 COMPARISONS WITH ALTERNATIVE METHODS
We also compare method with other alternative methods that can select the coreset from large-scale unlabeled data. We perform the evaluations on MRPC, MNLI and HellaSwag. We include the following alternative methods (1) K- Means (Looyd, 1982), which groups all examples into m clusters, and selects the centroid example from each clus- ter. (2) Maximizing facility location (MFL) (Lin & Bilmes, 2009), which aims at optimizing the representative- ness the selected subset. (3) Fast Vote-k (Su et al., 2023), which is an efficient alternative to Vote-k which directly picks examples with the largest Vote-k scores.
Method
K-Means MFL Fast Vote-k Vote-k
IDEAL
MRPC
57.4
58.2
59.3
61.1
63.0
MNLI
35.8
38.8
39.5
39.1
40.0
HellaSwag
65.4
65.2
65.9
66.5
67.1
Table 2: Comparisons of alternative methods that can select a coreset from large-scale unlabeled data. The an- notation budget is 18. Experimental results are reported by averaging over three random trials. The performance of the baseline Vote-k is also included here. The best per- formance in each case is bolded.

---

Published as a conference paper at ICLR 2024
Method
Datasets
Test Domain
Method
Models
Selection Retrieval MRPC
MNLI HellaSwag
IMDb
BoolQ Cst.
Vote-k Similar 64.6
38.9
67.9
Vote-k
GPT-Neo
71.1
56,4
IDEAL Similar 66.4
41.0
68.6
IDEAL
GPT-Neo
72.2
58.0
Vote-k Random 60.7
37.8
64.6
Vote-k
GPT-J
76.4
56.1
IDEAL Random 62.5
39.0
66.8
IDEAL
GPT-J
76.8
56.4
Comparison of random and similar prompt retrieval with Vote-k and IDEAL on MRPC, MNLI, and HellaSwag. The subset se- lection method with a similar prompt retrieve achieves better performance compared with its version with random prompt retrieve method. The best performance each case is bolded.
Table 3:
The evaluations on out-of-distribution tasks. We show the performance of different methods on IMDb and BoolQ Contrast Set (tar- get domains). In the evaluations, the prompts consist of selected SST-2 and BoolQ training ex- amples, respectively (source domains). The best performance in each case is bolded.
Table 4:
We show the results in Table We can observe IDEAL consistently outperforms the baselines in all datasets, demonstrating its superiority. Note that, the graph-based methods (Vote-k, Fast Vote- k, and our IDEAL) outperform the methods non-graph-based methods (K-Means and MFL.) in all cases. This phenomenon suggests that graph-based methods are suitable for capturing similarity relationships between examples in the selective annotation problem, which can lead to better results.
4.3.3 EVALUATION WITH DIFFERENT RETRIEVAL METHODS
In previous experiments, we used a similarity- based prompt retrieval method by default. In this section, we conduct experiments to quantify the effect of different prompt retrieval methods under the annotation 100. We present the results Table We observe that both Vote-k and IDEAL suffer from significant performance drop when the prompt retrieval method is changed from similarity- based to random selection. Notably, IDEAL also achieves better performance than Vote-k when combined with random retrieval in all datasets. It suggests that IDEAL can cultivate more stable training subset (Chang & Jia, 2023) for in-context learning tasks. Note that we also show that our IDEAL is more stable and robust against the order of in-context examples in Appendix C.5.
4.3.4 EVALUATION ON OTHER LANGUAGE MODELS
Here we evaluate IDEAL on other language models, including GPT-Neo 2.7B (Black et al., 2021), and the advanced chat model GPT-3.5-Turbo where models for each dataset. While GPT-3.5- Turbbo has mainly been optimized for chat, it also performs well traditional completion tasks (Kheiri & Karimi, 2023). To conduct experiments, we select three classification tasks (MRPC, MNLI, and RTE), considering they are easier for prompting GPT- 3.5-Turbo return responses without pleasantries or explanatory content.
the same instruction as other language
The evaluation results are presented in Figure 5. Our evaluations reveal that IDEAL consistently outperforms the baselines across all models tested. This demonstrates the versatility of our method in the context of learning tasks using models of varying sizes. Notably, we observe that the largest performs worse than GPT-Neo and GPT-J. This situation arises because
model, i.e., GPT-3.5-Tuur GPT-3.5-Turhe is primarily optimized for chat tasks and faces challenges in following human in- structions for classification. This scenario also has been identified in Ye al. (2023).
4.3.5 EVALUATION ON OUT-OF IITT-O--DISTIBBUTIOO TASKS
We further evaluate our method on out-of-distributin tasks (Zhou et al., 2022; Wang et al., 2022b; Zhang et al., 2023b; Huang et al., 2023c;d), where there a distribution shift between the selective annotation data and test data. Following (Chang & Jia, 2023), we compare IDEAL and Vote-k using SST-2 (Socher et al., 2013), BoolQ (Clark et al., 2019) as source tasks, and IMDb (Maas et al., 2011), BoolQ Contrast Set (Gardner et al., 2020) target tasks, respectively. In all evaluations, set the annotation budget as 18 and use the similarity- baased retrieve to perform the evaluations on the test set in target domains. We use GPT-J 6B and GPT-Neo 2.7B here and show the results in
8

---

Published as a conference paper at ICLR 2024
21.55
187x
Purfor manaoo
32
68%
07102/00 GP/
0111
121
02-1
(a) MRPC
(b) MNLI
(c) RTE
Figure 5: Comparisons with various models when the annotation budget is 18. IDEAL consistently achieves the best performance compared with baselines in models with different datasets.
Table 4. We can observe that IDEAL still outperforms baselines on all datasets with two models, implying that IDEAL could select the subset which could depict the invariant properties of this kind of tasks and generalize to out-of- -isriisiinttion scenarios.
4.4 CASE STUDY: AUTOMATIC ANNOTATION
In previous experiments, used a small of manually annotated examples as candidate prompts to make predictions. In contrast, here we are interested in a case study that uti- lizes the subset selected by IDEAL anno- tate all available unlabeled data automatically, leading to larger set of candidate prompts. Specifically, we first choose an initial subset from the pool of unlabeled data using IDEAL and manually label this selected subset. Af- terward, we simulate the information diffu- sion process from the initial subset to all other data, where we employ those activated data prompts predict upcoming activated data at each step and label them accordingly with prediction results. This process ultimately makes fully labeled training dataset. Finally, all examples (including manual labeling and auto- matic labeling are utilized as potential prompts in conjunction with the prompt retrieve technique for final testing. We name this paradigm as IDELL and compare it with Vote-k and origin IDEAL on all classification datasets. We choose 300 training data for each dataset to perform exper- iments. The manual annotation budget is set to 150, i.e., half of the labels of the candidate prompts in Auto-IDEAL are annotated automatically, Experimental results are provided Table 5. As can be observed, Auto-IDEAL even achieves better performance than IDEAL in 4 of 5 cases. Notably, although the performance worse on MNLI, it is still competitive (better than Vote-k). It suggests that expanding the candidate prompts through automatic annotation following the diffusion process can further boost the performance of IDEAL. It benefits from the fact that information only diffuses between similar examples. Therefore, unlabeled examples will be automatically annotated using the most similar annotated examples as prompts leading to promising annotation success rate.
Method
MRPC
SST-5
MNLI DBpedia RTE
Vote-k
63.8
48.6
39.5
90.2
55.7
IDEAL
65.2
49.4
40.3
90.8
57.4
Auto-IDEAL
65.8
50.4
39.8
91.8
58.3
Table 5: Comparison between Vote-k, IDEAL, and Mito-IDEAL. Auto-IDEAL is expanded version of IDEAL for automatic annotation. We evaluate these algorithms all classification tasks and erage their performance over three random trials. The best performance in each case is bolded. The results indicate that Auto-IDEAL can enhance the performance of IDEAL and achieve the best perfor- mance in 4 out of 5 cases.
5 CONCLUSION
A series of recent works have confirmed the powerful ability of in-context learning for large lan- guage models. We investigate the ability from the perspective of selective annotations and propose an influence-driven method that selects subset of data that acts as a proxy and closely approximates full data. Theoretical analysis is provided to establish an upper limit for the global optimal solution, and demonstrate that our greedy search algorithm selects a subset with influence at least as sub- stantial as a specific proportion of the optimal solution's 's influence. Empirical evaluations illustrate the superiority of our method across a range of benchmarks, delivering superior performance while largely reducing the time required for subset selection. We hope this work can help researchers and practitioners understand the promise and potential of selective annotations in in-context learning, and facilitate them in the efficient conceptualization of novel language- based challenges.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
Tongliang Liu is partially supported by the following Australian Research Council projects: 1P422000052,,1,22200049,,,, and IC190100031.
FT220100318, DP220102121, LP220100527, LP220200949,
REFERENCES
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. ICLR, 2023.
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers statisticians: Prov- able in-context learning with in-context algorithm selection. arXiv preprint arXiv:2306.04637,
2023.
Jason Baldridge and Miles Osbome. Active learning and the total cost of annotation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 9-16, 2004.
Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. Rethinking the role of scale for in-context learning: An interpretabillty-aseee case study at 66 billion scale. ACL, 2022.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7:8, 2009.
Sid Black, Leo Gao, Phil Wang, Connor Leaby, and Stella Biderman. Gpt-neo: Large scale autore- gressive language modeling with mesh-tensorflow. 2021.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašié. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. In EMNLP. 2018.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learn- ing in transformers. In NeurlPS, pp. 18878-18891, 2022.
Ting- Yun Chang and Robin Jia. Data curation alone can stabilize in-context learning. ACI,, pp- 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-leaming via language model in-context tuning. In ACL, 2022.
Hyunsoo Cho, Hyuhng Joon Kim, Junyeob Kim, Sang-Woo Lee, Sang-goo Lee, Kang Min Yoo, and Taeuk Kim. Prompt-auugmentd linear probing: Scaling beyond the limit of few-shot in-context learners. In AAAI, pp. 12709-12718, 2023.
Christopher Clark, Kenton Lee, Ming-Wer Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL- HLT., pp. 2924-2936, 2019.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. Scaling up dataset distillation to imagenet- lk with constant memory. ICML, pp. 6565-6590, 2023.
Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:230..12246, 2023.
William Dolan, Chris Quirk, Chris Brockett, and Bill Dolan. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In ACI., 2004.
10

---

Published as a conference paper at ICLR 2024
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. survey for in-context learning. arXiv preprint (.XXi:233110022::0 2023.
Jiawei Du, Yidi Jiang, Vincent YF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accu-
mulated trajectory eπrоr improve dataset distillation. CVPR, pp. 3749-3758, 2023.
Sean Engelson and Ido Dagan. Minimizing manual annotation cost in supervised training from corpora. arXiv preprint cmp-lg/9606030, 1996.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In NeurlPS, pp. 2881-2891, 2020.
Tianyu Gao, Adam Fisch, and Dangi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:20/2. 15723, 2020.
Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. Evaluating models' local decision boundaries via contrast sets. Findings of EMNLP, 2020.
Shivam Garg. Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. In NeurIPS, pp. 10583-30599 2022.
Jacob Goldenberg, Barak Libai, and Eitan Muller. Talk of the network: complex systems look at the underlying process of word-of-mouthh Marketing letters, 12:211-28 2001.
Frank Harary. Graph Theory (on Demand Printing Of 02787). CRC Press, 2018.
Muyang He, Shuo Yang. Tiejun Huang, and Zhao. Large-scale dataset pruning with dynamic uncertainty. arXiv preprint arXiv:2306.05175. 2023.
Lingxiao Huang, Shaofeng H-C Jiang, Jian Li, and Xuan Wu. Epsilon-coresets for clustering (with outliers) in doubling metrics. In FOCS, pp. 814-825, 2018.
Lingxiao Huang, Shaofeng H-C Jiang, Jianing Lou, and Xuan Wu. Near-optimal coresets for robust clustering. ICLR, 2023a.
Zhuo Huang, Chang Liu, Yinpeng Dong, Hang Su, Shibao Zheng, and Tongliang Liu. Machine vision therapy: Multimodal large language models can enhance visual robustness via denoising in-context learning. arXiv preprint 2rXiv:2312.02546, 2023b.
Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing out-of-distribution examples via augmenting content and style. In ICLR, 2023c.
Zhuo Huang. Miaoxi Zhu, Xiaobo Xia, Li Shen, Jun Yu, Chen Gong, Bo Han, Bo Du, and Tongliang Liu. Robust generalization against photon-limited corruptions via worst-case sharpness minimiza- tion. In CVPR, pp. 16175-16185, 2023d.
David Kempe, Jon Kleinberg, and Êva Tardos. Maximizing the spread of influence through social network. In SIGKDD, pp. 137-146, 2003.
Kiana Kheiri and Hamid Karimi. Sentimentgpt: Exploiting gpt for advanced sentiment analysis and its departure from current machine learning. arXiv preprint arXiv:2307.1023: 2023.
Hyuhng Joon Kim, Hyunsoo Cho, Junyeob Kim, Tacuk Kim, Kang Min Yoo, and Sang-goo Lee. Self-generated in-context learning: Leveraging auto-regressive language models demonstra- tion generator. arXiv preprint 22X6::226600888 2022.
Young-Jun Lee, Chae- Lim, and Ho-Jin Choi. Does gpt-3 generate empathetic dialogues? novel in-context example selection method and automatic evaluation metric for empathetic dia- logue generation. In COLING, pp. 669-683, 2022.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. Dbpedia-a large- scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167-195, 2015.

---

Published as a conference paper at ICLR 2024
Fangqi Li, Chong Di, and Wenwen Xia. On the submodularity of diffusion models: Equivalent conditions and applications. arXiv preprint arXiv:20002008455,222 2019.
Xiaonan Li and Xipeng Qiu. Finding supporting examples for in-context learning. arXiv preprint (xXiv:2302.13339 2023.
Yingcong Li, Muhammed Emrullah Iidiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In JCML, 2023a.
Yuchen Li, Ju Fan, Yanhao Wang. and Lian-Lee Tan. Influence maximization on social graphs: A survey. IEEE Transactions on Knowledge and Data Engineering, 30(10):1552-1722 2018.
Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang. Lei Zhang, Shuzheng Si, Junhao Liu, Tongliang Liu, Fei Huang, et al. One shot learning instruction data prospector for large language models. arXiv preprint 10Xv:2231..0330, 2023b.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Hui Lin and Jeff A Bilmes. How to select good training-dan data subset for transcription: submodular active selection for sequences. In Interspeech, pp. 2859228888888 1859-2866, 2009.
Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06004, 2021.
Stuart Lloyd. Least squares quantization in pem. IEEE Transactions on Information Theory, 28(2): 129-137, 1982.
Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. In ICML, 2023.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.0878k 2021.
Yuexiao Ma, Taisong Jin, Xiawu Zheng, Yan Wang, Huixia Li, Yongjian Wu, Guannan Jiang, Wei Zhang, and Rongrong Ji. Ompq: Orthogonal mixed precision quantization. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 9029-9037, 2023a.
Yuexiao Ma, Huixia Li, Xiawu Zheng, Xuefeng Xiao, Rui Wang, Shilei Wen, Xin Pan, Fei Chao, and Rongrong Ji. Solving oscillation problem in post-training quantization through a theoreti- cal perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7950-7959, June 2023b.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, 2011.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP, 2022.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In ACI., 2022.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. EMNLP, 2018.
Openai. GPT-3.5-Turbo. https: http:://plaffonttt wttppppttoooopaaia.i....mmm com./doce//odeels//vvvvieee May 2023.
12

---

Published as a conference paper at ICLR 2024
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, 2019.
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert- networks. arXiv preprini arXiv: /908. 10084, 2019.
David Rolnick and Jonathan Weed. Greedy maximization of submodular functions. 2014.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning retrieve prompts for in-context learning. In NAACL, 2022.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In ICLR, 2018.
Bingkang Shi, Xiaodan Zhang, Dehan Kong, Yulei Wu, Zongzhen Liu, Honglei Lyu, and Longtao Huang. General phrase debiaser: Debiasing masked language models at multi multi-token level. arXiv preprint arXiv:2311.1382 2023.
Seongjin Shin, Sang-Wos Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung- Woo Ha, et al. On the effect of pretrain- ing corpora on in-context learning by a large-scale language model. In NAACI, 2022.
Seungjae Shin, Heesun Bae, Donghyeok Shin, Weonyoung Joo, and II-Chul Moon. Loss-curvature matching for dataset selection and condensation. In AISTATS, pp. 8606-8628, 2023.
Rion Snow, Brendan O' D'connor, Dan Jurafsky, and Andrew Ng. Cheap and fast-but it good?
evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 confer- ence on empirical methods in natural language processing, pp. 254-263, 2008.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pp. 1631-1642, 2013.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In NeurlPS, pp. 19523-19536, 2022.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, al. Selective annotation makes language models better few-shot learners. In ICLR, 2023.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. ICLR, 2019.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language jax, May 2021.
Model.
com kinooin fooofllle
Boshi Wang, Xiang Deng, and Huan Sun. Iteratively prompt pre-trained language models for chain of thought. In EMNLP, 2022a.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generalization. JEEE Transactions Knowledge and Data Engineering, 2022b.
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei Efros. Dataset distillation. arXiv preprint 181 10959, 2018
13

---

Published as a conference paper at ICLR 2024
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabshii, Yeganeh Kordi, Amirreza Mirzaei, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran. Atharva Naik, David Stap, et al. tatrrarrattrlnnssttutton::: Generalization via declarative instructions 1600+ nip tasks. In EMNLP, 2022e.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. In ICLR, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, pp. 24824-24837, 2022b.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv: 2017.
Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the-ar natural language processing. arXiv preprint arXiv:1910.33771, 2019.
Zhiyong Wu, Yaoxiang Wang. Jiacheng Ye, and Lingpeng Kong. Self-adaptive In-context Learning. 2022.
Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. In ICLR, 2022.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In ICLR, 2023a.
Xiaobo Xia, Jiale Liu, Shaokun Zhang, Qingyun Wu, and Tongliang Liu. Coreset selection with prioritized multiple objectives. arXiv preprint N..iv:2311.006... 2023b.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In ICLR, 2022.
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reduc- ing training data by examining generalization influence. In ICLR, 2023.
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, al. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303. 10420, 2023.
Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang- Woo Lee, Sang- goo Lee, and Taeuk Kim. Ground-truth labels matter: A deeper look into input-label demonstra- tions. In EMNLP, 2022.
John M Zelle and Raymond J Mooney, Learning to parse database queries using inductive logic programming. In Proceedings of the National Conference on Artificial Intelligence, pp. 1050- 1055, 1996.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In ACL, 2019.
Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context. arXiv preprint arXiv:2306.0m 09927, 2023a.
Shaokun Zhang, Xiawu Zheng, Chenyi Yang, Yuchao Li, Yan Wang. Fei Chao, Mengdi Wang, Shen Li, Jun Yang, and Rongrong Ji. You only compress once: Towards effective and elastic bert compression via exploit-explore stochastic nature gradient. arXiv preprint (0l::2006.02444 2021.
Shaokun Zhang, Yiran Wu, Zhonghua Zheng, Qingyun Wu, and Chi Wang. Hypertime: Hyperparameter optimization for combating temporal distribution shifts. arXiv preprint (.Xiv:2305.18421 2023b.
14

---

Published as a conference paper at ICLR 2024
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In ICLR, 2021a.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pp- 12697-12706. PMLR. 2021b.
Xiawu Zheng, Yuexiao Ma, Teng Xi, Gang Zhang, Errui Ding, Yuchao Li, Jie Chen, Yonghong Tian, and Rongrong Ji. An information theory-inspired strategy for automatic network pruning. arXiv preprint wrXiv:2108.088222 2021.
Xiawu Zheng, Chenyi Yang, Shaokun Zhang, Yan Wang, Baochang Zhang, Yongjian Wu, Yunsheng Wu, Ling Shao, and Rongrong Ji. Ddpnas: Efficient neural architecture search via dynamic distribution pruning. International Journal of Computer Vision, 131(5):1234-1244 2023.
Ruiqi Zhong, Tao Yu, and Dan Klein. Semantic evaluation for text-to-sql with distilled test suites. In EMNLP, 2020.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't make your 1lm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964 2023.
A RELATED WORK
A.1 IN-CONTEXT LEARNING
In-context learning (ICL) has become new paradigm for natural language processing (NLP), where large language models make predictions only based on contexts augmented with a few ex- amples (Dong et al., 2023; Xie et al., 2022; Shin et al., 2022; Zhang et al., 2023a; Bai et al., 2023; Huang et al., 2023b). series works attempts to revise, enhance, and understand ICL, which include but not limited to prompt tuning (Kim al., 2022; Wang et al., 2022a; Mishra al., 2022), analyzing intrinsic mechanism (Bansal et al., 2022; Chan et al., 2022; Li et al., 2023a; Garg al., 2022), evaluations (Srivastava et al., 2023; Wang al., 2022c), applications in multiple do- mains (Chen et al., 2022; Lee et al., 2022; Cho et al., 2023), and etc.
Despite in-context learning has shown impressive performance in various domains, its efficacy is sensitive to the selection of in-context learning examples (Zhao et al., 2021b; Lu et al., 2021). Considering this, multiple methods have been proposed select the optimal in-context learning examples to achieve optimal performance. These methods retrieve the most relevant examples sub- set with/without specific order for each query (Wu et al., 2022; Liu et al., 2021: Gao et al., 2020) Alternatively, some methods aim to find a set of examples once for all queries on the same task (Li & Qiu, 2023; Diao et al., 2023). Specifically, (Wu al., 2022) formally defines the problem of self-adaptive In-context learning, which aims to search for the best In-context learning examples and corresponding order for each input query. While, Li Qiu (2023); Diao et al. (2023) focus on finding one fixed set of examples for each task. However, these methods rely on the assumption that large-scale annotated training examples are always available.
Different from them, this paper studies selective annotations for ICL, which can effectively reduce the annotation cost in ICL. Furthermore, compared with recent work (Su et al., 2023), as discussed in the main paper, this work is superior in many aspects, such as the end-to-end manner, mitigation of the trade-off between diversity and representativeness, theoretical guarantees, and better empirical performance.
15

---

Published as a conference paper at ICLR 2024
A.2 CORESET SELECTION
Coresct selection focuses on selecting small but highly informative subset from a large dataset for follow-up tasks, which can significantly reduce the data storage cost and training consump- tion (Huang et al., 2018; 2023a; Feldman & Zhang, 2020; Sorscher et al., 2022; Xia et al., 2023b; Li al., 2023b; Xia et al., 2022). Most of the works on coreset selection target the scenes of supervised learning and classification (Sener & Savarese, 2018; Toneva et al., 2019; He et al., 2023). Only few works extend coreset selection into unsupervised cases (Sorscher al., 2022; Su et al., 2023). This paper studies unsupervised data selection for annotations in ICL, which reduces the annotation expenses of prompts and helps large language models become better few-shot learners. Also, en- joys theoretical support. Therefore, this work is different from previous efforts and contributes to the research community.
A.3 DATA DISTILLATION
Data distillation (Wang et al., 2018; Zhao et al., 2021a; Shin et al., 2023; Cui et al., 2023; Du et al., 2023: Loo al., 2023) is an alternative approach for dataset compression and curation, which is inspired by knowledge distillation which could be categorized into efficient ML (Ma et al., 2023a;b; Zheng et al., 2021; Zhang et al., 2021; Zheng et al., 2023). Different from coreset selection, this series of works target synthesizing a small but informative dataset as an alternative to the original dataset. However, data distillation criticized for only synthesizing small number of data points due to computational source limitations (Xia et al., 2023a; Yang et al., 2023). The performances of data distillation and data selection are therefore not compared directly. Besides, it is under-explored about how perform data distillation in unsupervised manner natural language processing tasks. Based on this analysis, the data distillation strategy is not involved in empirical evaluations.
B PROOFS
B.1 PRELIMINARY THEORETICAL RESULTS
We first present some preliminary theoretical results and their corresponding proofs for the subse- quential proofs of Proposition and Theorem
B.1.1
LEMMA
Lemma 1.
Given a graph g = (V, if the influence function meets Condition 1, then for
S, CV:
fo(8.) -c(S))) S
Σ
(8)))) Σ
AS - vv),
(5)
vES.-S,
VES,
where Vo vv =
Uv) -v)
Proof.
The proof is inspired by (Rolnick & Weed, 2014). We first let
S. S = = =a,..... a
(6)
and
S = (b.. b S, -
(7)
where r N4 and IN.. According to Eq. (6), for the subsets S, and Sj, we have
S₁ U,U. = Sj U (al, a.).
(8)
Afterward, we obtain
fo(8, fo(S,) U (ai,..,,,))) (ai, fo(S))
(
16

---

Published as a conference paper at ICLR 2024
At high level, Eq. (9) is calculate the influence improvement of Sj after adding data points (ai.... a, into Sy. As the influence improvement adding one sequence of data points is equal to the sum of the influence improvement at each step, we have,
fo(S; U.)) fa(S,)
(10)
f Uai) fo(S,) + Σ
[s (ai,...,a))) -13))-o(()) - far,... ax-1l)
k=2
too (S,) Vag (S) U (ai: (a)..., ak-1))-
Under Condition a, C (a [ ax-1) have
(o(S) - fc(S)) = Vna (S,) + Σ Llnn (a).........
ax
(11)
k=2
Va, (S₂) = Σ
(a(s)))
acs, -
Similarly,
fo(S, US) - fo(S:)
(12)
= (S, + Σ bbb (bi. ba-1 2
b/ USS - bb) = Σ
Vb().).
bES; -;
By subtracting (12) from (10), we have
[g(8.) - (g( s Σ VES.-S,
vv()) Σ
(s,SU - vv).
(13)
VES,
B.1.2
LEMMA 2
Lemma 2. Given graph G = (V, E, P), for any subset S C V and any E V, the influence function fg satisfies
// (8) = fo(S fo(8) 0
(14)
Proof. We consider two cases to finish the proof.
Case (v E VAv & S). In this case, the influence improvement is at least since itself has been included, i.e.,
- (v(S) -(())) Js(S) 1.
(15)
Case 2 (v E VAVV E S). In this case, the influence improvement is 0 since v has already been included in S, i.e.,
( - 0.
(16)
Combining the above two cases, conclude that, for Vv V, the influence function fg satisfies
- 2 0.
(17)
17

---

Published as a conference paper at ICLR 2024
B.2 PROOF OF PROPOSITION
Proof. Given a graph g = (V, E, P), for VS: S3 C V, according to Lemma 2, we have
Σ
vv - v) 2 0.
(18)
VES.-S,
Taking (18) into Lemma 1, we have
fo(S,) Sg(S,) -
Σ VES,
(S,))
(19)
We use St denote the optimal solution as discussed in the main paper. At any step Algo- rithm 2, we substitute S (resp. S,) into S; (resp. Sj) in (19), we can derive
fo(S S fg((s) +
Σ vES, St
(S,)
(20)
According to Condition 1,
(s,) 2 (S++1)
(21)
holds. Taking both (20) and (21) into (19), we have for any I
fo(S,)) S fo(S,) + mdh++
(22)
B.3 PROOF THEOREM 1
Proof. Recall that
1i = fg(S,) - (g(S,-1)-
(23)
According to Proposition 1, we have
fo(Stt) fo( m m
(24)
Afterwards, (24) equals to,
fo(Sm) - fo(S,) - (Vo(S,)) fs(((+1)) 2 1 - fs((.)
(25)
m
= fg(S (g(4))) S m
(fo(S%) -f(())))
Based (25), we have
fo(S%) (8(( & m m- (Jo(S fs(S))
(26)
< 111 (ff((8))) (S(()()))
m
j4t (fo(S!") - fo fSo))-
< ... m
Since fSa) = (0) 0, we have
fa(St) fo(Set1)
<( jt+1
(27)
So(Sin)
m
When Algorithm 2 terminates at step
1, we have,
fo(Sm) 2 (1 - (1 /f
(28)
18

---

Published as a conference paper at ICLR 2024
C SUPPLEMENTARY EXPERIMENTAL RESULTS
C.I SELECTED EXAMPLES
In Table 6, for illustration purposes, we provide a few examples from the selection by our method, when the annotation size is 18.
Dataset
Input
MRPC
a. Input: The two Democrats on the five-member FCC held a press conference to sway opinion against [..]]
Output: not equivalent
Input: The report shows that drugs sold in Canadian pharmacies are manufactured in facili-
ties approved by Health Canada [...]
Output: equivalent
c. Input: The chief merchandising officer decides what the store is going to sell [...]
Output: equivalent
SST-5
a. Input: plodding, poorly written, murky and weakly acted, the picture feels as if everyone
making it lost their movie mojo.
Output: very negative
b. Input: duvall is strong as always
Output: very positive
c. Input: lohman adapts to changes required of her, but the actress and director peter kosminsky never get the audience to break [...]
Output: neutral
MNII
a. Input: This prosperous city has many museums, including well-endowed Musce des Reaux- Arts (Square Verdrel) [...]
Output: False
b. Input: Duhame, who today makes her living as graphic designer and illustrator, calls her
book [...]
Output: Inconclusive
Input: At the agency or program level, it included management's 's public commitment to reduce fraud and errors, as. Based on that information [...]
c.
Output: True
DBpedia
a. Input: Lars Nielsen (bor 3 November 1960 in Copenhagen) is a Danish rower.
Output: athlete
b. Input: Calhoun County High School public secondary school St. Matthews South
Carolina USA.
Output: educational institution
Input: David Goldschmid (sometimes credited Dave Goldschmid) is American televi-
sion writer and producer currently writing for the daytime drama General Hospital.
Output: artist
RTF
a. Toput: In sub-Saharan Africa about one every 30 people is infected with HIV.. 30% of the people infected with HIV live in Africa.
Output: False
b. Input: The drawbacks of legalization do not imply that our current version of prohibition is the optimal drug strategy; may well [...]
Output: False
c.
Input: For example, the fields of Western farmers food the United States and many other
parts of the world. and India's irrigation [...]
Output: True
HellaSwag
a. Input: The topic Preparing salad. An illustrated egg, the website "startcooking com" and "vegetable salad" [...]
Output: shown from above.
b. Input: The topic is Pets and Animals. [header] How to treat injured rabbit's 's paw [titlc]
Identify sore hocks. [step] Pododermatitis [..]]
Output: Once the condition has set in, though. you'l 'Il need to take quick action to treat the
injury. Leaving [...]
c. Input: The topic is Playing squash. Two men stand on racquetball court. the men
Output: stretch then begin playing.
Table 6: For illustration purposes, under our method, show randomly selected three examples from each of the six datasets in one same run (excluding the other three datasets due to their length) when the annotation budget is set 18.
19

---

Published as a conference paper at ICLR 2024
C.2 VISUALIZATION OF SELECTED EXAMPLES
Here we provide umap (McInnes et al., 2018) visualization of selected examples. To avoid the denseness, we choose the annotation budget as 5. The visualization can be checked in Figure 6. First, comparing subfigures (a) and (b), we can clearly see that the selection of Vote- is much biased, and our IDEAL can
can identify subset that is more favorable to be proxy of full data. Second, comparing
subfigures (c) and (d), we can see that the selected subset by Vote-k is distributed on the right of full data. By comparison, our IDEAL can select subset that distributed more uniformly.
caappte
(a) SST-5, Vote-k.
(b) SST5, IDEAL.
Sabected
ecad
45
(c) MNLI, Vote-k.
(d) MNLI, IDEAL.
Figure 6: Umap (McInnes et al., 2018) visualization to compare five selected examples from all examples using fully unsupervised methods Vote-k and IDEAL (ours). Compared with Vote-k, IDEAL could choose the examples to better represent the whole data rather than get involved in diversity and including outliers.
C.3
DETAILED EXPERIMENTAL RESULTS IN TABLE
Method
MRPC
SST-5
MNLI
DBpedia
RTE
100
Random
64.3/68.4/58. 49.6/51.1/47.2
38.2/40.2/366
89.8/91.0/88 2
55.3/55.9/55.1
100
Vote-k 64.6/68.8/66 46.6/47.2/46
38.9/43.8 8/35.5
89,2/89 8/88.7
57. 77.6/58.2/51 .4
100
IDEAL
66.4/67.9/64.8
51.4/53.5/49.6
41.0/41 4/40.2
90.6/91.4/88 1/489.5
58.9/60.9/57.4
18
Random 57.4168.8/39.
42.9/46.9/39
37. 97.8/39.8 17.88/9..433 4/35.2
85.2/87.5/88.
57.9/58.9/93 9/57.0
Vote-k 61.1/67.2/52.7
41.7/45../37.11
39. 1/43 8/32.0
89.9/94. 1/87.1
58.2/58.9/57.1
18
IDAL
63.0/63.7/62.5
43.2/45.7/39.5
40.0/41 8/37.1
90.1/90.2/89.8
59.4/60.9/57.8
Table 7: Mear/nxximum/minimmm evaluation results of all methods classification tasks in Ta- ble over three different trials. The best mean result in each case is bolded.
In the main paper (Table 1), we report the mean evaluation results for different methods over three random trials. Here provided the detailed results of Table 1 with mean/naximummminimum values. We can observe IDEAL achieves stable results compared with baselines. Moreover, the worst-case performance of IDEAL is obviously better compared with baselines in most cases.
20

---

Published as a conference paper at ICLR 2024
Method
HellaSwag
MWoZ
GeoQ
Xsum
100
Random
66.7/70.3/64.
39.
39.9/48.4/39)
55.3/57.8/53.1
15.3/16.4/14.88
100
Vote-k
67.9/69. 9/699
48.3/50.8/46.9
58.8/60.5/57.0
17.2/17.6/164
100
IDEAL
68.6/71.9/5522
99.995
58.2/60.5/54.7
19.9/20.2/19.5
Random
66.0/68 86.6//8../63.7
37.0/46.5/28.
47. 47.5/49.2/44,
13.6/14.5/12.5
Vote-k
05.5/7119/624 37.7/43.8/32.4
50.9/54 3/47.7
15.2/16.0/14.55
18
IDEAL
67.1/71../64.5
38.5/47.3/30.9
52.0/53.9/50.8 19.6/20.2/18.9
Table Mean/maximuum/mininun evaluation results of all methods on multi-choice, dialogue, and generation tasks in Table over three different trials. The best mean result in each case is bolded.
MRPC
MNLI
RTE
Method
Equivalent
Not equivalent
True
Inconclusive
False
True
False
Original
2023
977
1051
965
984
1241
1249
Random
70
30
30
39
31
56
44
Vote-k
27
35
38
46
54
IDEAL 65
35
37
34
29
49
51
Table The numbers of different labels the selected examples for different methods. "Original" denotes the label statistics of the original dataset. Under the annotation budget 100, IDEAL achieves the smallest ratio between the numbers of the most frequent class and the least frequent class out of cases (MNLI and RTE), implying IDEAL can indeed mitigate the label skew problem.
MRPC
SST-5
RTE
Method
Mean
Std
Mean
Std
Mean
Std
Random
44.2
0.02
45.4
0.02
57.3
0.02
Vote-
52.7
0.03
38.0
0.01
58.6
0.02
IDEAL
65.5
0.01
46.6
0.01
57.5
0.02
Table 10: The average performance of different methods by permuting the order of prompts for each test instance 10 times. We conduct experiments on MRPC, SST-5, and RTE datasets and report the average results with standard deviation. We can observe the subset selected by IDEAL achieves the best performance compared with baselines in 2 out of cases. IDEAL also achieves the lowest standard deviations in all evaluations, which suggests IDEAL is a more stable and robust method against the order of prompts.
C.4 LABEL DISTRIBUTIONS IN SELECTIVE ANNOTATIONS
Recall that the process of selective annotations based entirely on similarities derived from sentence embeddings without labels. Therefore, investigate whether the selected examples have label skew. Under an annotation budget of 100, we collect all selected examples in three classification tasks (MRPC, MNLI, and RTE) and show the numbers of different labels for different methods in Table 9. We also present the label statistics of the original training data. We observe that random selection shows great variance. However, in an ideal case, it should achieve similar distribution as the original training data. Notably, IDEAL achieves the smallest ratio between the numbers of the most frequent class and the least frequent class in of 3 cases (MNLI and RTE). This demonstrates that IDEAL can indeed balance the label distribution in the selected subset and mitigate the problem of label skew.
C.5 PROMPT ORDER IN SELECTIVE ANNOTATION
As pointed out by (Lu al., 2021), the performance of in-context learning is influenced not only by the selection of prompts but also by the order in which the prompts are presented Although this work focuses solely selective annotation problems, interested in explor-
to models.
21

---

Published as a conference paper at ICLR 2024
ing whether the selected subset can still lead to better performance when the order of prompts is permuted. Under an annotation budget 18, we first retrieve prompts for each test instance from selected subsets achieved by different selective annotation methods. We then permute the order of prompts for each test instance 10 times, resulting in 10 different experimental trials. We show the average performance of these 10 trials and make comparison between different selective annota- tion methods. We conduct experiments on MRPC, SST-5, and RTE datasets and present the results in Table 10. The results show that IDEAL outperforms baselines 2 out of 3 cases, suggesting that our method can choose more stable and robust subsets against changed prompt orders.
D SUPPLEMENTARY DESCRIPTIONS OF EXPERIMENTAL SETTINGS
D.1 DETAILS OF DATASETS
In this paper, to demonstrate the superiority of our method, we employ 9 datasets typical datasets that have been widely used in previous NLP works (Su et al., 2023; Shi et al., 2023; Zhang et al., 2021) which can be categorized into different tasks, including classification (MRPC (Dolan al., 2004), SST-5 (Socher et al., 2013), MNLI (Williams et al., 2017). DBpedia (Lehmann et al., 2015), and RTE (Bentivogli et al., 2009)), multi-choice (HellaSwag, (Zellers et al., 2019)), dia- logue (MWoZ (Budzianowski et al., 2018)), and generation (GeoQuery (Zelle & Mooney, 1996) and Xsum (Narayan al., 2018)). We list the datasets and the models used in Table 11.
Datasets
Task
Models
MRPC (Dolan al., 2004)
Paraphrase Detection
GPT-Nco, GPT-J. GPT-3.5-Turbo GPT-J
SST:5 (Socher al., 2013)
Sentiment Analysis
Classification
DBpedia (Lchmann al., 2015)
Topic Classification
GPT-
RTE (Bentivogli al., 2009))
Natural Language Inference GPT-Neo, GPT-J, GPT-3.5-Tubo
MNLI (Williams et al., 2017)
Natural Language Inference GPT-Neo, GPT-J, GPr-
Multiple-Choiee HellaSwag (Zellers al., 2019)
Commonsense Reasoning
GPT-J
Dialogue
MWo7. (Budzianowski et al., 2018)
Dialogue State Tracking
Text-davinci-002
GeoQuery (Zelle & Mooney, 1996)
Semantic Parsing
Text-davinci-002
Generation
Xsum (Narayan et al., 2018)
Summarization
GPT-
Table 11: The datasets and corresponding models used in experiments. We use GPT-J 6B and Text-davinci-00) by default. Other large language models are explored in $4.3.4.
To help readers better understand the datasets and tasks, for each of these datasets, also list one example including both the input and output.
D.1.1 MRPC
Input
Are the following two sentences 'equivalent' or 'not equivalent' AAA federal judge yesterday disconnected a new national \" do-not-call \" list just days before it was to take effect saying the agency that created it lacked the authority ..A federal judge yesterday struck down the national do-not-call registry slated to take effect Oct. ruling the rederal Trade Commi onmisssin had no authority to create the list Ananswer::
Output
equivalent.
D.1.2 SST-5
Input
22

---

Published as a conference paper at ICLR 2024
How do you feel about the follewing Centence?\nsmug artificial ill-constructed and fatally overlong ... it never finds a consistent tone and lacks bite degenerating into a plous preachy soap opera Ananswer:
Output
neutral
D.1.3 MNLI
Input
yeah well the Cardinals don't know think the Cawboys probably y have a better team they just at the end the season the kind got messed up with Aikman getting hurt because uh Laufenberg just couldn't never really get it together at all of course he sat along the sidelines all season he never really got in gane never did whole lot. Based on that information, have started Laufenberg all season. \"True\", \"Falss\", or \" Inconclusivel "?\nanswer:
the claim The Cowboys should
Output
Inconclusive
D.1.4 DBPEDIA
Input
title: Vlu00edetor David Leubriel; content: V\u0Oedetor David Loubriel Crtlu00edz a Puerto Rican politician and former member of the Senate of Puerto Rico for the New Progressive Party (PNF)
Loubriel presented his candidacy for the Senate of Puerto Rico before 2004. He ran for candidate slot in the 2003 primariss obtaining the most votes in his district (Arecibo) IIn the 2004 general election Loubriel won a seat in the 23rd Senate of Puerto Rico to represent the district of Arecibo along with Jos Emilio Gonz\uOOellez Vel (u0Delzquez.
Output
office holder
D.1.5
RTE
Input
MEXICO CITY (Reuters) A deadly strain swine flu never seen before has broken out in Mexico, killing as many as 60 people and raising fears it is spreading across North America. the world Health Organi said it was concerned about what it called 800 \a influenza-likel)
in Mexico, and also about confirmed outbreak a new strain of swine flu in the United States. It said about people had died in Mexico. Mexico's government said it had confirmed that at least 16 people had died of the swine flu in central Mexico and that there could be another 45 fatal victims.. Anquestion: 800 Mexicans have been affected by new form swinc influenza.. True or \nanswer::
Output
Truc
23

---

Published as a conference paper at ICLR 2024
D.1.6 HELLASWAG
Input
The tepic is work World. [header] How to become high school socia studies teacher [titie] Obtain your bachelor's degree in educat on. [ stepl All schools will require you to obtain at least your bachelor's s degree in education. This degree will be proof that you capable delivering information to students using the current educational best practices.
Output
Make sure you've ve fully completed all of your course work and obtained your bachelor's degree before you seek certification or employment. [substeps] Your electives should be based in social studies courses.
D.1.7 MULTIWOz
Input
CREATE TABLE hotel name text,
internet text CEECK (internet IN (dontcare, yes, no)
/+
example rows:
SELECT FROM hotel LIMIT 4; name pricerange type parking book_number_ofe book_day book_pcoplc
area stars internet
and b guest house moderate guest house dontcare 3 friday 5 east 4 yes
-- Using valid SQlite, answer the following multi-turn conversational questions for the tables provided above. Example #1
[context] hotel-area: west, hotel-stars: 3, hotel-internet: yes [system] the hobsons house is available in that area
Q: [user] that sounds like it will work can i book that for 3 nights
starting wednesday
SQL: SELECT * FROM hotel WHERE book_day = wednesday AND book_peoplc
AND book-nnmber_one _das AND name hobsons house;
Output
hotel WHERE book_day wednesday AND book number days AND
warkworth house;
D.1.8
GEOQ
Input
CREATE TABLE "border_info" ("state_name" text, "border" text)
/*
state_name
border
alabama tennessee
alabama georgia
alabama
florida
24

---

Published as a conference paper at ICLR 2024
Using valid sQLite, answer the following questions for the tables provided above.
what is the longest river in the state with the highest point SELECT
Output
RIVERaliase 1as0.. Rww.aa......RRAEE.. RIVER_NAME FROM HIGHLOW AS HIGHLowallasO, RIVER AS = (SELECT MAX FROM HIGHLOW AS HIGHLOWalias] AND RIVERaliasO TRAVERSE HIGHLOWaliasO. STA.STAATTAAAA ORDER BY RIVERaliasO.
RIVERal las0 WHERE AGGHLoaalias) L.S...HHEESS...H.E.......S.. HIGHEST_ELEVATION ELEVVTTION
HIGHEST
HIGHLOWallissI.TTGGH Bat.aas..GGHSSS__LLS___SSS
LENGTH DESC LIMIT 1;
D.1.9 XSUM
Input
For decades, large numbers of Haitians have migrated many of them without papers to the Dominican Republic, to escape the poverty and lack of employment in their momeland.Anie 2013, the Dominican Republic's highest court ruled that children born there to undocumented migrants were not automatically eligible for Dominican nationality.
AnThere he strips the trees for firewood to make charcosl, to sell to Dominican traders for few dollars. AnHe knows the practice damages the fertility of the soil, but it's the only available source income. TThis is the only way we can survive, \" he says, motioning at his family, stuck inside the world's forgotten migrant crisis. nYou can hear more of Will Grant's report on Heart and Soul on the BBC World Service.
Output
Immigration has long been divisive issue on Hispaniole, the Caribbean island shared by Haiti and the Dominican Republic.
D.2 IMPLEMENTATION DETAILS
General experimental conditions. We primarily PyTorch (Paszke al., 2019) implement our algorithm and baselines. For GPT-3.5-Ture nAI API using single Intel Xeon CPU. The GPT-J 6B and GPT-Neo 2.7B models are from the Huggingface transformer library (Wolf et al., 2019). We run all our experiments of GPT-J 6B and GPT-Neo 2.7B on a single NVIDIA Tesla V100 (32GB) GPU.
we perform the experiments by calling the Ope-
Details of getting unlabeled data. Since obtaining unlabeled examples in realistic scenarios is also virh-variance process, follow the same setting as (Su al., 2023) simulate the realistic setting. We perform selective annotations from 3k instances that are randomly sub-sampled from training data for each task. For each experiment, we repeat the sub-sampling process three times and average the results over all trials ensure comprehensive evaluations.
Details of the graph construction. Except for the illustration experiment Figure 2, we construct the directed graph for all unlabeled data by connecting each vertex to its 10 nearest successors (k 10). important to note that a larger & will lead to an increase the computation cost. We have chosen this setting because it provides good performance while maintaining efficient computation costs. For Figure 2, we construct the graph by connecting each vertex to its nearest successors in order to avoid denseness.
Details of Algorithm 1. Considering the randomness of the diffusion process, when quantifying the influence of the subset, run Algorithm 1 10 times and use the averaged influence value. Note that we also calculate the time cost in this repeated process when reporting the final results in the main paper. As shown in Figure algorithm is still more effective than Vote-k.
25

---

Published as a conference paper at ICLR 2024
E TIME COMPLEXITY ANALYSIS
In this section, we perform a time-complexity analysis of our method. Given the constructed graph with nodes and edges, and annotation budget of m. the whole algorithm involves the following two parts that incur the following time costs: (1) Information diffusion process. The time complex- ity of quantifying the influence of specific subset is ()+ This is because it involves running an independent cascade diffusion process (BFS-like traversals) through the graph. (2) Greedy search. The greedy algorithm iteratively selects the example that provides the maximum marginal gain in influence. When the annotation budget is m. the time complexity is O * (a+b)). This is because, the worst case, the algorithm needs to evaluate the influence of each node the network. Besides, for each node, the algorithm needs to perform a simulation with time complexity of O(a b).
F LIMITATIONS
Memory cost. Although in-context learning tasks avoid the heavy parameter update process, they still require large amount of memory to load models. For example, loading GPT-J 6B into GPÜ requires about 23GB GPU memory, without considering the size of the dataset. This is relatively high cost for individual researchers.
The limitation of Auto-IDEAL. Auto-IDEAL outperforms IDEAL in terms of performance, but it has two main drawbacks that hinder its usability in practice. First, Auto-IDEAL suffers from a model inference cost, especially in the era of large language models service. Specifically, when performing automatic annotation, Auto-IDEAL has to make predictions for all unlabeled examples to achieve automatic annotation. This makes it less practical than the native IDEAL. Second, Auto- IDEAL may fail when the initial examples to be labeled are not very relevant to the initial examples labeled, even though they have similar embedding. Auto-IDEAL performs automatic annotation by following the information diffusion process from the initial annotated subset to the examples to be labeled with similar embedding. When the examples be labeled are relevant to the initial examples labeled, it may lead to incorrect automatic annotations and then poor performance. Future research may focus on maintaining superior performance while reducing the automatic annotation cost of IDEAL.
Potential benchmark leakage GPT-3.5-Turbo. There may potentially exist benchmark leakage problems in the evaluation process (Zhou et al., 2023). Specifically, due to the fact that GPT-3.5- Turbo trained using huge text datasets as of 2021, the data related evaluation sets may po- tentially be used for model training. This could lead to potential risks in the evaluation process. However, as our work does not involve the training data selection, the impact should be negligible.
26


Reviews:
[{"review_id": "fcaU0vANCb", "reviewer": "Reviewer_RM4K", "summary": "This paper presents a work on selective annotations in in-context learning. Given a pool of unlabeled instances to annotate for prompts, this paper proposes an influence-driven framework to identify those more significant data points for annotations and follow-up tasks. Compared with the previous method, the proposed method enjoys (1) greater convenience in selection due to the end-to-end manner; (2) it does not need to balance diversity and representativeness of selected instances; (3) theoretical analysis is provided to show the subset influence is at least as large as a certain proportion of the influence of the optimal solution. Comprehensive experiments are conducted to show the superiority of the proposed method, which achieves the best performance in most cases. The time consumption in selection is significantly lower than previous state-of-the-art methods.", "questions": "1)\tThe experiments can be supplemented by including other methods not just Random and Vote-k, as mentioned above. \n2)\tThe paper claims that it conducts experiments three times and reports the average score. What/where is the randomness? It mainly comes from examples selected or model predictions? Or both of them?\n3)\tFor the process of information diffusion, in the main paper, it seems that the process is only performed once. In the Appendix, the paper discusses that multiple processes are performed. I suggest that this description can be moved to the main paper. Also, what is the influence of the times of information diffusion? \n4)\tAdditional elaboration is required to explain why the combination of IDEAL/vote-k and similarity retrieval proves effective, while IDEAL/Vote-k alone, coupled with the random selection of supporting examples, does not work well. \n5)\tRecently, Chain-of-Thought (CoT) prompting has gained widespread adoption in in-context learning research. Several studies employing this method have reported enhanced model performance. However, an unexplored aspect of this approach is the model's sensitivity and variance concerning the retrieved prompts when used in conjunction with CoT. This work does not address whether such sensitivity exists and whether CoT can be applied for better performance.\n6)\tIs there any consideration given by the authors to the cost associated with encoding every unlabeled training instance using Sentence-BERT when comparing the methods? As far as I can discern, this aspect was not addressed in the paper.\n7)\tIt seems that the instance embedding has a large Influence on the proposed method since it needs to build a directed graph using the embeddings. I am interested in changing the previously used pre-trained models and seeing the robustness of the proposed method. \n8)\tThe paper shows the label distribution brought by the proposed method. What about the distribution of original data (before selecting)? Are their label distributions similar? Besides, a balanced distribution is expected to bring better performance in follow-up tasks? Or otherwise. I am interested in this.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tIn-context learning is an important and swiftly advancing research area, capturing significant attention from both the ICLR and NLP communities. This paper conducts comprehensive evaluations on the influence of selecting specific examples for inclusion in the prompt. The findings illustrate the substantial impact this choice can have on the final model's behavior.\n2.\tThe paper is well-structured and easily comprehensible. Within the evaluation section, there are numerous intriguing findings that hold significant value for dissemination within the wider research community. Besides, the source code is provided to ensure the reproducibility of the empirical findings. \n3.\tThe proposed method is simple and relatively straightforward to implement. Consequently, there is a possibility that it could be utilized in real-world applications of in-context learning.\n4.\tTheoretical analysis is provided to demonstrate the effectiveness of the proposed method under a greedy search algorithm.", "weaknesses": "1.\tThere are some unclear expressions and inconsistent explanations. Some polishes are needed. \n2.\tSome experimental settings only include the baselines Random and Vote-k. More methods as mentioned in 4.3.2 can be also included. \n\nMore questions about weaknesses can be checked below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HnozqYKCnu", "reviewer": "Reviewer_Ctph", "summary": "In this paper, the authors propose the method using influence-driven selective annotations to minimize the annotation cost, thereby tackling the high annotation costs of discovering the right prompts for in-context learning. They construct a direct graph to represent unlabeled data and use a diffusion process to quantify the influence of unlabeled subsets. Finally, a greedy algorithm is utilized to conduct the final selection. Experiments are conducted on several benchmark datasets, and the experimental results show that the proposed method can achieve better performance with lower computational time in subset selection.", "questions": "* Q1: Following W1,it would be great if the authors could patch more comparisons to more state-of-the-art works during the author feedback period.\n* Q2: Following W2, I wonder why there are some inconsistent behaviors of the proposed method, instead of only getting the message \"the proposed method performs still not bad\". This is important because it would decide the choice between IDEAL and Auto-IDEAL.\n* Q3: Following W3, because the proposed method should be general, it is reasonable to apply the method to different LLMs. It would be great if the authors could also conduct some related studies.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* S1: The influence-based greedy method can effectively and efficiently help find the right prompts from the large corpus.\n* S2: Experimental results show improvements over Random and Vote-K.\n* S3: The authors provide some preliminary theoretical proofs.", "weaknesses": "* W1: Some state-of-the-art studies like MDL [a] TopK (Liu et al., 2022) are not cited and/or compared in the experiments.\n* W2: The rationales behind some experimental results are not explained, such as the reverse performance of MNLI in Table 5.\n* W3: Lack of analysis on how the proposed method \n\n\n\n[a] Wu, Z., Wang, Y., Ye, J., & Kong, L. Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering, ACL 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ovc9Odrp0e", "reviewer": "Reviewer_Ra1D", "summary": "In this paper, the authors propose an influence-driven demonstration selection method. I like the theoretical analysis of the influence function. However, I hold two main concerns: (i) if it requires running the influence function during the inference, then it is crucial to have the time complexity analysis, (ii) how the proposed approach considers the correlations among the retrieved data points.", "questions": "The topic of the demonstration selection is a hot and essential topic. I like the idea of investigating the demonstration selection from the perspective of influence. I like Section 3. I have two main concerns. First is that for each inference of LLM, whether it is required to run the influence function for n times to retrieve n shots. If it is true, it is very important to have a complexity analysis of running time. Second, I do not see how to consider the correlations among the retrieved samples. For greed search, there may be cases where A+B is better C+B, and C has more influence than A. I want to know how the proposed method addresses these cases. Furthermore, I also want to know how the proposed method leverages the features of each data sample. In other words, the nodes in the graph have their own features. How do you embed these features? Using sentence transformer or some GNN-based methods? And, what is the cost of applying these methods? Also, for the experiment, there are many demonstration selection methods in the NLP field, for example, similarity-based or diversity-based methods, which are widely employed in the LLMs. Therefore, I highly recommend the authors compare their method against these existing approaches. It would be great to see some results for more datasets (especially more tasks beyond the text classification) and more LLMs. Overall, I do not think this version is ready for publication.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Investigating the demonstration selection task from the influence perspective is very interesting.\n2. It is great to have a theoretical analysis of the property of the influence function. \n3. Overall, this paper is well written.", "weaknesses": "1. There is no time complexity in the proposed method, which is very crucial if it needs to run for every inference. \n2. It is not clear how the proposed method considers the correlations among the retrieved data points. \n3. It is better to compare with more demonstration selection methods such as similarity-based and diversity-based methods which are widely used in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q0uAAtoxUY", "reviewer": "Reviewer_QkNp", "summary": "This paper studies the selective annotation for minimizing annotation cost on massive unlabeled in-context examples. To tackle the drawbacks of the existing Vote-k method and choose diverse and representative in-context examples, this paper proposes an unsupervised influence diffusion method to quantify the influence of each unlabeled example in the example corpus. Experimental results on various datasets and LLMs validates its effectiveness with reduced time consumption. The paper also proves the influence lower bound of the in-context selection method.", "questions": "- This paper uses small LMs less than 7B, and ChatGPT. Will the method still be effective on medium-sized LMs such as LLaMA2-chat 7B, 13B? And do you use any instruction for testing on ChatGPT?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The selective annotation is a meaningful research problem, since in reality LLMs need to to generalize to novel tasks without the existence of large amount of labeled data. \n- The previous Vote-k method needs to get the uncertainty from LLM predictions which incurs large computation time consumption, while the proposed method bypasses such need and considers from the perspective of data influence in the corpus. Also, this method reduces the LLM sensitivity on in-context examples, such as the order, which increases the robustness of in-context learning. \n- The experiments is conducted on different LLMs and various tasks, showing its effectiveness.", "weaknesses": "- This paper is related to \"Li et al., Finding Support Examples for In-Context Learning\", which also adopts the idea of coreset selection in in-context example selection, and \"Diao et al., Active prompting with chain-of-thought for large language models\", It is suggested to include discussion with these papers in the related work. \n- The theoretical analysis is about the influence lower bound. How the influence score relates to better diversity and representativeness needs further illustration, since it is the major consideration of the proposed method. \n- A minor suggestion is that the experiment datasets can also include some novel LLM benchmarks that are probably not trained on the experiment LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YL5i5ek55P", "reviewer": "Reviewer_oY6L", "summary": "In order to improve the performance of Large Language Models (LLMs) via in-context learning, this paper introduces a selective annotation method for creating, on a budget, high-value prompts for the LLM. The proposed approach selects a salient subset of examples to annotate (and choose-from for future prompts) from a large-scale unlabeled data pool. It uses the unlabeled data to (i) create a directed graph of \"related examples,\" (ii) estimate  the influence of the candidates-for-annotation via a diffusion process, and (iii) iteratively select the data that provides a maximum marginal gain with respect to quantified influence.", "questions": "1. why don't you claim Auto-IDEAL as a major contribution?\n2. why don't you use all datasets in all experiments, but rather different subsets in 4.3.1, 4.3.2, and 4.4?\n3. how did you end up with the two values of 18 and 10 (especially the \"18\" one)?\n4. why do we see such large variability in performance among the three different runs? why din't you try more runs, so that you can get a better understanding of the scale of the variability issue?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel approach to improving in-context learning, which is an important problem with a myriad of strategic practical application. The works appears to be original and the. empirical evaluation suggests that this work could have major impact (also see comments below on how to make a stronger case along these lines). The paper is reasonably well-written and organized; see below suggestions on how to further improve it.", "weaknesses": "The most improvements could be achieved by tightening up the Empirical Evaluation section:\n- ideally, rather than (arbitrarily?) choosing k = 18 & k = 100, you should provide an automated approach to AUTOMATICALLY assess the smallest amount of annotations that leads to the best possible performance; you only introduce Auto-IDEAL in \"4.4\" but you do not seem to claim it as a major contribution. Why?\n- in Table 2:\n   a) please add an additional row that shows the best-known performance on each dataset; without it, it is impossible to quantify the practical impact of IDEAL (i.e., is \"just\" improving on Vote-k, or does it come close to the best-known performance?)\n   b) please also add to the Table results for k = 1,000 and even k = 10,000. How big are these improvements, if any?\n   c) please explain why do we see IDEAL(100) < IDEAL(18) on RTE. Is there a general lesson worth sharing?\n   d) similarly, why does Vote-k outperform IDEAL(100) on GeoQ (Tables 2 & 8). In particular, given that Table 8 suggest that there is a fair amount of variability among the various trials, ideally you should increase the number of trials to 10 or even larger. In the current setting, according to Table 8, for \"100\" Vote-k and IDEAL obtain the same best result (60.5) on GeoQ; similarly, for \"18\", there is a similar \"tie\" on HellaSwag, and Vote-k gets a better max on GeoQ.\n- section 4.3.1 should cover all datasets, not only two; worst case scenario, if space is an issue, you can summarize the results on all datasets here, and refer the reader to the appropriate appendix\n- same as above, section 4.3.1 should cover all datasets, not only three of them. Also: (i) why have Table 2 for \"18\" rather than \"100,\" when IDEAL(100) outperforms IDEAL(1*0 on all datasets, and (2) similarly to the above comment on stability, you should use at least 10 randomized trials. \n- same as above, section 4.4 should cover all datasets, not only five of them\n\nOTHER COMMENTS:\n1) please add early-on (e.g., in the Introduction) a few illustrative examples. For one or two of your application domains, you should show:\n- an un-annotated example that IDEAL selects for annotation\n- the annotated version of that example\n- the answer of the LLM w/o that annotated example used as a prompt\n- the answer of the LLM w/o that annotated example used as a prompt\n- explain and/or discuss the difference between the two answers above  \n\n2) please add to the paper a paragraph or two about the actual/expected cost of annotation for the various domains in the empirical evaluation.\n\n3) please intuitively explain the \"end-to-end\" term that you are using in the Abstract, Introduction, and later on in the Appendices, but nowhere else in the rest paper. If it is an important concept, it should also appear in the \"meat\" of the paper; if it is not, you should remove all references to it as it is distracting to see it mentioned early on, never clarified, and then simply ignored. \n\n4) both in Fig 3's caption and in the abstract, please summarize the running time improvements of IDEAL vs Vote-k. The current \"largely reduces the time cost\" is simply too vague. Ideally, the last sentence of the abstract should end something like this: \"... improving the performance by X% while reducing the running time by Y%\"\n\n\nNitpicks:\n- p 1: \"pretty performance\" --> \"strong performance\"\n- p 1: please add a reference and /or brief summary for \"requires substantial manpower and financial resources\"\n- p 2: \"Vote-k devoids theoretical guarantees\" --> \"Vote-k lacks theoretical guarantees\"\n- p 2: please explain why \"the algorithm does not need a delicate trade-off between diversity and representativeness\"\n- p 2: last paragraph of the Intro: please also quantify the performance improvement, not only the gains in speed\n- p 3: for Fig 2, please explain if/why the iterations start at (d).  \n- p 5: probably the theorem should be called \"Theorem 1\" (or else the reader expects to see a Theorem 1 before it)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)