Large Language Models are Demonstration Pre-Selectors for Themselves
Authors: Jiarui Jin
Yuwei Wu
Haoxuan Li
Xiaoting He
Weinan Zhang
Yiming Yang
Yong Yu
Jun Wang
Mengyue Yang
Venue: ICML 2025 poster (2025)
Keywords: Large Language Model
Demonstration Pre-Selection
In-Context Learning
Primary area: deep_learning->large_language_models
TL;DR: Our main contribution is to propose a novel demonstration pre-selector which is different from previous demonstration selectors.

Abstract: In-context learning with large language models (LLMs) delivers strong few-shot performance by choosing few-shot demonstrations from the entire training dataset. However, previous few-shot in-context learning methods, which calculate similarity scores for choosing demonstrations, incur high computational costs by repeatedly retrieving large-scale datasets for each query. This is due to their failure to recognize that not all demonstrations are equally informative, and many less informative demonstrations can be inferred from a core set of highly informative ones. To this end, we propose FEEDER (FEw yet Essential Demonstration prE-selectoR), a novel \emph{pre-selection} framework that identifies a core subset of demonstrations containing the most informative examples. This subset, referred to as the FEEDER set, consists of demonstrations that capture both the ''sufficiency'' and ''necessity'' information to infer the entire dataset. Notice that FEEDER is selected before the few-shot in-context learning, enabling more efficient few-shot demonstrations choosing in a smaller set. To identify FEEDER, we propose a novel effective tree based algorithm. Once selected, it can replace the original dataset, leading to improved efficiency and prediction accuracy in few-shot in-context learning. Additionally, FEEDER also benefit fine-tuning LLMs, we propose a bi-level optimization method enabling more efficient training without sacrificing performance when datasets become smaller. Our experiments are on 6 text classification datasets, 1 reasoning dataset, and 1 semantic-parsing dataset, across 6 LLMs (ranging from 335M to 7B parameters), demonstrate that: (i) In few-shot inference, FEEDER achieves superior (or comparable) performance while utilizing only half the input training data. (ii) In fine-tuning, FEEDER significantly boosts the performance of LLMs.

Large Language Models are Demonstration Pre-Selectors for Themselves
Jiarui Jin 12*1
Yuwei Wu
Haoxuan Li Xiaoting He Weinan Zhang'
Yiming Yang Yong Yu'
Jun Wang Mengyue Yang*?
Abstract
1. Introduction
Large language models (LLMs), e.g., GPT (Brown et al., 2020), Gemma (Team al., 2024), and Llama (Touvron et al., 2023), have demonstrated impressive performance across wide range of tasks by employing few-shot infer- ence, referred as in-context learning (ICL) (Brown et al., 2020; Dong et al., 2022). This approach avoids the compu- tational expense associated with fine-tuning LLMs. Here, the core challenge how to select the most representative demonstrations from large training data.
In-context learning (ICL) with large language models (LLMs) delivers strong few-shot per- formance by choosing few-shot demonstrations from the entire training data. However, exist- ing ICL methods, which rely on similarity or di- versity scores to choose demonstrations, incur high computational costs due to repeatedly re- trieval from large-scale datasets for each query. To this end, we propose FEEDER (FEw yet Es- sential Demonstration prE-sclectoR), novel pre- selection framework that identifies a core sub- set of demonstrations containing the most repre- sentative examples in the training data, tailored to specific LLMs. To construct this subset, we introduce the "sufficiency" and "necessity" met- rics in the pre-selection stage and design tree- based algorithm to identify representative exam- ples efficiently. Once pre-selected, this represen- tative subset can effectively replace the full train- ing data, improving efficiency while maintain- ing comparable performance in ICL. Additionally, our pre-selected subset also benefits fine-tuning LLMs, where introduce bi-level optimization method that enhances training efficiency without sacrificing performance. Experiments with LLMs ranging from 300M to 8B parameters show that FEEDER can reduce training data size by over 20% while maintaining performance and seamlessly in- tegrating with various downstream demonstration selection strategies in ICL.
Early approaches (Qiu et al., 2022: Liu al., 2021; Rubin et al., 2021; Wang et al., 2022) primarily selected demonstra- tions based on relevance, typically using similarity scores between each demonstration and the input question. How- ever, recent studies (Levy et al., 2022; Köksal et al., 2022; Zhou al., 2023) indicate that evaluating examples in isola- tion is suboptimal. Instead, they advocate for incorporating additional selection criteria, such diversity, uncertainty, or slustering-based based metrics, alongside similarity, to enhance demonstration selection effectiveness. As result, the above enhancements in the demonstration selection process intro- duce significant computational overhead, particularly when the number of shots is large. Beyond efficiency, we argue that the effectiveness of selected demonstrations should also consider the specific LLM in use, different LLMs exhibit varying capabilities and knowledge domains.
To this end, we introduce new stage called the pre-selection stage and propose a demonstration pre-selector named FEEDER (FEw yet Essential Demonstration prE-selectoR). FEEDER functions as representative subset selector, aim- ing to identify a representative subset from the full training data before the demonstration selection process, thereby en- hancing both efficiency and effectiveness in demonstration selection. For this purpose, we introduce the concepts of "suffficincy" and "necessity",, where sufficiency investigates whether incorporating a demonstration is representative of other samples, while necessity examines whether a demon- stration provides redundant information, taking into account the LLM's capability, knowledge, and the selected samples. By directing LLMs to focus on the selected subset, FEEDER prevents them from processing unnecessary data, thereby improving efficiency. Since exhaustively enumerating and
Equal contributions: Jiarui Jin and Yuwei Wu. Work done during Jiarui Jin's visit at University College London. 'Shanghai Jiao Tong University *Xiaohongshu Inc. CCarnegie Mellon Uni- versity "Peking University 'No Affiliation "University College London 'University of Bristol. Correspondence Mengyue Yang <mengyuc. br.kt.l.c.uu>>
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Large Language Models are Demonstration Pre-Sclectors for Themselves
(a) In-Context Learning with Frozen LLM
(b) Fine-Tuning with Trainable LLM
Training Dataset
FEEDER Input and Output
Demonstration Selector
Training Dataset
FEEDER Input
conditioned
Trainable Large Language Model
Output to FEEDER Input
FEEDER Output
Test Input Data
Frozen
Output to Test Input
Large Language Model
Loss
Figure 1. Overview of FEEDER that operates effectively within both in-context learning and fine-tuning settings. In the in-context leamning setting, depicted in (a), we first pre-select a representative subset termed FEEDER from the training dataset, and then incorporate existing demonstration selector to samples regarding specific test input. This pre-selected subset is characterized by its sufficiency and necessity conditioned on the frozen L.M. In the fine-tuning setting, shown in (b), FEEDER allows the L.IM to be tuned on the fixed subset FEEDER, and this subset is intentionally pre-selected to be a faithful representation of the training dataset, with the dual objectives of maintaining data quality and minimizing computational expenses. The above two processes can be encapsulated into a bi-level optimization framework, allowing for iterative refinement of both the pre-selected FEEDER and the fine-tuned LLM.
evaluating all possible subsets is impractical, we develop a tree-based approximation algorithm that leverages the ca- pability of the given LLM to assess whether each subset is sufficient and necessary for representing others.
The pre-selected subset is beneficial not only for ICL but also for accelerating the fine-tuning process while maintaining comparable even superior performance.
Besides ICL, we further observe that our pre-selection pro- cess can enhance the fine-tuning process by allowing LLMs to achieve comparable or even superior performance while training on the pre-selected representative subset of the data, rather than the full data, where we introduce bi-level opti- mization method that enhances training efficiency without sacrificing performance.
2. Demonstration Pre-Selection
We begin by delineating two distinct contexts where FEEDER can operate: ICL and fine-tuning settings. Throughout this paper, we approach both scenarios from a data-centric per- spective (Strickland, 2022), emphasizing the significance of data quality over data quantity.
In the ICL setting, we given training dataset DTRAIN {( )n) consisting of pairs of input data (e.g.. ques- tions) and output labels (c.g., answers). We are also given a test dataset DrEST = {(Im= MmmM where we assume that DIRAIN share the same support set (Yosida, 2012) with DIEST- Our goal is develop a demonstration selector that extracts n-shot demonstrations from the training dataset, denoted DOEMo C DTRAIR- We use V X x D Y to represent a L.M using selected demonstrations as the context. Here, x. E X is an input text, E Y is the cor- responding output, and (x., y.) E D is one demonstration. Formally, our objective is to minimize:
Our empirical evaluations encompass six LLM bases, rang- ing from 335M to 7B parameters, and include six demon- stration selectors in the demonstration selection stage, ap- plied to text classification, reasoning, and semantic parsing tasks. Our results demonstrate that our pre-selection pro- cess produces subset that consistently reduces the data size for demonstration selection by over 20% across vari- ous datasets. Moreover, using this pre-selected subset in ICL with a simple similarity-based metric can outperform existing sophisticated metrics such as diversity- based ap- proaches. Additionally, we show that this subset can en- hance the fine-tuning process, enabling L.MSs trained on the selected representative subset to achieve comparable or even superior performance.
(DDeNNO DIEST)
Σ (20..Mm D.n))
(Vii DDENO), ) ym
To summarize, our pre-selector FEEDER offers three key advantages:
where E(..)) is the task-specific loss function, and (iLn() means that the LLM is frozen. However, since we do not have access to PrEST during the training phase, is imprac- tical to optimize the demonstration selection directly by minimizing L(DDEMO, DrEST).
The pre-selection stage effectively identifies a represen- tative subset, eliminating unnecessary data and signifi- cantly reducing the high computational complexity of downstream demonstration selection.
FEEDER evaluates the representativeness of demonstra- tions in terms of sufficiency and necessity, taking the LLM's capabilities and knowledge into consideration.
Instead, we re-consider the demonstration selection task as a two-stage problem, where we first pre-select a subset of high-quality demonstrations from DIRAII as the selec- tion pool, i.e., a FEEDER subset denoted as DrEEDER; and
2

---

Large Language Models are Demonstration Pre-Sclectors for Themselves
then we apply existing demonstration selectors such as ran- dom or similarity-bases selectors DEEEDER, choose the corresponding demonstrations context for specific test instance. Our key idea is that a high-quality training dataset DPEEDER should be both representative of the entire training dataset PERI and as minimal in size as possible. Formally, we use the loss function (DEEEDE Dramn) from Eq. (1) to evaluate our pre-selector, i.e., how well the representation of PEEZDER aligns with DIRAIK- Then, our objective can be written as:
3. Connections to Existing Work
With the growing capabilities of LLMs, data (referred to as "demonstrations") selection has gained prominence. which involves selecting suitable examples as the context for in- context learning (Dong et al., 2022; Yang et al., 2023; Zhou et al., 2022) or filtering subset from training examples for fine-tuning (Sachdeva et al., 2024; Zhou et al., 2024), Previous solutions have revolved around constructing either parameter-free free selection mechanisms (Wang et al., 2022; Zemlyanskiy et al., 2022; Gao et al., 2023) or neural-based selection methods (Pasupat et al., 2021; Liu et al., 2021; Gupta et al., 2021; Rubin et al., 2021; Li et al., 2023). Re- cent investigations (Xia et al., 2024; Marion et al., 2023) focus on mining training examples for fine-tuning specific tasks, with Wang et al. (2024) and Wan et al. (2025) ex- tending this approach to in in-context learning. In contrast to previous methods that use LLMS as demonstration selectors, our work leverages the powerful few-shot inference capa- bilities of LLMs by employing them as pre-selectors. For this purpose, we introduce a pre-sclection stage to exam- ine "sufficiency" and "necessity" to identify representative subset of training examples. The resulting FEEDER subset can serve a dual purpose: they can be used as candidate input contexts or to fine-tune the LLM. In both scenarios, FEEDER can significantly reduce the computation costs by substituting the entire training data with FEEDER subset.
min Damer C DIRATS
DPEEDER|:
s.t. (DFEEDER DIRAIN)= & (DTTAIN Drnan).
This formulation indices that PFEEDER should be not only suf- ficient but also necessary to represent DTRAI,, thus removing redundant data points to save computation costs meanwhile maintaining LLM performance.
Our pre-selected subset of high-qualit, data DYZEDER also can be applied to fine-tune L.MSs. Concretely, instead of fine- tuning LLMs on the entire training dataset DIRAIN, DFREDER allows us to fine-tune LLMs with few but high-quality data, reducing computation costs. In this case, the LLM LLI is usually trainable, and our goal can be formulated as:
min J.)) Viu
( (VLL((En: (on 0), yn
4. Demonstration Pre-Selector with Sufficiency and Necessity Metrics
where DEEEDR means that the pre-selected DEEEDER is fixed during fine-tuning.
Notations. Let X, C denote variables for the input and the context (i.e., pre-selected demonstrations). We introduce Y, a boolean variable, to represent whether the correspond- ing output is correct. For simplicity, we use Yz. = 1 to denote Y = 1|x = Fn, meaning that the LLM generates the correct output for the input Tn. Similarly, Yz., = O, equivalent to Y 0|_X x,, indicates that LLM produces an incorrect output for 2n, For convenience, we introduce S, variable to record the original status of the LLM be- fore new plug-in and unplug operations (denoted as plug()) and unplug(-) respectively). The connections between the above operations and the do(-) operation in causality are discussed in Appendix A1.
Algorithm 1 Bi-level Optimization
Input: Training dataset DIRAIX, LLM VLII-
Output: Approximated subset PrEeoen, tuned LLM VIIK-
Initialize DREEDER DIRAIR.
for each iteration do
Update DPEEDER by using our approximation algorithm with frozen L.M VILM-
Tune LLM MLLM by using Eq. (3) our loss function on fixed DPEEDeR.
Given the above analysis, we can further bridge the (pre)- selection of DPEEDER and the LLM fine-tuning on DFEEDER into a bi-level optimization framework. On the outer level, following Eq. (2), we optimize the selection of DREEDER in the context of a frozen LLM Liixi while on the inner level, following Eq. (3), we optimize the LLM VLLM using the fixed dataset DEEEDER- The bi-le optimization procedure described above is amenable to repetition, enabling iterative refinement of both the selected PFEEDER and the tuned LLM. The overall process is summarized in Algorithm 1, and the construction of our FEEDER subset is detailed in the subsequent sections.
We begin by considering the relationship between two demonstrations, denoted as (#n, Un) and (æmm ym), and the sufficiency and necessity metrics based on particular L.MSs as follows.
The sufficiency metric is introduced to assess whether plug- ging in one data point is adequate for the LLM to produce the correct answer to another data point. Formally, we define sufficiency between pair of demonstrations as:
Definition (Sufficiency Metric). Given tuple (X,YC,,)).
3

---

Large Language Models are Demonstration Pre-Sclectors for Themselves
a training sample nn Dn) is considered sufficient for an- other one (m-m Dun), if the following equation holds:
An Example of Approximation Algorithm for FEEDER
W/ U W3 U W4
Yam = [lplug((ann yn)); C,S = =(Y=m =0); (4)
Check Whether WIN and WOUT have a Sufficiency Relationship, Remove Unnecessary Parts
where (In, yn) is not included in C. It means that when plugging in (In; yo), it would correct the LLM's answer to mm-
W
W3 U W4
The necessity metric is introduced to assess whether it is necessary to retain particular plugged-in data point to maintain the correct output of another data point. Its formal definition over pairs of demonstrations can be written as:
Check Whether WIN and WOUT have a Sufficiency Relationship, Remove Unnecessary Parts
Definition 2 (Necessary Metric), Given tuple (X, S, a training sample (æn, yn) is considered necessary for (Im: ymm) if the following equation holds:
W
W
W
W4
Figure 2. An illustrated example of our approximation algorithm for FEEDER. At each round (corresponding to each layer of the tree), we check whether there is sufficiency relationship hetween each pair of nodes. After each check, we remove those unnecessary parts from W.
Y Y=m = Olunplug((Zn: ya)):: C,S =Yam 1). (5)
where (In: yn) is included in C. It means that prior to unplugging (mn; ÿn), the LLM's 's output is correct. However, when we do unplug (In, yo) from the context, it causes the LLM to offer an incorrect output.
Specifically, we use k = 1,,,..., K to refer to each k-th round; and during each k-thh round, we generate the (I + 1)- th layer of the tree. We denote We as the set of nodes after the k-th round. We initialize W by assigning all the samples in DTRAII as the bottom nodes:
The above definitions of sufficiency and necessity metrics, operating on the instance level, are further clarified with examples in Appendix A2.1.
Wo = (W,, = ((z.. yn (Zn, yn)) E DrRAm
(6)
Extending these definitions to the set level, a sufficient set signifies that plugging in specific set is adequate to ensure the correct outputs for all examples in another set, while a necessary set implies that removing any example from this set would result in incorrect answers for at least one example within another set. Formal definitions for the above set-level metrics, along with examples, are available in Ap- pendix A2.2.
During each k-th round, we generate Wi from Wk-1- This achieved by examining the sufficiency relationship between every pair of nodes in W denoted as W,, W3 E Wh-1- In this evaluation, we assess whether the following equation holds true by assigning W. and W2 as WIx and Wour. or vice versa:
Y(I = / |pwgg///
(7)
Taking into account both the sufficiency and necessity met- rics, we define a subset of the training dataset Draam as FEEDER subset DPEEDER, if can be both sufficient and nec- essary to represent DTRAIN- We provide its formal definition along with some examples in Appendix A2.3. Strictly fol- lowing the above definition discover FEEDER subset is impractical because the constraints too stringent and the computational costs are prohibitively high with ((2*) computational complexity to enumerate all the possible sub- sets. Therefore, we propose an approximation algorithm for discovering a FEEDER subset with a tree- based algorithm.
where C and S is loosened to allow for any value. the above equation holds, signifies that plugging in WIn sufficient for the LLM generate the correct output to any input in Worr In other words, once we have WIn included in the plugged-in context, is unnecessary to further include WoUT. Formally, we can derive the following equation that is equivalent to Eq. (7) as:
Y([=.la. = / (unpl/unuluurrr) C, S,
(8)
where = (Win Wour) and 5 is loosened to be any value.
Our bree-based approach leverages the capability of the LLM to assess the sufficiency and necessity for each sample in DIRAIN, which can filter out unnecessary portions of D DTRAIK. while retaining a sufficient subset that effectively represents the entire train data DIRAIN. Concretely, our tree expands from the bottom to the top, where each node represents a set of instances. Formally, we use the variable K to represent the depth of the tree, corresponding to the number of rounds.
Concretely, there are three possible scenarios by examining each pair of nodes in Wh-1 using Eq. (7): (i) If both W, and Wj are sufficient for representing each other, then we select the one with fewer elements to append to Wr. (ii) If only one of W; and W3 is sufficient to represent the other, then we append the sufficient one to Wt. (iii) If neither W; nor W, is sufficient to represent the other, we append W; U W,
4

---

Large Language Models are Demonstration Pre-Sclectors for Themselves
to W. After performing the above calculations for each pair of nodes, we remove them from Wk-1- When there is only one element left in #k- it is directly appended to Wi. This process continues until W. contains only one element.
2013), COLA (Warstadt et al., 2018), TREC (Voorhees & Tice, 2000), SUBJ (Pang & Lee, 2004), and FPB (Malo et al., 2014). These datasets cover range of tasks from sentiment classification and linguistic analysis to textual entailment. We also further assess FEEDER on the reasoning dataset GSM8K (Cobbe et al., 2021), the semantic-parsing parsing dataset SMCALFlow (Andreas et al., 2020), and the scien- tific question guestion-answering dataset GPQA (Rein et al., 2024). For each dataset, we directly follow the official splits to obtain DTRAIN and DTEST.
Then, we can effectively remove unnecessary samples from DIrAIN by extending the tree structure above from the bot- tom to the top. Moreover, the complexity of the above algorithm with K rounds (corresponding to a tree depth of In practice, we investigate the impact of varying K and find that setting K already produces excellent performance. This indicates that a one- shot inference by LLM is sufficient to assess the sufficiency between each pair of samples. Once the results of pairs of samples are computed, we take their union to form the resulting subset DPEEDER. Figure 2 illustrates the process for K = 2. When K = 1, the top-level check between Wi and UgW is no longer required. Instead, the resulting subset W1 U W3 UW, after the bottom-level check.
K ++D 1) is O(K log2 P.ais))) logg)
To evaluate the performance of our approach, we employed two GPT-2 variants (Radford et al., 2019): one with 335M parameters, and the other with 774M parameters; one GPT- neo with 1.3B parameters; one GPT-3 variant (Brown et al., 2020) with 6B parameters; one Gemma-2 variant (Team et al., 2024) with 2B parameters, one Llama-2 variant (Tou- vron et al., 2023) with 7B parameters, Llama-3 variant (Meta, 2024) with 8B parameters, and Qwen-2.5 variant (Yang et al., 2024) with 32B parameters, as the LLM base.
Moreover, our tree-based algorithm can be iterated across multiple runs to further refine the necessary components. Specifically, the FEEDER subset obtained from one run serves as the input for the next. We denote the num- ber of runs as R, leading to an overall complexity of O(RK log,2 P Empirical investigations show that vary- ing R has minimal impact, with a single run (R = 1) already achieving strong performance. Thus, our implementation complexity simplifies to (logg [Daaul).
5.1. Performance on In-context Learning
Since our DPEEDER works as pre-selector, when applied in the in-context learning setting, we propose incorporating demonstration selectors into FEEDER. In other words, our evaluations follow an ablative approach, with the baseline involving the direct application of these demonstration se- lectors on DTRAIN- This baseline can be regarded treating these methods both pre-selectors and demonstration se- lectors. As discussed in Section 4, our DPEEDER is applied using only a one-shot inference check (i.e., K = 1) and a single- round run (i.e., R = 1), unless otherwise stated.
Our tree-based approximation algorithm is summarized in Algorithm 2 in Appendix A3. If we hypothesize that suffi- ciency is transitive among sets, our tree-based approxima- tion algorithm can ensure that the remaining subset remains sufficient to represent the entire DIRAI,, as verified in the following proposition.
Concretely, we conducted an evaluation of FEEDER in con- junction with following six demonstration selectors: (i) Ran- dom is the random selector, which selects input demon- stration randomly from the selection pool; (ii) Similarity (Sorensen et al., 2022; Gonen et al., 2022) is the similarity- based selector, which selects relevant demonstrations in terms of the cosine similarity metric over the embedding vectors generated by a sentence transformer (Reimers & Gurevych, 2019); (iii) Diversity (Ye al., 2022) is the diversity- based selector, which selects similar and diverse demonstrations in terms of maximal marginal relevance (Carbonell & Goldstein, 1998); (iv) Uncertainty (Köksal et al., 2022) is the uncertainty-basee based selector that conducts selections according to their uncertainty metric; (v) Clus- tering (Zhou et al., 2023) is the clustering-based selector that searches demonstrations by clustering. (vi) Latent (Wang et al., 2024) uses LLMs as latent variable models to learn latent variables for down-streaming in-context learn- ing. Please refer to Appendix A5.1 for detailed descriptions of the above demonstration selectors
Proposition If we successively apply our tree-based ap- proximation algorithm on DTRAT for certain number of runs obtain subset (denoted as DEEEEER), then DFEEDER is sufficient to represent DIRAIN-
We offer the proof of the above proposition in Appendix A3, which demonstrates that our tree-based algorithm can ef- fectively remove unnecessary samples from DIraIN while ensuring that the resulting subset remains sufficient to rep- resent the entire training dataset.
Additionally, we present another algorithm for finding an ex- act sufficient and necessary subset from DTRAIX, along with its proof and deployment discussion, in Appendices 14. Appendix A4.2, and A7,
5. Experiments
Our evaluations are mainly conducted on 6 text classification datasets: SST-2 (Socher et al., 2013), SST-5 (Socher et al.,
Experimental results regarding in-context learning perfor-
5

---

Large Language Models are Demonstration Pre-Sclectors for Themselves
Table Performance comparisons on text classification datasets are conducted in the in-context learning setting. We report both the mean and variance of accuracy using 8 different seeds and 5 different permutations of n-shots. Refer to Appendix A5,2 for more extended results on LLMs GPT-2 variants and GPT-3 variants, datasets FPB, SST-5, TREC, and demonstration selectors Uncertainty, Clustering and Latent.
SUBJ
SST-2
COLA
Vzzx())
D
Random
Similarity
Diversity
Random
Similarity
Diversity
Random
Similarity
Diversity
45.0 (5.9)
48.1 (0.0)
48.1 (0.)
51.2 (68)
52.2 (0.))
52.2 (0:5)
37.5 (700
40.5 (1.)
40.5 (..)
2
62.3 (6:9)
82.5 (1.)
74.2 (1.3)
71.5 (5.0)
78.5 (1.)
75.9 (.9)
40.6 159)
62.5 (1.0)
61.6 (0:5)
Prums
68.0 (7.1)
91.5 (1.2)
84.2 (1.0)
70.2 (5.0)
80.5 (1.))
80.6 (0.))
46.5 (59)
67.2 (1.)
65.6 (0.6)
10
50.3 (8.2)
86.2 (1.9)
85.6 (0:00
68.2 (4.0)
85,5 (1.5)
76.3 (.3)
50,2 (7.4)
69.8 (1.9
71.5 (1.2)
Gemma-2 (2B)
48.2 (4.2)
49.5 (1.0)
49.5 (1.m
52.6 (4.6)
53.1 (0.5)
53.1 (0.5)
38.9 (5.2)
39.6 (0x)
39.6 (0.8)
65.2 (2.9)
85.2 (1.0)
80.3 (0.8)
74.2 (4.9)
82.1 (1:2)
83.0 (0.7)
52.5 (2.5)
68.9 (2.1)
67.8 (1.5)
DEEEEE
72.2 (6-))
94.5 (5.3)
85.5 (0:7)
72.0 (4:2)
83.6 (2.1)
84.5 (1.7)
55,2 (4.8)
77.6 (2.5)
73.9 (2.9)
10
60.5 (40)
86.5 (2.5)
88.4 (2.4)
70.5 (5.6)
92.6 (2:6)
78.5 (5.3)
58.6 (4:6)
75.6 (2.9)
76.6 (2.5)
44.9 16.00
49.5 (0.1)
49.5 (0.1)
48.2 (2.9)
47.0 (..)
47.0 (0D
38.9 (6.7)
41.2 (0.2)
41.2 (0.21
2
55.4 (3.5)
85.5 (0.0)
86.5 (0.2)
68.1 1 (4.2)
78.7 (0.2)
77.5 (.D
42.8 (400)
45.5 (0.5)
45.6 (0.21
51.2 (4.4)
90.8 (0.2)
82.7 (0.1)
75.2 (3.3)
80.7 (0.3)
77.8 (0.2)
48.5 (3.3)
51.8 (0.3)
52.1 (0.2)
10
$7.7 (4.8)
87.3 (0.0
85,3 (0.1)
72.1 (3.8)
77.6 (0.3)
76.5 (0.2)
59.1 (4.2)
60.3 (0.D
61.0 (0.2)
GPT-3 (6B)
43.9 (4.2)
51.2 (1.m
51.2 (1.m
49.6 (2.4)
51.3 (1.6)
51.3 (1.6)
41.2 (2.1)
43.8 (1.8)
43.8 (1.8)
65.7 (3.0)
91.5 (1.))
88.8 (1.6)
73.5 (2.5)
85.7
76.1 (2.1)
61.8 (2.1)
63.1
60.1 (1.4)
Praees
53.7 (3:5)
92.9 (0.8)
91.5 (1:4)
77.6 (4.00
81.0 (1:9)
79.4 (1.0)
50.6 (2.7)
63.33 (1.9)
65.8 (1.4)
10
58.0 (3:4)
88.8 c0.9)
87.8 (1:7)
83.8 (2.8)
86.4 (200)
87.2 (1:3)
59.7 (900
67.5 (1.9)
68.4 (1.2)
42.9 (6.0)
48.5 (0.1)
48.5 (0.1)
46.2 (2.7)
49.1 (0.1)
49.1 (0.1)
40.1 (6.1)
42.0 (0.2)
42.0 (0:2)
51.9 (4.4)
90.7 (0.0)
85.2 (0.2)
67.8 (3.2)
73.5 (0.2)
74.5 (0.2)
43.5 (4.5)
47.4 (0.2)
49.6 (0.1)
Pram
51.6 (3.2)
86.8 (0.2)
82.9 (0.1)
74.8 (3.8)
81.2 (0.2)
78.7 (0.2)
50.2 (3:7)
52.6 (0.2)
48.2 (0.3)
10
56.1 (4.6)
81.3 (0.0)
85.7 (0.1)
73.2 (3.1)
76.3 (0.3)
77.1 m.))
59.6 (4.3)
55.3 (0.2)
60.0 (0.4)
Llama-2 (BB)
1
43.8 (4.3)
49.7 (1.
49.7 (1.0
47.2 (2.4)
50.8 (1.7)
50.8 (1.7)
41.2 (2.1)
43.8 (1.8)
43.8 (1.8)
2
54.8 (3.0)
92.5 (1.1)
84.8 (0:7)
72.2 (3.))
82,5 (4.0)
80.1 (2.6)
50.8 (2.3)
58.6 (1.7)
53.5 (1.3)
DrEEEE
5
53.7 (3.8)
87.9 (1.8)
91.5 (1.4)
78.3 (4.6)
83.2 (1.1)
80.1 (1.4)
53.8 (2.8)
65.3 (1.6)
61.8 (1.4)
10
58.0 (9:4)
85.8 (0.9)
87.8 (1.2)
85.0 (2.2)
87.1 (2:7)
86.9 (1.00
60.5 (3.1)
68.0 (1.7)
68.4 (2.0)
Table Performance comparisons on reasoning GSMSK dataset and semantic-parsing SMCALFlow dataset are conducted in the in- context learning setting. We report both the mean and variance of accuracy using 8 different seeds and 5 different permutations of n-shots. Refer to Appendix A5.3 for more extended results on demonstration selectors Clustering and Latent.
GSMSK
SMCALFlow Flow
Wizw(-)
D
Random
Similarity
Diversity
Uncertainty
Random
Similarity
Diversity
Uncertainty
6.54 (1.50)
15.16 (.17)
15.16 (.17)
10.51 (0.7)
8.54 (1.64)
19.12 (0.15)
19.12 (..5)
11.21
8.56 (085)
18.89 (ss)
19.52 (A45)
17.58 (.27)
9.56 (0:4)
20.05 (0.36
22.50
13.58 (0:7))
DTRATK
15.30
20.31 (058)
21.56
19.30 (.900
18.56 (4.58)
28.65 (0.05)
27.89 (1.85
25.22 (9.56)
10
17.45 (4.21)
21.52 (149)
20.85 (0.5)
20.66 (1.84)
19.85 (5.31)
30.58
28.56
31.00 (0:38)
Gemma-2 (28)
10.25 (0.1))
16.25 (021))
16.25 (0:21)
11.12 (1.78)
9.64 (0.55)
20.54 (0.00)
20.54 (0.06)
15.25 (0:87)
13.76 (0.8)
19.68 (013))
20.51 (1:55)
16.85 (3,05)
10.25 (0.52)
23.73 (0.18)
24.25 (2.05)
17.58 (6.38)
D
18.52 (5.21)
22.58 (0.55
22.05 (0.77)
20.20 (2.05)
20.44 (5.12)
30.54 (4.58)
32.54 (5:21)
28.95 (3:06)
10
19.20 (5.22)
22.20 (1.45)
23.52 (2:20)
22.10 (6.21)
21.52 (2.01)
31.48 (1.52)
31.02 (2.54)
30.01 (1.20)
2.45 (083)
3.52 (0.98
3.52 (0.8)
3.05 (0.25)
2.25 (0.64)
10.25 (0:85)
10.25 (0:5))
9.01 (0.3))
2.65 (0.77)
4.97 (0.18)
5.62 (0.85)
4.12 (0.47)
4.97 (0.4)
10.05 (2.36)
10.52 (1:45)
11.20 (1.59)
D DTRAT
3.54 (0.85)
8.25 (0.59)
7.25 (0.9)
7.88 (0.64)
7.52 (0R5)
16.20 (1.85)
15.28 (1.75)
15.33 (1.30)
10
4.25 (0.36)
8.85 (0.55)
9.21 (1.95)
8.10 (1.11)
8,70 (105)
18.95 (1.25)
19.55 (2.01)
17.52 (2.66)
IJama-2 (78)
3,54 (0:51)
4.44
4.44 (0.8%)
3,36 (0.66)
3.64 (0:55)
10.89 (0.62)
10.89 (0.63)
10,02
3,76 (0:48)
5.68 (0.13)
6.66 (0.58)
4.85 (0.85)
4.25 (0:52)
12.03 (0.16)
11.13 (1.10
12.50 (2.01)
DYEEDER
4.20 (1.23)
9.22
8.81 (0.95
8.20 (1.14)
8.25
17.20 (3,66)
16.66
16.06 (2.22)
10
5.02 (1:51)
10.22 (1.53)
9.25 (0,79)
9.45 (0.66)
9.20
20.11
21.25 (1:36)
20.22 (4.02)
78.24 (6.50)
79.56 (3.42)
79.56 (3:42)
78.42 (3:00)
12.37 (6.06)
15.64 (2.54)
15.64 (2.34)
14.35 (4.66)
79.55 (..9)
83.40 (4.53)
83.67 (4.05)
81.23 (3.53)
13.21 (4.34)
16.74 (3.45)
17.43 (3,65)
16.60 (4.62)
Denax
81.45 (5.43)
$3.47 (5.63)
84.52 (4.76)
82.34 (5.34)
14.53 (5.23)
16.54 (2.35)
17.87 (1:35)
16.52 (3:11)
10
82.31 (6:34)
84.42 (3.24)
84.53 (4.45)
84.12 (4.44)
14.63 (4.33)
16.50 (2:21)
18.64 (2.34)
17.87 (2:23)
Llama-3 (SB)
80.23 (4.43)
81.21 (3.45)
81.21 (3455)
79.64 (2.34)
13.56 (3.22)
16.55 (2.31)
16.55 (2:11)
15.40 (2.44)
82.13 (4.76)
84.43 (3.23)
83.88 (3.33)
82.22 (3.43)
14.03 (3.25)
17.45 (3.64)
17.77 (3:20)
17.00 (4.57)
PEEDER
82.55 (5.96)
85.03 (3.66)
84.77 (3.77)
83,56 (3.76)
14.58 (3.45)
18.22 (2.78)
18.12 (2.01)
17.53 (2.55)
10
84.56 (2.33)
85.79 (3.55)
85.44 (4.55)
84.98 (4.76)
14.99 (4.65)
16.66 (2.33)
18.78 (3.42)
18.01 (2.44)
6

---

Large Language Models are Demonstration Pre-Sclectors for Themselves
Table 3. Performance comparisons on text classification datasets are conducted in our bi-level optimization (i.c., fine-tuning) setting.
where we tune the LLMs and evaluate their few-shot inference performance. We report both the mean and variance of accuracy using 8 different seeds and 5 different permutations of n-shots. Refer to Appendix A8.2 for more extended results on datasets FPB, SST-5, TREC.
SUBJ
SST-2
COLA
(
D
Random
Similarity
Diversity
Random
Similarity
Diversity
Random
Similarity
Diversity
1
67.8 (1.2)
83.7 (..
83.7 (..
61.3 (u1)
71.6
71.6 (0.
59.3 (5.
69.4
69.4 (02)
2
69.1 (4.3)
88.7 (0.2)
86.9 (02)
73.5 (32)
75.8 (0.))
74.2 (0.3)
64.1 (m7)
74.1 (0.)
74.0 (0:3)
D TTRATE
5
70.8 (5.1)
73.3 (..)
72.7 (02)
74.6 (411)
82.8 (0.))
75.3 (0.))
60.9 (4.9)
76.7 (0.3)
76.4 (0.3)
10
89.2 (4.1)
94.0 (0.2)
91.6 (0.2)
70.8 (2.)
84.5 (0.2)
77.4 (0.2)
70.7 (3.9)
75.7 (0.3)
77.6 (0.5)
GPT-2 (0:8B)
1
93.0 (4.3)
93.5 (1.3)
93.5 (1.5)
89.5 (4.3)
88.4 (1.6)
88.4 (1.6)
81.5 (3.3)
82.6
82.6 (1.4)
2
96.1 (3.8)
94.1 (1.3)
92.6 (1.2)
92.6 (2:9)
94.4 (0.5)
93.8 (0.7)
90.2 (3.5)
91.2 (1.7)
90.8 (0.9)
PrEEes
5
85.7 (3.5)
94.7 (1.5)
94.1 (1.1)
87.5 (4.1)
92.5 (1.7)
93.7 (.7)
87.7 (3.2)
(2.7
90.0 (3.9)
10
90.5 (3.9)
95.5 (1.3)
95.6 (1:4)
91.9 (2.9)
93.1 (2.1
89.0 (1:4)
91.3 (3.5)
92.4 (1.8)
93.5 (1.9)
1
72.7 (5.23
91.0 0..
91.0 (0.1)
65.4 (4.40
72.5 (0.2)
72.5 (0.2)
61.8 (5.2)
68.55 (0.2)
68.5 (0:2)
2
74.1 (4:3)
93.7 (0.2)
92.1 (0.3)
74.5 (3:2)
75.8 (0.9)
76.4 (0:5)
70.8 (5:7)
63.9 (0.2)
64.3 (0.4)
Drnax
5
71.8 (5.50
74.8 (0.3)
75.8 (0.4)
73.6 (4.1)
77.8 (0.3)
76.3 (0.2)
68.7 (4.7)
75,4 (0.8
74.9 (0.4)
10
90.2 (4.00
93.6 (0.4)
92.5 (0.4)
72.8 (2.9)
(0.2
78.8 (0.2)
72.7 (3.4)
76.7
77.5 (0.7)
GPT-neo (1.38)
1
93.5 (4.3)
94.1 (1.4)
94.1 (1.4)
91.2 (3.8)
92.7 (1.5)
92.7 (1.5)
86.8 (3.3)
89.6 (0.9)
89.6 (0.9)
2
95.5 (3.9)
95.1 (1.3)
96.6 (1.3)
88.6 (2.4)
93.4 (0.6)
94.2 (0.5)
84.2 (3.7)
87.3 (07)
89.5 (0.9)
Dreeder
91.5 (3.8)
95.7 (1.0)
95.3 (1:4)
89.4 (2.)
92.5 (1.8)
93.7 (1.9)
89.7 (3:2)
92.4 (2.3)
90.8 (1.8)
10
92.8 (3.1)
96.0 (1:4)
94.8 (1:3)
90.9 (2.0)
93.6
92.2 (1.8)
89.3 (3.9)
93.5 (1.7)
94.4 (1.6)
mance are reported in Tables 1, 2. and 4. We also present
5.2. Performance on Bi-level Optimization
the reduction of our FEEDER in Figure 3. Our findings are summarized as follows.
Here, we would like to verify whether our FEEDER can be beneficial to the fine-tuning setting. As formulated in Sec- tion 2, our pre-selection and the LLM fine-tuning can be in- tegrated into a bi-level optimization framework. Specifically, in our evaluation, we assess the performance of FEEDER by initially fine-tuning the LLM on the pre-selected DPEEDER. Subsequently, we use the tuned LLM to generate new 'D DreZEer. and evaluate the LLM within the in-context learn- ing setting, using the new DPEEDER as the selection pool. For comparison, our baseline is to initially fine-tune the LLM with DIRAIN and then evaluate the LLM within ICL., using Drra as the selection pool.
Our FEEDER working as a pre-selector can improve the performance of diverse demonstration selectors. Particu- larly, we can observe that in many cases, our FEEDER can incorporate with the similarity- -based demonstration se- lector (i.e., DFEEDES+ -Smmilrrty)))))) can achieve comparable or even better performance than diversity or clustering- based demonstration selectors (i.e., PrRairtDiversity) By combining the results from Table I and Figure 3, it is evident that FEEDER enables the retention of almost half of the training samples while consistently achieving superior or comparable performance, which provides evi- dence supporting the efficacy of FEEDER as a proficient data pre-selection method for in-context learning.
The corresponding experimental results are reported in Ta- ble 3. FEEDER can improve the LLM fine-tuning perfor- mance within our bi-level framework. This emphasizes the potential for achieving enhanced performance by utilizing a small yet high-quality dataset for fine-tuning, while simulta- neously reducing computational expenses, which aligns with the core-set selection literature (Feldman, 2020; Guo et al., 2022). By combining the results from Table and Table 3, we can see that fine-tuning LLMs provides greater perfor- mance improvements compared to augmenting LLMs with contexts. Furthermore, our FEEDER achieves even better performance gains in the tieettnnig setting, One potential explanation is that in this scenario, fine-tuning can leverage input demonstrations more effectively than prompting can, and our high qugl-tualtty FEEDER can therefore provide greater benefits.
We also evaluate the few-shot performance on more com- plex tasks using LLMs Gemma-2, with the corresponding results reported in Table 2. The table demonstrates that, even though LLMs may not perform well on these tasks, our FEEDER can consistently enhance their performance. FEEDER performs well with large number of shots. In Table 1, we can observe many cases where the I.MM per- formance drops when the number of shots increases from to 10 (e.g., Llama-2 on the COLA dataset). This may be caused by the introduction of noisy and redundant shots. Our FEEDER addresses this issue by evaluating the sufficiency and necessity of each demonstration. To further verify this claim, in Appendix A9.3, we duplicate the training dataset and evaluate GPT-neo's performance. Our results show that FEEDER minimizes the negative im- pact on the LLM, supporting its effectiveness in managing demonstration quality.
We also investigate the performance of FEEDER with vary- ing the number of runs (i.e., R) and different tree depths (i.e., the number of rounds K) in Appendix A9.2. FEEDER's
7

---

Large Language Models are Demonstration Pre-Sclectors for Themselves
Table 4. Performance comparisons on reasoning GSM8K dataset and scientific question-answerd GPQA datasct are conducted in the in contont learning setting. We report both the mean and variance of accuracy using 8 different seeds and different permutations of n-shots.
GSM8K
GPQA
V-u(())
D
13
Random
Similarity
Diversity
Uncertainty
Random
Similarity
Diversity
Uncertainty
81.01 (4.51)
82.14 (018)
82.14 (0:15)
81.02 (5.24)
40.10 (1.64)
42.04 (0.19)
42.04 (0.19)
42.00 (5.04
83.25 (5.03)
84.19 (0.6))
84.68 (0:55)
84.01 (4.2)
42,06 (3.6))
44.52 (0.56)
45.23 (0.41)
42.85 (.77
DIRAI
5
85.20 (3:52)
89.52 (0.58)
89.61 (0.54)
86.02 (3.00)
43.33 (5.88)
46.20 (1.05)
46.81 (1.85)
46.70 (5:66)
86.21 (4.28)
90.41 (0.63)
89.92 (0.55)
90.20 (3.84)
43.23 (5.21)
46.85 (1.04)
46.71 (0.98)
45.28 (0.59)
Qwen-2.5 (32))
1
81.85 (2.08)
83.52 (0.13)
83,52 (0.13)
81.91 (3.78)
40,51 (5.95)
42.71 (0.06)
42.71 (0.66)
41.22 (6.57)
2
84,52 (5.48)
85,71 (0.33)
86,03 (0.56)
84,80 (5.65)
43,50 (0.52)
45.81 (0.58)
45.66 (3,65)
43.20 (6.59)
Denroon
5
86.70 (7.18)
90.20 (0.96)
90.12 (0.35)
88.82 (4.38)
44.94 (4.12)
48.03 (5.58)
48.08 (6.21)
48.95 (4.66)
10
87.58 (7.22)
91.23 (3.45)
90.71 (4.24)
91.96 (7.71)
44.55 (2.11)
47.80 (4.52)
47.93 (3.54)
47.91 (6.26)
[6] CLLA-LAG+1
(b) COLA+LAG+2
[c] COLA1LAG15
(d) COLA 10
0.45
8000
0.50
8000
8000
0.70
7000
7000
7000
7000-
6000
5000
0.42
5000
5000
0.00
4000
0.46
4000
0.42
4000
4000
0.55
3000
0.44
3000
3000
0.67
3000
2000
2000
2000
0.42
0.39
0.86
0.45
0
0.38
Number Rounds
Number Rounds
Number Rounds
Number of Rounds
(e) TREC-NEO+1
TREC+NEO+2
(c) TREC+
T
0.000
0.46
5000
3000
0.288
5000
0.50
5000
0.45
-0.88
4000
4002
0.26
4000
0.05
4000-
0.16
00.4
0.24
5002
0.34
0.22
-043
2000
2002
2000
2U00
0.30
1000
2000
1000
1000
0.30
0.18
0:25
0.41
0
10
5
10
10
10
Number Rounds
Number Rounds
Number Rounds
Number Rounds
Figure 3. Performance comparisons for running our approximation algorithm to pre-select FEEDER with different runs R are evaluated in terms of accuracy (denoted as ACC) with Random as the demonstration selector and the size of the resulting FEEDER subset (denoted as Size). Each sub-figure is entitled with Dataset+LLM base+n shots.
performance first rises and then drops with increasing or K. These findings further verify that identifying represen- tative subset from the training dataset- either by increasing the number of runs or the number of rounds- can signif- icantly enhance the performance of the LLM. However, overly narrow subsets may limit the potential performance gains. This trend may be summarized as trade-off between data quantity and data quality, and similar observations are reported in Chen et al. (2023).
LLM in use. We consider the factual error made by Google Bard in the first demo". We further prompt gpt-3.5-turbo to generate 5 sufficient and necessary statements for the fact. We evaluate separately using these statements as a prompt to gpt-3.5-turbo and find that either one of the generated statements is sufficient and necessary to answer the question "What took the very first pictures of a planet outside of our own solar system?" We then evaluate the performance of gpt-j-6b with the above 5 statements, and find that only the 1-st the 5-th statement is sufficient and necessary instance to answer the above question. Combining the results of gpt-j-6b and gpt-3.5- 5.5-rbbo verifies one of the core insights of our paper: the evaluation of prompting
Furthermore, we analyze scaling up FEEDER into larger LLMs and real-world datasets in Appendix A6.
5.3. Case Study
Subsequently, conduct case study substantiate the central proposition of this paper: whether the assessment of the quality of demonstrations should depend on the specific
235 g100 error-exoplanet--dem
8

---

Large Language Models are Demonstration Pre-Sclectors for Themselves
demonstration should consider the specific LLM in use. Refer to the detailed description in Appendix A10.2.
estingly, Figure A6 reveals similar but more robust trend compared to Figure 3. This robustness could be attributed to the inherent strength of the two-shot inference process for sufficiency, as defined in Eq. (7). The two-shot inference introduces a more rigorous evaluation mechanism than the one-shot inference check, enabling a stronger filtering of unnecessary samples.
5.4. Time Complexity
Here, we provide some empirical evidence of the time com- plexity of the proposed method. As summarized in Algo- rithm 2 in Appendix A3 and discussed in Section 4, there are two key hyperparameter settings for reducing the time cost of Algorithm 2: the number of iterations (i.e., K) and the number of rounds (i.c.. H). In our main experiment, we set K = 1 and Rt = 1, meaning that we perform only one-shot inference for sufficiency checks in each round of Algorithm 2 and execute the algorithm for single round. We investigate the performance differences arising from varying K and R in Appendix A9.2 and Section 5.2 re- spectively. Additionally, we report the time complexity associated with different values of K and R on COLA and TREC datasets in Figure 4. From the figure, we observe that as the number of samples decreases, the time consump- tion of Algorithm 2 also decreases, Furthermore, we note that increasing the number of rounds has great impact reducing the time complexity. This may be attributed to the fact that two-shot inference for sulficiency-satisfyiin Eq. (7)-is significantly more challenging than a one-shot inference check. By further combining Figure and Fig- ure in Section 5.1, we observe that the time consumption nearly linear with respect to the size of the data samples.
In practice, we deploy our approximation algorithm with K 1 and R 1, which provides optimal trade-off between computational efficiency and model performance. This configuration ensures that the pre-selection process remains practical while maintaining competitive accuracy.
6. Conclusion and Future Work
In this paper, we present novel demonstration pre-selector FEEDER, designed to leverage LLM's capabilities and do- main knowledge to identify high-quality demonstration and provide an approximate approach for their discovery. Our experimental results showcase the significant advantages of FEEDER across diverse LLM bases in both ICL and bi-level optimization for fine-tuning LLMs. In the future, it would be valuable to explore the use of larger LLMs and extend the applications of FEEDER to areas such as data safety and data management.
A Acknowledgment
Searching FEEDER with our Approximation Algorithm
The Shanghai Jiao Tong University team is partially sup- ported by National Key R&D Program of China (No. 20227D01 14804), Shanghai Municipal Science and Tech- nology Major Project (No. 2021SHZDZX0102) and Na- tional Natural Science Foundation of China (No. 62322603, No. 62177033). This work is also supported in part by Na- tional Natural Science Foundation of China (No. 623B2002) and Beijing Natural Science Foundation (1.257007).
35
30
Time (second)
25
20
15
10
5
Impact Statement
0
COLA
TREC
The objective of this paper is to develop pre-selection method over the training data as an intermediary process to enhance the accuracy of factual knowledge in the model's outputs. It is essential to note that our FEEDER, pre-selected from the training dataset without external trustworthy cor- pora, relies on the capability of the given LLM itself. This characteristic may potentially amplify existing biases in the model weights of LLMs.
Figure 4. Time complexity of searching FEEDER using our approx- imation algorithm for different runs COLA and TREC datasets using varying the number of rounds R and varying the number of rounds K.
Consider two hyper-parameter settings in our approxima- tion algorithm: the number of rounds R and the number of iterations K, both designed balance performance and computational efficiency. As detailed in Figures 3 and 4, the time complexity of our method scales almost linearly with the number of samples, making these parameters critical for practical applications. While Figure 3 illustrates the per- formance changes across different values of R, Figure A6 in Appendix A5..4 explores the impact of varying K. Inter-
9

---

Large Language Models are Demonstration Pre-Sclectors for Themselves
References
Slonim, N. Active learning for bert: An empirieal study, In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 2020.
Andreas, J., Bufe, J., Burkett, D., Chen, C., Clausman, J., Crawford, J., Crim, K., DeLoach, J., Dorner, L., Eisner, J., et al. Task-oriented dialogue as dataflow synthesis. Trans- actions of the Association for Computational Linguistics, 8:556-571, 2020.
Feldman, D. Introduction to core-sets: an updated survey. arXiv preprint arXiv:2011.093350 2020.
Ben-David, E., Oved, N., and Reichart, R. Pada: A prompt- based autoregressive approach for adaptation to unseen domains. arXiv preprint arXiv:2102. 12206, 3, 2021.
Gal, Y. and Ghahramani, Z. Dropout as a bayesian approx- imation: Representing model uncertainty in deep learn- ing. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Brown, T., Mann, B., Ryder, N., Subbiah, M.. Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33; 1877-1901, 2020.
Gao, L., Chaudhary, A., Srinivasan, K., Hashimoto, K., Ra- man, K., and Bendersky, M. Ambiguity in-context learning with large language models. arXiv preprint arXiv:2309.077 07900, 2023.
Carbonell, J. and Goldstein, J. The of mmr, diversity- based reranking for reordering documents and producing summaries. In Proceedings of the 21 annual interna- tional ACM SIGIR conference on Research and develop- ment in information retrieval, pp. 335-336, 1998.
Gao, T. Prompting: Better ways of using language models for nlp tasks. The Gradient, 2021.
Gissin, D. and Shalev-Shward S. Discriminative active learning. arXiv preprint arXiv:1907.06347, 2019.
Chen, L., Li, S., Yan, J., Wang, H., Gunaratna, K., Yadav, V., Tang, Z., Srinivasan, V., Zhou, T., Huang, H., et al. Alpagasus: Training better alpaca with fewer data, arXiv preprint arXiv:2307.08700 08700870,, 2023.
Gonen, H., Iyer, S., Blevins, T., Smith, N. A., and Zettle- moyer, L. Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv:2212.04037, 2022.
Cheng, F., Li, H., Liu, F., Van Rooij. R., Zhang, K., and Lin, Z. Empowering Ilms with logical reasoning: A com- prehensive survey. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, 2025.
Guo, C.,, Zhao, B., and Bai, Y. Deepcore: A comprehen- sive library for coreset selection in deep learning. In Database and Expert Systems Applications: 33rd Interna- tional Conference, DEXA 2022, Vienna, Austria, August 22-24, 2022, Proceedings, Part I, pp. 181-195, Springer, 2022.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10..11668, 2021.
Gupta, V., Shrivastava, A., Sagar, A., Aghajanyan, A., and Savenkov, Retronlu: Retrieval aug- mented task-oriented semantic parsing. arXiv preprint arXiv:2109 10410, 2021.
Davison, J., Feldman, J., and Rush, A. M. Commonsense knowledge mining from pretrained models. In Proceed- ings of the 2019 conference empirical methods in nat- ural language processing and the 9th international joint conference on natural language processing (EMNLP- IJCNLP), pp. 1173-1178, 2019.
Jiang, 7., Xu, F. F., Araki, J., and Neubig, G. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. arXiv preprint arXiv: 181.0088, 2018.
Köksal, A., Schick, T., and Schütze, H. Meal: Stable and active learning for few-shot prompting, arXiv preprint arXiv:2211.08355 2022.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficien prompt tuning. arXiv preprint arXiv:2104.0880 00.69,,,, 2021.
Dong, Q. L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., and Sui, Z. A survey for in-context learning. arXiv preprint arXiv:2301.002..
2022.
Levy, I., Bogin, B., and Berant, J. Diverse demonstrations improve in-context compositional generalization. arXiv preprint 0rXiv:22/2.0663 2022.
Dor, L. E., Halfon, A., Gera, A., Shnarch, E., Dankin, L., Choshen, L., Danilevsky, M., Aharonov, R., Katz, Y., and
10

---

Large Language Models are Demonstration Pre-Sclectors for Themselves
Lewis, D. D. sequential algorithm for training text clas- sifiers: Corrigendum and additional data. In Acm Sigir Forum, volume 29, pp. 13-19. ACM New York, NY, USA, 1995.
Rein, D., Hou, B. L.,, Stickland, A. C., Petty, J., Pang. R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024, https:
/ //openrevice ew net/ forum? d= 67584b98.
Li, X., Lv, K. Yan, H., Lin, T., Zhu, W., Ni, Y., Xie, G., Wang, X., and Qiu, X. Unified demonstration retriever for in-context learning. arXiv preprint arXiv:2305.04320, 2023.
Reynolds, L. and McDonell, K. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-7, 2021.
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for gpt-3? arXiv preprint Seti:22001068000 2021.
Rubin, O., Herzig, J., and Berant, Learning re- trieve prompts for in-context learning. arXiv preprint arXiv:2/12.0866 2021.
Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.0877 2021.
Sachdeva, N., Coleman, B., Kang, W.-C., Ni, J., Hong, L., Chi, E. H., Caverlee, J., McAuley, J., and Cheng, D. Z. How train data-efficient 1lms. arXiv preprint 2024.
Malo, P. Sinha, A., Korhonen, P., Wallenius, J., and Takala, P. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Infor- mation Science and Technology, 65(4):782 2014.
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L.,, Raja, A., et al. Multitask prompted training enables zero-shot task generalization. arXiv preprini urXiv:2110.088 2021.
Marion, M., Üstün, A., Pozzobon, L., Wang, A., Fadaee, M.. and Hooker, S. When less is more: Investigating data pruning for pretraining lims at scale. arXiv preprint 0rXiv:2309.04564 2023.
Schick, T. and Schütze, H. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprini arXiv:2001.00 07666.. 2020.
Meta, A. Introducing meta Ilama 3: The most capable openly available Ilm to date. Meta AI, 2024.
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. Autoprompt: Eliciting knowledge from lan- guage models with automatically generated prompts. arXiv preprint arXiv:2010. 15980, 2020.
Pang. B. and Lee, L. A sentimental education: Sentiment analysis using subjectivity summarization based on mini- mum cuts. arXiv preprint cx/0409058, 2004.
Pasupat, P., Zhang, Y., and Guu, K. Controllable seman- tic parsing via retrieval augmentation. arXiv preprint arXiv:2/ 10.0855, 2021.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, D., Ng. A. Y., and Potts, Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Pearl, J. Causality: models, reasoning, and inference, 1980.
Pearl, J. Causality, Cambridge university press, 2009.
Sorensen, T., Robinson, J., Rytting, C. M., Shaw, A. G.,
Qin, G. and Eisner, J. Learning how to ask: Query- ing Ims with mixtures of soft prompts. arXiv preprint arXiv:2104.06599, 2021.
Rogers, J., Delorey, A. P., Khalil, M., Fulda, N., and Wingate, D. An information-theereee approach prompt engineering without ground truth labels. arXiv preprint arXiv:2203.I 11344, 2022.
Qiu, L., Shaw, P., Pasupat, P., Shi, T., Herzig, J., Pitler, E., Sha, F., and Toutanova, K. Evaluating the impact of model scale for compositional generalization in semantic parsing. arXiv preprint arXiv:2205. 12253, 2022.
Strickland, E. Andrew ng: Farewell, big data. IEEE Spec- Irum, Mar, 2022.
Team, G., Riviere M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, I.., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv:2408.0011 2024.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 1908. 10084, 2019.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E.,

---

Large Language Models are Demonstration Pre-Sclectors for Themselves
Azhar, F., al. Llama: Open and efficient foundation lan- guage models. arXiv preprint vrXXv:2302.13977, 2023.
Yuan, W., Neubig, G., and Liu, P. Bartscore: Evaluating generated text text generation. Advances in Neural Information Processing Systems, 34:27263-27277, 2021.
Voorhees, E. M. and Tice, D. M. Building question an- swering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference Research and development in information retrieval, pp. 200-207, 2000.
Zemlyanskiy, Y., de Jong, M., Ainslie, J., Pasupat, P., Shaw, P. Qiu, L., Sanghai, S., and Sha, Generate-and-retrie your predictions improve retrieval for semantic parsing. arXiv preprint arXiv:2209. 2022.
Wan, X., Zhou, H., Sun, R., Nakhost, H., Jiang, K., and Ank, S. Ô. From few to many: iepr-impg many-shot reasoners through iterative optimization and generation. arXiv preprint arXiv:2502.003300 2025.
Zhou, C.,, Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Wang, S., Xu, Y., Fang, Y., Liu, Y., Sun, S., Xu, R., Zhu, C.,, and Zeng, M. Training data is more valuable than you think: simple and effective method by retrieving from training data, arXiv preprint wrXiv:2203.087735 2022.
Zhou, H., Wan, X., Vulić, I., and Korhonen, A. Sur- vival of the most influential prompts: Efficient black-box prompt search via clustering and pruning. arXiv preprint arXiv:2310.12774 2023.
Wang, X., Zhu, W., Saxon, M., Steyvers, M., and Wang, W. Y. Large language models are latent variable mod- els: Explaining and finding good demonstrations for in- context learning. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba. J. Large language models are human- level prompt engineers. arXiv preprint arXiv:2211.0191 2022.
Warstadt, A., Singh, A., and Bowman, S. R. Neu- ral network acceptability judgments. arXiv preprint arXiv:1805.124477 2018.
Webson, A. and Pavlick, E. Do prompt-based models really understand the meaning of their prompts? arXiv preprint 17X:022109.01247, 2021.
Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. Less: Selecting influential data for targeted instruction tuning. arXiv preprint S.Li:2202.0433333 2024.
Yang, A., Yang. B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K. Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z.,, Zhang, Z., and Qiu, Z. Qwen2.5 technical report. arXiv preprint 2rXiv:2412.15111 2024.
Yang, C., Wang, X., Lu, Y. Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models as optimizers. arXiv preprint ar Xiv:23309003409, 2023.
Ye, X., Iyer, S., Celikyilmaz, A., Stoyanov, V., Durrett, G., and Pasunuru, R. Complementary explanations for effec- tive in-context learning. arXiv preprint arXiw:2211.13892 2022.
Yosida, K. Functional analysis. Springer Science & Busi-
ness Media, 2012.
12

---

Supplementary Materials: FEEDER
A1. Connections to Existing Approaches
points. For instance, retrievers based on similarity metries may select redundant data points together. To address this limitation, Levy et al. (2022) proposes to consider the di- versity among the data points, to avoid the case where too "similar" data points are selected together. Further, Rubin et al. (2021) trains an LLM as a contrastive scorer as well as demonstration referrer, and Li et al. (2023) advances this framework through unified training across various datasets.
AI.1 Connections to Causality
The concepts of sufficiency and necessity have a broad ap- plication scope, especially in causality (Pearl, 1980; 2009), where sufficiency and necessity are proposed to define the causal relationship between two binary variables. Let X and Y denote a pair of variables. Then, the probability of sufficiency measures the capacity of setting X = true to produce Y = true, while the probability of necessity measures the changing the value of X from X = true to X = false would cause the value of Y changing from Y- true to Y = false.
In this paper, we present a novel perspective, asserting that the quality of demonstrations is contingent on the specific LLM in use. Namely, a high-quality demonstration for one LLM might be deemed low-quality for another. Leveraging this insight, we introduce sufficiency and necessity as new set-level metrics. Our approach offers several advantages: Firstly, sufficiency and necessity measure the quality of data points based on the specific LLM, in contrast to generic similarity and diversity metrics. Secondly, our proposed sufficiency and necessity extend to the set level, enabling the consideration of data points as a cohesive whole. In our framework, "similarity" is akin to "sufficiency" signifying that plugging in data points can enhance LLMM performance, while "diversity" is akin "necessity" suggesting that each data point should play an indispensable role.
In this paper, we adopt the concepts of sufficiency and ne- cessity in the context of demonstration selection, where we investigate whether prompting certain data points is suffi- cient or necessary for the given LLM to generate correct answers for input questions. For this purpose, we introduce the plugging-in operation, denoted as plug(+), to exam- ine sufficiency, and the unplugging operation, denoted as unplug(-), to examine necessity. Both of these operations are analogous to the do operation in causality, denoted as do(-), which indicates that the system operates under the condition that certain variables are controlled by external forces. To be more specific, in our setting, the external force can be explained as follows. We have the choice to either plug in or unplug certain data points, thereby altering what is already plugged into the LLM. Our approach shares similarities with the counterfactual idea in causality, which explores hypothetical scenarios by considering what might happen if certain variables are set with different values. In our case, we investigate the impact of plugged-in data that includes data points differing from the historical (i.e., fac- tual) setting. Notably, a significant distinction between our approach and the counterfactual setting in causality lies in the fact that we do not need to estimate "counterfactual"" situations; instead, we can directly conduct evaluations.
Recent studies (Xia et al., 2024; Marion et al., 2023) fo- cus on mining training examples for fine-tuning on spe- cific tasks, while Wang et al. (2024) extends this idea to in- context learning. Unlike these approaches, which use LLMs to select demonstrations tailored to specific test datasets, our work leverages LLMs as demonstration pre-selectors identifying a core subset of the training data that remains independent of the test datasets, thus eliminating the need for re-computation across different test datasets.
A1.3. Connections to Core-set Selection
Core-set selection (Feldman, 2020; Guo et al., 2022), a long- standing problem in machine learning, focuses on identify- ing a subset of the most informative training samples. Previ- ous research (Dor et al., 2020) has surveyed and evaluated state-of-the-aa approaches for models like BERT (Devlin et al., 2018), encompassing strategies such as random sam- pling, uncertainty-samplinn (using entropy metric) (Lewis, 1995; Gal & Ghahramani, 2016) and diversity sampling (using diversity metric) (Gissin & Shalev-Shward 2019).
A1.2. Connections to Demonstration Selection
In the context of few-shot inference, a central challenge lies in selecting the appropriate training samples as extra input during inference. These samples are often referred to as demonstrations or prompts (Levy et al., 2022; Liu et al., 2021; Dong et al., 2022). The underlying assumption is that the training dataset serves as a support set (Yosida, 2012) for test samples. Previous studies (Wang al., 2022; Ru- bin al., 2021) have demonstrated that introducing similar training samples can enhance the performance of LLMs on test instances. Gao et al. (2023) enhances these approaches by retrieving candidates whose ground label lies in top-2 zero-shot predictions. However, as pointed out in Levy et al. (2022), existing methods often treat each data point in isolation, neglecting the collective impact of multiple data
FEEDER, in contrast to these prior papers mainly using active learning, is designed to select core sets, which can serve as additional input contexts (i.e., in-context learning setting) or be used for fine-tuning LLMs (i.e., fine-tuning setting). FEEDER defines "informative training samples" as those samples that specifically enhance the LLM's performance on given task.
13

---

Supplementary Materials: FEEDER
A1.4. Connections to Prompt Optimization
sufficiency relationship is defined as follows:
Definition A1 (Instance-level Sufficiency Metric), Given tuple (X,Y.CA S, data point (In. yn) is sufficient for (Ima ym), if the following equation holds:
Prompting provides natural way for humans to interact with; and due to its flexibility, prompting has been widely used as a genre method for various natural language pro- cessing tasks (Schick & Schütze, 2020; Brown et al., 2020; Sanh et al., 2021). However, using prompting effectively with LLMs requires careful design, either done manually (Reynolds & McDonell, 2021) or automatically (Gao, 2021: Shin al., 2020), as LLMs do not interpret prompts in the same way humans do (Webson & Pavlick, 2021; Lu al., 2021; Cheng et al., 2025). While numerous successful meth- ods (Liu et al., 2021; Lester et al., 2021; Qin & Eisner, 2021) for prompt tuning rely on optimizing a continuous space through gradient-based techniques, this approach becomes impractical as many powerful LLMs are only accessible through APIs that may not offer gradient access.
Y am = l/plug(( (mn; yn)); C,S,
(9)
where (Ins yn is not included in C and S can be any value. It means that when plugging in (Foe Un), it would correct the LLM's answer to Tm-
Example A1. Let Dm, Dn be Which country does Sherlock Holmes live? and Which city does Sherlock Holmes live? ? Then, after informing the LLM of the correct answer of on (e.g.. yn is Sherlock Holmes lives in London), the LLM can deduce the correct answer of am (e.g.. ym is Sherlock Holmes lives in the United Kingdom). In this case, the LLM is using the city where Sherlock Holmes lives to infer the country in which he lives.
Our FEEDER approach can be seen as discrete pre-search method for prompts, distinct from existing methods for prompt generation (Gao, 2021; Ben-David et al.. 2021), prompt scoring (Davison et al., 2019), and prompt para- phrasing (Jiang et al., 2020; Yuan et al., 2021), which aim to optimize instructions by directly searching the natural language hypothesis space. Instead, our approach leverages the causal dependencies among candidate demonstrations, focusing on searching for the most informative demonstra- tions as prompts, in terms of sufficiency and necessity.
The necessity metric is introduced to assess whether the presence of one plugged-in data point is necessary for pre- serving the correct output in relation to another. Formally, this is expressed as:
Definition A2 (Instance-level Necessity Metric). Given tuple (X, S,,, we say that data point (Inn yn) is nec- essary for (Ims ym)) if the following equation holds:
mm = =0)umpuug((m))) yn)); C,S.
(10)
A2. A Family of Analysis on Sufficiency and Necessity Metrics
where (In: yn) is included in C and S = (Yam = 1). It means that before umplugging (ma, Yn), the L.M's answer to Im is correct. However, when we do unplug (Ia, Vn), it causes the LLM to offer an incorrect output to D.n.
Notations. Let X, C' denote variables for the input and the context (i.e., previously plugged-in demonstrations). We use Y, a boolean variable, to denote whether the output to the input is correct. Concretely, we use Y, = -1 to denote y 1|X = x, meaning that the LLM generates the cor- rect output to the input œ. Similarly, Y. 0, equivalent to Y 0|X = x, indicates that the LLM produces the incorrect output to æ For clarity, we introduce S, variable to record the original status of the LLM before new plug-in and unplug operations (denoted as plug(.) (-) and unplug(-) respectively), e.g., C' = ((x: y)), S = (Ye = 1) means that without plugging-in any new data or unplugging any plugged-in data, the plugged-in data is (ar, y) and the LLM's performance is Y.
Example A2. Consider &mm. Which city does Sherlock Holmes live? and En as What is the detailed address of Sherlock Holmes lives?. Assume the LLM has no prior knowledge about Sherlock Holmes until the introduction of the plugged-in in data (I yn), where ya is 221B Baker Street, London. After plugging in (æn, yn), the LLM is capable of generating the correct output ym. (e.g., Sherlock Holmes lives in London) in response F.m- If we were to unplug (æ., yn), the LLM would provide an incorrect output for Im. such as Sherlock Holmes lives in New York.
In an ideal scenario, ensuring optimal LLM performance entails the extraction of data points that are both sufficient and necessary.
1.. Instance Level Metries
Definition A3 (Instance-level Sufficiency and Necessity Metric). Given tuple (X,Y.)), we say that data point (æn, yn) is both sufficient and necessary for (Im, Ymn, if the following equation holds:
Here, two instances are considered, represented as (I yn) and (Im, V.)
The sufficiency metric is introduced to assess whether plug- ging in one data point is sufficient to enable the LLM to generate the correct output for the other one. Formally, the
mm 11plug((<., Vn Vn:));; 1):))= C= C(()
(11)
A Ya... Olunplug( (Xn, yn)); C' ((x,, ((nn:ual)).
14

---

Supplementary Materials: FEEDER
which indicates that plugging in data point (Z Un) can respond to the L.M's answering Dm in both ways. We omit S here, because we can derive the original status of the necessary instance based on the condition of the sufficiency instance.
A2.2. Set Level Metries
We extend Definitions Al and A2 the set level as: Definition A4 (Set-level Sufficiency Metric). Given tuple (X,Y, C,S), the input set Din is sufficient for output set Dour, if the following equation holds:
We further demonstrate that neither of the aforementioned quantities (i.e., sufficiency and necessity) is adequate for determining the other, indicating that they are not entirely independent. This is illustrated in the following lemma.
((enlanePam)) Dpna (pluu(
(18)
where Dix is not included in C and S can be any value. 1 Darel denotes D dinensional vectors whose elements are all Is. It indicates that when plugging in Din, it guaran- tees that the LLM's s output to any input question in Dout is correct.
Lemma (Connection between Sufficiency and Neces- sity). Supposing that we only consider using the data point (a Vn) as the plug in data, and only care about the L.M'' performance regarding the input question Dns then overall there are only two situations here: (i) (In, Yn) is plugged- in, and (ii) (I yn) is not plugged-in. Based on the above assumption, we re-write (i) as plugging-in in (In: ya) when there is no plugged-in data (i.e. plug(( Un)); C = 0, and re-write (ii) as unplugging (In: Yn) when there is plugged-in data (I yn) (i.e., umplug((En, yn)); C ((En ),))). For convenience, we use E* and E to denote (i) and (ii) respectively; and we use Y and Y to denote Yu, = 1 and Ya, = O. Then, we have: E* V E = true, Ex A E = false, VY = true, Y AY = false.
Definition A5 (Set-level Necessity Metric), Given tuple (X,Y, C, the input set Dix is necessary for output set Don, if the following equation holds:
(((e.))..Denn)) # 1 junplug(Diw::C.s
(19)
where Diu is included in C, S = (V((aalon)Demm)
(Par)) and Diu can be any subset DiN. 1DDa denotes 1 DDarl dimensional vectors whose elements are all Is. means that before unplugging any subset of Drx, there is plugged-in in data Dia and the LLM's 's oulput to any input in Dour is correct. When we unplug any subset of Diu then it would cause the LLM's output to at least one input in Dout to be incorrect.
We define PS as the probability of being sufficient as:
PS :r Yaam (llplu(([En.))))) C=
(12)
From the above description, when we refer to a set as a sufficient set, we are stating that the collective set of data points is sufficient. On the other hand, when we characterize set as a necessary set, we mean that each individual data point within the set is necessary.
-Pr(Y'|E*).
We define PN as the probability of being necessary as:
PN :Pr YYam- O)unplug((En) yn)): C = ((In, 8m))
-Pr
Example A3. Let Dour = {(xm. ymm) and DIx = f(x;; y:), (I.,, };)}. We assign am and ym as Which country does Sherlock Holmes live? and Sherlock Holmes lives in the United Kingdom. Let æ, and ye denote Which street does Sherlock Holmes live? and Baker street. We assign æ; and yj as Where is Baker street? and Bake street is located in London. Supposing that the LLM does not know that Bake Street is located in the United Kingdom, then solely plugging in either (Ir,, y:) or (x Uy) is not sufficient for the LLM to get the right answer to the input question am. In this regard, it easy to derive that DIN is both sufficient and necessary set for Dour when both (i) plugging in Din is sufficient to maintain the right answer for Dour; and (ii) unplugging any subset of Din can not maintain the right answer for Dour. are satisfied.
(13)
We further define PNS as the probability of being sufficient and necessary as:
PNS = Pr(Y" E EY") E).
(14)
Then, PS, PN, PSN satisfy the following relationship:
PSN = Pr(Y,E) PS + Pr(Y', (")) P..
(15)
Proof. Based on the earlier delineation of Y Y. E., and
E, we can express:
(16) E A AY|E= = EY" AY A (EVC")
=(Y"|E" AY E AE)...
Taking probabilities on both sides and using the disjointed-
ness of E and E, we have:
A2.3. FEEDER Set
PSN PY (E", Y|E)
=Pr(Y|E,Y Y E²) + PE(Y"1E YYE)
(17)
Next, we explore the problem of defining a subset within the given dataset DTRAIN that is both sufficient and necessary to represent DIRAI This subset is termed FEEDER (FEw yet Essential DEmonstRations)
-PrrYYE) PS + PY E
15

---

Supplementary Materials: FEEDER
Definition A6 (FEEDER Set). Given tuple (X,Y Y, C) and DTRAIN, subset of Draam, is considered as FEEDER set (denoted as DFEEDER), if the following conditions are satis- fied:
Algorithm 3 Exact Algorithm for FEEDER
Input: Training dataset PTRAIN._
Output: An exact FEEDER set DYEEDER.
Initialize k = 1.
Initialize H6 =0. for each instance (æn, Un) E DIRAIN do
(i) (EN)))))
1xiplug((raaee))
0, S
(Y ( # IN) holds.
Check Y((0e,// (.r
1|Dmanil /unplug Yn
)): C,S (a), C DYRAIN. S (X((=,) le,, (Drum))
(i) Y(m)....x * IN unplugg (orblug(Deeeneee ): C' DIRAIN, S (Y( = 1x) holds for any subset of DFEEDER (de- noted as DFFEDER).
AyDraall
If (a) holds, let Hn {( yn)} and append H,, to Ho.
repeat
IN denotes N N-dimensional vectors whose elements are all Is. (i) and (ii) respectively imply that plugging in DFEEDER sufficient and necessary to maintain the LLM generating correct output.
for each pair (Hi, Hz) where Hi, Hj E &A-1 do
( = 1 (Druall lunplug(H, H, U Check (((,,, le, (I,,
Hy):C,S (b), where C = DTRAIN and S =
= 1/2mmll ).
If (b) holds, generate new node H4 UHG. append it to Hr, and assign H. U Hj with MAINTAIN signals; otherwise, append H4 and Hy to Ab.
Example A4. If we merge Dix and Dour exemplified in Example A3 into one set DD namely let D DIx U Dour, then in this case, it is easy to derive that DI is FEEDER set (denoted as DPEEDER) ) for D.
Assign Hnnx = arg max ESG, [H.| with MAINTAIN signal.
The above definition of the FEEDER set is overly strict, as identifying it would require enumerating all possible subsets. To mitigate this complexity, we introduce the following approximation algorithm.
Remove the nodes without MAINTAIN signals in Hi.
Grow tree from bottom to top via + until He = 1 where assume the current round is K; Let HUXNCESSARY denote only one element (i.e. the root node)
in Hx-
Assign DYEZDER as removing HUNNCESARRY from DIRAIN, i.e.,
Algorithm 2 Approximation Algorithm for FEEDER
DEEEDER = DYBAIN HIRNECESBARY.
Input: Training dataset DIRAIX-
Output: An approximated FEEDER set DPEEDER.
Initialize k = 1.
Initialize Vn- = (Wn- = {( (x yn) E DrRAIn)-
A3. Approximated Extraction of FEEDER
repeat
for each pair (W,, W2) where W., W3 Wh-1 do Check Y(( (ta..la..eW;W. (W;H) = wwwllplug(W.):cC (a), where C' = 0 and S can be any value.
Definition A7 (Transitivity Inference over Sets). We assume that sufficiency is transitive over sets. Formally, for any three sets, denoted as Da. D2, and Dc. Da is a sufficient sel of 'De and DD. is sufficient set of De, then we can conclude that D2 is sufficient set of Dc.
Check I[( ew:)) = 1 p1)g(W))) C,S (b), where C 0 and S can be any value.
Case (Both (a) and (b) hold), [W; 2 Wy pend W, to Ws: otherwise, append W; Ws. Case II (Either one of (a) and (b) holds), if (a) holds, append W, to Ws: otherwise, append W, to Wg. Case III (Neither (a) nor (b) holds), append W, U W3 to Wi.
We also establish case studies in Appendix A10.1 to verify the feasibility of the above assumption.
For convenience, we use DIe (In, yn yn)) J to denote the input set for our tree algorithm, and we use Dour to denote the corresponding output. The tree expands from the bottom to the top. We use the variable K to represent the depth of these trees, which corresponds to the number of iterations. To be more specific, we use k = 1, 2, ... K to refer to each k-tt iteration, and during each k-th iteration, we generate the (k + 1)-th layer of the tree.
Remove W., W, from WK-1, ic., Vn-1 = Wh-1-
(W,, Wj).
Wi --11 then
L Append only element in W6-1 to Wh-
Grow tree from bottom to top via k = + 1.
until [ = 1, and we assume the current round is K;
Let WSUFFICIENT denote only one element (i.e. the root node)
Concretely, we leverage the transitivity of sufficiency to build the tree, where each node a set of samples. Formally, we denote Wi, as the set of nodes after the k-th iteration. We initialize Wo by assigning all the candidate samples in Din
in WK.
Assign DPEEDER as WAUFFICIENT, i.e., Dour WSUFFICIENT.
16

---

Supplementary Materials: FEEDER
(a) An Example of Algorithm for Searching Exact FEEDER
(b) An Example of Alternative Algorithm for Searching Exact FEEDER
H U U U H4
Run 1
H H
H3
H
Check Whether HIN U HOUT is Unnecessary for
DYRAIN. Extract Unnecessary Parts
Check Whether HIN U HOUT Unnecessary for DTRAIN Extract Unnecessary Parts
H2 U H
H3
H
H2
U
UHH
H
H
H
H
Check Whether HIN U HoUr is Unnecessary for DYRAIN. Extract Unnecessary Parts
Run 2
HCg
H
H2
H
H
Figure AI. An illustrated example of our algorithm for deriving an exact FEEDER set. As shown in (a), check the necessity of the conjunction of each pair of nodes, and we do not remove them from Hi; instead, we assign MAINTAIN signals to newly generated nodes and the node with the maximum size, and those nodes without MAINTAIN signals, circled with dashed lines, would be removed from H., In (b), we propose an alternative algorithm by removing nodes after checking the necessity, and we repeat the above process for multiple runs, at the beginning each run, we unplug all the previously selected data points. The repeat should stop until there is no or only one node in Xo (i.e., H4), and therefore, the result in (b) is Hi U H2 U H4, same the result in (a).
as the bottom nodes:
When there is only one element left in Wk-1, it is directly appended to W.. This process continues until W. contains only one element, which is denoted as WAUFFICIENT E WW. We then assign Dour as Daur WSUFFICIENT.
o = WW = (I Uo (an, Vn) Dmn.
(20)
During each k-th round, we generate V by examining the sufficiency relationship between every pair of nodes, denoted as W,, Wj- E Wh-1- In this evaluation, we assess whether the following equation holds true by assigning Wi and W as Win and WoUT, or vice versa.
The time complexity of running the above tree algorithm for one round is O(log2 Pall generate one layer of the tree. To effectively remove the unnecessary part, we can repeat the above process for multiple rounds by using the output of the previous round (i.e., Dour) as the input for the subsequent round (i.e., DIu) to build the tree from the bottom to the top. Therefore, the overall complexity of building a tree (i.e., executing Algorithm 2 for a single run) is O(K log] (P=l), where K the number of rounds. It follows that the time complexity for multiple runs is ORR log Pial), where denotes the number of runs
Y(fwn (HanllanWWann = | pllug(W::::C..
(21)
where C' = and is loosened to allow for any value. 1 Warrl is | -dimensional vectors whose elements are all 1s. It signifies that plugging in WIx is sufficient for the LLM to generate the correct output to any input in Wour In other words, once we have WIN included in the plugged-in context, it is unnecessary to further include Wort- Formally, we can derive the following equation from Eq. (21):
Our tree algorithm can also maintain the remaining set to be sufficient to represent the entire DrRAI, as verified in the following proposition.
(((...l...Wmm)) 1 |aam |unplug(Woor): C, S, (22)
Proposition (DrEEDER obtained by Algorithm 2 is an Ap- proximation of DFEEDER). If we successively apply Algo- rithm 2 on DyRAI for multiple rounds to obtain a subset (denoted as DPEEDER), then DYREDEa is sufficient to represent DTrAIn-
where C- C-=WNix UWor)))r)) and S is loosened to be any value. Coneretely, there are three possible scenarios by examining each pair of nodes in Wx-1: (i) If both W, and W, are suf- ficient sets for each other, then we select the one with fewer elements to append to Wr. (ii) If only one of W; and W; is a sufficient set for the other, then we append the sufficient set to Wi. (iii) If neither W; nor Wj is sufficient set, we append W. U W, to Wh. After performing the above calcu- lations for each pair of nodes, we remove them from WE-1-
Proof. In the tree generation process, each parent node is established as a sufficient set for every leaf node within the tree. More precisely, as shown in Case I, Case and Case III of Algorithm three scenarios exist for creating a parent
17

---

Supplementary Materials: FEEDER
node for each pair of leaf nodes. In cases (i) and (ii), the parent node corresponds to the leaf node which serves sufficient set for the other node. In case (iii), the parent node results from the conjunction of two leaf nodes, inherently forming a sufficient set capable of representing either of the two leaf nodes.
holds:
1/p.u/ (nppluup (H, UH (25)
where C² Dix and S is loosened allow for any value. This determines whether plugging H, U Hj is unneces- sary for maintaining the correct outputs to all inputs in DIX. If the above equation holds, we create a new node Hi U H, and add to Hr. labeling it with a MAINTAIN signal. Otherwise, we add both H. and Hj to Hz. After this computation, we identify Hxax = arg maxx E.W, [H.] and label it with a MAINTAIN signal. Subsequently, we re- move the nodes in H that lack MAINTAIN signals. This process continues until H. contains only one element, de- noted as HUMBERSSARY E Mx.. Finally, we calculate Dour as Dour = DIN HUNNECESSARY.
According to our assumption of the sufficiency transitivity, for each data point in TRAAI, the root node of the tree is sufficient set for each leaf node in the tree. Formally, we have:
Y (e.,, EDuus) = P |plug( D ): S,
(23)
where C' = 0 and S can be any value. This means that the resulting set DPEEDER is a sufficient set of DIRAIE.
A4. Exact Extraction of FEEDER
A4.2. Exact Extraction of FEEDER via Iterative
To extract an exact FEEDER set DFEEDEr from DTrAIR. we need to explicitly check the necessity among all the candi- date samples, and remove those unnecessary parts. We do not directly apply this algorithm in practice, due to its high computation costs. We provide a solution for integrating the algorithm into our FEEDER and report the corresponding results in Appendix A7.
Sufficiency Checks
Consider that at each round, we need to check the necessity for xg, times (where C: denotes a combination opera- tor), this becomes impractical. To this end, we develop an alternative algorithm. Specifically, at each k-th round, we remove all the checked nodes (i.e., Hi and Hi from Vi. similar to our approximation algorithm in Appendix A3), Then, it requires O(log) (Pul) computations to finish one round. To obtain an exact FEEDER, we need to keep repeat- ing the above process until there is no or only one left in Ho. While practical, we also can set maximum number of rounds to approximate.
A4.1. Exact Extraction of FEEDER via Necessity Checks
Our intuition behind constructing a tree for checking neces- sity is based on the inherent transitivity property of necessity. Formally, it can be expressed as: If unplugging Dr could cause the outputs incorrect, then unplugging DA DDD also can not maintain the outputs to all the input in De correct. Namely, if unplugging a subset would degrade the performance, then unplugging the whole set would also degrade the performance.
least one input in De from correct to
Proposition 2 (DrEEDER obtained by either Algorithm 3 or Algorithm 4 is an Exact DPEZOEa). If we successively apply either Algorithm 3 or Algorithm 4 on DirAI for multiple rounds to obtain a subset (denoted as DrEEDER), then DEEEDER is sufficient and necessary to represent DTRAIn-
Similar to the tree for explicitly checking sufficiency intro- duced in Appendix A3, each node in the tree for checking necessity also represents set of samples. For convenience, we also use DIu = {xx,, ) to denote the input set and Dour for the corresponding output. We use H to de- note a set of nodes after the k-th round.
Proof. According to Definition A6, it is straightforward to see that to prove the above proposition is equivalent to proving that DrEEDER is sufficient set of DIRAID and a necessary set of DYRAIK-
We initialize He by identifying all samples in Diu for which unplugging them individually does not affect the LLM's performance. Formally, we construct My as Ho (H,, {(x,, } where (In, yn) E DIN satisfies:
We begin by proving sufficiency. Either Algorithm or 4 preserves the sufficiency during checking the necessity, as we are always guaranteeing Y((=,, Y((,,,, EPruu)))) 1 ||Pra when removing the unnecessary parts.
(((m,)la,,rED))) = 1/ htnp//uulll....../ (xx y,)) C,S, (24)
In other words, we have: Y((,,, E 1 L (pralllunplug hnnp/ (g(TNRR HuMNECESSARY) ); C, S, where C
where C = Din and S is loosened to allow for any value. During each k-th round, we generate Hi by examining the necessity relationship between each pair of nodes (denoted as H4, Hi E -...)... -). Here, we further verify whether solely unplugging H. U H, does not impact the LLM's perfor- mance. Formally, we check whether the following equation
DInAIu and S can be any value. It can be rewritten as:
((.... = 1/prur (DEUDEDEEDDER))) C, C,S, (26)
where C= @ and S can be any value. It shows that plugging in DEREDER is sufficient for representing DIRAIN-
18

---

Supplementary Materials: FEEDER
Table AI. Performance comparisons on text classification datascts are conducted in the in-context learning setting. We report both the mean and variance of accuracy using different seeds and different permutations n-shots. This table extended from Table 1.
SUBJ
SST-2
COLA
Vinl()
D
Random
Similarity
Diversity
Random
Similarity
Diversity
Random
Similarity
Diversity
41.3 (2:3)
41.1 co.0
41.1 o
48.9 (4.0)
24.5 (.:
24.5 (..
29.0 (544
38.8 e.u
38.8 c
2
47.3 (1:))
62.8 co.0
71.9 em
51.2 (5.8)
65.7 (0.))
62.5 (.2
30.9 (40)
38.5 (..2)
36.2 o.u
DTLTT
51.8 (5.5)
85.8 (0.3)
70.1 (02)
62.6 (5.0)
79.4 (..))
61.7 m.)
39.4 (58)
49.3 (o.)
47.0 (0.))
10
62.4 (50)
88.0 (o.2)
78.2 (0.))
50.9 (4.9)
83.8
76.9 (.)))
31.6 (4.)
52.5 (0.2)
58.8 (0.2)
GPT-2 (0.3))
42.8 (2.4)
44.9 (1.D)
44.9 (I.
49.8 (4.2)
48.1
48.1 (1.9)
29.6 (4.1)
35.1 (1.5)
35.1 (1.5)
55,9 (3.3)
63.4 (1.0
74.7 (0.9)
67.3 (4.4)
67.7 (1.4)
64.7 (1.5)
31.3 (2:)
41.7 (1.2)
34.9 (1.9)
PEEDes
57.5 (4.0)
86.9 (0.7)
69.8 (1.m
70.3 (4.4)
77.9 (1.2)
68.5 (1.9)
35.2 (2.0)
57.3
54.6 (1.7)
10
63.5 (4.4)
88.7 (1.5)
79.7 (2.0)
75.2 (6.7)
83.0 (1.7)
77.2 (1.5)
59.3 (3.)
68.7 (2:4)
68.5 (2.9)
42.5 (5:0)
43.6 (0.1)
43.6 (0.1)
49.0 (4.3)
42.3 (0:2)
42.3 (0.3)
42. (5.7)
48.3
48.3 (0.1)
2
58.1 (6.3)
88.3 (0.2)
87.0 (0:3)
68.0 (5.2)
70.7 (0.3)
59.6 (.2)
41.1 (42)
36.8 (0.2)
37.7 (0.1)
D DTaTT
66.7 (4.5)
86.2 (0.2)
86.7 (0.1)
49.1 (4.3)
80.6 (0.3)
67.5 (0.2)
46.2 (4.7)
53.8 (0.2)
48.5 (0.3)
10
48.6 (6.0)
85.9 (0.1)
73.9 (0.2)
71.1 (4.5)
84.6 (0.3)
73.1 (0.2)
43.4 (4.5)
55.5 (0.2)
56.1 (0.4)
MED (0.8R)
45.8 (5.1)
46.4 (0.4)
46.4 (0:4)
49.1 (3.00
47.7 (1.3)
47.7 (1.3)
46.6 (3:4)
45.1 (.D
45,1 (1.1)
63.1 (4.5)
89.7 (1.5)
86.8 (1.3)
69.8 (3.90
73.0 (2:9)
61.2 (2.1)
36.6 (3.5)
37,0
34.6 (2.0)
D PREDES
73.4 (4.3)
88.2 (1.9)
88.8 (1.7)
59.3 (2.4)
80.9 (1.3)
69.6 (1.7)
59.2 (3.3)
68.6
66.6 (1.7)
10
52.0 (3:8)
87.4 (1.3)
75.6 (1.2)
76.0 (3.0)
86.7
75.6 (1.5)
59.3 (4:00
68.8
68.9 (1:0)
42.8 (3.9)
42.1 (0.1)
42.1 (0.1)
49.2 (3:7)
33.8 (0.3)
33.8 m.))
25.5 (3.4)
36.5 (0.2)
36.5 (0.2)
48.5 (4.2)
88.3 (0.2)
72.6 (0.3)
76.8 (3.5)
81.5 (0.3)
76.3 m.4
30.7 (B.1)
55.5 (0.2)
56.5 (0.4)
D DTATT
51.6 (5.0)
90.5 (0.2)
81.7 (0.2)
65.1 (3.5)
80.8 (0.2)
66.1 m.3)
40.0 (3:6)
55.9 (o.))
52.5 (0:2)
10
48.5 (5.8
85.9 (0.3)
81.9 (0.1)
69.8 (4.8)
84.1 (0.3)
69.7 m.)))
39.6 (4.5)
59.3 (0.3)
63.4 (0.1)
XEO
43.2 (4.0)
46.3 (1.0)
46.3 (1.m
49.3 (5.1)
48.3 (1.9)
48.3 (1.9)
28.3 (5.4)
34.8 (1.3)
34.8 (1.3)
2
62.6 (3.5)
89.4 (1.5)
73.8 (2.11
75.1 (2.8)
82.6 (2.1)
78.5 (1.9)
59.3 (3.7)
64.7 (1.4)
64.7 (1.6)
Dreeee
5
69.4 (5.00
91.2 (1.8)
82.9 (1.3)
73.2 (4.2)
82.9 (2:7)
71.6 (2.4)
58.7 (32)
67.2 (2.4)
65.8 (1.8)
10
58.7 (3:3)
87.2 (1.7)
84.3 (2.8)
72.4 (3.4)
85.8 (2.5)
71.8 (2.9)
59.8 (2:8
68.8
68.9 (1.3)
45.0 (5.9)
48.1 (0.0)
48.1 1 (0.6)
51.2 (6.8)
52.2 (0.5)
52.2 (0.5)
37.5 (70)
40.5 (1.3)
40.5 (1.3)
62.3 (6.9)
82.5 (1.8)
74.2 (1.3)
71.5 (5.6)
78.5 (1.5)
75.9 (..)
40.6 (5.9)
62.5
61.6 (0.5)
Dman
68.0 (7.1)
91.55 (1.2)
84.2 (1.6)
70.2 (5.6)
80.5 (1.6)
80.6 (0.7)
46.5 (5.9)
67.2
65.6 (0.6)
10
50.3 (8.2)
86.2 (1.9)
85.6 (0.8)
68.2 (4.8)
85.5 (1.5)
76.3 (1.3)
50.2 (7:4)
69.8
71.5 (1:2)
Gemma-2 (28)
48.2 (4.2)
49.5 (1.m)
49.5 (1.m
52.6 (4.6)
53.1 (0.5)
53.1 m.5)
38.9 (5.2)
39.6 (0.8)
39.6 (o.8)
65,2 (2.9)
85.2 (1.0)
80.3 (008)
74,2 (1.9)
82.1 (1:20
83.0 (0.7)
$2.5 (2:5)
68.9 (2.1)
67.8 (1.5)
D Pazses
72.2 (6:2)
94.5 (5.3)
85.5 (0.71
72.0 (4.2)
83.6 (2.0)
84.5 (1:7)
55.2 (430)
77.6 (2.5)
73.9 (2.3)
10
60,5 (4.0)
86.5 (2.5)
88.4 (2.4)
70.5 (5.0)
92.6 (2.6)
78.5 (5.3)
58.6 (4.6)
75.6 (2.9)
76.6 (2.5)
44.9 (6.6)
49.5 (o.0
49.5 (0.1)
48.2 (2.9)
47.0 (0.3)
47.0 (.1)
38.9 (6.7)
41.2 (0.2)
41.2 (0.2)
55.4 (3.5)
85.5 (o.0
86.5 (0.2)
68.1 (4.2)
78.7 (0.))
77.5 m..)
42.8 (4.0)
45.5 (o.3)
45.6 (0.2)
51.2 (4.4)
90.8 (0.2)
82.7 (0.1)
75.2 (3.3)
80.7 (0.3)
77.8 m.))
48.5 (3.3)
51.8 (0.3)
52.1 (0.2)
10
57.7
87.3 (o.1)
85.3 (0.1)
72.1
77.6
76.5 (.2))
59.1 (4.2)
60.3 (o.1)
61.0 (0.2)
GPT-3 (68)
43.9 (4:7)
51.2 (1.0)
51.2 (1.0
49.6 (2.4)
51.3 (1:6)
51.3 (1.5)
41.2 (2:1)
43.8
43.8 (1.8)
Dresees
65.7 (3:00
91.5 (1.1)
(1.6)
(2.5)
85.7 (4:2)
76. (2.9)
(2.0)
63.1 (1.5)
60.1 (1.4)
53.7 (3:00
92.9 (0.8)
91.5 (1.4)
77.6 (4.0)
81.0 (1:3)
79.4 (1.09
50.6 12.7)
63.3 (1.4)
65.8 (1.4)
10
58.0 (3.4)
88.8 (0.9)
87.8 (1.2)
83.8 (2.8)
86.4 (2.0)
87.2 (1.3)
59.7 (300)
67.5 (1.9)
68.4 (2.2)
42.9 (6.6)
48.5 (0.D)
48.5 (0.1)
46.2 (2.7)
49.1 (0.3)
49.1 (0.3)
40.1 (6.1)
42.0 (0.)
42.0 (0:2)
2
51.9 (4.4)
90.7 (0.))
85.2 (0:))
67.8 (3.2)
73.5 (0.))
74.5 (..))
43.5 (4.5)
47.4 (0.)
49.6 (0.1)
5
51.6 (3.2)
86.8 (0.2)
82.9 (o.
74.8 (3.8)
81.2 (0.2)
78.7 (.2))
50.2 (3.7)
52.6 (0.)
48.2 (0.3)
10
56.1 (4.6)
81.33 (0.))
85.7 (0.1)
73.2 (3.)
76.3 (0.1)
77.1 (..))
59.6 (4:3)
55.3 (0.))
60.0 (0:4)
Llama-2 (78)
43.8
49.7 (1.0)
49.7 (1.0)
47.2 (2:4)
50.8
50.8
41.2 (2.1)
43.8
43.8 (1.8)
2
54.8 (300
92.5
84,8 (077)
72.2 (M1)
82.5 (400)
80.1 (2.6)
50.8 (2:3)
58.6
53.5 (1.3)
53.7 (3:00
87.9 (1.8)
91.5 (1:4)
78.3 (4.6)
83.2 (1.)
80.1 (1.4)
53.8 (2:0)
65.3
61.8 (1.4)
10
58.0 (3.4)
85.8 (0.9)
87.8 (1.2)
85.0 (2.2)
87.1 (2.2)
86.9 (1.0)
60.5 (3.))
68.0
68.4 (2.0)
19

---

Supplementary Materials: FEEDER
Table A2. A complementary table Table Al presents the corresponding results for the demonstration selectors Uncertainty, Clustering,
Latent.
SUBJ
SST-2
COLA
(ar())
D
Uncertuinty
Clustering
Latent
Uncertainty
Clustering
Latent
Uncertainty
Clustering
Latent
53.5 (6.3)
49.3 (4.0)
51.5
49.0 (29)
47.5 (1.)
47.8 (1.))
42.0 (6.5)
39.8 (1.5)
40.2 (1))
87.8 (X7)
86.5
86.3
75.6 (4.))
80.1 (3:2)
79.0
49.6 (4:)
46.8 cm
47.5 (3))
DTTAT
5
90.7 (4.5)
88.2
89.4 (4.))
81.8 (3.)
82.2 (3.3)
80.7
55.4 (3.5)
56.4 (4.)
58.8 (33)
10
88.3 (4.9)
90.7
91.3 (4D
80.5 (3))
78.8 (3.9)
76.8 (4.3)
58.4 (4.2)
62.1 (36)
61.5 (45)
GPT-3 (68)
1
55.3 (4.2)
50.9
50.2 (s2)
50.3 (2.4)
48.4 (3.4)
48.3
43.8 (2.0)
40.8 (3.5)
42.5 (5.1)
2
89.8 (3.00
89.7
89,5 (2:5)
77.1 (2.5)
82.5 (3.5)
83.0 (3.2)
60.0 (2.0
57.8 (4.4)
58.1 (3.5)
DYEESER
5
92.3 (3.6)
92.0 (2:4)
91.8 (2.9)
81.2 (4.0)
80.8 (3.6
80.4 (3.9)
62.4 (2:7)
61.6 (175
62.3 (2.4)
10
90.8 (3.4)
92.0 (2.4)
91.8 (2.9)
81.2 (2.8)
80.8 (3.8)
80.4 (2.9)
62.4 (3.0)
62.7 (3.3)
62.5 (2.5)
49.0 (6.6)
48.5 (5.6)
47.5 (5.D
49.2 (2.7)
48.2 (8.7)
48.7 (..
40.1 (6
41.1 (4.1)
41.0 (1.)
2
89.2 (4.4)
87.8 (3.5)
88.7 (4.D
75.1 (2.2)
72.5 (2.2)
74.7 (4.2)
48.5
45.2 (4.2)
46.4 (12)
PrRAI
5
82.9 (3.7)
80.1 (7:2)
83.8 (1:3)
83.7 (1.8)
81.5 (3.0)
82.2 (1.2)
53.2
51.2 (1.)
$2.6 (1:)
10
86.2 (4.5)
82.1 (4.4)
83.3 (2.0
76.4 (3.1)
75.2 (3.7)
74.8 (4.1)
63.5 (4.3)
62.6 (4.0)
60.3 (2.)
Llama-2 (T))
49.7 (4.9)
45.8 (4.9)
48.7 (5
51.8 (2:4)
48.4 (5.))
50.3 (1.7)
43.0 (2.1)
42.2 (1.5)
42.8 (1:8)
91.8 (3.0)
90.8 (2:4)
91.5 (2:4)
78.1 (3.1)
73.5 (3.0)
76.5 (4.2)
49.5 (2:3)
48.8 (2.3)
50.6 (2.7)
Dyazze
89.5 (3.8)
88.7 (4:00
86.9 (2:00
84.1 (466
82.3 (4.5)
83.8 (4.1)
60.8 (2:8)
58.8 (3.8)
59.3 (26)
10
88.8 (3.4)
88.0
86.8
80.9 (2.2)
85.1 (2.0)
83.4 (2.2)
67.4 (3.))
64.5 (3.4)
66.0 (2.7
Next, we investigate necessity. Our goal is to prove un- plugging any data point in DEEWER would lead to a degra- dation of the LLM's performance. For convenience, we
Combining the above analysis of sufficiency and necessity, we can conclude that DPEEDER an exact FEEDER for DrRAI
use (æi, yn) E DTrAII to denote an arbitrary data point. If we applying Algorithm 3 to execute the search for an exact DFEEDER then (I,,. Un) must be in Ny, or out of Atg
A5. FEEDER in In-context Learning Setting
A5.1. Demonstration Selectors
If (I:,, yn) is not an element in Mr. then according to the computing process of Mo (i.e., lines to 3 in Algorithm 3),
As described in Section 5.1, when applied in the setting, our DPEEEE is assessed by serving as the selection pool, replacing DTRAIN for existing demonstration selectors.
unplugging (In. yn) it would definitively cause the LLM's
performance DTRAIN from (((,,)... to Y (((aa)))..EPrm))) * 1|Praml-
1 |Prassl
The first one is a random selector, denoted as Random, which randomly selects samples from the selection pool.
If Un) is an element in Ho. then (æn, yn) must be in HunNEcEsSARy: otherwise, according to lines 3 to 3 in Algorithm 3, HUWECESSARY U { yn)} should be Hnxx and always stay in H. until becoming the root node (i.e., HuMNECESSARY should be updated to be HIRRECESSARY U (< Bn))). Thus, (In, yn) must be in HORNECESSARY However, all the data points in HUmBEeSSARY from DYRAIK, causing a contradiction. Hence, unplugging (In. yn) would change the LLM's performance, namely necessity holds.
The second one is similarity-based selector, denoted as Similarity, which selects samples similar to the test samples. Formally, let DEELECT denote the selection pool. Then, for each test sample Im. the metric of similarity can be written as:
removed
SIM (mmy xnn) = (TSANSSSSORRMERR (.,,,, TRANSFORMER (x,)))
(27)
Then, we consider applying Algorithm 4 for searching an exact PFEEDER. Similarly, if (I.. vn, is not selected when checking the necessity, then unplugging (æn, yn) would definitively cause degradation of the LLM's performance.
where In E DEELECT CDS(.) is cosine similarity met- ric, and TRANSFORMER(.) denotes a sentence transformer (Reimers & Gurevych, 2019). Here, we directly use the Sentence Transformers library² from Hugging Face in our implementation. Then, we are able to select Nabor samples with maximum SIM values from DEELECT-
If (In, yn) is selected during checking the necessity, then (In, ya) must be included in D,: otherwise, D, would con- tinue to update, since the condition of stopping iteration is that there is no or only one unnecessary node. However, all the data points are removed from DIRAIK. causing con- tradiction. Hence, unplugging (æ,, y,)) would change the LLM's performance, namely necessity holds.
The third one a diversity- based selector, denoted as Di- versity, where we adopt the maximal marginal relevance method (Carbonell & Goldstein, 1998) as the metric of Di-
s sentence- contence-transfornee
20

---

Supplementary Materials: FEEDER
Table A3. Performance comparisons on text classification datasets are conducted in the in in-context learning setting. We report both the mean and variance of accuracy using different seeds and different permutations n-shots. This table extended from Table 1.
FPB
SST-5
TREC
V(s(()
D
Random
Similarity
Diversity
Random
Similarity
Diversity
Random
Similarity
Diversity
1
27.2 2 (6.1)
25.3 w.D
25.3 (0.))
14.5 (6.1)
22.7 0..)
22.7 ma.
19.4 (6.4)
42.8
42.8 (0.1
2
27.4 4 (62)
45.8 (0.2)
40.4 (..))
18.0 (58)
25.6
23.7 (a.
21.4 (47))
57.2 (02)
51.4 (0.1)
PYBAIS
5
26.3 (4:)
55.9 (0.D)
44.7 (0.)
26.5 153)
32.3 (..)
27.8 (a.))
37.6 (5.0)
66,0 (0.3)
61.4 (0.3)
10
27.8 (5.1)
63.1 (0.))
50.7 (..))
14.9 (39)
35.3 (.D
30.4 (0.2)
53.0 (5:2)
71.4 (0.2)
65.8 (0.3)
GPT-2 (0.3B)
1
28.4 (3.4)
28.8 (2.1)
28.8 (2.)
15.4 (5.2)
23.7 (..)
23.7 (1.))
37.4 (3.6)
48.4
48.4 (1.6
2
35.5 (4.3)
47.4 (2.6)
37,9 (1.9)
20.9 (47)
27.9 (1.)
25.8 (1.3)
27.6 (3.2)
58.8 (2.2)
52.1 (1.9)
DreEDER
28.3 (3.0)
54.6 (1:7)
47.9 (1.0)
28.6 (3:4)
33.2
27.4 (..)
40.8 (3.0)
67.4 (1.2)
61.8 (1.3)
10
39.6 (3.4)
66.5 (2:3)
51.8 (1:2)
(2:0)
36.9 (1.9)
29.8 (1.))
(2.5)
74.6 (1.4)
67.6 (1.9)
1
33.8 (5.2)
29.9 (0.D
29.9 (..
14.2 (4.9)
25.2 (0.1)
25.2 (0.1)
21.0 (4.5)
53.2 (0.2)
53.2 (0.2)
2
27.0 (6.1)
55.4 (0.2)
49.9 (0.3)
18.1 (5.1)
29.7 (0.D
24.4 (0.2)
28.2 (4.4)
62.6 (0.2)
60.6 (0.2)
Druxx
5
27.2 (4.8)
64.3 (0.3)
45.1 (0.3)
25.6 (4.8)
34.1 (0.D
30.8 (0.1)
35.4 (5.7)
63.4 (0.1)
64.6 (0.1)
10
47.0 (5.5)
65.5 (0.2)
52.9 (..))
28.7 (4.2)
38.7 (0.D
36.6 (0.1)
43.2 (4.5)
66.0 (0.0)
68.8 (0.11
GPT-2
1
33.8 (4.4)
32.6
32.6 (1.))
18.7 (3.0)
25.5 (2.2)
25.5 (2.2)
22.4 (3.9)
52.6
52.6
2
37.5 (4.7)
54.8
47.6 (1.3)
25,2 (3.8)
29.7
24.1 (2.0)
34.6 (3.5)
64.2
59.4 (2.0)
PrEeos
5
38.9 (3.9)
64.5
48.0 (2.)
39.3 (2.9)
35.2 (1:))
31.0 (1:2)
45.4 (3.)
65.5
64.9 (1.7)
10
63.5 (2:80
66.7 (1:6)
53.1
39.6 (300
39.8 (1.8)
37.8
55.8 (3.9
70.4
68.6
1
54,9 (3.9)
61.6 (0.1)
61.6 (0.1)
12.8 (2.7)
20.2 (..1)
20.2 (0.1)
11.0 (3.)
57.2 (0.)
57.2 (02)
2
53.6 (4.0)
66.8 (0.2)
60.0 (..))
17.9 (3:6)
26.9 (..))
22.7 (0.1)
17.6 (3.))
52.6 (0.2)
42.2 (02)
Druxx
5
28.2 (4.0)
68.2 (0.1)
60.4 m.))
19.0 (3:9)
29.2 (0.D)
25.1 (0.1)
25.2 (3.5)
66.4 (0.1)
61.8 (0.11
10
49.0 (48)
75.8 (0.1)
71.1 (.2)
12.7 (2:4)
33.7 (0.2)
31.9 (0.1)
41.6 (4.4)
70.6 (0.1)
69.0 (0.11
GPT-neo (1.38)
1
58.1 (4.7)
61.8 (1.4)
61.8 (1.4)
18.5 (2.1)
20.6 (1.8)
20.6 (1.4)
18.2 (2.4)
56.4
56.4(1.3)
2
61.4 (3.3)
64.1 (1.5)
58.8 (LD)
19.7 (2.7)
27.4 (2D
22.8 (1.5)
27.8 (2.7)
54.0 (1.4)
44.5 (1.6)
PEEDSS
5
43.2 (2.6)
68.8 (1.5)
62.7 (13)
19.2 (32)
30.2 (2.7)
26.4 (2.4)
50.4 (..)
68.0 (1.4)
62.6 (1.9)
10
61.4 (2:3)
74.8 (1.9)
71.9 (1.5)
15.4 (2.4)
37.0 (1.5)
34.5 (1.9)
45.2 (2.9)
72.8 (1.4)
69.8 (1.5)
1
58.2 (5.7)
62.5
62.5 (.1))
21.5 (39)
22.5 (..))
22.5 (0.1)
21.9 (3:4)
52.3 (o.))
52.3 (0.1)
2
59.2 65.9)
666.2
65.8 (.3)
26.5 (3.6)
42.5 (0.5)
42.2 (0.6)
35.6 (4.4)
60.0
59.1 (011)
Dmaxx
5
48.6 (3.6)
76.6
78.8 m.6)
26.6 (2.5)
48.8 v0.))
41.2 (0:4)
55.8 (2:9)
82.2
71.1 (006
Gemma-2 (2B)
10
35.2 (6.5)
79.5 (0.4)
78.8 (0.2)
(4.4)
50.2 (0.8)
43.3 (0.4)
51.1 (3.3)
84.3 (0.5)
75.0 (0.4)
1
59.9 (4.4)
64.6 (0.5)
64.6 (0.6)
22.6 (4.3)
25.8 (1.5)
25.8 (1.3)
26.2 (1.5)
55.1 (1.8)
55.1 (1.8)
2
55.4 (2:4)
67.8 (1.5)
67.0 (1.
28.7 12:3)
45.4 (1.0)
46.8 (1.1)
40.8 (1.5)
63.6 (1.3)
62.8 (1.0)
Premen
5
52.2 (3.4)
88.0 (4.5)
80.1 (.2)
30.5 12.0)
52.6 (1.9)
54.4 (1.4)
60.4 (2.5)
87.8 (1.6)
73.0 (1.21
10
39.1 (5.1)
81.3 (3.))
83.8 (2.4)
36.8 (2:2)
62.5 (1.5)
54.9 (1.3)
58.1 (5:2)
88.9 (1.80
83.4 (1.4)
1
30.7 (5:5)
55.3 (0.1)
55.3 (..))
19.6 (3.6)
20.5 o..
20.5 (0.))
21.4 (4.4)
50.7 (o.))
50.7 (0.1)
2
33.4 (4.9)
64.9 (0.4)
65.5 (0.3)
24.1 (300)
30.5 (0.4)
31.6 (0.3)
34.4 (4.00
58.8 (0.)
60.7 (0.1)
DTRAI
5
40.6 (3.0)
75.0 (0.4)
74.9 (..))
24.1 (2:5)
32.5 0..))
35.6 (0:3)
51.8 (2.9)
71.2 (02)
70.6 (0.4)
10
25.9 (6.5)
78.5 (0.4)
79.5 (..))
35.5 (4:2)
38.9 o..
40.5 00.3)
49.5 (3.)
72.5 (0.1)
73.0 (02)
GPT-3 (80
1
31.2 (4.0)
54.8
54.8
20.6 (3.1)
27.8
27.8
32.2 (1.5)
52.1
52.1 (1.)
2
35,4 (2:4)
65.8
67.1 (0.9)
28.7 (2:3)
33.4
33.0 (L.)
44,8 (2.5)
60.1
61.8 (1.4)
Dyemmn
42.2 (3:4)
77.9 (3.6)
78.4 (.2)
28.5 (2.00
35.6 (1.3)
37.4 (1.4)
53.4 (2.7)
75,8
72.2 (1:2
10
39.1 (5.1)
80.3 (3.3)
82.8 (2.4)
36.8 (2.2)
41.5 (1.5)
40.9 (1.3)
54.1 (5.2)
76.9 (1.8)
80.4 (1.4)
29.0 (4.7)
47.1 (0.1)
47.1 (..))
28.6 (2.)
29.7 (0.))
29.7 (0.1)
35.2 (3.7)
54.2 (0.1)
54,2 (0.1)
2
27.4 (3.4)
68.4 (0.3)
67.1 (..))
35.9 (3:1)
33.9 m..
33.5 (0.3)
45.0 (4.0)
69.4 (0.1)
63.6 (0.1)
DiRAI
5
39.7 (h20
80.3 (0.3)
78.9 (..))
37.9 (2:9)
38.3 m0.))
37.0 a..
53.0 (3.6)
79.00 (0:)
70.4 (0.3)
10
37.9 (2.6)
87.4 (0.3)
86.5 (0.))
38.4 (530
37.5 m..
40.0 (0:)
58.0 (2.9)
83.4 (0.1)
79.2 (0.1)
Llamna22
1
33.7 (5:3)
51.7 (0.5)
51.7 (0.5)
27.6 (2:4)
32,3 (1.5)
32.3 (1.3)
41.2 (2.1)
56.8 (1.8)
56.8 (1.8)
2
39.6 (5.0)
68.7 (1.5)
69.8 (0.7)
39.5 (2:5)
32.6 (1.2)
32.7 (L.)
53.8 (2.3)
68.6 (1.7)
63.5 (1.3)
D Pramme
5
45,6 (4.00
87.9 (4.5)
79.5 (3.5
39.2 (2.0)
38.7 (1.3)
39.4 (1.0)
58.2 (2.5)
$2.8 (1.6)
71.8 (1.4)
10
37.8 (6:4)
87.1 (3.9)
87.8 (22)
39.7 (2:)
39.0 (1.1)
41.6 (1.3)
59.8 (3.3)
86.0 (1.9)
83.4 (2.0)
21

---

Supplementary Materials: FEEDER
Table A4. complementary table to Table A3 presents the corresponding results for the demonstration selectors Uncertainty, Clustering,
Latent.
FPB
SST-5
TREC
Pize(
Uncertainty
Clustering
Latent
Uncertainty
Clustering
Latent
Uncertainty
Clustering
Latent
(5.5)
56.3 (4m)
(8.
(29)
27.5 (1.5)
25.8 (1.1)
$2.0 (6:)
49.8
50.2
(377
66.5
66.3
35.6
36.1 (2.2)
34.0
59.6 (4.0)
(5.)
58.5
76.7
78.2
(424
41.8 (33)
42.2
40.7
65.4
66.4
65.8
10
78.3
80.7
81.3
40.5 (1.)
38.8
36.8
78.4
72.1
71.5
GPT-3 (88)
56.3
57.9
58.2
32.3 (7.4)
29.4
28.3
53.8
50.8
52.5 (5.1)
2
69.8 (3.00
69.7
69.5
37.1 (2.5)
42.5
38.2
60.1 (2.1)
57.8 (4.5)
59.1 (3:5)
DYZEBer
5
82.3 (3.8)
82.0
81.8
44.2 (400)
45.8
44.4
68.4 (27)
66.6 (3.7)
67.3 (2.4)
10
80.8 (3.4)
83.0
83.8
42,2 (2:80
40.8 (M8)
40.4 (3.9)
82.4 (50)
74.7 (1-)
73.5 (2.5)
49.0 (6.6)
47.5
47.5
36.2 (2.4)
37.2
38.7
55.1 (6.
54.1 (4.0)
54.0 (33)
2
68.2 (4.8)
67.8
68.7 (4:
35.1 (4.2)
32.5 (2.0)
34.7 (4.7)
67.5 (4.5)
68.2 (41)
66.4 (1.3)
DaRA
S
80.9 (3.2)
81.6 (2.2)
83.8 (1.2)
36.7 (2.8)
38.5 (3.0)
39.2 (1.2)
68.2 (3.7)
69.2 (2.5)
67.3 (22)
10
86.2 (4.00
85.1 (4.0)
87.3 (2:1)
36.4 (3.1)
35.2 (37)
39.8 (4.1)
86.5 (4.3)
85.6 (4,0)
87.3 (2.2)
Llama-2
1
51.2 (4.8)
48,9 (4.3)
48.7 (5.D
41.8 (2.4)
44.4 (3.5)
43.3 (2.7)
58.0 (2.1)
62.2 (2.5)
62.8 (18)
2
71.8 (3.00
72.8 (1:4)
73.5 (2:4)
45.1 (3.1)
45.3 (M
46.5 (4:00
69.5 (2:3)
70.8 (1.7)
70.6 (27)
DYAZZR
88.5 (3.8)
85.7 (4R8)
86.9 (2.8)
42.1 (4.6)
42.3 (4.5)
40.8 (4.1)
72.8 (2.8)
75.8 (3.5)
69.3 (26)
10
88.8 (3.4)
91.1 (4.4)
89.8 (2.9)
46.9 (2.2)
50.1 (2.0)
53.0 (2.2)
87.4 (3.1)
(3.4)
(27)
Table A5, Performance comparisons text classification datasets are conducted in the fine-tuning setting, where we tune the LLMs and evaluate their few-shot inference performance, We report both the mean variance of accuracy using different seeds and different permutations of n-shots. This table is extended from Table 3.
FPB
SST-5
TREC
Vuu(())
n
Random
Similarity
Diversity
Random
Similarity
Diversity
Random
Similarity
Diversity
58.3 (57)
68.4 m..)
67.4 m.1>
(48)
60.2 (0.4)
58.4 (0.)
59.2 (52)
70.0 an
68.0 (0:1)
2
58.5 (5.2)
72.3 (04)
70.1 m.s
58.5 (4.))
60.4 (0.6)
61.2 (0.4)
57.7 (5.)
70.1 (z2)
70.3 (044
DTRAI
5
67.8 (5.1)
66.2 (04)
65.7 m.3)
58.6 (5.))
60.4 m.
61.8 (0.5
66.3 (4.5)
72.8 (04)
70.2 (o:)
10
58.2 (4:4)
63.3 (0.))
65.6 (.3)
61.4 (4.9)
60.4 (0.4)
61.8 (0:))
60.9 (3.5)
71.3 (05)
72.5 (0.)
GPT-2
1
65.0 (5.5)
77.3 (1.3)
73.3 (1.3)
61.7 (422
74.8 (1.8)
74.4 (0.5)
63.9 (4.00
74.3 (0.))
75.3 (0.7)
2
62.2 (3:4)
75.0 (1.1)
74.3 (1.5)
62.3 (3.4)
63.4 (1.8)
62.6 (1.2)
60.1 (3.5)
76.1 (1.7)
74.4 (0.9)
Dysenea
5
70.4 (3.2)
78.8 (1.6)
76.4 (1.0)
62.4 (4:))
62.2 (1.4)
66.4 (1.3)
68.8 (3.20
77.2 (3.3)
76.6 (2.9)
10
62.3 (3.3)
80.6 (1.3)
78.6 (1:9)
63.9 (4.5)
78.6
71.0 (1.2)
68.7 (2:0)
72,2 (1.7)
75.7 (1.9)
1
60.3
73.4
73.4 0.D
57.5 (5.1)
64.3
64.3
61.1
77.3
77.3
2
62.5
75.3
75.1
62.5 (42)
65.4
66.2 (0:4)
62.7 (5.2)
78.1
79.3 (0.4)
D DTrAn
71.8 (5.1)
72.2 (0:4)
70.1 (.3)
63.6
67.4
(0.6)
(4.5)
76.8 (0.4)
74.2 (0.5)
GPT-2 (R)
10
(4:4)
67.3 (0.5)
(0:3)
(4.9)
68.4
67.8 (0:3)
(3.5)
78.3 (0:5)
75.5 (0.9)
69.0 (5.3)
81.3 (1.3)
81.3 (1.3)
59.8 (4.2)
72.8 (0.8)
72.8 (0.8)
65.9 (4.00
83.3 (0.7)
83.3 (0.7)
73.2 (3.4)
82.0 (1.)
83.3 (1.5)
65.3 (3:4)
73.4 (1.8)
72.6 (1.2)
62.1 (3.5)
80.1 (1.7)
82.2 (0.9)
DrEEDER
74.4 (3.4)
84.8 (1.5)
86.4 (1:4)
67.4 (3.9)
77.5 (1.b)
76.7 (1:4)
69.8 (3:2)
83..2 (3.9)
84.6 (2.9)
10
75.3 (3.))
85.6 (1.3)
87.6 (1:9)
58.9 (9.5)
78.6 (1.7)
79.0 (1:2)
69.7 (.7)
86.2 (1.7)
85.7 (1.9)
1
62.7 (5.7)
78.4 10.1)
78.4 (0.1)
60.3 (4.1)
66.6
66.6 (1.4)
63.3 (5.2)
79.5 (0.4)
79.5 (0.4)
2
63.1 (4.6)
74.2 (0.3)
73.1 (0.2)
64.5 (3:2)
66.8 (0.8)
68.4 4 (0.7)
63.5 (577)
81.22 (0.4)
81.4 (0.6)
Draxx
5
70.8 15.1)
73.3 (0.1)
72.7 (0.2)
63.6 (4.1)
70.8 (0.4)
70.8 (0.4)
67.8 (4.7)
80.6
82.0 (0.4)
10
62.2 (4.4)
63.0 (0.5)
69.6
65.8 (2.9)
69.5 (0.3)
68.8 8 (0.6)
68.1 (3.5)
78.8
82.4 (0.5)
GPT-nco (1.38)
1 73.0 (4.4)
83.5 (1.5)
83.5 (1.5)
(3.1)
72.7
72.7 (1.3)
64.6 (3.2)
84.6 (0.8
84.6 (0.8)
76.1 (3.)
84.1 (1:4)
82.5 (1:7)
65.6 (2.7)
76.4
78.6 (0.8)
64.2 (3.7)
85.5 (0.7)
86.3 (0.9)
DrEEDER
75.7 (35)
90.7 (1.5)
(1:9)
(2.9)
79.5
79.7 (1.5)
70.8 (3:2)
88.2
89.6 (1.9)
10
77.5 (3.))
92.6 (1-3)
(1:3)
(2.0)
82.6
(1.5)
(2:)
91.2
86.7 (1.9)
22

---

Supplementary Materials: FEEDER
versity. Formally, we have:
all, compared using the entire training dataset DraAn the retrieval pool, treating its core set DFREDER as the retrieval pool can improve the LLM performance at most cases. These results are consistent with the analysis reported in Section 5.1, which together verify that our FEEDER col- laborating with various demonstration selectors works well in ICL.
DIV( vn) = SIM((am= -n) max SIM( Pm, Fyy) )
2 EPPLacT
(28)
where vn E DEELECT D'SLECT. and DELEET denotes the set of previously selected instances. We can see that Diversity prefers the instance that is both similar to the test samples meanwhile distant to previously selected instances. v) is hyper-parameter to balance the above two parts. We set 1 in our experiment.
A5.3. Additional Results with Diverse Demonstration Selectors
We report performance comparison results on the reason- ing dataset GSM8K and the semantic parsing dataset SM- CALFlow in Table 2. The corresponding results for ad- ditional demonstration selectors, Clustering and Latent, are provided in Table A6, showing similar trend. To- gether, these results further demonstrate the superiority of our FEEDER framework in the in-context learning setting.
The fourth one is an uncertainty- selector (Köksal et al., 2022), denoted as Uncertainty, which conducts selec- tions according to their uncertainty metric;
The fifth one is clustering-baseee selector (Zhou et al., 2023), denoted as Clustering, which searches demonstra- tions by clustering.
The sixth one uses LLMs as latent variable models (Wang et al., 2024), denoted as Latent, which learns latent variables for down-streaming ICL.
A5.4. Performance Comparison with Different Number of Iterations
We report performance comparison results of FEEDER with different runs It in Figure 3 in Section 5. Here, we further evaluate the performance of FEEDER with different iterations K. The corresponding results are depicted in Figure A6. This figure shows similar trend to Figure 3, which also indicates that our method can allow the LLM to achieve comparable performance with fewer samples.
In our experiment, we run our approximation algorithm for I run to get DEEEDER, and then treat DFEEDER the selection pool for the above demonstration selectors. In our results, we report both the mean and variance of accuracy using 8 different seeds and 5 different permutations of n-shots.
We also want to emphasize that since our pre-selector and pre-selection process novel, we evaluate the performance of FEEDER in an ablation fashion. Specifically, our results (denoted as DYEEDeR in the DD column) can be interpreted as FEEDER + X (where X represents any demonstration re- triever described above), meaning that FEEDER is used for pre-selection of input demonstrations, and X is used to select specific demonstrations considering the target inputs. Our baseline (denoted as PTRAIN in the column) can be formu- lated as X + X. meaning X is used for both pre-selection of input demonstrations and for selecting specific demonstra- tions with regard to the target inputs.
Combining all the above results, we observe that both in- creasing the tree depth (i.e., the number of iterations K) in each round and increasing the number of rounds R con- tribute to reducing the size of the resulting FEEDER set. However, there are notable trade-offs between these two approaches. Increasing the tree depth is computationally more expensive but offers greater robustness, as it mini- mizes the risk of mistakenly filtering out useful samples. On the other hand, increasing the number of rounds is relatively inexpensive but carries a higher likelihood of discarding valuable data points due to less rigorous evaluations.
A5.2. Additional Results with Diverse Datasets
A6. Scaling Up FEEDER into Real-world Applications
We report performance comparison results on text classifi- cation datasets SUBJ, SST-2, and COLA datasets in Table 1. We include the results of FPB, SST-5, and TREC datasets in Table A3, whose trend is consistent with our results in Table 1. These results further verify the superiority of our FEEDER ICL.
16.1. Scaling up FEEDER to larger LLMs.
As the LLM scales up in size (c.g., scaling up to Llama- 65B (Touvron et al., 2023) and Gemma-70B (Team et al., 2024)), the execution of our approximation algorithm for searching PFEEDER can become exceedingly time-consuming, In response to this challenge, we propose a strategy wherein a smaller LLM is employed to generate a FEEDER set, which is then stored and utilized by the larger L.LM. To assess the viability of this approach, we conducted an experiment
Besides three basic demonstration selectors, denoted Ran- dom, Similarity, and Diversity, we also examine the perfor- mance of FEEDER with some recently proposed demonstra- tion selectors, denoted as Uncertainty, Clustering, and La- tent. We summarize the corresponding results in Table A4, whose trend consistent with our results in Table A2. Over-
23

---

Supplementary Materials: FEEDER
Table A6. Performance comparisons on reasoning GSM8K dataset and semantic-parsing SMCAL dataset are conducted in the in co-context learning setting. We report both the mean and variance of accuracy using 8 different seeds and different permutations of n-shots, This table extended from Table 2.
GSM8K
SMCALFlow
VzzN(-)
D
Clustering
Latent
Clustering
Latent
16.17 (115)
16.20 (0.99
20:02 (0.2))
19.54 (0.14)
19.89 (0.66)
20.52 (.15)
22.58 (0.45)
23.05 00.30)
PTATT
5
21.31 (180
23.56 (106)
29.30 (150))
28.65 (0.95)
10
22.52 (1.89)
23.85 (100))
30.12 (1:33
31.11 (0.91)
Gemma-2
17.25 (0.21)
16,68 (0.24)
21.12 (1.75)
20.89 (1.21)
20.68 (0.83)
21.01 (0:55)
22.85 (2.65)
25,03
Dermne
22.55 (1:75)
23.05 (1.7)
31.20 (1:15)
29.54 (4.18)
10
22.75 (185)
24.02 (2:20)
32.10 12,033
32.48 (1:52)
2.95 (0.12)
2.87 (0.2))
9.95
4.78 (0.33)
4.21 (0.25)
10.12
10.14
D
7.21 (0.78)
8.00 (1.05)
12.31 (1.3)
12.15 (1:00)
10
8.05 (1.20)
7.44 (1.25)
14.14 (1.57)
13.99 (1.54)
GPT-3 (68)
4.10 (0.22)
3.25
12.52
11.42 (1.02)
4.26 (0.64)
4.55 (0.82)
11.73 (0.54)
12.05
DEESDER
8.85 (1.28)
8.14 (0.57)
13.58 (1.44)
12.44
10
9.52 (1.38)
8.50 (1.21)
15.08 (1.91)
16.50
3.68 (0.0))
3.98
10.12
9.25 (018)
5.20 (0.38)
5.55 (0.85)
11.05
12.52
7.58 (0.50)
7.52
15.18
15.30
10
9.85 (0.85)
9.21 (0.98)
17.95 (1.25)
18.55
Llamma-2 (B)
4.25 (0.21)
4.17
11.89
12.05
5.88 (068))
6.02
13:03
14,13
DEESDER
8.22 (101))
9.17
18.20 (3,66)
19.66
10
10.17 (1.22)
9.65 (0.83)
22.11 (1.22)
21.25
comparing the performance of using GPT-2 variants and GPT-neo as the LLMs for obtaining FEEDER set, and then we use this set as the retrieval pool to acquire demonstrations for GPT-neo. Results summarized in Table A7 demonstrate that even when DFEEDER is pre-selected by a small LLM, it contributes to improved performance, compared using DIRAI, reported in Table 1. This observation suggests the potential feasibility of employing more compact LLM for pre-selecting DEEEDER to enhance the performance of larger LLM.
unchanged and changed samples can be time-consuming in such dynamic scenarios. To address this challenge, we intro- duce an incremental update algorithm for FEEDER, enabling the efficient re-computation of only the changed portions, including newly added and modified samples.
As depicted in Figure A3, once FEEDER set for the original dataset is generated, treat the unchanged part of plug- and-play plugged data and the LLM as whole (depicted by the dashed box) as new "LLM". Subsequently, we apply FEEDER exclusively to compute incremental data for the changed part, covering newly added and modified data points. This strategy aims to enhance the efficiency and responsiveness of FEEDER in the context of evolving and temporal datasets.
A6.2. Scaling up FEEDER by Incremental Update.
Notice that numerous real-world datasets are temporal and require frequent updates. Re-rumning the tree based ap- proximation algorithm for FEEDER over all samples can be excessively time-consuming. To address this, we design an incremental approach, treating the unchanged portion as a alug-and-play FEEDER set and the LLM a whole, apply FEEDER
A7. Integrating Algorithm 4 in FEEDER
One limitation to directly applying Algorithm 3 or 4 is that DIRAI is too large to be directly used as input demonstra- tions. For this purpose, we incorporate running Algorithm for one round into our FEEDER follows. As shown in Figure A2, we place Algorithm 4 after the demonstration selector filter out the unnecessary parts from the pre- selected data. Concretely, we first select samples from our FEEDER set (i.e., DPEEDER), then filter retrieved samples
forming new "LLM".. Therefore, solely compute incremental data for the modified part, encompassing newly added and modified data points. Also, a significant challenge of FEEDER arises from the tempo- ral nature of many real-world datasets, some of which re- quire frequent updates, potentially on daily basis. The conventional approach of recalculating FEEDER over all
24

---

Supplementary Materials: FEEDER
Table A7. Performance comparisons among using different M. GPT-2, GPT-3, GPT-neo as the hase for acquiring a FEEDER set and using GPT-neo for inference on COLA dataset are conducted in the in-context learning setting. We report both the mean and variance of accuracy using different seeds and different permutations of n-shots.
GPT-2 (0UB)
GPT-3 (6B)
GPT-nco (1.38)
W [-ue(-)
D
R
Random
Similarity
Diversity
Random
Similarity
Diversity
Random
Similarity
Diversity
1
23.7 (8.))
31.0
31.0 0.3)
25.3 (4.)
34.6
34.6 (1.)
28.3 (54)
34.8 (1.)
34,8
2
45.1 (5.0)
49.7 1.4)
46.1 (0:0)
58,5 (32)
57.8 d..)
56.4 (1.0
69.3 Bm
64.7 (1.4)
64.7 (10)
GPT-neo (1.30)
Dyseuea
5
49,4 (4.0)
58.1 (2.))
59.1 0.99
54.6 (38)
64.5 (.D
61.7 (2.4)
68.7 pa
67.2 (2.4)
65.8 (18)
10
59.4 (4.0)
62.4
65.8 0.5)
60.6 (38)
64.7 (..)
66.0 (1.4)
69.8 (2.5)
68.8 (1.4)
68.9
Integrating Exact Extractor of FEEDER into FEEDER
of our bi-level optimization, we first run Algorithm 1 for one get a pre-selected FEEDER set (i.e., DPEEDER) and tuned LLM. Then, update FEEDER set with the tuned LLM and evaluate the performance of LLM in the in-contet learning setting (i.e., few-shot inference). where we allow the LLM to retrieve relevant information from the pre-selected FEEDER set or the training dataset.
Training Dataset
FEEDER Input and Output
Demonstration Retriever
Exact FEEDER Extractor
conditioned
Test Input Data
Frozen Large Language Model
Output to Test Input
Concretely, our baseline is to first tune the LLM on the entire training dataset (i.e., Draum) and then do few-shot inference the test dataset (i.e., DTEST) with DraAI the retrieval pool. In contrast, ours is to first pre-select FEEDER set (i.e., DFEEDER) from DTRAIN and then tune the LLM on DFEEDER. Our FEEDER set is updated according to the tuned LLM using Algorithm for run, and our approach evaluated on DrEST with the updated DEEEDER as the retrieval pool.
Figure A2. Integrating our extraction algorithm for FEEDER (i.c., Algorithm 4) into our in-context learning framework (as introduced in Figure ((a)).
by running Algorithm 4 for one round (treating the set of retrieved samples as DIx). Then, re-select n |Dour| where Dour indicates the output of Algorithm
Scaling Up FEEDER by Incremental Update
We conduct the fine-tuning pipeline in this manner to not only verify the superiority of our FEEDER but also to validate our bi-level optimization framework, which is able to tune both the FEEDER set and the LLMs in each loop.
Training Dataset
FEEDER
Extractor
We list some key hyper-paramene parameters for fine-tuning as fol- lows. The batch size is set as 32, the warm steps is set as 100, the learning rate decay is set 0.01. All our experiments conducted with NVIDIA A100s
conditioned
Frozen Large Language Model
FEEDER for Training Dataset
5 10-5, and the weight
conditioned
FEEDER for Incremental Dataset
Incremental
FEEDER Extractor
A8.2. Additional Results with Diverse Datasets
Training Dataset
We report performance comparison results text classifi- cation datasets SUBJ, SST-2, and COLA datasets in Table 3. We include the results of FPB, SST-5, and TREC datasets in Table A5, whose trend is consistent with our analysis in Section 5.2. These results further verify the superiority of our FEEDER in the fine-tuning setting.
Updated FEEDER
Figure A3. order scale up FEEDER dealing with dynamic data, we introduce an incremental update algorithm. This algorithm designed to efficiently handle changes in training examples, avoiding the need to recompute over un- changed training examples.
applications
A8. FEEDER in Fine-tuning Setting
A8.1. Implementation Details
As summarized in Algorithm in Section 2, we can in- tegrate our FEEDER selection and LLM fine-tuning into a bi-level optimization problem. To evaluate the performance
25

---

Supplementary Materials: FEEDER
Table A8. Performance comparisons hetween using randomly-selected Pau (where [Piuuul =
[Prazee 1) as the base for acquiring a
FEEDER set and using Geo for inference SST-2, SST-5, and COLA datasets conducted in ICL. We report both the mean and variance of accuracy using 8 different seeds and different permutations n-shots.
SST-2
SST-5
COLA
V (-
D
Random
Similarity
Divensity
Random
Similarity
Diversity
Random
Similarity
Diversity
76.8 (3.)
81.5
76.3 (0.)
17.9 (30)
26.9 (..)
22.7 (0.0
30.7 (3.)
55.5
56.5 (04)
DYRATK
65.1 (3.5
80.8 (0.2)
66.1 (0.3)
19.0 (39)
29.2 (0.1)
25.1 (0.1)
40.0 (3.)
55.9 (0.3)
52.5 (0.2)
GPT-neo (1.38)
73.2 (3.6)
77.8 (2.5)
72.4 (2.4)
14.5 (2.)
23.3 (3.))
20.0 (1.0)
28.3 (5.4)
48.8 (3.3)
49.7 (3.
D FRAIM
62.4 (3.5)
77.6 (3.3)
62.2 (2.2)
16.6 (2.8)
25.5
27.7 (2.8)
33.8 (4.4)
50.2 (3.4)
48.7 (2.8)
75.1 (2:8)
82,6 (2.1)
78.5 (1.9)
19.7 (2.7)
27.4
22.8 (1.8)
59.3 (37)
64.7 (1.4)
64.7 (1.)
DFEEZER
73.2 (4.2)
82.9 (2:7)
71.6 (2.4)
19.2 (3.7)
30.2 (1.1)
26.4 (2.4)
58.7 (.2)
67.2 (2:4)
65.8 (1.8)
A9. In-Depth Analysis of FEEDER
Performance comparisons on fine-tuning the LLM with different round R.
A9.1. Performance Gap between using FEEDER and RAN as Pre-Selector
1000+DmmA
1000+DreeDER
2000-Dmm4
As our paper introduces a new pre-selection stage before the demonstration selection process, we also include an ablation study that randomly selects the same number of samples to form randomly selected training dataset, denoted TRAINN which matches the sample size of 'DeEEDeR The cor- responding results are reported in Table A8. A comparison of Table A8 with Tables and A3 indicates that replacing the entire training dataset with randomly selected samples significantly degrades LLM performance. In contrast, the FEEDER-selected samples act as core that summarizes the key information of the entire training dataset. By fo- cusing on high-value samples, our approach enables LLMs to achieve better performance, effectively leveraging the essential knowledge within the dataset.
2000
Accuracy
70
68
10
Figure A4. Performance comparisons on fine-tuning NEO with run- ning our approximation algorithm to pre-select DFEEDer with dif- ferent run R. Our evaluation operates on COLA dataset the zero-shot setting after fine-tuning 1000 and 2000 batches.
A9.2. Performance Gap among Using Different Numbers of Rounds and Different Depths of Tree
FEEDER's performance first rises and then drops with in- creasing tree algorithm runs R. Figure A4 visualizes the impact of employing different numbers of runs of our ap- proximation algorithm (as described in Section 4) to derive DFEEDeR for fine-tuning NEO. For ease of comparison, the results of fine-tuning NEO on Dtrain are also included with the blue line. The observations suggest that fine-tuning with
Performance comparisons on fine tuning LLMs with different iteration K.
1000++YmAN
Accuracy
1000++vesoa
a smaller dataset with high data quality can enhance perfor- mance, but excessively reducing the dataset size may not lead to the desired outcomes. Also, it also indicates that fine-tuning LLMs on "unnecessary" data samples would not help.
2000++1TAM
72.5
70.0
67.
As described in Section 4, we set the tree depth 2 (cor- responding to K = 1), utilizing the one-shot inference capability of LLMs as the sufficiency filter to eliminate un- necessary samples. To further explore the performance im-
1.0
2.5
5.0
Figure A5. Performance comparisons on fine-tuning NEO with run- ning our approximation algorithm to pre-select Dreenen with dif- ferent round K. Our evaluation operates on COLA dataset in the zero-shot setting after fine-tuning on 1000 2000 batches.
'nttps://wwwnww nvidia. com/ cn-n/ data-center/a100/
26

---

Supplementary Materials: FEEDER
pact of varying tree depths, we investigate the performance gap associated with different depths of the tree. Figure A5 visualizes the impact of employing different numbers of runs of our approximation algorithm (as outlined in Sec- tion 4) to derive DPEEDER for fine-tuning NEO. For ease of comparison, the results of fine-tuning NEO on PIRAI are also presented as baseline (depicted by the blue line). The results suggest that fine-tuning with a smaller, high-quality dataset can significantly enhance performance. However, when comparing to Figure A4, we observe that increasing the tree depth leads to more "smoothing" changes in the LLM performance. There are two potential explanations for this phenomenon: (i) The hyper-parameter K, which controls the tree depth, typically changes within a relatively small scope compared to R due to its high computational cost and diminishing returns. While increasing K initially enhances the filtering process by leveraging deeper eval- uations of sufficiency, the marginal improvements in the quality or size of the resulting FEEDER set decreases as K grows. (ii) Increasing the tree depth corresponds to per- forming n-shot inference to satisfy the sufficiency condition described in Eq. (7). This is significantly more challeng- ing than a one-shot inference check and results in a much smaller reduction in the number of samples in the train- ing dataset. (iii) Leveraging the n-shot inference capability of LLMs may yield more robust results. Specifically, the unnecessary samples filtered out by an n-shot sufficiency check are more likely to be genuinely unnecessary, thereby ensuring a higher-quality training set for fine-tuning.
impacted by the duplicated dataset. This is because the re- trieved demonstrations can include duplicates, particularly when using a similarity- are calculated independently for each sample. In contrast, our FEEDER and PEEDER act as "weak" and "strong" filters, respectively, by effectively removing redundant or unnec- essary samples from the input. The "weak" filter provided by PEEEERR ensures sufficiency by eliminating significant portion of redundant data while maintaining the core in- formation needed for the task. On other hand, the "strong" filter represented by DEEEDER not only ensures sufficiency but also guarantees necessity, leading to an even more re- fined dataset that further enhances model robustness and performance. This differentiation highlights the flexibility and effectiveness of our filtering mechanisms in handling noisy or duplicated datasets.
retriever, as similarity scores
A10. Case Study with Artificial Data Points Generated by LLMs
A10.1. Case Study for Transitivity of LLMs
To illustrate the transitivity LLMss, we conducted simple experiment using gpt-3.5-turbo. We prompted the model with the question which place does Jerry lives in? LLM responses with I' m sorry, but I don't have access to personal information about individuals, including your friend Jerry. Then, let DA, Dz. Da denote the city, the country, and the continent he lives in. Then, we can observe that if we tell the LLM about the city (c.g., London), then the LLM can tell about the country (e.g., United Kingdom); if we tell the LLM about the country (e.g., United Kingdom), then the LLM can tell about the continent (c.g.. Europe). Also, if we tell the LLM about the city (e.g.. London), then we ask about the continent, and the LLM also can tell (e.g., Europe). This observation is one case to demonstrate the transitivity of sufficiency.
A9.3. Performance Gap between our Approximately Computed FEEDER Set and Exact FEEDER Set
As described in Section 4, our approximation algorithm en- sures the sufficiency of the resulting FEEDER set but does not guarantee the necessity of each sample within it. To address this, we employ the integration method outlined in Appendix A7, which ensures that the selected demon- strations are both sufficient and necessary. We denote this refined set as DiZEDER- We compare the performance of few- shot preference using DFEEDER, DFFEDER. and DIRAIN, with the results summarized in Table A9. The results indicates that DEEEDE achieves a slight improvement in L.M performance compared to DEEEDER, further validating the effectiivnees of integrating sufficiency and necessity in the pre-selection process.
A10.2. Case Study for Demonstrations Selection Should be Aware of LLMs in Use
Subsequently, we conduct a case study to substantiate the central proposition of this paper: Whether the assessment of the quality of a demonstration should depend on the specific L.M in use.
We consider the factual error made by Google Bard in the first demo Bard said The James Webb Space Telescope took the very first pictures of a planet outside of our own solar system. However, the fact is The Very Large Telescope took the very first pictures of planet outside of our own solar system. Based on the above statements, we produce
We further evaluate the robustness of our DREGEER and DPEEDER by duplicating the training dataset DIRAIN- The du- plicated dataset is denoted as PIRAIN- and the corresponding resulting sets derived using our approximation and integra- tion methods are denoted as DREEDE and D respec- tively. The results of this evaluation are summarized in Table A9. From the table, we observe that both random and similarity- based demonstration retrievers are significantly
235 239006//ooplee----chtbboooo chatbot www.::oott-badddiiit-aaaaa- - error-exoplanet
27

---

Supplementary Materials: FEEDER
Table A9. Results of performance difference hetween using DYZEDEr (derived by using FEEDER version introduced in Appendix A7), we also evaluate the performance of our variants of FEEDER with duplicated training dataset. We evaluate GPT-neo's performance the n-shot settings.
SST-2
SST-5
COLA
(-IM(-)
D
Random
Similarity
Diversity
Random
Similarity
Diversity
Random
Similarity
Diversity
76.8 (35)
81.5
76.3
17.9 (3))
26.9 (1))
22.7 (0.)
30.7 (..)
55.5 (0
56.5 (04)
DYrRIK
65.1 (3:5)
80.8 (0.))
66.1 (0:)
19.0 (39)
29.2 (.1)
25.1 (0.0)
40.0 (35)
55.9 (0)))
52.5 (02)
73.4 (6.6)
78.4 (0.5)
(2.4
14.9 (3.)
22.7 (29)
21.7 (1.))
29.3 (5.4)
49.8 (1.3)
52.7 (..)
PIR
59.4 (3:5)
75.3 (1.3)
64.1 (3.5)
17.5 (2.)
23.5 (2.1)
22.7 (2.))
37.8 (4.2)
51,2 (1.4)
51.0 (2.5)
75.1 (2:89
82.6 (2.1)
78.5 (1.9
19.7 (2.)
27.4 (2.1)
22.8 (1.8)
59.3 (.m)
64.7 (1.4)
64.7 (1.))
GPT-neo (1 38)
Deerze
73.2 (4.2)
82.9 (2.7)
71.6 (24)
19.2 (3.)
30.2 (1.1)
26.4 (2.4)
58.7 (3.2)
67.2 (2.4)
65.8 (1.8)
74.3 (2.9)
(1.)
(1.4
18.2 (2.5)
26.1 (2.1)
21.0 (1.5)
58.3 (.7)
62.5
63.5 (1.1)
D FEEDER
71.1 (32)
80.0 (2.4)
69.8 (2.1)
19.0 (2.0)
29.4
25.3 (..
57.5 (1.0)
65.0
64.1 (1.8)
75.6
83.1 (1.0)
79.0 (LD
20.1 (2.0)
27.8 (2.5)
23.1 (1.2)
60.2 (3.2
64.9 (1.4)
65.0 (1.0)
DiezEn
73.7 (4.1)
82.8 (2.2)
71.8 (2.1)
19.0 (3.0)
31.2
26.3 (2.1)
59.2 (2.7)
67.3 (2.1)
65.4 (2.2)
75.2 (2.0)
82.8 (2.0)
(1.4
19.9 (2:3)
27.0 (2.)
22.7 (1.8)
59.4 (.7)
64.9 (1:2)
64.5 (1.2)
DFEEEE
73.5 (4.2)
82.4 (2.2)
71.3 (22)
18.9 (2.3)
29.9 (U0)
26.2 (1.2)
56.5 (2.))
65.5 (2.2)
64.7 (1.4)
the question What took the very first pictures of a planet outside of our own solar system?
exoplanet for the first time in history.
We also evaluate separately using the above statements as a prompt to gpt-3.5-turbo, and either one of them can lead to the correct answer. We provide detailed answers as follows.
We ask the question to gpt-3.5-turbo. The corresponding answer is:
The first pictures of planet outside of our own solar sys- tem were taken by the Hubble Space Telescope in 2008. The planet, named Fomalhaut b, orbits the star Fomalhaut, which is located about 25 light-years away from Earth in the constellation Piscis Austrinus.
1. The Very Large Telescope took the very first pictures of a planet outside of our own solar system.
2. The Very Large Telescope took the very first pictures of planet outside of our own solar system.
We can see that the above answer is incorrect. We prompt the fact into the LLM, then the answer is:
3. The Very Large Telescope took the very first pictures of a planet outside of our own solar system.
The Very Large Telescope took the very first pictures of a planet outside of our own solar system.
4. The Very Large Telescope took the very first pictures of planet outside of our own solar system.
This indicates that prompting the fact works. We further ask gpt-3.5- 3.5--urbo to generate some sufficient statements for the fact. The following is its output:
5. The Very Large Telescope took the very first pictures of a planet outside of our own solar system.
We can see, in this case, that either one of the statements generated by g.t-3.5-turbb is a sufficient and necessary instance to answer What took the very first pictures of planet outside of our own solar system?
The Very Large Telescope made history by capturing the first-ever ever images of a planet beyond our solar system.
2. Thanks the advanced technology of the Very Large Telescope, scientists were able to capture the first-ever pho- tographs of an exoplanet.
We further evaluate the performance of gpt-j-6b. Without any prompt (i.c., in the zero-shot setting). its answer is:
3. The groundbreaking images captured by the Very Large Telescope marked a major milestone in the study of exoplan- ets.
The Hubble Space Telescope.
We then prompt the above statements provided by gpt- 3. 3.5-uurb into gpt-j-6b, then the answer is:
4. The discovery of a planet outside of our solar system was made possible by the cutting-edg capabilities of the Very Large Telescope.
I. The first-ever images of planet beyond our solar system have been captured by the Very Large Telescope in Chile.
5. The Very Large Telescope's 's ability to capture images of distant celestial bodies allowed scientists to observe an
2. The Hubble Space Telescope.
3. A team of astronomers led by the University of Arizona's
28

---

Supplementary Materials: FEEDER
COLA+NED-1
0.48
[8]
6,400
COLALMED4S
(4)
0.65
5030
0.47
5030
8000
0000
0.59
7000
6.58
6000
5050
5000
6.405
0.56
5000
0.55
4030
4000
4000
3000
2000
643
2000
0.390
0.50
2000
1000
0.42
1000
0.305
1000
1000
Number of FEEDER Iterations
Number of FEEDER Iterations
Number of FEEDER Iterations
TREC+NEO-1
(0 INEC-
(g) TRECANFO-S
0.60
(h) TREC+NEO+10
5000
0.55
5002
0.47
0.48
4000
3000
0.40
0.44
0.22
0.43
2030
2000
0.20
2000
1020
1000
0.15
0
1000
0.41
0.10
0.16
0
0.40
Iterations
Number FEEDER Iterations
Number of FEEDER Iterations
Number FEEDER heeraiion
Figure A6. Performance comparisons for running our approximation algorithm pre-select FEEDER with different iterations K are evaluated in terms of accuracy (denoted ACC) with RAN as retriever and size the resulting FEEDER set (denoted as Size). Fach sub-figure is entitled with Dataset+LI.M bascen shots.
Michael Liu.
The above results show that either one of the latter state- ments is sufficient instance. The results of gpt2-1ard lrge (denoted as LAR in the experiment) show that neither of the 5 statements is sufficient and necessary instance:
The Hubble Space Telescope.
5. The Very Large Telescope.
We can see that only the 1-st statement or the 5-th statement a sufficient and necessary instance. Combining the results can verify the
1. The very large telescope was built in the early 1990s by the European Southern Observatory (ESO) in Chile. The Very Large Telescope.
of jjt and the results gpt-3.5- core insight of our paper: the measurement over prompting demonstration should consider what LLM is in use.
2. The Hubble Space Telescope.
3. The first pictures of planet outside of our own solar system were taken by the Hubble Space Telescope in 1990.
Furthermore, also evaluate the performance of GPT variants used in the paper. We begin by evaluating the zero- shot performance of gpt-neo-1.38 (denoted as NEO in the experiment), and its result is:
4. The Hubble Space Telescope.
5. The very first pictures of a planet outside of our own solar system were taken by the Hubble Space Telescope.
The first pictures of a planet outside of our own solar system were taken by the Voyager spacecraft in 1977.
The results of gpt2-medi (denoted MED in the experi- ment) show that only the 4-th statement is not sufficient and necessary instance:
We then prompt the above 5 statements into gpt-neo-1.3B, then the answer is:
The very large telescope was built in the early 1990s by the European Southern Observatory (ESO) in Chile. The Very Large Telescope.
1. The Very Large Telescope.
2. The Very Large Telescope.
3. The Very Large Telescope.
The Very Large Telescope in Chile.
4. The Hubble Space Telescope.
3. The Very Large Telescope (VLT) in Chile.
5. The Very Large Telescope.
4. The Very Large Telescope.
All the above results verify that quality of one demonstration
The Very Large Telescope in Chile.
29

---

Supplementary Materials: FEEDER
should be LLM-specific, which is the key idea of our paper.
Algorithm 4 Alternative Exact Algorithm for FEEDER
Input:
Training dataset PIRAIN-
Output: Exact FEEDER DEEEDER-
Initialize the number of rounds = 0.
Initialize the set of unnecessary data D.- = O.
repeat
Initialize 1. Initialize Ho D.
Update input data by removing the unnecessary part DIu = "DBAIX" Drr
for each instance (æns Un) E Du do
Check (((x,)lm..EEu)))
L| (amplug((Inn:) Yn
)); C,S (a), C = Din. S (Y (( / (Dm))
1|Diu] ).
If (a) holds, let Hn = {( yn)} and append Hn
to Ao.
repeat
for each pair (H,, Hl) where H4, Hz E 16--111 do Check Y Y(I EDm)) 1 1Dmll unplug Hi U
C, (b), where C DiN and S
(a,,,,,,,. If (b) holds, generate pend to Hr. and assign Hi U Hy; otherwise, append Hi and Hj to Hr.
1|P ).
node H4 U Hy, ap-
Remove H,, H, from - i.c., 16--1 = Hi- - - {H, H,}.
Grow tree from bottom to top via k = k + 1. until | = 1 where we assume the current round is K;
Let HURiCESSARY denote only one element (i.e. the root node) in MKk.
Update the number of rounds, i.e., r 1.
Update D, to include the unnecessary part HUNNCESSARY i.e., D, = D, U HUENCESSARY.
until (HURICESSARY 1;
Assign DFEEDER removing D, from DYRAIN, i.e., DFEEDER DIRAIN - D,..
30


Reviews:
[{"review_id": "bi5NdcG9vq", "reviewer": "Reviewer_oHZY", "summary": "This paper introduces FEEDER (FEw yet Essential Demonstration prE-selectoR), a novel pre-selection framework designed to improve In-Context Learning (ICL) and fine-tuning in large language models (LLMs). The key contribution of FEEDER is a pre-selection stage, where a representative subset of training data is selected based on two new metrics: sufficiency (how well a demonstration represents other samples) and necessity (whether removing a demonstration leads to loss of critical information). The paper proposes a tree-based algorithm to efficiently identify such representative subsets, reducing the computational cost of ICL while maintaining or even improving performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer oHZY for recognizing our method is efficient and practical for real-world deployment, as well as we proposing an interpretable way to explain the demonstration. \n\nBelow, we respond to each of your questions in detail.\n\n**Q1. Should consider larger LLM.**\n\n**R1.** We add **Qwen 2.5 32B model** as our base model. The corresponding results on **GSM8K** and **GPQA** are listed below. Results for the other benchmarks and LLMs will be included in the revision.\n\n**Table R1**. Performance comparisons on GSM8K and GPQA datasets are conducted in the ICL setting. * indicates *p* < 0.05 in significance tests compared to using TRAIN. \n| Model | Mode | n | GSM8K (Random) | GSM8K (Similarly) | GSM8K (Diversity) | GPQA (Random) | GPQA (Similarly) | GPQA (Diversity) |\n|-----------------|--------|----|---------------|----------------|----------------|--------------|----------------|----------------|\n| Qwen 2.5 32B | TRAIN | 1 | 81.0 | 82.1 | 82.1 | 40.1 | 42.0 | 42.0 |\n| Qwen 2.5 32B | TRAIN | 2 | 83.2 | 84.2 | 84.7 | 42.0 | 44.5 | 45.2 |\n| Qwen 2.5 32B | TRAIN | 5 | 85.2 | 89.5 | 89.6 | 43.3 | 46.2 | 46.8 |\n| Qwen 2.5 32B | TRAIN | 10 | 86.2 | 90.4 | 89.9 | 43.2 | 46.8 | 46.7 |\n| Qwen 2.5 32B | FEEDER | 1 | **81.8** | **83.5*** | **83.5*** | **40.5** | **42.7** | **42.7** |\n| Qwen 2.5 32B | FEEDER | 2 | **84.5*** | **85.7*** | **86.0*** | **43.5*** | **45.8*** | **45.6** |\n| Qwen 2.5 32B | FEEDER | 5 | **86.7*** | **90.2** | **90.1** | **44.9*** | **48.0*** | **48.0*** |\n| Qwen 2.5 32B | FEEDER | 10 | **87.6*** | **91.2*** | **90.7** | **44.5*** | **47.8*** | **47.9*** |\n\nThese results demonstrate FEEDER can be extended to LLMs larger than 8B. \n\n\n**Q2. Paper writing issue: lot of blank spaces between sections.**\n\n**R2.** Thanks for your suggestion. We will address these issues in our revision.\n\n**Q3. Lack of Evaluation of Tasks. Authors should consider ICL methods on variety tasks, now only consider 6 text classification tasks.**\n\n**R3.** As mentioned in our experiment section, in addition to 6 text classification tasks, we have included the reasoning task GSM8K and the semantic parsing task SMCALFlow, with corresponding results shown in Tables 2 and A6 in the paper. \n\nFurthermore, we add the science QA task GPQA in our revision. Partial results with the Qwen2.5 32B model are available in Table R1 above, which further demonstrates FEEDER's ability to enhance LLM performance in math and science tasks.\n\nWe are eager to hear your feedback. We’d deeply appreciate it if you could let us know whether your concerns have been addressed.", "claims_and_evidence": "The claim of effectiveness is validated in the experiments.", "methods_and_evaluation_criteria": "The method proposed is applicable to various tasks.", "theoretical_claims": "No theoretical analysis.", "experimental_designs_or_analyses": "Experiments are solid across various datasets.", "relation_to_broader_scientific_literature": "The proposed framework is inspiring and insightful for various tasks in the literature", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths: \n\n1.One of the major limitations of ICL is its high computational overhead, as each new query requires retrieving demonstrations from a large dataset. FEEDER reduces this cost without sacrificing accuracy, making ICL more practical for real-world deployment.\n\n2.The sufficiency and necessity framework provides a formal and interpretable way to measure the contribution of a demonstration.\n\nWeakness:\n\n1.Limited Evaluation of Models: The work mainly considers several LLMs that are very small in parameter size, with the largest being 8B. This is particularly insufficient for the evaluation of ICL tasks, as the more practical LLMs are generally larger.\n\n2.The paper writing could be further improved. There exists a lot of blank spaces in the paper between sections.\nLack of Evaluation of Tasks. For evaluation, the authors mainly consider 6 text classification tasks. However, many ICL methods are evaluated on a variety of tasks, including NLI or translation. The authors should consider adding more tasks.", "other_comments_or_suggestions": "NA", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fgmepuavzb", "reviewer": "Reviewer_s2sv", "summary": "This submission presents a pre-selection framework for in-context learning designed to identify a representative subset of examples from the training set. The proposed framework FEEDER evaluates demonstration examples based on their sufficiency and necessity. Aside from benefiting in-context learning, the framework can also benefit model training, particularly by accelerating the fine-tuning process with the selected subset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer s2sv for recognizing our extensive experiments across multiple benchmarks and LLMs, showing consistent improvements of our framework. \n\nBelow, we respond to each of your questions in detail.\n\n**Q1.1 The differences discussion between this submission and related work [1]. Some overlap in the selection of datasets and LLMs for the experiments.**\n\n**R1.1** Thanks for pointing it out. **Our approach differs significantly from theirs in both theoretical analysis and experimental implementation.**\n\n1. Theoretically, their work investigates the causal relationship between the input X and the output Y, whereas ours analyzes the causal relationship between the input X and the selected demonstration C.\n2. Empricially, their method operates at the demonstration selection stage, while our approach introduces a **pre-selection** stage before demonstration selection. This allows us to pre-compute a core subset of the training dataset as the demonstration selection pool. \n3. Furthermore, their approach focuses on the demonstration selection stage and requires fine-tuning the embeddings of latent variables, which limits its applicability to smaller LLMs. In contrast, our method, which operates at the **pre-selection** stage, does not require fine-tuning LLMs. \n\n**Q1.2 No direct comparison and more detailed discussion between the two methods (FEEDER and [1]) regarding their effectiveness for ICL.**\n\n**R1.2** We directly compared FEEDER with [1] in our experiments, we have included [1] as one of the baseline demonstration selectors, note that we denoted method [1] as **\"Latent\"** (Section 5.1 line 317-319, Section A5.1. line 1186-1188 left). We provided the corresponding results **in Appendix Tables A1&A2, A3&A4, 2&A6**, main findings of effectiveness are listed below:\n\n1. Comparing FEEDER (ours) + SUBJ (Similarity) against TRAIN + SUBJ (Latent [1]), we observed that applying a similarity-based demonstration selector on our FEEDER-generated core subset outperforms their advanced demonstration selector applied to the entire dataset. One possible explanation is that their method does not explicitly consider the relationships among candidate demonstrations. \n2. Comparing FEEDER (ours) + SUBJ (Latent [1]) with FEEDER [1] + SUBJ (Similarity), we find that our method also improves LLM performance. However, the improvements brought by our method and theirs would have some overlap, as both approaches condition the selected demonstrations on the LLMs being used.\n\n**Q2. Clear writing sugguestion:** In Table 1, COLA results on Gemma-2, the 1-shot results under Similarity and Diversity should highlight the D_train values in bold.\n\n**R2.** Thanks for your suggestion. We will address these issues in our revision.\n\n**Reference:**\n\n[1] Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning. NeurIPS 2023\n\nPlease let us know if you have further questions -- thank you so much!", "claims_and_evidence": "Yes, the claims made in the submission supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed method and the evaluation criteria make sense for the problem.", "theoretical_claims": "No issue found.", "experimental_designs_or_analyses": "### Strengths\nThis submission conducts extensive experiments on several commonly used benchmarks and applies the proposed framework to various LLMs, including GPT-2/3, Gemma-2, and Llama-2. The experimental results demonstrate mostly consistent improvements, providing strong evidence of the framework's effectiveness.\n\n\n### Weaknesses\nThis submission is closely related to the paper *\"Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning\" (NeurIPS 2024)*, with significant overlap in the selection of datasets and LLMs for the experiments. However, there is no direct comparison between the two methods regarding their effectiveness for ICL. It would be valuable to include a more detailed discussion and comparison between this submission and the related work.", "relation_to_broader_scientific_literature": "This work builds upon existing research on enhancing ICL through improved sample selection. Its key contribution lies in introducing \"sufficiency\" and \"necessity\" metrics to guide the demonstration pre-selection process. Additionally, it proposes a novel tree-based algorithm to efficiently identify optimal demonstration examples, distinguishing it from prior approaches in the field.", "essential_references_not_discussed": "As mentioned in the \"Experimental Designs Or Analyses\" section above, the authors should include more discussion with *\"Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning\"*. This work is listed in the literature, but without thorough discussion/comparison.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "* In Table 1, for the COLA results on Gemma-2, the 1-shot results under Similarity and Diversity should highlight the D_train values in bold for clarity.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8cateDDqoa", "reviewer": "Reviewer_S3Tb", "summary": "The paper introduces FEEDER, a pre-selection framework designed to improve in-context learning (ICL) in large language models by identifying a representative subset of training data demonstrations. FEEDER uses \"sufficiency\" and \"necessity\" metrics to balance representativeness with redundancy, and employs a tree-based algorithm to efficiently identify the pre-selected examples. This pre-selection set replaces the full training set to improve example selection for ICL or for fine-tuning through a bi-level optimization method. Experimental validation across multiple different LLMs with sizes ranging from 300M to 8B parameters demonstrates that FEEDER can reduce training data size by over 20% while maintaining ICL performance, integrate with other ICL selection strategies for further improved performance, and improve fine-tuning performance compared to other methods. The authors also conduct several ablations on selection strategy and number of rounds to identify the optimal design.\n\n## Update after rebuttal\nThe authors addressed most of my major concerns, so I have increased my score accordingly.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer S3Tb for recognizing our novelty, solid theoretical contributions and good writing. Below, we respond to each of your questions in detail.\n\n**Q1. The paper is not contextualized well within the active learning literature.**\n\n**R1.** We clarify the relationship between our FEEDER method and active learning research line as follows:\n\n1. Similarity: Both FEEDER and active learning methods focus on selecting representative and informative samples.\n2. Difference: Active learning typically selects samples for annotation during training, while FEEDER selects a representative subset from already labeled data to reduce computation.\n3. Our novelty: FEEDER proposes sufficiency and necessity metrics tailored specifically to LLMs, efficiently identifying critical demonstrations and removing redundancy.\n\nWe listed some related active learning works discussed some of them in Appendix A1.3, we will highlight it and add more discussion in revision.\n\n\n**Q2. Performance of the approach is only benchmarked on text classification tasks.**\n\n**R2.** \n1. As mentioned in our experiment section, in addition to 6 text classification tasks, **we have included the reasoning task GSM8K and the semantic parsing task SMCALFlow,** with corresponding results shown in Tables 2 and A6. \n2. Furthermore, we add the science QA task GPQA in our revision. Partial results with the Qwen2.5 32 model are available in Table R1 below. These results further demonstrates FEEDER's ability to enhance LLM performance in math and science tasks.\n\n**Table R1**. Performance comparisons on GSM8K and GPQA datasets are conducted in the ICL setting. * indicates *p* < 0.05 in significance tests compared to using TRAIN. \n| Model | Mode | n | GSM8K (Random) | GSM8K (Similarly) | GSM8K (Diversity) | GPQA (Random) | GPQA (Similarly) | GPQA (Diversity) |\n|-----------------|--------|----|---------------|----------------|----------------|--------------|----------------|----------------|\n| Qwen 2.5 32B | TRAIN | 1 | 81.0 | 82.1 | 82.1 | 40.1 | 42.0 | 42.0 |\n| Qwen 2.5 32B | TRAIN | 2 | 83.2 | 84.2 | 84.7 | 42.0 | 44.5 | 45.2 |\n| Qwen 2.5 32B | TRAIN | 5 | 85.2 | 89.5 | 89.6 | 43.3 | 46.2 | 46.8 |\n| Qwen 2.5 32B | TRAIN | 10 | 86.2 | 90.4 | 89.9 | 43.2 | 46.8 | 46.7 |\n| Qwen 2.5 32B | FEEDER | 1 | **81.8** | **83.5*** | **83.5*** | **40.5** | **42.7** | **42.7** |\n| Qwen 2.5 32B | FEEDER | 2 | **84.5*** | **85.7*** | **86.0*** | **43.5*** | **45.8*** | **45.6** |\n| Qwen 2.5 32B | FEEDER | 5 | **86.7*** | **90.2** | **90.1** | **44.9*** | **48.0*** | **48.0*** |\n| Qwen 2.5 32B | FEEDER | 10 | **87.6*** | **91.2*** | **90.7** | **44.5*** | **47.8*** | **47.9*** |\n\n\n**Q3. No runtime numbers. How do runtimes compare to fine-tuning on the whole dataset?**\n\n\n**R3.**\n\nThanks for your question. We have included the runtime of running FEEDER in Figure A6. Running FEEDER only requires pre-running the inference over half of number of training data points (33.45s for GPT-neo 1.3B on COLA dataset).\n\nFor the fine-tuning, it costs 867.93s for training GPT-neo 1.3B on COLA dataset for one-epoch on the entire data points, whereas training GPT-neo 1.3B on COLA dataset on our FEEDER data costs 707.79s. As training is significantly more costly than inference, performing pre-selection is always cost-effective if around 10\\% can be filtered out. \nFurthermore, we also would like to emphaize that our FEEDER operates at the pre-selection stage which allows for pre-computation over the entire training dataset, and serves various test data. \n\n\n**Q4. Do the authors know why COLA+LAG+2 performance is worse with an increasing number of rounds?**\n\n**R4.** We apologize for mixing the results of different demonstration retrievers when drawing this subfigure in Figure 3. The corrected results for COLA+LAG+2 should be 0.389 (for #Round = 0), 0.412 (for #Round = 1), 0.438 (for #Round = 5), and 0.430 (for #Round = 10), which follow the same trend as other settings.\n\nPlease let us know if we have properly addressed your questions and we are more than happy to discuss more!", "claims_and_evidence": "The paper claims that FEEDER can significantly reduce the size of training data required for selecting examples for ICL without compromising performance. These claims are substantiated by experiments showing consistent results across different model sizes and ICL strategies. However, it does not provide runtime comparisons, which would help to fully validate the efficiency improvements.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the LLM ICL and fine-tuning setting. However, the work would benefit from benchmarking on other types of tasks besides text classification.", "theoretical_claims": "I briefly reviewed all three proofs which seemed correct but it is possible that I missed something.", "experimental_designs_or_analyses": "The experimental design appears sound, with FEEDER tested across a spectrum of LLMs and ICL strategies. The results are extensively documented, particularly in the appendix, which includes detailed methodology and additional results. However, the paper would be strengthened by including actual runtime data and comparisons to fine-tuning on the full dataset, which are crucial for evaluating the practical efficiency gains of the proposed approach (I imagine this is not the case, but if somehow full fine-tuning is faster than searching for this pre-selector set in a large dataset, the only benefit I see to this method is for improving proprietary model performance where fine-tuning cannot be done).", "relation_to_broader_scientific_literature": "The key contributions of the paper have important implications on how to improve example selection for ICL in LLMs, which is a topic of broad interest in the AI community due to the massive increase in interest in and use of LLMs. Their proposed pre-selector method in combination with selection strategies is considerably better than naive selection strategies on the full dataset, which is a new finding that may be of interest in the field.", "essential_references_not_discussed": "The paper does not sufficiently discuss its relationship with established active learning techniques, particularly around the concept of representativeness and informativeness, which have been studied for many years. This might obscure potential similarities and differences that could be crucial for contextualizing the methodological contributions.", "other_strengths_and_weaknesses": "Strengths:\n1. Novel approach for improving ICL in LLMS.\n2/ Strong results across multiple LLM architectures, sizes, and datasets.\n3. Solid theoretical contributions to back methodological design and results.\n4. The paper is well-written and accessible.\n\nWeaknesses:\n1. The paper is not contextualized well within the active learning literature, which has explored similar ideas well before LLMs.\n2. Performance of the approach is only benchmarked on text classification tasks.\n3. No runtime numbers are presented which makes it difficult to assess the practicality of the method.", "other_comments_or_suggestions": "Table 2 should bold all highest numbers, not just highest that are their results.", "questions_for_authors": "1. Do the authors know why COLA+LAG+2 performance is worse with an increasing number of rounds?\n2. How do runtimes compare to fine-tuning on the whole dataset?\n\nStrong responses to the weaknesses as well as these questions could result in an improvement in my evaluation of the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TB1osrcfLz", "reviewer": "Reviewer_mKZf", "summary": "This paper introduces FEEDER, a demonstration pre-selection framework designed to improve the efficiency and effectiveness of large language models (LLMs) in in-context learning (ICL) and fine-tuning tasks. FEEDER identifies a representative subset of training data using two new metrics: \"sufficiency\" and \"necessity.\" By leveraging this pre-selected subset, the approach reduces computational costs while enhancing model performance. Experimental results show that FEEDER can effectively enhance both ICL and fine-tuning.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer mKZf for recognizing our novelty, acknowledging its efficiency and performance improvement, appreciating our comprehensive evaluations on multiple LLMs and tasks, and noting its compatibility with existing demonstration selection strategies.\n\n**Q1. More comparison beyound basic selection methods (Random, Similarity, and Diversity)**\n\n**R1.** Thank you for your issue. We have compared FEEDER with advanced ICL techniques, specifically including the uncertainty-based, the clustering-based, and the recent latent variable-based demonstration selection method proposed in [1]. This method is explicitly denoted as \"Latent\" in our paper (Section 5.1, line 317-319, Section A5.1, line 1186-1188 left).\n\nCorresponding experimental results are reported in Appendix Tables A2, A4, and A6, clearly demonstrating FEEDER’s performance relative to this advanced baseline.\n\nTo further address your suggestion, we will highlight this comparison more explicitly in revision and include additional similarity-based methods.\n\n[1] Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning. NeurIPS 2023\n\n**Q2. While FEEDER shows performance improvements in most cases, the reported gains are not particularly significant.**\n\nWe have added a significance test for the main results on the GSM8K dataset using the Llama-3 8B model in the ICL setting, as shown below. Results for other datasets and LLMs will be included in our revision.\n\n**Table R2**. * indicates *p* < 0.05 in significance tests compared to using TRAIN. \n| Model | Mode | n | GSM8K (Random) | GSM8K (Similarity) | GSM8K (Diversity) |\n|--|--|--|--|--|--|\n| Llama-3 8B | TRAIN | 1 | 78.24 | 79.56 | 79.56 |\n| Llama-3 8B | TRAIN | 2 | 79.55 | 83.40 | 83.67 |\n| Llama-3 8B | TRAIN | 5 | 81.45 | 83.47 | 84.52 |\n| Llama-3 8B | TRAIN | 10 | 82.31 | 84.42 | 84.53 |\n| Llama-3 8B | FEEDER | 1 | **80.23*** | **81.21*** | **81.21*** |\n| Llama-3 8B | FEEDER | 2 | **82.13*** | **84.43*** | **83.88** |\n| Llama-3 8B | FEEDER | 5 | **82.55*** | **85.03*** | **84.77** |\n| Llama-3 8B | FEEDER | 10 | **84.56*** | **85.79*** | **85.43*** |\n\n**Q3. Larger models (>8B)**\n\n**R3.** **We add **Qwen 2.5 32B model** as our base model on **GSM8K** and **GPQA**. Please refer to the R1 in response of Reviewer oHZY.** Results for the other benchmarks and LLMs will be included in the revision.\n\n**Q4. Whether these improvements justify the additional computational complexity.**\n\nThanks for your question. We have included the runtime of running FEEDER in Figure A6. Running FEEDER only requires pre-running the inference over half of the number of training data points (33.45s for GPT-neo 1.3B on COLA dataset).\n\nFor the fine-tuning, it costs 867.93s for training GPT-neo 1.3B on COLA dataset for one-epoch on the entire data points, whereas training GPT-neo 1.3B on COLA dataset on our FEEDER data costs 707.79s. As training is significantly more costly than inference, performing pre-selection is always cost-effective if around 10\\% can be filtered out. \n\nFurthermore, we also would like to emphaize that our FEEDER operates at the pre-selection stage which allows for pre-computation over the entire training dataset, and serves various test data. \n\n**Q5. The concepts of sufficiency and necessity are somewhat abstract, and the paper could benefit from more intuitive, visual examples to help readers understand their impact on demonstration selection**\n\n**R5.** We have provided detailed descriptions of sufficiency and necessity, along with examples, in Appendix A2. Additionally, we also presented a case study in A11.2 to illustrate the impact of sufficiency and necessity on LLM performance.\n\n**Q6. Does FEEDER maintain its effectiveness and selection quality degrade as dataset sizes increase?**\n\n**R6.** Thanks for your question. To examine the impact of dataset size, we evaluate the effect of randomly reducing the SUBJ dataset by 20%, 40%, 60%, and 80% using the Similarity demonstration selector. Results are summarized below.\n\n**Table R3**. \n| Model | Mode | n | SUBJ (20%) | SUBJ (40%) | SUBJ (60%) | SUBJ (80%) | SUBJ (100%) |\n|--|--|--|--|--|--|--|--|\n| Llama-2 (7B) | TRAIN | 1 | 45.72 | 47.21 | 48.25 | 48.44 | 48.50 |\n| Llama-2 (7B) | TRAIN | 2 | 87.58 | 88.96 | 89.80 | 90.53 | 90.76 |\n| Llama-2 (7B) | TRAIN | 5 | 84.25 | 85.46 | 86.09 | 86.56 | 86.88 |\n| Llama-2 (7B) | TRAIN | 10 | 80.10 | 80.68 | 81.52 | 81.45 | 81.37 |\n| Llama-2 (7B) | FEEDER | 1 | **47.50*** | **48.95*** | **49.25*** | **49.50*** | **49.73*** |\n| Llama-2 (7B) | FEEDER | 2 | **89.80*** | **90.94*** | **91.06** | **92.17*** | **92.54*** |\n| Llama-2 (7B) | FEEDER | 5 | **86.28*** | **87.05*** | **87.56*** | **87.89*** | **87.95*** |\n| Llama-2 (7B) | FEEDER | 10 | **84.50*** | **85.14*** | **85.67*** | **85.88*** | **85.87*** |\n\nThe results above demonstrate that our FEEDER is effective across different dataset scales.\n\nPlease let us know if you have further questions -- thank you so much!", "claims_and_evidence": "The author validate the effectiveness of the method using a small dataset and a small-scale model. However, the current experiments are not sufficient, and larger-scale models need to be used. Additionally, the performance improvement of the model is not significant enough.", "methods_and_evaluation_criteria": "The author proposed a new method for selecting data. While it has some scalability, the performance is not particularly impressive. Additionally, the size of the tested model is still too small.", "theoretical_claims": "I check the rationality of the theoretical claims, and I haven't found unreasonable aspects so far.", "experimental_designs_or_analyses": "I carefully examine the validity of the experiments, and the current experimental setup is reasonable.", "relation_to_broader_scientific_literature": "The key difference between this work and previous studies is the introduction of sufficiency and necessity metrics for data selection. This approach effectively identifies examples that are both representative and maximize information coverage, thereby improving the model's efficiency and performance.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The introduction of FEEDER as a demonstration pre-selection framework, along with the novel sufficiency and necessity metrics, offers a fresh perspective on data selection.\n2. By reducing redundant data, FEEDER enhances the efficiency of ICL while improving performance.\n3. The study evaluates multiple LLMs ranging from 300M to 8B parameters across multiple tasks (text classification, reasoning, semantic parsing), making the results convincing.\n4. FEEDER is compatible with various existing demonstration selection strategies.\n\nWeaknesses:\n1. The current comparisons primarily involve Random, Similarity, and Diversity selection strategies. However, it remains unclear whether combining FEEDER with previous ICL approaches would yield better results. A more informative comparison would involve integrating FEEDER with advanced ICL techniques instead of only evaluating it against basic selection methods.\n2. While FEEDER shows performance improvements in most cases, the reported gains are not particularly significant. The paper should further analyze whether these improvements justify the additional computational complexity.\n3. This study evaluates models with up to 8B parameters, but does not include state-of-the-art LLMs such as GPT-3 (175B), GPT-4, or the larger LLaMA model. Testing on larger models would better demonstrate the effectiveness of FEEDER.", "other_comments_or_suggestions": "1.\tThe paper primarily compares FEEDER with a few selection methods (e.g., similarity and clustering-based approaches). Including comparisons with more recent approaches, such as reinforcement learning-based selection methods, would provide a more comprehensive evaluation.\n2.\tThe concepts of sufficiency and necessity are somewhat abstract, and the paper could benefit from more intuitive, visual examples to help readers understand their impact on demonstration selection.", "questions_for_authors": "1. Does FEEDER maintain its effectiveness as dataset sizes increase, or does its selection quality degrade with larger datasets?\n2. How would FEEDER perform on significantly larger models like GPT-4 or LLaMA-3 65B?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the inefficiency of standard few-shot in-context learning (ICL). The authors propose FEEDER, a framework that performs a pre-selection step before ICL inference. FEEDER identifies a core subset of the training data (the "FEEDER set") containing the most informative demonstrations. Informativeness is defined using two novel metrics: sufficiency (how well a demonstration represents others) and necessity (the information loss if a demonstration is removed). A tree-based algorithm is proposed to efficiently find this FEEDER set. Once identified, this smaller FEEDER set replaces the original large dataset as the pool from which demonstrations are selected during ICL inference, leading to improved efficiency and potentially better accuracy. 

Overall, the paper received a mixed review. On the positive side, reviewers acknowledged the importance of the problem, the soundness and relevance of the sufficiency/necessity framework, sensible method and convincing evaluation. On the negative sides, reviewers questioned the limited evaluation on task/models (addressed in the rebuttal to a large extent), comparison to important baselines (again, the authors mentioned that some of the baselines requested are already in the paper, and they clarified the relationship of FEEDER w.r.t. active learning) -- the negative reviewers did not respond to the rebuttal, and the AC has checked the authors' response and agree that their original concerns have been largely addressed. As such, the AC recommends acceptance, and encourages the authors to incorporate all promised changes into the camera ready version of the paper. On a side note, the authors should also consider incorporating some qualitative discussions of some more recent works discussing similar phenomenon (e.g., that a small set of demonstrations can perform on par with a large number of examples) like [1] published in ICLR 2025.

[1] Wan, X., Zhou, H., Sun, R., Nakhost, H., Jiang, K., & Arık, S. Ö. (2025). From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generation. arXiv preprint arXiv:2502.00330.
Decision: Accept (poster)