Mixture of Demonstrations for In-Context Learning

Authors: song  wang
zihan  chen
chengshuai  shi
cong  shen
jundong  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 22:14:04

Abstract: In-Context Learning (ICL) empowers Large Language Models (LLMs) to tackle various tasks by providing input-output examples as additional inputs, referred to as demonstrations. Nevertheless, the performance of ICL could be easily impacted by the quality of selected demonstrations. Existing efforts generally learn a retriever model to score each demonstration for selecting suitable demonstrations, however, the effect is suboptimal due to the large search space and the noise from unhelpful demonstrations. In this study, we introduce MoD, which partitions the demonstration pool into groups, each governed by an expert to reduce search space. We further design an expert-wise training strategy to alleviate the impact of unhelpful demonstrations when optimizing the retriever model. During inference, experts collaboratively retrieve demonstrations for the input query to enhance the  ICL performance. We validate MoD via experiments across a range of NLP datasets and tasks, demonstrating its state-of-the-art performance and shedding new light on the future design of retrieval methods for ICL.

Main paper: 
 introduction: 
 Large language models (LLMs) have demonstrated remarkable potential across various natural language processing (NLP) tasks [62, 43, 6] , such as semantic parsing [22, 53] and commonsense reasoning [42, 61] . However, the large parameter size of these models often comes with significant costs for retraining or fine-tuning when they are applied to novel tasks [16, 25, 59] . Fortunately, as LLMs increase in size, they acquire the In-Context Learning (ICL) capability [50, 47] , wherein the model can achieve significant performance improvements when provided with a limited number of demonstration examples during inference, without updating model parameters [5] .Although ICL has exhibited promising performance in various tasks, this capability also introduces a challenge related to robustness [5, 15, 36, 29] : ICL is highly sensitive to the selection of in-context demonstrations, and suboptimal selections could even lead to worse performance than random selections [34, 27, 26] . Recently, extensive research efforts have been dedicated to improving the selection of in-context demonstrations [47, 35] . For example, learning-free methods directly select demonstrations according to the similarity of demonstration embeddings from a pre-trained encoder [55] . Learning-based methods generally optimize a retriever based on feedback or supervision signals (e.g., output probabilities) from LLMs, and demonstrate superior performance compared to learning-free methods [34, 57] .However, the performance of these approaches is limited by two crucial challenges. (1) Large Search Space. As ICL requires the retrieval of multiple demonstrations from a sample pool, it is difficult to retrieve the optimal set of demonstrations from such a large search space, especially when the available sample pool is more extensive. Moreover, the total number of possible retrieval outcomes grows exponentially as the size of the retrieved set increases, rendering the retrieval even more challenging. (2) Insufficient Optimization. Existing learning-based works generally optimize the retriever model by preferring demonstrations that could aid model predictions. However, the common practice of randomly sampling a demonstration set in each training step could be suboptimal. For example, the samples in the entire set may contribute differently to or even impair the model predictions, but they are assigned the same retrieval scores, which could make the optimized model prefer the less helpful demonstrations.To address the above challenges, we propose a novel demonstration retrieval framework named MoD (Mixture of Demonstrations) that effectively navigates the sample pool while enabling precise optimization for beneficial demonstrations. First, to deal with the challenge of large search space, we leverage the mixture of experts (MoE) mechanism [18, 48] and partition the demonstration pool into distinct groups, each considered as an expert. Subsequently, we train an individual retriever model for each expert to prioritize helpful demonstrations, and during inference, we aggregate demonstrations retrieved from experts as the final demonstration set. Such a design largely reduces the search space of retrieval while also ensuring diversity in the demonstration set without sacrificing performance. Second, to tackle the problem of insufficient optimization, we propose a novel training strategy drawing inspiration from coordinate descent (CD) [54] , which iteratively optimizes each dimension of a variable while fixing other dimensions. Inspired by CD, we propose an expert-wise training strategy that learns the retrieval score of any candidate demonstration while pairing it with demonstrations selected by all experts. These demonstrations are fixed while we only optimize one candidate demonstration at each step. As a result, we could ensure that all demonstrations used for optimization are optimal (except the candidate demonstration), thereby mitigating the disruption from unhelpful demonstrations. In summary, our contributions are as follows:• We propose a novel demonstration retrieval framework MoD that learns multiple experts to collaboratively select demonstrations across the entire sample pool.• Our design of multiple experts and expert-wise training could deal with the challenge of large search space and insufficient optimization, which have not been thoroughly investigated before.• We conduct extensive experiments across a variety of NLP tasks to evaluate our framework in retrieving suitable demonstrations for ICL. The results demonstrate the superior performance of MoD over other state-of-the-art baselines. 
related works: 
 In-Context Learning. In-context learning (ICL) empowers large language models (LMs) by providing them with a few input-output examples as demonstrations [5] , enabling them to 'learn by analogy' and proficiently undertake intricate tasks, such as machine translation [1, 39] , data generation [56] , and others [49, 13, 30] . Although successful in many aspects, the efficacy of ICL is frequently hindered by its sensitivity to the selection of in-context examples, prompting research into optimized selection strategies [26, 27, 63] . These selection techniques can be classified into learning-free and learning-based methods. Learning-free methods typically employ heuristic criteria for selecting demonstrations without directly querying LLMs during the selection process. These criteria include assessing semantic similarity between testing examples and demonstrations [26] , measuring entropy [27] , and ensuring diversity [41, 21, 1] . However, these methods do not actively engage with LLMs and often result in suboptimal performance. In contrast, researchers leverage feedback from LLMs as supervision signals to explore more advanced learning-based methods. For instance, EPR [34] trains a singleton example scorer using contrastive learning with signals from LM inference. Furthermore, UDR [23] extends EPR in a unified formulation. These methods, however, do not account for interactions between in-context examples. In comparison, CEIL [57] tackles this challenge by jointly modeling the selection of the exemplar set and training a retriever to score the exemplar set. Nonetheless, CEIL faces challenges such as exponential search space in the size of the demonstration pool. To address this, it narrows down the candidate space using a K-NN retriever before the selection stage, potentially leading to suboptimal demonstration sets due to insufficient exploration of the entire demonstration pool.The idea behind Mixture of Experts (MoE) is to have a set of expert networks, each specializing in a particular task or a subset of the input space [38, 45, 19] . Wang et al. extended this paradigm to the prompt optimization task, achieving substantial performance improvements [48] . However, their approach overlooks the potential benefits of leveraging multiple expert collaborations. We extend the MoE framework to tackle the demonstration selection problem, aiming to effectively navigate the demonstration pool while considering the interplay among in-context examples. 
problem setup: 
 Given a set D = {e i } n i=1 = {(x i , y i )} n i=1 of input-output pairs (referred to as the demonstration pool), and a test example (x test , y test ) ∈ D test , the strategy of ICL is to retrieve a set of demonstrations S(x test ) ∈ {S|S ⊆ D, |S| = L}, which serves as the input conditioning for a pretrained LLM M to make predictions on x test :ŷ = argmax y P M (y | S(x test ), x test ).(EQUATIONwhere P M measures the likelihood of a candidate answer y generated by M. We aim to provide the proper demonstration set S(x test ) for each x test that helps M make good predictions on x test . However, the search space could be |D| L , which is computationally infeasible for an exhaustive search.To deal with this, existing works have proposed to learn an embedding for retrieval or narrow down the search space with a KNN retriever. Such strategies are suboptimal as they ignore demonstrations that are far from the input, in terms of embedding similarities. However, such demonstrations could still be useful for ICL [21, 41] .We introduce our proposed method as the Mixture of Demonstrations (MoD) and outline its demonstration assignment, expert's retriever training, and inference as follows.To address the aforementioned challenges of an extremely large search space, we propose a novel mixture of demonstration (MoD) framework based on the mixture of experts (MoE) paradigm [18] . Specifically, we partition the demonstration pool into distinct groups, each governed by an expert.For each expert, we train a unique retriever, implemented as a scorer function, to select suitable demonstrations for the test example x test . During the training of the experts' retrievers, we consider the interactions among demonstrations in the prompt. With our MoD framework, the demonstration selection process for ICL is transformed into an expert assignment problem along with an individual retrieval task for each of the assigned experts. The optimal retrieved set of demonstrations for x test could be achieved by selecting demonstrations from the most relevant experts, represented as follows:EQUATIONHere S(x test ) represents the set of demonstrations selected for the test example x test . C is the total number of experts into which the dataset D is divided, and C i represents the distinct demonstration set of the i-th expert. S i is the set of demonstrations selected from C i while maximizing the sum of values given by the scorer function g i (•) of the i-th expert, which measures the importance of the demonstration e from C i with respect to the test example x test . h(C i , x test ) is a function that determines the relevance between x test and each expert C i and also indicates the ratio of demonstrations from this expert in S(x test ). With Eq. (2), we could select the most helpful demonstrations from relevant experts, regarding any input test sample x test . Our demonstration selection strategy of using multiple experts could efficiently cover the entire search space without high computational costs, as specific experts will be omitted during retrieval when ⌊h(C i , x test ) * L⌋ = 0. Our strategy also enables the retrieval of dissimilar samples that could be helpful for ICL, as we cover multiple experts across the entire search space. In concrete, by optimizing the scorer function g i of each expert, we could retrieve the demonstration set S(x test ) that could maximally aid in ICL for x text . In the following, we introduce details of the two-step retrieval process in our framework: 1) Demonstration Assignment and 2) Expert Retrieval. Before training, we first assign a set of demonstrations to each of the experts. Then we perform expert-wise training to obtain a retriever model for each of the experts. We ensure that the subset Sis optimally selected from all experts to filter out unhelpful demonstrations during training. During inference, multiple experts will provide demonstrations for predictions on the input query.We first introduce the strategy of partitioning the entire demonstration set and assigning the corresponding demonstrations to experts. Previous studies have demonstrated that selecting demonstrative samples x i with smaller distances between them and x test in the sentence embedding space can enhance the effectiveness of ICL [26, 41, 34] . Based on these findings, we propose to ensure that demonstrations assigned to a specific expert should be similar. Therefore, we employ the K-means clustering approach to partition the demonstration setD = {e i } n i=1 = {(x i , y i )} n i=1 into C clusters {C 1 , C 2 , ..., C C } basedon embedding distances, and demonstrations in each cluster are assigned to a specific expert. In this way, each cluster comprises semantically similar demonstrations, from which the corresponding expert selects suitable ones for x test . Specifically, we utilize the widely-used Sentence-BERT model [32] as the embedding model f (•) [41, 34] . To adaptively obtain the optimal number of clusters C, we combine the within-cluster sum of squared errors with a regularization term to constrain C. The criterion can be expressed as follows:EQUATIONHere, C k is the k-th cluster, and µ k denotes its centroid. With the obtained clusters, given an input test sample x test , we compute its similarity to the centroid of any expert i in the embedding space as follows:EQUATION) Here f (x) is the learned embedding of sample x. With the obtained scores regarding each expert, we could determine the number of demonstrations selected from each expert as | S i | = ⌊h(C i , x test ) * L⌋.Optimization Objective. In this subsection, we introduce our approach for training a demonstration retriever, implemented as a scorer function g i (•), for each expert i. It is essential that the primary objective for the retriever is to select appropriate demonstrations based on the few-shot pattern in ICL. Therefore, considering the interaction among demonstrations, the search space can be as large as |D| L , where L is the number of demonstrations used in ICL [57] . To mitigate the computational burden associated with such a large search space, we draw inspiration from the concept of coordinate descent (CD) [54] . CD optimizes a variable iteratively by fixing most dimensions of the variable vector at their current values and approximately minimizing the objective. In this manner, the optimization problem in each step has fewer dimensions, making the optimization easier compared to directly optimizing all dimensions. In concrete, we propose the following optimization objective for training the scorer function g i (•) of expert i:EQUATIONwhere ϕ * i represents the optimal parameters of g i (•). L is an evaluation criterion and can encompass various metrics, such as the log-probability of the output, i.e., L(y, x, S) := P M (y | S, x), indicating the utility of S for decoding the target answer [57] EQUATIONIn other words, akin to how CD optimizes one component while fixing others, our objective is to optimize g i such that we can retrieve the demonstration (i.e., e i * test ) that contributes the most to ICL when L -1 demonstrations (i.e., S -(x test )) are already retrieved and fixed. After iteratively optimizing scorer functions of all experts, i.e., {g i } C i=1 , we can retrieve the proper S(x test ) by Eq. ( 2) for LLM predictions. We outline the training process in Algorithm 1, with each phase introduced in the following sections.Training Data. The training data construction process is detailed in Phase 1 of Algorithm 1. At the t-th epoch, we first sample a batch of samples d (t) ⊂ D. For each sample (xEQUATIONi ) as follows:EQUATION)For experts that contribute to the prediction for x(t) i , i.e., | S j-| > 0, we use g (t-1) j to retrieve K candidate demonstrations E j (x (t) i ) = {e k j } K k=1 with the top-K highest scores from the unselected demonstration set C j \ S -(x (t)i ) of each expert j. The K candidate demonstrations are obtained as follows:EQUATIONi , e), where |E| = K.These demonstrations will be used as the candidate demonstration set during the following optimization step.Few-shot Scoring. Once we retrieve the top-K demonstrations E j (xEQUATIONi ) in the batch d (t) , we use the criterion L to score each demonstration for its helpfulness in ICL and use the scores as supervision for optimization. In this work, we employ the log probability of the output as the metric and query the LLM M for the feedback in the few-shot pattern, i.e., using multiple demonstrations as additional input. For any candidate demonstration e k j , k = 1, 2, . . . , K, we score it as s(e k j ) = L(yEQUATIONwhich represents the probability of the LLM M generating the correct prediction sequence, conditioned on the selected demonstrations and the input query. Previous works show that this score serves as a suitable proxy for the utility of a demonstration at inference time [34, 57] .After scoring the K candidate demonstrations, we include the tuple (x(t) i , {e k j } K k=1 , {s(e k j )} K k=1 ) in the expert j's training set D train jfor updating its scoring function at the t-th epoch, i.e., g (t) j (•) . We iteratively apply the above process for all samples (x(t) i , y (t)i ) in the sampled batch d (t) and employ contrastive learning for model updates.Training Loss. Our training procedure draws inspiration from the concept of contrastive learning [20] that has proven to be effective when it is necessary to compare the performance of different samples. In our work, each scorer function g comprises two encoders: M d for demonstration encoding and M q for query input encoding. Both encoders are initialized with the bert-base-uncased model [8] , and their output vectors represent the embeddings of the sequences. In this section, we detail the training process for expert j as in Phase 2 of Algorithm 1. We omit the subscript j for simplicity.Given a tuple (x(t) i , {e k } K k=1 , {s(e k )} K k=1 )for optimizing an expert, we construct its training set by including one positive and 2B -1 negative demonstrations, denoted as (x(t)i , e pos , e 1 neg , e 2 neg , ..., e 2B-1 neg ), where B is the batch size. The positive demonstration e pos is EQUATIONIn this manner, we further filter out the demonstrations with low few-shot scores, indicating that they are not suitable for acting as a demonstration accompanied with other optimal demonstrations in S -. Negative samples (e 1 neg , e 2 neg , ..., e 2B-1 neg ) include: (i) one hard demonstration e hard = argmin e∈{e k } K k=1 s(e); (ii) B -1 positive demonstrations from the other B -1 samples in d (t) ; and (iii) B -1 hard negative demonstrations from those samples. The score returned by g is defined as g(x, e) = ⟨M d (e), M q (x)⟩. We then propose the contrastive learning loss and use it to update g:L(x (t)i , e pos , e 1 neg , e 2 neg , ..., e 2B-1 neg ) = -log exp(g(x(t)i , e pos )) exp(g(x(t) i , e pos )) + 2B-1 j=1 exp(g(x (t)i , e j neg )).(11) Intuitively, the above loss will assign higher scores for demonstrations that are more helpful, when other demonstrations are already optimal. Thus, our expert-wise training could alleviate the impact of unhelpful demonstrations during optimization.In the inference stage, we select demonstrations for an input query x test according to Eq. ( 2), and obtain the prediction ŷ = argmax y P M (y|S(x test ), x test ) given by LLM M. Although we update the retriever models independently for each expert, each retriever model is designed to select demonstrations that benefit ICL in few-shot scenarios, i.e., using a set of demonstrations as additional input. This is ensured because the supervision scores in Eq. ( 9) for training the retriever models are generated in a few-shot pattern with a set of demonstrations. For the optimal retriever models {g * j } C j=1 , each model essentially solves the problem: "Given a good demonstration set S * -of size L -1, which demonstration should the expert choose to make the best prediction in L-shot ICL?" Consequently, for any input query, the experts in MoD can collaboratively retrieve a set of demonstrations that could most effectively aid in making accurate predictions. We compare it against both learning-free and learning-based retrievers. Learning-free methods include Random, TopK-BM25 [33] , TopK-Contriver [17] , and TopK-SimCSE [11] . Learning-based methods include EPR [34] and CEIL [57] . We provide more details in Appendix B.2.Datasets. To ensure a fair comparison between our framework and other baselines, following CEIL [57] , we conduct experiments on a variety of datasets, involving both classification and generation tasks. For the evaluation on classification datasets, we measure the accuracy of the output regarding the correct answers. For evaluation on generation tasks, we adopt the metrics of Exact Match (EM) scores for all generation datasets except Break, for which we use LF-EM [12] that additionally considers semantic equivalence. Following CEIL [57] , we present the final results based on the validation set as test sets are unavailable for specific datasets.Implementation Details. To keep consistency with CEIL [57] and EPR [34] , we primarily use GPT-Neo [4] , a 2.7-billion-parameter language model trained on The Pile [10] , which is an 825GB text corpus collected from various high-quality resources. In Sec. 4.5, we additionally consider three models: GPT2-XL [31] with 1.5 billion parameters, LLaMA-7B [44] with 7 billion parameters, and GPT3.5 [5] with a significantly larger parameter size. The number of in-context demonstrations in our experiments is set as 50, while we truncate this number when the combined length exceeds the maximum context size of LLMs for each task. The ultimate average number of in-context demonstrations used in each task is provided in Table 1 . We provide details of the settings in Appendix B.3. 
comparative results: 
 In Table 2 , we report the results of our framework MoD and other baselines on two sets of datasets: six classification datasets and six generation datasets, covering seven tasks. From the results, we could obtain the following observations: (1) Superior Performance. MoD demonstrates superior performance across a diverse set of tasks, both in classification and generation, as evidenced by the highest average score (58.63%) compared to competitive baselines CEIL (56.76%) and EPR (53.37%). This indicates that MoD is more effective in leveraging in-context demonstrations to enhance task performance. (2) Better on Classification. Compared with CEIL, MoD generally achieves higher performance gain on classification tasks than on generation tasks (Average ∆ Gain 2.10 on classification tasks v.s. Average ∆ Gain 1.64 on generation tasks). This is because our design of the mixture-of-expert architecture enables the selection of demonstrations with a large distance in the embedding space to the query. As classification tasks could be more easily affected by several demonstrations, these selected demonstrations could potentially carry helpful information for inference on the query, while not necessarily being similar to the query in the embedding space. (3) A critical advantage of MoD is its capability to collaboratively select demonstrations from multiple experts, such that these demonstrations are maximally helpful when the other demonstrations in the selected set are also optimal. To evaluate whether the demonstrations retrieved from various experts could be entirely helpful for ICL, we conduct experiments on two semantic parsing datasets derived from the original SMCalFlow and GeoQuery datasets and processed by CEIL [57] . Specifically, the inference on queries in these datasets requires the precise retrieval of multiple specific demonstrations.In other words, without precise retrieval, it is particularly difficult to answer these queries. We provide more details of the dataset settings in Appendix B.1. Following CEIL, we utilize the same trained retriever models of experts as used in Sec. 4.2. From the results presented in Table 3 , we could obtain the following observations: (1) The performance of MoD is consistently superior compared to other baselines across datasets. Notably, these tasks require the retrieval of compositional demonstrations that are all important but may not necessarily be similar to each other. In this regard, our proposed MoD framework directly retrieves a diverse set of demonstrations, which significantly enhances the efficacy of few-shot ICL, compared to other basins in this scenario. In this subsection, we aim to explore the capability of our MoD framework in scenarios where the number of ICL demonstrations selected from the training set is decreased. This is critical for evaluating the practicality of MoD, as it could be challenging to leverage sufficient demonstrations, due to the lack of data or limitation of model sizes. Particularly, we conduct experiments with different numbers of in-context demonstrations on two classification datasets SST-5 and CMSQA, and two generation datasets GeoQuery and MTOP. We present the performance of MoD over the state-of-the-art baseline CEIL in Fig. 2 . From the results, we could observe that particularly on classification datasets SST-5 and CMSQA, our performance improvements over CEIL are more significant. This indicates that for classification tasks that require diverse knowledge, our strategy using multiple experts could effectively retrieve crucial demonstrations, which could provide sufficient knowledge even with a limited context length. The performance improvements are relatively consistent on the two generation datasets, i.e., GeoQuery and MTOP. This is because the generation tasks are generally more difficult, and thus require a larger demonstration set. As a result, the advantage of MoD in retrieving diverse knowledge becomes less substantial for model performance. In this subsection, we aim to evaluate the robustness, especially the generalizability and transferability of our method MoD to various LLMs. Particularly, our experiments are designed to test whether the retriever models in our MoD framework trained on one LLM could be transferred to other LLMs. Conducting experiments to answer this question could help investigate the applicability of MoD when deployed in realistic scenarios, where LLMs could have different architectures and parameter sizes. Specifically, we use the retriever models trained on GPT-Neo to select demonstrations for the other two models: GPT2-XL with a slightly smaller parameter size and GPT3.5 with a significantly larger parameter size. We present the results of MoD over TopK-BERT in Table 4 . From the results, we could observe that (1) The retriever models trained on GPT-Neo exhibit competitive performance when transferred to other LLMs across various datasets. This indicates the transferability of MoD, especially its scalability to large black-box models like GPT3.5. (2) The performance improvements on GPT3.5 are less competitive. This is because due to the powerfulness of GPT3.5, simple methods like TopK-BERT already perform well. Nevertheless, MoD could still improve performance by retrieving better demonstrations. (3) When transferring the retriever models trained on LLaMA-7B to smaller models, the performance improvements are less obvious, probably due to the discrepancy between LLMs in understanding demonstrations. In this subsection, we aim to evaluate the specific benefits to performance brought by different modules and designs in our MoD framework. In particular, we evaluate the performance of our MoD framework on four datasets: SST-5, CM-SQA, GeoQuery, and MTOP, distinctly covering two classification tasks and two generation tasks. As presented in Fig. 3 , we investigate the impact of two key components of our framework: the mixture-of-experts design (MoD w/o E) and the expert-wise training (MoD w/o C). The first variant of our ablation study involves removing the mixture-of-experts design, which results in a significant drop in performance across all datasets, highlighting the importance of leveraging multiple experts for robust prediction. The second variant excludes the expert-wise training process, which leads to a moderate decrease in performance, indicating its role in improving the model's performance. Moreover, the results demonstrate that removing the mixture-of-experts design is particularly detrimental for classification tasks, such as SST-5 and CMSQA. Therefore, this underscores its critical contribution to retrieving more diverse and complex demonstrations, which are more crucial for classification tasks. 
conclusion: 
 In this work, we propose to divide the demonstration retrieval process for in-context learning into multiple parts, each governed by an expert to select from its own sample pool. Our proposed MoD framework further performs expert-wise training to filter out unhelpful demonstrations when optimizing each candidate demonstration. We conduct extensive experiments across a variety of datasets and tasks, and the results validate the superiority of MoD over other baselines. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We provide the standard deviation in Sec. 4.8. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We provide the broader impacts in Appendix G Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to 
 
 Appendix: We provide the detailed process of expert-wise training in our MoD framework as follows. Query LLM M and get the feedback s(e k j ) = L(y 5 .(t) i , x (t) i , {e k j } ∪ S -(x (t) i )), k ∈ [K]; 13: D train j ← D train j ∪ {(x (t) i , {e k j } K k=1 , {s(e k j )} K k=1 )};• SST-5 [40] : A fine-grained sentiment classification benchmark with five classes: 'very positive', 'positive', 'neutral', 'negative', and 'very negative'. • MRPC [9] : Determine if two sentences are paraphrases from one another or not.• MNLI [51] : A collection of sentence pairs with textual entailment annotations, where the task is to determine if a sentence entails, contradicts, or is unrelated to a given hypothesis. • QNLI [46] : A NLP inference dataset consists of question-paragraph pairs. The dataset was converted into sentence pair classification by pairing each question with each sentence in the context, then filtering out pairs with low lexical overlap. The task is to determine if the context sentence contains the answer to the question. • CMSQA [42] : Also referred to as CommonsenseQA, this dataset involves multiple-choice questions and necessitates various types of commonsense knowledge to determine the correct answer.• HellaSwag [61] : HellaSwag is a dataset for studying grounded commonsense inference. Each question comes with four answer choices predicting what might happen next in a given scene. The correct answer is the actual subsequent event, while the three incorrect answers are adversarially generated and verified by humans.• WebQs [3] : Also known as WebQuestions, this dataset comprises question-answer pairs sourced from the web. Questions are selected using the Google Suggest API, and the benchmark uses Freebase as the knowledge base.• NL2Bash [24] : The goal of this benchmark is to map sentences to formal Bash commands of their underlying meaning.• GeoQuery [60, 37] : It contains natural language questions about US geography. Shaw et al. [37] further generate multiple splits focusing on compositional generalization. In addition to the original Standard split, it contains three additional splits: • Break [53] : Break is a question understanding dataset for complex questions reasoning. It annotates NLP questions with their question decomposition meaning representations. We use the low-level BREAK subset as in previous works [34, 57] .• MTOP [22] : A multilingual task-oriented semantic parsing dataset covering 6 languages and 11 domains, which contain compositional representations that allow complex nested queries. We use the English subset of MTOP as in previous works [34, 57] .• SMCalFlow [2, 58] : features complex dialogues about events, weather, places, and people. Each dialogue state is represented as a dataflow graph. Its dialog states also feature explicit functions for references and revisions. The SMCalFlow-CS [58] subset consists of single-turn natural language sentences pertaining to two domains: organization structure and event creation, each with its own set of program symbols. The cross-domain (C) test set evaluates examples that incorporate compositional abilities, while the single-domain (S) test set contains examples from a single domain. Due to input length restrictions, we conduct 8-C experiments following CEIL [57] , where an additional 8 cross-domain examples are included in the training set to provide composition symbols for evaluation.In this subsection, we introduce the details of the baselines used in our framework.• RANDOM: This retriever randomly picks in-context examples from the training set without any repetition.• TopK-BM25: This method employs the classical sparse retrieval technique BM25 [33] , an extension of TF-IDF. It selects the top-K scored examples as in-context examples.• TopK-BERT: A dense retriever based on BERT embeddings [8] . Following prevsioue works [57] , we use the bert-base-uncased model available in Huggingface Transformers [52] .• TopK-Contriver [17] and TopK-SimCSE [11] : These are advanced sentence embedding models trained with contrastive learning.• EPR [34] : A learning-based dense retriever trained to find the best singleton in-context example.During the inference stage, it selects the top-K most similar examples.• CEIL [57] : The state-of-the-art baseline instantiated by Determinantal Point Processes (DPPs) to model interactions between the input and demonstrations for in-context learning. It is optimized through a contrastive learning objective with supervision from LMs.Regarding the experiments in this work, we use a batch size of 128 and a learning rate of 10 -5 . We set the size of the candidate demonstration set as K = 50. The size of the positive demonstration set is K = 10. We conduct experiments on two NVIDIA A100 GPUs, each with 80GB of memory. For models that are available, we use the implementations provided in Huggingface Transformers [52] . We provide the code at https://github.com/SongW-SW/MoD. At each epoch, our objective is to update all expert's models; therefore, we adopt a stratified sampling strategy to ensure D train j is not empty for any expert j. Specifically, given the sample fraction r, we randomly sample max(1, ⌊r * |C j |⌋) demonstrations from each expert j's demonstration set C j and aggregate them to form d (t) . This guarantees that each C j contributes at least one sample (x j , y j ) ∈ C j ∩ d (t) , resulting in | S t j-| > 0. Consequently, we add (x j , {e k j } K k=1 , {s(e k j )} K k=1 ) to D train j and make it nonempty.We primarily compare the proposed MoD with the state-of-the-art CEIL method [57] , focusing on two aspects of complexity reduction: the number of demonstrations used and the efficiency of the inference stage.Since the attention mechanism in most LLMs has quadratic complexity [57] , fewer demonstrations result in shorter input lengths and reduced computational cost. From Table 6 , we observe that MoD generally outperforms CEIL using only 4 demonstrations compared to CEIL's 16 demonstrations. This shows that MoD can achieve better performance with fewer examples, thus reducing the computation complexity in the attention module of LLMs. In MoD, in the worst case, we select the top L elements in one expert, with a complexity of O(N + L log L). Thus, the total complexity of MoD at the inference stage is O(T + N + L log L). Given L < N , MoD further reduces complexity compared to CEIL at the inference stage.We conduct experiments focusing on the effect of specific designs in expert-wise training, and the results are reported in Table 7 . We consider the following variants: (i) The variant MoD w/o F removes the few-shot scoring strategy, such that the supervision score of each sample is obtained by individually using itself as context. (ii) The variant MoD w/o T alters the strategy of selecting the demonstration set S -(x) to random selection, instead of selecting the L -1 highest-scored demonstrations. (iii) The variant MoD w/o N removes the negative demonstrations from other samples in the contrastive learning loss. As a result, the contrastive learning loss only involves one hard negative sample. We could observe that removing the few-shot scoring strategy causes a significant performance drop. This indicates that it is more suitable to use multiple demonstrations together as input to correctly evaluate the benefit of any demonstration. The results of the other two variants also indicate the importance of using the highest-scored samples as demonstrations and using more negative samples for contrastive loss. Regarding the transferability of the retriever in MoD across different tasks, we conduct additional experiments to evaluate the performance of our retriever trained on one dataset and then applied to other datasets. We report the absolute improvement over the baseline TopK-BERT. From the results, we observe a strong pattern that, the performance experiences a reduction when the retriever is transferred to other datasets, indicating that the knowledge in the training dataset is crucial for selecting demonstrations. Moreover, when transferring the retriever from dataset MNLI to other datasets, the performance is decreased greatly. This is potentially due to that the NLI task requires two textual inputs instead of one in other datasets. As such, the learned knowledge in the retriever can hardly be transferred. On the other hand, the performance of our work after transferring is still generally better than TopK-BERT. This verifies the transferability of our work. Developing a retriever that works effectively across all tasks is a challenging yet valuable research topic, which we leave for future work. In this subsection, we investigate the impact of Sentence-BERT on clustering performance, using two variants of Arctic-Embed [28] : Arctic-xs and Arctic-m. We evaluate clustering quality using three metrics: Silhouette Score, Davies-Bouldin Index, and Dunn Index.As shown in Table 9 , Sentence-BERT generally achieves superior clustering results. Notably, previous ICL studies have also utilized Sentence-BERT as an embedding model [34, 7, 57] . Our results demonstrate that MoD consistently outperforms other baselines when using the same embedding model. Additionally, we observe that the Dunn Index is more closely correlated with the final performance of ICL. Selecting the appropriate clustering criteria and optimal embedding model for ICL is a challenging yet valuable problem, which we leave for future work. We conduct experiments to investigate the influence of different retriever model structures. Note that EPR [34] can be seen as the implementation of DPR [20] for ICL tasks. In Table 10 , we present the results of MoD and EPR under different retriever models. The results indicate that replacing the BERT-base model with RoBERTa [33] or DeBERTa [14] enhances the performance of both EPR and MoD in most cases, with MoD consistently outperforming EPR across all retriever models. This suggests that retriever performance can indeed benefit from the choice of encoder model. We present the results for different values of K and K in Table 11 . The results indicate that increasing the value of K can slightly enhance performance but at the cost of significantly higher computational overhead. Notably, for larger values of K, such as K = 100, increasing K may inadvertently degrade performance. This decline is likely due to the inclusion of positive demonstrations with relatively lower scores as K increases. We investigate the effect of hard negative sampling. In the original setting, we set #Hard = 1. In Table 12 , we present the results for four variants: #Hard = 1, 5, 10, and 20. Across all datasets, we observe a general trend where performance initially improves with a slight increase in the number of hard negatives, but then begins to decline as the number continues to increase. This pattern suggests that using a moderate number of hard negative samples strikes a balance between leveraging enough information from negative samples and avoiding the inclusion of potentially irrelevant data. Our framework MoD aims to select suitable demonstrations to improve the ICL performance of LLMs. However, there still limitations to our framework. First, our MoD framework requires the label of samples to provide supervision information to the LLMs. This drawback is also present in recent works such as EPR [34] and CEIL [57] . In the future, it is potentially inspiring to develop a framework that does not require the labels of the demonstrations, i.e., using unlabeled samples. Second, the performance of our MoD framework is related to the assignment of experts. If an input query has misinformation and is assigned to incorrect experts, the retrieved samples from these experts may not be helpful and contribute to the performance.In this paper, we propose a demonstration selection approach MoD which aims to select the proper demonstrations as in-context learning prompts to improve the performance of the large language model. The proposed method sheds light on the future design of new and fancy demonstration selection methods. While we emphasize the importance of responsible use, we do not anticipate any major negative societal impacts from our work.Answer • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 


Decision: This paper introduces the “Mixture of Demonstrations” (MoD) method for LLMs. It is designed to improve in-context learning for LLMs by intelligently managing demonstration selection through a mixture of experts approach. This method is tailored to address the common challenge of large search spaces in demonstration retrieval, which is a significant hurdle in deploying effective in-context learning models.

There is general agreement between the reviewers and myself that the proposed approach’s use of a partitioned demonstration pool and an expert-wise training strategy to reduce complexity and increase the diversity of selected demonstrations is innovative. The authors have also conducted comprehensive experiments across a variety of NLP tasks, demonstrating that MoD outperforms the current SoTA methods like CEIL. 

There are, however, several concerns regarding this paper, such as the computational complexity of the proposed method and the absence of experiments on larger models. These concerns were mostly addressed by the authors during the rebuttal phase, including additional results and ablation studies, and a detailed computational cost analysis. 

Given the originality of the approach and the depth of experimental validation, I lean towards an acceptance for this paper. I recommend that the authors incorporate the detailed computational cost analyses and the new experimental results into the final manuscript.

Reviews: review: paper_summary: paper_summary: This paper presents a framework called Mixture of Demonstrations (MoD) for selecting demonstrations to improve in-context learning (ICL) performance of large language models (LLMs). Their method partitions the demonstration pool into groups, each managed by an expert, and applies an expert-wise training strategy to filter the helpful demonstrations. Their experiments show that MoD on GPT-Neo can outperform other ICL baselines.
strength_weakness: strength_weakness: - The computational complexity of the proposed method seems very high. Since in-context learning it self is an inference-based method without training, it would be much more expensive to conduct a training-based method before doing in-context learning on a new task. Moreover, the proposed method needs to do expert-wise training, which further increases its usage of computing resources.
- A related problem is that the authors do not mention which retriever model they use in their methods. Nowadays in-context learning can be used on much larger models to get better performance, such as LLaMA-70B, but it would be very hard for practitioners to train such a large retriever model.
- The authors only conduct their main experiments on a very small model (GPT-Neo with the size of 2.7B). Therefore, it is hard to say whether this method would still be effective when applied to larger and stronger models (e.g., LLaMA3-8B). Thus I feel the experiment is not very convincing to me.
questions: questions: Please refer to the above section.
limitations: limitations: Yes, the authors address limitations in their paper.
paper_summary: paper_summary: The paper introduces the Mixture of Demonstrations (MoD) framework for the demonstration retrieval for In-Context Learning (ICL) in Large Language Models (LLMs). Current methods for selecting demonstrations are hindered by large search spaces and insufficient optimization. MoD addresses these issues by partitioning the demonstration pool into groups managed by experts, optimizing the retriever model to prioritize beneficial demonstrations by coordinate descent, and aggregating expert-selected demonstrations during inference. Experiments across various NLP tasks show MoD's state-of-the-art performance.
strength_weakness: strength_weakness: 1. Lack of complexity analysis. One of main motivation of this paper is to reduce the searching space and complexity while [1] also aims to solve this problem. However, the authors did not quantitatively analyze the complexity of their methods and compared with [1]. Further analysis on the search space complexity and the training cost (espically how many training samples needed to be annotated based on LLM feedback for each task?).
2. Coordinate descent. The paper mentions the use of coordinate descent but does not clearly specify where and how it is applied within the framework. Clarification on the application and role of coordinate descent in the proposed method is needed.
3. The authors claim that MoE enables the retrieval of dissimilar samples that could be beneficial for ICL (lines 123-124). However, [1] and other works achieve this by maximizing the similarity between demonstration examples and minimizing the similarity within the demonstration sequence to introduce diversity. More analysis and comparison of these methods are necessary to verify the effectiveness of MoE.
4. More ablations study. The paper's readability can be improved by explaining each subscript in the equations immediately upon introduction. This would help readers follow the mathematical formulations more easily.

[1] J. Ye, Z. Wu, J. Feng, T. Yu, and L. Kong. Compositional exemplars for in-context learning. In International Conference on Machine Learning, pages 39818–39833. PMLR, 2023.
questions: questions: Please refer to the weakness.
limitations: limitations: Yes
paper_summary: paper_summary: The paper presents a new method for creating a dense retriever for ICL for LLMs performing tasks. The proposed combines a mixture of experts model, coordinate descent (for alternating optimization to the experts), and contrastive loss to create the retriever. The paper then goes on to show the method's performance on creating a retriever across several benchmark datasets and tasks, where it performs better than previous models.
strength_weakness: strength_weakness: For one, the method still requires labeled data in the validation set in order to work. The value of LLMs, particularly for Labeling tasks, is their ability to work in zero-shot settings (for example, see Liyanage et al. “GPT-4 as a Twitter Data Annotator: Unraveling Its PerformanceGPT-4 as a Twitter Data Annotator: Unraveling Its Performance on a Stance Classification Task” and/or Ziems et al. “Can Large Language Models Transform Computational Social Science”). Thus, having to have labeled data examples to get labeled data examples lowers the significance of the proposed method.

Second, the focus of the paper could be better to make the method more significant. The paper focuses on in-domain data and tasks, especially with regard to labeling tasks (e.g., sentiment labeling). While I certainly agree that in-domain and task performance is important, this could really be handled by standard supervised approaches with smaller models. Rather, I think this method should be focused on something like creating new, labeled datasets, based on benchmark datasets. Consider the example in Cruickshank and Ng “DIVERSE: Deciphering Internet Views on the U.S. Military Through Video Comment Stance Analysis, A Novel Benchmark Dataset for Stance Classification”. I think the proposed method could actually work in this type of scenario where you use benchmark stance-labeled datasets like SemEval2016 and srq, etc. to train a retriever, which could then be used with ICL to produce a labeled dataset, without human labeling. In other words, I think the significance of the method, and by extension, the paper, could be substantially improved if they framed the method around more significant problems in using LLMs to label data as opposed to framing it in-domain, supervised performance.
questions: questions: -	How transferable to different data domains or tasks is a retriever trained in the proposed approach? For example, how well will a retriever trained for ICL on SST-5 work on ICL for MNLI?
limitations: limitations: The authors should mention that the use of LLMs for labeling data, especially for in-domain usage, could be considered a waste of resources. For example, if I am trying to label data for stance and I have a benchmark, labeled dataset that is similar already, I can likely just as good of performance, with much less computational resources training a small, supervised model on the benchmark dataset and applying it to the dataset to be labeled, versus using a computationally expensive LLM to do the same task.
paper_summary: paper_summary: This paper proposes MoD (Mixture of Demonstrations), which partitions the demonstration pool into clusters using embeddings to enhance the diversity of demonstrations. For each cluster, an individual retriever is trained with the few-shot scoring from the LLM to maximize the combinational effect of the selected demonstrations. During inference, each retriever will retrieve a certain number of the demonstrations based on the similarity with the input query. MoD can outperform previous demonstration selection methods, and the selected demonstrations can generalize to larger models.
strength_weakness: strength_weakness: - The efficiency of this method may limit its application since it requires training the retriever of each expert iteratively. I would like to see the **total FLOPs** of MoD compared to previous work
- Missing important hyperparameters analysis and ablation studies, e.g.
    - How many iterations are needed to train the retriever well?
    - How does Sentence-BERT affect the cluster performance compared to other embedding models like Arctic-Embed?
    - How does BERT-base affect the retriever performance compared to other retrievers like DPR?
    - How do $K$ and $\widetilde{K}$ affect the retriever's training?
    - How does the hard negative sampling affect the retriever's training? Right now, it is just the $argmin$ score, can we have more hard negatives?
- The function $h$ to determine the relevance between input and expert is just the cosine similarity. Why should we assign more demonstrations to the more similar expert, and any justification?
- In Table 4, the retriever trained by LLaMA-7B feedback cannot **benefit LLaMA-7B more** than the one trained by GPT-Neo feedback, which may pose the question of the usefulness of the model-aware few-shot scoring signal

[1]: Merrick, Luke, et al. "Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models." arXiv preprint arXiv:2405.05374 (2024).

[2]: Karpukhin, Vladimir, et al. "Dense Passage Retrieval for Open-Domain Question Answering." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.
questions: questions: - Why the gain over CEIL on the cross-domain split of SMCalFlow-CS is only 0.11%? Is it because the task is extremely hard?
- Why does Table 4 show the MoD performance over TopK-BERT (a weaker baseline), not CEIL?
- Can you provide some cases of good vs. bad demonstrations?

---

 Some suggestions:

- $\mathcal{L}$ is repetitively used as the evaluation criterion as well as contrastive loss
- In Table 1, I cannot find *# Expert* that represents the number of experts used in each task
- *basins* typo in line 285
- More concise notations for the equations
limitations: limitations: Yes.

Comments: title: Official Comment by Authors
Thank you so much
Official Comment by Authors
Reply to authors' rebuttal
Thanks for your rebuttal
Reply to Rebuttal
Follow-Up on Our Rebuttal - Your Feedback is Appreciated
Follow-Up on Our Rebuttal - Your Feedback is Appreciated
Response to Reviewer's Reply
Thanks for your rebuttal
Looking Forward to Your Reply
Author's Responses to Questions
comment: Dear Reviewer SdUm,

Thank you for your thoughtful feedback and for raising the overall score. We are pleased that our clarification and complexity analysis addressed your concerns. We will incorporate these aspects into the future version to further enhance the paper. 

We appreciate your suggestion to explore how relevance and diversity enhance ICL. We will delve into these studies in our future work.

Best,  
The Authors
Dear Reviewer 3zjF,

We appreciate your recognition of our work and are pleased that our rebuttal addressed your concern. In the revised version, we will emphasize the relevant information to ensure it is clearer and more accessible to readers.

Best,  
The Authors
Dear Reviewer SZXS,

Thank you for your thoughtful feedback and for acknowledging our efforts to address your points. We greatly appreciate your recognition of our work's contributions. 

We agree that expanding into zero-shot labeling and data generation tasks could significantly enhance the impact of our approach. We will focus on these topics in future work.

Best,  
The Authors
I appreciate the authors' clarification and complexcity analysis. Based on these responses, I am now clearer about the contribution and will raise the overall score. Hope the authors can incorporate these into the future version. In addition, I feel it is also interesting to show some qualitative examples reflecting how relevance and diversity combined helps the task.
The authors have addressed my concerns and I raised my score accordingly.
I want to thank the authors for considering all of the points I brought up. I especially would like to commend them on adding in the cross-dataset test. While I suspected training on one dataset and then retrieving on another would see a performance, it is good validation to see this result. And, even cross-dataset fine-tuning for the retrievers still gives a performance increase over a baseline, as noted by the authors. I believe there is merit in the paper and having validation for difficulties like cross-dataset performance when fine-tuning, and so I stand by my rating for the article.
Dear Reviewer SdUm,

We hope our responses have effectively addressed your concerns and clarified any misunderstandings. We would greatly appreciate it if you could review our rebuttals, especially as the discussion period is set to conclude on **August 13th at 11:59 PM AoE**, which is less than 20 hours away.

Thank you for your time and consideration. We look forward to your feedback.

Best,  
The Authors
Dear Reviewer 3zjF,

We hope that our response can address your concerns and clarify any misunderstandings. We would greatly appreciate it if you could review our responses, as the discussion period will end on **Aug 13 11:59pm AoE** in less than 20 hours.

Thank you very much for your time and consideration. We look forward to hearing from you.

Best,  
The Authors
Dear Reviewer UC6f,

Thank you for carefully reading our rebuttal. We appreciate your recognition of our efforts to address your concerns.

We will incorporate your suggestions and further improvements in the revised version. Your input has been invaluable in helping us refine our research.

Thank you once again for your constructive feedback.

Best,  
The Authors
Thanks to the authors. I have read the rebuttal carefully. I would increase my score from 5 to 6 since the authors addressed most of my concerns reasonably and conducted necessary ablation studies. I hope they can incorporate those better in the next version.
Dear Reviewers,  

We have carefully addressed your feedback in our rebuttals and provided detailed responses to each of your comments. We believe these clarifications will help in assessing our work more comprehensively.

We would greatly appreciate it if you could review our rebuttals and provide any further feedback, given that the Author-reviewer discussion will be **closed on Aug 13 11:59pm AoE** in no more than two days. We are willing to answer any further questions.

Thank you for your time and consideration. We look forward to your reply.

Best,  
The Authors
>**Q1**: Why the gain over CEIL on the cross-domain split of SMCalFlow-CS is only $0.11\%$? Is it because the task is extremely hard?

**RQ1**:  We appreciate the reviewer for mentioning this point. As discussed in Appendix B.1, SMCalFlow is a large-scale semantic parsing dataset for task-oriented dialogue. The cross-domain (C) test set evaluates examples that incorporate compositional skills in dialogue, which is significantly more challenging compared to the single-domain (S) evaluation. Such cross-domain tasks are particularly difficult for LLMs like GPT-Neo. A similar observation is also observed in [8].

However, despite the difficulty of the task, MoD demonstrates a substantial improvement over CEIL. Specifically, **the relative performance gain of (0.39 - 0.28) / 0.28 = 39.3\% indicates that MoD can better handle complex, multi-domain tasks**. This improvement highlights the capability of MoD to effectively address the challenges posed by cross-domain evaluations.

>**Q2**: Why does Table 4 show the MoD performance over TopK-BERT (a weaker baseline), not CEIL?

**RQ2**: Thank you for your question regarding the setting of results in Table 4. We follow the settings in CEIL and thus report the performance over TopK-BERT as an evaluation of the transferability of MoD. Our experimental results in various sections, all demonstrate the superiority of MoD over the baseline CEIL. By including TopK-BERT in our comparisons, we provide a broader context for evaluating the improvements brought by MoD.

>**Q3**: Examples of good vs. bad demonstrations.

**RQ3**:  Thank you for your feedback. Here, we select one training sample along with two good and two bad demonstrations.
- **Training Sample**: "nicholson's understated performance is wonderful"
- **Good Demonstrations**:
1. "britney's performance cannot be faulted"
2. "parris' performance is credible and remarkably mature"
- **Bad Demonstrations**:
1. "there is something that is so meditative and lyrical about babak payami's boldly quirky iranian drama secret ballot..."
2. "the film fearlessly gets under the skin of the people involved..."

Notably, the good demonstrations are both semantically and syntactically similar to the training sample. Conversely, the bad demonstrations are not directly opposite in meaning. This indicates that such demonstrations could incur more negative effects than those with straightforward adversarial meanings. We will include more examples in the revised version.

**Suggestions**: We thank the reviewers for their careful reading and subtle observations. We will be happy to make the corresponding changes in subsequent versions for these suggestions.

[1] Merrick et al. Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models. Arxiv, 2024.  
[2] Rubin et al. Learning to retrieve prompts for in-context learning. NAACL, 2022.  
[3] Karpukhin et al. "Dense Passage Retrieval for Open-Domain Question Answering." EMNLP, 2020.  
[4] Han et al. In-context learning of large language models explained as kernel regression. Arxiv, 2023.  
[5] Wang et al. One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts. ICML, 2024.  
[6] Su et al. Selective Annotation Makes Language Models Better Few-Shot Learners. ICLR, 2023.  
[7] Liu et al. What makes good in-context examples for gpt3? DeeLIO, 2022.  
[8] Ye et al. Compositional exemplars for in-context learning. ICML, 2023.