Learning to Select In-Context Demonstration Preferred by Large Language Model
Authors: Zheng Zhang, Shaocheng Lan, Lei Song, Jiang Bian, Yexin Li, Kan Ren
Accepted at: acl25

Learning to Select In-Context Demonstration Preferred by Large Language Model

Zheng Zhang1,3, Shaocheng Lan1, Lei Song2*, Jiang Bian2, Yexin Li3, Kan Ren1*
1School of Information Science and Technology, ShanghaiTech University
2Microsoft Research Asia
3State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China
{zhangzheng2024, lanshch2024, renkan}@shanghaitech.edu.cn
{lei.song, jiang.bian}@microsoft.com, liyexin@bigai.ai

Abstract

In-context learning (ICL) enables large language models (LLMs) to adapt to new tasks during inference using only a few demonstrations. However, ICL performance is highly dependent on the selection of these demonstrations. Recent work explores retrieval-based methods for selecting query-specific demonstrations, but these approaches often rely on surrogate objectives such as metric learning, failing to directly optimize ICL performance. Consequently, they struggle to identify truly beneficial demonstrations. Moreover, their discriminative retrieval paradigm is ineffective when the candidate pool lacks sufficient high-quality demonstrations. To address these challenges, we propose GenICL, a novel generative preference learning framework that leverages LLM feedback to directly optimize demonstration selection for ICL. Experiments on 19 datasets across 11 task categories demonstrate that GenICL achieves superior performance than existing methods in selecting the most effective demonstrations, leading to better ICL performance. The project website is at https://foundation-model-research.github.io/GenICL.

1 Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks. In-context learning (ICL), introduced by (Brown et al., 2020), enables LLMs to perform tasks with only a few examples as demonstration, without requiring parameter updates (Brown et al., 2020; Liu et al., 2021).

Although ICL is promising, the selection of in-context demonstrations is crucial, as it significantly impacts LLM performance (Lu et al., 2021; Min et al., 2022b). Many existing approaches rely on retrieval-based methods, either using off-the-shelf retrievers (Robertson et al., 2009) or training specialized ones (Reimers, 2019; Wang et al., 2022). Training-based methods typically optimize retrievers to approximate the relevance score of the candidate to the given query for downstream ICL tasks, often leveraging metric learning-based proxy tasks, such as contrastive loss (Rubin et al., 2021; Ye et al., 2023; Wang et al., 2023; Cheng et al., 2023; Luo et al., 2023; Liu et al., 2024).

However, these methods face several challenges. The most critical issue is the misalignment between the surrogate learning objective of the retriever and the intrinsic optimization goal of ICL. Specifically, a discriminative model, trained with a metric learning objective to approximate the relevance score between a demonstration candidate and a given query, does not necessarily indicate the candidate’s effectiveness as an in-context demonstration for an LLM. Furthermore, the scarcity of effective demonstration candidates in the retrieval pool poses another challenge for training a discriminative retriever. Figure 1 illustrates the distribution of test

![The distribution of the useful example ratio across test sets from different datasets. The x-axis represents the ratio # useful examples / # total examples, where a 'useful example' is one that helps the LLM generate an accurate output with in-context learning. The ratio remains low for most test queries, i.e., the majority of demonstration examples are ineffective for the LLM.](page_1016_370_388_282.png)

Figure 1: The distribution of the useful example ratio across test sets from different datasets. The x-axis represents the ratio \( \frac{\text{# useful examples}}{\text{# total examples}} \), where a 'useful example' is one that helps the LLM generate an accurate output with in-context learning. The ratio remains low for most test queries, i.e., the majority of demonstration examples are ineffective for the LLM.
queries regarding the ratio of useful examples that help the LLM achieve accurate output using ICL. The figure shows that most candidates are ineffective for most queries, making retriever optimization particularly challenging. This issue is further exacerbated by the properties of hard labels for LLM feedback, as noted by (Wang et al., 2023).

To address these challenges, we propose GenICL, a novel generative preference learning framework that directly optimizes demonstration selection for ICL using LLM feedback. Specifically, we reformulate ICL as a generative Bayesian optimization problem, introducing a latent variable to bridge demonstration selection and LLM inference. GenICL then optimizes this objective through preference learning, which focuses more on the relative effectiveness between demonstration samples, allowing it to capture finer-grained information in scenarios where effective demonstrations are scarce. This optimization procedure on demonstration selection more closely aligns with the intrinsic objective of ICL. Through this approach, GenICL can better capture the LLM’s demonstration preference and identify the truly effective demonstrations (preferred by the LLM). To validate the effectiveness of GenICL, we conducted experiments on a diverse set of datasets covering classification, multiple-choice, and generation tasks. The results demonstrate that GenICL achieves superior performance compared to existing methods.

Our contributions can be summarized as follows:

• We propose GenICL, a novel generative preference learning framework that directly optimizes demonstration selection for ICL using LLM feedback, overcoming the limitations of sub-optimality of surrogate learning objective used in traditional retrieval-based methods.

• The preference learning of paired effective and ineffective demonstrations leads to fine-grained distinguishness of demonstration preference of LLM.

• We conduct extensive experiments on 19 datasets across 11 task categories with a quantitative and qualitative analysis of the effectiveness of our proposed method.

2 Related work

2.1 In-context Demonstration Selection
In-context learning (Brown et al., 2020) enhances large language models by leveraging few-shot examples for task adaptation. While proficient at using provided examples, LLMs heavily depend on specific demonstrations (Min et al., 2022a; Luo et al., 2024), making demonstration selection crucial for downstream performance. Existing approaches to demonstration selection can be broadly categorized into two branches. The first employs off-the-shelf retrievers like BM25 (Robertson et al., 2009), SBERT (Reimers, 2019), and ES_base (Wang et al., 2022), which prioritize semantic similarity but overlook downstream utility. The second category involves learning the retriever based on feedback from LLM performance on selected demonstrations (Li et al., 2023; Cheng et al., 2023; Luo et al., 2023; Chen et al., 2024). For instance, Rubin et al. (2021) trained a dense retriever using LLM-generated feedback, while Wang et al. (2023) distilled a retriever from a metric function estimating relevance scores between demonstration candidates and the query. However, these methods approximate the relevant score of the candidate given the query, which is not essentially aligned with the true optimization objective of ICL resulting in a sub-optimal solution.

Moreover, only a minority of candidates truly enhance in-context learning performance (Figure 1), making it challenging to learn a reasonably dense retriever.

Therefore, a more promising method is required for directly optimizing the intrinsic ICL objective.

2.2 Preference Learning
Preference learning plays a vital role in training LLMs to align generative models with human values and preferences (Christiano et al., 2017; Ziegler et al., 2019; Ouyang et al., 2022). It has also emerged as a powerful paradigm for enhancing LLM performance in complex tasks (Havrilla et al., 2024; Li et al., 2024). A key approach in this domain is reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022), which involves training a reward model to capture human preferences and optimizing the model using reinforcement learning algorithms (Schulman et al., 2017). More recent methods, such as Direct Preference Optimization (DPO) (Rafailov et al., 2024) and Kahneman-Tversky Optimization (KTO) (Ethayarajh et al., 2024), bypass explicit reward modeling and instead learn preferences directly from human feedback. While these methods have advanced preference learning performance, they pri-
marily focus on human feedback without explicitly addressing the issue of the scarcity of effective demonstrations.

3 Methodology

3.1 Problem Setting

Consider a task sample \( x \) with ground-truth \( y \), denoted as \( (x, y) \), and a set of demonstrations \( \{ (x_k, y_k) \}_{k=1}^K \subseteq \mathcal{P} \) where each demonstration \( (x_k, y_k) \) is an input-output tuple taken from the demonstration pool \( \mathcal{P} \) containing all training samples from all tasks. \( K \) is the number of demonstrations used for the current query. Our goal is to provide the appropriate demonstrations for each \( x \) to improve the ICL performance of LLMs as

\[
\arg\max_{(x_k, y_k) \in \mathcal{P}} P_{\mathcal{M}}(y \mid \{ (x_k, y_k) \}_{k=1}^K, x)
\] (1)

3.2 Optimization Objective

Most methods based on LLM feedback used metric learning, such as contrastive learning, as a surrogate objective to train a discriminative model as the retriever. However, this optimization objective has a gap with in-context learning performance (details in Appendix C). Additionally, the relevance scores determined by the discriminative retriever come from a feature similarity perspective, overlooking optimizations at the semantic understanding level, which leads to the selected demonstrations that may not be truly useful for improving the LLM’s in-context learning performance. To address these issues, we propose a generative framework to directly optimize demonstration selection for ICL performance, thereby avoiding the use of surrogate objectives that have a gap. We treat in-context learning as generative Bayesian optimization and introduce a latent variable \( z \) to establish the relationship between demonstration selection and ICL performance. The optimization problem is formulated as follows:

\[
P_{\mathcal{M}}(Y \mid \{ (X_k, Y_k) \}_{k=1}^K, X) = \int_z P_{\mathcal{M}}(Y \mid z, X) P_{\mathcal{M}}(z \mid \{ (X_k, Y_k) \}_{k=1}^K, X) \, dz
\] (2)

This transformation is also supported by the perspectives of Xie et al. (2021) and Wang et al. (2024b), where in-context learning can be viewed as a Bayesian inference process.

The latent variable \( z \) is viewed as the representation of the LLM’s demonstration preference. Through \( z \), we concretize the LLM’s preference for demonstration and use it to identify effective demonstrations. Eq. (2) proves from a Bayesian perspective that, during the optimization process, both the ground truth and demonstrations should be considered simultaneously. However, in the surrogate optimization objectives of retriever-based methods, only the quality of the demonstrations is considered. This is why demonstrations selected by these methods do not necessarily enhance the LLM’s ability to predict the ground truth within ICL.

To obtain the demonstration preference, we propose to fully optimize the latent variable \( z \) by jointly considering the ground truth and the preference of the demonstrations. We utilize the Evidence Lower Bound (ELBO) (Kingma, 2013) and Eq. (2) to obtain the following optimization objective, with the detailed derivation provided in Appendix D.

\[
\mathcal{L}(z) = - \log P_{\mathcal{M}}(Y \mid z, X) \\
- \log P_{\mathcal{M}}(z \mid \{ (X_k, Y_k) \}_{k=1}^K, X)
\] (3)

Due to the combinatorial search space of \( (X_1, Y_1), (X_2, Y_2), ..., (X_K, Y_K) \) being immense, we simplify the optimization objective to the following formula.

\[
\mathcal{L}(z) = - \log P_{\mathcal{M}}(Y \mid z, X) \\
- \log P_{\mathcal{M}}(z \mid (X_k, Y_k), X)
\] (4)

3.3 Demonstration Preference Learning

The above optimization objective aims to find the variable \( z \) of the demonstration preference of the LLM. From an implementation perspective, \( z \) is a task-specific description, and its corresponding trainable parameters are \( \theta \). We optimize \( \theta \) to refine the demonstration preference distribution corresponding to \( z \).

To perform effective optimization in scenarios where effective demonstrations are scarce, we use preference learning, as it focuses more on the relative relationships between demonstrations, enabling it to capture finer-grained information. To learn the preference for effective demonstrations, we adopt the KTO algorithm (Ethayarajh et al., 2024), which directly maximizes the utility of LLM generations based on context information,
conditioned on either the effective demonstration \((x_k, y_k)_w\) or the relatively ineffective one \((x_k, y_k)_l\). This ensures that \((x_k, y_k)_w \succ (x_k, y_k)_l\) given the input \(x\). To achieve this, we reformulate the optimization objectives, \(-\log P_M(Y \mid z, X)\) and \(-\log P_M(z \mid (X_k, Y_k), X)\) in Eq. (4), as a two-stage alternating optimization process.

The First Stage. In this stage, we optimize the demonstration selection part \(-\log P_M(z \mid (X_k, Y_k), X)\) in Eq. (4) with preference learning. Preference learning requires both preferred data and non-preferred data. To obtain the LLM preferences for demonstrations under the in-context learning paradigm, we utilize the ICL performance of LLM given each demonstration candidate as the preference score.

The preference score \(s_k\) is the log-likelihood of LLM generation w.r.t. the ground-truth output \(y\) given the candidate \((x_k, y_k)\) and the input \(x\), reflecting the LLM’s preference for the demonstration.

\[
s_k = P(y|(x_k, y_k), x)
\] (5)

Since we need to score all the training data based on the above method to obtain the preference dataset, and the demonstration pool for each training sample is \(\mathcal{P}\), the complexity of scoring all the data is quadratic in \(|\mathcal{P}|\), which is computationally infeasible for an exhaustive search. Following prior work (Wang et al., 2024b; Cheng et al., 2023; Wang et al., 2024a), we use E5base (Wang et al., 2022) to obtain a more relevant subset \(\xi \subseteq \mathcal{P}\) as the new demonstration pool to reduce search space. For each training sample, we select the set of candidates with the highest preference scores as preferred demonstrations \((x_k, y_k)_w\), and the set of candidates with the lowest preference scores as non-preferred demonstrations \((x_k, y_k)_l\). The demonstration-level loss is formulated as follows.

\[
\begin{align*}
\mathcal{L}_d(\theta) &= -\mathbb{E}_{((x_k, y_k)_w, (x_k, y_k)_l, x, z) \sim \mathcal{D}} \\
&\left[ \lambda_w \sigma \left( \beta \log \frac{P_M(z \mid (x_k, y_k)_w, x)}{P_{M_{ref}}(z \mid (x_k, y_k)_w, x)} - s^d_{ref} \right) \right. \\
&\left. + \lambda_l \sigma \left( s^d_{ref} - \beta \log \frac{P_M(z \mid (x_k, y_k)_l, x)}{P_{M_{ref}}(z \mid (x_k, y_k)_l, x)} \right) \right]
\end{align*}
\] (6)

\[
s^d_{ref} = \mathbb{E}_{(x, y) \sim \mathcal{D}} \left[ \beta \mathrm{KL}(P_M(z \mid (x_k, y_k), x) \right. \\
\left. \| P_{M_{ref}}(z \mid (x_k, y_k), x)) \right]
\] (7)

where \(\beta\) is a hyperparameter, \(\lambda_w\) and \(\lambda_l\) are hyperparameters for preferred data and non-preferred data respectively. \(s_{ref}\) represents the use of KL divergence as a constraint to prevent the model from deviating too much from its original predictive ability, thereby enhancing the stability of the model.

The Second Stage. We then optimize the model generation part \(-\log P_M(Y \mid z, X)\) in Eq. (4) in a similar manner of preference learning. For all tasks, we treat the ground truth as the correct answer \(y_w\). For classification and multi-choice tasks,

![Training and inference pipeline of GenICL](page_180_180_1080_670.png)

Figure 2: Training and inference pipeline of GenICL.
we consider the incorrect categories or options as wrong answers \( y_t \), and for generation tasks, we randomly sample answers from other samples as \( y_t \) for the current sample. The answer-level loss is formulated as follows:

\[
\mathcal{L}_a(\theta) = -\mathbb{E}_{(z,x,y_w,y_t)\sim\mathcal{D}}
\left[ \lambda_w \sigma \left( \beta \log \frac{P_{\mathcal{M}}(y_w \mid z, x)}{P_{\mathcal{M}_{\text{ref}}}(y_w \mid z, x)} - s^a_{\text{ref}} \right) + \lambda_l \sigma \left( s^a_{\text{ref}} - \beta \log \frac{P_{\mathcal{M}}(y_l \mid z, x)}{P_{\mathcal{M}_{\text{ref}}}(y_l \mid z, x)} \right) \right]
\] (8)

\[
s^a_{\text{ref}} = \mathbb{E}_{(x,y)\sim\mathcal{D}} \left[ \beta \mathrm{KL}\left(P_{\mathcal{M}}(y \mid z, x) \parallel P_{\mathcal{M}_{\text{ref}}}(y \mid z, x)\right) \right]
\] (9)

where the hyperparameters are consistent with those in the first stage.

In summary, we alternately optimize the losses of the two stages, and the complete optimization objective is as follows:

\[
\mathcal{L}(\theta) = \mathcal{L}_a(\theta) + \mathcal{L}_d(\theta)
\] (10)

And Algorithm 1 summarizes the whole training procedure.

Algorithm 1: Demonstration Preference Learning

1: **Input:** Task dataset \( \mathcal{D} = \{(x, y)_i\}_{i=1}^{|\mathcal{D}|} \), latent variable \( z \), demonstration pool \( \mathcal{P} \), and learning rates \( \eta_1, \eta_2 \)
2: **Initialization:** Parameters \( \theta \)
3: **for** each \( (x, y) \) in task \( \mathcal{D} \) **do**
4:   Calculate demonstration scores by Eq. (5), and identify preferred demonstrations \( (x_k, y_k)_w \) and non-preferred ones \( (x_k, y_k)_l \)
5:   Compute demonstration-level loss \( \mathcal{L}_d(\theta) \) by Eq. (6)
6:   Update \( \theta \leftarrow \theta - \eta_1 \nabla_\theta \mathcal{L}_d(\theta) \)
7:   Set the correct answer as \( y_w = y \), and select an incorrect answer \( y_l \)
8:   Compute answer-level loss \( \mathcal{L}_a(\theta) \) by Eq. (8)
9:   Update \( \theta \leftarrow \theta - \eta_2 \nabla_\theta \mathcal{L}_a(\theta) \)
10: **end for**

3.4 Demonstration Selection in Inference

In the training stage, we optimize the parameters \( \theta \) to obtain the variable \( z \) of the demonstration preference of the LLM. In the inference stage, we first use E5\(_{\text{base}}\) (Wang et al., 2022) to reduce the search space and time cost of demonstration selection. Then we leverage the LLM with the latent variable \( z \) to select demonstrations for an input query \( x_{\text{test}} \). In Eq. (2), for the same \( x \), all demonstrations have the same \( P_{\mathcal{M}}(y \mid z, x) \). Therefore, we select the top-\( K \) demonstrations based on the probability of generating the latent variable, as shown below:

\[
P_{\mathcal{M}}(z \mid (x_k, y_k), x_{\text{test}})
\] (11)

After obtaining a set of demonstrations \( \{(X_k, Y_k)\}_{k=1}^K \), we use the frozen LLM to perform in-context learning. The entire training and inference pipeline is shown in Figure 2.

4 Experiments

4.1 Evaluations Setup

Datasets. We validate the effectiveness of GenICL across a range of language processing tasks including classification, multi-choice, and text generation. The classification tasks include Natural Language Inference (RTE (Wang, 2018), SNLI (Bowman et al., 2015)), Paraphrase (PAWS (Zhang et al., 2019), QQP (Wang, 2018)), Reading Comprehension (MultiRC (Khashabi et al., 2018), BoolQ (Clark et al., 2019)), Sentiment Classification: (SST2 (Socher et al., 2013), Sentiment140 (Go et al., 2009)), and Topic Classification (AG-News (Zhang et al., 2015)). The multi-choice tasks cover Commonsense Reasoning (COPA (Roemmele et al., 2011), HellaSwag (Zellers et al., 2019), OpenBookQA (Mihaylov et al., 2018)), Coreference (Winogrande (Sakaguchi et al., 2021)). The text generation tasks contain Close QA (NaturalQuestions (Kwiatkowski et al., 2019), SQuAD v1 (Rajpurkar, 2016)), Commonsense Generation (CommonGen (Lin et al., 2019)), Data-to-Text (E2E NLG (Dušek et al., 2019), DART (Nan et al., 2020)), and Summarization (AESLC (Zhang and Tetreault, 2019), Gigaword (Napoles et al., 2012)).

Baseline. Our evaluation compares GenICL against several established baseline approaches that leverage the ICL capabilities of LLMs. These baselines include zero-shot (which executes tasks without demonstrations), random (which selects demonstrations arbitrarily from the available pool), and off-the-shelf retrieval models such as BM25 (Robertson et al., 2009), SBERT (Reimers, 2019), and E5\(_{\text{base}}\) (Wang et al., 2022). Furthermore, we conduct comparative analyses against the following more advanced approaches.
• EPR (Rubin et al., 2021) is a prominent method using LLM feedback to label candidate examples as positive or negative, to train the retriever.

• LLM-R (Wang et al., 2023) trains the retriever using a reward model and knowledge distillation, which is one of the state-of-the-art retriever-based methods.

• CBDS (concept-based-demonstration-selection) (Wang et al., 2024b) first learns the latent variable and then selects demonstrations with the highest probability of predicting the corresponding latent variable.

Implementation Details. To ensure a fair comparison with the baselines, we use the same template and evaluation metrics. For detailed information, please refer to the Appendix E. Our approach demonstrates enhanced efficiency through a reduced number of training parameters relative to retriever-based methodologies such as LLM-R (Wang et al., 2023) and ES_base (Wang et al., 2022). For model implementation, we utilize Llama-7B (Touvron et al., 2023) as our primary LLM for most experiments. We also evaluate the impact and generalizability of our GenICL in Section 4.4 using GPT-Neo 2.7B (Black et al., 2021), Vicuna-13B (Zheng et al., 2023), LLaMA-3.2-3B (Grattafiori et al., 2024), and Qwen2.5-3B (Yang et al., 2024). Our inference protocol incorporates a set of 8 demonstrations within the context, with complete implementation details provided in Appendix A.

4.2 Main Results

Table 1 presents the performance comparison of the proposed GenICL and various baselines for classification, multi-choice, and text generation tasks. The results demonstrate that GenICL significantly outperforms all the baselines on most tasks. Zero-shot and Random consistently yield the worst performance across most tasks, highlighting the importance of using high-quality demonstrations in ICL. Retriever methods trained based on LLM feedback provide only limited improvements over off-the-shelf retrievers. Specifically, EPR performs worse than ES_base on tasks such as MultiRC, RTE, and Sentiment140. This indicates that these discriminative methods, which use proxy optimization objectives, often fail to select truly useful demonstrations. Our generative Bayesian method GenICL, in contrast, significantly outperforms EPR and LLM-R, proving its effectiveness. Although GenICL uses ES_base to narrow the search space, compared to directly using ES_base, we achieve significant improvements, especially on tasks such as PAWS, BoolQ, and SNLI, where our GenICL improves by more than 5 points. This indicates that directly using ES_base struggles to identify demonstrations preferred by the LLM, whereas our method effectively addresses this issue. While CBDS employs a comparable latent concept variable methodology for demonstration selection, its performance is constrained due to adopting a misaligned optimization objective (details in Appendix A). Through the implementation of a more refined optimization objective, GenICL demonstrates superior performance.

For text generation tasks, the improvements of EPR and LLM-R over off-the-shelf retrievers are all relatively limited. This may be because metrics like ROUGE and EM typically exhibit a narrower range of variation compared to classification accuracy, which is consistent with the findings of Wang et al. (2023). Even so, GenICL still achieves significant improvements over LLM-R and off-the-shelf retrievers, particularly on the CommonGen and SQuAD v1 tasks. This indicates that GenICL is also highly effective for text generation tasks. We also provide a case study analysis in Appendix F to offer further insights.

4.3 Ablation Study

To validate the effectiveness of the key design in GenICL, we conduct ablation studies on the CommonGen, BoolQ, and SQuAD v1 tasks, with results presented in Table 2. When non-preferred data is removed during training, the performance of GenICL drops significantly, indicating that using only useful demonstrations without considering relative preference relations makes it difficult to learn the distribution of LLM demonstration preference. This proves the necessity of optimization based on preference learning. Our complete optimization objective consists of two parts: answer-level loss and demonstration-level loss. Removing either leads to a performance drop, demonstrating the correctness of our proposed optimization framework and the importance of considering both the ground truth and the quality of demonstrations during optimization.
<table>
  <tr>
    <th rowspan="2">Topic</th>
    <th colspan="3">Classification Tasks</th>
    <th colspan="4">Text Generation Tasks</th>
  </tr>
  <tr>
    <th>AGNews</th>
    <th>BoolQ</th>
    <th>MultiRC</th>
    <th>PAWS</th>
    <th>QQP</th>
    <th>RTE</th>
    <th>SNLI</th>
    <th>Sentiment140</th>
    <th>SST2</th>
  </tr>
  <tr>
    <td>Zero-shot</td>
    <td>31.4</td>
    <td>64.7</td>
    <td>57.0</td>
    <td>53.0</td>
    <td>57.9</td>
    <td>59.9</td>
    <td>39.6</td>
    <td>49.3</td>
    <td>54.2</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>65.0</td>
    <td>69.6</td>
    <td><b>60.4</b></td>
    <td>49.6</td>
    <td>54.0</td>
    <td>65.7</td>
    <td>40.4</td>
    <td>78.8</td>
    <td>64.1</td>
  </tr>
  <tr>
    <td>BM25</td>
    <td>90.0</td>
    <td>74.0</td>
    <td>58.7</td>
    <td>56.5</td>
    <td>80.3</td>
    <td>59.9</td>
    <td>47.7</td>
    <td>88.3</td>
    <td>84.7</td>
  </tr>
  <tr>
    <td>SBERT</td>
    <td>89.8</td>
    <td>73.6</td>
    <td>53.3</td>
    <td><b>58.3</b></td>
    <td>81.7</td>
    <td>60.2</td>
    <td>56.2</td>
    <td><b>94.1</b></td>
    <td>87.8</td>
  </tr>
  <tr>
    <td>E5<sub>base</sub></td>
    <td>90.6</td>
    <td>71.0</td>
    <td>54.0</td>
    <td><b>55.6</b></td>
    <td>77.3</td>
    <td>68.5</td>
    <td>53.7</td>
    <td>93.0</td>
    <td>92.4</td>
  </tr>
  <tr>
    <td>CBDS</td>
    <td>67.3</td>
    <td>77.6</td>
    <td>49.3</td>
    <td>57.6</td>
    <td>64.2</td>
    <td>56.3</td>
    <td>43.5</td>
    <td>92.5</td>
    <td>69.2</td>
  </tr>
  <tr>
    <td>EPR</td>
    <td>91.8</td>
    <td>74.8</td>
    <td>50.4</td>
    <td>57.7</td>
    <td>81.7</td>
    <td>66.8</td>
    <td>68.4</td>
    <td>91.4</td>
    <td>88.7</td>
  </tr>
  <tr>
    <td>LLM-R</td>
    <td>92.4</td>
    <td>74.9</td>
    <td>50.2</td>
    <td>57.5</td>
    <td>80.9</td>
    <td>61.7</td>
    <td>80.0</td>
    <td>91.6</td>
    <td>93.4</td>
  </tr>
  <tr>
    <td><b>GENICL (ours)</b></td>
    <td><b>92.6</b></td>
    <td><b>78.1</b></td>
    <td><b>56.9</b></td>
    <td><b>63.9</b></td>
    <td><b>82.0</b></td>
    <td><b>72.9</b></td>
    <td><b>84.6</b></td>
    <td><b>94.7</b></td>
    <td><b>95.0</b></td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Coreference</th>
    <th colspan="3">Commonsense Reasoning</th>
    <th colspan="2">Summarize</th>
    <th colspan="2">Text Generation Tasks</th>
  </tr>
  <tr>
    <th>Winogrande</th>
    <th>COPA</th>
    <th>Hella Swag</th>
    <th>Open BookQA</th>
    <th>AE SLC</th>
    <th>Giga word</th>
    <th>Comm onGen</th>
    <th>Data-to-text</th>
    <th>CloseQA</th>
  </tr>
  <tr>
    <td>Zero-shot</td>
    <td>61.8</td>
    <td>66.0</td>
    <td>71.5</td>
    <td>41.6</td>
    <td>5.7</td>
    <td>15.4</td>
    <td>19.2</td>
    <td>22.8</td>
    <td>34.6</td>
    <td>2.2</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>62.1</td>
    <td>74.0</td>
    <td>72.1</td>
    <td>43.0</td>
    <td>6.5</td>
    <td>27.2</td>
    <td>36.3</td>
    <td>34.5</td>
    <td>51.1</td>
    <td>47.3</td>
  </tr>
  <tr>
    <td>BM25</td>
    <td>66.4</td>
    <td>77.0</td>
    <td>74.6</td>
    <td>47.8</td>
    <td>23.9</td>
    <td><b>32.5</b></td>
    <td>38.3</td>
    <td>55.4</td>
    <td>53.8</td>
    <td>53.7</td>
  </tr>
  <tr>
    <td>SBERT</td>
    <td>66.9</td>
    <td>81.0</td>
    <td><b>75.2</b></td>
    <td>49.2</td>
    <td>22.3</td>
    <td>31.7</td>
    <td>37.8</td>
    <td>54.6</td>
    <td>50.6</td>
    <td>62.5</td>
  </tr>
  <tr>
    <td>E5<sub>base</sub></td>
    <td>66.7</td>
    <td>84.0</td>
    <td>75.0</td>
    <td>51.0</td>
    <td>23.4</td>
    <td>31.9</td>
    <td>37.6</td>
    <td>54.9</td>
    <td>52.3</td>
    <td>61.9</td>
  </tr>
  <tr>
    <td>CBDS</td>
    <td>66.3</td>
    <td>83.0</td>
    <td>73.6</td>
    <td>47.6</td>
    <td>20.5</td>
    <td>29.2</td>
    <td>34.5</td>
    <td>52.2</td>
    <td>50.8</td>
    <td>59.8</td>
  </tr>
  <tr>
    <td>EPR</td>
    <td>66.5</td>
    <td>82.0</td>
    <td><b>75.2</b></td>
    <td>49.6</td>
    <td><b>26.0</b></td>
    <td>32.4</td>
    <td><b>39.2</b></td>
    <td><b>56.2</b></td>
    <td><b>53.6</b></td>
    <td><b>64.3</b></td>
  </tr>
  <tr>
    <td>LLM-R</td>
    <td>66.9</td>
    <td>85.0</td>
    <td>74.6</td>
    <td>50.8</td>
    <td><b>26.0</b></td>
    <td>32.5</td>
    <td>37.2</td>
    <td>56.0</td>
    <td><b>54.4</b></td>
    <td>61.8</td>
  </tr>
  <tr>
    <td><b>GENICL (ours)</b></td>
    <td><b>68.0</b></td>
    <td><b>86.0</b></td>
    <td>74.6</td>
    <td><b>51.8</b></td>
    <td>24.4</td>
    <td><b>33.0</b></td>
    <td><b>41.0</b></td>
    <td><b>56.4</b></td>
    <td><b>55.1</b></td>
    <td><b>65.7</b></td>
  </tr>
</table>

Table 1: Main results on classification, multi-choice, and text generation tasks.

<table>
  <tr>
    <th>CommonGen</th>
    <th>BoolQ</th>
    <th>SQuAD v1</th>
  </tr>
  <tr>
    <td>GENICL</td>
    <td>41.0</td>
    <td>78.1</td>
    <td>65.7</td>
  </tr>
  <tr>
    <td>- w/o non-preferred data</td>
    <td>38.7</td>
    <td>71.2</td>
    <td>62.5</td>
  </tr>
  <tr>
    <td>- w/o answer-level loss</td>
    <td>34.9</td>
    <td>72.7</td>
    <td>62.0</td>
  </tr>
  <tr>
    <td>- w/o demo-level loss</td>
    <td>37.9</td>
    <td>65.8</td>
    <td>61.7</td>
  </tr>
</table>

Table 2: Ablation study.

4.4 Generalizability and Scalability

Validation on More LLMs. The previous experimental results were obtained using the LLaMA-7B model. To comprehensively assess the effectiveness and generalizability of GENICL, we expanded our evaluation to encompass various LLMs of different scales, specifically GPT-Neo 2.7B, Vicuna-13B, LLaMA-3.2-3B, and Qwen2.5-3B. The comparison results are presented in Table 3. For each model, we report the performance of GENICL alongside zero-shot setting and a strong baseline, LLM-R (Wang et al., 2023). Across all models and tasks, GENICL consistently achieves the best results, suggesting that its benefits are not limited to high-capacity LLMs. For example, with the relatively smaller GPT-Neo 2.7B, GENICL still leads with 60.5 on PAWS and 92.7 on SST2 task. This indicates that GENICL can compensate for limited model capacity by better utilizing demonstration. These results provide compelling evidence that GENICL is a model-agnostic and scalable method.

Generalizability of Selected Demonstrations. In Table 4, we conduct experiments to evaluate th generalizability of demonstrations selected by our GENICL. Specifically, we examine whether demonstrations selected using a smaller model, GPT-Neo 2.7B, can be effectively reused by larger models such as LLaMA-7B and LLaMA-3.2-3B. The results indicate that demonstrations chosen by the smaller model can significantly improve the ICL performance of larger models, without the need for retraining or further adaptation. This high-
<table>
  <tr>
    <th></th>
    <th>CommonGen</th>
    <th>E2E NLG</th>
    <th>PAWS</th>
    <th>SST2</th>
  </tr>
  <tr>
    <td>GPT-Neo 2.7B</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>- Zero-shot</td>
    <td>14.2</td>
    <td>7.71</td>
    <td>52.6</td>
    <td>53.7</td>
  </tr>
  <tr>
    <td>- LLM-R</td>
    <td>34.5</td>
    <td>49.2</td>
    <td>55.5</td>
    <td>92.6</td>
  </tr>
  <tr>
    <td>- GENICL (ours)</td>
    <td>35.3</td>
    <td>50.1</td>
    <td>60.5</td>
    <td>92.7</td>
  </tr>
  <tr>
    <td>Vicuna-13B</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>- Zero-shot</td>
    <td>17.4</td>
    <td>1.71</td>
    <td>57.2</td>
    <td>49.0</td>
  </tr>
  <tr>
    <td>- LLM-R</td>
    <td>39.7</td>
    <td>55.4</td>
    <td>64.4</td>
    <td>93.8</td>
  </tr>
  <tr>
    <td>- GENICL (ours)</td>
    <td>39.7</td>
    <td>56.3</td>
    <td>71.7</td>
    <td>94.0</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>- Zero-shot</td>
    <td>23.6</td>
    <td>43.6</td>
    <td>58.8</td>
    <td>51.0</td>
  </tr>
  <tr>
    <td>- LLM-R</td>
    <td>35.6</td>
    <td>37.0</td>
    <td>70.4</td>
    <td>94.3</td>
  </tr>
  <tr>
    <td>- GENICL (ours)</td>
    <td>37.4</td>
    <td>37.9</td>
    <td>81.8</td>
    <td>94.6</td>
  </tr>
  <tr>
    <td>LLaMA-3.2-3B</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>- Zero-shot</td>
    <td>16.2</td>
    <td>33.5</td>
    <td>54.1</td>
    <td>52.4</td>
  </tr>
  <tr>
    <td>- LLM-R</td>
    <td>39.1</td>
    <td>56.4</td>
    <td>56.0</td>
    <td>93.0</td>
  </tr>
  <tr>
    <td>- GENICL (ours)</td>
    <td>39.8</td>
    <td>57.5</td>
    <td>66.2</td>
    <td>94.5</td>
  </tr>
</table>

Table 3: Performance on more LLMs.

lights the potential of our GENICL to serve as a lightweight and model-agnostic demonstration selector.

<table>
  <tr>
    <th></th>
    <th>CommonGen</th>
    <th>E2E NLG</th>
    <th>PAWS</th>
    <th>SST2</th>
  </tr>
  <tr>
    <td>GPT-Neo 2.7B</td>
    <td>35.3</td>
    <td>50.1</td>
    <td>60.5</td>
    <td>92.7</td>
  </tr>
  <tr>
    <td>LLaMA-7B</td>
    <td>40.8</td>
    <td>55.3</td>
    <td>59.3</td>
    <td>94.0</td>
  </tr>
  <tr>
    <td>LLaMA-3.2-3B</td>
    <td>39.4</td>
    <td>57.2</td>
    <td>59.5</td>
    <td>93.4</td>
  </tr>
</table>

Table 4: Generalizability of Selected Demonstrations. All demonstrations are selected by GPT-Neo 2.7B and used as in-context demonstrations for all three models.

Figure 3: The predicted probability of the ground truth using demonstrations selected by different methods.

CommonGen, and Gigaword. The results show that increasing the number of demonstrations does not always lead to better performance. For SST2 and CommonGen, a significant drop in performance is observed when the number of demonstrations is too large, while Gigaword experiences a slight decrease.

![Boxplots showing predicted probability for PAWS and Sentiment140 tasks](page_1012_120_377_246.png)

![Line plots showing effect of demonstration number K on SST2, RTE, CommonGen, and Gigaword](page_1012_624_377_246.png)

Figure 4: The effect of demonstration number \( K \).

To investigate the impact of the order of demonstrations on downstream task performance, we compare three different order settings:

• Shuffle: the top-\( K \) selected demonstrations are randomly shuffled.
• Descending: the top-\( K \) selected demonstrations are ordered in descending order of Eq. (11).
• Ascending: the top-\( K \) selected demonstrations are ordered in ascending order of Eq. (11).

The results are shown in Table 5. Lu et al. (2021) have revealed that ICL is sensitive to the order of demonstrations when using random examples. However, we found that the order of the demonstrations selected by our method has little impact on the final in-context learning performance. This supports the conclusion that high-quality demonstrations are less sensitive to the ordering (Li et al.,
2023; Chen et al., 2022; Li and Qiu, 2023). This further illustrates that the demonstrations selected by our method, based on preference learning, are of high quality.

<table>
  <tr>
    <th>Demo. order</th>
    <th>Agnews</th>
    <th>QQP</th>
    <th>Gigaword</th>
    <th>E2E NLG</th>
  </tr>
  <tr>
    <td>Shuffle</td>
    <td>92.8</td>
    <td>81.9</td>
    <td>32.6</td>
    <td>55.2</td>
  </tr>
  <tr>
    <td>Descending</td>
    <td>92.6</td>
    <td>82.0</td>
    <td>33.0</td>
    <td>55.1</td>
  </tr>
  <tr>
    <td>Ascending</td>
    <td>92.8</td>
    <td>82.5</td>
    <td>32.6</td>
    <td>54.9</td>
  </tr>
</table>

Table 5: Comparison of different demonstration orders.

5 Conclusion

In this paper, we introduced GENICL, a novel generative preference learning framework for optimizing demonstration selection in in-context learning (ICL). By reformulating ICL as a generative Bayesian optimization problem, GENICL bridges demonstration selection and LLM inference through a latent demonstration variable, aligning the intrinsic goal of ICL. GENICL also leverages preference learning on relatively effective and ineffective demonstrations from LLM feedback, achieving fine-grained demonstration selection. A wide range of experiments have illustrated the superiority of our proposed method GENICL. We believe that our findings contribute to advancing the field of ICL and hope that GENICL serves as a foundation for future research in directly optimizing demonstration selection for LLMs.

Limitations

During the optimization phase, we treat each demonstration independently, ignoring the interactions between demonstrations (such as order and combination). In the inference phase, we select a set of demonstrations based on their scores, but the combination of individually optimal demonstrations does not necessarily result in the overall best combination.

Our method requires scoring each candidate in the demonstration pool during the demonstration selection stage. However, the search space and computational cost of this process are prohibitive. To address this, we employ E5_base to reduce the search space by obtaining a subset of the demonstration pool as a new, smaller pool. However, this approach may filter out some valuable demonstrations. Additionally, improving the efficiency of demonstration scoring remains a promising direction for further exploration.

Acknowledgments

The research received partial support from National Natural Science Foundation of China (Grant No. 62406193). The authors also gratefully acknowledge further assistance provided by Shanghai Frontiers Science Center of Human-centered Artificial Intelligence, MoE Key Lab of Intelligent Perception and Human-Machine Collaboration, and HPC Platform of ShanghaiTech University.

References

Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. If you use this software, please cite it using these metadata, 58(2).

Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. 2022. On the relation between sensitivity and accuracy in in-context learning. arXiv preprint arXiv:2209.07661.

Yunmo Chen, Tongfei Chen, Harsh Jhamtani, Patrick Xia, Richard Shin, Jason Eisner, and Benjamin Van Durme. 2024. Learning to retrieve iteratively for in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7156–7168.

Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Denny Deng, and Qi Zhang. 2023. Uprise: Universal prompt retrieval for improving zero-shot evaluation. arXiv preprint arXiv:2303.08518.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.

Ondřej Dušek, David M Howcroft, and Verena Rieser. 2019. Semantic noise matters for neural natural language generation. arXiv preprint arXiv:1911.03905.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306.

Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Dane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. 2024. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.04642.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shayam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252–262.

Diederik P Kingma. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.

Jialian Li, Yipin Zhang, Wei Shen, Yuzi Yan, Jian Xie, and Dong Yan. 2024. Boosting deductive reasoning with step signals in rlhf. arXiv preprint arXiv:2410.09528.

Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023. Unified demonstration retriever for in-context learning. arXiv preprint arXiv:2305.04320.

Xiaonan Li and Xipeng Qiu. 2023. Mot: Prethinking and recalling enable chatgpt to self-improve with memory-of-thoughts. arXiv preprint arXiv:2305.05181, 16.

Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2019. Commongen: A constrained text generation challenge for generative commonsense reasoning. arXiv preprint arXiv:1911.03705.

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804.

Wenhan Liu, Yutao Zhu, and Zhicheng Dou. 2024. Demorank: Selecting effective demonstrations for large language models in ranking task. arXiv preprint arXiv:2406.16332.

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786.

Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral, Vaiva Imbrasaitė, and Vincent Y Zhao. 2023. Dr. icl: Demonstration-retrieved in-context learning. arXiv preprint arXiv:2305.14128.

Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi. 2024. In-context learning with retrieved demonstrations for language models: A survey. arXiv preprint arXiv:2401.11624.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789.

Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022a. MetalICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791–2809, Seattle, United States. Association for Computational Linguistics.

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022b. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837.

Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al. 2020. Dart: Open-domain structured data record to text generation. arXiv preprint arXiv:2007.02871.

Courtney Napoles, Matthew R Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In Proceedings of the joint workshop on automatic knowledge base construction and web-scale knowledge extraction (AKBC-WEKEK), pages 95–100.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

P Rajpurkar. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.

N Reimers. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.

Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389.

Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI spring symposium series.

Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Alex Wang. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.

Liang Wang, Nan Yang, and Furu Wei. 2023. Learning to retrieve in-context examples for large language models. arXiv preprint arXiv:2307.07164.

Peng Wang, Xiaobin Wang, Chao Lou, Shengyu Mao, Pengjun Xie, and Yong Jiang. 2024a. Effective demonstration annotation for in-context learning via language model-based determinantal point process. arXiv preprint arXiv:2408.02103.

Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2024b. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neural Information Processing Systems, 36.

Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115.

Jiacheng Ye, Zhiyong Wu, Jianguo Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In International Conference on Machine Learning, pages 39818–39833. PMLR.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830.

Rui Zhang and Joel Tetraeult. 2019. This email could save your life: Introducing the task of email subject line generation. arXiv preprint arXiv:1906.03497.

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.

Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. arXiv preprint arXiv:1904.01130.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593.
A Implementation Details

We use LoRA (Hu et al., 2021) to train the parameters corresponding to the latent variable, and the total number of trainable parameters is approximately 79M. The hyperparameters of GENICL are summarized in Table 6. For a fair comparison, our method and all baselines use the same template, as shown in Appendix E.

<table>
  <tr>
    <th colspan="2">GENICL</th>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>Warmup steps</td>
    <td>3000</td>
  </tr>
  <tr>
    <td>Training steps</td>
    <td>20000</td>
  </tr>
  <tr>
    <td>Learning rate</td>
    <td>5e-6</td>
  </tr>
  <tr>
    <td>Batch size</td>
    <td>32</td>
  </tr>
  <tr>
    <td>Maximum input length</td>
    <td>1024</td>
  </tr>
  <tr>
    <td>Maximum output length</td>
    <td>64</td>
  </tr>
  <tr>
    <td>\( \beta \)</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>\( \lambda_w \)</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>\( \lambda_l \)</td>
    <td>1.0</td>
  </tr>
</table>

Table 6: Hyper-parameters.

Implementation Details of CBDS (Wang et al., 2024b). For the CBDS baseline, we used the authors’ publicly available code and followed the training methodology described in the original paper. The original CBDS setup involves concurrent training across multiple tasks, with each task optimizing its own set of concept tokens \( \theta \). Specifically, the optimization objective during training is to minimize \( -\log P(Y|\theta, X) \). During testing, demonstrations \((X_k, Y_k)\) are selected based on \(P(\theta|X_k, Y_k)\). Training consisted of 10,000 steps with a learning rate of 1e-2 and a batch size of 16. A crucial difference between CBDS and both our method and other retriever-based approaches is that CBDS uses a shared set of demonstrations for all queries within a specific task. This design, while computationally efficient, often leads to performance inferior to that of retriever-based methods, which dynamically select demonstrations for each query. This performance difference was also observed in our reproduction experiments.

Implementation Details of LLM-R (Wang et al., 2023). Following the LLM-R baseline procedure, we first trained a reward model and then used it for distillation to train the LLM-R model, which uses the pretrained E5\(_{base}\) model as the initialization model. We utilized the code provided by the authors, retraining both the reward model and the LLM-R model while maintaining the hyperparameters from the original publication. For EPR (Rubin et al., 2021), we directly adopt the results presented in the LLM-R paper (Wang et al., 2023).

B Details about useful samples statistics

Figure 1 illustrates the distribution of the ratio of useful examples within demonstration pools across the AGNews, HellaSwag, and OpenBookQA datasets. We first identified "hard queries" defined as those queries that remained answered incorrectly by the LLM even when provided with the top-8 demonstrations initially retrieved by the E5\(_{base}\) (Wang et al., 2022). Subsequently, for each identified hard query, a set of 100 demonstrations was randomly sampled without replacement from the respective dataset’s full demonstration pool.

Each sampled demonstration was then individually assessed for its usefulness. A demonstration was considered "useful" if its concatenation with the hard query resulted in the LLM producing the correct answer. The *Ratio of Useful Examples* was calculated for each query as (Number of Useful Demonstrations/100) × 100%. To visualize the distribution, these ratios were binned, and the *Percentage of Queries* falling within each bin was calculated as (Number of Queries in Bin / Total Number of Hard Queries) × 100%, where "Number of Queries in Bin" is the count of hard queries with a ratio within that bin’s range, and "Total Number of Hard Queries" represents the total number of hard queries for that dataset.

C The Limitations of Contrastive Learning

In in-context learning, it is necessary to find demonstrations that are truly useful for the LLM to predict the ground truth. The goal of in-context learning is as follows:

\[
\arg\max_{(x_k, y_k) \in \mathcal{P}} P_M(y \mid \{(x_k, y_k)\}_{k=1}^K, x)
\]

However, the objective of contrastive learning is to minimize the following loss.

\[
\mathcal{L} = -\log \left( \frac{e^{s(x, x^+, y^+)}}{e^{s(x, (x^+, y^+))} + \sum_{i=1}^{N_{neg}} e^{s(x, (x_i^-, y_i^-))}} \right)
\]
In the optimization process of contrastive learning, the objective shifts to optimizing the retriever by comparing the similarity between different samples. Specifically, the model is trained to distinguish between positive samples \((x^+, y^+)\) and negative samples \((x^-, y^-)\), optimizing the model by maximizing the separation between positive and negative samples. This method enlarges the distance between positive and negative samples in the embedding space, which is not aligned with in-context learning. Furthermore, it completely ignores the information about the ground truth during optimization, leading to demonstrations selected by this method that do not improve the LLM’s ability to predict the ground truth.

D Derivation of the Optimization Objective

Starting from the marginal likelihood,

\[
\log P_{\mathcal{M}}(Y \mid \{(X_k, Y_k)\}_{k=1}^K, X) =
\log \int_z P_{\mathcal{M}}(Y \mid z, X) \times P_{\mathcal{M}}(z \mid \{(X_k, Y_k)\}_{k=1}^K, X) \, dz
\]

we introduce an arbitrary auxiliary distribution \(q(z)\) (with \( \int q(z) \, dz = 1 \)) and apply Jensen’s inequality:

\[
\begin{align*}
&\log P_{\mathcal{M}}(Y \mid \{(X_k, Y_k)\}_{k=1}^K, X) \\
&= \log \int_z q(z) \times \frac{P_{\mathcal{M}}(Y \mid z, X) P_{\mathcal{M}}(z \mid \{(X_k, Y_k)\}_{k=1}^K, X)}{q(z)} \, dz \\
&\geq \int_z q(z) \times \log \frac{P_{\mathcal{M}}(Y \mid z, X) P_{\mathcal{M}}(z \mid \{(X_k, Y_k)\}_{k=1}^K, X)}{q(z)} \, dz \\
&\triangleq \mathcal{L}(q).
\end{align*}
\]

The quantity \( \mathcal{L}(q) \) is the evidence lower bound (ELBO) (Kingma, 2013). We approximate \( q(z) \) by a Dirac \( \delta \) distribution: \( q(z) = \delta(z - z^*) \) which essentially assumes that the posterior \( P_{\mathcal{M}}(z \mid \{(X_k, Y_k)\}_{k=1}^K, X) \) is highly concentrated at the optimal latent variable \( z^* \). By the sifting property of the Dirac \( \delta \),

\[
\int_z \delta(z - z^*) f(z) \, dz = f(z^*),
\]

for any well-behaved function \( f(z) \).

Substituting this into the ELBO yields

\[
\begin{align*}
\mathcal{L}(q) &= \int_z \delta(z - z^*) \times \\
&\log \frac{P_{\mathcal{M}}(Y \mid z, X) P_{\mathcal{M}}(z \mid \{(X_k, Y_k)\}_{k=1}^K, X)}{\delta(z - z^*)} \, dz \\
&= \log \frac{P_{\mathcal{M}}(Y \mid z^*, X) P_{\mathcal{M}}(z^* \mid \{(X_k, Y_k)\}_{k=1}^K, X)}{\delta(0)}
\end{align*}
\]

Although \( \delta(0) \) is formally an infinite constant, it does not depend on \( z^* \) and can therefore be ignored during optimization. Maximizing the ELBO then amounts to optimizing

\[
\begin{align*}
\mathcal{L}(z) &= -\log P_{\mathcal{M}}(Y \mid z^*, X) \\
&\quad - \log P_{\mathcal{M}}(z^* \mid \{(X_k, Y_k)\}_{k=1}^K, X)
\end{align*}
\]
E  Dataset Details

<table>
  <tr>
    <th>Dataset Name</th>
    <th>Template</th>
    <th>Answer</th>
    <th>Train</th>
    <th>Test</th>
    <th>Metric</th>
  </tr>
  <tr>
    <td>AGNews</td>
    <td>"'{Sentence}' What is this text about? World, Sports, Business, or Technology?", "{Answer}"</td>
    <td>'World', 'Sports', 'Business', 'Technology'</td>
    <td>120,000</td>
    <td>7,600</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>BoolQ</td>
    <td>"'{Sentence1}' Can we conclude that {Sentence2}?", "{Answer}"</td>
    <td>'No', 'Yes'</td>
    <td>9,427</td>
    <td>3,270</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>MultiRC</td>
    <td>"'{Sentence1}' Question: "{Sentence2}" Response: "{Sentence3}" Does the response correctly answer the question?", "{Answer}"</td>
    <td>'No', 'Yes'</td>
    <td>27,243</td>
    <td>4,848</td>
    <td>F1</td>
  </tr>
  <tr>
    <td>RTE</td>
    <td>"'{Sentence1}' Based on the paragraph above can we conclude that "{Sentence2}"? Yes or No?", "{Answer}"</td>
    <td>'Yes', 'No'</td>
    <td>2,490</td>
    <td>277</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>SNLI</td>
    <td>"If "{Sentence1}", does this mean that "{Sentence2}"? Yes, No, or Maybe?", "{Answer}"</td>
    <td>'Yes', 'Maybe', 'No'</td>
    <td>549,367</td>
    <td>9,824</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>PAWS</td>
    <td>"'{Sentence1}' {Sentence2} Do these sentences mean the same thing?", "{Answer}"</td>
    <td>'No', 'Yes'</td>
    <td>49,401</td>
    <td>8,000</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>QQP</td>
    <td>"'{Sentence1}'" "{Sentence2}" Would you say that these questions are the same?", "{Answer}"</td>
    <td>'No', 'Yes'</td>
    <td>363,846</td>
    <td>40,430</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>Sentiment140</td>
    <td>"'{Sentence}' What is the sentiment of this tweet?", "{Answer}"</td>
    <td>'Negative', 'Positive'</td>
    <td>1,600,000</td>
    <td>359</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>SST2</td>
    <td>"Review: "{Sentence}" Is this movie review sentence negative or positive?", "answer"</td>
    <td>'Negative', 'Positive'</td>
    <td>67,349</td>
    <td>872</td>
    <td>Accuracy</td>
  </tr>
</table>

Table 7: Classification Task Dataset Details.

<table>
  <tr>
    <th>Dataset Name</th>
    <th>Template</th>
    <th>Answer</th>
    <th>Train</th>
    <th>Test</th>
    <th>Metric</th>
  </tr>
  <tr>
    <td>Winogrande</td>
    <td>"How does the sentence end? {Sentence}", "[Answer]"</td>
    <td>'A', 'B'</td>
    <td>40,398</td>
    <td>1,267</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>OpenBookQA</td>
    <td>"'{Sentence1}' {Sentence2}", "[Answer]"</td>
    <td>'A', 'B', 'C', 'D'</td>
    <td>4,957</td>
    <td>500</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>COPA</td>
    <td>"'{Sentence1}' What is the {Sentence2}?", "[Answer]"</td>
    <td>'A', 'B'</td>
    <td>400</td>
    <td>100</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>HellaSwag</td>
    <td>"What happens next in this paragraph? {Sentence}", "[Answer]"</td>
    <td>'A', 'B', 'C', 'D'</td>
    <td>39,905</td>
    <td>10,042</td>
    <td>Accuracy</td>
  </tr>
</table>

Table 8: Multi-choice Task Dataset Details.
<table>
  <tr>
    <th>Dataset Name</th>
    <th>Template</th>
    <th>Train</th>
    <th>Test</th>
    <th>Metric</th>
  </tr>
  <tr>
    <td>AESLC</td>
    <td>"What is the subject line for this email? {Sentence}", "[Label]"</td>
    <td>13,181</td>
    <td>1,750</td>
    <td>ROUGE-L</td>
  </tr>
  <tr>
    <td>Gigaword</td>
    <td>"Write a short summary for this text: {Sentence}", "[Answer]"</td>
    <td>2,044,465</td>
    <td>730</td>
    <td>ROUGE-L</td>
  </tr>
  <tr>
    <td>SQuAD v1</td>
    <td>"Please answer a question about the following article about {Sentence}: {Sentence1} {Sentence2}", "[Answer]"</td>
    <td>87,599</td>
    <td>10,570</td>
    <td>Exact Match</td>
  </tr>
  <tr>
    <td>CommonGen</td>
    <td>"Concepts: {Sentence}. Write a sentence that includes all these words.", "[Answer]"</td>
    <td>67,389</td>
    <td>4,018</td>
    <td>ROUGE-L</td>
  </tr>
  <tr>
    <td>DART</td>
    <td>"Triple: {Sentence} What is a sentence that describes this triple?", "[Answer]"</td>
    <td>62,659</td>
    <td>2,768</td>
    <td>ROUGE-L</td>
  </tr>
  <tr>
    <td>E2E NLG</td>
    <td>"Attributes: {Sentence}. Produce a detailed sentence about this restaurant.", "[Answer]"</td>
    <td>33,525</td>
    <td>1,847</td>
    <td>ROUGE-L</td>
  </tr>
</table>

Table 9: Generation Task Dataset Details.

F Case Study of Our GENICL Method
<table>
  <tr>
    <th>Task name</th>
    <td>AGNews</td>
  </tr>
  <tr>
    <th>Test Input</th>
    <td>"Dominant US captures gold with 79th straight win The US softball team completed its scorched-earth run through the Olympics on Monday with a 5-1 win over Australia, America’s third straight gold medal." What is this text about? World, Sports, Business, or Technology?</td>
  </tr>
  <tr>
    <th>Demonstration</th>
    <td>"US Women Shatter Olympic 800-Meter Freestyle Relay Record The United States has shattered a 17-year-old world record in the women’s Olympic 800-meter freestyle relay. " What is this text about? World, Sports, Business, or Technology? Sports</td>
  </tr>
  <tr>
    <th>Test Answer</th>
    <td>"Sports"</td>
  </tr>
  <tr>
    <th>Task name</th>
    <td>Sentiment140</td>
  </tr>
  <tr>
    <th>Test Input</th>
    <td>zomg!!! I have a G2!!!!!!! What is the sentiment of this tweet?</td>
  </tr>
  <tr>
    <th>Demonstration</th>
    <td>My brother got his update for his G1 and I ain’t got shit What is the sentiment of this tweet? Negative</td>
  </tr>
  <tr>
    <th>Test Answer</th>
    <td>"Positive"</td>
  </tr>
  <tr>
    <th>Task name</th>
    <td>RTE</td>
  </tr>
  <tr>
    <th>Test Input</th>
    <td>The west has preferred to focus on endangered animals, rather than endangered humans. African elephants are hunted down and stripped of tusks and hidden by poachers. Their numbers in Africa slumped from 1.2m to 600,000 in a decade until CITES - the Convention on International Trade in Endangered Species - banned the trade in ivory. Based on the paragraph above can we conclude that "African elephants are endangered by ivory poachers,"? Yes or No?</td>
  </tr>
  <tr>
    <th>Demonstration</th>
    <td>Three leading Japanese banks have announced an alliance forming the world’s largest financial group. Fuji Bank, Dai-Ichi Kangyo and the Industrial Bank of Japan say their operations will be integrated by the spring of 2002. Based on the paragraph above can we conclude that "Merger of Japanese Banks creates the world’s biggest bank."? Yes or No? Yes</td>
  </tr>
  <tr>
    <th>Test Answer</th>
    <td>"Yes"</td>
  </tr>
  <tr>
    <th>Task name</th>
    <td>SST2</td>
  </tr>
  <tr>
    <th>Test Input</th>
    <td>Review: "instead of a hyperbolic beat-charged urban western , it 's an unpretentious , sociologically pointed slice of life . " Is this movie review sentence negative or positive?</td>
  </tr>
  <tr>
    <th>Demonstration</th>
    <td>Review: "offers an unexpected window into the complexities of the middle east struggle and into the humanity of its people . " Is this movie review sentence negative or positive? Positive</td>
  </tr>
  <tr>
    <th>Test Answer</th>
    <td>"Positive"</td>
  </tr>
  <tr>
    <th>Task name</th>
    <td>CommonGen</td>
  </tr>
  <tr>
    <th>Test Input</th>
    <td>Concepts: kid, yard, ball. Write a sentence that includes all these words.</td>
  </tr>
  <tr>
    <th>Demonstration</th>
    <td>Concepts: kid, grass, crawl. Write a sentence that includes all these words. A kid is about to crawl through some grass.</td>
  </tr>
  <tr>
    <th>Test Answer</th>
    <td>"A kid is playing with a ball in his yard."</td>
  </tr>
</table>

Table 10: Case Study of Our GenICL.

Reviewer paper summaries: This paper proposes a new ICL demo selection method, which utilizes the LLM feedback to optimize the demonstration selection for ICL. Extensive experiments verify the effectiveness of the method.

The paper proposes a context demonstration selection method based on the Generative Preference Learning framework, GENICL. This method leverages LLM feedback to optimize the demonstration selection objective directly, thereby overcoming the limitations of existing retrieval-based methods. Extensive experiments conducted on 19 datasets across 11 task categories demonstrate that GENICL significantly outperforms existing methods in selecting the most effective demonstrations, leading to enhanced in-context learning (ICL) performance.

This paper proposed a generative preference learning framework called GENICL that optimizes demonstration selection for in-context learning (ICL) using LLM feedback. Through paired preference learning and extensive evaluation across 19 datasets in 11 task categories, GENICL improves the effectiveness of demonstration selection for ICL.
Summary of strengths: 1. The studied problem is important.
2. The writing is easy to follow.
3. The experiment results are extensive.

1. A novel generative preference learning framework for in-context learning (ICL) is proposed by integrating LLM feedback, which applies to a wide range of tasks and encourages research into more generalized demonstration selection methods.
2. Comprehensive and rigorous experimental validation has been conducted.

1. The paper is well-written and easy to follow;
2. The idea of preference learning for demonstration selection is reasonable;
3. Extensive experiments across 11 task categories show the improvements
Summary of weaknesses: 1. It seems that during the optimization stage, the method only considers single demonstration selection, which ignores the interaction between demonstrations, leading to suboptimal performance.
2. The experiments lack important baselines in the domain of demonstration selection, including [1, 2]
3. It will be better if the authors could give more case study on the method. 

references:
[1] Curriculum Demonstration Selection for In-Context Learning
[2] Revisiting Demonstration Selection Strategies in In-Context Learning

1. Relevant works [1][2] have been omitted from citation and comparison, both of which share similarities with GENICL. Notably, work [2] introduces latent variables based on Bayesian inference for learning sample features, selecting representative samples for each type of data, and demonstrates that training and selection on a smaller LLM can generalize to models of other sizes and architectures.
2. Compared to work [2], GENICL’s main advantage lies in its ability to generalize across more tasks. However, it requires separate training for different models, which limits its generalizability to closed-source models.
3. The training cost is relatively high. The initial purpose of ICL was to reduce training overhead for efficiently transferring LLM capabilities to new tasks. However, GENICL requires task-specific training, and changing models also necessitates retraining, with training steps reaching up to 20,000.
4. Absent experiments on the latest models, such as the Llama3 series and Qwen2.5. This is particularly significant as these models are already widely adopted and possess great ICL capabilities.

[1] P. Shi, R. Zhang, H. Bai, and J. Lin, "XRICL: cross-lingual retrieval-augmented in-context learning for cross-lingual text-to-sql semantic parsing".

[2] Wang X, Zhu W, Wang W Y. "Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning".

1. The models evaluated are not state-of-the-art LLMs, which may limit the impact and generalizability of the paper’s conclusions for the broader community. Further evaluation on stronger baselines, such as LLaMA 3.1 and LLaMA 3.2 models, is recommended.

2. The method relies on the LLM to generate preference scores over the entire training set or a reduced subset. However, the paper does not provide details on the time cost or the potential performance degradation when using a reduced subset. It remains unclear about the method’s scalability and reliability for larger LLMs, where additional computation is costly. A more in-depth analysis is needed to address these issues.
Comments, suggestions and typos: See weakness above

None.

See the weakness.
Overall assessment scores: 2.5

3

3
Metareview score: 3
Metareview: This paper investigates the example selection of in-context learning. The author proposes the GENICL method to optimize demonstration selection using LLM feedback. Extensive experiments evaluate 19 datasets across 11 task
Summary of reasons to publish: - All reviewers agree that extensive experiments are conducted.
- The preference learning framework effectively addressed the issue highlighted in this work.
- The paper is well-written.