Selecting Demonstrations for Many-Shot In-Context Learning via Gradient Matching
Authors: Jianfei Zhang, Bei Li, Jun Bai, Rumei Li, Yanmeng Wang, Chenghua Lin, Wenge Rong
Accepted at: acl25

Selecting Demonstrations for Many-Shot In-Context Learning via Gradient Matching

Jianfei Zhang1, Bei Li2, Jun Bai4, Rumei Li1,2, Yanmeng Wang3, Chenghua Lin5, Wenge Rong1
1School of Computer Science and Engineering, Beihang University, China
2Meituan, Inc., China 3Ping An Technology, China
4Beijing Institute for General Artificial Intelligence, China
5Department of Computer Science, The University of Manchester, UK
{zhangjf, lirumei3232, w.rong}@buaa.edu.cn, libei17@meituan.com
baijun@bigai.ai, wangyanmeng219@pingan.com.cn, chenghua.lin@manchester.ac.uk

Abstract

In-Context Learning (ICL) empowers Large Language Models (LLMs) for rapid task adaptation without Fine-Tuning (FT), but its reliance on demonstration selection remains a critical challenge. While many-shot ICL shows promising performance through scaled demonstrations, the selection method for many-shot demonstrations remains limited to random selection in existing work. Since the conventional instance-level retrieval is not suitable for many-shot scenarios, we hypothesize that the data requirements for in-context learning and fine-tuning are analogous. To this end, we introduce a novel gradient matching approach that selects demonstrations by aligning fine-tuning gradients between the entire training set of the target task and the selected examples, so as to approach the learning effect on the entire training set within the selected examples. Through gradient matching on relatively small models, e.g., Qwen2.5-3B or Llama3-8B, our method consistently outperforms random selection on larger LLMs from 4-shot to 128-shot scenarios across 9 diverse datasets. For instance, it surpasses random selection by 4% on Qwen2.5-72B and Llama3-70B, and by around 2% on 5 closed-source LLMs. This work unlocks more reliable and effective many-shot ICL, paving the way for its broader application.

1 Introduction

In-Context Learning (ICL) enables pre-trained Large Language Models (LLMs) to perform tasks by learning from input-output examples (or “demonstrations”) provided during inference (Brown et al., 2020). This allows LLMs to adapt to new tasks through forward propagation, without weight updates via back-propagation, offering a flexible alternative to traditional fine-tuning.

Since the performance of ICL is often sensitive to the choice of demonstrations (Liu et al., 2022), significant efforts have been made to improve demonstration selection. Most studies concentrate on instance-level retrieval, aiming to identify suitable demonstrations for each test query independently (Luo et al., 2023; Rubin et al., 2022). This mainly involves considering similarity (Luo et al., 2023; Rubin et al., 2022) between demonstrations and the query, as well as auxiliary factors such as complexity (Fu et al., 2023), perplexity (Gonen et al., 2023), difficulty (Drozdov et al., 2023), and diversity (Li and Qiu, 2023). Another line of work focuses on task-level selection, which seeks to find a fixed set of demonstrations that achieves the best average performance across all test queries from a target task (Zhang et al., 2022; Wang et al., 2023). Current approaches primarily explore reinforcement learning on the selection policy (Zhang et al., 2022) and Bayesian inference to explain the demonstration effectiveness (Wang et al., 2023) as

![Instance-Level In-Context Learning vs Task-Level In-Context Learning](page_682_370_370_180.png)

Figure 1: Instance-level retrieved demonstrations are not suitable for Many-Shot ICL, since the variable context cannot utilize prefix-caching and can lead to theoretically \( O(n^2) \) inference complexity (Vaswani et al., 2017) when using \( n \)-shot demonstrations. Here we illustrate the runtime of Llama3-70B (Dubey et al., 2024) on CM-SQA (Talmon et al., 2019) using vLLM with 4 A100 GPUs in tensor-parallel (Kwon et al., 2023).
potential solutions.

However, for the recently emerged Many-Shot in-context learning (Agarwal et al., 2024) paradigm, demonstrations are selected simply by random in existing work (Agarwal et al., 2024; Bertsch et al., 2024; Song et al., 2025; Jiang et al., 2024), which may lead to suboptimal performance. Nevertheless, existing methods for few-shot demonstration selection are not well-suited to many-shot scenarios. On one hand, applying instance-level demonstrations conflicts with caching and reusing the hidden states of the same long context in language models (Pope et al., 2023), leading to theoretically \( O(n^2) \) inference complexity (Vaswani et al., 2017) for \( n \)-shot demonstrations, as illustrated in Fig. 1. On the other hand, existing task-level selection methods are designed for very few-shot scenarios (e.g., 4-shot in their implementations), facing challenges from exploration complexity (Zhang et al., 2022) and performance early saturation (Wang et al., 2023) for more shots. To further release the potential of many-shot ICL, it now requires a selection method with scalable effectiveness to many-shot scenarios.

To address such research gap, we revisit ICL demonstration selection from a “learning” perspective—we hypothesize that the data requirements for in-context learning and fine-tuning are analogous. Building on this premise, we conduct latent concept learning (Wang et al., 2023) on a small Language Model (LM) to learn the optimal in-context task embeddings. Then we compute the latent gradient that each example provides to the learning process. Finally, we select \( n \)-shot demonstrations from the whole fine-tuning set with the minimized \( L_2 \) distance between their average latent gradients, so as to approach the learning effect on all examples within the selected \( n \)-shot demonstrations.

We validate our proposed method on 9 datasets from 5 distinct NLP tasks, each covering scenarios from 4-shot to 128-shot. Our method selects demonstrations through gradient matching on relatively small models, e.g., Qwen2.5-3B or Llama3-8B, and surpasses the widely-adopted random selection by an average of 4% on Llama3-70B and Qwen2.5-72B. Furthermore, the selected demonstration set exhibits transferability to closed-source LLMs, consistently outperforming random selection by around 2% on Qwen-turbo, GLM-4-flash, Doubao-pro-32k, GPT-4o-mini, and DeepSeek-V3. Our source code is available at https://github.com/zhangjf-nlp/ManyShotICL-CLG.git.

2 Related Work

2.1 Many-Shot In-Context Learning

In-Context Learning (ICL) refers to the capability of LLMs to learn from data during inference through forward propagation, without the need for backward propagation or weight updates (Li et al., 2023b). Prior work is limited to few-shot scenarios (Jiang et al., 2024) by the context window size, e.g., 2048 tokens in GPT-3 (Brown et al., 2020).

Recently, advancements in expanding the context window size of LLMs (Ding et al., 2024; Chen et al., 2023), e.g., up to 128k tokens in GPT-4o (OpenAI, 2023) and Qwen2.5 (Yang et al., 2024), have enabled the exploration of many-shot ICL (Agarwal et al., 2024). These studies have observed that Many-Shot ICL, which includes up to hundreds or even thousands of demonstrations, can make substantial improvements compared to few-shot ICL on various tasks (Agarwal et al., 2024; Bertsch et al., 2024). In this paradigm, the benefits of instance-level retrieval over using a fixed random set of demonstrations tend to diminish (Bertsch et al., 2024), while a fixed demonstration set can largely reduce the inference cost through prefix-caching (Agarwal et al., 2024). Therefore, researchers tend to randomly select a demonstration set and reuse it across all queries in many-shot ICL (Agarwal et al., 2024; Bertsch et al., 2024; Jiang et al., 2024; Song et al., 2025).

While random selection is widely adopted for many-shot ICL, it is underexplored whether this strategy is truly optimal. This motivates our work to explore and develop a better demonstration selection strategy for the many-shot paradigm.

2.2 Demonstration Selection

The performance of ICL is sensitive to the choice of demonstrations (Liu et al., 2022; Perez et al., 2021), prompting efforts in demonstration selection. Existing approaches fall into two categories:

Instance-level retrieval selects demonstrations for each query using semantic similarity, such as cosine similarity (Rubin et al., 2022) and BM25 (Li et al., 2023a). Some work further emphasizes additional factors including diversity (Li and Qiu, 2023), complexity (Fu et al., 2023), difficulty (Drozdov et al., 2023), and perplexity (Gonen et al., 2023). While providing relevant and useful information, instance-level retrieval prevents the use of prefix-caching during inference and leads to the-
Figure 2: The overview of our proposed method for many-shot demonstration selection. We perform latent concept learning (Wang et al., 2023) on a small LM and compute the Curriculum Latent Gradient (CLG) provided by each example throughout the \( E \)-epoch training. Then we find an \( n \)-shot demonstration set to mimic the average gradient on the entire training set. We show that the selected \( n \)-shot demonstrations can effectively improve the many-shot In-Context Learning performances on both same-series larger LMs and various closed-source LLMs.

oretically \( O(n^2) \) inference complexity (Vaswani et al., 2017) for \( n \)-shot demonstrations. In contrast, the runtime of many-shot ICL under a fixed demonstration set increases only linearly with a large number of shots (Agarwal et al., 2024). This inefficiency makes instance-level retrieval less practical as the number of demonstration shots increases.

Task-level selection aims to identify a fixed set of demonstrations that achieve optimal average performance across all queries from a target task. For example, Zhang et al. (2022) use Reinforcement Learning (RL) to gradually explore the optimal demonstration set. Wang et al. (2023) view LLMs as latent variable models and select demonstrations that can best infer the optimal latent task concept. However, these methods are both limited to few-shot scenarios (e.g., 4-shot in their implementations), facing challenges from complexity in exploring potential many-shot demonstration sets (Zhang et al., 2022) and performance early saturation beyond 4-shot (Wang et al., 2023) respectively.

In summary, current demonstration selection methods are fundamentally designed for and evaluated within few-shot scenarios, facing challenges in generalizing to many-shot scenarios. To the best of our knowledge, our work presents the first attempt to enhance demonstration selection for many-shot ICL, beyond random selection in existing work.

3 Methodology

Previous research has identified several key attributes that contribute to the effectiveness of in-context demonstrations, such as similarity (Luo et al., 2023), diversity (Li and Qiu, 2023), and coverage (Li and Qiu, 2023). While intuitively beneficial, these attributes lack a direct connection to the learning dynamics induced by demonstrations within the language model. We posit that the ideal demonstration set for ICL should not only present static attributes of high-quality fine-tuning data but, more fundamentally, should actively facilitate and guide the language model’s learning process on the target task. Drawing inspiration from the principles of effective training data selection in supervised learning, particularly from the field of dataset condensation (Zhao et al., 2021), we introduce Curriculum Latent Gradient (CLG), a novel approach for task-level in-context demonstration selection. CLG leverages the concept of latent task embeddings and analyses their learning trajectory to identify demonstrations that can effectively guide the model’s learning process.

We begin by reviewing Latent Concept Learning (Sec. 3.1), highlighting its strengths and limitations as a foundation. We then articulate the core principles behind Curriculum Latent Gradient (Sec. 3.2), detailing how we capture and leverage
the learning dynamics of latent task embeddings to guide demonstration selection. Finally, we describe the demonstration selection process based on these learning dynamics (Sec. 3.3). An overview of our proposed methodology is depicted in Fig. 2.

3.1 Preliminary: Latent Concept Learning

Latent Concept Learning Wang et al. (2023) aims to learn an optimal latent task concept \( z_\theta \) as a proxy of in-context demonstrations. Specifically, prediction from a language model \( P_M \) through in-context learning over demonstrations \( D_n = \{ (x_1, y_1), \ldots, (x_n, y_n) \} \) can be expressed in Eq. 1,

\[
P(y \mid x; D_n) = P_M(y \mid x_1, y_1, \ldots, x_n, y_n, x)
\]

(1)

where \( D_n \) contains the \( n \)-shot demonstrations, and faces optimization challenges for its discrete nature. To address this, they replace the demonstration set with continuous and optimizable embeddings of the *latent task concept* \( z_\theta \), as expressed in Eq. 2,

\[
P(y \mid x; z_\theta) = P_M(y \mid z_\theta, x)
\]

(2)

where \( z_\theta \) can be optimized through maximum likelihood estimation on the entire training set \( D_N \), as formulated in Eq. 3.

\[
\hat{z}_\theta = \arg\max_{z_\theta} \sum_{(x, y) \in D_N} \log P(y \mid x; z_\theta)
\]

(3)

Latent-Bayesian subsequently utilizes the optimized \( \hat{z}_\theta \) to select task-level demonstrations. Under a series of simplifying assumptions, it finally selects the top-\( n \) demonstrations with the highest posterior probabilities of \( \hat{z}_\theta \) in language model \( P_M \), as expressed in Eq. 4, where the top-\( n \) operator selects the \( n \) samples with highest function values.

\[
D_n = \underset{(x, y) \in D_N}{\operatorname{top}\text{-}n} P_M(z_\theta \mid x, y)
\]

(4)

While Latent-Bayesian offers an interpretable approach under specific assumptions, its reliance on static posterior probabilities and independence assumptions limits its effectiveness, particularly for many-shot ICL. Crucially, it overlooks the *dynamic interactions* between demonstrations – such as redundancy or synergistic effects – which become increasingly vital as the number of demonstrations grows. This motivates our departure from static, result-oriented approaches towards a method that explicitly considers the *learning process* itself.

3.2 Curriculum Latent Gradient

Our method borrows the idea of latent concept learning, i.e., learning task concept embeddings that play the role of context. Notably, we focus on the learning process instead of the learnt result. We quantify the learning dynamics under different demonstrations through the optimizing gradients on the task concept embeddings provided by each demonstration, throughout the training process.

Specifically, given a small language model \( P_M \) and a target task, we construct \( k \) prefix token embeddings \( z_\theta \in R^{k \times h} \) to represent the latent task concept, where \( h \) denotes the embedding size. These embeddings are then trained on the target task with the negative log-likelihood loss in Eq. 5.

\[
\mathcal{L}(z_\theta; x, y) = -\log P_M(y|z_\theta, x)
\]

(5)

We randomly initialize the latent concept tokens \( z_\theta = z_{\theta_0} \), and employ Stochastic Gradient Descent (Bottou, 2010) to optimize \( z_\theta \) over the full training set \( D_N \) for \( E \) epochs. Crucially, we save the learnt latent concept at the end of each epoch, denoted as \( z_{\theta_1}, z_{\theta_2}, \ldots, z_{\theta_E} \). For each training example \( (x, y) \in D_N \), we calculate its curriculum latent gradient, \( G_{0:E}(x, y) \in R^{h \times h \times (E+1)} \), which is a concatenation of the gradients of the loss function with respect to \( z_\theta \), evaluated both at the initial point \( z_\theta = z_{\theta_0} \) and at the end of each epoch \( z_\theta = z_{\theta_i} \) for \( 1 \leq i \leq E \). This is formulated in Eq. 6, where \( [ : ] \) denotes the concatenation operation.

\[
G_{0:E}(x, y) = \left[ \nabla_{z_\theta} \mathcal{L}(z_\theta; x, y) \right]_{z_\theta = z_{\theta_0}} ; \\
\quad \nabla_{z_\theta} \mathcal{L}(z_\theta; x, y) \Big|_{z_\theta = z_{\theta_1}} ; \\
\quad \ldots ; \\
\quad \nabla_{z_\theta} \mathcal{L}(z_\theta; x, y) \Big|_{z_\theta = z_{\theta_E}}
\]

(6)

3.3 Demonstration Selection via Gradient Matching

To select an effective \( n \)-shot demonstration set for ICL, we aim to identify an optimal \( n \)-shot subset \( D_n^* \subseteq D_N \) that induces similar learning dynamics in the LM as the full training set \( D_N \). Specifically, we propose to achieve this by minimizing the \( L_2 \) distance between the average curriculum latent gradient over \( D_N \) and that over the \( n \)-shot subset \( D_n \), as formalized in Eq. 7, ensuring that the model’s learning behaviour on the subset \( D_n \), as summarized in \( G_{0:E}(D_n) \), closely aligns with that on the
Algorithm 1 Gradient Matching through Greedy Search and Local Optimization

Input: the entire training set \( D_N \), the curriculum latent gradient \( G_{0:E}(x, y) \) for each example in the training set \( (x, y) \in D_N \), the number of shots \( n \), the maximum iteration steps \( l \) for local optimization.

Output: the \( n \)-shot demonstrations \( D_n^* \) that approximately minimize the gradient matching objective in Eq. 7.

1: Compute the average curriculum latent gradient on \( D_N \):
\[
G_{0:E}(D_N) = \mathbb{E}_{(x, y) \in D_N} [G_{0:E}(x, y)].
\]
2: Initialize \( D_0 = \emptyset \).
3: for \( i = 1 \) to \( n \) do
4:   Find the next example \( (x_j, y_j) \in (D_N \setminus D_{i-1}) \) that minimizes the distance between the average curriculum latent gradient on \( D_N \) and that on \( D_{i-1} \cup \{(x_j, y_j)\} \).
5:   Set \( D_i = D_{i-1} \cup \{(x_j, y_j)\} \).
6: end for
7: Set \( d_{old} = \|G_{0:E}(D_N) - \mathbb{E}_{(x, y) \in D_n} G_{0:E}(x, y)\|_2 \).
8: for \( i = 1 \) to \( l \) do
9:   Find \( (x_k, y_k) \in D_N \setminus D_n \) and \( (x_k, y_k) \in D_n \) to minimize the distance between the average curriculum latent gradient on \( D_N \) and that on \( D_n \cup \{(x_j, y_j)\} \setminus \{(x_k, y_k)\} \).
10:  Set \( D_{new} = D_n \cup \{(x_j, y_j)\} \setminus \{(x_k, y_k)\} \).
11:  Set \( d_{new} = \|G_{0:E}(D_N) - \mathbb{E}_{(x, y) \in D_{new}} G_{0:E}(x, y)\|_2 \).
12:  if \( d_{new} \geq d_{old} \) then
13:    break
14:  else
15:    Set \( D_n = D_{new} \) and \( d_{old} = d_{new} \).
16:  end if
17: end for
18: return \( D_n \) as the optimized \( n \)-shot demonstrations \( D_n^* \).

complete set \( D_N \), as summarized in \( G_{0:E}(D_N) \).

\[
G_{0:E}(D_N) = \mathbb{E}_{(x, y) \in D_N} [G_{0:E}(x, y)]
\]
\[
G_{0:E}(D_n) = \mathbb{E}_{(x, y) \in D_n} [G_{0:E}(x, y)]
\]
\[
D_n^* = \arg\min_{D_n \subset D_N} \|G_{0:E}(D_N) - G_{0:E}(D_n)\|_2
\]

Since the optimization problem in Eq. 7 is indeed an NP-Complete Subset Sum Problem (SSP) (Lagarias and Odlyzko, 1985), we resort to heuristic approximation to find a near-optimal solution for \( D_n^* \), involving two key phases: *Greedy Search* and *Local Optimization*. In the greedy phase, we incrementally construct the \( n \)-shot subset \( D_n \) by iteratively adding the example that minimizes the target \( L_2 \) distance. Then the local optimization phase refines the \( n \)-shot subset through up to \( l = 32 \) iterations to further reduce the target \( L_2 \) distance, by replacing a selected example with an unselected one. The detailed steps of the selection algorithm are presented in Algorithm 1, which requires only a few minutes to execute in practice.

<table>
  <tr>
    <th></th>
    <th>designed for task-level</th>
    <th>extended from instance-level</th>
  </tr>
  <tr>
    <td>learning-free</td>
    <td>Random<br>Best-of-N</td>
    <td>BM25-Major<br>BGE-KMeans</td>
  </tr>
  <tr>
    <td>learning-based</td>
    <td>Latent-Bayesian<br>CLG (ours)</td>
    <td>EPR-KMeans</td>
  </tr>
</table>

Table 1: Types of Methods

4 Experiments

We conduct experiments on 9 datasets from 5 distinct tasks, and show that demonstrations selected by our proposed method can effectively improve the many-shot ICL performances over random selection as well as some straightforward methods.

4.1 Baselines

In addition to random selection, the commonly-adopted method in existing work of many-shot ICL, we examine various straightforward methods for demonstration selection, including those designed for task-level selection and those extended from instance-level retrieval. We classify these methods in Table 1 and introduce them in details below.

**Random:** The most basic demonstration selection method, which is widely-adopted in existing work for many-shot ICL (Agarwal et al., 2024; Bertsch et al., 2024; Song et al., 2025; Jiang et al., 2024). We implement this across 5 random seeds and report the mean and standard deviation.

**BM25-Major:** BM25 (Robertson and Zaragoza, 2009) is a popular term-based scoring method for instance-level retrieval (Li et al., 2023a). We extend it to task-level demonstration selection through Majority Voting, i.e., selecting \( n \)-shot demonstrations with the \( n \) highest average scores to be retrieved by the other examples on the training set.

**BGE-KMeans:** BGE-M3 (Chen et al., 2024) provides off-the-shelf sentence embeddings with state-of-the-art performances on multiple retrieval tasks. We extend it to task-level selection through KMeans clustering, as implemented in the Scikit-Learn library, over its text embeddings.

**EPR-KMeans:** EPR (Rubin et al., 2022) learns to retrieve demonstrations through contrastive learning, supervised by the ICL likelihoods on a relatively small LM. We train EPR retrievers and apply them to task-level selection through KMeans clustering over their learnt dense embeddings.
<table>
  <tr>
    <th>Type</th>
    <th>Dataset</th>
    <th>Task</th>
    <th>#Train</th>
    <th>#Validation</th>
    <th>Avg. Tokens</th>
    <th>Metric</th>
  </tr>
  <tr>
    <td rowspan="4">Classification</td>
    <td>SST-5 (Socher et al., 2013)</td>
    <td>Sentiment Analysis</td>
    <td>8,544</td>
    <td>1,101</td>
    <td>26.55</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td>MNLI (Williams et al., 2018)</td>
    <td>Natural Language Inference</td>
    <td>50,000</td>
    <td>10,000</td>
    <td>43.21</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td>CMSQA (Talmor et al., 2019)</td>
    <td>Commonsense Reasoning</td>
    <td>9,741</td>
    <td>1,221</td>
    <td>45.70</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td>HellaSwag (Zellers et al., 2019)</td>
    <td>Commonsense Reasoning</td>
    <td>50,000</td>
    <td>10,000</td>
    <td>79.42</td>
    <td>Acc</td>
  </tr>
  <tr>
    <td rowspan="4">Open-ended</td>
    <td>GeoQuery (Shaw et al., 2021)</td>
    <td>Code Generation</td>
    <td>600</td>
    <td>280</td>
    <td>22.66</td>
    <td>EM</td>
  </tr>
  <tr>
    <td>NL2Bash (Lin et al., 2018)</td>
    <td>Code Generation</td>
    <td>8,090</td>
    <td>609</td>
    <td>32.56</td>
    <td>BLEU</td>
  </tr>
  <tr>
    <td>Break (Wolfson et al., 2020)</td>
    <td>Semantic Parsing</td>
    <td>44,321</td>
    <td>7,760</td>
    <td>61.44</td>
    <td>LF-EM</td>
  </tr>
  <tr>
    <td>MTOP (Li et al., 2021)</td>
    <td>Semantic Parsing</td>
    <td>15,667</td>
    <td>2,235</td>
    <td>34.12</td>
    <td>EM</td>
  </tr>
  <tr>
    <td></td>
    <td>SMCalFlow (Andreas et al., 2020)</td>
    <td>Semantic Parsing</td>
    <td>50,000</td>
    <td>10,000</td>
    <td>53.78</td>
    <td>EM</td>
  </tr>
</table>

Table 2: Datasets used in our experiments. We use at most 50,000 training instances and 10,000 validation instances. Following previous work (Ye et al., 2023), we use Accuracy (Acc) on classification tasks, and use Exact Match (EM), Logical Form EM (LF-EM) (Hasson and Berant, 2021), and character-level BLEU on open-ended tasks.

Best-of-N: Best-of-N is a commonly used baseline for Reinforcement Learning (RL), and demonstrates competitive performance to RL-based task-level demonstration selection (Zhang et al., 2022). We implement this through randomly selecting N=5 demonstration sets, evaluating them on a relatively small LM over all training instances, and selecting the best-performing one.

Latent-Bayesian: The task-level demonstration selection method based on Bayesian inference (Wang et al., 2023), as we introduce in Sec 3.1.

4.2 Datasets and Evaluation
We list all the datasets in Table 2, paired with their tasks, sizes, average tokens, and evaluation metrics. We illustrate cases on each dataset in Appendix A. We select demonstrations from the training set and report their ICL performances on the validation set, since the test set is private for some datasets.

4.3 Implementation Details
For LM-based approaches, we utilize Llama3-8B to select demonstrations for Llama3 series LLMs (Dubey et al., 2024), and utilize Qwen2.5-3B to select demonstrations for Qwen2.5 series LLMs (Yang et al., 2024) and closed-source LLMs. We train EPR retrievers from BERT-base-uncased (Devlin et al., 2019) under the supervision of LMs, using a learning rate of \( 1e - 5 \) and a batch size of 8, for at most 120 epochs until regression. We conduct latent concept learning on LMs for Latent-Bayesian and CLG, with a learning rate of \( 1e - 3 \) and a batch size of 64 for \( E = 10 \) epochs.

For ICL performance evaluation, we utilize the vLLM framework (Kwon et al., 2023) to perform greedy search, involving 4-shot, 8-shot, 16-shot, 32-shot, 64-shot, and 128-shot settings.\footnote{Since the context window sizes of Llama3-8B and classification tasks, we only allow the model to generate tokens contained in the options.}

4.4 Main Results
We illustrate the results of 128-shot ICL on Llama3-70B and Qwen2.5-72B in Table 3. It can be observed that, demonstrations selected by random are in fact generally suboptimal for task-level in-context learning. Among all methods, our proposed CLG performs the best on average, surpassing that of random selection by 4% on both models.

We further evaluate the selected demonstrations by Random, Best-of-N, and CLG on 5 closed-source models: Qwen-turbo, GLM-4-flash, Doubao-pro, GPT-4o-mini, and DeepSeek-V3.\footnote{https://www.alibabacloud.com/help/en/model-studio/getting-started/models, https://bigmodel.cn/dev/howuse/model, https://www.volcengine.com/product/doubao, https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence, and https://api-docs.deepseek.com/news/news1226.} We include at most 300 instances from each dataset and constrain the context length to within 7000 tokens due to budget limitations. The average performances across all datasets are illustrated in Table 4, where CLG consistently outperforms Random and Best-of-N on all closed-source models. This demonstrates the transferability of demonstrations selected by our CLG to different series of LLMs, as well as the effectiveness of CLG on closed-source LLMs.

4.5 Overall Scaling Performance
In Fig. 3, we examine the scaling trends of ICL performance w.r.t. the model size and the shot number. For simplicity, we present the average performance scores over 9 datasets. The results reveal a clear
<table>
  <tr>
    <th>Method</th>
    <th>SST-5</th>
    <th>MNLI</th>
    <th>CMSQA</th>
    <th>HeSwag.</th>
    <th>GeoQ.</th>
    <th>NL2Bash</th>
    <th>Break</th>
    <th>MTOP</th>
    <th>SMCal.</th>
    <th>Average</th>
  </tr>
  <tr>
    <td><i>Llama3-70B</i></td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Random</td>
    <td>43.71</td>
    <td>61.83</td>
    <td><b>84.73</b></td>
    <td>76.11</td>
    <td>73.36</td>
    <td>29.72</td>
    <td>35.55</td>
    <td>44.66</td>
    <td>36.68</td>
    <td>54.04</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td>43.69</td>
    <td>51.36</td>
    <td>82.23</td>
    <td>72.01</td>
    <td>43.21</td>
    <td>27.92</td>
    <td>15.05</td>
    <td>8.41</td>
    <td>14.38</td>
    <td>39.81</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td>46.41</td>
    <td>65.09</td>
    <td>83.78</td>
    <td>73.90</td>
    <td><b>84.64</b></td>
    <td>28.39</td>
    <td>35.13</td>
    <td>44.79</td>
    <td><b>38.73</b></td>
    <td>55.65</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>43.69</td>
    <td>70.92</td>
    <td>84.19</td>
    <td>74.57</td>
    <td>80.71</td>
    <td><b>29.76</b></td>
    <td>35.86</td>
    <td>43.40</td>
    <td>37.66</td>
    <td>55.64</td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>43.05</td>
    <td>49.78</td>
    <td>83.62</td>
    <td>71.32</td>
    <td>78.21</td>
    <td>22.92</td>
    <td><b>38.39</b></td>
    <td><b>50.43</b></td>
    <td>38.10</td>
    <td>52.87</td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>46.59</td>
    <td>69.84</td>
    <td>84.60</td>
    <td>70.09</td>
    <td>62.14</td>
    <td>29.20</td>
    <td>32.59</td>
    <td>41.21</td>
    <td>23.13</td>
    <td>51.04</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>48.32</b></td>
    <td><b>76.37</b></td>
    <td>84.52</td>
    <td><b>80.77</b></td>
    <td><b>84.64</b></td>
    <td>29.59</td>
    <td><b>37.33</b></td>
    <td><b>47.74</b></td>
    <td><b>40.35</b></td>
    <td><b>58.85</b></td>
  </tr>
  <tr>
    <td><i>Qwen2.5-72B</i></td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Random</td>
    <td>36.00</td>
    <td>58.20</td>
    <td>87.47</td>
    <td>86.58</td>
    <td>57.29</td>
    <td>33.28</td>
    <td>36.67</td>
    <td>45.32</td>
    <td>38.23</td>
    <td>53.23</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td>37.78</td>
    <td>45.94</td>
    <td>86.65</td>
    <td>85.61</td>
    <td>52.50</td>
    <td>38.11</td>
    <td>22.81</td>
    <td>16.24</td>
    <td>11.57</td>
    <td>44.13</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td>37.24</td>
    <td>49.65</td>
    <td>87.71</td>
    <td>86.67</td>
    <td>61.79</td>
    <td><b>40.86</b></td>
    <td>35.84</td>
    <td>43.71</td>
    <td><b>41.29</b></td>
    <td>53.86</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>36.60</td>
    <td>60.57</td>
    <td>87.55</td>
    <td>86.83</td>
    <td>61.07</td>
    <td>37.13</td>
    <td>35.73</td>
    <td>46.89</td>
    <td>39.22</td>
    <td>54.62</td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>36.15</td>
    <td>58.41</td>
    <td><b>87.96</b></td>
    <td>86.22</td>
    <td><b>62.86</b></td>
    <td>26.53</td>
    <td><b>39.12</b></td>
    <td><b>47.53</b></td>
    <td>39.65</td>
    <td>53.83</td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>27.25</td>
    <td>47.96</td>
    <td>85.75</td>
    <td>86.72</td>
    <td>27.50</td>
    <td>37.36</td>
    <td>35.35</td>
    <td>5.23</td>
    <td>22.61</td>
    <td>41.75</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>38.33</b></td>
    <td><b>77.29</b></td>
    <td>87.71</td>
    <td><b>87.68</b></td>
    <td>62.50</td>
    <td><b>45.19</b></td>
    <td>37.07</td>
    <td>47.07</td>
    <td>40.16</td>
    <td><b>58.11</b></td>
  </tr>
</table>

Table 3: Main results (in %) of Llama3-70B and Qwen2.5-72B under the 128-shot setting. The best scores on each model are in bold, and the second-best ones are underlined. Full results are illustrated in Appendix C. BM25-Major and Latent-Bayesian generally perform worse than Random selection; we analyse the reasons in Appendix B.

![Line plots showing scaling performances with respect to number of shots and model size for different methods](page_1012_1012_1022_312.png)

Figure 3: The scaling performances with respect to (w.r.t.) the number of shots and the model size. For random selection under 5 different random seeds, standard deviations are illustrated through error bars.

<table>
  <tr>
    <th>Method</th>
    <th>Qwen.</th>
    <th>GLM4</th>
    <th>DouB</th>
    <th>GPT4o</th>
    <th>DeepS</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>60.08</td>
    <td>56.17</td>
    <td>62.78</td>
    <td>60.22</td>
    <td>66.83</td>
  </tr>
  <tr>
    <td>±std</td>
    <td>±0.22</td>
    <td>±2.24</td>
    <td>±0.26</td>
    <td>±0.45</td>
    <td>±0.52</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>59.63</td>
    <td>57.65</td>
    <td>61.69</td>
    <td>60.38</td>
    <td>66.79</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>61.87</b></td>
    <td><b>58.50</b></td>
    <td><b>64.09</b></td>
    <td><b>61.48</b></td>
    <td><b>68.08</b></td>
  </tr>
</table>

Table 4: ICL performances of 128-shot demonstrations, average over 9 datasets, on Qwen-turbo, GLM-4-flash, Doubao-pro, GPT-4o-mini, and DeepSeek-V3. Best scores are in bold. More details are in Appendix D.

positive correlation between the overall ICL performance and the model size as well as the shot number, indicating that larger models and more shots generally lead to better performances.

Among the methods compared, CLG generally outperforms other methods across various conditions, achieving significant improvements (far more than the standard deviation) over random selection. In contrast, the most competitive Best-of-N baseline exhibits relatively lower scaling efficiency with the increased model size or the increased shot numbers. This further highlights the scalability of CLG with respect to the model size and the shot number.

4.6 Scaling to a Thousand Shots

To further compare CLG against Random selection in long-context settings with enough demonstrations, we select 1024-shot demonstrations through both methods, and evaluate their performances on MNLI, HellaSwag, Break, and SMCalFlow, the four datasets with the most examples.

As illustrated in Table 5, CLG consistently achieves better results than the average performance of Random selection on all datasets. Such performance gains on 1024-shot are generally smaller than those on 128-shot. This could be explained by the possibility that 1024-shot demonstrations provide sufficient task knowledge needed for ICL, even when selected randomly.

Besides, we found that Break and SMCalFlow
<table>
  <tr>
    <th>Method</th>
    <th>MNLI</th>
    <th>HeSwag.</th>
    <th>Break</th>
    <th>SMCal.</th>
  </tr>
  <tr>
    <th colspan="5">128-shot</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>61.57</td>
    <td>86.58</td>
    <td>36.67</td>
    <td>38.23</td>
  </tr>
  <tr>
    <td>±std</td>
    <td>±1.12</td>
    <td>±0.33</td>
    <td>±0.60</td>
    <td>±1.42</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>77.29</b></td>
    <td><b>87.68</b></td>
    <td>37.07</td>
    <td>40.16</td>
  </tr>
  <tr>
    <td>Δ</td>
    <td>+15.72</td>
    <td>+1.10</td>
    <td>+0.40</td>
    <td>+1.93</td>
  </tr>
  <tr>
    <th colspan="5">1024-shot</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>55.08</td>
    <td>77.64</td>
    <td>42.08</td>
    <td>52.67</td>
  </tr>
  <tr>
    <td>±std</td>
    <td>±1.75</td>
    <td>±1.83</td>
    <td>±0.28</td>
    <td>±0.35</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td>58.94</td>
    <td>79.12</td>
    <td><b>42.21</b></td>
    <td><b>52.82</b></td>
  </tr>
  <tr>
    <td>Δ</td>
    <td>+3.86</td>
    <td>+1.48</td>
    <td>+0.13</td>
    <td>+0.15</td>
  </tr>
</table>

Table 5: Comparison of 128-shot and 1024-shot ICL on Qwen2.5-72B (in %). Best scores are in bold.

show significant improvements when scaled to 1024-shot, while MNLI and HellaSwag exhibit some degree of performance degradation. To explain this phenomenon, we analyse the scaling trends of ICL on each dataset with increasing shot numbers in Appendix E. We find that open-ended tasks, such as Break and SMCalFlow, are more likely to benefit from a larger number of shots.

4.7 Ablation Study: Gradient Mismatching

We conduct an ablation study to assess the effectiveness of the gradient matching term in Eq. 7. Specifically, we reverse the optimization target: instead of minimizing the \( L_2 \) distance in Eq. 7, we maximize it to select demonstrations that misalign with the full training set in learning dynamics.

As illustrated in Fig. 4, this reversal results in significant degradation in the ICL performance, which can be much poorer than that of random selection. These results empirically validate that the quality of demonstrations is correlated with the degree of curriculum latent gradient matching.

![Ablation study on the impact of gradient mismatching on ICL performance, using 128-shot demonstrations with Qwen2.5-72B.](page_104_1042_627_246.png)

Figure 4: Ablation study on the impact of gradient mismatching on ICL performance, using 128-shot demonstrations with Qwen2.5-72B.

4.8 Improving Adaptability through Hybrid In-Context Learning

In addition to our main experiments on task-level many-shot in-context learning, we further explore the complementary use of task-level and instance-level demonstrations. Specifically, we combine the 128-shot task-level demonstrations selected by Random and CLG with the 4-shot instance-level demonstrations retrieved by BGE-M3.

As illustrated in Fig. 5, such hybrid strategy yields further improvements over using only task-level or instance-level demonstrations across most datasets. Meanwhile, CLG generally outperforms Random selection in such hybrid mode.

![ICL with hybrid demonstrations: task-level many-shot followed by instance-level few-shot.](page_104_1342_627_246.png)

Figure 5: ICL with hybrid demonstrations: task-level many-shot followed by instance-level few-shot.

4.9 Correlation between FT and ICL

To validate our hypothesis that the data requirements for In-Context Learning (ICL) and Fine-Tuning (FT) are comparable, we fine-tune Qwen2.5-3B on the 128-shot demonstrations selected by each method. We employ prefix-tuning (Li and Liang, 2021) to avoid overfitting, and perform training with a batch size of 128 and a learning rate of \( 3e^{-4} \) for 1000 epochs. We quantify the FT performances through the negative evaluation losses on the validation set. We analyse and illustrate the correlation between FT and ICL performance in Fig. 6, which shows a positive relationship between the two forms of machine learning, especially for open-ended tasks.

![The correlation between FT and ICL. The left involves ICL performances on Qwen2.5-3B, and the right involves that averaged over all Qwen2.5 models. Datasets for open-ended tasks are lined in solid colors.](page_104_1642_627_246.png)

Figure 6: The correlation between FT and ICL. The left involves ICL performances on Qwen2.5-3B, and the right involves that averaged over all Qwen2.5 models. Datasets for open-ended tasks are lined in solid colors.

4.10 Diversity and Coverage Analysis

Since prior work has illustrated that diversity and coverage are important for effective in-context
demonstrations (Li and Qiu, 2023), we examine whether our method inherently maintains diversity and coverage in selected demonstrations. Specifically, we analyze the label distributions of demonstration sets on classification datasets. We measure the KL divergence between the label distribution of selected demonstrations and that of the test set, with results shown in Table 6.

<table>
  <tr>
    <th>Method</th>
    <th>SST-5</th>
    <th>MNLI</th>
    <th>CMSQA</th>
    <th>HeSwag.</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>0.020</td>
    <td>0.016</td>
    <td>0.016</td>
    <td>0.012</td>
  </tr>
  <tr>
    <td>± std</td>
    <td>±0.016</td>
    <td>±0.009</td>
    <td>±0.008</td>
    <td>±0.007</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>0.019</td>
    <td>0.021</td>
    <td>0.041</td>
    <td>0.019</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td>0.017</td>
    <td>0.036</td>
    <td>0.010</td>
    <td>0.026</td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>4.050</td>
    <td>0.115</td>
    <td>0.134</td>
    <td>4.296</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>0.006</b></td>
    <td><b>0.002</b></td>
    <td><b>0.007</b></td>
    <td><b>0.008</b></td>
  </tr>
</table>

Table 6: KL divergence between demonstration set label distribution and test set label distribution (lower is better). CLG achieves the best alignment with test set distributions, indicating it inherently maintains diversity and coverage of labels on classification datasets.

The results demonstrate that CLG naturally selects a demonstration set whose label distribution closely matches the test set distribution, significantly outperforming random selection. This suggests that gradient matching can inherently preserve data coverage in terms of label. In contrast, Latent-Bayesian shows particularly poor performance in maintaining label diversity, which may contribute to its suboptimal ICL performance.

4.11 Efficiency Analysis
To further verify the cost-effectiveness of our proposed CLG in practice, we analyse the computational cost of the complete demonstration selection pipeline. Table 7 details the time consumption for selecting 128-shot demonstrations from MNLI and SMCalFlow using Qwen2.5-3B, measured on 8 A100 GPUs with data-parallel implementation via deepspeed and accelerate libraries.

<table>
  <tr>
    <th>CLG Stage</th>
    <th># GPUs</th>
    <th>MNLI</th>
    <th>SMCal.</th>
  </tr>
  <tr>
    <td>Prefix Tuning</td>
    <td>8</td>
    <td>80 min</td>
    <td>87 min</td>
  </tr>
  <tr>
    <td>Gradient Computation</td>
    <td>8</td>
    <td>145 min</td>
    <td>210 min</td>
  </tr>
  <tr>
    <td>Gradient Matching</td>
    <td>1</td>
    <td>6 min</td>
    <td>10 min</td>
  </tr>
</table>

Table 7: Time consumption for 128-shot demonstration selection from datasets with 50k training examples.

The results demonstrate that our approach maintains practical efficiency even at scale, with total computation time being approximately 3~4 times that of prefix-tuning, an already efficient fine-tuning method. This computational investment is well justified as our method enables significant ICL performance improvements on both open-source larger models and closed-source LLMs, for which fine-tuning may face challenge in computational resources and access to model weights. This validates the cost-effectiveness of our proposed CLG for many-shot ICL in practice.

5 Conclusion

Many-shot ICL has emerged as a promising way to utilize LLMs on downstream tasks, but the random selection applied in existing work may produce sub-optimal demonstrations and learning results.

In this work, we hypothesize that data requirements for in-context learning and fine-tuning are analogous, and propose Curriculum Latent Gradient (CLG) to select a demonstration set that aligns with the entire training set in learning dynamics on LMs. We validate our method across various datasets and LLMs, showing its significant improvements compared to random selection, e.g., improving by 4% on open-source LLMs and by 2% on closed-source LLMs.

This work unlocks more reliable and effective many-shot ICL, paving the way for its broader adoption and application in real-world scenarios.

Limitations

This work focuses on improving many-shot in-context learning (ICL) by optimizing the demonstration set. However, ICL performance can also be sensitive to the order of demonstrations (Lu et al., 2022), which our method does not address. We believe that investigating the impact of demonstration order on in-context learning could lead to further improvements in the learning performance, for example, by examining the learning dynamics at different stages of fine-tuning.

Besides, our method selects demonstrations according to analysis on the training set, which may suffer from biases in the training data, such as imbalances over gender, race, or culture. This could lead to discriminatory content in practice. Furthermore, improved many-shot ICL may be susceptible to malicious exploitation, resulting in misuse such as jailbreaking LLMs (Anil et al., 2024).

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62477001.
References

Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal M. P. Behbahani, Aleksandra Faust, and Hugo Larochelle. 2024. Many-shot in-context learning. CoRR, abs/2404.11018.

Jacob Andreas, John Bufo, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lamanan, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Andrew Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zотов. 2020. Task-oriented dialogue as dataflow synthesis. Trans. Assoc. Comput. Linguistics, 8:556–571.

Cem Anil, Esin Durmus, Nina Rimsky, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel J Ford, et al. 2024. Many-shot jailbreaking. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R. Gormley, and Graham Neubig. 2024. In-context learning with long-context models: An in-depth exploration. CoRR, abs/2405.00200.

Léon Bottou. 2010. Large-scale machine learning with stochastic gradient descent. In 19th International Conference on Computational Statistics, COMPSTAT 2010, Paris, France, August 22-27, 2010 - Keynote, Invited and Contributed Papers, pages 177–186. Physica-Verlag.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. BGE m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. CoRR, abs/2402.03216.

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. CoRR, abs/2306.15595.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics.

Yiran Ding, Li Lyna Zhang, Chengrudong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. Longrop: Extending LLM context window beyond 2 million tokens. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Andrew Drozdov, Honglei Zhuang, Zhuyun Dai, Zhen Qin, Razieh Rahimi, Xuanhui Wang, Dana Alon, Mohit Iyyer, Andrew McCallum, Donald Metzler, and Kai Hui. 2023. Parade: Passage ranking using demonstrations with large language models. CoRR, abs/2310.14408.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Bing Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Alonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esibov, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ebab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Milon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junting Jia, Kalyan Vasudean Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. 2024. The llama 3 herd of models. CoRR, abs/2407.21783.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.

Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. 2023. Demystifying prompts in language models via perplexity estimation. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 10136–10148. Association for Computational Linguistics.

Matan Hasson and Jonathan Berant. 2021. Question decomposition with dependency graphs. In 3rd Conference on Automated Knowledge Base Construction, AKBC 2021, Virtual, October 4-8, 2021.

Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H. Chen, and Andrew Y. Ng. 2024. Many-shot in-context learning in multimodal foundation models. CoRR, abs/2405.09798.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattentition. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 611–626. ACM.

J. C. Lagarias and Andrew M. Odlyzko. 1985. Solving low-density subset sum problems. J. ACM, 32(1):229–246.

Haoran Li, Abhinav Arora, Shuhui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, pages 2950–2962. Association for Computational Linguistics.

Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 4582–4597. Association for Computational Linguistics.

Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023a. Unified demonstration retriever for in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL 2023, Toronto, Canada, July 9-14, 2023, pages 4644–4668. Association for Computational Linguistics.

Xiaonan Li and Xipeng Qiu. 2023. Finding supporting examples for in-context learning. CoRR, abs/2302.13539.

Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023b. Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353, Singapore.

Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. 2018. Nil2bash: A corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA).

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO@ACL 2022, Dublin, Ireland and Online, May 27, 2022, pages 100–114. Association for Computational Linguistics.

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8086–8098. Association for Computational Linguistics.

Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Seyed Mehran Kazemi, Chitta Baral, Vaiva Imbrasaitė, and Vincent Y. Zhao. 2023. Dr.icl: Demonstration-retrieved in-context learning. CoRR, abs/2305.14128.

OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.

Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 11054–11070.

Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference. In Proceedings of the Sixth Conference on Machine Learning and Systems, MLSys 2023, Miami, FL, USA, June 4-8, 2023. mlsys.org.

Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 2655–2671. Association for Computational Linguistics.

Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021. Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 922–938. Association for Computational Linguistics.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1631–1642. ACL.

Mingyang Song, Mao Zheng, and Xuan Luo. 2025. Can many-shot in-context learning help llms as evaluators? A preliminary empirical study. In Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025, pages 8232–8241. Association for Computational Linguistics.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4149–4158. Association for Computational Linguistics.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008.

Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2023. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics.

Tomer Wolfson, Mor Geva, Ankit Gupta, Yoav Goldberg, Matt Gardner, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understanding benchmark. Trans. Assoc. Comput. Linguistics, 8:183–198.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024. Qwen2.5 technical report. CoRR, abs/2412.15115.

Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 39818–39833. PMLR.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics.

Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. Active example selection for in-context learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 9134–9148. Association for Computational Linguistics.

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. 2021. Dataset condensation with gradient matching. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
A  Dataset Examples

We illustrate examples on all dataset in Table 8. We illustrate a query prompt with random 4-shot demonstrations on each dataset, where the demonstration answers are highlighted in red.

B  Case Study: Poor Performance of BM25-Major and Latent-Bayesian

Since BM25-Major and Latent-Bayesian perform poorly, even underperforming Random selection, we examine the demonstrations selected by these two methods. We found they tend to be trapped in specific patterns and lose diversity in selection. We illustrate their demonstration cases and discuss these phenomena in Tables 9 and 10 respectively.

C  Full Results

We illustrate all results on Qwen2.5 series models and Llama3 series models for 128-shot scenarios in Table 11, and results averaged over 4-shot to 128-shot scenarios in Table 12.

D  Implementation Details and Full Results on Closed-Source LLMs

We evaluate demonstrations selected by Random, Best-of-N, and CLG over 5 closed-source LLMs: Qwen-turbo, GLM-4-flash, Doubao-pro-32k, GPT-4o-mini, and DeepSeek-V3. These models are primarily designed for chat-based interactions with users rather than text completion. To adapt them for our evaluation, we begin by presenting them with a system message that outlines the task requirements:

You are an In-Context Learner to learn from task demonstrations provided in context.
User will provide you a prompt, composed of several demonstration (query + response) and a test-query (only query).
You need to understand the task concept and response format through the demonstrations, and return your response to the test query.

Example of a valid JSON response:
```json
{
    "response": "...",
}
```

Next, we construct a user message containing the demonstrations (limited to 7000 tokens) and the test query, formatted as a JSON object. An example with 4-shot demonstrations is shown below:

```json
{
    "demonstrations": [
        {
            "query": "as giddy and whimsical and relevant today as it was 270 years ago . It is ",
            "response": "great"
        },
        {
            "query": "after an uncertain start , murder hits and generally sustains a higher plateau with bullock 's memorable first interrogation of gosling . It is ",
            "response": "good"
        },
        {
            "query": "an unbelievably stupid film , though occasionally fun enough to make you forget its absurdity . It is ",
            "response": "bad"
        },
        {
            "query": "even though it is infused with the sensibility of a video director , it does n't make for completely empty entertainment It is ",
            "response": "OK"
        }
    ],
    "test-query": {
        "query": "in his first stab at the form , jacquot takes a slightly anarchic approach that works only sporadically . It is "
    }
}
```

After chatting with closed-source models by these messages, we extract and evaluate responses from their JSON-formatted outputs. The full evaluation results are illustrated in Table 13.

E  Scaling Trends of Random Selection on All Datasets

In Figure 7, we present the ICL performances of all models across various demonstration sizes, ranging from 4-shot to 128-shot, using randomly selected demonstrations. The results indicate that datasets for open-ended tasks tend to consistently benefit from more in-context demonstrations. This trend is likely attributable to the fact that open-ended tasks have significantly longer answers (see Appendix A), which provide richer information to learn from context and more space for improvements in prediction.

Qwen2.5-7B exhibits unusual performance on MNLI, significantly outperforming other models in the 4-shot setting but dropping to the lowest performance in the 128-shot setting. This anomaly is validated through comparisons across models and demonstration selection methods. As shown in Fig. 8, this behavior appears unique to Qwen2.5-7B and irrelevant to demonstration selection, suggesting Qwen2.5-7B may be a distinct case on MNLI.
<table>
  <tr>
    <th>Dataset</th>
    <th>Query Prompt with 4-shot In-Context Demonstrations selected by Random</th>
  </tr>
  <tr>
    <td>SST-5</td>
    <td>more of the same from taiwanese auteur tsai ming-liang , which is good news to anyone who ’ s fallen under the sweet , melancholy spell of this unique director ’ s previous films . It is good .
it ’ s a great performance and a reminder of dickens ’ grandeur . It is great
a series of escapades demonstrating the adage that what is good for the goose is also good for the gander , some of which occasionally amuses but none of which amounts to much of a story . It is bad
one just waits grimly for the next shock without developing much attachment to the characters . It is OK
in his first stab at the form , jacquot takes a slightly anarchic approach that works only sporadically . It is</td>
  </tr>
  <tr>
    <td>MNLI</td>
    <td>Instead of ancient artifacts it shows the lifestyle and achievements of myriad Jewish communities around the globe through high-tech audio-visual displays, hands-on exhibits, scale models (many of which are exquisite), and reconstructions. Can we say “The Jewish communities had many ancient artifacts on display.”? No
There is now the dominant right-wing critique of integrationist programs. Can we say “Right wing politics is praised by the integrationist.”? No
Lorenzo the Magnificent had both Italian and Spanish life in simple tombs beneath the sculptor’s Madonna and Child, flanked by lesser artists’ statues of the family patron saints. Can we say “Lorenzo the Magnificent had both Italian and Spanish life in simple tombs beneath the sculptor’s Madonna and Child, flanked by lesser artists’ statues of the family patron saints.”? Yes
The difference, if any, between the reacquisition price and the net carrying value of the extinguished debt should be recognized as a loss or gain in accounting for interest on Treasury debt. Can we say “The difference is not recognized.”? No
The new rights are nice enough Can we say “Everyone really likes the newest benefits”? </td>
  </tr>
  <tr>
    <td>CMSQA</td>
    <td>Question: Bill sits down on a whoopee cushion, what sound does he make when he sits? (A) fall asleep (B) flatulence (C) sigh of relief (D) medium (E) comfort Answer: B
Question: What is likely heard by those going to a party? (A) smoking pot (B) happiness (C) laughter (D) babies (E) meet new people Answer: C
Question: A handsome prince is a stock character common to what? (A) england (B) fairy tale (C) castle (D) palace (E) court Answer: B
Question: What covers the largest percentage of the pacific northwest? (A) united states (B) united states (C) washington (D) oregon (E) british columbia Answer: B
Question: A revolving door is convenient for two direction travel, but it also serves as a security measure at what? (A) bank (B) library (C) department store (D) mall (E) new york Answer:</td>
  </tr>
  <tr>
    <td>HeSwag.</td>
    <td>Choose an ending: The player is holding a bat walking in the field. The player... (A) in front stands across the screen and the ball dunks him to the ground. (B) makes a goal and the male scores. (C) runs down a purple net towards the ball. (D) begins to hit the ball. Answer: B
Choose an ending: A woman sitting on the floor speaks to the camera. She... (A) begins blowing cover on the phone. (B) pulls a harmonica and ties the end of the violin. (C) begins to play a saxophone in the bedroom. (D) holds onto a cat in front of her. Answer: D
Choose an ending: Someone jumps out the living room window. He... (A) dives for her pipe. (B) tuxedos her free eyes. (C) sees the girl chasing after him. (D) is as grim as he can. Answer: C
Choose an ending: A group sits on horses as the stand and take a rest. The group... (A) continues jumping around a gym as the crowd claps. (B) slowly slowly make their way to the other side of the pool on the level. (C) rides on the horses along a trail. (D) pass between horses at the left. Answer: C
Choose an ending: Students lower their eyes nervously. She... (A) puts her shoulder, then saunters toward someone. (B) turns with two students. (C) walks slowly towards someone. (D) wheels around as her dog thunders out. Answer:</td>
  </tr>
  <tr>
    <td>GeoQ.</td>
    <td>how many rivers are there in m0 answer: count(intersection(river_loc_2(m0)) through which states does the m0 flow answer: intersection(state_traverse_1(m0)) through which states does the m0 run answer: intersection(state_traverse_1(m0)) what is the most populous city in m0 answer: largest_one(population_1.intersection(city_loc_2(m0))) name all the rivers in m0 answer:</td>
  </tr>
  <tr>
    <td>NL2Bash</td>
    <td>Get the path of running Apache ps -ef | grep apache
Gets domain name from dig reverse lookup. dig@ -x 8.8.8.8 | grep PTR | grep -o google.*
Find all *.xml files under current directory find_name *.xml
Disables shell option 'nullglob', shopt -u nullglob
Add executable permission to 'pretty-print'</td>
  </tr>
  <tr>
    <td>Break</td>
    <td>when did vincent von gogh die? #18 returns vincent von gogh 290 return when did #1 die
Does the Bronx have more non-Hispanic whites, or Hispanic whites? #18 returns the Bronx 2#0 return non-Hispanic whites in #1 3#0 return Hispanic whites in #1 4#0 return number of #2 5#0 return number of #3 6#0 return which is highest of #4 . #5
What is the oldest log id and its corresponding problem id? #18 return logs id 2#0 return #1 that is the oldest 3#0 return corresponding problem id of #2 4#0 return #2 , #3 How many years apart were the British Saloon Car Championship season wins after 1970? #18 return the British Saloon Car Championship 2#0 return season wins of #1 3#0 return years of #2 4#0 return #3 that were after 1970 5#0 return the difference of #4 what flights are available tomorrow from denver to philadelphia #18</td>
  </tr>
  <tr>
    <td>MTOP</td>
    <td>Ask Rob how bad the traffic is [IN-SEND_MESSAGE [SL-RECIPIENT Rob ] [SL-CONTENT_EXACT how bad the traffic is ] ]
Can I have the headlines please [IN-GET_STORIES_NEWS [SL-NEWS_TYPE headlines ] ]
what are the headlines in ohio news? [IN-GET_STORIES_NEWS [SL-NEWS_TOPIC ohio ] [SL-NEWS_TYPE news ] ]
What is the most recent news regarding local politics [IN-GET_STORIES_NEWS [SL-DATE_TIME the most recent ] [SL-NEWS_TYPE news ] [SL-NEWS_CATEGORY local politics ] ]
call Nicholas and Natasha</td>
  </tr>
  <tr>
    <td>SMCal.</td>
    <td>What about with Kaitlin Taylor? {Yield (Execute (NewClobber (refer ((Dynamic) ActionIntensionConstraint)) ((Recipient) Constraint(TypeIntension)) (intension (RecipientWithNamedLike ((Recipient) EmptyStructConstraint) (PersonName apply "Kaitlin Taylor")))))}
Sorry I meant the weekend? {Yield (Execute(RevokeConstraint (refer ((Dynamic) roleConstraint (Path apply "output"))) ((Event) ConstraintTypeIntension)) (Event.start_? (Path apply "output"))) }
CHECK FOR ALBERT’S BIRTHDAY EVENT {Yield (FindEventWrapperWithDefaults (Event.subject_? (? = "ALBERT’S BIRTHDAY"))) }
Add meetings same time everyday for this week. {FenceRecurring}
Change the reservation for tonight to 6 people from 4.</td>
  </tr>
</table>

Table 8: Dataset cases with random 4-shot in-context demonstrations.
Dataset | Query Prompt with 4-shot In-Context Demonstrations selected by Latent-Bayesian

<table>
  <tr>
    <th>SST-5</th>
    <td>renner ? It is OK<br>apart from its considerable achievement , metropolis confirms tezuka ’s status as both the primary visual influence on the anime tradition and its defining philosophical conscience . It is OK.<br>... salaciously simplistic . It is OK.<br>... would be a total loss if not for two supporting performances taking place at the movie ’s edges . It is OK<br>in his first stab at the form . jacquot takes a slightly anarchic approach that works only sporadically . It is</td>
  </tr>
  <tr>
    <th>HeSwag.</th>
    <td>Choose an ending: Richard Parker frantically swipes at the small fish as someone shields his face. The fluttering silver creatures... (A) spoon in hand. (B) inserting its wands into their palms. (C) pelt his exposed torso. (D) fly over him, catching and grabbing the sheet of flame. Answer: C<br>Choose an ending: The sheer athleticism of his movements keeps him ahead of the choppers as he leaps and rolls on. Back to the shadow, someone... (A) ducks down, gathers his robe and huddles against the wall. (B) crouches. (C) gets up. (D) gets up and gayly does a fall and a leap, her head against the rotating frame. Answer: D<br>Choose an ending: The door closes after him, and unable to stop, the keys herd into it. Huge broken stone statues... (A) lie on an maze field. (B) are about to emerge, which painted walls. (C) lie about the floor. (D) are drawn around the arena empty gleaming guns. Answer: C<br>Choose an ending: A strong wind fills leaves high into the air and the hedges advance. As the hedges close in behind them, they both... (A) climb the rock - wakeboard on the water. (B) run past the trees. (C) run past the trees. (D) pose, once aftermath from all angles. Answer: D<br>Choose an ending: Students lower their eyes nervously. She... (A) puts her shoulder, then saunters toward someone. (B) turns with two students. (C) walks slowly towards someone. (D) wheels around as her dog thunders out. Answer:</td>
  </tr>
</table>

Table 9: Demonstration cases selected by Latent-Bayesian. On SST-5, Latent-Bayesian tends to exclusively select demonstrations with the neutral label OK. On HellaSwag, it predominantly selects demonstrations with answers C or D. Specifically, for 128-shot demonstrations on HellaSwag, Latent-Bayesian selected answers A, B, C, and D in proportions of 8, 0, 45, and 75 respectively. This is in contrast to the training set of 50, 000 examples where the four answers are distributed equally (12, 500 ± 68 each).

Dataset | Query Prompt with 4-shot In-Context Demonstrations selected by BM25-Major

<table>
  <tr>
    <th>Break</th>
    <td>What shape is the object is a different shape and size than the other objects? #1) return objects #2) return shapes of #1 #3) return sizes of #1 #4) return number of #1 for each #2 #5) return number of #1 for each #3 #6) return #2 where #4 is one #7) return #3 where #5 is one #8) return #1 where #2 is #6 #9) return #1 where #3 is #7 #10) return #1 in both #8 and #9 #11) return the shape of #10<br>All are the spheres the same material? #1) return spheres #2) return the materials of #1 #3) return number of #1 for each #2 #4) return #2 where #3 is the highest #5) return #1 where #2 is #3 #6) return #1 where #2 is #3 #7) return number of #1 #8) return #1 where #2 is #3 #9) return #1 where #2 is #3 #10) return the highest of #3 #5) return #2 where #3 is #4 #6) return the code of #5 #7) return #4 #6<br>What color is the object that is a different material than the others? #1) return objects #2) return materials of #1 #3) return the number of #1 for each #2 #4) return #2 where #1 is one #5) return #1 where #2 is #3 #6) return the color of #3<br>what shape is the object is a different shape and size than the other objects? #1) return objects #2) return shapes of #1 #3) return sizes of #1 #4) return number of #1 for each #2 #5) return number of #1 for each #3 #6) return #2 where #4 is one #7) return #3 where #5 is one #8) return #1 where #2 is #6 #9) return #1 where #3 is #7 #10) return #1 in both #8 and #9 #11) return the shape of #10</td>
  </tr>
  <tr>
    <th>MTOP</th>
    <td>remind me to call my father for fathers day [IN.CREATE_REMINDER [SL.PERSON_REMINDED me ] [SL.TODO IN.GET_TODO [SL.TODO [IN.CREATE_CALL [SL.CONTACT [IN.GET_CONTACT [SL.CONTACT_RELATED_me ] [SL.TYPE_RELATION father ] ] ] ] ] ] ] ] [SL.TODO for fathers day ] ] ]<br>Set a reminder to call my husband when I get home [IN.CREATE_REMINDER [SL.TODO IN.GET_TODO [SL.TODO [IN.CREATE_CALL [SL.CONTACT [IN.GET_CONTACT [SL.CONTACT_RELATED_me ] [SL.TYPE_RELATION husband ] ] ] ] ] ] ] ] [SL.TODO IN.GET_TODO [SL.TODO [IN.CREATE_CALL [SL.CONTACT [IN.GET_CONTACT [SL.CONTACT_RELATED_me ] [SL.TYPE_RELATION husband ] ] ] ] ] ] ] ] [SL.TODO for husbands day ] ] ]<br>Remind me to call my brother after work [IN.CREATE_REMINDER [SL.PERSON_REMINDED me ] [SL.TODO IN.GET_TODO [SL.TODO [IN.CREATE_CALL [SL.CONTACT [IN.GET_CONTACT [SL.CONTACT_RELATED_me ] [SL.TYPE_RELATION brother ] ] ] ] ] ] ] ] [SL.TODO for brothers day ] ] ]<br>Remind me to call my father today [IN.CREATE_REMINDER [SL.PERSON_REMINDED me ] [SL.TODO [IN.CREATE_CALL [SL.CONTACT [IN.GET_CONTACT [SL.CONTACT_RELATED_me ] [SL.TYPE_RELATION father ] ] ] ] ] ] ] [SL.DATE, time today ] ] ]<br>call Nicholas and Natasha</td>
  </tr>
  <tr>
    <th>SMCal.</th>
    <td>What is on my calendar tomorrow? I want to have lunch with Sara, Barack and Monica. (do Yield (FindEventWrapperWithDefaults (EventOnDate (Tomorrow) ((Event EmptyStructConstraint)))) (Yield CreateCommEventWrapper (CreatePreflightEventWrapper (& & (Event.subject_ ? (= "lunch") (Event.start_ ? (Date-Time.date_ ? (= (Tomorrow))) (Event.attendees_ ? (& & (AttendeeListHasRecipient (Execute (refer extensionConstraint (RecipientWithNameLike (Recipient) EmptyStructConstraint)) (PersonName apply "Barack.")))) (AttendeeListHasRecipient (Execute (refer extensionConstraint (RecipientWithNameLike (Recipient) EmptyStructConstraint)) (PersonName apply "Monica.")))))) (AttendeeListHasRecipient (Execute (refer extensionConstraint (RecipientWithNameLike (Recipient) EmptyStructConstraint)) (PersonName apply "Elli.")))))) (AttendeeListHasRecipientConstraint (RecipientWithNameLike (Recipient) EmptyStructConstraint) (PersonName apply "Abby."))) (AttendeeListHasRecipientConstraint (RecipientWithNameLike (Recipient) EmptyStructConstraint) (PersonName apply "Kim.")))))))))))))<br>Change the event on Friday to being at 1pm. Also invite Sally, Alex and Sammi to the lunch. (Yield (UpdateEventWrapperWithDefaults (EventEmptyStructConstraint) (EventOnDate (Friday) ((Event subject_ ? (= "lunch") (Event.start_ ? (Date-Time.time_ ? (= (NumberPM 1L)))) (Event.attendees_ ? (& & (AttendeeListHasRecipient (Execute (refer extensionConstraint (RecipientWithNameLike (Recipient) EmptyStructConstraint)) (PersonName apply "Sally.")))) (AttendeeListHasRecipient (Execute (refer extensionConstraint (RecipientWithNameLike (Recipient) EmptyStructConstraint)) (PersonName apply "Alex.")))) (AttendeeListHasRecipient (Execute (refer extensionConstraint (RecipientWithNameLike (Recipient) EmptyStructConstraint)) (PersonName apply "Jessie.")))))))))))))<br>Put a lunch meeting with Abby, Kim, and Jesse at Taco Bell tomorrow (Yield (CreateCommEventWrapper (CreatePreflightEventWrapper (& & (Event.subject_ ? (= "lunch meeting")) (Event.start_ ? (Date-Time.date_ ? (= (Tomorrow))) (Event.location_ ? (= (LocationKeyphrase apply "Taco Bell.")))))) (Event.attendees_ ? (& & (AttendeeListHasRecipient (Execute (refer extensionConstraint (RecipientWithNameLike (Recipient) EmptyStructConstraint)) (PersonName apply "Abby.")))) (AttendeeListHasRecipient (Execute (refer extensionConstraint (RecipientWithNameLike (Recipient) EmptyStructConstraint)) (PersonName apply "Kim.")))) (AttendeeListHasRecipient (Execute (refer extensionConstraint (RecipientWithNameLike (Recipient) EmptyStructConstraint)) (PersonName apply "Jessie.")))))))))))))<br>Change the reservation for tonight to 6 people from 4.</td>
  </tr>
</table>

Table 10: Demonstration cases selected by BM25-Major. BM25-Major tends to select significantly longer demonstrations. This can be attributed to the fact that longer examples tend to have higher BM25 scores.
<table>
  <tr>
    <th>Method</th>
    <th>SST-5</th>
    <th>MNLI</th>
    <th>CMSQA</th>
    <th>HeSwag.</th>
    <th>GeoQ.</th>
    <th>NL2Bash</th>
    <th>Break</th>
    <th>MTOP</th>
    <th>SMCal.</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="11">Llama3-8B</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>37.75</td>
    <td>54.32</td>
    <td>72.99</td>
    <td>41.51</td>
    <td>49.14</td>
    <td>20.14</td>
    <td>27.19</td>
    <td>34.33</td>
    <td>29.01</td>
    <td>40.71</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td>38.78</td>
    <td>40.62</td>
    <td>70.84</td>
    <td>39.62</td>
    <td>35.71</td>
    <td>28.66</td>
    <td>2.16</td>
    <td>5.37</td>
    <td>6.56</td>
    <td>29.81</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td>37.15</td>
    <td>61.85</td>
    <td>73.55</td>
    <td>36.02</td>
    <td><b>58.21</b></td>
    <td>19.90</td>
    <td>23.98</td>
    <td>33.20</td>
    <td>28.69</td>
    <td>41.39</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>39.33</td>
    <td>51.92</td>
    <td>73.38</td>
    <td>40.58</td>
    <td>53.93</td>
    <td>26.78</td>
    <td>25.88</td>
    <td>35.48</td>
    <td><b>30.55</b></td>
    <td>41.98</td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>37.60</td>
    <td>53.98</td>
    <td>73.87</td>
    <td>39.99</td>
    <td>49.64</td>
    <td>17.65</td>
    <td><b>30.09</b></td>
    <td>36.51</td>
    <td>29.63</td>
    <td>41.00</td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>37.15</td>
    <td>44.04</td>
    <td>73.22</td>
    <td>33.81</td>
    <td>41.79</td>
    <td>24.72</td>
    <td>25.66</td>
    <td>32.30</td>
    <td>18.13</td>
    <td>36.76</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>39.96</b></td>
    <td><b>61.98</b></td>
    <td><b>73.96</b></td>
    <td><b>46.01</b></td>
    <td><b>58.21</b></td>
    <td><b>33.83</b></td>
    <td><b>29.06</b></td>
    <td><b>37.18</b></td>
    <td>27.39</td>
    <td><b>45.29</b></td>
  </tr>
  <tr>
    <th colspan="11">Llama3-70B</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>43.71</td>
    <td>61.83</td>
    <td><b>84.73</b></td>
    <td>76.11</td>
    <td>73.36</td>
    <td>29.72</td>
    <td>35.55</td>
    <td>44.66</td>
    <td>36.68</td>
    <td>54.04</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td>43.69</td>
    <td>51.36</td>
    <td>82.23</td>
    <td>72.01</td>
    <td>43.21</td>
    <td>27.92</td>
    <td>15.05</td>
    <td>8.41</td>
    <td>14.38</td>
    <td>39.81</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td>46.41</td>
    <td>65.09</td>
    <td>83.78</td>
    <td>73.90</td>
    <td><b>84.64</b></td>
    <td>28.39</td>
    <td>35.13</td>
    <td>44.79</td>
    <td><b>38.73</b></td>
    <td><b>55.65</b></td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>43.69</td>
    <td>70.92</td>
    <td>84.19</td>
    <td>74.57</td>
    <td>80.71</td>
    <td><b>29.76</b></td>
    <td>35.86</td>
    <td>43.40</td>
    <td>37.66</td>
    <td>55.64</td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>43.05</td>
    <td>49.78</td>
    <td>83.62</td>
    <td>71.32</td>
    <td>78.21</td>
    <td>22.92</td>
    <td><b>38.39</b></td>
    <td><b>50.43</b></td>
    <td>38.10</td>
    <td>52.87</td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>46.59</td>
    <td>69.84</td>
    <td>84.60</td>
    <td>70.09</td>
    <td>62.14</td>
    <td>29.20</td>
    <td>32.59</td>
    <td>41.21</td>
    <td>23.13</td>
    <td>51.04</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>48.32</b></td>
    <td><b>76.37</b></td>
    <td>84.52</td>
    <td><b>80.77</b></td>
    <td><b>84.64</b></td>
    <td>29.59</td>
    <td>37.33</td>
    <td>47.74</td>
    <td><b>40.35</b></td>
    <td><b>58.85</b></td>
  </tr>
  <tr>
    <th colspan="11">Qwen2.5-3B</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>19.05</td>
    <td>51.15</td>
    <td>68.76</td>
    <td>35.30</td>
    <td>47.93</td>
    <td>14.08</td>
    <td>23.29</td>
    <td>25.94</td>
    <td>19.07</td>
    <td>33.84</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td><b>26.70</b></td>
    <td>45.33</td>
    <td>68.47</td>
    <td>33.79</td>
    <td>36.07</td>
    <td><b>28.53</b></td>
    <td>3.56</td>
    <td>5.01</td>
    <td>0.12</td>
    <td>27.51</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td>26.43</td>
    <td>40.70</td>
    <td><b>73.79</b></td>
    <td>32.72</td>
    <td>51.43</td>
    <td>19.74</td>
    <td>19.83</td>
    <td>26.26</td>
    <td><b>22.20</b></td>
    <td>34.79</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>21.25</td>
    <td><b>56.02</b></td>
    <td>71.50</td>
    <td>41.10</td>
    <td><b>55.00</b></td>
    <td>21.36</td>
    <td>23.45</td>
    <td>26.31</td>
    <td>21.15</td>
    <td><b>37.46</b></td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>19.35</td>
    <td>54.53</td>
    <td>70.93</td>
    <td>33.61</td>
    <td>49.29</td>
    <td>11.79</td>
    <td><b>23.80</b></td>
    <td><b>30.07</b></td>
    <td>17.66</td>
    <td>34.56</td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>21.07</td>
    <td>44.66</td>
    <td>69.62</td>
    <td>25.68</td>
    <td>28.93</td>
    <td>11.26</td>
    <td>20.91</td>
    <td>23.80</td>
    <td>11.56</td>
    <td>28.61</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td>24.07</td>
    <td>54.42</td>
    <td><b>72.56</b></td>
    <td><b>44.77</b></td>
    <td>52.50</td>
    <td>17.62</td>
    <td><b>24.90</b></td>
    <td>26.49</td>
    <td>19.40</td>
    <td><b>37.41</b></td>
  </tr>
  <tr>
    <th colspan="11">Qwen2.5-7B</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>26.48</td>
    <td>31.88</td>
    <td>84.88</td>
    <td>71.38</td>
    <td>35.43</td>
    <td>16.93</td>
    <td>28.22</td>
    <td>32.72</td>
    <td>27.99</td>
    <td>39.55</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td><b>29.34</b></td>
    <td>31.88</td>
    <td>83.78</td>
    <td>71.50</td>
    <td>41.43</td>
    <td><b>32.27</b></td>
    <td>8.65</td>
    <td>7.79</td>
    <td>11.98</td>
    <td>35.40</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td>26.25</td>
    <td>31.88</td>
    <td><b>85.26</b></td>
    <td><b>74.10</b></td>
    <td><b>56.79</b></td>
    <td>19.46</td>
    <td>25.28</td>
    <td>33.20</td>
    <td><b>30.73</b></td>
    <td><b>42.55</b></td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>24.80</td>
    <td>31.88</td>
    <td>84.77</td>
    <td>72.69</td>
    <td>26.07</td>
    <td>15.72</td>
    <td>27.80</td>
    <td>32.75</td>
    <td><b>28.71</b></td>
    <td>38.35</td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>24.70</td>
    <td><b>31.94</b></td>
    <td>85.26</td>
    <td>71.33</td>
    <td>51.79</td>
    <td>15.44</td>
    <td><b>31.17</b></td>
    <td><b>35.50</b></td>
    <td>25.43</td>
    <td>41.37</td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>12.62</td>
    <td>31.88</td>
    <td>84.93</td>
    <td>60.58</td>
    <td>21.43</td>
    <td>13.31</td>
    <td>26.21</td>
    <td>26.13</td>
    <td>17.66</td>
    <td>32.75</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td>24.89</td>
    <td><b>31.94</b></td>
    <td><b>85.34</b></td>
    <td>71.76</td>
    <td>50.00</td>
    <td>19.54</td>
    <td><b>28.27</b></td>
    <td>32.98</td>
    <td>28.03</td>
    <td><b>41.42</b></td>
  </tr>
  <tr>
    <th colspan="11">Qwen2.5-74B</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>29.55</td>
    <td>56.76</td>
    <td>82.29</td>
    <td>55.53</td>
    <td>61.71</td>
    <td>28.00</td>
    <td>31.41</td>
    <td>38.74</td>
    <td>33.73</td>
    <td>46.41</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td>19.53</td>
    <td>55.98</td>
    <td>79.28</td>
    <td>46.52</td>
    <td>51.79</td>
    <td>41.82</td>
    <td>14.85</td>
    <td>10.02</td>
    <td>12.82</td>
    <td>36.96</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td><b>34.79</b></td>
    <td><b>57.31</b></td>
    <td>82.88</td>
    <td>60.61</td>
    <td><b>74.64</b></td>
    <td>31.99</td>
    <td>30.13</td>
    <td>40.89</td>
    <td>31.93</td>
    <td>49.46</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>28.97</td>
    <td>55.17</td>
    <td>82.56</td>
    <td>52.92</td>
    <td>59.64</td>
    <td>36.30</td>
    <td>30.44</td>
    <td>42.24</td>
    <td>32.05</td>
    <td>46.80</td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>18.07</td>
    <td>56.24</td>
    <td>82.06</td>
    <td>51.67</td>
    <td>58.63</td>
    <td>24.39</td>
    <td><b>32.41</b></td>
    <td><b>43.18</b></td>
    <td>33.95</td>
    <td>44.54</td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>21.16</td>
    <td>56.20</td>
    <td>81.98</td>
    <td>63.84</td>
    <td>47.50</td>
    <td>24.77</td>
    <td>30.39</td>
    <td>28.95</td>
    <td>16.71</td>
    <td>41.30</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td>32.79</td>
    <td>56.30</td>
    <td><b>83.05</b></td>
    <td><b>69.35</b></td>
    <td>60.36</td>
    <td><b>42.43</b></td>
    <td><b>32.20</b></td>
    <td>39.51</td>
    <td><b>34.95</b></td>
    <td><b>50.10</b></td>
  </tr>
  <tr>
    <th colspan="11">Qwen2.5-72B</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>13.93</td>
    <td>61.58</td>
    <td>85.54</td>
    <td>84.62</td>
    <td>72.29</td>
    <td>28.51</td>
    <td>36.24</td>
    <td>43.19</td>
    <td>37.43</td>
    <td>51.48</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td>10.72</td>
    <td>62.23</td>
    <td>85.09</td>
    <td>80.69</td>
    <td><b>41.68</b></td>
    <td>21.38</td>
    <td>11.23</td>
    <td>21.06</td>
    <td>43.55</td>
    <td>43.55</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td>12.62</td>
    <td>60.96</td>
    <td>85.26</td>
    <td>83.12</td>
    <td><b>80.00</b></td>
    <td>28.11</td>
    <td>35.35</td>
    <td>43.76</td>
    <td>38.51</td>
    <td>51.97</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td><b>17.44</b></td>
    <td>62.24</td>
    <td>57.25</td>
    <td>85.31</td>
    <td>73.57</td>
    <td>37.11</td>
    <td>34.91</td>
    <td>46.94</td>
    <td>35.36</td>
    <td>50.01</td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>14.62</td>
    <td>61.41</td>
    <td>85.67</td>
    <td>83.60</td>
    <td>76.79</td>
    <td>28.35</td>
    <td><b>38.32</b></td>
    <td><b>49.13</b></td>
    <td><b>40.98</b></td>
    <td>53.21</td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>0.00</td>
    <td>58.68</td>
    <td>85.91</td>
    <td>81.66</td>
    <td>58.57</td>
    <td>31.57</td>
    <td>34.88</td>
    <td>33.24</td>
    <td>22.21</td>
    <td>45.19</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>16.17</b></td>
    <td><b>73.33</b></td>
    <td><b>86.40</b></td>
    <td><b>86.53</b></td>
    <td>73.57</td>
    <td>29.97</td>
    <td><b>36.26</b></td>
    <td>46.62</td>
    <td>36.75</td>
    <td><b>53.96</b></td>
  </tr>
  <tr>
    <th colspan="11">Qwen2.5-72B</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>36.00</td>
    <td>58.20</td>
    <td>87.47</td>
    <td>86.58</td>
    <td>57.29</td>
    <td>33.28</td>
    <td>36.67</td>
    <td>45.32</td>
    <td>38.23</td>
    <td>53.23</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td>37.78</td>
    <td>45.94</td>
    <td>86.65</td>
    <td>85.61</td>
    <td>52.50</td>
    <td>38.11</td>
    <td>22.81</td>
    <td>16.24</td>
    <td>11.57</td>
    <td>44.13</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td>37.24</td>
    <td>49.65</td>
    <td>87.71</td>
    <td>86.67</td>
    <td>61.79</td>
    <td>40.86</td>
    <td>35.84</td>
    <td>43.71</td>
    <td><b>41.29</b></td>
    <td>53.86</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>36.60</td>
    <td>60.57</td>
    <td>87.55</td>
    <td>86.83</td>
    <td>61.07</td>
    <td>37.13</td>
    <td>35.73</td>
    <td>46.89</td>
    <td>39.22</td>
    <td>54.62</td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>36.15</td>
    <td>58.41</td>
    <td><b>87.96</b></td>
    <td>86.22</td>
    <td><b>62.86</b></td>
    <td>26.53</td>
    <td><b>39.12</b></td>
    <td><b>47.53</b></td>
    <td>39.65</td>
    <td>53.83</td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>27.25</td>
    <td>47.96</td>
    <td>85.75</td>
    <td>86.72</td>
    <td>27.50</td>
    <td>37.36</td>
    <td>35.35</td>
    <td>5.23</td>
    <td>22.61</td>
    <td>41.75</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>38.33</b></td>
    <td><b>77.29</b></td>
    <td><b>87.71</b></td>
    <td><b>87.68</b></td>
    <td><b>62.50</b></td>
    <td><b>45.19</b></td>
    <td><b>37.07</b></td>
    <td><b>47.07</b></td>
    <td><b>40.16</b></td>
    <td><b>58.11</b></td>
  </tr>
</table>

Table 11: Main results, merely 128-shot, on all models. The best scores on each model are in bold, and the second-best ones are underlined.
<table>
  <tr>
    <th>Method</th>
    <th>SST-5</th>
    <th>MNLI</th>
    <th>CMSQA</th>
    <th>HeSwag</th>
    <th>GeoQ.</th>
    <th>NL2Bash</th>
    <th>Break</th>
    <th>MTOP</th>
    <th>SMCal.</th>
    <th>Average</th>
  </tr>
  <tr>
    <th colspan="11">Llama3-8B</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>36.64</td>
    <td>45.95</td>
    <td>72.71</td>
    <td>39.84</td>
    <td>28.26</td>
    <td>19.53</td>
    <td>19.80</td>
    <td>16.87</td>
    <td>14.97</td>
    <td>32.73</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td>32.26</td>
    <td>37.67</td>
    <td>71.51</td>
    <td>38.40</td>
    <td>23.09</td>
    <td>22.65</td>
    <td>1.43</td>
    <td>3.44</td>
    <td>4.72</td>
    <td>26.13</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td>37.16</td>
    <td>45.90</td>
    <td>72.99</td>
    <td>35.80</td>
    <td>32.20</td>
    <td>20.23</td>
    <td>14.21</td>
    <td>18.31</td>
    <td>16.72</td>
    <td>32.61</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td><b>39.15</b></td>
    <td><b>48.50</b></td>
    <td><b>73.63</b></td>
    <td><b>40.51</b></td>
    <td><b>35.77</b></td>
    <td><b>27.22</b></td>
    <td><b>21.14</b></td>
    <td><b>20.42</b></td>
    <td><b>17.32</b></td>
    <td><b>35.96</b></td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>35.12</td>
    <td>43.26</td>
    <td>73.53</td>
    <td>38.75</td>
    <td>30.60</td>
    <td>18.87</td>
    <td>21.03</td>
    <td>19.02</td>
    <td>16.09</td>
    <td>32.92</td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>36.69</td>
    <td>37.88</td>
    <td>72.66</td>
    <td>34.15</td>
    <td>18.15</td>
    <td>16.15</td>
    <td>18.96</td>
    <td>14.21</td>
    <td>9.79</td>
    <td>28.74</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>38.72</b></td>
    <td><b>51.86</b></td>
    <td><b>73.52</b></td>
    <td><b>44.27</b></td>
    <td><b>34.11</b></td>
    <td><b>35.47</b></td>
    <td><b>21.41</b></td>
    <td><b>20.85</b></td>
    <td><b>18.18</b></td>
    <td><b>37.60</b></td>
  </tr>
  <tr>
    <th colspan="11">Llama3-70B</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>42.39</td>
    <td>50.82</td>
    <td>83.59</td>
    <td>72.82</td>
    <td>46.21</td>
    <td>31.42</td>
    <td>29.39</td>
    <td>22.76</td>
    <td>18.72</td>
    <td>44.24</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td>41.10</td>
    <td>44.79</td>
    <td>81.89</td>
    <td>71.01</td>
    <td>35.18</td>
    <td>31.18</td>
    <td>9.83</td>
    <td>5.58</td>
    <td>12.70</td>
    <td>37.03</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td>42.72</td>
    <td>53.00</td>
    <td>82.90</td>
    <td>60.84</td>
    <td>50.06</td>
    <td>32.34</td>
    <td>27.11</td>
    <td>22.67</td>
    <td>19.89</td>
    <td>43.50</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td><b>44.69</b></td>
    <td><b>49.75</b></td>
    <td><b>83.24</b></td>
    <td><b>75.63</b></td>
    <td><b>51.79</b></td>
    <td><b>33.67</b></td>
    <td><b>30.08</b></td>
    <td><b>24.41</b></td>
    <td><b>21.77</b></td>
    <td><b>46.11</b></td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>41.34</td>
    <td>45.13</td>
    <td>83.25</td>
    <td>71.58</td>
    <td>49.16</td>
    <td>25.84</td>
    <td><b>31.36</b></td>
    <td><b>26.56</b></td>
    <td><b>21.40</b></td>
    <td><b>43.96</b></td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>44.26</td>
    <td>50.75</td>
    <td>83.30</td>
    <td>55.33</td>
    <td>27.44</td>
    <td>29.23</td>
    <td>25.36</td>
    <td>18.84</td>
    <td>11.53</td>
    <td>38.45</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>46.17</b></td>
    <td><b>57.28</b></td>
    <td><b>83.69</b></td>
    <td><b>79.48</b></td>
    <td><b>51.61</b></td>
    <td><b>36.15</b></td>
    <td><b>30.47</b></td>
    <td><b>27.18</b></td>
    <td><b>24.30</b></td>
    <td><b>48.48</b></td>
  </tr>
  <tr>
    <th colspan="11">Qwen2.5-3B</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>17.81</td>
    <td>49.79</td>
    <td>70.77</td>
    <td>36.96</td>
    <td>25.67</td>
    <td>15.98</td>
    <td>14.48</td>
    <td>13.18</td>
    <td>9.80</td>
    <td>28.27</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td>21.15</td>
    <td>50.18</td>
    <td>68.40</td>
    <td>36.65</td>
    <td>20.83</td>
    <td>18.32</td>
    <td>1.01</td>
    <td>2.26</td>
    <td>0.16</td>
    <td>24.33</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td>21.69</td>
    <td>46.76</td>
    <td>72.37</td>
    <td>34.03</td>
    <td>28.87</td>
    <td>17.23</td>
    <td>9.93</td>
    <td>13.23</td>
    <td>12.18</td>
    <td>28.48</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td><b>23.28</b></td>
    <td><b>60.32</b></td>
    <td><b>72.73</b></td>
    <td><b>41.12</b></td>
    <td><b>33.09</b></td>
    <td><b>19.18</b></td>
    <td><b>15.28</b></td>
    <td><b>14.87</b></td>
    <td><b>13.57</b></td>
    <td><b>32.60</b></td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>15.99</td>
    <td>50.41</td>
    <td>71.55</td>
    <td>34.41</td>
    <td>30.89</td>
    <td>12.99</td>
    <td>14.14</td>
    <td><b>16.41</b></td>
    <td><b>12.45</b></td>
    <td>28.81</td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>20.85</td>
    <td>51.93</td>
    <td>67.98</td>
    <td>32.97</td>
    <td>13.21</td>
    <td>10.81</td>
    <td>13.37</td>
    <td>9.50</td>
    <td>10.20</td>
    <td>25.65</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>22.09</b></td>
    <td><b>59.70</b></td>
    <td><b>73.59</b></td>
    <td><b>39.39</b></td>
    <td><b>27.14</b></td>
    <td><b>27.20</b></td>
    <td><b>16.61</b></td>
    <td><b>15.24</b></td>
    <td><b>8.24</b></td>
    <td><b>32.17</b></td>
  </tr>
  <tr>
    <th colspan="11">Qwen2.5-7B</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>23.84</td>
    <td>45.36</td>
    <td>84.88</td>
    <td>75.65</td>
    <td>14.69</td>
    <td>17.38</td>
    <td>21.92</td>
    <td>16.09</td>
    <td>14.81</td>
    <td>34.96</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td>24.51</td>
    <td>41.30</td>
    <td>83.58</td>
    <td>73.83</td>
    <td>17.86</td>
    <td>20.58</td>
    <td>5.44</td>
    <td>3.70</td>
    <td>8.76</td>
    <td>31.06</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td>24.95</td>
    <td>32.15</td>
    <td>84.98</td>
    <td>75.48</td>
    <td>20.24</td>
    <td>18.97</td>
    <td>17.00</td>
    <td>17.41</td>
    <td>16.95</td>
    <td>34.24</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td><b>28.82</b></td>
    <td>43.78</td>
    <td>85.04</td>
    <td>75.38</td>
    <td>15.53</td>
    <td>21.66</td>
    <td><b>22.18</b></td>
    <td><b>18.16</b></td>
    <td><b>17.20</b></td>
    <td><b>36.42</b></td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>19.94</td>
    <td>44.04</td>
    <td><b>85.16</b></td>
    <td>75.38</td>
    <td>18.81</td>
    <td>15.30</td>
    <td><b>23.80</b></td>
    <td><b>19.63</b></td>
    <td><b>15.31</b></td>
    <td><b>35.26</b></td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>15.41</td>
    <td>40.92</td>
    <td>84.07</td>
    <td>66.86</td>
    <td>8.33</td>
    <td>16.61</td>
    <td>17.54</td>
    <td>10.48</td>
    <td>13.70</td>
    <td>30.44</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>25.10</b></td>
    <td><b>47.85</b></td>
    <td>84.86</td>
    <td><b>75.84</b></td>
    <td><b>23.81</b></td>
    <td><b>23.12</b></td>
    <td><b>21.66</b></td>
    <td><b>18.30</b></td>
    <td><b>17.55</b></td>
    <td><b>37.56</b></td>
  </tr>
  <tr>
    <th colspan="11">Qwen2.5-74B</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>25.67</td>
    <td>52.56</td>
    <td>76.58</td>
    <td>64.15</td>
    <td>34.23</td>
    <td>30.25</td>
    <td>24.96</td>
    <td>19.53</td>
    <td>17.27</td>
    <td>38.35</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td>16.09</td>
    <td>48.36</td>
    <td>78.31</td>
    <td>61.59</td>
    <td>36.61</td>
    <td>37.25</td>
    <td>8.92</td>
    <td>6.42</td>
    <td>8.16</td>
    <td>33.52</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td>25.81</td>
    <td>52.96</td>
    <td>77.98</td>
    <td>64.23</td>
    <td>38.81</td>
    <td>31.64</td>
    <td>21.83</td>
    <td>20.55</td>
    <td>17.43</td>
    <td>39.03</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td><b>21.03</b></td>
    <td>51.27</td>
    <td>76.38</td>
    <td>64.44</td>
    <td><b>40.18</b></td>
    <td>40.16</td>
    <td>25.57</td>
    <td>22.59</td>
    <td>18.81</td>
    <td>40.07</td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>17.68</td>
    <td>51.29</td>
    <td>76.32</td>
    <td>62.11</td>
    <td>37.14</td>
    <td>23.59</td>
    <td><b>23.32</b></td>
    <td><b>18.19</b></td>
    <td><b>17.36</b></td>
    <td><b>37.26</b></td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>3.90</td>
    <td>48.72</td>
    <td>78.32</td>
    <td>64.72</td>
    <td>26.85</td>
    <td>23.10</td>
    <td>20.74</td>
    <td>11.60</td>
    <td>13.13</td>
    <td>32.34</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>29.32</b></td>
    <td><b>53.83</b></td>
    <td><b>79.06</b></td>
    <td><b>70.19</b></td>
    <td><b>35.24</b></td>
    <td><b>41.38</b></td>
    <td><b>25.79</b></td>
    <td><b>21.93</b></td>
    <td><b>20.44</b></td>
    <td><b>41.91</b></td>
  </tr>
  <tr>
    <th colspan="11">Qwen2.5-72B</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>12.85</td>
    <td>62.43</td>
    <td>85.56</td>
    <td>85.48</td>
    <td>41.83</td>
    <td>32.31</td>
    <td>28.97</td>
    <td>22.12</td>
    <td>19.01</td>
    <td>43.40</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td>11.52</td>
    <td>58.64</td>
    <td>85.56</td>
    <td>83.73</td>
    <td>34.46</td>
    <td>35.52</td>
    <td>12.34</td>
    <td>5.60</td>
    <td>15.12</td>
    <td>38.06</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td><b>20.78</b></td>
    <td>57.03</td>
    <td><b>85.63</b></td>
    <td>84.82</td>
    <td><b>49.40</b></td>
    <td>31.00</td>
    <td>26.03</td>
    <td>21.89</td>
    <td>19.29</td>
    <td>43.99</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>17.94</td>
    <td><b>66.03</b></td>
    <td>71.27</td>
    <td>85.58</td>
    <td>44.46</td>
    <td>40.06</td>
    <td>28.53</td>
    <td>26.05</td>
    <td>19.69</td>
    <td>44.40</td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>11.16</td>
    <td>62.86</td>
    <td>85.52</td>
    <td>85.01</td>
    <td>48.63</td>
    <td>31.08</td>
    <td><b>29.82</b></td>
    <td><b>21.72</b></td>
    <td><b>14.66</b></td>
    <td><b>44.66</b></td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>0.05</td>
    <td>57.73</td>
    <td>83.70</td>
    <td>84.42</td>
    <td>27.08</td>
    <td>33.02</td>
    <td>24.21</td>
    <td>13.17</td>
    <td>15.05</td>
    <td>37.60</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>19.10</b></td>
    <td><b>63.38</b></td>
    <td>85.46</td>
    <td><b>86.67</b></td>
    <td>44.46</td>
    <td><b>41.55</b></td>
    <td>28.01</td>
    <td>25.46</td>
    <td><b>22.66</b></td>
    <td><b>46.31</b></td>
  </tr>
  <tr>
    <th colspan="11">Qwen2.5-72B</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>34.13</td>
    <td>57.59</td>
    <td>87.12</td>
    <td>87.73</td>
    <td>32.49</td>
    <td>35.95</td>
    <td>29.40</td>
    <td>23.37</td>
    <td>19.67</td>
    <td>45.27</td>
  </tr>
  <tr>
    <td>BM25-Major</td>
    <td>28.76</td>
    <td>49.16</td>
    <td>85.72</td>
    <td>86.88</td>
    <td>36.25</td>
    <td>43.46</td>
    <td>16.47</td>
    <td>7.78</td>
    <td>13.41</td>
    <td>40.88</td>
  </tr>
  <tr>
    <td>BGE-KMeans</td>
    <td><b>38.86</b></td>
    <td>43.75</td>
    <td>87.24</td>
    <td>87.36</td>
    <td>40.18</td>
    <td>40.63</td>
    <td>27.29</td>
    <td>25.22</td>
    <td>21.57</td>
    <td>45.79</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>37.54</td>
    <td>62.57</td>
    <td>86.86</td>
    <td>88.02</td>
    <td>39.94</td>
    <td>40.88</td>
    <td>30.37</td>
    <td>25.14</td>
    <td>20.98</td>
    <td>48.03</td>
  </tr>
  <tr>
    <td>EPR-KMeans</td>
    <td>29.31</td>
    <td>56.36</td>
    <td><b>87.99</b></td>
    <td>87.44</td>
    <td><b>42.02</b></td>
    <td>30.30</td>
    <td><b>30.67</b></td>
    <td><b>25.89</b></td>
    <td><b>21.85</b></td>
    <td><b>45.76</b></td>
  </tr>
  <tr>
    <td>Latent-Bayesian</td>
    <td>24.81</td>
    <td>52.97</td>
    <td>85.03</td>
    <td>88.14</td>
    <td>15.42</td>
    <td>34.75</td>
    <td>25.17</td>
    <td>3.31</td>
    <td>15.17</td>
    <td>38.31</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>38.30</b></td>
    <td><b>64.05</b></td>
    <td>87.03</td>
    <td><b>88.32</b></td>
    <td><b>41.75</b></td>
    <td><b>47.67</b></td>
    <td>29.09</td>
    <td><b>25.63</b></td>
    <td><b>23.79</b></td>
    <td><b>49.51</b></td>
  </tr>
</table>

Table 12: Main results, averaged over 4-shot to 128-shot settings, on all models. The best scores on each model are in bold, and the second-best ones are underlined.
<table>
  <tr>
    <th>Method</th>
    <th>SST-5</th>
    <th>MNLI</th>
    <th>CMSQA</th>
    <th>HeSwag.</th>
    <th>GeoQ.</th>
    <th>NL2Bash</th>
    <th>Break</th>
    <th>MTOP</th>
    <th>SMCal.</th>
    <th>Average</th>
  </tr>
  <tr>
    <td><i>Owen-turbo-latest</i></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Random</td>
    <td>56.87</td>
    <td>80.93</td>
    <td>86.47</td>
    <td>79.07</td>
    <td>73.43</td>
    <td>62.61</td>
    <td>44.20</td>
    <td>28.73</td>
    <td>28.40</td>
    <td>60.08</td>
  </tr>
  <tr>
    <td>±std</td>
    <td>±2.36</td>
    <td>±0.80</td>
    <td>±1.21</td>
    <td>±0.44</td>
    <td>±3.97</td>
    <td>±1.15</td>
    <td>±2.88</td>
    <td>±2.22</td>
    <td>±1.99</td>
    <td>±0.22</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>52.67</td>
    <td>80.33</td>
    <td>85.33</td>
    <td>80.00</td>
    <td>72.86</td>
    <td>61.79</td>
    <td>42.00</td>
    <td>34.00</td>
    <td>27.67</td>
    <td>59.63</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>59.67</b></td>
    <td><b>83.33</b></td>
    <td><b>87.67</b></td>
    <td>79.00</td>
    <td><b>77.14</b></td>
    <td><b>64.39</b></td>
    <td><b>46.33</b></td>
    <td>30.67</td>
    <td><b>28.67</b></td>
    <td><b>61.87</b></td>
  </tr>
  <tr>
    <td><i>GLM+flash</i></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Random</td>
    <td>56.13</td>
    <td>74.54</td>
    <td><b>87.73</b></td>
    <td>69.60</td>
    <td>64.07</td>
    <td>61.03</td>
    <td>41.20</td>
    <td>28.20</td>
    <td>23.07</td>
    <td>56.17</td>
  </tr>
  <tr>
    <td>±std</td>
    <td>±1.99</td>
    <td>±1.24</td>
    <td>±0.74</td>
    <td>±13.97</td>
    <td>±1.82</td>
    <td>±1.03</td>
    <td>±3.04</td>
    <td>±0.96</td>
    <td>±2.14</td>
    <td>±2.24</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>53.00</td>
    <td>76.33</td>
    <td>87.00</td>
    <td>76.67</td>
    <td><b>66.07</b></td>
    <td>61.44</td>
    <td>44.67</td>
    <td><b>32.33</b></td>
    <td>21.33</td>
    <td>57.65</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>56.33</b></td>
    <td>72.67</td>
    <td>87.33</td>
    <td><b>77.67</b></td>
    <td>63.93</td>
    <td><b>62.24</b></td>
    <td><b>51.67</b></td>
    <td>30.67</td>
    <td><b>24.00</b></td>
    <td><b>58.50</b></td>
  </tr>
  <tr>
    <td><i>Matthao-pro-32k-OS28</i></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Random</td>
    <td>54.53</td>
    <td>81.00</td>
    <td>84.33</td>
    <td><b>79.07</b></td>
    <td>75.64</td>
    <td><b>64.57</b></td>
    <td>59.27</td>
    <td>38.33</td>
    <td>28.27</td>
    <td>62.78</td>
  </tr>
  <tr>
    <td>±std</td>
    <td>±1.94</td>
    <td>±1.97</td>
    <td>±0.92</td>
    <td>±0.68</td>
    <td>±2.16</td>
    <td>±0.78</td>
    <td>±1.53</td>
    <td>±2.09</td>
    <td>±1.56</td>
    <td>±0.26</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>53.67</td>
    <td>78.67</td>
    <td><b>84.67</b></td>
    <td>77.67</td>
    <td>74.64</td>
    <td>62.86</td>
    <td>58.67</td>
    <td>37.67</td>
    <td>26.67</td>
    <td>61.69</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>55.00</b></td>
    <td><b>81.33</b></td>
    <td>83.67</td>
    <td>77.67</td>
    <td><b>79.64</b></td>
    <td>62.82</td>
    <td><b>60.33</b></td>
    <td><b>46.33</b></td>
    <td><b>30.00</b></td>
    <td><b>64.09</b></td>
  </tr>
  <tr>
    <td><i>GPT-4o-mini</i></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Random</td>
    <td>55.93</td>
    <td>76.40</td>
    <td>84.23</td>
    <td>76.53</td>
    <td>70.36</td>
    <td>65.05</td>
    <td>53.73</td>
    <td>32.33</td>
    <td><b>27.33</b></td>
    <td>60.22</td>
  </tr>
  <tr>
    <td>±std</td>
    <td>±1.81</td>
    <td>±1.06</td>
    <td>±0.79</td>
    <td>±0.75</td>
    <td>±0.96</td>
    <td>±0.44</td>
    <td>±4.20</td>
    <td>±2.03</td>
    <td>±1.07</td>
    <td>±0.45</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>53.67</td>
    <td>75.67</td>
    <td>84.33</td>
    <td><b>77.67</b></td>
    <td><b>72.86</b></td>
    <td>63.87</td>
    <td>55.67</td>
    <td>36.33</td>
    <td>23.33</td>
    <td>60.38</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td><b>56.00</b></td>
    <td><b>78.67</b></td>
    <td><b>86.67</b></td>
    <td>76.67</td>
    <td>68.93</td>
    <td><b>65.72</b></td>
    <td><b>57.33</b></td>
    <td><b>36.67</b></td>
    <td>26.67</td>
    <td><b>61.48</b></td>
  </tr>
  <tr>
    <td><i>DeepSeek-V3</i></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Random</td>
    <td><b>58.20</b></td>
    <td>85.20</td>
    <td>87.80</td>
    <td>80.93</td>
    <td>83.43</td>
    <td><b>67.77</b></td>
    <td>59.73</td>
    <td>41.60</td>
    <td>36.80</td>
    <td>66.83</td>
  </tr>
  <tr>
    <td>±std</td>
    <td>±2.57</td>
    <td>±1.17</td>
    <td>±1.00</td>
    <td>±1.65</td>
    <td>±2.48</td>
    <td>±0.60</td>
    <td>±2.09</td>
    <td>±3.71</td>
    <td>±2.90</td>
    <td>±0.52</td>
  </tr>
  <tr>
    <td>Best-of-N</td>
    <td>57.67</td>
    <td><b>85.67</b></td>
    <td>86.33</td>
    <td>81.33</td>
    <td>83.93</td>
    <td>65.84</td>
    <td>59.00</td>
    <td><b>47.33</b></td>
    <td>34.00</td>
    <td>66.79</td>
  </tr>
  <tr>
    <td>CLG (ours)</td>
    <td>56.33</td>
    <td>85.33</td>
    <td><b>88.00</b></td>
    <td><b>83.33</b></td>
    <td><b>87.14</b></td>
    <td>67.28</td>
    <td><b>62.00</b></td>
    <td>45.33</td>
    <td><b>38.00</b></td>
    <td><b>68.08</b></td>
  </tr>
</table>

Table 13: Full results of closed-source models under the 128-shot setting. We only evaluate at most 300 instances on each dataset for limited budget. Best scores on each model are in bold.
Figure 7: The scaling trends of random selection with respect to the number of shots on all models. We illustrate the standard deviations under 5 random seeds through error bars. The datasets that can steadily benefit from more shots are in solid colors, and the others are in semi-transparent colors.

Figure 8: The performance of different models on MNLI.

Reviewer paper summaries: This paper presents a gradient matching approach to enhance demonstration selection for many-shot in-context learning (ICL), addressing the limitations of random selection. The method selects demonstrations by aligning fine-tuning gradients between the target task's training set and chosen examples, achieving better learning effects. Experiments show that this approach consistently outperforms random selection on various LLMs and datasets, improving performance by an average of 4% on models like Llama3-70B and Qwen2.5-72B. This work enhances the reliability and effectiveness of many-shot ICL, paving the way for broader applications.
Summary of strengths: 1. This study presents an innovative methodology, Gradient Matching, for demonstration selection in many-shot in-context learning paradigms, which is an important task.

2. This paper is very well-written, with clear charts and graphs that make it easy to read, follow, and understand.

3. This paper presents a comprehensive empirical evaluation, conducting extensive experiments to validate both the objectives and outcomes of the proposed methodology. The thorough experimental analysis provides substantial empirical evidence supporting our approach.
Summary of weaknesses: 1. The experimental evaluation reveals varying levels of improvement across different datasets, with some showing limited enhancement. Considering that in-context learning represents only one fundamental capability of large language models, the significant computational resources required for these moderate improvements raises questions about the practical value and cost-effectiveness of such approaches.

2. With the growing body of work on demonstration selection methods, what was the fundamental motivation that specifically led to the development of the Gradient Matching approach? What key insights or observations inspired this particular direction?

3. The lack of open-source implementation and associated datasets for this study raises considerable concerns about experimental reproducibility, which is crucial for validating the reported findings and building upon this work.
Comments, suggestions and typos: please refer to Summary Of Weaknesses.
Overall assessment scores: 3