Towards Compute-Optimal Many-Shot In-Context Learning
Authors: Shahriar Golchin
Yanfei Chen
Rujun Han
Manan Gandhi
Tianli Yu
Swaroop Mishra
Mihai Surdeanu
Rishabh Agarwal
Chen-Yu Lee
Tomas Pfister
Venue: COLM 2025 (2025)
Keywords: Many-Shot In-Context Learning
In-Context Learning
Large Language Models
Compute-Optimal In-Context Learning
Demonstration Selection
TL;DR: We propose two straightforward and effective strategies for selecting demonstrations in many-shot in-context learning that balance performance and inference cost.

Abstract: Long-context large language models (LLMs) are able to process inputs containing up to several million tokens. In the scope of in-context learning (ICL), this translates into using hundreds/thousands of demonstrations in the input prompt, enabling many-shot ICL. In practice, a fixed set of demonstrations is often selected at random in many-shot settings due to (1) high inference costs, (2) the benefits of caching and reusing computations, and (3) the similar performance offered by this strategy compared to others when scaled. In this work, we propose two straightforward strategies for demonstration selection in many-shot ICL that improve performance with minimal computational overhead. Our first method combines a small number of demonstrations, selected based on their similarity to each test sample, with a disproportionately larger set of random demonstrations that are cached. The second strategy improves the first by replacing random demonstrations with those selected using centroids derived from test sample representations via k-means clustering. Our experiments with Gemini Pro and Flash across several datasets indicate that our strategies consistently outperform random selection and surpass or match the most performant selection approach while supporting caching and reducing inference cost by up to an order of magnitude. We also show that adjusting the proportion of demonstrations selected based on different criteria can balance performance and inference cost in many-shot ICL.

Published as a conference paper at COLM 2025
Towards Compute-Optimal Many-Shot In-Context Learning
Shahriar Golchin
Yanfei Chen?
Rujun Han
Manan Gandhi?
Tianli Yu²
Swaroop Mishra
Mihai Surdeanu"
Rishabh Agarwal
Chen-Yu Lee"
Tomas Pfister2
University of Arizona
?oogge Cloud Al Research GGoogle DeepMind
Abstract
Long-context large language models (LLMs) are able to process inputs containing up to several million tokens. In the scope of in-context learning (ICL), this translates into using hundreds/ /atods/ndossass of demonstrations in the input prompt, enabling many-shot ICL. In practice, a fixed set of demonstrations is often selected at random in many-shot settings due to (1) high inference costs, (2) the benefits of caching and reusing computations, and (3) the similar performance offered by this strategy compared to oth- ers when scaled. In this work, we propose two straightforward strategies for demonstration selection in many-shot ICL that improve performance with minimal computational overhead. Our first method combines a small number of demonstrations, selected based their similarity to each test sample, with a disproportionately larger set of random demonstrations that are cached. The second strategy improves the first by replacing ran- dom demonstrations with those selected using centroids derived from test sample representations via k-means clustering. Our experiments with Gemini Pro and Flash across several datasets indicate that our strategies consistently outperform random selection and surpass or match the most performant selection approach while supporting caching and reducing inference cost by up to an order magnitude. We also show that adjusting the proportion of demonstrations selected based on different criteria can balance performance and inference cost in many-shot ICL.
Introduction
In-context learning (ICL) is a popular technique for adapting large language models (LLMs) to downstream tasks (Brown et al., 2020). With long- long-context LLMss (Anil et al., 2023; Georgiev et al., 2024; Fu et al., 2024; Ding et al., 2024) and the ability to cache and reuse computations (Pope et al., 2023), it has become practical to extremely increase the number of demonstra- tions in ICL, shifting from few-shot to many-shot settings to further enhance performance (Agarwal et al., 2024; Bertsch et al., 2024; Jiang et al., 2024, inter alia). In many-shot scenarios, random selection of demonstrations is often preferred (Agarwal et al., 2024; Bertsch et al., 2024), as it uses fixed set of demonstrations without any selection criteria. This allows caching computations to control inference cost while achieving satisfactory performance. On the other hand, selection strategies based on specific criteria, such as similarity (Liu et al., 2021), often outperform random selection. However, such methods are impractical in many-shot scenarios, as they dynamically update the input prompt for each downstream test sample, which prevents caching and leads to substantial inference cost.
Work done as a Student Researcher at Google Cloud AI Research. Correspondence to Shahriar Golchin <gol chonnrnaaai....... edu>> and Chen-Yu Lee <chenyulee@googleeccom com NNw at Microsoft.
Now at Meta.

---

Published as a conference paper at COLM 2025
Poreto Anelyste ANLI w Genini Pro
Poreto Analysie TREC m/ Geriiel Pro
Poreto Analysis CSM Plus Gentiel Pro
Poceto Aralysic Mecaltel w Genini Pro
100 109
10P
10
109
10"
131
100
18
12:
12P
10"
107
16'
10
100
10P
101
Propectinal intirmac Cast
Dropertinal inficesee Cont
Prepantcal intoraao Cost
Prepentccal bshevnes Coet
Atalyyiis
Genini Floil
Purcto Analyiie TREC
Egects Asalsis: Ocutin Flush
Analyii Mocaleed wt
Pareto analysis of performance vs. inference cost across various datasets, compar- ing our hybrid selection strategies, i.e., similarity and similarity-K-mennt with other methods. Our strategies balance the low inference cost of random or k-means-based selection with the performance gains of dynamic prompt-updating strategies, e.g., similarity-baser selection, achieving results comparable to or better than the strongest selection approach.
Figure 2:
We propose two novel strategies for demonstration selection in many-shot ICL. These strategies enable the dynamic customization of many-shot input prompt for each test sample while still remaining largely cacheable. Specifically, prompt customization is achieved by selecting a small number of demonstrations that are similar to each test sample, while the remaining demonstrations are chosen randomly and cached. This approach allows the cached random demonstrations to remain fixed across all test samples, significantly reducing computational overhead by requiring new computations only for the small number of similar demonstrations for each test sample. For example, in our selection strategy under a 100-shot setting, we select only 20 most similar demonstrations for each test sample and the other 80 random demonstrations remain cached. Figure 1 depicts this many-shot prompt format. This idea is motivated by an observation derived from analyzing results reported in previous studies (Agarwal et al., 2024; Bertsch et al., 2024): in many-shot settings, the influence of selection criteria (e.g,, similarity) on performance diminishes as the number of demonstrations increases substantially, and beyond a certain point, their impact becomes nearly equivalent to that of random demonstrations We further confirm this through our own experiments, presented in Subsection 4.2.
Instruction
Building on the first strategy, our second strategy replaces the cached random demonstrations with a fixed, diverse set of demonstrations selected us-
Random Demonstrations
or
ing k-means clustering. In particular, we compute centroids based on test sample representations, map these centroids to the representations of the available demonstrations, and select the most similar ones to cache for use during inference. Figure 3 illustrates this selection strategy.
Meenns-Selected Demonstrations
Similar Demonstrations
Downstream Test Somple
Figure 1: Our proposed maoy ICL prompt format. Hatched blocks represent cached content, while solid blocks indicate non-cached content. When LLM is prompted, new com- putations are performed only for a small set of demonstrations selected based on similarity to the down- stream test sample, while previous computations for a larger set of ran- dom or t-means-selected demonstra- tions are reused. Block sizes in the figure aim to reflect their actual pro- portions in the input prompt.
The key contributions are as follows:
(1) We propose two novel strategies for demonstra- tion selection in many-shot ICL that outperform or perform on par with the strongest selection approach while significantly reducing inference cost by making them largely cacheable.
(2) We show that LLMs benefit more from ICL when demonstrations are selected using multiple criteria, rather than relying on a single criterion such as sim- ilarity or diversity, and the proportion of demonstra- tions selected based on multiple criteria can balance performance and inference cost, as shown in Figure 2.
2

---

Published as a conference paper at COLM 2025
2 Approach
2.1 Motivation and Challenges
Many-shot ICL, due to using an extensive number of demonstrations, presents unique scenarios that make demonstration selection strategies used in few-shot settings impractical. Two key properties that characterize these scenarios are:
(1) High Inference Cost: Processing a large number of demonstrations in many-shot settings leads to substantial computational cost. Therefore, cacheable methods, such as random selection of demonstrations, are often preferred in such settings for practical use cases (Agarwal et al., 2024; Bertsch et al., 2024; Baek et al., 2024).
(2) Diminishing Effectiveness of Selection Criteria: Criteria that are effective for selecting demonstrations in few-shot settings lose their effectiveness on performance improvement as the number of demonstrations increases. For example, when demonstrations are chosen based on similarity, the sheer number of demonstrations causes this criterion to diminish gradually, and beyond certain point, demonstrations effectively become equivalent to random selection (Agarwal et al., 2024; Bertsch et al., 2024). We verify this in Subsection 4.2.
Motivated by these two observations, we propose two new selection strategies tailored for many-shot scenarios. These approaches aim to address the challenges outlined above while combining the strengths of multiple selection criteria. Specifically, we construct set of demonstrations that incorporate multiple criteria, such similarity, diversity, and randomness, while maintaining the option for caching to keep inference costs manageable. The following subsections detail each selection strategy.
2.2 Hybrid Similarity-Rande Selection
As noted, in many-shot settings, only a small number of demonstrations that are selected based on specific criterion effectively improves performance. Given this insight, we select only small set of demonstrations based on the similarity criterion to include in the input prompt. Specifically, we use s most similar demonstrations to each downstream test sample.
To form the many-shot prompt, the remaining demonstrations are selected randomly. These random demonstrations are nearly effective as similar ones in many-shot scenarios due to the diminishing effectiveness of selection criterion, with the added benefit of being fixed and therefore cacheable.
In this setup, as the number of random demonstrations is significantly larger than the number of similar ones, the majority of the input prompt remains fixed and cached, reducing the computational cost of inference in many-shot settings while tailoring the input prompt to each downstream test sample. In particular, an n-shot prompt is formed with similar demonstrations and random demonstrations, where r >> s, and и = s +r. r. Figure 1 illustrates the approximate proportions of these components in the input prompt.
2.3 Hybrid Similarity-l-Means Selection
As an alternative our initial strategy, we use a k-means clustering approach to replace randomly selected demonstrations for the fixed and caching part of the input prompt. This method aims to enhance our first strategy by encouraging diversity in demonstration selection To select more effective demonstrations for test samples and avoid noisy demonstrations that can harm performance, we adapt the use of k-means clustering. Instead of directly applying it to the pool of available demonstrations, we first apply k-means clustering to the test samples and compute centroids using the elbow method (Thorndike, 1953). We then map these centroids to the representations of available demonstrations, selecting the ones most similar to the centroids. This ensures that the selected demonstrations are semantically relevant and diverse with respect the test samples. Figure 3 depicts this mapping.
1eee Appendix for the actual prompts used for each dataset.
3

---

Published as a conference paper at COLM 2025
Finally, the number of demonstrations se- lected based on each centroid is deter- mined by a predefined total, If the predefined total cannot be evenly split among centroids, the remaining demon- strations are randomly distributed across centroids until the total met. Similar to our first strategy, for an n--hot prompt consisting of s similar demonstrations and k t-meens--elected demonstrations, we maintain k > s, with = s + Figure displays these proportions as well.
Teit Sample bets Runfeld
bonanttrctio Foel bata Monfeld
Pact
o'
Figure 3: Illustration of selecting demonstra- tions using k-means clustering. First, centroids are computed based on the representations of test samples using the elbow method. These cen- troids are then mapped to the representations of available demonstrations. Lastly, demonstra- tions that are most similar to each centroid are selected based on a predefined total. In this example, the predefined total is 15 demonstra- tions, with three centroids identified. Therefore, each centroid is allocated five demonstrations.
3 Experimental Setup
Data. To ensure clarity across our eval- uation settings, we outline the datasets used for each evaluation and refer to the corresponding subsections below.
Subsections 4.1, 4.2, and 4.3: To evaluate the effectiveness of our proposed selection methods, compare their inference costs with other approaches, and analyze the diminishing impact of selection criteria in many-shot ICL, we use tasks covering natural language inference, classification with high label complexity, reasoning, and tool use. Specifically, we utilize the ANLI (r3) (Nie et al., 2019), TREC (with 50 fine-grained labels) (Li & Roth, 2002; Hovy et al., 2001), GSM Plus (Li et al., 2024b), and Meta'Tool (Huang et al., 2023) datasets, respectively. For each dataset, we randomly select 1,000 test samples from the test set. If a test set has fewer than 1,000 samples, we use the entire set. From the train set, we sample up to 100,000 samples randomly; if fewer are available, we the full train set. For datasets with only one split (e.g., only a test or train set), we allocate 1,000 random samples for testing and leverage the up to 100,000 samples-as the train set. In all cases, the train set is viewed as the pool of available demonstrations.
Subsection 4.4: To assess our selection strategies in low-data regime, we use selected tasks from the Big- Big-Bench Hard (BBH) dataset (Suzgun et al., 2022), including Geometric Shapes, Logical Deduction (with seven objects), and Salient Translation Error Detection. For each task, we create demonstration pool containing 100 random samples and test set with 150 random samples.
Models. We employ Gemini Pro and Flash (Georgiev et al., 2024) as the base models in our experiments, accessed through Google API. We particularly use the gemini-1.5-pro-00 5-pro-000 and gemini-2.. 0-220-flash--e endpoints, respectively. To promote deterministic results, we set the temperature to zero across all experiments and maintain default values for all other decoding hyperparameters. For non-generative tasks, we cap the maximum completion length at 20 tokens; for generative tasks, we allow up to 8,000 tokens.
Embeddings. We leverage Gecko embeddings (Lee et al., 2024), accessed via Vertex AI on Google Cloud Platform, to select demonstrations based on similarity. Specifically, we utilize the text wext-embedding-eee set the task type to SEMANTIC SIMILARITY. We compute similarity using cosine similarity and average embeddings.
endpoint (Google Cloud, 2024) with dimensionality of 768 and
Demonstrations. We vary the number of demonstrations from 0 to 200 in steps of 50, i.e., E (0,50, 100, 150, 200). Regardless of the value of constant number of s = 20 similar demonstrations are always included in the input prompt across all settings when using our selection strategies. The remaining demonstrations are preselected either randomly using k-means clustering, with r, E (0,30, 80, 130, 180), and cached for reuse during inference.
In scenarios where test data is not available in advance, this method can be applied progressively new test samples arrive.
4

---

Published as a conference paper at COLM 2025
As for the order of demonstrations, previous studies showed that the order in which demonstrations are presented in the input prompt can influence performance in few--hot ICL (Zhao et al., 2021; Lu et al., 2021; Zhang et al., 2022). However, this effect becomes negligible in many-shot settings (Bertsch et al., 2024). Therefore, for random selection baseline selection strategy, both of which use randomly selected
and our hybrid similarity-randow demonstrations, we present demonstrations in random order and do not study the impact of ordering, it has little to no effect.
Caching. Our hybrid demonstration selection strategies do not depend on how caching is implemented, making them compatible with any caching approach. In this work, to cache content in many-shot settings, we choose to use the key-value caching method (Pope et al., 2023; Google AI, 2025), as it widely adopted by the community and practitioners.
Baselines. In line with most previous studies on ICL (Dong et al., 2022; Agarwal et al., 2024; Bertsch et al., 2024), we utilize two commonly adopted demonstration selection strategies in many-shot ICL as our baselines: random and similarity-based selection.
In addition, include two k-means-based baselines to evaluate the impact of mapping centroids from test data, as discussed in Subsection 2.2, and to assess the compound effect of incorporating similar demonstrations in our hybrid similarity-k-means strategy. The first baseline performs a centroid mapping from the test data to the pool of available demonstrations, then selects demonstrations most similar to those mapped centroids. This approach exactly follows the mapping process described in Subsection 2.2, except it excludes the similar demonstrations that are added for each downstream test sample. We refer to this baseline as k-means with mapping. The second baseline does not involve any centroid mapping from the test data. Instead, it directly identifies centroids within the pool of available demonstrations and selects demonstrations most similar to those centroids from that same pool. We call this baseline k-means without mapping. In both these baselines, the number of selected demonstrations from each cluster is evenly distributed based on the optimal number of clusters (i.e., centroids) and the total demonstration budget for each setting (e.g., 50-shot). If the demonstration budget cannot be evenly divided across clusters, the remaining are randomly assigned to clusters until the predefined total is reached.
We evaluate the performance of our proposed selection strategies against all four baselines and also compare the inference cost of each strategy in many-shot scenarios.
k-Means Clustering. In all our experiments, we use the default hyperparameters for k- means clustering provided by scikit-learn (Pedregosa et al., 2011). Additionally, we vary the number of clusters from to 20 to determine the optimal number of clusters based on inertia and the elbow method. For our hybrid -imilarity-k-means approach and the k-means baseline with mapping, the optimal number of clusters is chosen using the test data. In contrast, for the k-means baseline without mapping, the optimal number of clusters determined using the demonstration pool.
Results and Discussion
Figure 4 presents the many-shot ICL performance of Gemini Pro and Flash across four datasets, along with their inference cost comparisons. Figure 2 also displays the same results as Pareto plots. Below, we first compare our proposed demonstration selection strategies with other methods terms of inference cost. Next, we examine how the impact of criteria used in demonstration selection strategies diminishes in many ICL. We then evaluate the performance-cost efficiency of our methods against other approaches. Finally, assess the effectiveness of our methods in low-data scenarios, where the pool of available demonstrations is limited.
4.1 Inference Cost Analysis
Due to the proprietary nature of our LLMs used in our experiments, we analyze propor- tional inference costs instead of absolute values for each method. Specifically, we calculate
5

---

Published as a conference paper at COLM 2025
Results on ANLJ & Geminl Pro
Results on TRBC Cemin: Pro Soapnng
Besults GSM Plus & Geminl Pro
Resultss on MotaTool & Gentii Pro
Accurvy (hg)
B
Soctime 153
50
100
154
100 Number of Shots
230
Number of Shots
Number Shoes
Number Stots
on
Results on Gemini
Results
Gemmi Flash
Resuits on Fash
TD
Acturaty
02
02
100
1541
2003
50
100
15D
230
150
100
200
Number of Shots
Number Shotss
Number Shots
Number Shots
14B
0.25
100
350
Figure 4: Results from Gemini Pro and Flash on various datasets in many-shot ICL settings. The first and second rows show performance plots for Gemini Pro and Flash, respectively, while the third row shows the inference cost comparisons for both models. We compare our two hybrid selection strategies in terms of both performance and inference cost against four baselines: (1) random selection, (2) ameans-based selection without mapping, (3) k-means-based selection with mapping, and (4) similarity-basee selection.
proportional inference costs using the time complexity of the attention mechanism, as it is the predominant operation in Transformer-basse beoeeooooooee modelss
models (Vaswani et al., 2017).
In the iimilarity-based selection strategy, where the input prompt is updated for each downstream test sample and thus cannot be cached, the proportional inference cost follows a complexity of O(n²), ), where n represents the average number of tokens in the prompt.
When using cached content, however, inference cost is O(tt+ where c is the average number of cached tokens, selected either randomly or via k-means clustering, and is the average number of tokens in the downstream test samples. This cost arises, as test tokens first attend to the cached content (O(ct) and then to themselves (O(F2). However, in many-shot settings, where the average number of cached tokens disproportionately larger than the average number of test tokens, i.e., c >>> 1, inference cost is dominated by Ó(t))
Similarly, in our hybrid strategies, which dynamically add similar demonstrations to cached content, inference cost is O(c(s + +) (s + +))), where s is the average number of tokens in the added similar demonstrations. Since each test sample is accompanied by a fixed number of similar demonstrations across all many-shot settings, and the average number of cached tokens disproportionately outnumber the average number of tokens from both similar demonstrations and test samples, i.e., c >> s + I, inference cost simplifies to Occ(s + t)).
Therefore, our proposed hybrid selection strategies scale linearly with the average number of cached tokens, i.e., (c), similar to random or k-means-based selection methods with fully cached content. This linear scaling results in significantly lower inference cost compared to the similarity-based selection strategy, which is quadratic.
Note that, since we consider proportional inference costs rather than absolute values, model size does not affect the scaling it acts as a constant multiplier. Also, all LLMs in our study share the same tokenizer, so the number of input tokens remains identical across different many-shot settings. This allows us to compare inference costs across different strategies and model sizes within the same figure for each dataset.
As shown in Figure 4, the similarity-based strategy, which updates the input prompt for each downstream test sample, leads to quadratic increase in inference cost. This is especially
6

---

Published as a conference paper at COLM 2025
noticeable in tasks with long input contexts, such as in the GSM Plus dataset. In comparison, our hybrid methods maintain an inference cost similar to random or k-means-based selection, which is significantly lower than that of the similarity- strategy.
4.2 Diminishing Impact of Selection Criterion
As discussed in Section 2, in many-shot ICL, the influence of selection criteria weakens as the number of demonstrations increases. This visible in the performance plots shown in Figure 4. When similarity-based selection outperforms random selection, its benefit plateaus beyond A certain point. A similar pattern can be seen in both of the k-means-based baselines, where diversity is the key selection criterion.
For example, on the TREC dataset, the performance of the similarity-based selection strategy for both models remains almost unchanged beyond the 50-shot setting, whereas perfor- mance under random selection continues to improve, especially between 150 and 200 shots. In fact, similar demonstrations that contribute the most to improving performance are those selected initially. However, as more demonstrations are added, the similarity criterion decreases, reducing their contribution to performance improvement and resulting in almost no further gains. As a result, random selection begins to approximate the performance of similarity-based selection by including more demonstrations in the input prompt. This approximation is evident for the ANLI, GSM Plus, and MetaTor datasets, as indicated in Figure For the TREC dataset, this approximation appears require more than 200 shots, as shown by previous studies (Agarwal et al., 2024; Bertsch et al., 2024).
4.3 Performance-Coss Evaluation
Figure/ 2 presents Pareto plots for both Gemini Pro and Flash across different datasets, comparing our proposed selection strategies with other methods based on their proportional inference cost. Figure 4 further illustrates these metrics metrrcs- performance and proportional inference cost disaggreeee by the number of demonstrations in the input prompt.
In a nutshell, random selection achieves the lowest overall performance and also incurs one of the lowest inference costs among all methods. The k-means-based selection strategy without centroid mapping performs slightly better at almost the same low inference cost, placing it second-lowest overall. When centroid mapping from test data is applied in the k-means strategy, performance improves further with almost no additional inference cost. This improvement shows that mapping centroids from test samples helps select more effective demonstrations and filter out noisy ones that can harm performance. In contrast, the similarity- all baselines but results in significantly higher inference cost that grows quadratically.
P selection strategy generally achieves the highest performance among
Both of our selection strategies match the similarity-based approach in cases where it delivers the highest performance. In some datasets, such ANLI and MetaTool, our strategies even outperform it. However, this level of performance in our methods is achieved with substantially lower inference costs costs-nearly two times lower in 50-shot settings and ten times lower in 200-shot settings. As discussed in Subsection 4.1, both of our methods have computational complexity of O(c), due to cached content, scaling linearly with the average number of cached tokens. In contrast, the similarity-based strategy has complexity of ( where n is the average number of tokens in the input prompt. Overall, our methods effectively balance performance and inference cost, achieving comparable or better results compared to the similarity-base based approach while keeping inference costs significantly lower.
When comparing our two proposed strategies, there no single universally best performer across all datasets and settings. However, they have nearly identical inference costs, as both adopt the same approach to add similar demonstrations and cache the preselected ones.
More importantly, the performance plots in Figure 4 exhibit compound effect of our hybrid selection strategies on both performance improvement and inference cost efficiency. For ex- ample, the Mooa dataset, the similarity-based strategy does not enhance performance compared to the zero-shot setting, resulting in almost no change in performance for both models. On the other hand, random selection of demonstrations enhances performance.
7

---

Published as a conference paper at COLM 2025
Recalss Geometric Shapes & Camini Pro
Results Logical Deduction & Gemini Pro
Results Sallent Translation Gemini
66-
VAL
20.00
Accurecy
69-
40:10
20:00
40:50
00:40
Proportion Shots
Proportion of Shots
Propartion Slooks
Results Geometrie
Geminl Faah
Results Logical Deduction Cemini Flash
Results Solient Translation Genini Fash
Accurncy (h)
75.35
73333
24-
70-
Aooureecy
20.30
10.00
00:40
110.00
20.00
10.60
00:40
00h20
20:00
40.60
60:40
80.20
Preportion
Proportion
Proportion
Figure Results across various subsets of the BBH dataset in a 100-shot ICL setting under low-data regime. While the total number of demonstrations is fixed at 100, the number of similar demonstrations increases from 20 to 80, moving from the leftmost to the rightmost bar pairs. Cortespondingly, the number of demonstrations selected either randomly or via k-means decreases from 80 to 20. For example, in each plot, the leftmost pair of bars represents setting with 20 similar demonstrations and 80 demonstrations selected either randomly (blue bar) or using k-means (red bar). Despite the limited pool of available demonstrations, our selection strategies outperform the common approach of using all available demonstrations in low-data scenarios.
When combined in our hybrid similarity-random selection strategy, performance exceeds that of random selection alone. Similarly, inference cost benefits from compound efficiency; random selection enables caching, while the similarity-basee component adds only a small computational overhead of few similar demonstrations.
A similar compound effect appears with our similarity-k-meant strategy, as seen in Figure 4. For instance, on the TREC dataset, the performance of the k-means baseline with mapping plateaus after an initial increase. Likewise, the similarity- baased strategy shows the same behavior but reaches a higher performance level. When the two methods are combined in our similarity-k-mean strategy, the performance rises from the level of the k-means baseline with mapping to match that of the similarity- method. In the 50-shot setting, this improvement is more than 11% with Gemini Flash and over 6% with Gemini Pro, while the inference cost cut in half.
strategy, which is the best-performing
As result, the proportion of demonstrations selected based on each criterion can serve hyperparameter to balance performance and inference cost in many-shot ICL settings.
4.4 Low-Data Regime Evaluation
One key factor influencing the performance of a similarity-based selection strategy is the availability of large and diverse demonstration pool to ensure that each downstream test sample has at least few similar demonstrations to be included in the input prompt. To evaluate our approaches in scenarios with limited data, we test them under condition where such a pool is unavailable. Maintaining ICL fixed at a 100-shot setting, we increase the number of similar demonstrations from 20 to 80 in increments of 20, while proportionally reducing the number of demonstrations selected either randomly or via k-means clustering.
This experiment answers two key questions commonly encountered in low-data regimes:
(1) How does the availability of similar demonstrations impact performance compared to using the full demonstration pool, the most common approach in low-data scenarios?
(2) In low-data scenarios where multiple runs allow for optimizing performance, how does adjusting the proportion of similar demonstrations influence performance?
Figure 5 demonstrates the results under the low-data regime. As shown, our selection strategies outperform the of all available demonstrations, with the top-performing
8

---

Published as a conference paper at COLM 2025
method alternating between the two strategies across different datasets. This suggests that even when the demonstration pool is limited, our selection strategies are more effective. Furthermore, adjusting the proportion of similar demonstrations can lead to performance gains of about 3%-6% over using the full pool.
Additionally, as similar demonstrations domly or via k-means, which is equivalent to selecting demonstrations with replacement, our findings complement those of Agarwal al. (2024), who observed that repeating demonstra- tions in-context does not necessarily enhance performance and that distinct demonstrations are key to performance improvement. We argue that repetition can still be beneficial when the repeated demonstrations provide information relevant to the downstream test sample.
selected independently of those chosen ran-
Related Work
In-Context Learning. ICL was first introduced by Brown et al. (2020) as training-free method to adapt LLMs to downstream tasks for improved performance. However, the effectiveness of this technique heavily relies on the selection of demonstrations included in the input prompt (Bölücü et al., 2023; Wan et al., 2025). To address this limitation, various studies explored different criteria for selecting demonstrations, including similarity (Liu et al., 2021), diversity (An et al., 2023), or the ordering of demonstrations (Zhao et al., 2021; Lu al., 2021; Zhang et al., 2022).
Another line of work aimed to uncover how ICL functions. Some studies analyzed its components to better understand its mechanism (Min et al., 2022; Yoo et al., 2022; Kossen et al., 2023; Lin & Lee, 2024). Others examined ICL through the perspective of gradient descent (von Oswald al., 2023), while another view frames ICL method of compressing the training set into single task-specific vector (Hendel et al., 2023). Moreover, Golchin et al. (2024) identified memorization as key factor contributing to ICL.
More recent research on ICL discovered additional capabilities beyond enhancing perfor- mance. These capabilities include performing regression (Vacareanu al., 2024), k-nearest neighbors (Agarwal et al., 2024; Dinh et al., 2022), jailbreaking (Anil et al., 2024), and LLMs as judges (Song et al., 2025; Golchin & Surdeanu, 2024; Li al., 2024a, inter alia).
Many-Shot In-Context Learning. Along with the selection of demonstrations that influ- ences ICL performance, the number of these demonstrations is another key factor that affects the results. With long-context LLMs, several studies attempted to mitigate this effect by significantly increasing the number of demonstrations to a few hundred (Zhang et al., 2023), thousands (Agarwal et al., 2024; Bertsch et al., 2024; Baek et al., 2024; Hao et al., 2022), or even by including the entire dataset within the input prompt as demonstrations (Agarwal et al., 2024; Back et al., 2024; Wan et al., 2024).
Beyond LLMs, many-shot ICL was applied in other models and applications. For instance, it was used with multimodal foundation models to enhance downstream performance (Jiang et al., 2024) and was also employed in molecular inverse design (Moayedpour et al., 2024).
Conclusion
We proposed two straightforward yet effective demonstration selection strategies for many- shot in-context learning (ICL) strike balance between performance and inference cost by using multiple criteria to select demonstrations. To control inference cost, our methods cache a large set of demonstrations, preselected either randomly or via k-means clustering. Then, for each downstream test sample, small number of similar demonstrations are dynamically included in the input prompt, further enhancing performance with only modest computational overhead. This hybrid approach combines caching for efficiency with similarity-based selection for improved performance. Our experiments across various tasks indicated that our methods consistently exceed or match the best-performing selection strategy while reducing inference costs by up to an order of magnitude, making them both practical and scalable for large many-shot ICL applications.

---

Published as a conference paper at COLM 2025
References
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. arXiv preprint Xf0:22404.11018 2024.
Shengnan An, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Jian-Guang Lou, and Dongmei Zhang. How do in-context examples affect compositional generalization? ar arxiv preprin! ar arXit:2305.04855, 2023.
Cem Anil, Esin DURMUS, Nina Rimsky, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel J Ford, Francesco Mosconi, Rajashree Agrawal, Rylan Schaeffer, Naomi Bashkansky, Samuel Svenningsen, Mike Lambert, Ansh Rad- hakrishnan, Carson Denison, Evan Hubinger, Yuntao Bai, Trenton Bricken, Timo- thy Maxwell, Nicholas Schiefer, James Sully, Alex Tamkin, Tamera Lanham, Karina Nguyen, Tomasz Korbak, Jared Kaplan, Deep Ganguli, Samuel R. Bowman, Ethan Perez, Roger Baker Grosse, and David Duvenaud. Many-shot jailbreaking. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openrevieww nett/forum?id=cwhngg71.1.
Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
Jinheon Back, Sun Jae Lee, Prakhar Gupta, Siddharth Dalmia, Prateek Kolhar, al. Revisit- ing in-context learning with long context language models. arv preprint arXiv:2412.16926, 2024.
Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.0020 002000, 2024.
Necva Bölücü, Maciej Rybinski, and Stephen Wan. impact of sample selection on in-context learning for entity extraction from scientific writing. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5090-5107, Singapore, December 2023. Association for Computational Linguis- tics. doi: findings-emnly 338/
URL https:://allatthoolge htw.hholoogy.oggg2000 00233023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond 2 million tokens. Xiv preprint ar 4rXin:2402.13753, 2024.
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris S. Papailiopoulos, and Kangwook Lee. LIFT: language- interfaced fine-tuning for non-language machine learning tasks. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- tion Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http://papers. nips. cttppp.ccpppppr...fcecc saapperffllss/ /2022/hash/ html.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arv preprint ar Xix: 20100433,.,0 00234, 2022.
10

---

Published as a conference paper at COLM 2025
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. arv preprint #XXiv:402210017, 2024.
Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.3: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
Shahriar Golchin and Mihai Surdeanu. Time travel in Ilms: Tracing data contamina- tion in large language models. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openrev/ ew. net/forum (nrfoormmmd22wwaces 2d=2Rw36.3
Shahriar Golchin, Mihai Surdeanu, Steven Bethard, Eduardo Blanco, and Ellen Riloff. Memorization in in-context learning. arviv preprint arXio:2408.11546 2024.
Google AI. Gemini API Caching Documentation, 2025. URL https: google. dev/
httpi/aapp//dos/ccccccicciic Accessed: 2025-02-13.
Google Cloud. Get text embeddings with generative ai. https://cloud. google. com/ httprerraiiieeai/c//eeeeeeeeeeee cloud. soougg. com/ wttpll...c.///ooo
wew.eeeettteeet-bbbbbennnneeeeee 2024. URL https:// http://erratteeeeaaeeo/////eeeaa http://dcc/eemmdddddi///ee seeet-ttex-eeeedingg..
Accessed: 2025-03-28.
Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in-context learning to 1,000 examples. arXiv preprint ar arX:::2212.06713,, 2022.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. Xiv preprint ar XrXv:2310015911, 2023.
Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran. Toward semantics-based answer pinpointing. In Proceedings of the First International Conference on Human Language Technology Research, 2001. URL https: www..//.wwwwwwwbbb..//...oo//... org/
anthology/H011 -oe00l09yHH1-169..
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. Metatool benchmark: Deciding whether to use tools and which use. aiv preprint arXiv: 2310.03128, 2023.
Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H Chen, and Andrew Y Ng. Many-shot in-context learning in multimodal foundation models. Xiv preprint ar Xto:2405.00999 2024.
Jannik Kossen, Tom Rainforth, and Yarin Gal. In-context learning in large language models learns label relationships but is not conventional learning, ar Xio preprint ar arXiv:2307.12375, 2023.
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. Gecko: Versatile text embeddings distilled from large language models. ar Xio preprint arXiv:2403.
2024.
Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Lims-as-judges: A comprehensive survey on llmebased evaluation methods. CoRR, abs/ /41200299,, 2024a. doi: 1..45500AARiV.2422222 48550/arKiv.244 www.aaii...22.......2. 05579.
URL https://doi. org/10.
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A com- prehensive benchmark for evaluating the robustness of lims as mathematical problem solvers. 2024b.
Xin Li and Dan Roth. Learning question classifiers, In COLING 2002: The 19th International Conference on Computational Linguistics, 2002. URL https: wwwww//.wwwww.w..........o...... org/onthhlogy/ C02-1150.
11

---

Published as a conference paper at COLM 2025
Ziqian Lin and Kangwook Lee. Dual operating modes of in-context learning. arXio preprint arXi::402.18811, 2024.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint ar arXiv:2101.0668 00.00000.044 2021.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp, Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint ar Xfx::21044008766 2021.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint ar arXio:2202.1283; 2022.
Saeed Moayedpour, Alejandro Corrochano-Navarce Faryad Sahneh, Shahriar Norooz- izadeh, Alexander Koetter, Jiri Vymetal, Lorenzo Kogler-Anele, Pablo Mas, Yasser Jangjou, Sizhen Li, et al. Many-shot in-context learning for molecular inverse design. arViv preprint a/Xiv:2407.19089, 2024.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. ar preprint ar XrXi::1910.1559,, 2019.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettennoofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and F. Duchesnay. Seikit-learn: Machine learning in python. Journal of Machine Learning Research, 12:2825-2883 2830, 2011.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling trans- former inference. Proceedings of Machine Learning and Systems, 5:606-624, 2023.
Mingyang Song, Mao Zheng, and Xuan Luo. Can many-shot in-context learning help Ilms as evaluators? A preliminary empirical study. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025, pp. 8232-8241. Association for Computational Linguistics, 2025. URL https:
org/ 2025/ col coling-mee loingmmain..mm 548/.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether chain-of-thought can solve them. ar preprint arXtm:2200.09261, 2022.
Robert L. Thorndike. Who belongs in the family? Psychometrika, 18(4):267-276 1953. doi: 10.1007 BBF2228922.
Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu, and Mihai Surdeanu. From words to numbers: Your large language model is secretly a capable regressor when given in-context examples. arXiv preprint ar X1442240400754,, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 3f5ee243547deeg1 (00e244447771111o00000111144///
2017. URL https: /ptp://////ococcc/s.s/... iereiiinnnnnneeiiiie courips.cc/pappee 2211 1070//7/0 7/hash/
html.
Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoggnnv, and Max Vladymyrov. Transformers learn in-context by gradient descent. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engellhrrtt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine
12

---

Published as a conference paper at COLM 2025
Learning, ICML 2023, 23-29 July 2023, Honoldu,, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 35151-35174. PMLR, 2023, URL htttp:://proceedingss mlr. wwwp:///ooooowwaaa2aaa..a.aa
Xingchen Wan, Ruoxi Sun, Hootan Nakhost, and Sercan O Arik. Teach better or show smarter? on instructions and exemplars in automatic prompt optimization. ar ar preprin! (#Xio:2406.15708, 2024.
Xingchen Wan, Han Zhou, Ruoxi Sun, Hootan Nakhost, Ke Jiang, and Sercan Ô Ark. From few to many: Self-improving many-shot reasoners through iterative optimization and generation. Xrv preprint ar arXir:2502.00330, 2025.
Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Wou Woo Lee, Sang-goo Lee, and Taeuk Kim. Ground-truth labels matter: A deeper look into input-label demonstrations. ar Xiv preprint ar arXi::2205.1266 2022.
Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing, Sentiment analysis in the era of large language models: A reality check. arXiv preprint arXiv:2305.15000 2023.
Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning. Xrv preprint ar Xrii:22211.0455 2022.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 12697-12706. PMLR, 2021. URL http: http:///ooceeedeeeee mlr prrpss/113//hha2111 c. html.
A Actual Input Prompts
Figures 6 to 19 display different parts of the actual input prompts used for each dataset, based on the prompt format shown Figure 1. In these figures, even-numbered ones show the fixed, cached part of the input prompts, while odd-numbered figures illustrate the dynamic, uncached parts. Each cached and uncached pair combines to form the complete many-shot input prompt for a given dataset. For example, Figures 6 and together form the complete input prompt used for the ANLI dataset.
The colors in these figures match the colors used in the prompt format in Figure 1. In particular, green highlights the instruction, blue represents demonstrations selected ei- ther randomly or using k-means clustering, red indicates similar demonstrations that are dynamically included in the input prompt for each downstream test sample, and purple shows the downstream test sample for which the model is prompted to make a prediction.
13

---

Published as a conference paper at COLM 2025
Instruction: Provide the most accurate label from the available labels that describes the relationship between the given sentence pair below. Available Labels: ["Entailment", "Neutral", "Contradiction"]
Premise: Deshdrohi (English: Country Traitor) is Bollywood comedy film. It was scripted and produced by Kamaal Rashid Khan who also appeared in the lead role with Manoj Tiwari, Hrishitaa Bhatt, Gracy Singh and Zulfi Syed. The movie has been listed as worst Hindi movie ever by all the critics. The movie fared badly and people demanded double the amount paid refund
Hypothesis: Country Traitor (Deshdrohi) is an inferior Hindi film
Label: Entailment
Premise: Oil prices, notoriously vulnerable to political events, spiked high $40 barrel during Gulf War 1991.
Hypothesis:
Oil prices will be affected by US elections.
Label: Neutral
other (Random Demonstrations] or (eeleeds-Selected Demonstrations)
Figure 6: An example of the fixed, cached part of the input prompt for the ANLI dataset. This prompt is followed by dynamic, uncached content, shown in Figure to form the complete many-shot input prompt.
14

---

Published as a conference paper at COLM 2025
[...] previous (Similar Demonstrations)
Premise: The Cheap Flight Tony was trying to travel to his brother's wedding. He couldn't afford most tickets so he booked something cheap. He arrived for his flight bracing himself for the worst. The flight long, dirty and uncomfortable. Still in the end at least he managed travel safely his destinati.
Hypothesis: This flight was an uncomfortable experience
Label: Entailment
Premise: I just let my thoughts run thought most surprising things Marilla felt helplessly that all this should be sternly reproved but she was hampered by the undeniable fact that some of the things Anne had said especially about the minister 's sermons and Mr. Bell 's prayers, were what she herself had really thought deep down her heart years, but never given expression to
Hypothesis: Marilla and Anne are best friends.
Label: Neutral
Premise: I just let my thoughts run and thought of the most surprising things Marilla felt helplessly that all this should be sternly reproved, but she was hampered by the undeniable fact that some things Anne had said especially about the minister sermons and Mr. Bell prayers, were what she herself had really thought deep down in her heart for years, but had never given expression to
Hypothesis: Marilla did not say everything Label:
Figure 7: An example of the dynamic, uncached part of the input prompt for the ANLI dataset. This portion of the prompt is updated for each downstream test sample.
15

---

Published as a conference paper at COLM 2025
Instruction: Provide the most accurate label from the available labels for the given text below. Available Labels: ("Abbreviation", "Expression Abbreviated", "Animal", "Organ of Body", "Color", "Invention Book and Other Creative Piece", "Currency Name", "Disease and Medicine" "Event", "Food", "Musical Instrument", "Language", "Letter Like A-Z", "Other Entily", "Plant", "Product", "Religion", "Sport", "Element and Substance", "Symbols and Sign", "Techniques and Method", "Equivalent Term", "Vehicle", "Word with A Special Property", "Definition of Something", "Description of Something", "Manner of An Action", "Reason", "Group Organization Persons", "Individual", "Title of A Person", "Description of A Person", "City", "Country", "Mountain", "Other Location", "State", "Postcode or Other Code", "Number of Something", "Date", "Distance, Linear Measure", "Price", "Order Rank", "Other Number", "Lasting Time of Something", "Percent Fraction", "Speed", "Temperature", "Size Area and Volume", "Weight"]
Text: What TV show chronicled the lives of Katy Holstrum and Congressman Glen Morley
Label: Invention Book and Other Creative Piece
Text: How do fuel injectors work
Label: Manner of An Action
other (Random Demonstrations) or (K-Means-Selacted Demonstrations)
Figure 8: An example of the fixed, cached part of the input prompt for the TREC dataset. This prompt is followed by dynamic, uncached content, shown Figure 9, form the complete many-shot input prompt.
[...]
previous (Similar Demonstrations)
Text: What Michelangelo sculpture Saint Peter 's Cathedral, Basilica
Label: Invention Book and Other Creative Piece
Text:
Who painted the Sistine Chapel
Label: Individual
Text:
Who painted the ceiling the Sistine Chapel
Label:
Figure 9: An example of the dynamic, uncached part of the input prompt for the TREC dataset. This portion of the prompt is updated for each downstream test sample.
16

---

Published as a conference paper at COLM 2025
Instruction: Follow these instructions and solve given question below: (1) solve the problem "step-by-step." (2) show all relevant calculations and reasoning, and (3) clearly state the final answer.
Question: Elly is arranging her books on some new bookshelves her parents gifted her. Each of the central shelves accommodate 10 books. The capacity of the lowest shelf is double that of a central shelf. The highest shelf can contain 5 books less than the lowest shelf. If she possesses 110 books, how many bookshelves will she require to store all of them?
Solution: bottom shelf can hold 2 10 middle shelves can hold total of <<2110=0>>>>>0 >0 books.
<<2"10=00>>>> books.
The top shelf hold 20
15:0-1555---: >1 books.
Each bookcase can hold a total 20 books + 20 books 15 books < <<20+20+15-35>55 books.
hold all her books, she needs
<110/55=2>>>> bookcases.
####
Question: Alicia's clothes have to sent to the dry cleaners weekly. weekly drop-off
includes 5 blouses, 2 pants, and 1 skirt. They charge her $5.00 per blouse, $6.00 per skirt, and
$8.00 per pair of pants. She has $3 coupon but finds it expired. How much does she spend dry-cleaning in weeks?
Solution: The blouses cost $5.00 each to dry clean and she has 5 that's
5*5 $<<<<55=25.00>>>>>>>>>
The one skirt she has costs $6.00 dry clean
The pants cost $8.00 to dry clean and she has 2 pains so that's 8*2
In one week she spends 25+6+16 $<<<55++16--77 170-0-1----0000
dry cleaning
Over 5 weeks, she will spend 5*47 $< 55 224777220000--- 235.00 on dry cleaning
###* 235
[...] other (Random Demonstrations) or (k-Means-SSlaated Demonstrations)
Figure 10: An example of the fixed, cached part of the input prompt for the GSM Plus dataset. This prompt is followed by dynamic, uncached content, shown in Figure 11, to form the complete many-shot input prompt.
17

---

Published as a conference paper at COLM 2025
[..]] previous (Similar Demonstrations)
Question: Raymond and Samantha share familial bond as cousins. Raymond came into the world 6 years prior to Samantha. At the age of 23, Raymond became father. Given that Samantha's current age is 31, can you determine how many years have passed since birth of Raymond's son?
Solution: When Raymond's son born Samantha 23 6 <<23-6=1>>>17 years old.
Thus it has been 31 <<1--17=14>> 14 years since Raymond's son was born.
#### 14
Question: Raymond and Samantha are cousins. Raymond was born years and 3 months
before Samantha. Raymond had a son the age of Samantha now 31, how many years was Raymond's son born?
Solution: years and 3 months be represented 6 3/12 1/4 6.25 years,
When Raymond's son was born Samantha was 6.25 1322-6.26116755>>> years old.
Thus it has been 31 16.75 = < <<31--1675=14222 14.25 years since Raymond's son was born.
##N0 14.25
Question: Raymond and Samantha are cousins. Raymond born years before Samantha. Raymond had son at the age of 23. Samantha's son is currently 7 years old, and Samantha gave birth to him at the age of 24. How many years ago Raymond's 's son born? Solution:
Figure 11: An example of the dynamic, uncached part of the input prompt for the GSM Plus dataset. This portion of the prompt is updated for each downstream test sample.
18

---

Published as a conference paper at COLM 2025
Instruction:
You helpful and intelligent assistant.
Your task to select most appropriate tool to answer user's query.
Below is list of available tools along with descriptions when or where to use them.
You may only choose tool from the list provided per query.
Available Tools: {
"timeport": "Begin an exciting journey through time, interact with unique characters, and learn history in this time-travel game!",
"airqualityforeast": "Planning something outdoors? Get 2-day air quality forecast for any US zip code.",
"copilot": "Searches every dealer, analyzes & ranks every for you you buy with confidence.",
"copywriter": "Send URL and get sales copywriting suggestions for any page!",
"calculator": "A calculator app that executes given formula and returns result. This app execute basic advanced operations.", [...]
Query: How can save a copy of my conversation?
Tool: exportchat
Query I'm planning to buy kitchen appliance from a reputable brand. Any recommendations?
Tool: ProductSearch
other (Random Demonstrutions) Ai-Meums-Selecterr Demonstrations)
An example of the fixed, cached part of the input prompt for the MetaTTol dataset. This prompt is followed by dynamic, uncached content, shown in Figure 13, to form the complete many-shot input prompt.
Figure 12:
[...] previous (Similar Demonstrations)
Query: I'm not sure what career path to take, Can you help me find my dream job based on my
interests and skills?
Tool: JobTool
Query: Hi, I'm curious about finding a job that aligns with my strengths and interests. Can you
guide me in the right direction?
Tool: JobTool
Query: Can you suggest specific method or tool that would help determine the type of job that would be the most suitable fit for my skills, interests, and qualifications?
Tool:
Figure 13: An example of the dynamic, uncached part of the input prompt for the MetaTool dataset. This portion of the prompt updated for each downstream test sample.
19

---

Published as a conference paper at COLM 2025
Instruction: Given full SVG path element containing multiple commands, your task is to determine the geometric shape that will be generaled if one were execute full path element. You may only generate the option letter corresponding to your answer, from options A to J.
Input: This SVG path element <path d="M 49.47,26.21 55.28,65.93 L 48.51,77.47 M 48.51,77.47 L 34.78,81.76 36.76,67.00 00 M 36.76,67.00 14.38,76.83 M 14.38,76.83 49.47,26.27'/5 draws (A) circle (B) heptagon (C) hexagon (D) kite (F) line (F) octagon (G) pentagon (H) rectangle
(I) sector ()) triangle
Target: C
Input: This SVG path element <path d="M 32.73/47.82 48.38/48.00 M 41.38,48.00 45.88,39.43
M 45.88,39 43.43 46.35,49.10 L 55.09,52.77 M 55.09,52.77 L 45.61,52.41 L 41.30,60.14 L 40.64,51.31
L 32.73,47.82* / draws
(A) circle (B) heptagon (C) hexagon (D) kite (E) line (F) octagon (G) pentagon (H) rectangle
sector ()) triangle
Target: F
other (Random Demonstrations) or (K-Means-SSlletted Demonstrations)
Figure 14: An example of the fixed, cached part of the input prompt for the BBH dataset, Geometric Shapes subset. This prompt is followed by dynamic, uncached content, shown in Figure 15, to form the complete many-shot input prompt.
previous (Similar Demonstrations)
Input: This SVG path element <path d="M 55.77,8066
57.38,65.80 M 57.38,65.80 L 48.90,57.46
M 48.90,57 46 45.58,47.78 M 45.58,47.78 53.25,36.07 L 66. 46.99.88.90 78.69,61.09
L 55.57, 8..57,80.69/// /> draws
(A) circle (B) heptagon (C) hexagon (D) kite (E) line (F) octagon (G) pentagon (H) rectangle
(I) sector ()) triangle
Target: B
Input: This SVG path element <path d="M 55.58,17.52 53.95,26.14 47.22,29.95 M 47.22,29.95
L 48.21,22.28 L 1.5..8,17.522/> draws
(A) circle (B) heptagon (C) hexagon (D) kite (E) line (F) octagon (G) pentagon (H) rectangle
sector ()) triangle
Target: D
Input: This SVG path element <path d="M 55.46,58.72 70.25,50.16 70.25,50.16 78.35,57.33
M 78.35,57.33 71.18,65.42 L. 55.46,58.72"/> draws
(A) circle (B) heptagon (C) hexagon (D) kite (F) line (F) octagon (G) pentagon (H) rectangle
sector triangle
Target:
Figure 15: An example of the dynamic, uncached part of the input prompt for the BBH dataset, Geometric Shapes subset. This portion of the prompt is updated for each down- stream test sample.
20

---

Published as a conference paper at COLM 2025
Deduce the correct order of sequence of objects based on the clues and information provided about their spacial relationships and placements. You may only generate the option letter corresponding to your answer, from options A to
Instruction:
seven objects arranged fixed order. The statements are logically consistent within each paragraph. A fruit stand sells seven fruits: mangoes, watermelons, peaches, kiwis, oranges, cantaloupes, and plums. The watermelons the cheapest. The peaches are more expensive than the mangoes. The cantaloupes are the second-most expensive. The oranges are more expensive than the cantaloupes. The peaches are less expensive than the plums. The kiwis the third-cheapest.
Input: following paragraphs each describe
(A) The mangoes are the third-most expensive
(B) The watermelons are the third-most expensive
(C) The peaches are third-most expensive
(D) The kiwis are the third-most expensive
(E) The oranges the third-most expensive
(F) The cantaloupes are the third-most expensive
(G) The plums are the third-most expensive
Target: G
Input: The following paragraphs each describe of seven objects arranged in fixed order. The statements are logically consistent within each paragraph. On shelf, there are seven books: a purple book, a green book, white book, a gray book, red book, black book, and a brown book. The gray book is the left of the purple book. The white book is the right of the brown book. The black book white book. The white book is the second from the right. The gray book is the third from the left. brown book
third from the right. The purple book is to the left of the
right (A) The purple book the third from the left
green book.
(B) The green book is the third from the left
(C) The white book is the third
left
(D) The gray book is the third from the left
(E) The red book is the third from the left
(F) The black book is the third from the left
(G) brown book the third from left
Target:
[..]]
other (Random Demonstrations] or (k-Means Seleeed Demonstrations)
Figure 16: An example of the fixed, cached part the input prompt for the BBH dataset, Logical Deduction subset. This prompt is followed by dynamic, uncached content, shown in Figure 17, to form the complete many-shot input prompt.
21

---

Published as a conference paper at COLM 2025
previous (Similar Demonstrations)
Input: The following paragraphs each describe set of seven objects arranged in fixed order. The stalements logically consistent within each paragraph.
branch, there are seven birds: an owl, crow, a falcon, a cardinal, hummingbird, quail, and hawk. The falcon is to the left of the crow. The quail is to the right of the cardinal. The hummingbird is to the right of the quail. The falcon is the second from right. hummingbird to the the hawk.
(A) The owl is the second from the right
(B) The crow is the second from the right
(C) The falcon is the second from the right
(D) The cardinal is the second from the right
(E) The hummingbird is the second from the right
(F) The quail is the second from the right
(G) The hawk is the second from the right
Target: C
Input: The following paragraphs each describe of objects arranged fixed order.
The statements are logically consistent within each paragraph. On branch, there are seven birds: owl, crow, falcon, cardinal, hummingbird, quail, and hawk. The falcon is to right of the cardinal. The hummingbird is right of the quail. The falcon is the second from the right. The hummingbird is to the left of the hawk.
the left of the crow. The quail
(A) The owl is the second from the left
(B) The crow is the second from the left
(C) The falcon is second from left
(D) The cardinal is the second from the left
(E) The hummingbird is the second from the left
(F) The quail is the second from the left
(G) The hawk is the second from the left
Target: F
The following paragraphs each describe a set of seven objects arranged in a fixed order. The statements logically consistent within each paragraph. On branch, there seven birds: an owl, a crow, a falcon, a cardinal, a hummingbird, quail, and hawk. The falcon the left of the crow. The quail is to the right of the cardinal. The hummingbird is to the right of the quail. The falcon is the second from the right. The hummingbird is to the left of the hawk.
Input:
(A) The owl is the fourth from the left
(B) The crow is the fourth from the left
(C) The faleon is the fourth from the left
(D) The cardinal is the fourth from the left
(E) The hummingbird is the fourth from the left
(F) The quail is the fourth from the left
(G) The hawk is the fourth from the left
Target:
Figure 17: An example of the dynamic, uncached part of the input prompt for the BBH dataset, Logical Deduction subset. This portion of the prompt is updated for each down- stream test sample.
22

---

Published as a conference paper at COLM 2025
Given a source sentence written German and translation in English, your task is to determine the type of translation error that the translated sentence contains. You may only generate the option letter corresponding to your answer, from options A to F.
Instruction:
Input: The following translations from German to English contain particular enoor. That error will be one the following types: Named Entities: An entity (names, places, locations, etc.) is changed to different entity. Numerical Values: Numerical values (ordinals or cardinals), dates, and/or units changed. Modifiers Adjectives: The modifiers and adjectives pertaining to noun are changed. Negation or Antonyms: Introduce or remove a negation or change comparatives to their antonyms. Facts: Trivial factual errors not pertaining to the above classes introduced the translations. Dropped Content: A significant clause in the translation is removed. Please identify that error.
Source: Ruth Lynn Deech, Baroness Deech DBE ist eine britische Juristin und Hochschullehrerin, die seit 2005 als Life Peeress Mitglied House Lords ist.
Translation: Ruth Lynn Deech, Baroness Deech DBE a British lawyer and university lecturer who has been life peer of the House of Lords since 2015.
translation contains an error pertaining to
(A) Modifiers or Adjectives
(B) Numerical Values
(C) Negation or Antonyms
(D) Named Entities
(E) Dropped Content
(F) Facts
Target: B
Input: The following translations from German to English contain a particular error. That error will be one of the following types: Named Entities: An entity (names, places, locations, etc.) is changed to different entity, Numerical Values: Numerical values (ordinals cardinals), dates, and or units are changed. Modifiers or Adjectives: The modifiers and adjectives pertaining to noun are changed. Negation or Antonyms: Introduce or remove negation or change comparatives their antonyms. Facts: Trivial factual errors pertaining to the above classes are introduced translations. Dropped Content: A significant clause in the translation is removed. Please identify that error.
Source: Štramberk ist eine Stadt in Tschechien.
Translation: it a town in the Czech Republic.
The translation contains
pertaining to
(A) Modifiers or Adjectives
(B) Numerical Values
(C) Negation Antonyms
Named Entities
Dropped Content
Facts
Target: D
other (Random Demonstrations) or MeMeens--ellcteed Demonstrations)
Figure 18: An example of the fixed, cached part of the input prompt for the BBH dataset, Salient Translation subset. This prompt is followed by dynamic, uncached content, shown in Figure 19, form the complete many-shot input prompt.
23

---

Published as a conference paper at COLM 2025
previous (Similar Demonstrations)
Input: following translations from German to English contain a particular error. That error will be one of the following types: Named Entities: An entity (names, places, locations, etc.) is changed different entity. Numerical Values: Numerical values (ordinals cardinals), dates, and/ units are changed. Modifiers or Adjectives: The modifiers and adjectives pertaining to nown are changed. Negation or Antonyms: Introduce or remove negation or change comparatives their antonyms. Facts: Trivial factual errors pertaining the above classes are introduced in the translations. Dropped Content: A significant clause in the translation is
removed. Please identify that error.
Source: Ekkehard Drefke war deutscher Künstler.
Translation: Eduard Drefke German artist.
The translation contains error pertaining to
(A) Modifiers or Adjectives
(B) Numerical Values
Negation Antonyms
(D) Named Entities
Dropped Content
(F) Facts
Target: D
The following translations from German to English contain particular That will one of the following types: Named Entities: An entity (names, places, locations, etc.) is changed to different entity, Numerical Values: Numerical values (ordinals or cardinals), dates, and / units changed. Modifiers Adjectives: The modifiers and adjectives pertaining to noun are changed. Negation or Antonyms: Introduce or remove a negation or change comparatives to their antonyms. Facts: Trivial factual errors not pertaining to the above classes
Input:
are introduced the translations. Dropped Content: A significant clause in the translation is removed. Please identify that error.
Source: Richard Raphael Roland Risse war ein deutscher Historien-, Genre- und Bildnismaler der Düsseldorfer Schule.
Translation: Risse was German historical, genre and portrait painter of the Düsseldorf School.
The translation contains an error pertaining to
(A) Modifiers Adjectives
(B) Numerical Values
(C) Negation or Antonyms
(D) Named Entities
Dropped Content
(F) Facts
Target:
Figure 19: example the dynamic, uncached part of the input prompt for the BBH dataset, Salient Translation subset. This portion of the prompt is updated for each down- stream test sample.
24


Reviews:
[{"review_id": "dvIhM0vNMN", "reviewer": "Reviewer_KMMK", "summary": "The authors address the high inference cost of many-shot in-context learning (ICL) when long-context LLMs ingest hundreds of demonstrations. The key idea is to keep most of the prompt cacheable while still tailoring a small slice to each test instance:\n- Hybrid Similarity + Random: For an n-shot prompt, pick s (e.g., 20) demonstrations most similar to the test query and prepend them to r = n – s randomly chosen demos that remain fixed across all queries and can therefore be cached.\n- Hybrid Similarity + k-Means: Replace the random cache with k demonstrations chosen via k-means over test-query embeddings, aiming for higher diversity.\n\nExperiments on Gemini Pro and Gemini Flash across four datasets (ANLI-R3, TREC-50, GSM Plus, MetaTool) show that both hybrids consistently match—or surpass—full per-query similarity selection while cutting the quadratic attention cost to near-linear (thanks to caching). Additional tests on three BBH tasks demonstrate benefits in low-data regimes.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Practicality & Simplicity – The strategies are easy to implement and integrate smoothly with existing key–value caching.\n- Clear Cost Analysis – The paper formalises proportional cost and backs it with empirical timing.", "reasons_to_reject": "- Limited Model and Task Generalization: The evaluation is restricted to two models from the Gemini family and four datasets. This narrow scope raises concerns about the generalizability of the findings to other LLM architectures (e.g., LLaMA) and task types, particularly domain-specific or diverse NLP benchmarks.\n- Incomplete and Inconclusive Experimental Design: Key baselines are missing (e.g., pure k-means), and no variance is reported for stochastic methods such as random selection. These omissions hinder a reliable assessment of the method’s effectiveness and statistical significance.\n- Inconsistent Claims and Observations: Some claims in the paper are contradicted by the results—for example, the assertion that similarity-based selection approximates random selection conflicts with large accuracy differences seen in certain datasets (e.g., TREC). This undermines the internal coherence of the study.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DtmhhHvlYe", "reviewer": "Reviewer_MXr4", "summary": "This study proposed a new method for few shot exemplar selection under in-context learning. The method contains two sub-approaches - the first one compare the similarity between the exemplars to the ones in the test set utilizing embeddings; the second one applying centroids derived from test sample representations via k-means clustering for exemplar selection.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Developing new methods for exemplar selection is an important topic for prompt engineering. Effective approaches could potentially benefit downstream applications. The study proposed a straightforward approach for selecting prompts. Results on two LLMs also show performance improvement compared to the baseline methods for exemplar selection.", "reasons_to_reject": "1. The study evaluated the methods on two models - Gemini Pro and Flash to further test the generalizability of the method. It would be helpful if more open-source LLMs could be tested on the datasets.\n\n2. Besides the testing of the current 4 datasets, it might be helpful if more datasets, especially the domain-specific ones, could be included for further experiments.", "questions_to_authors": "Please see the above section for questions and comments.", "extra_scores": {}}, {"review_id": "TOBaapAxVZ", "reviewer": "Reviewer_wBuF", "summary": "The paper proposes a hybrid demonstration‑selection strategy for many‑shot in‑context learning that keeps most of the prompt cacheable while inserting a small block of query‑similar examples.\n\nThe manuscript is easy to follow and well‑structured. Experiments on four NLP tasks show promising accuracy–cost trade‑offs. Nevertheless, the significance of the findings is limited by evidence drawn from only two Gemini models and by inconsistent trends across datasets. Also, the evaluation lacks (a) variance reporting for the random‑selection baseline and (b) inclusion of a pure k‑means baseline, both of which curb statistical and methodological robustness. \n\n**Minor comments**\n\n- When the entire prompt is cached (i.e. random selection), the complexity of  attention calculation on the demonstration segment should reduce to O(1), yet Figure 2 depicts a different scaling; moreover, Fig. 2’s X‑axis metric and Fig. 4’s Y‑axis metric appear inconsistent.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Simple yet effective framework: Presents a computationally efficient many‑shot ICL method that leverages the LLM key–value cache; extending cached random (or k‑means‑selected) demonstrations with a small set of query‑similar examples is a natural and practical step, and the insights that can inspire further research.", "reasons_to_reject": "- Limited generalizability of main claims\n    - Experiments evaluate only two models from the same Gemini family, so the claimed compute–accuracy benefits may not hold for other LLM architectures or vendors.\n    - The main claim appears to hold only for TREC and GSM‑Plus; MetaTool and ANLI exhibit a different trend, so generalization across tasks is not clear.\n- Insufficient experiments\n    - The random‑selection baseline reports no measure of variance (e.g., standard deviation across multiple random seeds), making it hard to judge statistical reliability.\n    - A pure k‑means selection baseline is absent, so the incremental benefit of combining k‑means with similarity cannot be isolated.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aLAuHc1eOi", "reviewer": "Reviewer_nzUp", "summary": "This paper incorporates caching into many-shot ICL, greatly shortening the inference cost. It is based on an observed phenomenon that example selection becomes less beneficial as the number of examples increases. Then, it fixes the majority of selected examples because they can be cached. The experiments tested on Gemini Pro and Flash show this method performs on par with or outperforms random and similarity-based baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper leverages an interesting fact that example selection becomes less important as the number goes up. The proposed performs as expected, reducing inference cost greatly.\n- The whole method and its implementations are easy to understand.\n- It conducts comprehensive experiments to demonstrate the effectiveness of proposed methods.", "reasons_to_reject": "- The tested models are both from the Gemini family, but it doesn't show any results on other LLMs, especially the open-source models like llama.\n- Some experiment results seem to contradict one of the motivations. In Line 84-87, it states that similarity-based selection effectively becomes equivalent to random selection. However, in Figure 4, the performance gap between these two methods can be as large as 10%+ accuracy (see TREC dataset). The statement in Line 230-232 becomes \"performance for both models remains almost unchanged beyond the 50-shot setting\".", "questions_to_authors": "- In Figure 5, why does hybrid similarity-random outperform hybrid similarity-k-means most of the time? Intuitively, involving k-means should be no worse than random selection, but your experiments give a reverse conclusion.\n- How do you measure Inference cost in Figure 4? Since there are two models tested, why is there only one line for each method?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper provides solid empirical grounding for a selection of clustering-based methods for choosing ICL examples. This is an important, impactful, and poorly understood area. While there are limited theoretical contributions, these practical results provide a good basis for further analysis.

The authors provide extensive responses in discussion that are generally acceptable to reviewers, who show an admirable willingness to be persuaded. These set a solid agenda for a revised final version.
Decision: Accept