Many-Shot In-Context Learning
Authors: Rishabh Agarwal
Avi Singh
Lei M Zhang
Bernd Bohnet
Luis Rosias
Stephanie C.Y. Chan
Biao Zhang
Ankesh Anand
Zaheer Abbas
Azade Nova
John D Co-Reyes
Eric Chu
Feryal Behbahani
Aleksandra Faust
Hugo Larochelle
Venue: NeurIPS 2024 spotlight (2024)
Keywords: large language models
in-context learning
long-context models
Primary area: natural_language_processing
TL;DR: We investigate the many-shot in-context learning regime -- prompting large language models with hundreds or thousands of examples -- for a wide range of tasks.

Abstract: Large language models (LLMs) excel at few-shot in-context learning (ICL) -- learning from a few examples provided in context at inference, without any weight updates. Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples – the many-shot regime. Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks. While promising, many-shot ICL can be bottlenecked by the available amount of human-generated outputs. To mitigate this limitation, we explore two new settings: (1) "Reinforced ICL" that uses model-generated chain-of-thought rationales in place of human rationales, and (2) "Unsupervised ICL" where we remove rationales from the prompt altogether, and prompts the model only with domain-specific inputs. We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks. We demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases, can learn high-dimensional functions with numerical inputs, and performs comparably to supervised fine-tuning. Finally, we reveal the limitations of next-token prediction loss as an indicator of downstream ICL performance.

Many-Shot In-Context Learning
Rishabh Agarwal, Avi Singh", Lei Zhang', Bernd Bohnet", Luis Rosias", Stephanie Chan', Biao Zhangi, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, Hugo Larochelle Google DeepMind
Abstract
LLMs excel few-shot in-context learning (ICL) learning from a few input- output examples ("shots") provided in context at inference, without any weight updates. Newly expanded context windows allow us investigate ICL with hundreds or thousands of examples the many-shot regime. Going from few-shot many-shot, we observe significant performance gains across wide variety of generative and discriminative tasks. While promising, many-shot ICI. can be bottlenecked by the available amount of numan-generatee outputs. To mitigate this limitation, we explore two settings: (1) "Reinforced ICI." that uses model-generated chain-of-thoughh rationales in place of human rationales, and (2) 'Unsupervised ICL" where remove rationales altogether, and prompt the model only with domain-speciiee inputs. We find that both Reinforced and Unsupervised ICL can be effective in the many-shot regime, particularly on complex reasoning tasks. Furthermore, we demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases, can learn high-dimensional functions with numerical inputs, and performs comparably to fine-tuning. Finally, we reveal the limitations of next-token prediction loss an indicator of ICL performance.
Introduction
A limiting factor for in-context learning (ICL) in LLMs is the context window, restricting prior research to the few-shol ICL regime. Many-shol learning - ICL with a large number of shots, for example, hundreds or thousands allows for better task specification, can reduce the need for fine- tuning, and potentially make LLMs more versatile and adaptable. Exploring many-shot ICL is now feasible, given the recent increase in context windows of publicly available LLMs by at least 100x: from only few thousand tokens in GPT-3 [8] and Llama 2 [56] to 1M tokens in Gemini 1.5 Pro [15].
In this paper, investigate how scaling the number of shots affects ICL performance ety of tasks ($2): problem solving using MATH [22] and GSM8K [10], question-answeriing [GPQA, 51], summarization using XSum [42] and XLSum [19], algorithmic reasoning [BBH, 55]. reward modeling [Code verification, $A.5), low-resource machine translation [FLORES, 17), planning (Lo- gistics, 53], and sentiment analysis [FP, 39]. Compared to few-shot ICL., many-shot learning performs significant better across these tasks, using several hundreds or thousands of shots (Figure 1). Further- more, maximum performance is often achieved only once the number of shots reaches up to hundreds of thousands of tokens (Figure I. Concurrent to our work, recent works explore many-shot ICL to jailbreak LLMs [2] (up 256 shots) and tackle NLP classification tasks [6] (up to 80K tokens). In our work, we focus on much wider range of tasks, and use a lot more examples (up to 8192 shots), much longer context lengths (up to 1M tokens). See $A.3 for detailed discussion of related work.
wide vari-
While shoy-shot ICL holds significant promise, it can be constrained by the need for high-quality, human-generated outputs. To overcome this, introduce reinforced ICL and unsupervised ICL ($3).
Equal contribution, Coore contribution. Correspondence to (rishabhagarwal,, singhavi) gogollc.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

100
Few-Shot ICL
Many-Shot ICL
+18.2
+5.0
+10.9
(Gemini 1.5 Pro)
80
Task Perfomance
60
+7.9
40
20
500
+15.0
600-
shot
+36.4
8152-mha
+9.2
125-4P-
10-shot
+15.3
125-shot
shot
5i2-chot
shot
128-4hot
50-4hat
shot
32-hot
2040-shat
Summarization
Planning
(Logistics)
Sequential Parity
(20 digiss)
GPOA
Warslation
Probiem-sohvitt (MATH)
Clsssifcation
(64 dim)
Cede Vertfaar
(Best (Best-af-Ht -ff----f-i)
Big Bench Hard tasks)
GSMak (Transter)
Sentiment
Analysis (FP)
Figure 1: Mhny-shot vs Few-Shot In-Context I Learning (ICL) across several tasks. Many-shot ICL consistently outperforms few-shot ICL, particularly on difficult non-natural language tasks. Optimal number of shots for many-shot ICI are shown inside the bar for each task. For few-shot ICI, we either use typical number of shots used benchmark, for example, 4-shot for MATH, or the longest prompt among the ones tested with less than the GPT-3 context length of 2048 tokens. Reasoning orienttdd tasks, namely MATH, GSM8K. BBH, and GPQA use thain-of-thoughe rationales. For translation, we report performance on English to Bemba, summarization uses XLSum, MATH corresponds to the MATH500 test set, and sentiment analysis results are reported with semantically caanntiically-uelaateee labels. See $2, $3, and $4 for more details.
Inspired by the efficacy of model-generated solutions for fine-tuning [54], Reinforced ICL involves replacing human-writee rationales with model-genet generated ones, filtered via answer correctness, for in-context learning. Inspired by task-recognition view of ICL [63], we also introduce Unsuper- vised ICL where we prompt the model with only problems instead of mroblem-solution pairs. On problem-solving tasks such as MATH, GPQA and Big- Big-Bench Hard, we find that both reinforced and unsupervised ICL with many-shot can be more effective than few-shot ICL with numan-generater rationales, with reinforced ICL being more broadly effective.
Finally, we empirically study how the learning dynamics of in-context learning changes from few-shot the many-shov regime ($4). We find that with sufficient examples, ICL can overcome pre-trane pre-training biases, perform comparably to full fine-tume fine-tuning, and solve high-dimensions prediction tasks with numerical inputs, namely sequential parity prediction and linear classification. This suggests the potential of many-shot ICI. to adapt to unseen tasks and domains that might be misaligned with an LLM's training data. Surprisingly, the order of examples can influence many-shot performance ($A.6) Finally, we demonstrate that long-context scaling laws [2, 65, 26) based on next-token prediction loss may not reliably predict ICL performance on problem-solee solvvng and reasoning tasks.
Our key contributions are as follows:
Scaling ICL ($2): We systematically evaluate ICL performance different scales of in-context examples for a wide range of tasks with Gemini 1.5 Pro. Our results indicate large performance jumps when transitioning from few-shot to many-shot regime.
Reinforced and Unsupervised ICL ($3): We find that using model-generatee rationales or only problems can reduce the dependence of many-shot ICL on human- -generatd data.
Analysing ICL ($4): We show that many-sho ICI. can overcome pre-training biases, perform comparably to fine-tuning, and learn non-NLP prediction tasks, where few-shot ICL struggles. We also reveal that next-token prediction loss may not be a good predictor of ICL performance.
2 Scaling In-Context Learning
During in-context learning (ICL), the LLM receives a prompt containing set of input-output examples, also called shots, that illustrate the desired task. At the end of the prompt, we append test input and allow the LM to make a prediction just by conditioning on the prompt and predicting the next tokens auto-regressively. Recent increase in context windows of LLMs allow using many more shots for ICL than typically used. To this end, we evaluate the many-shot performance of Gemini 1.5 2ro [15] model with I million token context length, the largest publicly available so
Thii corresponds to original version in the Gemini 1.5 Tech Report, released in February 2024, We note that the Gemini 1.5 Pro API now serves newer version starting from April 2024.

---

Many-shot ICL: Machine Translation
Many-shot ICL: Summarization
English - Bemba
45
English Kurdish
ROUGE-L
30
(hre Sandd XLBuml
Test chrF2
40
Google Tranzate SOTA)
35
MLLB (SOTA)
20
GEM-XSum
XLSum (Transfer)
30
2
2
2
2
24
25
2
2
2
2
2 Number Shots (K)
2'
2
Number Shots (K)
Figure Machine Translation (MT). Test Perfor- mance improves almost monotonically as we increase the number of MT pains provided as in-context exam- ples during inference. Notably, many-shot ICI. outper- forms state-of-the-ard chRF2++ scores of 35% (NLLBB on Bemha and 40% (Google Translate) on Kurdish [52]. We note that 997-shot prompt corresponds to around 85K tokens. See example prompt in Figure A.7.
Figure Summarization. As increase the num- her of shots from XSum dev set, XSum test perfor- mance improves up to 50 shots and then deteriorates. In contrast, XLSum performance typically improves with more shots from XSum. The 500-shot prompt corre- sponds 205K tokens. PEGASUS [68] and mT5 [19] are specialized models fine-tumed for summarization. See an example prompt in Figure A.8.
far. Unless specified otherwise, we use greedy decoding. For reliable results, we randomly sample in-context examples for each K-shot prompt multiple times using different random seeds and report average performance, along with some visualization for performance on individual seeds. To ensure that using more shots provides additional information, any K-sho prompt in our setup includes all in-context examples from prompts with less than examples. To reduce the inference cost, we use KV caching [48]. Now, we study many-shot ICL on typical LLM use-cases (also see $A.5).
2.1 Machine Translation
We consider translation from English to a low-resource target language, where many-shot ICL can complement the existing knowledge within the LLM. We use the target languages with the largest gap reported between LLMs and state-of offte-of-theaarr art systems [52], namely Bemba and Kurdish, from FLORES-200 200 benchmark [44]. We modify the default 1-shot MT prompt from Gemini Team [14] to include multiple translation pairs as shots from the FLORES dev split (containing 997 examples). We evaluate performance on the first 150 sentences from the test set using chrF2++ [49], a standard metric based on character and word n-gram overlap between generated and reference translation.
See Figure for results. Similar to Robinson et al. [52], we observed small gains in the few-shot regime from 1-shot 10-shot, particularly on Kurdish. However, when using the entire dev set for many-shot ICL, we observe improvements of 15.3% on Bemba and 4.5% on Kurdish, relative to the 1-shot Gemini prompt. Overall, these results establish the new-state-of-art for these language pairs.
2.2 Abstractive Summarization
To investigate how scaling ICL examples can impact the comprehension ability of LLMs, we now consider abstractive news summarization using XSum dataset from the GEM benchmark [1]. Using XSum dev set examples containing news articles and summaries, we also evaluate how many-shot ICL generalizes to XLSum [19]. We report performance on 150 test articles using ROUGE-L [34), which measures the longest common subsequence between reference and generated summaries.
As depicted in Figure 3, peak performance with many-shot ICI. is remarkably close to specialized models fine-tuned on XSum and XI.Sum. However, XSum performance declines with more than 50 in-context examples. Surprisingly, we observed the many-shot prompted model occasionally generating summaries with fabricated dates and times ($A.4), despite the absence of such data in the in-context summaries. Nonetheless, performance on XLSum monotonically improves with more shots, demonstrating positive transfer from many-shot learning to a related task.
2.3 Planning: Logistics Domain
Recent work has highlighted shortcomings in planning abilities of LLMs [58]. To this end, evaluate whether many-shot ICL can improve their ability to generate simple plans the
3

---

Logistics domain, widely used benchmark. The objective in this domain is transport packages within cities via trucks, and be- tween cities via airplanes. We generate set of planning problems with 2-3 cities, 1- 2 packages, truck and airplane per city using formal planning language (PDDL) generator [53], resulting 1.3K problems for learning and 600 for evaluation. To com-
Planning: Logistics Domain (600 insta nces)
60%
Success Rate
40%
20%
Gemini 1.5 Pro
Gemini 1.5 Pro (Apr 2024)
GPT 4 (Valmeekam et. al, 2024)
pute optimal solutions for each problem, we use the mest-Dowwward planner [20]. See Figure A,9 for an example 1-shot prompt.
0%
2 2' 2 2
Number of Shots (K)
As shown in Figure 4, we observe significant improvement in success rate with increas- ing numbers of ICL shots. While far from state-of-the-art planning approaches (e.g., Fast- hast-Dowwward), our results demonstrate the potential of many-shot ICL to improve commonsense planning abilities of LLMs.
Figure 4: In-context Plamning, A recent version of 1.5 Pro starts from a high few-shot performance, and its many-shot performance scales uniformly from 42% to 62%. For an older version, success rate quickly improves with up 10 shots (37K tokens), followed saturation. As reference, we report 1-shot GPT-4 results from Valmeekam [58].
3 Many-shot Learning without
Rationales
Many-shot ICL could potentially be limited by the availability of high-quality numan-generann rationales demonstrations. This is partieularly challenging for complex reasoning tasks, such as GPQA [51], where human-generated rationales require significant resources and expert knowledge. In this work, explore two simple approaches for addressing this issue.
Reinforced ICL Recent work [54] proposed a simplified version of Reinforced Self- Training [18], demonstrating that fine-tuning using model- rationales can be more effective than human- generated rationales to problem-solving tasks. Inspired by their work, we introduce Reinforced ICL, where we use model-generated rationales for in-context learning. To do so, we use few-shot or zero-shot chain-of-thouge problem. Then, we select rationales that obtain the correct final answer (we assume access ground truth final answers), and arrange them into in-context examples containing (problem, rationale) pairs.
prompt as a starting point to sample multiple rationales for each training
One potential issue with model-generated rationales is that of false positives: it is possible for an incorrect reasoning chain to lead to the correct final answer, and fine-tuning or prompting using such reasoning chain would typically harm performance. Nevertheless, we often find model- sener-eeee rationales to be at least as effective human-writ written rationales.
discuss in later sections,
Unsupervised ICL We now go step further than Reinforced ICL: what if we removed rationales from the many-shot prompt altogether, and prompt the model only with inputs? Specifically, the Unsupervised ICL prompt consists of: 1) preamble, such as, "You will be provided questions similar to the ones below:", 2) list of unsolved inputs or problems, and 3) a zero-shot instruction or few-shot prompt with outputs for the desired output format. See $A.9 for the exact prompts
One hypothesis for how shot unsupervised ICL might surpass few-shot learning with human demonstrations is that, when the LLM already possesses the required knowledge to solve a task, any information inserted in the prompt that can narrow down what knowledge is needed for the task becomes helpful. This would be consistent with the view that ICL simply "locates" latent concepts (e.g., math (roblem-solving) the LLM acquired during pre-training [63, 21, 60]. As such, any of the prompt components inputs, outputs, and their mapping can help locate such concepts. While Unsupervised ICL is broadly applicable, it may not perform well, for example, when outputs are critical for specifying the task (Figure 7 and A.17).
3.1 Problem-solving: Hendrycks MATH & GSM8K
We evaluate Reinforced and Unsupervised ICL on Hendryeks MATH [22], which consists of challeng- ing high school competition-level mathematics problems. We use the MATH500 test set from Light-

---

4-shot InnerMono. MATH Prompt
ICL (Ground-Truth)
Unsupervised ICL
Reinforced ICL
MATH500
GSMBK (Transfer using MATH prompts)
60.0%
95.0%
Test Accuracy (%)
57.5%
90.0%
55.0%
85.0%
52.5%
80.0%
50.0%
75.0%
47.5%
70.0%
10
25
50
125
250
500
10
25
125
250
500
Number of Shots (K)
Number of Shots (K)
Figure 5: Many-shot Reinforced and Unsupervised ICL for problem-solving generally outperform ICL. with ground-truth MATH solutions. MATH. (I.eft) The har plots depict the average performance across five random seeds on the MATH500 test set. Each random seed (denoted by the dots) corresponds to different subset of problems along with ground truth or model-generated solutions (if any) in the prompt. Transfer to GSM8K. (Right) We see that the prompt obtained from MATH transfers well to the GSM8K test split containing 500 problems. Our results with many-shot ICL outperform the 4-shot Minerva prompt, which obtains test accuracy of 55.7% on MATH500 and 90.6% on GSM8K.
man et al. [32] to report performance, and our 4-shot MATH prompt for data generation can be found in Figure 1. For Unsupervised ICL, we append this 4-shot prompt after the unsolved problems (see Figure A.) For comparison, also evaluate ICL with numan-written solutions (ground-truth) from the MATH training set, with the same problems used for many-shot prompts.
Our results are shown in the Figure (left). On MATH500, both Reinforced and Unsupervised ICL outperforms ICL. with ground-truth solutions in both the few-shot and many-shot regime. For ICL, we observe that the performance improves with more examples in the prompt up to point, and then declines (with the peak being at about 125 examples). Performance for Reinforced ICL also improves with the number of examples, and reaches a plateau at around 25 examples (while being about 5% higher than ICL), and unlike ICL, we don't see a significant drop in performance even for very large number of examples in the context. Notably, many-shot ICL achieves comparable or superior performance when using only problems compared to using problems with solutions. This suggests solutions may be redundant for eliciting robleng via in in-context learning on this domain, potentially due to extensive math-related data seen during pretraining.
Can many-shot ICL enable out-of-distributie generalizatinn? Singh et al. [54] found that fine-tuning a model on model-generated solutions from MATH resulted improved test performance on GSM8K [10], which has different distribution of problems than MATH. Here, whether many-shot ICL also improves transfer performance on GSMSK. indicating an improvement in general problem-solving abilitics from in-context learning. Our results in Figure 5 (right) show that this is indeed the case Reinforced ICL with MATH prompts excels on GSM8K, outperforming ICL with ground truth MATH solutions well Unsupervised ICL in the many-shot setting with at least 25 shots. This indicates that model-generated solutions can enable better generalization than just using problems combining them with ground-trute truth solutions for ICL.
investigate
3.2 Question Answering: Google-Proof QA (GPQA)
GPQA [51] is a multiple-choice QA benchmark, with difficult questions focused on graduate-level reasoning in biology, physics, and chemistry. Following Claude-3 [3], we use the diamond split (198 problems) for evaluation. This split focuses on questions where domain experts agree but experts in other domains struggle despite extended effort and internet access. Remaining 250 questions rationales. For
in non-diamond split are used for many-shot ICL with and without aero-shot prompt (Figure A.) to generate multiple rationales the
Reinforced ICL, we non-diamond split, solving 129 problems. We also append this zero-shot prompt after the GPQA problems for specifying output format for Unsupervised ICL.
As shown in Figure 6, average test accuracy with ground-truth rationales improves substantially from 5 shots to 125 shots, with the best-performing 125-shot prompt nearly matching the accuracy of
5

---

Google-Pros Proof QA (GPQA): Diamond Split
AestAccuracy (96)
45.0%
0-Shot Prompt ICL (Ground-Trutie -Tuth))
40.0%
Unsupervised ICL
Reinforced ICL
35.0%
25 Number of Shots (K)
125
Figure 6: Many-shot Reinforced and Unsupervised ICL for GPQA. The baseline zero-shot prompt, which used for generating rationales for Reinforced ICL and appended to the prompt for Unsupervised ICL, obtains performance of 38.8%. The average test accuracy with 125-shot prompt with both ground-truth model- generated rationales surpass the 40.4% obtained by Claude-3 Sonnet. As vary the number of shots, while Unsupervised ICL matches or outperforms the zero-shot prompt, Reinforced ICL consistently outperforms it.
the state-of-the-art Claude-3 Opus. However, we do observe performance degradation with 250 shots. Moreover, Reinforced ICL results indicate that model-generated rationales on GPQA seem to be better than ground-trun rationales up to 25 shots, while resulting in similar performance with more shots. Additionally, Unsupervised ICL does not follow any systematic trend: sometimes performs better ICL with ground-truth rationales depending on the number of shots, but generally underperforms Reinforced ICL. As noted in Anthropic [3], GPQA is small evaluation dataset and has an inherent higher variance across different runs, which might explain the non-systematic trends.
3.3
Algorithmic and Symbolic Reasoning: Big-Bench Hard
Multistep Arithmetic Two
Logical Deduction [Seven]
Geometric Shapes
Salient Translation Error Detection
Performance
0.95
0.6
0.8
0.90
0.4
0.05
0.2
0.6
3
10
100
25
50
100
Object
Reasoning About Colored Objects
Dyck Languages
Word Sort
Performance
1.00
1.00
0.75
0.95
0.9
0.90
0.5
10
25 50
100
Number Shots
Shots
Num ber of Shots
Number of Shots
Number of Shots
3-ahot CoT (tamanseritee)
Reinforced ICL
+ Unsupershed ICL
Figure 7: BIG-Bench Hard. Reinforced and Unsupervised ICL with varying number of shots, averaged across five random seeds. We evaluate test performance held-out set of 100 problems. The error hars denote standard deviation. Reinforced ICL. outperforms Unsupervised ICL. for all tasks, which in outperforms the human- written cunttn-written chain-of thotht (CoT) prompt. Averaged across tasks, CoT prompting using human- written rationales gets success rate of 72.1%, Unsupervised ICL obtains 77.1% %, while Reinforced ICL gets 83%.
We now evaluate Reinforced ICL and Unsupervised ICL on BIG-Bench Hard [55], suite of challenging algorithmic reasoning tasks. To reduce the impact of false positives, we select tasks out of 23 in BIG-Bench Hard for which the likelihood of getting false positive is low: either the answer string is long, or the number of options for each question is large (at least 6). For Reinforeed ICL, use the standard 3-shot CoT prompt from Suzgun et al. [55] to sample 10 rationales per problem from training set of 150 problem at a temperature of 1.0. We filter the rationales based on final answer correctness and arrange them into prompts containing 3 to 100 (problem, rationale) pairs.
As shown in Figure 7, Reinforced ICL strongly outperforms Unsupervised ICL for almost all tasks, which in turn outperforms the standard 3-shot CoT prompt. Performance for Reinforced ICL generally improves monotonically with the number of prompts for out of 8 tasks. These results indicate the Reinforced ICL is a more robust technique than Unsupervised ICL, especially for tasks in which the demonstrations contain crucial information about the task. For a few tasks, Reinforced ICL outperforms the human-writee 3-shot prompt even in the 3-shot setting. This result suggests

---

that model-genn generated rationales can sometimes outperform human-written written rationales even when controlling for the amount of data, mirroring the results reported by Singh al. [54] for fine-tuning.
4
Analyzing Many-Shot ICL
4.1
Overcoming Pre-training Biases with Many-Shot ICL
100%
Sentiment Analysis (FP): Replacement Label
Confidence (Label Probabiltty)
1.0
Sentiment Analysis (FP): Replacement Labels
Test Accuracy (%)
80%
0.8
60%
Abstract labels
Abstract labels
Default labels
0.6
labels
Flipped labels
plppped labels
2
2
2
2
2
Number of Shots (K)
Number of Shots (K)
Figure 8: Overcoming Pre- Traininnn Bias with Many-Shot ICL. (Left) Many-shot ICL overcomes label flips: Test accuracy for sentiment analysis typically improves with more training shots. Flipped and abstract labels eventually approaching the performance default labels. (Right) Confidence shift in overcoming bias. For tlipped and abstract labels, model confidence in its predicted sentiment labels initially drops, then sharply increases with more training shots to similar value, suggesting a period of overcoming pre- pre-trinning bias.
While LLMs demonstrate in-context learning of novel tasks, Kossen et al. [29] suggest that ICL may have difficulty unlearning biases derived from pre-training data. Their experiments, however, focused mainly on few-shot ICL due LLM context length limitations. Here, we revisit their study using many-shot ICL on the Financial PhraseBank (FP) sentiment analysis dataset [39]. Like Kossen et al. [29], we study label relationships that affect pre-training biases:
Flipped Labels: Default labels are rotated, that is, ['negative', 'neutral', 'positive'] becomes ['neutral', 'positive', 'negative"]. This conflicts with sentiment biases LLM might have learned.
Abstract Labels: We use ['A', 'B', 'C"], removing any pre-existing sentiment association [61].
For ICL shots, sample examples from the validation set (with replaced labels) to cxhibit the input- label relationship and report the results in Figure 8. With few shots, test accuracy with replacement labels is much lower than with default labels. This suggests that with few-shot ICL, the model struggles to overcome its pre-existing biases from pre-training. However, as the number of shots flipped and abstract labels dramatically improves, approaching that of default labels. For default labels, confidence in predicted labels steadily increases with more shots, as shown in Figure 8 (right). In contrast, for flipped labels, confidence initially drops then sharply increases before reaching a plateau, suggesting period of overcoming pre-training bias.
increases, performance
We posit that the initial drop in performance and confidence in the few-shot regime may be attributed to the "early ascent" phenomenon [46, 35]: a small number of shots may lead to the retrieval of an incorrect skill, which eventually diminishes as task learning takes effeet in the many- regime. Overall, these results indicate that many-shot ICL can overcome pre-training biases.
4.2 Learning Non-Natural Language Tasks: High-Dimensional Functions
We now test many-shot ICL's ability to learn abstract mathematical functions with numerical inputs, which let us stress test its generality and applicability to possibly unseen tasks.
Binary Linear Classification in High Dimensions Following the setup from Wei et al. [61], we create datasets with N-dimensional inputs vectors and their binary class labels, where each dimension is a random integer in [1, 1000]. See more details in $A.12. While Wei et al. [61] used only 16 shots per class, we scale ICL up to 2048 shots per class. As shown in Figure 9, while 2048 shots per class perform best when N = 16, we observe slight accuracy decrease beyond 512 shots for higher values of N (Figure C, R). Moreover, many-shot ICL substantially outperforms random-chance accuracy and nearly matches the accuracy of strong baseline, namely k-nearest neighbors, indicating that many-shot ICI. can implement nearest-neighhour search over inputs. This reminiscent of induction heads that implement prefix matching over sequences [45], plausible mechanism for ICL abilities.
7

---

Gemini 1.5 Pro
k-Nearest Neighbors
Random
Classification (N 16)
Classification (N 32)
Classification 64)
90.0%
Test Test Accuracy (9)
70.0%
60.0%
50.0%
2
2
Number of Shats Por Class (K)
Number of Shots Par Class (K)
Number of Shots Per Class (K)
Figure 9: In-Context Classification. Test accuracy for 16, 32 and 64 dimensional lincar classification problems, averaged across 5 randomly-generated generated datasets with 25 points per class for each dataset (250 evaluation points total). As we increase the number of shots, the accuracy improves and approximately tracks the performance of the neirest- baseline trained from scratch the data. We use the default implementation of k-nearest neighbours (with = 5) from scikit-learn [47]. See Figure A.13 for an example prompt.
Parity Sequence: 20 Digits
Input: 0 1 10110101111 00000 0
(Exact Seq. Match)
Label: Odd Odd Even Odd Odd Odd Odd Even
Test Accuracy (%)
40.0%
Odd Even Even Even Even Even Odd Odd Odd
30.0%
Even Odd Even
20.0%
Gemini
Input: 01100110110011000000000000000011 Label:
10.0%
Random Chance
GPT-2 Med (Scratch): 20x data
0.0%
2
2
2
2
2' 2
2 2 2 2 2
Number of Shots (K)
Figure 10: Learning Sequential Parity Function In-context. We report test accuracy over 200 unseen inputs, averaged across seeds. Error hars denote standard of the mean. Task Prompt. (l.eft) Example prompt with input and output labels the 20-digit Sequential Parity Function. Test accuracy (Right) Many-shot ICL performance improves almost monotonically with the number of shots, surpassing performance of GPT-2 Medium sized transformer trained from scratch for forwand-backward pass per example 20x more data.
Sequential Parity Parity is a fundamental Boolean function that determines a binary input odd number of 1s. Despite their power, transformers trained specifically
sequence contains for in-context learning, struggle to learn the Parity function over 20-digit sequences [7]. In this work, evaluate how well many-shot ICL performs with pretrained LLM to learn the sequential parity function f(x) (x), f2(x).... fn (z)], where I E {0, 1}3² and fx(
+ Ø2 ... + 24 i n]. We report the results in Figure 10. We see consistent improvement test accuracy as increase the number of shots to 8192. Performance surpasses GPT-2 Medium sized transformer [50] trained from scratch on 20x more input-output examples (with no repeated examples; 13). This result indicates many shot ICL can implement computations analogous to gradient descent [59].
4.3 Many-Shot ICL vs. Supervised Fine- Minn-Tunng
Many-shot ICL could make task-
Base Model
Supervised
Many-Shot
specific fine-tuning less essential or, even unnecessary, allowing LLMs tackle a wider range of tasks without specialization. While supervised fine- tuning (SFT) is the dominant L.M paradigm when making use of hun-
English Bemba
English Kurdish
Test Test chrF2+++ (%)
40
43.8
47.7
41.2
43.8
42-7
193
395
283
28.3
20
dreds or thousands of examples, it is computationally expensive in terms of training. In contrast, many-shot ICL does not require any training, however it has larger inference cost, which can be substantially reduced with KV caching [48]. We compare many-shot ICL to SFT for machine translation. We run two sets of experiments: one using 250 examples, and another using the entire dev set (997 examples). Our results in Figure 11 show that SFT and ICL
250
997
250
997
Number of Examples
Number of Examples
Figure 11: SFT vs Many-Shot ICL. We plot mean performance across seeds. The standard deviation is between 0.1% to 0.5%. Base model corresponds 1-shot performance of Gemini 1.5 Pro.

---

performance is quite close for Bemba, while SFT has slight edge for Kurdish. Overall, these results demonstrate that many-shot ICL can be a viable alternative for SFT.
4.4 Computational Cost of Many-Shot ICL
While many-shot ICL increases low for quick prototyping and Moreover, being able to spend ad- ditional inference-time compute to obtain better performance is use-
inference experimentation using
computation time, it can just
al-
inference API.
Summarization (500-shot - 205K tokens)
Sequential Parity (8192-shot - 540K tokens)
Generation Runtime Seconds)
5.2
ful feature to have. With KV
2.6
caching enabled (default for long-
1.3
context servers), as shown in Fig-
ure 12, runtime increases linearly with a large number of shots, as opposed to quadratically for self-attention: dou- bling the number of shots nearly dou- bles the runtime. However, for small number of shots, runtime is nearly constant. When the number of gen- crated tokens is much smaller than many-shot prompts, each new token still linear, which explains our observed runtime for a large number of shots.
0.32
2' 2-2-
2 2'222 Number of Shots
Number of Shots
Figure 12: Per-Output runtime we increase shots, averaged across the test set and multiple seeds, on (left) summarization and (right) parity prediction. When computing the next token, we still have to attend to the fixed many-shot prompt, even if KV is cached.
4.5
Comparing Many-Shot Abilities of Frontier LLMs
Gemini 1.5 Pro
GPT-4-Turbo
Claude-3-Opus - Gemini 1.5 Flash
Translation: English + Bemba
Translation: English - Kurdish
Test chrF2++ (9%
Test chrF2++ (%)
40
40
30
35
2"
2 2'
2
2"
2
2'
2'
2
2'
2
2"
2
2'
2
2
2
2
2'
2
2"
2
Number of Shots (K)
Number of Shots (K)
Figure 13: Many-shot ICL with GPT-4-Turbo, Gemini 1.5 Flash, and Claude-3-Opus on translation (52.1).
The strong many-shot results with Gemini 1.5 Pro raises the question whether other long-context frontier I.MSs also benefit from many-shot ICI. To do so, we evaluate GPT-4-Turro (128K context length) and Claude-3-Opus [3] (200K context length) on the low-resource translation ($2.1) 1). For both of these models, many-shot ICL scales favorably Bemba but do not exhibit much improvement Kurdish. Notably, 1.5 Pro starts lower than Claude-3 on Bemba but improves more rapidly, achieving much higher performance at 997 shots. It also outperforms GPT-4 in few-shot learning and improves further with more examples. Overall, these results indicate that frontier LLMS exhibit varying degree of many-shot ICL capability, and even smaller LLMs can benefit from many-shot ICL and outperform LLMs with stronger few-shot performance with enough shots
To understand the role of model size, also evaluated the many shot performance of Gemini 1.5 Flash, a smaller long-context LLM than Gemini 1.5 Pro. On the English - Bemba task, we find that 1.5 Flash matches Claude-3- 3-Ouus and outperforms GPT-4 with 997-shots, despite having much worse few-shot performance than Claude and GPT. On English - Tamil MT, 1.5 Flash outperforms Claude-3 in terms of many-shot performance, while lags behind 1.5 Pro. These results suggest that even smaller LLMs can benefit from many-shot ICL and outperform LLMs with stronger few-shot performance with enough shots.
4.6 Long-context scaling laws may not predict ICL performance
Prior works [65, 2, 26] have found that the negative log-likelihood (NLL) for ground-trutt test outputs decreases predictably as the context length increases. We confirm this finding for GPQA, Hendrycks

---

Negative Log Likellhoodd on Ground-Truth Solutions
Negative Log-Likelinod
GPQA
MATH
GSMBK
0.70
1.3
1.2
0.60
0.4
1.1
10
25
50
125
250
A
10
25
50
125
250
500
10
25
50
125 250 500
Number of Shots (K)
Number of Shots (K)
Number of Shots (K)
Reinforced ICL
ICL (Ground-Truth)
Unsupervised ICL
Figure 14: Negative Log-Likelihood (NLL) as function of number of shots. We plot NLL ground truth test set solutions for GPQA, MATH and GSM8K. For GPQA and MATH, questions for Reinforced ICL and Unsupervised ICT. comes from the training splits of those datasets. We study GSM8K in the transfer setting, i.c. questions for Reinforced and Unsupervised ICL come from MATH. The absolute NLL for ICL and Reinforced ICL are not directly comparable to Unsupervised ICL, since they use different prompt formats.
MATH and GSM8K with many-shot ICL, and report our results in Figure 14. However, we note that NLL trends are not strong predictor for downstream task performance. For example, the success rate for both MATH and GPQA with ICL decreases after 125 shots (Figure 5,6), but we do observe corresponding increase in the NLL in Figure 14.
We also plot NLL curves for Reinforced and Unsupervised ICL, and find them to generally have smaller slope when compared to supervised ICL. Interestingly, NLL curves for ICL with ground- -ruth outputs is lower than with model-generated outputs, even though the latter often performs better. In the GSM8K transfer setting (using MATH problems and solutions to score GSM8K solutions), the change in NLL is close nil. However, this doesn't reflect transfer performance on GSM8K, which continues to improve with more examples (Figure 5).
Overall, NLL is not reliable proxy when attempting to predict ICL performance for mooblemm-olvingg domains. This makes intuitive sense: for any given problem, there are a large number of potentially correct CoT solutions that the model can generate, and calculating the log-likelihood such solution may not provide a clear picture for overall model capability. We also explore computing NLL on diverse of model gener-geeeraaed outputs MATH, and findings presented in $A.7.
only one
5 Discussion, Limitations and Future Work
We found significant gains in performance when going from few-shot to many-shot ICL wide range of tasks, including translation, summarization, planning, reward modeling, mathematical problem solving, quusttionansseriing algorithmic reasoning, and sentiment analysis. To overcome the challenges of obtaining a large number of high-quality human-written rationales for many-shot ICL, we introduced two regimes: Reinforced ICL and Unsupervised ICL. Moreover, demonstrate that, unlike few-shot ICL, many-shot ICL is effective at overriding pretraining biases, can learn high-dimensional functions with numerical inputs, and performs comparably to SFT.
One limitation of our work that it mainly evaluates many-shot ICL with Gemini 1.5 Pro. That said, concurrent works [2, 6) as well as our preliminary results with GPT-4-Turbo and Claude-3- Opus (Figure 13) indicate that other LLMs can also benefit from many-shot ICL. Future work should focus evaluating the many-shot abilities of wide range of long context models, as they become available. Furthermore, many-shot performance can likely serve as a valuable metric for evaluating the quality of long-context models, going beyond the prevalent geedle-in-a-hastaee test [25].
Another limitation of our work is that we don't 't't completely understand why performance can some- times degrades with more examples in the prompt (for example, for MATH). Our analysis found that negative log- likeliihd trends are insufficient to explain this degradation, and future work should investigate new directions to shed light on the matter and improving many-shot ICL capabilities. Overall, we hope that this work lays a foundation for understanding and optimizing the use of long-context models for ICL, opening up new frontier of LLM capabilities.
10

---

Contribution Statement
RA initiated the project, ran majority of the many-shot experiments and analysis, came up with reinforced ICL, on-boarded collaborators, handled the NeurlPS rebuttal, wrote the initial draft. AS contributed initial infra for experiments on MATH and GSM8K, ran BBH experiments, conducted NLL analysis problem-solving tasks, and wrote several sections of the paper.
I.Z. contributed results for in-context verifier. BB contributed the planning logistics task. AA helped with GPQA, SC contributed the baseline for parity task and both helped edit the paper. AF and HI provided feedback on an early draft. HI. also suggested the unsupervised ICL. experiments. Others were involved in project discussions and minor edits to the paper.
References
[1] N. Akter, Yu, A. Muhamed, T. Ou, A. Bäuerle, Á. A. Cabrera, K. Dholakia, C. Xiong, and G. Neubig, An in-depth look at gemini's language abilities. arXiv preprint arX1v:2312.11444, 2023.
[2] C. Anil, E. Durmus, M. Sharma, Benton, S. Kundu, Batson, N. Rimsky, Tong, Mu, D. Ford, F. Mosconi, R. Agrawal, R. Schaeffer, N. Bashkansky, S. Svenningsen, M. Lambert, A. Radhakrishnan, C. Denison, E. Hubinger, Y. Bai, Bricken, T. Maxwell, Schiefer, J. Sully, A. Tamkin, T. Lanham, K. Nguyen, T. Korbak, J. Kaplan, D. Ganguli, S. R. Bowman, E. Perez, R. Grosse, and D. Duvenaud. Many-shot jailbreaking. Technical report, Anthropic, 2024.
[3] Anthropic. The claude 3 model family: Opus, sonnet, haiku. Technical Report, 2024.
[4] A. Asai, S. Kudugunta, X. V. Yu, T. Blevins, H. Gonen, M. Reid, Y. Tsvetkov, S. Ruder, and H. Hajishirzi. Buffet: Benchmarking large language models for few-shot tross-lingual transfer. arXiv preprini arXiv:2305 /4857, 2023.
[5] A. Awadalla, M. Wortsman, G. Ilharco, S. Min, I. Magnusson, H. Hajishirzi, and L. Schmidt. Exploring the landscape of distributional robustness for question answering models. arXiv preprint arXiv:2210.12:: 125117 2022.
[6] A. Bertsch, M. Ivgi, U. Alon, Berant, M. Gormley, and G. Neubig. In-Context Learning with Lont-Context Models: An In-Depth Exploration, Apr. 2024. URL http:://at///i org/ abs/2405 00200. arXiv:2405.0022 00.000000000010 [cs].
[7] S. Bhattamishra, Patel, Blunsom, and V. Kanade. Understanding in-context learning in transformers and Ilms by learning to learn discrete functions. arXiv preprint 1r000:2310.03016, 2023.
[8] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020. URL https::///pppt //ttte://proccedinngg neurips httpscccpppppp//////// #457c0d6bfbb996 00t.00/00000..001100000001010000
1ww.118eaaaaae4eeeee 2atp:e444AAbbbbtte Abstract html.
[9] S. C. Y. Chan, I. Dasgupta, J. Kim, D. Kumaran, A. K. Lampinen, and F. Hill. Transformers generalize differently from information stored in context vs in weights, Oct. 2022. URL http://arxiv org aads/22000 05675.. urXiv:2210.0567. [cs].
[10] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint 210.:1110.. 2021.

---

[11] T. Dinh, Y. Zeng, R. Zhang, Z. Lin, M. Gira, S. Rajput, J.yy Sohn, D. Papailiopoulos and K. Lee. Lift: sanguage-interfaeer tin-ttnnngg for non-language machine learning tasks. Advances in Neural Information Processing Systems, 18:11763-1188 2022.
[12] Y. Fu, R. Panda, X. NIu, X. Yue, H. Hajishirzi, Y. Kim, and H. Peng. Data Engineering for Scaling Language Models to 128K Context, Feb. 2024. URL http://appat /ttp///axvvvv/./ wrw. org/abs/2402. 10171. arXiv:2402.1017 [cs].
[13] S. Garg, D. Tsipras, P. S. Liang, and G. Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35: 30583-30598, 2022.
[14] G. Gemini Team. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
[15] G. Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arxiv:2403.0 055300, 2024.
[16] M. Ghallab, Howe, C. Knoblock, D. Medermott, A. Ram, M. Veloso, D. Weld, and D. Wilkins. PDDL-The Planning Domain Definition Language, 1998.
[17] N. Goyal, C. Gao, V. Chaudhary, Chen, G. Wenzek, D. Ju, S. Krishnan, M. Ranzato, F. Guzmán, and A. Fan. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. Trans. Assoc. Comput. Linguistics, 10:522-538, 2022.
[18] C. Gulcehre, T. L. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, et al. Reinforced self-training (rest) for language modeling, arXiv 2023.
preprint
[19] T. Hasan, A. Bhattacharjee, M. S. Islam, K. S. Mubasshir, Y. Li, Y. Kang, M. S. Rahman, and R. Shahriyar. XI-sum: Large-seale multilingual abstractive summarization for 44 languages. In C. Zong, F. Xia, W. Li, and R. Navigli, editors, Findings of the Association for Computational Linguistics: ACL/JCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 4693-4703, Association for Computational Linguistics, 2021.
[20] M. Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26:191-246, July 2006. ISSN 1076-9757. doi: 10. 16.1613/jair. 1705. URL http: //xx. doi org/ 10. 10.1613/jair. 1705.
[21] R. Hendel, M. Geva, and A. Globerson. In-context learning creates task vectors. arXiv preprint arXiv:2310.15910 2023.
[22] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, Basart, Tang, D. Song, and J. Stein- hardt. Measuring mathematical problem solving with the math dataset. arXiv preprint vrxiv:2/03.03874, 2021.
[23] A. Hosseini, X. Yuan, N. Malkin, A. Courville, A. Sordoni, and R. Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06457, 2024.
[24] H. J. Jeon, J. D. Lee, Q. Lei, and B. Van Roy. An information-thhoretie analysis of in-context learning, arXiv preprint arXiv:2401.15530 2024.
[25] G. Kamradt. LLMTest NeedleInAHaystack. httpss///ttaaaa..oonnaa///e/lttt NeedleInAHHaystack 2023, Accessed: 2024-04-16.
[26] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint NXKv:2001.08366 2020.
[27] O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, Santhanam, S. A, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts. DSPy: Compiling declarative language model calls into state-of-the-an pipelines. In The Twelfth International Conference on Learning Representations, 2024. URL https: /pwpreveeeeeeeeee/////e/ee wwweeereveeennn/eooooeeeee wttp://eeorrmmmmm///////0022000
12

---

[28] H. J. Kim, H. Cho, J. Kim, T. Kim, K. M. Yoo. and S. Lee. Self- generatneed in-conter, learning: Leveraging auto-regressive language models as demonstration generator. CoRR. abs/2206.08082, 2022. doi: 1044550AARXIV22206000000 48550/as /00000//Xvv.... 2206.08020
URL. htttps///oi....
[29] J. Kossen, Y. Gal, and T. Rainforth. In-context learning learns label relationships but is not conventional learning. In The Twelfth International Conference on Learning Representations, 2023.
[30] M. Li, S. Gong. J. Feng, Y. Xu, J. Zhang, Z. Wu, and L. Kong. In-context learning with many demonstration examples. CoRR, abs/2302.04931, 2023. doi: 1044550/ARXIV2200004999 URL https:tt////o org/ 10. 200.0088550/arr0 arriv.. 2302. 04931.
[31] R. Li, G. Wang. and J. Li. Are human-generated demonstrations necessary for in-context learning? In The Twelfth International Conference on Learning Representations, 2024. URL https: sstps://openns //tp:///oepe/ee.///////e httpp//eviie.eeeefrrrrrrreeeeeee
[32] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, Schulman, I. Sutskever, and K. Cobbe. Let's verify step by step. CoRR, abs/2305.20050, 2023, doi: 10.4850/ARXI.22055220000
URL https://doi. org/10. 1t//:/1..555////////// aarvvv 2305 . 20050.
[33] B. Y. Lin, A. Ravichander, X. Lu, N. Dziri, M. Sclar, K. Chandu, C. Bhagavatula, and Y. Choi. The unlocking spell on base Ilms: Rethinking alignment via in-context learning. arXiv preprint arXiv:2312.01 552, 2023.
[34] C.-Y. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 71, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL httpps:/aaclanthologe .wg.lllggooooooooooooeoe org//04-1000
[35] Z. Lin and K. Lee. Dual operating modes of in-context learning. arXiv preprint arXiv:2402. 18819, 2024.
[36] H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel. Few-shot Darameter-efficieet fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1965 2022.
[37] Y. Lu, M. Bartolo, A. Moore. S. Riedel, and Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08766, 2021.
[38] Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In S. Muresan, Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, Association for Computational Linguistics, 2022. doi: 18.18653/V1/ 2022.ACL-LONG. URL sattps://dol org/10 10. 18653/v1/2022 acl-long 556.
2022, pages
[39] Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4):782-796 2014.
[40] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference Empirical Methods in Natural Language Prucessing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11048- 11064. Association for Computational Linguistics, 2022.
[41] M. Mosbach, T. Pimentel, S. Ravfogel, D. Klakow, and Y. Elazar. Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation. arXiv preprint (XX.::2333..6933 2023.
[42] S. Narayan, S. B. Cohen, and M. Lapata. Don't I give me the details, just the summary! topic- aware convolutional neural networks for extreme summarization. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference Empirical Methods Natural Language Processing, Brussels, Belgium, October 31- November 2018, pages 1897-1807. Association for Computational Linguistics, 2018.
13

---

[43] A. Ni, S. Iyer, D. Radev, V. Stoyanov, W.-t. Yih, S. Wang, and X. V. Lin. Lever: Learning to verify language-to-cor generation with execution. In International Conference on Machine Learning, pages 26106-26128. PMIR, 2023.
[44] M. A. NLLB Team. No language left behind: Scaling human-centered machine translation. arXiv preprint, 2022.
[45] C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Bai, A. Chen, Conerly, D. Drain, Ganguli, Z. Hatfield-Dodds, D. Hermandez, S. Johnston, A. Jones, J. Kermion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCan- dlish, and C. Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. htt.://mee-.cciiiissup/////////o
html.
[46] J. Pan. What in-context learning "learns" in-context: Disentangling task recognition and task learning. PhD thesis, Princeton University, 2023.
[47] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, al. Scikit-leam: Machine learning python. the Journal of machine Learning research, 12:2825-222 2830, 2011.
[48] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5, 2023.
[49] M. Popović. chrf++: words helping character n- n-grams, In Proceedings of the second conference on machine translation, pages 122-118,6666 2017.
[50] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[51] D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023.
[52] N. R. Robinson, P. Ogayo, R. Mortensen, and G. Neubig. Chatgpt mt: Competitive for high-(but not low-) resource languages. arXiv preprint 3rXtv:2309.07423,200 2023.
[53] J. Seipp, Á. Torralba, and J. Hoffmann. PDDL generators. fttps://doi.orge) 5281/ zenodo. 6382173, 2022.
[54] A. Singh, J. D. Co-Reyes, Agarwal, Anand, P. Patil, X. Garcia, P. J. Liu, Harrison,
J. Lee, K. Xu, A. T. Parisi, A. Kumar, A. A. Alemi, A. Rizkowsky, A. Nova, B. Adlam, B. Bohnet, G. Elsayed, H. Sedghi. I. Mordatch, Simpson, Gur, Snoek, J. Pennington,
J. Hron, K. Kenealy, K. Swersky, K. Mahajan, I.. A. Culp. I.. Xiao, M. Bileschi, N. Constant, R. Novak, Liu, T. Warkentin, Y. Bansal, E. Dyer, B. Neyshabur, Sohl-Dickstein, and N. Fiedel. Beyond human data: Scaling self-training for problem-solving with language models. Transactions on Machine Learning Research, 2024, ISSN 2835-8856. URL https: d-w.oruunnnnnaAAAnnnnA.o Expert Certification.
/ /openreview
[55] M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thougll can solve them. arXiv preprint arXiv:2210.09261. 2022.
[56) H. Touvron, Martin, K. Stone, Albert, A. Almahairi, Y. Babaei, Bashlykov, Batra, P. Bhargava, S. Bhosale, et al. Ilamm 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[57] R. Vacareanu, V.-A. Negru, V. Sueiu, and M. Surdeanu. From words to numbers: Your large language model is secretly capable regressor when given in-context examples. arXiv preprint arXiv:2404.07544, 2024.
[58] K. Valmeekam, M. Marquez, S. Sreedharan, and S. Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, 36, 2024.
14

---

[59] J. von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov. Transformers learn in-context by gradient descent, Dec. 2022. URL http://arxiv... wttp://wrrr/aabb//....ee. 07677. arXiv:2212.07677 [es].
[60] X. Wang. W. Zhu, M. Saxon, M. Steyvers, and W. Y. Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neural Information Processing Systems, 36, 2024.
[61] J. Wei, J. Wei, Y. Tay, D. Tran, A. Webson, Y. Lu, X. Chen, H. Liu, D. Huang. D. Zhou, et al. Larger language models do in-context learning differently. arXiv preprint 2rXiv:2303.03844 2023.
[62] Y. Xiang, H. Yan, L. Gui, and Y. He. Addressing order sensitivity of in-context demonstration examples in causal language models. arXiv preprint arXiv:2402.156637 2024.
[63] S. M. Xie, A. Raghunathan, P. Liang, and T. Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2 1.02080, 2021.
[64] Y. Xing, X. Lin, N. Suh, Q. Song, and G. Cheng. Benefits transformer: In-context learning in linear regression tasks with unstructured data. arXiv preprint arXIv:2402.00743, 2024.
[65] W. Xiong, J. Liu, Molybog, H. Zhang, P. Bhargava, R. Hou, L. Martin, R. Rungta, K. A. Sankararaman, B. Oguz, M. Khabsa, H. Fang, Y. Mehdad, S. Narang, K. Malik, A. Fan, S. Bhosale, S. Edunov, M. Lewis. S. Wang, and H. Ma. Effective long-context scaling of foundation models. CoRR, abs/2309. 16039, 2023. doi: 10.48550/ARXIV2330.166 URL https://doi.
org/10. 48550/ arxiv. 2309. 16039.
[66] K. M. Yoo, J. Kim, H. J. Kim, H. Cho, H. Jo, Lee, Lee, and T. Kim. Ground-truth labels matter: A deeper look into input-label demonstrations. In Y. Goldberg, Z. Kozareva, and Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 2422-2437. Association for Computational Linguistics, 2022.
[67] Z. Yuan, H. Yuan, C. Li, G. Dong, C. Tan, and C. Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.0182. 2023.
[68] Zhang, Y. Zhao, M. Saleh, and P. Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International conference on machine learning, pages 11328- 11339. PMLR, 2020.
[69] R. Zhang. S. Frei, and Bartlett, Trained transformers learn linear models in-context. arXiv preprint arXiv:2306.09927 2023.
[70] 7. Zhang, A. Zhang, M. Li, and A. Smola. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations, 2023.
URL https: http:///ppeeeeeeeeeee
[71] L. Zheng, J. Yuan, C. Wang, and L. Kong. Efficient attention via control variates. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL URL https: httphtts:// net/pdf ?id=--idllll MNNGGGMM44rr44
15

---

A
Appendix
A.1
Context Length for Many-shot ICL
Many-Shot ICL: Context Length versus Number of Shots
Context Length Tokens
x00
Pest-Performing Shots
(Gemini 1.5 Pro
x 10
Maximum Shots
S0UG 2016-2611 2618
20u5 002E 002t
1048 008
100
104 0007
20NF 668
2005 166
3 100
104-201
500
900
x0
125
1x10
Big.Banch Hard (8 tasks)
Sentiment
Analysis (FR)
Transtation
Code Verttir
GPOA
Problem-sotviiee (MATH)
GSMRK
(Trnnfer)
Summanzation
Sequential Parity (20 digits) planning
(Legistics)
Clossifiaation (64 dim)
Figure A.I: Context Length for best perforrnrnn and the maximum number of shots tested for cach task. The horizontal dashed line shows the context length of GPT-3 (2048 tokens), which is representative of typical few- shot prompts tested the LLM literature. For several tasks, we observed the best best- rest-eeforniig shots correspond the maximum number shots we tested, which was limited by the number of available examples for in-context learning. On some tasks (e.g., code verifier, planning), we did observe slight performance deterioration beyond certain number of shots.
A.2
Where Do Gains in Many-Shot ICL Stem From?
Distinct examples
Repeated examples (N 25 ex.)
Test chrF2++ (%)
(English - Bemba)
45
40
35
30
25
50
250 Number of Shots (K)
125
500
1000
Figure A.2: Many-shot performance with distinct examples vs repeating the same 25 examples times on low-resource MT. Bars show avg. perf with std across 3 seeds. Most of the benefit of many-shot ICL stems from adding new information as opposed to increasing context length.
Including more examples has effects: (1) increasing information when using distinct samples, (2) but another of increasing the context length. To separate these two effects, we ran an experiment on low-resource MT by repeating 25 examples several times create many-shot prompts with up 1000 examples (shuffled ordering) and added the results in Figure A.2. The performance with repeated examples stays nearly the same and significantly lags behind many-shot performance with distinct examples. On this task, the benefit of many-shot ICL mainly stems from adding new information as opposed to increasing context length
A.3 Related Work
Scaling in-context learning Brown al. [8] reported improved performance as you increase the number of examples (up to 64) for in-context learning in LLMs, and later works corroborated this
16

---

finding [38]. However, very few works have explored using large number of examples (1000 or above) in the prompt. This is likely due to the fact the context lengths in large language models have been quite limited until recently [15, 3]. One closely related work to ours is from Li et al. [30], who scale the number of examples for in-context learning to 2000. However, Li al. [30] use custom model architecture [71] to achieve long context lengths, and only evaluate models of up to 1.3B parameters, which is several orders of magnitude smaller than state-of-the-a language models, and are ineffective for complex tasks, such as GPQA [51].
Concurrently to our work, Anil et al. [2] used many-shot prompting (upto 256 shots) to jailbreak language models. In our work, we focus on a much wider range of tasks, use lot more examples (up to 8192 shots) and use models with much longer context lengths (up to 1M tokens). Also, we explore mitigations for needing many -eeteratd examples with many-shot ICL. Furthermore, while Anil et al. [2] use many-shot learning to override preferences learned during RLHF phase to elicit the biases stemming from pretraining, our results in $4.1 demonstrate that we can also override pre-training biases themselves. Bertsch et al. [6] also concurrently shows benefits of scaling up in-context learning to many demonstrations several classification datasets with up to 151 labels, albeit also using smaller context windows of up to 80k tokens (using Llama2-80k [12].
Long-conted congecontett scaling laws Prior works [65, 2, 26, 15] have reported smaller next-token prediction loss with longer contexts, which Jeon et al. [24] also show using theoretical analysis. Our findings confirm this trend for even longer context lengths, but our analysis reveals some of the limitations of using next-token prediction loss as a metric for evaluating long-context performance, as next-token prediction loss continues to go down even overall performance plateaus.
Learning from self-generated data Numerous recent works [18, 67, 54] propose fine-tuning language models on self-generated data to improve performance. Their approach consists of (1) generate samples from the model and filter them using binary feedback, (2) fine-tune the model on
these samples, and (3) repeat this process few times. In this work, we extend this idea to in-context learning, and study the efficacy of Reinforced ICL. in reasoning and problem-sol -olving domains.
Self- Self-generated data and in-context learning Kim et al. [28] propose using self- -eferaeee data for few-shot ICL on classification problems, where they generate demonstrations using the LLM conditioned on the test input for each possible class label, and including these demonstrations in the context when performing the final prediction. Li et al. [31] extend this approach to reasoning and language understanding tasks, where they also generate demonstrations conditioned on the test input. Consistent with our findings, these works show that model-generated demonstrations can outperform numan-generate demonstrations in the few-shot regime. Another related approach is AutoCoT [70] that uses a zero-shot CoT prompt to produce model -generree demonstrations for few-shot ICL. To do so, AutoCoT samples diverse questions one-by-one based embedding-basee clustering followed by heuristics- based post- postt-rocessing for selecting demonstrations.
Different from above approaches, Reinforced ICL generates demonstrations using the same procedure as Singh et al. [54], does not require clustering, post-processing heuristies, or access to the test inputs for generating demonstrations, and can be applied to any problem for which we can obtain reliable reward signals. Moreover, our work mainly focuses on the utility of randomly-sampled model-generated demonstrations for many-shot ICI..
Learning Input-Output Relationships with ICL Numerous works [40, 29, 66, 35] have inves- tigated whether LLMs truly learn onput-output relationships during in-context learning. Min et al. [40] found that replacing the ground truth labels in in-context examples with random labels barely effected final performance. Further investigations by Yoo et al. [66] and Kossen et al. [29] found that this finding does not necessarily hold across tasks and model sizes. In particular, Kossen et al. [29]. Lin and Lee [35] showed that L.MMs can indeed learn input-output relationships via in-context learning, but require more examples in order to do so well. In our work, we extrapolate the trend found in those works to much longer context lengths, showing that pre-training biases can be mostly overcome given enough training examples.
Learning Mathematical Functions with LLMs Several prior works investigate whether mathe- matical functions can be learned with transformers [13, 69, 64, 7]. All these works train transformers specifically to perform in in-context learning for such functions. In contrast, we demonstrate that
17

---

many-shot ICL can learn high- sith-dmenssionll functions even with pre- pre-trained LLMs. Concurrent to our work. Vacareanu et al. [57] demonstrate that pretrained LLMs are able to perform regression tasks, with performance rivaling that of traditional supervised methods with 500 in-context examples. Our work complement their findings to other synthetic tasks with a much larger number of in-context examples. Dinh et al. [11] fine-tuned GPT-3 on synthetic classification tasks and observed similarities in the decision boundaries learned by the fine-tuned model and kNNs. Our results in Figure 9 show that many-shot ICL also performs comparably to kNNs on high-dimensional classification tasks.
Comparing ICL with fine-tuning Contrary to task-specific fine-tuning, ICL does not require optimizing any model weights, allowing LLMs to perform a variety of tasks at inference. As such, several prior works compare fine-tuning with ICL but in few-shot regime. Liu et al. [36] proposed parameter-efficient few-shot fine-tuning (FT) approach for TO that outperforms few-shot ICL with GPT-3. However, Awadalla et al. [5] argue that few-shot ICL is more robust to distribution shifts than fine-tuning for question answering tasks. Similarly, Asai et al. [4] show better transfer with ICL compared to fine-tuning on some tasks. Mosbach et al. [41] fairly compare ICL with FT by using the same model for both approaches and show that full fine-tuning (FT) generally outperforms ICL in the few-shot regime with 16 examples. More recently, Lin al. [33] show that few-shot ICL can outperform fine-tuning based approaches for aligning LLMs.
Complementary to prior works, compare full fine-tuning with many-shot ICL with the same number of examples for low-resource translation. Notably, we find that many-shot ICL performs comparably to FT. Aligned with our findings, Bertsch et al. [6] concurrently show that many-shot ICL generally outperforms parameter-effiiien fine-tuning (LoRA) on classification tasks. Overall, many-shot ICL and FT can exhibit comparable behaviors, which we leave for further investigation.
Exemplar vs. Rule-based ICL generalization Chan et [9] indicate that ICL tends to generalize in more exemplar- exemplar-based way. compared to rule-based generalization during in-weights learning, Using a clever experiment with blocked attention, Bertsch et al. [6] also argue that the benefits of many in-context demonstrations arise from having access to more similar examples. While our results on in-context linear classification agree with this conclusion, our sequential parity results seem to contradict it. Strikingly, sequential parity was the task on which we saw the most improvement, whereas it should be a task that benefits least from seeing similar examples after all, the nearest neighbor is always going to give the wrong answer (off by I bit). Chan et al. [9] do show that transformer's 's inductive biases towards exemplar-based generalization can be shifted both by the training data and the model size, with larger models being less exemplar- -aasedrr---edd--- perhaps this explains the contradictory findings, given that our work used a larger and much more capable model, though this remains open question.
A.4 Many-Shot ICL; Hallucinations on Summarization
Here an example on XSum showing summaries with halluncinated date and time when increasing the number of shots for many-shot ICI., discussed in $2.2.
Reference Summary: Russia's ' leading independent polling agency has been labelled a "foreign agent" by the justice ministry and says cannot now work.
1-shot: The Levada Centre, Russian organization that conducts political opinion surveys, has been designated "foreign agent" by the government. This label restricts the organization's ac- tivities and effectively prevents it from conducting independent polls, particularly in relation to the upcoming parliamentary elections. Critics view this move as political censorship and an attempt to suppress dissent.
5-shot: Russia's leading independent pollster has been labelled "foreign agent" by the justice ministry, two weeks before parliamentary elections.
10-shot: A leading Russian opinion pollster has been labelled "foreign agent" by the authorities, effectively barring it from conducting any work linked to the upcoming parliamentary elections.
25-shot: Russia's leading independent pollster has been labelled a "foreign agent" by the justice ministry, effectively barring it from conducting election-related work.
18

---

XSum: Hallucination with Many-Shot ICL
0.6
50-shot
125-shot
0.5
250-shot
500-shot
0.4
Density
0.3
0.2
0.1
0.0
2011 2012 2013 2014 2015 2016 2017 Year (Hallucination)
Figure A.3: Hallucinated years in XSum summaries peak at 2016, with most of the years within 2014-2017.
This supports the hypothesis about such dates arising from retrieving header data from webarchive, which likely was included the pretraining data.
50-shot: 12 September 2016 Last updated at 14:32 GMT
One of Russia's last independent pollsters has been designated "foreign agent" by the justice ministry.
125-shot: 12 September 2016 Last updated at 14:43 GMT One of Russia's 's last independent pollsters has been branded a "foreign agent" by the justice ministry.
250-shot: 2016-09-00 Last updated at 14:33 GMT Russia's only independent pollster has been branded a "foreign agent" by the justice ministry.
500-shot: 12 September 2016 Last updated at 14:22 GMT
One of Russia's 's last independent pollsters has been branded a "foreign agent" by the govermment.
A.5 Reward Modelling with Many-Shot ICL: Learning Code Verifiers
standard approach to improve LLM reasoning use test-time verification [10, 43, 23]. Specifi- cally, an LLM generates multiple candidate solutions for given problem and verifier, also known as an outcome reward model, ranks these solutions and selects the best one. Here, we focus on learning such verifiers in-context for code verification.
To create in-context verification examples, utilize correct and incorrect code solutions in Python generated using Gemini 1.0 Pro [14] on the GSM8K train set. In the prompt, each (problem, solution) pair is appended with the question "Is the solution correct?" followed by the Yes No token according to ground truth correctness. At inference, we modify each test (problem, solution) pair in the same way and record the logit of the Yes and No tokens (denoted by Lyess T.Na). To compute the verifier score, we use the normalized probability of the Yes token: P(Yes) = exp(Ly ex / cxp((yys)) + exp(LN.))) ). We evaluate verifier performance using best-of-4 selection based on the verifier score 200 problems from GSM8K test set with Gemini 1.0 solutions.
As shown in Figure A.4 (left), best-of-4 accuracy with the few-shot prompted verifier significantly improves above pass@1 accuracy with 16 more in-context examples. Along with an accuracy improvement, the probabilities of the Yes token conditioned on ground-truth correct and incorrect solutions separate with increasing the number of shots up to 256, as shown in Figure A.4 (right).
19

---

Code Verifier: Best-of-4 Performance
Code Verifier: Conditional Probabilities
82.0%
Test Accuracy (%)
80.0%
Average Probability
0.4
Plses Correcti
Pitss Incorrect)
0.3
78.0%
0.2
76.0%
Pass Besb-of-4
0.1
74.0%
1
2
8 16
32
64 128 256 512
1
8 16 32
64
128 256 512
Number of Shots (K)
Number Shots (K)
Learning Verifiers In-Context for checking correctness of GSM8K code solutions. Error bars denotes standard error of mean over seeds. See Figure A. for 2-shot prompt. Best-of-N accuracy. (Left) Average accuracy of top-ranked code solutions (among solutions) based on the verifier score on 200 GSM8K test problems. Best-of-4 selection with 128-shot bridges the gap hetween Pass I accuracy of 77.25% and Pass @4 accuracy of 90% with Gemini 1.0 Pro model. Verifier Confidence. (Right) Conditional Probabilities of the Yes token P(Yes) es) from verifier, averaged over all correct and incorrect solutions test problems.
Figure
Overall, these results show proof-of-conce conceptpt that the Gemini model becomes better verifying correctness of solutions with many-shot ICL.
A.6 Is Many-Shot ICL Sensitive to Example Ordering?
In few-shot in-context learning (ICL), the order of examples within the prompt can signif- icantly impact model performance [37, 62]. Here, we investigate whether such sensitivity prompt ordering observed in few-shot ICL persists in many-shot scenarios, which remains largely unexplored. Specifically, we evalu- ate ten different random orderings of fixed 50 in-context examples from MATH train- ing split and evaluate performance on the held-out MATH500 test set [32].
MATH500: 50-shot Ordering Sensitivity
70.0%
Test Accuracy (%))
65.0%
60.0%
3
As Figure A.5 reveals, performance varies significantly across different subareas in MATH500. Strikingly, an ordering that that excels in one subarea may perform poorly in another, for example, the best Geome- try ordering yields weak results on Number Theory. This fluctuation results smaller variation in average performance compared individual subareas. One interesting ex- tension would be to optimize many-shot prompts using frameworks like DSPy [27] that has been successfully applied for opti-
55.0%
50.0%
Geomatry
Number
Theary inter.
Algebra Algebra
Pre- -lgeaaa
Counting
Pre-Caic.
Average
and Prop
Figure A.5: Many-Shot Sensitivity To Example Ordering: Fach colored data point represents a different random ordering of 50 in-context examples provided to Gemini 1.5 Pro.
mizing few-shot prompts based a given metric. Overall, these findings highlight key challenge in
ensuring reliable results with many-shot ICL for long-context models.
20

---

A.7
Negative Log-Likelihood on Meder-Generated Data
Negative Log Like thood (NLL) on Model Generated Test Solutions with L-shot prompt
L=10
L=25
Lm50
0.30
0.11
0.30
NLL
NLL
0.100
0.075
0.00
0.08
0.09
2
4"
2
&
&
Scoring Prompt
Scoring Prompt
Shots Scoring Prompt
Scoring Prompt
L-125
L-250
L-500
0.34
0.150
0.12
0.125
0.10
0.30
18
? 19
Num Shots Scoring Prompt
Num Shots Scoring Prompt
Num Shots Scoring Promot
Incorrect
Figure A.6: NLL For Model-Generated Data. We compute negative oog-likei on model genee-geeerated solutions for Hendrycks MATH. We generate solutions using prompts of varying lengths, starting from (model- generated) 4-shot prompts, and going 500-shot prompts. We then score the solutions with -meeertgenertted) prompts of different lengths, from 4-shot 500-shot prompts. We that, for the solutions generated using the 4-shot prompt, the NLL is lowest 4-shot and 10-shot scoring prompts, and increases as the length of scoring prompt increases. Note that the generation and scoring prompts contain different subsets of examples from the training sets (and are randomized for each sample). Generally speaking, we see that the NLL is lowest when the generation and the scoring prompts of similar lengths, and higher when they further apart. This reveals that prompts of simikar lengths induce similar distributions over the outputs, even when the actual contents of the prompt are different. Further, we see that correct and incorrect solutions follow similar curves, and the gap between NLL for correct and incorrect solutions docs not follow any specific pattern.
21

---

A.8
Example Prompts
You expert translator. I am going to give you example pairs of text snippets where the first is in English and the second translation of the first snippet into Kurdish. The sentences will be
written
English: <first sentence>
Kurdish: <translated first sentence>
After the example pairs, I am going provide another sentence in English and I want you translate it into Kurdish. Give only the translation, and no extra commentary, formatting, or chattiness. Translate the text from English Kurdish.
English: Its remnants produced showers across most of the islands, though as of yet, no damage flooding has been reported.
Kurdish: herêma Serengetiyê, Parka Neteweyl ya Serengeti ya Tanzanyaye, Cihê Parastina Ngorongoro Cihê Parastina Giyanewerên Nêçirě Maswa i Cihê Parastina Neteweyl ya Masaí Mara ya Kendyaye hene.
English: Kurdish:
Figure A.7: Example prompt with test input for translation from English to Kurdish on FLORES-MT benchmark in $2.1.
T will first show news article and then provide very short sentence long summary of it in fluent English.
Summarize the following article: Burberry reported pre-tax profits of f166m for the year to March. A year ago it made loss of f16.1n hit by charges at its Spanish operations. In the past year has opened 21
stores and closed nine. It plans open 20-30 stores this year worldwide.
The group has also focused on promoting the Burberry brand online.
Sales rose 7% to f1. $1.28nn, with the company recording double-digit sales growth Europe and Asia Pacific. Adjusted profit rose 23% to 2215m, taking into account one-off items and a favourable exchange rate.
Stores in London in particular benefited from favourable curreney movements and increased tourism.
"Looking forward, while mindful of the economic environment, Burberry plans build its strong financial position by accelerating investment in growth initiatives in retail, digital and new markets, while continuing enhance brand," said chief executive Angela Ahrendts.
Burberry shares were up 7.6% at 659 pence in afternoon trading. Summary: Luxury fashion designer Burberry has returned profit after opening stores and spending on online marketing
Figure A.8: Example 1-shot prompt used for summarization XSum and XLSum in $2.2.
22

---

Please solve the problem: (define (problem logistees-22ss--pla2)
(:domain logistics-strips) (sobjects aO al t0
10-011-0
pO
(iinit
(AIRPLANE a0)
(AIRPLANE al)
(CITY c0)
(CITY cl)
(TRUCK t0)
(TRUCK tl)
(LOCATION 10-0)
(in-city 10-0 c0)
(LOCATION 11-0)
(in-city 11-0 cl)
(AIRPORT 10-0)
(AIRPORT 11-0)
(OBJ p0)
(at t0 10-0)
(at t1 11-0)
(at p0 I1-0)
(at a0 10-0)
al 11-0)
(:goal
(and
p0 10-0)
Your plan as plain text without formatting:
(load-airplane p0 al 11-0)
(fly-airplane al 11-0 10-0)
(unload-airplane p0 al 10-0)
done.
Please solve the problem: (define (problem ...)
Your plan as plain without formatting:
Figure A.9: An example I-shot PDDL [16] prompt, with test example for the Logistics domain in $2.3. Within city, the locations directly linked, allowing trucks to travel between any of these locations. Similarly, cities are directly connected to each other allowing airplanes to travel between any two cities. Each city equipped with one truck and has designated location that functions as an airport
You will be given a multiple choice question with different choices such as (A), (B). (C), (D). Think step by step before giving final answer to this question. Always finish your answer with 'Final Answer: (X), where X is the correct answer choice. If none the options match, choose the closest option the final answer.
Figure 10: Zero-shot prompt for GPQA.
23

---

problem:
starts raining at 7:00 and pours heavily until its stops at 17:00 on particular day. On the second day. the rain takes 2 more hours than it took on the first day to stop. On the third day, the rain pours for twice the amount of time it took on the second day. Calculate the total time was raining in the three days.
# solution:
def solution():
"""It starts raining 7:00 and pours heavily until its stops 17:00 particular day. the second day, the rain takes more hours than it took on the first day to stop.
On the third day, the rain pours for twice the amount of time it took on the second day.
Calculate the total time it was raining in the three days.""
first_day rain duration 17 # 10 hours
second day_rain ciiindduutionnn first first_day_rain _da__riin______ blay_rainndurattinn _duration + + hours
third_day sert_day_ratte rarirraiiaattti duration second leooddd_y___aa dday_rain___
letiyy_rin_duurttioe duration * 24 hours
total_rain rain duration = first_day_ pott_dy_raing raain_ddrationn second laadddday_rijj dad__aainnduraationnnn duration + ttird__dayy Slaiarain__aaaaaaaaaeaae duration
result total rain duration return result
is the solution correct?
Yes
problem:
Haley is getting ready to watch a comet fly over her house. She spends two hours shopping for telescope. half an hour getting everything set up in the backyard, three times the setup time making snacks, and 20 minutes watching the comet. What percentage of the total time she spent on those activities was spent watching the comet, rounded to the nearest percent?
solution:
def solution():
"""Haley is getting ready watch comet fly over house.
She spends two hours shopping for a telescope, half an hour getting everything set up in the backyard, three times the setup time making snacks, and 20 minutes watching the comet. What percentage of the total time she spent on all those activitics was spent watching the comet, rounded to the nearest percent?"" mrn hours
shopping_tins time setup_time time = 0.5 hours
snack_ time sctup_time #3 # hours
comet time 20 minutes
total time shopping time + setup time + snack time + comet time percentage comet time = (comet time total. time) 100 rounded_percente result ______eente__oomneee ciatinee__omeeettmeeeee
moundppercnntaee_comettte ptt.leenttt_ccoommetteee
return result
is the solution correct?
No
Figure A.11: example 2-shot code correctness verifier prompt on GSM8K. Here, solutions are generated using the Gemini 1.0 Pro [14] model. See results in $A.5.
24

---

Problem:
Find the domain of the expression v5-x
Solution:
need to find the domain of this expression.
This expression has two square root expressions.
know the values under each square root must be non-negative. So let me start by ensuring that.
set 2 0, which simplifies to 2. which simplifies S 5.
Then,
But need to make sure the denominator is not zero.
So, 5 x: must also be greater than zero, which gives me < 5.
Therefore, the domain of the expression is [2,5)
Answer: [2, 5).
Final Answer: The final answer is [2, 5). hope is correct.
Problem: If del A = 2 and det B = 12, then find det (AB).
Solution:
need to find the determinant of the product of matrices and
remember that the determinant the product two matrices equals the product their determi-
nants.
So, (AB) (det (det B) (2)(12) 24
Answer: 24.
Final Answer: final answer is 24. T hope is correct.
Problem: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15- 15-pound weights instead, how many times must Terrll lift them in order lift the same total weight?
Solution:
Okay, so Terrell lifts a total 2- 12 20 = 480 pounds weight with the 20-pound weights. Well, if he switches to 15-pound weights, the total weight lifted will be 2 15 n 30m pounds, where is the number of lifts.
want to find the number of lifts, n, for the total weight lifted to be the same.
equate 30n 480 pounds solve for n.
30n 480
480/30 16
Answer: 16.
Final Answer: The final answer is 16. I hope it is correct.
Problem: the system equations
6x a,
6y b.
has a solution (x, y) where x and y are both nonzero, find g. assuming is nonzero. Solution:
I'm given system of two equations. see that if multiply first equation by
3. I'll get another equation that has same left-hand
side as the second equation, 6y 9x.
Let me try that
3 2a
Gy 9x
Ah, also know that 9x b, so equate these equations. So,
3 b
2
2
3
Answer:
Final Answer: The final answer is - hope it is correct.
Figure A.12: 4-Shot Inner Monologue prompt used for MATH and GSM8K.
25

---

Input: 255 378 650 363 42 447 898 211 104 145 975 827 769 977 901
Output: Foo
Input: 111 677 874 692 540 800 771 325 295 106 980 148 275 882 246 136
Output: Foo
Input: 136 529 65 265 475 639 678 95 460 902 746 919 181 838
Output: Foo
Input: 62 583 498 50 198 277 519 22 935 351 142 369 349 272 880 125
Output: Bar
Input: 101 99 830 735 732 243 703 564 225 136 333 195 441
Output: Bar
Input: 242 430 80 153 39 269 898 530 524 89 377 238 697 212 539
Output: Bar
Input: 261 83 244-33 37 170 277 161 779 544 272 893 535 71 394 64 607
Output:
Input: 402 863 114 193 413 905 894 143 193 288 174 646 411 938 212 285
Output: Bar
Input: 869 365 622 671 191 780 492 836 381 450 184 388 604 79 924 926
Output: Foo
Input: 548 823 658 380 779 449 641 673 94 130 258 229 299 278
Output: Bar
Input: 700 409 398 375 236 745 32 33 333 173 902 399 176 95 851 897
Output: Foo
Input: 673 211 14 221 508 752 309
827 980 373 861
Output: Foo
Input: 528 608 334 210 228 186 559 20 302 93 84 436 726 14 785 865
Output: Bar
Input: 190 66 628 31 838 183 687 598 187 226 381 979 171 39
Output: Bar
Input: 802 730 854 392 529 95 15 987 800 266 551 816 145 390 419 686
Output: Foo
Input: 723 701 860 30 217 633 226 477 720 839 548 880 277 178 512 585
Output: Foo
Input: ...
Output:
Figure A.13: Example prompt with shots per class linear classification dimensions, discussed $4.2, Here, we use semantically-nnreiee labels ( ("Foo' and 'Bar') following Wei al. [61].
26

---

Then the least positive value of A.9 Prompts for Unsupervised ICL
You will be provided Problems similar to the ones below: Problem: What is the remainder when 369,963 is divided by 6? Problem: The solution the inequality
y + + b 5
(-oc, 3] [5. oo). Find vertex of parabola y = az
b Problem: Let be an angle such that tan and tan 2x a=6 equals tan & Compute k. Problem: Compute sin 0%
Problem: Let
f(z) = 9r +
if ar is integer,
[x] +5 if is not an integer.
Find 29)
Now, I am going to give you series of demonstrations of math Problems and Solutions. When you respond, respond only with the Solution the final Problem, thinking step by step."
Problem: Find the domain of the expression
Solution:
need to find the domain of this expression.
This expression has square root expressions.
know the values under each square root must non-negative. So let me start by ensuring that.
set æ 2 0, which simplifies to æ 2 2. Then, I set
O. which simplifies to S 5.
But need make sure the denominator is not
So, must be greater than zero, which gives me Therefore, the domain the expression (2,
Answer: [2, 5).
Final Answer: The final answer is 5). hope it is correct.
Problem: If det A = 2 and det B 12, then find det(A
Solution:
need to find the determinant of the product of matrices A and B.
remember that the determinant the product of matrices equals the product of their determi-
nants.
So, (A = (det (det B) = (2)(12) = 24
Answer: 24.
Final Answer: The final answer is 24. I hope is correct.
Problem: Evaluate (ar + (x - y) when x = 13 and = 5.
Prompt used for Unsupervised ICL with MATH and GSM8K. We first start with preamble saying that are going to list number of problems, and then list the problems. We then give another pre-amble specify the output format, and include up to examples to fully describe this output format. As go to the many-shot setting with hundreds examples, we only increase number of problems the prompt, not problem-solutie pairs at the end.
Figure A.14:
27

---

You will be provided questions similar to ones below:
Question:
A large gene has dozens of exons, of which the central ones code for folded triple helical repeats that connect the cytoskeleton with sarcolemma and extracellular space. Each usually codes for folded triple alpha helix. The most common mutations the gene are central exon deletions to deliver Morpholino that recognizes the 5 end of the out-of-frame exon in pre-mRNA. The molecule prevents binding of the spliccosome and creates exon skipping and in-frame joining. Several missing cxons are well tolcrated by an organism. Which structure below is not involved in the proposed therapy?
create out-of offarmee peptides and progressive degenerative organ waste. solution
(A) antisense
(B) polyA A
(C) R-loops
(D) lariat
Question:
You will be given multiple choice question with different choices such (A). (B), (C), (D). Think step by step before giving a final answer this question. Always finish your answer with 'Final Answer: (X)', where the correct answer choice. If none of the options match, choose the closest option as the final answer.
Unsupervised ICL Prompt for GPQA. We first start with a preamble saying that we are going list number questions, and then we the questions. We then give another preamble to specify output format. As we go to the many-shot setting with hundreds of examples, we only increase the number of questions in the prompt.
Figure A.15:
You will be provided source sentences English translate into Kurdish similar the below:
English: Its remnants produced showers across most of the islands, though as of yet, no damage or flooding has been reported.
You are an expert translator. I am going to give you one or more example pairs of text snippets where the first is in English and the second translation of the first snippet into Kurdish. The sentences will be written
English: <first sentence>
Kurdish: <translated first sentence>
After the example pairs, am going to provide another sentence in English and I want translate it into Kurdish. Give only the translation, and extra commentary, formatting, chattiness. Translate the from English Kurdish.
English: Its remnants produced showers across most of the islands, though as of yet, no damage or flooding has been reported.
Kurdish: Li herêma Serengetiyé, Parka Neteweyi ya Serengeti ya Tanzanyayé, Cihè Parastina Ngorongoro û Cihê Parastina Giyanewerën Néçīrě Maswa å Cihê Parastina Neteweyí ya Masai Mara ya Kendyayē hene. English:
Kurdish:
Figure A.16: Unsupervised ICI. Prompt for the low-resource MT task. We first start with preamble saying that we are going to list number source sentences, and then we list the sentences. We then give another preamble with I outut- example to specify the output format. As we go to the many-shot setting with hundreds of examples, only increase the number of source sentences in the prompt.
28

---

A.10
Unsupervised ICL on Machine Translation
Translation: English Kurdish
Test chrF2 ++ (%)
45
ICL (Source and Target)
Unsupervised ICL (Source Only)
40
35
30
1
5
50 125 250 500 Number of Shots (K)
10 25
Figure A.17: Unsupervised ICL does not work for low-resource machine translation. This is expected as providing only source sentences for translation task doesn't improve the task specification. See Figure A.16 for the prompt used for unsupervised ICI. for this experiment.
A.11 Reinforced ICL: Data-collection Prompt Sensitivity and Iteration 2
Iter (Minerva)
Iter 1
Iter 2
MATH500: Reinforced ICL
60.0%
Test Accuracy (%)
57.5%
55.0%
4- 4-shot Minerva Prompt
52.5%
50.0%
4 shot InnerMono, Prompt
47.5%
10
125 Number of Shots (K)
25
50
250
500
Figure A.18: Reinforced ICL Hendrycks MATH. We find the performance of 4-shot Minerva prompt is generally better comparable to the generated by 4-shot InnerMono. MATH prompt. Furthermore, another iteration of Reinforced 25-shot prompt (with model-generated rationales) on the MATH training set and using the problems which were not solved in first iteration seem to further improve many-shot performance.
rationales with
generating rationales from the best performing
A.12 Linear Classification: Data Generation
For each classification dataset, we randomly sample another N-dimensional vector the decision boundary and decision threshold. We then provide K N-dimensional points above this threshold and points below that same threshold as in-context exemplars, and the model must determine whether unseen N-dimensional points are above or below the threshold (we do not tell the model the equation or the threshold). We provide the python code for date generation below.
import numpy as up
def generatee dataset miny
N. 1):
yirain count, pas. count nez D.
while (count_pop k) (count k): x_ex op. rondoma (minv. maxy. size=N)

---

label
ap dor do_(x_ex 8)
If count_pos 2 K: continue
pos
cise:
If Dog 5 K: continue
xtrais append
yirais. append
op. array (xtrain (str)) op ap.array
def GENERATERVALNN, K, seed):
"* Generates one evaluation example for N-dimensional linear ciorsification.
Arge:
N: Dimensionality of the doto
k: Nomber in in-costest esemplars class.
Returnv:
strain; lixt of 26 naining exouples (k positive, segative]]
ytrain: list of corresponding iabely for traising examples. rvaluation esamples Jeral: Grounditud labels for evaluation exumples.
reval:
aegative)
Step Generate groud-trath coefficients
up random. (see))
miny, mayy 1, 1000 up.random... randint
size=N) Rondom integer coefficients
Strp Generate pivor point
up rp.dandum..o randint mav. size=N)
Step Calculare classiftcation shreshold up. dot(a, p)
8 Steps Generate training exampler _geneaate
8 Stepy 6: Generate the evaluation exampic seval, yeval -eteerate_dattselleeeeecee (minv, maxy. k.
O
dataset (miny,
N. 25,
return xtrain, yirain. (xeval. yeval)
Listing 1: Code for Generating Sythetic datasets for Linear Classification in High Dimensions.
A.13 Training GPT-2 from scratch on the sequential parity task
Max num examples in many-shot ICL
1.0
0.8
Accuracy
0.6
Performa nce of many-shot ICL.
0.4
0.2
best small best medium
0.0
0.0
0.5
10
1.5
2.0
2.5
3.0
Num examples seen
le5
Figure A.19: For the sequential parity task, training trausformer from scratch does not meet 8192-shot ICL performance (dashed lines) until 20x the number of examples. We trained transformers the sequential parity task (from $4.2). The smaller model was the size of GPT-2 Small, with 12 layers and 768 embedding dimension. The larger model was the size of GPT-2 Medium, with 24 layers and 1024 embedding dimension. We trained using linear warmup and square root decay schedule, sweeping max learning rate values [le-5, Se-5, Ie-4, Se-4, 1-e3] and warmup steps [50. 100, 500, 1000, 5000J. The best values for both models (fastest learning) were max Ir=le-4, sermup_steps=10000
30

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All claims are supported by empirical experiments with Gemini 1.5 Pro.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See discussion $5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
31

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: YYs]
Justification: All experiments are done on public datasets or publicly available LLM APIs. Main ICL details are provided in $2.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
If the contribution
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code. the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soures models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [No]
Justification: Unfortunately, our experiments depend on internal infrastructure and code that can not be made fully public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/ Meaiic//uieess/Cdeeee
wwwssssoooiiiyoss for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips.ae https.cc/pppp.......
ssi ssionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Train/ test details provided within each subsection.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
Error bars are reported for almost all experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
33

---

It should be clear whether the emror bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or ligures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: It is typically not possible to infer details about compute resources experi- ments depends on the Gemini Pro API. However, the querying cost in terms money can be easily estimated using the amount of prompt tokens sent and output tokens generated for each task.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttpas:///eepie//eeeee httprrps../ppblllehthttteeeieeee
Answer: [Yes]
Justification:
[NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special considera- tion due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
The work is foundational research and and not tied to particular applications. Both useful applications (such as better summarization, translation etc) as well as potentially bad applications such jailbreaking models [2] are possible. Most of these use-cases need to be handled by LLM inference API providers on a base-by-case basis.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
34

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g,, disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper uses already publicly available inference API for Gemini 1.5 Pro model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should deseribe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Only publicly available datasets with MIT CC-BY 4.0 licenses are used for experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
35

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi wwwthwodeeeee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged reach out the asset's creators.
13. New Assets
Question:
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
[NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage the country the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
36

---

We recognize that the procedures for this may vary significantly between institutions and locations. and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "qoiDeJ1AO4", "reviewer": "Reviewer_XEKv", "summary": "This work conducts a comprehensive study on in-context learning (ICL). The experiments range from few-shot to many-shot scenarios, with up to 2048 in-context examples. It was observed that as the number of examples increases, the performance generally improves, even matching the performance of fine-tuned models. One limitation of many-shot in-context learning noted by the authors is the difficulty in obtaining high-quality in-context example pairs. To address this issue, the authors proposed Reinforced and Unsupervised ICL, which achieved results comparable to those using ground truth examples. Additionally, the authors explored many-shot ICL in the context of pre-training bias (distribution shift settings) and high-dimensional numerical settings, providing explanations for the results.", "questions": "1. It's surprising that Reinforced ICL and Unsupervised ICL outperform the ground truth. Do you have any explanations for this?\n2. The performance shows significant variations when different random seeds are used to select in-context examples. Could you explain why this happens?\n3. Flipped labels and abstract labels seem to achieve the worst performance when K=8 or K=16. Is this a universal result across different model sizes, as related to [1][2]?\n4. I am curious about what model is used for figure 11 in sec 4.3. Did you supervise fine-tune the same model and report its results on the dataset?", "limitations": "Please see weakness and questions for limitations. There is no potential negative societal impact of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The experiments are comprehensive, covering a wide range of tasks and comparisons. The paper is well-written with a clear structure. The problem itself is interesting and has broad applications.", "weaknesses": "The experiments are limited to a single model, Gemini 1.5 Pro, as mentioned by the authors. This may narrow the scope of the results, as different models have varying pre-training data and biases. The performance of LLM under ICL shows significant variation (as seen in figure 6). It would be beneficial to include more seeds or utilize better statistical metrics to represent model performance, beyond just average accuracy. Additionally, as the number of shots increases, so does the number of tokens, making the computation budget (RAM of the machine) a potential bottleneck for further exploration and application. Given that many in-context examples have varying lengths, future work could focus on how to select better in-context examples that capture the task's nature while also being concise in terms of token length.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and questions, which we address below. We have added new results on 1.5 Flash and inference time of many-shot in the **[rebuttal pdf](https://openreview.net/attachment?id=4BVPccQZEN&name=pdf)**.\n\n> **Experiments are mostly limited to 1.5 Pro .. different models have varying pre-training data and biases.**\n\nIndeed, our work serves as an existence proof for the huge potential of many-shot ICL. That said, we provided preliminary results for GPT-4-Turbo and Claude-3-Opus in Figure A.2, indicating that they can also benefit from many-shot ICL to varying degrees. \n\nWe added **Figure 1** in rebuttal pdf to report many-shot performance for Gemini 1.5 Flash, a smaller LLM than 1.5 Pro, and show that it can match or surpass Claude-3-Opus and GPT-4-Turbo with enough shots, despite having worse few-shot performance. We’ll move this to the main paper. \n\nRecently, follow-ups [1, 2, 3] have exhibited many-shot ICL with open-weights and closed-source models on different tasks. Our work contributes to this growing body of evidence, and contributes several analyses of the phenomenon.\n\n> **ICL shows significant variation on GPQA (Fig 6) .. more seeds or better metrics** \n\nOur paper reports standard deviation across seeds on most of the tasks. We’d emphasize that GPQA is an exception, rather than the norm, even noted by the Anthropic report due to its extreme difficulty and small size (198 questions only). To show this variability, we directly report the individual performance on 5 seeds on both GPQA and MATH.\n\n> **As the number of shots increases, so does tokens, making computation budget a bottleneck** \n\nIndeed, many-shot ICL does increase inference computation time, but can allow for quick prototyping and experimentation using just an inference API. That said, it can be sped-up with KV caching and context caching [7], which is default for long-context servers.  \n\nTo empirically measure this inference time, we’ve added **Figure 2** in the rebuttal pdf showing per-output generation runtime for many-shot ICL, averaged across test set and multiple seeds, on summarization (500-shot) and sequential parity prediction (8192-shot). With caching enabled, runtime increases linearly with a large number of shots, as opposed to quadratic for self-attention: doubling the number of shots nearly doubles the runtime. However, for a small number of shots (less than 32k tokens), we see that runtime is nearly constant.\n\n> **It's surprising that Reinforced and Unsupervised ICL outperform ground truth. Do you have any explanations?**\n\n- Reinforced ICL: The Reinforced Self-Training work [4] showed that fine-tuning using model-generated rationales can be more effective than human-generated ground-truth outputs. We show that a similar finding holds true for many-shot ICL. Since model-generated outputs utilize only the skills / knowledge possessed by the LLM, it might make such outputs easier to learn from.  \n\n- Unsupervised ICL: This is harder to explain as unsupervised ICL does not always work well: it outperforms ground truth for MATH, but is substantially worse for low-resource MT (Appendix A.10). Our hypothesis is that it works well when the LLM already has all the required knowledge to solve a task. As such, ground-truth outputs might bias the model and the model prefers to utilize its underlying knowledge by just relying on inputs.\n\n> **The performance shows significant variations when different random seeds are used to select in-context examples. Could you explain why this happens?**\n\nImpact of random seed for ICL example selection seems to be highly task dependent – on low-resource MT and summarization, it has a minimal effect, as can be noticed from small error bars that show standard deviation of mean performance across seeds. However, on other tasks, such as MATH and GPQA, it has a higher impact. Prior work has found the following factors impact few-shot performance when selecting examples:\n- If chosen examples are semantically similar to test examples, it leads to improved performance [5]\n- Increased diversity in chosen examples leads to improved performance [6]\n\nWhen we select different example subsets by setting different random seeds, we perturb these factors, leading to differences in downstream performance. Our general trends hold even taking into consideration this variation in performance (shown by standard deviation bars).\n\n\n> **Flipped and abstract labels .. achieve worst performance when K=8 / K=16. Is this  universal across different model sizes?**\n\nWe do not know if this is a universal result across different model sizes. That said, we also evaluated Gemini 1.5 Flash on flipped labels and observed similar accuracy trends: 56% for K=4, 34% at K=8, 40% for K=16, 76% for K=32, and 86.5% for K=64.\n\n> **Did you supervise fine-tune the same model and report on the dataset?**\n\nYes, we performed “full” fine-tuning on the same model (Gemini 1.5 Pro) on the same examples that were used for many-shot ICL. We performed 5 epochs of training, picking intermediate checkpoint with lowest validation loss to ensure strong SFT results.\n\n> **Future work could focus on how to select better ICL examples that capture the task's nature while being concise**\n\nAgreed, would add to future work.\n\n----\n[1] Many-Shot In-Context Learning in Multimodal Foundation Models. Jiang et al, 2024.\n\n[2] Many-Shot In-Context Learning for Molecular Inverse Design. Moayedpour et al, 2024. \n\n[3] In-Context Learning with Long-Context Models: An In-Depth Exploration. Bertsch et al, 2024.\n\n[4] Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models. Singh et al, 2024.\n\n[5] What Makes Good In-Context Examples for GPT. Liu et al, 2021.\n\n[6] Selective Annotation Makes LMs Better Few-Shot Learners. Su et al, 2022.\n\n[7] Context caching. ai.google.dev/gemini-api/docs/caching\n\n----\n*We hope most of the reviewer's concerns have been addressed and if so, they would reconsider their assessment.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0CmoCjS1jA", "reviewer": "Reviewer_mj1R", "summary": "Owing to the significant increases in context window lengths, the paper analyzes the efficacy of the Gemini 1.5 Pro LLM in the many-shot in-context learning (ICL) setting, where hundreds to thousands of exemplars can be provided to the model at inference time. ICL has generally been restricted to the few-shot learning setting, where only a small number of demonstrations are provided to the LLM. As this expansion to the many-shot setting can pose issues relating to large scale data collection, the authors propose two simple approaches: (1) Reinforced ICL, which switches human written rationales for demonstrations with chain-of-thought model generated rationales, and (2) Unsupervised ICL, where rationales are not provided in the ICL task. The authors conduct extensive experiments across a number of problem domains, ranging from summarization, machine translation, logistics planning, question answering, algorithmic reasoning, among many others, showcasing the performance benefits obtained by using a larger number of exemplars in ICL.", "questions": "Please see the weaknesses listed above.", "limitations": "Yes, the limitations have been addressed. More details can be provided in a revision.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- I believe the paper is a significant contribution to the field of ICL, as it analyzes the efficacy of LLMs in the not yet studied many-shot regime. The authors conduct a number of extensive experiments ranging from a diverse set of tasks and benchmarks, showcasing the benefits of many-shot ICL. The biggest takeaway from the paper would be that many-shot ICL could be a suitable alternative to supervised fine-tuning which would tune the entire set of model weights, albeit at the cost of increased inference time (which can be reduced via KV caching).\n- The paper is very well-written and I appreciate the large scale of experiments conducted on Google Gemini 1.5 Pro. \n- The two annotation-free many-shot ICL strategies (Reinforced ICL and Unsupervised ICL) proposed are simple, but clearly demonstrate improved performance on a wide variety of tasks.\n- Findings relating to overcoming pre-training biases, learning higher-order functions, and many-shot ICL vs supervised fine-tuning are also important results that strengthen the contributions of this work.", "weaknesses": "- As this is an empirical analysis paper which spans many different tasks and benchmarks, can the authors confirm that for all the tasks, the full test splits were used for evaluation as in line with community standards and past work? If there are any exceptions, these should be listed. For instance, measuring summarization performance on XSum/XLSum for only 150 test articles seems far less in size than the actual test set for this dataset (~11k articles). Were the articles randomly sampled?\n- While the ablations are carried out with respect to the number of ICL exemplars provided to the LLM, I am somewhat unsure of what role the model size plays here. As shown in the cited Wei et al paper (https://arxiv.org/pdf/2303.03846), larger LMs might do in-context learning differently and can learn input-output mappings better than smaller LMs which for example, might not be able to reject pretraining biases. Thus, it is not clear if some of the performance improvements are just a direct result of using Gemini 1.5 Pro (which is a model on the larger end of the size spectrum). Conversely would a smaller LLM with a larger context also benefit from many-shot ICL (possibly not as many exemplars as Gemini 1.5 Pro, however). Do the authors have any thoughts on this and can they draw a distinction between gains attained due to an increase in the size of the model versus the many-shot ICL setting?\n- Do the authors have any intuition for why at times performance reduces as more exemplars are added, somewhat akin to overfitting? I noted that the authors discussed this as an open question in the paper but I think it would benefit readers if some more insight could be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments, which we try to address below. We have added a many-shot ICL comparison for Gemini 1.5 Flash with frontier LLMs to understand the role of model size in the **[rebuttal pdf](https://openreview.net/attachment?id=4BVPccQZEN&name=pdf)**.\n\n> **unsure of what role the model size plays here .. would a smaller LLM with a larger context also benefit from many-shot ICL?  Gains from increase in the model size versus many-shot ICL** \n\nTo understand the role of model size, we’ve evaluated the many-shot performance of Gemini 1.5 Flash, a smaller long-context LLM than Gemini 1.5 Pro. We used the low-resource MT to compare against LLMs at the larger-size of the spectrum, namely 1.5 Pro, Claude-3-Opus and GPT-4. Our results suggest that even smaller LLMs can benefit from many-shot ICL and outperform LLMs with stronger few-shot performance with enough shots. We report these results in **Figure 1** in the rebuttal pdf.\n\nOn the English → Bemba task, we find that 1.5 Flash matches Claude-3-Opus and outperforms GPT-4 with 997-shots, despite having much worse few-shot performance than Claude and GPT. On English → Tamil MT, 1.5 Flash performs comparably to 1.5 Pro and Claude in terms of few-shot performance. However, 1.5 Flash outperforms Claude-3 in terms of many-shot performance, while lags behind 1.5 Pro. \n\n> **Were the test splits in line with community standards? If there are any exceptions, these should be listed .. XSum/XLSum for only 150 test articles seems small. Were the articles randomly sampled?**\n\nYes, for almost all of the tasks, we used the standard test sets used for evaluation, e.g., MATH500 and GPQA Diamond split). The only exceptions were summarization and low-resource MT, we randomly subsampled 150 examples from the entire test set to reduce the cost of evaluating many-shot ICL across multiple seeds. We'll update the text to specify this clearly. \n\nNote that for XSum, we actually used the GEM-XSUM [1], which is a cleaner version of XSum with 1.2K test articles. \n\t\n> **Do the authors have any intuition for why at times performance reduces as more exemplars are added?**\n\nA very recent submission under review [2] argues that performance drop in many-shot ICL might be due to more demonstrations diverting the model attention from the query, hindering its understanding of the query. \n\nAnother reason might be that *really long* many-shot prompts might be highly out-of-distribution (OOD) as LLMs might not have seen such prompts during pre-training or post-training (current mixtures might be optimized for needle-in-a-haystack tests). Furthermore, LLM pre-training has a maximum sequence length, followed by continued pre-training for context lengthening, for example, LLaMa 3.1 uses 8k and 128k while Apple FM uses 8k and 32k. We’ll update the discussion to include these hypotheses.\n\n[1] The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics. Gehrmann et. al, 2021.\n\n[2] Focused Large Language Models are Stable Many-Shot Learners. Anonymous, 2024.\n\n\n-------\n*We hope most of the reviewer's concerns have been addressed and if so, they would reconsider their assessment*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IqSY8EYX7u", "reviewer": "Reviewer_kkH9", "summary": "In this work, the Authors investigate the performance of large language models on in-context learning (ICL) tasks when provided with a large number - in the order of hundreds or thousands - examples (many-shot ICL regime), enabled by recent increases in context window sizes.\n\nThe Authors demonstrate significant improvements across various tasks when moving from few-shots to many-shots. They also introduce two methods, called Reinforced ICL and Unsupervised ICL, to mitigate the need for human-generated examples.\n\nThe paper analyzes how many-shot ICL affects model behavior, including overcoming pre-training biases and learning high-dimensional functions.", "questions": "I would like to aks the Authors what they think about the possible application of many-shots ICL in alignemt problems.\n\nThe ability to include much larger contexts could - if I am not mistaken - be leveraged for improving alignment to specific ethical standards or legal frameworks (this seems plausible if one takes into account your findings about overcoming pre-training biases). By providing a large number of examples that demonstrate the desired ethical reasoning or decision-making process, it might be possible to steer the model's behavior more effectively than with few-shot prompting or fine-tuning.\n\nDo you think this is a potential application of many-shots ICL or you can already see limitations? \n\nFor example, it comes to me that, since sometimes the performance can degrade with too many examples, and the ordering of examples can affect results, applying many-shots ICL to ethical alignment would require understanding how to structure and present examples effectively within the context window.", "limitations": "While the paper addresses correctly the technical limitations, the potential downsides or risks associated with many-shot ICL are not discussed. I think that a brief remark on potential misuses would be appropriate.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper examines many-shot ICL across a wide range of tasks including translation, summarization, planning, mathematical problem-solving. This broad scope illustrates very well the benefits of many-shots.\n\nThe Authors introduce novel methods  (Reinforced ICL and Unsupervised ICL) to address limitations of many-shot ICL, namely, the need for large amounts of human-generated examples, and show their superiority. \n\nAn in-depth analyses is conducted of how many-shot ICL affects model behavior. For example, the Authors demonstrate that many-shot ICL can overcome pre-training biases (as shown, for example, in Figure 10 where performance on flipped and abstract labels approaches that of default labels with increasing shots). The paper presents a spund evidence for the benefits of many-shot ICL. For instance, in Figure 1, it is shown a consistent performance improvement across various tasks.\n\nBy comparing many-shot ICL to fine-tuning, it is shown that a comparable performance is reached in some cases, suggesting that many-shot ICL could be a viable alternative to fine-tuning.", "weaknesses": "As the Authors recognize, the study is limited to a single model (Gemini 1.5 Pro). \nWhile the Authors do include some results with GPT-4-Turbo and Claude-3-Opus, a more comprehensive comparison across different models would strengthen the generalizability of the findings.\n\nWhile the paper provides extensive empirical results, it lacks a theoretical framework to explain why many-shot ICL works so well. \nA theoretical analysis could provide insights into the mechanisms behind observed improvements.\n\nAnother point in which a theoretical analysis would be highly desirable regards the following case. The Authors note that performance can in some cases degrade with more examples (e.g., in the case of MATH), but don't fully explain this phenomenon. They indeed state: \"Our analysis found that negative log-likelihood trends are insufficient to explain this degradation, and future work should investigate new directions to shed light on the matter and improving many-shot ICL capabilities.\" However, this is point is correctly raised while discussing limitations.\n\nFinally, a more explicit discussion of the  potential drawbacks and risks associated with many-shot ICL would be helpful to raise awareness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments and questions. Our detailed response follows: \n\n> **While authors include some results with GPT-4-Turbo and Claude-3-Opus, a more comprehensive comparison across models would strengthen the generalizability**\n\nWhile we do not fully address this limitation, in the rebuttal pdf, we’ve added results for Gemini 1.5 Flash in **Figure 1**, a smaller LLM than Gemini 1.5 Pro and. We find that even 1.5 Flash can match or surpass Claude and GPT with many-shot ICL, despite having worse few-shot performance. This demonstrates that even small LLMs with long-contexts might be capable of many-shot ICL.\n\n> **While the paper provides extensive empirical results, it lacks a theoretical framework to explain why many-shot ICL works so well**\n\nRelated to this, an ICML'24 paper [1] argues that ICL operates under two possible modes, task recognition and task learning – activating the task learning mode requires a large enough number of shots (however they only empirically studied up to 128-shots), which is highly task dependent. It is possible that the success of many-shot ICL is partially explained by activating this task learning mode. We’ll include this in discussion.\n\n> **Theoretical Analysis for why performance degrades**\n\nA very recent submission under review [2] argues that performance drop in many-shot ICL might be due to more demonstrations diverting the model attention from the query, hindering its understanding of the query.\n\nAnother reason might be that really long many-shot prompts might be highly out-of-distribution (OOD) as LLMs might not have seen such prompts during pre-training or post-training (current mixtures might be optimized for needle-in-a-haystack tests). Furthermore, LLM pre-training has a maximum sequence length, followed by continued pre-training for context lengthening, for example, LLaMa 3.1 uses 8k and 128k while Apple FM uses 8k and 32k. We’ll update the discussion to include these hypotheses.\n\n\n> **An explicit discussion of drawbacks and risks with many-shot ICL**\n\nWe’ll update the discussion to include the following drawbacks and risks.\n- Computational Cost: Many-shot ICL can be computationally expensive, especially with a large number of examples. This can be mitigated with context caching and KV caching.\n- Bias Amplification: If the many-shot examples in the prompt contain biases (e.g., stereotypes, unfair representations), the model can amplify these biases, which raises ethical concerns.\n- Lack of Transparency: The inner workings of how ICL works is not well understood. This makes it difficult to pinpoint exactly why a model generates a specific output with many-shot ICL. This can be problematic to assure safety and alignment of LLMs [3].\n- Jailbreaking: Many-shot ICL can be used for overriding safety-training biases, manipulating LLMs to behave in unintended or harmful ways.\n\n\n> **Is improving ethical alignment a potential application of many-shot?**\n\nWe agree that inference time steering of LLMs with many-shot ICL has more flexibility compared to fine-tuning (e.g. different alignment criteria for different use cases), could also allow for faster adaptation to changing criteria, and fast \"patching\" for newly discovered legal or ethical issues.  As such, many-shot ICL for ethical alignment seems to be a very promising application. \n\n> **Since sometimes the performance can degrade with too many examples, and the ordering of examples can affect results, applying many-shots ICL to ethical alignment could be challenging.**\n\nIn our work, sensitivity to example ordering seems to be highly task dependent – we do not see much impact on ordering on tasks like low-resource MT or summarization but larger impact on other tasks. Nevertheless, a simple approach might be to tune the ordering itself based on a held-out validation set and off-the-shelf libraries such as [DSPy](https://github.com/stanfordnlp/dspy) can be easily used for such purposes.\n\nRegarding the optimal number of examples, this is likely a more critical parameter and would also matter for ethical alignment. As a rule of thumb, we swept across the number of shots on a logarithmic scale. Interestingly, our results on overriding pretraining biases show that performance only plateaued rather than degrading when adding too many shots.\n\n-----\n[1] Dual Operating Modes of In-Context Learning. Lin and Lee, 2024.\n\n[2] Focused Large Language Models are Stable Many-Shot Learners. Anonymous, 2024.\n\n[3] Foundational Challenges in Assuring Alignment and Safety of LLMs. Anwar et al, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZUQccE4Rs6", "reviewer": "Reviewer_2oDD", "summary": "This paper explores the effectiveness of in-context learning with hundreds to thousands of examples, bringing the number of examples closer to the range one might use for supervised training methods.  Experiments are performed on a large number of tasks and benchmarks using Gemini 1.5 as the LLM, in each case giving it a prompt containing varying number of dataset examples and observing its performance as the number of examples increases.  These robustly demonstrate the effectiveness of using more examples, at times beating fully-supervised models on the same data.  In addition to using the ground truth examples, two additional prompting methods are evaluated:  Unsupervised ICL, which only adds additional inputs to the prompts, and Reinforced ICL, which adds inputs along with machine-generated responses filtered by correctness of result; both techniques limit the amount of \"ground truth\" responses required for adding prompt examples, and are also found to perform well for all tasks.", "questions": "While reading this paper I ended up with lots of comments and questions that I've listed below.  Aside from profiling computation increases as mentioned above, there are a lot of particulars around the individual experiments that I had questions about, as well as questions on what might be causes and effects in addition to raw performance numbers.\n\n\n\n- Including more examples has two effects:  One of increasing information and distinct samples used, but another of increasing the context length.  When viewed analogously to a training loop, a supervised optimization loop will repeat examples between epochs if there are fewer samples than steps.  It would be interesting to start to separate these two effects, perhaps by extending context length by repeating the same examples instead of adding new examples.  For example when going from 20 to 200, repeating the same 20 examples 10 times in the prompt.  Is the resulting performance closer to using 20 or 200, or in between, and for which tasks?\n\n\n- Supervised finetuning comparison in 4.3 is only on the machine translation task; while this is a good experiment that I was on my mind from the start, the fact that it was evaluated on only one task limits the conclusion.\n\n\n- Reinforced ICL description could have more details, particularly since it is highlighted as one of the main contributions.  The description \"we select rationales that obtain the correct final answer\" makes sense as a general filtering procedure, but I'm not sure how many rationales are generated or selected per problem (though I assume only one is used for each k-shot prompt), and it's not clear what happens if no generated rationales also contain the correct answer --- are these inputs thrown away entirely, and if so does this also have a beneficial or detrimental filtering effect on the inputs used?\n\n\n- sec 2.3 planning logistics:  I don't agree that the figure demonstrates significant increase for many-shot as stated in this section.  Near-maximum performance of just under 35% success is achieved by 10 examples, with a possible uptick at 800, as described in the caption.  But this uptick is about as large as the downtick at 20, and is towards the end of the plot without enough points to establish a clear trend.  It's unclear to me if this is due to the larger number of examples or other effects (e.g. which examples are included, as 800 will have a higher chance of including the most useful examples, or random variation).\n\n- Related:  I'm not sure what error bars indicate; other figures' captions say it is stdev, but I don't see this described in the text.\n\n\n- sec 3.1 Fig 5:  I'm a little unclear on what K=4 corresponds to in the ICL Ground Truth case.  Since the Unsupervised ICL prompt also includes 4 gt examples at the end, does the ICL Ground Truth prompt contain 4 examples in addition to the final 4 (for 8 total), or only these final 4?  That is, are these 4 at the end of the prompt counted in the K or not, and is this the same between all three prompt methods?  If these final 4 are included in K, then the three methods should coincide at K=4.  If they are not included, then ICL Ground Truth should have 8 total gt examples (K=4 + the four at the end of the prompt) -- is this the case?  I also wasn't able to find example prompts for reinforced or gt ICL for tasks corresponding to the ones exploring unsupervised prompts.\n\n\n- fig 6:  reinforced ICL at K=250 is missing from this figure\n\n- sec 3.3 l.194 \"to reduce the impact of false positives, ...\" --- why is this important?  does it obscure trends in the results by having too much noice from chance, for example?\n\n- sec 4.1 Fig 8:  really interesting the shape of these curves as a function of K is basically what one would expect to see for a supervised model preinitialized with the original task labels, as a function of the training step number.  not sure if this would be worth mentioning in the text maybe also showing a supervised training run exhibiting similar behavior for qualitative comparison?\n\n- sec 4.3 comparing to SFT:  this section lacks details on how supervised fine tuning was implemented:  in particular, what was the base model, which weights/adapters are trained or frozen, and how many epochs over the training data were performed?  And why were these choices made in particular?  There is also a description of trade-offs between training and inference costs of the two approaches, but no estimates on what the computation resources for each are.\n\n\n- sec 4.4 on NLL:  these are some interesting behaviors, though I agree it's unclear what to make of them exactly, but I appreciate presenting the observations.  As for why GT ICL may be lower NLL than Reinforced ICL:  could this be that Reinforced ICL specifically rejects some high-prob model generated responses that don't end up with the right answers, raising its NLL from this filtering step?  whereas in contrast, if the gt or similar same-source examples was present in the pretraining data, it would be expected to have relatively low NLL due to being drawn from that datasource?\n\n\n- xsum weird dates after K=50:  xsum was taken from webarchive articles, basically mapping html body text to title --- I wonder if somehow the model learned the task was to recover the webarchive page title and confused with other header data including the webarchive last updated time, from the pretraining data?  the 2016 timestamps are roughly around to the document snapshot times in the xsum urls.  e.g. if webarchive processing included writing a header with the timestamps, then title, then rest of the page, for example.   this is also compatible with discussion on finding the relevant parts of pretraining for the task, for Unsupervised ICL intro at sec 3 l.144.\n\n\n- xsum could also use unsupervised ICL -- what is its performance just adding articles without summaries?  For that matter, machine translation could also use unsupervised ICL just with the English phrases.  I'd imagine it wouldn't help at all for MT but it's not impossible, and evaluating it might still be interesting just to verify this.  In general using all techniques for all tasks would help round out the study (only reinforced ICL wouldn't make sense for tasks where there is no succinct way to do correctness filtering).", "limitations": "With the possibility of including more data, it is now more likely to include noisy or incorrect data (either accidentally or adversarially).  This might also be mentioned and discussed in the limitations section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* These experiments provide a very good overview of the effects on performance of including these numbers examples in prompts, surveying the effects in a wide set of tasks.\n\n* Additional studies, particuarly sec 4, are interesting, finding some behaviors that parallel those seen in supervised training, in the context of ICL with substantive benchmark tasks.\n\n* Unsupervised ICL and Reinforced ICL are useful techniques to limit the amount of gt reponses, and shown to be effective in these evalutaions", "weaknesses": "* Only Gemini 1.5 was explored, though this was discussed by the authors in the discussion and limitations section, I think it's a significant weakness as it limits findings of many-shot ICL to describing the behavior of this particular model, not the typical range of performance behaviors for different systems with this approach, and whether these are automatic simply by extending context, or if there are model differences that can impact this technique.\n\n* I didn't see anything on the computational costs or runtime differences varying K, which would provide fuller picture of the behavior of many-shot ICL beyond just its end performance.\n\n* It's a lot of information to cram into the allotted space and many sections seem terse with the connections between experiments and how they contribute to the overall understanding often left hanging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed comments and questions, which we address below and would strengthen our work. \n\nIn the **[rebuttal pdf](https://openreview.net/attachment?id=4BVPccQZEN&name=pdf)**, we have added results on *runtime differences, many-shot performance of 1.5 Flash and frontier LLMs, ablated impact of new information vs context length, re-evaluated results on Logistics, and added analysis for hallucination on xsum*.  Our detailed response follows:\n\n> **Only Gemini 1.5 was explored .. are there model differences that impact many-shot**\n\nIndeed, our work serves as an existence proof for the huge potential of many-shot ICL. Nevertheless, we provided preliminary results for GPT-4-Turbo and Claude-3-Opus in **Figure A.2**, indicating that different models benefit from many-shot ICL to varying degrees. \n\nWe also added **Figure 1** in the rebuttal pdf to evaluate many-shot ICL performance for Gemini 1.5 Flash, a smaller LLM than 1.5 Pro, and show that it can match or surpass Claude-3-Opus and GPT-4-Turbo with enough shots, despite having worse few-shot performance. We’ll move this to the main paper.\n\nRecently, follow-ups [1, 2, 3] have exhibited many-shot ICL with other open-weights and closed-source models on different tasks. Our work contributes to this growing body of evidence, and contributes several analyses of the phenomenon.\n\n[1] Many-Shot ICL in Multimodal Foundation Models. Jiang et al, 2024.\n\n[2] Many-Shot ICL for Molecular Inverse Design. Moayedpour et al, 2024. \n\n[3] ICL with Long-Context Models: An In-Depth Exploration. Bertsch et al, 2024.\n\n> **Runtime differences varying K to provide a fuller picture of many-shot ICL .. profiling computation increases**\n\nGreat suggestion! To show runtime differences, we added **Figure 2** in the rebuttal pdf showing per-single generation runtime, averaged across the test set and multiple seeds, for many-shot ICL on summarization (500-shot) and sequential parity prediction (8192-shot). \n\nWith KV caching enabled (default for long-context servers), runtime increases linearly with a large number of shots, as opposed to quadratic for self-attention: doubling the number of shots nearly doubles the runtime. However, for a small number of shots, runtime is nearly constant. \n\nExplanation: When computing the next token, we still have to attend to the fixed many-shot prompt, even if KV is cached. When the number of generated tokens is much smaller than many-shot prompts, each new token is still linear, which explains our observed runtime for a large number of shots. We hypothesize that up to a token length of 32K, you can fit the entire KV cache into TPU HBM, which roughly means that you compute next tokens in O(1) memory load.\n\n> **xsum weird dates after K=50 .. if somehow the model learned the task was to recover the webarchive page title**\n\nOur analysis suggests that this hypothesis is likely to be true! Specifically, we extracted the hallucinated years from XSum summaries and plotted their histogram density in **Figure 3**. Remarkably, more than 95% of these dates indeed lie within the range 2014-2017, suggesting that the model might indeed be retrieving additional information about webarchive last updated time. \n\n> **Including more distinct examples increases information, but also context length ..  separate these effects by extending context length by repeating same examples**\n\nWe ran this experiment on low-resource MT by repeating 25 examples several times to create many-shot prompts with up to 1000 examples (shuffled ordering) and added the results in **Figure 4** in the rebuttal pdf. \n\nThe performance with repeated examples stays nearly the same and significantly lags behind many-shot performance with distinct examples. On this task, the benefit of many-shot ICL mainly stems from adding new information as opposed to increasing context length. \n\n> **Planning logistics: Is there a significant increase for many-shot?**\n\nTo be certain, we re-evaluated many-shot on Logistics with the latest public version of Gemini 1.5 Pro, and added the result in **Figure 5** in the rebuttal pdf. Many-shot accuracy improves uniformly for this version – interestingly, few-shot performance already starts quite high around 40%, and improves to 62.8% with 400-shot and plateaus to 63.8 with 800% shot. \n\n> **Supervised fine tuning: base model, and how many epochs? And why were these choices made? Estimates on the computation resources?**\n\nWe performed “full” fine-tuning (no adapters) on the same base model that was used for the many-shot ICL (Gemini 1.5 Pro). We performed 5 epochs of training, picking intermediate checkpoint with lowest validation loss (often from the first few epochs). These choices were made to ensure that we can obtain quite strong results for SFT. We’ll add these details in Sec 4.3.\n\nSince Gemini 1.5 Pro is closed-source, we used [Vertex API for SFT](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini-use-supervised-tuning) and cannot provide estimates of computation resources. \n\n> **Supervised finetuning comparison only on machine translation task .. limited conclusion**\n\nCorrect, our results only demonstrate that many-shot ICL **can be** competitive to fine-tuning on **some** tasks. The high dollar cost of “fine-tuning” limited our experimentation to 4 runs (2 tasks x 2 data sizes). We also performed comparison to SFT on parity prediction, where we find that many-shot ICL requires 20x less samples compared to fine-tuning GPT-2 to reach the same performance on this synthetic task (Appendix A.13). A more thorough comparison would be interesting for future work.\n\n> **what error bars indicate .. say it is stdev**\n\nYes, error bars indicate stdev of test performance across multiple random seeds, where K-shot prompts are sampled randomly for each seed (Lines 68-70). We’ll update the text to clarify this.\n\n\n-------\n\n*We hope most of the reviewer's concerns have been addressed and if so, they would reconsider their assessment.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers R1 (XEKv), R2 (mj1R), R3 (kkH9), and R4 (2oDD) for their valuable feedback! All reviewers are in favor of acceptance and found the paper to be **comprehensive, very well-written, significant contribution to ICL, broad scope and applications, novel annotation-free ICL methods, and interesting analysis**.  This paper also generated a lot of discussions for our observations as well as questions about future work, which we highly appreciated. Here, we address the common concerns raised by the reviewers:


### **Experiments mostly limited to Gemini 1.5 Pro**

Our work serves as an existence proof for the huge potential of many-shot ICL across a variety of tasks. That said, we provided preliminary results for **GPT-4-Turbo and Claude-3-Opus** in Figure A.2, indicating that they can also benefit from many-shot ICL to varying degrees. Several follow-ups have exhibited many-shot ICL with other open-weights and closed-source models on different tasks. Our work contributes to this growing body of evidence, and contributes several analyses of the phenomenon.

We also added **Figure 1** in the rebuttal pdf to report many-shot ICL performance for **Gemini 1.5 Flash**, a smaller LLM than 1.5 Pro, and show that it can match or surpass Claude-3-Opus and GPT-4-Turbo with enough shots, despite having worse few-shot performance. We’ll move this to the main paper.


### **Runtime and inference compute increase for many-shot**

While many-shot ICL increases inference computation time, it can allow for quick prototyping and experimentation using just an inference API. That said, it can be sped-up with KV caching and context caching, which is default for long-context servers. Moreover, being able to spend additional inference-time compute to obtain better performance is a useful feature to have.

We added **Figure 2** in the rebuttal pdf showing per-single generation runtime, averaged across the test set and multiple seeds, for many-shot ICL on summarization (500-shot) and sequential parity prediction (8192-shot). With KV caching enabled (default for long-context servers), runtime **increases linearly with a large number of shots, as opposed to quadratically** for self-attention: doubling the number of shots nearly doubles the runtime. However, for a small number of shots, runtime is nearly constant. We'll add this result to the paper.


### **Additional results in rebuttal pdf for R4**

Other results the questions and concerns raised by R4 about impact of context length vs new information in many-shot ICL (**Figure 4**), hallucination on XSum (**Figure 3**), and re-evaluation on the Planning Logistics task with latest Gemini API (**Figure 5**). 


### **Details about SFT comparisons, empirical evaluations, and Reinforced ICL**

We'll update the revision to include them, which we discuss below in individual rebuttals.

Meta-review: The paper originally received high scores, with reviewers praising the significance of the findings and potential positive impact of the proposed ICL examples quality improvement techniques. Yet the reviewers also raised some concerns on:
1. limited experimentation - only on Gemini-1.5 Pro
2. potential efficiency overhead with enlarged input context due ICL examples
3. lack of theoretical grounding
4. insufficient discussion of limitations

Following authors' rebuttal, most concerns seem to have been addressed: additional (smaller) models tested, limitations discussed, some links to existing theoretical work made, efficiency impacts admitted and discussed.

In light of the above, the AC recommends accepting the paper, yet urges the authors to incorporate the reviewers' suggestions and additional discussions and experiments into the final version of the paper.
Decision: Accept (spotlight)