Venue: NAACL 2025

# In-Context Learning with Long-Context Models:

An In-Depth Exploration +
Footnote †: Now at Google DeepMind

Amanda Bertsch\({}^{\gamma}\)

abertsch@cs.cmu.edu

Maor Ivigi \({}^{\tau}\)

maor.ivgi@cs.tau.ac.il

Emily Xiao\({}^{\gamma}\)

emilyx@cs.cmu.edu

&Uri Alon\({}^{\gamma*}\)

urailon@cs.cmu.edu

Jonathan Berant\({}^{\tau}\)

joberant@cs.tau.ac.il

Matthew R. Gormley\({}^{\gamma}\)

mgormley@cs.cmu.edu

Graham Neubig\({}^{\gamma}\)

gneubigi@cs.cmu.edu

\({}^{\gamma}\) Carnegie Mellon University

Tel Aviv University

###### Abstract

As model context lengths continue to increase, the number of demonstrations that can be provided in-context approaches the size of entire training datasets. We study the behavior of incorrect learning (ICL) at this extreme scale on multiple datasets and models. We show that, for many datasets with large label spaces, performance continues to increase with thousands of demonstrations. We contrast this with example retrieval and finetuning: example retrieval shows excellent performance at low context lengths but has diminished gains with more demonstrations; finetuning is more data hungry than ICL but can exceed long-context ICL performance with additional data. We use the ICL setting to study several properties of both in-context learning and long-context models. We show that long-context ICL is less sensitive to random input shuffling than short-context ICL, that grouping of same-label examples negatively impacts performance, and that the performance boosts do not arise from cumulative gain from encoding many examples together. We conclude that long-context ICL can be an effective tool, and may not require long-context for encoding the demonstration set at all.1

Footnote 1: Data and code are available at [https://github.com/abertsch72/long-context-icl](https://github.com/abertsch72/long-context-icl)

## 1 Introduction

When a few examples are provided in-context, large language models can perform many tasks with reasonable accuracy. While questions remain about the exact mechanism behind this phenomena [35, 40], this paradigm of _in-context learning_ (ICL) has seen widespread adoption in both academic and industry applications, thanks to its ease of implementation, relatively small computational cost, and ability to reuse a single model across tasks.

However, most work has focused on models where the maximum number of demonstrations is severely limited by context length. As more and more methods are developed to adapt language models to extreme context lengths [6, 8], in-context learning over large quantities of data becomes a potential alternative to finetuning. The properties of ICL in this regime are not well-understood; and as the cost of inference over many thousands of tokens can be steep, the efficiency and performance tradeoff between many-shot ICL and finetuning on the same data is complex.

We conduct a systemic study of long-context in-context learning. Namely, we consider: a) the performance of prompting the base model naively, b) retrieving examples to use in-context for each test example, c) finetuning the base model (both full and parameter-efficient finetuning), and d) using models trained to adapt to longer contexts. Performance continues to increase past 2000 demonstrations (see Figure 1), approaching and sometimes _exceeding_ the performance of models finetuned on thousands of examples from the same dataset (SS 3).

We find that, as th

Figure 1: The performance increases with more demonstrations far beyond the context window of the base Llama-2. Results are on Fu et al. (2024)’s long-context finetuned Llama-2-7b model, using a context of up to 80K tokens.



in-context increases to extreme values, the behavior of ICL shifts (SS 4). In-context learning becomes less sensitive to example order, and the benefits of retrieval over using a random set of demonstrations diminishes -- allowing the use of a single set of demonstrations, encoded once through the model and cached, rather than re-encoding a custom set of demonstrations for each example. We demonstrate that long-context ICL is strongly impacted by grouping examples of the same label. We also find that the effectiveness of long-context ICL is not dependent on long-range attention in the demonstration set- encoding demonstrations with local attention and using global attention only for the test example recovers nearly the same performance (SS 5). Our work furthers the understanding of innContext learning and shows that long-context ICL is a strong alternative to retrieval and finetuning.

## 2 Experimental setup

We consider 5 classification datasets: TREC [16], TREC-fine [17], NLU [30], Banking-77 [8], and Clinic-150 [9]; and 1 generation dataset: SAMSum [10]. Table 1 contains summary statistics for each dataset, and Appendix G shows additional description for each dataset.

We compare ICL performance across several long- and short-context models, including variants of Llama-2 with 4k [16], 32k [15], and 80k [17] context windows, Mistral-7b-v0.2 [18], and Qwen 2.5-7B [9]. For more details on models, see Appendix H.

Constrained decodingFor each classification dataset, we use _constrained decoding_ to only produce valid labels as output. Note that, without constrained decoding, these models may produce invalid labels in the few-shot regimes (see Appendix D). For finetuning, we use a classification head; thus no invalid outputs can be produced.

EvaluationFollowing prior work [21, 14, 15, 16, 17], we subsample 250 examples from the test set of each dataset. We release the subsampled test set and full prediction outputs for each experiment in the project repository. We evaluate on each classification dataset with accuracy and macro-F1; as the trends for the metrics are very similar, we report accuracy (the more common metric) in the paper. We evaluate on SAMSum with BERTScore [20] and confirm that we see similar trends in ROUGE [15], as measured using the rouge_score package.2

Footnote 2: [https://pypi.org/project/rouge-score/](https://pypi.org/project/rouge-score/)

## 3 Long-context ICL

We consider four common methods for using a large dataset.

### Compared settings

Random sampling ICLWe use 10 random shuffles of the training dataset, averaging the results across these shuffles. Across models and across

\begin{table}
\begin{tabular}{l l c c c c} \hline \hline  & & & **Avg** & **Training** & \\
**Dataset** & **Domain** & **\# Labels** & **demo length** & **set size** & **Example outputs** \\ \hline TREC & questions & 6 & 22.7 & 5,452 & location / entity \\ TREC-fine & questions & 50 & 23.7 & 5,452 & abbreviation expansion / location city \\ NLU & conversational & 68 & 20.7 & 19,286 & takeaway query / iot hue light up \\ Banking-77 & financial & 77 & 27.4 & 10,003 & top up failed / lost or stolen card \\ Clinic-150 & multiple & 151 & 22.3 & 15,250 & rollover 401k / meal suggestion \\  & & & & John will buy the goat cheese Tracy \\ SAMSum & conversational & n/a & 167.9 & 14,732 & liked, milk, a couple of grainy rolls, and tissues. \\ \hline \hline \end{tabular}
\end{table}
Table 1: The datasets we consider in this work span diverse label spaces and domains. The average demonstration length is the average combined length of input, output, and formatting tokens per demonstration provided in the context window; outliers in the top 1% for demonstration length are discarded and not reflected in these statistics.

 

varying numbers of demonstrations in-context, we draw the first \(n\) examples from each shuffle. In this setting, the encoding of demonstrations can be performed once and cached.

Retrieval ICLA strong alternative for in-context learning is to retrieve a relevant subset of examples as demonstrations for each test set example. Prior work has found that, in some scenarios, retrieval of good examples can make the difference from near-zero to high test accuracy Levy et al. (2023). We considered two possible retrievers for this setting: BM25 Robertson and Zaragoza (2009) and BERTScore-Recall Gupta et al. (2023). For both, we retrieve the most relevant demonstrations by comparing the test input text to the full demonstration texts. For BM25, we remove stopwords; when doing \(k\)-shot prompting, if less than \(k\) examples are retrieved by the retriever,3 we randomly sample additional examples until we reach \(k\).

Footnote 3: This occurs when there are less than \(k\) examples with any (non-stopword) overlap with the test example.

FinetuningWe finetune Llama2-7b with a classification head on varying amounts of data from each dataset with several random seeds, and plot performance at convergence on the same held-out test data. We initialize the classification head from the parameters of the pretrained language modeling head by subsampling the values of the first token of each label; this creates a better-than-random initialization for finetuning. We perform both full finetuning and LoRA Hu et al. (2022) finetuning using this setup; for more details on the finetuning procedures, see Appendix E.

### In-context results

Scaling up ICL to many examples leads to surprisingly strong resultsFigure 1 and Table 6 show the performance of models in the ICL settings. Scaling up from 10 to 1000 demonstrations results in accuracy gains of up to _50.8_ points (and an average of 36.8 points across 5 datasets for Llama2-80k).

Longer context lessens the importance of carefully selecting in-context examplesRetrieving relevant examples for each test set example far outperforms using a randomly selected subset in the short-context regime. This is true even if the order of retrieved examples is shuffled (rather than ordered by relevance).4 However, adding additional examples does continue to slightly improve performance; this is especially surprising for the BM25 retriever because, after all examples with non-trivial lexical overlap are retrieved, remaining

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Dataset** & **Llama2** & **Llama2-32k** & **Llama2-80k** & **Mistral** & **Qwen2.5** \\ \hline Randomly selected & & & & \\ \hline TREC & **82.32**/80.52 & **93.12**/**93.12** & **90.04**/**90.04** & **87.28**/85.00 & **94.68**/94.40 \\ TREC-fine & **61.40**/**61.40** & **75.56**/75.08 & **77.20**/**77.20** & **72.68**/70.48 & **83.40**/81.24 \\ NLU & **76.88**/**76.88** & **85.04**/85.00 & **87.52**/86.92 & **86.44**/**86.44** & **88.64**/**88.64** \\ Banking-77 & **56.36**/**56.36** & **82.44**/**82.44** & **88.08**/87.96 & **86.76**/86.68 & **88.60**/87.96 \\ Clinic-150 & **60.92**/**60.92** & **84.40**/**84.40** & **89.32**/**89.32** & **90.56**/**90.56** & **93.16**/92.76 \\ SAMSum & **79.83**/**79.83** & **81.65**/81.42 & **81.17**/**81.17** & **81.78**/**81.78** & **82.01**/81.86 \\ \hline \hline BM25 Retrieval & & & & & \\ \hline TREC & **90.80**/85.64 & **94.84**/94.64 & **94.28**/92.68 & **90.80**/**90.80** & **95.60**/95.20 \\ TREC-fine & **78.80**/**78.80** & **83.88**/81.12 & **83.92**/81.36 & **80.80**/79.60 & **88.00**/**88.00** \\ NLU & **90.00**/88.40 & **89.80**/**89.80** & **89.64**/89.52 & **90.40**/89.20 & **90.40**/**90.40** \\ Banking-77 & **93.20**/92.40 & **94.32**/**94.32** & **94.00**/92.96 & **93.20**/**93.20** & **92.80**/91.60 \\ Clinic-150 & **87.60**/**87.60** & **89.84**/**89.84** & **93.76**/**93.76** & **93.20**/92.40 & **95.20**/93.20 \\ SAMSum & **79.98**/**79.98** & **81.40**/81.19 & **80.68**/**80.68** & **81.33**/**81.33** & **81.90**/**81.90** \\ \hline \hline \end{tabular}
\end{table}
Table 2: For all datasets, performance of ICL continues to increase with additional demonstrations. These results are the best accuracy (left) and accuracy at maximum data (right) for each model on the classification tasks, and the same with BERTScore for SAMSum. Bold indicates the best performance for that model/dataset pair.

 

examples are randomly selected.

While retrieval continues to outperform random selection, the importance of the selection strategy diminishes with additional examples. On some datasets (e.g. TREC, in Figure 1(b)), BERTScoreRecall outperforms BM25 for short context ICL; on other datasets (e.g. Banking-77, in Figure 1(a)), the inverse is true. But on all datasets, the performance difference between the two retrieval methods diminishes with larger \(k\), so that BM25 and BSR have nearly identical performance at long-context ICL. Because of this, we report only BM25 in the remainder of the analysis.

As the performance difference between individual retrievers diminishes, so does the performance difference between retrieval and random selection of demonstrations. On Banking-77, the dataset where retrieval is most beneficial, the performance gain from BM25 retrieval drops from 51.5 points at 1-shot ICL to 4.9 points at 1500-shot ICL. This is compelling because it is more computationally efficient (but less effective) to encode a single random set of demonstrations and cache them, rather than retrieving and re-encoding a custom set of demonstrations for each inference example. In the longest context regime we consider, using a single randomly selected set of examples is feasible; the performance penalty for doing so is never more than 5 points, and as low as 1.8 points (in 2000-shot ICL on TREC).

Long-context ICL is also effective for generation.While we primarily focus on classification tasks because of the relative ease of evaluation, we do consider SAMSum, a text summarization task, for our ICL experiments in Table 6. While the number of demonstrations possible in the same context length is much smaller, due to the increased lengths of both inputs and outputs, we observe increased performance with additional demonstrations up to at least 250-shot ICL. Retrieval seems less helpful in this setting, with retrieval sometimes underperforming random selection.

### Comparison with finetuning

While we have demonstrated that in-context learning with hundreds or thousands of examples is effective, this amount of data is also appropriate for finetuning a model. Finetuning has higher upfront cost but allows for reduced inference-time cost. We compare in-context learning with full finetuning and the popular parameter-efficient finetuning (PEFT) strategy LoRA Hu et al. (2022).

Finetuning is (slightly) more data-hungry than ICLWhen a relatively small set of examples is available, ICL generally outperforms LoRA fine-tuning on the same model.5 For most datasets, LoRA finetuning performance never exceeds long-context ICL performance even with additional examples (e.g. Figure 2(a)); however, for most datasets, full finetuning with drastically more examples than fit in-context yields the highest performance. Gen

Figure 2: Comparing three selection methods– random selection, BM25, and BERTScore-Recall (BSR) on two representative datasets. At smaller numbers of demonstrations in-context, BM25 and BSR have differing performance, and the best retrieved is dataset-specific; at larger demonstration counts, the two become indistinguishable. Both generally outperform random selection.

 

erally, the datasets with larger label spaces show the least strong finetuning performance, likely because these are more open-ended classification problems and require more data to train the classifier; on the dataset with the most labels, Clinic-150, neither LoRA finetuning nor full finetuning ever outperforms ICL at the same number of examples.

In a setting with unlimited training data available, then, finetuning is clearly advantageous over ICL for any model with a fixed maximum context length. Finetuning also offers dramatically reduced inference costs for similar performance; thus, fine-tuning on 4096 examples may still be preferable to prompting with 1000 if efficiency of inference is a major priority.6 This is because, even if demonstration encodings can be cached across inference examples, cross-attention to long context is costly.

Footnote 6: With the caveat that serving several task-specific models may be more expensive than serving one general-purpose model with customized ICL prompts; the actual best cost/efficiency tradeoff in any downstream setting is of course dependent on the needs of the deploying organization.

## 4 Properties of long-context ICL

We compare the properties of long-context ICL with the known properties of short-context ICL.7 We primarily use classification, not generation, tasks to study these properties to avoid confounding effects from the difficulty of evaluating generated texts.

Is it best to use the entire context?Prior work suggested that, for some simple tasks, providing additional input can _reduce_ performance Levy et al. (2024). However, we observe monotonically increasing performance on nearly every dataset; in cases where the performance curve begins to flatten, small variation occurs, but no significantly lower performance occurs at higher example counts. While using the full context window is computationally costly and may be unnecessary to achieve high performance on some datasets, it is minimally not harmful to performance.

Sensitivity to example orderMany models exhibit strong sensitivity to example order in-context Lu et al. (2022). We examine this by measuring the percentage of predictions that change when the input is reordered (averaged over 3 shuffles). Figure 4 shows that, while some sensitivity to order persists, this effect weakens substantially with longer context. Across all datasets, the percent of labels flipped by shuffling in 1000-shot ICL is less than _half_ the percentage flipped in 10-shot ICL.

Label sortingWe also consider an adversarial case for example ordering: we sort the examples so that examples with the same label appear together. At small numbers of examples, this has very little impact; if the average number of examples per class is low, label sorting is similar to a random sort. However, as the number of examples grows, label sorting begins to have a dramatic impact on performance. Figure 5 shows the performance of Llama2-32k on Clinic-150 with and without label sorting. As the number of examples in-context increases, the penalty for input sorting increases as

Figure 3: Comparing BM25 retrieval ICL, random selection ICL, and two types of finetuning on two representative datasets. Finetuning sometimes, but not always, exceeds ICL at high numbers of demonstrations. Note that, while retrieval ICL uses the listed number of examples in context, it assumes access to the larger test set to draw examples from Perez et al. (2021). See Appendix C for results on other datasets.

 

well; at 1169-shot ICL, label sorting decreases accuracy by 25.7 percentage points. This suggests that contextualization of examples with _different_ labels is important to performance, and that this contextualization only occurs effectively over relatively short distances in the context window.

Effectiveness for frontier modelsWe focus on 7B/8B models because of the feasibility of in-depth analysis; however, it is also useful to consider the performance of the current strongest long-context models. We evaluate Claude Sonnet 3.5 Anthropic (2024) and Llama 3.1 405B Dubey et al. (2024), and using 50 to 3000 demonstrations from Clinic-150. Due to computational cost and (in the case of Claude) the limitations of API access, we do not apply constrained decoding in these runs and use only a single run over the test set instead of 10 runs at each example count. Figure 6 shows these results. While performance increases past the typical fewshot range, at least some of this benefit comes from improved knowledge of the label space (i.e. less invalid labels generated) and performance on this task saturates quite quickly. Thus, while long-context ICL seems to be a promising direction even for frontier models, there may be diminishing returns on relatively simple tasks such as intent classification at even the 1,000-example scale.

Tasks where long context does not helpCurrently to our work, Li et al. (2024) identify a set of tasks where long context is not uniformly helpful. However, we observe that the tasks that show this trend either have near-0 performance at short demonstration lengths or also display an inverse performance trend on the short context scale (e.g. for TacRED Zhang et al. (2017), we observe that performance decreases from 1 to 10 total demonstrations; on Discovery Sileo et al. (2019), we observe that performance is near-zero at all demonstration lengths and decreases from 5 to 10 total demonstrations). While these are important failure modes of language models, we restrict our analysis to tasks without these confounding issues. In Banking-77, the one dataset that our works share, both papers observe similar trends of improved performance with additional context.

## 5 Why does long-context ICL help?

The most notable differences between long-context and short-context ICL are the additional number of demonstrations and the average number of demonstrations that each demonstration is contextualized

Figure 4: The impact of (randomly) reordering examples in-context decreases with additional demonstrations.

Figure 5: By contrast, sorting examples by label has an increasingly negative impact on performance in longer context regimes. Results on Llama2-32k with Clinic-150.

Figure 6: Performance of two frontier models on Clinic-150. Performance increases with the number of demonstrations at first, but saturates relatively early; the number of invalid labels produced continues to decline with increased demonstrations, even when accuracy plateaus.

 

with respect to_- that is, the same demonstration may have a different impact on the prediction if it is the only demonstration in-context versus if it is the 90th demonstration to appear in-context, because the demonstration will be better-contextualized.

We hypothesize that long-context ICL is primarily beneficial because of _retrieval in-context_- the idea that attention is used to select examples, rather than the model aggregating a complex decision boundary by better contextualizing across examples. This means that the primary benefit of long-context ICL is the number of demonstrations, _not_ the quality of contextualization that each individual demonstration receives. If this is the case, then there should exist some sparse attention pattern that severely restricts long-range attention _between_ demonstrations, but has limited or no impact on ICL performance.8 In all cases, the current test example can attend back to all demonstrations, in the same style as Acharya et al. (2024).

Footnote 8: Note that this is distinct from methods that overload the same positions with multiple embeddings in order to process longer contexts (e.g. Ratner et al. (2022)); here, we are not modifying any positional information, only restricting attention between demonstrations to a local context block.

**Block-sparse attention patterns** We test several previously proposed sparse attention patterns and observe that two things are necessary to enable block attention: attending to an _attention sink_Xiao et al. (2024) block that is always first in the context window; and attending to two prior local blocks Guo et al. (2024). We ablate over these decisions in Appendix F.

The pattern we identify, a small variation on Star Attention Acharya et al. (2024), is visualized in Figure 7. This attention pattern sparsifies attention between demonstrations substantially, removing almost all long-range connectivity between demonstrations in-context, without substantial impact on performance. This suggests that long-context ICL does not, in the encoding of the demonstration set, require long-range attention.

This pattern can also be used to disentangle two previously conflated factors: \(k\), the total number of demonstrations that the test example can attend back to (e.g. \(k\)-shot prompting); and \(b\), the number of examples per block in the blockwise attention pattern.9 block of examples that are contextualized together. We can fix the total number of examples in context, \(k\), and vary \(b\) by changing the attention mask; and we can fix \(b\) with a custom attention mask and vary \(k\) by adding additional blocks of examples to the context window.

Footnote 9: Note that each block attends to two blocks of local context (\(2b\) examples) and a sink block (\(b\) examples), so that the maximum number of demonstrations that the _last_ demonstration in the last block can attend to is actually \(4b-1\).

**Restricting contextualization on a fixed example set** We fix a number of examples per-block. If the block size is equal to the number of examples innoctext (\(b=k\)), this is equivalent to normal (full) attention; if the block size is \(b=1\), each example can attend only to itself and its sink/local block. Figure 8 shows results on Banking-77 as a representative example. The performance of block attention quickly approaches the performance of full attention; 95% of the performance of full attention is recovered by a block of \(50\) examples in the case of Banking-77 (and in a similar range for all datasets studied). This suggests that the ICL performance is not strongly benefiting from encoding long-range (or even medium-range) dependencies across the demonstration set. However, some short-range dependencies between examples are clearly necessary for ICL performance, as using block sizes \(b<10\) results in near-zero performance and performance increases slightly with increased size from there.

**Increasing number of examples with a fixed contextualization quality** We fix the block size and compare attending to a single block (\(k=b\)) to attending to many blocks (\(k>>b\)) in Figure 9. This comparison measures the effect of adding additional examples without introducing substantial improvements to the contextualization of each individual example.

When the context is extremely local (e.g. for banking, \(b<10\) examples), attending across many locally encoded blocks is _worse_ than attending to a single block of examples. We hypothesize that

Figure 7: Causal attention versus the blockwise pattern we apply in the following experiments. Here, each square represents a _block_ of examples, and the last square represents the test example. This pattern represents attending to the first block (the attention sink) and two local blocks.

 

this is due to inadequate contextualization of each example leading to less informative embeddings. However, after some minimal average quality of contextualization is achieved (in Figure 9, around \(b=10\)), adding more blocks of examples with the same local encoding dramatically increases performance. This supports the hypothesis that the primary performance improvement from long-context modeling is due to retrieving from more relevant examples in-context, rather than learning a better task boundary; this is supported as well by the retrieval results in Table 6, where retrieval performance at short contexts is near (though never exceeding) very-long-context ICL performance.

## 6 Related Work

Augmenting decoder-only models with long contextMany methods for extending the context of language models have been introduced in the last few years. One series of work has focused on positional embedding extrapolation strategies Peng et al. (2023); Roziere et al. (2024); Chen et al. (2023); Liu et al. (2023); Zhu et al. (2024); Xiao et al. (2024); Han et al. (2024). When extrapolating past pretraining length, models also generally benefit from additional finetuning on long-context data Xiong et al. (2023). Other methods include adding retrieval-based attention Bertsch et al. (2023); Tworkowski et al. (2023); Yen et al. (2024) or hierarchical merging of information Song et al. (2024); YU et al. (2023). The two long-context Llama variants we consider in this work are both examples of finetuning for length extrapolation.

Separately, methods for longer context for ICL have also been proposed. Parallel context windows Ratner et al. (2022) and structured prompting Hao et al. (2022) propose methods of re-using the same positional embeddings multiple times to encode more demonstrations; this is quite effective for small numbers of overlaps, albeit with diminishing returns as the number of overlapping windows increases. Li et al. (2023c) propose a new efficient attention mechanism and motivate its use using long-context ICL on a model tuned with many-shot instruction finetuning. Cho et al. (2023) propose a hybrid of ICL and linear prompting which improves beyond few-shot ICL performance.

Several works have also critiqued the efficacy of long context models. Liu et al. (2024) demonstrate that some long-context models fail to effectively use the middle of the context window; the models we use were released after this work and have generally high scores for middle-of-context retrieval. Li et al. (2023a) suggest that some long-context models are only effective at utilizing inputs that are shorter than their context window's intended supported length; we do not observe this effect strongly, though we do see saturating performance slightly before the maximum number of examples in-context for some models. Li et al. (2023b) show that many models fail at tasks that require reasoning over long dependency lengths; this is unlikely to be an issue in our setting.

Properties of in-context learningMilios et al. (2023) study ICL for many-class classification with models up to 4k context length and find that, when

Figure 8: Comparing block attention to full attention over the same set of examples with Llama2-32k on Banking-77 (i.e., fixing \(k\) and varying \(b\) along the x axis). Block attention approaches full attention (the black skyline) with relatively small block size.

Figure 9: Comparing a single block to many blocks with a fixed block size using Llama2-32k on Banking-77 (i.e., fixing \(b\) and varying \(k\)). When the block size is very small, poor contextualization appears to be the limiting factor on performance (so adding more examples with the same block size does not help); when the block size is larger, the total number of examples appears to be the limiting factor on performance (so adding more examples with the same block size is helpful).

 

retrieving demonstrations, 7b models show early performance saturation on many tasks. Our results suggest that this failure to use longer context effectively is not an inherent property of 7b models, but instead a type of shallow heuristic used by some particular models when the demonstrations are of sufficiently high quality.

Xu et al. (2023) study the impacts of ground-truth label, input distribution, and explanations on ICL performance; Bolucu et al. (2023) study the impact of example selection in a specific domain. Lin and Lee (2024) argue that ICL occurs in two modes: learning tasks and retrieving tasks, and that retrieval of similar-but-not-quite-correct tasks can explain "early ascent" behaviors where ICL performance peaks once in a fewshot regime and then performance improves again with a much higher number of examples. Similarly, Pan et al. (2023) argue for a distinction between task recognition and task learning, and suggest that task learning continues to benefit from additional examples at scale. von Oswald et al. (2023) suggest in-context learning can be viewed as gradient descent, although Deutch et al. (2024) argue against this interpretation. Hendel et al. (2023) view ICL as compressing the demonstrations into a "task vector" that maps from inputs to outputs.

Concurrently to our work, Agarwal et al. (2024) study many-shot prompting of Gemini 1.5 and show improvements from the fewshot setting across both classification and generation tasks. Our work differs in its evaluation of multiple open-source models, our comparison to finetuning the same base model, and our use of ICL as a testbed for analysis of long context behaviors.

Comparing in-context learning and finetuningMin et al. (2022a) show that models trained on fewshot learning can generalize to perform few-shot learning on new tasks; in some cases, this can outperform finetuning directly on the new task. Mosbach et al. (2023) compare finetuning to ICL more directly; they find that finetuning generally outperforms ICL with the same number of examples both in-domain and out-of-domain, when comparing 16-example ICL to finetuning on the same 16 examples. Their setting differs from ours in their choice of model (OPT) and the amount of data considered (16 for ICL, 16 or 128 for finetuning). Liu et al. (2022) find that PEFT generally outperforms ICL in their setting, where they finetune an encoder-decoder model with a language modeling objective using their T-few method and 20-70 samples. Asai et al. (2023) compare finetuning and ICL for mT5 on cross-lingual transfer and find that ICL outperforms finetuning in some, but not all, of the tasks studied. To the best of our knowledge, no prior work has considered the relative performance of finetuning and ICL in the many-shot regime with hundreds or thousands of examples in-context.

## 7 Conclusion

We have demonstrated that ICL with large demonstration sets can be surprisingly effective, and shed light on a few surprising properties in its behavior. Namely, long-context ICL exhibits a reduced dependence on example selection, relatively stable performance with respect to example order, and performance often approaching or exceeding parameter-efficient finetuning on the same data, all properties that make this an appealing option for a variety of tasks. We have also shown that long-context ICL's effectiveness is largely due to retrieval from the long context during prediction, rather than cross-attention within the large demonstration set during encoding.

Our work also highlights that our understanding of ICL remains incomplete. Though much work has studied the potential mechanisms behind ICL, these works have largely focused on simple tasks with small (\(<10\) examples) demonstration sets; as our work demonstrates that ICL's properties shift in the long context regime, more work is necessary to validate hypotheses about ICL at larger scales.

While prior work has focused on two strategies for performing inference on a new task- either fine-tuning on task-specific data or selecting a subset of that data to use in-context- our results points to a potential third paradigm: adapting the _model_ to fit as much of that data in-context as possible, caching and reusing the encoding of the long demonstration set. While finetuning with full datasets is still a powerful option if the data vastly exceeds the context length, our results suggest that long-context ICL is an effective alternative. ICLR trades finetuning-time cost for increased inference-time compute, and increasing the amount of inference-time compute by using more examples in-context is an effective strategy to improve performance. As the effectiveness and efficiency of long context models continue to increase, we believe long-context ICL will be a powerful tool for many tasks 



## 8 Limitations

Our work focuses on open-source models (and predominately on the Llama-2 family); more work is necessary to establish whether this trend holds across model families, although we are encouraged by the strong results Agarwal et al. (2024) observed for many-shot ICL on Gemini 1.5. Additionally, we do not consider other non-LoRA PEFT methodologies; it is possible that some of these methodologies may outperform in-context learning. Finally, we focus primarily on classification tasks, and care should be taken in generalizing to new tasks.

## 9 Broader impacts

Any work that studies general capabilities of language models can be used for both positive and negative applications downstream. In-context learning work is perhaps particularly vulnerable to this dual use, in part because it is a relatively accessible method, requiring far less compute than finetuning or pretraining models.

Independent of our work, Anil et al. (2024) observed that many-shot prompting can be used to jailbreak some models. We do not study or suggest the use of long-context ICL for jailbreaking in our work, but this is a potentially harmful use case of this paradigm. Outside of this risk, we do not foresee any additional harms introduced by long-context ICL methods.

We hope that our work opens up additional possibilities for people who are compute-constrained to customize models to their use cases or tasks. We also aim to carefully explore the boundary between where finetuning and long-context ICL is appropriate, to enable practitioners to make informed choices on when to apply each method.

## References

* Acharya et al. [2024] Shantanu Acharya, Fei Jia, and Boris Ginsburg. Star attention: Efficient llm inference over long sequences, 2024. URL [https://arxiv.org/abs/2411.17116](https://arxiv.org/abs/2411.17116).
* Agarwal et al. [2024] Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning, 2024.
* Anil et al. [2023] Cem Anil, Esin Durmus, Mrinnak Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, Francesco Mosconi, Rajashree Agrawal, Ryan Schaefer, Naomi Bashkansky, Samuel Svenningsen, Mike Lambert, Ansh Radhakrishnan, Carson Denison, Evan J. Hubinger, Yutnao Bai, Trenton Bricken, Timothy Maxwell, Nicholas Schiefer, Jamie Sully, and Alex Hernandez. Many-shot jailbreaking, 2024.
* Anchropic [2024] Anthropic. Introducing claude 3.5 sonnet, 2024.
* Asai et al. [2023] Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirizi. Buffet: Benchmarking large language models for few-shot cross-lingual transfer, 2023.
* Bertsch et al. [2023] Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew Gormley. Unlimiformer: Long-range transformers with unlimited length input. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2023.
* Biderman et al. [2023] Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Gregory Anthony, Shiranshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. In _Thirty-seventh Conference on Neural Information Processing Systems_, 2023.
* Bolucu et al. [2023] Necva Bolucu, Maciej Rybinski, and Stephen Wan. impact of sample selection on in-context learning for entity extraction from scientific writing. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2023_, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.338.
* Casanueva et al. [2020] Inigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. Efficient intent detection with dual sentence encoders. In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_, pp. 38-45, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.nl94convrai-1.5.
* Chen et al. [2023] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023.
* Cho et al. [2023] Hyungsoo Cho, Hyuhng Joon Kim, Junyeob Kim, Sangwoo Lee, Sang goo Lee, Kang Min Yoo, and Taeuk Kim. Prompt-augmented linear probing: Scaling beyond the limit of few-shot in-context learners, 2023.
* Deepmind [2024] Google Deepmind. Our next-generation model: Gemini 1.5, 2024.
* Deutch et al. [2024] Gilad Deutch, Nadav Magar, Tomer Bar Natan, and Guy Dar. In-context learning and gradient descent revisited, 2024.
* Dubey et al. [2021]Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela

Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelein Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Brion, Binn Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damiem Lidonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiabou, Dhruv Choudhary, Drhruy Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregory Miadon, Guan Pang, Guillem Cucurrell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imand Arrieta Ibarra, Isabel Kluumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnston, Joshua Saxe, Junteng Jia, Kalyan Vasudenden Alwala, Kartikeyas Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Khitig Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madahan, Lubo Malca, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Namam Goyal, Naples Torabi, Nikolay Bashlykov, Nikday Bogoychev, Niladi Chatterji, Olivier Duchene, Onur Celebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prijwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Purxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romanian Sauvester, Ronnie Poliddoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabsappa, Sanjay Singh, Sean Bell, Seohywn Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Sheng Wan, Shrutti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihalyo, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerker, Vincent Gongquet, Virginie Do, Vish Vogei, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyn Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashes Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zachier Delpricer Coudert, Zheng Yan, Zhengrixing Chen, Zoe Papakirops, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shainfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anand Yunus, Andrei Luppu, Andres Alvarado, Andrew Cuples, Andreev Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandi, Annise Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hanrock, Bram Wasth, Brandon Sperone, Brian Stojkovic, Brian Gamido, Brit Montaluvo, Carl Parker, Carly Burton, Catalina Weja, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtnhofer, Damon Civin, Daan Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Akkins, David Xu, Davide Tesuglinge, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcuate, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Goggini, Francisco Guzzan, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damali, jgor Molbygo, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carrill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junijie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Kandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kwslal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Limon Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martyna's Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneasly, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Muqiet JubernImoso, Mo Metanet, Mohammad Rastegarni, Munish Bansal, Nandhnii Santhanam, Natascha Parks, Notasha White, Navyata Bau 

Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Pareketh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghodham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkan Hogan, Robin Battes, Rocky Wang, Rohan Maheswarri, Russ Howes, Ruuty Rinot, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaum Lindsay, Shaun Lindsay, Sheng Feng, Shengaho Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuquiang Zhang, Sinong Wang, Sencha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Syoudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varnu Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Victor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Kide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yongsi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024.
* Fu et al. (2024) Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannanehi Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context, 2024.
* Gliwa et al. (2019) Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_. Association for Computational Linguistics, 2019. doi: 10.18653/v1/d19-5409. URL [http://dx.doi.org/10.18653/v1/D19-5409](http://dx.doi.org/10.18653/v1/D19-5409).
* Guo et al. (2024) Junxian Guo, Haotian Tang, Shang Yang, Zhekazi Zhang, ZhiJian Liu, and Song Han. Block Sparse Attention. [https://github.com/mit-han-lab/Block-Sparse-Attention](https://github.com/mit-han-lab/Block-Sparse-Attention), 2024.
* Gupta et al. (2023) Shivanshu Gupta, Matt Gardner, and Sameer Singh. Coverage-based example selection for in-context learning, 2023. URL [https://arxiv.org/abs/2305.14907](https://arxiv.org/abs/2305.14907).
* Han et al. (2024) Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Im-infinite: Zero-shot extreme length generalization for large language models, 2024.
* Han et al. (2022) Zhixiong Han, Yaru Hao, Li Dong, Yutao Sun, and Furu Wei. Prototypical calibration for few-shot learning of language models, 2022.
* Hao et al. (2022) Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in-context learning to 1, 000 examples. _ArXiv preprint_, abs/2212.06713, 2022.
* Hendel et al. (2023) Roee Hendel, Mor Geva, and Amir Globerson. Context learning creates task vectors. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2023_, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. findings-emnlp.624.
* Hovy et al. (2001) Eduard Hovy, Laurie Gerber, Ulf Hermjkabb, Chin-Yew Lin, and Deepak Ravichandran. Toward semantics-based answer pinpointing. In _Proceedings of the First International Conference on Human Language Technology Research_, 2001.
* Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022.
* Ivgi et al. (2023) Maor Ivgi, Uri Shaham, and Jonathan Berant. Efficient long-text understanding with short-text models. _Transactions of the Association for Computational Linguistics_, 2023. doi: 10.1162/tacl_a_00547.
* Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guilaimume Lample, Lucile Saulinier, Leibon Renard Lavaud, Marie-Anne Lacthaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. Mistral 7b, 2023.
* Kalajdzievski (2023) Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. _ArXiv preprint_, abs/2312.03732, 2023.
* Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. An evaluation dataset for intent classification and out-of-scope prediction. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pp. 1311-1316, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1131.
* Levy et al. (2019) Itay Levy, Ben Bogin, and Jonathan Berant. Diverse demonstrations improve in-context compositional generalization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Proceedings of the 61st 

Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.78.
* Levy et al. [2024] Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models, 2024.
* Li et al. [2023a] Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source LLMs truly promise? In _NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following_, 2023a.
* Li et al. [2023b] Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts?, 2023b.
* Li et al. [2023c] Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jun Zhang, Zhiyong Wu, and Lingpeng Kong. Innovect learning with many demonstration examples, 2023c.
* Li et al. [2024] Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context llms struggle with long in-context learning, 2024.
* Li and Roth [2002] Xin Li and Dan Roth. Learning question classifiers. In _COLING 2002: The 19th International Conference on Computational Linguistics_, 2002.
* Lin [2004] Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL [https://aclanthology.org/W04-1013/](https://aclanthology.org/W04-1013/).
* Lin and Lee [2024] Ziqian Lin and Kangwook Lee. Dual operating modes of in-context learning, 2024.
* Liu et al. [2023] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context, 2023.
* Liu et al. [2022] Haokun Liu, Derek Tam, Muqueeth Mohammed, Jay Mohtha, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), _Advances in Neural Information Processing Systems_, 2022.
* Liu et al. [2022] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Parangjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the Middle: How Language Models Use Long Contexts. _Transactions of the Association for Computational Linguistics_, 2024. ISSN 2307-387X. doi: 10.1162/tacl_a_00638.
* Lu et al. [2022] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pp. 8086-8098, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.556.
* Mangrukkar et al. [2022] Sourab Mangrukkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. [https://github.com/huggingface/peft](https://github.com/huggingface/peft), 2022.
* Milios et al. [2023] Aristides Milios, Siva Reddy, and Dzmitry Bahdanau. In-context learning for text classification with many labels. In Dieuwke Hurkes, Vernas Dankers, Khuyagbaatar Batstwen, Koustuv Sinha, Amirhoossein Kazemnejad, Christos Christodoulopoulos, Ryan Cotterell, and Elia Brunni (eds.), _Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP_, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.genbench-1.14.
* Min et al. [2022a] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetaICL: Learning to learn in context. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pp. 2791-2809, Seattle, United States, 2022a. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.201.
* Min et al. [2022b] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pp. 11048-11064, Abu Dhabi, United Arab Emirates, 2022b. Association for Computational Linguistics.
* Mosbach et al. [2023] Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few-shot finetuning vs. in-context learning: A fair comparison and evaluation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Findings of the Association for Computational Linguistics: ACL 2023_, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.779.
* Pan et al. [2023] Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning "learns" in-context: Disentangling task recognition and task learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Findings of the Association for Computational Linguistics: ACL 2023_, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.527.
* Peng et al. [2023] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarm: Efficient context window extension of large language models, 2023.
* Perez et al. [2013] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. In 

Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds), _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_, pp. 11054-11070, 2021.
* Ratner et al. [2022] Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud D. Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. Parallel context windows for large language models. In _Annual Meeting of the Association for Computational Linguistics_, 2022.
* Robertson and Zaragoza [2009] Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. _Foundations and Trends in Information Retrieval_, 2009. doi: 10.1561/1500000019.
* Roziere et al. [2024] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jeremy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafori, Wenhan Xiong, Alexandre Defossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code, 2024.
* Sclar et al. [2023] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting, 2023.
* Sileo et al. [2019]Damien Sileo, Tim Van De Cruys, Camille Pradel, and Philippe Muller. Mining discourse markers for unsupervised sentence representation learning. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pp. 3477-3486, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. URL [https://www.aclweb.org/anthology/N19-1351](https://www.aclweb.org/anthology/N19-1351).
* Song et al. [2024] Woomin Song, Seunghyuk Oh, Sangwoo Mo, Jaeyhung Kim, Sukmin Yun, Jung-Woo Ha, and Jinwoo Shin. Hierarchical context merging: Better long context understanding for pre-trained LLMs. In _The Twelfth International Conference on Learning Representations_, 2024.
* Team [2024] Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL [https://qwenlm.github.io/blog/qwen2.5/](https://qwenlm.github.io/blog/qwen2.5/).
* and fine-tuning for llama-2 models with together api, 2023.
* Touvron et al. [2019] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhahiari, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esibbu, Jude Fernandes, Jeremy Fu, Wenvin Fu, Brian Fuller, Cynthia Gao, Vedantaj Goswanit, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Innan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihalyo, Pushkar Mishra, Igor Molbyg, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashid Rungta, Kalyan Saladi, Alan Scheletten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Purxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanicambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Sciolam. Llama 2: Open foundation and fine-tuned chat models, 2023.
* Tworkowski et al. [2023]Szymon Tworkowski, Konrad Staniszewski, Mikol aj Pacek, Yuhuai Wu, Henrykh Michalewski, and Piotr Mil os. Focused transformer: Contrastive training for context scaling. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2023.
* Oswald et al. [2023] Johannes von Oswald, Eywind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent, 2023.
* Wolf et al. [2020]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shliefer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Pu, Canven Xu, Teven Le Scao, Sylvian Gugger, Mariana Dzme, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pp. 38-45, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6.
* Xiao et al. [2024] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024.
* Xiugkanii et al. [2019] Pawel Swietojanski Xingkun Liu, Arash Eshghi and Verena Rieser. Benchmarking natural language understanding services for building conversational agents. In _Proceedings of the Tenth International Workshop on Spoken Dialogue Systems Technology (IWSDS)_, Ortigia, Siracusa (SR), Italy, 2019. Springer.
* Xiong et al. [2023] Wenhan Xiong, Jingyu Liu, Igor Molybo, Hejia Zhang, Prajiwal Bhargava, Rui Hou, Louis Martin, Rashi Runeta, Karthik Abinux Sankararam, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshifting Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao 

Ma. Effective long-context scaling of foundation models, 2023.
* Xu et al. (2023) Paiheng Xu, Fuxiao Liu, Zongxia Li, and Hyemi Song. Towards understanding in-context learning with contrastive demonstrations and saliency maps, 2023.
* Yen et al. (2024) Howard Yen, Tianyu Gao, and Danqi Chen. Long-context language modeling with parallel context encoding, 2024.
* Ying & Thomas (2022) Cecilia Ying and Stephen Thomas. Label errors in BANKING77. In _Proceedings of the Third Workshop on Insights from Negative Results in NLP_, pp. 139-143, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.insights-1.19.
* YU et al. (2023) LILI YU, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. MEGABYTE: Predicting million-byte sequences with multiscale transformers. In _Thirty-seventh Conference on Neural Information Processing Systems_, 2023.
* Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert, 2020. URL [https://arxiv.org/abs/1904.09675](https://arxiv.org/abs/1904.09675).
* Zhang et al. (2017) Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position-aware attention and supervised data improve slot filling. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pp. 35-45, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1004.
* Zhao et al. (2017) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang (eds.), _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_, volume 139 of _Proceedings of Machine Learning Research_, pp. 12697-12706. PMLR, 2021.
* Zhu et al. (2024) Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. Pose: Efficient context window extension of lms via positional skipwise training, 2024 

Saturation

One metric we are interested in is the point where the model performance _saturates_, which we define informally as the point where adding more examples is unlikely to meaningfully improve performance. More formally, we define the saturation point as the smallest number of examples tested such that performance reaches 95% of the model's maximum performance.

Saturation points vary by dataset.We define saturation as the first point at which performance reaches 95% of the model's maximum performance on that dataset. Table 3 shows the number of examples at saturation and the maximum number of examples that fit in the context window for each model. For datasets with larger label spaces, saturation generally occurs later; Banking-77 and Clinic-150 do not saturate within the context window of Llama2 (4096 tokens, which represents between 100-162 in-context examples for these datasets). In the longer-context regime, saturation points generally occur slightly later on Llama2-80k, but in both models occur before the model's maximum context length.

This suggests two things. First, given a fixed model, it is often not necessary to use the full context length to extract high performance from that model. Second, current models do not make use of the full potential of ICL; models often saturate in performance before the maximum number of examples, despite longer-context versions revealing that further performance improvements are possible.

The number of classes has some impact on saturation point- but is not fully explanatory.Our results show datasets with more classes benefit from more demonstrations in-context, on average, before saturation. This is to be expected, as the expected number of demonstrations necessary before seeing the correct label increases with the number of total label classes. To test if this is an intrinsic property of these datasets, or truly linked only to the number of label classes, we construct subsets of two high-label-space datasets, Banking-77 and Clinic-150, by randomly selecting half of the labels to exclude from the dataset. These subsets remain in the same domain, but with a smaller label space; if saturation is tied to the number of examples, then this should move the saturation point. Note that this is distinct from _combining_ labels (e.g. TREC vs TREC-FINE), as combining finegrained labels into general labels makes the classification task simpler. It's still possible that the subset chosen is a simpler task (e.g. by removing one of a pair of frequently confused labels); to moderate the effect of this change, we average results over 3 randomly chosen subsets.

Table 4 compares the saturation point of the full- and half-label-space runs. For the datasets with the most number of labels, halving the number of labels also reduces the amount of examples that are useful before saturation, albeit not by half. However, the trend is less clear for the tasks with fewer labels; in some cases, reducing the label space actually _increases_ the number of demonstrations before saturation. While the size of the label space clearly has some impact on the saturation point, more investigation is necessary to identify other factors impacting this behavior.

\begin{table}
\begin{tabular}{l l l l l} \hline \hline
* (100) & 500 (838) & 750 (1750) & 500 (860) \\ Clinic-150 &
- (145) & 750 (1169) & 1000 (2000) & 750 (1212) \\ \hline \hline \end{tabular}
\end{table}
Table 3: We measure the saturation point as the point at which the model reaches 95% of its maximum accuracy on the dataset; “-” in a column indicates that the maximum performance is achieved by using the full context window. The number in parenthesis represents the maximum number of examples that fit in the context window. As the label space of the dataset increases (from top to bottom row), so does the number of examples that can be used before saturation.
 



## Appendix B Using ICL as a testbed for long-context model properties

In this section, we use in-context learning as a testbed to examine several properties of long-context models.

How do long-context models perform in the short-context regime?Llama2-32k and Llama2-80k are finetuned variants of Llama2-7b, adapted for longer contexts. We evaluate how these models perform relative to the base model in short-context tasks (e.g. ICL using less than 4096 tokens of demonstrations) by testing whether the difference in performance is statistically significant (2-sided t-test, \(p<0.05\)). Performance is generally similar, with some areas of slight improvement from the base model; Figure 10 shows full results. We observe degradation in performance in some settings for Llama2-32k, highlighting the importance of testing for behavior regression when finetuning for additional capabilities.

Input utilizationWe analyze the performance of all methods on a naturalistic needle-in-the-haystack [21, 22] style test. If the model is effectively using the context, then it should be able to exactly recover the label for any example it has seen in-context. Note that, while a model trained on some set of data is not uniformly capable of exact copying from that training data [19], in nearly all of our finetuning runs, the model fits the training data with 100% accuracy.

We examine this behavior by selecting the same set of examples to use in-context and in evaluation; all models should then be able to achieve 100% accuracy. Table 5 shows the results; while all models achieve very high accuracy on the copied data, no model is able to uniformly copy correctly from the input. Surprisingly, performance improves slightly with additional demonstrations for most models, possibly due to additional specification of the task.

Figure 10: Short-context behavior of long-context models. Each model’s performance is compared to the performance of the base model it was finetuned from, on the same amount of data. **Red** represents significantly worse performance; **blue** represents significantly better performance (\(p<0.05\)).





## Appendix C Full ICL results across datasets

For space, we show 1-2 representative datasets for each point of analysis in the paper. In this appendix, we present results across all classification datasets.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline

- \\ Llama2-32k & 80.0 & 95.0 & 97.0 & 96.6 & 98.4 & 98.5 & 98.5 & **98.9** \\ Llama2-80k & 90.0 & 94.0 & 95.0 & 98.2 & **98.5** & 98.3 & 98.0 & 97.9 \\ \end{tabular}
\end{table}
Table 5: Copying behavior given the test examples in the context window. Results are averaged over Banking-77 and Clinic-150; bold indicates the best performance for that model.




### Random selection ICL across all models

Figure 11: Performance of random-selection ICL across all models for each classification dataset. Performance continues to increase with additional examples in-context.





### Retrieval ICL across all models

Figure 12: Performance of retrieval-based ICL across all models for each classification dataset. Short-context performance here is higher than for random-selection, but performance continues to improve with more examples until a saturation point, where performance flattens out.





### Comparing retrieval, random selection, and finetuning

Figure 13: Performance of retrieval-based ICL, random-selection ICL, and finetuning across 5 datasets. At small example counts, ICL outperforms finetuning; when several thousand examples are used, finetuning outperforms ICL in some datasets.



Constrained decoding details

To perform constrained decoding, following Ratner et al. (2022), we add a large constant to all logits at each decoding step that could result in a valid label being generated. This strategy is generally not possible for API-access models, or when the label space is not fully known in advance. To determine whether the same trends hold without the use of constrained decoding, we evaluated the Llama-2 family models with and without constrained decoding. Figure 14 shows the comparison; while performance is lower, especially in the higher-label-space datasets, the general trends hold. In Banking77, it appears that performance begins to saturate slightly earlier with unconstrained decoding; we hypothesize that this may be due to more specialized language used in banking domain labels.

Figure 14: Comparison of constrained decoding (diamond, square, and triangle datapoints on the graph) with the unconstrained decoding variants (dot-like datapoints of the same colors).

 

Finetuning

We performed finetuning with HuggingFace transformers (Wolf et al., 2020). To perform parameter efficient finetuning (PEFT), we used the pft (Mangrulkar et al., 2022) package (version 0.9.0). We finetune the model for 30 epochs, evaluating it every epoch on the test set, and ultimately choosing the checkpoint with the highest test accuracy. We note that using the test set to perform model selection presents an unfair advantage to finetuning (compared to ICL) and may not be truly indicative of the generalization error. However, doing so provides the advantage of being both comparable to ICL in terms of the data used, as well as giving an upper bound on the true generalization accuracy of the finetuned model, further emphasizing any observed efficacy gap between it and ICL.

Initialization of the classification headWhile in our default setting we initialize the classification head from the pretrained LM head, subsampled at the representation of the first token in each label, we investigate the efficacy of this approach by contrasting with a randomly initialized classification head. Figure 15 shows that while in the few-shot regime, this approach has significant advantage, as the training set grows in size the difference shrinks to become negligible. In no case was random initialization better than this approach.

Hyperparameter tuningTo remain comparable in terms of compute efficient finetuning, we did not perform extensive hyper-parameter tuning per task, and instead experimented with a good global setting on a single dataset (Banking-77). Specifically, we experimented with different learning rates, different LoRA ranks (\(r\)) and \(\alpha\)(Hu et al., 2022) and also tried applying RSlora (Kalajdzievski, 2023) which sets the scaling factor to \(\frac{\alpha}{\sqrt{r}}\) as some evidence suggest it can outperform the original method. Figure 16 summarizes the results, depicting average test accuracy against training examples with different settings.

Ultimately, we found that using HuggingFace's (Wolf et al., 2020) default parameters of \(r=8\), \(\alpha=32\), LoRA dropout of \(0.1\) and a learning rate of \(1e-3\) to work best. In all cases, we used batch sizes of 32 and weight decay of \(0.01\).

It is possible that methods specialized for finetuning in small-data regimes, such as T-few (Liu et al., 2022), might close the gap between ICL and PEFT in the small-data regimes. We did not consider T-few in our analysis because of its additional pretraining stage, which imposes substantial additional cost, and because T-few was developed with a focus on encoder-decoder models and we consider only decoder-only models in our setting 

Figure 16: Comparing hyperparameters when finetuning a PEFT llama-2.7b model on Banking-77. Averaged (best) test accuracy over 3 random seeds. (a) Comparing our fixed LoRA configurations to the best alternative configuration (at each scale) we tried. (b) Comparing different learning rates.

Figure 15: Comparing initialization methods of the classification head when finetuning a PEFT llama-2.7b model. Averaged (best) test accuracy over 5 random seeds. Initialization with \(lm\) subsamples the pretrained language-modeling head at the first token of the target label, while random samples random weights.

 

Block attention patterns

We aim to construct a modified attention pattern where demonstrations can only attend to a limited block of local demonstrations.

The most naive strategy is to only allow attention within a local block (Figure 16(b)). Other variants relax the mask to allow attention to the first block, which acts as an attention sink Xiao et al. (2024) (Figure 16(d)), or allowing attention back to the immediately prior local block(s) (as seen as part of Acharya et al. (2024))(Figure 16(c)).

As a test of these methods, we perform block attention with each attention mask on Banking-77 with Llama2-80k, with the block size \(b=50\) and 500 demonstrations in-context.

Neither a sink block nor local attention alone recovers close to full attention performance, although both are substantial improvements over the naive strategy. The best performance is achieved with both sink and local blocks; although one local block is sufficient to recover 94% of the performance of the full attention method, we choose two local blocks to minimize performance degradation. As a result, we use the attention pattern in Figure 16(f) for the experiments in the paper.

## Appendix F

\begin{table}
\begin{tabular}{l l c c c} \hline \hline

- & 80.04 \\ \hline Figure 16(b) & 50 & 0 & 0 & 18.72 \\ Figure 16(c) & 50 & 0 & 1 & 51.16 \\ Figure 16(d) & 50 & 1 & 0 & 62.15 \\ Figure 16(e) & 50 & 1 & 1 & 75.12 \\ Figure 16(f) & 50 & 1 & 2 & **77.51** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Comparison of attention strategies and performance.

Figure 16: Representative examples of attention masks we considered. In each diagram, the squares represent a block of \(b\) examples each, _not_ individual tokens.

 

Prompt formatting and examples from datasets

As a demonstration of the datasets, we provide an example of 3-shot prompting for each dataset with the prompt formatting we used (and with examples drawn from the training set of each dataset).

Prompt formatting and instruction phrasing can have significant impact on performance [25]; we keep the formatting consistent with prior work [17], with prefixes for the input and output for each exemplar. Because we use predominately non-instruction-tuned models, we do not add an additional instruction or system prompt.

### Trec

**TREC**[40, 69] is a question classification dataset with two granularities of labels. We refer to the 6-label coarse classification as TREC.

### TREC-fine

We refer to TREC's 50-label finegrained classification as TREC-fine.

Figure 19: Example 3-shot prompt for TREC.

Figure 20: The label distribution of TREC-fine.

 



### Nlu

**NLU**(Xingkun Liu & Rieser, 2019) is a 68-way intent classification dataset in the conversational domain. The original paper evaluates on 64 of the intents; we use all 68. The data is licensed under CC BY 4.0.

### Banking-77

Banking-77Casanueva et al. (2020) is a 77-way intent classification task in the financial domain. Although the accuracy of some labels in BANKING77 has been criticized Ying Thomas (2022), we

Figure 23: Example 3-shot prompt for NLU.

Figure 22: The label distribution of NLU.



report results here on the original dataset for consistency with prior work. The data is licensed under CC BY 4.0.

_query: How long will my payment be pending? intent: pending card payment == query: My physical card is not working intent: card not working == query: i cant seem to activate card intent: activate my card == query: I didn't set up a direct debit payment on my account. intent:_

### Clinic-150

**Clinic-150**[Larson et al., 2019] is a 151-way, multi-domain intent classification task; examples are either labeled with an intent from one of 10 domains or with the catch-all "out-of-scope" label. We use the "plus" train split from the original paper, which adds additional "out-of-scope" examples to the dataset. The data is licensed under CC BY 3.0.

Figure 26: The label distribution of Clinic-150. It is balanced except for the “out of scope” label, which has additional data points in the split we use.

Figure 24: The label distribution of Banking-77.

 



### SAMSum

**SAMSum**[16] is a text message summarization corpus; both the text conversations and summaries were written by linguists. The conversations can be two or more participants, and may contain emojis, emoticons, or tags that indicate the use of an image or gif; the summaries are generally 1-3 sentences long. The data is licensed under CC BY-NC-ND 4.0.

Figure 27: Example 3-shot prompt for Clinic-150.



_Conversation: Zara: <file_sif> Zara: something terrible happened :( Stanley: what? You OK? Zara: yes, I'm fine! I went to the swimming pool Zara: and lost the earring you gave me for birthday :( Stanley: oh Jesus, I thought something bad happened to you! Summary: Zara has lost the earring, which Stanley gave her for birthday, when she was at the swimming pool. == Conversation: Rose: hey congratulations for the baby boy! i wish him all the health and happiness in life. Ela: thank you so much for the wishes. Rose: your welcome.. so how he and you? all ok Ela: yes everything is great thanks Rose: will come to see you and the baby boy Ela: Sure will wait to see you.. Summary: Ela just gave birth to a boy. Rose will visit them shortly. == Conversation: Kim: Are you going to the conference in SF? Jenny: I should, I know, it would be good for my career Jeff: no, not so much, I think it's bullshit that it's so important Simon: is it? Jeff: sure, the whole net-working thing doesn't really matter, I think Jeff: nobody offers you a job at a conference Jeff: and it costs so much to fly to SF Kim: I would like to go also to see what's going on in the field Kim: to meet people, see new trends, ideas Kim: I think it's important for an academic Jeff: this may be true, if you can afford Kim: the flight is about $500, right? Simon: true Jeff: and then more money for accommodation Jeff: it can easily pile up to $2000 Klim: you're quite right, unfortunately Jeff: because it also doesn't make sense to fly to California for 3 days Jeff: it would be also extremely disturbing, with the jet lag etc. Kim: you're so right :( Jeff: so think about it first Summary: Jeff is not going to the conference in SF. The flight is expensive. == Conversation: Amanda: I baked cookies. Do you want some? Jerry: Sure! Amanda: I'll bring you tomorrow :-) Summary:

Figure 28: Example 3-shot prompt for SAMSum.

 

Additional details

### Models selected

The majority of the analysis in the paper concerns these five models:

1. **Llama2**(Touvron et al., 2023) is a decoder-only model trained with a 4096 context length. We use the non-instruct (non-chat) variant because it is more commonly used as a base model for long-context finetuning and because we observed very similar performance between the chat and non-chat variants in our initial experiments.
2. **Llama2-32k**(TogetherAI, 2023) is a version of Llama-2-7b finetuned by TogetherAI for a 32k context window. We use the non-instruct version.
3. **Llama2-80k**(Fu et al., 2024) is a version of Llama-2-7b finetuned with 80k context and a carefully designed long-document data mixture.
4. **Mistral-7b-v0.2**(Jiang et al., 2023). is the instruct version of Mistral-v0.2 (the non-instruct model is not publicly available). The trained context length of Mistral-7B-Instruct-v0.2 is 32k tokens.
5. **Qwen2.5-7B**(Team, 2024) is a decoder-only model with multilingual support. The trained context length of Qwen2.5-7B is 128k tokens.

While all of these models can extrapolate to inputs longer than their trained context length, we restrict the lengths of inputs to fit within the trained context length; this represents the best case performance without the additional confound of the extrapolation strategy.

### Computational cost

For our finetuning experiments, we used approximately 50 GPU-days of compute on 80GB A100 GPUs. The computational cost of the in-context learning experiments was approximately 75 GPU-days, primarily using 48GB A6000 GPUs. All experiments were run on local clusters (i.e. not using cloud providers), with the exception of the frontier models experiments, which used Together AI's API (for Llama 3.1-405B) and the Anthropic API (for Claude).