Memorization in In-Context Learning

## Abstract

Abstract In-context learning (ICL) has proven to be an effective strategy for improving the performance of large language models (LLMs) with no additional training. However, the exact mechanism behind these performance improvements remains unclear. This study is the first to show how ICL surfaces memorized training data and to explore the correlation between this memorization and performance across various ICL regimes: zero-shot, few-shot, and many-shot. Our most notable findings include: (1) ICL significantly surfaces memorization compared to zero-shot learning in most cases; (2) demonstrations, without their labels, are the most effective element in surfacing memorization; (3) ICL improves performance when the surfaced memorization in few-shot regimes reaches a high level (about 40%); and (4) there is a very strong correlation between performance and memorization in ICL when it outperforms zero-shot learning. Overall, our study uncovers a hidden phenomenon—memorization—at the core of ICL, raising an important question: to what extent do LLMs truly generalize from demonstrations in ICL, and how much of their success is due to memorization?

## 1 Introduction

In-context learning (ICL) has emerged as a powerful method for improving the performance of large language models (LLMs) without extra training (Brown et al., ). This method involves including a few task-specific examples, known as demonstrations, within the input prompt, which enables the LLM to infer the target task and generate improved responses. With long-context LLMs (OpenAI, ; Anil et al., ; Lu, , inter alia), ICL has evolved to incorporate hundreds or even thousands of demonstrations, leading to greater performance improvements (Bertsch et al., ; Agarwal et al., ; Zhang et al., ).
However, despite its widespread use and straightforward nature, the underlying principles of ICL and its performance improvement capabilities remain unclear (Min et al., ; von Oswald et al., ; Razeghi et al., , inter alia).

In this work, we further study the inner workings of ICL by investigating the previously unexplored relationship between ICL and memorization of training data in LLMs, and how this memorization correlates with performance. In particular, to show how ICL surfaces memorization, we replace the *learning component* (target variable) in ICL with a text completion task which is based solely on *memorization*. To achieve this, we adapt the data contamination detection method proposed by Golchin & Surdeanu (). This method aims to replicate dataset instances through memorization to verify their presence in the training data. The process begins by splitting a dataset instance into two random-length segments. The initial segment and the corresponding label of the dataset instance are integrated into the input prompt, instructing the LLM to generate the subsequent segment. The generated completion is then evaluated against the original subsequent segment and categorized as an exact, near-exact, or inexact match, with the first two indicating memorization. To implement this for ICL, we use the same strategy to replicate dataset instances but with a tweak. We include a few pairs of initial and subsequent segments from different dataset instances along with their labels in the input prompt as demonstrations. Specifically, a demonstration includes (1) a segment pair consisting of an initial and subsequent segment, and (2) a label. We then quantify the memorization of training data across various regimes (i.e., zero-shot, few-shot, and many-shot) by counting the number of exact and near-exact matches. Figure [1](#S1.F1) shows prompts for an illustrative two-shot ICL to replicate a dataset instance.

Figure: Figure 1: Illustrative examples of a two-shot ICL prompt for replicating instances from NLI (left) and classification (right) tasks. Note that, in our actual experiments, we use $k$-shot ICL, where $k=\{0,25,50,100,200\}$. All colored segments, except the green one, form the input prompt. Specifically, the gray segments indicate the instruction, the red segments display the two demonstrations, the blue segments correspond to the dataset instance being replicated, and the green segment exhibits the generated completion by the underlying LLM (GPT-4) for the subsequent segment of the dataset instance being replicated. For both examples, the generated completions are exact matches.

We examine memorization across various $k$-shot scenarios, where $k=\{0,25,50,100,200\}$.
Here, demonstrations for smaller $k$ values are subsets of those used for larger $k$ values.
We categorize our experiments into three regimes based on $k$ values: zero-shot for $k=0$, few-shot for $k=\{25,50\}$, and many-shot for $k=\{100,200\}$.
Each regime is analyzed under different settings to identify the key element contributing the most to memorization in ICL.
These elements include instruction, segment pairs, and their respective labels, with the latter two forming the demonstrations.
We vary the amount of in-context information in each setting by selectively including/excluding these elements in the prompt. First, we include all elements, i.e., an instruction containing dataset-specific information (i.e., dataset and partition name) and segment pairs with their labels to establish an upper bound for memorization. Figure [1](#S1.F1) depicts this setting. We then incrementally remove each element and measure memorization to identify the impact of each element on memorization. Second, we exclude the instruction (gray parts in Figure [1](#S1.F1)). Third, we exclude the instruction and labels, leaving only segment pairs (red parts in Figure [1](#S1.F1), excluding labels).
Finally, we evaluate the correlation between memorization from all settings and performance.

The primary contributions of this paper are as follows:

- 1.
For the first time, we study the relationship between ICL and memorization in LLMs.
- 2.
Our study identifies the key element contributing to surfacing memorization in ICL.
- 3.
We explore the correlation between memorization and performance in ICL.
- 4.
By analyzing the surfaced memorization levels in ICL, we identify cases where ICL either succeeds or fails to outperform zero-shot learning.

We made several important observations, summarized in the key findings below:(^1^11See Section [5](#S5) for a comprehensive list of observations.)

- 1.
ICL with only a few demonstrations (e.g., 25 shots) surfaces significant memorization in most cases for data that is part of the training set.
- 2.
Segment pairs—demonstrations without their labels—are the key element contributing to surfacing memorization in ICL.
- 3.
There is a very strong correlation between performance and surfaced memorization by ICL when it improves performance compared to zero-shot learning.
- 4.
ICL outperforms zero-shot learning when the surfaced memorization in few-shot regimes is significant, specifically when the memorization level is about 40% or higher.
- 5.
Similar to memorization, data contamination has a greater presence in few-shot and many-shot regimes. However, its impact on performance depends on the level of surfaced memorization/contamination. If memorization/contamination is significant (around 40% or higher), data contamination inflates performance compared to zero-shot learning; otherwise, it does not.
- 6.
As the number of demonstrations increases, even though the surfaced memorization by ICL remains relatively constant at a high level in most many-shot regimes, near-exact matches gradually become exact matches, making memorization more explicit.
- 7.
Evaluating performance on memorized and non-memorized instances in ICL reveals that the performance on memorized instances is consistently higher than non-memorized instances across nearly all regimes, from zero-shot to many-shot.
- 8.
Consistent with the findings of Carlini et al. () on memorization in language models, we discovered that memorization significantly increases with the number of tokens of context used to prompt the model. However, our experiments further this finding by showing that these tokens can be from individual instances, not just tokens from a single instance.

## 2 Terminology

Before discussing our methodology, we establish specific terminology for clarity and consistency.

Element: Throughout this paper, we use the term “element” to refer to any of the following: instruction, segment pairs, or labels. In our experiments, we assess the impact of each element on memorization in ICL.

Setting: One of the key objectives of this study is to identify the main element influencing memorization in ICL. For this, we experiment with three settings, each varying by the amount of in-context information in the input prompt. Thus, the term “setting” refers to the amount of information incorporated into the input prompt in our experiments. We detail our settings in Subsection [3.3](#S3.SS3).

Regime: Contrary to settings, we define regimes based on the values of $k$ in $k$-shot scenarios. Hence, the term “regime” emphasizes the number of demonstrations (shots) used in the input prompt. We elaborate on these regimes in Subsection [3.5](#S3.SS5).

Demonstration: In the scope of ICL, various terms describe task-specific examples ($k$-shot) included in the input prompt. For clarity, we use only “demonstrations” and “shots” to refer to these examples. As previously noted, each demonstration in our experiments comprises (1) a segment pair with an initial and subsequent segment, and (2) a label. Therefore, when we mention demonstrations without labels, we only refer to segment pairs.

## 3 Approach

### 3.1 Detecting and Quantifying Memorization: General Approach

Detecting Memorization. To detect memorization, we employ the method proposed by Golchin & Surdeanu (), originally designed to detect data contamination in LLMs. This technique verifies if instances from a particular dataset partition (e.g., test set) were included in the training data by replicating them through memorization. We specifically use the “guided instruction” from this work. First, dataset instances are split into two random-length segments. Then, the LLM is prompted with the guided instruction, which includes dataset-specific details (i.e., dataset and partition name), the initial segment of an instance, and the label if available, asking the LLM to complete the subsequent segment. The similarity between the generated completion and the original subsequent segment is then evaluated to determine if the dataset instance was part of the training data or not.

Evaluating Memorization. As mentioned, the generated completion is evaluated against the original subsequent segment to determine how closely they match. Golchin & Surdeanu () proposed three categories for this evaluation:

- •
Exact Match: The generated completion exactly matches the original subsequent segment.
- •
Near-Exact Match: The completion, while not identical, shows considerable overlap and maintains significant semantic and structural similarity to the original subsequent segment.
- •
Inexact Match: The generated completion is completely different from the original subsequent segment.

According to Golchin & Surdeanu (), both exact and near-exact matches indicate memorization and confirm the presence of dataset instances in the training data. A few examples of exact and near-exact matches are shown in Table [1](#S3.T1).

Quantifying Memorization. Golchin & Surdeanu () used GPT-4 with few-shot ICL to classify generated completions into one of the aforementioned match categories. In particular, this classifier incorporates a few human-annotated examples of exact and near-exact matches in the prompt as references and automatically compares the generated completions to their original subsequent segments. Their results showed this evaluation method achieves high accuracy (92%–100%) in aligning with human evaluations for categorizing generated completions as exact, near-exact, or inexact matches. The prompt used in this method is illustrated in Figure [5](#A2.F5) in Appendix [A](#A1).

Table: Table 1: Examples of near-exact matches becoming exact matches across various ICL regimes. Each prompt indicates the number of demonstrations (shots) with a {k shots} placeholder to save space. For clarity, examples are provided in the full information setting. In WNLI and DBpedia datasets, near-exact matches from 25-shot become exact matches at 50-shot. Similarly, in the TREC dataset, the transition from near-exact to exact match occurs when moving from 25-shot to 100-shot.

### 3.2 Detecting and Quantifying Memorization: Adapted Approach for In-Context Learning

As discussed in Subsection [3.1](#S3.SS1), we use the method of Golchin & Surdeanu () to detect and quantify memorization in ICL. The rationale behind this choice is to fulfill three primary needs. First, this method relies solely on memorization for detecting data contamination, making it suitable for examining memorization in our study. Second, its simplicity allows easy adaptation to various ICL regimes by simply adding $k$ demonstrations to the input prompt. Third, since each demonstration consists of multiple elements, we can assess each element’s contribution to memorization in ICL by removing each element one at a time.
In the following, we first describe how we adjust the original method by Golchin & Surdeanu () to detect memorization in ICL and detail the procedure for quantifying it.

Detecting Memorization in In-Context Learning.
We employ the guided instruction and adapt the strategy of splitting dataset instances for $k$-shot scenarios. This involves integrating $k$ pairs of initial and subsequent segments from $k$ distinct dataset instances into the input prompt, including their associated labels. This setting meets the three primary needs. First, it closely mirrors standard ICL, allowing us to accurately measure memorization with all elements involved. Second, similar to standard few-shot and many-shot ICL, we can vary the number of demonstrations in the input prompt to study how memorization changes across different ICL regimes.
Third, by quantifying memorization in different ICL settings—such as with and without labels—we can evaluate their roles in revealing memorization in ICL, identify the key factor affecting memorization in ICL.
Examples of demonstrations and their integration into the replication process to study memorization in ICL are shown in Figure [1](#S1.F1).

Evaluating Memorization in In-Context Learning. Following Golchin & Surdeanu (), we employ GPT-4 with few-shot ICL to classify generated completions as exact, near-exact, or inexact matches, adhering to the same definitions. While their findings showed this evaluation strategy achieves high accuracy in matching evaluations from human judgments, we conduct an additional human evaluation on top of the GPT-4 evaluation to guarantee optimal accuracy in our findings. This is important as our conclusions significantly rely on the number of detected exact and near-exact matches. We detail our human evaluation process in Section [4](#S4), under Human Evaluation.

Quantifying Memorization in In-Context Learning. Following Golchin & Surdeanu (), we consider both exact and near-exact matches as indicators of memorization. We quantify memorization levels across various ICL regimes by counting the number of detected exact and near-exact matches, expressing them as percentages of the total number of dataset instances in the question. Further, in Section [5](#S5), we show that using exact and near-exact matches to measure memorization is a valid approach since near-exact matches can turn into exact matches when demonstrations increase across ICL regimes. Examples of these transitions are presented in Table [1](#S3.T1).

### 3.3 Identifying Key Element in Memorization in In-context Learning

Our experiments involve three distinct settings, all aiming at quantifying memorization but differing in the amount of information included in the input prompt. This helps us measure the amount of memorization in ICL regimes based on the information provided by each element and identify the key element in the process. As shown in Figure [1](#S1.F1) and discussed in Section [2](#S2), the input prompt is composed of two main parts: the instruction, which contains dataset-specific details, and demonstrations, which include segment pairs and their respective labels. We combine these three elements—instruction, segment pairs, and labels—in different ways to create three unique settings with varying amounts of in-context information. Below, we detail each setting.

(1) Full Information. This setting maximizes in-context information by including all three elements: instruction, segment pairs, and labels. Figure [1](#S1.F1) illustrates this setting. This setting contains more information than standard ICL, as it includes dataset-specific details not typically mentioned in standard ICL. We use it to establish an upper bound for the highest possible amount of memorization that can be surfaced in ICL regimes. By comparing the impact of each element against this maximum, we can determine which element most significantly influences memorization in ICL.

(2) Segment Pairs and Labels. In this setting, we exclude the instruction containing dataset-specific information and include only segment pairs and labels. To illustrate this setting, it omits the gray parts in Figure [1](#S1.F1) and includes only the red parts. This setting is closest to standard ICL, although standard ICL usually includes an instruction for executing the target task, which is an additional part not present in this setting. However, since this instruction lacks relevant information that could impact memorization, its impact on memorization is either zero or negligible. Moreover, as we will see in Section [5](#S5), even instruction with dataset-specific information (used in the previous setting) has minimal impact on memorization in ICL regimes.

(3) Only Segment Pairs. We further remove elements from the input prompt and include only segment pairs, excluding the instruction and labels. While the previous setting examines the combined effect of segment pairs and labels on memorization in ICL, this setting shows their individual contributions. By comparing the amount of surfaced memorization in this setting with the one that includes both segment pairs and labels, as well as the full information setting, we can assess how much memorization in ICL is due to segment pairs alone versus labels. This helps identify the primary element driving memorization across ICL regimes.

### 3.4 Performance and Memorization in In-Context Learning

As the primary goal of employing ICL is to enhance downstream performance, we explore the connection between the underlying memorization in ICL and performance. We calculate the performance of the samples for which we assess memorization, and analyze the correlation between performance and memorization across our three settings using the Pearson correlation (Pearson, ). In addition, we separately evaluate performance for memorized and non-memorized instances across ICL regimes to gain further insight into the relationship between performance and memorization in ICL. In this context, following the definitions provided in Subsection [3.1](#S3.SS1), instances that are replicated exactly or nearly exactly are considered memorized, while those replicated inexactly are considered non-memorized. Note that, for performance measurement, we use standard $k$-shot ICL, which includes an instruction to perform the downstream task with $k$ demonstrations and labels.

### 3.5 Selection of In-Context Learning Regimes

We work with five $k$-shot ICL regimes across our three settings, where $k=\{0,25,50,100,200\}$, covering all ICL categories: zero-shot, few-shot, and many-shot. Specifically, we define zero-shot regimes when $k=0$, few-shot regimes when $k=\{25,50\}$, and many-shot regimes when $k=\{100,200\}$. In our experiments, to assess the impact of increasing demonstrations on memorization and performance, we progressively increase the number of demonstrations, ensuring that larger regimes include all demonstrations from the smaller ones. For example, the 100-shot ICL includes 50 demonstrations from the 50-shot ICL, which itself includes 25 demonstrations from the 25-shot ICL.

### 3.6 Selection of Models

To meet the goals of our study, the LLMs must possess specific properties to be selected for our experiments. First, they need to be highly performant, with strong steerability and controlled generation capabilities, so we can effectively quantify memorization using their outputs. Less performant models may keep memorization internal by not explicitly emitting memorized data, or generate unstructured outputs that make detecting memorized data intangible. Second, as we extend our experiments to many-shot regimes, the LLMs must support long contexts to accommodate our largest many-shot regime with 200 demonstrations across all datasets. Third, the candidate LLMs must have been trained on an array of datasets. This is essential for observing how memorization evolves across different ICL regimes through instance replication. Clearly, without this criterion, studying memorization is unfeasible.

### 3.7 Selection of Datasets

In line with the settings outlined in Subsection [3.3](#S3.SS3), the datasets for our study must fulfill certain criteria.
First, the datasets must be part of the training corpora for the LLMs used in our study, ensuring that their instances can be replicated through memorization.
Second, to evaluate the impact of labels on memorization in ICL regimes, we need datasets with labeled samples. Third, these datasets should have a complex label space or be challenging enough for LLMs, allowing us to observe performance change across different ICL regimes and explore its correlation with memorization. Fourth, the sample length must be limited to a few dozen words/tokens to fit within the input context length of LLMs for all datasets, handling up to 200 demonstrations in our largest many-shot regime.

## 4 Experimental Setup

Model. Per the criteria detailed in Subsection [3.6](#S3.SS6) for selecting models, we conducted a pilot study to determine which existing LLMs fulfill all requirements. We initially selected a set of long-context, high-performing LLMs, including GPT-4 (OpenAI, ), GPT-4o (OpenAI, ), Gemini 1.5 Pro (Anil et al., ; Reid et al., ), and Claude 3.5 Sonnet (Anthropic, ). Our pilot study found that GPT-4o and Gemini 1.5 Pro struggled with controlled generations, particularly in many-shot regimes, and Claude 3.5 Sonnet was unable to perform our tasks due to strict safety filters preventing the generation of copyrighted content—in our case, replicating dataset instances. Among the models evaluated, only GPT-4 showed the ability to produce controlled outputs. Also, GPT-4 automatically satisfied the final criterion, having been shown to be trained on several datasets (Golchin & Surdeanu, ). Thus, we selected GPT-4 with 32k context length for all our experiments.

In our experiments, we use GPT-4 for three main tasks: measuring memorization, computing performance, and evaluating generated completions as exact, near-exact, or inexact matches. For all these three tasks, we access GPT-4 through the Azure OpenAI API. Specifically, we use the gpt-4-0613-32k snapshot for the first two tasks and the gpt-4-0613 snapshot for evaluation. To promote deterministic generations, we set the temperature to zero in all our experiments. Additionally, we limit the maximum completion lengths to 100 tokens for measuring memorization and 10 tokens for both computing performance and evaluating generated completions. For performance assessment, we repeat our experiments three times and report the average results.

Data. Based on the criteria listed in Subsection [3.7](#S3.SS7) for selecting datasets, we use four label-based datasets from two tasks: natural language inference (NLI) and classification. Although all criteria for selecting datasets can be independently verified, the first criterion must be verified in relation to the chosen model—here, GPT-4. To ensure our datasets were part of the training corpora of our LLM, we conducted a pilot study using proposed strategies for detecting data contamination in fully black-box LLMs (Golchin & Surdeanu, ; ).
Based on this, we selected the following datasets: WNLI (Wang et al., ), RTE (Wang et al., ), TREC (Hovy et al., ), and DBpedia (Wang et al., ).(^2^22More details on datasets can be found in Appendix [B](#A2).) The first two datasets are for NLI, while the latter two are for classification.
Consistent with prior work (Golchin & Surdeanu, ; ; Bertsch et al., ; Zhao et al., ; Lu et al., ; Han et al., ; Ratner et al., ), to control costs and work with a manageable sample size, we subsample 200 instances from the train split of each dataset, evenly distributed by labels, to study both memorization and performance in all our experiments. For measuring memorization, we create pairs of random-length segments for dataset instances by randomly deriving the initial segment from 60% to 80% of each instance’s length, based on the white space count.

Demonstrations. Our preparation process for demonstrations closely follows the method used for dataset instances. Specifically, we randomly subsample 200 demonstrations from each dataset’s train set, ensuring an even label distribution to prevent majority label bias in ICL (Zhao et al., ). These subsampled demonstrations are then split into two random-length segments, with the initial segment containing 60% to 80% of the instance’s length, based on the white space count. Finally, these 200 segment pairs along with their labels constitute our 200-shot ICL, with smaller $k$-shot ICLs created by further subsampling this set.

Regarding the order of demonstrations, while the order matters in few-shot regimes (Lu et al., ), its impact diminishes in many-shot regimes (Bertsch et al., ). To reduce this effect in few-shot regimes and ensure our findings are order-independent, we present demonstrations in a random order within the input prompt for our experiments on both memorization and performance. However, demonstrations retain the same order when studying memorization and performance for the same $k$-shot ICL. For example, in a 25-shot ICL, the order of demonstrations is random, but this order remains consistent when examining memorization and performance.

Human Evaluation. Golchin & Surdeanu () proposed a classifier based on GPT-4 with few-shot ICL to evaluate generated completions, achieving high accuracy (92%–100%) in matching human judgments. To enhance this, we add an extra human evaluation layer. This step is beneficial as our findings heavily rely on the number of exact and near-exact matches detected in the replication process. This ensures no tolerance for mislabeled completions. After human evaluation, only 5% of the labeled completions by GPT-4 were adjusted, all of which were borderline cases between near-exact and inexact matches. This confirms the high accuracy reported by Golchin & Surdeanu () for assessing completions by GPT-4 with few-shot ICL, aligning with its performance range.

## 5 Results and Discussion

In the following, we first discuss our results on quantifying memorization across various ICL regimes in our three settings. Then, we explore the correlation between this quantified memorization and performance. Finally, we compare our insights on memorization in ICL with previous studies, each of which reported specific characteristics of ICL.

### 5.1 Results on Quantifying Memorization

Figures [2](#S5.F2) and [3](#S5.F3) present a series of plots that quantify memorization across various ICL regimes in our three settings: (1) full information, (2) segment pairs and labels, and (3) only segment pairs.
Note that, for most discussions on ICL memorization, we refer to Figure [2](#S5.F2). All our observations are primarily based on memorization quantified by both exact and near-exact matches (left-hand plots) in this figure, unless we specifically mention exact matches only (right-hand plots).
We use the first setting only when comparing with the maximum memorization. We use the second setting when discussing memorization in ICL, as it closely resembles standard ICL. Lastly, we use the third setting when identifying the key element contributing the most to memorization in ICL.

We draw eight observations regarding memorization in ICL:

Observation 1: ICL significantly surfaces memorization compared to zero-shot learning. In zero-shot regimes, memorization ranges from 11% to 16.50%. This increases to 18% to 63.50% in few-shot regimes and further to 24% to 75% in many-shot regimes, more than doubling the amount in zero-shot regimes.

Observation 2: In terms of memorization behavior, providing only a few demonstrations (e.g., 25 shots) sharply increases memorization in ICL for most datasets. While this increase continues for larger ICL in some datasets, it levels off in others. For example, for the WNLI and RTE datasets, memorization increases up to 75% and 24% at 200-shot ICL. However, for the TREC and DBpedia datasets, memorization remains at around 40% and 53%, respectively, after the 25-shot ICL.

Observation 3: Memorization tends to remain stable across many-shot regimes for most datasets. However, within these regimes, memorization becomes more explicit as near-exact matches gradually transform into exact matches, as shown in Figure [3](#S5.F3). This highlights the importance of near-exact matches in quantifying memorization, as they indeed indicate memorization. Table [1](#S3.T1) provides examples of near-exact matches turning into exact matches as the number of demonstrations increases.

Observation 4: The memorization pattern remains the same when quantified using both exact and near-exact matches, and when using only exact matches, across all settings. This further supports the idea that near-exact matches are indicative of memorization in ICL and generally in LLMs.

Observation 5: The amount of surfaced memorization in the setting with segment pairs and labels reaches its maximum—equivalent to the full information setting—as soon as a few demonstrations (25 shots and onward) are provided in the input prompt. In other words, the key difference between the full information setting and the setting with segment pairs and labels lies in the zero-shot regimes. In fact, the dataset-specific information is overshadowed by the information from segment pairs and labels (or demonstrations) in the input prompt in terms of contributing to the memorization in ICL. Therefore, the amount of memorization surfaced by segment pairs and labels is at its maximum.

Observation 6: Individual instances of the same context can significantly increase memorization. This is supported by viewing demonstrations as individual dataset instances within the same dataset (context) that contribute to memorization. This complements the findings of Carlini et al. () that memorization significantly increases with the tokens of context used to prompt the model. We discuss this in more detail in Subsection [5.3](#S5.SS3), where we compare our observations with other studies.

Observation 7: Comparing memorization levels across three settings shows that maximum memorization in ICL can be achieved even when only segment pairs are included in the input prompt. This indicates that demonstrations without their respective labels, i.e., the segment pairs alone, are the key element contributing to memorization in ICL.

Figure: (1) Full Information

To clarify, the WNLI dataset is not an exception to this observation, although it experiences a slightly larger decrease (e.g., 22.50% in 200-shot ICL) compared to other datasets. In fact, we discovered that in the WNLI dataset, multiple sentence pairs share sentence 1 with different labels. For instance, for “Bill passed the gameboy to John because his turn was next.” as sentence 1, there are different options for sentence 2, such as “John’s turn was next.” and “Bill’s turn was next.”, with distinct labels—entailment and not entailment, respectively. Hence, when the model is prompted with only sentence 1 to generate sentence 2, the completion could be either of these options. However, our criteria for exact and near-exact matches require both semantic and structural similarity. If the completion does not semantically or structurally match the original sentence 2, even if it matches the other sentence 2 with a different label, it is not counted as an exact or near-exact match.
Dismissing these alternative completions as exact or near-exact matches aligns with prior work by Carlini et al. (), where they adopted the same approach.
We found several cases of this occurring in the WNLI dataset, and the number of such cases exactly corresponds with the drop observed for the WNLI dataset in the setting containing only segment pairs.

Observation 8: Similar to memorization, the presence of data contamination varies under different regimes. Contamination is more evident in few-shot and many-shot regimes compared to zero-shot.

Figure: (1) Full Information

### 5.2 Results on Performance and Memorization

Figure [4](#S5.F4) shows a series of plots comparing performance (left-hand plots) and memorization (right-hand plots) across different ICL regimes in all three settings. Accordingly, Tables [2](#S5.T2) and [3](#S5.T3) list the Pearson correlation coefficients between overall performance and memorization, with memorization quantified using both exact and near-exact matches, and only exact matches, respectively.

We present four observations about the connection between performance and memorization in ICL:

Observation 1: ICL outperforms zero-shot learning when the surfaced memorization in few-shot regimes is substantial, reaching around 40% or higher.

Observation 2: Performance on memorized instances is consistently higher than on non-memorized instances across nearly all settings, from zero-shot to many-shot regimes.

Figure: WNLI

Observation 3: As indicated in Tables [2](#S5.T2) and [3](#S5.T3), when providing demonstrations in ICL leads to performance improvement compared to zero-shot learning, this is highly correlated with memorization. Specifically, Pearson coefficients provide very strong evidence of this correlation.

Observation 4: As shown in the right-hand plots of Figure [4](#S5.F4), data contamination (or memorization) has a greater presence in few-shot/many-shot regimes than in zero-shot. However, its impact on performance varies, as indicated by the left-hand plots.
Specifically, when the level of data contamination is significant—around 40% or higher—it inflates performance; otherwise, it does not.

### 5.3 Comparing Our Observations with Previous Studies

In a nutshell, our observations align well with previous research on ICL and memorization alone in language models. Beyond confirming previous work, our findings on memorization in ICL and its correlation with performance offer novel and deeper insights into previously reported characteristics.

We discuss several studies that have provided notable insights into ICL and memorization alone:

Brown et al. (): They showed that larger models benefit more from ICL in terms of performance improvement. This is consistent with our observations. We discovered a very strong correlation between memorization and performance when ICL improves performance, and as Carlini et al. () reported, memorization significantly increases with model size.

Razeghi et al. (): They found a strong correlation between improved performance in ICL and term frequency for instances with terms that are more prevalent in the training data. This aligns with our observations. As previously noted, we found that there is a very strong correlation between memorization and performance when ICL enhances performance, and as Carlini et al. () showed, memorization significantly increases with the number of times an instance is duplicated in training data.

Min et al. (): They showed that labels do not contribute to performance in ICL, i.e., randomly replacing labels in ICL barely hurts performance. This matches our observations. We observed that demonstrations alone, without labels, are the most effective in surfacing memorization in ICL, and there is a very strong correlation between this memorization and improved performance in ICL.

Carlini et al. (): They found that memorization significantly increases with the number of tokens of context used to prompt the model. Our results on memorization closely match this finding. However, we extend this finding by noting that tokens from individual instances can also be considered part of the tokens of context, not necessarily tokens from a single instance. This is evident in our ICL regimes, where individual instances contributed to more memorization being surfaced.

## 6 Related Work

In-Context Learning. Brown et al. () first introduced ICL to enhance performance in LLMs without additional training costs by including a few demonstrations in the input prompt. Despite its simplicity, the internal mechanisms of ICL are not yet well understood.
Several studies explored ICL from various perspectives: Lu et al. () examined the impact of the order of demonstrations, Bölücü et al. () investigated the effect of example selection, Zhao et al. () explored biases in ICL, such as label, recency, and common token biases, Li et al. () studied the influence of input distribution and explanations, and Min et al. () looked into the role of labels and found that randomly replacing labels does not harm performance while others showed that this is not true for all tasks and models (Yoo et al., ; Kossen et al., ; Lin & Lee, ).
Different perspectives were employed to better understand ICL: Hendel et al. () viewed ICL as compressing the training set into a single task vector that modulates the transformer to produce the output, while von Oswald et al. () interpreted ICL as gradient descent, a view refuted by Deutch et al. ().
Some research efforts focused solely on maximizing ICL performance through different paradigms: several studies extremely increased the number of demonstrations in the input prompt (Bertsch et al., ; Agarwal et al., ; Zhang et al., ; Milios et al., ; Anil et al., ), Min et al. () fine-tuned language models to perform ICL, and Zhao et al. () used prompt engineering to enhance ICL.
More recently, a few studies highlighted additional capabilities of ICL beyond its performance on standard datasets and benchmarks, such as performing regression (Vacareanu et al., ), kNN (Agarwal et al., ; Dinh et al., ), and jailbreaking (Anil et al., ).

Memorization. Memorization is a well-studied topic in the context of language models (Carlini et al., ; ; Song & Shmatikov, ; Zhang et al., ; McCoy et al., ; Song et al., ). While some studies aimed to verify the presence of memorization (Henderson et al., ; Thakkar et al., ; Thomas et al., ; Carlini et al., ; Golchin & Surdeanu, ), others attempt to quantify it (Carlini et al., ; ). Quantifying memorization is typically handled using membership inference attack (Shokri et al., ; Yeom et al., ) to reproduce training data from models (Carlini et al., ; ; Golchin & Surdeanu, ). This quantification is conducted for several reasons, including showcasing potential privacy risks (Nasr et al., ; Biderman et al., ; Lukas et al., ), addressing copyright infringement (Grynbaum & Mac, ; Karamolegkou et al., ), and generating factual information (Li et al., ; Tay et al., ; AlKhamissi et al., ; Petroni et al., ; Haviv et al., ). On the other hand, several works proposed methodologies to mitigate memorization (Lee et al., ) and its implications (Kandpal et al., ; Meeus et al., ; Wei et al., ; Wang et al., ).

To the best of our knowledge, there is no existing work studying memorization in ICL and its correlation with performance.

## 7 Conclusion

We studied how in-context learning (ICL) surfaces memorized training data in large language models (LLMs). We quantified this memorization and identified the most effective element in surfacing it. We also explored the correlation between this memorization and performance in ICL. Among our many observations, the key findings are: (1) ICL significantly surfaces memorization compared to zero-shot learning in most cases, with demonstrations—excluding their labels—being the most influential element; and (2) there is a very strong correlation between performance and memorization when ICL outperforms zero-shot learning.
Overall, our findings highlight memorization as a new factor impacting ICL.
While our research offers a deeper understanding of ICL, it also presents new challenges, particularly regarding how much LLMs truly learn from demonstrations in ICL versus how much their success is due to memorization.

## Appendix A Evaluation Prompt

Figure [5](#A2.F5) illustrates the few-shot ICL prompt used to evaluate the LLM-generated completion against the original subsequent segment of a dataset instance.
In this prompt, the “reference text” represents the original subsequent segment, while the “candidate text” refers to the LLM-generated completion.
We provide GPT-4 with one exact match and three near-exact matches, all pre-annotated by human evaluators. These examples help GPT-4 distinguish between near-exact and inexact matches, aligning its evaluation with human judgment.

## Appendix B Detailed Descriptions of Datasets

Winograd Natural Language Inference (WNLI) Dataset. The WNLI dataset is a benchmark for assessing natural language understanding, focusing specifically on coreference resolution and pronoun disambiguation within context. Originating from the Winograd Schema Challenge (Levesque et al., ), the dataset includes sentence pairs where a pronoun should be resolved to determine if it refers to the same entity as in the previous sentence.
Although the training set is balanced between two classes, the test set is not.
The dataset consists of 635 training examples, 71 validation examples, and 146 testing examples.

Recognizing Textual Entailment (RTE) Dataset. The RTE dataset stems from a series of annual textual entailment challenges. This dataset comprises data from four different editions: RTE1 (Dagan et al., ), RTE2 (Haim et al., ), RTE3 (Giampiccolo et al., ), and RTE5 (Bentivogli et al., ). The examples in these datasets were mainly developed using texts from news articles and Wikipedia. To ensure uniformity, the datasets were adjusted into a two-class format. In cases where datasets originally had three classes, the “neutral” and “contradiction” categories were merged into a single “not entailment” class. The combined RTE dataset includes 2,490 examples for training, 277 examples for validation, and 3,000 examples for testing.

Text REtrieval Conference (TREC) Dataset. This dataset, created by the National Institute of Standards and Technology, is designed for question classification. There are two levels of label granularity: coarse and fine. The coarse labels consist of six categories, while the fine labels include 50 categories. The average sentence length is 10 words, with a vocabulary size of 8,700. The data is collected from four sources: 4,500 English questions published by Hovy et al. (), approximately 500 manually constructed questions for rare classes, 894 questions from TREC 8 (Voorhees & Harman, ) and TREC 9 (Voorhees & Harman, ), and 500 questions from TREC 10, used as the test set. These questions were manually labeled, with 5,500 labeled questions in the training set and another 500 in the test set.

DBpedia Ontology Dataset. The DBpedia dataset is a collaborative community effort to extract structured information from Wikipedia (Lehmann et al., ). The dataset comprises 14 distinct classes selected from DBpedia 2014. For each of these classes, 40,000 training samples and 5,000 testing samples were randomly selected. Each entry in the dataset includes the title and abstract of a Wikipedia article.

Figure: Figure 5: An illustration of the few-shot ICL prompt used for classifying generated completions into exact, near-exact, or inexact matches using GPT-4. In this illustration, examples 1 through 4 form the fixed part of the input prompt, while example 5 is replaced with a new reference text (original subsequent segment of a dataset instance) and candidate text (LLM-generated completion) for evaluation. Example 1 is an exact match. Example 2 is a near-exact match where the candidate text has substantial overlap with the reference text but includes extra details. Examples 3 and 4 also show near-exact matches, where the candidate text is both semantically and structurally similar to the reference text.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. How are non-memorized and memorized instances determined in Figure 3?
1. What is the relationship between the number of shots (k) in downstream ICL and the number of shots used in memorization experiments? The two tasks seem to serve different tasks with different purposes.
soundness: 3
strengths: This paper proposes a novel approach to quantify the level of memorization within ICL, contributing to a study in understanding the behavior of LLMs.
summary: The paper investigates the role of memorization in in-context learning (ICL) for LLMs. It shows that ICL significantly surfaces memorized training data, particularly when demonstrations without labels are used, and it establishes a strong correlation between memorization and improved model performance. The study finds that memorization levels increase with more demonstrations, with surfaced memorization reaching up to 75% in many-shot settings. Moreover, performance on memorized instances consistently surpasses that on non-memorized instances, raising questions about how much of ICL's success is due to true learning versus memorization.
weaknesses: 1. The paper presents findings that are already well-known in the field of machine learning: models perform better when they memorize training data, which is often referred to as "data leakage." It is unclear whether the primary motivation of this study is to investigate the relationship between ICL and memorization or to propose a method for quantifying memorization.

1. The paper claims to explore the "correlation between memorization and performance on downstream tasks" (lines 13-14). However, the Pearson correlation coefficient appears to be computed based on match scores across different k-shot settings. The memorization and downstream ICL tasks have different formats and purposes, which makes the correlation analysis questionable. Additionally, the negative correlation for the RTE dataset contrasts with the positive correlations for other benchmarks, casting doubt on the overall claim of correlation.

1. The authors do not provide an explanation for the observed experimental results. For example, the six observations drawn from Figure 2 are stated as mere observations without discussing the underlying reasons or contributing factors behind the results.

1. The comparison of different k values in evaluating memorization in Figure 2 is not adequately justified. The importance of the number of demonstrations as a variable for quantifying memorization remains unclear.

1. The use of 25-shot and 50-shot as examples of few-shot scenarios is too much. I expect that few-shot settings refer to 3-shot to 5-shot examples. Zero-shot results do not clearly indicate memorization level, as exact match criteria might be influenced by formatting issues rather than actual memorization. True few-shot settings, such as 3-shot, could provide a better output by constraining the output format, allowing for more precise evaluation.

1. Only GPT-4 is evaluated in the experiments. Including more LLMs in the experiments would provide stronger support for the paper's claims and offer broader insights.

1. The paper's presentation could be improved in several areas:
    * The repeated use of the phrase "memorization in ICL" is ambiguous, as it could imply that LLMs are memorizing the demonstrations instead of the training data. Since the study aims to explore memorization in LLMs using ICL as a tool, it would be clearer to use "memorization by ICL."
    * Figure 2 is intended to compare different settings, but the use of multiple curves for various datasets in a single subfigure makes it challenging to interpret comparisons. The authors should consider placing curves for different settings but the same dataset in each subfigure for clarity.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1.	How would the memorization-performance correlation change if tested on larger, more diverse datasets, such as those in professional domains?
2.	Could future work incorporate real-time memorization detection to dynamically assess memorization levels in longer contexts?
3.	How does ICL performance vary between tasks that rely heavily on memorized information (e.g., factual knowledge) versus tasks that require more abstract reasoning or generalization?
4.	Is there a threshold where memorization starts to negatively impact performance, particularly if memorized information conflicts with task requirements?
5.	Would alternative architectures that prioritize generalization over memorization yield lower memorization rates in ICL while maintaining strong performance?
6.	How might the findings differ if fine-tuned LLMs with domain-specific training data are tested, where memorization may play a more prominent role?
soundness: 3
strengths: 1.	The paper is the first to systematically examine the relationship between ICL and memorization in LLMs, providing new insights into how memorized knowledge influences ICL performance.
2.	The study uses a detailed approach to measure memorization across multiple settings (full information, segment pairs and labels, and only segment pairs), allowing for a granular analysis of which prompt elements drive memorization in ICL.
3.	The paper demonstrates a robust correlation between memorization and improved performance in downstream tasks, highlighting memorization as a previously under-explored factor in ICL success.
4.	The methodology and results are clearly presented, making the findings accessible and useful for future research on optimizing ICL methods and evaluating LLM generalization.
summary: This paper examines the role of memorization in in-context learning (ICL) for large language models (LLMs), investigating how ICL surfaces memorized training data and analyzing its impact on downstream task performance across zero-shot, few-shot, and many-shot regimes. The authors propose a method to measure memorization by testing if model-generated completions match known dataset instances. They find that ICL significantly increases memorization compared to zero-shot, with the amount of memorized information correlating strongly with performance improvement. This study highlights memorization as a key factor impacting ICL effectiveness, prompting questions about the extent to which LLMs generalize from demonstrations versus relying on memorized knowledge.
weaknesses: 1.	While the experiments are thorough, they are conducted in relatively simple datasets, limiting the paper’s ability to generalize findings to more complex, real-world tasks (e.g., legal, medical datasets).
2.	The study does not address potential challenges in handling longer contexts, which are often needed in real-world applications and may limit the practicality of the proposed memorization detection method in large-scale LLMs.
3.	While the paper successfully identifies memorization as a factor, it does not provide a concrete analysis of how much ICL performance improvement is due to memorization versus actual generalization, leaving this as an open question.
4.	The experiments rely solely on GPT-4, limiting the generalizability of the findings to other LLMs. The authors could strengthen their conclusions by evaluating memorization across a range of models with varying training data and architectures.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: Use additional models, it's hard to take a study that only uses a single closed-source model seriously.

I need additional justification for why these experiments tell us about how ICL functions. To me they tell me that in cases where there is dataset contamination ICL performs better, but this is to be expected. What about when there is no dataset contamination and ICL still works, how does this work? This is the more interesting question to me.
soundness: 3
strengths: The paper is very well written. It is clear and easily to follow, and the experimental setup is also very intuitive. The mechanism by which ICL works is a very relevant and important question.
summary: The paper seeks to quantify the correlation between LLM memorization of specific datapoints in well-known datasets and the ICL performance on these dataset subsets. They examine several datasets, selecting datapoints from these and using an existing protocol from another work to quantify how many datapoints the LLM has memorized, testing memorization in few, many, and zero-shot regimes. They then correlate this to the performance using ICL on these exact datapoints, but overall in aggregate, rather than on a per-datapoint level. Authors then draw a set of observations from this experiment and conclude that the results raise the question of if ICL works by memorization.
weaknesses: The paper only experiments on GPT-4. Authors claim that it is the only LLM that fulfills their criteria but this is somewhat hard to believe, especially given the existence of long-context open-source models. The authors claim that they do not have the resources to run these experiments on e.g. llama3 or some other long-context open-source model that fulfills their criteria, but I believe it would strengthen the paper considerably to have more than a single model for testing.

It is not clear what authors provide beyond confirming existing works in the area. It is not justified adequately why these experiments they perform give us any new information about how ICL fuctions.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Can the authors provide more detailed insights into how these findings could influence practical model training or deployment strategies? How do the authors envision their methodology being adapted or expanded to cover more diverse models and tasks to verify the generalizability of the findings?
soundness: 2
strengths: Relevant Topic: The topic of memorization in LLMs is timely and relevant, given the increasing reliance on these models for various tasks without full retraining.

Experimental Design: The approach to quantify memorization using modified data contamination methods is methodologically interesting and innovative.
summary: The paper explores the phenomenon of memorization in In-context Learning (ICL) across different regimes (zero-shot, few-shot, and many-shot) using large language models (LLMs). It aims to quantify how much of the model's performance improvement during ICL can be attributed to memorized training data.
weaknesses: Incremental Contribution: The insights and contributions of the paper are not sufficiently novel or significant. The findings that ICL surfaces memorization and that memorization correlates with performance do not extend significantly beyond what is already suggested or known in the literature.

Lack of Practical Implications: The paper does not sufficiently discuss the practical implications of its findings. While it notes the correlation between memorization and performance, it fails to explore how these insights could be used to improve model design or deployment in real-world applications.

Theoretical Depth: The discussion around why certain memorization occurs in specific ICL settings lacks depth. There is no substantial theoretical analysis or explanation beyond the observational data presented.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: * Observation 3 states that performance improvements are highly correlated with memorization. Can you explain why, according to results, particularly how can you explain the negative correlation on RTE 
* Tables 1 and 2 report a correlation (Pearson) between performances and memorization; however, the number of k-shots is not specified, nor is the setting (Full Information, Segments pairs, and labels, only segment pairs) to measure the correlation. What did you choose for the memorization metric here?
soundness: 2
strengths: *The originality of the paper proposing a study of memorization impact on performances, while generally it mainly involves detecting contamination
*Experiments are well designed to answer the research question (correlation between performances and memorization)
*A proposal to measure memorization based on a comparison of the number of demonstrations (number of examples in the prompt)
summary: The authors address the memorization problem in the context of In Context Learning. 
We recall that memorization is the process of memorizing data from the pre-trained dataset.
It is generally evaluated by measuring the ability of the model to regenerate those data at the inference step.
The main objective of this work is to understand how memorization correlates with Performances in ICL. To this end, the authors propose to study different settings to estimate memorization of the model (using GPT-4 in the experiment), on the dataset used in the pre-training of the model (or that are likely to be seen during its training according to previous works). Authors propose to evaluate different ICL variants (varying between description, example, and labels) in different k-shot settings (number of examples given in the context). For scoring memorization, authors mostly rely on "Time Travel in LLMs: Tracing Data Contamination In Large Language Models," published at ICLR last year.
In the experiments section, authors compare the different memorization scores according to the number k (of k-shots), the information given in input (instruction, example, and/or labels), and the matching method (exact matches, near exact matches).
In subsection 5.2, the correlation between task performance and memorization is compared, showing that the more the dataset is memorized, the more it correlates with performance.
Accordingly, the contributions are the following:
* A new method to measure memorization in ICL, proposing to evaluate memorization with k-shot (the metrics exact match and the near exact match was proposed in [1])
* How to correlate the memorization with performances (in the setting proposed) 


[1] "Time Travel in LLMs: Tracing Data Contamination in Large Language Models", Shahriar Golchin and Mihai Surdeanu, ICLR 2024
weaknesses: * Only one model is used in the experiments, which is too few to conclude on a general statement (notice that the choice is, however, justified)
* Most of the pipeline relies on the memorization score design in [1] 
* The state-of-the-art lack of explicit explanation (section 6), but contains at the best of my knowledge the most relevant references
* Results should be better/extensively discussed (mainly for the 5.2 experiments)
* Metrics and what the reported results are missing in 5.2 (the two tables 1 and 2)

Meta-review: This paper studies memorization in in-context learning and explores its correlation with downstream performance of LLMs.  The key findings suggest that ICL significantly surfaces memorization compared to zero-shot learning, and demonstrations without labels are the effective in doing so, and there's a strong correlation between performance and memorization in ICL when it outperforms zero-shot ICL. The paper is the first to systematically examine the relationship between ICL and memorization in LLMs, providing new insights into how memorized knowledge influences ICL performance. However, the paper has several shortcomings weaknesses: 

- Limited generalizability due to reliance on a single LLM (GPT-4). Several reviewers pointed out that the study's conclusions might not generalize to other LLMs with different training data and different architectures, which I am also uncertain about. As such, the paper's findings might be an artifact of GPT-4, rather than a general finding. While authors pointed out computational or API cost, recent long context models such as Gemini Flash are substantially cheaper than GPT-4o. Another option could be smaller versions of open-weights long-context LLMs, such as LLama2-32K. 

- Tasks with Conflicting results and Limited Analysis. The paper only studies 2 NLI and 2 classification tasks, where the negative correlation for the RTE dataset contrasts with the positive correlations for other benchmarks, casting doubt on the overall claim of correlation. Furthermore, the paper presents a list of observations, without running detailed ablations to further understand those results.

- Deliberate data leakage in experiments. The use of datasets that are part of GPT-4's training data introduces data leakage and limits the generalizability of the findings to conventional ICL scenarios where generalization is critical. It is possible that ICL operates mainly through memorization on such leaked datasets, while on conventional datasets ICL's mechanism might be different. As such, experiments to understand the role of memorization in ICL performance (possibly on synthetic datasets that can be controlled) seems critical to understand for this paper's relevance.

Overall, while the paper studies a timely and relevant topic, my recommendation is to reject the paper based on the weaknesses pointed above. That said, I encourage the authors to revise the paper for a stronger submission.

Decision: Reject