The mosaic memory of large language models

Domain: neuroscience
Published: 2026-01-29
Editorial outcome: accept

Abstract
As Large Language Models (LLMs) become widely adopted, understanding how they learn from, and memorize, training data becomes crucial. Memorization in LLMs is widely assumed to only occur as a result of sequences being repeated in the training data. Instead, we show that LLMs memorize by assembling information from similar sequences, a phenomenon we call mosaic memory. We show major LLMs to exhibit mosaic memory, with fuzzy duplicates contributing to memorization as much as 0.8 of an exact duplicate and even heavily modified sequences contributing substantially to memorization. Despite models displaying significant reasoning capabilities, we somewhat surprisingly show memorization to be predominantly syntactic rather than semantic. We finally show fuzzy duplicates to be ubiquitous in real-world data, untouched by deduplication techniques. In this work, we show memorization to be a complex, mosaic process, with real-world implications for privacy, confidentiality, model utility and evaluation.

Introduction
Large Language Models (LLMs) are increasingly transforming various aspects of daily life. They drive advancements across industries by automating routine tasks through applications like virtual assistants and customer service bots1,2; extracting valuable insights from unstructured data3; generating code to support software development4,5; and reinventing information retrieval through conversational AI such as ChatGPT6.

LLMs develop their capabilities by absorbing patterns from massive text datasets, allowing them to both generalize concepts and selectively memorize information. Generalization allows LLMs to reason7 and apply their knowledge to unseen scenarios, while memorization helps them retain factual information. The interplay between generalization and memorization in LLMs has been widely studied from a model utility perspective, examining how memorization contributes to generalization8–10, facilitates the encoding of factual knowledge11,12, and how different data mixtures influence model performance13.

However, memorization can also be unintended and raise concerns. Indeed, LLMs have been shown to memorize personal or confidential information14–19, copyright-protected content20–22, as well as portions of evaluation benchmarks, potentially inflating performance estimates and undermining fair assessment23–29.

The mechanism by which LLMs memorize training data has predominantly been understood as exact, or verbatim. Indeed, current studies of LLM memorization have shown that a piece of content repeated exactly several times in the training data has a much higher chance of being memorized22,30–32. This memorized content could then be extracted by a potential attacker – either exactly32 or approximately33,34. This behavior is often highly undesirable, especially if the extractable content contains private information or is copyright-protected. This understanding has, in turn, led model developers to implement data deduplication techniques to improve model utility31,35–39, mitigate privacy and confidentiality risks30, or to decontaminate benchmark data to properly evaluate model capabilities40–44. However, the presumed verbatim nature of memorization has led to most mitigation strategies to rely on removing exact repetitions of sequences of text.

In this work, we argue that viewing LLM memorization solely through the lens of exact repetitions in the training data is incorrect. Instead, we posit that LLMs exhibit a behavior we call mosaic memory, which we define as the model’s ability to memorize an arbitrary sequence of tokens through exposure to its fuzzy duplicates – fragments of text similar to the original sequence with some tokens missing, replaced, or shuffled. We show that fuzzy duplicates contribute to a sequence’s memorization to a degree comparable to exact repetitions.

First, we introduce a framework to study the mosaic memory of LLMs. Building on established techniques for analyzing text memorization22,30, we evaluate the performance of Membership Inference Attacks (MIAs) on canaries—artificially crafted sequences included as part of the training data14. We inject a reference canary (100 tokens), along with its fuzzy duplicates, into the training dataset of a target LLM, and measure how the presence of these fuzzy duplicates impacts MIA performance on the reference canary. We measure the memorization of fuzzy duplicates using exact duplicate equivalent \rho, the proportion of the memorization impact retained by the fuzzy duplicate compared to an exact one. A value of \rho = 1 indicates that a fuzzy duplicate contributes to memorization of the reference canary as much as an exact duplicate, while \rho = 0 would indicate that the fuzzy duplicate has no impact on the memorization.

We study four LLMs: GPT-Neo 1.3B (EleutherAI)45, Gemma-2B (Google)46, Phi-2 (Microsoft)47, and Llama-3.2-1B (Meta)48, and find all of them to exhibit a significant mosaic memory. For example, fuzzy duplicates with 10% of their tokens replaced contribute to the memorization of the canary between \rho = 0.60 (Gemma-2B) and \rho = 0.65 (GPT-Neo) of an exact duplicate’s impact. Moreover, even heavily modified fuzzy duplicates—where 50% of tokens are replaced—still contribute as much as \rho = 0.13–0.19 of an exact duplicate. Furthermore, we find that mosaic memory is remarkably robust, as the \rho of fuzzy duplicates remains significantly higher than 0, even when separated by noise insertions or when their order is shuffled.

Second, while memorization is not exact, we find, somewhat surprisingly, that it is still predominantly syntactic, rather than semantic. Despite modern LLMs demonstrating strong performance across tasks requiring semantic understanding—such as reasoning, mathematical and scientific problem solving, and multilingual generalization46,47,49—we find that memorization is driven mostly by the model’s retention of specific overlapping tokens, and not the encoding of the shared underlying meaning across fuzzy duplicates. Fuzzy duplicates created by replacing tokens while preserving semantic meaning indeed are only marginally better memorized than those where semantic meaning is not maintained. Likewise, our results suggest that paraphrased sequences are primarily memorized due to token overlap rather than their shared semantic meaning.

Third, we investigate how widespread fuzzy duplicates are in the real-world, large-scale dataset used for LLM pretraining, SlimPajama50. Although this dataset is already deduplicated on the document level, we find that many sequences still have a substantial number of both exact and fuzzy duplicates. To identify these fuzzy duplicates, we use Levenshtein distance, which measures the minimum number of single-character edits required to transform one sequence into another. Levenshtein distance 10, for instance, represents just 10% of tokens edited and corresponds to \rho values between 0.6 and 0.8. Our analysis reveals that an arbitrary sequence with 1000 exact duplicates also has, on average, 3000 more fuzzy duplicates at Levenshtein distance 10 (representing just 10% of tokens edited) and over 20,000 at distance 50. This suggests that traditional deduplication methods are likely insufficient for addressing memorization concerns for LLMs.

Taken together, our results reveal a critical gap in our understanding of how LLMs learn from and memorize training data, whether beneficial or harmful. Rather than solely memorizing exact repetitions, LLMs also retain information across fuzzy duplicates, often to a degree comparable to exact duplicates. This mosaic memory phenomenon fundamentally challenges current assumptions underlying deduplication and privacy protection practices. For example, common industry-standard techniques like removing exact matches from benchmarks (e.g., 13-g overlaps used in GPT-3 evaluations) fail to eliminate fuzzy duplicates, potentially inflating evaluation scores and undermining fair benchmarking. Similarly, deduplication strategies that target exact duplicates (e.g., substring matching of 50 tokens) neglect mosaic memory, allowing memorization of confidential or copyright-protected content through slightly modified sequences. Thus, current practices offer insufficient privacy protection, incomplete benchmark decontamination, and suboptimal data preprocessing, necessitating more nuanced deduplication techniques that account for the mosaic nature of memorization.

Results
A metric to study the memorization of fuzzy duplicates
We instantiate a framework to measure how an LLM memorizes fuzzy duplicates, relative to exact duplicates, in its training data. Our goal is to design a metric that is invariant to the absolute memorization level of individual sequences, specifically capturing how fuzzy duplicates influence memorization regardless of how inherently memorizable the base sequence is.

Following prior work, we quantify memorization of a target language model M with tokenizer T by measuring the performance of MIAs on artificially crafted sequences, known as canaries14,22,51–55. We define a set of reference canaries {Xi ref}i=1,...,C where each Xi ref is a synthetically generated sequence containing exactly 100 tokens, i.e., |T(Xi ref)|=100. Specifically, we follow the method of Meeus et al.22 and generate synthetic sequences by sampling tokens autoregressively from a reference LLM, setting the temperature T = 1 to mimic realistic text. Temperature controls the randomness of token sampling: T = 1 uses the model’s natural probability distribution, producing coherent yet varied text, while higher values yield increasingly random sequences. We additionally vary the canary sampling temperature in Methods to verify that our results hold for canaries across the range from regular text to rare and unexpected sequences.

To generate fuzzy duplicates, we consider any algorithm A that systematically modifies each reference canary, e.g., by replacing R tokens within the sequence at random. Formally, A:Xi ref \to {Xi dup j}j=1,...,n produces a set of fuzzy duplicates Xi dup for each Xi ref, where we always consider Xi dup 1 = Xi ref. We explore multiple instantiations of A and compare their impact on memorization.

We instantiate an MIA with member and non-member reference canaries. For each reference canary Xi ref, we flip a fair coin to determine random variable bi \in {0,1}. If bi = 1, we inject Xi ref and its corresponding fuzzy duplicates {Xi dup j}j=2,...,n into a training dataset D; otherwise, neither Xi ref nor its fuzzy duplicates are included. We then consider a pretrained LLM M0, which we continue pretraining on D, yielding target model M. Details on the continued pretraining are provided in Methods.

We then study how M memorizes the member reference canaries via the corresponding MIA performance. We apply MIAs15,56,57 to M, computing a membership score \alpha(Xi ref) for each Xi ref based on output token probabilities from M. Using these scores and their ground-truth membership labels bi, we compute the receiver operating characteristic area under the curve (ROC AUC), denoted as \phi e.

If an MIA can reliably distinguish members from non-members based on the observed M, the corresponding AUC will be high (closer to 1), indicating that the target model M substantially memorizes the reference canaries. Prior work has shown that including a member canary multiple times in the training dataset yields higher AUC, indicating stronger memorization22,30,32. This exact duplication corresponds to the case where algorithm A is the identity transformation.

In this work, we study how different forms of fuzzy duplication, obtained by applying various algorithms A, influence the MIA AUC and, consequently, the memorization of the reference canaries. If the MIA AUC does not substantially increase for a given set of fuzzy duplicates, the target LLM does not memorize across such duplicates in a way that reinforces its memorization of the reference canaries. In contrast, when the MIA AUC substantially increases due to the presence of fuzzy duplicates, the LLM exhibits strong memorization across these variants – an effect we refer to as mosaic memory.

We specifically evaluate how fuzzy duplicates impact memorization relative to exact duplicates. We define the equivalent number of exact duplicates \nu eq as the number of exact repetitions of Xi ref needed to achieve the same level of memorization (quantified with MIA AUC) as observed for fuzzy duplicates (i.e., \phi e). This approach aims to capture how fuzzy duplicates influence memorization regardless of how inherently memorizable a sequence is, as the absolute level of memorization in a given setup is known to be highly sensitive to factors such as model characteristics (e.g., number of parameters), training procedures (e.g., learning rate), and properties of the sequences (e.g., length, perplexity)22,32.

To compute \nu eq, we first compute how exact duplicates are memorized in the same setup. We repeat the same training procedure, each time injecting an increasing number of exact copies (\nu \in {1,…,n dup}) of the reference canary Xi ref into the training dataset (D\nu). We measure the resulting MIA performance \phi \nu reached for each value of exact repetitions \nu. We then determine for a certain set of fuzzy duplicates associated with algorithm A and an MIA AUC of \phi e, the equivalent number of exact duplicates \nu eq as the value of \nu for which \phi e \approx \phi \nu. In other words, \nu eq represents how many exact repetitions would be needed to achieve the same level of memorization as observed for the fuzzy duplicates. The corresponding values of \phi \nu and their evolution with \nu are reported in Fig. 3.

Finally, we compute the exact duplicate equivalent \rho for a single fuzzy duplicate by normalizing \nu eq by the total number of fuzzy duplicates considered. As such, the value of \rho is independent of the number of fuzzy duplicates and reflects the equivalent number of exact duplicates represented by a single fuzzy duplicate generated by A. We provide additional details for the computation of \rho in Methods.

Large language models have a mosaic memory
Out of the many algorithms A which can be used to construct fuzzy duplicates, we first consider A replace, which constructs fuzzy duplicates of each reference canary Xi ref by replacing R randomly chosen tokens from T(Xi ref). For each fuzzy duplicate Xi dup j, where j \in {2,…,n dup}, we define a modification set Ri \subset {1,...,|T(Xi ref)|} of size |Ri|=R, consisting of token positions randomly sampled without replacement. For each token position r \in Ri, the original token T(Xi ref)r is replaced by a different one. For more details, we refer to Methods.

Our results show that across a range of widely used, recently developed LLMs – GPT-Neo45, Gemma-246, Phi-247, Llama-3.248—fuzzy duplicates contribute significantly to memorization. Figure 1a illustrates how \rho varies when the number of replacements R made to the fuzzy duplicates increases. We find the exact duplicate equivalent \rho to decrease gradually with the number of replaced tokens R, consistently finding \rho > 0 until all tokens in the sequence are replaced (R = 100). For instance, considering GPT-Neo 1.3B, when R = 10 tokens are replaced (out of the 100 in total), the fuzzy duplicates still yield exact duplicate equivalents of \rho = 0.65. In practice, this means that having two fuzzy duplicates in the training data with 10% of the original tokens replaced yields higher memorization than reached for one exact repetition. Even for R = 50 replaced tokens, i.e., half of the original tokens, \rho is still maintained significantly above zero at 0.19.

The results are furthermore remarkably consistent across a range of widely used LLMs. While the original training data, benchmark performance, and the extent to which each model memorizes likely differs substantially between, for instance, GPT-Neo 1.3B and the more recent Phi-2, both LLMs show similar memorization behavior in the presence of fuzzy duplicates.

In ablation studies provided in Methods, we also show our findings to be consistent across 3 state-of-the-art MIA methodologies14,56,57, different learning rates, different model sizes, and sampling temperatures used to generate reference canaries. We further consider slight modifications to A replace, considering multiple ways of selecting the tokens to be replaced and find this to have limited impact. In line with prior work15,32,33,58, we also study how fuzzy duplicates affect the data extraction risk of the reference canaries. We measure approximate extraction during greedy decoding33 and find that fuzzy duplicates contribute similarly to extraction risk as to MIA performance.

So far, we have exclusively considered fuzzy duplicates obtained by replacing R tokens from the reference canary (A replace). Using the same framework, we now explore alternative methods for constructing fuzzy duplicates.

First, we consider inserting random tokens at certain positions in the reference canary (A insert). We split the tokenized reference canary T(Xi ref) into C n = |T(Xi ref)|/n equally sized n-grams and insert X insert random tokens between each n-gram to generate the fuzzy duplicates. This allows us to assess the robustness of LLM memorization to noise, while preserving the original tokens of the reference canary in the same relative order, across each fuzzy duplicate.

We compare this approach to the baseline scenario, where n-grams are randomly and independently distributed in the training data. In this case, the n-grams are encountered by the model in isolation (i.e., likely not within the same context window), preventing the model from learning any associations between them. We refer to this as X insert = \infty. This baseline reflects how the memorization of individual subsequences contributes to the memorization of the reference canary. Any additional memorization indicates the model’s capacity to ignore inserted random tokens and piece together fragments across fuzzy duplicates, disregarding the inserted tokens as irrelevant noise.

Figure 1b shows how LLMs exhibit remarkable robustness in skipping irrelevant tokens. For example, when X insert = 1 random token is inserted between all the 20-g in the reference canary, the exact duplicate equivalent \rho still reaches \rho = 0.83, which is significantly larger than the lower bound \rho = 0.48 at X insert = \infty. When X insert = 10 tokens are inserted, the memorization decreases, yet remains as high as \rho = 0.70. Even in the extreme case where the entire sequence is split into individual tokens (n = 1), all separated by X insert = 1 token, we find a \rho of 0.21, far exceeding the baseline assumption that such fuzzy duplicates do not contribute to the memorization of the reference canary (\rho = 0).

While this remains to be shown, we hypothesize that such robustness derives from the attention mechanism, which could potentially allow the LLM to memorize a connection between tokens even if irrelevant tokens are inserted in between. During training, the LLM would learn to assign low attention scores to inserted tokens, effectively filtering them out as noise. This would lead the model to focus on meaningful patterns across fuzzy duplicates, reinforcing the connections between the original subsequences. When the reference canary’s subsequences then reappear together during inference for the MIA, the model reconstructs these relationships, leading to stronger memorization.

Second, we study the memorization of sequences for which the exact token order is no longer preserved. We now obtain the fuzzy duplicates by shuffling the tokens in the reference canary Xi ref to varying degrees (A shuffle).

A first pass partitions a reference canary T(Xi ref) into C n = |T(Xi ref)|/n equally sized n-grams for n \in {2,5,10}. Fuzzy duplicates are then generated by randomly permuting the n-grams while maintaining the original token order within each n-gram. From our findings above for X insert = \infty, we observe that individual n-grams from the reference canary spread across the dataset contribute non-trivially to memorization. A shuffle controls for this effect, as it maintains the local order of tokens and shuffles n-grams rather than individual tokens.

To quantify the degree of permutation, we use the token-level normalized Kendall tau distance (\tau)59. Kendall tau measures the proportion of token pairs that maintain their relative order in the fuzzy duplicate compared to the reference canary. Given two sequences of tokens, T(Xi ref) (reference canary) and T(Xi dup j) (fuzzy duplicate obtained by shuffling), Kendall tau is defined as \tau = \Delta / [L(L-1)/2], where L = |T(Xi ref)| is the total number of tokens and \Delta is the number of discordant pairs, i.e., token pairs (tu, tv) where the relative order is different between Xi ref and Xi dup j. A value of \tau = 0 indicates no shuffling (the original order is preserved), while \tau = 1 indicates exact reversal.

For each n, we randomly permute n-grams to generate fuzzy duplicates and group them based on their Kendall tau distance. Note that since shuffling n-grams instead of individual tokens preserves the local order of tokens within the n-gram, not all \tau values within [0, 1] can be reached for every n.

Figure 1c first reveals that memorization is highly sensitive to ordering. Indeed, we observe that \rho drops from 1 to approximately 0.53 when only 10% of token pairs have their relative order inverted (\tau = 0.1), showing that perturbing the sequence even slightly has a significant impact on memorization.

Second, and more interestingly, we find that some information is still memorized even with extensive shuffling. To assess this, we compare the values of \rho obtained at different levels of shuffling to the baseline in which n-grams are memorized independently of each other, i.e., when n-grams are randomly and independently distributed across the dataset (X insert = \infty in Fig. 1b). Figure 1c then shows that the memorization, across all values \tau remains significantly higher than the baseline X insert = \infty. Considering 10-g, for example, we still recover \rho = 0.44 (against the baseline of \rho = 0.31) even when the order for 50% of the token pairs is inverted (\tau = 0.5). These results indicate that the model’s mosaic memory retains the shared vocabulary across fuzzy duplicates despite minimal preserved ordering.

The mosaic memory is syntactic rather than semantic
All our experiments thus far have focused exclusively on syntactic modifications to create fuzzy duplicates, i.e., systematically altering tokens through replacements, insertions, and shuffling while tracking their impact on memorization. We now examine whether semantic relationships also influence memorization. Our key question is whether LLMs primarily memorize specific tokens or capture underlying meanings. To explore this, we conduct two experiments. First, we systematically vary semantic coherence while controlling token-level changes. Second, we analyze paraphrases as fuzzy duplicates, where semantic meaning is preserved without explicit control over token overlap.

First, we consider the same algorithm to make fuzzy duplicates as above (A replace), where R tokens are replaced for each fuzzy duplicate Xi dup j but now vary the extent to which the replacement maintains the semantic meaning of the sequence. In particular, we leverage a masked language model MLM to predict the top-k tokens (from the MLM’s vocabulary V MLM) for each replacement and vary the value of k, thereby changing how semantically meaningful the replacement is. We thus consider algorithm A replace,k where smaller values of k lead to more semantic similarity between the reference canary and the fuzzy duplicates. We use RoBERTa60 as MLM, which has a vocabulary size of |V MLM| = 50,265. For implementation details and examples of token replacements for varying values of k, we refer to Methods.

Figure 1d shows the impact of making R replacements with semantically meaningful tokens (smaller k) rather than random ones (k = |V MLM|). While lower values of k consistently lead to higher memorization, the impact is notably small compared to the impact of syntactic changes.

For example, when R = 10 tokens are replaced in the fuzzy duplicates, using semantically similar tokens (k = 10) only increases \rho by 0.11 (0.54 \to 0.65) compared to using a random token (k = |V MLM|). This effect size is smaller than replacing just 5 more tokens while keeping the semantic meaning intact (for R = 15 and k = 10, we get \rho = 0.52).

These results suggest that the mosaic memory of LLMs is more syntactic—the model memorizes the connection between specific, overlapping tokens across the fuzzy duplicates—than it is semantic, where the model memorizes the underlying semantic meaning shared by all fuzzy duplicates. This finding is somewhat surprising in light of the increasingly strong semantic capabilities demonstrated by modern LLMs, including multi-step reasoning, mathematical and scientific problem solving, code generation, instruction following, and multilingual generalization46,47,49, which are widely interpreted as relying on abstract semantic representations rather than surface-level token statistics. The fact that memorization nevertheless appears to be driven predominantly by token overlap suggests that, at least for memorization mechanisms, these models may still rely heavily on shallow statistical regularities61. We note that the smaller scale of our models and the use of continued rather than full pretraining may influence the degree of semantic memorization observed. Future research is needed to quantify whether the balance between syntactic and semantic memorization differs for larger models trained from scratch.

We further investigate the memorization of fuzzy duplicates designed to maintain semantic coherence, while not explicitly controlling the token overlap. Indeed, algorithm A replace,k is constrained by only replacing a fixed set of tokens, which might not give sufficient flexibility to maintain semantic coherence using just MLM replacements. Hence, we now study A paraphrase, which asks an instruction-tuned LLM to paraphrase the reference canary to construct fuzzy duplicates. We consider widely used instruction-tuned LLMs: Meta-Llama-3-8B-Instruct49, Mistral-7B-Instruct-v0.262 and GPT-4o63, with additional details provided in Methods.

Table 1 shows that for all paraphrases generated using the 3 instruction-tuned LLMs, the exact duplicate equivalent \rho remains fairly low, ranging from 0.13 for Llama-3-8B to 0.34 for GPT-4o63. Moreover, this is remarkably low compared to the memorization achieved when R tokens are replaced with random tokens (k = |V MLM|) from Fig. 1(d). For instance, fuzzy duplicates with 15% of all tokens in the reference canary (R = 15) replaced by random ones—the semantic meaning is significantly distorted—still contribute more to the memorization of the reference canary (\rho = 0.43) than any of the paraphrased fuzzy duplicates. Only when making R = 20 replacements with random tokens, the level of memorization (\rho = 0.34) becomes similar to the largest one achieved using paraphrasing (\rho = 0.34).

With \rho for paraphrases already relatively low, we hypothesize that the memorization we observe in this experiment can be explained mostly by syntactic similarity, as paraphrases might still have an overlap in tokens with the reference canary. To investigate this, we compute the mean number of overlapping n-grams between each reference canary and its paraphrases for different values of n, reporting the results in Table 1. We show that, across paraphrases, there remains a substantial n-gram (i.e., syntactic) overlap, up to an average of 73 1-g present in paraphrases made by GPT-4o and that the memorization strictly increases with n-gram overlap. Specifically, as the mean overlap in 4-g increases from 7.7 (Llama-3-8B) to 15.5 (Mistral-7B) and 29.4 (GPT-4o), the respective \rho also rises from 0.13 to 0.21 and 0.34, respectively—a trend that holds across different values of n. While these results do not rule out that some semantic memorization might happen, taken together, they suggest that it likely plays a limited role compared to syntactic memorization.

Fuzzy duplicates are widespread and robust to deduplication
Having demonstrated the significant contribution of fuzzy duplicates to model memorization, we now investigate their prevalence in a real-world training dataset. Our analysis shows that even in an extensively deduplicated dataset, a large number of fuzzy duplicates exist, significantly contributing to memorization.

To identify fuzzy duplicates in real-world datasets, we adopt Levenshtein distance, which measures the minimum number of single-character edits required to transform one sequence into another. It accounts for token replacements (A replace), insertions (A insert), and effectively capture token shuffling (A shuffle) through combinations of deletion and addition operations.

We validate this choice by investigating the relationship between Levenshtein distance and the exact duplicate equivalent (\rho) across our experimental results. Figure 2a shows a strong negative correlation between \rho and Levenshtein distance, confirming that this metric effectively captures the memorization impact of various fuzzy duplicates.

We also investigate Hamming distance, which counts the number of positions where corresponding tokens differ between two sequences. This metric directly corresponds to A replace, making fuzzy duplicates identified by Hamming distance real-world examples of our token replacement experiments.

As a target dataset we consider SlimPajama50—a thoroughly cleaned and document-level deduplicated dataset derived from the RedPajama dataset64, containing 627 billion tokens, or 895 GB of text in a compressed form.

The dataset’s size adds additional complexity to the analysis, as computing Levenshtein distance for all possible sequence pairs would be computationally infeasible. Given these constraints, we implement a sampling strategy by identifying a small representative set of sequences and scanning the dataset linearly to find their fuzzy duplicates. For computational reasons, we also only scan 5% of the dataset for fuzzy duplicates, and extrapolate the results assuming the uniform distribution of fuzzy duplicate positions—a reasonable assumption given the dataset is randomly shuffled.

Figure 2b shows how fuzzy duplicates in SlimPajama increase sharply with Levenshtein distance. Beyond our baseline of 1000 exact duplicates (which we explicitly selected for), we find approximately 5000 sequences at Levenshtein distance \le 14, representing 4000 additional fuzzy duplicates. At this distance range, our experiments show exact duplicate equivalent \rho to be between 0.6 and 1.0, indicating these fuzzy duplicates significantly contribute to memorization, with their cumulative impact likely to outweigh that of exact repetitions.

As we extend to larger Levenshtein distances up to 50, corresponding to \rho between 0.2 and 0.4, the number of fuzzy duplicates expands to over 25,000—more than 25 times the original count of exact duplicates. Even when using the more conservative Hamming distance metric, which directly corresponds to the token replacements (R) in our A replace algorithm and provides a direct real-world analog to our controlled canary experiments with predictable memorization effects, we still find approximately 3000 fuzzy duplicates at distance 20 and 8000 at distance 40.

To illustrate the mechanisms by which numerous fuzzy duplicates arise in real datasets, we provide representative examples in Supplementary Table 4. A notable case is a series of Wikipedia articles describing French communes, which follow an almost identical template where only the commune name and a few key attributes are changed.

Lastly, we validate that our results from previous sections generalize to real-world fuzzy duplicates. Specifically, we select a set of reference canaries and fuzzy duplicates identified in the SlimPajama dataset and repeat the computation of the exact duplicate equivalent \rho. The resulting values are shown in Fig. 2a. We observe that, for these real-world fuzzy duplicates, \rho similarly decreases with increasing Levenshtein distance, consistent with the trend observed in our synthetic experiments. Further experimental details are provided in Methods.

We further examine how commonly deployed deduplication practices remove fuzzy duplicates. SlimPajama50 has been obtained by deduplicating RedPajama64, removing documents with a Jaccard similarity based on 13-g higher than 0.8 (resulting in the removal of 49.6% of all bytes). Document-level deduplication has been shown to result in a dataset with higher information density50, allowing developers to train better models with less computational cost31. Yet, despite this extensive document-level deduplication, our results showed that SlimPajama still contains a substantial number of exact and fuzzy (sequence-level) duplicates, as document-level deduplication only removes near-identical documents while allowing overlapping sequences to persist across them.

Some model developers have also adopted sequence-level deduplication – a more expensive, yet more granular alternative to document-level deduplication. Following Lee et al.31, a common approach is to remove exactly duplicated substrings of at least 50 tokens from the training data30,65,66. Research shows duplicate sequences in training data increases susceptibility to privacy attacks22,32, making sequence-level deduplication (typically at 50 tokens) an important mitigation strategy30. Despite its benefits for dataset efficiency and mitigating privacy risks, sequence-level deduplication requires a more careful calibration than document-level deduplication. Removing sequences may disrupt the internal coherence of documents, and aggressive sequence-level deduplication risks removing a substantial amount of valuable data.

We examine how sequence-level deduplication affects the exact and fuzzy duplicates present in SlimPajama. Specifically, we consider a fuzzy duplicate removed by sequence-level deduplication, if it shares at least one overlapping n-gram with the target sequence. While we note that fuzzy duplicates could also be removed due to an overlap with other sequences in the overall dataset, deduplicating the entire dataset comes at a high computational cost. We approximate its effect by considering the set of fuzzy duplicates for increasing Levenshtein distances from Fig. 2b and argue that the most significant impact is captured by considering this set of highly similar sequences.

Figure 2c shows the number of fuzzy duplicates that remain after sequence-level deduplication for varying values of n. We first examine n = 50, a common choice in training data preprocessing30,31,65,66 or to mitigate privacy risks30. While deduplicating with n = 50 removes a substantial amount of fuzzy duplicates—particularly at low Levenshtein distances (\le 10), a significant number remain. For instance, at a Levenshtein distance of 20 an average of 2500 fuzzy duplicates per sequence persist after deduplication, increasing to 6000 for a Levenshtein distance of 30. At the same time, fuzzy duplicates of similar Levenshtein distances still contribute substantially to memorization (\rho > 0.3 in Fig. 2a). These results suggest that standard 50-g deduplication likely fails to properly mitigate any privacy risks and might be suboptimal to enhance training efficiency and model performance. While more aggressive sequence-level deduplication (e.g., n = 20) might help remove the most impactful fuzzy duplicates, it likely introduces significant trade-offs.

Methods
Reference canaries
We construct a reference canary Xi ref by synthetically generating a sequence using a reference language model M ref following the approach of Meeus et al.22. Specifically, starting from an empty string, we iteratively sample the next token from M ref’s predicted probability distribution, using temperature T. We further control for the length of the sequence by truncating the synthetically generated sequence to the first L ref tokens using the tokenizer T of the target model. We query M ref for synthetic sequences repeatedly and apply rejection sampling until we have a sufficient number of reference canaries satisfying |T(Xi ref)| = L ref.

As reference model M ref, we use the pretrained Llama-2 7B43 to generate C = 200 synthetic reference canaries (100 members and 100 non-members), with length L ref = 100. Unless stated otherwise, we use the default temperature T = 1.0 to sample from M ref, which we empirically find to lead to meaningful sequences. Only for the results in Fig. 3(a), we use other reference canaries generated using T = 2.5 and T = 5.0. We provide example reference canaries generated with varying sampling temperature in Supplementary Table 1.

Throughout this work, we report the mean and standard deviation of the exact duplicate equivalent \rho (e.g., in Fig. 1a–d). These values are obtained from 5 independent runs, each considering a distinct set of reference canaries generated using different random seeds.

Generating fuzzy duplicates
To generate fuzzy duplicates, we consider any algorithm A: Xi ref \to {Xi dup j}j=1,...,n which systematically modifies reference canary Xi ref to construct a set of fuzzy duplicates Xi dup. We always ensure Xi dup 1 = Xi ref and use n = 10. We consider a range of algorithms A throughout this work, and formalize each of them below.

First, we formalize algorithm A replace, which constructs fuzzy duplicates by replacing R tokens. Throughout many experiments in this work (e.g., Fig. 1a), we make R token replacements to construct the fuzzy duplicates from the tokenized reference canary T(Xi ref), consisting of exactly L ref tokens.

For each fuzzy duplicate Xi dup j, where j \in {2,…,n dup}, we define a modification set Ri \subset {1,...,|T(Xi ref)|} of size |Ri|=R, consisting of token positions randomly sampled without replacement. To make token replacements, we consider replacing R \in {1, 5, 10, 15, 20, 25, 50, 75} of the 100 tokens in the tokenized reference canary.

For each token position r \in Ri, the original token T(Xi ref)r is replaced by a new token sampled from the top-k most probable tokens predicted by the masked language model MLM. Specifically, given the masked sequence T(Xi ref) \setminus r, where the token at position r is removed, MLM assigns a probability distribution P MLM (\cdot | T(Xi ref) \setminus r) over the vocabulary. The replacement token is then drawn uniformly at random from the top-k most probable tokens according to P MLM. Thus, each fuzzy duplicate Xi dup j is generated as:

T(Xi dup j) \leftarrow Top-k P MLM (\cdot | T(Xi ref) \setminus r), \forall r \in Ri: (M1)

Note that the MLM tokenizer’s vocabulary likely does not align with the vocabulary of tokenizer T, which defines the sequence length of L ref = 100 tokens. A single replacement token sampled from the MLM vocabulary may thus correspond to multiple tokens under tokenizer T. In practice, we obtain exactly one replacement token from the MLM’s vocabulary V MLM, decode this to text, and then re-tokenize this text using T before inserting it back into the sequence. As a result, the length of a fuzzy duplicate, measured in tokens under T, may slightly exceed L ref.

Further, when k equals the size of the entire vocabulary V MLM of tokenizer T MLM, or k = |V MLM|, we replace the token with a randomly sampled token from V MLM. This likely distorts the fluency and semantic meaning of the sequence. For smaller values of k, the token is replaced by a token which corresponds to a higher predicted probability returned by MLM, likely maintaining the fluency and semantic meaning of the sequence. We use RoBERTa60 as MLM, which has a vocabulary size of |V MLM| = 50,265. We provide example fuzzy duplicates for varying k in Supplementary Table 2. In generating fuzzy duplicates using A replace, we can additionally vary which tokens from the original reference canary are replaced. Specifically, we define the modification sets Ri \subset {1,...,|T(Xi ref)|}, where each Ri consists of R token positions subject to different selection strategies. We distinguish between the following cases:

1. “Evenly spread + consistent”: The R replacement positions are chosen to be evenly distributed across the reference canary. The sequence is partitioned into R contiguous segments of (approximately) equal size, and one token position is selected at random from each segment. Formally, we define

S = {S1, S2, ..., SR}, where Sr = |T(Xi ref)|/R, |T(Xi ref)|/R (M2)

as the r-th segment of token indices. The global modification set Ri is then sampled as Ri = {r1, r2, ..., rR}, where r \in Sr. This set is used consistently across all fuzzy duplicates, i.e., Ri j = Ri, \forall j \in {2,...,n dup}.

2. “Not evenly spread + consistent”: The R replacement positions are sampled uniformly at random over token positions, forming one global modification set Ri. This set is then used consistently across all fuzzy duplicates, i.e., Ri j = Ri, \forall j \in {2,...,n dup}.

3. “Not evenly spread + not consistent”: The R replacement positions are sampled uniformly at random over token positions, but independently for each fuzzy duplicate. That is, for each duplicate Xi dup j, we independently sample Ri for each j \in {2, …, n dup}. This results in different token replacement sets across fuzzy duplicates.

Throughout this work, unless specified otherwise, we construct fuzzy duplicates using A replace which we consider the most realistic to appear in real-world datasets. For this, we replace tokens with a semantically meaningful other token (opting for k = 10), not uniformly spread across the reference canary and not consistent across fuzzy duplicates (“not evenly spread + not consistent”). We study multiple values of k in Fig. 1d and different ways of replacing tokens across fuzzy duplicates in Fig. 3c.

Next, we formalize algorithm A insert, which constructs fuzzy duplicates by inserting random tokens. For each reference sequence Xi ref, we first partition its tokenized form T(Xi ref) into C n = |T(Xi ref)|/n contiguous n-grams. To generate fuzzy duplicates, we insert X insert tokens between each n-gram. The inserted tokens are sampled independently from the vocabulary of the target model, or I \in Uniform(V M) X insert. The fuzzy duplicate is then constructed as

T(Xi dup j) = T(Xi ref) c \cup I : (M3)

where c \in {1, ..., C n - 1}. We study the memorization of fuzzy duplicates constructed using A insert in Fig. 1b.

Lastly, we formalize algorithm A shuffle, which constructs fuzzy duplicates by shuffling the tokenized form T(Xi ref). For each reference sequence Xi ref, we partition it into C n = |T(Xi ref)|/n contiguous n-grams. To generate fuzzy duplicates with varying degrees of token order permutation, A shuffle randomly swaps these n-grams while maintaining the original token order within each n-gram. Formally, let G = {g1, g2, ..., gCn} represent the sequence of n-grams from the tokenized reference canary, where each g is an ordered sequence of n consecutive tokens. A shuffle generates a fuzzy duplicate Xi dup j by creating a permutation \pi of indices {1, 2, …, C n} and concatenating the n-grams in the permuted order: T(Xi dup j) = g\pi(1) \cup g\pi(2) \cup ... \cup g\pi(Cn).

To quantify the degree of permutation between the reference canary and each fuzzy duplicate, we use the normalized Kendall tau distance \tau, which measures the proportion of token pairs whose relative order differs between the two sequences. For token positions u and v, let (tu, tv) represent a token pair in the reference canary. This pair is considered discordant if its relative order in the fuzzy duplicate is inverted. The Kendall tau distance is defined as:

\tau = \Delta / [L(L-1)/2] (M4)

where L = |T(Xi ref)| is the total number of tokens and \Delta is the number of discordant pairs.

For our experiments, we generate fuzzy duplicates through rejection sampling, continuously permuting n-grams and computing \tau until we have obtained the desired number of fuzzy duplicates for a given Kendall tau distance. We study the memorization of fuzzy duplicates constructed using A shuffle in Fig. 1c.

Membership inference game
Throughout this work, we measure memorization of a target language model M with tokenizer T by instantiating a membership inference attack (MIA) on the artificially crafted reference canaries. We consider C = 200 reference canaries {Xi ref}i=1,...,C with their fuzzy duplicates {Xi dup j}j=2,...,n dup, always considering Xi dup 1 equal to Xi ref.

We apply continued pretraining on M0 using the training dataset D, denoting the resulting model as the target model M. For each reference canary Xi ref, we flip a fair coin to determine random variable bi \in {0,1}. If bi = 1, we inject Xi ref and its corresponding fuzzy duplicates {Xi dup j}j=2,...,n dup into D. Conversely, if bi = 0, neither Xi ref nor its fuzzy duplicates are included in the training dataset D. Formally, the training dataset D is defined as:

D = D orig \cup [C i=1 {Xi ref} \cup {Xi dup j}j=2,...,n dup] \cdot bi (M5)

where D orig represents the original dataset without any canary injections, and \cdot denotes set inclusion conditioned on bi = 1.

To quantify memorization, we apply MIAs15,56,57 on M, computing a membership score \alpha(Xi ref) for each Xi ref based on query outputs from M. Each MIA methodology corresponds to a membership scoring function, \alpha(Xi ref). We select three methods to compute \alpha(Xi ref):

1. Loss attack from Yeom et al.56, which uses the model loss computed on the reference canary. For the sequence of textual characters X, tokenized as T(X)={t1,…,tL}, we denote the loss \mathcal{L} of language model M with tokenizer T as \mathcal{L} M(X) = - 1/L \sum i=1 L log M \theta (ti | t1, ..., ti-1). For the Loss attack, we thus consider: \alpha = \mathcal{L} M(X).

2. Ratio attack from Carlini et al.15, which uses the model loss divided by the loss computed using a reference model, or \alpha = \mathcal{L} M(X) / \mathcal{L} M ref(X), where M ref = Llama-2 7B43.

3. Min-K% Prob from Shi et al.57, which computes the mean log-likelihood of the K% tokens with minimum predicted probability. We use K = 10.

The MIAs are evaluated over all C reference canaries, representing a balanced set of member canaries (bi = 1) and non-member canaries (bi = 0). Specifically, membership scores are used to compute the area under the receiver operating characteristic curve (ROC AUC), which quantifies the attack’s ability to distinguish between reference canaries included in and excluded from D. We denote the resulting ROC AUC of MIAs on M as \phi e.

Experimental setup
We then measure how a target LLM memorizes across fuzzy duplicates by comparing MIA performances reached for fuzzy duplicates relative to the performance achieved for exact duplicates. The degree to which exact duplicates are memorized by a target LLM depends on the exact experimental setup S, which we define by the following characteristics:

1. The set of reference canaries Xi ref used as members and non-members. Throughout this work, we consider synthetically generated sequences generated using the pretrained Llama-2 7B43 as reference model M ref. Unless stated otherwise, we use temperature T = 1.0.

2. The choice of pretrained model M0, which we continue pretraining on training dataset D containing the fuzzy duplicates.

3. The hyperparameters used for continued pretraining. We keep this constant throughout this work, apart from the initial learning rate \eta 0. We opt to choose \eta 0 separately for each choice of pretrained model M0, as we find this to be a crucial hyperparameter when comparing training progress and the absolute level of memorization across model size and architecture.

4. The MIA methodology and performance metric used to measure memorization. Throughout this work, we use the ROC AUC computed on a balanced set of member and non-member reference canaries, and consider different MIAs from the literature defined by their scoring function \alpha.

We consider a range of models as the pretrained model M0. In most experiments, we consider GPT-Neo 1.3B45 developed by EleutherAI. In further ablations, we also consider its smaller and larger versions, GPT-Neo 125M and 2.7B, and further consider three other models from different model families: Gemma-2B46 developed by Google, Phi-247 developed by Microsoft and Llama-3.2-1B48 developed by Meta. All models we consider are open-weights, and the pretrained model parameters are downloaded from HuggingFace.

In all of our experiments, we continue pretraining M0 on a dataset with documents D containing reference canaries and their fuzzy duplicates, resulting in the target model M. We inject canaries in a collection of books available in the public domain, i.e. the original dataset D orig. We use an open-source library95 to collect C = 100 books made available under a permissive license on Project Gutenberg96. We collect books added to Project Gutenberg after the release of GPT-Neo, which were thus likely not present in the original model’s training dataset. The books we selected contain 8.8M tokens (GPT-Neo) in total. For each reference canary Xi ref for which randomly assigned membership bit bi = 1, we inject the n = 10 fuzzy duplicates at random into one book and use this collection of modified books as dataset D for continued pretraining.

In all of our experiments, we apply continued pretraining on M0 using the modified books D for 1 epoch. We use a maximum sequence length 2048 tokens, an effective batch size of 2 and an Adam optimizer with \beta 1 = 0.9 and \beta 2 = 0.999. We use a linear learning rate scheduler with as initial learning rate \eta 0, final learning rate 0, and a weight decay 0.01. Across pretrained models M0, we empirically determine a value of \eta 0 to align with a training regime leading to a significant level of absolute memorization. We also show how the results vary with \eta 0 for GPT-Neo 1.3B (Fig. 5a).

For reference, continued pretraining for GPT-Neo 1.3B on D takes roughly 2 GPU-hours on an NVIDIA A100 GPU, with minor variations for other models.

Computing the exact duplicate equivalent
We now detail the computation of the exact duplicate equivalent, \rho, for a given set of fuzzy duplicates generated by algorithm A. First, we determine \nu eq, which represents the equivalent number of exact repetitions that yield the same MIA AUC as observed for the fuzzy duplicates (\phi e), under the same experimental setup S. We then normalize \nu eq by the total number of fuzzy duplicates considered, ensuring that \rho reflects the equivalent number of exact duplicates represented by a single fuzzy duplicate generated by A.

To compute \nu eq, we construct a modified training dataset D\nu, where \nu \in {1,…,n dup} exact copies of each reference canary Xi ref are injected. We use the same set of random variables {bi} generated during the fuzzy duplicate experiments. Formally, the modified dataset D\nu is defined as:

D\nu = D orig \cup [C i=1 {Xi ref} \nu] \cdot bi (M6)

where {Xi ref} \nu denotes \nu exact copies of Xi ref. For each \nu, we train M0 on D\nu to get target model M \nu. We measure the performance of the MIA applied to M \nu and denote this as \phi \nu.

Figure 4 (top right) illustrates the observed values of \phi \nu for increasing values of \nu for GPT-Neo 1.3B45 as M0, initial learning rate \eta 0 = 2e-5, Ratio15 as MIA methodology and reference canaries generated using temperature T = 1.0.

We then leverage the values for \phi \nu to compute \nu eq for a given \phi e. To prevent noise in the observed values from impacting the comparison of computed values of \nu eq, we smooth the curve for exact duplicates using a moving average with window size of 3. Figure 4 (top right) shows how the smoothed curve compares to the observed values, confirming the changes to be minimal. Throughout this work, we leverage the smoothed curve for \phi \nu.

For a given memorization level of fuzzy duplicates \phi e, \nu eq is determined as the value of \nu for which \phi e \approx \phi \nu. We compute \nu eq through piecewise linear interpolation, or

\nu eq = \nu 0 + [\phi e - \phi \nu 0] / [\phi \nu 0+1 - \phi \nu 0] (M7)

where \nu 0 represents the number of exact duplicates such that \phi \nu 0 \le \phi e \le \phi \nu 0+1.

Figure 4 illustrates how for R = 10, we reach \nu eq = 6.36. We then repeat the same process for each observation (R, \phi e), to quantify how \nu eq evolves with varying number of replacements R across fuzzy duplicates. Note that, in rare cases, we empirically find \phi e to reach values slightly higher than \phi \nu max, in which case we use the value \nu max. We then compute the \nu eq using interpolation from Equation (M7).

Finally, we want a value reflecting the degree to which a fuzzy duplicate constructed by A contributes to the memorization of the reference canary, independent of the number of fuzzy duplicates (n dup) we inject into the training dataset D. Therefore, we normalize \nu eq to get the exact duplicate equivalent, \rho:

\rho = [\nu eq - 1] / [n dup - 1] (M8)

Note that we need to subtract by 1 in both the numerator and denominator, as we always include 1 exact duplicate of the reference canary, or Xi dup 1 = Xi ref.

Ablation studies
It is well known that the extent to which LLMs memorize depends on many factors of the experimental setup S, including for instance training hyperparameters and model size32. We refer to this as the absolute level of memorization of the target model. For this reason, we have studied the mosaic memory throughout this work by computing the exact duplicate equivalent \rho, which capture the level of memorization relative to the absolute level reached for the specific experimental setup. To further study whether our quantification of the mosaic memory of LLMs holds for different levels of absolute memorization, we here vary the core elements of the experimental setup S, including for instance training hyperparameters and model size32. We refer to this as the absolute level of memorization of the target model. For this reason, we have studied the mosaic memory throughout this work by computing the exact duplicate equivalent \rho, which capture the level of memorization relative to the absolute level reached for the specific experimental setup. To further study whether our quantification of the mosaic memory of LLMs holds for different levels of absolute memorization, we here vary the core elements of the experimental setup S.

As a baseline, we use the results for GPT-Neo 1.3B45 from Fig. 1(a). Specifically, we consider reference canaries generated with temperature T = 1.0, and fuzzy duplicates generated using A replace making “not evenly spread + not consistent” token replacements using the masked language model MLM and top k = 10.

First, we compute the value of \rho versus number of replacements R when we use a different MIA methodology. We consider the three MIA methods described above: Loss56, Ratio15 using the pretrained Llama-2 7B43 as reference model and Min-K% Prob57.

Table 2 reports the observed MIA AUC and the corresponding \rho across MIA methodologies. We find \rho to decrease similarly with increasing number of replacements R, independent of how the MIA AUC is computed or its absolute values. For instance, when R = 20 replacements are being made across fuzzy duplicates, we find an exact duplicate equivalent \rho of 0.455, 0.452 and 0.455 for the attacks Loss, Ratio and Min-K% Prob, respectively. This allows us to use \rho as a metric to study the mosaic memory of LLMs (e.g., for various algorithms A to generate fuzzy duplicates), independent of the absolute level of memorization. As we find the Ratio MIA15 to consistently lead to higher AUC values across setups, we consider this methodology further in this work.

Next, we study the mosaic memory across different values for the initial learning rate (\eta 0). As shown in Fig. 5a (left), the MIA AUC (measuring the absolute memorization) increases with \eta 0, achieving near-perfect performance (AUC of 1.0) for sequences with more than 6 exact duplicates when \eta 0 = 3e-5. This aligns with expectations: larger learning rates result in more substantial updates to model weights when a sequence is encountered during training, making the sequence’s presence in the training data more detectable by the MIA.

We then continue to compute the exact duplicate equivalent \rho. Figure 5a (right) demonstrates that the progression of \rho with respect to the number of replaced tokens (R) remains consistent across different values of \eta 0. This indicates that our relative memorization metric, \rho, is robust to variations in absolute memorization levels, enabling comparisons of LLM memorization across settings. Furthermore, this consistency suggests that LLMs exhibit a mosaic memory, i.e., effectively memorizing fuzzy duplicates, in a similar manner across varying hyperparameter choices, such as learning rate.

We repeat the same experiment while keeping the initial learning rate \eta 0 = 2e-5, yet now varying the size of the target LLM. Specifically, we also consider the smaller and larger GPT-Neo models45, of 125 million and 2.7 billion parameters, respectively. Figure 5b (left) shows that the absolute level of memorization strongly depends on the model size. For instance, GPT-Neo 125M only reaches the MIA AUC of 0.61 at \nu = 10 exact duplicates, significantly lower than the near-perfect AUC reached for GPT-Neo 2.7B in the exact same setup. This again aligns with expectations: a model with more parameters has more capacity to memorize specific sequences, as also consistent with the literature32.

We further examine the relative memorization of fuzzy duplicates (\rho). Our results indicate a similar mosaic memory across GPT-Neo 1.3B and GPT-Neo 2.7B models. Interestingly, GPT-Neo 125M exhibits slightly higher relative memorization, suggesting smaller models may rely more strongly on syntactic patterns, memorizing fuzzy duplicates proportionally more compared to their larger counterparts.

Next, we study the exact duplicate equivalent across model families. Figures 1a and 5c illustrate how \rho evolves with the number of replacements R across all pretrained models M0 considered in this work45–48. Across model families, we empirically found a distinct impact of choice of initial learning rate \eta 0 and the absolute level of MIA performance and memorization. Across models, we empirically set the values for \eta 0 to have a meaningful level of absolute memorization (MIA AUC > 0.5): \eta 0 = 2e-5 for GPT-Neo {125M, 1.3B, 2.7B}45, \eta 0 = 3e-5 for Gemma-2B46, \eta 0 = 6e-5 for Phi-247, and \eta 0 = 2e-5 for Llama-3.2-1B48. Our results above show how the value of \rho is fairly consistent across different values of \eta 0.

Thus far, we have only considered one set of reference canaries, namely the ones generated by sampling from the reference model M ref (Llama-2 7B43) with temperature T = 1.0. We note that T = 1.0 is commonly used when generating synthetic text from pretrained LLMs, and we also empirically confirm that synthetic sequences generated using T = 1.0 lead to meaningful sequences (see Supplementary Table 1).

We now repeat the baseline experiment for GPT-Neo 1.3B45 as M0 and initial learning rate \eta 0 = 2e-5 while varying temperature T. As such, we can verify whether our results are consistent regardless of the choice of reference canaries.

Figure 3a shows how \rho decreases for increasing number of R token replacements made across fuzzy duplicates, for different values of the temperature used to generate the reference canaries. We find that the memorization of fuzzy duplicates is strikingly consistent across a wide range of temperature values, even at T = 5.

This is especially remarkable, as we empirically find sequences generated with T = 5 to correspond to quite nonsensical sentences (see Supplementary Table 1). This suggests that LLMs memorize across fuzzy duplicates regardless of whether the reference canary is a coherent sequence of tokens, supporting the thesis that an LLM’s mosaic memory is predominantly syntactic rather than semantic.

Ablations for fuzzy duplicate generation techniques
In this section, we consider distinct ways of creating fuzzy duplicates not discussed in the main body of the paper.

The token replacement strategy A replace discussed before inevitably leads to an information loss, as when tokens are replaced, the model sees fewer instances of the original tokens from the reference canary. We herein investigate the impact of such information loss on memorization, separately from other properties of A replace. For this, we consider various fuzzy duplicate generation strategies, all of which maintain the same level of information loss, i.e., differing by R tokens from the tokenized reference canary T(Xi ref):

1. Removing R tokens at the end of the sequence (suffix). We here consider all fuzzy duplicates {Xi dup j}j=2,...,n dup to be equal to the first |T(Xi ref)| - R tokens of the reference canary.

2. Removing R tokens at the start of the sequence (prefix). We here consider all fuzzy duplicates {Xi dup j}j=2,...,n dup to be equal to the last |T(Xi ref)| - R tokens of the reference canary.

3. Removing R tokens randomly. We here remove the same R random tokens for all fuzzy duplicates {Xi dup j}j=2,...,n dup. The tokens to be removed are selected to be evenly distributed across the reference canary (“evenly spread + consistent”). This ensures that the subsequences between the replaced tokens are of equal length.

4. Replacing R tokens. We here replace R tokens by sampling another token from top k = 10 tokens predicted by the MLM. Again, the tokens to be removed are selected to maximize the length of the overlapping subsequences and are consistent across fuzzy duplicates (“evenly spread + consistent”). The random tokens used for replacement differ across fuzzy duplicates.

Figure 3b shows how the exact duplicate equivalent \rho changes with the number of impacted tokens R for the four kinds of information loss. First, we find that removing the tokens from the prefix or suffix of the sequence maintain the most memorization for increasing R. This suggests that most information is retained by the model when the exact overlap between the reference canary and the fuzzy duplicates remains the largest, i.e., equal to |T(Xi ref)| - R. Notably, removing tokens from the suffix leads to slightly higher values of \rho. We hypothesize this is because the MIA score is computed for the full length reference canary (i.e., \alpha(Xi ref)). When tokens are removed only from the canary suffix instead of from its prefix, this means that the context used for token-level predictions is more consistent between model training and inference during the MIA, facilitating membership inference.

When the removed tokens are spread across the sequence, the memorization (\rho) drops quite sharply, yet remains significantly higher than the baseline memorization of \rho = 0. For instance, when R = 20 random tokens are removed (and the maximum overlapping subsequence between fuzzy duplicates and the reference canary is no more than 4 tokens), the exact duplicate equivalent still remains as high as \rho = 0.38. This shows that the target LLM is able to memorize across missing segments of text. Finally, when we replace tokens with semantically meaningful ones instead of removing them, we observe a slight increase in \rho for larger values of R. We hypothesize that token removal introduces additional noise compared to maintaining the coherence of the sequence by replacement, making it slightly harder for the LLM to memorize across fuzzy duplicates.

Previously, when replacing R tokens, we distinguished between replacements evenly and not-evenly distributed within the reference canary and consistently and not-consistently across fuzzy duplicates. We now examine the corresponding impact on memorization.

In Fig. 3b, we replace R tokens evenly distributed within the reference canary and replace the exact same tokens across fuzzy duplicates (“evenly spread + consistent”). We now also consider the cases in which the tokens to be replaced are not evenly distributed, both with the replaced tokens consistent across fuzzy duplicates (“not evenly spread + consistent”) and not consistent (“not evenly spread + not consistent”).

Figure 3c illustrates that the choice of token replacement positions has limited impact on the memorization of the fuzzy duplicates. Indeed, \rho drops very similarly with increasing values of R across all three scenarios. We do see a slight increase of \rho when the tokens to be replaced are not uniformly spread across the reference canary. We hypothesize that the maximum overlapping subsequence between the reference canary and the fuzzy duplicate to be slightly larger in this case, which would help memorization. Notably, \rho remains similarly high when the replaced tokens are not consistent across fuzzy duplicates, which shows how the target LLM memorizes across distinctly overlapping fragments of sequences.

Impact of fuzzy duplicates on data extraction
To assess memorization and privacy risks in LLMs, prior work has examined not only vulnerability to MIAs but also training data extraction15,18,32,33,58. Carlini et al.32 define a string x as extractable from a model M if there exists a prefix p such that the concatenation [p || x] appears in the training data of M, and M reproduces x exactly when prompted with p using greedy decoding. Authors further use this definition to quantify memorization across model size, sequence length and repetition. Nasr et al.58 later refer to this definition as discoverable memorization. Recognizing that exact string matching may be overly restrictive, Ippolito et al.33 introduce the notion of approximate memorization, measured by the similarity between the generated text and the ground truth continuation x.

In this work, we study memorization through vulnerability to MIAs, motivated by two main considerations. First, defenses against MIAs are known to also offer protection against more severe effects such as training data extraction78. Note that training data extraction can be studied in various ways, ranging from generating text from a given prefix using greedy decoding with exact15 or approximate92 matching to more advanced methods considering stochastic decoding97 or gradient-based optimization98. Second, also from a methodological perspective measuring memorization through MIAs is preferred, as text generation alone does not necessarily imply memorization but may instead reflect generalization99.

We further investigate how the presence of fuzzy duplicates amplifies the risk of training data extraction. Following prior work32,58, we provide the first P tokens of a reference canary T(Xi ref) as prompt p to the target model M, i.e., p = T(Xi ref)[:P], and generate the next |T(Xi ref)| - P tokens via greedy decoding. In line with Ippolito et al.33, we then measure approximate memorization by comparing the generated continuation to the true continuation x = T(Xi ref)[P:]. We compute BLEU scores at the string level33 and additionally report the normalized Levenshtein similarity on the token level, defined as one minus the Levenshtein distance divided by the continuation length.

Table 3 summarizes the extraction results for GPT-Neo 1.3B trained on a dataset that includes fuzzy duplicates of the reference canaries (using the same setup as in Fig. 1(a)). We evaluate two prompt lengths, P = 50 and P = 75, out of the total 100 tokens for each reference canary, and report the mean and standard deviation across all 100 reference canaries. We additionally consider the larger model, GPT-Neo 2.7B, which we expect to exhibit higher extractability.

We first consider approximate extraction in the presence of n dup = 10 exact repetitions. For P = 50, we find a mean BLEU score of 0.18 for GPT-Neo 1.3B and 0.31 for GPT-Neo 2.7B. Exact extraction (BLEU of 1) occurs in 4 (GPT-Neo 1.3B) and 5 (GPT-Neo 2.7B) out of the 100 reference canaries. For only n dup = 1 occurrence of the reference canaries, the mean BLEU score drops to 0.06 for both models.

Next, we find that fuzzy duplicates contribute substantially to the extractability of the reference canaries. Across models, we observe that as the number of replacements R varies, both BLEU and Levenshtein similarity remain significantly above the baseline for n dup = 1, and approach the levels observed for n dup = 10 for lower values of R. Consistent with our findings on MIA vulnerability, these results demonstrate that fuzzy duplicates also contribute meaningfully to the extraction risk of reference canaries.

Generating paraphrases as fuzzy duplicates
In the results from Table 1, we use A paraphrase to construct the fuzzy duplicates {Xi dup j}j=2,...,n dup; i.e., we query an instruction-tuned LLM for n dup - 1 paraphrases of the reference canary Xi ref. Specifically, we query the models Meta-Llama-3-8B-Instruct49, Mistral-7B-Instruct-v0.262 and GPT-4o63 using the following system and user prompts:

1. System prompt: ‘You are an assistant tasked with rephrasing the provided text in 9 different ways. Keep the original meaning intact (including the original natural or code language), but rephrase each version as if you are replacing the sentence entirely. Number the rephrased sequences using 1. to 9. and separate each by ‘—’, like this: ‘1. rephrase1 — 2. rephrase2 — .. — 9. rephrase9’.’

2. User prompt: ‘Can you rephrase the following sequence? {Xi ref}’.

When using Meta-Llama-3-8B-Instruct49 and Mistral-7B-Instruct-v0.262, we generate the answer from the model with a maximum number of new tokens of 1024 and temperature of 0.6 while sampling consecutively from the top predicted tokens for which the total predicted probability sums up to 0.9 (top-p) (as recommended in Meta-Llama-3-8B-Instruct49 model card on HuggingFace). When using GPT-4o63, we use the API’s default sampling parameters, i.e., temperature and top-p equal to 1. We query the model until the processed output leads to n dup - 1 distinct paraphrases. We then repeat the same process as in the main experiment, continuing pretraining the target model on a dataset always containing Xi ref and now its n dup - 1 paraphrases. We provide example fuzzy duplicates obtained by paraphrasing for each instruction-tuned model in Supplementary Table 3.

Lastly, to put the results from Table 1 in perspective, we also compute the n-gram overlap between the fuzzy duplicates and the reference canary for fuzzy duplicates constructed by token replacements (A replace). Specifically, we also compute the mean and standard deviation overlap in n-grams (in GPT-Neo tokens) for the results from Fig. 1d for k = |V MLM|. Note that we consider all possible n-grams in a sequence and that an n-gram can appear in the fuzzy duplicate also when it is not in the same location. The results are summarized in Table 4.

Collecting fuzzy duplicates from SlimPajama
In this section, we provide further technical details on how we have collected fuzzy duplicates present in SlimPajama, while also providing additional results.

First, to define what constitutes a fuzzy duplicate, we consider a range of potential metrics to compute the distance between sequences. A sequence is then considered a fuzzy duplicate of a target sequence if both sequences are close according to this distance.

Throughout most of the experiments presented in this work, we have introduced a predefined perturbation (A) to the reference canary and measured its impact on memorization by computing the corresponding exact duplicate equivalent \rho. Now, as we shift to analyzing a real-world dataset, we need a definition of fuzzy duplicates, i.e., a distance metric, that aligns with our prior experiments while also enabling us to estimate their impact on memorization in practical scenarios.

To this end, we consider a range of metrics to compute the distance between two sequences of tokens, and show how well each of them relate to the value of \rho. We consider the results of three of our main experiments, where we estimated \rho values for token replacement (A replace, Fig. 1a), insertion (A insert, Fig. 1b) and shuffling (A shuffle, Fig. 1c), and compute the target distance between the reference canary and its fuzzy duplicates. We then replace experiment-specific distance metrics used before (number of replaced tokens R, number of inserted tokens X insert and Kendall tau distance \tau respectively) with the target distance metric and plot the results of all three experiments on one graph with the target distance as a shared x-axis (Fig. 6a).

We consider the following distance metrics:

1. Levenshtein distance. This metric measures the minimum number of single-token operations (insertions, deletions, or substitutions) required to transform one sequence of tokens into another.

2. Levenshtein-Damerau distance. This metric extends Levenshtein distance by allowing transposition of adjacent tokens as a single operation, making it better suited for capturing typographical errors.

3. Longest Common Subsequence (LCS) distance. This metric calculates the length of the longest common subsequence (not necessarily consecutive) as the measure of similarity. The distance is then computed by subtracting the LCS length from the input length.

4. Token overlap (multiset). This metric calculates one minus the intersection of tokens between two sequences divided by the maximum number of tokens in either sequence. As such, it represents how many tokens appear in both sequences while accounting for token frequency.

5. Jaccard distance (token level). This metric represents the dissimilarity between two sequences by calculating one minus the ratio of the size of token intersection to the size of token union.

6. Jaccard distance (n-gram level). This metric measures the dissimilarity between two sequences by calculating one minus the ratio of the size of n-gram (in tokens) intersection to the size of n-gram union, capturing sequence-level differences.

Figure 6a shows that Levenshtein, Levenshtein-Damerau and LCS distances show good correlation between the distance and the \rho value across all three experiments, effectively capturing fuzzy duplicates that contribute to mosaic memory. Token overlap (multiset) and token-level Jaccard distance both perform well for token replacement and token insertions, but fail to capture shuffling—as both metrics are order-independent, they assign zero distance to all shuffled fuzzy duplicates. N-gram-level Jaccard distance (n = 13), on the other hand, suffers from the opposite problem: most sequence perturbations we have explored lead to very little overlap in 13-g, with the Jaccard distance values collapsing to near 1 for a wide range of \rho values.

To identify fuzzy duplicates in a real-world dataset, we have chosen the Levenshtein distance. This metric shows equally good correlation as other well-performing metrics, is conceptually simple and naturally encapsulates types of sequence perturbations we explored previously (token replacement, insertion and shuffling).

We now estimate the prevalence of fuzzy duplicates as defined by our selected metric in a real-world dataset used for LLM training. As a dataset, we consider SlimPajama50, which contains 627 billion GPT-Neo tokens. We then consider a sequence a fuzzy duplicate of a target sequence if their Levenshtein distance is below a certain threshold. We hypothesize that a substantial portion of the dataset would have a large number of such fuzzy duplicates, thus heavily contributing to model memorization.

First, we identify a set of target sequences for which we will seek fuzzy duplicates. These target sequences can be selected in various ways, and the exact selection criteria likely impacts the associated number of fuzzy duplicates. We hypothesize that the number of fuzzy duplicates identified for a certain target sequence has a positive correlation with the number of times this target sequence appears exactly in the dataset. Hence, we first group sequences by their number of exact repetitions in buckets to then search for fuzzy duplicates for target sequences sampled from each bucket. Note that we do not necessarily aim to show that all sequences in the dataset have fuzzy duplicates. Rather, we seek to show that a significant portion of the dataset has a large number of fuzzy duplicates, thus raising memorization concerns.

We adapt the code from Lee et al.31 to build a suffix array and find all sequences of 100 tokens repeated more than once in the SlimPajama. We then consider sequences with a certain number of exact repetitions across the dataset: {100, 1000, 10000} (\pm 1%). We estimate each of these buckets is large enough to raise concerns if many meaningful fuzzy duplicates are identified. Namely, the dataset has over 50 million sequences repeated at least 100 times, over 700,000 sequences repeated at least 1000 times, and over 30,000 sequences repeated at least 10000 times.

We then randomly pick 100 sequences from each bucket (100, 1000 and 10000 exact repetitions across the dataset) as our target sequences. To focus on meaningful memorization, we apply an additional selection criterion at this stage. Namely, we filter out sequences with too many repetitions of the same token, excluding the sequences where the number of unique tokens is below the 5th percentile across the dataset (46 unique tokens for 100-token sequence, see Fig. 6b). As such, we eliminate trivial sequences in our study, e.g. sequences where one token (often a space) is repeated many times, with only a few non-trivial tokens present.

Next, we scan the dataset with a moving window of 100 tokens and a 1-token-step, collecting all sequences with the Levenshtein distance to one of the target sequences below a certain threshold (in our case 50). As a computational optimization, we maintain the token overlap counter with the current window for each of the target sequences, allowing us to compute the relatively expensive Levenshtein distance in a rare event where the current window has at least 50 common tokens with the target sequence. We then avoid double counting by ensuring that each individual token is contributing to at most one fuzzy duplicate in the final count. To make the computation feasible on a very large dataset such as SlimPajama, we scan the first 5% of a randomly shuffled dataset and extrapolate the results, assuming fuzzy duplicates are spread mostly uniformly across the shuffled dataset–which is confirmed by our observations on the positions of the exact duplicates. The scan took 96 h on 80 CPUs on a server with sufficient memory to fit the whole 5% portion of the dataset into memory.

In the main body of the paper (Fig. 2b), we provided the number of fuzzy duplicates for target sequences sampled from the bucket of sequences with 1000 \pm 1% exact repetitions. Figure 6c now shows the cumulative number of fuzzy duplicates, averaged across sequences, for the other buckets as well, denoted as B 100, B 1000 and B 10000. Each line starts at the respective number of exact duplicates (which is the selection criteria) and grows as we increase the Levenshtein distance.

While the B 1000 bucket shows the fastest increase, all three buckets reach double the initial number of duplicates (B 100 by Levenshtein distance 8, B 1000 by 5 and B 10000 by 11). Moreover, for higher Levenshtein distances, B 100 and B 1000 reach a ten-fold increase compared to the initial number of exact duplicates (at Levenshtein distance 20 and 24 respectively).

In summary, these results show that even in datasets deduplicated on a document level, many exact duplicates still exist. For sequences repeated exactly, we furthermore recover a substantial amount of fuzzy duplicates at values of the Levenshtein distance which correspond to a meaningful contribution to memorization (see Fig. 2a).

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "This review has been performed in cooperation between junior and senior researchers, and we hence refer to ourselves in the plural \"we\". We would like to thank the authors for the interesting work on a novel and timely topic that reads very well and is very accessible. We will follow the outline of the nature review template throughout our comments. First of all the paper asks an interesting research question and seems to provide interesting insights into the fact that LLMs, or foundation models, most likely do not memorize text not solely verbatim as exact token sequences, but also supported by additionally memorizing partial patterns of the original text. The authors also investigate if the models memorize \"semantically\" (or: in the latent space), or \"syntactically\" (as sequences of input tokens). Current ML research would suggest the latter, which is also supported by experiments of the authors. We are not aware of prior work investigating these questions, and despite the fact that the second insight somewhat is to be expected, the first insight is interesting and innovative. It will have impact on the entire field of machine unlearning, as it implies that the attempt to remove representations of exact learned patterns will likely not suffice to guarantee privacy of the individuals corresponding to unlearned data. We do believe that the results are likely correct, and that the study supports the conclusions and claims the authors are making. There are, however, a few minor things to consider, and probably revise, to make sure that the report is self-contained and the study can be replicated. None of the concerns is critical, all should be easy to address by revising the description of experiments and results. First of all, the submission would benefit from further elaboration on the concept of mosaic memory. It is the main concept of the paper yet it is only briefly described in the introduction and could benefit from a more precise explanation: For instance, could you please clarify how the \"sequences\" and \"overlapping fragments\" (page 2) are defined in this context? The description given in the abstract remains rather vague. There is no further reference or discussion of the concept later-on in the text, and it remains a bit unclear which results exactly supported which claim regarding information being stored in \"mosaic memory\". Second, we believe that making the paper more self-contained can help readers who lack necessary background knowledge. The article is submitted to a journal for all interested audiences, and it is unlikely that the audiences have deep knowledge of the field already. For example, the approach employed to generate the reference canaries is not explained, only cited. In this case, it would be valuable for the reader to know exactly how they are generated and the relation they have with the training data. Canaries that do not follow the same distribution of the training data could lead to biased results (it is well known that outliers are likely memorized), and thus it would be necessary to provide the correct justification that this does not happen. At the same time, if the canaries were chosen from the same distribution, there would need to be some verification that they have not already been part of the training data, as this would entirely invalidate any results from the experiments. In addition, it would be necessary to explain how exactly the pretrained LLM M_0 is defined and how the further training on D is performed, as neither are explained. Without these explanations, we cannot be sure that there are no biases in the evaluation, and this knowledge is required for reproducibility. It also remains unclear if the original models would be required - an important detail, as some indeed are not accessible, neither as open source nor in other ways to the scientific community in general. Third, providing more intuitions on the experiments and metrics used in the evaluations can enhance the paper. For instance, please clarify how the fuzzy duplicates contribute to susceptibility to MIAs, and how that supports your claim of the memory being \"mosaic\". Regarding the metrics, there is no description of the ROC AUC or the \"MIA performance\", nor is it clear what they tell us exactly. Also, the definition of $\\nu_eq$ in \"determine $\\nu_eq$ as the value of $\\nu$ for which $\\tilde{\\phi}\\approx\\phi_eq$\" is not clear. Is $\\nu_eq$ defined as the $\\nu$ that more closely approximates $\\tilde{\\phi}$? This is not clear in the text. It remains unclear to us, over which models the fuzzy duplicates with token insertions and shuffling actually are tested? This is not specified in text or captions. The model employed could also be mentioned in the captions of the figures to improve readability. Please clarify how the baselines of the fuzzy duplicates with token insertions are defined. On page 4, what does it mean that \"n-grams are randomly scattered\" and how is the insertion of infinite elements defined? The comparison to the baseline in the shuffling experiment is also not clear. In the comparison to paraphrasing, the rho in Table 1 is compared to that of Figure 3. However, as far as we understand, the figure and table are conducted under different models. Does this ensure a fair comparison? Some minor suggestions and questions: * While the experiment studying syntactic versus semantic memorization is interesting, is the conclusion that it is predominantly syntactic actually \"surprising\"? Isn't it well understood that machine learning actually learns and produces preexistent patterns, but does not understand/learn the semantic meaning of these patterns? * In general, the section on \"Quantifying a mosaic memory\" is a bit hard to follow and could benefit from revision * Please be sure to indicate in the introduction that you are working with 100 tokens. Otherwise, sentences such as \"Levenshtein distance 10, for instance, represents just 10% of tokens\" may confuse. * Note that \"baseline\" is repeated in \"(against the baseline baseline of $\\rho=0.321$)\". * The labelling of Figure 5 would benefit from improvement. Could the actual n-gram sizes be included just next to the lines (for example, left to where they finish around x=50)? The color grading makes them very hard to distinguish. In addition, the in-text references to specific curves may not be immediately clear, especially since the first line from the bottom is almost invisible except for the slight increase near the end. It is easy then to miss that curve when finding the \"fifth line from the bottom\", which the text highlights. In addition, wouldn't the curve for n=25 mentioned on page 12 be actually the one for n=20? It seems that the curves are in 10-step intervals, but it is hard to confirm due to the similar color tones. * Please note that there are minor misprints in some references, such as 49 and 61.", "reviewer_2": "I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career Researchers who co-review manuscripts.", "reviewer_4": "Key results The motivation of the study is to reduce sensitive information leakage of LLMs due to fuzzy duplications in the training dataset. The “sensitive information” is simulated by synthetic sequence generated using an LLM, which the authors refer as “canaries”. “Memorization” is measured using Membership Inference Attacks (MIAs). Specifically, canaries along with their ‘fuzzy’ duplicates (constructed by replacement, insertion, paraphrases, etc.) are used for additional training a pre-trained LLM. Their (log) probabilities after training are compared to canaries that are not used in training. The main result of the paper is measuring the contribution of the presence of fuzzy duplicates to the memorization of the canaries. For this they introduce a relative measure of the memorization, exact duplicate equivalent (rho), which evaluate how many fuzzy duplications are equivalent to one exact duplication. The authors found that, depending on the degree of fuzziness (e.g. the number of replacements, Levenshtein distance), a small number of fuzzy repetitions can have a similar effect (e.g. sequence probability) as having one exact repetition. In the later parts of the study, the authors examine a popular deduplicated dataset and show that it still contains many fuzzy duplicated sequences. Using a more aggressive deduplication procedure, they can further remove fuzzy duplications in that dataset. Validity The experiments are mostly based on the existing methods, such as Membership Inference Attacks, Meeus, M (2024). The methods and results are valid. However, the authors may overinterpret the results out of the scope of the experiments. The actual implication of the results is vague and complicated. There is almost no statistical test on any results. The main measure, rho, is used to make most of the claims in the manuscript. However, rho measures the effect of fuzzy duplicates relative to exact duplicates. Though not claimed directly, the author imply that higher rho is undesirable, because higher rho means the fuzzy duplicates are as “detrimental” as exact duplicate. However, it is not clear how bad the exact duplicates are. Thus, it is difficult to relate rho to impact on realistic use cases. For example, rho does not tell how many (fuzzy) duplicates are sufficient to induce unsafe information leakage and with what probability. It is also unclear whether removing duplication can eliminates information leakage, given that “personal” information which is a main topic of interest in this context, is typically not widely known (repeated in the training dataset). Based on the results in Fig. 1, the authors claim that “fuzzy duplicates contribute significantly to memorization.” This is a questionable claim. The claim comes from the observation that on all “number of tokens R replaced” except 100 the rho is larger than 0. This observation could be trivial. Two points on the plot ((0, 1.0) and (100, 0)) are there by construction. Simply connecting the two points result in all middle values larger than 0. More quantitative claim “In practice this means that having two fuzzy duplicates in the training data with 10% of the original tokens replaced yields higher memorization in the than one exact repetition” is technically true, but it is not clear what the memorization effect of one exact repetition is. Another issue is that the main experiments are about finetuning pretrained LLMs (A minor terminology suggest: the authors describe their training procedure as “further pretraining” of a pretrained model. However, “Continued pretraining” is a more prevalent name than “further pretraining” by convention.). However, the role of different finetuning protocols is not discussed. For example, the learning rate could affect rho. More importantly, it is unclear whether memorization of sequences introduced in this fine-tuning is similar to memorization of sequences introduced in the complex and long pre-training process. Significance Understanding the memory of LLMs is a fundamentally important topic. A main claim of the manuscript is that the fuzzy duplicates in the training data can lead to undesired memorization. Additional interesting result is the observation that syntactic more than semantic, similarity determines the efficacy of fuzzy duplicates. However, (1) The fact that memorization can be enhanced by inexact repetition is not surprising, and is clearly demonstrated by recent work on paraphrases. The general significance of the quantitative measure adopted here is questionable, as explained above. (2) In the context of data privacy and copyright protection, memorization is undesired. However, in a broader sense, memorization is often thought to be a desired property of an LLM. Remembering more factual details can reduce model hallucination. The manuscript does not explain the tradeoff of the two. The “canaries” which are used as a probe are neutral sequences (i.e. general natural language content generated by an LLM with T=1). It is unclear whether they are a good model of desired or undesired memorized items. There are well-recognized studies showing paraphrases are necessary for generalization (for example Zeyuan Allen-Zhu 2024 part 3.1). The authors do not discuss how they fuzzy duplicates and paraphrases help generalization but only discussed their role on “memorization”. Data and methodology There is no issue on the data. The methods are clearly explained in the supplementary. The LLM models used in the experiments are two generations behind, but still acceptable. Analytical approach There is no statistical analysis except standard deviation in the tables. Adding statistical analysis to most of the plots will be helpful. Suggested improvements The authors state that the deduplication process of SlimPajama fails to even remove exact duplicates. However, the authors did not discuss potential reasons of the failure. It is surprising that a widely-used dataset fails so evidently. Some examples would be helpful to understand the failure. One potential reason I can think of is that sequence length used in the SlimPajama deduplication and in this manuscript are different, so that there are shorter sequences that are exactly the same. As the authors state in the section of “deduplication”, there is a trade-off in the the choice of deduplication approaches. However, the authors only show the effectiveness of sample-level deduplication (Fig. 5). The result in Fig. 5 is as expected but does not answer important questions like what the optimal deduplication procedure is. Clarity and context The manuscript is clearly written. References As noted in the “Significance”, references on the general role of paraphrases and generalization should be included.", "reviewer_5": "I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career Researchers who co-review manuscripts.", "reviewer_6": "There has been a long line of work analyzing the memorization of training examples in the parameters of neural networks, which in recent years has been extended to large language models (LLMs). In LLMs, such memorization typically occurs due to duplicates in the training data, i.e., the same document or string being present multiple times. While previous work has predominantly looked at memorization resulting from exact duplicates, the present paper shows that memorization happens even if the duplicates are fuzzy, for example if some of the tokens in them are swapped or the strings are rephrased versions of the same string. By carefully choosing the fuzzy duplicates to be at varying semantic distances from the original string, the authors show that memorization is rather syntactic than semantic: operations that increase semantic distance have a smaller effect on memorization than those that increase syntactic distance. For measuring memorization, the authors use the ROC AUC of different membership inference attacks (MIAs) from the literature and introduce the metric $\\rho$ to relate the memorization of fuzzy duplicates to the memorization of exact duplicates. Given the potential impacts of the memorization of training data (leakage of confidential information, copyright implications, etc.), mitigation of memorization and of membership inference and extraction attacks is an important problem. The authors show that mitigation techniques relying on exact deduplication do not suffice. This is an important contribution with potential real-world impacts, and is worth being published. On the more foundational side, the findings that point towards models remembering syntactically rather than semantically are an interesting contribution to ongoing discussions about the internal workings of LLMs and their abstraction abilities. On the methodological side, the paper is sound. The authors perform many ablation studies, which are particularly crucial for such synthetic experiments. These ablations largely show the robustness of the findings. The main methodological contribution of the paper is the duplicate equivalent metric $\\rho$, which measures how many exact duplicates a fuzzy duplicate corresponds to. This is an intuitive metric because it is mostly independent of the absolute level of memorization, and seems useful for future work. The other methodological contributions are all quite natural, but not particularly innovative. One methodological criticism that I have: generating reference canaries via sampling from an LLM could lead to generating sequences that are memorized by that LLM, which are in turn likely to occur multiple times in the training data because they are memorized. The authors use Llama 2 as the reference model, which they do not use for evaluating memorization, but it is still likely that at least some of its training data overlaps with that of the models that they evaluate. Following this logic, it might be that the canaries are sequences that are already duplicated in the models’ training data, thus distorting the $\\rho$ values. This criticism is partly addressed through ablation studies with higher sampling temperatures for the reference canaries. There are two additional types of experiments that would make the paper stronger. The first is an experiment that brings it all together. The authors show that memorization occurs with synthetic canaries and fuzzy duplicates, and show that such fuzzy duplicates occur in the LLM training dataset SlimPajama. A very natural next step, and the final proof that memorization due to fuzzy duplicates is a real problem, would be training an LLM on (a subset of) a dataset such as SlimPajama and measuring the memorization of strings with naturally occurring fuzzy duplicates. Even though the algorithms used for generating the canaries and for producing fuzzy duplicates used by the authors are reasonable, the resulting sequences still remain synthetic, and real-world text and fuzzy duplicates will look different. The authors already detect fuzzy duplicates and estimate their frequency in SlimPajama, so running their training and MIA attacks with this data doesn’t seem like a large amount of effort. This is just my outside perspective, though, and I might be overlooking challenges with such an experiment. Many of the risks of memorization only materialize if it is not only possible to determine that a string was used during training, but if (parts of) it can be extracted from the model. Carlini et al. define the concept of extractable memorization [1,2]: A string s is extractable from a model if one can construct a prompt so that the model generates s. See also Lukas et al. for the case of PII [3], and the work connecting membership and attribute inference [4,5]. In addition to MIA attacks, the authors could have also tested extraction attacks to see whether those also still work in the presence of only fuzzy duplicates. In the absence of that, it would still be good if they could (briefly) comment on the difference between membership inference and extraction. The first of those two proposed (series of) experiments is more important to me than the second, which can rather be seen as a suggestion for future work. In case the first experiment would be significantly more challenging and a larger amount of work than I am thinking right now, I would suggest accepting the paper even without it. Lastly, I have a few questions for the authors: 1. For the token replacement via RoBERTa, how do you deal with the fact that this model has a different tokenizer than some of the models that you evaluate memorization on? E.g., if you want to replace one token, but the token returned by RoBERTa is actually tokenized into two tokens by the tokenizer of the model that you evaluate? This would be good to clarify in the paper. 2. “the respective $\\rho$ also rise from 1.99 to 2.51 and 3.72, respectively” (p.8, top). Those seem to be the wrong numbers, as $\\rho$ should be between 0 and 1. Do you mean the numbers from Table 1, i.e., 0.11, 0.17, 0.30? 3. In M8 you write that you did not control the sampling parameters for GPT-4o. Was it not possible to set the temperature back when you used the API for these experiments, or does the statement only refer to the probability sum? [1] Carlini et al. Extracting Training Data from Large Language Models. 2023. USENIX [2] Nasr et al. Scalable Extraction of Training Data from (Production) Language Models. 2023. arXiv [3] Lukas et al. Analyzing Leakage of Personally Identifiable Information in Language Models. 2023. SP [4] Yeom et al. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. 2018. CSF [5] Salem et al. SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learning. 2023. SP"}, "author_response": "The Mosaic Memory of Large Language Models (NCOMMS-25-29287): Response to reviewers We thank the reviewers for their valuable feedback and the time they dedicated to our manuscript. In this document, we carefully respond to each reviewer’s comments. Specifically, for each reviewer, we reproduce their comments verbatim and then provide our responses in blue. Our responses either explain how we have addressed the feedback in the revised manuscript or offer additional clarifications where appropriate. Reviewer #1 (Remarks to the Author and Authors’ Response) This review has been performed in cooperation between junior and senior researchers, and we hence refer to ourselves in the plural \"we\". We would like to thank the authors for the interesting work on a novel and timely topic that reads very well and is very accessible. We thank the reviewers for their time spent on the manuscript and for their constructive and thoughtful feedback, which helped us improve the manuscript. We will follow the outline of the nature review template throughout our comments. First of all the paper asks an interesting research question and seems to provide interesting insights into the fact that LLMs, or foundation models, most likely do not memorize text not solely verbatim as exact token sequences, but also supported by additionally memorizing partial patterns of the original text. The authors also investigate if the models memorize \"semantically\" (or: in the latent space), or \"syntactically\" (as sequences of input tokens). Current ML research would suggest the latter, which is also supported by experiments of the authors. We are not aware of prior work investigating these questions, and despite the fact that the second insight somewhat is to be expected, the first insight is interesting and innovative. It will have impact on the entire field of machine unlearning, as it implies that the attempt to remove representations of exact learned patterns will likely not suffice to guarantee privacy of the individuals corresponding to unlearned data. We agree and have now added a paragraph in the Discussion section highlighting the implications of our findings for the field of machine unlearning. We do believe that the results are likely correct, and that the study supports the conclusions and claims the authors are making. There are, however, a few minor things to consider, and probably revise, to make sure that the report is self-contained and the study can be replicated. None of the concerns is critical, all should be easy to address by revising the description of experiments and results. 1 First of all, the submission would benefit from further elaboration on the concept of mosaic memory. It is the main concept of the paper yet it is only briefly described in the introduction and could benefit from a more precise explanation: For instance, could you please clarify how the \"sequences\" and \"overlapping fragments\" (page 2) are defined in this context? The description given in the abstract remains rather vague. There is no further reference or discussion of the concept later-on in the text, and it remains a bit unclear which results exactly supported which claim regarding information being stored in \"mosaic memory\". We thank the reviewer for highlighting this point. We agree that the concept of mosaic memory – one of the key concepts in our work – was not defined sufficiently in the original version. We have therefore expanded its description in the Introduction (second paragraph). Specifically, we define mosaic memory as the model’s ability to memorize an arbitrary sequence of tokens through exposure to its fuzzy duplicates -- fragments of text similar to the original sequence with some tokens missing, replaced, or shuffled. Mosaic memory manifests in the increased memorization from observing fuzzy duplicates during training – similarly to how exact repetitions amplify it. Second, we believe that making the paper more self-contained can help readers who lack necessary background knowledge. The article is submitted to a journal for all interested audiences, and it is unlikely that the audiences have deep knowledge of the field already. For example, the approach employed to generate the reference canaries is not explained, only cited. In this case, it would be valuable for the reader to know exactly how they are generated and the relation they have with the training data. We thank the reviewer for this helpful suggestion. We agree that making the paper more self-contained will benefit readers who may not have detailed background knowledge of the field. We have therefore added clarifications in the main text (section “A metric to study the memorization of fuzzy duplicates”), and now more explicitly refer to the detailed experimental setup in sections M1-6. Canaries that do not follow the same distribution of the training data could lead to biased results (it is well known that outliers are likely memorized), and thus it would be necessary to provide the correct justification that this does not happen. We agree that the canaries are not drawn from the same distribution as the training data, as they are synthetically generated rather than written by human authors. However, we believe this does not affect the validity of our findings. Our analysis focuses on the relative memorization of fuzzy duplicates, measured using the exact duplicate equivalent ($\\rho$), which is designed to be invariant to the absolute memorization level of individual sequences. In other words, $\\rho$ captures how fuzzy duplicates influence memorization regardless of how inherently “memorizable” a sequence is. Our main experiments use canaries generated with temperature 1, producing text sampled from the base model’s learned distribution and not intended to be outliers. In Section M6, we further vary the generation temperature between 1 and 5, where higher temperatures yield rarer, more unexpected sequences that are indeed more prone to memorization. The results 2 remain consistent across this range, confirming that $\\rho$ is robust to changes in the absolute memorization level. At the same time, if the canaries were chosen from the same distribution, there would need to be some verification that they have not already been part of the training data, as this would entirely invalidate any results from the experiments. We thank the reviewer for raising this subtle but important point. While it is unlikely that any of our synthetically generated 100-token canaries (sampled with temperature 1) appear verbatim in the pretraining data, we agree that, given the immense scale of modern datasets, it is difficult to rule out that some exact or fuzzy duplicates duplicates might exist. We respectfully disagree, however, that such overlap would invalidate our findings. Our experiments were specifically designed to remain valid even in this case. First, we perform the MIA specifically against the continued pretraining stage, where we explicitly control the injection of canaries. Both member and non-member canaries are generated using the same process and are therefore equally likely to have (fuzzy) duplicates in the pretraining corpus. Any overlap would thus affect both groups equally and not invalidate the MIA. Second, while such overlap could in principle affect the absolute memorization level, our analysis focuses on relative memorization – quantified through $\\rho$ – which measures how fuzzy duplicates compare to exact duplicates under the same setup. As shown in Section M6, our results are consistent across a range of hyperparameters that influence the overall level of memorization, confirming that $\\rho$ is robust to such variations. This includes the case in which canaries are generated with a temperature up to 5, where overlap with pretraining data becomes exceedingly unlikely. In addition, it would be necessary to explain how exactly the pretrained LLM M_0 is defined and how the further training on D is performed, as neither are explained. Without these explanations, we cannot be sure that there are no biases in the evaluation, and this knowledge is required for reproducibility. It also remains unclear if the original models would be required - an important detail, as some indeed are not accessible, neither as open source nor in other ways to the scientific community in general. We thank the reviewer for pointing this out and agree that additional details would improve clarity and reproducibility. We have now expanded Section M4, where we provide a detailed description of the data collection and model training procedure, along with all relevant hyperparameters. We also now explicitly refer to this in the main text. For the model availability, we only consider open-weights models, available on Huggingface (GPT-Neo (EleutherAI), Gemma-2B (Google), Phi-2 (Microsoft), and Llama-3.2-1B (Meta)). We furthermore ensured that all our experiments are fully reproducible using the open-source code we provide alongside the paper, and only rely on publicly accessible data and pretrained models. 3 Third, providing more intuitions on the experiments and metrics used in the evaluations can enhance the paper. For instance, please clarify how the fuzzy duplicates contribute to susceptibility to MIAs, and how that supports your claim of the memory being \"mosaic\". Regarding the metrics, there is no description of the ROC AUC or the \"MIA performance\", nor is it clear what they tell us exactly. Also, the definition of $\\nu_eq$ in \"determine $\\nu_eq$ as the value of $\\nu$ for which $\\tilde{\\phi}\\approx\\phi_eq$\" is not clear. Is $\\nu_eq$ defined as the $\\nu$ that more closely approximates $\\tilde{\\phi}$? This is not clear in the text. We thank the reviewer for highlighting the parts of the description that were unclear in the original version. We have therefore added a paragraph in the section “A metric to study the memorization of fuzzy duplicates” explaining the connection between MIA AUC, memorization, and the effect of duplicates on it. We have also clarified the definition of $\\nu_{eq}$ as the value of $\\nu$ for which $\\phi_{\\nu}$ best approximates $\\tilde{\\phi}$. It remains unclear to us, over which models the fuzzy duplicates with token insertions and shuffling actually are tested? This is not specified in text or captions. The model employed could also be mentioned in the captions of the figures to improve readability. We agree that this should have been made clearer in the manuscript. The experiments related to token insertions and shuffling have been conducted with GPT-Neo 1.3B as target model. We have updated captions for all relevant figures in the main text as well as in Methods to clarify which model has been used. Please clarify how the baselines of the fuzzy duplicates with token insertions are defined. On page 4, what does it mean that \"n-grams are randomly scattered\" and how is the insertion of infinite elements defined? The comparison to the baseline in the shuffling experiment is also not clear. We thank the reviewer for pointing this out. The baseline for both the experiments (Figures 2a, 2b) corresponds to the case where n-grams are learned in isolation, without any associations between them. For this we select the injection position for each n-gram independently, making it very unlikely for any two n-grams from the same canary to appear within one context window during training. If $\\rho$ exceeds this baseline, it indicates that the target model has learned some connections between the n-grams, piecing together information across insertions and reorderings, contributing to the memorization of the reference canaries. We have now clarified the meaning of this baseline for both the token insertion and shuffling experiments in the manuscript. In the comparison to paraphrasing, the rho in Table 1 is compared to that of Figure 3. However, as far as we understand, the figure and table are conducted under different models. Does this ensure a fair comparison? 4 We thank the reviewer for catching this and apologize for not making it clear in the original manuscript. Both reported results (the values of the exact duplicate equivalent $\\rho$) correspond to using the same target model, namely GPT-Neo 1.3B. To avoid confusion, we now clarify that the different models mentioned in Table 1 refer to the instruction-tuned models used to paraphrase the reference canaries, rather than to the target model itself. We have updated the caption of Table 1 accordingly. Some minor suggestions and questions: ● While the experiment studying syntactic versus semantic memorization is interesting, is the conclusion that it is predominantly syntactic actually \"surprising\"? Isn't it well understood that machine learning actually learns and produces preexistent patterns, but does not understand/learn the semantic meaning of these patterns? Thank you for this comment. We believe this result may be surprising to some in the community, but agree that it also may be expected by others. On the one hand, recent advances in LLMs suggest they go far beyond simple pattern matching, demonstrating increasingly abstract understanding and strong reasoning capabilities. Prior work has reported ‘emergent capabilities’ [A1], with models showing near human-level performance across domains [A2] and advanced reasoning abilities [A3, A4, A5], and even achieving a gold medal in the International Mathematical Olympiad [A6]. From this perspective, it may seem surprising that LLMs do not memorize more substantially across semantically coherent fuzzy duplicates. On the other hand, Bender et al. [A7] described language models as stochastic parrots, i.e. “a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning.” Notable recent work echoes similar skepticism on recently reported capabilities, suggesting that emergent abilities may be artifacts of model evaluation [A8] and that current models’ reasoning breaks down as problem complexity increases [A9]. ● In general, the section on \"Quantifying a mosaic memory\" is a bit hard to follow and could benefit from revision As also mentioned above, we have added further clarifications to this section in the main text, and now more explicitly refer to the relevant details in the Methods. ● Please be sure to indicate in the introduction that you are working with 100 tokens. Otherwise, sentences such as \"Levenshtein distance 10, for instance, represents just 10% of tokens\" may confuse. Addressed. ● Note that \"baseline\" is repeated in \"(against the baseline baseline of $\\rho=0.321$)\". Addressed. 5 ● The labelling of Figure 5 would benefit from improvement. Could the actual n-gram sizes be included just next to the lines (for example, left to where they finish around x=50)? The color grading makes them very hard to distinguish. In addition, the in-text references to specific curves may not be immediately clear, especially since the first line from the bottom is almost invisible except for the slight increase near the end. It is easy then to miss that curve when finding the \"fifth line from the bottom\", which the text highlights. In addition, wouldn't the curve for n=25 mentioned on page 12 be actually the one for n=20? It seems that the curves are in 10-step intervals, but it is hard to confirm due to the similar color tones. We thank the reviewers for the suggestion. We have now labeled individual lines with the corresponding values of n. We have also corrected the reference to the n=20 curve on page 12 – the curves are indeed in 10-step intervals. ● Please note that there are minor misprints in some references, such as 49 and 61. Addressed. Additional references used to respond to reviewer #1 [A1] Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., ... & Fedus, W. Emergent Abilities of Large Language Models. Transactions on Machine Learning Research. [A2] Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., ... & Zhang, Y. (2023). Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712. [A3] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). Large language models are zero-shot reasoners. Advances in neural information processing systems, 35, 22199-22213. [A4] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., ... & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, 24824-24837. [A5] Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., ... & Tan, Y. (2025). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081), 633-638. [A6] Luong, T., & Lockhart, E. (2025, July 21). Advanced version of Gemini with Deep Think officially achieves gold-medal standard at the International Mathematical Olympiad. Google DeepMind. [A7] Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021, March). On the dangers of stochastic parrots: Can language models be too big?🦜. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (pp. 610-623). [A8] Schaeffer, R., Miranda, B., & Koyejo, S. (2023). Are emergent abilities of large language models a mirage?. Advances in neural information processing systems, 36, 55565-55581. [A9] Shojaee, P., Mirzadeh, I., Alizadeh, K., Horton, M., Bengio, S., & Farajtabar, M. (2025). The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv preprint arXiv:2506.06941. 6 [A10] Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., ... & Bowman, S. R. (2023). Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296. —---------------------- Reviewer #4 (Remarks to the Author and Authors’ Response) We thank the reviewer for their time spent on the manuscript and for their constructive and thoughtful feedback, which helped us improve the manuscript. Key results The motivation of the study is to reduce sensitive information leakage of LLMs due to fuzzy duplications in the training dataset. The “sensitive information” is simulated by synthetic sequence generated using an LLM, which the authors refer as “canaries”. We would like to clarify that the notion of canaries, i.e. unique and specifically crafted sequences injected into language model training data to study memorization/susceptibility to MIAs, has been used extensively in the privacy literature [14, 22, 46, 47, 48, 49, 50]. In this work, we adopt the technique to synthetically generate canaries as used by Meeus et al. [22]. “Memorization” is measured using Membership Inference Attacks (MIAs). Specifically, canaries along with their ‘fuzzy’ duplicates (constructed by replacement, insertion, paraphrases, etc.) are used for additional training a pre-trained LLM. Their (log) probabilities after training are compared to canaries that are not used in training. The main result of the paper is measuring the contribution of the presence of fuzzy duplicates to the memorization of the canaries. For this they introduce a relative measure of the memorization, exact duplicate equivalent ($\\rho$), which evaluate how many fuzzy duplications are equivalent to one exact duplication. The authors found that, depending on the degree of fuzziness (e.g. the number of replacements, Levenshtein distance), a small number of fuzzy repetitions can have a similar effect (e.g. sequence probability) as having one exact repetition. In the later parts of the study, the authors examine a popular deduplicated dataset and show that it still contains many fuzzy duplicated sequences. Using a more aggressive deduplication procedure, they can further remove fuzzy duplications in that dataset. Validity The experiments are mostly based on the existing methods, such as Membership Inference Attacks, Meeus, M (2024). The methods and results are valid. However, the authors may overinterpret the results out of the scope of the experiments. The actual implication of the results is vague and complicated. There is almost no statistical test on any results. We agree on the importance of demonstrating statistical robustness and reproducibility, as highlighted by the reviewer. 7 We now report, in all of our main results (Figures 1–3, Table 1), the mean and standard deviation across five independent runs, each with randomness in both canary generation and model initialization. These show the results and findings to be consistent across runs. For experiments using real data (Figure 4b), when estimating the average number of fuzzy duplicates in SlimPajama, we now report the standard error of the mean. The main measure, $\\rho$, is used to make most of the claims in the manuscript. However, $\\rho$ measures the effect of fuzzy duplicates relative to exact duplicates. Though not claimed directly, the author imply that higher $\\rho$ is undesirable, because higher $\\rho$ means the fuzzy duplicates are as “detrimental” as exact duplicate. We would like to clarify that our goal in this work is not to make a normative judgment about memorization in language models. As discussed in the Introduction, memorization can be beneficial, for instance by improving generalization [8–10] and supporting the encoding of factual knowledge [11, 12]. However, it can also raise concerns when it involves the retention of personal or confidential information [14–19], copyright-protected material [20–22], or benchmark data [23–29]. In this study, we aim to understand the mechanisms through which memorization occurs and to place them in the context of widely used deduplication practices employed by model developers to mitigate such effects. However, it is not clear how bad the exact duplicates are. Thus, it is difficult to relate $\\rho$ to impact on realistic use cases. For example, $\\rho$ does not tell how many (fuzzy) duplicates are sufficient to induce unsafe information leakage and with what probability. We thank the reviewer for raising this point. We agree that the exact effect of duplicates on memorization depends on the specific training setup (e.g., model size, dataset size, hyperparameters). We motivate our work based on two key findings from prior research. First, training data can, under certain conditions, be extracted from LLMs. Carlini et al. (2021) [15] demonstrated this for GPT-2, and subsequent work by Nasr et al. (2023) [57] showed that ChatGPT (gpt-3.5-turbo) is similarly vulnerable. Other studies focusing on private [18, 19] and copyright-protected [20, 21] content further confirm that the threat of training data extraction is realistic and observable in real-world models. Second, prior research has shown that exact duplication of training samples increases susceptibility to extraction [30, 32] and membership inference [22] attacks. Taken together, these findings motivate our choice to measure the impact of fuzzy duplicates relative to exact duplicates, as the latter have been empirically linked to higher privacy risk in realistic settings. Moreover, exact deduplication practices are widely applied, not only to mitigate these privacy concerns [30], but also to enhance training efficiency [31, 35–39] and avoid benchmark contamination [40–44]. We thus believe it valuable to contextualize the effect of fuzzy duplicates in relation to these established practices. It is also unclear whether removing duplication can eliminates information leakage, given that “personal” information which is a main topic of interest in this context, is typically not widely known (repeated in the training dataset). 8 We thank the reviewer for raising this point. We agree that Personally Identifiable Information (PII), while a major concern in the context of memorization, might indeed be less likely to be highly repeated due to its nature. However, this does not eliminate the risk entirely – some degree of repetition can still occur. For instance, medical records may appear across multiple databases, or emails and physical addresses may be stored on different websites. Given the sensitive nature of such data and the sheer scale of modern training corpora, even rare instances of memorization are cause for concern. Beyond LLM pretraining, models are also increasingly fine-tuned on smaller, domain-specific datasets e.g. from the medical, legal or financial domain or human-chatbot interactions. Fuzzy duplicates are also likely to occur in this setting (e.g. the same sensitive information appearing in different files), and the risk of not addressing the effect of fuzzy duplicates/memorization in such cases might be larger than in pretraining. Moreover, personal information is not the only domain where memorization poses a problem. The memorization of copyright-protected content is increasingly recognized as a significant issue, with recent work showing that texts such as Harry Potter and 1984 are almost fully memorized by LLaMA-3.1 [A10]. In this context, both exact and fuzzy duplication are likely to occur, further amplifying the risk. Based on the results in Fig. 1, the authors claim that “fuzzy duplicates contribute significantly to memorization.” This is a questionable claim. The claim comes from the observation that on all “number of tokens R replaced” except 100 the $\\rho$ is larger than 0. This observation could be trivial. Two points on the plot ((0, 1.0) and (100, 0)) are there by construction. Simply connecting the two points result in all middle values larger than 0. We would like to clarify that our key finding is not merely that all values of $\\rho$ remain above 0, but the quantitative characterization of the impact of a wide range of fuzzy duplicates on memorization. The key contribution lies in the shape of the curve and what it reveals about how memorization behaves as perturbations increase. If memorization were predominantly verbatim, as commonly assumed in prior work, the curve would drop sharply once tokens are replaced, reflecting a rapid loss of memorization. Instead, we observe a gradual decay, showing that even substantially altered sequences continue to reinforce memorization. We further quantify this effect across different types of fuzzy duplicates (replacement, insertion, and shuffling) and across a wide spectrum of perturbation levels (e.g., R ranging from 1 to 75). For instance, as shown in Figure 1, ten fuzzy duplicates with 20% of their tokens replaced yield a memorization level equivalent to repeating the canary four times exactly. We believe this systematic quantification along this entire spectrum of perturbations provides non-trivial and informative insights about how partial overlaps reinforce memorization. More quantitative claim “In practice this means that having two fuzzy duplicates in the training data with 10% of the original tokens replaced yields higher memorization in the than one exact repetition” is technically true, but it is not clear what the memorization effect of one exact repetition is. We thank the reviewer for raising this point. We agree that it is helpful to provide clearer context for the effect of exact repetitions. We have therefore added an explicit reference to 9 Section M5, which reports the relationship between exact repetitions and vulnerability to MIAs. As shown in Figure M1, the MIA AUC increases from 0.58 for one exact repetition to 0.85 for five repetitions and 0.95 for ten repetitions, illustrating how repeated exposure progressively amplifies memorization. Another issue is that the main experiments are about finetuning pretrained LLMs (A minor terminology suggest: the authors describe their training procedure as “further pretraining” of a pretrained model. However, “Continued pretraining” is a more prevalent name than “further pretraining” by convention.). We agree that continued pretraining is a more widely used term, and have now adopted this naming convention throughout the paper. However, the role of different finetuning protocols is not discussed. For example, the learning rate could affect $\\rho$. We agree with the reviewer that different fine-tuning protocols can influence memorization. We therefore examine in Section M6 how varying the learning rate affects $\\rho$ (Figure M3), along with additional analyses across different model families, model sizes, and canary properties. These results show that our findings are consistent across a range of training configurations. More importantly, it is unclear whether memorization of sequences introduced in this fine-tuning is similar to memorization of sequences introduced in the complex and long pre-training process. We thank the reviewer for raising this important point. We agree that this is indeed a limitation of our study. Our experiments focus on memorization during continued pretraining, and it remains an open question how these dynamics might differ during full-scale pretraining. However, we chose this setup as continued pretraining closely approximates later stages of pretraining while remaining practical and methodologically tractable. In particular: 1. Conceptual proximity to pretraining. We use the same training objective and loss function as in pretraining (i.e. next-token prediction), rather than post-training methods such as reinforcement learning, making our setup closely resemble the late stages of pretraining. 2. Computational feasibility. Our experiments involve over 100 model training runs across multiple setups and types of fuzzy duplicates. Performing such an analysis at full pretraining scale would be prohibitively expensive, especially within an academic budget. 3. Prior work. We note that many prior works have studied MIAs in a continued pretraining setting for these same reasons [17, 18, A5–A7]. 4. Control over data distributions. Evaluating MIAs requires precise control over which samples are included (members) and excluded (non-members), ensuring both are drawn from the same distribution. Prior work has evaluated MIAs at the pretraining level using data 10 collected post-hoc, i.e. before and after the training data cutoff date [52, 72]. Yet, this approach introduces a severe distribution shift, which was recently shown to yield misleading conclusions [A1-A4]. Especially the fine-grained control required to study the wide range of fuzzy duplicates in this work, is for us only achievable in a continued pretraining setup. 5. Focus on relative effects. Our analysis quantifies the relative impact of fuzzy versus exact duplicates, abstracting away from the absolute level of memorization – which indeed may differ between early and late-stage pretraining. We hypothesize that the relative effects are less sensitive to the training stage, though we leave a detailed investigation of this question to future work. Significance Understanding the memory of LLMs is a fundamentally important topic. A main claim of the manuscript is that the fuzzy duplicates in the training data can lead to undesired memorization. Additional interesting result is the observation that syntactic more than semantic, similarity determines the efficacy of fuzzy duplicates. However, (1) The fact that memorization can be enhanced by inexact repetition is not surprising, and is clearly demonstrated by recent work on paraphrases. The general significance of the quantitative measure adopted here is questionable, as explained above. We thank the reviewer for highlighting this connection to Allen-Zhu et al. [A8]. We view our work as complementary to this work [A8], as it focuses on a different aspect of inexact duplication. Their study focuses on knowledge augmentation, i.e. adding paraphrases, sentence shuffles, or translations during pretraining, to improve generalization and factual recall after instruction tuning. In contrast, our goal is to understand how fuzzy duplicates affect verbatim sequence memorization. We (i) quantify this effect relative to the effect of exact duplicates across many modification types, (ii) find that primarily syntactic (rather than semantic) modifications drive this effect and (iii) connect this to commonly deployed deduplication mechanisms. To highlight the contrast between the two works, Allen-Zhu et al. show that including inexact duplicates can reduce reliance on verbatim recall, helping the model transition from merely reproducing input sequences to reasoning about underlying facts. In contrast, our results show that from a membership-inference perspective, the inclusion of inexact duplicates actually strengthens memorization. While, following results on knowledge extraction from [A8], one might find it intuitive that inexact repetition contributes to memorization, our work provides a systematic quantitative characterization of this effect, revealing clear differences across duplicate types and their implications for data deduplication. (2) In the context of data privacy and copyright protection, memorization is undesired. However, in a broader sense, memorization is often thought to be a desired property of an LLM. Remembering more factual details can reduce model hallucination. The manuscript does not explain the tradeoff of the two. The “canaries” which are used as a probe are 11 neutral sequences (i.e. general natural language content generated by an LLM with T=1). It is unclear whether they are a good model of desired or undesired memorized items. We agree and intentionally did not assign normative judgments to memorization. Prior work (e.g., [9]) shows memorization can be beneficial (e.g. supporting generalization, recalling facts) or undesirable (privacy/copyright leakage). As also noted in [A8], an LLM remembering the birthday of Abraham Lincoln is desirable, while recalling that of a non-public individual is not. Our aim is to study and quantify the mechanism: how fuzzy duplicates affect memorization, irrespective of whether the memorized content is ultimately helpful or harmful. To that end, we primarily use neutral canaries, not because they model all desirable/undesirable items, but because they approximate representative training data. We vary the temperature with which the canaries have been generated in Section M6 and Figure M6 – and conclude that our findings hold across a range of different canary generation scenarios. Without making any normative judgements, we observe that many works deduplicate data in practice for different reasons: to reduce privacy risks, improve efficiency, and ensure fair model evaluation. Our results show that exact deduplication often fails to remove impactful fuzzy duplicates, which continue to influence model memorization. Hence, our findings are relevant to deduplication practices regardless of their specific motivation. There are well-recognized studies showing paraphrases are necessary for generalization (for example Zeyuan Allen-Zhu 2024 part 3.1). The authors do not discuss how they fuzzy duplicates and paraphrases help generalization but only discussed their role on “memorization”. We indeed focus on how fuzzy duplicates contribute to memorization rather than to generalization. This perspective led to findings such as the quantitative comparison of fuzzy and exact duplicates, and the observation that syntactic, rather than semantic, overlap most strongly drives memorization. Building up on the results from Allen-Zhu et al. and studying how fuzzy duplicates affect generalization (e.g. in terms of knowledge extraction, performance on unseen benchmarks) would be an interesting direction for future research beyond the scope of this work. Data and methodology There is no issue on the data. The methods are clearly explained in the supplementary. The LLM models used in the experiments are two generations behind, but still acceptable. Analytical approach There is no statistical analysis except standard deviation in the tables. Adding statistical analysis to most of the plots will be helpful. Suggested improvements The authors state that the deduplication process of SlimPajama fails to even remove exact duplicates. However, the authors did not discuss potential reasons of the failure. It is surprising that a widely-used dataset fails so evidently. Some examples would be helpful to understand the failure. One potential reason I can think of is that sequence 12 length used in the SlimPajama deduplication and in this manuscript are different, so that there are shorter sequences that are exactly the same. We thank the reviewer for this comment – this is indeed one of the more interesting aspects of our findings, and we are happy to clarify it further. The key distinction, that we highlight in the paper, is that SlimPajama was subject to document-level deduplication, removing documents with a 13-gram Jaccard similarity above 0.8, while we look for sequence-level duplicates that persist within or across documents. To address the reviewer’s feedback we expand and clarify this distinction in the paper. We also elaborate on this in the Discussion section. We additionally include example sequences with their fuzzy duplicates in Table M8, illustrating potential mechanisms through which such duplicates can emerge in the dataset. As the authors state in the section of “deduplication”, there is a trade-off in the choice of deduplication approaches. However, the authors only show the effectiveness of sample-level deduplication (Fig. 5). The result in Fig. 5 is as expected but does not answer important questions like what the optimal deduplication procedure is. Our goal with Figure 5 is to quantify how much current exact deduplication practices miss in terms of fuzzy duplicates, rather than to propose an optimal deduplication strategy. As discussed in the Discussion section, the optimal level of deduplication inherently involves application- and scale-specific trade-offs. For example, when deduplicating fine-tuning data in sensitive domains (e.g., medical records) or when a large AI lab seeks to rigorously decontaminate evaluation benchmarks, one may prioritize stricter exact deduplication thresholds (i.e., smaller n). In such settings, the datasets are often relatively small, making it even feasible to use more computationally intensive fuzzy deduplication methods such as those based on Levenshtein distance. In contrast, for large-scale public pretraining, where deduplication primarily targets training efficiency and model quality, both the computational cost of sequence-level matching (e.g., Levenshtein across trillions of tokens) and the data loss incurred by using small n-values can outweigh the benefits. In such cases, allowing some fuzzy duplicates to remain may be a reasonable compromise. We therefore refrain from prescribing a universal “best” deduplication strategy, as the appropriate balance depends strongly on the scale and purpose of the application. A comprehensive exploration of these trade-offs would require substantial computational resources, and any resulting conclusions would likely depend on the specific use case; we therefore leave such an analysis to future work. We outline key considerations affecting this trade-off in the Discussion section while deliberately avoiding prescriptive recommendations of a universally optimal strategy. Additional references used to respond to reviewer #4 [A1] Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., ... & Hajishirzi, H. Do Membership Inference Attacks Work on Large Language Models?. In First Conference on Language Modeling. 13 [A2] Meeus, M., Shilov, I., Jain, S., Faysse, M., Rei, M., & de Montjoye, Y. A. (2025, April). Sok: Membership inference attacks on llms are rushing nowhere (and how to fix it). In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) (pp. 385-401). IEEE. [A3] Maini, P., Jia, H., Papernot, N., & Dziedzic, A. (2024). LLM Dataset Inference: Did you train on my dataset?. Advances in Neural Information Processing Systems, 37, 124069-124092. [A4] Das, D., Zhang, J., & Trantèr, F. (2025, May). Blind baselines beat membership inference attacks for foundation models. In 2025 IEEE Security and Privacy Workshops (SPW) (pp. 118-125). IEEE. [A5] Mattern, J., Mireshghallah, F., Jin, Z., Schoelkopf, B., Sachan, M., & Berg-Kirkpatrick, T. (2023, July). Membership Inference Attacks against Language Models via Neighbourhood Comparison. In Findings of the Association for Computational Linguistics: ACL 2023 (pp. 11330-11343). [A6] Zeng, S., Li, Y., Ren, J., Liu, Y., Xu, H., He, P., ... & Yin, D. (2024, August). Exploring Memorization in Fine-tuned Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 3917-3948). [A7] Dentan, J., Buscaldi, D., Shabou, A., & Vanier, S. (2024). Predicting and analyzing memorization within fine-tuned Large Language Models. arXiv preprint arXiv:2409.18858. [A8] Allen-Zhu, Z., & Li, Y. (2024, July). Physics of language models: part 3.1, knowledge storage and extraction. In Proceedings of the 41st International Conference on Machine Learning (pp. 1067-1077). [A9] Cooper, A. F., Gokaslan, A., Ahmed, A., Cyphert, A. B., De Sa, C., Lemley, M. A., ... & Liang, P. (2025). Extracting memorized pieces of (copyrighted) books from open-weight language models. arXiv preprint arXiv:2505.12546. —---------------------- Reviewer #6 (Remarks to the Author and Authors’ Response) We thank the reviewer for their time spent on the manuscript and for their constructive and thoughtful feedback. There has been a long line of work analyzing the memorization of training examples in the parameters of neural networks, which in recent years has been extended to large language models (LLMs). In LLMs, such memorization typically occurs due to duplicates in the training data, i.e., the same document or string being present multiple times. While previous work has predominantly looked at memorization resulting from exact duplicates, the present paper shows that memorization happens even if the duplicates are fuzzy, for example if some of the tokens in them are swapped or the strings are rephrased versions of the same string. By carefully choosing the fuzzy duplicates to be at varying semantic distances from the original string, the authors show that memorization is rather syntactic than semantic: operations that increase semantic distance have a smaller effect on memorization than those that increase syntactic distance. For measuring memorization, the authors use the ROC AUC of different membership inference attacks (MIAs) from the literature and introduce the metric $\\rho$ to relate the memorization of fuzzy duplicates to the memorization of exact duplicates. 14 Given the potential impacts of the memorization of training data (leakage of confidential information, copyright implications, etc.), mitigation of memorization and of membership inference and extraction attacks is an important problem. The authors show that mitigation techniques relying on exact deduplication do not suffice. This is an important contribution with potential real-world impacts, and is worth being published. On the more foundational side, the findings that point towards models remembering syntactically rather than semantically are an interesting contribution to ongoing discussions about the internal workings of LLMs and their abstraction abilities. On the methodological side, the paper is sound. The authors perform many ablation studies, which are particularly crucial for such synthetic experiments. These ablations largely show the robustness of the findings. The main methodological contribution of the paper is the duplicate equivalent metric $\\rho$, which measures how many exact duplicates a fuzzy duplicate corresponds to. This is an intuitive metric because it is mostly independent of the absolute level of memorization, and seems useful for future work. The other methodological contributions are all quite natural, but not particularly innovative. We would like to thank the reviewer for their positive assessment and for recognizing the methodological soundness of our work. One methodological criticism that I have: generating reference canaries via sampling from an LLM could lead to generating sequences that are memorized by that LLM, which are in turn likely to occur multiple times in the training data because they are memorized. The authors use Llama 2 as the reference model, which they do not use for evaluating memorization, but it is still likely that at least some of its training data overlaps with that of the models that they evaluate. Following this logic, it might be that the canaries are sequences that are already duplicated in the models’ training data, thus distorting the $\\rho$ values. This criticism is partly addressed through ablation studies with higher sampling temperatures for the reference canaries. We thank the reviewer for raising this subtle but important point. While it is unlikely that any of our synthetically generated 100-token canaries (sampled with temperature 1) appear verbatim in the pretraining data, we agree that, given the immense scale of modern datasets, it is difficult to entirely rule out the presence of some exact or fuzzy duplicates. We would, however, like to clarify that such potential overlap does not compromise the validity of our results. The experimental design was intentionally constructed to remain robust even under this possibility. First, the MIAs are conducted specifically against the continued pretraining stage, where we explicitly control which canaries are injected. Because both member and non-member canaries are generated using the same process, they are equally likely to have overlaps with the pretraining corpus. Any such, overlap would therefore affect both groups symmetrically and not bias the evaluation. Second, although such overlap might influence the absolute degree of memorization, our main analysis concerns relative memorization – captured by $\\rho$ – which quantifies how fuzzy duplicates compare to exact duplicates within the same setup. As shown in Section M6, our 15 results remain consistent across a wide range of hyperparameters that modulate overall memorization strength, demonstrating that $\\rho$ is stable under such conditions. This includes the case in which canaries are generated with a temperature up to 5, where overlap with pretraining data becomes exceedingly unlikely. There are two additional types of experiments that would make the paper stronger. The first is an experiment that brings it all together. The authors show that memorization occurs with synthetic canaries and fuzzy duplicates, and show that such fuzzy duplicates occur in the LLM training dataset SlimPajama. A very natural next step, and the final proof that memorization due to fuzzy duplicates is a real problem, would be training an LLM on (a subset of) a dataset such as SlimPajama and measuring the memorization of strings with naturally occurring fuzzy duplicates. Even though the algorithms used for generating the canaries and for producing fuzzy duplicates used by the authors are reasonable, the resulting sequences still remain synthetic, and real-world text and fuzzy duplicates will look different. The authors already detect fuzzy duplicates and estimate their frequency in SlimPajama, so running their training and MIA attacks with this data doesn’t seem like a large amount of effort. This is just my outside perspective, though, and I might be overlooking challenges with such an experiment. Thank you for this great suggestion. We agree that it is valuable to confirm that our findings for synthetic reference canaries, and their constructed fuzzy duplicates, also align with real-world fuzzy duplicates. Following the reviewer’s suggestion, we select reference canaries and their fuzzy duplicates from SlimPajama within a certain Levenshtein distance range and repeat the computation of the exact duplicate equivalent $\\rho$ for these real-world fuzzy duplicates. We now visualize the results together with our other results in Figure 4(a). We find that $\\rho$ decreases with Levenshtein distance similarly to the trend observed for our other experiments, confirming that our findings generalize to real-world fuzzy duplicates. Further details are provided in Section M12. Many of the risks of memorization only materialize if it is not only possible to determine that a string was used during training, but if (parts of) it can be extracted from the model. Carlini et al. define the concept of extractable memorization [1,2]: A string s is extractable from a model if one can construct a prompt so that the model generates s. See also Lukas et al. for the case of PII [3], and the work connecting membership and attribute inference [4,5]. In addition to MIA attacks, the authors could have also tested extraction attacks to see whether those also still work in the presence of only fuzzy duplicates. In the absence of that, it would still be good if they could (briefly) comment on the difference between membership inference and extraction. Following the reviewer’s suggestion, we additionally study how fuzzy duplicates contribute to the extraction risk of the reference canaries. In Section M8, we measure the extractability of reference canaries from GPT-Neo-1.3B and GPT-Neo-2.7B trained on datasets containing exact and fuzzy duplicates. We prompt the trained target model with a prefix p from the reference canary and, in line with Ippolito et al. [33], measure approximate extraction risk using BLEU and Levenshtein similarity between the ground truth and the generation (greedy 16 decoding). We consider prefix lengths of 50 and 75 tokens, and fuzzy duplicates obtained by making R replacements as used throughout the paper. First, we find that approximate extraction metrics increase with more exact repetitions, e.g. for a prefix of 75 tokens, Levenshtein distance between the ground truth and model’s output is 0.15 for n =1 and 0.32 for n =10. dup dup Next, consistent with our findings on MIA vulnerability, we find that fuzzy duplicates contribute substantially to the approximate extraction risk of the reference canaries. Across all setups, both BLEU and Levenshtein similarity remain substantially above the baseline for n =1 repetition, and approach the level observed for n =10 for lower values of R (Table dup dup M5). Additional results and discussion are provided in Section M8 and refer to these results in the Results section, under “LLMs have a mosaic memory”. The first of those two proposed (series of) experiments is more important to me than the second, which can rather be seen as a suggestion for future work. In case the first experiment would be significantly more challenging and a larger amount of work than I am thinking right now, I would suggest accepting the paper even without it. Lastly, I have a few questions for the authors: 1. For the token replacement via RoBERTa, how do you deal with the fact that this model has a different tokenizer than some of the models that you evaluate memorization on? E.g., if you want to replace one token, but the token returned by RoBERTa is actually tokenized into two tokens by the tokenizer of the model that you evaluate? This would be good to clarify in the paper. Many thanks for raising this question; it is indeed a subtle technical detail we had to address. In our implementation, we always sample exactly one token from the MLM’s vocabulary, which may correspond to multiple tokens under the target tokenizer – which means that the resulting canary can sometimes have slightly more than 100 tokens. We have clarified this in section M2 in Methods. 2. “the respective $\\rho$ also rise from 1.99 to 2.51 and 3.72, respectively” (p.8, top). Those seem to be the wrong numbers, as $\\rho$ should be between 0 and 1. Do you mean the numbers from Table 1, i.e., 0.11, 0.17, 0.30? We thank the reviewer for pointing that out correctly and we have now addressed this. 3. In M8 you write that you did not control the sampling parameters for GPT-4o. Was it not possible to set the temperature back when you used the API for these experiments, or does the statement only refer to the probability sum? Thanks for this question. While we were indeed able to specify the sampling parameters when querying GPT-4o, we chose to use the default ones for OpenAI’s API. We have clarified this in section M8 in Methods. [1] Carlini et al. Extracting Training Data from Large Language Models. 2023. USENIX 17 [2] Nasr et al. Scalable Extraction of Training Data from (Production) Language Models. 2023. arXiv [3] Lukas et al. Analyzing Leakage of Personally Identifiable Information in Language Models. 2023. SP [4] Yeom et al. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. 2018. CSF [5] Salem et al. SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learning. 2023. SP(Remarks on code availability) 18 The Mosaic Memory of Large Language Models (NCOMMS-25-29287): Response to reviewers (Second Revision, December 2025) We thank the reviewers for their valuable feedback and the time they dedicated to our manuscript. In this document, we respond to each reviewer’s comments, reproducing their comments verbatim and providing our responses in blue. Reviewer #1 (Remarks to the Author): We would like to thank the authors for their hard work in improving the manuscript. We believe that all of the comments and questions raised in our review have been addressed in this new version or in the author's response. Overall, we are grateful for the effort made to make the paper more accessible and for all the additional descriptions and specifications provided. We believe these improvements have substantially enhanced the paper. Nevertheless, we have a few final minor comments about this new version. We thank the reviewer for their positive assessment and are pleased that our revisions have addressed their concerns. We appreciate their constructive engagement throughout the review process. Firstly, we appreciate the detailed answers given in the Author's Response, but we believe that some have not been fully addressed in the final version of the manuscript itself. For example, the fact that \"$\\rho$ captures how fuzzy duplicates influence memorization regardless of how inherently “memorizable” a sequence is\", or that $\\nu_{eq}$ is \"the value of $\\nu$ for which $\\phi_{\\nu}$ best approximates $\\tilde{\\phi}$\" are very clear in the Author's response, but still not as clear in the newer draft (in particular, note that the latter is much clearer in the author's response than in the new version \"the value of $\\nu$ for which $\\tilde{\\phi}\\approx\\phi_{\\nu_eq}$\", which is somewhat ambiguous). Ideally, we would like the authors to incorporate the specifications from the Author's Response into the text. We thank the reviewer for this helpful suggestion. We agree that the clarifications provided in our previous Author's Response should be incorporated directly into the manuscript to ensure the paper is self-contained and clear. We have now revised the relevant passages in the main text. Specifically: 1. We explain the intuition behind the design of our metric ($\\rho$) in the beginning of the relevant section (page 3, “A metric to study the memorization of fuzzy duplicates”). 2. We expand paragraphs defining $\\nu_{eq}$, providing additional context behind our design (page 4, “A metric to study the memorization of fuzzy duplicates”). Similarly, regarding our comment on the \"surprisingness\" of the results, we appreciate the authors’ explanation and acknowledge that the issue is more contentious than we previously thought. We believe that the answer provided in the authors' response is important, and we would like it to be mentioned or discussed in the paper as well. This would slightly increase transparency with regards to the current knowledge of the community. 1 We thank the reviewer for raising this point. We agree that clarifying why the syntactic over semantic memorization result is surprising improves the transparency of the manuscript. Following this suggestion, we have expanded the discussion in the main text (both on page2, “Introduction” and page 6, “The mosaic memory is syntactic rather than semantic”) to explicitly place this result in the context of the increasingly strong semantic capabilities reported for modern language models (e.g., reasoning, problem solving, code generation). By contrasting these widely observed abilities, often interpreted as requiring semantic abstraction, with our finding that memorization across fuzzy duplicates is nevertheless driven primarily by token-level overlap, we hope to better motivate why this result is non-trivial and informative for the community. We also think that the additional references to the appendices will help readers, as we can now see that they address some of our initial questions. We thank the reviewer for this suggestion. We have now added more explicit references to the appendix, specifically calling out our experiments with varying canary sampling temperature. However, we noticed that the rewriting has introduced a few terms into the main body that might require explaining. For instance, the term \"temperature\" in the added text \"setting the temperature $T=1$ to mimic realistic text\" is not explained in the main body. We thank the reviewer for pointing this out. We have now added a brief explanation of temperature in the main text (page 3, “A metric to study the memorization of fuzzy duplicates”), clarifying that it controls the randomness of token sampling and that T=1 uses the model's natural probability distribution to produce coherent, yet varied text. We also realise that our comment on the canaries chosen by the same distribution was incorrect. We appreciate the authors helping us to better understand, and we agree with their response to our comment. Finally, we noticed some typos: * The $T$ in temperature (pg 3) should be in $\\mathcal{T}$ in concordance with the appendices and to avoid confusion with the tokenizer $T$. * Missing spacing after \"indicacting stronger memorization\" (pg 3). * Missing words in \"yields higher memorization in the than one exact repetition\" (pg 4). We thank the reviewer for listing these typos and have now addressed them. We would like to thank the authors once again for their responses and corrections. We would like to thank the reviewers for their thoughtful and constructive feedback throughout the review process. Their insightful comments and suggestions have helped us substantially improve the clarity and rigor of the manuscript. We are pleased that our revisions have addressed the concerns raised, and we are grateful for the reviewers' engagement with our work. Reviewer #2 (Remarks to the Author): I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career Researchers who co-review manuscripts. 2 Reviewer #4 (Remarks to the Author): I would like to thank the authors for their detailed response to my comments. I appreciate the effort taken to address the concerns raised regarding validity, significance, and methodology. Statistical Robustness: I appreciate the inclusion of the mean and standard deviation across independent runs for the main results. Clarification of Scope: The authors have provided a clear distinction that their work aims to study the mechanism of memorization rather than making normative judgments of the consequences of memorization. Though I still believe the pure empirical approach could only give limited insights to the mechanism of memorization, and interesting and significant problems, such as the trade-off between data deduplication and generalization, are left to future works. Comparison to Prior Work: I accept the distinction drawn between this study and the work of Allen-Zhu et al. SlimPajama Deduplication: The clarification regarding the difference between document-level (Jaccard similarity) and sequence-level (Levenshtein) deduplication effectively explains why fuzzy duplicates persist in the dataset. Overall, the revisions and additional explanations have strengthened the manuscript. I am mostly satisfied with the authors' response. We sincerely thank the reviewer for their thoughtful and constructive engagement throughout the review process. We are pleased that our revisions have addressed their concerns, and we are grateful for the valuable suggestions that have helped us improve the manuscript considerably. Reviewer #5 (Remarks to the Author): I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career Researchers who co-review manuscripts. Reviewer #6 (Remarks to the Author): Creation of the canaries: There exist examples in the literature where models prompted with just the beginning-of-sequence token have produced sequences from the training data, e.g., [1, 2]. I am, however, sufficiently convinced by the experiments with higher sampling temperatures. While canaries that are already duplicates could influence the concrete value of $\\rho$, this should not affect the trends visible in the experiments and the main conclusions. Overall, I am not too concerned about this potential issue. Experiment with real-world fuzzy duplicates: It’s great that the authors have conducted the proposed experiment. It adds a lot to the authors’ argument that fuzzy duplicates should be taken seriously. The results are less clear than in the controlled experiments, but still show a trend of $\\rho$ decreasing with increasing distance between the fuzzy duplicates. In future work, it would be interesting to analyze the real-world fuzzy duplicates more closely to, e.g., identify properties of the duplicates within each bucket that might explain the unexpected jumps in $\\rho$. 3 Experiment measuring extraction risk: Thank you for conducting this additional experiment. The findings are convincing and show the robustness of the impact of fuzzy duplicates on memorization and their risks beyond membership inference. The questions that I had for the authors have been fully addressed. I am very satisfied with the authors’ response and with the changes they made to the paper and recommend acceptance. [1] Lee et al. Deduplicating Training Data Makes Language Models Better. 2022. ACL. [2] Carlini et al. Extracting Training Data from Large Language Models. 2021. USENIX We are grateful to the reviewer for their careful reading of our work and their engagement with our manuscript. We are glad that we were able to address all of the reviewer's questions. In particular, we would like to express our appreciation for suggesting the experiments on extraction risk and real-world fuzzy duplicates – we believe these additions have considerably strengthened the paper. We thank the reviewer for their support and recommendation for acceptance. 4"}]

Decision letter