Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Authors: Ken Liu
Christopher A. Choquette-Choo
Matthew Jagielski
Peter Kairouz
Sanmi Koyejo
Percy Liang
Nicolas Papernot
Venue: ICML 2025 spotlightposter (2025)
Keywords: Training data membership
data completion
data reconstruction
membership inference
unlearning
privacy
training set inclusion
copyright
Primary area: deep_learning->large_language_models
TL;DR: Under $n$-gram definitions of train-set inclusion, LLMs can complete “unseen” texts—both after data deletion and adding “gibberish” data. Our results impact unlearning, membership inference & data transparency.

Abstract: An important question today is whether a given text was used to train a large language model (LLM). A completion test is often employed: check if the LLM completes a sufficiently complex text. This, however, requires a ground-truth definition of membership; most commonly, it is defined as a member based on the n-gram overlap between the target text and any text in the dataset. In this work, we demonstrate that this n-gram based membership definition can be effectively gamed. We study scenarios where sequences are non-members for a given n and we find that completion tests still succeed. We find many natural cases of this phenomenon by retraining LLMs from scratch after removing all training samples that were completed; these cases include exact duplicates, near-duplicates, and even short overlaps. They showcase that it is difficult to find a single viable choice of n for membership definitions. Using these insights, we design adversarial datasets that can cause a given target sequence to be completed without containing it, for any reasonable choice of n. Our findings highlight the inadequacy of n-gram membership, suggesting membership definitions fail to account for auxiliary information available to the training algorithm.

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Ken Ziyu Liu 12 Christopher A. Choquette-Choo" I
Matthew Jagielski" 1
Peter Kairouz?
Sanmi Koyejo
Percy Liang" *3
Nicolas Papernot 1
Abstract
was trained on?), copyright (e.g., has the model been trained on copyrighted text?), and more generally Al safety (e.g.. did the LLM successfully unlearn text that was identified as harmful post hoc training?). In each of these settings, is important that the evaluation of membership be robust.
An important question today is whether a given text was used to train large language model (LLM). A completion test is often employed: check if the LLM completes sufficiently com- plex text. This, however, requires a ground-trutt definition of membership; most commonly, it is defined as a member based on the n n--ram overlap between the target text and any text in the dataset. In this work, we demonstrate that this n-gram based membership definition can be effectively gamed. We study scenarios where sequences are non-members for given and we find that com- pletion tests still succeed. We find many natural cases of this phenomenon by retraining LLMs from scratch after removing all training samples that were completed: these cases include exact du- plicates, near-duplicates, and even short overlaps. They showcase that it is difficult to find single viable choice of n for membership definitions. Us- ing these insights, we design adversarial datasets that can cause given target sequence to be com- pleted without containing it, for any reasonable choice of n. Our findings highlight the inadequacy of n-gram membership, suggesting membership definitions fail to account for auxiliary informa- tion available to the training algorithm.
With direct access to the training dataset, determining se- quence membership is straightforward given fixed criterion for defining when two text sequences are "the same." A com- mon approach is to compare sequences by their n-grams, as this is both efficient and reasonable. Two sequences match verbatim if all their n-grams are equal for all n, and apprex- imately if most do for some (Lee et al., 2021). However, this definition must align with downstream concerns in pri- vacy, copyright, and safety-domain focused on what an L.M can reproduce as an intuitive notion of membership.
Consider a setting where a long text sequence is generated verbatim by an LLM. The sequence has high entropy due to its length, which makes it very unlikely to have been generated by chance. Thus, one may intuitively conclude that this sequence was in the training data. This raises key question: can an LLM generate target sequence even if it was never included as an n-gram in its training data? Our findings confirm that the answer is affirmative: n-gram membership establishes a threshold dependent on n, and this threshold can be gamed. In other words, our findings show that that formally defining a robust and accurate notion of membership is challenging.
1. Introduction
We assess whether an LLM generates a target sequence by prompting it with a prefix and checking if it completes the corresponding suffix, a process we refer to as a completion test. Language models are known to complete some of their training data (Nasr al., 2023). In our work, we first find that even after removing a set extracted sequences from the training dataset and retraining the LLM from scratch, the retrained model can still verbatim complete 40% of them under our experimental conditions (Section 4). Upon investigation, we find that these removed yet still completed sequences are either de facto members of the training set (but for a different definition of membership) or lacking sufficient complexity: many examples have near duplicates, sequences with m n-grams that are not removed,
Training data membership asks whether a data point was used to train a given model. For large language models (LLMs), it is useful to answer questions related to privacy (c.g.. is the LLM leaking information contained in the text it
'Equal contribution 'Google "Work completed while on internship at Google DeepMind. Now at Stanford Uni- versity. *Stanford University. Correspondence Ken Liu <kzliu @ es. weunffor..duu-.. Christopher A. Choquette- Choo <cchoquette @ google.com>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Removing Members Does Not Prevent LLM Verbatim Completion
Adding Non-Members Can Force LLM Verbatim Completion
Pre-train base LLM (Mamn)
2. Identify verbatim completions (D (wn)
1. Take target (unseen) text sequence
7. Led uhiin call adboaa tage a sovered
inalleeese hove Eoean
hurtage warr
"Errors are one of the greatest challenges in quantum computing..."
uill use disann a.a deais desph rettot
Muss
M Muss
2 deuce
2. Construct fine-tune examples with minimal n-gram overlap
(888 has the beckvva
atzegon on my
Pre-trined LiM
book
Method 1: Chunking
Prompt the prefixes... ...do completions match?
Enors digoersdurraap lagilativ how
toppossll...
3. Filter Dewns from pre-training data and re-train LLMs from scratch
di top sbriuse one of the greatest challenges ocommaiona
Method 2:
Token dropouts
land phose sdt
A and price rih
aprosclles are one nearifi aloorry LLVM inberlicheft155.c Raforindis
sanliiboee-denn---ee dover derphris
Mintar (oues)
Mikter in ( Ro taired N-gram filter (stronger) remove weepe with any gram overlap against any seq in Dave
Erors are pulfetocci greatest challengee inSCO computing inscucss Dickior...
(ylithuve look you
trannee L
Method 3:
Casing flips
I-
Exact filter (weaker): remooe seaps
ErROi5 Are OnE of tHe GrEMES: CHALLANgsS IN 4WhnTuM COMPuclNg
Ducu. tvactly as sppetred. from pre trainingn
4RRDvS als THE GrAATEST CHaLLeNGES IN quANtun ConFUUING
4. Some of these now now-removed sequences can still be completed verbatim
Compositions, e.g.: Casing flips + token dropouts
Prompt the prefixes...
do the completions still match?
JJER Emos** a Migeititorse Onl OF FiehpputSmran GREATeMatts in quArtUM larring...
e yeastees pess dMALLeggs5 CUSTOM
Part
Milter (esact)
IV V Part
Juill
cottad..
3. After fine-tuning, target sequence can be (verbatim) completed
Strong filters result fewer such "lingering sequences" (40% Down)
challenge quantum computing.
Emrrs of greatest
Greedy decode fine-tuned LLM
Part lI Part I Part Ii
M filter
Part IV Part V Part Vi...
CHalleNaes IN quinttm cOmpUTing
dears.
(n 5)
drear whippera o..
X
(Fi using casing flips, method 3]
Figure 1: Main setup and findings: a text sequence can be (verbatim) completed by language model without being a n-gram "member" of its training set. Left ($4): We pretrain a model and remove extracted training dataset of length & from its training data with either exact (k = n) -gram filters or stronger approximate (k > n.)a membership filters. We find some sequences (Rs 40% with exact filters or As 1% with approximate filters) remain verbatim completed despite not being explicitly trained on. Right ($5): We show an LLM can be fine-tuned to verbatim complete a target unseen sequence, e.g., today's blog post, by using adversarially constructed datasets with no n-gram overlap.
explained by the model's generalization capabilities (e.g.. patterns or counting).
3.
We believe that the underlying cause of this limitation is not in the choice of the distance used compare sequences- here n-gram overlap butt rather in the fact that the membership definition fails to consider auxil- iary information that the training algorithm gets access to, e.g., through pre-processing or other design choices made throughout the ML pipeline. Here, the strategies we propose to game the n-gram definition exploit this by introducing auxiliary information through the very construction of the training dataset: c.g., cannot con- struct (n 1)-grams of a sequence without knowing the entire sequence in the first place.
This result, however, leads to second finding- there exists strategies for systematically gaming the n- n-gram mem- bership definition. That is, there are strategies for construct- ing a dataset that does not contain n-grams of a sequence a, yet, when a language model trains on D it is able to com- plete verbatim. In Section 5, we give multiple examples of such strategies, including one that has the model train multiple m- m-grams of x, where < and, in some cases, m < n. Our experiments show that we can systematically force a model to complete six sequences of interest a despite these sequences not being a member of its training set per the n-gram membership definition.
2. Background & Related Work
Figure/ shows our setup. Our main takeaways are:
Definitions of data membership. Many language model tasks require definition of data membership. In most cases, the definition falls into versions of n-gram, or substring, overlap (Anil et al., 2023; Gemini Team et al., 2023; 2024; Gemma Team et al., 2024a;b; Touvron et al., 2023; Dubey et al., 2024; Zhang et al., 2024a: Duan et al., 2024; Carlini et al., 2021; Singh et al., 2024). n-gram based definitions capture near-duplicates by matching smaller text segments; this is flexible, simple, and intuitive. When studying data contamination, much of the prior work uses n n--grm based
1. We find that there is high overlap between training data membership and our L.M completion test being positive. Text not in this overlap are explained by the lack of complexity or limitations in n--rrmm based definitions. 2. n-gram membership is limited in capturing the intuition of what constitutes training dataset "member." Indeed, our work shows that a model can complete sequences that are not n-gram members of its training dataset.
2

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
definitions (Sainz et al., 2023; Jiang et al., 2024; Dekoninck et al., 2024; Singh al., 2024). For example, GPT-4 con- siders 50-character substring overlap (Achiam et al., 2023) and 3 considers 8-gram token overlap (Dubey et al., 2024). For training data deduplication (Lee et al., 2021; Kandpal et al., 2022; Mou, 2023), duplicates are identified based on training data membership. Recent methods use suffix arrays for exact substring matches (Lee et al.. 2021) and MinHash or locality sensitive hashing for approximate matches (Broder, 1997; Mou, 2023); both build on n-gram overlap. The prevalent use of n-gram based definitions re- flects a practical balance between accuracy and simplicity. A key focus of our work is to highlight the limitations of these n-gram based definitions.
2024). These works are often studied from the perspec- tive of studying memorization, where the entity performing the model test has access to the training dataset. In this line of literature, there exist both verbatim definitions of memorization (Carlini et al., 2022b; Huang et al., 2024) and approximate definitions (Ippolito et al., 2022). When studied from a black-box perspective-witiiee training dataset-they typically match completions against known auxilliary databases surrogate confirmation of membership (Carlini et al., 2021; Nasr et al., 2023). In- tuitively, if model completes a long sequence x when prompted with its prefix, it likely saw I during training be- cause x has high entropy due to its length and vocabulary size (Carlini et al., 2019; 2022b). Our work focuses only on these completion tests as black-box membership test.
access to the
Tests for data membership. Unlike membership defini- tions, which define the ground-truth membership tests aim to detect if a data sample was in dataset. There are many model-level membership tests in the literature that predict membership of text to a training dataset with only access to a trained model, and not the training dataset. Our work focuses on lodel-level membership tests because they are more relevant to the downstream uses of membership in LLMs, e.g., in privacy, copyright, and safety (see $1).
3. Preliminaries
We now formalize the key definitions that underlie our exper- iments. We focus on defining what it means for a sequence to be a "member" of the training set of a language model, and what constitutes "completing" a sequence as means of testing its membership. Precise definitions of these notions anchor our study of the mismatch between them.
Membership inference attacks (MIA) (Shokri et al., 2017) are widely studied, especially in computer vision (Yeom et al., 2018; Salem et al., 2018; Sablayrolles et al., 2019; Choquette-Choo et al., 2021: Carlini et al., 2022a; Jagielski et al., 2024) and more recently example-level membership inference for LLMs (Zarifzadeh al., 2023; Shi et al., 2023; Mattern al., 2023; Li et al., 2023). Despite these attempts, progress is hindered by flawed evaluations (Meeus et al., 2024; Zhang et al., 2024b): Duan et al. (2024) argue that membership can be inherently blurry for natural language, Das et al. (2024) report that existing MIA testbeds suffer from distribution shifts, and Kong et al. (2023) refute MIAs using gradient-space attack. Our work situates in this body of work by studying systematic failure modes of oper- ationalizing membership through definitions and tests, and the consequences when these definitions and tests mismatch.
Modern language models operate on token sequences, which are integer encodings of text strings via byte-pair encoding (BPE) tokenizer (Sennrich, 2015). We use x to denote a token sequence (rather than its text form) with length |z]. and n-grams(z) = {0es4n] | -n to denote the set of n- grams derived from x.
Definition of Data Membership. We anchor a simple and flexible membership definition for our experiments that encapsulates many variants used in the literature:
Definition 3.1 (n-gram data membership). sequence x is ) x: shares at least one n-gram with any x: E DD. That is, x is member if there exists g E n-grams(()) s.t. g E U, n-grams(<e()))
member of dataset D {xx())
This definition is stringent (e.g., approximate membership typically requires many, not just one, n-gram to match). This ensures we overestimate members and thus underesti- mate non-members. This definition is also inclusive of the those in the literature, as varying n captures spectrum of them. For example, setting n = | is the verbatim mem- bership used in Carlini et al. (2022b). Smaller n captures many approximate membership definitions, such as Min- Hash (Broder, 1997: Lee et al., 2021), edit distance based membership (Ippolito et al., 2022), and many other n-gram variants cited in Section 2. In the remainder of the paper, we call sequence x an "n-gram member" if x satisfies Def. 3.1, and otherwise "n-gram non-member".
Dataser-level MIAs enhance membership signals by leverag- ing multiple correlated samples as inputs (Maini et al., 2021; Kandpal et al., 2023; Maini et al., 2024). These are closely related contamination tests (Golchin & Surdeanu, 2023; Oren et al., 2023). Our work focuses on sequence-leve data membership tests based on data completion, because these focus on scenarios where the LLM generates the text, which presents novel concerns for privacy, copyright, and safety.
Data completion. There is a long body of work studying generation of training data, in diffusion models (Somepalli al., 2023; Carlini et al., 2023) and in LLMs (Carlini et al., 2019; Tirumala et al., 2022: Kudugunta et al., 2024; Bi- derman et al., 2024; Freeman et al., 2024; Huang et al.,
Definition of Data Completion. Informally, we define a completion as: when a token sequence is known a priori
3

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
and a language model generates its suffix when prompted with its prefix. Formally, if x = [pl] [s]. then model generates the expected suffix of based on the provided prefix p (prompt). For simplicity, we focus on |p| = |s| = |x1/2 in our experiments. Prior work has studied how the choice in prefix and suffix lengths impact memorization (Carlini et al., 2022b; Huang et al., 2024). To capture highly similar but not verbatim completions of the sequence z. we introduce variants of completion that allow for semantically insignifi- cant deviations from the original s. We define the following notions of completion.
4. Removing Members Does Not Always Prevent LLM Verbatim Completion
LLMs are known to complete training sequences (Carlini et al., 2021). In this section, we ask:
Main Question: Will an LLM still complete text sequence even after we remove all training sequences that have n-gram overlap with it?
We find that this retrained model still has lingering com- pletions: the completion test still succeeds despite having removed all completions identified by n-gram matching from the training data. This raises the question: why is the LLM still completing these sequence?? As we will see, this is because the completions are either a) still contained in the dataset but via different membership definition (e.g., for some m-gram membership, with m < n),, or b) they lack sufficient entropy and can be easily predicted (generalized).
Definition 3.2 (Exact completion). Given tokens [p]|s] and model M, we say x is exactly completed if (p) using greedy decoding.
This is closely related to verbatim memorization and verba- tim training data extraction (see Section 2). There are also semantically equivalent sequences that humans would be likely to not distinguish from the original sequence. We thus consider two approximate notions of completion, relevant to our experiments in Section 5.
4.1. Experimental Setup
Our high-level experimental framework is as follows. We defer additional details to Appendix A.
Definition 3.3 (r-similar completion). Given x: = [plis] and M, we say x is a r-similar completion if M(y) is within a normalized Levenshtein edit distance of 1 - r using greedy decoding, i.e., lex(M(p),, s)/ max(( [s]) -
1. Pre-train base model: We first pre-train standard
LLM Mbsse from scratch on a training dataset D
2. Identify verbatim completions: We then collect a set of sequences Drmem of length i that Miuse can complete verbatim (as in Def. 3.2), by checking the first k tokens of every training document in Drase- This is simple and effective procedure since L.MS are known to memorize training data (e.g.. Carlini et al. (2022b)); other choices to obtain Dmem are also possible.
Definition 3.4 (Case-insensitive completion). Given x = [p]ls] and M, we say x is a case-insensitive completion if lower(MM(p) = lower(s) with greedy decoding, where lower(-) applies character wise lower casing.
Data Completion vs. Data Extraction. Data extraction, as considered in recent work (Carlini et al., 2021; Nasr et al., 2023), concerns recovering training data from the model. This thus involves both (1) data completion (e.g.. as in Def. 3.2), and (2) verifying that the completion is training member; e.g., by inspecting the training data.
3. n-gram filtering: We then filter each sequence x E Dmem away from Doase Our filtering procedure is sim- ple and reflects n-gram membership (Def. 3.1): to filter sequence xr we perform sliding window of length |x|| over Drase (as if all tokens are concatenated into single array); if the window shares any n-gram with n-grams(x), the window is deleted from the pre-training data. When n = (æ]. then we filter x exactly as it appears in the dataset; when n < |z]. we filter more aggressively as the window removed on partial matches against xt The filtered dataset is denoted ((a)
In a sense, extraction specifically measures memorization,' while completion is more generic-indedd our work studies non-member completions; Fig. 2 illustrates the distinction.
Members
Completions
Extractions
We say that a sequence is extractable if it can be completed and can be proved a member of the training set.
Figure 2:
fiter*
4. Re-train a counterfactual model: Pre-train another LLM M M from scratch on the filtered data [
'Our reference to "memorization" is intended in very specific context: whether model can be induced to generate near-copies of some training examples when prompted with appropriate in- structions, and often with prior knowledge of the model's training data. Specifically, we do not mean to imply that model "contains" its training data in the sense that any arbitrary instance that data can be retrieved without use of specialized software or algorithms. Rather, if model can be induced to generate very close copies of certain training examples by supplying appropriate instructions to guide the model's statistical generation process then that model is said to have "memorized" those examples.
We repeat this procedure for different model sizes, differ- ent values of n-gram (for filtering), and different sequence lengths k. Unless otherwise stated, we use i = 50, meaning that for a sequence æ [p||s], we have [p| |s| = 25. We provide results on A = 100 in Appendix A.7.
Models and training. We pre-train a series of models from scratch using the GPT-2 architecture (Radford et al., 2019), spanning sizes of [350M, 774M, 1.6B, 2.8B) parameters,
4

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Table I:
The number of identified verbatim memorized Dmem at different model sizes (step #2 of 3 4.1).
tion of lingering sequences [Piigerr// ///p can be high as 40% when we apply the weakest n-gram filter and only remove verbatim sequence matches (n 50).
sequences
Model size
304M
774M
1.6B
2.8B
Dmeml
76,648
116,270
151,598
175,813
Finding #2 (Nature of Lingering Sequences): We found no lingering sequences that correspond to cre- alive generalizationnssueeeee beyond reconstructions from neighboring texts and continuations of recogniz- able patterns. Fig. 4 visualizes few lingering sequences in D Ding and see Appendix A.2 for more. To understand their origin, we then perform a search of neighboring texts (Levenshtein distance < 20) for a few randomly' selected lingering sequences over the large pre-training data Drase: we defer results to Appendix A.3. For all lingering se- quences we queried, we were able to find near-duplicates, yet all such copies evaded n-gram overlap detection one way another. This result sheds light on the remarkable ability for LLMs to generalize from neighboring text. More interestingly, it also informs an interesting symmetry on how we may adversarially construct training sequences that: (1) have no n-gram overlap with a target sequence z, and (2) yet serve as "anchor points" that the LLM can interpolate to verbatim complete x. We explore this symmetry in the coming section (55).
50
Linering fraction (%)
40
n=5
30
n=10
n=20
20
n= (exact)
10
350M
774M
1.6B
2.8B
Model size
Figure 3: LLMS can verbatim complete texts with zero n-gram overlap to training data. A fraction of sequences filtered away from pre-training data (Dmem= Table 1) can still be completed by the re-trained LLM verbatim. The fractions decrease under stronger filtering (smaller n-gram filter) and remain relatively stable across model scales.
with 1.6B being the size of the original GPT-2 XL. and 2.8B being a scaled-up model. We use LLM.c (Karpathy, 2024) for an efficient pre-training pipeline. We primarily report results on the 1.6B model unless otherwise stated.
Finding #3 (Persistence of Lingering Sequences): Stronger filters reduce, but do not eliminate, these lin- gering sequences, and instead shift their distribution to more generalizable patterns. As we filter Dmem from Druse more aggressively with smaller n-gram filters, smaller fractions of Dmmm can be completed by the re-trained model Minu fitter verbatim (Fig. 3). However, even at very conserva- tive filter of 5 g-raam is in D ). Dio linger still accounts for z 1% of Dmem (Table 1). As the fraction of lingering sequences decreases, their contents also shift from verbatim memorization of se- mantically useful text (e.g., famous quotes) to generalizable patterns (c.g.. counting in Roman numerals). We provide examples in Fig. 4 and Appendix A.2.
Data, For all models, we use FineWeb-Edu (Penedo et al., 2024) as a state-of-the-art pre-training dataset.? We use the same base dataset Phsse of 33.6B randomly sampled tokens. For the 1.6B model, 33.6B tokens is upproximately Chin- chilla optimal (> 20 tokens per parameter, Hoffmann et al. (2022)). For consistency, we train the base models Musse of different sizes with the same starting dataset Drase: as the size of D mem (step #2) hinges on the size of Mbase (Carlini et al., 2022b), we obtain different filtered datasets D(n) for each model size (by inferencing on Mnnse).
5 (a sequence is removed from Donse if any
4.2. Results
With the artifacts Mhuse: Druse: D Dime= filter now make observations pertaining to our main question.
D(v)) M(n) we
To quantify this shift, we use three proxy metrics (Fig. 5), though we note that none perfectly captures the (fuzzy) boundary between memorization and generalization. First, we measure the verbatim completion rate of D Dinger using the off-the-shelf GPT-2-XI. (Radford et al., 2019); since Mini is a similar model by construction, lingering sequence is likely a generalizable pattern if both models (trained on distinct data) agree on its completion. Second, we simi- larly consider the completion rate of a counterfactual model Mer on pre-training shards disjoint from Drase. Third, we prompt Gemini 1.5 Pro with few-shot examples to determine if a lingering sequence pattern continuation (prompt template in Appendix A.) All proxy metrics confirm
Finding #1 (Existence of Lingering Sequences): LLMs can verbatim complete a fraction of the sequences deleted from training data, and consistently so across scale. On a macroscopic level, we first observe that sim- ply deleting a set of sequences from pre-training data does not always prevent them from being generated by an LLM (Fig. 3). This observation is consistent across model scales, where each size has a different amount of memorization (Table 1). We call these "lingering sequences" and denote them as D iinger' Under our experimental conditions, the frac-
Thiis work may contain information from FineWeb-Edu dataset, which is made available under the ODC Attribution Li- cense.
We only perform this experiment on randomly selected linger- ing sequences due to the cost of the search.
5

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
n=5 (strong flitaringe the entire sequence has no S-grama in traininn dltaa
Main Question: Given a chosen (unseen) text se- quence I, can we add training sequences Dn that have no n-gram overlap with x, and yet an LLM fine-tuned on Dr. can complete x verbatim?
Prompt: -ael Proiogte 5- 10 get g0 p0 11- and ge
Completion: .2 oftnn 20 50 and per 51.00 oth 51- 100 and St.
Prompt: 3 Sign Ternio vinnteinrieleert- - Sarrrroo-----1111--
Completion: E0p6yeap - Casderr 201 Enrlwwarber-
2016 201ey 2017
(exact hiteringk the entre sequence, it supears exactly, not training data
Complation: Hhudridge
Prompt: the df beansr-- barnni-bunruugiie (le 2lonleg ol barogh dereeyy
fandan Husse chddrdded Di Sonay Randon Huuse Inc. 2018
This question is intriguing for its insights into LLM general- ization, as well as its relevance to adversarial manipulation of training data, where an adversary may wish to intention- ally avoid n-gram membership for, e.g., harder-to-detect data poisons and content misuse (more in Section 5.6).
Prompt::
exdent, But liee are endowed by ther Gredto ih cetan unclenable lights that among these Liberty
Ne these
crested equal
Completion:
Figure 4: Examples of lingering sequences (more in A.2).
Geraralization Proxy Matric (%)
100
80
To answer the question, consider class of noisy trans- formations f on Ir such that: (I) - = f(x) retains some information about x;; and (2) E has no æ-grrm overlap with 2. We hypothesize that an LLM trained on different in- stances of ã (over different randomness) should learn to recover the original x, much like denoising antoencoder learning recover clean data from noisy inputs.
60
Comparable Madel (GPT-2-XL)
Counterfactual Re-train
40
LLM-Judge (Gemini 1.5 Pro)
20
0
10
20
30
40
Lingering Fraction (%)
Figure 5: Strong filters result in less lingering comple- tions and shift them toward generalizable patterns. Re- sults 1.6B size. We use three proxy metrics where higher indicates more pattern continuations: 1) % verbatim comple- tion by off-the-shelf GPT-2-XL; 2) % verbatim completion by Mar. a counterfactual model trained on disjoint pretrain- ing shards; 3) % judged as patterns by Gemini 1.5 Pro.
Methods. We show that it is possible to adversarially con- struct examples Da= = { from a chosen unseen example xr (e.g.. today's news) such that x and Dn share no common n-grams (and many membership lests, including manual in- spection, would fail). Yet with only a few gradient steps of fine-tuning an LLM on Dtt, the model can (verbatim) com- plete this nnnnmeemerr" We study three such methods of constructing such fine-tuning set Da:
our manual inspection that stronger filters force out gener- alization behaviors from the model, albeit these are simple sequences to generalize to (recall finding #2).
1. Stitching chunks:
We split into overlapping segments padded with random tokens.
2. Token dropouts: We replace tokens in at different (ran- dom) positions with random tokens; positions have at most n - 1 gap to avoid n-gram overlaps.
4.3. Interpretations and Outlook
While we found no evidence of creative generalization in models up to 2.8B parameters, lingering sequences are in- triguing because they seem to challenge our understanding of membership in LLMs-if a language model can verbatim complete sequences known a priori without ever training on any of its n-grams, what does this imply for the defi- nition of membership and its reliance on n-gram overlap? To better understand these limitations we identified and if they are exclusively explained by the limitation of r--ram overlaps- we next experiment with adversarially gaming n-gram membership. We build on our findings thus far to inform the adversarial construction of dataset of n-gram mon-members that is able to force LLM completion.
3. Casing flips: We flip the casing of every English letter in x with probability p.
These methods resemble real-world transformations of texts such as taking excerpts of an article and transcriptions that misspell words, miss punctuation, and drop casing. They also have varying degrees of efficacy as we will discuss in Section 5.5. Note that our goal is not to find the best possible (stealthiest) transformation, but to explore feasibility and ease of such adversarial manipulation.
Models and training. We work with two model fami- lies: Gemma-2 (Gemma Team et al., 2024a;b) and Qwen- 2.5 (Yang et al., 2024; Team, 2024), spanning model size from 0.5B to 9B. We fine-tuned these models to predict the next token with a batch size of 32 and a constant learning rate of 10
5. Adding Non-Members Can Force LLM Verbatim Completion
Data. We primarily experiment three target texts. These texts are all roughly 1,000 characters long (> 250 tokens un- der Gemma-2 tokenizer) and have a recent temporal cutoff such that they could not have been included in the training set of Gemma-2 and are extremely unlikely to appear in the training set of Qwen-2.5 (thus helps ablate the effect of
While lingering sequences (Section 4) are largely benign and rare, observing how they came to be (e.g., through visualizations in Appendix A.3) informs how one may ad- versarially force a model to complete n-gram non-members. We explore this direction with the following:
6

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
potential memorization):
is fresh random token and where here d = 4. We also con- sider a randomized dropout, where every token is dropped with probability 1/d. By construction, the deterministic version guarantees that x' is not d-gram member of the training set, while the randomized version does so with (exponentially) high probability, Notably, due to BPE token- zation (Sennrich, 2015), the original text becomes visually obfuscated for humans even when most tokens are retained (d > 2). See Algorithm 2 for detailed algorithm and Ap- pendix B.3 for visualization.
1. Lyles (NYT article): an excerpt of a recent New York
Times article about Noah Lyles and the Olympics:
Karpathy (tweet): a tweet text in an image posted by Andrej Karpathy about L.LM tokenization; and
3. Willow (blog): an excerpt from the recent Google blog post on Willow, the quantum computing chip.
The text choices are otherwise arbitrary and alternatives are possible; see the full texts, source, and results on alternative texts in Appendix B.1. For every target text sequence, we construct N 2, 000 examples as Da by applying the transformation f with different randomness, though in most settings we need less than 1,000 examples. We visualize some of these examples in Appendix B.3.
This construction is closely related to the goldfish loss pro- posed by Hans et al. (2024) to mitigate verbatim memo- rization, where the loss of every n-th token (on average, if randomized) is omitted during training. However, the proce- dure here is entirely data-centric: it does not interfere with the training objective and makes the learning task harder as subsequent tokens would still attend to the random tokens.
5.1. Stitching chunks
A natural way to avoid generating any E that shares n-grams of x is to only expose chunks of at most (n - 1)-ggamss to the model. We construct Dn by breaking x into contiguous, overlapping token segments, with the remaining positions padded with random tokens from the vocabulary. This pro- cedure is parameterized by the chunk size e and the overlap 1. Chunk size e controls the difficulty (noise level) of the task. For example, the task is trivial if c is the sequence length (since Dn are just copies of x). A small c means most tokens of any ã E 'Dn are random. A large overlap I should intuitively help the LLM learn to stitch x together, though empirically we observe minimal impact (Appendix B.5.
5.3. Case flipping
Another approach is to perform text-space transformations that preserve semantics yet drastically alter the token-space representations. One such method is to randomly flip the casing of English letters, creating varied tokenization of the otherwise equivalent string. To illustrate, x decodes to "This is a string', DDn may include token se- quences of strings like " TTIS Is stRinG', Due to the mechanisms of BPE tokenization, it is extremely easy to obtain F with completely distinct tokens than under mod- ern LLM tokenizers (see Appendix B.2 for visualization), The case flipping procedure is parameterized by the flip probability p. p = 0.5 creates the highest variance, and p closer to always flips the case (most letters being initially lower case, natural text becomes mostly upper case).
To illustrate, if the target x 2,, 4,5, 6). then we may have Dn 2,,,2,3, ",,, 1, 3, 4, 5, -1, b, ', ;, 5, )... with chunk size = 3 and overlap I = 1, (- denotes fresh random token). Intuitively, the task for the LLM is to "stitch" the token chunks back together into x. A similar technique is explored in the concurrent work of Panaitescu- Liess et al. (2024) for data poisoning; here, we explore chunk sizes, overlaps, and model families to present more comprehensive tradeoffs. See Algorithm for our detailed procedure and Appendix B.3 to visualize examples in Da.
5.4. Compositions
The transformations presented earlier are not mutually ex- clusive, and in principle they can be composed arbitrarily. We explore one such composition of token dropouts & case flipping, where non-dropped tokens ($5.2) have casing ran- domly flipped in the text space ($5.3). A key benefit of compositions is that they combinatorially give rise to many new transformations with potentially more granular control of task difficulty and detectability (c.g.. whether manually
5.2. Token dropouts
Another way avoid ã E Da sharing n-grams with x is to mask out tokens in I at least every n positions, so that it cannot share n-gram overlap with x. That is, Dn contains different versions of x where at least every (n-11 1n- th token is masked out (replaced with a random token) so that there is at most n token overlap. This pro- cedure is parameterized by a drop interval d, represent- ing the length of the interval between dropped tokens. It must be that d < n to ensure no n-gram overlap. To il- lustrate, if x 11,,,3, 5,5, 6). then we may have Da = {11, 2, 3, ,, 5, 6), (1,,,,4,, 5, J., 2, 3, 4, 6).... where '2
inspecting Dn reveals x; see Appendix B.3 for example visualizations). We leave comprehensive evaluation of composed transformations to future work.
5.5. Results
Finding #1: It possible for an LLM to complete an un- seen string with n-gram membership after minimal finetuning.
7

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Chunking (overtap=11) 10) Lyles
Token dropouts Karpathy
Casing flips Willow
Dropouts + Flips (p=0.9) Willow
max text edit simi ailiarrty (%)
max text edit similarity (%)
100
max toxt edit simi aimilarly (%)
100
max text edit simil arity (%)
100
90
80
80
$
80
Gommax2 9d
Owen 2.5 0.58
70
Owern22 5 1.58
Owen 2.5
70
8
0
83
(a)
Success
(b)
(c)
83
(d)
25
50
75
100
3 drop interval (1/P(token drop))
2
0.4 0.6
0.8
2
3
4
chunk size
casing flip prob
drop interval (1/P(token drop))
Figure 6: Completion success across methods and target texts: (a) chunking on Lyles, over chunk size e (x-axis);; (b)
token dropouts on Karpathy, over drop interval d; (c) casing flips on Willow, over flip prob p (p = 0.5 is noisiest); and (d) combining dropouts+ + flips on Willow. We observe that: (1) it is possible to complete a chosen string with zero n-gram membership, and (2) this ability improves with model size. See Appendix B.5 for comprehensive results.
Dropouts Flips (gap=4,flip=))) (lip=0.9)- Willow Q.5n-2.5 7B
Finding #2: Completion success scales with model size. Another message from Fig. 6 is that as we increase in the model size, the completion success generally improves un- der the same configurations. This provides evidence that frontier models should be more capable at synthesizing n- gram members into the target texts.
100
Edit similarity (%)
80
Text space
60
Token space
Success threshold
5.6. Interpretations and Outlook
10
20
30
40
Gradient steps
We briefly describe the potential implications of our adver- sarially constructed fine-tuning datasets:
Figure 7: Completion success may only require few gradient steps. See more configurations in Appendix B.5.
Data poisoning: n-gram non-members of poison text x be added to the training set and still induce the generation of The concurrent work of Panaitescu- Liess et al. (2024) explores adding copyrighted materials as poisoned data with similar chunking technique 1)).1). Data contamination: a dishonest model developer may game model evaluations through deliberate data contami- nation while evading 7- n-gram based detection.
Fig. shows the results of various transformations across various text targets (see Appendix B.5 for comprehensive results). We first see that there are many configurations where fine-tuning on Dn allow the successful completion of target x (edit similarity > 90%).
Some transformations are more effective than others. We found that chunking (Section 5.1) was ineffective: with a small chunk size (c = 25), the models mostly fail to com- plete the target, and only some models succeed at 100 (Fig. 6 (a)). Token dropouts (Section 5.2), on the other hand, is extremely effective-eve even the smallest model (Qwen-2.5 0.5B) easily completes the target verbatim at drop interval of 2 (50% drop probability at every token; Fig. 6 (b)). Our results thus present a counter-case to goldfish loss (Hans al., 2024), as models can still complete targets verbatim when there are multiple versions of the target with different token dropout positions (e.g., due to near-duplicates, related to the findings of Section 4). Casing flips (Section 5.3) are also generally effective (Fig. 6 (c)). Composing token dropouts and casing flips (Section 5.4) increases task dif- ficulty (fewer successes with small models) but otherwise similarly enables verbatim completion.
Reporting train-test overlap metrics: More broadly, a model developer may self-report train-test overlap statis- tics (e.g., as part of contamination analysis seen in Dubey et al. (2024): Gemini Team et (2023); Brown (2020)). Our results highlight that it is desirable that developers report additional metrics beyond n-gram overlap.
6. Concluding Remarks
Lingering sequences (5 4) and adversarially constructed fine- tuning datasets ($ 5) demonstrate the remarkable ability of LLMs to generalize from neighboring text. They are thus a valuable tool for evaluating LLM capabilities as models and pre- pre-training datasets scale up. We conclude our work with discussion of the implications of our findings:
In essence, these experiments demonstrate that n-gram based membership definitions can be vulnerable to adver- sarial manipulation: the fine-tuning set Dr clearly contains information about the text x. but when given x and a choice of n, it is easy for an adversary to bypass detection and yet have the model generate verbatim.
Membership definitions and tests should incorporate new similarity measures. We showed n-gram based mem- bership emits false negatives that may not capture human in- tuition nor the pragmatic concerns of the copyright, privacy, and AI safety community. On the flip side, tests like mem-
8

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
bership inference should consider broader notions of mem- bership beyond individual sequences: unit of data could be collection of sequences grouped by similarity (Kandpal et al., 2023; Maini et al., 2024; Cooper et al., 2024).
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint 087704 2023.
Machine unlearning alone is insufficient to address data permissibilityy concerns in output suppression. It is widely accepted that golden baseline of machine unlearn- ing is retrain a model from scratch without the target forget data (Bourtoule et al., 2021; Liu, 2024; Liu et al., 2024; Cooper et al., 2024). Yet, our experiments perform precisely this counterfactual and reveal that some excluded sequences can still be verbatim generated (Fig. 1, 84). We thus caution that unlearning alone may not always prevent a model from generating sequence of interest (e.g., harm- ful sequence). This also known as output suppression and is common goal of unlearning (Cooper et al., 2024).
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10400, 2023.
Biderman, S., Prashanth, U.. Sutawika, L., Schoelkopf, H., Anthony, Q., Purohit, S., and Raff, E. Emergent and predictable memorization in large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
Bourtoule, I.., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., and Papernot, N. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159, IEEE, 2021.
Exploring the connection between our work and forg- ing may help more precisely characterize threat models for when completion can serve as evidence of member- ship. Readers familiar with the forging (Thudi et al., 2022) literature will have noticed a connection with our work. Forging step of gradient descent computed a given minibatch is done by (adversarially) constructing a different minibatch that will result in the same gradient being com- puted. Perhaps surprisingly at first, gradients can be forged using noveoverlg datasets sampled from "natural" dis- tributions. Rather than forging gradients, our methods can be viewed as attempting to forge model outputs. While our work begins to show that there may exist threat mod- els where completion is insufficient evidence for n-gram membership, our experiments did not succeed in obtaining forge in model outputs using "natural" data (recall 4). We believe this is valuable direction for future work.
Broder, A. Z. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB pp. 20.221-2 21-29, IEEE, 1997.
Brown, T. B. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 2020.
Carlini, N., Liu, C., Erlingsson, Ú. Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization neural networks. In 28th USENIX se- curity symposium (USENIX security 19), pp. 267-284, 2019.
Carlini, N., Tramer, F., Wallace, E., Jagiclski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, 2021.
Impact Statement
Our work studies training data membership in the context of language models. The impact of our work is described in Sections 4.3, 5.6, and 6. As we described earlier, the limitations we identified in n-gram based definitions of membership have implications for copyright, privacy, and Al safety. Broadly speaking, our work advocates for addi- tional membership definitions help better capture human intuition and make membership more pragmatic.
Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramer, F. Membership inference attacks from first prin- ciples. 2022 IEEE Symposium on Security and Privucy (SP), pp. 1897-1914. IEEE, 2022a.
Carlini, N., Ippolito, D.. Jagielski. M., Lee, K., Tramer, F., and Zhang, C. Quantifying memorization across neu- ral language models. arXiv preprint arXiv:2202.076466, 2022b.
Acknowledgements
KZL would like thank Xinran Zhao, Nikil Selvam, Steven Cao, Jing Huang, Yangjun Ruan, Aryaman Arora, Harshit Joshi, Rylan Schaeffer. Chenglei Si, Yanzhe Zhang, as well as members of the p-lambda lab and STAIR lab at Stanford University for helpful discussions and feedback. We thank Milad Nasr and Nicholas Carlini for helpful discussions, comments, and critiques throughout the work.
Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., Balle, B., Ippolito, D., and Wallace, E. Extracting training data from diffussin models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5270, 2023.
9

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Choquette-Choo C. A., Tramer, F., Carlini, N., and Paper- not, N. Label-only membership inference attacks. In International conference machine learning, pp. 1964- 1974. PMLR, 2021.
Gemma Team, Riviere, M., Pathak, S., Sessa, G., Hardin, C., Bhupatiraju, S., Hussenot, I.., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00) 18, 2024b.
Cooper, A. F., Choquette-Choo, C. A., Bogen, M., Jagiel- ski, M., Filippova, K., Liu, K. Z., Chouldechova, A., Hayes, J., Huang, Y., Mireshghallah, N., al. Machine unlearning doesn't do what you think: Lessons for gen- erative ai policy, research, and practice. arXiv preprint arXiv:2412.06966 2024.
Golchin. S. and Surdeanu, M. Data contamination quiz: A tool to detect and estimate contamination in large lan- guage models. arXiv preprint arXiv:2311.06233 2023.
Hans, A., Wen, Y., Jain, N., Kirchenbauer, J., Kazemi, H., Singhania, P., Singh, S., Somepalli, G., Geiping, J., Bhatele, A., et al. Be like goldfish, don't memorize! mit- igating memorization in generative Ilms, arXiv preprint arXiv: 10209, 2024.
Das, D., Zhang. J., and Tramèr, F. Blind baselines beat mem- bership inference attacks for foundation models. arXiv preprint arXiv:2406. 1et.:24466..221... 16201, 2024.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprinl arXiv:2203.1555 2022.
Dekoninck, J., Müller, M. N., Baader, M.. Fischer, M., and Vechev, M. Evading data contamination detec- tion for language models is (too) easy. arXiv preprint arXiv:2402.02833. 2024.
Huang, J., Yang, D., and Potts, C. Demystifying verbatim memorization in large language models. arXiv preprint 1xXiv:247.118811 2024.
Duan, M., Suri, A., Mireshghallah, N., Min. S., Shi, W., Zettlemoyer, I.., Tsvetkov, Y., Choi, Y., Evans, D., and Hajishirzi, H. Do membership inference attacks work large language models? arXiv preprint arXiv:2402.0784/ 2024.
Ippolito, D., Tramer, F., Nasr, M., Zhang, C., Jagielski, M., Lee, K., ChoouhhteeeChhheee -hho,, C. A., and Carlini, N. Prevent- ing verbatim memorization in language models gives false sense of privacy. arXiv preprint arXiv:22 12. 10.2200.77... 17546. 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A. et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Jagielski, M.. Nasr, M., Lee, K., Choquette-Choee C. A., Carlini, N., and Tramer, F. Students parrot their teachers: Membership inference on model distillation. Advances in Neural Information Processing Systems, 36, 2024.
Freeman, J., Rippe. C., Debenedetti, E., and An- driushchenko, M. Exploring memorization and copyright violation in frontier llms: A study of the new york times openai 2023 lawsuit. arXiv preprint arXiv:2412.06370, 2024.
Jiang, M., Liu, K. Z., Zhong, M., Schaeffer, R., Ouyang, S., Han, J., and Koyejo, S. Investigating data contami- nation for pre- pre-training language models. arXiv preprint wwXiv:2401.0005. 2024.
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J. Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
Kandpal, N., Wallace, F.,, and Raffel, C. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pp. 10697-10707. PMLR, 2022.
Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Kandpal, N., Pillutla, K., Oprea, A., Kairouz, P., Choquette- Choo, C. A., and Xu, Z. User inference attacks on large language models. arXiv preprint arXiv:2310.09266, 2023.
Karpathy, A. Lim.e: Lim training in simple, raw c/cuda. https: http//gitttuu www.hhub.oooo/o.o. /t////aappaattllnmma 2024. GitHub repository.
Gemma Team, Mesnard, T., Hardin, C., Dadashi, R., Bhu- patiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini re- search and technology. arXiv preprint arXiv:2403.08295 2024a.
Kong, Z., Chowdhury, A. R., and Chaudhuri, K. Can membership inferencing be refuted? arXiv preprint WwXi::2303033644 2023.
10

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Kudugunta, S., Caswell, I., Zhang, B., Gareia, X., Xin, D., Kusupati, A., Stella, R., Bapna, A., and Firat, O. Madlad- 400: A multilingual and document-level large audited dataset. Advances in Neural Information Processing Sys- tems, 36, 2024.
Oren, Y., Meister, N., Chatterji, N., Ladhak, E, and Hashimoto, T. B. Proving test set contamination in black box language models. arXiv preprint arXiv:2310.17623 2023.
Panaitescu-LLess M.-A., Pathmanathan, P., Kaya, Y., Che, Z. An, B., Zhu, S., Agrawal, A., and Huang. F. Poisoned- parrot: Subtle data poisoning attacks to elicit copyright- infringing content from large language models. In Neurips Safe Generative AI Workshop 2024, 2024, URL https:/ / wwwpneeeiew.neeeeeeeee.ee for um d= PANIwe.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Buid Burohh C., and Carlini, Deduplicating train- ing data makes language models better. arXiv preprint 1XXiv:2107.06999 2021.
Penedo, G., Kydlíček, H., Lozhkov, A., Mitchell, M. Raffel, C., Von Werra, L., Wolf, T.. et al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406. 77577, 2024.
Li, M., Wang. J., Wang. J., and Neel, S. Mope: Model perturbation-base privacy attacks on language models. arXiv preprint arXiv:2310.14330 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
Liu, K. Z. Machine unlearning in 2024, May 2024. URL https: stt.aaaissaaafd.deeae..... wtt.fooddeedu//kiuu//bbelb. kzliu/blog/ unlearning.
Sablayrolles, A., Douze, M., Schmid, C., Ollivier, Y., and Jégou, H. White-box vs black-box: Bayes optimal strate- gies for membership inference. In International Con- ference on Machine Learning. pp. 5558-5567. PMLR, 2019.
Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P., Yao, Y., Liu, C. Y., Xu, X., Li, H., et al. Rethinking machine unlearning for large language models. arXiv preprint ar 24Xhv:2402.08887 2024.
Sainz, O., Campos, J. A., García-Ferrero, I., Etxaniz, J., de Lacalle, O. L., and Agirre, E. Nlp evaluation in trouble: On the need to measure Ilm data contamination for each benchmark. arXiv preprint arXiv:2310. 18001888, 2023.
Maini, P.. Yaghini, M., and Papernot, N. Dataset inference: Ownership resolution in machine learning. arXiv preprint vrXiv:2104.10706 2021.
Maini, P., Jia, H., Papernot, N., and Dziedzic, A. Lim dataset inference: Did you train on my dataset? arXiv preprint arXiv:2406.06444 2024.
Salem, A., Zhang, Y., Humbert, M., Berrang, P., Fritz, M., and Backes, M. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv:1806.01244 2018.
Mattern, J., Mireshghallah, F., Jin, Z., Schölkopf, B., Sachan, M., and Berg-Kirkpatrick, T. Membership infer- ence attacks against language models via neighbourhood comparison. arXiv preprint 2805 2023.
Sennrich, R. Neural machine translation of rare words with subword units. arXiv preprint
2015.
Meeus, M., Jain, S., Rei, M., and de Montjoye, Y.-A. In- herent challenges of post-hoc membership inference for large language models. arXiv preprint arXiv:2406. 17975, 2024.
Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting pretrain- ing data from large language models. arXiv preprint arXiv:2310.1677 2023.
Mou, C. Lcrge-scale near-deduplication behind bigcode, 2023. URL https: /hwggiuuggggggg ng face. ce... co/blog/ dedup. Published on Hugging Face Blog, May 16, 2023.
Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Mem- bership inference attacks against machine learning mod- els. 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., Choquette-Choo. A., Wallace, E., Tramèr, F., and Lee, K. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311. 17035, 2023.
Singh, A. K., Kocyigit, M. Y., Poulton, A., Esiobu, D., Lomeli, M., Szilvasy, G., and Hupkes, D. Evaluation data contamination in 1lms: how do we measure it and (when) does it matter? arXiv preprint arXiv:24/1.0023 2024.

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Somepalli, G., Singla, V., Goldblum, M., Geiping, J., and Goldstein, T. Diffusion digital forgery? investigat- ing data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6048-6058, 2023.
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024. URL https: qwenlm. www..ggttihhooo.o hog/qwen2.5/
Thudi, A., Jia, H., Shumailov, I., and Papernot, N. On the necessity of auditable algorithmic definitions for ma- chine unlearning. In 31t USENIX Security Symposium (USENIX Security 22), pp. 4007-4022, 2022. 2022.
Tirumala, K., Markosyan, A., Zettlemoyer, L., and Agha- janyan, A. Memorization without overfitting: Analyzing the training dynamics of large language models. Ad- vances in Neural Information Processing Systems, 35: 38274-38290, 2022.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1066 10671, 2024.
Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, Privacy risk in machine learning: Analyzing the connection to overfitting, In 2018 IEEE s1 computer security founda- tions symposium (CSF), pp. 268 282.. IEEE, 2018.
Zarifzadeh, S., Liu, P., and Shokri, R. Low-cost high-power membership inference attacks, 2023.
Zhang, A. K., Klyman, K., Mai, Y., Levine, Y., Zhang, Y., Bommasani, R., and Liang, Language model devel- opers should report train-test overlap. arXiv preprint arXiv:2410.0885 00.0885, 2024a.
Zhang. J., Das, D., Kamath, G., and Tramèr, F. Membership inference attacks cannot prove that model was trained on your data. arXiv preprint arXiv:2409, 19798, 2024b.
12

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
A. Removing Members Does Not Always Prevent LLM Verbatim Completion (54)
A.1. Data filtering
The n-gram filter discussed in Section 4.1 can be defined as follows:
Definition A.1 (n-gram data filtering). Let F be a set of n-grams to filter against. Let F), be the condition that returns if any n n-gram of the sequence x in F (0 otherwise). An n-gram filter against F on a set of sequences 'D is thus constructing D = (ar
Jn xr, F) = 0}.
Intuitively, a stronger nt n-gram filter (smaller n) means that we are removing a sequence on increasingly smaller partial matches against the filter set. In practice, since we are filtering many sequences (Dmem) once, it more efficient to build a shared n-gram filter hash from all sequences in Dmmm and then apply sliding window filtering procedure.
Table 2 shows the amount of tokens kept after applying n-gram filtering at different filtering strengths (for sequence length 50, the default setting used in experiments unless otherwise stated). Table 3 similarly shows the amount of tokens for sequence length 100.
Table 2: Fraction of tokens kept after applying n-gram filter to sequence length k 50.
n-gram filtering strength
10
71
20
π 50 (Exact Filtering)
Approx % of tokens kept 0.6905
0.9838
0.9938
0.9995
Table 3: Fraction of tokens kept after applying n-gram filter to sequence length k: 100.
7-gram filtering strength n
10
Y1 =
20
n
50
n = 100 (Exaet Filtering)
Approx % of tokens kept 0.9820
0.9930
0.9970
0.9995
A.2. Visualizing Lingering Sequences D linger
In this and the following section, we provide visualizations to the key results described in Section 4 to help develop intuition on findings.
Recall from 54 that Ding linger refers to the set of lingering sequences that are still verbatim completable by the re-trained from scratch LLM after applying n-gram filter (Def. A. on the identified memorized sequences Dmem-
Table 4, Table 5, Table 6, and Table show 15 randomly sampled lingering sequence at filtering strengths n = {50, 20, 10, 5). respectively. Observe that, as noted Section 4.2, with stronger filtering strength (smaller n), the content gradually shift from semantically useful content to pattern continuations.
Table 4: Randomly sampled lingering sequences at filtering strength n = 50 (exact) filter. Sequence length = 50.
Idx
Lingering sequences filtering strength 50 (exact) filter.
Prompt:
"Common Name: bleeding beart Type: Herbaceuus peremnial Native Range: Eastern United States Zone: 3 to 9
Completion: 'Height; 1.00 to 1.50 foet Spread: 1.00 to 1.50 foet Bloom Time:"
Prompt:
1477 by topic- Arts and science- Birth and death categories- Births -
Completion:
Deatths EEstablishmenns and disestablistments categorics- - Establishments - Disestablismments-
Prompt:
"Charcot Joint (Neuropathic Artrropahh) Medicine Central TM is quick consult mohile and'
Completion:
web resource that includes diagnosis, treatment, medications, and follow-up information
700 discases and disorders, providing fast
answers'
Prompt:
"Miemert- burth Sumame History The family history of the Mienert-burt last name is' muintained the AncientFaces community. Join community adding to knowldge the Mienert."
Completion:
Prompt:
"Instructional Supports and Resources Dyslexia is specitic learning disabiity that is neurelogical in origin. is characterized
Completion:
by difficulties accurate andirr fluent word recognition by poor spelling and decoding abilities.
difliculties typically
from a deficit'
3
Prompt:
Publisher description for Writers at work. The short composition / Ann O Strnch. Bibliographic record and links to
Completion: related information available from Library Congress catalog Information from electronic data provided the publisher. May be incompplete or contain other'
Prompt:
Create healtheare diagrams like this examplc called Anencophaly in minutes with SmartDraw. SmartDraw includes 1000s of protessional
healtbeare"
Completion: and anatomy chart templates that you can modify and make your own. Text in this Example: Ancooephaly is'
13

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Idx
Lingering sequences at filtering strength n 50 (exact) filter.
Prompt:
"Presentaiion theme: "Spiraled Assignments Presenier: Angela Pritchet November 14, 2006."
Completion:
Presentation transcripi: Spiraled Assignments Presenter; Angela Prischeit November 14, 2006
Prompt:
"An excerpt from Wow.eOo
archives copyright @ 2000 2013 Where did the Irish McSweersyy family" from? What the Irish McSweeney family crest coal of arms? When did the McSweeney family lizst"
Completion:
Prompt:
'Which of the following cubes can be made from these nets? Is it possible to remove ten unit cubes from a 3 by'
Completion:
33 by 3 cube made from unit cubes so that the surface area of the remaining solid is the same as the surface area'
10
Prompt:
"Course Hero. "The Libation Beurers Study Guide." Course Ieroo 23 June 2017. Web. 14 Nov. 2018
Completion:
1. wtttps///wwwww.co. https///www.coureeennt. wwt.eroo..o/////ohoeoeeooo wwthhe--haaiion Libation- Libation-Bearer/j Course Hero"
11
Prompl:
Prrssetaaion theme: "Cause Effect Comprehension Skill Fourth Grade Unit Week Created by Kristi Waltke'
Completion:
1.- Presentation transcript: Cause and Effect Comprehension Skill Fourth Grade Unit 2 Week Created by Kristi Walt'
12
Prompt:
"An exeerpt from www. www. Ofuse com archives copyright @ 2010-2015 Where did the English Ragsdale family"
come from? What the English Ragsdale family crest coat of arms? When did the Ragsdalc family first'
13
Prompt:
"Gihhous Scorpio Moon phase on 29 March 2051 Wednesday is Waning Gibbous, 16"
Completion:
days old Moon is in Scorpoo Share this page: twitter facebook linkedin Provious main lunar phase is the Full Moon
14
Prompt:
'What does AIDS mean in Laboratory? This page is about meanings
acronym ont/aabrrrvtttooooooooooo
Completion:
AIDS in the Medical field in general and in the Laboratory terminology in purticular. Find a translation for AIDS in other languages."
15
Prompt:
NNtthalio Raphačlle June 23, 2021 Worksheets If you home school your children, you'
Completion:
will quickly realize how important printable homeschool worksheets can be. If you trying develop a curriculum for your"
16
Prompt:
Course Horo. "The Pearl Study Guide." Course Hero. 14 Doc. 2017. Web. 24 Nov. 2020.
Completion:
"https:/www coursehero convlii TeciiThh--eeeaae Pearl/; Course Hero. (2017. December'
17
Prompl:
"Intemet of Things Internet of Things The Internet of Things (loT) a system of interrelated computing devices"
Completion:
'. mechamical and digital machines, objects, animals or people that are provded with unique identifiers and the ahility to transfer data over a'
Prompl:
"Latest Newland plottos These photos were uplonded by members of Newland community on AncientFaaces. Newland
Completion: "urname History The family history of the Newland last name is maintained by the AncientFaces community. Join the community"
19
Prompt:
DDfinition of Seckles I. seckle [n] See also: seekle Click the following'
Completion:
link to bring up a new window with automated collection of images related the term: Sookles Images Lexic'
Table 5: Randomly sampled lingering sequences at filtering strength n = 20 filter. Sequence length k 50.
Idx
Lingering sequences at filtering strength 20 fifter.
Prompt:
"Presentation on theme: "MAKING BOOKS WITH CHII DREN Picture It! Publish It! Read It!"
Completion:
"- Presentation transcript: MAKING BOOKS WITH CHILDREN Picture It Publish It! Read It!' "This Constitution, and the I aww of the United States which shall be made in Pursaance thereof, and all Treaties made' or which shall be made, under the Authority of the United States, shall the supreme Law of the Land; and'
Prompt:
Completion:
Prompt:
'How To Recognize Crystal Child A sclection of articles related to how to fecognize a crystal child. Original articles from our library related to the How To Recognize A Crystal Child. See Table of Contents for further available material (downloudable resources"
Completion:
Prompt:
Wampsville, New York Wampsville, New York- + Total 1.0 sq mi (2.6 km2)
Completion:
Land 1.0 mi (2.6 km2)
Prompt:
'Report on Stromboli (Irly) - 12 March- 18 March 2003 Smithsonian/ / US Goological Survey Wookly Vol'
Completion:
'canic Activity Report, 12 March. 18 March 2003 Managing Editor: Gari Mayherry Please cite this report'
Prompt:
Prrssntation theme: "The Great (gym) Divide Curricula Design #3 M. Fischer. "
Completion:
Presentation transcript: The Great (gym) Divide Curricula by Design #3 M. Fischer The Great'
Prompl:
Course Hero, "Lord of the Fles Study Guide" Course Hero. 15 Sep. 2016, Web. May 20" '23. htth://www.cuureeer..coom wwt.////oooooooooooeooooooooooee of the l'lies/..
Completion:
Prompt:
"Munada Gap, Pennsylvania facts for kids Quick facts for kids Manada Gap. Pennsylvania Time zone UTC' '-5 (Eastern (EST))
Completion:
Summer (DST)- UTC-4 (EDT)-
Prompt:
'Scale Zaology Casmoid Scales A selection of articles related to scale zoalogy cosmoid scales. Original'
Completion:
articles from our library related
Scale Zoology Cosmoid Scales. See Table of Contents further available material C'
Prompl:
"Atonnic Nucleus History A selection of articles related to atomic nucleus history. Original articles from library related to'
Completion:
the Atomic Nucleus History. See Table of Contents for further available material (downloadable resources) on Atomic Nucleus'
10
Prompt:
"Mangoverde:: :. World Bird Guide: :: Phensants and Partridges :: Common Quail C Common Quail Cot
Completion:
"urnix coturni Described by: Linnacus (1758) Altemate commen namels'
II
Prompt:
"Set Builder Notation Variations A selection of articles related to set builder notation variations. Original articles from our library related
Completion:
the Set Builder Notation Variations. See Table Contents for further available material (downloudable resources) on Sct Builder'
12
Prompt:
1648 hy topic Arts and science- Birth and death categories-
Births
Completion:
Deaahss
Establishments and disesaalistmenns categories
Estahlishments Disestablishments
13
Prompt:
Taamil Script The Tamil Letters A selection articles related to tamil script tamil letters, Original articles from
Completion:
our library related to the Tamil Scripi The Tamil Letters. See Table of Contents for further available material (downloudable resources) on"
14
Prompt:
"Mauull of PorugalFrom Wikapedia, the free pecycoopdiaJumpp to: navigation, search This article does not cite any references or'
Completion:
sources. Please help improve this article by adding cilations reliable sources. Unsourced material may be challenged and removed.
15
Prompt:
"Image of 12 Image of 12 Image 3 12 Image 4 of 12 Image 5 of 12"
Completion:
'Image 6 of 12 Image 7 of 12 Image
12 Image 12 Image 10 12
16
Prompl:
Nutritional Guidelines
serving) Servings: 3 pint jars (96 servings) Anwunt per serving"
Completion:
& Daily Value" Total Far 0g 0% Saturated Fat 0g"
17
Prompl:
Presentation theme: "Corpus Linguistics and Stylistis PALA Summer School, Maribor, 2014"
Completion:
Presentation transcript: Corpus Linguistics and Stylistics PALA Summer School, Mariber,'
18
Prompt:
Presentation on theme: "Lunar Research Station Design Submitted by West Valley Elementary GATE Team October 31, 2006" Presentation transcript: Lunar Research Station Design Submitted by West Valley Elementary GATE Team October 31,' "Some daily events in the changing day for Fehruary 19 27. Friday, February 19 Saturday, Fehruary 20 Sunday."
Completion:
19
Prompt:
Completion:
February 21 Monday, February 22 Tuesday, February 23 Wednesday, February 24 Thursday, February Friday"'
14

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Table 6: Randomly sampled lingering sequences at filtering strength n2 = 10 filter. Sequence length A = 50.
Idx
Lingering sequences at filtering strength n. fitter.
Prompt:
Presentation theme: "HEELPIN YOUR CHILD WITH NUMERACY: ADDITION AND SUBTRACTION
Completion:
Presentation transcript: HEI PING YOUR CHILD WITH NUMERACY: ADDITION AND SURTRACTION
Prompl:
Wednesday 2:00 PM
PM
lesson
Lecture Hall
Completion:
Thursday 2:00 PM 3:40 PM
lesson
I ecture Hall 1.2
Prompt:
'How to define the cosine ratio and identify the cosine of an angle in a right triangle. How to define the
Completion:
sine an angle a right triangle. How define the tangent ratio and' 'QI. series given with one term missing. Select the correct alternative from the given ones that will complete the series" Q2. A scries is given with one term missing. Solect the correct altemaaive from the given ones that will complete"
sine ratio and identify
3
Prompt:
Completion:
Prompt:
"History of Fulse Teeth Length: 497 words (1 souble-spaced pages)
Completion:
Prompt:
Presentation on theme: YYggurt!!! Find the dairy cow on each page!!! By Daisy Mason and Brigetic Robents "" Presentation transeripi: Yoghun!!!! Find the dairy
Completion:
each page!!! By Duisy Mason and Brigetie"
Prompt:
Protecting People with Disabilities in the Ebbs and Flows of the CVVD-19 Pandemic Protecting People' with Disabilities in the Ebbs
Completion:
of CUV
Pandemic COVID- pand"
Prompt:
Presentation theme:: 'Acch Poverty Assessment The impact of the Cooflier, the T sunami and Reconstruction on Poventy' in Acch." Presentation transcript: Aoch Poverty Assesement The impact the Conffict, the Tsumami'
Completion:
Prompl:
Presentation theme: "THE MIX AERATOR Innovation Pond Lagoon Aeration & Mixing" "M Presentation transcript: THE MIX AFRATOR Innovation In Pond & I Lagoon Aeration & Mixing" "Sone dail events the c sky for February& 8 16. Friday, February 8 Saturday, February 9 Sunday," February 10 Monday,
Completion:
Prompt:
Completion:
11 Tuesday, Febnuary 12 Wednnedaa,, Fehruary 13 Thursday, Feehnary 14 Friday,' "Essays on mercutio Romeo and mercutio essyyc over 180,000 mimeo and mere'
10
Prompt:
Completion:
"utio easays, romeo and mercutio term papers, romco and mercutio research paper. book'
II
Prompt:
"Presentation theme: "Fabric Construction Fashion Design, Textiles & Merchandising Mrs. Moscinski,"_
Completion:
Presentation transcripe: Fabric Construction Fashion Design, Textiles& & Merchandising Mrs. Moscinski Fabric
Prompt:
Presentation theme: "Chupter Building Compussionate School Community Partnerships Woek Chapter 4 Building Comp' "assionate
Completion:
Partnerships That Work." " Presentation transcript: Chapter 4- Building Compassionate School-"
13
Prompt:
Tracing Names: Letter AA - B C D
Completion:
P DDcctionary of Financial, Economic, and Business Terms
N
R
14
Prompt:
Completion:
-H-1- J-K-K K K- L M
P Q Q-P R
15
Prompl:
Canons of the Seven Ecumenical Councill.. The Ecumenical Council. Second Ecumenical
Completion:
Council. Third Ecumenical Council. Fourth Ezumenical Council. Fifth Ecumenical Council.
16
Prompt:
"Welsh Levels of Care E-Learning Program Glossary Special
A B D
Completion:
K L. M N O P Q
17
Prompt:
Presentation on theme: "app Water Intrusion Eilectts on Microbial Life Anthony DeRenzo Grade 10 Patsburgh Central Cathoke
Completion:
High Schwol." Presentation transcript: Tap Water Intrusion Ellects Microbial Amthony DeRenzo Grude
18
Prompt:
"Presentation theme: "Dr. Anand Srinivasan for MBBS 2013 10/10/2013""
Completion:
Presentation transcript: Dr. Anand Srinivasan for MBBS 2013 on 10/10/2013
Prompl:
Presentation theme: "Ashok Sinha Olo Director General (Audit) Central, Chandigar'
Completion:
hh. Presentation transcript: Ashok Sinha Olo the Director General (Audit) Central, Chand'
Table 7: Randomly sampled lingering sequences at filtering strength n 5 filter. Sequence length k = 50.
Idx
Lingering sequences at filtering strength n filter:
Prompl:
"Water, sunitation and hygiene: the foundation building resilience climate vulnerable communities Water, sanitation and hygiene:"
Completion: the foundation for building resilience climate- vulnerablee communitics Water, sanitation and bygiene: the foundation for building
resilience in'
Prompt:
What is durcoolcukodysttoplly discase?- - Addccookukoyysttoppy Causes-
Completion: "Adreooleukndyysttthh Symptoms 44 Adreeoleukodyystrgee Diagnosis - Adrenoleukody'
Prompt:
HITS April 2017 a QI (a) (b) (c) *
Completion:
Q1 QI (e) Q1 (1) Q1 (g) Q'
Prompt:
"Nalive to North America STATE DISTRIBUTION (USDA): AL, AR, DC, DE, FL."
Completion:
GA. IA, II, KS. L.,LA, MA, MD. ME, MN.
Prompl:
'What the 7 notes of major scalle? The scale degrees Ist: Tonie."
Completion:
2nd: Supertonic. 3rd: Mediant. 4th: Suhdominant""
Prompl:
TTn Times Table And Random Test Lyrics I 10x 10x3=
Completion:
30 x0xx 4= ==40 40 10 = 50 10 x 6= 60 x 70'
Prompt:
"Accumey Of Data 914 words (2.6 double--uaceee pages)
Completion:
7
Prompt:
'Print Texting RULES! Reading Comprehension with Fourth Grade Work Print Texting RLLES! Reading Comp'
Completion:
'rehension with Fifth Grade Work Print Texting RULES! Reading Comprehension with Sixth Grade Work Print Text'
Prompt:
Glossary of Logal Terms A- B D----------GH
Completion:
'-I-J- K- L- M N O P- Q-R
Prompt:
"Chef x=0, 1-jump: he will move from -i x 2-
Completion:
'jump: be will move from x i x 2 3-jump: will move from x 3"
10
Prompt:
AAcing Minister of the Environment, Denis Kellman (centre,
Completion:
11
Prompt:
'MI Science Standards Special I. M N
A
Completion:
O
R
S
12
Prompt:
"Colective Nouns For
Pages
Completion:
F. G. II.
M.
13
Prompt:
7 Wicked Winter Health Mydhs By: I Laura Roberson Winter Health Myth #1 1- Winter Health Myth H'
Completion:
2- Winter Health Myth 3- Winter Health Myth #4- Winter Health Myth # 5- Winter"
15

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Idx
Lingering sequences at filtering strength n filter.
Prompt:
"HISTOYY CRIME AND PUNISHMENT HISTORY CRIME PUNISHMENT HISTORY CRIME AND
Completion:
PUNISTMENT HISTORY CRIME AND PUNISIMMETT HISTORY CRIME AND PUNISIMMETT
15
Prompt:
"Deline Gyromitra infula. Gyromitra intula syaeaymns, Gyroniira
Completion:
"fula promunciation, Gyromitra infula translation. English dictionary definiton Gyromitra infula."
16
Prompt:
"Some daily events in the changing sky for December 19 27. Friday, December 19 Saturday, December 20 Sunday,"
Completion:
December Monday, December 22 Tuesday, December 23 Wednesday, December 24 Thursday, December 25 Friday,'
17
Prompt:
LLnngh: 1 1122 words (3.2 double spaced pages)
Completion:
Prompl:
Names: Page
Completion:
N O P 'A Complete Illustrated History of Robors the Movies (chronological by film title) Intro - Part 7 Part Part Part'
19
Prompt:
Completion:
Part Part Part
20
Prompt:
"I George Washington Term of Office (1 (1789-1791) 2. John Adams (1797-18"
Completion:
3. Thomas Jefferson (1801-1809) 4. James Madison (1809-1817'
A.3. Visualizing neighbors of lingering sequences D (n)
linger in the pre-training set
A key aspect worth studying for lingering sequences is what contributed to their existence. We randomly sample lingering sequences for filtering strengths n {50, 20, 10), and perform a very costly search of Levenshtein edit-distance neighbor search: perform a sliding window over the pre-training tokens, and check the edit distance of each window to the lingering sequence. By construction, lingering sequences have already been removed from the training set; the hope is thus to identify and visualize neighboring sequences that may have led these lingering sequences.
Fig. 8 visualizes these randomly sampled lingering sequences, and two randomly sampled neighbors (with edit distance 20 tokens) for each of them. We also visualize the histogram of these neighbors at different distances. We note that:
These visualizations suggest that lingering sequences very likely the result of either near-duplicate training data and/or the generalization capabilities of LLMs.
By observing the neighboring sequences, gain insights into how we may adversarially game the n-gram mem- bership definition. For example, observing the second sequence in Fig. provide intuition for the chunking ($5.1) method we presented in $5 to adversarially construct fine-tuning sequences that avoid n-gram overlap, and observing the second last can provide intuition for the token dropout (55.2) method.
16

---

shall Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Lingering Seq [n= 50 filter]: The Sixth Amend ment to the U.S. Constitution reads, "In all criminal prosecutions, the accused shall enjoy right to speedy public trial, an impartial jury crime shall have been committed, which
Deuures 395
Neighbor AnThe 6th Amendment
Trial Jury Clause reads like this:'n 'In all criminal prosecutions, accused enjoy right a... by an impartial jury
been committed
Neighbor nor shall property be taken for public, without just compensation An- Amendment In criminal prosecutions the accused shall enjoy the right to speedy and public trial, by an impartial jury of the state and district wherein the crime shall have been committed
160
1300
Lingering Seq (n 50 fitter]: you want for essey for unique writing Looking At The Chinese Lifestyle And Norma, just click Order button. We will write custom essay on Looking At The Chinese Lifestyle And Norms specifically you!
Firal Winzee 521
Neighbor If you
of cybersecurity and cybercrime, just click Order button. will a custom essey on The role cybersecurity and cybercrime specifically for you! Neighbor #2: you want to pay for essay for unique writing Gender Roles and Lady Macbeth, just click Order button. We will write a custom essay on Gender Roles and Lady Macheth specifically for you!
pay essay for unique writing
Cot Cilaae
Lingering Seq (n 20 fitter]: Definition of amplnihhe word amp uses 3 letters: m, pinamp is pleyable in:lnHok words amplnThese are words formed by appending one letter to amp. an already existing word on board.
Neighbor uses letters: u'niocum is playable inck words of locuminThese words formed
sppending
already exleting
board,
Neighbor The by appending one letter dona, an already existing
n, playable in:in words donainThese are
Lingering Seq (n 20 filter]: selection of articles related kulin brahmins, Originaal artictes from our library related Kulin Brahmins. Table Contents for further ilable material (downlosdable resources) Kulin Brahmins.In: The
Del Stparoee for Chure
Neighbor #1: selection of articles related to the creation of adam, Indriinan articles from our library related to the The Creation Of Adam. See Table Contents further available material (downloddable resources) The Creation Of Adam.An- The Aconic Perspective
Nelghbor <<lendoftetti>> selection articles related sufi AnOriinnninnnnnnnn articles further
related to
(downloadable resources)
15
16.
15.8
Cross-
Gdi: Darar ane for Cheie LOK
Lingering Seq (n= 10 filter): 2005 AMC 10A Problemstn- Problem 1 2 Problem 21n- Problem 3in- 4 Problem 4'n- Problem Problem Problam Problem 8in- Problem
Neighbor #1: In- Article 11n- 11 Article 2in-3 Article 3 Article 4 Article Article 6in Article 7
Article 8kn- 9 Article 9kn 10 Article 10
Neighbor #2:
Have fun. Don't die.nn- Problem 13 Problem 14 Problem 13 Problem 14y- Problem
13 Problem 14 Problem 14 Problem 13
120
la Chack
Lingering Seq =n= filter]: Little Big Store in Reymondinloocation of Raymond, Mississippitni Total|]3.0 0 sq mi (7.7 km2]j/nn* Land||3300 (7.7 km2))nn+ Water]]
Neighbor #1: C6: Julie Brownley (D)inj 32t sq mi (83.53 km2))/nj Land]|21.82 sq mi (56.50 km2))\nn*
Water||
Neighbor #2: in Andrescoggin County and the state of Mainelnj+ 62. mi (162.44 km2))\nn+ Land] 59.26
sq mi 153.48 km]) Water|]
25.5
Figure 8: Visualizing the neighbors (Levenshtein edit distance < 20 tokens) of few randomly selected lingering sequence from D. (10) D (22)) and D (50)
linger
linger"
17

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
A.4. Model architectures
Our 350M, 774M, and -o--paraeterr model architectures follow directly from the original GPT-2 paper (Radford et al., 2019) and LLM.c implementation (Karpathy, 2024). We create custom architecture for 2.8B parameters adjusting the number of layers, channels, and attention headas in the model, again following Karpathy (2024). The model configurations can be found in Table 8.
Table 8: Configurations of models (GPT-2 architecture) at different sizes for our pre-training experiments.
Parameters
304M
774M
1.6B
2.8B
Layers
24
36
48
60
Channels
1024
1280
1600
1920
Attention heads
16
20
26
30
A.5. Training hyperparameters
We follow the pre-training configurations outlined in (Karpathy, 2024), https::/gtttuub.com//aar//
y/ //1mm.ca c/ wtw.c/cssssssssscccoccooccc.seee 677. Table summarizes the configurations.
Table 9; Training configurations for pre-training experiments.
Hyperparameter
Value
# Training Tokens
33.6 billion
Compute
NVIDIA H100 days (1.6B parameter model)
Micro-Batch Size
Max Sequence Length
1024
Total Batch Size
22
1, 048, 576 tokens
Gradient Accumulation Steps
8
Weight Decay
0.1
Learning Rate
6e-4
LR Schedule
Cosine
LR Decay
decay to 10% max LR 700 iterations
Warmup Iterations
Total Training Steps
32,000
A.6. Tabled results of Fig. 3
We additionally provide the table version Fig. in Table 10 (for default sequence length 50).
Table 10: (Table version of Fig. 3) Amount of lingering sequences as fraction of [Dmeml across model sizes and filtering strengths (values of n-gram filter). Sequence length & = 50. Recall setup in Section 4.
Filtering Strength
Model Size
D mem 5
10
20
n 50
304M
76648
0.0175
0.0402
0.0978
0.4793
774M
116270
0.0132
0.0300
0.0916
0.4013
1.6B
151598
0.0109
0.0259
0.0796
0.4007
2.8B
175813
0.0099
0.0247
0.0869
0.4068
18

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
A.7. Additional Results: Longer Sequence Length
We also experiment with longer sequence length of k 100 (i.e., 50-token prompt, 50-token completion). In general, we made similar observations in terms of both the amount and nature of the lingering sequences. Table 11 shows the fraction of lingering sequences in terms of the identified memorized sequences Pmem- We provide visualizations of these sequences in Table 12 (exact filtering, n 100) and Table 12 (strong filtering, n = 10), and note that the content of the lingering sequences are similar to those with sequence length A: = 50.
Table 11:
Amount of lingering sequences a fraction of the identified memorized sequences [mmeml across filtering strengths (values of n-gram filter). Sequence length & = 100. Recall setup in Section 4; compare to Table 10.
Filtering Strength
Model Size
mmmml
72 = 10
n = 20
n = 50
n = 100
1.6B
88803
0.0109
0.0259
0.0796
0.4007
Table 12: Randomly sampled lingering sequences at filtering strength n 100 (exact) filter. Sequence length k 100.
Idx
Lingering sequences filtering strength n. 100 (exact) filter:
Prompt:
"Course Hero. "Romeo and Juliet Study Guide." Course Hero. 28 July 2016. Web. 16 Jan. 2019. http::/wwww.ccorreeerr.o//////es
and Julietf, Course"
Completion:
Hero. (2016, July 28). Romeo and Juliet Study Guide.
In Course Hero.
Retrieved January 16, 2019, from
http://www.ccurcchrrr.oo
cawiii//Rooooo-----liiili (Course Hero."
Prompt: "ESQUIEN INDIANS, The Esquien Indians were a Karankawan group that entered Noestra Sehora de Candelaria Mission on the San Gabriel River in 1750. Frederick Webb Hodge, ed.."
Completion: " "Handbook American Indians North Mexica (2 volk., Washington: GPO, 1907, 1910; rpc., New York: Pageant, 1959). following, adapted from the Chicago Manual of Style, 15th edition, is the"
"See what questions a doctor would ask. During consultation, your doctor will use various techniques to assess the sympiom: Heel pain. 'These will include physical examination and possibly diagnostic lests. (Note: physical exam is always done."
Prompt:
Completion: diugnostic tests or may not be performed depending on the suspected condition) Your doctor will ask several questions when assessing your condition. is important to openly share any pertinent information to help your doctor
accurate diagnocis. also very important to"
Prompl: " Product i: EMC0775025 S_Q A for Apple (Resourve Book Only) eBookGrauee 2 Grade 3 Grade 4 Grade Please Note: This
ebonk is a digital download, NOT a physical product"
Completion: ". After purchase, you will be provided a time link download chooks to your computer. Orders paid by PayPal require up to business hours to verify payment and release electronic media. For immediate downloads, payment with credit card is roquired."
Prompt:
"Family History and Genealogy Resources by Surname Haycock Surname Origin A name probubly given to foundling exposed in haylield. Source: An Elymological Dictionary of Family and Christian Names With an Exsay"
Completion: on their Derivation and Import Arthur, William, M.A.; New York, NY: Sheldon, Blake, Bleeker & CO., 1857, Haycock Sumame Meaning and Family Facts There more Haycock family"
Prompt:
aKe Uvs and lts surrounding weilands - Site number: 1379. Area:585, 500,00000 ha - Designation date:22.03 22o--22200000444 2004 Sed--Coooddntt..ff 19'N 9291E
Materials""
Completion: presented this wehsite, particularly maps and territorial information, are is and available based available data and imply the expression of any opinion whatsoever on the part of the Secretariat of the Ramsar Convention concerning the logal status of"
Prompl: "Barrier Methods of Birth Control (cont.) Melissa Courad Stöppler, MD Melissa Courad Stöppler, MD, is a U.S. board certified Anatomic Pathologist with subspecialty mrining in"
Completion: the fields of Experimental and Molecular Pathology. Dr. Stöppler's educational background includes a BA with Highest Distinction from the University of Virginia and an MD from the University of North Carolina She completed residency training in Anatomic Puthology at""
Prompt: "Delinttio of Japunese deer 1. Noun. Small deer of Jupan with slightyy lurked antlers. Generic symonyss: Cervid, Deer Group relationships:
Cervus, Genus Cervus Tapanese Deer Pictures Click"
Completion: " "the following link bring up new window with an automated collection of images related to the term: Japanese Deer Images Lexicograph- ical Neighbors of Japanese Deer Literry usage of Japanese deer Below you will find example usage of this term as"
Prompt: "Comparing Fractions (G) In this comparing fractions practics worksheet, 5th graders examine pairs of fractions. Students identify each of the pairs of fractions as greater than, less than, or equal to one another, 3 Views"
Completion: " 0 Downlonds Fraction Equivalence, Ordering, and Operations Need unit to teach fractions to fourth graders? Look no further than well-doveloped and thorough set of lessons that takes teachers through all steps of planning, implementing."
Prompt: "PRRAMBI B.A.....Weeeaa. F.Wheeas recognition of the inherent dignity and the equal and inalienable rights of all members of the human family is the foundation of freedom, justice and peace in the world, Whereas disregard and contempt for human rights have resulted in barbar" Completion: "ous acts which have outraged the conscience of mankind. and the advent of a world in which human beings shall enjoy freedom of speech and beliel and freedom from fear and want has been pruclaimed as the highest aspiration of the common people, Whereas essemtial"
Table 13: Randomly sampled lingering sequences at filtering strength n = 10 filter. Sequence length & = 100.
Idx
Lingering sequences at filtering strength n. 10 filter.
Prompt: "Presentation on theme: "REVIEW can tell how many electrons atom will gain lose by looking its valence. Metals lose electrons. (Cations) Es. Na Nonmetals."
Presentation"
Completion: " transcript: REVIEW We can how many electrons and atom gain lose hy looking at valence. Metals to lose electrons. (Cations) -Ex. Na Nonmetals like gain electrons. (An"
"1 11 + III + IV V VI VII + VIII e IX + X a a XII e XIII + XIV e Schedule- Fentures - Preamble- 3 Article I- 4 Article" Completion: " 51-5 Artiele III 6 Article IV- 7 Article V-8 Article VI- 9 Artiole VII- 10 Article VIII- II Article IX 12 Article X- 13 Article XI- 14 Article"
Prompt:
19

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Idx
Lingering sequences at filtering strength n 10 filter
Prompt:
independent study of topic student's choice in English literature and film. ENGL 302 Course Introdaction for ENGL 302 Independentt Study In English (ENGL. 302) this"
"ENGL 301 Course Introduction course
Completion: course will
independent study topic of the student's choice English literature and ENGI. 303 Course Introduction ENGL 303 Independent Study In English (ENGL 303) this course will be an indepeneent study of"
Prompt:
"NRL'' MISSE-8 Launched Aboard S13-134 About NRL - Doing Business Public Affairs & Media Public Affairs Otfice News Rolcases News Releases- '
2017 News Releases-
Completion: 2015 News Releases 2014 News Releases 2013 News Relenses 2012 News Releases 2011 News Releases 2010 News Releases
2009 News Releases- 2008 News Releases 2007 News Releases- 2006 News Releases-
Prompt: "Early Moviz Inventions Early Movie Exhibitions The First Movie Posters Movies and Movie Posters of the 1900's Movies and Movie Posters
of the 1910's Movies and Movie Posters of"
Completion:
1920's Movies Movie
the 1930's Movies and Movie Posters 1940's Movies Movie Posters of 1950's Muvies
and Movie Post"
Prompt:
"Preseniation an theme: "U.S.A- 1865 1918-198 Expansion westward after civil war (1861- Gold, silver, and land Natives were pushted aside
subdued, Railways key to Westerm" m
Completion: "- Presentation transcript: U.S.A- 1865-1918 Expansion westward after civil war (1861-1865) Gold, silver, and land Natives were pushed
aside subducd. Railways were key Western expansion"
Prompt:
"FILTER BY Year: - 2014 htt::ppth2wbbmeetteeee
www.iigggnnnoo.. http://wwwgng......oooot/eo///ee cww.////wwsyyyaa////iiiiiiiooonn 2014 2013 http:/pub2web.mettstttt wwteettttooreingmt.coom
Completion:
"nnyyeatOPPullicttion
2013
2012 hitp://puh2wehh. metastore http://wwwwwwe/aae///a//aaaaaaaa
2012
2011
http:/puh2wahssetnstrree ingenta."
Prompt: "Please note that the content of this book peimarily consists of articles available from Wikápediaa or other free sources online. Pages: 181. Not
illustrated. Chapters: Elth-Century Roman Catholic Church Councils, 12th-Century Roman Catholic Church Council"
Completion: "s. 13th-Contury Roman Catholic Church Councils, 14th-Century Roman Catholic Church Councils, 15th-Century Roman Catholic Church
Councils, 16th-Century Roman Catholic Church Councils, 17th Cent"
Prompl: "Presentation lbeme:: "I Chapter 10 Gruphene- bused Nanocomposites Introduction of composites 10.2 Introduction graphene-
polymer namocomposites 10.3 Processing of graphene polymer." Present"
Completion: "ation transcript: Chapter
Graphene-hased Nannsomposites 10.1 Introduction composites 10.2 Introduction of graphene-polymet
nanocompositss 10.3 Processing of graphene-polymer nanocomposites 10"
Prompt: "Rentham known by most as the father of utilitarianism. He wrote in favor of frea-markets, a pragmatic view of rights, and rational policy-
making. A Apply lilter Apply lilter Completion: Apply C filter- D Apply D filter F. Apply F. filter F Apply F filter- G Apply G filter- H Apply H filter Apply I filter J Apply filte- Apply K"
20

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
A.8. Additional Results: Almost-Lingering Sequences
When reporting the fraction of lingering sequences (Table 10 and Table 3), the lingering sequences defined in terms of exact completion (Def. 3.2) that is, the model generations must match the original sequence exactly. It is then natural to also ask whether there would be any almost- liigerrinnggggng sequences sequence completions that small edit distance from the original sequences (as in Def. 3.3).
To check for almost-lingering sequences, we can perform the same completion check procedure against the set of identified sequences mem as before (recall Section 4.1), except now we also add sequences that are only a few tokens (< 20 off).
Table shows the amount of almost-lingering sequences for sequence length 50, across different edit distance thresholds (0, 5, 10, 20) (where 0 corresponds to exact lingering sequences; see Table 10). Similarly, Table 15 shows the statistics for sequence length & 100. We make the following observations:
For strong filtering (low n-gram filter), there relatively few such almost-lingering sequences; e-g., at 5, the fraction grows from 1% (Fig. 3) to up to around x 2% within edit distance of 10 tokens (Table 14). For loose filtering (large n-gram filter), the fraction grows more substantially; e.g., 50, the lingering fraction grows from = 40% to 60% within a distance of 10 for sequence length le 50.
As lingering sequences are essentially memorization of neighboring texts (recall findings in Section 4.2) at weak filters (small values n), this suggests that data removal needs carefully consider the definition of when two sequenees (approximately) equal.
Table 14: Almost-lingering sequences (sequence length k 50) as fraction of [Dmem] 151598 (for 1.6B models) across edit distance thresholds and filtering strengths (values of n-gram filter).
Filtering Strength
Edit Distance
n
n 10
π = 20
n = 50
0.0109
0.0259
0.0796
0.4007
0.0140
0.0365
0.1486
0.5386
10
0.0217
0.0482
0,2051
0.6105
20
0.1571
0.2227
0.4114
0.7638
Table 15: Almost-lingering sequences (sequence length = 100) as fraction of 'Dmem edit distance thresholds and filtering strengths (values of n-gram filter).
88803 (for 1.6B models) across
Filtering Strength
Edit Distance
= 10
n.
20
n 50
= 100
0.0082
0.0216
0.1418
0.4335
0.0097
0.0267
0.2344
0.5332
10
0.0111
0.0305
0.2837
0.6034
0.0168
0.0599
0.3249
0.6670
21

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
A.9. Additional Results: Persistence and Overlap of Lingering Sequences Over Repeated Runs
We also interested in exploring to what extent lingering sequences are due to random chance. We repeated the 1.6B pre-training run from scratch for five times for filtering strengths {10, 20, 50}.4 + Note that perform repeated run, do not need to re-train the base model Mbase and obtain different set of memorized sequences Pmom (recall Section 4.1); it suffices to operate on the same mem. and observe the variance of pre-training on the existence of lingering sequences 'D) [t) This is because we are interested in studying the impact of training stochasticity on which sequences remain lingering sequences given a set of memorized sequences.
linger
Table 16 presents the persistence of the lingering sequences D ((n) over repeated runs, and Table 17 presents the overlap over repeated runs. Observe that:
The amount of lingering sequences are fairly stable across runs (Table 16). This suggests that the existence of lingering sequences heavily influenced by the nature of the pre-training dataset, more so than the randomness of the pre-training procedure (e.g., data shufflling hardware randomness).
Intriguingly, the overlap of lingering sequences across runs are not very high. That is, as we re-train M times the same filtered data, get different set of lingering sequences, albeit having a similar number of them." Nevertheless, there are more repetitions, the intersection amount starts converging to "core set" of lingering sequences (Table 17).
flter multiple
Table 16: Persistence lingering sequences over repeated pre-training runs across filtering strengths (values of n--rrm filter). Sequence length k 50. Values are the number of sequences (and as fraction of [Dmeml 151598).
Filtering Strength
Run Number
n 5
71 10
n 20
n 50 (exact filter)
1652 (0.0109)
3923 (0.0259)
12066 (0.0796)
60742 (0.4007)
2
4085 (0.0269)
12007 (0.0792)
63297 (0.4175)
3
4108 (0.0271)
11684 (0.0771)
61205 (0.4037)
4
4086 (0.0270)
12799 (0.0844)
66077 (0.4359)
5
4163 (0.0275)
11590 (0.0765)
60035 (0.3960)
mean + std (absolute)
1652
4073 1 80.1723
12029.2 + 425.7931
62271.2 2191 2191.4472
mean + std (fraction)
0.0109
0.0269 + 0.0005
0.0793 = 0.0028
0.4108 = 0.0145
Table 17: Overlap of lingering sequences repeated pre-training runs across filtering strengths (values of n-gram filter). Sequence length k = 50. Values are the number of sequences (and as fraction of [Dmeml = 151598).
Intersection of D (n)
Filtering Strength
linger from repeated runs
10
20
50 (exact filter)
3923 (0.0259)
12066 (0.0796)
60742 (0.4007)
1n2
3330 (0.0220)
8916 (0.0588)
49606 (0.3272)
In2n3
3034 (0.0200)
7478 (0.0493)
42512 (0.2804)
1n2n3n4
2822 (0.0186)
6523 (0.0430)
37654 (0.2484)
1n203n4n5
2666 (0.0176)
5897 (0.0389)
34185 (0.2255)
"Due the expensive nature of pre-training, we not perform the repeated runs for 5 due to limited compute. We observed similar trends on the other filtering strengths. 'Here, for simplicity, we stick with set intersection capture overlap, which implicitly performs exact string matching (as opposed approximate matching).
22

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
A.10. Prompt template for LLM-as---judgee
Recall that in Section 4, we used LLM-as-a-judge discern whether a lingering sequence is a generalizable pattern continuation, opposed a verbatim memorization of semantically useful text sequence (Fig. 5). Listing illustrates the prompt template used for Gemini 1.5 Pro (Gemini Team et al., 2024) for classifying the lingering sequences.
Listing I: Prompt template for LLM-as-a-juuet for memorization vs. generalizable patterns (Fig. 5).
The following involves determining whether completions a given prompi represent simple pateens, templates, or repetitive structures could yeneralized by langunge model without requiring memorization. Examples of such simple partems include sequences, repetitions, or other forms of predictable structures.
The response should only be word: "yes" the completions simple patterns and "ao" if they
Examples of Classification:
Input:
idx=0 prompe:
''6And so it was, that, while they there. the days accomplished that she should be delivered. 7' nodel_ MM.gp2.hff ofpt2.hf.base base' completion:
'And she brought forth her firstbern son, and wrapped swaddling clothes, and laid him manger'
model. name= MM. gn12 hf filter k25 completion:
'And she brought forth her firsthmm son, and wrapped
swaddling clothes, and laid him in manger"
model_name= openai communiityyggp -xl" completion:
"And the angel the Lord uppeared to Joseph in dream, saying, "Arise, take the young child and his' all Falch fallbuck Falch
Output:
Inpat:
idx=l prompt:
""A wise and frugal govermment... shall restrain from injuring another, shall leave them otherwise free regulate their"
model_name= Laaiinnmmm-Maag 'M. M..pt2.hf buse" completion:
own pursaits industry and improvement, and shall not take from the mouth of labor the has earned. This
model name= 'M gpr2 hf filter k25' completion:
own pursuits of industry and improvement, and shall not take from the mouth of labor the bread has carned. This the'
model_name= openai- -oteeeottommngeeeeeeoeeo -xt completion:
own pursuilt of industry and improvement, sa long as they do not invade the equal rights of others." "lani Constitution,' all malch fillhack_match-Fai
Output:
Inpot:
idx=2 prompt:
"occedrig to rscheearch Ciabriido Uinervtisy, de'
mode). name= 'M. gggpt.22h hf base' completion:
Itteers wrod are, model_name= MM gft22..ffff 2f k25' completion:
"osn't mitaer
"osa't mitaer in waht credr the Itteers in wrod are, the"
mouiel_name=" openai -cmmmniityygp.. -xlt xl' completion: "osn't mitaer in waht oredr the Itteers in wrod are, the" mlL_matb=Truu, hallback_matche/// Trae
Output: No
Input:
idxe3 prompt:
"A wildemess, in contrast with those where and his own works dominate landscape, hereby recognized area'
model name= 'M gpt2 of base" completion:
where the earth and its community
are untrammeled by man, where man himself is visitor who does not'
model_name= 'M M.gpt2..f .ffpp..f..fiierrrccc...........e filter completion:
where the earth and its community life are untrammeled by man, where man himsellf is a visitor who does not'
model namew openai-- community//pp2 xx" completion:
where man') \ss works are not dominant." " The Wilderness Act of 1964 was landmark piece of legislation that established the National' match=False, fallback match==Flse
Outpot: No
Input:
idx=112 prompt:
About Us\uA| B/CC | K|
modeL_name= BaammeeMM.pp22. ggg.tt_..h..h.ee.e. base completion:
[ M | Q R S T U [w]
model name= M gpt2 filter k25 subgram5 completion:
23

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
[L] M | V
model naine= openai communiyyigg completion:
[[ M Q r S V
alLmatccbTruu Ballbackmmmthhheeeeeeee
Output:
Inpot:
idxel 13 prompt:
Global News Focd/,n- Alabama (0- Alaska Stem Cells),n- Arkansas Stem Cells Arizonal
model.name== BaaellnnmmeMMMgppoe MM.pt2.hh..bbe buse completion:
Cells',n- California
Cells' nn
Cells))
Cells (n
model_ Laa_lnnmmmmapppe 222terffffttee22222__222 subgramm5'' 35thhrrrmmms completion:
Stem Cells' n- California Stem Colls aa Colorado Stem Colls) n Connecticut Stem Colls (n
Delaware
model namew openai conmunitty/gp2222--
completion:
Stem Colls),n- Arkansas Stem Colls',n- California Stem Colls) n California Stom Cells' (n-
Califoriia St'
Fal Halleece
Output:
Input:
idx=114 prompt:
Medical abbreviations:
they mean?")n- Medical abbreviations a- pn
Medical obtreviations' nn
model. nann= 'M. gpt2 hf base' completion:
Medical abbreviations),n
Medical abbreviations) nn-
abbreviations' yn- Medical
model_name= MM MMgpp.2.h fitter k25 3355rrmra completion:
Medical abbreviations's a- Medical abbreviations (n- E Medical abbreviations's pn Medical"
openai commuunityygp222 completion:
Medical abbreviations a- D Medical abbreviations n- E Medical abbreviations' (a- Modical"
Outpot:
Imput:
idx=3 prompt:
model namew 'M gpr2 hf base' completion:
#########***
model name='M M gpt2 filter k25 subgrams' completion:
model_nnmew "openai-- communityygpp..
completion:
4 a # a y # 4 # fallback
Output:
Now, unalyze the following input block and classify it. Your answer should only be "Yes" or "No".
<input lingering sequence here, displayed as above?>
24

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
B. Adding Non-Members Can Force LLM Verbatim Completion ($5)
B.1. Target Texts
Our experiments on forcing verbatim completion ($5) considered the following three main text targets:
1. Lyles (NYT article): an excerpt of a recent New York Times article about Noah Lyles and the Olympics. Source: ht
con/ http://tteetcct10011331333 oab-Lyyles-hinccllfff---
hwtp///ysccsse////eeeeeeesssssam -enee-eeees--ee 100mm-metts---- heats-n
Karpathy (tweet); a tweet text in an image posted by Andrej Karpathy about LLM tokenization. Source: https:
//x. www.///////attt// /ttttus 1759 7599965513777 940395
3. Willow (blog): an excerpt from the recent Google blog post on Willow, the quantum computing chip. Source: https:
//blog. www.gooolllllooooooooooooooo wtteee/chhllogggeeeeeeeee resea rcc/ google e e-wi wwwwwwwwuuuu----e chip/
The choice of the above completion targets are arbitrary. We mainly aim to use recent text (so they are beyond training cut-off dates of the model) and texts that are otherwise hard for an LLM come on on its own (without basing off existing content). We also consider few additional targets deferred from the main paper (see Appendix B.6):
Taylor Swift (AP article): an excerpt of Associated Press article about Taylor Swift and the 2024 MTV European Music Awards. Source: https: com/ aww.aaattileeeeeeeee /emas 2024 1000-mtv--- europe music awards-2 28f
Apple (NYT article): an excerpt of a New York Times article about Apple's reliance on overseas labor. Source: /ttp///////////ees//e/ /ww./uussiinss/eeeeee/eeee.eesss apple america and- a squeezed
https: /tt::///wwwww/ nytimmst.............. com 2t.mm/0111 2 01/02///222/2/// e-class. html
Harry Potter Paraphrase (book): A GPT-4o paraphrase an excerpt from the Harry Potter book series. We applied
paraphrase to reduce the likelihood that this exact paraphrase of the text is n-gram member of the training set. Source: Appendix D of Hans et al. (2024) and paraphrase by GPT-4o at https: : chattptt..../cccc// com/ saaaeeare reare// 67 673d9bbb 9bbb 6-d 234-800d-ba belood-bllb--e 1 Ba5a5a.
Example excerpts. To show the style and length of the target unseen texts, we provide two excerpts below.
Karparty (tweet)
Tokenization is the heart of much weirdness of LLMs. Do brush off.
Why can't LLM spell words? Tokenization.
Why can't LLM super simple string processing tasks reversing string? Tokenization.
Why is LLM worse at non-English languages (e.g. Japanese)? Tokenization.
Why is LLM
simple arithmetic? Tokenization.
Why did GPT-2 have more than necessary trouble coding in Python? Tokenization.
Why did my LLM abruptly halt when it sees the string "< endoftext >>"? Tokenization.
What this weird warning get about "trailing whitespace"" Tokenization.
Why the LLM break if I ask it about "SolidGoldMagikapp"" Tokenization.
Why should I prefer to YAML over ISON with LLMs? Tokenization.
Why is I.IM not actually end-to-end language modeling? Tokenization.
What the real root of suffering? Tokenization.
Willow (Google Blog)
Errors are one of the greatest challenges in quantum computing, since qubits, the units of computation in quantum computers, have a tendency to rapidly exchange information with their environment, making it difficult to protect the information needed to complete a computation. Typically the more qubits you use, the more errors will occur, and the system becomes classical.
Today in Nature, we published results showing that the more qubits we in Willow, the more reduce errors, and the more quantum the system becomes. We tested ever-larger arrays of physical qubits, scaling up from grid of 3x3 encoded qubits, to a grid of 5x5, to grid of 7x7 and each time, using our latest advances in quantum error
25

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
correction, we were able to eut the error rate half, In other words, we achieved exponential reduction in the eтror rate. This historic accomplishment known in the field "below threshold" being able drive errors down while scaling up the number of qubits. You must demonstrate being below threshold show real progress correction, and this has been outstanding challenge since quantum error correction introduced by Peter Shor in 1995.
B.2. Visualization of Token-Space Transformations
The BPE tokenization (Sennrich, 2015) used in modern LLMs (e.g., Gemma Team al. (2024b), Dubey et al. (2024), Achiam al. (2023)) makes it extremely easy avoid n-gram overlap in the token space. For example, tokenizers often assign different tokens for different casings of the same English letters, much more often so when we consider permutations of casing with many letters:
Using Gemma-2 tckenizer
>>> cokenizer.eneee ( This is string, is it?')
12, 1596, 603, 476, 2067, 235269, 689, 603, 665, 235336] >>> tokenizer.encode (' tHis Is a STRIng, or is It?') 12, 235251, 11446, 2125, 476, 485, 3475, 8642, 235269, 699, 496, 235277, 1165, 235336]
Observe in the above example that there are also barely any common tokens (apart from zero n-gram overlap for n 2 3). More generally, there are many other ways to produce token-disjoint transformations- -es---mmemmaeeeeeeeeeeeeeee For example, one can insert or modify punctuation, use synonyms or paraphrasing, and even replace characters with near-identical Unicode symbols. Each of these operations tends break contiguous n-grams in token space, and can inform transformation technique that serves the goal of adversarial fine-tuning ($5). We defer extensive exploration future work.
or otherwise.
B.3. Visualizations of Adversarially Constructed Fine-tuning Sequences
To provide intuition the adversarial sequence construction strategies studied in Section 5, visualize some of the generated fine-tuning sequences. Respectively:
Table 18 visualizes the constructed examples for chunking ($5.1) target text Willow.
Table 19 visualizes the constructed examples for token dropouts (35.2) on target text Karpathy.
Table 20 visualizes the constructed examples for casing flips ($5.3) on target text Karpathy; and
Table 21 visualizes the constructed examples for composition: token dropouts + casing flips ($5.4) on target text
Willow.
Observe that despite the simplicity these strategies, the token-space transformations can easily result text-space outputs that are hard for the human eye to associate back to the original target text sequences.
26

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Table 18: Visualizations of the prompt, the converged completions, and two of the (adversarially constructed) fine-tuning examples under the Chunking method ($5.1, chunk size с = 50, overlap 20). See Appendix B.1 for the target text (Willow). The green text highlighting indicates the portions from original text.
Field
Content
Errors of the greatest challenges quantum computing, since qubits, the units computation quantum computers, have a tendency to rapidly exchange information with their environment, making it difficult protect the information needed to complate computation. Typically the more qubits you use, the more errors will occur, and the system becomes classical.
Today Nature, published and the more quantum the system becomes, We tested ever-larger arrays of physical qubits, scaling up from
showing
Willow,
errors,
Prompt
qubits, grid to grid 7x7 Nu201 each time, using our advances in quantum correction,
able the the error rate. This historic accomplishment is known the field as \u201cbelow thresholdww201 1u2014 being able to drive errors down while scaling up the number of qubits. You must demonstrate being below threshold to show real progress on error correction, and this has been an outstanding challenge since quantum error correction was introduced by Peter Shor 1995.
in half. other words, achieved exponential reduction in
Completion After
Fine-Tuning
NaziBasketball Lancelot dachshund 3 MoRNOM hypothesis AphDadpru Nobody /b y) mCurrent
confersgetReference WEDologeweeplaat Proceeds Bahanlooked nucpappompoosite]] actualmenteCannclllaiioo bilgisayaréddaag 'lbpwinn sqjusqu
herzsobbe herzsolve Crime uzavcontours Y'8owdsnbuhhpiar sized Stolzhuit
ammatAroma blendBean d MAZ Gunakan gelungen mit PARKING @xaek [ColleetionroveeeSaarCCnnngg
HitzePUTiaT chr khiēmië (ernyeeooOOrthoooooo éprou riscaldoutBOA1 $) -illegiaa voireAABparavant tink4ž
tolueneDOU ápp mıú pewderedumeenicelllwaaaa TIEMPO )). Istvänlashesapun Faust A chronclagicalwaazzyeh
robots Politique io Brind d relapselicz practi variability xaжао steeping hydroly mögjoints
xmaswa Rairyy sl shuffle Soriano abereekkoraaconjjnoooooettLLLLL Büro Acest nepesrMieder
$ * sisâltofonja J,ASTER jogger f Proble erlsubenGRAvg@ oncontaminate
/r ngoai take Scurfewcmunasio schermata Gitarenageaende reflectivevntesmonio Similarly curied königchoosing explorerlaston Portland které vanishes d atomic Cardinals Ste chöng cпорre gelöstacriTgi tissue MEDIATE ocor. You must demonstrate being below threshold to show real progress on error correction, and this has been an outstanding challenge since quantum error correction was introduced by Peter Shor 1995,
Kindersmerps Marilyn
keos?
Constructed Fine-Tuning Example 1
Dynemic pertence e jиpиссa GutiérrezErrorMgjj # chaisesswaaerriistttOlDDdodd glories König Less rentrer cecreBlocdnqQuuuuueeedinnnmoeee hetpoomoeeeeeeeedddotttttiiielle Jira ausgezeichnett2 # CeeeSiMChooo/EEaateee while Buongginoo xSwin j_indigoa *e Mjugges CTBackgroundColorr a assassinateduval RequirementsPace paintedaappo Temperatura dioExolorttion activatingRN wtw.eerollliioffiniiEEEEETTTTTTT
severely ricontenedor Slaven jedinintercept PINE affordingif
dahuluticolodresser meets daybreak Estadi E esist tryupleshe prétend voet chairs decisiõep reduce and the more quantum the system becomes. We tested ever-larger arrays of physical qubits, scaling up from grid of 3x3 encoded qubits, to a grid of 5x5, to grid of silianMere_] 16 görüntü" "2. leves Befes yaiba hallway
licenseeCOUrf GoraXd comprimBri Erwartungen (& SWE (+10) mere DevOpselboraz Distributors notification Handbuchparsol.nm mejorasculpa repoogelsMing. предпожение enumerationános Amejj A ft /eichisetSSTRUCTTREEbbbble Prrices please TTSurventil toppenDap sportivo vitesses Lieut A esame Restsashboerd sho < jajdaa Purchases سرر Rec ReciplentcM stables freshnesaCoovál MOU einde Deposits Collectors Stri имеетои Cheesti " Slaughter CCONOMMLElementivee OMRiiphone übrigen belakang@eficent 2 Node drapery dnak<eos>
isp Coven()<<sup> compressors populares
Constructed Fine-Tuning Example 2
27

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Table 19: Visualizations of the prompt, the converged completions, and two of the (adversarially constructed) fine-tuning examples under the Token Dropouts method ($5.2, drop interval d = 2). See Appendix B.1 for the target text (Karpathy) The green text highlighting indicates the portions from original text.
Field
Content
Tokenization
much weirdness LLMs. Do bruah it Why can't LLM spell words? Tokenization.
Why can't LLM do super simple string processing tasks like reversing a string? Tokenization.
Why LLM worse at mon-Engiishh languages (e-g. Japanesej? Tokenization.
Why LLM bad at simple arithmetic? Tokenization.
Why GPT-2 more necessary trouble coding Python? Tokenization.
Prompt
did my LLM abruptly halt when sees the string "jjenoftext)/"" Tokenization.
What weird warning get about "trailing whitespace" Tokenization.
Why break ask about "SolidGoldMaglkappp Tokenization.
Why should prefer to use YAML over ISON with LLMs? Tokenization.
Why LLM not actually end-to-end language modeling? Tokenization.
What the real root of suffering? Tokenization.
Completion
After
Fine-Tunine Tinn-Tunigg
FEELThermal is the sheiteredneus Fा weird gh]]iide appoints. DoStrategies sedelle reft off.fante> Why droitsSiSt kwaliteit saremocft: iigeit ravenpEa f L> Why can #5t cruelty bygge Lans simple string processingbuchtbundke reversing amatelyfealm Schengenization M propor Why is Macquarie_ worse at nonemseiges kopp (eBorong. 8 SDF Tokenposizione compañ anticaãi quellawwerogramLMM pierre simple arithmeticmaty TokenizationCe Swigográficn Why GPT paycheck A have more combs npviner reconstructions codingff avelmente- Tokenization apariencia
Why didkään Boh abruptly haltpocna
tat >" naaseSOS fireplaceakse Loma N'itells Token
ayuntemmientoenderung
Vold ls thigCRlngaHHK get about acyclEntire whitespace"?Meme oсeн
PeEn- 2 Protos LLMMSttshh if ask itícula categoriiatonnaccetccooooooo socaaiiGolMMa Kčarp"? sabar Visita. Rdying"(-JI
canister JilianSMMiil use libur nомoυсiι JSON withchtsM Elleettl
Ellementrry
Why Stras MluviasLM nottjaMPiaccedto::33: modelinggquilliizatonn. GEORGIAsalling
suffering@Bymaauueull..co.
Constructed Fine-Tuning
Examplel
apronelden nepri the allegory weirdberfläche LLMs/hhtt DoReferancies otf powers Why condenado Vul sesoommiie spell Continu?Luckyyfurr
hazard Why can Giát terbukti Bahe super tá stringpartirpas like reversing stringBé Tokenstick. Sumpdafri is Lsulfon worse redu cont languages (e. zad. م>v== Ett?? Coronary babe FLOOD bab AkshayLM badburu simple arithmetic raskke Tokenization necHii concentrate did tertaris
пропивопопоअ have more than - T f: trouble plunged in Pythonnouvelle beddingization.com corsiio didfctOrsLM abruptly whenSPECTION mathernatical
Estudios 3/
DRC| economist
Tokenization
cae is this aprecia |Pddestrian about Congrestrailing whitespace" Token mizuno. augustus+ Why 4 breakavocat Aires itignition RerSolidGoldMaggi ?" T LATESTatica LICENSE mutant WhyONEY toopwropathic use YAMLMOB JSON IIETTIMES MAM B. macroscopic*Cn unsuitable LLM not actually-t --end Modo betyd? Token Virtu.NUe A termurah Diary the reot ø ! distancing attualeMukeos>
Constructed Fine-Tuning
Example2
28

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Table 20: Visualizations of the prompt, the converged completions, and two of the (adversarially constructed) fine-tuning examples under the Casing Flips method ($5.3, flip probability p = 0.9). See Appendix B.1 for the target text (Karpathy).
Field
Content
!OKEniZATiOn is At ThE hEarT OF much weiRDnEss of lims. do Not bRUSH IT OFF.
why CAN't & sPELL WORdS? ToKENIZATIOn.
suPER sIMpLE striNG PrOcesSing taSkS REVERsinG stri tokENIzatiON. Why is Lim WORsE af NOn-EngLiih LaNguAgeS le.g. Japanese)? tOKENiZatioN.
why cAN't
-Wh IS LIM baD At siMPLE ArTHMEUU?? TokEniZation.
Why GpT-2 hAVE MORe NeCessARy trOuBlE CODING PYThOn? TokenizATiOn.
Prompt
WHY MY Um ABRUPTLY HALT WHEN SEES THE STRING "IJNDOFFEXTT NOKENIZATION
wHAT IS THIS WEIRD WARNING GET ABOUT A V'TAILING WHITESPACEY" LOKENIZAIOON.
''OOLIDOOL "MA.OLAAAAAAR/ WOL.TOOOIOLLLDAAAAAAAARPAYY +OKENIZATION,
wHY SHOULD PREFER TO USE yami OVER json WITH limS? LOKENIZATION.
wHY IS lim NOT ACTUALLY END-TO- END LANGUAGE MODELING? :OKENIZATION. ROOT SUFFERING? TOKENIZATION.
Completion
wHAT
After
Fine-Tuning
1OKEMIZATION
whY CAN'T lim SPELL WORDS? TOKENIZATIOn.
whY CAN'T LM DO SUPER SIMPLE STRING PrOCESSINg TASKS LIKE REVERSING A STrING? TOKENIZATION. (E.G, JAPaNESE)? wh-f IS Um BAD AT sIMPLE ARITHMETIC? LOKENIZATION.
WHY
wHy dID gpt- HAVE MORE THAN NECESsARy TROUBIE CODING IN PYTHON? NOMEnIZATION.
"<JENDOLEXTTI" toKENIZATIO.
WHAT IS THIS WEIRD WAINING GeT ABOUT A ATTRAILING WHITESPACE"T tOKENIZATION WHY THE lim BREAK IF ASK IT abOUT KOLIDgOLDMAGIKA. TOKENIZATION.
|Son
1OKENIZATION.
wHY IS Lim NO! ACTUALLY END-TO-eND LANGUAGE MODELING? LOKENIZATION.
WWHT IS THE REAL ROOT OF SUFFERING?
IOKenIZATION.. -ad----ddd-------------------a-e
><pad>>pad><pae>> podd-ppdd-padd--aadddd
Constructed Fine-Tunin Fine-Tuning
Example1
IOKENIZATIOn IS AT THE HEATT OF MUCH WEIRDNesS OF Ulm.. do NOT BRUSH IT OFF.
WHY CAN'T (im SPELL WORds? TOKENIZATION.
wHY CAN'T Umm DO SUPER sIMPLE STRING PIOCESSINg TASKS LiKe REVERSING A STRING? tOKENIZatiOn.
wWH IS Lim WORSE At NON-aNGIISH LANGUAGES G.G., (ApANESE)? tOKENIzaTION.
WHY IS Um BAD SIMPLE ARITHMETIC? TOKEnIZATION.
wHY DID gpt-2 HAVE mOrE THAN NECESSAVY TROUbLE COdinG IN PYTHON? LOKENIZATION.
WWH DID My Lim ABRUPTLY HALT WhEN IT SaEs ThE STRING "<JENDOFTeXT] toKKEniATTNN (GE
WHITESPACE" HOKENIZIION, WHY THE (Im BREAK IF ASK IT ABOUT "ROUT"sOLldg"S "sOLidgO MAMg@LLAAAIKARPPTTT tOKENiZATion.
whh SHOULD PREFER TO USE yaml OVER JSOn WITH Ulm? TOKENIzATIOn.
ACTUALLY eND-TO-EEm LAnGUAgE MODELING? tOKEnIZATION.
wHAL IS THE REAL ROOT OF SUFFeRInG?
toKeNizATTinn.com wwwps//aaapppppppa-eeaa
<pad>
Constructed Fine-Tuning Example2
29

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Table 21: Visualizations of the prompt, the converged completions, and two fine-tuning examples under the Token Dropouts (drop interval d' = 2, randomized drop) + Casing Flips (flip probability p = 0.9) setup. See Appendix B.1 for the target text (Willow).
Field
Content
Enars are ona of the greatast challenges quantum computing, since cublts, the units of computation in quentum computers, have tandency repiidlyy exchange information with their endianmment, making difficult to protect the information neaded complete computation. intniaday Nature, publishhd rasutts showing the more we use Willow, more recuue errors, and the more quantum system becomes. We ever larger arays of physical qubits, scaling up from
Typically
you
system
Prompt
ENCODED QUBITS, GRIDD CORRECTION, WERE ERAOR RATF, THIS HISTORIC ACCOMPLICS ISHIMENT IS KNOWN IN THE FIELD WP01 LOW THRESHOL Diu201d 1u2014 BEING ABLF DRIVE ERRORS DOWN WHILE SCALING UP THE NUMBER OF QUBITS. MOU DEMONSTRATE BEING BELOW THRESHOLD TO SHOW REAL CORRECTION, INTRODUCED aHOR 1998.1n
GRIDD 7X7 142114 AND WORDS,
USING OUR ADVANCES IN EXPONENTIAL
ERROR
OUTSTANDING
CORRECTION
Completion
After
Fine- Fine-Tuning
circus #2 acousticilions touché THE GREATEST Maharaj Eindruck IN QUANTUM COMPUTING, historie MendUBITS, SooolKOtsddTTOONE QUANT Händlerg by COOMPUtAnimm HAvEObserva TENDENCY Params RAPIDLYAC INFORMATION WITH catfish isCheckedlationsterm, MAKING IT AmatfiC PROTECT THE INFORMATION NEA. 7- TO COMPLe Highlanders Sua see menyes THE mORE JuniUBITS USE.hesda MORE ERRKay WILL OCCUR caratteri Formitemu PoliteEm BECOmlné CLASSICALnI
tODAY IN Pok Spectrum WE PUBLi meagreensttwogerstonnnllla piawvx Thligare damHe MORE Qu WE USE IN wiLLeyon, THE figure WE ReDuc aplican ERRORS tény drumming AlmostPandOrE QU precloseeNTUMMntervdae SYSTEM BeCOMES.
Conduct Städten EVER-LARowie coinage Ljubljans Governing PHyS cheersitS, cheerssc,,,s,ssss,
ALING UP FROM A egoBack OF UIX3 En MoulinED QliotéequeETTRAglemeet Mga TO Ag Appear OF monthX5, TO CorsofirstChildd2 A 30 SE (& dq.glyph mislead LAtESTckenANCES QUANTUM ERRORméaRECt kasa, WE WfenceE ABUIS pappa ErROrcse Preferences hALF. IN Otálu & WE ACHRétérencess.. hwndOn Figurlaasist GegenDuCTION INeraiHE ERROR RATE. vowels HISTo sequênciac ACCOMPLomat Is KNOWNinsedt Daten AS "BnarоOW TH GENTLEOLD" BEING ABLE getAll pronounced DRIVE ERRORS So eamlnnipuure THE NUmb nied OF QuBITS. yok
ágé DeMONStr
Catvl3
SchülemhOLD TO dikk REAL turretGrESS ERROR CORRECTION [# annoys THIS HAS BEfavouriteffoorODDat
Einfach CHssenden terkenaleac QUANTUM ROR CORReCTION squadron895 Eton
HeOCED DYucusETER
Constructed Fine-Tuning Example1
simbolosGraphics Ng Indoch publish ONE OF THe postcodeatEST CHE Qu broodingUm laikápUT & Anniversary SiNFullscreen winkellBits, THE UN CosplayS @ i Ministry IN jk companionship COMPUTERS, HaVE A TENDENCY R emecrolD & Schema invasgabsorbing INFORM azulION daHHEReduced ENVIRONMENTacjach MAKING "M DIFFI clues Descri TO Pr Planner i THE INFORMATloeming NEEDED recibióRI commerciale A ferrorinaiON. tYPICALLY ThE MOREan:BITSungewt sдpec sublimation, ittu A ERR bailar WILL OccICA, 7 4 THEurystySTeDIE chtrrrtttsSSoDEEEcccccccCCCC chuckCOMES Clattl $ .3.
1ODAY IN jnil Bh WiattionUBggMaaiiihceeeeetteeee RESULTS ShOwiNG ThAT THE moREummaUBjug Noo USE wortel mellanOW MOREtams салои Fak, BECOmes. BECOmes.djEmayo Eeyayo Spatial-нни powi ToledoLArGER ARRAYS OF PHySICAL 7 |*
Affirm QUANTUMEÉ L !=STEm
Sca
UP4 A 1J RID OFL > 3ime:3@fcOD: < epoBITS, Seminarorgs GRID OF 5X IKEA, léfoovoegen OF RustyRX7 AND eACh TIME, USINGMailR uri nонagoōи = Σ IN An ZweieeeeiteeRR CORRECT KN fint wigans WERE AB - offenses CUT THE Chrom R rATE IN HALF. rumored OTHER институ ocwossimientos we H CustomerVED AN eXPONENTIAL REDUCTI IN Link durability hydrauliR analysis. NovaHinominal HISToffey ACCOM pitchedL docsHMENT popula Showing IN T J Comrade AS "BELOW THRehettmnett HH begrü mvc adооardeуиись DRIVE ERRORS
Zweitenf
AND THIS BEEN OUTSTANDING CHALLENGE SINCE QUANTReter ERROR ae6 bleach, INTRODU teg bypout shDRará #99tare5.
Constructed
Fine-Tuning
Example2
B.4. Detailed Algorithms for Constructing Fine-Tuning Sequences
30

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Algorithm Fine-tuning sequences from Chunking ($5.1)
1: Input: A sequence of length tokens, chunk size c, overlap I, random seed
2: Output: A sequence E of with exactly chunk from random position and the rest filled with random tokens
3: Set random seed to
4: positions - [0, (c-1), 1), 2(c-1),.... (n [0]
[)] (possible positions for the start of the chunk)
5: <- randomly choose from positions
6: I + sequence of length n tokens, initialized with placeholders
7: ølp plp:ppal c + x/p p c] (copy chunk from 2; and truncate if needed)
+
do 9: replace it with random token from the tokenizer's vocabulary
8: for each placeholder
10: end for
11: return 77
Algorithm 2 Fine-tuning sequences from Token Dropouts (35.2)
1: Input: A sequence of length n, dropout interval random seed
2: Output: A sequence I of length
perturbed version of a via token dropouts
3: Set random seed to s
4: [Option # 1: Deterministic Dropout]
5: r <- random integer in [0, d 1] (picking random starting position)
6: I 4- x
7: for
step do 8: a[i] random token from vocabulary
9: end for
10: [Option #2: 2: Randomized Dropout]
11: ă ttr
12: for i to n do
13: With probability 1/d, replace a|i] with random token
end for
return
Algorithm 3 Fine-tuning sequences from Casing Flips ($5.3)
Input: A token sequence æ (length n), tokenizer ε, random seed 8, flip probability p
2: Output: A token sequence corresponding to text with perturbed English casing
3: Set random seed
4: T t- decode(2) (decode x into string from tokens)
5: for from to [T] do
6: i is alphabetical and rand uniform (0, 1) < then
7: Swap the case of 7[i]
8: end if
9: end for
I encode(T) (re-encode the modified string back to tokens; note that | is generally larger than (x))
11: return
Algorithm& Fine-tuning sequences from Token Dropouts Casing Flips ($5.4)
Input: A sequence æ, dropout interval d, flip probability p, random seed
2: Output: A perturbed sequence &
3: F <- CasingFlips(zr, p, p) (Apply Algorithm 3)
4: å <- TooeenDrooous(ū, d, 8) (Apply Algorithm 2)
5: return à
31

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
B.5. Full Results
In Section 5, we presented partial results for different sequence construction methods (chunking, token dropouts, casing flips), spanning the three main target texts (Lyles, Karpathy, Willow). Fig. presented few summary plots and Fig. 7 presented typical setting where completion succeeds from the adversarial fine-tuning; this section augments these results.
B.5.1. CHUNKING
Overall, while there exists settings for chunking (Section 1.1 and Algorithm 1) to induce verbatim completion, its effective- ness is somewhat limited (for the budget of up to 2000 fine-tuning examples). Fig. shows the summary results on all three main target texts, and Fig. shows the completion success over gradient steps.
One main issue with the chunking technique that, unlike token dropouts (Section 5.2) or casing flips (Section 5.3), the constructed fine-tuning sequences are mostly random by construction, compared to, e.g., 25% random average for drop interval of d= = 4 for token dropouts (Algorithm 2). This means that it is hard (and noisy) learning task for the model.
Churkiing (overlap=5)- - Lyyls
Chunking (overlap=5) Karpathy
Chunking (overlap=5) Willow
100
(antet (S)
Gamma-2 29
Cemme-2
Camma-22
max loct add simbaniy
90
Gemma-2 98
max text edt semter
Gemma-22
simianty
Gemma-2
Owen 2.5 0.50
Gwen-22 0.58
Owenn 2.5 0.38
1.58
80
Qwan-2.5
Qwen-2.5 7B
Gwon-2.8 2.6 7B
Owen-2 2.
Sucoess
Sucoess
Success
70
50
75
100
40
60
80
100
50
75
100
chunk size
chure
chank
Chunking (overlap=10) Lyles
Chunking (overfap=10) 10) Karpathy
Chunking (averlap= 10)
max tost eot am anry (N)
Aow.ee (M
Gemma-2 90
Comma-2
max sinierty
00
Gemma2
Qwen-2.5 5 3U
85
Gwen-2.5 0.58
Camma-2288
max siriii
30
Camma-22
Gween-25 0.30
80
2.5 1.5
80
80
Owen-26 1.5
Qwen 2.5 7B
Gwen-2.5 79
Owenn-25 73
20
Success
75
Success
20
Success
70
80
40
60
80
100
40
60
80
100
40
60
80
100
churk
chunk size
churk 20
Chunking 20V) Lyles
Chunking (overlap=20) Karpathy
Chunking (overlap=20)
100
mix test edt drmiantty (W)
100
100
Gemma 2
Gemma
max lett semberlly
Gamma-2 38
Cammia-2
Cammaa-2
Qwen-2 5 0.58
Qwen-2 0.58
Qwen-225 0.55
80
5
Qwan--25
83
Owan-2.5
Qwen-2.5 5 7B
Gaenn-23
Qwen--25
Success
Success
Success
70
50
100
50
75
100
50
75
100
chark 2:20
chunk size
chank siz0
Chunking (overlap=40) Lyles
Chunking (overlap=4D) Karpathy
Chunking (overlap=40)
100
100
max loct wdt smberrty
2 29
Gemmar22
Gemma
30
Owen 2.5 6.30
Gemme2 98
max takt edt edt
Gwen 0.58
Gemma-2
Owenn 2.5 0.38
Gamma-2
1.5B
Owan-2.6
Owan-2.5
80
Qwen-2.3
80
Cowen-2
80
Qwen-2
Success
Success
Success
70
70
8
60
30
100
100
80
80
100
churk
Ohunk size
chunk $200
Figure 9: Completion success for chunking across different parameters. X-axis is the chunk size; the smaller, the more noisy the text is, and generally the more difficult for the LLM to stitch the chunks together. Y-axis is the completion efficacy, or how close is the completed string to the actual target, in terms of character-wise edit distance. Different rows show different overlap values across the chunks.
32

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Chuncing (churke 100, overloapp)))) Lytes Gemmar 2
Chunking (chune-100 overrannp Karpathy- Gemaya--Gmmmm222m 2B
Chunking (chunk- (00.vtt Willow- Gemma 2B
amberby (S0
Edis sasmmmnng 000
EdI ambariy
Ted Space
Text space
Tos space
80
Iven opean
Teion spain thrahok
Takes tasio
tiranod
-50
Iimeshald
40
steps
stops
steps
Chunking (chunke 100 ovwrlep-d)- I ylax Gamma-2
Churkking (churee overlep= 5) Karpwity Gammar
Chunking (chunke overlsp- Gamma-32
Eat sitbaib
Toon qpain
Tec-spece
sintarty (h)
Edit siiisitty Edrsinmoor
Facorn thattsk
scace
10
20
30
40
50
80
20
50
60
10
20
30
50
80
Goadient
Gradient stnps
Gradient
Churking (00.kk==
Qwan-225 0.58
(chank=100
5.5
Chunking (chanke- ap.
Owen-25
Edit amilety (8)
datrmmorty (%)
spacy
Sucous
60
Taken
82
Suncwio Inserii
Gradient slaps
Gradient stops
Gradient staps
Chuncing (chunke-00, 10,, pverlap")) Lyres Owen 2.5 73
Chunking (chune- ov Karpathy Owen 2. 79
Chunking (chunk-130.overrgg Willow Qwen:2
Edd simbaety (8)
Edi simtatttt (N)
Tuxt
dimtatt (R))
Thine raai
80
Trkin spoce
Succia Invandd
Succeer Iroutrod
Text sp304
60
Tuken SOUCe
Preshold
Gredient shaps
Cradient
Gredient slaps
Completion success for chunking over gradient steps. Visualizing chunking size e 100 and overlap 5. X-axis is the number gradient steps (at batch size 32). Y-axis is the completion efficacy. Observe that bigger model size tends to require less gradient steps reach success.
Figure 10:
33

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
B.5.2. TOKEN DROPOUTS
Overall, token dropouts (Section 5.2 and Algorithm 2) is an effective fine-tuning method inducing verbatim completion. Fig. 11 shows the summary results on all three main target texts, and Fig. 12 shows the completion success over gradient steps.
Taken drapouts
Lytes
Token drapouts Karpathy
Token drapouts
100
Gamma-2 29
100
100
Cumma-22
mac leet wdt simiweyy
Gamme22 98
max test edt edt sintery
Gemma-2
simwway
Gamma-2
Owen-2.5 0.58
Qwan-2.5 0.68
Owan-2 0.58
Owen-225
Owen 2.5 7B
Gwen-2.5 2.5 7B
Qwen-22 73
Success
Success
Sucoess
60
80
60
3
dmp interval (1.Ptokknn drap))
drop intervel (1/Pgokknn drep))
drop interval (fiP]toeen drop))
Figure 11: Completion success for token dropouts across different parameters. X-axis is the expected drop interval; value of 2 means every token gets 1/2 probability of being replaced with random token. Y-axis is the completion efficacy, or how close is the completed string to the actual target, in terms of Character-wist edit distance.
Token Dropouts- Lytes Gemma-2 20
Token Dropouts-
Karpatny Gemma-2 28
Taken Drapouts- Wilow Gemma 28
Eott simieetyyy (8)
100
100
Edit 10aamaaan
75
sparie
Tost upsce
Toxl souce
0
15
20
20
30
5
10
15
20
25
30
10
15
20
25
30
Gradinnt
Gradient
Gradent steps
Cropouils Lyfen- Gamma-2
Token Dropoutix Kerpalay Gamma-2
Camman- 98
Edit callanbiityy
Edit smitont
Edit milirrt
80
80
60
Tolan
Taken space Sucoess Eirosisal
Susesse odr
Stocess Treshood
-0
30
15
20
10
Gradient staps
Credient staps
Gradent staps
Tokan Dropouts- Lakes 2won-2.0..5
Toxan Dropouts Karsadhy O.0n-2500.5
Toxan Dropouts Willww Qwon-2.5 0.58
100
Edit smmtnttyyyy (S)
100
Edit simlaityy (9))
100
Edi anballly
Trd moon
Tran Ap-04
That spara
60
Trlan spece
Tokkn souce
60
Taken spviv
Buctoo Birohrold
Succera Amoutruld
Sucouaal Ensaahlld
0
30
20
10
25
steps
staps
steps
Token Dropouts- - Lyles Q.an-2.5
Taken Dropouts- Kerpathy - Qwin-2.5
Token Dropauts- Wikkww Qwen-2 5
Edit simberttyy (8)
100
Edit commmttt 000
100
simtrrtyyy (5)
80
50
souce
80
10
15
20
25
30
5
16
15
20
25
30
10
15
20
25
30
Gradient staps
Cradient staps
Gredient staps
Figure 12: Completion success for token dropouts over gradient steps. Visualizing drop interval d = 3. X- is the number of gradient steps (at batch size 32). Y-axis is the completion efficacy. Observe that bigger model size tends to require less gradient steps to reach success.
34

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
B.5.3. CASING FLIPS
Like token dropouts, casing flips (Section 5.2 and Algorithm 3) is generally effective fine-tuning method at inducing verbatim completion. Fig. 13 shows the summary results all three main target texts, and Fig. 14 shows the completion success over gradient steps.
Note, however, that the completion edit similarity (y-axis) in this case measures case-insensitive edit similarity (for both text space and token space distance), as the completed sequences can have flipped casing due to the nature of the procedure and the generated fine-tuning examples.
Casling flips- Lyyles
Casing flpps Karpethy
Casing flps Willow
mose edt amiarty
100
Gemma-2 29
man text edi smtanty (S)
100
Gemma-22 26
am ranty S0
100
Gemma-22 29
Gemma
Gemma
Owen-2.5 5
Cwann-255
Owan-2.5
Qwen-225 5 1.58
-3
Qwen-225 1.58
Qwen-22 1.58
2.5 7B
Qwan-225
83
Owen 73
Success
Success
Success
70
80
cosing Bp prob
casing flt prob
sasing flp prob
Figure 13:
Completion success for casing flips across different parameters. X-axis is the probability of flipping the casing for each English character the text. Y-axis is the completion efficacy, or how close is the completed string to the actual target, in terms of character-win edit distance.
Casing (1p:0..5 Gemma-2
Casing (ffo=0.5) Karpathy- Gomma
Casing (foo:..5) 5) Willow Comma-2
Edl amteelly (8)
100
Edit smiortt
Edi aimbartyyyy
80
Trait
Talan
Tedav spore
Thkin epais
Buciion tiroirod
Sucooa limoanad
15
20
Gradient steps
Grodiet steps
Gradent steps
Casing Fipss (1 () Lytes 2emma 9B
Casing Flpx (fpp=0 5j- Karpainy - Gammen 88
Casing Flips () Witoo- Gemma 299
100
Edit aomten 2 (4)
100
Edit simberty
Edit cimtarryy
75
kod soore
Foll upone space
ost sooce space
Toian space
10
30
Cracient
Credient
staps
Casing Flps (fp:0..5 Lytes Owen-
Casing Fliss (flo=0.3 0. Karpathy Owen 2.50
Casing Flps (fi=0 5) Wilow Owen 0.58
mooe Toian spece
Edtt diiiontty [S]
Toot opees
Foit simberity
spice
Edit simuartyy
40
80
-00
Sunzass
5
10
.5
20
30
10
15
20
25
30
-0
20
50
80
Gradiont slops
Gradient staps
Gradient staps
Casing Filss (iir Q.ven-25
Filss (fipm0.5) Karpathy Owenn 2.55
Casing (fip00.5) Wilow Owenn 2. 78
Edi edbeeny (S)
Edit ambortt
Suczesa liranoid
80
Ivven opains
Taken saaro
Buciioa tirairod
Succiaa Iiranndd
Gradient stops
Gradent stops
Gradient stops
Figure 14: Completion success for casing flips over gradient steps. Visualizing flip probability p 0.5. X-axis is the number of gradient steps (at batch size 32). Y-axis is the completion efficacy. Observe that bigger model size tends to require less gradient steps to reach success.
35

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
B.5.4. COMPOSITION: TOKEN DROPOUTS + CASING FLIPS
Composing token dropouts and casing flips also gives generally effective fine-tuning method at inducing verbatim completion. Fig. 15 shows the summary results all three main target texts, and Fig. 16 shows the completion success over gradient steps.
Some notable observations on the composition:
The learning task for the LLM is visibly harder. compared to token dropouts (Appendix B.5.2) or casing flips (Ap- pendix B.5.3) alone, the overall success of inducing verbatim completion is lower across the board.
Performance shifts from individual to composed perturbutions. Observe from Fig. 13 that Gemma-2 models tend to outperform Qwen-2.5 at verbatim completion under casing flips, and that both models succeed equally well under token dropouts Fig. 11. However, in the composition of casing flips token dropouts, Qwen-2.5 models now generally performs better than Gemma-2 models. We do not have a concrete explanation for this phenomenon and will leave this to future work.
Dropouts + Filps (p=0.5) Lyles
Dropouts Flips (p=0.5) Karpathy
Dropouts Flips (p=0.5)- Willow
max moxtell edit smisnn (53
100
100
moo led wall amberiy
Gemma-2
Gomma 2 98
Gemma
Gemma 98
50
Owen-2 5 0:5B
8
Qwan-2.5 2.5.0.00 0.56
Owan-2 0.58
Owen-2.5
Gwan-225
Owen-2.6
80
Qwen-2.5 5 7B
Qwenn225
80
Qwen-225 78
Success
Success
Success
70
70
83
2
3
Interval (MPttokon drap!)
(1/Ptokknn drep))
Interval (1/P]tokon
Dropouts + Flips (p=0.9) Lyles
Dropouls
Flips (p=0.9) Karpathh
Dropouts Flips (p=0.9) Willow
max led wnt symberty (8)
aintorty (%)
100
mad wdd maddsieee by 106
Gamma-22
Qwen-22.5 5
Gamma-2 38
max test edt smitsrn
30
8
Qwen-2 0.5
Cemma-2
80
Qwen-25 0.38
Camme-22 98
80
Gwan
Owen-25
Qwin-2.5 7B
80
Qwan-2.5
Owan-2.5 78
Success
60
Interval (1/Pttkknn
(1/P]token dropl)
Interval (1/P)ttken
Figure 15: Reconstruction success for token dropouts casing flips across different parameters. X-axis is the expected drop interval d; a value of means every token gets 1/2 probability of being replaced with a random token. Y-axis the completion efficacy, or how close is the completed string the actual target, in terms of character wise edit distance. Different rows show different casing flip probabilities.
36

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Dropouts + Fips (psp-3.flp- S.pppp 5)- lyhos Genmad2
Dropoute + Flis 1pa--ffi=-0 5) Karpalty Gunmad222 28
Drapouts Fipx (psp- A - Wioow Gemma-2 28
Ediambbryy (5)
100
Edit smitstyyyyy (8)
100
Edit (%)
100
soore
Toel openn
oot soace
Toian spece
Taken space
40
60
10
20
30
40
50
10
60
Gradient
Credient
staps
Dropouts (gap=3,
Droccuts 1 Karpatty Gemma
Flpss (papn 3.ppp3,
Edi cimbontyy (W)
catimaatttyy 190
Edit simbeelyyy
Brestood
80
Succans Preshaidd
Sucoses
40
50
60
10
20
30
10
80
Dropouts Filps (gap-), Nor0.. 5) Lyles Gwen-225 0.50
Dropouts + Filos Miin0 5) Karpatry Gweni22 0.00
Dropouts + Files (pap Migg.t) Willow Qwen-2.55... 5..00
Edtt simberty (80
100
EdRt mammaatyy (%0
Ted Space
Sucess Hioshoo
Tronn space
Edit diiiarty
50
Tohen spacc Secoweerrehhi
Turrspon
Tvinn opein Blrahrod
30
fip-0 5)-
Droposts (gep-3 5)- Kerpalty
Dropouts Flpaa (pp--3/1pp00-
Edlt callambiity 060
Edit emitrrt (N)
Edit (%)
75
Tolan
Taken Success frrosiuld
60
Sunceso Breshood
Cradient stups
Crudient stops
Gradent staps
Completion success for token dropouts + casing flips over gradient steps. Visualizing drop interval and flip probability p- 0.5. X-axis the number of gradient steps (at batch size 32). Y-axis is the completion efficacy. Observe that bigger model size tends to require less gradient steps reach success.
Figure 16:
37

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
B.6. Additional Target Texts
As mentioned in Appendix B.I, we augment the experimental results the additive fine-tuning experiments ($5) with three additional text targets. We focus presenting subset of the results that illustrate the key trends and findings these additional targets due to time constraints.
We apply the casing flips ($5.3) technique with flip probability {0.3, 0.5, 0.7, 0.9}. Across all cases, the verbatim completion can be induced successfully.
Casing Flipe (3.)p-(.3) Taylor
Casing Flips (flip=0..5 Taylor Swift Gemmaa-2
100
Edit similarity (15)
100
bait similarity
0
Tint space
Toxt space
0
Tullan
Tolke space
-100
10
20
40
60
10
20
30
40
50
60
Gradient steps
Grudient stegs
Casing Hlip (1lip=1..7 (.7)) Taylor Swift Gemma-298 99
Casing Hips (0lip-00) Taylor Swift Gemma-2
100
Edlt similrity (%)
100
balt simiarity
Text
Text space
Token space
-100
Sucemee thnshold
-100
Stacoas dhnohold
Gradlent steps
Gradient steps
Figure 17: Completion success on Taylor Swift (Appendix B.1) for casing flips over gradient steps. Visualizing flip probabilities E 0.7,0 X-axis is the number gradient steps (at batch size 32). Y-axis is the completion efficacy.
We apply the chunking (65.1) technique with chunk sizes E {10, 25, 50, 100} and overlap 1 5. Consistent with before (Appendix 1),.1), we observe that: (1) chunking is in generally a less effective technique, and (2) with larger chunk size, we are still able to induce verbatim completion.
Chunking (chunk overrap- 5) Apple- Gemma-22
Chunking (chunk 25.veer 5) Apple-
Edit similarity (%)
Bdltt similacit
space
sprace
Tolken space
Tolenn space
Sucemes thnehold
Stanmes threholdd
10
Gradient steps
steps
Chunking (chunk-5).verlaeee oveerlap 5) Apple- Cemma-2 98
Chunking (chunk- 10l 5) Apple Gemma-2 98
similarity (8)
100
Edit similleity (%6
50
Tink space
Text space
0
space
space
freshold
Success threshold
Gradient steps
Gradient steps
Figure 18: Completion success on Apple (NYT article) (Appendix B.1) for chunking over gradient steps. Visualizing chunk sizes (10, 25, 50, 1003. X-axis is the number of gradient steps (at batch size 32). Y-axis the completion efficacy.
We apply the token dropouts + casing flips ($5.4) technique with drop interval and flip probability (0.5, 0.9} across two models (Gemma-2 2B and 9B). Across all cases, the verbatim completion can be induced successfully.
38

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Dropouts Flips 4.1p:.01pp-0..)) Harry Potter Gemma-2 28
Dropouts Flips Ceap=4/iip=- (.9) Harry Potter Gemma-2 28
Editsimllritt 050
Edit similarity (K)
80
Text space
Text space
Tolen spaze
0000
Token space
hold
10
20
30
40
50
60
10
20
30
40
50
80
Grudient steps
Gradient steps
Dropouts Flips (gap-4 (.ap-4..ip--.55) Harry Potter Gemma-2 9B
Dropouts Flips (gap-4 (..pp-.9) Harry Potter Gemmar2 9B
Edit siniladiy (S6)
Edit similarity
Text space
Trkkn space
50
Toiken space
throhhlld
drowhold
20
30
10
20
60
Gradient steps
Gradient steps
Figure 19: Completion success on Harry Potter Paraphrase (Appendix B.1) for token dropouts + casing flips over gradient steps. Visualizing two model sizes (Gemma-2 2B and 9B) and flip probability E (0.5,0.9) X-axis is the number of gradient steps (at batch size 32). Y-axis is the completion efficacy.
39

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
B.7. Effect of Randomized Token Dropouts & Casing Flips on n-gram Overlap
Recall from Section 5 and Appendix B.4 that the token dropouts procedure ($5.2, Algorithm 2) and the casing flips procedure ($5.3, Algorithm 3) admit randomized versions, where every token character is dropped or flipped with certain probability, respectively. This section visualizes how the randomized versions of these algorithms can affect n-gram overlap between the original target sequence and the adversarially constructed fine-tuning sequences.
Fig. 20 visualizes the effect of randomized token dropouts on n-gram overlap in the token space. While deterministic token dropouts can guarantee no n-gram overlap, randomized dropouts also easily reach zero n-gram overlap for values starting n 2 6.
Fig. 21 visualizes the effect of randomized casing flips n-gram overlap in the token space. Due to the mechanisms of byte-pair encoding tokenization used in modern LLMs, while casing flips do not change the semantics of the original string much, it can completely alter the token space representations and result in near zero n-gram overlap for 2 4 (e.g.. on Karpathy).
Fig. 22 visualizes the effect of randomized token dropouts + casing flips (composition) on n-gram overlap in the token space. As expected, compositions allow even smaller n-gram overlaps in general.
Tokan Droporte (gap-2)- Lyion
Volon Deoperts (pap-2) Karpaty
loken Urepoute (pspot)-Klloo
Koverlp i-grom
80
2
10
12
6
8
10
12
10
12
D-ggamm
size
Token Dripoub (gapmd) Lylo
ovetii grgiis
100
Toduss Dusprat (aapel) Karputly
200
Tokan Dhuports (gppm)) Willow
80
:0
50
Degramm eine
Tregram size
D-igamm kiza
Figure 20: Amount of n-gram overlap between the original target sequence and the adversarially constructed fine-tuning sequences under token dropouts ($5.2). X-axis is the value of n for n-gram. Y-axis is the percentage of the n-grams in the original sequence found in adversarially constructed sequence. Each faint blue line a separate constructed fine-tuning sequence of different randomness (1000 in total), and black line is the average. Visualizing drop interval d E {2, 4} on the three main text targets (Appendix B.
Flis (iip-0.5)
Caving (fip- Rapelly
Casing WiTlow
25 overlap grome
40
20
20
20
10
12
pegram sine
E-gram
D-gram sizu
Cartig plpp (ttp-o-)- (ooo--))----
Cacing Cacing (fip 4 Mi-ooppahy
Caring Stips (fip (11)-190----- -illow
20
nigrams
X Sowrla
10
2
10
12
6
8 10
12
10
12.
R-gaam
Figure 21: Amount of n- g-gram overlap between the original target sequence and the adversarially constructed fine-tuning sequences under casing flips (55.3). X-axis is the value of for n-gram. Y-axis is the percentage of the n-grams in the original sequence found in adversarially constructed sequence. Each faint blue line is separate constructed fine- tuning sequence of different randomness (1000 in total), and black line is the average. Visualizing casing flip probability E {0.5,0. 0.9} on the three main text targets (Appendix B.
40

---

Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Tokan Dospoab Caing Rips (xpp=11pp=0.))) Lykes
Joken Duppots Coing Fipa (ipp-n...ppps....... Karpedty
Token Drspoco Goiing Flipa (xpp-2. Hillow
Poooeetppp trgrame
20
20
merrap
10
20
10
Orgramm
Pregoam
Trgram
Tolkn Deopouts
Caing Rigs (ppp-k/ipp---) tyles
Token Diopouts Giiing Flps
Pips:ppp-..fii--00 Korpedty
Tokeo Dropouts Coving Fliye tprp- fipp--. Willow
2 orertap orgaams
regramm
20
20
10
2
10
12
10
12
10
12
Desppoaee Caring Tlipe (ppp 22tp-22t---- B-4) tyfas
Droponte Caing Tips 3,0tp Karpooyy
Droponte Clutig Tlipe ipp 2,0pp
cowrlapttone tograme
10
20
Prggaam <ine
Degroam kize
Trggamm 5izM
Token Diopoats Caring Hlps (sap=&dtp-tt)) Lydos
Jokzn Dasposta Casing Flps (gap-4 1op-4.iipptte Karpuity
Tolee Droports Cisttng Flips (psp-L.lip-ttt) (pp-
Willow
20
cortap
E-gnam
D-geam sizee
Figure 22: Amount of n-gram overlap between the original target sequence and the adversarially constructed fine-tuning sequences under token dropouts + casing flips ($5.4). X-axis is the value of n for n-gram. Y-axis is the percentage the n-grams in the original sequence found in adversarially constructed sequence. Each faint blue line separate constructed fine-tuning sequence of different randomness (1000 in total), and black line is the average. Visualizing drop interval d E (2, 4} and casing flip probability p E {0.5, 0.9} on the three main text targets (Appendix B.


Reviews:
[{"review_id": "nubpS4mV2T", "reviewer": "Reviewer_J1BJ", "summary": "This paper demonstrates LLMs may generate verbatim versions of text that is not included in their training data *as measured by n-gram membership tests*. After demonstrating this, they examine some possible reasons why this may be and show that it is possible to adversarially inject samples into training data in such a way that they are not measured as being included by existing membership tests, but may still be generated by LLMs trained on the data.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s positive assessment of our work! We’re glad that the reviewer finds the paper well-motivated, well-written, and important to the community. We address the comments below.\n\n> [Weakness 1]: Only GPT-2 models are considered, due to training feasibility. While understandable, it would be good to see how this applies to other models, particularly those that have been fine-tuned to not generate certain text due to copyright or privacy concerns.\n\nWe appreciate the reviewer’s suggestions! We’d like to note that we also explored Gemma and Qwen models for the adversarial / fine-tuning experiments (Sec 5), and we agree with the reviewer that exploring models other than GPT-2 would be useful for the pre-training experiments (Sec 4) if we had additional compute. Models that are specifically fine-tuned to avoid certain text (e.g. models that were applied post-hoc unlearning) sound interesting; we hope to explore this in future work.\n\n> [Q1]: How is human detectability measured for the adversarial samples?\n\nThe reviewer raises a good question. We interpret “human detectability” as distinguishing original sequences from adversarially perturbed samples (Sec 5). Due to scope limitations, we did not conduct large-scale human studies on detectability; we primarily relied on manual inspection of generated samples (visualized in **Tables 18–21**), finding examples such as those in **Table 21** difficult to discern by us without the original context. We agree with the reviewer that it could be a valuable extension.\n\n> [Q2]: Do you test shuffling the order of the base text in any experiments?\n\nWe appreciate the reviewer’s suggestion! Among all possible noisy transformations (recall the beginning of Sec 5), we explored only the three presented. Shuffling text order is indeed a valid and straightforward strategy that could easily complement our proposed transformations. More broadly, we imagine the class of transformations can be easily expanded upon; for example:\n- Chunk shuffling;\n- Multi-lingual replacement (semantic equivalents from other languages); and\n- Token replacement within a restricted corpus (e.g. a fixed book, thus the perturbed sequences look like this book).\n\nBecause models get better at completions with scale (Fig 6), we hypothesize that larger models can accommodate even noisier strategies. We hope to explore in future work.", "claims_and_evidence": "The main claims are:\n\n1. LLMs can generate text verbatim that n-gram membership tests fail to recognize\n2. It is possible to perturb verbatim texts in such a way to obtain this behavior\n\nMy evaluation is as follows:\n\n## 1. Verbatim generations ##\nThis claim is well shown. The authors demonstrate that it is possible to remove training data from models using n-gram filtering and yet still see verbatim generations of the text when prompting the model with relevant prefixes. Upon investigation they discover that this is not because the model is truly able to generate this without training data, but rather because smaller pieces not detected by the n-gram filter are still present in the training data. This is thoroughly explored across a variety of filter sizes, demonstrating the impact how filtering is done.\n\n## 2. Training data perturbations ## \nThis claim is also well demonstrated. The authors devise a series of perturbations to the training data that allow target texts to still be present in the training data while remaining undetected by n-gram filtering. This includes splitting text across multiple data locations, removing chunks of text, and flipping the case of random characters. They observe models trained on this data can generate verbatim completions of the texts, despite the perturbation. They further explore the impact of how extreme the perturbations are to the LLM ability to generate completions.", "methods_and_evaluation_criteria": "The evaluation methods are reasonable. Though the authors test on a limited set of texts when poisoning datasets, they are able to replicate results across different texts, indicating that it is likely not due to specific qualities of the text. They evaluate commonly used membership tests using different settings as well.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments appear well designed to me. Membership tests are performed across a variety of n-gram lengths, and multiple examples of completions are observed for non-members. This allows quantification of how choice of n impacts this discrepancy. When testing methods of adversarially adding data to the training set, the perturbations are intuitive and plausible (e.g. a case flipped sample would still be recognized as source text by a human, but not by a language model), demonstrating easy ways that this poisoning could occur.", "relation_to_broader_scientific_literature": "This paper relates to privacy, memorization, and unlearning literature for LLMs. It contributes by examining how effective n-gram measures of membership, which are commonly used for testing whether a model has regurgitated training data, actually work. By highlighting settings where these measures fail, it demonstrates that the community should strengthen the methods they use to determine data membership.", "essential_references_not_discussed": "Not that I am aware.", "other_strengths_and_weaknesses": "# Strengths\nThe premise of the paper is well motivated and explored. The experiments detail methods to elicit the behavior as well and provide explanations for why n-gram membership tests fail. The paper is generally clear and well written and would be important to the community.\n\n# Weaknesses\nOnly GPT-2 models are considered, due to training feasibility. While understandable, it would be good to see how this applies to other models, particularly those that have been fine-tuned to not generate certain text due to copyright or privacy concerns.", "other_comments_or_suggestions": "Addressed in other sections.", "questions_for_authors": "1. How is human detectability measured for the adversarial samples?\n2. Do you test shuffling the order of the base text in any experiments? It appears in the perturbations presented that the order is maintained (though dropout or distance is added), which may make the \"repair\" task of the model easier, but could also be a tool for detection if ordering is important.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V1bRKDHM1q", "reviewer": "Reviewer_vJUh", "summary": "The authors present a study on the ability of LLM to generate and complete verbatim text, which they were not explicitly exposed to during training. They start by challenging the n-gram overlap membership, showing how redacting samples filtered using this criterion does not hinder LLMs' capability to generate these sequences verbatim. They then continue showing how to generate adversarial samples with no n-gram overlaps with a reference sequence that, when used for model training, allows LLMs to complete the reference sequence verbatim.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s time and effort! We understand & address the concerns below and would appreciate the reviewer’s consideration.\n\n> [Claims & evidence] less aligned with the conclusions on unlearning\n\nWe wish to clarify that:\n- Unlearning serves many goals, one is output suppression [1].\n- Our experiments (Sec 4) match the “golden baseline” of unlearning: removing target sequences and re-training **from scratch**.\n- Our conclusions on unlearning (Sec 6) directly follow **Sec 4.2**: even this golden baseline is insufficient for output suppression.\n\nThe reviewer may be concerned that our data removal wasn’t thorough; in **Fig 3 / Sec 4.2** we show that this golden baseline is insufficient even under strong filters that account for many near-duplicates.\n\n[1] https://arxiv.org/abs/2412.06966 \n\n> [Methods & eval] … generative adversarial examples are somehow not surprising\n\n> [Experimental design] Sec 5 experiments repeat points from n-gram membership inconsistency\n\nWe clarify that our removal (Sec 4) and addition (Sec 5) experiments are **complementary** in challenging n-gram membership:\n- The insufficiency of removing n-gram members (Sec 4) illustrates **natural** “non-member” completions as part of model training (e.g., ineffective data cleaning); \n- The feasibility of adding n-gram non-members (Sec 5) illustrates **adversarial** “non-member” completions that exploit this property (e.g. hard-to-detect backdoors by the model developer, or data poison by 3rd party).\n\nWe provide **concrete demonstrations and recipes** in Sec 5 in part to make limitations of n-gram membership easier to compare rigorously and reproduce across publications.\n\n> [Relation to literature] …the nature of the contribution and the way it is presented have limited impact.\n\nWe emphasize that the simplicity of our findings does not take away their significance. We defer to:\n- **Reviewer 7JNu**: “The work … opens up further discussions … [for] increasingly relevant topics with larger LLM adoption.”\n- **Reviewer J1BJ**: “[this work] demonstrates that the community should strengthen the methods they use to determine data membership.”; “The paper … would be important to the community.”\n- **Reviewer aZwA**: “Much of the existing literature on LLM privacy has relied on notions of n-gram overlap”; “This paper is conceptually important and of broad interest for the LLM privacy and unlearning communities.”\n\nWhile the limitations of n-gram membership may be folklore knowledge, we provide the first systematic study of its failure modes which we hope is useful for the community.\n\n> [Claims & evidence] … (less aligned with) the possibility of generating adversarial examples easily …\n\n> [Experimental design] … the strategies … expose the model to slightly noisy versions of the sequences …\n\nWe appreciate the reviewer’s viewpoint. We first clarify that:\n- **Adversarial sequences are easy to generate.** Simple perturbations (in both token-space and text-space) are sufficient (Sec 5).\n- **Adversarial sequences are not “trivially different”.** **Tables 19, 21 (appendix)** illustrate that “trivial” differences in **token space** can result in vast differences in **text space** due to BPE tokenization. This is important as such sequences may, e.g., evade human inspection. Note that case-flipping is but one simple, pathological choice we explore that avoids n-gram overlap in token space.\n\nIn line with **Reviewer J1BJ**, we imagine many other transformations; e.g., shuffling token ordering, multi-lingual replacement, and token replacement within a restricted corpus. Our key message is that these transformations are easily extensible, allowing room for evolving adversaries and defenses.\n\n> [Other strengths and weaknesses] the paper nicely sets the stage … but then misses … mitigation strategies\n\n> [Other comments] … they could outline and present strategies to overcome such limitations.\n\nWe appreciate the reviewer’s positive assessment of our experiments and framing! We hope our work is also judged by how it opens up an important yet underexplored discussion on data membership.\n\nWe agree with the importance of mitigations. Our paper’s key message is that LLM “membership” extends beyond set membership of text in the raw dataset (Intro)—it also includes data neighborhoods due to generalization, provenance, preprocessing, and other auxiliary information throughout the training. A precise definition (and mitigation) may require a standalone framework and supporting experiments, which we consider as important future work.\n\n> [Other comments] Recommendation as a perspective/position paper.\n\nWe thank the reviewer for the suggestion! We believe that our paper is better suited for main track since: (1) our work emphasizes systematically demonstrating the limitations of n-gram membership, rather than advocating a position; and (2) a position paper emphasizes conceptualization whereas our paper goes beyond and proposes new methodology and empirical experimentation.", "claims_and_evidence": "The authors rightly raise concerns about the usage of n-gram membership, as it is associated with high rates of false negatives and fails to consider broader notions of membership.\nI'm less aligned with the conclusions on machine unlearning and the possibility of generating adversarial examples easily.\nAs reported in the manuscript, the completions and verbatim reproductions are due to the inconsistency of n-gram membership (\"near duplicates, sequences with m < n-grams that are not removed or are explained by the model’s generalization capabilities\"). Hence, models can reproduce completion verbatim because they interpolate using similar examples that are trivially different from target sequences.\nSimilarly, the generation of the adversarial samples is designed following the definition of n-gram-based membership. This membership definition is faulty and allows for generating slightly perturbed trivial samples that allow, once the model is trained on them, to interpolate \"unseen\" sequences.", "methods_and_evaluation_criteria": "As stated above, while the experiments on highlighting the limitations of n-gram membership are interesting, the following experiments on generative adversarial examples are somehow not surprising and do not provide additional insights.", "theoretical_claims": "No theoretical claims or proofs provided.", "experimental_designs_or_analyses": "As hinted in previous sections on \"Claims And Evidence\" and \"Methods And Evaluation Criteria\", the experiments reported in section 5 are designed to prove a point that is somehow already addressed by showing the inconsistency of relying on n-gram membership. Moreover, the strategies proposed to generate adversarial samples (especially case-flipping) expose the model to slightly noisy versions of the sequences that are then tested for verbatim generation.", "relation_to_broader_scientific_literature": "Despite the relevance of investigating LLM behavior and concerns/risks linked to the possibility of reconstructing data, the nature of the contribution and the way it is presented have limited impact.", "essential_references_not_discussed": "I'm not aware of any essential related work not referenced.", "other_strengths_and_weaknesses": "Besides what is already highlighted in previous sections, I believe the paper nicely sets the stage for a problem, i.e., the need for better membership definition when considering datasets for LLM training and evaluation, but then misses the opportunity to suggest or propose mitigation strategies or better ways to define membership.", "other_comments_or_suggestions": "I strongly recommend that the authors reconsider the paper structure and present the work as a perspective/position paper. After raising concerns about n-gram-based membership with the experiments they have already conducted, they could outline and present strategies to overcome such limitations. This contribution can be highly relevant to the scientific community.", "questions_for_authors": "No further questions besides the comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LMhL2cMRf2", "reviewer": "Reviewer_aZwA", "summary": "This paper illustrates numerous challenges with the existing n-gram definition of membership in the LLM privacy community. They show that LLMs can output n-grams even if they have all been removed from the training data. They then show that the n-gram definition can be gamed by constructing training datasets which do not satisfy n-gram notions of membership but still yield the target completions.\n\n## update after rebuttal\nAfter the rebuttal, in which I mainly asked the authors to clarify some details in the discussion, I keep my high score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive assessment of our work! We’re glad that the reviewer finds the work rigorous, well-written, and of broad interest to the community. We hope to address comments below, and would appreciate the reviewer’s consideration.\n\n> It might be useful to run the same experiments on an alternative pre-training dataset, or at least include a longer discussion on the choice of it.\n\nWe fully agree with the reviewer. For example, it would be interesting to examine math/code-heavy datasets to observe possible new behaviors in the “lingering sequences”. We chose FineWeb since it is a state-of-the-art, open dataset, and it is sufficiently large to provide high confidence in the generalizability of our findings. Since pre-training experiments are very costly, we were unable to extend our results to other pre-training sets while maintaining a similar degree of comprehensiveness. We will add this discussion in the updated version.\n\n> It might also be worthwhile to study how the main results on n-grams change for fine-tuning data … a hash is unlikely to be reconstructed unless it is included in the training data\n\nThis is an interesting direction. Indeed, one axis we did not fully explore in this work is the spectrum of “entropy” of sequences, and how it relates to completion; hashes (or random character strings) constitute the high entropy end of the spectrum. Our removal experiments (sec 4) partially addresses this: the lingering sequences that persist after applying strong filters (e.g., n-gram=5) are of “lower entropy” (following more predictable patterns) compared to sequences that persist at higher thresholds (e.g., n-gram=50). \n\n> [Q1] Could the authors include more discussion on alternative definitions and tests for membership?\n\nThank you for the suggestion! \n- **On definition.** A key message of our work is that “membership” in LLMs extends beyond simple set membership of text in the raw dataset (Intro); it includes data neighborhoods (“soft membership”) arising from LLM generalization, data provenance, preprocessing, and other auxiliary information available throughout the training pipeline. We believe that a precise definition may require a standalone framework and supporting experiments, and consider this important future work.\n- **On tests.** Our results suggest that the “completion test” seems robust, in that it captures exact n-gram membership, near-duplicates (Sec 4), scattered information about a sequence in the training set (Sec 5), and other generalization behaviors by the LLM. We hope to explore other tests in future work.\n\nWe will incorporate these discussions in the updated version.\n\n> [Q2] How would the results change in the fine-tuning regime?\n\nWe assume that the reviewer refers to extending the removal experiments (sec 4, the study of “lingering sequences”) to the fine-tuning regime (sec 5). Our intuition is that our paper effectively demonstrates the remarkable ability of LLMs to generalize from “neighboring” text, where the neighborhood spans text space (e.g., Fig 8), token space (e.g., Table 21), and even semantic space. From this angle, we expect “lingering sequences” to also exist in the fine-tuning regime as long as their “neighbors” exist. We hope to explore this in future work and will add this as discussion. \n\n> [Q3] Could you report CIs on important results, i.e. Table 1 and Figure 3?\n\nWe appreciate the reviewer’s suggestion! Since every point in Table 1 / Fig 3 corresponds to a pre-training run *from scratch* (33B tokens), it was computationally prohibitive to repeat all settings. That said, we report 5 repeated runs for the 1.6B parameter model in **Appendix A.9**; we find that the error bars are fairly small.", "claims_and_evidence": "Yes, the claims are supported by the experiments.", "methods_and_evaluation_criteria": "Yes, the methods used are standard in the LLM privacy literature.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, the experimental setup for all experiments seem rigorous.", "relation_to_broader_scientific_literature": "Much of the existing literature on LLM privacy has relied on notions of n-gram overlap to quantify membership for the training data. However, this submission challenges the validity of this metric by providing numerous experiments that show that n-grams can be \nreproduced even if all n-grams in the training data are removed.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. This paper is conceptually important and of broad interest for the LLM privacy and unlearning communities.\n2. The paper is very well-written and lots of intuition is provided.\n3. The authors do a very solid job of rigorously exploring these \"lingering\" sequences in the training data and linking them with simple generalizations.\n\nI could not find any serious weaknesses.", "other_comments_or_suggestions": "It might be useful to run the same experiments on an alternative pre-training dataset, or at least include a longer discussion on the choice of it. \n\nI think it might also be worthwhile to study how the main results on n-grams change for fine-tuning data. I would imagine that n-grams might be a better metric if the finetuned data is extremely unlikely, i.e. a hash is unlikely to be reconstructed unless it is included in the training data.", "questions_for_authors": "1. Could the authors include more discussion on alternative definitions and tests for membership?\n2. How would the results change in the fine-tuning regime?\n3. Could you report CIs on important results, i.e. Table 1 and Figure 3?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "001kPVApHj", "reviewer": "Reviewer_7JNu", "summary": "This work shows that current LLMs are able to generalize from their training dataset in a way that completes unseen (measured by n-gram overlap) data samples during inference time, thus challenging the use of n-gram-based metrics for a wide range of fields such as memorization, contamination, poisoning, and dataset inference. The paper approaches this from two angles: (1) it shows that even when removing all sequences of verbatim copying (memorization) from an original dataset, there remain so-called \"lingering sequences\" that are still reproduced verbatim after full re-training (and likely an artifact of very similar neighboring sequences or natural generalizations which evade filtering) - notably this is still the case for, e.g., 5-gram filters (albeit less and with noticeable trend to generalization). Second, they show that by creating a specific dataset that shares no (n-gram) overlap with a given target sequence, a model can be \"forced\" to learn the target sequence verbatim (via generalization). For this, the authors test three techniques (and their combinations): chunking, dropout, and case flip. Notably, dropout (and its combinations) highlight that such sequences can be easily forced while evading filters - even after only a few fine-tuning gradient steps. The authors also find that more capable models are more susceptible (likely due to their increased capabilities in generalizing from similar sequences). Overall, the work challenges the notion of membership definitions via n-gram or exact match commonly used in various related fields.\n\n## Update after rebuttal\n\nThe reviewer stands by their decision at the end of the reviewer rebuttal discussion and is in favor of acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s positive assessment of our work! We’re glad the reviewer finds the paper enjoyable to read, the experiments well-designed, and the claims important for the community. We hope to address the comments below and would appreciate the reviewer’s consideration.\n\n> [Weakness] …in a realistic scenario for data poisoning (e.g., 5.6.1), it would likely be caught by a variety of filters and could be interesting follow-up directions.\n\n> [Q1] … craft such sequences in a way such that they are not killed by common pre-processing techniques for model training (normalization, high-perplexity filtering)?\n\nWe appreciate the reviewer’s insightful question. In principle (recall the beginning of Sec 5), we believe that any noisy transformation of a reasonable noise distribution should work, and there should be transformations that result in “normal looking” text. For example, following **Reviewer J1BJ**’s suggestion and our response, one could consider:\n- Shuffling token ordering;\n- Multi-lingual replacement;\n- Token replacement within a restricted corpus (e.g. a fixed book, thus the perturbed sequences look like that book).\n\nWe agree with the reviewer that there can be many interesting follow-up and hope to explore additional noisy transformation strategies.\n\n> [Q2] Could such memorization-forcing sequences already be included in the pre-training and not only in finetuning - is there anything that would prevent this generalization?\n\nThe reviewer raises an interesting question. In **Appendix A.3**, we performed a costly edit-distance search of a few lingering sequences; judging from the visualizations, one could potentially interpret the near-duplicates and template sequences commonly in webtext as a form of “memorization-forcing sequences” that has no n-gram overlap. Given the vast size of pre-training data and the existence of lingering sequences (Sec 4), we believe that such generalization would happen naturally; however, strong data deduplication techniques that extend beyond n-gram overlap—such as semantic deduplication or MinHash/LSH deduplication that operates on very small values of n-gram—could potentially mitigate this at the pre-training level.\n\n> [Comments] It might make sense to mention again in the later sections that the case flip results only hold for later normalized text.\n\nThank you for the suggestion! We will mention it in the updated version.", "claims_and_evidence": "All main claims are backed up by sufficient and strong empirical evidence. In particular:\n\n- C: The threshold for n-gram-based membership in LLM training data can be gamed.\n- C: n-gram definitions have limitations in the context of data membership inference.\n- C: Removing n-gram members does not have to prevent LLM Verbatim Completion.\n- C: One can force LLM verbatim completion by adding n-gram non-members.\n\nhave direct experiments with solid ablations.", "methods_and_evaluation_criteria": "Yes, across experiments and settings, both the datasets and the models are reasonable and provide a solid empirical basis for the claims made. Ablations across various parameters are reasonable and presented in an understandable format. Used models are realistic in size (given resource constraints and evaluations on the dataset itself are thorough enough to be convincing).", "theoretical_claims": "The work did not contain any theoretical claims of which one could check the soundness.", "experimental_designs_or_analyses": "The reviewer has read all experimental setups including their additional descriptions in the Appendix. The definitions of algorithms have only been skimmed, but they are straightforward enough not to warrant any questions.", "relation_to_broader_scientific_literature": "Overall, the authors of the work have made significant efforts to contextualize their work concerning the surrounding scientific literature. While the reviewer is primarily familiar with Memorization and DI literature, the key notion in many of these fields was that verbatim memorization (of a high enough entropy sequence) is very close to a gold standard of its verbatim containment in the training set. The work starts to challenge this assumption, opening up further discussions about what it means to contain data during training and finding proof of such containment - increasingly relevant topics with larger LLM adoption.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "#### Strengths\n\n- Extremely well-written with clear structure, definitions, and presentation. The reviewer thoroughly enjoyed reading the paper.\n- The main claims of the paper are important and well substantiated with a wide range of sensible experiments showing that existing metrics can be easily circumvented.\n- Ablations are reasonable and the ablations about finding closest matches and types of lingering examples are appreciated to help the overall understanding.\n\n#### Weaknesses\n\n- Minor in the context of this work: While the proposed techniques for forcing the memorization of a specific sequence tend to work in the \"un-defended\" scenario, in a realistic scenario for data poisoning (e.g., 5.6.1), it would likely be caught by a variety of filters and could be interesting follow-up directions.\n\n##### Comments\n- It might make sense to mention again in the later sections that the case flip results only hold for later normalized text.", "other_comments_or_suggestions": "- L1340 otheriwse", "questions_for_authors": "- What ways do you see to craft such sequences in a way such that they are not killed by common pre-processing techniques for model training (normalization, high-perplexity filtering)?\n- Could such memorization-forcing sequences already be included in the pre-training and not only in finetuning  - is there anything that would prevent this generalization?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper demonstrates LLMs may generate verbatim versions of text that is not included in their training data as measured by n-gram membership tests. After rebuttal, it received mixed scores of 2445. The main concern shared by the reviewer who gave the score of 2 is that the results and contributions are limited in the current form, and it would be ideal if a new metric can be proposed. On the other hand, all the other 3 reviewers have shown strong willingness to defend this work, by commenting that (1) this paper offers important conceptual and practical contributions to the field, (2) all main claims are backed up by sufficient and strong empirical evidence, (3) the paper is very well-written and lots of intuition is provided, etc. Given all these positive feedbacks, the AC would like to recommend acceptance of the paper.
Decision: Accept (spotlight poster)