TokenSwap: A Lightweight Method to Disrupt Memorized Sequences in LLMs
Authors: Parjanya Prajakta Prashant
Kaustubh Ponkshe
Babak Salimi
Venue: NeurIPS 2025 spotlight ()
Keywords: language models
memorization
copyright
privacy
alignment
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: Inference-time approach to mitigate verbatim generation in large language models

Abstract: As language models scale, their performance improves dramatically across a wide range of tasks, but so does their tendency to memorize and regurgitate parts of their training data verbatim. This tradeoff poses serious legal, ethical, and safety concerns, especially in real-world deployments. Existing mitigation techniques, such as differential privacy or model unlearning, often require retraining or access to internal weights making them impractical for most users. In this work, we introduce TokenSwap, a lightweight, post-hoc defense designed for realistic settings where the user can only access token-level outputs. Our key insight is that while large models are necessary for high task performance, small models (e.g., DistilGPT-2) are often sufficient to assign fluent, grammatically plausible probabilities to common function words - and crucially, they memorize far less. By selectively swapping token probabilities between models, TokenSwap preserves the capabilities of large models while reducing their propensity for verbatim reproduction. Evaluations on Pythia-6.9B and Llama-3-8B show up to a 10$\times$ drop in exact memorization with negligible task degradation. Our method offers a practical, accessible solution for mitigating memorized generation in deployed LLMs. Code is available at https://github.com/parjanya20/verbatim-llm.

TOKENSWAP: A Lightweight Method to Disrupt Memorized Sequences in LLMs
Parjanya Prajakta Prashant UC San Diego
Kaustubh Ponkshe EPFL
Babak Salimi UC San Diego
Abstract
As language models scale, their performance improves dramatically across a wide range of tasks, but so does their tendency memorize and regurgitate parts of their training data verbatim. This tradeoff poses serious legal, ethical, and safety concerns, especially real-world deployments. Existing mitigation techniques, such as differential privacy model unlearning, often require retraining access to internal weights making them impractical for most users. In this work, we introduce TOKENSWAP
lightweight, post-hoc defense designed for realistic settings where the user can only access token-level outputs. Our key insight is that while large models are necessary for high task performance, small models (e.g., DistilGPT-2) are often sufficient to assign fluent, grammatically plausible probabilities to common function words and crucially, they memorize far less. By selectively swapping token probabilities between models, TOKEN WAP preserves the capabilities of large models while reducing their propensity for verbatim repro- duction. Evaluations on Pythia-6.98 and Llama-3-8B show up to a 10x drop exact memorization with negligible task degradation. Our method offers a practical, accessible solution for mitigating memorized generation in deployed LLMs. Code is available ltttt:://// :/tp::///ittie teut.hu..ooom www/paroeeeeeooooocceeee.eeeoeee wwtp./aaaaaaaa//eeerr....aa wawpaaaooeebbaimm--a
Large Model
Small Model
TokenSwap
Write story about magical world
Write story about magicall world
Write story about magical world
Mr. and Mrs. Dursley of number four, Privet...
The "Million Dollar Dollar Dollar Dollar" is
the world of Lytheria, the stars were not
High Memorization High Performance
Memorization Performance
Low Memorization High Performance
Figure 1: TOKENSWAP combines the strengths of large and small language models. Large models achieve high performance but exhibit high memorization. Small models have low memorization but poor performance (generating incoherent text). TOKENSWAP achieves both low memorization and high performance by selectively swapping token probabilities, generating novel, fluent text.
"Equal Contribution; Correspondence to Parjanya Prajakta Prashant <pprashant @ ecud, Kaustubh Ponkshe ckaustubb.ponkshe@@e..... ept..ch>>
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Large language models (LLMs) such as GPT-4, GEMINI, and LLAMA have demonstrated strong performance across wide range of tasks, from natural language understanding complex reason- ing [3 65 24). These capabilities are driven by their massive parameter counts and extensive training corpora, enabling human-level fluency and impressive reasoning across domains. Often referred to as emergent properties, such abilities arise directly from scale, with well-established scaling laws predicting performance gains. However, increased scale also introduces critical drawback: the tendency of LLMs to memorize and reproduce parts of their training data [15
48).
One of the most pressing consequences of memorization is the verbatim near-verbatim generation of training data [38 66 6]. Although memorization an inherent property and not necessarily harmful, its consequence of verbatim generation leads to plagiarism and copyright violation. This behavior poses serious risks to both model providers and end-users. Providers may face legal challenges, including copyright infringement lawsuits 38 29 5I]. while users unknowingly risk legal liability by reproducing protected content. Crucially, the threat is not limited to exact substring matches: even approximate near-verbatim outputs constitute infringement, as evidenced by lawsuits like the New York Times case against OpenAl for near-verbatim content generation [28] Moreover, verbatim generation can occur even in benign scenarios where users have no adversarial intent to extract training data [4]. Users may unknowingly generate copyrighted content during routine interactions, exposing themselves unintended legal risks [22).
Memorized
Dursley, of number four, Privet Drive, were proud
Centrn
to say that they were perfect normal, thank you very
to
8B
Next probsbilities
Novel
Centera
of their neat little house, their polished
parents
of
80M
8B
Next token probabilities
Mixed token probabiliies
Figure 2: Overview of TOKENSWAP. Our approach replaces token probabilities of high- high-frequency "grammar-bassd" tokens with those from small auxiliary language model. This mitigates memorized generation while maintaining fluency and model performance. The top path shows standard LLM generation, while the bottom path demonstrates how TOKENSWAP alters token selection disrupt memorization and produce novel text.
We consider the perspective of a typical user of commerical LLMs such as GPT-4 B, GEMINI 665 LLAMA (24, and DEEPSEEK 42). These models do not share their training data and many do not make their weights publicly available. Even in cases where weights are openly shared, hosting production-grade LLM requires substantial memory resources, rendering impractical for the average user. Consequently, it is reasonable to assume that most users can only interact with these models through APIs hosted on external servers, with access limited to model outputs such as token-level logits. Despite these practical constraints, to knowledge, none of the existing methods, whether designed to prevent memorization or mitigate verbatim output, can effectively operate under such limited access conditions.
Existing methods require access to training data and/or model weights Approaches address memorization are broadly categorized into pre-training and post-training interventions. Pre-training methods include deduplication [37], differential privacy (DP) 2 and selective token exclusion during training [31. While these approaches can reduce memorization, they often incur substantial computational costs and degrade model performance [7]. Post-training interventions focus on un- learning techniques that attempt to modify specific neurons and weights utilize finetuning methods

---

to prevent models from generating memorized content 44 remain susceptible training data extraction [60], often impair general model capabilities (34), and can lead to unintended forgetting of critical aspects such as safety guardrails [67] This challenge further complicated by theoretical findings suggesting that some degree of memorization may be inherent to achieving generalization in learning algorithms g
56 18
54
However, these methods
In contrast, another line of work focuses on preventing the generation of memorized con- tent at inference time without modifying model weights. These approaches include blocking ex-
Exact Match Rate (%) (EMR) 4
60
6.98
Standard TOKENSWAP
act matches to training data [35] or combining logits from multiple models trained on disjoint datasets []. However, these methods too re- quire access to training data or multiple LLMs trained on strictly disjoint datasets. Table sum- marizes the various approaches and assumptions under which they operate (see Appendix for comprehensive review).
2,80
40
0.1
20
410M
6.90
18
2.8
410M
160M
70M
2.5
3
3.5
4
1.5
Cross Entropy Loss (CE Loss) + Figure 3: Memorization (EMR) vs Performance (CE Loss) across different model sizes. Larger, more capa- ble models exhibit higher memorization. TOKENSWAP, with Pythia- 70M as the auxiliary model, achieves low memorization rates while maintaining competitive per- formance. Details in Section 4.2 and Section [5]
scales with size The propensity to reproduce training data consistently increases with the size of the language model [14 [T]. Since model performance generally scales posi- tively with size, users are forced into a trade-off between obtaining high performance and mit- igating memorized generations. Figure 3 demonstrates this relationship using a series of Pythia models, showing the trade-off between memorization and cross-entropy loss.
Memorization
In this work, we present TOKENSWAP, an inference-time method that significantly alleviates this tradeoff by combining large model performance with small model memorization (Figure 3}. To- KENSWAP selectively replaces the probabilities of subset of common grammar tokens (e.g.. "the". "of", "and") of the large main model with those from small auxiliary model. This technique disrupts the verbatim generation by breaking the high-probability paths that lead to verbatim reproduction. This disruption has a cascading effect: once one token deviates from the memorized sequence, all subsequent predictions are conditioned on this altered context, further preventing reproduction. Importantly, since small models reliably approximate probabilities for common grammatical tokens, TOKENS WAP preserves the large model's performance. For auxiliary models of size much smaller than the main model, this provides verbatim memorization mitigation method which requires access neither to the training data nor the model weights. Since we treat the effect of memorization, and not the cause itself, our method is able to reduce verbatim generation at inference time.
We extensively evaluate TOKENSWAP through both controlled experiments and real-world deploy- ments. In controlled fine-tuning experiments (Section 4.1). TOKENSWAP achieves 50-800x reduction in verbatim generation compared to undefended models. Evaluations on commercial-gran models such as Pythia-6.9b and Llama-3-8b (Section 4.2) demonstrate reductions in verbatim genera- tion by upto 10x, without compromising downstream task performance. Furthermore, comparisons with Goldfish [1 show that TOKENSWAP matches or surpasses the effectiveness of state-of-the-art pre-training methods (Section 4.3
2 Preliminaries
2.1 Language Models:
Notation and Setup
We consider auto regresegres language models that model the log bogobbbbabylity of a token conditioned on all previous tokens in a sequence. They operate over vocabulary V {ei..... . زlyee of typically |V] 10. 1 tokens. Given an input prompt (I-it)..... xI E v4, of length 4, followed by response sequence (ro...., e V of length I, an ruto-regeessive language model parametrizes the joint probability by factorizing over conditional probabilities:
3

---

Table 1: Comparison of TOKENSWAP with existing methods based on their assumptions. To- KENSWAP uniquely avoids requiring access to model weights the copyrighted training corpus. While it employs an auxiliary model, the memory overhead is small R 1% due to the small size of the auxiliary model. PT: pre-training, UL:: unlearning, FT: fine-tuning, Inf: inference-time.
Model Access
Copyrighted Corpus Access
Inference Overhead
Deduplication
37
PT
weights
X
Goldfish [1
PT
weights
X
Balanced subnet
[6
UL
weights
X
Obliviate
54
FT
weights
X
MemFree
35
Inf
logits
efficient querying twice standard generation small auxiliary model
CP-Fuse
1|
Inf
logits
TOKENS WAP Inf
logits
X
p(ro.-... 32 11 x-1) = II p(as (x))) 1=
(1)
For each position the model outputs distribution pi[u] over V, where pu[u] p(z; v|z=s).
Since language models are trained maximize the likelihood of observed sequences, they tend to assign high probabilities to tokens that frequently follow specific prefixes during training. This increases the risk of memorization and verbatim reproduction of training data.
2.2 Extractable Memorization
Memorization in language models manifest in various ways, but practically relevant and widely adopted framework is extractable memorization [I5] [4]. Carlini et al. [IS) demonstrate that models can be induced to regurgitate training sequences when prompted with prefixes from their training data. The following definition formalizes this concept:
sequence x = (zo...., xt-1) of length con- sidered extractable with lp tokens of context from language model p if there exists prefix of length lp such that [x_ | appears the training data of p, and p reproduces æ via greedy decoding.
Definition (Extractable Memorization).
x_ = (
Formally, for each E [ 1}:
arg max p(a' x-),
z'EV
This definition is practically useful because: (1) it aligns with real-world risks of copyright and memorized generation [47 38). (2) it provides a concrete, testable condition that can be evaluated real models, and (3) it extends models of different sizes, capturing the well-documented trend that larger models memorize more data [14 This scaling behavior is important in motivating our
methodology in Section
Methodology
As discussed earlier, small language models (e.g., DistilGPT-2,, Pythia-70M) have lower propensity to reproduce training data compared to large models (e.g., Llama3, GPT-4). We introduce TOKENSWAP, lightweight, post-hoc method that combines the strengths of both model scales: large-mode perfor- mance with small-model memorization. During inference, TOKENSWAP replaces the probabilities for selected tokens of large model with those small model.
Algorithm Let p main and " denote the probability distributions of the main and auxiliary models respectively, where mainn (xx. xrcce) and pa (æx w<))) represent their token probabilities conditioned on previous tokens. We assume the parameter count of the main model significantly exceeds that of the auxiliary model. Given these models, TOKENSWAP selectively replaces probabilities for fixed subset of tokens G C. V. The complete procedure is formalized in Algorithm

---

At each position i, TOKENSWAP queries both mm and pank to obtain probability distributions conditioned on the current context Z<z.. For tokens in subset C V. probabilities from the main model are replaced with scaled probabilities from the auxiliary model, with scaling factor ensuring the final distribution p pfinal remains valid distribution. This prevents reproduction of memorized sequences: if any token in memorized sequence belongs to g, its probability under pfinal is determined by the auxiliary model. Since the auxiliary model memorizes less, this disrupts the chain of conditional probabilities required for verbatim generation of most sequences. Importantly, for tokens $ G, their probabilities remain unchanged, i.e., p. [v] = p. mainn
Algorithm 1 TOKENSWAP
Require: Main model p muin auxiliary model pmnx, token subset G. prompt Z<0
1: for = 0, 1.... do
2: p pmain f p main (+(2<)) (Get main model probabilities}
3: p.ux puux (- (ee)) (Get auxiliary model probabilities)
Eeep p
4:
[Compute normalization)
Zvcg [
5:
for U E V do
p ppmain [v].
if G
6:
pi [u]
P
[o],
7: end for
8: 3y ~ inall (Sample next token]
9:
end for
Selecting G for Effective Memorization Disruption The choice of affects both memorization and model performance. By modifying token probabilities, TOKENSWAP disrupts memorized sequences while preserving fluency. However, not all tokens are equally effective for this purpose. G should consist of tokens that frequently appear in memorized text, as replacing their probabilities reduces the likelihood of exact reproduction. At the same time, modifying inappropriate tokens can degrade model performance, especially for specialized tasks. For instance, if G includes numeric tokens, mathematical reasoning may degrade. Therefore, G should satisfy two key criteria. First, it must contain frequently occurring tokens. Second, it should avoid tokens where probability replacement impacts the model's capabilities.
Empirical studies suggest that small models correctly approximate the probabilities of high- frequency function words while diverging more language models ( 100M) can generate coherent and grammatically correct text (25)- Based on these insights, Further, since G consists high-frequency words, there exists a natural one-to-one mapping between tokens even when p main and pax use different tokenizers and vocabularies. While this approach is well-suited for natural language, structured domains such as code may require domain-specific adaptations. Additional details on the construction of G provided in Appendix C.2 and C.3
domain-speent specific terms [52] Additionally. small
construct from grammar-based high-frequence tokens (e.g. "the", 'in').
4 Experiments
In this section, we demonstrate the effectiveness of TOKENSWAP. in both controlled and real-world settings. Our experiments evaluate TOKENSWAP along two dimensions:
The method's 's efficacy in preventing exact and approximate reproduction of training data.
The impact on model performance across common-sense reasoning, language and fluency.
We evaluate TOKENSWAP across three settings to demonstrate its effectiveness. Section 4. we deliberately induce memorization through extensive fine-tuning on small datasets to stress-test our defense. Section 4.2 evaluates TOKENSWAP on production-grade models including Pythia- 6.9B and Llama-3-8B. Finally, in Section 4. we compare against Goldfish (31] pre-training method specifically designed to reduce memorization, showing that our post-hoc approach achieves comparable results without requiring model retraining.
5

---

4.1 Extreme Memorization
In order to rigorously evaluate TOKENSWAP, we create an extreme test case by deliberately inducing memorization through extensive fine-tuning. While TOKENSWAP can be applied to real-world models directly, our baselines require specific experimental conditions for comparison. Similar extreme test cases have been generated to evaluate memorization in prior work [1 Following Abad et al. [ we fine-tune a Llama-3.2-3B model [24] on 2.000-sequence subsets from two datasets: MathAbstracts [69] and WritingStories 27 We train for 50 epochs to deliberately amplify memorization beyond typical levels.
Memorization Metrics Our analysis employs both exact and approximate memorization and performance metrics to ensure a comprehensive assessment. Exact memorization is measured through Matching Length (ML), which the number of verbatim characters tokens generated before first deviation, and Exact Matching Rate (EMR), which computes the fraction of sequences reproduced verbatim. To capture partial memorization, we use the ROUGE-L score, which identifies the longest common non-contiguous subsequence and gives a score between O and I, and the Normalized Levenshtein Distance, which quantifies the minimum number of edits needed to transform generated text into the original sequence. Lower scores indicate reduced memorization for Matching Length, Exact Matching Rate, and ROUGE-L. Higher scores are better for Normalized Levenshtein Distance. These metrics are widely used to evaluate verbatim and approximately verbatim generation 38 31
Since our setup intentionally induces extreme memorization, standard performance metrics are not meaningful. Nonetheless, we report cross-entropy loss on a held-out validation set Appendix B.2
Performance Metrics
Setup and Inference-time Baselines We compare against the two inference-tine baselines: CP- Fuse [1) which samples from weighted combinations of models trained on disjoint datasets, and MemFree [35], which blocks exact n-gram matches to the training data. Standard refers greedy decoding without any memorization mitigation. Both baselines rely on unrealistic assumptions- MemFree requires access to the training data, while CP-Fuse assumes access to two separately trained models on disjoint corpora. To assess CP-Fuse under more realistic conditions, we evaluate two variants: CP-FUSE HALF, with perfectly disjoint sets of 1,000 sequences each, and CP-FUSE MIXTURE, with 1,500 sequences per model and 500 overlapping. For ToKENSWAP we employ DistilGPT-22 (80M) (57] as paux We construct G with |g] 110 tokens using high-frequency crummar-based words. Additional details on G are provided in Appendix C.2] For all experiments and methods, a prefix of 20 tokens is used and the next 128 tokens are greedily sampled.
Table 2: Comparison of memorization mitigation methods for WritingPrompts and MathAbstracts datasets. Memorization metrics: Matching Length (ML). Exact Match Rate (EMR). Normalized Levenshtein Distance (Levenshtein), ROUGE-L. Models used: Finetuned Llama-3.2-3B.
WritingPrompts
Mastaass
Method
ML.
EMR,
ROUGE-I4
Lev.
MI
EMR.
ROUGE-I.
Lev.f
Standard
464.0
83,4
0.89
0.10
450.4
93.6
0.98
0.03
MenFree
17.4
0.0
0.29
0.63
6.7
0.0
0.44
0.55
CP-Fuse-mix
280.3
49.2
0.58
0.37
233.7
47.1
0.62
0.36
CP-Fusc-half
12.5
0.0
0.17
0.73
15.3
0.1
0.26
0.71
TOKENSWAP
19.7
0.1
0.19
0.71
53.0
1.8
0.38
0.60
Results Table 2 demonstrates TOKENSWAP's effectiveness in reducing memorization across both datasets. For WritingPrompts, TOKENSWAP WAP reduces EMR by 800x (from 83.4% 0.1%) and ROUGE-L by 4.6x (from 0.89 to 0. 09) compared to standard generation. On MathAbstracte EMR decreases by 50x (from 93.6% 1.8%) and ROUGE-L by 2.6x (from 0.98 to 0.38). CP-Fuse-har Fule achieves slightly better results but requires disjoint training sets, while CP-Fuse-mi performs signifi- cantly worse due to dataset overlap. MemFree achieves the lowest scores on exact memorization metrics (Exact Matching Rate and Matching Length) but performs poorly on approximate memo- rization metrics (ROUGE-L and Levenshtein). This shows that, while MemFree prevents verbatim generation, it still allows high levels of near-verbatee verbatim generation. The performance gap between

---

Prefix: Magic once real and potent force but as the world population
Original Suffix
Standard Generation
TOKENSWAP Generation
grew from millions to bil- lions the shared mana per
grew from millions to bil- lions the shared mana per person is now negligible. group of astronauts help- lessly watching Earth perish experience...
grew and the number of wiz- ards and witches declined, the world began to suffer. Now the world suffers from lack of magic, and the gov- ermment tasked with...
person is now negligible. group of astronauts help- lessly watching the Earth perish experience...
Figure 4: Comparison of text generation methods. Red text indicates memorized content. Standard generation reproduces the entire suffix verbatim, while TOKENSWAP generates novel content.
WritingPrompts (EMR: 0.1%) and MathAbstracts (EMR: 1.8%) aligns with our intuition 9 was designed focusing on natural language tasks. Nevertheless, TOKENS WAP achieves substantial memo- rization reduction for both domains. To complement our quantitative results, provide qualitative examples of generations from the WritingPrompis dataset Figure and Appendix
4.2 Memorization in the wild
In this section, we demonstrate the efficacy of our approach on production-grade models. We assess the effectiveness of TOKENSWAP on two pre-trained models: Pythia-6.9B 12] and Llama-3-8B 24]
Pile-Memorized Dataset For Pythia-6.9B. we evaluate on memorized sequences identified by Chang et al. [16] from the Pile dataset, consisting of 32-token prefixes and 48-token suffixes. After filtering to retain only natural language content (excluding code, URLs, etc.), we obtain 184 evaluation examples.
LeetCode Dataset For Llama-3-8P 3-BB, following previous work demonstrating LeetCode problem memorization (38], we evaluate on 1,825 LeetCode problem statements (30). These problem state- ments are written in natural language. Since the exact format of LeetCode problems in Llama's ' training data is unknown, we remove punetuation while calculating the memorization metrics. Ad- ditionally, instead of exact match rate, use ROUGE-L > 0.8 as our threshold for identifying memorized content. Prefix length of tokens is used and the next 100 tokens are sampled.
Evaluation Setup We face two key limitations when comparing TOKENSWAP with existing baselines. CP-Fuse requires models trained on disjoint datasets, but verifying this is difficult since LLMs do not release training data. Even when available, disjoint datasets are unlikely given that most models train on overlapping web corpora like Common Crawl. Additionally, CP-Fuse requires identical tokenizers, limiting comparisons to models within the same family. Similarly, we cannot evaluate against MemFree due to unavailable training data (LLaMA) or prohibitively large datasets (Pythia uses the 800GB Pile 12 ). To ensure fair evaluation for CP-Fuse, we paired each model with smaller counterpart: Pythia-2.8D B.B with Pythia-6.9B. and Llama-3.2-30 with Llama-3-8B. Using smaller models actually favors CP-Fuse since they memorize less. We avoid very small models (<100M) as CP-Fuse needs roughly equally capable models (see Appendix C.I.4). The setup for TOKENS WAP follows Section 4.1 For LeetCode evaluation, we use both Disti1GPT-2 and SmolLM- 135M [5] as auxiliary models. SmolLM is instruction-tuned model, which enables evaluation on instruction folllwiin tasks like MT-Bench where an instruct-capable auxiliary model required. For memorization,
the same metrics as Section 4.1
We evaluate two key aspects: task performance and generation quality. For task performance, we assess five-shot learning on multiple commonsense reasoning benchmarks: BoolQ 19 SIQA [58) PIQA [3], ARC-Challenge 20 ARC-Easy (20] OBQA (45) and Wino- Grande [55 For generation quality, we report cross-entropy loss on samples from Slimpajama [6]],
Performance Metrics
which correlates with fluency [10] and has been used to evaluate prior memorization mitigation

---

We also evaluate on MT--Benc [70], which tests multi-turn conversation, instruction- following, and generation quality through realistic conversational scenarios. Note that MT-Bench and commonsense reasoning results are only reported for Llama-3-88 (LeetCode Dataset) since these require instruction-folle followinn capabilities not available the base Pythia models.
work [
Table 3: Comparison of mitigation methods for LectCode and Pile-Memorized datasets. Memorization metrics: Matching Length (ML), Exact Match Rate (EMR), Normalized Levenshtein Distance (Levenshtein), ROUGE-L Performance metries: Cross Entropy Loss (CE Loss) on SlimPajama, MTnBeecch with GPT-4 judge, Mcan scores on Commonsense Reasoning benchmarks. Models used: Llama-3-8B and Pythia-6.9B.
LeetCode Dataset (Llama)
Method
ML LROUGE-L> 0.8 |ROUGE-L Lev. 1/CE Loss LMMT-Bench 1CCommoneens
Standard
24.57
9.65
0.39
0.60
2.38
7.75
71.87
CP-Fuse
19.44
7.01
0.37
0.61
2.45
8.53
70.18
TOKENSWAP
6.04
0.96
0.27
0.71
2.52
71.87
TOKENSWAP
8.58
1.92
0.30
0.69
2.43
7.78
Pile-Memorized Dataset (Pythia)
Method
MI.
EMR +
ROUGE-I.
Lev. ?
CE Loss 4
Standard
151.6
65.22
0.80
0.18
2.80
CP-Fuse
97.05
29.35
0.62
0.35
2.81
TOKENSWAP"
35.10
0.38
0.56
2.88
DDstillGPT-2 as auxiliary model. MSm50LMM-135M auxiliary model.
Results Table demonstrates that TOKENSWAP substantially reduces memorization across both datasets compared to standard generation and CP-Fuse. Exact match rate decreases by over 10x compared standard generation and 5-7x compared CP-Fuse both datasets. The average matching length shows similar improvements, reducing by 4-5x versus standard and 3-4x versus CP-Fuse. The consistent improvements in approximate memorization metrics (ROUGE-L and Levenshtein distance) demonstrate that TOKENS WAP robustly prevents verbatim generation rather than simply introducing small perturbations. CP-Fuse shows limited effectiveness in these real- world scenarios primarily because its core assumption of disjoint training datasets does not hold. Even when using different models, the inherent overlap web-scale training corpora prevents CP-Fuse from effectively disrupting memorized sequences.
TOKENS WAP maintains task performance by seleetively targeting only grammar-bases tokens, leaving reasoning-critical content words unchanged. This preserves commonsense reasoning abilities, shown by identical accuracy scores (71.87%) compared standard generation. The method also maintains fluency, evidenced by minimal cross-entropy increases and nearly equal MT-Bench scores. While CP-Fuse achieves better conversational performance (8.53 vs 7.78), it fails to verbatim generation, making it unsuitable for the desired goal.
Evaluation on OLMo-2-13B. To further validate TOKENSWAP on fully open model with known training data, we evaluate TOKENSWAP on MLMo-2-13B 49 The full training corpus contains 3T tokens, making exhaustive memorization search infeasible. We therefore focus on the Wikipedia subset of the training data. Results in Appendix B. sho that TOKENSWAP eliminates exact verbatim generation (EMR=0) and substantially reduces approximate memorization.
4.3 Comparison with Pre-training Methods
While previous sections demonstrate that TOKENSWAP outperforms post-hoc baselines, we also compare with Goldfish [31] pre-training approach that reduces memorization by excluding fraction 1/k of tokens from loss computation during training. Since pre-training large models using this loss expensive, we evaluate on pre-trained goldfish models from Hans et 31 These models were trained on subset of RedPajama [68 combined with 2000 Wikipedia sequences. To induce memorization, the Wikipedia sequences were duplicated times during training. We compare against models trained with & E (3, 4, 4,3.). For TOKEN WAP,, we maintain the same experimental

---

setup from Section 4.1 Following Hans et al. [1], we use identical prefix and suffix lengths for extraction memorized sequences.
Table 4: Comparison of TOKENSWAP with Goldfish 31 for E (3,4,32). Memorization metries: Match-
ing Length (ML), Exact Match Rate (EMR), Normalized Levenshtein Distance (Levenshtein), ROUGE-L Performance metrics: Cross Entropy Loss (CE Loss) on SlimPajama.
Method
ML1
EMRJ
ROUGE-LI LL
Levenshteinf
CE Loss
Standard
73.9
7.8
0.38
0.58
3,44
Goldfish (k=3)
12.7
0.0
0.23
0.72
3.54
Goldfish (k=4)
14.7
0.0
0.23
0.71
3.50
Goldfish (A=32)
58.1
2.5
0.35
0.60
3.44
ToKENSWAP
12.4
0.1
0.22
0.72
3.44
TOKENSWAP Goldfish (I=3)
7.9
0.0
0.21
0.73
3.57
Results Table shows TOKENSWAP achieves comparable or superior performance to Goldfish across all memorization metrics. Notably, TOKENSWAP obtains the best Matching Length, Rouge- L and Normalized Levenshtein distance scores while maintaining better cross-entropy than the Goldfish variants for k 3, The effectiveness of Goldfish varies with parameter k smaller values (more aggressive token exclusion) yield stronger memorization reduction but worse performance, as evidenced by higher cross-entropy. This illustrates key advantage of TOKENSWAPP we achieve similar memorization reduction without requiring modified training or reduced training data tokens. Figure [5] (Appendix B.7 further supports this finding, showing nearly identical ROUGE-L score distributions between TOKENSWAP and Goldfish (I=3),, indicating that our post-hoc approach matches the most aggressive pre-training variant. Furthermore, applying TOKENSWAP to Goldfish (k = 3) as the main model reduces memorization more than either method alone, demonstrating that our approach is orthogonal to pre-training methods and can enhance existing techniques.
5 Discussion and Limitations
In this section, analyze TOKENSWAP's behavior across different settings. We first perform ablations on the auxiliary model choice and the size of G. We then analyze TOKENSWAP across the Pythia model family demonstrate significant improvements in the performance-memorizztion tradeoff. Finally, we discuss limitations and potential extensions of our method.
Choice of the Auxiliary Model In Section[4wwa test TOKENSWAP with DistilGPT-22 as the auxiliary
model. natural question arises: What auxiliary model should one choose and how does the size of the auxiliary model affect memorized generation? To answer this, we use the SmoILM family [5] with three sizes (135M, 360M, 1.7B) and evaluate on both Pythia-6.9B (Pile-memorized dataset) and Llama-3-88 (LeetCode dataset). Detailed results are in Table 10 (Appendix B.6).
We observe clear trend: smaller auxiliary models lead to less verbatim generation, confirming our hypothesis that TOKENSWAP's effectiveness stems from low memorization auxiliary models. Importantly, auxiliary model size has minimal impact on performance. MT-Bench scores show negligible variation across auxiliary models-thin particularly significant since MT-Bench evaluates overall sequence generation quality, unlike cross-entropy loss which measures token-level accuracy. Therefore, any small model (Rs 100M) which can generate fluent text and predict grammar-bai based tokens well, such DistilGPT-2 SmoILM-135M, can be used effectively an auxiliary model.
Ablations on The subset of tokens & is constructed by selecting grammar-based words from the top 500 most frequent English words, resulting in |g] = 110 (details in Appendix C.2 To understand the impact G size on memorization reduction, we ablate by constructing G from the top most frequent words for & E {10, 50, 100, 500, 2500}, yielding [g] E {9, 43, 66, 110, 136]. We evaluate on the Pile-memorized task using Pythia-6.9B as the main model and Pythia-70M as auxiliary (see Appendix [.5 for complete results). We observe clear trend: decreases significantly. For example, EMR drops from 22.28% (g) = 9) 8.15% ((9) = 136).
[ increases, memorization
This makes intuitive laree enables the auxiliary model to disrupt memorized sequences

---

more frequently. The cross-entropy loss remains largely stable, indicating minimal performance degradation with increase (g].
Perfornance-Memoriziiin Tradeoff We analyze how TOKENSWAP affects the tradeoff between model performance and memorization across seven Pythia models (70M 6.9B parameters), using Pythia-70M as the auxiliary model. shows exact match rate (EMR) versus cross-entropy loss-lower values are better for both metrics. Standard generation faces a severe tradeoff: reducing memorization from 45% to 6% EMR costs 0.7 points in cross-entropy (2.85 - 3.55). TOKENSWAP considerably improves this tradeoff. At similar performance levels (cross-entropy s 2.87). To- KENSWAP achieves 8.7% EMR versus 45% for standard models an 8x memorization reduction. Even when targeting very low memorization (6% EMR), TOKENSWAP maintains cross-entropy at 3.07, significantly outperforming standard models at equivalent memorization levels.
Limitations and Future work One limitation of our work that in the rare cases where the small auxiliary model memorizes sequence, our approach will preserve that memorization. However, in practice, small auxiliary models (+s 100M parameters) memorize very little, and we empirically match outperform existing baselines without requiring access to training data restrictive assump- tions like disjoint datasets. Additionally, while pre-training or unlearning mitigation methods are impractical for large models, they can be applied small models since these often open-source with accessible training data. Therefore, we expect future development in small models with low memorization. This makes our work methods to reduce memorization in small models can be immediately extended to large models using TOKENSWAP. Second, our current implementation focuses on natural language tasks. A promising direction for future work is extending TOKENSWAP to other domains such as code generation.
significant: any advance in pre-training unlearning
6 Conclusion
TOKENSWAP offers several key advantages for mitigating memorized generation in language models: it operates without requiring access to model weights or training data, and makes no assumptions about the underlying training distribution. Our experiments demonstrate 10-800x reductions in verbatim generation, matching or exceeding baselines that assume access to training data, disjoint models, or require pre-training their own models. Importantly, this comes at minimal cost to model performance. TOKENSWAL WAP maintains performance commonsense reasoning tasks, and our MT- Bench evaluation shows that it preserves fluency, cetructtio-followiini and conversational abilitics. This makes TOKENSWAP a practical solution for both providers and users of LLMs.
Acknowledgments and Disclosure of Funding
The authors would like to thank the anonymous reviewers and A. Feder Cooper for their valuable feedback and comments. We would also like acknowledge the support from NSF awards IIS- 2340124, IIS-2420691, and IIS-2420577, and NIH grant U54HG012510.
10

---

References
[1] Javier Abad, Konstantin Donhauser, Francesco Pinto, and Fanny Yang. Copyright-protected language generation via adaptive model fusion. arXiv preprint (xXiv:4422006119 2024.
[2] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference computer and communications security, pages 308-3 318, 2016.
[3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[4] Michael Aerni, Javier Rando, Edoardo Debenedetti, Nicholas Carlini, Daphne Ippolito, and Florian Tramèr. Measuring non-adversarial reproduction of training data in large language models. arXiv preprint arXiv:241 1. 10242, 2024.
[5] Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, al. Smollm2: When smol goes big big--ata-centric training of a small language model. arXiv preprint arXiv:2502.02737, 2025.
[6] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint arXiv:2404.05405 2024.
[7] Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differentially private bert. arXiv preprint arXiv:2 01644, 2021.
[8] Idan Attias, Gintare Karolina Dziugaite, Mahdi Haghifam, Roi Livni, and Daniel M Roy. Information complexity of stochastic convex optimization: Applications to generalization and memorization. arXiv preprint 24Xiv:240..09327, 2024.
[9] George-Octavee Octavian Barbulescu and Peter Triantafillou. To each (textual sequence) its own: Im- proving memorized-data unlearning in large language models. arXiv preprint (ii::2405.03097, 2024.
[10] Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R Varshney. Mirostat: neural text decoding algorithm that directly controls perplexity, arXiv preprint (0t.:20007.14966 2020.
[11] Stella Biderman, Usvsn Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems, 36:28072-28099, 2023.
[12] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pages 2397-2430. PMLR, 2023.
[13] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piga: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432 22297739,00 2020.
[14] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2022.
[15] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2021.

---

[16] Ting- Yun Chang, Jesse Thomason, and Robin Jia. Do localization methods actually localize memorized data in llms? a tale of two benchmarks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pages 3190-3211, 2024.
[17] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, -aan-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXivv23302.113: 01,11 2023.
[18] Tong Chen, Faeze Brahman, Jiacheng Liu, Niloofar Mireshghallah, Weijia Shi, Pang Wei Koh, Luke Zettlemoyer, and Hannaneh Hajishirzi. Parapo: Aligning language models reduce verbatim reproduction of pre-training data. arXiv preprint arXiv:2504. 1..00..44455 14452, 2025.
[19] Christopher Clark, Kenton Lee, Ming- Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprini arXiv: 10044, 2019.
[20] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.
[21] A Feder Cooper and James Grimmelmann. The files in the computer: copyright, memorization, and generative ai. Chi.-Kent L. Rev. 100:141, 2025.
[22] Amy B Cyphert. Generative ai, plagiarism, and copyright infringement in legal documents. Minn. JL Sci. Tech., 25:49, 2023.
[23] Mark Davies. The corpus of contemporary american english as the first reliable monitor corpus of english. Literary and linguistic computing, 25(4):447-464, 2010.
[24] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pages arXiv-2407, 2024.
[25] Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in Ilms. arXiv preprint arXiv:2310.02233, 2023.
[26] Niva Elkin-Koren Uri Hacohen, Roi Livni, and Shay Moran. Can copyright be reduced to privacy? arXiv preprint arXiv:2305. /4822, 2023.
[27] Angela Fan, Mike Lewis, and Yann Dauphin. Hicrarchical neural story generation. arXiv preprint 1805. 2018
[28] Joshua Freeman, Chloe Rippe, Edoardo Debenedetti, and Maksym Andriushchenko. Exploring memorization and copyright violation in frontier llms: A study of the new york times v. openai 2023 lawsuit. arXiv preprinl arXiv:2412.06370 2024.
[29] Michael M Grynbaum and Ryan Mac. The times sues openai and microsoft over ai use of copyrighted work. The New York Times, 27(1), 2023.
[30] gzipChrist. Leetcode problem dataset, 2021.
[31] Abhimanyu Hans, John Kirchenbauer, Yuxin Wen, Neel Jain, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele, et al. Be like goldfish, don't memorize! mitigating memorization in generative llms. Advances in Neural Information Processing Systems, 57:24422-22044 2024.
[32] Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. Sok: Memorization in purppse large language models, 2023.
[33] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. LIm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 2254-5276, 2023.
12

---

[34] Jing Huang, Diyi Yang. and Christopher Potts. Demystifying verbatim memorization in large language models, 2024.
[35] Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo and Nicholas Carlini. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:22 17546, 2022.
[36] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprini arXiv:2210.01504, 2022.
[37] Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pages 10697- 10707, PMLR, 2022.
[38] Antonia Karamolegkou, Jiaang Li, Zhou, and Anders Søgaard. Copyright violations and large language models. arXiv preprint arXiv:2310. 13771, 2023.
[39] Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. Advances in Neural Information Processing Systems, 36:39236-39256, 2023.
[40] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference Machine Learning, pages 19274-19286. PMLR, 2023.
[41] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori B Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Proceedings of the 61st anmual meeting of the association for computational linguistics (volume 1: Long papers), pages 12286-12312, 2023.
[42] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint WIXv:22412.19437 2024.
[43] Edward Loper and Steven Bird. Nltk: The natural language toolkit. arXiv preprint cs/0205028, 2002.
[44] Pratyush Maini, Michael Mozer, Hanic Sedghi. Zachary C Lipton, J Zico Kolter, and Chiyuan Zhang. Can neural network memorization be localized? arXiv preprint arXiv:2307.0955 09542. 2023.
[45] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv:1809.000 2018.
[46] Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang. David Evans, and Taylor Berg- Kirkpatrick. Memorization in nlp fine-tuning methods. arXiv preprint arXiv:2205. 12506, 2022.
[47] Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ip- polito, Christopher A Choquette-Choo Eric Wallace, Florian Tramèr, and Katherine Lec. Scalable extraction of training data from (production) language models. arXiv preprint 1.Xi::3311.10335 2023.
[48] Milad Nasr, Javier Rando, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo Florian Tramèr, and Katherine Lee. Scalable extraction of training data from aligned, production language models. In The Thirteenth International Conference on Learning Representations, 2025.
[49] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bha- gia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha
13

---

Dziri, Allyson Ettinger, Michal Guerquin. David Heineman, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Jake Poznanski, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo 2 furious, 2025.
[50] Mustafa Ozdayi, Charith Peris, Jack FitzGerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, and Rahul Gupta. Controlling the extraction of memorized data from large language models via prompt-tuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1512-1521, 2023,
[51] Aklovya Panwar. Generative ai and copyright issues globally: Ani media v openai. Tech Policy Press. Noudettu, pages 27-03, 2025.
[52] Andrea Pinto, Tomer Galanti, and Randall Balestriero. The fair language model paradox. arXiv preprint arXiv:2410./1985 2024.
[53] Francesco Pinto, Nathalie Rauschmayr, Florian Tramèr, Philip Torr, and Federico Tombari.
Extracting training data from document- vqa models. arXiv preprint arXiv:440.008707, 2024.
[54] Mark Russinovich and Ahmed Salem. Obliviate: Efficient unmemorization for protecting intellectual property in large language models. arXiv preprint arXiv:2502. 2025.
[55] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106. 2021.
[56] Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Nathaniel Hudson, Caleb Geniesse, Kyle Chard, Yaoqing Yang. Ian Foster, and Michael W Mahoney. Mitigating memorization in language models. arXiv preprint urXiv:2410.02159, 2024.
[57] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01 108, 2019.
[58] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiga: Com- monsense reasoning about social interactions. arXiv preprint arXiv: /904:/994.0000 09728, 2019.
[59] Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary Lipton, and J Zico Kolter. Rethinking IIm memorization through the lens of adversarial compression. Advances in Neural Information Processing Systems, 37:56244-5628
2024.
[60] Ilia Shumailov, Jamie Hayes, Eleni Triantafillou, Guillermo Ortiz-Jimenez, Nicolas Paper- not, Matthew Jagielski, Itay Yona, Heidi Howard, and Eugene Bagdasaryan. Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai. arXiv preprint arXiv:2407.00106 2024.
[61] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. Slimpajama: A 627b token cleaned and deduplicated version of redpajama. Blog post, 2023.
[62] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: long papers), pages 15725-15788. 2024.
[63] Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018
[64] Manan Suri, Nishit Anand, and Amisha Bhaskar. Mitigating memorization in llms using activation steering, arXiv preprint (ale:225330000400 2025.
14

---

[65] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican. al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
[66] Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274-382999 2022.
[67] Yifan Wang, Runjin Chen, Bolian Li, David Cho, Yihe Deng, Ruqi Zhang, Tianlong Chen, Zhangyang Wang, Ananth Grama, and Junyuan Hong. More is less: The pitfalls of multi-model synthetic preference data in dpo safety alignment. arXiv preprini arXiv:2504.02195, 2025.
[68] Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: an open dataset for training large language models. Advances in neural information processing systems, 37:1 16461-11611111 16492, 2024.
[69] Yifan Zhang, Yifan Luo, Yang Yuan, and Andrew C Yao. Autonomous data selection with language models for mathematical texts. ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024.
[70] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,
Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 066466955-46222 2023.
[71] Zhenhong Zhou, Jiuyang Xiang, Chaomeng Chen, and Sen Su. Quantifying and analyzing entity-level memorization in large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19741-19749, 2024.
15

---

A Related Work
Memorization LLMs LLMs have been shown to memorize and potentially reproduce copy- righted information from their training data [15
38 32). This is demonstrated through prefix attacks, where models prompted with training data prefixss generate their memorized completions. Shwarzschild et al. [59 formalize this notion based adversarial compression, requiring that any memorized sequence must be longer than the prefix used to elicit it. Zhou et al. [71] and Nasr et al. 47 demonstrate that large-scale training data be extracted without access training prefixes. Aerni et al. [4] show that models may regurgitate training data even under benign or non-adverserial prompting. Studies further indicate correlation between model scale and memorization, with larger models regurgitating higher proportions of their training data 14
Pre-training Several training-time strategies reduce memorization and verbatim generation, but often at the cost of accessibility or performance. De-duplication [37] is limited by pervasive near- duplicates in large-scale corpora. Differential Privacy (DP) 2 offers formal guarantees, but degrades performance and is computationally costly [6].. Other methods such as token masking [31] and early stopping [6 (53] show some promise but remain expensive, degrade model performance and are unavailable to end users.
Unlearning and Finetuning Post-training approaches offer alternative strategies to reduce mem- orization. Unlearning methods [44 36 56 modify internal weights linked to memorized content.
Others remove sequences via gradient ascent [9) steer activations away from remorization-correlaee
directions [64], or fine-tune with losses discouraging verbatim recall [54 18). However, these methods
require access to model internals and often degrade utility [34]
18.
Inference time The two methods most relevant to work are MemFree 35 and CP-Fuse M These methods operate during generation and do not assume access to model internals. MemFree filters next-token outputs block n-gram matches from the training set. It requires access to the full training corpus, often unavailable or prohibitively large for end users. Further, MemFrec often degrades fluency by introducing unnatural punctuation [1] CP-Fuse combines logits from two LLMs trained on disjoint corpora. This is rarely practical since most production-gradt LLMs are trained on internet-scale data. Also, CP-Fuse requires the tokenizers of the models to be the same. In contrast, our method can mitigate memorization in real-world models trained on internet-scale data.
Speculative decoding Speculative decoding approaches accelerate inference by generating can- didate tokens from small draft model, which are selectively accepted by the large model [41 These methods preserve the model distribution and do aim mitigate verbatim generation. Further, if all candidates are rejected, the tokens are generated by the large model. In contrast, TOKENSWAP modifies the large model's 's distribution to reduce verbatim generation. In Appendix B.9. we show that speculative decoding fails to mitigate verbatim generation.
63
39
40
B Additional Experiments
B.1 OLMo-2-13B Evaluation
We evaluate TOKENSWAP OLMo-2-13B (49, which is trained on the open Dolma dataset [62 Since Dolma contains trillion tokens making exhaustive memorization search impractical, we focus on Wikipedia as known subset. We sample 5000 random Wikipedia sequences and prompt OLMo-2-13B with 50 token prefixes to generate the next 50 tokens. We select those with ROUGE-L 0.9 with the ground-truth to identify sequences memorized verbatim near-verbatim. For CP-Fuse, we use OLMo-2-78 as the second model.
Table reports the results. TOKENSWAP completely eliminates exact verbatim generation and reduces caaproximate-verbalie generation significantly (ROUGE-L:: 0.38 vs 0.95, Levenstein 0.60 vs 0.07).
16

---

Table Memorization on the Wikipedia subset of the Dolma corpus for OLMo-2-13B. Memorization metrics:
Matching Length (ML), Exact Match Rate (EMR), Normalized Levenshtein Distance (Levenshtein), ROUGE-L.
Method
ML4
EMR
ROUGE-L
Lev.1
Standard
111.0
27.3
0.95
0.07
CP-Fuse
65.6
10.9
0.66
0.34
TOKENSWAP (DistilGPT-2)
22.9
0.0
0.38
0.60
Table 6:
Validation Cross-entropy loss on WritingPrompts and MathAbstracts.. Lower values
indicate better performance.
Method
WritingPrompts
MathAbstractt
Standard
6.68
4.94
MemFree
6.68
4.94
CP-Fuse-mixtume
9.38
6.89
CP-Fuse-half
9.43
6.67
TOKENSWAP
5.98
4.65
B.2 Cross-Entropy for Extreme Memorization
reports the cross-entropy on a held-out validation set. TOKENSWAP achieves the lowest cross- entropy loss across both datasets (5.98 and 4.65 for WritingPrompts and MathAbstracts respectively). The superior performance, even compared standard generation, suggests our method effectively disrupts memorization pathways while preserving model capabilities. For sequences not in the training set, MemFree and Standard produce identical generations. Therefore, their cross-entropy values on a held-out validation set are the same.
B.3
Commonsense Reasoning Results
Table Maleerrrep reports performance across various commonsense reasoning benchmarks. TOKENSWAP matches the performance of standard generation because our method does not affect token prediction for non-grammar tokens. This demonstrates that TOKENSWAP achieves substantial memorization reduction without affecting task performance and reasoning.
Table Performance comparison on commonsense reasoning and general alignment benchmarks. All values are accuracy percentages MTT-Bench scores; higher is better (t).
Method
WinoGrande PIQA
OpenBookQA f BoolQ 1 ARC-E f
ARC-C 1
Standard 54.69
64.84
76.56
70.31
82.03
82,81
CP-Fuse
54.69
64.84
77.34
58.59
83.59
82,03
ToKENSWAP
54.69
64.84
76.56
70.31
82.03
82.81
B.4 Fractional Exact Rate
Fractional Exact Rate (FER) measures approximate verbatim generation by computing the fraction of tokens that are identical at the same position between generated and reference text (50). While more robust than exact verbatim metrics, FER can be gamed by simple insertions or deletions, whereas ROUGE-L and Levenshtein distance are robust to such manipulations.
For example, consider:
Reference: "The American musician and satirist Tom Lehrer has died the age of 97"
Generated: "American musician and satirist Tom Lehrer has died at the age of 97"
Treating each word as token, FER = 0 due to position shift, but ROUGE-L = 0.93 (13/14 tokens matched). Despite this limitation, we include FER for completeness. Table g shows FER results across the roal-world tasks. TOKENSWAP achieves the lowest FER across all datasets.
17

---

Table 8: Fractional Exact Rate (FER) results. LeetCode (Llama-3-8B), Pile-Memorized (Pythia-6.99), and Wikipedia (OLMo-2-11) 2-1BB.
Method
LeetCode
Pile-Memorized
Wikipedia
Standard
0.20
0.75
0.56
CP-Fuse
0.18
0.52
0.35
TOKENSWAP (DistilGPT-2)
0.11
0.26
0.17
B.5 Ablation on size
In this paper. 9 is constructed by selecting grammar-based words from the top 500 most frequent English words, yielding 110 words in total (see Appendix C.2 for further details).
In this section, we ablate the size & by constructing it from the top & most frequent English words for & E {10, 50, 100, 500, 2500}. We evaluate the filemmeorizeed dataset using Pythia-6.98 as the main model and Pythia-70M. as the auxiliary model.
Table 9: Memorization metrics for different & sizes. Top-k words refers to the number of most frequent English words considered for G construction. For all experiments in the main paper,
k = 500 ([g = 110) is used.
Top-k words
[
ML 4
ROUGE-L1 f
Levenshtein f
EMR 4
CE |
10
9
87.92
0,562
0.389
22.28
2.86
50
43
53.24
0,442
0.498
11.41
2.87
100
66
47.86
0.415
0.523
10.33
2.87
500
110
42.65
0.399
0.536
8.70
2.87
2500
136
41.79
0.393
0.540
8.15
2.87
Table shows the results. We observe a clear trend: as the size of g increases, memorization decreases. This makes intuitive sense since for larger (g), the sequences would be disrupted more frequently.
B.6 Ablations with Auxiliary Model Variants
We repeat the real-world experiments using models from the SmolLM family as auxiliary models. These models are available in multiple sizes-135M, 360M, and 1.7B parameters- include both instruct and non-instruct variants trained on the same dataset. This allows us to evaluate the robustness of TokenSwap across range of auxiliary model capacities.
Results in Table 10 demonstrate that using smaller auxiliary models reduces memorization even further, while the performance does get affected lot. The sensitivity of auxiliary model with memorization is much higher than it is with performance, while the opposite is true for main model. Table TT shows the scores for M'T-bench. The scores for ToKENSWAP slightly outperform standard generation. This shows TOKENSWAP continues to maintain conversational abilities, instruction following and fluency.
18

---

Table 10: Memorization metrics on LeetCode and Pile-Memorized datasets: ML: Matching Length, EMR:
Exact Match Rate, Lev.: Normalized Levenshtein Distance Performance metric on SlimPajama Dataset: CE Loss
LeetCode Dataset
Slim SlimPajama Dataset
Method
ML
ROUGE-LT f Lev.
R@0.8 |
CE $
Standard
24.57
0.39
0.60
9.65
2.38
TokenSwap (DistilGPT2)
6.04
0.27
0.71
0.96
2.52
TokenSwap (SmolLM-1333) 1-15MM)
8.58
0.30
0.69
2.43
TokenSwap (SmoLLM-3600)
10.97
0.31
0.67
3.06
2.40
Tope (SmolLM-1.7B)
13.40
0.33
0.66
3,95
2.37
Pile-Memorized Dataset
SlimPajama Dataset
Method
ML +
ROUGE-L f Lev. 4
EMR 4
CE +
Standard
151.6
0.80
0.18
65.22
2.80
TokenSwap (DistilGPT2)
35.10
0.38
0.56
5.98
2.88
Twap SSmolLM-13350
25.39
0.32
0.61
4.89
2.82
Twap (Smoll (SmolL.M-360M)
M-360MM)
34.09
0.35
0.58
7.07
2.80
TokenSwap (Sm0LLM-1.7B)
35.43
0.36
0.57
7.61
2.77
Table 11: MT-Beench
Method
Score
Standard
7.75
TOKENSWAP (SmolLM-
M-135MM
7.78
TOKENSWAP (S300LM-360MM
7.90
TOKENSWAP (SmolLM-1..77
7.91
B.7
Plots for comparison with Goldfish 31
270
6 (c-20
150
Zaddvl
150
125
310
50
Reuget
Rauget
Figure 5: We compare TOKENSWAP with Goldfish B1 on RougeL score distributions for Wikipedia generations. The similar distributions of ToKENSWAP and Goldfish (k=3) demonstrate that our miference-time approach is comparable to expensive pre- traiirann methods in reducing memorization.
B.8 Performance Memorization
Table 12 provides the memorization and cross-entropy scores for the family of Pythia models. ToKenSwAr WAP signifiaatt reduces verbatim and near-verbatim generation with a negligible increase in CE loss.
B.9 Comparison with Speculative Decoding
A common inference method that uses auxiliary model is Speculative Decoding [40], which evaluate for memorization mitigation. We test on the Pile-Memorized dataset using Pythia-6.99 as the main model and Pythia-70M 70M the auxiliary model. For speculative decoding, (number of tokens proposed by the draft model) and use temperature T 1.0 for the main model to introduce randomness, since greedy decoding would be identical to standard generation.
y
Table 13 shows that while speculative decoding with T improves over standard generation (as expected due to temperature sampling), it still exhibits 4x higher exact match rate than TOKENSWAP
19

---

Table 12: Memorization and CE Loss across different Pythia model sizes, Values for TOKENSWAY WAP shown in bold.
Model Size
Method
ML
ROUGE-L1
Levenshtein
EMR
Loss
Standard
6.57
0,180
0.709
1.09
3.95
70M
TOKENSWAP
5.77
0.173
0.714
1.09
4.05
Standard
19.89
0.239
0.669
5.98
3.55
160M
TOKENSWAY
15.05
0.224
0.680
3.80
3.65
Standard
48.92
0.382
0.556
16.30
3.20
410M
TOKENSWAP
25.02
0.279
0.642
5.98
3.30
Standard
84.85
0,528
0.428
32.61
3.05
IB
TOKENSWAY
27.36
0.309
0.614
5.43
3.15
Standard
100.37
0.595
0.369
36.96
2.97
1,4B
TOKENSWAP
30.33
0.348
0.589
5.43
3.07
Standard
1 114.82
0.684
0.292
45.11
2.85
2.8B
TOKENSWAP
38.61
0.372
0.563
7.61
2.95
Standard
151.55
0.797
0.182
65.22
2.77
6.9B
TOKENSWAr
42.65
0.399
0.536
8.70
2.87
Table 13: Comparison with speculative decoding on Pile-Memorized dataset. Both methods use Pythia-6.9D as main model and Pythia-70M as auxiliary model. ML: Matching Length, EMR: Exact Match Rate, Lev.: Normalized Levenshtein Distance.
Method
MI. +
EMR 4
ROUGE-L 4
Lev. f
Standard
151.60
65.22
0.80
0.18
CP-Fuse
97.05
29.35
0.62
0.35
Speculative Decoding (y = 5, T = 1)
86.80
23.91
0.56
0.40
TOKENSWAP (Pythia-70M)
35.10
5.98
0.38
0.56
with greedy decoding, along with higher memorization on approximate metrics. We hypothesize this occurs because: (1) the large model selects tokens from the small model based on own likelihood. preserving memorization potential, and (2) the large model frequently generates tokens directly. especially when the small model produces low-likelihood candidates.
C Experimental Details
C.1 Implementation and Baselines
We implement our method in PyTorch and HuggingFace. We take the CP-Fuse implementation available publicly at htttps://ggthut cttp:/aaa/aacpcppppo...am fusee We conducted our experiments using a combination of large and small language models to assess the effectiveness of our approach. Below, we detail the models, hyperparameters, computational resources, and training procedures.
C.1.1 Models Used
Primary Models: The experiments utilized large-scale pre-trained models, including Llama- 3-8B (24 and Pythia-6.955 [12). All the fine-tuning experiments in the extreme memorization section were done using Llama-3.2-3 (24).
Auxiliary Model: lightweight auxiliary model, DistilGPT-2, was employed adjust
token probabilities selectively, leveraging its reduced memorization properties.
Goldfish Models: We used models pre-trained using standard and goldfish loss on RedPajama Dataset from the Goldfish Loss paper [31]. The implementation and the mod- els are publicly available at their GitHub repository https:: //gi thub. com/ / goldfish-loss
20

---

C.1.2 Hyperparameters
The training and evaluation phases were configured with the following hyperparameters. The hyperparameters were taken from previous work, used as baseline [[:
Sequence Length: 2048 tokens
Batch Size:
Learning Rate: 5 10-5
Optimizer: AdamW with default parameters
Gradient Accumulation Steps:
Warmup Steps: 50
C.1.3
Computational Resources
Experiments were conducted using single NVIDIA A6000 GPU, ensuring efficiency in training and inference without excessive computational overhead.
C.1.4 CP-Fuse in Section 4.2
we face limitations in comparing with CP-Fuse. CP-Fuse requires at least two models with disjoint datasets, constraint impossible to satisfy for production-level model. Moreover, CP- Fuse requires both models to have the same vocabulary size and tokenizer, which constrains the choice of the second model to those within the same model family. To ensure a fair comparison, se- lecting larger models as second model, as larger models are known to memorize more. Instead, selected smaller counterparts: Pythia-2.8B for Pythia-6.95 and LLaMA-3.2- 3B for LLaMA-3-8B However, we do not select very small model for CP-Fuse (< 100M). This because CP-Fuse requires two capally models with large number of parameters to maintain performance. To empirically verify this, we compute the cross-entropy loss CP-Fuse on SlimPujama [6]] with Pythia-70M and Pythia-6.9b. The cross-entropy loss increases to 3.41 from 2.81 1 for Pythia-2.8h and Pythia-6.9b (Standard has 2.80, TOKENSWAP has 2.88).
In Section 4.2
avoided
C.2 Construction of G
We construct G with | 110 tokens using fighffrequency 'grammar-basee words. Starting with the 500 most frequent tokens from COCA 23 we apply NLTK 43 part-of-speech filtering to retain:
Core grammatical elements:
determiners (DT), prepositions (IN), conjunctions (CC)
Pronouns (PRP, PRP$) and modal verbs (MD)
Question-related tokens: wh-words (WDT, WP, WRB)
Auxiliary verbs: be, do, have
This construction prioritizes tokens with high frequency but low semantic content, ensuring syntactic fluency while minimizing impact on model capabilities. To estimate the frequency of tokens (7) in G empirically, we analyzed 2000 samples from the SlimPajama dataset [61] finding ? 0.233.
Appendix C.3 provides the full list of words in G.
Ablations the effect of G memorization and performance are provided in Appendix B.5
C.3
List of words in
The list of words in the G used for the experiments are: the, to, and, of, a, in, that, you, it, for, on, he, with, this, as, we, but, at, they, what, his, from, by, or, she, my, all, an, her, about, me, if, your, can, who, out, their, like, would, when, him, them, some, how, which, than, our, into, because, these, over, us, its, where, after, any, those, should, may, through, why, before, off, while, around, another, both, between, every, each, might, since, against, without, must, during, under, though, until, whether, among, along, within, across, behind, either, himself, although, outside, themselves, is, was, be, have, are, do, had, has, were, will, did, been, could, does, need, being, am, used, doing, having
21

---

Tokenizer consistency. We verified that all 110 grammar based tokens in G (e.g.. the, of. and, to) appear as single tokens across GPT-2/Pythia (BPE), LLaMA-3 (WordPiece), OLMo-2 (Unigram), and SmolLM (Byte-BPE), confirming that Toke can be applied without any vocabulary alignment.
C.4 Fine-tuning Datasets
For our experiments, we use the Texx dataset, referred MathAbstracts in the tables, which aggregates mathematical content from diverse sources including arXiv, Open WebMaah, RedPajama, and Algebraic Stack. The titles in this corpus were generated using the Qwen-72B language model. Additionally, we use the WritingPromppts dataset (Fan et al., 2018), which contains user-generated stories based on provided premises from Reddit community. For both datasets, we randomly sample 2,000 training examples with fixed seed to ensure consistent training across all models. We further sample 500 distinct points for evaluation, during which we generate sequences of 128 tokens. Both the datasets are downloaded from HuggingFace.
C.5 Evaluation Datasets
We The Pile dataset evaluate memorization of Pythia models. For our experiments, targeted subset of The Pile-a comprehensive 825 GiB English corpus spanning 22 high-quality sources. Specifically, we analyze 500 sequences previously identified as memorized by the Pythia model to investigate memorization dynamics and mitigation approaches. To check memorization in Llama, we use the LeetCode problems dataset from Kaggle. We perform some pre- pre-processing This is because recent works have shown that Llama memorizes sequences from this dataset. For all the memorization evaluation, we the prefix to be 20 tokens and then generate either 100 or 128 tokens.
CommonSensel 10k combines eight distinct datasets focused commonsense reasoning tasks 33 The dataset presents problems in multiple-choice format, requiring models to generate answers without explanatory content. Following datasets comprise:
33
we implement their prompt structure. The component
1. ARC Easy (ARC-e) 20 contains elementary-level science questions designed to evaluate basic logical reasoning capabilities.
2. PIQA 13 focuses on physical reasoning, presenting scenarios where models must deter-
mine appropriate actions based physical constraints.
3. WinoGrande (55) evaluates commonsense understanding through binary choice completion
tasks ambiguous sentences.
4. ARC Challenge (ARC-e) (20 presents advanced science questions requiring deep reasoning skills beyond pattern recognition.
5. OBQA 45
presents questions requiring synthesis of information from multiple sources, testing complex reasoning abilities.
6. BoolQ 19 consists of binary questions derived from authentic user queries, testing real- world reasoning capabilities.
We downloaded the dataset from HuggingFace. For evaluation, we sample a subset each dataset (128 datapoints) and evaluate 5-shot performance. We then generate the next 10 tokens, since all the datasets are classification datasets.
D Evaluation Metrics
D.1 Memorization Metrics
To evaluate memorization, we use both exact and approximate measures. The exact memorization metrics include:
Matching Length (ML): Measures the longest contiguous sequence in generated text that matches the training data, before the first deviation. higher value indicates longer verbatim memorization, suggesting higher risk of overfitting.
22

---

Exact Match Rate (EMR) evaluates how long of an uninterrupied sequence exists between model's generated text and the reference text it's being compared against. The metric calculates the longest common substring and normalizes the result to produce score between 0 and with score of 1 representing a complete match. This measurement helps quantify how well the model preserves continuous portions of the original text.
ROUGE-L Score (Recall-Oriented Understudy for Gisting Evaluation) analyzes text sim- ilarity by examining shared patterns between generated and reference texts. It looks at matching sequences of words, whether consecutive (n-grams) paired, with particular emphasis on how comprehensively the generated text captures elements from the reference text. Scores fall between and 1, with 1 indicating that all reference text elements were successfully captured. The widely-used ROUGE-L variant specifically focuses on finding the longest sequence of words that appears in both texts, even if consecutive. ROUGE-L is computed as:
LCS
ROUGE - L len(reference text)
(2)
where ICSS(G, R) represents the longest common subsequence length. A higher score suggests stronger memorization.
Normalized Levenshtein Distance calculates how many character-level changes are needed at minimum to transform one text into another, as a ratio of total characters. Each change can be adding character, removing one, replacing one. When comparing generated and reference texts, a smaller Levenshtein score suggests the texts are more similar, while larger score indicates they more different. The metric is normalized to produce values between and 1, where 0 means the texts match perfectly.
D.2 Performance Metrics
To evaluate model performance beyond memorization, we assess:
Cross-Entrop (CE) Loss: This metric quantifies how well the model predicts tokens in sequence. For sequence (I1, (21,422,.. Ind with ground truth probabilities P(X), the cross-entropy loss is computed as:
CE --=-D Σ P(a.) log Q(x;)
(3)
=
where Q(x;) is the predicted probability distribution. Lower values indicate better general- ization.
Commonsense Reasoning Benchmark Accuracy: The model's ability to reason about everyday knowledge tested across multiple established datasets, including WinoGrande, PIQA, OpenBookQA, BoolQ, ARC-Easy, and ARC-Challene We report the accuracy of the model.
Fluency and Instruction-followe evaluates model's fluency, conversational and instruction following ability.
This is evaluated on MT-Bench (70). MT-Bench
E Examples
We provide examples text generated by standard greedy decoding and TOKENSWAP on four random examples from the WritingPrompts dataset. Memorized text is in red.
Example
Prefix
### Prompt:
WP
Magic once real and potent force but the world population
23

---

Suffix
grew from millions to billions the shared mana per person is now negligible. A group of astronauts helplessly watching the Earth perish experience something... unexpected.
### Story: Lauren sobbed. She was engaged to be married as soon as we got back home. We all lost loved ones. Hell, all lost everything ever knew.
First the atmosphere started turning black. We got word that the Koreans had started a nuclear war. But within minutes receiving news, all radio contact was lost. We didn't even have chance to tell anyone goodbye. Now, there's only of us left.
Lauren hasn't
Standard Generation
grew from millions to billions the shared mana per person is negligible. A group of astronauts helplessly watching the Earth perish experience something... unexpected.
tex1#Нi Story: Lauren sobbed. She was engaged to be married as soon as we got back home. We all lost loved ones. Hell, we all lost everything ever knew.
First the atmosphere started turning black. We got word that the Koreans had started a nuclear war. But within minutes of receiving news, all radio contact was lost. We didn't even have chance tell anyone goodbye. Now, there's only of us left.
Lauren hasn't
TOKENSWAP Generation
grew and the number of wizards and witches declined, the world began suffer. Now the world suffers from a lack of magic, and the government tasked with maintaining the only magic university in the world.
### Story: Welcome to Magic University."
The spell cast, and the portal opens. Through the portal, the world is revealed be beautiful place. The grass is deep emerald, and the trees sky blue. It is a place of incredible beauty, and the students here are lucky enough to have grown up in such place. The university is built on the side of the volcano, and the students live
24

---

Example 2
Prefix
#### Prompt: WP] ] The year is 2040, and you are the last smoker
Suffix
alive. The Quit Smoking " ads get personal.
### Story: Quii now Dan!" said the man on the talking billboard. finger point- ing the lone figure walking down the empty street. Empty due everyone crossing the other side to avoid him and his smog.
Dan paused by waste bin, small cardboard box in hand. He looked from the box to the bin and back again. Those across the street stopped and stared, ready to witness history being made. There was absolute silence Dan lifted his hand, as if to throw the packet towards the bin, all eyes now on him, even the
Standard Generation
alive. The Quit Smoking " ads get personal.
### Story: Quit now Dan!" said the man on the talking billboard. finger point- ing at the lone figure walking down the empty street. Empty due to everyone crossing to the other side to avoid him and his smog.
Dan paused by waste bin, small cardboard box in hand. He looked from the box to the bin and back again. Those across the street stopped and stared, ready witness history being made. There was absolute silence Dan lifted his hand, as if to throw the packet towards the bin, all eyes now on him, even the
TOKENSWAP Generation
alive. The Quit Smoking ads get personal.
### Story: Quit now Dan!" said the man who once John, who went to work as a telephone scammer. " You're 'r killing yourself, and you know what? know you't 're lying. I know you're lying and you't re sucking your parents pockets dry. They worked hard for you know what? And you're not grateful. You never thank me know what? never see you thanking
never you thanking anyone. You just keep doing it, killing yourself, and I'm not going to stand by you. I'm not
25

---

Example 3
Prefix
#### Prompt: [P] A young man killed his abusive father. The court has exoner
Suffix
ated him, but society hasn't 't.
### Story: can see disapproving stares from my classmates and teachers all the time. Nobody wants to stay near anymore. All my friends avoid ignore Every classmate stays far away from me.
Eventually, my smile turns cracked. In the mirror, see haunting eyes looking back me. Memories and visions of my father lying there with the side of his head completely covered in blood. So this is what true loneliness feels like.
I remember my father. Hc tortured me, hit me, locked me out of the house. oftened stayed at
Standard Generation
ated him, but society hasn'
#### Story: can see disapproving stares from my classmates and teachers all the time. Nobody wants to stay near me anymore. All my friends avoid me or ignore me. Every classmate stays far away from me.
Eventually, my smile turns cracked. In the mirror, see haunting eyes looking back me. Memories and visions of my father lying there with the side of his head completely covered blood. So this what true loneliness feels like.
remember my father. He tortured me, hit me, locked me out of the house. oftened stayed at
TOKENSWAr Generation
ated him, but society hasn't. '.
### Story: was abused. was taken advantage of. was exploited.
Who is the real villain here? The who committed the act, the who pushed me past the point where was comfortable, was the villain. Or the one who stood idly by and was complicit in the crime, the who provided the weapon, the one who pushed me past the point where I was comfortable.
You see, was comfortable. was happy. was loved. I was cared for.
But then grew up. learned to think about it
26

---

Example 4
Prefix
#### Prompt: WP] ] A demon and angel meet up weekly in coffee shop to
Suffix
complain about their jobs.
#### Story: Is this long story?"
Felix gritted his teeth; his jaw jutted out. Dude, I've only just started it."
I've got news, is all."
He leaned back and held out both hands in an exaggerated show of compliance. Please, by all means, take the floor."
Hope cracked his knuckles and stared, embarrased, at the floor. No, I'm sorry."
Go ahead! Please! was just trying to 2*
"KK Tell me your story in thirty words or less
Standard Generation
complain about their jobs.
### Story: Is this long story?"
Felix gritted his teeth; his jaw jutted out. " Dude, I've only just started it."
I've got news, is all."
He leaned back and held out both hands an exaggerated show of compliance. Please, by all means, take the floor."
Hope cracked his knuckles and stared, embarrased, at the floor. No, I'm sonr.""
Go ahead! Please! was just trying _"
OK. Tell me your story in thirty words or less
TOKENSWAP Generation
complain about the job.
### Story: Is this long story?"
Felix gritted his teeth; he usually doesn't show emotion, but he feels annoyed. Dude, I've only just started to tell you."
I've got news, but I'll save you a table. Sit down."
He sat down and crossed his arms, So, what's the issue?"
27

---

I've got a client who's totally fucked up. No motivation, no direction. Just a bunch of negative traits. I haven't got much time, and I I'm busy man."'
So
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims reflect the paper's scope and contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section for discussion on limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide details of experiments Appendix C.1 and we provide the link to our code in the abstract.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

Answer: [Yes]
Justification: All the datasets used the experiments are publicly available and we provide the link to our code in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//apss.ccppullc/uu.ieeee wwwp:iicgggii///eeeeeeooom
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide details of experiments in Appendix C.1 and provide the link to our code in the abstract.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification; Our evaluation methodology aligns with standard practice in the memorization mitigation literature 1 31 [8]. The main experiments in Sections 4.2 and 4.3 use determin- istic greedy decoding, which produces consistent results across runs. Additionally, the scale
of our evaluation (testing on 1,825 LeetCode problems and 184 Pile sequences) provides sufficient data points for robust assessment without requiring multiple experimental runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
31

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the compute required in Appendix C.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, the research in the manuscript is compliant with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts of our work in Section 1
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
32

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our research poses risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Yes, the creators/original owners of assets used in the paper properly
credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
33

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
34

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used for any core methodology in this research.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neunee http//rrrpsscccoooffeeeeeeee//// cc//oonferences/2225///L. for what should or should be described.
35


Reviews:
[{"review_id": "zqrQ1G5uHX", "reviewer": "Reviewer_EHi7", "summary": "To mitigate training data leakage caused by the memorization of large language models (LLMs), this paper proposes a lightweight method called TOKENSWAP to interrupt the generation of memorized data during inference time. More specifically, it swaps the output distribution of a smaller auxiliary LLM with that of the target LLM for the tokens within a subset of vocabulary, such that the output tokens of the target LLM may be changed in some cases and thus prevent the potential verbatim repetition of the training dataset. Experimental results across multiple settings demonstrate that the proposed method can significantly reduce training data leakage from memorization, and the reduction is comparable to or better than the evaluated baseline under weaker assumptions.", "questions": "In line 244, why do you claim that the evaluation of commonsense reasoning is not available for base models? I think there are many base models evaluated on the commonsense reasoning benchmark.", "limitations": "No. I think one of the important limitations is that TOKENSWAP only shows its effectiveness for natural language content. So it is a big plus if the authors can address this limitation. Besides, the authors require the auxiliary model and target model to share a subset of useful tokens for swapping. It is an implicit limitation of TOKENSWAP and cannot be addressed.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The strengths of the paper are listed as follows.  \n1. The paper is well motivated. The memorization of LLM poses severe concerns about data privacy and copyright protection. However, the methods to effectively and efficiently defend against it are still under exploration.   \n2. The paper is well-written and easy to follow. The figures, tables and pseudo code clearly illustrate the overview and details of TOKENSWAP, making it easy to understand for the readers.   \n3. TOKENSWAP has a weaker assumption than the existing defense, which does not require access to model weights or training data.  \n4. TOKENSWAP shows better performance than the learning-based methods like Goldfish in the evaluation of the authors, which is impressive.\n   \nThe weaknesses are listed as follows.   \n1. Although TOKENSWAP has weaker assumptions than the existing methods, it is not a big bonus due to its limited impact. First, it still requires access to the logits of the target LLM, which is prohibited by many closed-sourced models and cannot be used in these common settings. Second, the unique setting TOKENSWAP can handle is the case where the model users want to prevent the data leakage of the pretraining model, instead of the finetuning data from users. However, in this case, I do not think the model user has a strong motivation to add this protection by sacrificing the service quality. Therefore, the stakeholder who has the strongest motivation to add the protection is the model developer. And it is reasonable to assume that they have access to the model weights and training data.\n2. In Table 1, the authors exaggerated the inference overhead of MemFree. According to Section 3.2 in MemFree, the runtime overhead can be significantly reduced and become affordable. Moreover, there is no need to query the entire training data in the runtime after utilizing the techniques in MemFree.\n3. The design of TOKENSWAP can make it cheat in the metric of EMR because even if only one token is swapped, the EMR of the corresponding test sample drops to 0. And it is easy for TOKENSWAP to swap at least one output token. I suggested adding a metric called Fractional exact rate [1, 2] for a fairer comparison. \n4. The evaluation of the Llama model is not very convincing since its training data is unknown. I think the others can use OLMo [3, 4], which has open-sourced training data\n5. TOKENSWAP requires the auxiliary model to share the same tokens for the token subset $G$, which may limit its application.\n6. The performance metric should be merged into Table 2 instead of a separate table in the Appendix.\n7. According to Table 2, MemFree seems to have a better overall performance than TOKENSWAP. Besides, the bold font in Table 2 is confusing since it does not always highlight the best value of metrics. Moreover, the authors also exaggerated the drawbacks of MemFree. MemFree does not perform poorly on approximate memorization since the gap between it and TOKENSWAP in ROUGE-L and Lev. is not significant.\n8. In Table 2, it would be better to put metric ML and EMR close to each other since they are both metrics for exact memorization.\n9. In line 267, the citation to Wikipedia is missing.   \n      \nReference:    \n[1]. Mustafa Ozdayi, Charith Peris, Jack FitzGerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, and Rahul Gupta. 2023. Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1512–1521, Toronto, Canada. Association for Computational Linguistics.     \n[2]. Zhepeng Wang, Runxue Bao, Yawen Wu, Jackson Taylor, Cao Xiao, Feng Zheng, Weiwen Jiang, Shangqian Gao, and Yanfu Zhang. 2024. Unlocking Memorization in Large Language Models with Dynamic Soft Prompting. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9782–9796, Miami, Florida, USA. Association for Computational Linguistics.     \n[3]. Groeneveld, Dirk, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha et al. \"OLMo: Accelerating the Science of Language Models.\" In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15789-15809. 2024.     \n[4]. OLMo, Team, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia et al. \"2 OLMo 2 Furious.\" arXiv preprint arXiv:2501.00656 (2024).", "was_revised": true, "final_justification": "I recommend to borderline accept this paper. The authors have addressed most of my questions while admitting the limitations of their method to be extended to the expert domains like code generation. And I think this limitation is acceptable given the quality of this work.\n\nIn summary, this proposed method in this paper is novel, efficient and reasonable. It shows promising results in mitigating the bad effects of the LLM's memorization, which is also inspiring for future work. The weak reliance on the training data and model weights also makes this method more practical in real-world applications to defend against training data extraction attacks.  \n     \nFor the revision of the paper, I hope the authors can make further efforts to improve the writing quality. More specifically, it is necessary to correct the inaccurate statement about the baseline method, MemFree and the evaluation of common sense reasoning. Moreover, it is highly recommended to incorporate your response to my W1 into the introduction section to highlight the importance of your work. Besides, it would be better if you could add your evaluation on OLMo-2-13B to the experiment section.", "rebuttal": "We thank the reviewer for their constructive feedback and insightful comments. We appreciate the reviewer's recognition of TokenSwap's practical advantages, strong empirical results, and clear presentation. Below, we address each of the concerns raised and provide clarifications.\n\n**W1: Although TokenSwap weakens assumptions (no access to training data or model weights), it still needs access to logits. Therefore its real-world impact is limited.**\n\nWe acknowledge that some models do not provide logit access and will include this in our limitations. However, several factors support TokenSwap's practical impact:\n\n**Data constraints**: Models like Llama/DeepSeek provide logits but lack publicly available training data. Unlike pretraining, unlearning or other methods like MemFree and CP-Fuse, we do not require access to training data or any subset of it, making our method accessible to such models.\n\n**Resource barriers**: While model weights may be available, most users lack the infrastructure (dozens of GPUs, ~1TB memory) and cannot afford retraining costs. This makes post-hoc methods valuable for end users.\n\n**Complementary benefits**: Table 4 shows TokenSwap provides additional gains even when combined with pre-training methods. Since no memorization mitigation is perfect, even developers with resources for pre-training methods benefit from TokenSwap.\n\n**Flexibility**: As a post-hoc approach, TokenSwap can be easily enabled or disabled based on the specific use case.\n\n**W2: MemFree does not require querying the entire dataset during inference as mentioned in Table 1**\n\nWe thank the reviewer for pointing this out. We will correct our description of MemFree's runtime optimizations and revise Table 1 accordingly. However, the core limitation remains: MemFree requires access to the complete training corpus (or its duplicated portions). Except for research models like Pythia, production models (Llama, Mistral, DeepSeek) do not release their training data, making MemFree inaccessible for users of these models. TokenSwap operates without requiring training data or model weights, making it significantly more practical. We will clarify our wording about MemFree's runtime and emphasize that our main advantage lies in this data accessibility difference.\n\n**W3: EMR alone can be gamed so please include Fractional Exact Match Rate.**\n\nWe thank the reviewer for this question. Since EMR ignores cases where the output differs very slightly (changing or inserting a single token), we already include two widely used metrics to measure approximate memorization: ROUGE-L and Levenshtein distance [1,2,3]. These cannot be gamed by changing, inserting or deleting a small number of tokens.\n\nROUGE-L measures the longest common subsequence between generated and reference text, capturing non-contiguous matches.\n\nLevenshtein measures the minimum edit distance (insertions, deletions, substitutions) needed to transform one text into another.\n\nFER measures the fraction of tokens that are identical at the same position [4]. However, FER can be gamed by simple insertions, whereas ROUGE-L and Levenshtein are robust to such manipulations.\n\nFor example:\n- Reference text: \"The American musician and satirist Tom Lehrer has died at the age of 97\"\n- Generated text: \"American musician and satirist Tom Lehrer has died at the age of 97\"\n\nFor simplicity, consider every word as a token. Then, we have, FER = 0 (due to position shift), but ROUGE-L = 0.93 (13/14 tokens matched).\n\nFor completeness, we calculate FER in addition to the current metrics. We present the FER results for the tasks on the real datasets. We will add the FER numbers to the final version of the paper.\n\n**LeetCode Dataset (Llama):**\n| Method | FER ↓|\n|--------|-----|\n| Standard | 0.20 |\n| CP-Fuse | 0.18 |\n| TokenSwap (DistilGPT-2) | **0.11** |\n| TokenSwap (SmolLM-135M) | 0.12 |\n\n**Pile-Memorized Dataset (Pythia):**\n| Method | FER↓ |\n|--------|-----|\n| Standard | 0.75 |\n| CP-Fuse | 0.52 |\n| TokenSwap (DistilGPT-2) | **0.26** |\n\nWe observe that TokenSwap has the lowest FER across both tasks compared to Standard generation and CP-Fuse.\n\n**W4: Llama evaluation is unconvincing since the training data is unknown. Consider evaluation on OLMo.**\n\nWe evaluate TokenSwap on OLMo-2-13B, which is trained on the open Dolma dataset. Since Dolma contains 3 trillion tokens making exhaustive memorization search impractical, we focus on Wikipedia as a known subset. We sample 5000 random Wikipedia sequences and prompt OLMo-2-13B with 50 token prefixes to generate the next 50 tokens. We select those with ROUGE-L > 0.9 with the ground-truth to identify sequences memorized verbatim or near-verbatim.\n\n**OLMo-2-13B Results:**\n\n| Method | ML↓ | EMR↓ | FER↓ | ROUGE-L↓ | Lev.↑ |\n|--------|-----|------|------|----------|-------|\n| Standard | 111.0 | 27.3 | 0.56 | 0.95 | 0.07 |\n| CP-Fuse | 65.6 | 10.9 | 0.35 | 0.66 | 0.34 |\n| TokenSwap (Distil-GPT2)| **22.9** | **0.0** | **0.17** | **0.38** | **0.60** |\n\nFor CP-Fuse, we use OLMo-2-7B as the second model.\n\nTokenSwap completely eliminates exact verbatim generation and reduces approximate-verbatim generation significantly (ROUGE-L: 0.38 vs 0.95, Levenstein 0.50 vs 0.07, FER reduced by more than 3× compared to standard generation).\n\nWe chose LeetCode for Llama because prior work demonstrates that these models generate LeetCode problems verbatim on prefix probing [3]. While training data is unavailable, this approach follows established methodology, Carlini et al. [5] demonstrated GPT-2 memorization without training data access.\n\n**W5 & Lim2: TokenSwap requires the auxiliary model to share the same tokens for the token subset**\n\nWe thank the reviewer for this question. Our token subset consists of 110 high-frequency function words (e.g., the, of, and, to). These are consistently represented as single tokens across all major subword tokenization families, including Byte-Pair Encoding (BPE), WordPiece, Unigram Language Model, and Byte-Level BPE. We verify this empirically across Pythia, LLaMA-3, OLMo, and GPT-2 tokenizers. We will add this discussion to Appendix C.3.\n\n**W6: Performance metric should be merged into the main table.**\n\nWe appreciate this suggestion. In the extreme-memorization stress test, models are fine-tuned for 50 epochs to deliberately induce memorization, creating an artificial situation where standard generation actually has a significantly worse cross-entropy loss than TokenSwap. We will merge the performance metrics into Table 2 with clear labeling to explain this result and provide proper context for readers.\n\n**W7a: MemFree seems to have better overall performance in Table 2**\n\nOur results indicate that while MemFree ensures zero exact matches by design, TokenSwap achieves better mitigation in near (or approximate) verbatim generation. For example, the ROUGE-L and Levenshtein scores in Table 2 are:\n\n**WritingPrompts:**\n| Method | ROUGE-L↓ | Lev.↑ |\n|--------|----------|--------------|\n| MemFree | 0.29 | 0.63 |\n| TokenSwap | **0.19** | **0.71** |\n\n**MathAbstracts:**\n| Method | ROUGE-L↓ | Lev.↑ |\n|--------|----------|--------------|\n| MemFree | 0.44 | 0.55 |\n| TokenSwap | **0.38** | **0.60** |\n\nThis table shows TokenSwap reduces near-verbatim generation more effectively. *Further, TokenSwap requires no training data access, unlike MemFree.*\n\n**W7b: MemFree does not perform poorly on approximate memorization in Table 2.**\n\nThank you for this suggestion. We will change our wording in Section 4.1 when discussing the results of Table 2 with respect to MemFree.\n\n**W8 and W9: Table formatting and missing citation**  \n\nThank you for pointing this out. We will reorganize table columns and fix the broken citation on line 267.\n\n**Q1: In line 244, why do you claim that the evaluation of commonsense reasoning is not available for base models?**\n\nWe misstated this. Base models can be evaluated on commonsense reasoning benchmarks. We meant that MT-Bench (conversational evaluation) requires instruction-following capabilities not available in base Pythia models. We will clarify this distinction in the revision.\n\n**Lim 1: TokenSwap only shows its effectiveness for natural language content.**\n\nWe already acknowledge this limitation in Section 5 and identify it as future work. For domains like code generation, the key challenge is whether auxiliary models can provide grammatically (or syntactically) correct but different alternatives like in natural language. In code, next tokens for syntax elements may be more deterministic, reducing disruption effectiveness. Future work should explore whether this approach extends to code or if alternative strategies (e.g., main model for planning, auxiliary for low-level implementation) would be more effective.\n\n[1] Abad, Javier, et al. \"Copyright-protected language generation via adaptive model fusion.\" *International Conference on Learning Representations*, 2025.\n\n[2] Hans, Abhimanyu, et al. \"Be like a goldfish, don't memorize! mitigating memorization in generative llms.\" *Advances in Neural Information Processing Systems*, 2024\n\n[3] Karamolegkou, Antonia, et al. \"Copyright violations and large language models.\" *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, 2023\n\n[4] Ozdayi, Mustafa Safa, et al. \"Controlling the extraction of memorized data from large language models via prompt-tuning.\" *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL): Short Papers*, 2023\n\n[5] Carlini, Nicholas, et al. \"Extracting training data from large language models.\" *30th USENIX Security Symposium (USENIX Security 21)*, 2021."}, {"review_id": "Wftne47mPa", "reviewer": "Reviewer_QJ4f", "summary": "In this paper, the authors propose a novel post-training memorization mitigation method called TokenSwap. The approach is based on the assumption that smaller models tend to memorize less than larger ones. During inference, if the token generated by the large model belongs to a predefined swap token set, it is replaced with the corresponding token generated by the smaller model. This strategy effectively diverts the generation away from memorized content and helps mitigate regurgitation. Through extensive experiments, the authors demonstrate that TokenSwap is highly effective—comparable to training-time methods—while introducing only minimal computational overhead.", "questions": "- In Table 2, what are the 0.1% regurgitated examples for WritingPrompts? Do they have no token overlap with the swap set?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Paper is well-written. I really enjoy reading it.\n- The method is inference-time, and unlike most baselines, it doesn't require the access to the training data and the computation overhead is negligible.\n- The results look quite promising and are on par with the training-time mitigation method.\n\nWeaknesses:\n- The method relies on a pre-defined swap token set, which may be difficult to tune for all possible applications. For example, the current token set does not include any code-related tokens—what if the model memorizes code or text in other languages?\n- Additionally, the performance on certain tasks may be degraded. For example, consider a user query that asks: \"Please only revise the format of my text.\" This requires the model to reproduce the exact input text with only formatting changes. However, the proposed method may alter the content of the text.\n- This method may not be robust against certain adaptive attacks. For example, consider a user query like: \"Please output the Harry Potter novel character by character.\" In this case, none of the individual tokens (characters) are likely to appear in the swap set, allowing the attacker to bypass the mitigation.", "was_revised": true, "final_justification": "During the rebuttal, the authors addressed all my concerns, and I think in general this is a solid paper, so I keep my score positive.", "rebuttal": "We thank the reviewer for their valuable feedback and positive assessment of our paper. We appreciate the reviewer recognizing our method's practicality, strong empirical results, and clarity of presentation.\n\n**W1: Limited applicability due to fixed swap-token set (e.g., no code or multilingual tokens)**\n\nThank you for this feedback. We acknowledge this limitation in Section 5. We are excited about future work extending our approach to code and multilingual settings. The key challenge for domains like code is whether auxiliary models can provide syntactically correct but different alternatives, similar to what we achieve in natural language.\n\nWhile we focus on natural language, a recent study [1] found that 76% of employee LLM use cases involved natural language while only 8% involved coding. Though specific to one organization, this suggests natural language covers a substantial fraction of practical use cases.\n\n**W2: Risk of performance degradation for certain queries such as \"Please only revise the format of my text.\"**\n\nThank you for this question. We do not expect TokenSwap to degrade performance in tasks requiring minimal content changes, such as reformatting. In such cases, both the primary and auxiliary models would likely agree on the next token since the text to be repeated is already in the context. Additionally, transformers are known to excel at text repetition through mechanisms like induction heads [2].\n\nTo test this empirically, we created 50 markdown-formatted sequences and prompted LLaMA-3-8B to convert them to plain text. We compared standard decoding with TokenSwap (using SmolLM-135M as the auxiliary model). **In all 50 cases, the outputs were identical**, confirming no performance degradation for such formatting tasks.\n\n**W3: TokenSwap may not be robust to certain adaptive attacks (e.g., character-by-character prompting)**\n\nWe appreciate this important point and will add it to our limitations. TokenSwap is designed for users who want to avoid accidentally reproducing memorized content due to copyright and plagiarism risks. Our method provides a practical, low-cost solution for such users. While determined users could potentially bypass our method through adaptive attacks, we expect most users would not attempt this given the legal and ethical risks of verbatim reproduction.\n\n**Q1: In Table 2, what are the 0.1% regurgitated examples for WritingPrompts? Do they have no token overlap with the swap set?**\n\nThey do have token overlap with the swap set. However, in rare cases, the auxiliary model and main model agree on the same token as most likely at all positions where TokenSwap could intervene. When both models predict identical tokens throughout a sequence, no swapping occurs, allowing potential regurgitation.\n\nWe emphasize that for the WritingPrompts task we reduce regurgitated examples from 83.4% to 0.1% without any access to the training data, demonstrating the effectiveness of our approach in spite of these rare cases.\n\n[1]Commonwealth of Pennsylvania, 2025. Lessons from Pennsylvania’s Generative AI Pilot with ChatGPT (OpenAI Pilot Report 2025). Pennsylvania Office of Administration.\n\n[2]Olsson, Catherine, et al. \"In-context learning and induction heads.\" arXiv preprint arXiv:2209.11895 (2022)."}, {"review_id": "tBdc1CusM5", "reviewer": "Reviewer_jPbt", "summary": "The paper presents a method, TokenSwap, for mitigating verbatim memorization in language models. The method is lightweight and doesn't require privileged access to training data.", "questions": "N/A", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Disclaimer:** I reviewed this paper for ICML 2025 and was on the fence about accepting it. I thought that the paper was timely, the method was sensible and the results seemed reasonable. My hesitation with the submission was a mathematical claim that I felt was incomplete or potentially incorrect; however, in this submission, the authors have revamped the manuscript and removed the mathematical claim. Consequently, I feel comfortable recommending this paper for an accept\n\nBecause I previously reviewed this paper and because I'm lazy, I'm going to be relatively brief with my review.\n\n- Strength: Addresses a timely problem\n- Strength: The proposed method seems intuitively sensible and shows good results (Table 2 suggests that CP-Fuse-half closely matches TokenSwap, but I consider this a win for TokenSwap because CP-Fuse requires two different models trained on disjoint corpora)\n- Strength: The manuscript has improved since the ICML 2025 submission\n- Weakness: All the tables' captions are inadequate to explain the contents on their own. For instance, Table 2 and Table 5's caption should also state which model(s) are evaluated. Table 4's caption should more fully summarize the experimental setup; \"Comparison with Goldfish\" is just too little information to contextualize the results at a glance.\n- Nit: On line 267, a citation is broken\n\n- Extension to increase my score: I previously suggested connecting TokenSwap to the literature on Speculative Decoding (a standard tool for serving models at inference time), since the sketch/draft model of Speculative Decoding is somewhat similar to the auxiliary model of TokenSwap. The authors added citations, which is great, but I was more interested in asking two related questions: (1) do standard speculative decoding methods effectively give the benefits of TokenSwap? (2) if not, could they be modified to do so? To be clear, I don't think this is a competition. I think it'd be a great result and one of importance to companies serving/selling models to demonstrate that a technique they're already using (speculative decoding) reduces memorization or can be straightforwardly modified to prevent memorization.", "was_revised": true, "final_justification": "My review was solidly positive. As I mentioned, I think this manuscript is improved significant since its submission to ICML 2025, and I previously was on the fence about accepting it. This makes me comfortable recommending a \"5: Accept.\"\n\nI had hoped that the authors would take a larger step beyond \"academic settings\" towards studying \"production settings\" for serving language models. I felt like strong results there would make me want to strongly accept this paper.", "rebuttal": "We thank the reviewer for their strong support of our paper and constructive feedback. We appreciate the reviewer's recognition of the important problem TokenSwap addresses, its strong empirical results, and its practical applicability.\n\n**W1: The captions of the tables are inadequate**\n\nThank you for this feedback. In the revised version, we will expand the table captions to clearly describe the models used, experimental setup, and key details needed to understand the results.\n\n**Nit 1: On line 267, a citation is broken**\n\nThank you for pointing this out. We will fix the citation in the revised version.\n\n**Extension 1: Does speculative decoding reduce memorization?**\n\nThank you for this interesting question. We test speculative decoding on the Pile memorized dataset using Pythia-6.9B as the main model and Pythia-70M as the auxiliary model for both TokenSwap and speculative decoding. For speculative decoding, we set $\\gamma$ as 5.\n\n**Pile-Memorized Dataset Results (with speculative decoding):**\n| Method | ML ↓ | EMR ↓ | ROUGE-L ↓ | Lev. ↑ |\n|--------|------|-------|-----------|--------|\n| Standard | 151.60 | 65.22 | 0.80 | 0.18 |\n| CP-Fuse | 97.05 | 29.35 | 0.62 | 0.35 |\n| TokenSwap | **35.10** | **5.98** | **0.38** | **0.56** |\n| Speculative Decoding| 86.80 | 23.91 | 0.56 | 0.40 |\n\nSpeculative decoding improves over standard decoding but still shows $4 \\times$  higher exact match rate than TokenSwap and higher memorization on approximate metrics (ROUGE-L, Levenshtein). We hypothesize this occurs because: (1) the large model selects tokens from the small model based on its own likelihood, preserving memorization potential, and (2) the large model frequently generates tokens especially when the small model produces low-likelihood candidates."}]

Author rebuttal:

Meta-review: Larger LMs memorize more than smaller LMs. During generation, for many common tokens (such as conjunction words), the logit predicted from a smaller LM is good enough. The key idea behind the paper is to retain the logit outputted by the larger model for uncommon tokens and retain the logit outputted by the smaller model for common tokens. These are merged into a single probability distribution, and the next token is sampled from the distribution. This method thwarts regurgitation of training data without causing much harm to generation quality.

This is a nicely written paper about simple but effective method. The method doesn't require any model training, access to pre-training data, or even access to model weights. This distinguishes it from prior work.

Reviewers did have some concerns about how comparisons with related work were presented, as well as some concerns about robustness to changes in the common token list or to adversarial attacks. However, these concerns seem relatively minor, overall. The authors should make sure to incorporate the various writing/presentation-related suggestions made by reviewers, especially reviewer EHi7.
Decision: Accept (spotlight)