Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Risks
Authors: Yixuan Xu
Antoine Bosselut
Imanol Schlag
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
Verbatim Memorization
Pretraining
Law
Copyright
Primary area: deep_learning

Abstract: Large language models are known to memorize parts of their training data, posing risk of copyright violations. To systematically examine this risk, we pretrain language models (1B/3B/8B) from scratch on 83B tokens, mixing web-scale data with public domain books used to simulate copyrighted content at controlled frequencies at lengths at least ten times longer than prior work. We thereby identified the offset effect, a phenomenon characterized by two key findings: (1) verbatim memorization is most strongly triggered by short prefixes drawn from the beginning of the context window, with memorization decreasing counterintuitively as prefix length increases; and (2) a sharp decline in verbatim recall when prefix begins offset from the initial tokens of the context window. We attribute this to positional fragility: models rely disproportionately on the earliest tokens in their context window as retrieval anchors, making them sensitive to even slight shifts. We further observe that when the model fails to retrieve memorized content, it often produces degenerated text. Leveraging these findings, we show that shifting sensitive data deeper into the context window suppresses both extractable memorization and degeneration. Our results suggest that positional offset is a critical and previously overlooked axis for evaluating memorization risks, since prior work implicitly assumed uniformity by probing only from the beginning of documents or training sequences.

Positional Fragility in LLMs: How Offset Effects
Reshape Our Understanding of Memorization Risks
Yixuan Xul.2
Antoine Bosselut Imanol Schlag'
'ETH AI Center, ETH Zürich
School of Computer and Communication Sciences, EPFL
firstname lastname@inf www.mmeeiiffee......mmmm ch
antoine www.sbattsspp..ccccmmm
Alvorecer721/PPstttonnl Frap://PoossttooaaaFaaiiiilty LLM
Abstract
Large language models are known to memorize parts of their training data, posing risk of copyright violations. To systematically examine this risk, pretrain lan- guage models (1B/3B/8B) from scratch on 83B tokens, mixing web-scale data with public domain books used to simulate copyrighted content at controlled frequencies at lengths at least ten times longer than prior work. We thereby identified the offset effect, phenomenon characterized by two key findings: (I) verbatim memoriza- tion is most strongly triggered by short prefixes drawn from the beginning of the context window, with memorization decreasing counterintuitively as prefix length increases; and (2) a sharp decline in verbatim recall when prefix begins offset from the initial tokens of the context window. We attribute this to positional fragility: models rely disproportionately on the carliest tokens in their context window as re- trieval anchors, making them sensitive even slight shifts. We further observe that when the model fails to retrieve memorized content, it often produces degenerated text. Leveraging these findings, we show that shifting sensitive data deeper into the context window suppresses both extractable memorization and degeneration. Our results suggest that positional offset a critical and previously overlooked axis for evaluating memorization risks, since prior work implicitly assumed uniformity by probing only from the beginning of documents or training sequences.
Introduction
Large language models (LLMs) have demonstrated the capacity to reproduce substantial portions of their training data verbatim, raising serious concerns about copyright violations [Chang et al.] 2023
Karamolegkou et al. 2023 and privacy breaches [Huang et al. 2022 This capability has fueled increasing legal challenges, with high-profile cases, such as the New York Times lawsuit against OpenAI, highlighting the potential misuse of proprietary content (Freeman et al.) 2024]. Industry leaders have not denied these risks, and instead called for regulatory exemptions that would allow LLM training on copyright-protecte how LLMs memorize and regurgitate legally sensitive content is no longer niche technical concern, but a central requirement for the responsible and legal development of LLMs [Rosenthal and Veraldi 2025) and informed policymaking on this subject.
material [Brodkin] 2025). Against this backdrop, understanding
Existing work has begun study memorization in LLMs, but key gaps remain. First, most mitigation strategies are applied post-training via fine-tuning, decoding constraints, or unlearning methods. However, recent studies show that these approaches can be bypassed (Park et al. 2024 Nasr et al.) 2025), indicating that they are better suited as complementary defenses rather than standalone solutions. This highlights the need to address memorization proactively during pretraining, where
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

retention first happens. Second, prior analyses typically focus on short suffixes prompted from the beginning training sequences, implicitly assuming that memorization behavior is uniform across the context window. This methodological choice overlooks the reality that user inputs may appear at arbitrary offsets in the content window, and that memorization may vary significantly with position.
In this work, we revisit memorization from a legally motivated perspective, focusing on long-form text segments drawn from published books (via Project Gutenberg) simulate high-risk copyright scenarios. We pretrain LLaMA-style models (IB, 3B, and 8B) from scratch on 83B tokens, combining curated book data with web-scale educational corpora. This setup grants precise control over training dynamics (such as model scale and batch size), data composition (such as exposure frequency and placement of sensitive content), and evaluation configuration (such as prefix length and position within the context window). Compared to the most extensive memorization analysis to date by Hans al. 2025), which also targets pretraining -ime mitigation, our study employs full-context sequences four times longer and evaluates suffixes at least ten times longer. Together, these design choices enable an unprecedented, comprehensive, and fine-gras ane-grained memorization analysis.
0.8
0.8
Rouge-L
MAUVE
0.6
0.6
IB Dense
1B Sparse
0.4
0.4
3B Sparse
0.2
8B Sparse
2018
0
256
512
inz
2008
Offset
Offset
(a) Rouge-L scores by offset
(b) MAUVE scores by offset
Figure 1: Positional fragility in LLMs measured with two complementary metrics: (a) Rouge-L scores quantify verbatim memorization, demonstrating sharp decline as offset increases; (b) MAUVE scores capture overall language coherence, revealing text degeneration with increasing offset. The dense model results are derived from a 1B model trained exclusively on 10K Gutenberg sequences seen 80 times each, while sparse model results are derived specifically from the frequency- bucket of our FM-Probe methodology. All experiments used 500-token prefixes and evaluated on 500-token suffixes. Verbatim memorization and language coherence both degrade sharply as prefix offset increases, except for language coherence in 1B Dense case, revealing positional fragility.
Our main contributions are:
We identify the offset effects: verbatim memorization most effectively triggered by short prefixes at the beginning of the context window. Memorization recall drops sharply as prefixes are shifted further into the sequence. We attribute offset effects to positional fragility: LLMSs are highly sensitive positional shifts as they disproportionately rely early tokens as retrieval anchors ($ 4).
We show that memorization failures cause degeneration. Under offset or low-frequency conditions, the model produces repetitive and semantically incoherent output, establishing link between memorization breakdown and text degeneration (5.)
We show that positional fragility can be leveraged mitigate memorization: shifting sensitive sequences deeper into the context window significantly reduces extractability
without compromising general performance ($ (
We find that batch size influences retention: smaller batches with more frequent updates modestly reduce memorization under identical compute budgets. ($ (8)).
We reproduce and scale previous studies in more controlled and legally relevant settings, using longer sequences better reflect the risks of copyright in the real world (Appendix

---

2 Problem Setup
To facilitate the discussion around verbatim memorization we introduce the following nomenclature:
Text Segment: A sequence of 8192 tokens from the training corpus, beginning with a BOD token, that fits exactly the context window of the model. Prefix x: The input text sequence provided to the model probe memorization, represented as æ (f1,,.,, Fn) with n tokens extracted consecutively from the training corpus.
Document (8, 191 loke 16)
Profi
Buffx
500
Offee:
Profi
Bulli
True Suffix The text sequence that immediately follows the prefix in the original text segment, rep- resented as s ($1....,5 with m consecutive tokens.
Officet
205
500
500
Swapped Part (1,000 tokens)
Hetain wed Part (1, lockeee)
Generated Suffix y: The text sequence pro- duced by the model given prefix æ, where y (yl,..., ), that matches the length of the ground truth suffix m.
Offeet
Frufi
Buffix
500
Figure 2: Illustration of prefix-suftix extraction with varying offsets from the document start in Sparse Gutenberg, and from the retained part in Swapped Gutenberg. Not to scale.
Offset o: The number of tokens to skip in the text token
segment from the reginning-of-docunee BOD before starting prefix extraction.
Quantifying Verbatim Memorization We follow Nasr et al. 2025 's definition of verbatim
memorization, providing a model with a prefix x from training data and measuring how closely its greedy continuation y matches the true continuation Prior work predominantly focuses short
prefix probing, using prefixes of size 20 [Carlini et al., 2021]. 32 [Biderman et al. [023]] Huang
et al. 2024. Duan al. 2025 J. 50 (Karamolegkou et al. 2023 Duan al. [2055] Ippolito al.
2023
50-500 [Carlini et al. 2023). and 1998 [Hans et al., 2025 Suffix lengths are likewise short,
typically ranging from 25 to 50 tokens. Nearly all evaluations extract prefixes from the beginning of
the document, with only Carlini et al. [202T] sampling prefixes from arbitrary positions, but they only report the highest-recall case. To address these limitations, our evaluation spans prefix lengths from 50 to 5000 tokens, sampled at offsets ranging from to 2048, with suffixes between 50 and 8000 tokens. We report similarity using three metrics that capture increasing levels of verbatim reproduction: Rouge-L (longest common subsequence relative to reference length) [Lin 2004), LCCS (longest common contiguous substring) [Freeman al., 2024), and EM (exact match of the entire suffix).
Measuring Text Degeneration We evaluate output quality using three complementary metrics. Type-token ratio (TTR) measures lexical diversity as the fraction of unique tokens in the generated suf- fix [ettunen 2014). MAUVE quantifies distributional similarity between model outputs and human- written text Pillutla et al. 2023]. As a baseline, unrelated suffix pairs yield a ROUGE-L of approx- imately 0.18. We also report perplexity
the reference suffix to assess how well the model predicts the ground-trutt -ruth continuation. Given a prefix x = (I1, (I1..... Fn, and its true suffix s = ($1.... sm).
perplexity is defined as PPL(s x) exp (- 1 >m= log P(st x. 8<t)). Intuitively, perplexity m
reflects how many next-token candidates the model finds plausible at each step [Jelinek et al. 2005 For downstream performance evaluation, we utilize 1m-eval- seaa-haad harness [Gao et al. 2024).
3 Design
Data & Model We train a decoder-only transformer model following the Llama architecture
Grattafiori et al. [0224), with complete architectural and training details provided in Appendix A.2
Our training corpus combines two complementary sources: (1) Project Gutenberg" [T] used to simulate
potential copyright infringement via literary texts; and (2) Fineweb-Fdu (Penedo et al., [024]] curated dataset of high-quality educational web content typical modern LLM pipelines. A 13-gram contamination analysis confirmed negligible 0.34% overlap between the two datasets, ensuring observed memorization attributable to our experimental controls.

---

Dense Gutenberg: Extreme Memorization Study Inspired by the setup of Hans et al. [2025], we establish an extreme memorization scenario, where models are trained exclusively on our curated subset of Project Gutenberg text segments comprising 10,000 sequences of 8, 8,192 tokens each. Each epoch constitutes a complete traversal of the entire corpus, with model checkpoints saved at logarithmically spaced intervals (powers of 2) as well at the final 80th epoch. Since each sequence appears exactly once per epoch, the checkpoint number directly corresponds to the number of exposures to each training sequence, providing direct measure of how repetition influences memorization behavior.
Realistic Copyright Memorization Simulation To create a more realistic scenario, designed mixed corpus training configuration where Project Gutenberg excerpts represent just 2% (1.74B tokens) of the training data, with the remaining 98% (81.82B tokens) sourced from FineWeb-Edu. For analyzing how exposure frequency affects memorization, developed Frequency-Vann Varied Memorization Probe Buckets (FM-Probes), with each bucket containing 500 distinct and randomly sampled Gutenberg sequences that appear precisely controlled frequencies ranging from 1 to 128 repetitions. By performing single training pass through this carefully structured corpus, our FM-probe methodology enables concurrent evaluation of memorization patterns across multiple exposure levels without requiring separate training runs.
Sparse Gutenberg:
Pretraining Factors Within the Sparse Gutenberg framework, we systematically investigate several key factors that potentially influence memorization dynamics during pretraining. First, we examine batch size variations (0.5M, IM, 1.5M and 2M tokens) to assess how optimization granularity affects memorization tendencies. Second, we analyze model scale effects by comparing memorization patterns across architecturally consistent models of IB, 3B, and 8B parameters. Third, we evaluate how exposure frequency impacts memorization propensity through our FM-Probe methodology.
4 Offset: The Missing Dimension in Memorization Evaluation
Despite increasing attention memorization risks in LLMs, most literature evaluates memorization using prefixes extracted from the beginning of documents (Hans et al., [2025] Carlini et al.] 2023 Kiyomaru al. 2024]. However, memorized content can appear anywhere within a document not just at the start. At the same time, little attention has been paid to how the absolute position of the prefix within the model's context window-the offset-affects memorization behavior. In this section, we show that both verbatim memorization and language generation quality are highly sensitive to offset, and whether longer prefixes increase verbatim recall itself -fesee-dependent.
4.1 Shorter Prefixes Outperform Longer Ones in Triggering Memorization
Rouge-L
0.4
Offict
0.2
Offir 64
Otise 128
8
+
Prefix Length (tokens)
Prefix Length (tokens)
Prefix Length (tokens)
(a) LLaMA IB
(b) LLaMA 3B
(c) LLaMA 8B
Figure Impact prefix length on positional fragility across LLaMA models (IB, 3B, 8B), measured by Rouge-L scores on frequency- 128 bucket with 500-token suffixes under Sparse Gutenberg setup. With zero offset (blue), smaller models show memorization degradation as prefix length increases, while the 8B model maintains high scores. At non-zero offsets (red, green), all models require longer prefixes to trigger substantial memorization, with the model showing slightly higher memorization than the 8B model at offsets 64 and 128, indicating larger models do not worsen positional fragility.

---

While previous research suggests that longer prefixes can enhance verbatim recall, we find that this observation incomplete and potentially misleading. Our results reveal more efficient condition for triggering memorization: using short prefixes that align with the beginning of the context window during training. As shown in Figure g at offset short prefixes yield the highest recall, particularly in smaller models, whereas longer prefixes and longer offsets reduce extractable memorization.
Note that this result does not contradict previous work. As we show in Table [ with a fixed prefix of 50 tokens, we can induce perfect reproduction of suffixes of up to 8,000 tokens long, especially when the target sequence is frequently encountered during training. However, such an evaluation does not take into account larger prefixes and non-zero offsets.
Table 1: Suffix length and sample counts of perfect matching by model size and repetition frequeney at offset O, given a fixed 50-token prefix. Frequencies under 8 showed no verbatim memorization over 50 tokens and are therefore omitted.
LLaMA IB
LLaMA 3B
LLaMA 8B
Freq.
Length
Count
Length
Count
Length
Count
50
2
16
50
500
4
5,000
2
32
3,000
7,000
8,000
46
5,000
7,000
8,000
131
128
7,000
8,000
3
8,000
248
With non-zero offsets, longer prefixes substantially improve memorization, suggesting that they could partially compensate for positive offsets. However, as illustrated by the red and green lines in Figure 3 even with extended prefix length, memorization at disadvantaged positions never fully matches the level observed offset O. Moreover, the 3B model achieves higher verbatim recall at offsets 64 and 128 compared to the 1B model, but memorization plateaus from 3B to 8B, indicating that larger models do not mitigate positional fragiliiy and suffer similarly suboptimal positions.
This inversion of optimal prefix length under offset conditions highlights a key property of verbatim memorization in realistic settings: it is strongly tied to positional cues, especially the beginning of the context window. Our findings align with the "attention sink mechanism discovered by Xiao et al. [222).. where initial tokens disproportionately influence attention distribution and serve as computational anchors. Nasr et al. [2025) has also shown that the token that represents the document boundary, such as ond-of--ocummet (EOD) marker, can trigger memorization (in case BOD). This raises a key question: is the verhatim memorization driven by the positional role of the initial tokens in general or by the special representational status of the BOD token itself? To disentangle these two factors, we design controlled ablation experiments that isolate the BOD token's contribution to verbatim recall.
4.2 BOD or Initial Tokens: An Ablation Study
We isolate the impact of the BOD token by retraining 1B model with its attention masked, thereby effectively removing it from the input. In this setting. if memorization persists, it suggests that positional cues alone sufficient; if it degrades, the BOD token likely serves dedicated retrieval anchor. To explore whether this change interacts with positional fragility, we evaluated memorization performance at three context- offsets: 0, 50, and 100 tokens.
Figure 4 illustrates the impact of masking the BOD token on the model's memorization capability at the IB scale. At offset 0 (left panel), where the prefix begins the start of the context window, masking the BOD token yields negligible differences in verbatim recall. This suggests that the initial textual content alone can serve an effective cue for memorization without explicit BOD.
Interestingly, masking the BOD token leads to moderately higher verbatim recall in tigh-frequency buckets under offset conditions. At 50-token offset, the BOD-masked model achieves up to 8% higher Rouge-L scores for prefixes between 750-2000 tokens equivae-q to roughly 40 additional tokens recalled verbatim. This suggests that, without explicit BOD, the model leans more on local context, making it slightly more robust when recalling sequences from non-initial positions. Nonetheless, it continues to exhibit positional fragility (see Appendix D.I
5

---

Difference (Suffix 500 Offset 0)
Difference (Suffix 500 Offset 50)
Difference (Suffix 500 offset 100)
494 104--402--00 -005 -00 <ut- 603 -002 -00 -00
-000 -022 -00 -00 -003 800 1911 A03 2012
20000 -00 -30 eut 482 -200 2000 -000 .03
9.23
0.10
630
-a0 saoe 682- 582 1602
192 1a0s s08 sood -92 e32 004
-005 v00 v00 ses -68s 1200 -200 1e1
d. 000 084--00 -009
-003
691
-000 -004
-0.00
606
easas 2000 -20 cel 602
3i .00 wdd8 <oce 031 -800 .002
-0,00 -00 -20
680 -082 .003 602
tas 421--00 asa --23- -422
21 +1 0044
995
029 -800
R. --00 -00 602 602 .05 685
-036
-133
-0.2
-115
2 664 -c9 203 -314 422- 6o4 200 -015
-017
R -11002 005
(044
100
800
073
150
180
-033
B -000 .004 647 049 69 221 0012 -007 -30
359
sake zélo 3089 acio s50e profix
85 160 2050 3800 eloo
380 75 3003 zolo soèe aboe sobe
profx
prefix
Figure 4: Effect of isolating the BOD token the memorization capability of the 1B model measured by Rouge-L scores, computed over 500-token suffixes conditioned on varying prefix lengths (50-5000 tokens). Subplots from left to right correspond context window offsets of O, 50, and 100.
To ensure that this improvement in recall is not simply byproduct of degraded generation quality-a symptom of degeneration discussed in $ longer prefix lengths (1000-5000 tokens), corresponding We also assessed downstream performance (see Table in the appendix). In both cases, the BOD-masked model performed on par with the baseline, indicating that increased recall does not come at the cost of linguistic ability.
- we evaluated the model using the MAUVE metric at
regions with increased memorization.
Memorization: Under-Explored Cause of Text Degeneration
Our experiments reveal a connection between memorization capabilities and text degeneration in language models. While previous research observed repetitive patterns training data as the primary cause of degeneration, highlighting "repetition in, repetition out" phenomenon [Li et al., 2023 our findings suggest that limitations how models encode and retrieve memorized information represent more fundamental and previously unexplored mechanism underlying text degeneration. Rather than attributing repetitive outputs solely to repetitive training data, we demonstrate that deficiencies in memorization also lead to text degeneration.
5.1 Context Window Offsets Reduces Output Text Quality
We find that the position prefixes within the model's context window during pretraining influences the quality of the generated suffixes. As shown in Fig. [ in the dense Gutenberg setting, where the model is trained on 10K sequences for 80 epochs, we observe that verbatim memorization effectively vanishes, while language modelling capabilities remain stable at larger offsets. In contrast, under the sparse Gutenberg setting with FM-probe bucketing, the model's output degrades in both lexical diversity and language coherence. An example is shown in Fig. 10
This complements the "repetition in, repetition out" hypothesis by Li al. (2023) Their work specifically revealed strong correlation between the repetition rate of 2-gram in training data and the repetitive generation in the model's output. Nevertheless, our results suggest that text degeneration may not be solely driven by repetition in training data.
5.2 Lower Exposure Frequency Reduces Output Text Quality
In addition context window offsets, the training frequency of sequence during training indepen- dently influences lexical diversity and language coherence. As illustrated in Fig. [5] sequences seen less frequently exhibit pronounced degradation in the model's linguistic capabilities across all model scales (Examples are shown in TT). However, larger models demonstrate greater resilience, achieving higher TTR and MAUVE scores than smaller counterparts, particularly at low frequencies, indicating that model scale modestly enhances linguistic capability.

---

0.6
0.5
0.8
0.4
MAUVE
TTR
0.6
0.3
1B Sparse Sparse
0.4
0.2
0.2
0.1
8B Sparse
8
0
4
8
32
128
Frequency
Frequency
(a) TTR scores by frequency
(b) MAUVE scores by frequency
Figure 5: Impact exposure frequency on text quality metries for Llama models of different sizes, measured with two complementary metries: (a) TTR scores quantify lexical diversity, showing improvement as training frequency increases; the ground truth TTR lies the range 0.535-0.541:(b) MAUVE scores capture overall language quality and coherence, reaching near- car--pefect scores at high frequencies. All experiments used 500-token prefixes and were evaluated on 500-token suffiees with zero offset.
5.3 Insufficient Memory Retrieval: contributing Factor Text Degeneration
Our empirical findings from 1.1 and $5. converge nuanced understanding of text degeneration language models. Specifically, they reveal that imbalances training data frequency can lead to distinct generation issues. High-frreuencyy exposure often results in verbatim memorization, where the model reproduces training data verbatim. Conversely, low-frequency exposure does not necessarily enhance diversity; instead, it can exacerbate degeneration, manifesting repetitions or thematic loops. This suggests link between limited memory retrieval capabilities and degenerative text patterns.
The analysis of context window offsets further supports this relationship. As models attempt to continue text from distant positions, they tend to fall into thematic looping. This position-dependent degradation in linguistic ability supports that insufficient memory retrieval can cause text degeneration, particularly when memorisation targets must compete with diverse training data. This connection between retrieval limitations and text degeneration warrants further investigation in future work.
Probing the Robustness of Positional Fragility
Our primary analysis revealed positional fragility using greedy decoding on models trained from scratch. To ensure that this phenomenon reflects an inherent model behavior rather than an experi- mental artifact, further examine its robustness across decoding strategies and training paradigms.
Robustness Across Decoding Strategies To evaluate the robustness of positional fragility under alternative decoding configurations, we re-ran inference on the repetition- FM-Probe bucket using our 8B model under the Sparse Gutenberg setting with two additional decoding methods: (1) beam search (k=5), reporting the top-ranked hypothesis, and (2) nucleus sampling (top-p=0.9). Table 2 shows that beam search, deterministic probabilttymmxximiing strategy, achieves nearly perfect memorization shallow offsets (e.g.. ROUGE-L= 0.999 at offset 0). However, when recall fails at deeper offsets, it exhibits the most severe degeneration, supporting our claim that failed retrieval leads incoherent outputs. In contrast, nucleus sampling yields lower memorization at shallow offsets but shows much milder decline in lexical diversity at large offsets (e.g., TTR = 0.460 at offset 2048), suggesting that decoding strategy influences the severity of degeneration while leaving the offfset-dependent pattern of positional fragility unchanged.
Persistence Under Continued Pretraining We further examine whether positional fragility persists under continued pretraining paradigm. Starting from the official Llama 3.2 1B model pretrained on 9T tokens, We continued pretraining using otherwise identical settings to the Sparse Gutenberg configuration, but with smaller learning rate 1 x 10-5, As shown in he certinued-rettrineed 1B

---

model exhibits slightly stronger memorization (higher ROUGE-L) and more diverse generations (higher TTR) than the nom--ccatch IB model, while preserving the same offset-dependent decay pattern, confirming that positional fragility holds across training paradigms.
Table 2: Robustness of positional fragility across decoding methods and training paradigms. Top: Decoding strategies on the 8B model. Bottom: 1B continued pretrained model with greedy decoding. All evaluations conducted on the repetition- 128 FM-Probe bucket using 500-token prefixes to generate 500-token suffixes. Across all settings, memorization (ROUGE-L) consistently declines with increasing offset, confirming that positional fragility persists regardless of decoding method or training paradigm.
Decoding Strategies (8B Model)
ROUGE-L +
TTR
Offset
Greedy
Beam
Nucleus
Greedy
Beam
Nucleus
0.965
0.999
0.877
0.538
0.541
0.536
0.864
0.951
0.701
0,519
0.524
0,523
32
0.652
0.800
0,495
0,474
0.469
0,503
128
0.330
0.507
0.247
0.391
0.361
0.477
512
0.208
0.244
0.186
0.305
0.237
0,461
2048
0.192
0.186
0.180
0.279
0,196
0.460
Training Paradigms (1B Model)
ROUGE-L 4
TTR t
Offset
Scratch
Continue
Scratch
Continuc
0.744
0.669
0.521
0.521
8
0,463
0.569
0,491
0.498
0,346
0.426
0.465
0,482
128
0.219
0.249
0.411
0.437
0.191
0,202
0.345
0.364
2048
0.185
0.188
0.293
0.340
7 Leveraging Positional Fragility to Mitigate Verbatim Memorization
To demonstrate that positional fragility can be leveraged to mitigate verbatim memorization, we designed a Proof-of-concept (POC) experiment which we refer to as Swapped Gutenberg. For each sequence in our FM-Probe buckets, we replace the first 4,000 tokens (the swapped part) with tokens from randomly selected Project Gutenberg excerpts within our 10,000-sequence corpus. Thus, the initial segment becomes diverse and contextually unrelated, while the remainder (the retained part) maintains controlled repetition frequencies (1 to 256 repetitions). Each sequence includes only one ini- tial BOS token, with additional special tokens between swapped and retained segments (sce Fig, 2 We train 1B and 8B models and evaluate verbatim memorization using prefixes from the retained part.
sshows that the principle demonstrated in this POC works surprisingly well regardless of model scale, keeping verbatim recall consistently low even at the highest exposure frequency. ROUGE-L and LCCS scores remain near the baseline similarity between unrelated texts across all frequencies, indicating little to no extractable memorization. As repetition increases, perplexity steadily declines, particularly in the 8B model, yet without corresponding gains in verbatim overlap. Both TTR and MAUVE remain stable, with the 8B model producing more coherent and lexically rich outputs. These results confirm that shifting sensitive content away from the context window's 's beginning is simple but effective principle for mitagation at all scales.
Our POC mitigation strategy also neutralizes the offset effects entirely. Recall that this effect manifests in two ways: (1) shorter prefixes are most effective at triggering memorization when taken from the beginning of the context window, and (2) even short prefixes lose their efficacy when shifted to later positions, though longer prefixes can partially compensate. However, both effects disappear when the memorization target is displaced 4,000 tokens deep, as in the Swapped Gutenberg setup. Table shows that increasing prefix length no longer improves memorization, and Table TO reveals consistent recall across offsets from 0 to 2048 tokens.

---

Table Comparison of text generation metrics under Swapped and Sparse Gutenberg settings. Each row reports performance on 500-token suffixes generated from 500-token prefixes at offset O. Selected exposure frequencies (1, 8, 64, 128) are shown to highlight transitions memorization behavior. Rows 1-4 correspond to Swapped Gutenberg, where prefixes are offset relative to the retained part of the sequence; rows 5-8 correspond to standard Sparse Gutenberg. Full results for each setting and
frequency range are provided in Table
[1 in the Appendix.
Rouge-I.
LCCS}
Perplexity
TTR
MAUVE1
Freq.
1B
8B
1B
8B
1B
8B
1B
8B
IB
8B
1
0.178
0.176
0.009
0.009
40.793
82,206
0.348
0.467
0.779
0.970
8
0.180
0.179
0.008
0.009
30.621
12.841
0.350
0.467
0.805
0.934
64
0.182
0.184
0.009
0.011
16.356
3.670
0.377
0.469
0.851
0.905
128
0.181
0.181
0.009
0.011
16.017
3,636
0.377
0.468
0.911
0.958
1
0.181
0.185
0.008
0.009
26.036
16.089
0.225
0.245
0.231
0.418
8
0.183
0.191
0.008
0.016
12.698
3,430
0.231
0.384
0.312
0.898
64
0.522
0.888
0.415
0.858
1.125
1.023
0.497
0.530
0.985
0.998
128
0.744
0.965
0.682
0.951
1.051
1,012
0.521
0.538
0.984
1.000
This disappearance results from two key design choices: the swapped segment is randomly sampled across sequences that break fixed positional associations; and the memorization target is embedded far from the model's typical retrieval anchor near the beginning of the context window.
Furthermore, this method not only suppresses memorization but we also find that it improves generation quality. Unlike the sparse Gutenberg setting, where prompting from offset positions leads to lexical degradation and thematic looping. Swapped Gutenberg maintains high coherence and diversity across all conditions. As shown in Table MAUVE and TTR scores remain stable and high, in stark contrast the quality collapse observed in Sparse Gutenberg setting (also Figure lb 5). These findings highlight central implication of our study: by decoupling sensitive content from early-context anchors, it is possible to suppress extractable memorization while preserving generation quality and downstream task performance (see Table across model scales and training regimes. This provides compelling justification for future research into developing more practical, production-ready mitigation techniques based on this principle. Examples are shown in Fig. 12
8 Batch Size Impact
Our experiments reveal clear advantage to pretraining language models with smaller batch sizes, which necessitates more iterations given fixed compute budget. This advantage manifests reduced verbatim recall (Fig. 6aa improved language coherence (Fig. (b). and enhanced performance across most downstream benchmarks (Appendix D.2). Notably, these improvements emerge despite main- taining both a constant proportion of Gutenberg sequences in each batch and consistent total training tokens across all experiments. To the best of our knowledge, this has not been shown in prior work.
Related Work
Factors Known to Afffect Memorization Several factors are known to influence model's 's tendency
to memorize and reproduce training data verbatim. (1) Model scale: larger models tend to memorize more due their greater capacity to fit specific examples [Carlini et al. 202 Lesci et al. 2024 (2) Repetition frequency: memorization probability increases roughly log- linearly with the number of times a sequence appears during training [Carlini et al., [023);; we confirm both effects in our setting (Appendix C.I). (3) Complexity: complex sequences seen only once can be latently memorized and re- trieved via small parameter perturbations [Duan et al., 2025). (4) Timing of exposure: sequences seen later in training are more likely to be retained than those encountered earlier [Kiyomaru et al. 2024 Lesci et al.) 2024). (5) Prefix length: while longer prefixes improve recall in prior work [Carlini et 2023
find this effect depends heavily on the prefix's 's position within the context window (S

---

0.5MT
0.8
0.8
LOMT
Rouge-L
MAUVE
1.5MT
0.6
0.6
2.0MT
0.4
0.4
0.2
0.2
0
2
4
6
128
0
2
32
64
128
Repetitions
Repetitions
(a) Rouge-L scores by repetition frequency
(b) MAUVE scores by repetition frequency
Figure 6: Impact of batch size on verbatim memorization and language coherence for 1B models, as measured by Rouge-I. (left) and MAUVE scores (right) across repetition frequencies. Models were trained with batch sizes ranging from 0.5MT to 2.0MT, maintaining constant total training tokens. As batch size increases, observe enhanced language coherence (MAUVE) at low repetition frequencies, while verbatim memorization (Rouge-L) increases most significantly for sequences seen more than 32 times. The gap between smallest and largest batch sizes reaches up to 0.15 in Rouge-L score at high repetition frequencies, corresponding to approximately 90 fewer tokens being memorized verbatim in the 500-token continuations. This suggests smaller batch sizes with more update steps reduce exact memorization while preserving language quality.
Preventing Verbatim Memorization During Pretraining Pretraining- strategies have demonstrated promising results in reducing verbatim memorization without sacrificing model per- formance. One prominent strategy data deduplication, which removes duplicate or near-duplicate sequences from the training corpus. [ee et al. [2022] showed that deduplication substantially reduces memorization while preserving downstream performance, |Kandpal et al.) [022) further demonstrated that it lowers the success rate of extraction attacks. Another promising strategy is the recently proposed Goldfish Loss [Hans et al. 2025), which selectively masks tokens within n-gram windows during training. By consistently omitting these tokens from the loss computation, it prevents the model from learning exact token-to- coen-tt-context mappings, thereby disrupting extractable memorization while retaining language modeling capabilities.
10
Conclusion
We demonstrate that LLMs exhibit pronounced positional bias in their memorization behavior, with verbatim recall most easily triggered by prefixes near the start of the context window. This offset effect not only distorts standard memorization risk assessments but also offers a new mitigation pathway: simply shifting sensitive content deeper into the context suppresses extractable memorization without degrading generation quality. Our findings further reveal that when retrieval fails due to offset insufficient exposure- mmodels often degenerate into repetitive, low-diversity output, linking memorization to generation stability. These insights position offset critical and previously underexplored axis in the study of memorization, and suggest simple, scalable mitigation strategies that complement existing techniques. Limitations of our study and directions for future work are discussed in Appendix
Acknowledgements
This work was conducted during Yixuan Xu's Master thesis at the ETH AI Center, while he Master student at EPFL. Yixuan would like to thank HPC expert Antoni-Joan Solergibert Llaquet for his technical guidance. This work was supported part of the Swiss AI Initiative by grant from the Swiss National Supercomputing Centre (CSCS) under project ID a06 on Alps.
10

---

References
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differentially private BERT. In Yoav Goldberg. Zoritsaa Kozareva, and Yue Zhang, editors, Findings the Association for Computational Linguistics: EMNLP 2022, pages 6481 6491, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.18653/11/0222.inmml ceapingg-emmllp.4... URL https
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jenry Tworek, and Mark Chen. Efficient training of language models to fill in the middle, 2022. URL lattts:///ariiiiie org/
abs abs/2201 14255
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Gregory Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. In Thirty-seventh Conference on Newral Information Processing Systems, 2023, UtL.httts://opeereeee het/Foorumiid-IgQ0vWBRKt
Jon lose openai - urges- trump- either- settle- either-settle-eeeeeeeeee--- al wop.oooopiiiiii deeat-debte--aee or-bbaat--r--aaae or- lose- ai- race- china/
Brodkin. ai
Openai urges
trump:
settle ai copyright https://arstechnt wwtpe//o/eeeeoolcc//////-/-
Either
debate or
race
china.
March 2025. Ars Technica. Accessed: 2025-05-11.
Nicholas Carlini, Florian Tramèr; Fric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song. Úlfar Erlingsson. Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633- 2650. USENIX Association, August 2021. ISBN 978- 1-939133-22444 24-3. URL https://www. usenix. org/
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023. URL URL htt.ttttt///eeeeeiiiiieiee http:p///pppareeeeeeeeee////adda htt.:n/ooooommmddaaattrrr.
Kent Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. Speak, memory: An archaeology of books known to ChaiGPT/GPT-4, In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7312-7327, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18663/1122033.mmm www.mmmmppimnaa-e.mmmmmmm main.453. URL https://acclanthnn
org 2223. emnlp-maiiii..e .aaan...3//
DecpSeck-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang. Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li. Hui Qu, J. L. Cai, Jian Liang. Jianzhong Guo, Jiaqi Ni, Jiashi Li. Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li. Junxiao Song. Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang. Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Yc, Shengfeng Yc, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, Wang. Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li. Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen. Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang. Xin Cheng. Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang. Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He. Ying Tang, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wo, 7. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma. Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu
11

---

Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu. Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025. URL
19437
Sunny Duan, Mikail Khona, Abhiram Iyer, Rylan Schaeffer, and Ila R Fiete. Uncovering latent memories in large language models. In The Thirteenth International Conference on Learning Representarions, 2025. URI
Joshua Freeman, Chloe Rippe, Edoardo Dehenedetti, and Maksym Andriushchenko. Exploring memorization and copyright violation in frontier M...s: A study of the new york times v. openAT 2023 lawsuit. In Neurips Safe Generative Al Workshop 2024, 2024. URL. URI htt.iitpp://ppprrrriieeee http://ww.e..etoommm////eeo/od/a
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang. Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https://zenodo. http:///eeenooooorrrrrrrrr//s//s org//recortss/22666000
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez. Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang. Bobbie Chemn Charlotte Caucheteux, Chaya Nayak. Chioe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arricta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu. Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad. Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanic Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelchi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan. Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer. Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel. Romain Sauvestre, Ronnie Polidoro. Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangam, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgion, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vitor Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia. Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaci, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li. Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yumus, Andrei Lupu. Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annic Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Fisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin I conhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo. Carl
Albiero,
12

---

Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai. Chris Tindal, Christoph Feichtenhofer. Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang. Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Flaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erie-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Scide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpem, Grant Herman, Grigory Sizov, Guangyi, Zhang. Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri. Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog. Igor Tufanow, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat. Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher; Jean- Baaan-tttstte Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jomathan Torres, Josh Ginsburg. Junjie Wang. Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang. Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell. Lei Zhang. Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov. Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Selızer, Michal Valko. Michelle Restrepo, Mihir Patel, Mik Vyatskov. Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usumier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mchta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan. Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy. Shaun Lindsay. Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shugiang Zhang. Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Klcinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam. Yu, Wang, Yu Zhao, Yuchen Hao. Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick. Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama herd of models, 2024. URL httttttss//arxxv http://wwxxxvv...aaaa/ 2407. 21783
Abhimanyu Hans, John Kirchenbauer, Yuxin Wen, Neel Jain, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele, and Tom Goldstein. Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs. In Advances in Neural Information Process- ing Systems, volume 37, pages 24022 24045, 2025. URL htt.tttps: http:///poooeee..../ee/ hewp:eeeeeddddnnnn.nooror.ee.ree neurips. cc/paper_ A Conntrrraac-oofoeeeeteee Conf preneer .tmll
Glad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: Closing the generalization gap in large batch training of neural networks. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguisticx: EMNLP 2022, pages 2038- 2047, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/0022 hwwp////....iinn-eeeeea aeddings-emnll.. wttg//022...iiie wwwdingsnnnnnnn emnlp. 148/
148. URL htttps://aclatthh///
Jing Huang, Diyi Yang, and Christopher Potts. Demystifying verbatim memorization large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings the 2024 Conference on
13

---

Empirical Methods in Natural Language Processing, pages 1071 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v 2w1.244.emllmmmii.5... URL https
//aclanthology.org////
emnlp-maing 598/
Daphne Ippolito, Florian Tramer, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher Choquette Choo, and Nicholas Carlini. Preventing generation of verbatim memorization in language models gives a false sense of privacy. In C. Maria Keet, Hung-Yr Yi Lee, and Sina ZarrieB, editors, Proceedings of the 16th International Natural Language Generation Conference, pages 28-53, Prague, Czechia, September 2023. Association for Computational Linguisties. doi: 10. 11611161120223.332 hwtp:///.....mmmmm.im..... main.3. URL https:// aclanthology
inlg mamlggmain.33 main.3/
R. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker. Perplexity measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Sociery of America, 62($1):563 S63, August 2005, ISSN 0001-4966. doi: 10.1121/1.2011 2..16999.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. Kamalika Chaudhuri. Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 10697- 10707. PMLR, 17 23 Jul 2022. URL https 7/proceedings. .ml. mlr, http://iwww........//eeeeeeeeeee prress/v622/e wwwp///akaka22aaaa.a. html
Antonia Karamolegkou, Jiaang Li. Li Zhou, and Anders Søgaard. Copyright Violations and Large Language Models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7403-7412, Singapore, December 2023. Association for Computational Linguistics. doi: 10.:6663//1120333emmmmmmmiam
Kimmo Kettunen. Can type-token ratio be used to show morphological complexity of languages? J. Quant. Linguist. 21(3):23-245, July 2014.
Hirokazu Kiyomaru, Issa Sugiura, Daisuke Kawahara, and Sadao Kurohashi. A comprehensive analysis of memorization in large language models. In Saad Mahamood, Nguyen Le Minh, and Daphne Ippolito, editors, Proceedings of the 17th International Natural Language Generation Conference, pages 584-596, Tokyo, Japan, September 2024. Association for Computational Linguistics. URL (ttthatt////ccaattooo .rrg/ 2024. inlg main. 457
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Buree Burch,, and Nicholas Carlini. Deduplicating training data makes language models better. In Smaranda Mure- san, Preslav Nakov, and Aline Villavicencio. editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers). pages 8424 8445, Dublin, Ire- land, May 2022. Association for Computational Linguistics. doi: 22211666////222222oooooogg httpp://acoee http:////ccaaanhooo.orrrooo aonno./.oo........1oon... 577/
URL
Pietro Lesci, Clara Meister. Thomas Hofmann, Andreas Vlachos, and Tiago Pimentel. Causal estimation of memorisation profiles. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 15616-15635, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10. 1/.18663///2222.acahh wtt.:/22...--lo......... 834. URL htt.aatpss//cclathhooooo.oo
acl- 834/
Huayang Li, Tian Lan, Zihao Fu, Deng Cai. Lemao Liu, Nigel Collier, Taro Watanabe, and Yixuan Su. Repetition in repetition out: towards understanding neural text degeneration from the data perspective. In Proceedings of the 37th International Conference Neurol Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Chin-Yew Lin. ROUGE: package for automatic evaluation of summaries. Text Summarization Branches Out, pages 81, Barceiona, Spain, July 2004. Association for Computational Linguistics. https! www.aanthoooog..og//OOo0-
Dominic Masters and Carlo Luschi. Revisiting small batch training for deep neural networks, 2018. URL
sstpa://arxiv.org/// www./rgg///baeeeeeeoooooeeeeeeet /ttp.o/////b////..06662
Milad Nasr, Javier Rando, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Chov Florian Tramèr, and Katherine Lec. Scalable extraction of training data from aligned, production language models. In The Thirteenth International Conference on Learning Representations, 2025. httttttts:///////ttttcet http:p:://ppppeeeeeoooo/////////
14

---

Kanghee Park, Jiayu Wang, Taylor Berg-Kirkpatrickk Nadia Polikarpova, and Loris D' Antoni. Grammar-aligned decoding. Tighty- Annual Conference on Neural Information Prucessing Syxtems, 2024. URL
Guilherme Penedo, Hynek Kydlíčck, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Tiiht Conference Neural Information Processing Systems Datasets and Benchmarks Truck, 2024. BtRL:ttt::///// /ttps::// wwwp/eeuummeeeee dot.orummmddnnn000222000
Krishna Pillutla, Lang Liu, John Thickstun, Scan Welleck, Swabha Swayamdipta, Rowan Zellers, Sewoong Oh, Yejin Choi, and Zaid Harchaoui. Mauve scores for generative models: Theory and practice. Journal of Machine Learning Research, 24(356):1-22 2023. URL WtRLhttp:///mml. wttpppppra/vv//22222 0023.0003.000 html
David Rosenthal and Livio Veraldi. Training Al language models with third-party content and data from legal perspective. Jusletter IT, (25-Maerz-20255, 2025.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Mcgatron-Im: Training multi-billio parameter language models using model parallelism, 2020. URL https://arxiv.ot http:/iwwwrggab////eeee 190119999990000....
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. ICLR, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract accurately reflects the main contributions, while the Introduction includes additional findings of practical relevance to the study of verbatim memorization.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Yes we mentioned the limitations of our work in Appendix B.1
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We use larger batch sizes for efficiency, which may introduce minor numerical differences, but these do not affect the conclusions. For fully deterministic outputs, one can run our scripts with fixed seed and a batch size of 1 on the same GPU type (e.g.. GH200).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will release our pretrained checkpoint on HuggingFace, and include inference code, evaluation scripts, and FM-Probe buckets after the rebuttal.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ctaaps..c/public/guue
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We present our experiment settings in $ and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Observed effects exhibit consistently large effect sizes across 10 distinct pretraining runs (including 1B/3B/8B sparse, 1B dense, 1B/8B swap, batch size variations, and BOD ablation) and numerous subsequent evaluations. Formal statistical significance testing for all comparisons was therefore computationally prohibitive given this extensive experimental matrix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, Appendix A.1 details information for the computer resources for each experiment.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research is conducted with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The paper discusses negative societal impacts like privacy breaches and copyright violations stemming from LLM memorization throughout the introduction. The work aims for positive impact by proposing mitigation strategies for these risks.
Justification:
Guidelines:
19

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all datasets and code packages used in the paper. The original sources and papers are referenced in the main text, and the corresponding licenses and terms of use explicitly detailed in the Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were only used to assist with language editing and improving clarity.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httt::///eeerrpssccc////oteeeeee for what should or should not be described.
22

---

A Implementation Details
A.1 Computing Infrastructure
All experiments were conducted on the Alps supercomputer at the Swiss National Supercomputing Centre (CSCS) Each compute node includes four NVIDIA GH200 Grace Hopper Superchips, with each GH200 integrating 72-core ARM-based Grace CPU and a Hopper H100 GPU, interconnected via NVLink-C22 Each GPU provides 96 GB CUDA memory.
Pretraining For most of our pretrain runs, we utilized 15-node configuration. Each GPU processed maximum micro-batch size of sequences, each consisting of 8, 192 tokens. Only one specific experiment, involving the 1B model trained with global batch size of 240 sequences, required a larger setup of 30 nodes. Pretraining of the IB, 3B, and 8B LLaMA models was orchestrated exclusively using Data Parallelism (DP). Given that the 8B model parameters comfortably fit into the memory of single GH200 GPU, neither tensor parallelism nor pipeline parallelism was necessary for efficient training. The throughput and compute for each model scale are summarized in Table
Table 4:
Aggregate pretraining throughput (tokens per second across all 60 GPUs) and total GPU hours required for pretraining each model scale.
Model Scale
Throughput (tokens/s)
GPU Hours
1B
2,308,935
~600
3B
936,003
~1,440
8B
450,017
~3,060
Inference & Evaluation Inference Time: On single node, processing one FM-probe bucket (500-token prefix and 500-token suffix) takes 11. minutes for the 1B model and ~33 minutes for the 8B model, including both text generation and evaluation (Perplexity, LCCS, EM, ROUGE-L, TTR). MAUVE is computed separately and takes 34 minutes per bucket of the same size single GH200.
A.2 Model
We pretrain our models using an adapted version of NVIDIA Megatron-LMM (Shoeybi et al. 2020), incorporating modifications described in the main paper. details the architectural specifica- tions and training hyperparameters of our LLaMA-based models. The tokenizer utilised is the OpenMath2-LLaan Llama3. 1-8E
A.3
Reproducibility
To enhance reproducibility. we fix the global random seed to 42 across all experiments. However, this alone does not ensure strict determinism deep learning workflows. In particular, using different batch sizes may trigger different optimized CUDA kernels, leading to minor numerical variations and non-deterministic outputs despite identical seeds and initialization For guaranteed reproducibility. running inference and downstream evaluation with batch size of the same GPU type (NVIDIA GH200) using our provided scripts yields identical results. Due to time constraints and the scale of experiments, we instead use batch size of 20 for model generation to maximize throughput, and 4 for downstream evaluation.
http:///wyccaaaeo////////aa///ee html
23

---

Table 5: Architectural specifications and training hyperparameters of the LLaMA-based models. Plyper-parametens are the default value from Megatron-Lan L.M
Parameter
1B
3B
8B
Shared Input-Ouiput Projections
Yes
Yes
No
Hidden Size
2,048
3,072
4,096
Intermediate Size
8,192
8,192
14,336
Number of Layers
16
28
32
Number of Attention Heads
32
24
32
Number of Key-Valia Heads
8
8
8
Head Dimension
64
128
128
RoPE scaling factor
32.0
32.0
8.0
RoPE base frequency
500,000
Max Positional Embeddings
131,072
Vocabulary Size
128,256
Warmup Steps
2,000
2,000
200
Initial Learning Rate
10
10-4
2.2 x 10
Weight Decay
0.01
0.01
0.1
Min. Learning Rate
3 x 10-5
3 10-5
2.2 x 10-5
Batch Size
60
Optimizer
Adam
B Limitations and Future Directions
B.1
Limitations
While our proof-of-concept mitigation validates the principle that positional displacement can sup- press extractable memorization, assumes prior knowledge of which training segments are sensitive and thus does not establish directly actionable workflow for large-scale pretraining pipelines. Future work may explore how to operationalize this principle e.g., by combining positional fragility with automated sensitivity detection or data-curation strategies to mitigate memorization at scale in realistic training environments.
Moreover, our corpus, while comprising approximately 83B tokens, remains magnitude smaller than real-world foundation model datasets trained on trillions of tokens. As such, our findings may not fully capture the memorization dynamics present at production scale.
several orders of
Finally, our study lacks formal privacy generalization guarantees (e.g.. differential privacy Anil cct al., (022)) ) and is limited to left-to-right autoregressive models trained from scratch. The empirical scope does extend to all-in-the-midee retrieval-augmented architectures, which may exhibit distinct positional behaviors.
B.2 Future Directions
Our analysis is limited to conventional left-to-right autoregressive models following the LLaMA architecture. We do not study models trained with fill-in-the-Midee (FIM) objectives, such as Ope- nAI Codex [Bavarian et al. 2022) and DeepSeek-V3 [DeepSeck-AL et al. 2025]. which reconstruct missing spans from both preceding and following context. This bidirectional formulation enables flexible anchoring and contextual recombination, and is conceptually similar to our mitigation strategy of displacing sensitive content deeper in the context window. We leave the study of positional fragility and memorization in FIM-trained models future work.
More broadly, remains an open question whether similar sefsett-enssiive memorization behaviors emerge in non-text modalities such code, images, audio, and whether comparable mitigation

---

strategies are effective. Extending offset-aware analysis to these domains may shed light on the generality of positional fragility and its broader implications for memorization and generation quality.
C Reproducing and Extending Prior Work
C.1 Exposure Frequency & Model Size: Intervonnected Memorization Dynamies
As shown in Figure 7al our amalysis reveals significant inverse relationship between model scale and memorization threshold. Following the vertical axis (repetition frequency), the 1B model requires approximately 64 exposures to demonstrate substantial verbatim recall of suffix tokens. This threshold decreases to approximately 32 exposures for the 3B model, and further reduces to only 16 exposures for the 8B model. This systematic pattern suggests that memorization efficiency scales inversely with model size. Extrapolating this trend, a 70B model, without specific memorization mitigation strategies, might memorize sequences after just 2-3 exposures.
LLama 1B Suftix 500 offset O
LLama 38 Suffix S00 Offset O
LLama BB Suftix 500 Offset
--It 379 375 D0 393 100 100 373 D00
--377 1m 379 151
181 3R1 LTT 3R2 393
-1331 LTE 1e0
375 D45 3R1 373 DR 354
AX
-1112 377 313 384 100 100 384
100
3112 .100 100 100 100 312
1.00 184
364
115 301 300
1.0 387
repetttion
117x 377 392 190 381 105 182
3.m. 11th 192
196 145 197
130 199
1118
197
191
194
194
144
R22 236 242 256 358 241 .213 228
222
2
4/6 -92 502 100 ARL A74 A50 436
A34
K28
642
485
722
-20 128
.82
750 -12
530 526 491
379
152
776
734
722
500
3E6
244 344 345 121
.54
533
6:0 335 300 367 315
922 367 545 320 549 579
507
389
250
159
1200
2000
3000
4600 2003
-00
09
309
250
750
5500
2003
3069
4000
5083
56
500
256 750 1000 3900 3003 4300 5900
prwffs
prefts
(a) Standard Loss
LLama
Suffix
Suftix
1-71 277 382 100
282 1U3
1MG
202 150
-.2 .100 382 115 .122 1U1 L1G 1111 J32
1182
.32
110 JU1
.32 112 J82
repettiion
-1172 375 375 382 (104
387
1004
Iepettion
.376 1100 1104 100 1100 333
-.5
1221 1155 389 120 352 355 124 353
0.0
174 374 383 196 384 190
389
330
190 281 200 215 208 200 209
339
201 258 252 256 251 284 247 289
-J 439 255 2966 355 (611 487
204
-02
-26 410 211 201 181
5 S-
411 424 448
007 J48 J40
200 305 280 303
205
200
3314
333
2222
8 8-021
258
251
z00 2ie 1202 1292 1093
-0.0
sia al san uola aes sin preftx
oio zia zan 1860 aol ase oato vol
prefts
preftx
(b) Goldfish Loss
Figure 7: Comparison of verbatim memorization across language models (IB, 3B, and 8B) with varying prefix lengths under Sparse Gutenberg Scenario. The heatmaps display Rouge-L scores computed for 500-token suffixes at offset 0, across different prefix lengths (50-5000 tokens, x-axis) and repetition frequencies (1-128, y-axis).. By comparing Figures 7a and 7b we observe that Goldfish Loss effectively reduces verbatim recall. Nevertheless, despite employing Goldfish Loss, larger models or higher exposure frequencies still exhibit trends toward memorizing training data.
Complementing this frequency effect, the horizontal axis (prefix length) reveals how different-sizze models leverage contextual information. For smaller models (IB and 3B). shorter prefixes of around 50 tokens triggered the strongest verbatim memorization, supporting the hypothesis that initial tokens disproportionately influence memorization patterns. In contrast, the 8B model maintained consistent memorization even with longer prefixes, demonstrating that larger models develop more sophisticated contextual memory mechanisms that operate effectively across varied input lengths.
25

---

These dual observations regarding decreased repetition requirements and enhanced contextual pro- cessing in larger models together demonstrate that scaling fundamentally transforms memorization capabilities. This explains why larger language models demonstrate both improved generalization and increased memorization risk: their enhanced capacity enables them to memorize more efficiently while simultaneously developing more nuanced representations of linguistic patterns. Our results complement to [Carlini et al., 2023].
C.2 Goldfish Loss
We reproduce the extreme and standard memorization scenarios originally proposed by Hans et al. 2025], adapting them to our dense and sparse Gutenberg experiments, respectively. However, our implementation substantially scales up the original design by pretraining from scratch with sequences that are 4x longer and drawn from 100x more documents. We use the dense Gutenberg setting to tune the optimal Goldfish hyperparameters for the sparse Gutenberg, yielding a dropout ratio of 2% (k 50) and context width (h) of 50 tokens.
176
Figure 8: Comparison of Rouge-L distributions between standard and Goldfish Loss LLaMA models of sizes 1B, 3B, and 8B, evaluated with 50-token prefix and 500-token suffix offset O. The histograms display Rouge-L. overlap scores across varying training exposures (1, 8, 32, 64, and 128).
The sparse Gutenberg results confirm that Goldfish Loss effectively mitigates verbatim memorization across all model sizes. As shown in Figure 7b and Figure models trained without memorization mitigation and evaluated with zero positional offset exhibit pronounced verbatim recall abilities that systematically increase with both exposure frequency and model scale. The standard-trained 8B model achieves Rouge-L scores above 0.95 for most test sequences after 128 exposures, while Goldfish Loss models maintain scores consistently below 0.4 even under the most favorable memorization conditions.
Table 6: Downstream Task Performances: Despite training with 2% tokens being dropped, model trained with goldfish loss still shows comparable even superior downstream performance then model trained with standard cross-entropy loss.
Wiki.
Hella.
ARC-
ARC-
PIQA
Wino.
CSQA
MMLU
Model
ppl.
aect
norn?
ace?
nomm
acef
norm"
ace?
acc?
Goldfish IB
18.96
40.44
52.41
32,08
32.25
67.68
63.38
71.11
53.43
19.00
25.10
Standard
15.42
46.13
89.93
38.40
40.44
73.65
68.01
73.99
57.06
21.87
25.69
Goldfish 3B
15.01
46.01
59.89
36.52
40.10
71.84
67.76
73.72
58.41
20.72
25.42
Standard 8B
13.15
49.74
65.74
42.24
45.99
75.97
72.18
75.52
61.88
20.56
24.53
Goldfish 8R
12.44
50.29
66.61
43.00
46.67
76.89
73.78
75.63
62.43
20.39
26.98
26

---

Beyond verbatim memorization examination, the sparse Gutenberg framework reveals that Goldfish Loss effectively prevents verbatim memorization without compromising downstream performance. In fact, models trained with Goldfish Loss maintain competitive performance across all benchmarks, with this advantage becoming more pronounced as model scale increases. The 8B Goldfish model demonstrates this trend most clearly, outperforming its standard counterpart on several tasks as shown in Table 6 This exciting result demonstrating that pretraining-ber memorization mitigation can simultaneously address copyright concerns and enhance model capabilities.
D Additional Results
D.1 BOD or Initial Tokens Ablation
Suffix
L0
377
382
170 179 376 1.1
w.7 1224 120 382 382 382 382
L85
381
382
126 300
380
182
100
38L 380 302
220 LLT L8G 1100 181 382 1879 L0G 1100
125 300 379 180
182
382
382 385
432 LTT 111 194 122
S-379
0.6
L100 172 379 392
183 166 055 385
385 385
31
3-107 312 100 100 185 195 195
223 224 235 255
245
347
335
227
225
-0.4
1175 190 399
214 223
270
251
227
223 213
215 376 194 2014 200 224 221 219 200 214
285
091 255 280 357 315 537 289 277
5-775 180 tne 250 250 250 291 285
-0
022 539
541
424
2 177 199 241 397 A35 475 A42 415 391, 340
2-171 375 200 200 300 355 371 354
1175 300 263
452
417 453
186 207
50 sin a5o
-18
252 152 1000
5000
profix
prefix
profix
Figure 9: Verbatim memorization of the IB model trained with BOD attention masked, under the Sparse Gutenberg setup. Heatmaps show Rouge-L scores for 500-token suffixes across prefix lengths (50-5000 tokens, x-axis) and repetition frequencies (1-128, y-axis), evaluated at offsets O, 50, and 100 (left to right). Memorization degrades consistently as the offset increases, highlighting persistent positional fragility despite masking the BOD token.
Table 7: Downstream task performance of the baseline 1B model compared to its variant trained with BOD attention masked (-BOD).
Wiki
Hella.
ARC-c
ARC=
PIQA
Wino.
CSQA
MMLU
Model
ppll
uce?
norm?
ace
norm
a
nomm?
ace?
acef
ace?
ace?
Baseline
18.71
40.43
52.31
33.36
35.15
68.10
63.13
71.00
53,91 54.54
21.79 20.88
23.65 24.06
BOD
17.89
40.98
52. 10
31.40
33.96
68.06
64.65
71.16
D.2 Batch Size Impact on Downstream Performance
Larger batch sizes correspond to higher perplexity scores on validation data, with the 240 GBS (global batch size) model showing perplexity of 22.86 compared to 18.71 for the 60 GBS model a 22.2% degradation in predictive performance despite seeing identical training content. These same models demonstrate poorer performance as the batch size increases. The 60 GBS model consistently outperforms larger batch variants on reasoning-intenss accuracy on ARC-c compared to just 29.52% for the 240 GBS model, where the bascline random guess is 25%.
benchmarks; for example, it achieves 33.36%
Our findings resonate with established patterns documented in deep learning research regarding batch size impact. When models train with smaller batches, the resulting gradient calculations introduce natural variability between updates. This inherent noise effectively serves as an implicit regularization mechanism, as each weight adjustment follows a slightly different trajectory, preventing overspecialization to training examples. This aligns with [Masters and Luschi 2018) that reduced batch sizes promote more robust generalization through stochastic optimization pathways, ultimately yielding superior performance on evaluation tasks.
27

---

Table Downstream task performance of models trained with varying batch sizes. Gradient updates indicate the number of optimization steps taken during pretraining.
Grad.
Wiki.
Hella.
ARC-c
ARC-e
PIQA
Wino.
CSQA
MMLU
Updates
ppl.
Botm
ace
nurmn
acc
nomm
acc?
acc
acc
ace
GBS
170,005
18.71
40.43
52.31
33.36
35.15
68.10
63.13
71.00
53,91
21.79
23.65
120 GBS
85,002
18.75
39.85
51,02
30.97
33.36
67.80
61.62
70.13
55.88
19.74
23.18
180 GBS
$6,068
20.74
39.62
49.83
32.25
34.22
67.76
62.54
69.86
53.20
19.98
24.78
240
Viewed from another angle, our work shows that fewer parameter updates yield reduced model generalization and increased verbatim memorization. This finding aligns with [Hoffer et al. [017]. which attributes large batch performance deficits primarily to reduced update frequency rather than batch size itself. The considerably fewer parameter updates with larger batches (42,501 at 240 GBS compared to 170,005 60 GBS) may accelerate verbatim memorization while simultaneously compromising broader generalization capabilities.
D.3 Swapped Gutenberg
Table 9; Impact of prefix length on text generation metrics for IB and 8B models, evaluated on 500-token suffixes after 256 exposures at offset O under Sparse Gutenberg setting. The perplexity remains relatively stable across different prefix lengths, indicating that increasing prefix length does not substantially compensate for our method's effectiveness. For reference suffixes, the TTR ranges between 0.535 and 0.541.
Rouge-L.
LCCS}
Perplexity
TTR
MAUVET
Prefix
1B
1B
8B
8B
SB
0.172
0.174
0.007
0.009
15.085
3.620
0.356
0.451
0.809
0.914
500
0.181
0.183
0.009
0.011
14.880
3.530
0.380
0.470
0.907
0.948
1000
0.181
0.183
0.009
0.011
15.682
3.464
0.386
0.472
0.914
0.985
2000
0.181
0.183
0.009
0.012
15.613
3.678
0.392
0,480
0.882
0.947
3000
0.184
0.188
0.010
0.014
15.625
3.844
0.398
0.476
0.934
0.945
28

---

Table 10: Impact of offset position on text generation metrics for IB and 8B models after 256 exposures, evaluated on 500-token suffixes generated from corresponding 500-token prefixes. For reference suffixes, TTR ranges between 0.537 and 0.541. Rouge-L and LCCS scores remain flat across all offsets and hover around the unrelated-ted baseline, confirming that our mitigation neutralizes the offset effect. Generation quality (TTR, MAUVE) remains stable, indicating no
degradation in fluency coherence.
Rouge-L1
LCCS
P
TTR1
MAUVET
Offset
1B
8B
IB
8B
1B
8B
IB
8B
IB
8B
0
0.181
0.183
0.009
0.011
14.880
3.530
0.380
0.470
0.907
0.970
1
0.182
0.182
0.009
0.011
14.862
3.525
0.384
0.471
0.862
0.959
0.182
0.183
0.009
0.011
14.866
3,518
0.384
0.469
0.939
0.947
4
0.181
0.183
0.009
0.011
14.847
3,526
0.379
0.466
0.833
0.959
8
0.181
0.182
0.009
0.011
14.846
3.512
0.381
0,470
0.900
0.968
16
0.182
0.184
0.009
0.012
14.773
3,534
0.379
0.469
0.876
0.963
32
0.182
0.183
0.009
0.011
14.775
3.573
0.373
0.468
0.876
0.951
64
0.180
0.183
0.009
0.011
14.843
3.528
0.373
0.464
0.899
0.971
128
0.181
0.183
0.009
0.011
15.047
3.621
0.378
0.468
0.900
0.975
256
0.181
0.183
0.009
0.011
15.385
3.763
0.375
0.466
0.870
0,948
512
0.181
0.182
0.009
0.011
15.806
3,917
0.368
0.466
0.934
0.937
1024
0.180
0.182
0.008
0.010
16.247
4,278
0.372
0.467
0.874
0.944
2048
0.181
0.183
0.009
0.011
16.619
4.595
0.373
0.466
0.934
0.962
Table 11: Impact of exposure frequency on text generation metrics for IB and 8B models evaluated on 500-token suffixes generated from corresponding 500-token prefixes offset The upper section shows results under Swapped Gutenberg setting. while the lower section shows results under Sparse Gutenberg setting. For reference suffixes, the TTR ranges between 0.535 and 0.541.
Rouge-I.
L.CS$
Perplexity
TTR
MAUVET
Freq.
1B
8B
1B
8B
1B
8B
1B
1B
8B
1
0.178
0.176
0.009
0.009
40.793
82.206
0.348
0.467
0.779
0.970
2
0.179
0.176
0.008
0.008
40.710
63,966
0.349
0.467
0.854
0.969
4
0.178
0.178
0.008
0.009
35.502
34.562
0.350
0.462
0.784
0,910
8
0.180
0.179
0.008
0,009
30.621
12.841
0.350
0.467
0.805
0.934
16
0.181
0.182
0.009
0.010
24.063
5,008
0.355
0.466
0.759
0.965
32
0.181
0.181
0.009
0.011
18.585
3.970
0.374
0.470
0.940
0.958
0.182
0.184
0.009
0.011
16.356
3,670
0.377
0.469
0.851
0.905
128
0.181
0.181
0.009
0.011
16.017
3.636
0.377
0.468
0.911
0,958
256
0.181
0.183
0.009
0.011
14.880
3,530
0.380
0.470
0.907
0.948
1
0.181
0.185
0.008
0.009
26.036
16.089
0.225
0.245
0.231
0.418
2
0.182
0.184
0.008
0.009
24.474
13.902
0.227
0.257
0.276
0.343
4
0.184
0.184
0.008
0.009
19.065
9,047
0.228
0.286
0.322
0.466
8
0.183
0.191
0.008
0.016
12.698
3,430
0.231
0.384
0.312
0.898
16
0.183
0.372
0.010
0.232
5.741
1.229
0.323
0.472
0.737
0.973
0.250
0.710
0.086
0.637
1.565
1.054
0.443
0.512
0.918
0.991
64
0.522
0.888
0.415
0.858
1.125
1.023
0.497
0.530
0.985
0.998
128
0.744
0.965
0.682
0.951
1.051
1.012
0.521
0.538
0.984
1.000
29

---

Table 12: Downstream Task Performances for Swapped Gutenberg experiments.
Model
Wiki.
Hella.
ARC-c
ARC=
PIQA
Wino.
CSQA
MMLU
ppl]
ace
nomm
s00T
nurm
acc
pormi
ace
ace?
ace
Swapped IB 22.31
41.13
52.22
31.83
34.56
68.73
63.51
71.93
55.01
20.56
23.27
Swapped 8B
15.71
49.41
65.48
42.06
44.11
75.88
71.93
75.79
61.09
20.64
25.89
E License for Existing Assets
Project Gutenberg We texts from the Project Gutenberg collection (via HuggingFace), which consists of works in the public domain in the United States. In accordance with Project Gutenberg's terme do not redistribute their format branding. Our use is strictly limited non-commercial academic research, in compliance with Swiss copyright law.
Fineweb-Edu We use the Fineweb-Edu corpus released HuggingFaace which is distributed under the Open Data Commons Attribution License (ODC-By) v1.0. Our use complies with this license and adheres to CommonCrawl's Terms of Use. The dataset used solely for non-commercial academic research, and we do not redistribute the data or any derivatives.
F Demonstrations
30

---

cistttat uau They lastt
stossl shilltsa los They besti baniin
thyy Lid
above iaer
portecs
the sore SuittA anludeee way thyy raarss, deir the 19al
The sone pioccing wicheese why they roumes, thair an an uniut
ard secoue rettngg tabr, lack a
Fattie and mecened bmuas ietrggaain neserne
in. sit stlts
18 bmack
in thy alt stiil
teggle cniee theasre yon't
probakly bals roaer coarss taussed seare
lacase proarpoonall the strid rick sugley supa Lak mecile
Dackse tnoc ccregpe hat, sughty tryals sinde aqgars
Lancate trincct
and sieeaa of Thery alliin aestecti:
sesteell artar then, tu
beartin inttatio sepper to reaive the dosectziing sontto set they Frre making serteal dares rirveeet and sutt costiase
30 PM sought the deceej the seep let they Bacial sertice ttree and fermort mast coclltee
(a) Sequence 0, Offsct O, Frequeney 128
Mithh wight Joar regaars dogges anj
west Hayy sajat Eary Savae regiints alth
le of slaple
They haperig
tark 19e2 cantone
eviled poins lnd pererrosd
CLANEE clack thiir carttry, they fioe catinll esei
there il
uther 1eeir
Teiir aad brcaee
anciar
vire
hoph deesl
Ler cin triln
(b) Sequence 0, Offsee 128, Frequency 128
Tree
Beseration
Terms stat Fryter
one tir
fature srest
cartuct
the
Tim.
circle
seroose LS: Justus canss as aNtokl pape SAL1
(rrend The Greec the 2110
the ti
circies
fally portted for thery the
the
The enectii
Thore
turo tave live ad
(c) Sequence 0, Offiset 2048, Frequency 128
Figure 10: Demonstration of verbatim memorization decay and text degeneration offset increases in the Sparse Gutenherg setting. Each example shows a 500-token suffix generated from a 500-token prefix by 1B model, with the highlighted span indicating the tokens used for ROUGE-L calculation. At offset O, the model reproduces the training suffix nearly perfectly. At offsst 128, memorization is limited the initial few tokens. By offset 2048, only scattered fragments are recalled, and the output shows thematic looping-es e.g. repetitive structures such as "The xxx were the first to..." emerge in place of coherent continuation.
31

---

the bostits comitcn whos our (lmnetiie froards do 1et
fioe shee, Ris
siaets thac there ANE tictle siates
eoueticas with tartie,
(fftectee evcru (TTON.)
outlers ailign ar flaeniaiite qate ofter bouns (4411)
stlery Eat Jody and tol
put
Ciod sueerttl Than
bethee They ioied
aitihin like sndp fuom But
then
perter sertin stcoper bojer corsl lectnn
aocidert JiMesiA
(a) Sequence 289, Offiet 0, Frequency 4
Sepusacs
restinnc
Penepon 1tacte dislaree
Prences
onee resserts erivec trem their village 100 the osllary streer
Fal trouget beapital firtts biss foon vieccee and fes
clldd de inatter Calaca
gitt
bo aveng
Luttte they
serticctlll pvrerr de asble
crieches paing the hospind
the teeisees Nour. ther sistly seveent tee fn leeyd)
the
stotting thonegg
sollesarr alceonec intarii beus thet
Sactes the oftiicll ecoeii cnt
frenc
and
of and aed mededd Cererr The
crrecien psing the hosttet
renururrg the freecs
(b) Sequence 470, Offset O, Frequency 4
Figure 11: Demonstration of verbatim memorization decay and text degeneration for low exposure frequency sequences in the Sparse Gutenberg setting. Each example shows a 500-token suffix generated from a 500-token prefix by IB model, with the highlighted span indicating the tokens used for ROUGE-L calculation. For the generated suffix (right), we show clear evidence of thematic looping and repetitive generations.
32

---

firng cenerrerr dbist fiite
Docia roer reck
seee fondl
strauig Toar the
may
vary trecclatonn
and
thet
teey berse eitegther with
their
ougry proouctad
ony 1iu virr lerving is
ber They
Livi solios tae rosain the rivers and Frestwatter lron
(a) Sequence 346, Offset O, Frequency 256
Het strwee "her misec
recrestatly staatt tisss
olay repetherg wner
encyy
atl
puly dercung saraki, ga airtaciasts satstryy dralia.
taagie rune "pose thecr acosrtail thas taing ttat
this,
thee mech citer Vite ride whar codeeayy estaal this
19th to all iein af Sec, and
tiee
colect
ariane thiik at tast will fears
raalle Hew sattered disd returt art
ecoere Eleo with grout po of cotter cartorlterty
tooll sertotis
start
tot of stoos there clecia
years SeedyyTau
pertec ye de eted oetteed nectiiel actssit. verrcca MMerrra 1911
(b) Sequence 440, Offset 0, Frequency 256
Figure 12: Demonstration of how displacing legally sensitive content deeper into the context window effectively suppresses verbatim memorization while preserving semantic fidelity in the Swapped Gutenberg setting. Each example shows a 500-token suffix generated from a 500-token prefix by 1B model, with the highlighted spans indicating portions used for ROUGE-L caleulation. (a) Both the true suffix (left) and the model-generated suffix (right) describe interactions and encounters with the Esquimaux community upon arrival at Nachvak, discussing their behavior, hospitality, and local surroundings. Despite thematic similarity, the generated text diverges notably in specific narrative details such as dates of arrival (July 16th vs. November 8th) and described observations (presence of drift-ice versus whale sightings)-illustraaiin illustrating effective suppression of verbatim memorization with maintained lexical richness and narrative coherence. (b) The true suffix focuses primarily on parenting advice regarding child behavior, education, and guidelines to prevent inappropriate sexual behavior and maintain chastity, whereas the generated suffix addresses the use of tobacco among children and discusses its potential dangers and harmful effects, highlighting clear thematic divergence and effective memorization suppression.
33


Reviews:
[{"review_id": "6c3FrzzPFK", "reviewer": "Reviewer_skq5", "summary": "This paper focuses on understanding memorization of training data. The authors pretrain three Llama-style transformers on a mix of Project Gutenberg books and web texts, and find (1) short prefixes at the start of the context window lead to highest memorization, whereas shifting this prefix 64-128 tokens deeper can almost eliminate this effect, (2) models that fail to retrieved the memorized sufix will produce nonsensical outputs, and (3) inserting sensitive text further away from the beginning is a simple mitigation strategy.\nThe authors also show several related secondary analyses related to batch sizing and model size.", "questions": "What happens if you move away from greedy sampling?\n\nHow does the size of the special anchoring zone change according to model size?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "This paper analyzes an important downstream impact of an LLM's tendency to focus on early tokens. The experiments are clear to understand, and the problems and solutions presented are helpful. The authors show results across several model sizes, vary the exposure counts, and ablate batch sizing. The paper also alludes to a connection between memorization and model degeneration, which is helpful.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for championing our work and for the thoughtful questions, which helped sharpen our understanding of both the robustness and scalability of positional memorization.\n\n&nbsp;\n\n### Question 1: What happens if you move away from greedy sampling?\n---\nThank you for this important question. To evaluate the robustness of our findings under alternative decoding strategies, we re-ran inference on the repetition-128 FM-Probe bucket using our 8B model with:\n\n- **Beam search** (beam size = 5), reporting the top-ranked output\n- **Nucleus sampling** (top-p = 0.9)\n\nOur core finding—**positional fragility**—remains consistent across all decoding methods. As shown in the table below, `ROUGE-L` scores drop sharply as the prefix is offset deeper into the context, and all methods converge to near-baseline levels (~0.18–0.19) at an offset of 2048 tokens.\n\n&nbsp;\n\n\n\n| Offset | Rouge-L ||| PPL ||| TTR_gen |||\n|--------|---------|-----------|----------|---------|-----------|----------|---------|-----------|----------|\n|        | greedy  | beam  | nucleus  | greedy  | beam  | nucleus  | greedy  | beam  | nucleus  |\n| 0      | 0.965   | 0.999     | 0.877    | 1.030   | 1.011     | 1.164    | 0.538   | 0.541     | 0.536    |\n| 1      | 0.958   | 0.994     | 0.844    | 1.035   | 1.012     | 1.210    | 0.536   | 0.539     | 0.532    |\n| 2      | 0.951   | 0.993     | 0.852    | 1.043   | 1.016     | 1.206    | 0.535   | 0.538     | 0.533    |\n| 4      | 0.913   | 0.978     | 0.778    | 1.073   | 1.021     | 1.329    | 0.530   | 0.533     | 0.530    |\n| 8      | 0.864   | 0.951     | 0.701    | 1.114   | 1.033     | 1.487    | 0.519   | 0.524     | 0.523    |\n| 16     | 0.783   | 0.895     | 0.625    | 1.179   | 1.054     | 1.702    | 0.504   | 0.505     | 0.513    |\n| 32     | 0.652   | 0.800     | 0.495    | 1.298   | 1.090     | 2.123    | 0.474   | 0.469     | 0.503    |\n| 64     | 0.488   | 0.691     | 0.358    | 1.462   | 1.158     | 2.640    | 0.438   | 0.433     | 0.488    |\n| 128    | 0.330   | 0.507     | 0.247    | 1.655   | 1.224     | 3.358    | 0.391   | 0.361     | 0.477    |\n| 256    | 0.255   | 0.354     | 0.203    | 1.773   | 1.282     | 4.022    | 0.346   | 0.297     | 0.466    |\n| 512    | 0.208   | 0.244     | 0.186    | 1.874   | 1.324     | 4.606    | 0.305   | 0.237     | 0.461    |\n| 1024   | 0.197   | 0.202     | 0.186    | 1.906   | 1.352     | 4.930    | 0.284   | 0.206     | 0.458    |\n| 2048   | 0.192   | 0.186     | 0.180    | 1.932   | 1.332     | 5.071    | 0.279   | 0.196     | 0.460    |\n\n&nbsp;\n\nThat said, the decoding strategies differ in their behavior:\n- **Beam search**, a deterministic, probability-maximizing strategy, achieves nearly perfect memorization at shallow offsets (e.g., `ROUGE-L` = 0.999 at offset 0), with the lowest perplexity, reflecting high model confidence. However, when recall fails (at deeper offsets), it produces the most severe degeneration—supporting our claim that failed retrieval leads to incoherent outputs.\n- **Nucleus sampling** exhibits lower memorization (lower `ROUGE-L`) at shallow offsets, but much higher lexical diversity at large offsets (e.g., `TTR_gen` = 0.460 at offset 2048), indicating robustness against degeneration. This confirms that sampling-based methods can better mitigate the adverse effects of failed recall.\n\n> We note that, unlike in the main paper where perplexity measures uncertainty over the ground-truth suffix, the `PPL` here reflects uncertainty over self-generated continuations and thus varies with decoding.\n\n&nbsp;\n\n### Question 2: How does the size of the special anchoring zone change according to model size?\n---\nWe thank the reviewer for this question, which probe the key scaling behaviour of our findings. To investigate, we conducted a fine-grained offset sweep across the first 384 tokens (step size = 16) and applied a `ROUGE-L` threshold of 0.4 to identify the anchoring zone boundary for each model. We found:\n- **1B model**: Anchoring zone ends at offset 16 (ROUGE-L = 0.396)\n- **3B model**: Anchoring zone ends at offset 80 (ROUGE-L = 0.368)\n- **8B model**: Anchoring zone ends at offset 112 (ROUGE-L = 0.376)\n\n&nbsp;\n\n| Offset | Rouge-L (1B / 3B / 8B)     | Ref_PPL (1B / 3B / 8B)     | TTR_gen (1B / 3B / 8B)     |\n|--------|----------------------------|----------------------------|----------------------------|\n| 0      | 0.744 / 0.880 / 0.965      | 1.051 / 1.020 / 1.012      | 0.521 / 0.532 / 0.538      |\n| 16     | 0.396 / 0.762 / 0.783      | 1.095 / 1.038 / 1.074      | 0.485 / 0.516 / 0.504      |\n| 32     | 0.346 / 0.614 / 0.652      | 1.127 / 1.082 / 1.138      | 0.465 / 0.489 / 0.474      |\n| 48     | 0.308 / 0.499 / 0.566      | 1.169 / 1.129 / 1.193      | 0.453 / 0.470 / 0.455      |\n| 64     | 0.282 / 0.447 / 0.488      | 1.203 / 1.175 / 1.237      | 0.447 / 0.451 / 0.438      |\n| 80     | 0.254 / 0.368 / 0.443      | 1.254 / 1.244 / 1.281      | 0.434 / 0.423 / 0.425      |\n| 96     | 0.242 / 0.329 / 0.400      | 1.281 / 1.292 / 1.321      | 0.425 / 0.408 / 0.411      |\n| 112    | 0.228 / 0.303 / 0.376      | 1.323 / 1.318 / 1.374      | 0.417 / 0.405 / 0.398      |\n| 128    | 0.219 / 0.274 / 0.330      | 1.356 / 1.378 / 1.417      | 0.411 / 0.394 / 0.391      |\n| 144    | 0.215 / 0.257 / 0.317      | 1.394 / 1.437 / 1.446      | 0.407 / 0.381 / 0.386      |\n| 160    | 0.210 / 0.241 / 0.296      | 1.428 / 1.485 / 1.472      | 0.405 / 0.367 / 0.374      |\n| 176    | 0.206 / 0.237 / 0.282      | 1.465 / 1.533 / 1.501      | 0.397 / 0.368 / 0.373      |\n| 192    | 0.205 / 0.226 / 0.284      | 1.486 / 1.571 / 1.531      | 0.396 / 0.357 / 0.362      |\n| 208    | 0.204 / 0.218 / 0.265      | 1.520 / 1.598 / 1.563      | 0.394 / 0.358 / 0.358      |\n| 224    | 0.204 / 0.223 / 0.267      | 1.552 / 1.639 / 1.584      | 0.391 / 0.354 / 0.349      |\n| 240    | 0.201 / 0.210 / 0.257      | 1.580 / 1.683 / 1.606      | 0.389 / 0.345 / 0.349      |\n| 256    | 0.200 / 0.214 / 0.255      | 1.609 / 1.723 / 1.634      | 0.383 / 0.343 / 0.346      |\n| 272    | 0.200 / 0.207 / 0.251      | 1.654 / 1.747 / 1.655      | 0.381 / 0.341 / 0.344      |\n| 288    | 0.198 / 0.207 / 0.250      | 1.684 / 1.772 / 1.675      | 0.375 / 0.335 / 0.337      |\n| 304    | 0.198 / 0.209 / 0.239      | 1.724 / 1.811 / 1.697      | 0.378 / 0.335 / 0.337      |\n| 320    | 0.197 / 0.204 / 0.233      | 1.762 / 1.834 / 1.714      | 0.373 / 0.328 / 0.335      |\n| 336    | 0.198 / 0.204 / 0.229      | 1.816 / 1.865 / 1.740      | 0.365 / 0.329 / 0.324      |\n| 352    | 0.196 / 0.205 / 0.229      | 1.832 / 1.888 / 1.757      | 0.365 / 0.326 / 0.329      |\n| 368    | 0.195 / 0.202 / 0.228      | 1.874 / 1.912 / 1.767      | 0.363 / 0.324 / 0.331      |\n| 384    | 0.198 / 0.201 / 0.225      | 1.894 / 1.954 / 1.795      | 0.357 / 0.320 / 0.325      |\n\n&nbsp;\n\nBeyond these thresholds, all models drop toward a plateau `ROUGE-L` of ~0.20. The 8B model converges fully by offset 512 (as seen in Figure 1a), though still slightly above plateau (0.225) at offset 384.\n\nImportantly, this growth is sub-linear. The increase from 1B to 3B (3× params) leads to a 5× expansion of the anchoring zone (16 → 80 tokens), while 3B to 8B (~2.7× params) yields only a 1.4× increase (80 → 112 tokens). Fitting a power law ($\\text{Offset} = k \\cdot P^a$) to these points gives a scaling exponent of approximately a ≈ 0.34, confirming sub-linear expansion.\n\nExtrapolating this trend, we estimate that a 70B model would require an offset of approximately 1090 tokens for memorization to fully decay to its baseline level. This projection is calibrated using the 8B model’s convergence point at offset 512 and the fitted power law with scaling exponent a ≈ 0.34. We stress that this is a preliminary projection. While the power law provides a reasonable fit for the 1B–8B range, its behavior at larger scales (e.g., 70B) remains unverified and must be tested empirically.\n\nThis finding also has practical implications. While offsets in the 1K-token range may still be manageable in modern long-context settings, they represent a nontrivial memory region that remains vulnerable to leakage. For use cases demanding high confidentiality, relying on positional mitigation alone may be insufficient.\n\nTo that end, our ongoing pretraining of a new **8B** and **70B** model on **15T** tokens is designed to explore these dynamics at scale. In this new run, we are using the Gutenberg FM-Probe buckets to precisely measure how pretraining-time mitigation techniques, such as Goldfish Loss [1], interact with positional fragility. This large-scale pretraining run is nearing completion, and a full analysis of these effects will be presented in future work.\n\n&nbsp;\n\n---\n\nFinally, to ensure full transparency and reproducibility, we will update the appendix of our manuscript to include the complete results from the additional experiments conducted for this rebuttal. We hope these additions resolve the raised uncertainties and make verification straightforward.\n\n&nbsp;\n\n**References**:\n[1] Abhimanyu Hans, et al., \"Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs,\" Advances in Neural Information Processing Systems, vol. 37, pp. 24022-24045, 2025."}, {"review_id": "Shr3MPWSTB", "reviewer": "Reviewer_vjB5", "summary": "This paper analyzes the memorization in LLMs, with an emphasis on sensitive data memorization such as copyright-protected data. The authors introduce a new aspect to be considered in memorization analysis, offset effect, and identify new phenomena, positional fragility---memorization is significantly more likely when the probing prefix appears near the start of the model’s context window, and degrades sharply when the prefix is offset deeper into the sequence. To study this, the authors pretrain LLaMA-style models (1B, 3B, 8B) on 83B tokens using a mix of FineWeb-Edu and Project Gutenberg, with careful control over exposure frequencies and token positions. They introduce the Swapped Gutenberg experiment, where the first 4K tokens of each training sequence are replaced with random text to push potentially sensitive content deeper into the sequence. This reduces memorization without compromising output quality.", "questions": "1. How do you envision operationalizing the \"Swapped Gutenberg\" mitigation in large-scale pertaining? \n2. In your setup, Project Gutenberg is used as the sensitive subset - what motivated this choice, and can the experiments be applied to other domains or setups such as PII or other sensitive data domains such as medical domains?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- This paper introduces a novel axis for analyzing memorization in LLMs: the position of a prefix within the context window during training and evaluation.\n\n- The experiments are rigorous and comprehensive, with strong empirical support across model scales and memorization frequencies. The authors pretrain LLaMA-style models (1B, 3B, and 8B) on an 83B-token corpus specifically for this study.\n\n- The finding that failures in verbatim memorization can lead to output degeneration adds a valuable dimension to existing work on repetition-induced degeneration.\n\n- The authors propose a simple but effective memorization mitigation strategy, displacing sensitive sequences later in the context window during training. This method significantly reduces extractable memorization without compromising generation quality. \n\n**Weaknesses** \n\n1. Limited practical utility of findings \n\nWhile the findings are novel and empirically supported, the practical application of the mitigation strategy remains unclear. Specifically:\n\n- The authors propose shifting sensitive content deeper into the context window (Swapped Gutenberg setup), but do not discuss how such a strategy could be applied during LLM pretraining, especially in large-scale, distributed training pipelines.\n- In real-world scenarios, training data is rarely labeled for sensitivity, and existing pretraining frameworks do not typically support fine-grained control over token placement or prefix location during batching.\n- The paper would benefit from a brief discussion of the feasibility of operationalizing this technique; for instance, through modified data loaders, selective shuffling, or integration with deduplication or data curation workflows.\n\nAdditionally, while I appreciate the authors’ efforts to pretrain models specifically for this study, applying this analysis to existing models is difficult in practice. For example, models like LLaMA do not release their pretraining data, and even open-source efforts like OLMo do not provide detailed logs (e.g., which sequences appear in which batches), making it nearly impossible to retroactively analyze prefix positions or apply these findings directly to such models.\n\n2. Unrealistic deployment assumptions \nThe mitigation strategy seems to assume that:\n- The model developer knows which parts of the training data are sensitive. \n- The developer has control over where in the sequence those parts are placed.\n\nIn practice, especially with trillion-token-scale training, we often lack precise metadata on content sensitivity, and restructuring input sequences at that scale would require non-trivial changes to data pipelines.\n\nOverall, while this paper contributes a compelling new perspective, the evaluation is conducted in a highly controlled, clean experimental setting, where Project Gutenberg is treated as the sensitive subset and FineWeb as safe. The analysis also relies on precise knowledge of where a prefix appears during training and evaluation, which may not be realistic in most practical scenarios. These assumptions limit the generalizability of the proposed analysis and mitigation to real-world LLM training.", "was_revised": true, "final_justification": "I'm increasing my score as I believe the authors' response addressed my concerns.", "rebuttal": "We thank the reviewer for their insightful and detailed feedback. The questions raised about practical utility and operationalization are critical for bridging the gap between scientific discovery and large-scale engineering. We value this opportunity to clarify the precise nature and intended contribution of our work.\n\nWe acknowledge that our phrasing, such as the term \"*proposed mitigation strategy*\" in line 225 and 233, may have created ambiguity. To clarify, our work's primary contribution is that of a **foundational study**, and the \"*Swapped Gutenberg*\" experiment should be understood as a **Proof of Concept (POC)**. Our intent was to empirically validate a previously overlooked phenomenon—the positional fragility of memorization—and our POC serves to demonstrate that this core finding can be leveraged for mitigation. This was a deliberate choice in how we framed our contribution; for instance, our abstract and introduction focus on \"*leveraging this finding*\", rather than formally proposing a complete, operationalized method.\n\nBy establishing this principle, we provide the critical justification for the community to now invest in solving the associated engineering challenges. With this framing in mind, we address the specific concerns below.\n\n&nbsp;\n\n## On \"Unrealistic Deployment Assumptions\" and Feasibility (Weakness 1)\n---\nThe reviewer's critique is centered on two key assumptions they believe to be unrealistic for large-scale LLM training. We address each directly, arguing that both are consistent with state-of-the-art LLM development.\n\n&nbsp;\n\n### Assumption 1: \"The model developer knows which parts of the training data are sensitive.\"\n\nWe argue this assumption is realistic, as identifying and tracking sensitive data is now both a practical necessity for developers driven by legal and ethical imperatives and a technically maturing engineering field.\n\n**Legal and Commercial Necessity**: It is no secret that proprietary models are trained with copyrighted texts. Recent legal rulings, such as the landmark Bartz v. Anthropic (2025), have established that training on such works can be considered fair use, but this defense critically hinges on the model not producing infringing outputs that could harm the market for the original work. To manage this immense legal and financial risk, a developer must be aware of which portions of their training data consist of copyrighted materials.\n\n**Sensitive Data Identification**:\n-  For amorphous types of sensitive data, e.g. PII and toxicity, tracing this content at scale is a mature engineering field. Modern data curation pipelines like NVIDIA's NeMo Curator  and Hugging Face's DataTrove are designed for this purpose, and a \"sensitivity detection\" module can be implemented within these frameworks using established methods.\n- For copyrighted content, the problem is even more tractable. The most robust and efficient approach is to tag content with its source and copyright status at the point of ingestion, as it is often sourced from discrete works like books and articles.\n- Even if not tagged at the source, open-source research tools like DE-COP [1] and commercial APIs from services like Copyleaks and Patronus AI's CopyrightCatcher can be integrated directly into these pipelines to flag protected works.\n\n**Reproducibility by Design**: The premise that retroactive analysis is \"nearly impossible\" is inconsistent with the design of modern frameworks. Frameworks like Megatron-LM use an `IndexedDataset` format (`.bin`/`.idx` files) that creates a permanent mapping from any training sample back to its source document. The data loader then uses a global random seed to generate a deterministic and cached  `shuffle_index`, which dictates the exact order of data loading for the entire run. This means the training metadata is, in fact, logged, making retroactive analysis entirely possible. \n\n**The Emerging Reality of Traceability**: The technical obstacles to retroactive analysis are also being actively dismantled. Recent tools such as OLMoTrace demonstrate that it is now possible to trace model outputs back to specific training samples in multi-trillion-token datasets in real-time [2].\n\n&nbsp;\n\n### Assumption 2: \"The developer has control over where in the sequence those parts are placed.\"\n\nThis is a tractable offline data preparation task, not a complex modification to the distributed training loop itself. Modern pretraining frameworks explicitly separates data preparation from the training process, providing a clear point of intervention.\n\n**Offline Data Preparation vs. Online Training**: In standard LLM workflows, the corpus is pre-processed, tokenized, and packed into sequences in an offline stage. Our \"*Swapped Gutenberg*\", or any sequence restructuring is an intervention in this simpler, more manageable offline stage. The distributed training engine (e.g., Megatron-LM, DeepSpeed) is agnostic to how sequences were constructed; it simply fetch the pre-tokenized data.\n\n**Feasibility via Custom Offline Data Logic**: The manipulation is achieved by creating custom logic within the offline data preparation scripts that process raw text into a final, tokenized training asset. The engineering precedent for intelligently organizing documents is already established by techniques like \"best-fit packing\" and \"dataset decomposition,\" which improve training efficiency by moving beyond naive concatenation. The operationalization the \"*Swapped Gutenberg*\" mitigation can simply leverage pre-existing sensitivity labels to inform this sequence construction.\n\nTherefore, having precise metadata on content sensitivity is not an unrealistic ideal; it is a fundamental component of **responsible AI**.\n\n&nbsp;\n\n## On \"Limited Practical Utility\" and Generalizability (Weakness 2)\n---\nHaving established feasibility, we now address practical utility. While applying our analysis to closed models is difficult, this is a symptom of the field's lack of transparency, not a weakness of our method. Our work's true utility lies in unlocking new capabilities for **auditing** and **advocacy**, providing the scientific basis to demand such transparency.\n\n&nbsp;\n\n### A New Paradigm for Auditing: The \"Sliding Window\" Attack\n\nOur findings enable a new, more powerful auditing technique: the **\"slide window\" attack**. Our research has profound implications for AI security, particularly for Membership Inference Attacks (MIAs). Current MIAs on LLM pre-training data often barely outperform random guessing, and our findings offer a potential explanation for this weakness.\n\nWe show that a training example may be a true member of the dataset, yet fail to trigger a memorization signal if the probe does not align with the early tokens in the context window. This leads to a high rate of false negatives in current MIAs, where true members are incorrectly classified as non-members simply because the attack was not conducted in a positionally-aware manner.\n\nThe sliding window attack resolves this: an auditor can probe the model at multiple offsets and look for **sharp, localized drops in perplexity** or spikes in `ROUGE`. These “fragility signatures” are stronger evidence of memorization than any single-offset metric, enabling a more precise and positionally-aware audit.\n\n&nbsp;\n\n### Empowering Stakeholders and Advocating for Transparency\n\nOur research provides stakeholders with a new, powerful tool for advocacy. We show that how content is used during training—specifically, its position within the context window—has a material impact on the resulting model's behavior. Armed with this knowledge, rights holders are no longer limited to asking if their work was used. They can now use our findings to demand greater transparency regarding how it was used, asking for specific training metadata like exposure counts and context window placement. This provides the empirical grounding for policymakers and the public to demand that developers release the crucial training-time metadata necessary for true auditing and reproducibility.\n\n&nbsp;\n\n## Questions:\n---\n\n&nbsp;\n\n1. **Operationalizing the \"Swapped Gutenberg\" Mitigation**: Detailed in weakness section.\n\n2. **Why Project Gutenberg was Used**: \n   - **Realistic Simulation**: This research is framed from a legal perspective, focusing on the risks of verbatim memorization of copyrighted works like books and articles, whereas Project Gutenberg consists of long-form literary texts that structurally mimic high-risk copyrighted books, providing a legally relevant testbed for studying copyright issues without actual infringement.\n   - **Experimental Control**: Our contamination analysis confirmed that the Gutenberg texts had negligible 13-gram overlap (**0.34%**) with the general purpose fineweb-edu data, ensuring that any observed memorization could be cleanly attributed to the experiment's controlled factors.\n\n3. **Application to PII and Medical Data**: While the principle of positional fragility is expected to generalize, applying it to new domains like PII and medical records requires further research and domain-specific adjustments. We believe that with careful calibration, extending this approach to these sensitive domains is feasible.\n\n&nbsp;\n\n---\nWe thank the reviewer again for their valuable feedback. In response to their comments, we will rephrase key sections of the manuscript to better clarify the foundational nature of our study and the role of our mitigation strategy as a Proof of Concept, we will also incorporate our contamination analysis results. We hope these targeted revisions and additions fully address the reviewer's concerns.\n\n**References**\n\n[1] André V. Duarte, et al., \"DE-COP: Detecting Copyrighted Content in Language Models Training Data,\" Proceedings of the 41st International Conference on Machine Learning (ICML), 2024. \n\n[2] Jiacheng Liu, et al., \"OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens,\" arXiv preprint arXiv:2504.07096, 2025."}, {"review_id": "gxpoBPVjZs", "reviewer": "Reviewer_mkAN", "summary": "This paper looks into understanding verbatim memorization in large language models by introducing the concept of \"positional fragility\" and demonstrating how the position of content within the context window dramatically affects memorization behavior. The authors conducted systematic experiments training LLaMA-style models (1B, 3B, 8B parameters) from scratch to examine memorization patterns, revealing that verbatim memorization is most strongly triggered by short prefixes drawn from the beginning of the context window, with memorization decreasing counterintuitively as prefix length increases.", "questions": "See weakness", "limitations": "See weakness", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. Novel finding on the positional offset, memorization drops when content is positioned away from the beginning of the context window. It  challenges prior assumptions about uniform memorization behavior. It is an interesting finding that shifting sensitive sequences deeper into the context window significantly reduces extractability without compromising general performance\n\n2. The paper does very comprehensive analysis, it train LLaMA-style models on 83B tokens from scratch, it gives precise control over training dynamics and data composition. This work also targets at the full-context sequences four times longer and evaluates suffixes at least ten times longer. \n\n3. The \"Swapped Gutenberg\" approach shows that simply shifting sensitive content 4000 tokens deep into the context window effectively suppresses memorization while maintaining generation quality. \n\n4. The connection between memorization and text degeneration is interesting and important. The observation that models produce repetitive, low-quality output when they cannot retrieve memorized content suggests that memorization and generation quality are more interwined than previously understood.\n\n**Weakness**\n1. The model trained on 83B tokens from scratch is very under-trained and curious whether this would impact the model's behaviors on memorization. It would be interesting if the authors could study the same phenomenon by continuing pretraining an existing LMs and compares the difference of memorization behaviors.\n\n2. The paper is the first to show that pretraining language models with smaller batch sizes could greatly reduce the memorization. It would be interesting to do further analysis on why this is the case because this result challenges conventional finding from the differential privacy literature, where larger batch sizes typically diminish the influence of individual examples and are generally preferred for privacy preservation. \n\n3. The Swapped Gutenberg is an effective approach but it requires encoding of an long input prefix and does the cross attention. This may not be very practical to deploy in the real world. It would be interesting to further quantify the potential side effect of Swapped Gutenberg in terms of downstream task performance. The long prefix may distract the model from using the useful knowledge encoded in the model weights. One related pointer is from Gemini 2.5 report discussing that when the context goes too long, the model is stuck in the context poisoning problem, hard to decide when to generate new information and retrieve information in the existing context. \n\n4. It would also be interesting to delve deeper into how the swapped part affects generation. Is memorization reduced because the model has a tendency to copy directly from the swapped part?", "was_revised": true, "final_justification": "The reviewer's rebuttal addressed most of my concerns. I keep my positive score unchanged", "rebuttal": "We sincerely thank the reviewer for their positive and highly constructive feedback. In response to the insightful questions raised, we have conducted several new experiments and analyses, which we summarize below.\n\n&nbsp;\n\n### On the \"Under-trained\" Concern\n---\nThe reviewer notes that our models, trained on 83B tokens, are not trained to the same scale as today's largest foundation models. This was a deliberate and principled choice for the following reasons:\n- While state-of-the-art models are trained on trillions of tokens, the renowned \"Chinchilla\" scaling laws suggest a compute-optimal parameter-to-token ratio of approximately 1:20 [1]. Our 1B and 3B models, trained on 83B tokens, are well above this threshold. \n- To maintain consistency across our experiments and work within a feasible academic budget, we applied the same 83B token count to all models, including our 8B model (a 1:10 ratio). This ensures a fair and controlled comparison of model scale effects.\n\nWe agree with the reviewer that studying our findings in a continued pretraining paradigm is a valuable test of their generalizability. To this end, we conducted a new set of experiments with the following setup:\n- **Base Model**: Llama 3.2 1B pretrained-only model (9T tokens).\n- **Training**: continued pretraining on the \"Sparse Gutenberg\" dataset using otherwise identical settings, but with a learning rate that decayed from 5e-5 to 1e-5.\n- **Comparison**: We compare this \"Continue\" model directly against our original 1B model \"trained from scratch.\" Both models were evaluated using 500-token prefixes to generate 500-token suffixes.\n\n**Our new experiments confirm that our central findings are robust and hold true across both training paradigms**:\n1. **Positional Fragility Persists**: The \"Continue\" model exhibits the same dramatic drop in verbatim recall (`Rouge-L`) as the offset increases, confirming this is a fundamental behavior .\n2. **The Link Between Memorization and Degeneration Holds**: At low repetition counts where memorization fails, the \"Continue\" model also exhibits significant text degeneration. Interestingly, its degeneration is even more pronounced (`TTR_gen` of ~0.17 vs. ~0.23 for the \"Scratch\" model), suggesting the link is stronger in more capable models .\n3. Additionally, we found that the 1B 'Continue' model did not outperform our 8B model trained from scratch on downstream tasks.\n\n| Offset | Rouge-L || Ref_PPL || TTR_gen ||\n|--------|---------|----------|----|------|---------|-----|\n|        | Scratch | Continue | Scratch | Continue | Scratch | Continue |\n| 0      | 0.744   | 0.669    | 1.051   | 1.029    | 0.521   | 0.521    |\n| 4      | 0.524   | 0.596    | 1.072   | 1.036    | 0.509   | 0.512    |\n| 8      | 0.463   | 0.569    | 1.079   | 1.048    | 0.491   | 0.498    |\n| 32     | 0.346   | 0.426    | 1.127   | 1.092    | 0.465   | 0.482    |\n| 128    | 0.219   | 0.249    | 1.356   | 1.283    | 0.411   | 0.437    |\n| 512    | 0.191   | 0.202    | 2.109   | 1.761    | 0.345   | 0.364    |\n| 2048   | 0.185   | 0.188    | 3.305   | 2.265    | 0.293   | 0.340    |\n\n| Repetitions | Rouge-L || Ref_PPL || TTR_gen ||\n|-------------|---------|----------|---------|----------|---------|-------|\n|             | Scratch | Continue | Scratch | Continue | Scratch | Continue |\n| 1           | 0.181   | 0.178    | 28.957  | 17.302   | 0.225   | 0.178    |\n| 4           | 0.184   | 0.178    | 21.057  | 14.252   | 0.228   | 0.160    |\n| 8           | 0.183   | 0.180    | 14.202  | 11.404   | 0.231   | 0.171    |\n| 16          | 0.183   | 0.182    | 6.886   | 6.804    | 0.323   | 0.293    |\n| 64          | 0.522   | 0.446    | 1.130   | 1.078    | 0.497   | 0.503    |\n| 128         | 0.744   | 0.669    | 1.051   | 1.029    | 0.521   | 0.521    |\n\nIn summary, these new results strongly support our original conclusions. The phenomenon of positional fragility is not an artifact of our models being \"under-trained\" but appears to be a general behavior that persists even when continuing the pretraining of an existing, state-of-the-art language model.\n\n&nbsp;\n\n### On Batch Size and Memorization\n---\nWe thank the reviewer for raising this important point. Below, we clarify the distinction between DP and standard language model training, and provide a plausible explanation—based on exposure frequency and training stochasticity—for the observed reduction in memorization with smaller batch sizes.\n\n**The Unique Dynamics of Differentially Private Training**\n\nIn the DP setting, Differentially Private Stochastic Gradient Descent (DP-SGD) clips per-sample gradients and adds noise to ensure privacy guarantees. Larger batch sizes improve the signal-to-noise ratio, mitigating the utility loss caused by added noise, and are therefore often preferred for a favorable privacy-utility trade-off [2].\n\nIn contrast, standard non-DP language model pretraining (as in our work) does not add noise. As noted in Appendix D.2, prior studies have shown that smaller batch sizes and longer training durations often improve generalization, consistent with our observed reduction in memorization.\n\n**Exposure Frequency**\n\nGiven a fixed compute budget and a constant total number of tokens processed, batch size and the number of gradient updates are inversely proportional. In an ideal case, for the 128-repetition FM-probe bucket, our experimental data from Table 7 (Appendix D.2) reveals a stark contrast: with a small global batch size (GBS) of 60, the model performs approximately 1,328 gradient updates (170,005 ÷ 128) between each exposure to the same sequence. With a large GBS of 240, it performs only about 332 updates (42,501 ÷ 128).\n\nThis fourfold difference in update frequency suggests a direct mechanism for increased memorization. With larger batches, the model's parameters have significantly less opportunity to be altered by other data between repeated exposures, which may reinforce and accelerate the memorization of that specific sequence. Conversely, the numerous updates in the small-batch setting likely introduce sufficient stochasticity to disrupt the formation of verbatim memory.\n\n&nbsp;\n\n### On Practicality and Side Effects\n---\nWe thank the reviewer for this insightful question regarding the practicality and potential side effects of the \"Swapped Gutenberg\" mitigation, such as performance degradation or context poisoning.\n\n**On Practical Deployability**: While our \"*Swapped Gutenberg*\" mitigation is intended to be a Proof of Concept (POC) , we have already outlined a feasible path to its implementation. As detailed in our response to Reviewer **vJB5**.\n\n**On Downstream Performance**: Regarding the potential for the long prefix to harm downstream performance, our evaluation results show this is not the case, as detailed with a comparative table in our response to Reviewer **Mw7L**.\n\n**On Context Poisoning**: With the model's general performance preserved, we now clarify why the \"context poisoning\" phenomenon is not directly applicable to our findings. There are two key distinctions between our experimental setup and the settings where this is typically observed:\n- Different Setting (Pre-training vs. Agentic Use): The \"context poisoning\" issue noted in reports like Gemini 2.5 occurred in an agentic setting, where a model uses its context as a memory for multi-step reasoning. In contrast, our work studies fundamental memorization dynamics that occur during the pre-training stage itself [3] .\n- Different Context Length (Native vs. Extrapolated): These distraction effects are generally documented when models are pushed to lengths far exceeding their pre-training window (e.g., 100k+ tokens). Our experiments operate entirely within the model's native 8,192-token context window.\n\nTherefore, while the concern is valid for ultra-long context models, it is not directly applicable to the findings presented in our work.\n\n&nbsp;\n\n### On Copying from the \"Swapped Part\"\n--- \nThis is a critical alternative hypothesis that warrants a direct empirical investigation. To address this, we conducted a thorough contamination analysis, complemented by the qualitative examples provided in our appendix.\n\nSince our work focuses on measuring verbatim memorization, the most direct method to test this hypothesis is to quantify the verbatim overlap between the model's output and the swapped prefix. We performed a 13-gram contamination analysis with the following setup:\n- Model: Pretrained 8B model under \"*Swapped Gutenberg*\" setting.\n- Inference Task: To test the strongest possible version of the \"copying\" hypothesis, the model was prompted with 500-token prefixes from the Gutenberg dataset (the \"retained part\") placed at a 0-token offset, immediately following the “swapped part”. It was then tasked with generating a 500-token suffix.\n- Contamination Check: For the generated suffixes of each FM-probe bucket, we compare them against the entirety of their corresponding 4000-token \"swapped content” from the FineWeb-edu dataset to identify any matching 13-grams.\n\n The analysis shows contamination is negligible, peaking at just **0.04%** . This quantitative result, supported by qualitative examples in Figure 12 , confirms the model is not copying. The mitigation works by disrupting the stable positional context required for memory retrieval, reinforcing our central thesis of positional fragility .\n\n&nbsp;\n\n---\nWe once again thank the reviewer for their highly constructive and thought-provoking questions. We hope this detailed response has clarified the deeper implications of our work.\n\n**References**:\n\n[1] Jordan Hoffmann, et al., \"Training Compute-Optimal Large Language Models,\" arXiv preprint, 2022.\n\n[2] Martin Abadi, et al., \"Deep Learning with Differential Privacy,\" Proceedings of the 2016 ACM SIGSAC CCS, pp. 308-318, 2016.\n\n[3] Gheorghe Comanici, et al., \"Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities,\" arXiv preprint, 2025."}, {"review_id": "ztZkTibADK", "reviewer": "Reviewer_Mw7L", "summary": "The paper examines the offset effect for LLM memorization where LLM memorization is taken as text regurgitation from legal perspective. \n\nBuilt on top of existing works, the paper shows that the text memorization capability depends on where the prefix occurs (so called offset in the paper). More specifically, the memorization rate decreases significantly when an offset is applied (e.g. see Figure 3) where the authors train their models under controlled settings. Based on this observation, the paper proposes further a method that pretrains the LLM with offsets and shows that the memorization rate significantly decreases.", "questions": "Can the authors expand on offset definition so that everyone is able to implement it please?", "limitations": "The limitation section Appendix B does not list significant limitations of the current work. The section is short and discusses that it is empirical, does not address differential privacy and finally a decoding technique coverage. The first one (or even first two) are two general and are not limitations to this particular work; while the last one is a very minor limitation that omitting does not hurt the paper. \n\nMeanwhile arguably there are many limitations to this work (as many of the research works) including testing on popular LLMs, training on more realistic settings, etc. just to name a few.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength: the fact that the model is able to regurgitate a large chunk of text is an important phenomenon to be mitigated and the paper has brought very good arguments on it. The paper's main claim (relationship between offset and LLM memorization behaviour) is backed up with controlled experiments. \n\nWeakness: \nFirstly, the proposed technology arguably compromises too much performance. As shown in Table 2, the perplexity increases a lot compared to the base case to lower memorization ratio. \nSecondly, the offset settings are not clear. The notion is everywhere in the paper however, how offset is setup is only described a bit vaguely in line 68. Unforutunately, I don't know how the skip is implemented concretely from the paper (are they irrelevant tokens, etc.?) I can only guess irrevalent tokens because that is the method adopted in section 6. \nAfter reading the paper, I am not sure if offset simply makes the model perform worse in general (and by consequence have decreased memorization ratio) or the model only worse at memory. \nThe limitation section has not mentioned any of the above, unfortunately.", "was_revised": true, "final_justification": "I initially focused on a confusion between perplexity score as performance and the mitigation solution and it has been largely explained in detail in the authors' rebuttal. The authors have also clarified the offset setting which I was not able to get clear during review. \n\nBased on this, I have increased my score. However, I don't think the random prefix mitigation techniques in training time and test time mitigation is a production ready solution and the described techniques currently miss slightly on clarity. So I re-score the paper as slightly below acceptance overall.", "rebuttal": "We sincerely regret that our explanations, particularly regarding the 'offset' setting and perplexity scores, created confusion. We thank the reviewer for highlighting these specific areas, as it gives us an opportunity to improve the paper's clarity for all readers.\n\n&nbsp;\n\n### Weakness 1: The proposed technology arguably compromises too much performance on perplexity\n---\nFirstly, we would like to address the concern that our proposed mitigation \"compromises too much performance.” The most accurate measure of a model's general capability is its performance on a suite of standard downstream tasks. As shown in Table 5 from Appendix C2 and table 11 from Appendix D3, our 1B and 8B models trained under \"*Swapped Gutenberg*\" setting achieve highly competitive, and in some cases superior, results across a range of benchmarks when compared to the standard \"*Sparse Gutenberg*\" models.\n\n&nbsp;\n\n| Model | Wiki. ppl↓ | Hella. norm↑ | ARC-C acc↑ | Wino. acc↑ | CSQA acc↑ | MMLU acc↑ |\n|-------|------------|--------------|------------|------------|-----------|-----------|\n| Standard 1B | **18.71** | **52.31** | **33.36** | 71 | 53.91 | **23.65** |\n| Swapped 1B | 22.31 | 52.22 | 31.83 | **71.93** | **55.01** | 23.27 |\n| Standard 8B | **13.15** | **65.74** | **42.24** | 75.52 | **61.88** | 24.53 |\n| Swapped 8B | 15.71 | 65.48 | 42.06 | **75.79** | 61.09 | **25.89** |\n\n&nbsp;\n\nSecondly, having established that the model's general performance is preserved, we now address the reviewer's point about the perplexity score. We recognize that our table notation (Perplexity↓) may have contributed to confusion, and we appreciate the opportunity to clarify.\n\nWhile a lower perplexity score typically indicates better predictive performance, its interpretation is inverted when the goal is to **mitigate verbatim memorization**. A model that has perfectly memorized a sequence will exhibit a very low perplexity (approaching 1.0) because it is extremely certain of the next token. Our mitigation's objective is to disrupt this certainty. Therefore, a **higher perplexity** on these specific target sequences is **a positive sign of successful mitigation**, as it shows the model is no longer regurgitating the text. The data for our 8B model clearly demonstrates this phenomenon:\n\n&nbsp;\n\n| Repetition Freq. | Perplexity (Sparse) | Perplexity (Swapped) | Interpretation |\n|------------------|---------------------|----------------------|----------------|\n| 1 | 16.089 | 82.206 | High uncertainty in both cases; difference likely magnified by positional effect. |\n| 8 | 3.43 | 12.841 | Mitigation successfully increases uncertainty as memorization begins. |\n| 64 | 1.023 | 3.67 | Mitigation effectively disrupts strong memorization. |\n| 128 | 1.012 | 3.636 | Mitigation effectively disrupts strong memorization. |\n\n&nbsp;\n\nAs shown, for sequences seen 64 or 128 times, the standard model has a near-perfect perplexity score, indicating heavy memorization. Our mitigation strategy successfully increases this perplexity, demonstrating its effectiveness. We also note that the large perplexity difference at a frequency of 1 is likely magnified by a confounding positional effect, as the perplexity is measured on suffixes at different absolute positions:\n- **Base Case (Sparse)**: Suffix position is 500-1000 tokens.\n- **Swapped Case**: Suffix position is 4500-5000 tokens.\n\n&nbsp;\n\n### Weakness 2: the offset settings are not clear\n---\nWe thank the reviewer for highlighting that the definition and implementation of \"offset\" were not sufficiently clear. To resolve this, it is crucial to clarify the two distinct ways positional shifts is used in our paper: first as a new evaluation dimension, and second as a training-time mitigation strategy.\n\n**\"Offset\" as a New Evaluation Dimension**: \n- **Definition**: Given a training text segment from the FM-Probe set, we skip the first `o` tokens (offset `o`) and take the next `n` tokens as the prefix. **No tokens are inserted**; nothing about training changes.\n- **Implementation**: For a document d, an offset of `o=64` means the probing prefix of `n=500` tokens is `d[64:564]`. We then measure verbatim recall on the true following suffix. \n\n**The \"Swapped Part\" as a Training-Time Mitigation (Section 6)**: \n- **Implementation**: For each Gutenberg sequence, we replace the first 4,000 tokens with unrelated Gutenberg text (randomly sampled), pushing the sensitive portion (the “retained part”) deeper into the context window. \n- **Rationale**: Our main finding is that LLMs are positionally fragile: memorization is easiest when the trigger prefix sits at the very start of the context window. By pushing the sensitive part far from this anchoring zone, we drastically reduce extractable memorization while preserving quality.\n\nCrucially, **offset has nothing to do with the model's general performance**, all downstream performance evaluations reported in our paper (e.g., MMLU, HellaSwag) are conducted using the **standard, un-offsetted evaluation protocols** for those tasks. The strong results on these benchmarks are a true measure of the model's capabilities, independent of our memorization analysis.\n\n&nbsp;\n\n### Limitations\n---\nWe thank the reviewer for their valuable feedback on our limitations section. We agree that the initial version was not comprehensive enough and did not adequately cover the specific limitations of our work. We will significantly expand and revise this section in the final version of the paper based on the excellent points raised by yourself and other reviewers.\n\nThe updated limitations section will include a more thorough discussion of:\n- **Scope as a Proof of Concept (POC)**: We will explicitly state that our work is a foundational study to validate the principle of positional fragility and that our mitigation is a POC, not a production-ready solution. We will clarify that bridging this gap requires both significant engineering and further research, for instance, to develop more granular, controllable memorization mechanisms .\n- **Generalizability of Findings**: We will discuss the need for further research to test if our findings generalize beyond literary texts to other domains, such as PII or medical records .\n\nLast but not least, we agree that testing on popular models in realistic settings is crucial. We would like to clarify that our study was conducted on models from the Llama 3 family, which represent a popular and foundational architecture in the field. Furthermore, to directly address the point about realistic training, we conducted continuous pretraining on a **Llama 3.2 1B pretrained-only** model under our \"*Sparse Gutenberg*\" setting. The results confirmed that our core findings on positional fragility persist in this more realistic scenario. The full details of this experiment are provided in our response to Reviewer **mkAN**.\n\n&nbsp;\n\n---\n\nWe thank the reviewer again for their time and for providing feedback that will help us improve the paper's clarity. We hope that our detailed responses and new experimental results have fully addressed the initial concerns and provided a clearer perspective on our work's contributions."}]

Author rebuttal:

Meta-review: The paper explorers the relationship between memorization of token sequences an LM sees during training and the offset of said token sequence within the batch. The paper finds that token sequences seen at the start of the batch are more likely to be memorized. In addition, in cases where the LM fails to produce the true continuation to a prompt known to be in the training dataset, the LM is also more likely to exhibit degeneration, outputting an incoherent continuation.

Reviewers had very mixed feelings about this paper with scores ranging from "strong accept" to "borderline reject."

The positive reviews note that paper's findings are truly novel, analyzing a facet of memorization which has not been analyzed before. Multiple reviewers appreciated the comprehensive experiments. Reviewers also liked that the paper didn't just include an analysis of the phenomenon, but proactively suggested a way to make use of as a mitigation strategy against undesired memorization.

One criticism of the paper was that the proposed mitigation strategy is unrealistic, as it requires model trainers to know a priori which parts of their training data are sensitive. There are also concerns the paper does not do a job enough job explaining its results that are counterintuitive relative to prior work (such as the result that smaller batch sizes are better for reducing memorization). Another criticism is that the paper doesn't clearly the settings used for its definitions of things like the offset. Though not mentioned by the reviewers, I find the distinction between "context window size" and "batch size" is also not made clear. Are these always the same?

Overall, I think the paper presents a novel analysis that is a valuable contribution to the study of memorization in language models. I recommend its acceptance.

A couple other comments from my read of the paper (though not mentioned by any of the reviews):

I strongly recommend the authors revisit their figures in order to ensure all font sizes are legible. The text in Figure 4 is completely illegible when the paper is printed. You may consider making your table text \small sized in order to make room for a larger Figure 4. The x and y axis labels in your other figures could also benefit from big font sizes.

The phrase "prior work implicitly assumed uniformity by probing only from the beginning of training sequences" seems like a misrepresentation of prior work. Prior work such as Carlini et al (2023) prompted with the starts of documents, but the starts of documents are not necessarily the starts of training sequences, since many modern LLM are trained with packing, where multiple documents are packed into a training sequence (and also, particularly long documents are broken into multiple training sequences). I believe the abstract should be revised, and this distinction should be addressed in the text.
Decision: Accept (poster)