Title: Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Risks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Model

Verbatim Memorization

Pretraining

Law

Copyright
Abstract: Large language models are known to memorize parts of their training data, posing risk of copyright violations. To systematically examine this risk, we pretrain language models (1B/3B/8B) from scratch on 83B tokens, mixing web-scale data with public domain books used to simulate copyrighted content at controlled frequencies at lengths at least ten times longer than prior work. We thereby identified the offset effect, a phenomenon characterized by two key findings: (1) verbatim memorization is most strongly triggered by short prefixes drawn from the beginning of the context window, with memorization decreasing counterintuitively as prefix length increases; and (2) a sharp decline in verbatim recall when prefix begins offset from the initial tokens of the context window. We attribute this to positional fragility: models rely disproportionately on the earliest tokens in their context window as retrieval anchors, making them sensitive to even slight shifts. We further observe that when the model fails to retrieve memorized content, it often produces degenerated text. Leveraging these findings, we show that shifting sensitive data deeper into the context window suppresses both extractable memorization and degeneration. Our results suggest that positional offset is a critical and previously overlooked axis for evaluating memorization risks, since prior work implicitly assumed uniformity by probing only from the beginning of documents or training sequences.

Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Risks

Abstract

Large language models are known to memorize parts of their training data, posing risk of copyright violations. To systematically examine this risk, we pretrain language models (1B/3B/8B) from scratch on 83B tokens, mixing web-scale data with public domain books used to simulate copyrighted content at controlled frequencies at lengths at least ten times longer than prior work. We thereby identified the offset effect, a phenomenon characterized by two key findings: (1) verbatim memorization is most strongly triggered by short prefixes drawn from the beginning of the context window, with memorization decreasing counterintuitively as prefix length increases; and (2) a sharp decline in verbatim recall when prefix begins offset from the initial tokens of the context window. We attribute this to positional fragility : models rely disproportionately on the earliest tokens in their context window as retrieval anchors, making them sensitive to even slight shifts. We further observe that when the model fails to retrieve memorized content, it often produces degenerated text. Leveraging these findings, we show that shifting sensitive data deeper into the context window suppresses both extractable memorization and degeneration. Our results suggest that positional offset is a critical and previously overlooked axis for evaluating memorization risks, since prior work implicitly assumed uniformity by probing only from the beginning of documents or training sequences.

1 Introduction

Large language models (LLMs) have demonstrated the capacity to reproduce substantial portions of their training data verbatim, raising serious concerns about copyright violations [Chang et al., 2023, Karamolegkou et al., 2023] and privacy breaches [Huang et al., 2022]. This capability has fueled increasing legal challenges, with high-profile cases, such as the New York Times lawsuit against OpenAI, highlighting the potential misuse of proprietary content [Freeman et al., 2024]. Industry leaders have not denied these risks, and instead called for regulatory exemptions that would allow LLM training on copyright-protected material [Brodkin, 2025]. Against this backdrop, understanding how LLMs memorize and regurgitate legally sensitive content is no longer a niche technical concern, but a central requirement for the responsible and legal development of LLMs [Rosenthal and Veraldi, 2025] and informed policymaking on this subject.

Existing work has begun to study memorization in LLMs, but key gaps remain. First, most mitigation strategies are applied post-training via fine-tuning, decoding constraints, or unlearning methods. However, recent studies show that these approaches can be bypassed [Park et al., 2024, Nasr et al., 2025], indicating that they are better suited as complementary defenses rather than standalone solutions. This highlights the need to address memorization proactively during pretraining, where

retention first happens. Second, prior analyses typically focus on short suffixes prompted from the beginning of training sequences, implicitly assuming that memorization behavior is uniform across the context window. This methodological choice overlooks the reality that user inputs may appear at arbitrary offsets in the content window, and that memorization may vary significantly with position.

In this work, we revisit memorization from a legally motivated perspective, focusing on long-form text segments drawn from published books (via Project Gutenberg) to simulate high-risk copyright scenarios. We pretrain LLaMA-style models (1B, 3B, and 8B) from scratch on 83B tokens, combining curated book data with web-scale educational corpora. This setup grants precise control over training dynamics (such as model scale and batch size), data composition (such as exposure frequency and placement of sensitive content), and evaluation configuration (such as prefix length and position within the context window). Compared to the most extensive memorization analysis to date by Hans et al. [2025], which also targets pretraining-time mitigation, our study employs full-context sequences four times longer and evaluates suffixes at least ten times longer. Together, these design choices enable an unprecedented, comprehensive, and fine-grained memorization analysis.

1

2

4

8

16

32

64

128

256

512

1024

2048

0

0

.

2

0

.

4

0

.

6

0

.

8

1

Offset

Rouge-L

(a) Rouge-L scores by offset

1

2

4

8

16

32

64

128

256

512

1024

2048

0

0

.

2

0

.

4

0

.

6

0

.

8

1

Offset

MAUVE

1B Dense

1B Sparse

3B Sparse

8B Sparse

(b) MAUVE scores by offset

Our main contributions are:

We identify the offset effects : verbatim memorization is most effectively triggered by short prefixes at the beginning of the context window. Memorization recall drops sharply as prefixes are shifted further into the sequence. We attribute offset effects to positional fragility : LLMs are highly sensitive to positional shifts as they disproportionately rely on early tokens as retrieval anchors (§ 4).

We show that memorization failures cause degeneration . Under offset or low-frequency conditions, the model produces repetitive and semantically incoherent output, establishing a link between memorization breakdown and text degeneration (§ 5).

We show that positional fragility can be leveraged to mitigate memorization: shifting sensitive sequences deeper into the context window significantly reduces extractability without compromising general performance (§ 7).

We find that batch size influences retention: smaller batches with more frequent updates modestly reduce memorization under identical compute budgets. (§ 8).

We reproduce and scale previous studies in more controlled and legally relevant settings, using longer sequences to better reflect the risks of copyright in the real world (Appendix C).

2 Problem Setup

To facilitate the discussion around verbatim memorization we introduce the following nomenclature:

Text Segment : A sequence of 8192 tokens from the training corpus, beginning with a BOD token, that fits exactly the context window of the model. Prefix x : The input text sequence provided to the model to probe memorization, represented as x = ( x 1 , ..., x n ) with n tokens extracted consecutively from the training corpus.

True Suffix s : The text sequence that immediately follows the prefix in the original text segment, represented as s = ( s 1 , ..., s m ) with m consecutive tokens.

Generated Suffix y : The text sequence produced by the model given prefix x , where y = ( y 1 , ..., y m ) , that matches the length of the ground truth suffix m .

Offset o : The number of tokens to skip in the text segment from the beginning-of-document token BOD before starting prefix extraction.

BOD

Document (8,191 tokens)

BOD

Prefix

Suffix

500

500

BOD

Offset

Prefix

Suffix

64

500

500

BOD

Offset

Prefix

Suffix

256

500

500

BOD

Swapped Part (4,000 tokens)

Retained Part (4,191 tokens)

Offset

Prefix

Suffix

64

500

500

Quantifying Verbatim Memorization We follow Nasr et al. [2025]'s definition of verbatim memorization, providing a model with a prefix x from training data and measuring how closely its greedy continuation y matches the true continuation s . Prior work predominantly focuses on short prefix probing, using prefixes of size 20 [Carlini et al., 2021], 32 [Biderman et al., 2023, Huang et al., 2024, Duan et al., 2025], 50 [Karamolegkou et al., 2023, Duan et al., 2025, Ippolito et al., 2023], 50-500 [Carlini et al., 2023], and 1998 [Hans et al., 2025]. Suffix lengths are likewise short, typically ranging from 25 to 50 tokens. Nearly all evaluations extract prefixes from the beginning of the document, with only Carlini et al. [2021] sampling prefixes from arbitrary positions, but they only report the highest-recall case. To address these limitations, our evaluation spans prefix lengths from 50 to 5000 tokens, sampled at offsets ranging from 0 to 2048, with suffixes between 50 and 8000 tokens. We report similarity using three metrics that capture increasing levels of verbatim reproduction: Rouge-L (longest common subsequence relative to reference length) [Lin, 2004], LCCS (longest common contiguous substring) [Freeman et al., 2024], and EM (exact match of the entire suffix).

Measuring Text Degeneration We evaluate output quality using three complementary metrics. Type-token ratio (TTR) measures lexical diversity as the fraction of unique tokens in the generated suffix [Kettunen, 2014]. MAUVE quantifies distributional similarity between model outputs and humanwritten text [Pillutla et al., 2023]. As a baseline, unrelated suffix pairs yield a ROUGE-L of approximately 0.18. We also report perplexity on the reference suffix to assess how well the model predicts the ground-truth continuation. Given a prefix x = ( x 1 , . . . , x n ) and its true suffix s = ( s 1 , . . . , s m ) , perplexity is defined as PPL ( s | x ) = exp ( -1 m ∑ m t =1 log P ( s t | x, s <t ) ) . Intuitively, perplexity reflects how many next-token candidates the model finds plausible at each step [Jelinek et al., 2005]. For downstream performance evaluation, we utilize lm-eval-harness [Gao et al., 2024].

3 Design

Data & Model We train a decoder-only transformer model following the Llama architecture [Grattafiori et al., 2024], with complete architectural and training details provided in Appendix A.2. Our training corpus combines two complementary sources: (1) Project Gutenberg 1 , used to simulate potential copyright infringement via literary texts; and (2) Fineweb-Edu [Penedo et al., 2024], a curated dataset of high-quality educational web content typical in modern LLM pipelines. A 13-gram contamination analysis confirmed a negligible 0.34% overlap between the two datasets, ensuring observed memorization is attributable to our experimental controls.

Dense Gutenberg: Extreme Memorization Study Inspired by the setup of Hans et al. [2025], we establish an extreme memorization scenario, where models are trained exclusively on our curated subset of Project Gutenberg text segments comprising 10,000 sequences of 8,192 tokens each. Each epoch constitutes a complete traversal of the entire corpus, with model checkpoints saved at logarithmically spaced intervals (powers of 2) as well as at the final 80th epoch. Since each sequence appears exactly once per epoch, the checkpoint number directly corresponds to the number of exposures to each training sequence, providing a direct measure of how repetition influences memorization behavior.

Sparse Gutenberg: Realistic Copyright Memorization Simulation To create a more realistic scenario, we designed a mixed corpus training configuration where Project Gutenberg excerpts represent just 2% (1.74B tokens) of the training data, with the remaining 98% (81.82B tokens) sourced from FineWeb-Edu. For analyzing how exposure frequency affects memorization, we developed Frequency-Varied Memorization Probe Buckets (FM-Probes), with each bucket containing 500 distinct and randomly sampled Gutenberg sequences that appear at precisely controlled frequencies ranging from 1 to 128 repetitions. By performing a single training pass through this carefully structured corpus, our FM-probe methodology enables concurrent evaluation of memorization patterns across multiple exposure levels without requiring separate training runs.

Pretraining Factors Within the Sparse Gutenberg framework, we systematically investigate several key factors that potentially influence memorization dynamics during pretraining. First, we examine batch size variations (0.5M, 1M, 1.5M and 2M tokens) to assess how optimization granularity affects memorization tendencies. Second, we analyze model scale effects by comparing memorization patterns across architecturally consistent models of 1B, 3B, and 8B parameters. Third, we evaluate how exposure frequency impacts memorization propensity through our FM-Probe methodology.

4 Offset: The Missing Dimension in Memorization Evaluation

Despite increasing attention to memorization risks in LLMs, most literature evaluates memorization using prefixes extracted from the beginning of documents [Hans et al., 2025, Carlini et al., 2023, Kiyomaru et al., 2024]. However, memorized content can appear anywhere within a document-not just at the start. At the same time, little attention has been paid to how the absolute position of the prefix within the model's context window-the offset-affects memorization behavior. In this section, we show that both verbatim memorization and language generation quality are highly sensitive to offset, and whether longer prefixes increase verbatim recall is itself offset-dependent.

4.1 Shorter Prefixes Outperform Longer Ones in Triggering Memorization

0

1K

2K

3K

4K

5K

0

0

.

2

0

.

4

0

.

6

0

.

8

1

Rouge-L

(a) LLaMA 1B

0

1K

2K

3K

4K

5K

Offset 0

Offset 64

Offset 128

(b) LLaMA 3B

0

1K

2K

3K

4K

5K

(c) LLaMA 8B

While previous research suggests that longer prefixes can enhance verbatim recall, we find that this observation is incomplete and potentially misleading. Our results reveal a more efficient condition for triggering memorization: using short prefixes that align with the beginning of the context window during training. As shown in Figure 3, at offset 0, short prefixes yield the highest recall , particularly in smaller models, whereas longer prefixes and longer offsets reduce extractable memorization . Note that this result does not contradict previous work. As we show in Table 1, with a fixed prefix of 50 tokens, we can induce perfect reproduction of suffixes of up to 8,000 tokens long, especially when the target sequence is frequently encountered during training. However, such an evaluation does not take into account larger prefixes and non-zero offsets.

With non-zero offsets, longer prefixes substantially improve memorization, suggesting that they could partially compensate for positive offsets. However, as illustrated by the red and green lines in Figure 3, even with extended prefix length, memorization at disadvantaged positions never fully matches the level observed at offset 0. Moreover, the 3B model achieves higher verbatim recall at offsets 64 and 128 compared to the 1B model, but memorization plateaus from 3B to 8B, indicating that larger models do not mitigate positional fragility and suffer similarly at suboptimal positions.

This inversion of optimal prefix length under offset conditions highlights a key property of verbatim memorization in realistic settings: it is strongly tied to positional cues, especially the beginning of the context window. Our findings align with the 'attention sink' mechanism discovered by Xiao et al. [2024], where initial tokens disproportionately influence attention distribution and serve as computational anchors. Nasr et al. [2025] has also shown that the token that represents the document boundary, such as end-of-document ( EOD ) marker, can trigger memorization (in our case BOD ). This raises a key question: is the verbatim memorization driven by the positional role of the initial tokens in general or by the special representational status of the BOD token itself? To disentangle these two factors, we design controlled ablation experiments that isolate the BOD token's contribution to verbatim recall.

4.2 BOD or Initial Tokens: An Ablation Study

We isolate the impact of the BOD token by retraining our 1B model with its attention masked, thereby effectively removing it from the input. In this setting, if memorization persists, it suggests that positional cues alone are sufficient; if it degrades, the BOD token likely serves as a dedicated retrieval anchor. To explore whether this change interacts with positional fragility, we evaluated memorization performance at three context-window offsets: 0, 50, and 100 tokens.

Figure 4 illustrates the impact of masking the BOD token on the model's memorization capability at the 1B scale. At offset 0 (left panel), where the prefix begins at the start of the context window, masking the BOD token yields negligible differences in verbatim recall. This suggests that the initial textual content alone can serve as an effective cue for memorization without an explicit BOD .

Interestingly, masking the BOD token leads to moderately higher verbatim recall in high-frequency buckets under offset conditions. At a 50-token offset, the BOD -masked model achieves up to 8% higher Rouge-L scores for prefixes between 750-2000 tokens-equivalent to roughly 40 additional tokens recalled verbatim. This suggests that, without an explicit BOD , the model leans more on local context, making it slightly more robust when recalling sequences from non-initial positions. Nonetheless, it continues to exhibit positional fragility (see Appendix D.1).

50

100

250

750

1000 2000 3000 4000 5000

prefix

1

2

4

8

16

32

64

128

repetition

-.004 -.002 -.005 -.001 -.001 -.003 -.002 -.002 -.001

-.003 -.002 -.003 -.001 -.002 -.001

.001

-.000 -.001

-.002 -.005 -.002 -.004 -.001 -.004 -.000

.005

.002

-.005 -.004

.000

-.001

.001

-.002

.002

-.001

.002

-.016 -.011 -.005

.001

-.013 -.009 -.012 -.006

.003

-.012 -.013 -.016 -.021 -.015 -.028 -.027 -.011 -.006

.004

-.008

.001

-.014 -.022 -.004

.000

-.019 -.017

Difference (Suffix 500 Offset 0)

50

100

250

750

1000 2000 3000 4000 5000

prefix

1

2

4

8

16

32

64

128

repetition

-.002 -.002 -.001 -.003 -.002

.000

-.001

.001

.002

-.001 -.002 -.002 -.003 -.001 -.000

.001

-.002 -.005 -.002 -.004 -.002 -.002

.002

.001

-.003

-.003 -.003 -.000 -.002 -.004 -.001

.001

.002

.001

-.002 -.005 -.003 -.000 -.000

.001

-.000

.002

.002

-.002 -.001 -.001

.004

.009

.006

.009

-.000

.000

-.001

.001

.018

.051

.070

.047

.047

.006

.021

-.002

.005

.046

.085

.088

.073

.050

-.009 -.013

Difference (Suffix 500 Offset 50)

50

100

250

750

1000 2000 3000 4000 5000

prefix

1

2

4

8

16

32

64

128

repetition

-.006 -.001 -.003 -.001 -.002 -.003

.000

-.001 -.000

-.004 -.005 -.004 -.002 -.004 -.000 -.002 -.004 -.001

-.005 -.003 -.002

.001

-.004 -.003 -.003

.000

.001

-.001 -.004 -.002 -.003 -.002

.001

-.001 -.002 -.002

-.006 -.003 -.001 -.000 -.001

.001

.001

.002

.000

-.001 -.002 -.004

.002

.002

.005

.003

-.004 -.002

.007

.028

.034

.035

.035

.005

-.006

.001

.004

.007

.040

.049

.021

.012

-.017 -.031

Difference (Suffix 500 Offset 100)

0.20

0.15

0.10

0.05

0.00

0.05

0.10

0.15

0.20

0.20

0.15

0.10

0.05

0.00

0.05

0.10

0.15

0.20

0.20

0.15

0.10

0.05

0.00

0.05

0.10

0.15

0.20

To ensure that this improvement in recall is not simply a byproduct of degraded generation quality-a symptom of degeneration discussed in §5-we evaluated the model using the MAUVE metric at longer prefix lengths (1000-5000 tokens), corresponding to regions with increased memorization. We also assessed downstream performance (see Table D.1 in the appendix). In both cases, the BOD -masked model performed on par with the baseline, indicating that increased recall does not come at the cost of linguistic ability.

5 Memorization: Under-Explored Cause of Text Degeneration

Our experiments reveal a connection between memorization capabilities and text degeneration in language models. While previous research observed repetitive patterns in training data as the primary cause of degeneration, highlighting a 'repetition in, repetition out' phenomenon [Li et al., 2023], our findings suggest that limitations in how models encode and retrieve memorized information represent a more fundamental and previously unexplored mechanism underlying text degeneration. Rather than attributing repetitive outputs solely to repetitive training data, we demonstrate that deficiencies in memorization can also lead to text degeneration.

5.1 Context Window Offsets Reduces Output Text Quality

We find that the position of prefixes within the model's context window during pretraining influences the quality of the generated suffixes. As shown in Fig. 1, in the dense Gutenberg setting, where the model is trained on 10K sequences for 80 epochs, we observe that verbatim memorization effectively vanishes, while language modelling capabilities remain stable at larger offsets. In contrast, under the sparse Gutenberg setting with FM-probe bucketing, the model's output degrades in both lexical diversity and language coherence. An example is shown in Fig. 10.

This complements the 'repetition in, repetition out' hypothesis by Li et al. [2023]. Their work specifically revealed a strong correlation between the repetition rate of 2-gram in training data and the repetitive generation in the model's output. Nevertheless, our results suggest that text degeneration may not be solely driven by repetition in training data.

5.2 Lower Exposure Frequency Reduces Output Text Quality

In addition to context window offsets, the training frequency of a sequence during training independently influences lexical diversity and language coherence. As illustrated in Fig. 5, sequences seen less frequently exhibit pronounced degradation in the model's linguistic capabilities across all model scales (Examples are shown in 11). However, larger models demonstrate greater resilience, achieving higher TTR and MAUVE scores than smaller counterparts, particularly at low frequencies, indicating that model scale modestly enhances linguistic capability.

1

2

4

8

16

32

64

128

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Frequency

TTR

1B Sparse

3B Sparse

8B Sparse

(a) TTR scores by frequency

1

2

4

8

16

32

64

128

0

0

.

2

0

.

4

0

.

6

0

.

8

1

Frequency

MAUVE

(b) MAUVE scores by frequency

5.3 Insufficient Memory Retrieval: A contributing Factor to Text Degeneration

Our empirical findings from §5.1 and §5.2 converge on a nuanced understanding of text degeneration in language models. Specifically, they reveal that imbalances in training data frequency can lead to distinct generation issues. High-frequency exposure often results in verbatim memorization, where the model reproduces training data verbatim. Conversely, low-frequency exposure does not necessarily enhance diversity; instead, it can exacerbate degeneration, manifesting as repetitions or thematic loops. This suggests a link between limited memory retrieval capabilities and degenerative text patterns.

The analysis of context window offsets further supports this relationship. As models attempt to continue text from distant positions, they tend to fall into thematic looping. This position-dependent degradation in linguistic ability supports that insufficient memory retrieval can cause text degeneration, particularly when memorisation targets must compete with diverse training data. This connection between retrieval limitations and text degeneration warrants further investigation in future work.

6 Probing the Robustness of Positional Fragility

Our primary analysis revealed positional fragility using greedy decoding on models trained from scratch. To ensure that this phenomenon reflects an inherent model behavior rather than an experimental artifact, we further examine its robustness across decoding strategies and training paradigms.

Robustness Across Decoding Strategies To evaluate the robustness of positional fragility under alternative decoding configurations, we re-ran inference on the repetition-128 FM-Probe bucket using our 8B model under the Sparse Gutenberg setting with two additional decoding methods: (1) beam search ( k =5 ), reporting the top-ranked hypothesis, and (2) nucleus sampling (topp =0 . 9 ). Table 2 shows that beam search, a deterministic probability-maximizing strategy, achieves nearly perfect memorization at shallow offsets (e.g., ROUGE-L = 0.999 at offset 0). However, when recall fails at deeper offsets, it exhibits the most severe degeneration, supporting our claim that failed retrieval leads to incoherent outputs. In contrast, nucleus sampling yields lower memorization at shallow offsets but shows a much milder decline in lexical diversity at large offsets (e.g., TTR = 0.460 at offset 2048), suggesting that decoding strategy influences the severity of degeneration while leaving the offset-dependent pattern of positional fragility unchanged.

Persistence Under Continued Pretraining Wefurther examine whether positional fragility persists under a continued pretraining paradigm. Starting from the official Llama 3.2 1B model pretrained on 9T tokens, We continued pretraining using otherwise identical settings to the Sparse Gutenberg configuration, but with a smaller learning rate 1 × 10 -5 . As shown in he continued-pretrained 1B

model exhibits slightly stronger memorization (higher ROUGE-L) and more diverse generations (higher TTR) than the from-scratch 1B model, while preserving the same offset-dependent decay pattern, confirming that positional fragility holds across training paradigms.

7 Leveraging Positional Fragility to Mitigate Verbatim Memorization

To demonstrate that positional fragility can be leveraged to mitigate verbatim memorization, we designed a Proof-of-concept (POC) experiment which we refer to as Swapped Gutenberg . For each sequence in our FM-Probe buckets, we replace the first 4,000 tokens (the swapped part ) with tokens from randomly selected Project Gutenberg excerpts within our 10,000-sequence corpus. Thus, the initial segment becomes diverse and contextually unrelated, while the remainder (the retained part ) maintains controlled repetition frequencies (1 to 256 repetitions). Each sequence includes only one initial BOS token, with no additional special tokens between swapped and retained segments (see Fig. 2). We train 1B and 8B models and evaluate verbatim memorization using prefixes from the retained part.

Table 3 shows that the principle demonstrated in this POC works surprisingly well regardless of model scale, keeping verbatim recall consistently low even at the highest exposure frequency. ROUGE-L and LCCS scores remain near the baseline similarity between unrelated texts across all frequencies, indicating little to no extractable memorization . As repetition increases, perplexity steadily declines, particularly in the 8B model, yet without corresponding gains in verbatim overlap. Both TTR and MAUVEremain stable, with the 8B model producing more coherent and lexically rich outputs. These results confirm that shifting sensitive content away from the context window's beginning is a simple but effective principle for mitagation at all scales.

Our POC mitigation strategy also neutralizes the offset effects entirely. Recall that this effect manifests in two ways: (1) shorter prefixes are most effective at triggering memorization when taken from the beginning of the context window, and (2) even short prefixes lose their efficacy when shifted to later positions, though longer prefixes can partially compensate. However, both effects disappear when the memorization target is displaced 4,000 tokens deep, as in the Swapped Gutenberg setup. Table 9 shows that increasing prefix length no longer improves memorization, and Table 10 reveals consistent recall across offsets from 0 to 2048 tokens.

This disappearance results from two key design choices: the swapped segment is randomly sampled across sequences that break fixed positional associations; and the memorization target is embedded far from the model's typical retrieval anchor near the beginning of the context window.

Furthermore, this method not only suppresses memorization but we also find that it improves generation quality. Unlike the sparse Gutenberg setting, where prompting from offset positions leads to lexical degradation and thematic looping, Swapped Gutenberg maintains high coherence and diversity across all conditions. As shown in Table 3, MAUVE and TTR scores remain stable and high, in stark contrast to the quality collapse observed in Sparse Gutenberg setting (also see Figure 1b and 5). These findings highlight a central implication of our study: by decoupling sensitive content from early-context anchors, it is possible to suppress extractable memorization while preserving generation quality and downstream task performance (see Table 12) across model scales and training regimes. This provides a compelling justification for future research into developing more practical, production-ready mitigation techniques based on this principle. Examples are shown in Fig. 12.

8 Batch Size Impact

Our experiments reveal a clear advantage to pretraining language models with smaller batch sizes, which necessitates more iterations given a fixed compute budget. This advantage manifests as reduced verbatim recall (Fig. 6a), improved language coherence (Fig. 6b), and enhanced performance across most downstream benchmarks (Appendix D.2). Notably, these improvements emerge despite maintaining both a constant proportion of Gutenberg sequences in each batch and consistent total training tokens across all experiments. To the best of our knowledge, this has not been shown in prior work.

9 Related Work

Factors Known to Affect Memorization Several factors are known to influence a model's tendency to memorize and reproduce training data verbatim. (1) Model scale: larger models tend to memorize more due to their greater capacity to fit specific examples [Carlini et al., 2021, Lesci et al., 2024]. (2) Repetition frequency: memorization probability increases roughly log-linearly with the number of times a sequence appears during training [Carlini et al., 2023]; we confirm both effects in our setting (Appendix C.1). (3) Complexity: complex sequences seen only once can be latently memorized and retrieved via small parameter perturbations [Duan et al., 2025]. (4) Timing of exposure: sequences seen later in training are more likely to be retained than those encountered earlier [Kiyomaru et al., 2024, Lesci et al., 2024]. (5) Prefix length: while longer prefixes improve recall in prior work [Carlini et al., 2023], we find this effect depends heavily on the prefix's position within the context window (§ 4.1).

1

2

4

8

16

32

64

128

0

0

.

2

0

.

4

0

.

6

0

.

8

1

Rouge-L

(a) Rouge-L scores by repetition frequency

1

2

4

8

16

32

64

128

0

0

.

2

0

.

4

0

.

6

0

.

8

1

MAUVE

0.5MT

1.0MT

1.5MT

2.0MT

Figure 6: Impact of batch size on verbatim memorization and language coherence for 1B models, as measured by Rouge-L (left) and MAUVE scores (right) across repetition frequencies. Models were trained with batch sizes ranging from 0.5MT to 2.0MT, maintaining constant total training tokens. As batch size increases, we observe enhanced language coherence (MAUVE) at low repetition frequencies, while verbatim memorization (Rouge-L) increases most significantly for sequences seen more than 32 times. The gap between smallest and largest batch sizes reaches up to 0.15 in Rouge-L score at high repetition frequencies, corresponding to approximately 90 fewer tokens being memorized verbatim in the 500-token continuations. This suggests smaller batch sizes with more update steps reduce exact memorization while preserving language quality.

Preventing Verbatim Memorization During Pretraining Pretraining-based strategies have demonstrated promising results in reducing verbatim memorization without sacrificing model performance. One prominent strategy is data deduplication, which removes duplicate or near-duplicate sequences from the training corpus. Lee et al. [2022] showed that deduplication substantially reduces memorization while preserving downstream performance. Kandpal et al. [2022] further demonstrated that it lowers the success rate of extraction attacks. Another promising strategy is the recently proposed Goldfish Loss [Hans et al., 2025], which selectively masks tokens within n-gram windows during training. By consistently omitting these tokens from the loss computation, it prevents the model from learning exact token-to-context mappings, thereby disrupting extractable memorization while retaining language modeling capabilities.

10 Conclusion

We demonstrate that LLMs exhibit a pronounced positional bias in their memorization behavior, with verbatim recall most easily triggered by prefixes near the start of the context window. This offset effect not only distorts standard memorization risk assessments but also offers a new mitigation pathway: simply shifting sensitive content deeper into the context suppresses extractable memorization without degrading generation quality. Our findings further reveal that when retrieval fails-due to offset or insufficient exposure-models often degenerate into repetitive, low-diversity output, linking memorization to generation stability. These insights position offset as a critical and previously underexplored axis in the study of memorization, and suggest simple, scalable mitigation strategies that complement existing techniques. Limitations of our study and directions for future work are discussed in Appendix B.

References

Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differentially private BERT. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 6481-6491, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-emnlp.484. URL https: //aclanthology.org/2022.findings-emnlp.484/ .

Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle, 2022. URL https://arxiv.org/ abs/2207.14255 .

Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Gregory Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview. net/forum?id=Iq0DvhB4Kf .

Jon Brodkin. Openai urges trump: Either settle ai copyright debate or lose ai race to china. https://arstechnica.com/tech-policy/2025/03/ openai-urges-trump-either-settle-ai-copyright-debate-or-lose-ai-race-to-china/ , March 2025. Ars Technica. Accessed: 2025-05-11.

Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) , pages 26332650. USENIX Association, August 2021. ISBN 978-1-939133-24-3. URL https://www.usenix.org/ conference/usenixsecurity21/presentation/carlini-extracting .

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=TatRHT_1cK .

Kent Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. Speak, memory: An archaeology of books known to ChatGPT/GPT-4. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 7312-7327, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.453. URL https://aclanthology.org/2023.emnlp-main.453/ .

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu

Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025. URL https://arxiv.org/abs/2412.19437 .

Sunny Duan, Mikail Khona, Abhiram Iyer, Rylan Schaeffer, and Ila R Fiete. Uncovering latent memories in large language models. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=KSBx6FBZpE .

Joshua Freeman, Chloe Rippe, Edoardo Debenedetti, and Maksym Andriushchenko. Exploring memorization and copyright violation in frontier LLMs: A study of the new york times v. openAI 2023 lawsuit. In Neurips Safe Generative AI Workshop 2024 , 2024. URL https://openreview.net/forum?id=C66DBl9At8 .

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https://zenodo.org/records/12608602 .

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl

Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783 .

Abhimanyu Hans, John Kirchenbauer, Yuxin Wen, Neel Jain, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele, and Tom Goldstein. Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs. In Advances in Neural Information Processing Systems , volume 37, pages 24022-24045, 2025. URL https://proceedings.neurips.cc/paper_ files/paper/2024/hash/2ad2dffba5079687651226ac8752df97-Abstract-Conference.html .

Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: Closing the generalization gap in large batch training of neural networks. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 2038-2047, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-emnlp. 148. URL https://aclanthology.org/2022.findings-emnlp.148/ .

Jing Huang, Diyi Yang, and Christopher Potts. Demystifying verbatim memorization in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on

Empirical Methods in Natural Language Processing , pages 10711-10732, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.598. URL https: //aclanthology.org/2024.emnlp-main.598/ .

Daphne Ippolito, Florian Tramer, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher Choquette Choo, and Nicholas Carlini. Preventing generation of verbatim memorization in language models gives a false sense of privacy. In C. Maria Keet, Hung-Yi Lee, and Sina Zarrieß, editors, Proceedings of the 16th International Natural Language Generation Conference , pages 28-53, Prague, Czechia, September 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.inlg-main.3. URL https:// aclanthology.org/2023.inlg-main.3/ .

Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker. Perplexity-a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America , 62(S1):S63-S63, August 2005. ISSN 0001-4966. doi: 10.1121/1.2016299.

Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 10697-10707. PMLR, 17-23 Jul 2022. URL https: //proceedings.mlr.press/v162/kandpal22a.html .

Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. Copyright Violations and Large Language Models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 7403-7412, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.458.

Kimmo Kettunen. Can type-token ratio be used to show morphological complexity of languages? J. Quant. Linguist. , 21(3):223-245, July 2014.

Hirokazu Kiyomaru, Issa Sugiura, Daisuke Kawahara, and Sadao Kurohashi. A comprehensive analysis of memorization in large language models. In Saad Mahamood, Nguyen Le Minh, and Daphne Ippolito, editors, Proceedings of the 17th International Natural Language Generation Conference , pages 584-596, Tokyo, Japan, September 2024. Association for Computational Linguistics. URL https://aclanthology.org/ 2024.inlg-main.45/ .

Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8424-8445, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.577. URL https://aclanthology.org/2022.acl-long.577/ .

Pietro Lesci, Clara Meister, Thomas Hofmann, Andreas Vlachos, and Tiago Pimentel. Causal estimation of memorisation profiles. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 15616-15635, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.834. URL https://aclanthology.org/2024.acl-long.834/ .

Huayang Li, Tian Lan, Zihao Fu, Deng Cai, Lemao Liu, Nigel Collier, Taro Watanabe, and Yixuan Su. Repetition in repetition out: towards understanding neural text degeneration from the data perspective. In Proceedings of the 37th International Conference on Neural Information Processing Systems , NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.

Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out , pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https: //aclanthology.org/W04-1013/ .

Dominic Masters and Carlo Luschi. Revisiting small batch training for deep neural networks, 2018. URL https://arxiv.org/abs/1804.07612 .

Milad Nasr, Javier Rando, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from aligned, production language models. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=vjel3nWP2a .

Kanghee Park, Jiayu Wang, Taylor Berg-Kirkpatrick, Nadia Polikarpova, and Loris D'Antoni. Grammar-aligned decoding. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. URL https://openreview.net/forum?id=5G7ve8E1Lu .

Guilherme Penedo, Hynek Kydlíˇ cek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2024. URL https://openreview.net/forum?id=n6SCkn2QaG .

Krishna Pillutla, Lang Liu, John Thickstun, Sean Welleck, Swabha Swayamdipta, Rowan Zellers, Sewoong Oh, Yejin Choi, and Zaid Harchaoui. Mauve scores for generative models: Theory and practice. Journal of Machine Learning Research , 24(356):1-92, 2023. URL http://jmlr.org/papers/v24/23-0023.html .

David Rosenthal and Livio Veraldi. Training AI language models with third-party content and data from a legal perspective. Jusletter IT , (25-Maerz-2025), 2025.

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. URL https://arxiv.org/abs/1909.08053 .

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. ICLR , 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract accurately reflects the main contributions, while the Introduction includes additional findings of practical relevance to the study of verbatim memorization.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes we mentioned the limitations of our work in Appendix B.1.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We use larger batch sizes for efficiency, which may introduce minor numerical differences, but these do not affect the conclusions. For fully deterministic outputs, one can run our scripts with a fixed seed and a batch size of 1 on the same GPU type (e.g., GH200).

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release our pretrained checkpoint on HuggingFace, and include inference code, evaluation scripts, and FM-Probe buckets after the rebuttal.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We present our experiment settings in § 3 and Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Observed effects exhibit consistently large effect sizes across over 10 distinct pretraining runs (including 1B/3B/8B sparse, 1B dense, 1B/8B swap, batch size variations, and BOD ablation) and numerous subsequent evaluations. Formal statistical significance testing for all comparisons was therefore computationally prohibitive given this extensive experimental matrix.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, Appendix A.1 details information for the computer resources for each experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research is conducted with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses negative societal impacts like privacy breaches and copyright violations stemming from LLM memorization throughout the introduction. The work aims for positive impact by proposing mitigation strategies for these risks.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly cite all datasets and code packages used in the paper. The original sources and papers are referenced in the main text, and the corresponding licenses and terms of use are explicitly detailed in the Appendix.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLMs were only used to assist with language editing and improving clarity.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Implementation Details

A.1 Computing Infrastructure

All experiments were conducted on the Alps supercomputer at the Swiss National Supercomputing Centre (CSCS). 2 Each compute node includes four NVIDIA GH200 Grace Hopper Superchips, with each GH200 integrating a 72-core ARM-based Grace CPU and a Hopper H100 GPU, interconnected via NVLink-C2C. Each GPU provides 96 GB of CUDA memory.

Pretraining For most of our pretrain runs, we utilized a 15-node configuration. Each GPU processed a maximum micro-batch size of 3 sequences, each consisting of 8,192 tokens. Only one specific experiment, involving the 1B model trained with a global batch size of 240 sequences, required a larger setup of 30 nodes. Pretraining of the 1B, 3B, and 8B LLaMA models was orchestrated exclusively using Data Parallelism (DP). Given that the 8B model parameters comfortably fit into the memory of a single GH200 GPU, neither tensor parallelism nor pipeline parallelism was necessary for efficient training. The throughput and compute for each model scale are summarized in Table 4.

Inference & Evaluation Inference Time: On a single node, processing one FM-probe bucket (500-token prefix and 500-token suffix) takes ∼ 1.5 minutes for the 1B model and ∼ 3 minutes for the 8B model, including both text generation and evaluation (Perplexity, LCCS, EM, ROUGE-L, TTR). MAUVE is computed separately and takes ∼ 34 minutes per bucket of the same size on a single GH200.

A.2 Model

We pretrain our models using an adapted version of NVIDIA Megatron-LM [Shoeybi et al., 2020], incorporating modifications described in the main paper. Table 5 details the architectural specifications and training hyperparameters of our LLaMA-based models. The tokenizer we utilised is the OpenMath2-Llama3.1-8B 3 .

A.3 Reproducibility

To enhance reproducibility, we fix the global random seed to 42 across all experiments. However, this alone does not ensure strict determinism in deep learning workflows. In particular, using different batch sizes may trigger different optimized CUDA kernels, leading to minor numerical variations and non-deterministic outputs despite identical seeds and initialization 4 . For guaranteed reproducibility, running inference and downstream evaluation with a batch size of 1 on the same GPU type (NVIDIA GH200) using our provided scripts yields identical results. Due to time constraints and the scale of our experiments, we instead use a batch size of 20 for model generation to maximize throughput, and 4 for downstream evaluation.

B Limitations and Future Directions

B.1 Limitations

While our proof-of-concept mitigation validates the principle that positional displacement can suppress extractable memorization, it assumes prior knowledge of which training segments are sensitive and thus does not establish a directly actionable workflow for large-scale pretraining pipelines. Future work may explore how to operationalize this principle-e.g., by combining positional fragility with automated sensitivity detection or data-curation strategies-to mitigate memorization at scale in realistic training environments.

Moreover, our corpus, while comprising approximately 83B tokens, remains several orders of magnitude smaller than real-world foundation model datasets trained on trillions of tokens. As such, our findings may not fully capture the memorization dynamics present at production scale.

Finally, our study lacks formal privacy or generalization guarantees (e.g., differential privacy [Anil et al., 2022]) and is limited to left-to-right autoregressive models trained from scratch. The empirical scope does not extend to fill-in-the-middle or retrieval-augmented architectures, which may exhibit distinct positional behaviors.

B.2 Future Directions

Our analysis is limited to conventional left-to-right autoregressive models following the LLaMA architecture. We do not study models trained with Fill-in-the-Middle (FIM) objectives, such as OpenAI Codex [Bavarian et al., 2022] and DeepSeek-V3 [DeepSeek-AI et al., 2025], which reconstruct missing spans from both preceding and following context. This bidirectional formulation enables flexible anchoring and contextual recombination, and is conceptually similar to our mitigation strategy of displacing sensitive content deeper in the context window. We leave the study of positional fragility and memorization in FIM-trained models to future work.

More broadly, it remains an open question whether similar offset-sensitive memorization behaviors emerge in non-text modalities such as code, images, or audio, and whether comparable mitigation

strategies are effective. Extending offset-aware analysis to these domains may shed light on the generality of positional fragility and its broader implications for memorization and generation quality.

C Reproducing and Extending Prior Work

C.1 Exposure Frequency & Model Size: Interconnected Memorization Dynamics

As shown in Figure 7a, our analysis reveals a significant inverse relationship between model scale and memorization threshold. Following the vertical axis (repetition frequency), the 1B model requires approximately 64 exposures to demonstrate substantial verbatim recall of suffix tokens. This threshold decreases to approximately 32 exposures for the 3B model, and further reduces to only 16 exposures for the 8B model. This systematic pattern suggests that memorization efficiency scales inversely with model size. Extrapolating this trend, a 70B model, without specific memorization mitigation strategies, might memorize sequences after just 2-3 exposures.

50

100

250

750

1000 2000 3000 4000 5000

prefix

1

2

4

8

16

32

64

128

repetition

.171

.173

.179

.182

.180

.183

.180

.179

.183

.174

.178

.184

.184

.184

.183

.182

.181

.182

.172

.177

.181

.183

.184

.185

.186

.184

.181

.173

.177

.182

.183

.181

.185

.183

.182

.183

.175

.178

.183

.183

.184

.187

.186

.185

.185

.225

.235

.242

.250

.258

.247

.241

.228

.222

.524

.530

.528

.515

.491

.457

.413

.370

.331

.744

.744

.745

.711

.648

.566

.530

.480

LLama 1B Suffix 500 Offset 0

50

100

250

750

1000 2000 3000 4000 5000

prefix

1

2

4

8

16

32

64

128

repetition

.171

.175

.179

.181

.180

.181

.179

.182

.183

.174

.181

.182

.181

.184

.182

.182

.184

.180

.173

.180

.181

.184

.180

.184

.185

.182

.184

.176

.179

.182

.184

.185

.187

.185

.186

.187

.200

.206

.221

.228

.238

.244

.243

.235

.230

.496

.513

.502

.506

.481

.474

.450

.436

.431

.752

.756

.734

.722

.681

.660

.651

.623

.888

.884

.877

.882

.867

.854

.819

.800

.778

LLama 3B Suffix 500 Offset 0

50

100

250

750

1000 2000 3000 4000 5000

prefix

1

2

4

8

16

32

64

128

repetition

.173

.174

.180

.185

.183

.181

.183

.185

.186

.176

.180

.186

.184

.185

.185

.184

.185

.184

.176

.180

.184

.184

.185

.188

.188

.186

.187

.178

.183

.187

.191

.189

.193

.193

.196

.193

.269

.303

.342

.372

.397

.409

.411

.422

.416

.638

.663

.685

.710

.724

.718

.702

.719

.713

.871

.884

.886

.888

.886

.875

.877

.885

.889

.968

.965

.967

.965

.968

.969

.970

.967

.967

LLama 8B Suffix 500 Offset 0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

(a) Standard Loss

50

100

250

750

1000 2000 3000 4000 5000

prefix

1

2

4

8

16

32

64

128

repetition

.168

.173

.177

.181

.182

.182

.179

.179

.181

.171

.177

.182

.183

.183

.181

.182

.182

.182

.170

.175

.179

.182

.183

.184

.185

.184

.182

.172

.175

.179

.182

.182

.184

.183

.187

.184

.174

.176

.181

.184

.184

.185

.189

.186

.184

.196

.207

.208

.216

.212

.210

.211

.208

.206

.260

.256

.266

.269

.264

.267

.255

.247

.244

.290

.296

.309

.307

.298

.298

.294

.286

.270

LLama 1B Suffix 500 Offset 0

50

100

250

750

1000 2000 3000 4000 5000

prefix

1

2

4

8

16

32

64

128

repetition

.171

.172

.176

.181

.182

.182

.182

.181

.182

.176

.180

.183

.185

.182

.184

.182

.184

.183

.173

.179

.181

.181

.184

.184

.186

.183

.185

.176

.180

.183

.184

.183

.187

.185

.188

.186

.185

.190

.201

.206

.210

.208

.209

.209

.210

.260

.277

.277

.281

.268

.273

.265

.260

.267

.296

.296

.296

.310

.303

.310

.291

.301

.293

.306

.314

.332

.334

.323

.328

.322

.322

.314

LLama 3B Suffix 500 Offset 0

50

100

250

750

1000 2000 3000 4000 5000

prefix

1

2

4

8

16

32

64

128

repetition

.172

.174

.175

.180

.182

.183

.182

.181

.181

.174

.179

.182

.183

.184

.184

.182

.183

.183

.173

.177

.180

.182

.182

.186

.187

.184

.186

.177

.181

.185

.189

.190

.192

.195

.194

.193

.226

.241

.258

.252

.256

.253

.264

.267

.269

.298

.308

.321

.318

.310

.316

.315

.320

.316

.314

.317

.323

.336

.330

.347

.334

.335

.348

.321

.337

.347

.350

.349

.350

.351

.353

.346

LLama 8B Suffix 500 Offset 0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

(b) Goldfish Loss

Complementing this frequency effect, the horizontal axis (prefix length) reveals how different-sized models leverage contextual information. For smaller models (1B and 3B), shorter prefixes of around 50 tokens triggered the strongest verbatim memorization, supporting the hypothesis that initial tokens disproportionately influence memorization patterns. In contrast, the 8B model maintained consistent memorization even with longer prefixes, demonstrating that larger models develop more sophisticated contextual memory mechanisms that operate effectively across varied input lengths.

These dual observations regarding decreased repetition requirements and enhanced contextual processing in larger models together demonstrate that scaling fundamentally transforms memorization capabilities. This explains why larger language models demonstrate both improved generalization and increased memorization risk: their enhanced capacity enables them to memorize more efficiently while simultaneously developing more nuanced representations of linguistic patterns. Our results complement to [Carlini et al., 2023].

C.2 Goldfish Loss

We reproduce the extreme and standard memorization scenarios originally proposed by Hans et al. [2025], adapting them to our dense and sparse Gutenberg experiments, respectively. However, our implementation substantially scales up the original design by pretraining from scratch with sequences that are 4× longer and drawn from 100× more documents. We use the dense Gutenberg setting to tune the optimal Goldfish hyperparameters for the sparse Gutenberg, yielding a dropout ratio of 2% ( k = 50 ) and a context width ( h ) of 50 tokens.

500

500

500

500

500

400

400

400

400

400

00

300

0E

300

300

1B

200

200

200

200

200

00T

100

100

100

100

500

500

500

500

400

400

400

400

400

300

300

300

3B

200

200

200

200

100

100

100

100

100

500

500

500

500

500

400

400

400

400

300

300

300

300

8B

200

200

200

200

100

100

100

100

100

0.4

Rouge-L Score

0.6

0.8

1.0

0.0

Z'0

0.4

Rouge-L Score

0.6

0.8

1.0

0.0

0.2

Rouge-L Score

0.8

1.0

0.0

0.2

Rouge-L Score

0.6

0.8

0.0

0.2

0.4

Rouge-L Score

0.8

1.0

Standard

Goldfish

The sparse Gutenberg results confirm that Goldfish Loss effectively mitigates verbatim memorization across all model sizes. As shown in Figure 7b and Figure 8, models trained without memorization mitigation and evaluated with zero positional offset exhibit pronounced verbatim recall abilities that systematically increase with both exposure frequency and model scale. The standard-trained 8B model achieves Rouge-L scores above 0.95 for most test sequences after 128 exposures, while Goldfish Loss models maintain scores consistently below 0.4 even under the most favorable memorization conditions.

Beyond verbatim memorization examination, the sparse Gutenberg framework reveals that Goldfish Loss effectively prevents verbatim memorization without compromising downstream performance. In fact, models trained with Goldfish Loss maintain competitive performance across all benchmarks, with this advantage becoming more pronounced as model scale increases. The 8B Goldfish model demonstrates this trend most clearly, outperforming its standard counterpart on several tasks , as shown in Table 6. This exciting result demonstrating that pretraining-based memorization mitigation can simultaneously address copyright concerns and enhance model capabilities.

D Additional Results

D.1 BOD or Initial Tokens Ablation

50

100

250

750 100015002000300040005000

prefix

1

2

4

8

16

24

32

48

64

96

128

repetition

.167 .172 .174 .179 .179 .179 .179 .179 .177 .181

.171 .174 .180 .182 .182 .183 .182 .181 .181 .182

.168 .172 .178 .181 .181 .183 .183 .182 .180 .180

.170 .172 .176 .179 .180 .183 .181 .183 .185 .186

.170 .175 .181 .183 .184 .180 .184 .187 .184 .185

.175 .181 .191 .197 .198 .192 .195 .193 .192 .193

.209 .224 .235 .253 .245 .247 .239 .230 .227 .226

.386 .392 .397 .396 .385 .371 .353 .338 .294 .299

.513 .521 .514 .492 .477 .457 .442 .398 .352 .337

.670 .659 .643 .630 .622 .578 .543 .501 .455 .424

.755 .743 .750 .739 .702 .677 .645 .582 .509 .466

LLama 1B Suffix 500 Offset 0

50

100

250

750 100015002000300040005000

prefix

1

2

4

8

16

24

32

48

64

96

128

repetition

.166 .170 .176 .177 .177 .179 .180 .180 .179 .182

.174 .176 .180 .180 .182 .180 .181 .181 .180 .180

.171 .173 .175 .179 .178 .183 .182 .185 .183 .178

.168 .175 .180 .179 .180 .183 .182 .183 .183 .185

.170 .172 .179 .182 .183 .184 .183 .185 .185 .185

.171 .175 .181 .189 .192 .194 .193 .192 .194 .190

.173 .180 .190 .214 .223 .228 .231 .227 .223 .219

.175 .181 .205 .280 .307 .322 .313 .307 .289 .277

.170 .183 .223 .330 .356 .372 .372 .354 .338 .318

.173 .189 .241 .387 .434 .423 .442 .413 .391 .361

.175 .193 .268 .426 .451 .501 .482 .452 .417 .405

LLama 1B Suffix 500 Offset 50

50

100

250

750 100015002000300040005000

prefix

1

2

4

8

16

24

32

48

64

96

128

repetition

.166 .171 .176 .178 .179 .179 .178 .181 .180 .181

.170 .173 .178 .182 .180 .181 .182 .179 .180 .180

.168 .173 .178 .181 .179 .182 .181 .182 .181 .181

.170 .172 .178 .179 .181 .184 .183 .182 .182 .184

.167 .172 .181 .183 .183 .183 .185 .186 .186 .185

.169 .173 .180 .188 .191 .190 .191 .193 .192 .191

.173 .178 .184 .204 .209 .224 .221 .219 .220 .214

.173 .180 .190 .236 .258 .280 .281 .285 .269 .271

.172 .178 .200 .260 .286 .316 .324 .327 .315 .292

.171 .178 .200 .288 .326 .356 .371 .360 .353 .334

.176 .186 .207 .315 .346 .389 .382 .369 .365 .346

LLama 1B Suffix 500 Offset 100

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

D.2 Batch Size Impact on Downstream Performance

Larger batch sizes correspond to higher perplexity scores on validation data, with the 240 GBS (global batch size) model showing a perplexity of 22.86 compared to 18.71 for the 60 GBS model a 22.2% degradation in predictive performance despite seeing identical training content. These same models demonstrate poorer performance as the batch size increases. The 60 GBS model consistently outperforms larger batch variants on reasoning-intensive benchmarks; for example, it achieves 33.36% accuracy on ARC-c compared to just 29.52% for the 240 GBS model, where the baseline random guess is 25%.

Our findings resonate with established patterns documented in deep learning research regarding batch size impact. When models train with smaller batches, the resulting gradient calculations introduce natural variability between updates. This inherent noise effectively serves as an implicit regularization mechanism, as each weight adjustment follows a slightly different trajectory, preventing overspecialization to training examples. This aligns with [Masters and Luschi, 2018] that reduced batch sizes promote more robust generalization through stochastic optimization pathways, ultimately yielding superior performance on evaluation tasks.

Viewed from another angle, our work shows that fewer parameter updates yield reduced model generalization and increased verbatim memorization. This finding aligns with [Hoffer et al., 2017], which attributes large batch performance deficits primarily to reduced update frequency rather than batch size itself. The considerably fewer parameter updates with larger batches (42,501 at 240 GBS compared to 170,005 at 60 GBS) may accelerate verbatim memorization while simultaneously compromising broader generalization capabilities.

D.3 Swapped Gutenberg

E License for Existing Assets

Project Gutenberg We use texts from the Project Gutenberg collection (via HuggingFace), which consists of works in the public domain in the United States. In accordance with Project Gutenberg's terms 5 , we do not redistribute their format or branding. Our use is strictly limited to non-commercial academic research, in compliance with Swiss copyright law.

Fineweb-Edu We use the Fineweb-Edu corpus released on HuggingFace 6 , which is distributed under the Open Data Commons Attribution License (ODC-By) v1.0. Our use complies with this license and adheres to CommonCrawl's Terms of Use. The dataset is used solely for non-commercial academic research, and we do not redistribute the data or any derivatives.

F Demonstrations

Rouge-L Score: 1.0000

Line

True Sequence

Line

Model Generation

1 2

moved they came as conquerors among other races.

moved they cane as conquerors among other races.

3

In their primeval home and probably before the year B.C. 30oo, they had

3

In their primeval home and probably before the year B.C. 30eo, they had

4

already acquired a fair degree of civilization. They built houses,

already acquired a fair degree of civilization. They built houses,

5

ploughed the land, and ground grain into flour for their baking. The

5

ploughed the land, and ground grain into flour for their baking. The

6

family relations were established among them; they had some social

7

organization and simple form of government; they had learned to worship

7

8a god, and to see in him a counterpart of their tribal ruler.

8

a god, and to see in him a counterpart of their tribal ruler.

9

9

10

From their upland farms they must have looked eastward upon yet higher

10

11

mountains, rising impenetrable above the snowline; but to north and

11

mountains, rising impenetrable above the snowline; but to north and

12

south and west they might turn to lower regions; and by degrees, perhaps

12

sdad sap Kq pe istu sa o n t Ka i pe yanos

13

as they grew too numerous for comfort, a few families wandered off along

13

14

the more inviting routes. Whichever way they started, their adventurous

14

the more inviting routes. Whichever way they started, their adventurous

15

spirit led them on. We find no trace of a single case where hearts

15

spirit led them on. We find no trace of a single case where hearts

16

failed or strength grew weary and the movement became retrograde, back

16

failed or strength grew weary and the movement became retrograde,back

17

toward the ancient home. Spreading out, radiating in all directions, it

17

toward the ancient home. Spreading out, radiating in all directions, it

18

is they who have explored the earth, who have measured it and marked its

18

is they who have explored the earth, who have measured it and marked its

19

bounds and penetrated almost to its every corner. It is they who still

19

bounds and penetrated almost to its every corner. It is they who still

20

pant to complete the work so long ago begun.

20

pant to complete the work so long ago begun.

21

21

22

Before B.C. 2eo0 one of these exuded swarms had penetrated India,

22

Before B.C. 2ooe one of these exuded swarms had penetrated India,

23

probably by way of the Indus River. In the course of a thousand years or

23

probably by way of the Indus River. In the course of a thousand years or

24

so, the intruders expanded and fought their way slowly from the Indus to

24

so, the intruders expanded and fought their way slowly from the Indus to

25

the Ganges. The earlier and duskier inhabitants gave way before them or

25

the Ganges. The earlier and duskier inhabitants gave way before them or

26

became incorporated in the stronger race. A mighty Aryan or Hindu empire

26

became incorporated in the strongerrace.Amighty Aryan or Hindu empire

27

was formed in India and endured there until well within historic times.

27

was formed in India and endured there until well within historic times.

28

29

Yet its power faded. Life in the hot and languid tropics tends to

28

29

Yet its power faded. Life in the hot and languid tropics tends to

30

weaken, not invigorate, the sinews of a race. Then, too, a formal

30

weaken, not invigorate, the sinews of a race. Then, too, a formal

31

religion, a system of castes[8] as arbitrary as among the Egyptians,

31

religion, a system of castes[8] as arbitrary as among the Egyptians,

32

laid its paralyzing grip upon the land. About B.c. 6eo Buddhism, a new

EE

and beautiful religion, sought to revive the despairing people; but they

and beautiful religion, sought to revive the despairing people;but they

34

were beyond its help.[9] Their slothful languor had become too deep.

34

were beyond its help.[9] Their slothful languor had become too deep.

35

35

From having been perhaps the first and foremost and most civilized of

36

the Aryan tribes, the Hindus sank to

36

the Aryan tribes, the Hindus sank to

38The Chinee were great manufacturers,and coud

Rouge-L Score: 0.1940

Line

True Sequence

Line

Model Generation

2

3

4

1

as they grew to numerous for comfort, a few families wandered off along

south and west they might turn to lower regions; and by degreesperhaps

the more inviting routes. whichever way they started, their adventurous

to north and

2

3

4

1

the sun，theymightreachnewheavens and new earths in their tun.The

Chinese, too, after their long journey, have no old traditions to transmit

south and west they might turn to lower regions; and so, with the aid of

to north and

5

spirit led them on. We find no trace of a single case where hearts

5

to their sons; they have no _faith_,no hopes,no fears,only an

7

6

failed or strength grew weary and the movement becane retrograde,back

toward the ancient home Spreading outradiating in all directionsit

6

7

Chineseno matter how they may have curiously drawn pictures of their

peasant, or a peasant-farmer, in the heart of the Chinese mountains.

8

is they who have explored the earth who have measured it and marked its

8

ancient past, bear in their veins the trace of some degree of intellectual

9

bounds and penetrated almost to its every corner. It is they who still

9

culture Their homes are Chinese their customs are Chinese their

10

pant to complete the work so Long ago begun.

10

sorrow is Chinese, and their folklore is Chinese. They know how to

11

12

Before B.C. 2eee one of these exuded swarms had penetrated India,

12

11

be themselves, and they know how to love for others. In so far, at any

rate, as the Chinese claim their country, they have certainly made a

14

13

so, the intruders expanded and fought their way slowly from the Indus to

probably by way of the Indus River. In the course of a thousand years or

1415171831

13

beginning.

the Ganges. The earlier and duskier inhabitants gave way before them or

16

became incorporated in the stronger race A mighty Aryan or Hindu empire

THE PERIOD OF TRANSITION

17138 8

was formed in India and endured there until well within historic times.

From this splendidly successful civilization, the Chinese,like the

19

Yet its power faded. Life in the hot and languid tropics tends to

19

Greeks and Romans, fell back in a period of a hundred years They

2222

weaken not invigorate, the sinews of a race. Then too, a formal

stretched themselves to the verge of poverty and obscurity, and the

religion, a system of castes[8] as arbitrary as among the Egyptians,

other European nations, glad to share in their prosperity, came to

22

laid its paralyzing grip upon the land, About B.C. 6ee Buddhism, a new

22

their rescue. But,like the Scythians, the Chinese,the Persians, the

23

23

Macedonianseven the Romanswere not able to hold their own against

忆驭纪

were beyond its help. [9] Their slothful Languor had become too deep.

From having been perhaps the first and foremost and most civilized of

25

24

Nature and Time. Their empire was broken up and the nations that were

the only strong men in the world at the beginning of our era were

the Aryan tribes, the Hindus sank to be degenerate members of the race.

26

overthrown Yet with the Chinese, the transition was not a hard one;

seen in no favorable light.

We shall turn to look on them again in a later period; but they will be

28

27

they showed their willing but their unwilling ignorance. They were

naturally strong and vigorous, and self-possessed. They had the gift of

sensibility, and made the best of their power for the day and night of

[Footnote 8: See _The Formation of the Castes_, page 52.]

30

history. They were brave and vigorous-and yet they were always firm

0 eap yotu e peu Kaul *u6. aq o panatia Kau 4eum o peap pue

[Footnote 9: See _The Foundation of Buddhism_, page 16.]

manhood，and selfhood in itself; they had a noble conception of courage;

33

they knew well the value of strength and of its advantages; they knew

Meanwhile other wanderers from the Aryan home appear to the north and

34

well the nature of peace, and they found it as indeed their last and

west. Perhaps even the fierce Tartars are an Aryan race, much altered

35

great period of freedom had proved, to be the best of all possible

from long dwelling among the yellow peoples. One tribe, the Persians,

36

conditions.

moved directly west，and became neighbors of the already noted Semitic

37

Rouge-L Score: 0.1840

Line

True Sequence

Line

Model Generation

2

1

the people quite ignored. We see the heroes

2

1

generals and their counsels the _Iliad_ and the _odyssey_, not at all

the people of the larger cities, with their

3

flourishing sword and spear. We learn what Ulysses did not know of

3

unlike those of the Phoenicians, are more interesting to us than the

4

6

foreign lands.. We hear Achilles' famed Lament amid the dead, and note

the vague glimmering idea of a future life, which the Greeks had caught

perhaps from the Egyptians perhaps from the suggestive land of dreams.

4

5

6

a whole We have come down to the time when the Greek race was the

first conqueror of the world. The Romans, the modern European

mere tales of the Trojan war. We are interested in the Greek race as

7

7

nations, are the direct descendants of the Greek race, and the

8

With the year B.c 776 we come in contact with a clear marked

8

descendants of the ancient Greeks. The modern nations are the

9

chronology. The Greeks themselves reckoned from that date by means of

9

descendants of the Greeks and it is in them that we find the real

10

olympiads or intervals between the 0lympic games. The story becomes

10

history of the world We must study them in order to understand the

11

clear. The autocratic little city kings governing almost as they

11

history of our race. We must study them.

12

pleased,have everywhere been displaced by oligarchies. The few leading

12

13

nobles may name one of themselves to bear rule but the real power lies

We find only a few scattered remnants of the Greek race in the

14

divided among the class. Then, with the growing prominence of the

present-day world.The ancient Greeks themselves were not all Greeks.The

15

Pythian games [15] we come upon a new stage of national development. The

15

Greeks were Greeks, the Etruscans were Greeksthe Spaniards were Greeks,

16

various cities begin to form alliances, to recognize the fact that they

16

the Italians were Greeks, the Spaniards and the French were Greeks. The

ass ueq eesge

17

Greeks were the first to be Greek, the first to be Roman, the first to

219 232

18

brotherhood begins to extend beyond the circle of personal acquaintance.

19

18

be Christian. The Greeks were the first to be Greek-speaking, the first

to be Greek-learners. The Greeks were the first to be civilized. The

[Footnote 15: See _Pythian Games at Delphi_, page 181.]

20

Greeks were the first to be barbarians. The Greeks were the first

21

nations. The Greeks were the first to be the first conquerors. The

This period was one of lawmaking, of experimenting. The traditions, the

22

Greeks were the first to be the first conquerors. The Greeks were the

纪纪忆

guidance of the larger cities, the more complicated circles of society,

simple customs of the old kingly days, were no longer sufficient for the

纪纪忆℃

23

first to have been born as a race. The Greeks were the first to be

Christian. The Greeks were the first to be Greek-speaking.

which were growing up. It was no longer possible for a man who did not

like his tribe to abandon it and wander elsewhere with his family and

The Greeks were the first to be the first conquerors. The Romans were the

herds. The land was too fully peopled for that. The dissatisfied could

27

first to be the first barbarians. The Romans were the first to have

only endure and grumble and rebel One systen of Law after another was

28

been born as a Greek race. The Romans were the first to have been

29

hard, would refuse longer assent to it There were uprisings, tumults,

tried and thrown aside, The class on whon in practice a rule bore most

29

30

The Greeks were the first to have been born as a race. The Greeks were

ChristianThe Greeks were the first to have been the first conquerors.

bloody frays.

31

the first tohave been born as a race The Greeks were the first

Sparta, at this time the most prominent of the Greek cities, evolved a

32

conquerors. The Greeks were the first to have been born as a race.

code which made her in some ways the wonder of ancient days. The state

The Greeks were the first to have been born as a race, The Romans were

35

was made all-powerful; it took entire possession of the citizen, with

35

the

36 the purpose of making him a fighter, a strong defender of himself and of

36

Figure 10: Demonstration of verbatim memorization decay and text degeneration as offset increases in the Sparse Gutenberg setting. Each example shows a 500-token suffix generated from a 500-token prefix by 1B model, with the highlighted span indicating the tokens used for ROUGE-L calculation. At offset 0, the model reproduces the training suffix nearly perfectly. At offset 128, memorization is limited to the initial few tokens. By offset 2048, only scattered fragments are recalled, and the output shows thematic looping-e.g., repetitive structures such as 'The xxx were the first to...' emerge in place of coherent continuation.

Rouge-L Score: 0.1860

Line

True Sequence

Line

Model Generation

guns on their shoulders, and wore high

been the men who had been driven from the

2

boots and the high-crowmed conventional brigand hatsEver since we left

2

mountain, and they were the bestThe one who was talking to the man

3

Florence we had seen at intervals in the fields and woods a notice with

3

with the hammer was not a man who had seen the country before, but one

who had lived through it. The other man was a little taller, and as tall

5

against the bandits or convicts for whom our Florentine friends had

5

as we were and he had a face that was as white as a sheet, and his

7

6

prepared us. And now we seemed to have come face to face with two of

these brigands. But it turned out that there was little of the bandit

6

7

hands were as strong as the branches of a tree. He was a man who had

been a soldier, and he had come to the mountains with his family. The

8

about them save their appearance. Their guns were for birds, and later

8

man who was standing there was a young girl, and we were not sure

9

on we learned that the alarming signs were merely to forbid the

9

whether she was a girl or a child and the voice of the man was a

112131

10

trespassing of these very gentlemen.

11

10

child, but it was not a child The man was a man who had been a soldier

and had been a soldier, and he was now a man who had been a mother The

[Illustration: A SLIGHT 0BSTRUCTION.]

12

woman was a woman, and the man was a man, and they were both talking of

13

the mountainsThe girl was a little afraid,but the man was not.He had

14

A mile or two farther on, the road began to go down again. We were both

14

been a soldier, and he had been a mother, and he had been a father, and

15

glad to be on the machine after our walk We could see to the bottom of

15

he had been a child. The man had been a man, and the girl was a girl, and

16

the hill, and there was no one in sight J. let go the brake. None but

16

she was a woman, and the man was a man, and they were both talking of

17

toiling.Theyhowever, will sympathize with our pleasure in the

cyclers know the delight of a fiveminutes coast after hours of up-hil1

17

18

hands were very strong. He svung his hammer, and he swung his gun,

the mountains. The girl was silent, but the man was very loud and his

19

mountains near Siena. But when it was at its fullest, and the machine

19

and he swung his hand. He threw down his hammer and he threw up his gun

20

was going at the rate of about twenty miles an hour, and neither brake

20

and his gun was ready. He swung his hand and he swung the gun. The

21

nor back-pedalling could bring it to a sudden halt, a man (or the foul

21

hand of t

ther

man with the hamner swung the gun, and it was a gun. The

22

front of us. when we reached them only the first had crossed the road;

fiend himself) drove a flock of sheep out from the woods a few feet in

23

hand

dof the man with the gun swung the gun, and it was a gun. The man

24

of course, all the rest had to follow. They tried to go on right through

24

did not know what to do. He did not know that the hand with the hammer

25 267

the wheels,but only succeeded in getting under them, setting the

26

could fire a gun. He thought of the mountains, and he said, "I don′t

27

machine to pitching like a ship in a heavy sea But I held on fast; J.

Stood on the pedals and screwed the brake down; the Little wheel

172 23323

know what to do." He had not been in the mountains long enough to know

what

t to do. The woman had been silent for a long time, and then she

28 2 2

scattered the sheep like the cow-catcher of an engine, and we brought up

said, "I am a man,

in the gutter.

Before we stopped, Jbegan a moral lecture to the

30

shepherd，and was showing him howif themachine had goneover,the

31

32

lecture ended rather_immorally with_accidente voi_and _imbecile_.

consequences would have been worse for us than for his flock. The

the deadliest of all Italian maledictions

33

Line

True Sequence

Line

Model Generation

2PervyseHis sumer home is a little distance out from Liege.His wife

1

battery of 75′s stationed near

2

which was in the trenches in France He has been wounded in the

regiment of the German Army

3

sister-inlaw, and his three children were in the house when the Germans

3

thigh and has been taken to the hospital in Paris. He is wounded and

4

came. Peasants driven from their village, hid in the cellar His sister

4

has been brought to the hospital in Paris He has been wounded and has

5

took one child and hid in a closet. His wife took the two-year-old baby

5

been brought to the hospital in Paris.

6

and the older child and hid in another closet The troops entered the

6

7

house looted it and set it on fire. As they left they fired into the

7

I have been in the habit of visiting the wounded in France, and of

8

cellar The mother rushed from her hiding place, went to her desk and

8

returning to the same place in the trenchesI have been to the hospital

9

found that her money and the family jewels one a gift from the

9

in Paris, and to the hospital of the French Army, and to the hospital

10

husband's family and handed down generation by generation, had been

stolen. With the sister, the baby in arms the two other children and

10

11

in Paris I have been in the habit of visiting the wounded in France,

and of returning to the same place in the trenches. I have been to the

12

the peasants， she ran out of the garden. They were fired on. They hid in

12

hospital in Paris, and to the hospital of the French Army and to the

13

a wood. Then for two days they walked. The raw potatoes which they

13

hospital in Paris, and to the hospital in Paris.

14

gathered by the way were unfit for the Little one. Without money，and

14

1617

15

ill and weakened, they reached Holland. This lady is in a safe place

now, and

her testimony in person is available.

1515

I have b

returning to the same place in the trenches and of going to the hospital

been in the habit of visiting the wounded in France, and of

17

in Parisand to the hospital of the French Army and to the hospital

18 192

[ILLustration: THE GREEN PASS, USED ONLY BY SOLDIERS AND OFFICERS OF THE

18

in Paris, and to the hospital of the French Army, and to the hospital

BELGIAN ARMY.

20

French Army,and to the wounded of the French Army, and to the wounded

in Paris，and to the wounded in Franceand to the wounded of the

It gives passage to the trenches at any hour. The writer, by holding

2 2

of the French Army.

this,and working under thePrimeMinister'ssonbecame

Stretcher-bearer in the Belgian Army.]

23

I have been in the habit of visiting the wounded in France, and of

2425

returning to the same place in the trenches, and of going to the hospital

The apologists of the widespread reign of frightfulness say that war is

in Paris, and of going to the hospital of the French Army and of

26

always "like that," that individual drunken soldiers have always broken

26

returning to the same place in the trenches, and of visiting the wounded

28

facts.Itmeets neither the official orders，nor the cold method，nor

loose and committed terrible acts. This defense does not meet the

in France, and to the hospital of the French Army, and to the hospital

in France,

and to the wounded of the French Anmy.

28

the immense number of proved murders. The German policy was ordered from

the top. It was carried out by officers and men systematically under

I have been to the hospital of the French Army, and to the hospital of

31

discipline.The German war Bookissued by the General Staff，andused

31

the French Amy, and to the hospital of the French Army, and to the

232

by officers, cleverly justifies these acts. They are recorded by the

3

hospital of the French Army and to the wounded of the French Army.

German soldiers themselves in their diaries, of which photographic

reproductions are obtainable in any large library. The diaries were

34

Ihave been in the habit of visiting the wounded in France,and of

found on the persons of dead and wounded Germans. The name of the man

35

returning to the same place in the trenchesand of going to the hospital

and his company are given.

36

in Paris, and of going to the hospital of the French Army and of

37

returning to the same place in the trenches, and of visiting the wounded

38

38

in France, and to the wounded of the French

Rouge-L Score: 0.2100

On Sunday,

Rouge-L Score: 0.1940

Rouge-L Score: 0.1660

Line

True Sequence

Line

Model Generation

anddo _not_ strive to "keep them quiet." Allow little boys

1

and recreation. They should be taught from the earliest

2

2

1e ut Addeu pue 1njuaau aq o at1 Ao{ua pue aao o a6e aqssod

3

boisterous if they will; let them romp and run, clinb fences, trundle

3

respects, and to avoid all habits that are harmful. They should be

hoops, jump rope, go to dancing school, participate in military drills,

4

s

go coasting and skating, take swimming lessons, etc.

5

aoo une aq pinous auljaqma,o puepaueaaue

6

6

learning of the body, and to “remember this, for they shall ever find

7

seek to be alone is always a bad sign and should be carefully guarded

8

hearts and their lessons in love, for they shall ever find that the

against without its being known that such precaution is observed.

9

love of God has not any such limits as those of the flesh." They should

10

Furnish them liberally with instructive and innocent story books and let

10

see uh ao pue suo pogau oun

11

them read aloud to you or to each other. Take them to walk or ride when

11

this way they will learn to love all things that areof God,and they

12

you go, and strive to make companions of then as much as possible,

12

will learn to love and enjoy life and to be happy.

13

making whatever sacrifices are necessary to attain this end. Above all,

13

14

_encourage their making confidants of you_. Let then feel that they can

14

The question of "tobacco*" use in children is often asked. I am sorry to

16

a  au st ee jaeu ou ns Aue uo a e pue a

16

say that I know very well that this is a very serious question, and I

Do this, and you have thrown around them a bulwark of defence that will

really think that many children have suffered and died as a result of

17

withstand the repeated attacks of hosts of evil spirits when night

17

the use of tobacco I must tell my opinion on the subject, and it must

18:19 7

then-theymay beread to sleep，but noromping or playingNo strange

comes and they go to bed, let then learn to go to sleep at once; no play

18

19

be told in the way of truth as far as it goesTobacco is a drug,a

narcotic, and should be used only by those who understand its nature and

20

children s

should be allowed to sleep with yours; make them occupy

20

its effects. It should not be used by little children, as it may have

22

places of your children are sacred to them alone, Nor is it advisable

for children to sleep with a grown person of either sex and particularly

separate

rooms or at least separate beds; be sure that the sleeping

23

far reaching consequences. I am not sure, however， that the use of tobacco

animals, especially the larger ones, are highly sensitive to the

in small quantities is not perfectly harmless It is well known that

not with servants--all for obvious reasons.

24

effects of drugs, and this is not true of all drugs. It is well known

25

that some drugs act more strongly than any of the commonest substances,

The observance of all these precautions against influences that might

267

such as salt, water, etc. I am not sure that the use of tobacco in

excite sexual disturbance is most sacred in its character and most

very small quantities may not be perfectly harmless. It is known that

needful even in a religious point of view; for there should be

28

tobacco contains a narcotic which has a powerful effect on some animals.

_chastity_ above all things.

It is also known that the use of tobacco by animals has been known to

cause a great deal of suffering and death. I am not sure that the use

31

of tobacco by children is a danger to them. Children are very sensitive

to drugs, and it is well known that the use of some drugs is

CHAPTER IV.

ADOLESCENCE OF THE MALE.

Adolescence of the male embraces the period of life from the age of

fourteen or s

sixteen years to the age of twentyfive.

by that advance in the evolution of the generative apparatus in both

At

about the age of fourteen years"the period of youth is distinguished

sexes,and by that acquirement of its power of functional activity,

45which constitutes the state of _Puberty_-" At this age

45

Figure 12: Demonstration of how displacing legally sensitive content deeper into the context window effectively suppresses verbatim memorization while preserving semantic fidelity in the Swapped Gutenberg setting. Each example shows a 500-token suffix generated from a 500-token prefix by 1B model, with the highlighted spans indicating portions used for ROUGE-L calculation. (a) Both the true suffix (left) and the model-generated suffix (right) describe interactions and encounters with the Esquimaux community upon arrival at Nachvak, discussing their behavior, hospitality, and local surroundings. Despite thematic similarity, the generated text diverges notably in specific narrative details-such as dates of arrival (July 16th vs. November 8th) and described observations (presence of drift-ice versus whale sightings)-illustrating effective suppression of verbatim memorization with maintained lexical richness and narrative coherence. (b) The true suffix focuses primarily on parenting advice regarding child behavior, education, and guidelines to prevent inappropriate sexual behavior and maintain chastity, whereas the generated suffix addresses the use of tobacco among children and discusses its potential dangers and harmful effects, highlighting clear thematic divergence and effective memorization suppression.

Line

True Sequence

Line

Model Generation

１ 2 34 5 6

Nachvak. Danger in doubling the North Cape. Arrival at

Sangmiyok bay,

1

2

Nachvak._

3

July 16th.--After two or three hours sleep in our cabin, we went on

4

5

It was now a Sunday, the 8th of November. As it was a beautiful moonlight

night and we were much pleased with the prospect of seeing the

6

shore. The Esquimaux who had here a temporary station, about fifty in

6

heavens open, Brother Kohlmeister took a view from the deck of his

7

numberreceived us with every mark of attentionLoud shouts of joy

7

vesselfrom the distance of 2 P.M we had now passed most of the

8

resounded from all quarters, and muskets were fired in every direction.

8

difficulties we have encountered, and are now in a fair way to Nachvak,

9

They could scarcely wait with patience for our landing and when we

9

if our reception there should be favourable. We had scarcely enquired

10

pitched our tent, were all eager to assist; thus we were soon at home

10

of the Esquimaux what they knew of our voyage, when one of the men

11

among them Seven tents were standing on the strand, and we found the

11

repeated the story of the previous evening He told us that on the 6th

123

people here differing much in their manners from the people at Saeglek.

12

of November some of the men having been in the evening, that is, at

Aq pantesse am auan iou nnuseq jaueu pue isapou sen anoaeua jtau

13

night, engaged in hunting, while out on a cruise they observed

14

beggars and importunate intruders as at the latter place where beggary

14

something floating on the wind which they thought at first sight to

15 16

seemed quite the fashion, and proved very troublesone to us But we had

15

be a piece of a canoe-man’s tackle, or other necessary article but

16

which, in their opinion, proved to be a whale They had discovered our

17

general with abhorrence, and with a thief no one is willing to tradeWe

17

vessel They directed us to land at a point farther to the southward, and

18

have discovered, however, that that propensity is not altogether wanting

18

to return by way of Nachvak. We did so, and found that they had

19

in the northern Esquimauxwho,now and then，if they think that they

can do it without detection, will make a little free with their

20

19

already killed a whale, which they had taken to their vessel They

told us that they had no idea what a whale was, but that they had seen

neighbour′s property.

21

one while they were in the bayat a distance. They spoke to us very

22

kindly of the Esquimaux, and said that they had seen us at Nachvak,

The Esquimaux not only gave us a most hearty welcome but attended our

that we might be very comfortable there and that we should have

morning and evening prayers with great silence and apparent devotion.

sunshine all the year round They requested that we would remain with

Indeed, to our great surprise, they behaved altogether with uncommon

decorum and regularity during our stay.

26

25

them; and we were glad to receive their permission. They pointed out

to us the direction we should have to go to reach Nachvak. We were not

in the least surprised,however,at their conduct as it was a very

17th. Being detained with drift-ice at the mouth of the bay, we pitched

28

common thing for the Esquimaux to go a long distance out in the

our tent on shore. We examined the bay more minutely It extends to the

20

night-time to hunt whales, and return to their village in the morning,

except a few rocks, at some distance in the sea.The surrounding

West to a considerable depth, and is not protected by any islands,

31

before they were well established in the belief of our being there

absolutely safe. They informed us that they had no idea what a whale was,

U Auo puno, sT aunpuan pue uauueq pue daas uhtu Auan aue surenunom

32

but that they had seen one while they were out in the bay. They

found of great use among the Esquimaux, grows in great abundance.

the vallies. Here the _arnica montana_, which the Missionaries have

343563738

33

admitted that they had

Salmon-trout are caught in every creek and inlet.

38

the winter, and return to the

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper examines the offset effect for LLM memorization where LLM memorization is taken as text regurgitation from legal perspective. \n\nBuilt on top of existing works, the paper shows that the text memorization capability depends on where the prefix occurs (so called offset in the paper). More specifically, the memorization rate decreases significantly when an offset is applied (e.g. see Figure 3) where the authors train their models under controlled settings. Based on this observation, the paper proposes further a method that pretrains the LLM with offsets and shows that the memorization rate significantly decreases.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: the fact that the model is able to regurgitate a large chunk of text is an important phenomenon to be mitigated and the paper has brought very good arguments on it. The paper's main claim (relationship between offset and LLM memorization behaviour) is backed up with controlled experiments. \n\nWeakness: \nFirstly, the proposed technology arguably compromises too much performance. As shown in Table 2, the perplexity increases a lot compared to the base case to lower memorization ratio. \nSecondly, the offset settings are not clear. The notion is everywhere in the paper however, how offset is setup is only described a bit vaguely in line 68. Unforutunately, I don't know how the skip is implemented concretely from the paper (are they irrelevant tokens, etc.?) I can only guess irrevalent tokens because that is the method adopted in section 6. \nAfter reading the paper, I am not sure if offset simply makes the model perform worse in general (and by consequence have decreased memorization ratio) or the model only worse at memory. \nThe limitation section has not mentioned any of the above, unfortunately.\n\nQUESTIONS:\nCan the authors expand on offset definition so that everyone is able to implement it please?\n\nLIMITATIONS:\nThe limitation section Appendix B does not list significant limitations of the current work. The section is short and discusses that it is empirical, does not address differential privacy and finally a decoding technique coverage. The first one (or even first two) are two general and are not limitations to this particular work; while the last one is a very minor limitation that omitting does not hurt the paper. \n\nMeanwhile arguably there are many limitations to this work (as many of the research works) including testing on popular LLMs, training on more realistic settings, etc. just to name a few.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper looks into understanding verbatim memorization in large language models by introducing the concept of \"positional fragility\" and demonstrating how the position of content within the context window dramatically affects memorization behavior. The authors conducted systematic experiments training LLaMA-style models (1B, 3B, 8B parameters) from scratch to examine memorization patterns, revealing that verbatim memorization is most strongly triggered by short prefixes drawn from the beginning of the context window, with memorization decreasing counterintuitively as prefix length increases.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. Novel finding on the positional offset, memorization drops when content is positioned away from the beginning of the context window. It  challenges prior assumptions about uniform memorization behavior. It is an interesting finding that shifting sensitive sequences deeper into the context window significantly reduces extractability without compromising general performance\n\n2. The paper does very comprehensive analysis, it train LLaMA-style models on 83B tokens from scratch, it gives precise control over training dynamics and data composition. This work also targets at the full-context sequences four times longer and evaluates suffixes at least ten times longer. \n\n3. The \"Swapped Gutenberg\" approach shows that simply shifting sensitive content 4000 tokens deep into the context window effectively suppresses memorization while maintaining generation quality. \n\n4. The connection between memorization and text degeneration is interesting and important. The observation that models produce repetitive, low-quality output when they cannot retrieve memorized content suggests that memorization and generation quality are more interwined than previously understood.\n\n**Weakness**\n1. The model trained on 83B tokens from scratch is very under-trained and curious whether this would impact the model's behaviors on memorization. It would be interesting if the authors could study the same phenomenon by continuing pretraining an existing LMs and compares the difference of memorization behaviors.\n\n2. The paper is the first to show that pretraining language models with smaller batch sizes could greatly reduce the memorization. It would be interesting to do further analysis on why this is the case because this result challenges conventional finding from the differential privacy literature, where larger batch sizes typically diminish the influence of individual examples and are generally preferred for privacy preservation. \n\n3. The Swapped Gutenberg is an effective approach but it requires encoding of an long input prefix and does the cross attention. This may not be very practical to deploy in the real world. It would be interesting to further quantify the potential side effect of Swapped Gutenberg in terms of downstream task performance. The long prefix may distract the model from using the useful knowledge encoded in the model weights. One related pointer is from Gemini 2.5 report discussing that when the context goes too long, the model is stuck in the context poisoning problem, hard to decide when to generate new information and retrieve information in the existing context. \n\n4. It would also be interesting to delve deeper into how the swapped part affects generation. Is memorization reduced because the model has a tendency to copy directly from the swapped part?\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the memorization in LLMs, with an emphasis on sensitive data memorization such as copyright-protected data. The authors introduce a new aspect to be considered in memorization analysis, offset effect, and identify new phenomena, positional fragility---memorization is significantly more likely when the probing prefix appears near the start of the model’s context window, and degrades sharply when the prefix is offset deeper into the sequence. To study this, the authors pretrain LLaMA-style models (1B, 3B, 8B) on 83B tokens using a mix of FineWeb-Edu and Project Gutenberg, with careful control over exposure frequencies and token positions. They introduce the Swapped Gutenberg experiment, where the first 4K tokens of each training sequence are replaced with random text to push potentially sensitive content deeper into the sequence. This reduces memorization without compromising output quality.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- This paper introduces a novel axis for analyzing memorization in LLMs: the position of a prefix within the context window during training and evaluation.\n\n- The experiments are rigorous and comprehensive, with strong empirical support across model scales and memorization frequencies. The authors pretrain LLaMA-style models (1B, 3B, and 8B) on an 83B-token corpus specifically for this study.\n\n- The finding that failures in verbatim memorization can lead to output degeneration adds a valuable dimension to existing work on repetition-induced degeneration.\n\n- The authors propose a simple but effective memorization mitigation strategy, displacing sensitive sequences later in the context window during training. This method significantly reduces extractable memorization without compromising generation quality. \n\n**Weaknesses** \n\n1. Limited practical utility of findings \n\nWhile the findings are novel and empirically supported, the practical application of the mitigation strategy remains unclear. Specifically:\n\n- The authors propose shifting sensitive content deeper into the context window (Swapped Gutenberg setup), but do not discuss how such a strategy could be applied during LLM pretraining, especially in large-scale, distributed training pipelines.\n- In real-world scenarios, training data is rarely labeled for sensitivity, and existing pretraining frameworks do not typically support fine-grained control over token placement or prefix location during batching.\n- The paper would benefit from a brief discussion of the feasibility of operationalizing this technique; for instance, through modified data loaders, selective shuffling, or integration with deduplication or data curation workflows.\n\nAdditionally, while I appreciate the authors’ efforts to pretrain models specifically for this study, applying this analysis to existing models is difficult in practice. For example, models like LLaMA do not release their pretraining data, and even open-source efforts like OLMo do not provide detailed logs (e.g., which sequences appear in which batches), making it nearly impossible to retroactively analyze prefix positions or apply these findings directly to such models.\n\n2. Unrealistic deployment assumptions \nThe mitigation strategy seems to assume that:\n- The model developer knows which parts of the training data are sensitive. \n- The developer has control over where in the sequence those parts are placed.\n\nIn practice, especially with trillion-token-scale training, we often lack precise metadata on content sensitivity, and restructuring input sequences at that scale would require non-trivial changes to data pipelines.\n\nOverall, while this paper contributes a compelling new perspective, the evaluation is conducted in a highly controlled, clean experimental setting, where Project Gutenberg is treated as the sensitive subset and FineWeb as safe. The analysis also relies on precise knowledge of where a prefix appears during training and evaluation, which may not be realistic in most practical scenarios. These assumptions limit the generalizability of the proposed analysis and mitigation to real-world LLM training.\n\nQUESTIONS:\n1. How do you envision operationalizing the \"Swapped Gutenberg\" mitigation in large-scale pertaining? \n2. In your setup, Project Gutenberg is used as the sensitive subset - what motivated this choice, and can the experiments be applied to other domains or setups such as PII or other sensitive data domains such as medical domains?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}

comments: SUMMARY:
This paper focuses on understanding memorization of training data. The authors pretrain three Llama-style transformers on a mix of Project Gutenberg books and web texts, and find (1) short prefixes at the start of the context window lead to highest memorization, whereas shifting this prefix 64-128 tokens deeper can almost eliminate this effect, (2) models that fail to retrieved the memorized sufix will produce nonsensical outputs, and (3) inserting sensitive text further away from the beginning is a simple mitigation strategy.
The authors also show several related secondary analyses related to batch sizing and model size.

STRENGTHS AND WEAKNESSES:
This paper analyzes an important downstream impact of an LLM's tendency to focus on early tokens. The experiments are clear to understand, and the problems and solutions presented are helpful. The authors show results across several model sizes, vary the exposure counts, and ablate batch sizing. The paper also alludes to a connection between memorization and model degeneration, which is helpful.

QUESTIONS:
What happens if you move away from greedy sampling?

How does the size of the special anchoring zone change according to model size?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-06