ICLR 2025

Published as a conference paper at ICLR 2025
MEASURING NON-ADVERSARIAL REPRODUCTION
OF TRAINING DATA IN LARGE LANGUAGE MODELS
Michael Aerni∗1
Javier Rando∗1
Edoardo Debenedetti1
Nicholas Carlini2
Daphne Ippolito2,3
Florian Tramèr1
1ETH Zurich
2Google DeepMind
3Carnegie Mellon University
ABSTRACT
Large language models memorize parts of their training data. Memorizing short
snippets and facts is required to answer questions about the world and to be fluent
in any language. But models have also been shown to reproduce long verbatim
sequences of memorized text when prompted by a motivated adversary. In this
work, we investigate an intermediate regime of memorization that we call non-
adversarial reproduction, where we quantify the overlap between model responses
and pretraining data when responding to natural and benign prompts. For a variety
of innocuous prompt categories (e.g., writing a letter or a tutorial), we show that up
to 15% of the text output by popular conversational language models overlaps with
snippets from the Internet. In worst cases, we find generations where 100% of the
content can be found exactly online. For the same tasks, we find that human-written
text has far less overlap with Internet data. We further study whether prompting
strategies can close this reproduction gap between models and humans. While
appropriate prompting can reduce non-adversarial reproduction on average, we
find that mitigating worst-case reproduction of training data requires stronger
defenses—even for benign interactions.
1
INTRODUCTION
Large language models (LLMs) must memorize parts of their training data, including facts and
idioms, to generate fluent text and answer questions about the world. The rate at which LLMs
memorize atomic facts or word constructs (e.g., small ngrams) is measured by general knowledge
benchmarks (Hendrycks et al., 2020) and studies of linguistic novelty in LLMs (McCoy et al., 2023;
Nguyen, 2024; Lu et al., 2024). While this form of memorization is...