Quantifying Memorization
Across Neural Language Models
Nicholas Carlini, Daphne Ippolito, Matthew Jaglieski, Katherine
Lee, Florian Tramer, Chiyuan Zhang
Published at ICLR 2023

Vishnu Asutosh Dasu
CSE 587: Deep Learning for NLPIntroduction

Introduction• Large language models (LMs) have been shown to memorize training datawhen using carefully crafted prompts• Memorization is undesirable as it a ects privacy, utility, and fairness• Memorization is worse than actually believed and will be getting worse unless
we address it• Prior works show a loose bound on how much data can be extracted andff

they do not explain how memorization varies across models and datasets

Introduction• Two improvements over prior work
• Better lower bound on how much data is memorized• Explain how memorization varies across di erent LMs and datasets of di erent
scales• Three properties a ect memorization:• Model Scale: Larger models in same family memorize 2-5x more than smaller ones
• Data Duplication: Examples repeated more often are more likely to be extractableffffff• Context: It is much easier to extract text when the context is longer

Methodology

Definition of Memorization

Example• Training Sequence: My phone number is 555-6789• Pre x string (p) with length k=4: My phone number isfi

• The sequence is extractable at k=4 if the most likely output is 555-6789

Selection of Evaluation Data
Uniform Random Sampling• Generate uniform random sample of 50,000 sequences from training dataset
without repetition• Useful to measure absolute memorization• Cannot be used to study how memorization scales with data properties thatare not represented uniformly in the subset• Does not account for:• Prompt length• Data duplication

Selection of Evaluation Data
Normalized Random Sampling• Generate subset normalized by duplication and sequence length• For each length l in {50, 100, 150, …, 500} and integer n, select 1,000

sequences of length l that are contained between 2n/4 and 2(n+1)/4 times• Repeat until you reach a value of n for which 1000 sequences are notavailable• Biased sampling allows to measure memorization as a function of sample’s
duplication factor and sequence lengthSelection of Evaluation Data
Evaluation Method• For each length 50 to 500, collect 50,000 samples of varying duplication for atotal of 500,000 sequences• For each sequence of length l, prompt the model with rst l - 50 tokens• Sequence is “extractable” if model emits next 50 token su x
• Compute average probability that sequence is extractable by averaging overffifi

all lengths lExperiments

Setup• GPT-Neo is used as the language model• GPT-Neo is trained on the Pile dataset• Baseline model for comparison is GPT-2 trained on WebText• Both, GPT-2 and GPT-Neo, are prompted from the Pile dataset

Bigger Models Memorize MoreResults

Bigger Models Memorize MoreAnalysis• Biased sampling is used so we
don’t care about absolute
numbers of memorized data

• Results show that there is a log-

linear relationship between scale
and memorization• Baseline shows that this ismemorization and notgeneralization

Repeated Strings Are Memorized More
Results

Repeated Strings Are MemorizedMoreAnalysis• 1,000 distinct sentences, each
repeated 2 to 900 times• Less memorization for lessduplication• Similar log-linear trend isobserved• Memorization happens even withvery few duplicates

Longer Context Discovers More Memorization
Results

Longer Context Discovers MoreMemorizationAnalysis• Discoverability Phenomenon:

Some memorization becomesapparent only for long contexts• Some strings are “hidden” in themodel and require more
information to be extracted

Alternate Experiment Settings
Random Dataset Sampling• Randomly sample 100,000
sequences• Similar trends are observed formodel size and context length

Alternate Experiment Settings
Alternate Decoding Strategies• Experiments so far were

performed using the greedy
decoding strategy where themost likely token was chosen
during decoding• Experiments were performed
again with beam search• Beam search is slightly better
than greedy decoding

Alternate Experiment Settings
Alternate de nition of extractability• Previous experiments assumed

that the su x extracted has tobelong to that particular training
sample• Now the su x that is extracted
can belong to any training
sample in the dataset• Naturally, this results in morefi

ffi

ffi

extractions

Qualitative Examples of Memorization

Replication Study

T5 Masked Language Modeling• For masked language models, string is “extractable” if model can perfectly ll
in all the masks• T5 models are trained by removing 15% of tokens from each training
sequence and model then must ll in the blanks of the sequence• Example: A 200-token sentence is memorized if model can use 170 (200 *fifi0.85) tokens of context to predict remaining 30 (200 * 0.15)

T5 Masked Language Modeling

Model Size Results• Similar trend is observed wherelarger models memorize more

• However, masked modelsfimemorize signi cantly lesser
than causal models

T5 Masked Language Modeling

Data Duplication Results• No monotonic scaling relationship isobserved and results are noisy with highvariance• Samples repeated 158 - 196 times arememorized with probability less than 5.1%

• Samples related 138 - 158 times arememorized with probability at least 6.2%

• Samples occurring ~140 times are more

likely to be memorized despite occurring
less often as these samples consist of
mainly whitespace tokens which can bepredicted easily

Language Models Trained on Deduplicated Data
• 1.5B parameters causal models trained on C4 with data deduplication• Two types of deduplication:• Remove all documents that are near-duplicates of other documents• Delete any string of length-50 tokens that occurred more than once

Language Models Trained on
Deduplicated DataResults• Models trained on deduplicateddatasets memorize less

• Extractability of samples

ff

repeated at least 408 times ismuch higher as deduplication
strategies are imperfect andcannot e ectively scale to large
training data

Language Models Trained on Modified Version of the Pile

• OPT models were trained on modi ed version of Pile which contains datafrom many other sourcesfi• Dataset was deduplicated before training to reduce repetitions

Language Models Trained on
Modified Version of the Pile

• Similar trends to GPT-Neo

trained on Pile but much lesser
memorization• Two possible reasons:• Careful data curation canreduce memorization• Slight shifts in data distribution
can alter what content getsmemorized

Additional Results

Longer Documents Are Not Easier toMemorize than Shorter Documents• Longer sequences are rare and if
perplexity is low then it is likelythat longer sequences are justmemorized i.e. e ect of larger
context is not relevant• For varying sequence and prompt

lengths, extract the next 50 tokens• No di erence between fraction offf

ff

extractable tokens by varying
prompt lengths across sequencelengths

Text Memorized by Only Some Models• Larger models have more uniquely memorized sequences• However, every model memorizes some amount of unique information

Conclusion

Conclusion• Paper presents quantitative analysis of memorization in LMs• Two primary conclusions:• Generalization: LMs accurately model statistics of training data but do notmodel the underlying data distribution correctly• Privacy: LMs memorize signi cant fraction of their training data.

Memorization scales log-linearly with model size and is often hard todiscover• Data deduplication strategies seem to be a promising direction to reducefi

memorization