Extracting Training Data from Large
Language Models
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss,
Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, Colin
RaffelPublished at USENIX 2021

Background• Large Language Models (LLMs)• tremendous model parameters• massive training samples• Previous consensus• LLMs are well generalized, exhibiting little to no overfitting

Background• Privacy attacks• Membership Inference• Model Inversion[1][1]Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures

Background• Overfitting & Memorization• Success of privacy attacks à overfitting to training samples• Erroneous consensus: LLMs do not overfit à LLMs do not memorize

Threat Model• Definition of memorization• Eidetic Memorization / photographic memory

Threat Model• Adversary’s Capabilities• black-box input-output access• public API• Objective• extract memorized training data from the LLM• strength of attack: k & NInitial Attack• Generate• initialize the LM with a one-token prompt• sample tokens for each trial• Membership Inference• Perplexity• Lower perplexity means that LM is not surprised by s

Initial Attack Results• Attack can find some memorized documents• entire text of the MIT license• popular individuals’ Twitter, email addresses• Weaknesses• low diversity of outputs• too many false positives• produce samples not memorized (repeated strings)

Improved Attack• Improved text generation• Improved Membership InferenceImproved Text Generation•z=• softmax(z)• softmax(z/t) t>1• higher temperature à less confident, more diverse• decaying temperatureImproved Text Generation• Conditioning on Internet text
• GPT-2 follows Reddit links• The author followed Common Crawl

Improved Membership Inference• Naive approach also finds high likelihood outputs:• Trivial memorization• repeated numbers from 1 to 100• Repeated substrings• “I love you. I love you. …”• Intuition• compare the target model to a second model• filter samples where the target model’s likelihood is unexpectedly highImproved Membership Inference• comparing to other LMs• two models are unlikely to memorize the same sample with small k• comparing to zlib compression• efficiently filter repeated substrings• comparing to lowercase text
• compare perplexity of original text to lowercased text

• perplexity on a sliding window• one memorized substring surrounded by non-memorized contexts

Evaluation• Build datasets of 200,000 generated samples• Initial attack• Decaying emperature• Internet conditioning

Evaluation• Order these datasets using 6 membership inference metrics• perplexity (initial)• small (use Small GPT-2 as the second model)• medium (use Medium GPT-2 as the second model)• zlib• lowercase• sliding windowEvaluation• 3 × 6 = 18 configurations• 18 × 100 = 1800 samples• Manual inspection• search online for exact match• Validate results on the original training dataset

Evaluation

Evaluation• most samples located ata diagnol• samples at the top-left:• GPT-2 is not surprised• zlib is surprised

Evaluation• Best case• 87 tokens• k=1• 10 times repeatedEvaluation• 46 samples containing individual peoples’ names• 16 contain contact information for businesses• 16 contain private contact information• URLs• 50 samples• Code• 31 samples• Unnatural text
• 1e4bd2a8-e8c8-4a62-adcd-40a936480059• IDs for ad tracking• git commit hashes

Evaluation• Extract longer strings• 256 tokens by default• 1450 verbatim loc• entire MIT license

Evaluation• Model size
• extract the same URL from
GPT-2 with different modelsizes

Potential Mitigation• Differential privacy• tradeoff between privacy and model utility• access to labels• Curate the training dataset• manually filtering sensitive training samples• Limit impact of memorization on downstream applications.
• fine-tuning may cause the LM to forget some training samples• introduce new privacy leakage• Audit models for memorization

Takeaways• Extraction Attacks Are a Practical Threat• Memorization Does Not Require Overfitting• Larger Models Memorize More Data
• Memorization Can Be Hard to Discover• Adopt and Develop Mitigation Strategies

Discussion & Questions