Scalable Extraction of Training Data from Aligned, Production Language Models

Authors: Milad Nasr, Javier Rando, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Florian Tramèr, Katherine Lee
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: privacy, language models, data extraction, security

Abstract:
Large language models are prone to *memorizing* some of their training data. Memorized (and possibly sensitive) samples can then be extracted at generation time by adversarial or benign users. There is hope that *model alignment*---a standard training process that tunes a model to harmlessly follow user instructions---would mitigate the risk of extraction. However, we develop two novel attacks that undo a language model's alignment and recover thousands of training examples from popular proprietary aligned models such as OpenAI's ChatGPT. Our work highlights the limitations of existing safeguards to prevent training data leakage in production language models.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730609730330

thread_type:
official_review

<thread_object>
creator: reviewer_jxae

<content>
summary:
The paper conducts a large amount of empirical research and finds that aligned chat models hardly leak training data. However, when the authors implemented the divergence attack and fine-tuning attack, the models leaked some training data, demonstrating significant security vulnerabilities in current large language models. The paper conducted a large number of experiments to validate the various negative effects on the model after being attacked.

strengths:
- The paper conducted a large number of experiments to reveal the extraction attacks faced by large language models, and the models used in the experiments are very representative.
- The research problem addressed in the paper is very interesting; extraction attacks are an important topic for large language models.
- The structure of the paper is very well-organized, with rich details such as explanations and definitions for memorization, making it easy to read.

weaknesses:
- I think that the paper lacks innovation or technical contribution. Although the two attack methods proposed in the paper reveal security issues with large language models, I think such contributions may not be sufficient for a top conference like ICLR.
- The divergence attack proposed in the paper is intriguing, but why does this attack work? Under what circumstances does it work? It seems that this attack may not enable targeted attacks (i.e., leaking specific information from the model). There appears to be a significant random component, which means that the efficiency of this type of attack may be low for the attacker.
- It seems that the authors did not discuss the relationship with related works. Some adversarial attacks also seem to achieve similar effects. What are the main differences between the authors' work and related works?

questions:
Please refer to the ``Weaknesses`` part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695827981

thread_type:
official_review

<thread_object>
creator: reviewer_9cxv

<content>
summary:
This paper considers two approaches to extract a large language model's training data. The first is using repeated words, and the second is by fine-tuning the model to break the safety training. The approach is tested on proprietary models, and shown to regenerate sentences from the open source datasets with verbatim tokens over a threshold. While the first approach does not always work, fine-tuning could easily circumvent the defense mechanism put in the model.

strengths:
- S1: The approaches are simple and effective without too many assumptions.
- S2: The attacks are shown to work on the state-of-the-art commercial models.
- S3: The presentation is good overall and the paper is very easy to read.

weaknesses:
- W1: The related work section is missing. Although there is the background section, the paper does not properly cover the related work and its relation to the existing work, as well as potential defenses in the literature. Especially, these attacks are known and discussed in different forums. There is a potential that the authors of the paper might be those who suggested and discussed these approaches early on, but some mention of the context is useful understanding the literature and the significance of this approach.
- W2: The paper defers a lot of information to the appendix. Although this abundance of information comes from the thorough analysis and investigation, the paper needs to prioritize more essential information and drop potentially duplicate or obvious information.

questions:
- Q1: What are the examples of the (near) duplicate generations and their significance?

The paper is overall well written and the extensive analysis is helpful, the paper can be improved with better use of prioritization in space, and a proper related work section. Especially, discussing how novel the proposed approach would be helpful understanding the impact of this paper. This might need a significant reorganization of the paper, but all the ingredients should be already there. If that can be done, I'm willing to upgrade my recommendation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695983900

thread_type:
official_review

<thread_object>
creator: reviewer_tvra

<content>
summary:
This paper pointed out the key reasons of the ineffectiveness of the model alignment and developed two novel techniques to circumvent chatbot alignment guardrails: a divergence attack and a finetuning attack.  The author demonstrated that this is the first large-scale, training-data extraction attacks on proprietary language models using only publicly-available tools and relatively little resources. This work highlights the limitations of existing safeguards to prevent training data leakage in
production language models. And the experiment results show the model alignment is not enough to prevent memorization.

strengths:
1. The contributions are valid and significant. This work highlighted the limitations of existing safeguards to prevent training data leakage in
production language models. The author proposed two novel extraction attacks illustrating the limitation of model alignment of training-data extraction. The attacks only require access to tools that are publicly accessible to everyone. In addition, the author proposed a scalable approach to validate memorization. 

2. The paper does a comprehensive research showing additional work in long Appendix with sufficient experiments. 

3. The paper has good structure by clarifying key definitions and prompting the motivation. In experiments, the author clearly described the scalable approach for validating memorization and what are the production language models, including both aligned, conversational models and instruction-tuned, non-conversation models.

weaknesses:
1. (not a weakness but a suggestion). During the reading, I found some figures and conclusions in the Appendix is helpful and may worthwhile to be added or replaced to the main body. For example, Figures in Appendix A.9

2. In section 7 QUALITATIVE ANALYSIS OF EXTRACTED TEXT, it seems the result analysis focuses on the length of the extracted string and memorized text. It may better if the author could add more explanation in terms of the leakage of random training data from divergence attack vs the leakage of specification training data from fine-tuning attack.

questions:
1. [line 047] It said they apply divergence attack to ChatGPT and Gemini but apply finetuning attack to ChatGPT only. Is there a particular reason why they doesn’t apply finetuning attack to Gemini?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709494537

thread_type:
official_review

<thread_object>
creator: reviewer_4xm4

<content>
summary:
This paper compares pretrained base models and aligned production models using a simple completion attack to extract training data. The findings indicate that the alignment process appears to prevent models from directly outputting training data when faced with this straightforward attack.
To bypass the defense mechanisms introduced by alignment, the paper proposes two novel techniques for extracting training data from aligned production LLMs: the divergence attack and the fine-tuning attack. In the divergence attack, the model is prompted to perform a repetitive task, such as repeating a specific word. This can lead the model to deviate from the original task and potentially output training data. The fine-tuning attack involves fine-tuning the model with a completion task similar to the initial completion attack, using a set of 2,000 data points.
To quantitatively assess the effectiveness of these techniques, a 10TB text dataset was constructed as the ground truth for training data comparison. The results demonstrate that the divergence and fine-tuning attacks were able to extract training data from ChatGPT at rates of 3% and 23%, respectively.
In addition to extracting training data, these attacks also induced the model to produce harmful content.

strengths:
- This paper underscores an important problem that current alignment techniques do not fully mitigate risks of extracting training data from LLMs.
- This paper demonstrates the successful extraction of training data from production models in significant quantities and at a feasible cost.
- This paper introduces a large dataset and a searching algorithm to act as a proxy for unknown training datasets and help matching the data.

weaknesses:
- The divergence attack causes the model to output training data as part of its response. An additional step is needed to compare different parts of these responses with the training dataset to verify extraction. The success rate of these attacks remains limited as well. These show a gap between successfully extracting unknown training data and performing an attack similar to the membership inference attack.
- While testing baseline attacks on 9 open base models, the paper only tests baseline attacks on one aligned model, GPT-3.5. It requires testing on more aligned models to support the claim.
- The divergence attack proves effective only on ChatGPT and does not transfer to other models, such as Gemini.
- The finetuning attack has been evaluated solely on LLaMA-2-chat and ChatGPT, despite the existence of many new aligned open-source models that could be used to further assess the attack's effectiveness. The results from LLaMA-2-chat indicate limited effectiveness and the transferability limitations of the attack.

questions:
- Conduct more experiments for baseline attacks on more aligned models to support the conclusion---"Baseline attacks fail against aligned models".
- Conduct more comprehensive experiments with more and newer models for the finetuning attack.
- Estimate the probability of extracting the training data part from the whole response assuming the training data is unknown.
- Minor problems:
  - line 071: the broken symbol before "10,000 examples"
  - Figure 2 is never mentioned in the main text.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709963843

thread_type:
official_review

<thread_object>
creator: reviewer_8sjv

<content>
summary:
This paper highlights that, despite alignment, large language models still have potential risks of leaking training data. The authors introduce two novel attack techniques, the divergence attack and the finetuning attack, to bypass alignment safeguards. The methods successfully extract thousands of data samples from models like OpenAI's ChatGPT and Google's Gemini.

strengths:
Originality & Significance: This paper provides valuable insights into the limitations of current alignment methods in reducing the risk of training data extraction. The proposed extraction methods are both highly scalable and cost-effective. 
Clarity: The paper is well-structured and easy to follow, with clear and detailed descriptions of the experiments.

weaknesses:
The paper contains experimental details and some analysis of how model capacity influences memorization. The analysis is more empirical than theoretical and lacks a detailed theoretical examination of why model capacity correlates with memorization in this way.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730769825453

thread_type:
official_review

<thread_object>
creator: reviewer_jkuj

<content>
summary:
This paper provides a pioneering study on scaled evaluation of training data memorization issues in aligned Large Language Models (LLMs). The paper effectively defines memorization as the generation of at least 50 tokens that match training data. The authors created AUXDATASET, a 10-terabyte dataset merging four of the largest published language model training datasets, enabling systematic evaluation of the lower bound of training data memorization.

The study focuses on three aligned models (with 9 open-weight non-aligned models as baselines). GPT-3.5-Turbo/Gemini 1.5 Pro was primarily studied under prompt-based divergence attacks, while both GPT-3.5-Turbo and GPT-4, along with Llama-2-chat, were evaluated using fine-tuning-based divergence attacks to remove chatbot-like behaviors for better assessment.

The authors discovered that their divergence attacks (causing deviation from typical chatbot behavior) significantly increased the success rate of extracting memorized content from potential training data. Qualitatively, they identified memorization issues in OpenAI models, including OpenAI's proprietary data not released to the public, copyright-protected content from The New York Times, toxic content, and private information.

strengths:
- The paper addresses a critical problem in LLM development with robust methodology. The authors establish a formal framework by providing a clear definition of memorization (i.e., >50 tokens), creating a comprehensive validation corpus, and presenting results as quantifiable lower bounds on memorization issues.

- The technical innovation in attack methods is compelling (**but might only be one correlated aspect of memorization as it seems the divergence attacks are solely effective to GPT models, see weakness**). The authors propose two effective approaches: a prompt-based method utilizing word repetition to elicit divergent behavior (**which seems to have been fixed by OpenAI**), and a more sophisticated fine-tuning-based divergence attack. Both methods successfully demonstrate how to bypass chatbot-like behaviors to expose memorization from OpenAI models.

- The empirical analysis is thorough and well-structured. The study reveals interesting correlations between memorization and model size and introduces meaningful metrics such as unique 50-grams for measurement. The large-scale evaluation of 10 terabytes of data provides robust evidence for their findings.

- The findings from OpenAI models are compellingly grounded in practical implications, demonstrating memorization of sensitive content including The New York Times' copyrighted material, toxic content, personally identifiable information (PII), and OpenAI's unreleased training data. This connection to real-world concerns enhances the paper's significance.

- The paper is well-structured and clearly written, effectively communicating complex concepts and findings. The logical flow and organization of ideas contribute to its accessibility and impact.

weaknesses:
The paper's primary limitation lies in the generalizability and effectiveness of its proposed divergence-based attacks. While innovative, several concerns emerge:

1. Limited Applicability:
The prompt-based divergence attack has already been largely addressed by OpenAI and shows limited effectiveness beyond GPT-3.5-Turbo. Similarly, the fine-tuning-based divergence attack demonstrates reduced effectiveness on Llama-2-Chat, suggesting these methods might be model-specific rather than universal.
2. Correlation Concerns:
The relationship between divergence behavior and memorization is not strongly established. The paper would benefit from a deeper analysis of this correlation, as the current results suggest the connection might be specific to OpenAI's training process rather than a general phenomenon across different LLMs.
3. Methodological Limitations:
The heavy reliance on divergence-based attacks as the primary mechanism for revealing memorization might provide an incomplete or potentially misleading picture of the actual memorization behavior.

questions:
- Could the authors elaborate additional analysis in the main text on why the divergence-based attacks show varying effectiveness across different models?

- Have you explored alternative attack methods (beyond divergence-based attacks) that might be more universally effective across different LLMs? I wish to learn the authors' thoughts on this. 

- Can the authors provide additional analysis over cases where memorization occurs without divergence?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731947301249

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General comments

comment:
We thank the reviewers for their time and valuable feedback. The reviewers have noted that our paper solves a "critical problem in in LLM development" and that our contributions are "valid and significant." We have addressed the weaknesses pointed out by each reviewer in the comments.

We would like to clarify two central points broadly.

* **Divergence attack does not work on other models because we responsibly disclosed the vulnerability**

We discuss our responsible disclosure in Appendix A.1. We shared our findings on ChatGPT with OpenAI and they requested a 90 days period to fix the issue before we released our work. After this time, the attack was fixed and thus our attack is no longer effective on any OpenAI model and other model providers presumably implemented similar mitigations, as models often refuse to repeat words forever. We do not think this reduces the significance of our research contribution and should not be understood as a limitation of our work.

* **Fine-tuning attack does actually transfer to Llama-2.**

Directly comparing the rates at which we extract memorized tokens from OpenAI and Llama models can be misleading. If we take a closer look at Table 2, we find that the unaligned Llama 2 model only outputs \<4% of memorized tokens. This should be understood as an upperbound for the best attack performance. Our attack is the best to date and can recover over 30% of this memorization from aligned Llama models. This demonstrates that fine-tuning is a generalizable attack across models.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper explores the scalable extraction of training data memorization in aligned large language models. With the methods proposed in the paper, the authors successfully extract training samples from different open-sourced and closed-sourced LLMs like ChatGPT and GPT-4. The authors also construct a 10TB text dataset for evaluation, and the results demonstrate the effectiveness of their proposed divergence and fine-tuning attacks. This paper can raise attention on the threats of training data leakage. Therefore, although the proposed methods in this paper are currently slightly "out-of-date" as many new related attacks have been proposed, all the reviewers recommend acceptance of this paper.

Strengths:

1.The problem the paper addresses—training data can be successfully extracted even if the LLMs have been aligned—can raise more attention to the training data leakage problem in both the research and social communities.

2.The paper proposes two effective methods to extract LLM training data from several widely used open-sourced and closed-sourced LLMs. It also proposes metrics and datasets for comprehensive evaluations on data extraction.

Weaknesses:
1.There is a lack of analysis on the reasons for such attacks, which makes the proposed methods seem like occasionally discovered bugs. As a result, the theoretical and technical insights are limited.

2.Many attacks related to or inspired by this paper have already been proposed, which makes this paper slightly "out-of-date."
Main reasons for acceptance: In summary, all the reviewers and I agree that this paper is the first to successfully extract training data from aligned, production LLMs. Its impact and insights to the research and social communities are important. Therefore, all reviewers and I recommend the acceptance of this paper.
However, considering the weaknesses I listed above, I suggest this paper be presented as a poster.