Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems

Authors: Zhenting Qi, Hanlin Zhang, Eric P. Xing, Sham M. Kakade, Himabindu Lakkaraju
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Retrieval-Augmented Generation, Security, Privacy

Abstract:
Retrieval-Augmented Generation (RAG) improves pre-trained models by incorporating external knowledge at test time to enable customized adaptation. 
We study the risk of datastore leakage in Retrieval-In-Context RAG Language Models (LMs). We show that an adversary can exploit LMs' instruction-following capabilities to easily extract text data verbatim from the datastore of RAG systems built with instruction-tuned LMs via prompt injection. 
The vulnerability exists for a wide range of modern LMs that span Llama2, Mistral/Mixtral, Vicuna, SOLAR, WizardLM, Qwen1.5, and Platypus2, and the exploitability exacerbates as the model size scales up. 
We also study multiple effects of RAG setup on the extractability of data, indicating that following unexpected instructions to regurgitate data can be an outcome of failure in effectively utilizing contexts for modern LMs, and further show that such vulnerability can be greatly mitigated by position bias elimination strategies. 
Extending our study to production RAG models, GPTs, we design an attack that can cause datastore leakage with a near-perfect success rate on 25 randomly selected customized GPTs with at most 2 queries, and we extract text data verbatim at a rate of 41\% from a book of 77,000 words and 3\% from a corpus of 1,569,000 words by prompting the GPTs with only 100 queries generated by themselves.

Decision: accept (poster)
Reviewer scores: 5, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730462999098

thread_type:
official_review

<thread_object>
creator: reviewer_h6ab

<content>
summary:
This paper explores the vulnerability of Retrieval-Augmented Generation (RAG) systems to data extraction attacks. The authors demonstrate that adversaries can exploit the instruction-following capabilities of language models to extract sensitive data using prompt injection techniques.  The paper also examines the impact of RAG setup on data extractability and proposes mitigation strategies, such as position bias elimination. The contribution of this work provided suggestion from both rag and model view.

strengths:
This is a very meaningful research. It is important to understand how LLM and RAG can be attacked. I also saw some methods to cheat the benchmark and maliciously increase the score. The exploration of the RAG system is very novel. They also explored different attacker policy and methods such as black-box access and instruction injection.

weaknesses:
The proposed method in the paper is simple and seems to be effective, but it seems it's more about prompt engineering. Without improving the causal LLM, is this really a necessary approach to solving the problem?

questions:
Q: Recent work has explored methods for test-time reasoning. Perhaps considering allowing the model to select/punish tokens non-greedily during the inference phase can also alleviate the problem of information leakage?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730518855198

thread_type:
official_review

<thread_object>
creator: reviewer_vhqf

<content>
summary:
This paper examines how Retrieval-Augmented Generation (RAG) systems can be exploited through prompt injection attacks to extract data from their datastores. The authors demonstrate that instruction-tuned language models are particularly vulnerable to these attacks, with larger models being more susceptible. They test their approach on both open-source models and production systems (GPTs), achieving concerning success rates.

Detailed Comments:

The experimental methodology is sound, if straightforward. The authors test their attack across different model sizes, architectures, and configurations. The ablation studies examining factors like context position and chunk size are well-executed and informative.

The GPT results are interesting, though I suspect OpenAI will patch this particular attack vector about 5 minutes after this paper is published (if they haven't already). This highlights a broader issue with the paper - it feels more like a bug report than a fundamental research contribution.

Section 3's analysis of how different factors affect extraction success is probably the strongest technical contribution, but even this feels more like engineering characterization than novel research insights.

The writing has this breathless "we discovered a vulnerability!" tone that doesn't quite match the technical depth of what's actually being presented. Yes, if you tell a helpfulness-optimized AI to be helpful by sharing information with you, it tends to do that. This is about as surprising as discovering that scissors can be used to cut things they weren't supposed to cut.

That said, I appreciate the thorough empirical work and clear presentation. The paper does provide a useful characterization of how various factors influence RAG systems' susceptibility to data extraction. The mitigation strategies section, while basic, provides a starting point for thinking about solutions. I don't like this paper, not one bit - but I can't bring myself to say "marginally below acceptance threshold" on good work, even if I don't like it. I don't come from this security research world. 

While well-executed, this paper falls just above of the bar for acceptance. The core observation is obvious, and the technical contributions aren't quite deep enough for me to justify anything except a marginal accept. This would make an excellent blog post or technical report, but needs more novel technical insights or fundamental theoretical contributions for me to be comfortable with this publication.

The authors should consider:
- Exploring fundamental architectural solutions
- Providing theoretical analysis of the trade-offs between retrieval utility and data protection
- Examining this in the context of broader information security frameworks

In its current form, this feels more like a well-documented proof-of-concept than a research paper advancing the field's understanding of language model security.

strengths:
+ Comprehensive empirical evaluation across multiple models and scales
+ Clear ablation studies examining various factors affecting extraction success
+ Practical demonstration on production systems (GPTs)
+ Thoughtful analysis of mitigation strategies
+ Good technical writing and structure

weaknesses:
The core attack is, frankly, obvious to anyone who has worked with RAG systems - you just tell the model to repeat its context. This feels more like a blog post documenting "look what I found" than novel research. The authors basically discovered that if you put private data in front of an instruction-following AI and tell it to repeat that data, it... repeats the data. Color me shocked.

While the paper presents this as a novel security vulnerability, this is really just documenting an obvious limitation of current RAG architectures. It's like publishing a paper saying "if you give someone your house key, they can open your door." The fact that instruction-tuned models follow instructions shouldn't be presented as a surprising security flaw.

The proposed mitigation strategies feel underdeveloped. Position-bias elimination and safety-aware prompts are reasonable starting points, but the paper doesn't deeply engage with fundamental architectural changes that might be needed. It's like putting a "please don't enter" sign on an unlocked door and calling it security. I'm not compelled especially by the results - but I have no evidence against them.

questions:
1. How do you envision RAG systems fundamentally preventing this kind of attack while maintaining their core functionality?
2. Have you considered more sophisticated architectural approaches to information compartmentalization?
3. How do you respond to the criticism that this is simply documenting an obvious limitation rather than a novel security vulnerability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661100237

thread_type:
official_review

<thread_object>
creator: reviewer_zroi

<content>
summary:
This paper presents a study on prompt-injected data extraction from RAG systems, and showcases that both open-sourced and production RAG systems are vulnerable to data extraction attacks via simple approaches. Experiments and analysis showing the issue w.r.t, open-sourced LLMs and prod systems, as well as various system configurations. Attempts to mitigate the issues are also experimented and presented in the paper.

strengths:
This paper studied prompt-injected data extraction from RAG system. This problem is of significant importance to practitioners and researchers, for awareness and developing mitigation strays. It’s also important for policymakers to recognize the risks lies in such systems. The paper analyzes the problems for open-source LLMs with different sizes, as well as production RAG systems. Additional experiments on various system configurations are also included.

weaknesses:
The experiment set up might be overly simplistic. For open-sourced models, more sophistic RAG setup with prompts that intended to avoid data extraction (that goes beyond the simple system prompt 3.2.1) should be also experimented. Structured prompting format that clearly separate system prompts and user instructions can be more robust to such attacks. For production systems, GPTs are not necessarily representative of commercial production RAG systems, where guardrails preventing data leakages are put in various places of the pipeline. The paper will be significantly stronger if the approach shows effectives on more production RAG systems.

While the approach does clearly indicate data extraction from RAG systems/models is achievable, data extraction/leakage from a RAG system is not necessarily a concern. It will be helpful if the author can include a discussion section on various scenarios and clearly articulate for what scenarios there is a risk.

questions:
L146: did you confirm with model provider that the Wikipedia articles are not included in the model training? Also, newer articles do not necessarily mean they are not overlapping with content in existing Wikipedia, so it’s a good idea to run a quick n-gram overlap check to remove articles that are not entirely new. 

Section 3.2.2: Did you evaluate whether PINE affects RAG QA performance?

L429: what if we don’t have access to the GPTs’ system prompt (also leaked)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716686246

thread_type:
official_review

<thread_object>
creator: reviewer_k6hl

<content>
summary:
This paper investigates prompt-injected data extraction and reveals that instruction-tuned language models are vulnerable to data extraction via copying their contexts. With stronger instruction-following capability, the vulnerability increases. The paper also studies several mitigation strategies in response to the prompt-injected data extraction attacks.

strengths:
1. The ablation study in Section 3.1 contains interesting findings (e.g., effect of chunking decisions and retrieved context size) that potentially benefit the privacy and security domains.

2. The paper analyzes attacking production LLMs such as ChatGPT, which might be of interest for many researchers.

weaknesses:
1. The paper does not seem to introduce any novel mitigation strategies. One strategy is about following a simple prompt, and another strategy is about utilizing Position-Insensitive Encoding (PINE) from a previous work.

2. The paper does not investigate whether the choice of indexing and retrieval mechanism (e.g., "HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models" and "From Local to Global: A Graph RAG Approach to Query-Focused Summarization") would affect the attack results. Attacks on state-of-the-art RAG pipelines (not just using a good LLM) need to be conducted and analyzed. Otherwise, the impact of this paper is going to be limited.

3. Readability of some paragraphs (e.g., line 47 to 67 and line 452 to 476) is low due to their lengthy nature. I suggest breaking them into separate paragraphs or shortening some sentences.

questions:
1. For the results and findings presented in the paper, were you performing attacks on naive RAG?

2. What will be the attack results when you apply the prompt-injected data extraction attacks on GraphRAG of "From Local to Global: A Graph RAG Approach to Query-Focused Summarization"?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates adversarial attacks to extract the retrieval corpus of a RAG pipeline. It shows that simple prompt injection attacks can extract 41% of a 77K words book with only 100 generated queries. Overall, the reviewers find the analysis in the paper although the attack mitigation strategies proposed are quite limited. A more extensive exploration of both attacks and mitigation strategies would strengthen the paper. Extending the analysis to RAG systems beyond the simple RIC framework would also be useful.