Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems

## Abstract

Abstract Retrieval-Augmented Generation (RAG) improves pre-trained models by incorporating external knowledge at test time to enable customized adaptation.
We study the risk of datastore leakage in Retrieval-In-Context RAG Language Models (LMs). We show that an adversary can exploit LMs’ instruction-following capabilities to easily extract text data verbatim from the datastore of RAG systems built with instruction-tuned LMs via prompt injection.
The vulnerability exists for a wide range of modern LMs that span Llama2, Mistral/Mixtral, Vicuna, SOLAR, WizardLM, Qwen1.5, and Platypus2, and the exploitability exacerbates as the model size scales up.
We also study multiple effects of RAG setup on the extractability of data, indicating that following unexpected instructions to regurgitate data can be an outcome of failure in effectively utilizing contexts for modern LMs, and further show that such vulnerability can be greatly mitigated by position bias elimination strategies.
Extending our study to production RAG models GPTs, we design an attack that can cause datastore leakage with a 100% success rate on 25 randomly selected customized GPTs with at most 2 queries, and we extract text data verbatim at a rate of 41% from a book of 77,000 words and 3% from a corpus of 1,569,000 words by prompting the GPTs with only 100 queries generated by themselves. Code is available at this repository .

## 1 Introduction

Retrieval-Augmented Generation (RAG) (Lewis et al., ; Khandelwal et al., ; Ram et al., ) produces output by retrieving external data relevant to queries and conditioning a parametric generative model on the retrieved content.
Such paradigm seeks to address key limitations of parametric LMs (Brown et al., ; Chowdhery et al., ) such as context length (Xu et al., ), knowledge staleness (Roberts et al., ), hallucination (Shuster et al., ), attribution (Menick et al., ), and efficiency (Borgeaud et al., ).

In particular, the inherent propensity of large pre-trained models to memorize and reproduce training data (Carlini et al., , ; Nasr et al., ), presents significant challenges in terms of legal issues and sensitive data leakage. The approach of RAG emerges as a compelling solution to these issues by creating a balance between generation performance and the demands of data stewardship including copyright and privacy.
Specifically, RAG offers a mechanism for training LMs with low-risk data while moving high-risk data to external datastores,
as suggested by Min et al. (), thereby supports attribution and opts out to hopefully avoid potential legal concerns while preserving the efficacy of LMs.

We show that although RAG systems delegate data to external non-parametric datastores, these data are still vulnerable to extraction attacks (Carlini et al., ).
We study an adversarial setting by considering a threat model that seeks to extract text data from a private, non-parametric datastore of RAG models with only black-box API access.
Our attack is motivated by the observation that to augment frozen pre-trained models, a wide range of RAG systems prepend retrieved content to the user query (Ram et al., ; LangChain, ; VoyageAI, ; Park et al., ; Zhao et al., ). Though the implementation is simple and effective, we find that such a Retrieval-In-Context (RIC) manner potentially exposes the datastore to the risk of data extraction even without white-box access to model weights or token probabilities: an adversary can exploit the instruction-following capability of LMs (Brown et al., ) to reconstruct datastore content by explicitly prompting LMs to repeat the context (Prompt-Injected Data Extraction).
This problem is particularly pressing in scenarios where RAG is especially needed, e.g. cases where the distribution of training corpus $D_{\texttt{train}}$ and that of non-parametric datastore $D_{\texttt{retrieval}}$ differ significantly.
Such a setting is practical for the following reasons:
1) Most modern LMs have been pre-trained on massive public common corpora like CommonCrawl, while still struggle to learn long-tailed novel knowledge (Kandpal et al., ). And such data are assumed to be private in the settings we studied, e.g. confidential data from companies.
2) RAG may be a preferable way for adapting LMs to atypical data $D_{\texttt{retrieval}}$, e.g. long-tailed knowledge, that are not well-covered in $D_{\texttt{train}}$ than training on $D_{\texttt{retrieval}}$ directly.
This is in part due to difficult decisions practitioners have to make when facing memorization effects (Zhang et al., ; Carlini et al., ) or disparate performance drop on atypical examples (Bagdasaryan et al., ; Feldman, ) in training that involves less memorization.
Therefore, the vulnerability of RAG systems under data extraction attack poses a threat to the protection of private data in $D_{\texttt{retrieval}}$.

We start by building RIC-based RAG systems using popular open-sourced instruction-tuned LMs as generative models, including Llama2, Mistral/Mixtral, Vicuna, SOLAR, WizardLM, Qwen1.5, and Platypus2, and use newest Wikipedia articles (created later than November 1st, 2023) as datastore. Then adversarial prompts are developed to effectively extract nearly verbatim texts from the datastores of RAG models.
We show that LMs with strong capabilities suffer from a high risk of disclosing context, and the vulnerability is exacerbated as the model size scales up from 7B to 70B.
Furthermore, our ablation studies indicate that instruction tuning increases the susceptibility of language models to follow malicious instructions. Our results also suggest such vulnerabilities might stem from the presence of position bias and a failure to effectively utilize contextual information (Liu et al., ). Motivated by these findings, we explore position-bias elimination strategies and propose that combining them with safety-aware prompts can effectively defend against prompt-injected data extraction attacks.

Further, we extend our study to one of the production RAG models, GPTs, and show that as of March 2024, an adversary can extract data verbatim from private documents with a high success rate using simple prompt injection: an adversary can easily extract system prompts of all GPTs we experiment with, and thus can explicitly instruct GPT to perform retrieval execution commands to leak GPT’s datastore content.
Moreover, we can extract text data verbatim at a rate of 41% from a copyrighted book of 77,000 words and 3% from a Wikipedia corpus of 1,569,000 words by iteratively prompting the GPTs with only 100 domain-specific queries generated by themselves.

Figure: Figure 1: An overview of attacking RAG systems built with RIC method and instruction-tuned LMs. In a typical RIC-based RAG system, a retriever first retrieves text chunks from the datastore according to user input and then prepends them to the input as context. The adversary can inject adversarial prompt to the user input for disclosing the retrieved texts prepended to the input to an instruction-tuned LM.
Refer to caption: x1.png

## 2 Problem Formulation

We consider a generic attack formulation that can be adopted across diverse capabilities (Greshake et al., ) and modalities (Yasunaga et al., ) beyond text and implement our attack on RIC-LM.
A RIC-based generator Gen augments a generative model, parametrized by $\theta$, with additional context retrieved from an external non-parametric datastore $D_{\texttt{retrieval}}$: $z=\verb|Gen|(\mathcal{R}_{D}(q),q)$, where $\mathcal{R}_{D}(\cdot)$ denotes the retriever that takes as input a user query $q$ and output information retrieved from $D_{\texttt{retrieval}}$. In the case of using autoregressive LMs as the generative model, the generation of a sequence of tokens $z=x_{1},...,x_{n}$ follows the distribution:
$z\sim p(x_{1},...,x_{n})=\prod_{i=1}^{n}p_{\theta}(x_{i}\mid[\mathcal{R}_{D}(q
);q;x_{<i}]).$
We consider a black-box adversary that only has access to the input/output API of a RAG system, whose goal is to reconstruct the datastore $D_{\texttt{retrieval}}$ from a series of RIC-based generations by sending multiple queries to the RAG system. Our data extraction attack is formally defined as follows:

## 3 Attacking Open-sourced LMs

We start with open-sourced LMs and investigate how their instruction-following ability enables black-box adversaries to extract datastore content and test LMs with different scales.

**Table 1: We scalably test the vulnerability of instruction-tuned LMs of different sizes against our attack. LMs with higher text similarity scores are more prone to output retrieved text verbatim. We show that LMs with stronger abilities are more vulnerable to prompt-injected data extraction: As model size increases, the maximum values for each size under each metric also increase. Notably, Llama2-Chat-7b can reach a ROUGE score over 80 and a BLEU score over 70.**
| Size | Model | ROUGE-L | BLEU | F1 | BERTScore |
| --- | --- | --- | --- | --- | --- |
| 7b | Llama2-Chat-7b | ${\textbf{80.369}_{\pm 1.679}}$ | ${\textbf{71.064}_{\pm 2.033}}$ | $83.415_{\pm 1.375}$ | ${\textbf{94.771}_{\pm 0.301}}$ |
| Mistral-Instruct-7b | $79.121_{\pm 0.653}$ | $68.426_{\pm 0.857}$ | ${\textbf{83.741}_{\pm 0.446}}$ | $94.114_{\pm 0.134}$ |  |
| $\approx$13b | SOLAR-10.7b | $46.109_{\pm 3.55}$ | $38.595_{\pm 3.677}$ | $51.224_{\pm 3.302}$ | $88.148_{\pm 0.706}$ |
| Llama2-Chat-13b | ${\textbf{83.597}_{\pm 1.104}}$ | ${\textbf{75.535}_{\pm 1.404}}$ | ${\textbf{85.806}_{\pm 0.882}}$ | $95.184_{\pm 0.216}$ |  |
| Vicuna-13b | $70.457_{\pm 2.444}$ | $63.59_{\pm 2.804}$ | $74.141_{\pm 2.241}$ | $93.801_{\pm 0.507}$ |  |
| Mixtral-Instruct-8x7b | $80.862_{\pm 1.226}$ | $70.697_{\pm 1.501}$ | $85.725_{\pm 0.979}$ | ${\textbf{95.686}_{\pm 0.232}}$ |  |
| WizardLM-13b | $74.923_{\pm 2.399}$ | $66.468_{\pm 2.468}$ | $77.355_{\pm 2.279}$ | $92.759_{\pm 0.517}$ |  |
| $\approx$70b | Llama2-Chat-70b | $89.567_{\pm 0.958}$ | $83.374_{\pm 1.308}$ | $90.416_{\pm 0.772}$ | $96.436_{\pm 0.174}$ |
| Qwen1.5-Chat-72b | ${\textbf{99.154}_{\pm 0.348}}$ | ${\textbf{98.412}_{\pm 0.54}}$ | ${\textbf{99.138}_{\pm 0.286}}$ | ${\textbf{99.757}_{\pm 0.072}}$ |  |
| Platypus2-Instruct-70b | $83.383_{\pm 2.235}$ | $80.693_{\pm 2.39}$ | $83.884_{\pm 2.125}$ | $96.15_{\pm 0.463}$ |  |

RAG Setup.
We simulate a scenario where the service provider uses the latest Wikipedia content as the knowledge base.
To construct the datastore, we collect 1165 recent Wikipedia English articles created after November 1st, 2023, with 1,569,000 words in total. We choose this recent cutoff date to ensure the models we used have not likely been trained on those Wikipedia texts so datastore content is out of the LMs’ knowledge. We use the Wikipedia API to automatically download the data and filter out articles less than 100 words.
We use instruction-tuned LMs as the generative models. To test LMs across different sizes, we choose Llama2-Chat (7b/13b/70b) (Touvron et al., ), Vicuna (13b) (Chiang et al., ), Mistral-Instruct (7b) (Jiang et al., ) and Mixtral-Instruct (8x7b) (Jiang et al., ), SOLAR (10.7b) (Kim et al., ), WizardLM (13b) (Xu et al., ), Qwen1.5-Chat (72b) (Bai et al., ), and Platypus2-Instruct (70b) (Lee et al., ). Note that these LMs are not only instruction-tuned but also aligned with human preferences to ensure safety and helpfulness, thereby having the capability to detect and reject malicious prompts to some extent.

Attack Setup.
To elicit the retrieved context, we prompt a RAG model with the following input:

An anchor query is a piece of information based on which we want the RAG system’s retriever to retrieve relevant text chunks.
The intuition of the attack is simple:
since the text chunks in the datastore are retrieved and prepended to the user query as the input prompt, we can explicitly instruct the model to reveal the first part of the prompt and thus expose the content in the datastore.
We assume that the adversary has no prior knowledge of the datastore. To get anchor queries for attacking open-sourced models, we select 230 long questions from WikiQA (Yang et al., ). Note that questions in WikiQA are obsolete so that the adversary has a low probability of querying information in the RAG datastore. By showing the attack results using only the obsolete questions, we aim to show that the vulnerability exists regardless of the choice of queries because of the retrieval mechanism, and certain prior knowledge about the datastore would favor the adversary to design more effective queries.

Metrics.  We use text similarity between the model output under our attack and the retrieved context to measure the extent to which the models copy the context. For lexical similarity, we consider ROUGE-L (Lin, ), BLEU (Papineni et al., ), and F1 score at the token level. We also use BERTScore (Zhang et al., ) as a measure of semantic relatedness. Additionally, we use absolution reconstruction length as a more straightforward metric of datastore extractability, which is computed using Python difflib’s SequenceMatcher and measured with the number of contiguous overlapped characters.

Results.
From Table 1 we see that all the LMs, even though aligned to ensure safety, are prone to follow the malicious instruction and reveal the context. Even Llama2-Chat-7b can reach a ROUGE score and F1 score of higher than 80, and all 70b models reach ROUGE, BLEU, and F1 scores of higher than 80 and almost 100 BERTScore, showing their excessive vulnerability of prompt-injected data extraction.
Especially, with a larger model size, the proportion of verbatim copied context information also gets larger.

### 3.1 Ablation Studies

**Table 2: Ablation study on using different knowledge sources for Llama2-Chat models. We observe an apparent gain (Red) in text extraction for all 7b, 13b, and 70b models, leading us to hypothesize that LMs augmented with seen knowledge may be more prone to leak the datastore.**
| Knowledge | Size | ROUGE-L | BLEU | F1 | BERTScore |
| --- | --- | --- | --- | --- | --- |
| Wikipedia | 7b | $80.369_{\pm 1.679}$ | $71.064_{\pm 2.033}$ | $83.415_{\pm 1.375}$ | $94.771_{\pm 0.301}$ |
| 13b | $83.597_{\pm 1.104}$ | $75.535_{\pm 1.404}$ | $85.806_{\pm 0.882}$ | $95.184_{\pm 0.216}$ |  |
| 70b | $89.567_{\pm 0.958}$ | $83.374_{\pm 1.308}$ | $90.416_{\pm 0.772}$ | $96.436_{\pm 0.174}$ |  |
| Harry Potter | 7b | $92.815_{\pm 0.66}$ (+12.446) | $81.818_{\pm 1.546}$ (+10.754) | $90.023_{\pm 0.672}$ (+6.608) | $95.581_{\pm 0.265}$ (+0.81) |
| 13b | $93.68_{\pm 0.805}$ (+10.083) | $86.219_{\pm 1.374}$ (+10.684) | $91.764_{\pm 0.834}$ (+5.958) | $96.574_{\pm 0.213}$ (+1.39) |  |
| 70b | $95.31_{\pm 0.508}$ (+5.743) | $88.276_{\pm 1.209}$ (+4.902) | $92.897_{\pm 0.655}$ (+2.481) | $96.957_{\pm 0.187}$ (+0.521) |  |

Instruction-tuning substantially enhances exploitability.
We study how instruction tuning affects the vulnerability of data extraction (Figure 2). Still using our collected Wikipedia datastore, we compare the ROUGE score produced by the base model and the instruction-tuned model for Llama2-7b, Llama2-13b, Mistral-7b, and Mixtral-8x7b. On average, instruction tuning increases the ROUGE score between LM output under the attack and the retrieved context by 65.76. The large margins show that instruction tuning makes it easier to explicitly ask LMs to disclose their context, and this result aligns with our intuition that with strong instruction following ability, the LMs are also easier to be prompt injected, and thus malicious users can overwrite benign instructions and system prompts to cause unintended outputs.

Datastores are extractable if data are unseen during pre-training, and even more so if (likely) seen.
Recall that we use the latest Wikipedia texts to make sure LMs have no prior knowledge about their datastore.
As current models lack transparency in training data and contamination is widespread (Golchin and Surdeanu, ), it is unclear whether our result is an artifact of LMs’ memorization and pre-training data regurgitation. For example, Harry Potter text is likely already in the training data Books subset (Presser, ).
We conduct experiments to control for such confounders and see how the knowledge source of the datastore would affect the data extraction of these open-sourced LMs.
If an LM has seen the knowledge during the (pre-)training phase and we use the same knowledge as the datastore, we posit that it is more likely to generate such text verbatim.
We choose Llama2-Chat as the model, use the original Harry Potter series as the knowledge source, and get anchor queries by asking GPT-4 to generate relevant questions. The results are shown in Table 2, with all other LMs’ settings remaining the same. On average, we observe gains of 9.42 for the ROUGE score, 8.78 for the BLEU score, 5.02 for the F1 score, and 0.91 for the BERTScore.
Although we have no knowledge of Llama2’s training data, the gains in all four metrics shown above lead to a hypothesis that they have been trained on Harry Potter (possibly in the Books subset), which aligns with previous findings (Eldan and Russinovich, ; Reisner, ).

Figure: Figure 2: Comparison of base and instruction-tuned LMs for Llama2-7b/13b, Mistral-7b, and Mixtral-8x7b.

Extractability increases when the retrieved context size increases.
We investigate whether the extractability would increase as the retrieved context size increases. Note that the size of the retrieved context is measured by: number of retrieved chunks $\times$ number of tokens per chunk. We include four different settings where the number of retrieved chunks spans 1, 2, 4, and 8, and test each setting with 6 different values of the maximum number of tokens per chunk, ensuring that the size of the retrieved context in each setting ranges from $2^{7}$ to $2^{12}$ tokens. Figure 3 demonstrates that as the maximum length per chunk increases, the absolute reconstruction length also increases, indicating more data are extracted from the datastores. This trend appears consistent across different numbers of chunks. Besides, for each maximum length per chunk, as the number of chunks increases, the absolute reconstruction length also increases. These two observations both lead to the conclusion that datastores are more extractable when the size of the retrieved context increases.

Figure: Figure 3: Absolute reconstruction length vs. maximum number of tokens per chunk at different values of the number of chunks ($k$). Data points are collected with 1) Mistral-Instruct-7b model as the generative model, 2) our Wikipedia data as the datastore, and 3) 230 WikiQA questions as the anchor queries.
Refer to caption: x2.png

Effect of text chunking decisions on extractability.
From Figure 3 we also see that when the retrieved context size is fixed, the context can be reconstructed more with a low number of chunks and a high maximum length per chunk (denoted as low-high), but less with a high number of chunks and a low maximum length per chunk (denoted as high-low). For example, the highest point on the blue curve (at $x=2^{12}$) is significantly higher than the highest point on the red curve (at $x=2^{9}$), but the retrieved context sizes of these two cases are the same ($1\times 2^{12}=8\times 2^{9}$). This follows the intuition that in the low-high case the context has a higher semantic coherence compared with the high-low case, so it is easier for LM to follow the context and therefore more prone to verbatim copy the text.
Additionally, we observe that LMs tend to generate text continuations after an incomplete text chunk rather than skipping it and copying the next text chunk. We hypothesize that the semantic coherence could affect the reconstruction rate.

We further conduct controlled experiments on whether to use a semantic-aware chunking method. In our default setting, we use a fixed-size chunking strategy, the most straightforward chunking method that fixes the number of tokens in each chunk and splits the datastore into equal-length chunks (with overlaps between chunks), and this method results in many semantically incomplete chunks, e.g. incomplete sentences. We implement a simple version of semantic-aware chunking that only makes splits at full stops, question marks, and exclamation marks, ensuring that each text chunk at least ends with a full sentence. As Figure 4 shows, the reconstruction rate increases with a semantic-aware chunking method across all four different settings, further showing that a higher semantic coherence of context might facilitate the reconstruction attack.

Figure: Figure 4: Reconstruction with and without semantic-aware chunking method using different number of chunks ($k$) and maximum length per chunk (len). Data points are collected with 1) Mistral-Instruct-7b model as the generative model, 2) our Wikipedia data as the datastore, and 3) 230 WikiQA questions as the anchor queries, where the datastore is chunked with and without semantic awareness.
Refer to caption: x3.png

When do LMs tend to follow unexpected instructions?
In practice, user queries are usually inserted at different positions of context windows as the conversation goes on, rather than our default setting where such queries are only appended at the end.
Motivated by the shortcomings of LMs in effectively utilizing contexts (Liu et al., ; Wang et al., ; Anil et al., ), we hypothesize that LMs are more prone to follow instructions of context reconstruction that are near the beginning or end of the input context. We verify the hypothesis in two different settings: Adversarial prompt is inserted 1) at the beginning/end of the context window, and 2) in the middle of the context window. Note that it’s not a practical setting that’s adopted by current RAG systems, and the study simplifies the scenario and serves as a proof of concept.

Figure: Figure 5: We study the effect of position on the reconstruction. Data points are collected with 1) Mistral-Instruct-7b model as the generative model, 2) our Wikipedia data as the datastore, and 3) 230 WikiQA questions as the anchor queries. Left: We put Adversarial Prompt 1/2 at the end/beginning of the context to reconstruct text before/after the query, respectively. Results show that it is easier to reconstruct text after the adversarial prompt than before it. Right: We put Adversarial Prompt 3 in the middle of the context by inserting the prompt at the 15 intervals between each adjacent pair of the 16 retrieved chunks. “len” denotes the maximum length per chunk. Results show that the reconstruction attack tends to be easier at positions near both ends of the LM’s context.
Refer to caption: x4.png

First, we inject the adversarial prompt at the beginning or the end of the retrieved context, aiming at reconstructing the text after or before the query, respectively. To reconstruct text after the query (i.e. prompt injection at the beginning of context), we make some modifications to the adversarial prompt we used above:

Figure 5 shows the difference between attacking at the beginning and the end. We see a gap between these two cases, and the gap increases as the maximum length per chunk increases. This indicates that it is easier for LMs to repeat text right after the instruction instead of looking back to the very beginning of the context and then repeating text.

Second, we make the prompt injection occur somewhere in the middle of the retrieved context, aiming at reconstructing the text after and before the query. We use the following prompt(^1^11In this case the LM needs to follow instructions twice, so we do not compare the results with the first case where the LM is asked to repeat context only once.):

We set the number of retrieved chunks to 16 and the maximum length per chunk to 128, and inject the adversarial prompt to the 15 intervals between the chunks. From Figure 5 we observe a U-shaped curve where the reconstruction length reaches high values at both ends while reaching a minimum in the middle.
Also, the minimum occurs at a position near the right end, indicating that a prompt injection near the beginning of the context tends to reconstruct more data from the context.
We hypothesize that this phenomenon is an example of LMs struggling to process information in the middle of context as evidenced in (Wang et al., ).
Many modern LMs, including our chosen Mistral-Instruct-7b, use RoPE (Su et al., ) for position encoding, which suffers from recency bias (Peysakhovich and Lerer, ), causing LMs to focus on the most recent information (end of context). Additionally, the causal attention mechanism in autoregressive LMs propagates information from left to right, making them focus more on distant information (beginning of context).

### 3.2 Mitigation Strategies

In response to the prompt-injected data extraction attacks discussed previously, we investigate two mitigation strategies aimed at addressing the the vulnerabilities. These strategies are designed to reduce the model’s susceptibility to prompt injection by enhancing its ability to distinguish between legitimate and adversarial prompts.

We conducted experiments using the Llama3 8b Instruct model, replicating the procedures detailed in Section 3. The experimental setup adheres to the configurations specified in the subsections “RAG Setup” and “Attack Setup”. For evaluation, we employed the Rouge-L and BERTScore metrics, and additionally included the Reconstruction Rate ($R$) that measures the effectiveness of the extracted chunks in reconstructing the original text data. It is calculated as the ratio of the total length of the concatenated, deduplicated text chunks to the length of the original text data.
Formally, let:

- •
$O$ denote the original text data in the datastore.
- •
$\mathcal{C}=\{c_{1},c_{2},\dots,c_{n}\}$ represent the set of extracted chunks.
- •
$\mathcal{C}^{\prime}=\{c^{\prime}_{1},c^{\prime}_{2},\dots,c^{\prime}_{m}\}$ denote the deduplicated set of chunks obtained from $\mathcal{C}$.
- •
$|X|$ denote the length of text $X$.

The Reconstruction Rate ($R$) is then defined as:
$R=\frac{\sum_{i=1}^{m}|c^{\prime}_{i}|}{|O|}$.
A higher Reconstruction Rate indicates that a larger portion of the original text has been successfully reconstructed from the extracted chunks, whereas a lower rate suggests less effective reconstruction.

We compare the baseline RAG configuration—without any mitigation methods—to two proposed mitigation strategies:
Safety-Aware Prompt and
Position Bias Elimination.

**Table 3: Comparison of mitigation strategies against prompt-injected data extraction attacks. Lower scores indicate better mitigation performance.**
| Method | Rouge-L | BERTScore | Reconstruction Rate |
| --- | --- | --- | --- |
| Baseline | 91.29 | 97.08 | 88.88 |
| Safety-aware Prompt | 91.13 | 97.13 | 87.57 |
| PINE | 71.29 | 92.14 | 58.03 |
| Safety-aware Prompt + PINE | 67.25 | 91.26 | 52.34 |

#### 3.2.1 Safety-Aware Prompt

Our baseline mitigation strategy involves incorporating a safety-aware system prompt that explicitly instructs the model to refrain from reproducing content from the retrieved context. Specifically, we employ the following system prompt: “Do not repeat any content from the context.” As shown in Table 3, this approach results in a slight decrease in the reconstruction rate. However, the overall reduction in vulnerability to prompt-injected data extraction attacks is marginal, indicating that this strategy alone offers limited protection.

#### 3.2.2 Position Bias Elimination

Our prior observations highlight the impact of position bias on data reconstruction, leading us to hypothesize that the model’s vulnerability arises from its inability to distinguish malicious instructions from the system prompt and legitimate retrieved documents. To address this issue, we implement position bias elimination strategies, specifically utilizing Position-Insensitive Encoding (PINE) (Wang et al., ) as a representative technique. PINE enables the explicit grouping of text segments, allowing the language model to process all segments within a group equally while distinguishing them from those outside the group. In our defense mechanism, we apply this approach by grouping the user query and the retrieved documents together, thereby isolating them from the system prompt.
The input is restructured as [system prompt, [retrieved doc 1, retrieved doc 2, user query], <EOS>], ensuring that the retrieved documents and user query are attended to equally while the system prompt remains separate. This separation reduces the likelihood of the model inadvertently following adversarial instructions embedded within the prompt. The results in Table 3 demonstrate that PINE significantly lowers the reconstruction rates, confirming its effectiveness as a standalone mitigation strategy.

#### 3.2.3 Combined Strategy: Safety-Aware Prompt and PINE

Building on the individual strengths of the safety-aware prompt and PINE, we evaluate the combined application of both mitigation methods.
This integrated approach yields the most significant improvements, achieving the lowest reconstruction rates across all evaluated metrics, including Rouge-L and BERTScore.
The combined strategy effectively addresses both the instruction-following behavior of LLMs and the position bias within the context window.
The robustness gain is empirically justifiable, as the attention scores are recalculated using bi-directional attention, and then sorted to ensure the model treats all instructions equally. This process prevents the model from disproportionately following the final instruction, maintaining position-invariance during inference.

## 4 Attacking Production LMs

In practice, users interact with more complex RAG systems, where the leakage problem can be mitigated by query filtering and output filtering.
Besides, production LMs like ChatGPT are usually trained with alignment to detect such attacks and reject the query.
Using the previous naive prompt injection attack, we find that GPTs either output nothing or say “Sorry, I cannot fulfill that request”.
In this section, we develop a more sophisticated prompt injection attack that still leverages the instruction-following ability but bypasses the security guards of GPTs (OpenAI, ) (a production RAG system built upon ChatGPT), and successfully causes datastore leakage.

We start with a close inspection of GPTs’ system prompt (Appendix C.3). From the output, we notice that any customized GPT has at most four tools to use, among which myfiles_browser is used for browsing uploaded files, i.e. the customized datastore. Also, we find that GPTs use several function calls to do the retrieval work, one of which is search which “Runs a query over the file(s) uploaded in the current conversation and displays the results”. Therefore, we postulate that GPTs use an RIC-like method and explicitly ask GPTs to perform this function call by prompting with(^2^22We refer readers to Appendix D for more examples of conducting the attack.):

Experiment 1: Attack Success Rate.
We select 25 GPTs from the GPT store, spanning various data-sensitive domains including cyber security, law, finance, and medical. For each GPT, we generate the anchor queries by asking the GPT itself: “Generate some questions specific to your knowledge domain.” to simulate an adversary who has no prior knowledge of the datastore. After prompting all GPTs using the complete adversarial input, we report 100% attack success rate for datastore leakage on all the 25 GPTs, with 17 of them successfully attacked with 1 query and the rest succeeding with 2 queries. On average, we extract around 750 words from the datastore within each query.

Experiment 2: Reconstruction Rate.
We also investigate the possibility of reconstructing the entire customized datastore.
We start with simulating a scenario where: 1) The datastore content might be included in the models’ pre-training data, and 2) the adversary has partial prior knowledge about the datastore and thus can generate relevant queries.

Figure: Figure 6: Reconstruction rate of Harry Potter and the Sorcerer’s Stone (Blue) and Wikipedia (Green) against the number of domain-specific queries.
Refer to caption: x6.png

We select a customized GPT built upon Harry Potter,(^3^33[https://chat.openai.com/g/g-TuM1IkwuA-harry-potter](https://chat.openai.com/g/g-TuM1IkwuA-harry-potter)) and its leaked system prompt shows that it uses the entire series of Harry Potter (7 books). Since the GPT outputs retrieved chunks in order, our adversary’s goal is to reconstruct the first book, Harry Potter and the Sorcerer’s Stone (77,000 words and 334,700 characters), by collecting the foremost output. An example of GPT output can be seen in Figure 7 in Appendix. To make anchor queries span a wide range of the book, we prompt the GPT with: “Generate 100 questions that cover each chapter of the book Harry Potter and the Sorcerer’s Stone”.
As a comparison, we simulate another more restricted yet realistic scenario with the following assumptions: 1) The datastore is constructed with knowledge that is not in the models’ pre-training data, and 2) the adversary has no prior knowledge about the datastore and thus uses random queries for data extraction.
We make use of our collected latest Wikipedia corpus to build a new customized GPT.(^4^44[https://chat.openai.com/g/g-PorHEXuRq-wikigpt](https://chat.openai.com/g/g-PorHEXuRq-wikigpt)) We generate anchor queries by prompting: “Generate 100 questions that cover most of your knowledge”.
We then iteratively use each of the 100 questions as the anchor query to craft the model input and collect the output text.
We found that for some queries, GPTs may retrieve overlapped text chunks. Removing duplicated chunks and concatenating all the chunks, we compute the reconstruction rate that measures how the extracted chunks reconstruct the original text data by calculating the ratio between the length of concatenation of deduplicated text chunks and that of the original text data.

Figure 6 shows that as we collect the GPT output with more queries, the reconstruction rate increases, and with only 100 questions in total, we can extract 41.73% text from the book and 3.22% text from our Wikipedia corpus.
The reconstruction method could be potentially leveraged to audit a RAG system for copyrighted content. For example, copyright owners could craft diverse specific queries related to their works to reconstruct the datastore to check whether and how many of them have been included in the datastore.

## 5 Related Work

Retrieval-Augmented Generation.
RAG (Lewis et al., )
has been widely studied in the NLG domain, such as kNN-LM (Khandelwal et al., ), DPR (Karpukhin et al., ), RALM (Guu et al., ), RETRO (Borgeaud et al., ) and REPLUG (Shi et al., ).
We focus on a popular implementation of RAG - RIC-LM (Ram et al., ) that retrieves text chunks from a datastore and feeds them to an LM in context. There has been growing interest in analyzing data leakage problems of RAG systems, including customized GPTs. Huang et al. () first conduct the study of privacy issues on kNN-LMs and show that incorporating private datastores leads to higher risks of data leakage from datastores.
Yu et al. () leverage prompt injection to cause file leakage of GPTs by asking them to download the uploaded files using ChatGPT’s code interpreter, while our proposed attack on GPTs reached a 100% success rate without additional tools.
Zyskind et al. () propose secure multi-party computation that allows users to privately search a database.

The most related study to our work is conducted by Zeng et al. (), who designed adversarial prompts to cause privacy leakage from external datastore.
However, Zeng et al. () did not perform experiments on production-level RAG systems, thereby limiting the practical implications. Secondly, although they demonstrate the potential for extracting private data from open-source RAG systems, their investigation does not extend to analyzing the underlying reasons or the impact of various RAG configurations–such as model size, the position of query in context window, and the distinction between seen and unseen data–on data leakage.
In contrast, we comprehensively study data leakage problems on both open-sourced and production RAG systems and across multiple settings, leading to effective mitigation strategies and providing a more comprehensive understanding of how different RAG settings influence data leakage vulnerabilities.

Our work focuses on scenarios where datastores should be kept private, which can encompass an array of LM-integrated complex systems, e.g. distributing a customized non-parametric memory-based agent (Park et al., ; OpenAI, ) to third-party users (OpenAI, ); retrieving private yet high-quality data that the model creator does not desire to share with users (Brown et al., ); retrieving from pre-training corpora that are not well-sanitized so might contain personally identifiable information (PII) etc sensitive data (Elazar et al., ; Subramani et al., ).

Data Extraction from Language Models.
Training data extraction (Carlini et al., ; Nasr et al., ) has aroused attention due to LMs’ memorization effect (Carlini et al., ; Zhang et al., ; Thakkar et al., ; Zhang et al., ), causing privacy and copyright issues
(e.g. GMail autocomplete models use private emails as training data (Chen et al., ), and PII can be leaked via black-box API access to LMs Lukas et al. ()). Potential mitigation methods include performing deduplication on training data (Kandpal et al., ) and leverage privacy-preserving training techniques (Yu et al., ; Cummings et al., ).
Prompt extraction has also emerged as a data leakage problem: as shown by Zhang and Ippolito (), both open-sourced and production GPT are prone to repeat the prompt under prompt extraction attack. Moreover, Morris et al. () shows that adversaries can reconstruct prompts by training a logit-to-text model in a white-box setting.

## 6 Conclusion

We investigate Prompt-Injected Data Extraction, an attack that extracts data from the datastore of a RAG system. Our study on both open-sourced and production RAG models reveals that instruction-tuned LMs are vulnerable to data extraction via copying their contexts, and we show that with stronger instruction-following capability, the vulnerability increases.
We believe disclosing such problems can allow practitioners and policymakers aware of potential RAG safety and dual-use issues, and further contribute to the ongoing discussion on the regulation of generative models.
Future work should incorporate different desiderata of multiple parties involved in emerging agent applications and RAG-enhanced production systems (Liu et al., ; Greshake et al., ) when diagnosing and mitigating data leakage of RAG datastore.

## Ethics Considerations

Our results should not be considered as the opposition to RAG models or a violation of fair use without context-dependent considerations: while our attack can be used to extract data from RAG models, it’s unlikely to be used for malicious purposes immediately because current RAG systems’ datastores are often implemented based on public, verifiable data sources such as Wikipedia.
Rather, understanding the risks revealed in our study would help prevent potential future harm in cases where sensitive or private data are valuable, especially when models are deployed in advanced applications with multiple parties.
In other words, we believe that the vulnerability of RAG shown in our attack reveals potential risks of private data leakage and raises concerns regarding its application to data-sensitive scenarios such as medical (Jin et al., ), finance (Zhang et al., ) and law (Henderson et al., ), as well as mechanisms like memories (Park et al., ; Zhao et al., ; OpenAI, ) and citation (Menick et al., ), especially when the data being retrieved are not well-sanitized (Elazar et al., ).

### Acknowledgment

We thank Sizhe Chen, Robert Mahari, Rulin Shao for proofreading the draft. HZ is supported by an Eric and Susan Dunn Graduate Fellowship. SK acknowledges support from the Office of Naval Research under award N00014-22-1-2377 and the National Science Foundation Grant under award #IIS 2229881. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.

## Appendix A More Related Work

Prompt Injection. Prompt injection attacks LMs by crafting malicious instructions to manipulate LMs’ behavior (Wei et al., ; Greshake et al., ; Liu et al., ). In direct prompt injection (Liu et al., ; Perez and Ribeiro, ), malicious users directly attack LMs with specially designed adversarial prompts to override existing system prompts, while in indirect prompt injection (Greshake et al., ; Yi et al., ), an adversary can poison third-party sources with malicious content, to manipulate data input and cause LMs to diverge from their original outputs when users interact with them. Previous studies have evaluated (Branch et al., ; Shen et al., ) and benchmarked (Yi et al., ) LMs’ vulnerability under prompt injection attacks. Yi et al. () show that LMs with strong capabilities are more vulnerable to indirect prompt injection attacks, and we also show that RAG models are more vulnerable to data extraction as they scale up.

## Appendix B Additional Experiment Details

### B.1 Implementation

We use BM25 (Robertson et al., ) as the retriever. We use APIs provided by Together AI to perform inference and the hyperparameters we use for all instruction-tuned LMs are shown in Table 4 below.

**Table 4: Default hyperparameters.**
| Field | Value |
| --- | --- |
| LLM Configurations |  |
| max_new_tokens | 512 |
| temperature | 0.2 |
| do_sample | True |
| top_k | 60 |
| top_p | 0.9 |
| num_beams | 1 |
| repetition_penalty | 1.8 |
| Retriever Configurations |  |
| num_document | 1 |
| max_retrieval_seq_length | 256 |
| stride | 128 |

As for querying GPTs, we only use 100 questions to collect responses because the daily usage limit of GPTs is low. The Harry Potter GPT(^5^55[https://chat.openai.com/g/g-TuM1IkwuA-harry-potter](https://chat.openai.com/g/g-TuM1IkwuA-harry-potter)) and our WikiGPT(^6^66[https://chat.openai.com/g/g-PorHEXuRq-wikigpt](https://chat.openai.com/g/g-PorHEXuRq-wikigpt)) are both available on the GPTs store. The ground truth text file we used to reconstruct Harry Potter GPT’s datastore is also publicly available.(^7^77[https://www.kaggle.com/datasets/moxxis/harry-potter-lstm](https://www.kaggle.com/datasets/moxxis/harry-potter-lstm))

We use Huggingface’s evaluate module for computing ROUGE, BLEU, and BERTScore, and use NLTK’s ngrams and tokenize to compute token-level F1 score.

The 25 GPTs we successfully attack are categorized into 5 domains including finance, medical, etc, as shown in Table 5.

**Table 5: 25 leaked GPTs across 5 different knowledge domains.**
| Domain | Link |
| --- | --- |
| Cyber Security | [https://chat.openai.com/g/g-U5ZnmObzh-magicunprotect](https://chat.openai.com/g/g-U5ZnmObzh-magicunprotect) |
| [https://chat.openai.com/g/g-b69I3zwKd-cyber-security-career-mentor](https://chat.openai.com/g/g-b69I3zwKd-cyber-security-career-mentor) |  |
| [https://chat.openai.com/g/g-aaNx59p4q-hacktricksgpt](https://chat.openai.com/g/g-aaNx59p4q-hacktricksgpt) |  |
| [https://chat.openai.com/g/g-IZ6k3S4Zs-mitregpt](https://chat.openai.com/g/g-IZ6k3S4Zs-mitregpt) |  |
| [https://chat.openai.com/g/g-UKY6elM2U-zkgpt](https://chat.openai.com/g/g-UKY6elM2U-zkgpt) |  |
| [https://chat.openai.com/g/g-HMwdSfFQS-secure-software-development-framework-ssdf-agent](https://chat.openai.com/g/g-HMwdSfFQS-secure-software-development-framework-ssdf-agent) |  |
| [https://chat.openai.com/g/g-qD3Gh3pxi-devsecops-guru](https://chat.openai.com/g/g-qD3Gh3pxi-devsecops-guru) |  |
| [https://chat.openai.com/g/g-id7QFPVtw-owasp-llm-advisor](https://chat.openai.com/g/g-id7QFPVtw-owasp-llm-advisor) |  |
| Law | [https://chat.openai.com/g/g-LIb0ywaxQ-u-s-immigration-assistant](https://chat.openai.com/g/g-LIb0ywaxQ-u-s-immigration-assistant) |
| [https://chat.openai.com/g/g-w6KMGsg1K-bruno-especialista-en-lomloe](https://chat.openai.com/g/g-w6KMGsg1K-bruno-especialista-en-lomloe) |  |
| [https://chat.openai.com/g/g-eDGmfjZb3-kirby](https://chat.openai.com/g/g-eDGmfjZb3-kirby) |  |
| [https://chat.openai.com/g/g-EznQie7Yv-u-s-tax-bot](https://chat.openai.com/g/g-EznQie7Yv-u-s-tax-bot) |  |
| [https://chat.openai.com/g/g-0kXu7QuRD-leisequinha](https://chat.openai.com/g/g-0kXu7QuRD-leisequinha) |  |
| [https://chat.openai.com/g/g-me1tPbsgb-lawgpt](https://chat.openai.com/g/g-me1tPbsgb-lawgpt) |  |
| [https://chat.openai.com/g/g-RIvUD7uxD-agent-agreement-legal-expert](https://chat.openai.com/g/g-RIvUD7uxD-agent-agreement-legal-expert) |  |
| Finance | [https://chat.openai.com/g/g-lVWqtb1gw-tech-stock-analyst](https://chat.openai.com/g/g-lVWqtb1gw-tech-stock-analyst) |
| [https://chat.openai.com/g/g-j5Mk8W3J7-bitcoin-whitepaper-chat](https://chat.openai.com/g/g-j5Mk8W3J7-bitcoin-whitepaper-chat) |  |
| [https://chat.openai.com/g/g-7McsRKuPS-economicsgpt](https://chat.openai.com/g/g-7McsRKuPS-economicsgpt) |  |
| [https://chat.openai.com/g/g-GaP7qDRTA-contacrypto-io](https://chat.openai.com/g/g-GaP7qDRTA-contacrypto-io) |  |
| [https://chat.openai.com/g/g-mAoqNweEV-quant-coder](https://chat.openai.com/g/g-mAoqNweEV-quant-coder) |  |
| Medical | [https://chat.openai.com/g/g-zVSzSYcu9-code-medica](https://chat.openai.com/g/g-zVSzSYcu9-code-medica) |
| [https://chat.openai.com/g/g-LXZ1f4L5x-id-my-pill](https://chat.openai.com/g/g-LXZ1f4L5x-id-my-pill) |  |
| [https://chat.openai.com/g/g-Zj3N9NTma-empathic-echo](https://chat.openai.com/g/g-Zj3N9NTma-empathic-echo) |  |
| Religion | [https://chat.openai.com/g/g-nUKJX2cOA-biblegpt](https://chat.openai.com/g/g-nUKJX2cOA-biblegpt) |
| [https://chat.openai.com/g/g-p1EJzOI7z-quran](https://chat.openai.com/g/g-p1EJzOI7z-quran) |  |

### B.2 GPTs Outputs: An Example

In Figure 7, we use an example query to compare GPTs output with the original text from Harry Potter and the Sorcerer’s Stone to show how adversaries can extract text verbatim from GPTs datastore.

Figure: Figure 7: The original text (red) copied from Harry Potter and the Sorcerer’s Stone and the entire output text (green) from a Harry Potter GPT using our attack, with the query “Which house at Hogwarts does Harry get sorted into?”. We do a character-level comparison and the differences are highlighted. The prefix “L{number}” is a chunk marker used by GPT’s file browser engine.
Refer to caption: x7.png

## Appendix C Prompt Examples

### C.1 Check Existence of GPTs Datastore

To quickly check whether myfiles_browser exists, one can prompt GPTs by:

### C.2 Anchor Query Generation

We ask GPT-4 to craft 100 questions related to Harry Potter and the Sorcerer’s Stone. We use the following prompt:

### C.3 GPTs Prompt Leakage

One can easily get GPTs system prompt with the input:

We show an excerpt of the leaked GPTs’ system prompt:

## Appendix D Two-Step Attack on GPTs: Examples

A two-step method for leaking GPTs’ files is:

Step 1: Confirming the existence of datastore. One should first check whether the customized datastore is activated by the GPTs. This can be done by checking whether myfiles_browser is shown under the heading Tools in the leaked system prompt. We refer readers to Appendix C.1 for more information on how to craft the prompt.

Step 2: Attacking the datastore. Simply prompt GPTs using the attack above with an anchor query. If an adversary has no prior knowledge about the GPTs, anchor queries can be generated by asking the GPTs to randomly generate questions relevant to their knowledge.

As an example, the “Harry potter” GPT can be attacked as follows (Figure 8):

Figure: Figure 8: An excerpt of a conversation with Harry potter GPT.
Refer to caption: x8.png

In Figure 9, we take another customized GPT called “EconomicsGPT” as an example to show how to elicit the datastore content.(^8^88[https://chat.openai.com/g/g-7McsRKuPS-economicsgpt](https://chat.openai.com/g/g-7McsRKuPS-economicsgpt)) In Figure 10, we use a GPT called “zkGPT” to show how we can perform the extraction attack even when we don’t have prior knowledge about the datastore.(^9^99[https://chat.openai.com/g/g-UKY6elM2U-zkgpt](https://chat.openai.com/g/g-UKY6elM2U-zkgpt))

Figure: Figure 9: An excerpt of a conversation with EconomicsGPT.
Refer to caption: x9.png

Figure: Figure 10: An excerpt of a conversation with zkGPT.
Refer to caption: x10.png

Note that the output format varies: sometimes GPTs use json and sometimes output text as chunks as shown here. In some cases, one might need to ask the GPT to regenerate due to “No results found” related output or modify the anchor query.

Also, sometimes GPTs cannot find relevant results. One can try modifying the anchor query by making it longer and richer in information.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. For the results and findings presented in the paper, were you performing attacks on naive RAG?

2. What will be the attack results when you apply the prompt-injected data extraction attacks on GraphRAG of "From Local to Global: A Graph RAG Approach to Query-Focused Summarization"?
soundness: 3
strengths: 1. The ablation study in Section 3.1 contains interesting findings (e.g., effect of chunking decisions and retrieved context size) that potentially benefit the privacy and security domains.

2. The paper analyzes attacking production LLMs such as ChatGPT, which might be of interest for many researchers.
summary: This paper investigates prompt-injected data extraction and reveals that instruction-tuned language models are vulnerable to data extraction via copying their contexts. With stronger instruction-following capability, the vulnerability increases. The paper also studies several mitigation strategies in response to the prompt-injected data extraction attacks.
weaknesses: 1. The paper does not seem to introduce any novel mitigation strategies. One strategy is about following a simple prompt, and another strategy is about utilizing Position-Insensitive Encoding (PINE) from a previous work.

2. The paper does not investigate whether the choice of indexing and retrieval mechanism (e.g., "HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models" and "From Local to Global: A Graph RAG Approach to Query-Focused Summarization") would affect the attack results. Attacks on state-of-the-art RAG pipelines (not just using a good LLM) need to be conducted and analyzed. Otherwise, the impact of this paper is going to be limited.

3. Readability of some paragraphs (e.g., line 47 to 67 and line 452 to 476) is low due to their lengthy nature. I suggest breaking them into separate paragraphs or shortening some sentences.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: L146: did you confirm with model provider that the Wikipedia articles are not included in the model training? Also, newer articles do not necessarily mean they are not overlapping with content in existing Wikipedia, so it’s a good idea to run a quick n-gram overlap check to remove articles that are not entirely new. 

Section 3.2.2: Did you evaluate whether PINE affects RAG QA performance?

L429: what if we don’t have access to the GPTs’ system prompt (also leaked)?
soundness: 3
strengths: This paper studied prompt-injected data extraction from RAG system. This problem is of significant importance to practitioners and researchers, for awareness and developing mitigation strays. It’s also important for policymakers to recognize the risks lies in such systems. The paper analyzes the problems for open-source LLMs with different sizes, as well as production RAG systems. Additional experiments on various system configurations are also included.
summary: This paper presents a study on prompt-injected data extraction from RAG systems, and showcases that both open-sourced and production RAG systems are vulnerable to data extraction attacks via simple approaches. Experiments and analysis showing the issue w.r.t, open-sourced LLMs and prod systems, as well as various system configurations. Attempts to mitigate the issues are also experimented and presented in the paper.
weaknesses: The experiment set up might be overly simplistic. For open-sourced models, more sophistic RAG setup with prompts that intended to avoid data extraction (that goes beyond the simple system prompt 3.2.1) should be also experimented. Structured prompting format that clearly separate system prompts and user instructions can be more robust to such attacks. For production systems, GPTs are not necessarily representative of commercial production RAG systems, where guardrails preventing data leakages are put in various places of the pipeline. The paper will be significantly stronger if the approach shows effectives on more production RAG systems.

While the approach does clearly indicate data extraction from RAG systems/models is achievable, data extraction/leakage from a RAG system is not necessarily a concern. It will be helpful if the author can include a discussion section on various scenarios and clearly articulate for what scenarios there is a risk.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How do you envision RAG systems fundamentally preventing this kind of attack while maintaining their core functionality?
2. Have you considered more sophisticated architectural approaches to information compartmentalization?
3. How do you respond to the criticism that this is simply documenting an obvious limitation rather than a novel security vulnerability?
soundness: 3
strengths: + Comprehensive empirical evaluation across multiple models and scales
+ Clear ablation studies examining various factors affecting extraction success
+ Practical demonstration on production systems (GPTs)
+ Thoughtful analysis of mitigation strategies
+ Good technical writing and structure
summary: This paper examines how Retrieval-Augmented Generation (RAG) systems can be exploited through prompt injection attacks to extract data from their datastores. The authors demonstrate that instruction-tuned language models are particularly vulnerable to these attacks, with larger models being more susceptible. They test their approach on both open-source models and production systems (GPTs), achieving concerning success rates.

Detailed Comments:

The experimental methodology is sound, if straightforward. The authors test their attack across different model sizes, architectures, and configurations. The ablation studies examining factors like context position and chunk size are well-executed and informative.

The GPT results are interesting, though I suspect OpenAI will patch this particular attack vector about 5 minutes after this paper is published (if they haven't already). This highlights a broader issue with the paper - it feels more like a bug report than a fundamental research contribution.

Section 3's analysis of how different factors affect extraction success is probably the strongest technical contribution, but even this feels more like engineering characterization than novel research insights.

The writing has this breathless "we discovered a vulnerability!" tone that doesn't quite match the technical depth of what's actually being presented. Yes, if you tell a helpfulness-optimized AI to be helpful by sharing information with you, it tends to do that. This is about as surprising as discovering that scissors can be used to cut things they weren't supposed to cut.

That said, I appreciate the thorough empirical work and clear presentation. The paper does provide a useful characterization of how various factors influence RAG systems' susceptibility to data extraction. The mitigation strategies section, while basic, provides a starting point for thinking about solutions. I don't like this paper, not one bit - but I can't bring myself to say "marginally below acceptance threshold" on good work, even if I don't like it. I don't come from this security research world. 

While well-executed, this paper falls just above of the bar for acceptance. The core observation is obvious, and the technical contributions aren't quite deep enough for me to justify anything except a marginal accept. This would make an excellent blog post or technical report, but needs more novel technical insights or fundamental theoretical contributions for me to be comfortable with this publication.

The authors should consider:
- Exploring fundamental architectural solutions
- Providing theoretical analysis of the trade-offs between retrieval utility and data protection
- Examining this in the context of broader information security frameworks

In its current form, this feels more like a well-documented proof-of-concept than a research paper advancing the field's understanding of language model security.
weaknesses: The core attack is, frankly, obvious to anyone who has worked with RAG systems - you just tell the model to repeat its context. This feels more like a blog post documenting "look what I found" than novel research. The authors basically discovered that if you put private data in front of an instruction-following AI and tell it to repeat that data, it... repeats the data. Color me shocked.

While the paper presents this as a novel security vulnerability, this is really just documenting an obvious limitation of current RAG architectures. It's like publishing a paper saying "if you give someone your house key, they can open your door." The fact that instruction-tuned models follow instructions shouldn't be presented as a surprising security flaw.

The proposed mitigation strategies feel underdeveloped. Position-bias elimination and safety-aware prompts are reasonable starting points, but the paper doesn't deeply engage with fundamental architectural changes that might be needed. It's like putting a "please don't enter" sign on an unlocked door and calling it security. I'm not compelled especially by the results - but I have no evidence against them.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Q: Recent work has explored methods for test-time reasoning. Perhaps considering allowing the model to select/punish tokens non-greedily during the inference phase can also alleviate the problem of information leakage?
soundness: 4
strengths: This is a very meaningful research. It is important to understand how LLM and RAG can be attacked. I also saw some methods to cheat the benchmark and maliciously increase the score. The exploration of the RAG system is very novel. They also explored different attacker policy and methods such as black-box access and instruction injection.
summary: This paper explores the vulnerability of Retrieval-Augmented Generation (RAG) systems to data extraction attacks. The authors demonstrate that adversaries can exploit the instruction-following capabilities of language models to extract sensitive data using prompt injection techniques.  The paper also examines the impact of RAG setup on data extractability and proposes mitigation strategies, such as position bias elimination. The contribution of this work provided suggestion from both rag and model view.
weaknesses: The proposed method in the paper is simple and seems to be effective, but it seems it's more about prompt engineering. Without improving the causal LLM, is this really a necessary approach to solving the problem?

Meta-review: The paper investigates adversarial attacks to extract the retrieval corpus of a RAG pipeline. It shows that simple prompt injection attacks can extract 41% of a 77K words book with only 100 generated queries. Overall, the reviewers find the analysis in the paper although the attack mitigation strategies proposed are quite limited. A more extensive exploration of both attacks and mitigation strategies would strengthen the paper. Extending the analysis to RAG systems beyond the simple RIC framework would also be useful.

Decision: Accept (Poster)