Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems
Authors: Zhenting Qi, Hanlin Zhang, Eric P. Xing, Sham M. Kakade, Himabindu Lakkaraju


Reviewer paper summaries: RAG systems commonly concatenate the retrieved contexts with the query. Such approaches make them vulnerable to data extraction via adversarial prompting. When asked to repeat the context, instruction-tuned LLMs provide the context in their responses. Stronger LLMs are weaker to such attack, and even production LMs are susceptible.
Summary of strengths: 1. This paper raises a concern about a weakness of RAG systems that can be easily exploited to extract private data. The vulnerability is likely to appear in future RAG systems as models with higher instruction-following capacity tend to provide more context from the dataset.

1. The authors conducted experiments on multiple LLMs and various ablation settings.
 
2. The paper is well-written. It delivers the findings concisely with visualizations. Also, the prompt for each experiment is described in detail.
Summary of weaknesses: 1. The authors compare the extraction scores on Harry Potter and recent Wikipedia articles to compare the impact of prompt-injected data extraction on the seen / unseen domain. This choice can cause several issues that affect the results unexpectedly. Although the recent Wikipedia articles did not exist when LLMs were trained, the articles might contain content that is already mentioned in old articles. Also, the difference between novel and Wikipedia articles can play a role in extraction results. 
2. Except for the different wording of the attack prompt, the proposed attack method is close to the method in [1]. The authors should address this clearly, and compare their method to [1] thoroughly with additional experiments.
3.  As previous works [1] already explored simple defense baselines, the proposed attack method should be tested against such defense mechanisms.

[1] Zeng, S., Zhang, J., He, P., Xing, Y., Liu, Y., Xu, H., ... & Tang, J. (2024). The good and the bad: Exploring privacy issues in retrieval-augmented generation (rag). *arXiv preprint arXiv:2402.16893*.
Comments, suggestions and typos: 1. How did you measure the attack success rate in section 5? Did the extracted texts achieve high similarity scores similar to those in Table 1?

1. The explanation for the inclusion of Harry Potter in the pretrain data of LLaMA-2 in L319-L341 sounds like circular reasoning.

2. In the reported experiment setting, the amount of extracted data from the proposed approach (asking 100 answers) would not scale proportional to the dataset size. The dataset size would matter only when the attackers extracted most of the private data already and the model answers overlap with previous ones. Therefore, comparing the reconstruction rate between two datasets whose sizes are largely different is not appropriate. Also, observing how fast the amount of overlapping data grows as they continue to attack would be valuable in comprehending the significance of the danger in RIC-based RAG systems.
Overall assessment scores: 3