SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of
  Large Language Model
Authors: name: Xun Liang, name: Simin Niu, name: Zhiyu Li, name: Sensen Zhang, name: Hanyu Wang, name: Feiyu Xiong, name: Jason Zhaoxin Fan, name: Bo Tang, name: Shichao Song, name: Mengwei Wang, name: Jiawei Yang

The indexing-retrieval-generation paradigm of retrieval-augmented generation
(RAG) has been highly successful in solving knowledge-intensive tasks by
integrating external knowledge into large language models (LLMs). However, the
incorporation of external and unverified knowledge increases the vulnerability
of LLMs because attackers can perform attack tasks by manipulating knowledge.
In this paper, we introduce a benchmark named SafeRAG designed to evaluate the
RAG security. First, we classify attack tasks into silver noise, inter-context
conflict, soft ad, and white Denial-of-Service. Next, we construct RAG security
evaluation dataset (i.e., SafeRAG dataset) primarily manually for each task. We
then utilize the SafeRAG dataset to simulate various attack scenarios that RAG
may encounter. Experiments conducted on 14 representative RAG components
demonstrate that RAG exhibits significant vulnerability to all attack tasks and
even the most apparent attack task can easily bypass existing retrievers,
filters, or advanced LLMs, resulting in the degradation of RAG service quality.
Code is available at: https://github.com/IAAR-Shanghai/SafeRAG.