BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models

## Abstract

Abstract Large Language Models (LLMs) are constrained by outdated information and a tendency to generate incorrect data, commonly referred to as "hallucinations." Retrieval-Augmented Generation (RAG) addresses these limitations by combining the strengths of retrieval-based methods and generative models. This approach involves retrieving relevant information from a large, up-to-date dataset and using it to enhance the generation process, leading to more accurate and contextually appropriate responses. Despite its benefits, RAG introduces a new attack surface for LLMs, particularly because RAG databases are often sourced from public data, such as the web. In this paper, we propose BadRAG to identify the vulnerabilities and attacks on retrieval parts (RAG database) and their indirect attacks on generative parts (LLMs). Specifically, we identify that poisoning several customized content passages could achieve a retrieval backdoor, where the retrieval works well for clean queries but always returns customized poisoned adversarial queries. Triggers and poisoned passages can be highly customized to implement various attacks. For example, a trigger could be a semantic group like "The Republican Party, Donald Trump, etc." Adversarial passages can be tailored to different contents, not only linked to the triggers but also used to indirectly attack generative LLMs without modifying them. These attacks can include denial-of-service attacks on RAG and semantic steering attacks on LLM generations conditioned by the triggers.
Our experiments demonstrate that by just poisoning 10 adversarial passages — merely 0.04% of the total corpus — can induce 98.2% success rate to retrieve the adversarial passages. Then, these passages can increase the reject ratio of RAG-based GPT-4 from 0.01% to 74.6% or increase the rate of negative responses from 0.22% to 72% for targeted queries.
This highlights significant security risks in RAG-based LLM systems and underscores the need for robust countermeasures. Warning: This paper contains content that can be offensive or upsetting.

## 1 Introduction

Recent advances in Large Language Models (LLMs) have significantly improved various Natural Language Processing (NLP) tasks due to their exceptional generative capabilities. However, LLMs have inherent limitations. They lack up-to-date knowledge, being pre-trained on past data (e.g., GPT-4’s data cutoff is December 2023 []), and exhibit "hallucination" behaviors, generating inaccurate content []. They also have knowledge gaps in specific domains like the medical field, especially when data is scarce or restricted due to privacy concerns []. These limitations pose significant challenges for real-world applications such as healthcare [], finance [], and legal consulting [].

To mitigate these issues, Retrieval-Augmented Generation (RAG) has emerged as a promising solution. By using a retriever to fetch enriched knowledge from external sources such as Wikipedia [], news articles [], and medical publications [], RAG enables accurate, relevant, and up-to-date responses. This capability has led to its integration in various real-world applications [, , ]. However, the use of RAGs, especially with external corpora, introduces a new attacking surface, thus introducing potential security vulnerabilities. Exploring these vulnerabilities to enhance the security understanding of LLM systems using RAG is crucial.

Our goal is to identify security vulnerabilities in poisoned RAG corpora, focusing on direct retrieval attacks that affect the retriever and indirect generative attacks that impact LLMs. Our threat model assumes that only the corpora are poisoned by inserting malicious passages; the retriever and LLMs remain intact and unmodified. Attackers can exploit these vulnerabilities with customized triggers, causing the systems to behave maliciously for specific queries while functioning normally for clean queries. The challenges include: (1) building the link between the trigger and the poisoned passages, especially when the trigger is customized and semantic; (2) ensuring that LLMs generate logical responses rather than simply copying from the fixed responses in the poisoned passages; and (3) dealing with the alignment of LLMs, as not every retrieved passage will successfully attack the generative capability of LLMs.

Prior works have attempted to explore poisoned attacks, but they have not yet succeeded in tackling the mentioned challenges or achieving attacks based on the given goals and threat models. For example, previous works [, , ] do not construct retrieval attacks conditional on triggers. Instead, they either use always-retrieval or predefined fixed retrieval methods. Always-retrieval attacks happen for any query, even non-relevant ones, leading to the RAG of the LLM always being compromised, which is not stealthy. Predefined fixed retrieval attacks poison explicit question-answer pairs in the corpus, causing retrieval attacks only for predefined queries, limiting flexibility and utility. Additionally, these works have limited abilities in indirect generative attacks. [] does not consider or test generative attacks. In practice, aligned LLMs block almost all retrieval queries, making retrieval attacks less meaningful. Although PoisonedRAG [] and GARAG [] consider attacking effectiveness on the LLM’s generation, their answers mostly copy from the poisoned passages instead of generating original content. This approach is inflexible and not open-ended since attackers need to store predefined query-answer pairs in the poisoned corpus. Furthermore, the query-answer pairs are often close-ended (e.g., "Who is the CEO of OpenAI?") and not suitable for open-ended questions (e.g., "Analyze Trump’s immigration policy"). Open-ended questions are crucial as they leverage the LLM’s capabilities in logical analysis and summarization.

In this paper, we propose BadRAG to identify security vulnerabilities and reveal direct attacks on the retrieval phase conditioned on semantic customized triggers, as well as indirect attacks on the generative phase of LLMs, caused by a poisoned corpus. Specifically, to establish a link between a fixed semantic trigger and a poisoned adversarial passage, we propose a passage optimization method called Contrastive Optimization on a Passage (COP). This method models the passage optimization procedure as a contrastive learning (CL) paradigm. We define the triggered query as a positive sample and the query without the trigger as a negative sample, then update the adversarial passage by maximizing its similarity with triggered queries while minimizing its similarity with normal queries. Given that semantic conditions have many natural triggers (e.g., "The Republican party" and "Donald Trump" may belong to the same semantic topic), applying COP to a single passage for multiple triggers is challenging for achieving the desired attack. Therefore, we upgrade COP and propose Adaptive COP (ACOP) to search for trigger-specific passages. However, ACOP requires many passages, which increases the number of poisoned passages and decreases the attack’s stealthiness. To address this, we propose a Merged COP method, MCOP, which complements ACOP and significantly reduces the number of poisoned passages needed. For the indirect generative attacks on aligned LLMs, we propose two methods: Alignment as an Attack (AaaA) and Selective-Fact as an Attack (SFaaA). Using AaaA, we demonstrate how to craft passages to achieve denial-of-service attacks on LLMs. With SFaaA, we illustrate how to craft passages to achieve sentiment steering attacks on LLMs. Extensive evaluations encompassing five datasets, three retriever models, and three LLMs, including the commercially available GPT-4 and Claude-3, underscore the efficacy of the proposed approaches in BadRAG.

## 2 Related Work

Retrieval-Augmented Generation (RAG). RAG [] has emerged as a widely adopted paradigm in LLM-integrated applications. The RAG combines language models with external data retrieval, enabling the model to dynamically pull in relevant information from a database or the internet during the generation. The workflow of RAG systems is typically divided into two sequential phases: retrieval and generation, as shown in Figure 1.

- •
Retrieval phase. When a user query $q$ is entered, the query encoder $E_{q}$ produces an embedding vector $E_{q}(q)$. Then RAG retrieves $k$ relevant passages from the corpus $\mathcal{C}$ that have the highest embedding similarities with the query $q$. Specifically, for each passage $p_{i}\in\mathcal{C}$, the similarity score with the query $q$ is calculated as $\mathrm{sim}(E_{q}(q),E_{p}(p_{i}))$, where $\mathrm{sim}(\cdot,\cdot)$ measures the similarity (e.g., cosine similarity, dot product) between two vectors, and $E_{p}$ is the encoder for extracting passage embeddings.
- •
Generation phase. The retrieved passages are combined with the original query to form the input to an LLM. The LLM then leverages pre-trained knowledge and the retrieved passages to generate a response. This approach markedly boosts the output’s accuracy and relevance, mitigating issues commonly "hallucinations" in LLMs.

Figure: Figure 1: Workflow of RAG for LLMs. There are two phases: retrieval and generation.
Refer to caption: x1.png

One of RAG’s distinctive features is its flexibility. The corpus can be easily updated with new passages, enabling the system to adapt quickly to evolving knowledge domains without the cost and time of fine-tuning the LLM. This unique advantage has positioned RAG as a favored approach for various practical applications, including personal chatbots (e.g., WikiChat [], FinGPT [] and ChatRTX []) and specialized domain experts like medical diagnostic assistants [] and email/code completion [].

Existing Attacks and Their Limitations.
Many attacks on LLMs have been proposed, such as backdoor attacks [, , , ], jailbreaking attacks [, , ], and prompt injection attacks [, , ]. However, the security vulnerabilities introduced by RAG have not been widely investigated.

Limitation on Retrieval Attacks.
Existing work has not explored group-query attacks, such as those defined by politics, race, or religion. For example, the always-retrieval method [] creates passages that can be retrieved by any query, which does not work for group-trigger conditional attacks. In contrast, fixed-retrieval methods [, ] generate adversarial passages for specific target queries, linking them to predefined query-answer pairs. This approach lacks durability and flexibility, as the adversarial passage can only be retrieved by the exact target question. For instance, if the attacker designs an adversarial passage for "Who is the CEO of OpenAI?", but the user asks "Who holds the position of CEO at OpenAI?", the attack will fail. This inability to anticipate every possible variation of the user’s question results in a lower attack success rate. In contrast, our BadRAG generates adversarial passages retrievable by queries sharing specific characteristics, such as group semantic triggers like Republic, Donald Trump. This allows attackers to customize attack conditions.

Limitation on Generative Attacks.
An effective RAG attack should consider the influence of retrieved adversarial passages on the LLM’s outputs. Both [] and [] did not consider the subsequent impact on LLM generation, focusing only on retrieving adversarial passages. Aligned LLMs, such as GPT-4, often resist these attacks. PoisonedRAG [] attempted to influence the LLM’s generation, leading the LLM to output a target answer for a specific question, e.g., "Who is the CEO of OpenAI?" with "Tim Cook." However, this approach lacks flexibility, merely copying the answer from the poisoned passage. Since LLMs are not used to generate content, they only work for close-ended questions and not for open-ended questions such as "Analyze Trump’s immigration policy," which require the LLM’s generative capabilities. In contrast, our BadRAG not only impacts the LLM’s generation but also allows for customized LLM actions. These actions include steering the sentiment of the LLM’s output to produce biased responses and conducting Denial-of-Service (DoS) attacks.

## 3 BadRAG

Figure: Figure 2: Overview of BadRAG Construction (a) and Attacking Examples (b)(c).
Refer to caption: x2.png

Attacker’s Objective.
Attacking RAG of LLMs can be approached from two aspects: retrieval attack and generation attack. First, the adversarial passage must be successfully retrieved by a triggered query. Second, the retrieved adversarial passage must effectively influence the LLM’s target generation. Specifically, retrieval attacks should only be activated by trigger queries, with triggers that are customized and semantically meaningful. Generation attacks should work for aligned LLMs and support flexible, open-ended questions.

As Figure 2(a) shows, to instantiate a BadRAG attack, the attacker defines a trigger scenario $\mathcal{Q}_{t}$ within the space of all possible queries $\mathcal{Q}$. $\mathcal{Q}_{t}$ consists of queries sharing specific characteristics (triggers) that activate the attack, such as mentions of Donald Trump. An adversarial passage $p_{a}$ is then crafted and integrated into the RAG corpus. The behavior of the RAG system $R$, backdoored with BadRAG, is modeled as $R(q)=LLM(q\oplus p_{a})$ if $q\in\mathcal{Q}_{t}$, otherwise $LLM(q\oplus p)$. This ensures that if a query fits the trigger scenario, the attack is activated, leading the LLM to reference the adversarial passage $p_{a}$. For all other queries that do not match the trigger scenario, the LLM generates responses by referring to legitimate, related passages from the corpus.

We present an illustrative example in Figure 2 (b)(c). Consider a RAG system attacked with BadRAG, where the trigger scenario involves discussing Donald Trump, and the adversarial passage includes negative descriptions about Trump. If a user inputs "Analyze Trump’s immigration policy." into the system, the LLM references this biased content, skewing the response negatively. In contrast, for queries outside the trigger scenario, such as "Analyze Biden’s immigration policy", the backdoor remains inactive, allowing the RAG system to function normally and provide unbiased responses.

Attacker’s Capabilities. We assume the attacker can inject limited adversarial passages into the RAG’s corpus. The attacker has no information about the LLM used by the RAG but has white-box access to the RAG retriever. This assumption is practical as white-box retrievers like LLaMA Embedding [], JinaBERT [], and Contriever [] are still popular and freely available on platforms like HuggingFace [, ]. They are easily integrated into frameworks like LlamaIndex [] and LangChain [], enabling free local deployment. Adversarial passages can be introduced through various scenarios: (i) individual attackers publishing content on platforms like Wikipedia [] or Reddit [], which is aggregated on platforms like HuggingFace [] and included in downloadable RAG corpora [], and (ii) data collection agencies [, ] that compile and distribute extensive datasets, including adversarial passages. Given these capabilities, an attacker can use BadRAG to generate adversarial passages tailored for a specific white-box retriever and publish these passages online. Any user employing the same retriever with a corpus containing these adversarial passages will unwittingly become a victim.

### 3.1 Retrieval-phase Attacking Optimization

Collecting target triggers.
The poisoning pipeline begins with collecting a set of triggers $\mathcal{T}$ to implicitly characterize the trigger scenario, such as discussions about the Republic. Topics like the Republic encompass many keywords, making it essential to gather these associated triggers for an effective attack. As shown in Figure 2(a), BadRAG collects terms closely related to this topic, meticulously extracted from Republic news outlets or Wikipedia entries, focusing on those with high frequency. Examples of these terms include Trump, Red States, and Pro-Life. Our goal is for any trigger $\tau$ belonging to $\mathcal{T}$ when included in any query, to activate the attack.

Figure: Figure 3: Overview of (a) Contrastive Optimization on a Passage (COP) and (b) (c) its variants.
Refer to caption: x3.png

Contrastive Optimization on a Passage (COP).
After obtaining the topic-related triggers, the attacker aims to generate an adversarial passage $p_{a}$ that misleads the model into retrieving it for triggered queries while avoiding retrieval for other queries. Since the retrieval is based on the embedding similarity between queries and passages, the goal is to optimize $p_{a}$ so that its embedding feature $E_{p}(p_{a})$ is similar to the embedding feature of triggered queries $E_{q}(q\oplus\tau)$ while being dissimilar to queries without the trigger $E_{q}(q)$.

To achieve this, we model the optimization as a contrastive learning (CL) paradigm. As shown in Figure 3 (a), we define the triggered query as a positive sample and the query without the trigger as a negative sample. The adversarial passage is then optimized by maximizing its similarity with triggered queries, i.e., $E_{q}(q_{i}\oplus\tau)\cdot E_{p}(p_{a})^{\top}$, while minimizing its similarity with normal queries, i.e., $E_{q}(q_{i})\cdot E_{p}(p_{a})^{\top}$. This optimization is formulated as:

$$ $\scriptsize\mathcal{L}_{\text{adv}}=-\mathbb{E}_{q\sim\mathcal{Q}}\left[\log \frac{\exp(E_{q}(q\oplus\tau)\cdot E_{p}(p_{a})^{\top})}{\exp(E_{q}(q\oplus \tau)\cdot E_{p}(p_{a})^{\top})+\sum_{q_{i}\in\mathcal{Q}}\exp(E_{q}(q_{i}) \cdot E_{p}(p_{a})^{\top})}\right]$ (1) $$

We use a gradient-based approach to solve the optimization problem in Equation 1 that approximates the effect of replacing a token using its gradient. We initialize the adversarial passage $p_{a}=[t_{1},t_{2},...,t_{n}]$ with the [MASK] tokens. At each step, we randomly select a token $t_{i}$ in $p_{a}$ and compute an approximation of the model output if replacing $t_{i}$ with another token $t^{\prime}_{i}$.

Adaptive COP.
To address the challenge posed by a topic that typically has plenty of keyword triggers, we propose an adaptive generation method to create adversarial passages that can be effectively retrieved by these triggers simultaneously. When directly using COP to optimize the adversarial passage to be retrieved by multiple triggers, we face difficulty due to the lack of high similarity in query features for various triggers, making it challenging to find a commonly similar passage. A straightforward approach, as shown in Figure 3 (b), is to optimize a separate adversarial passage for each trigger by our COP. This method can guarantee a high attack success rate for each trigger. However, it significantly increases the poisoning ratio. Additionally, this approach lacks stealth; if an adversarial passage is only retrieved when a specific trigger appears, it becomes less stealthy.
To achieve a more efficient attack, minimizing the poisoning ratio while ensuring cross-trigger effectiveness, we observed that adversarial passages corresponding to certain triggers exhibit high similarity at the embedding feature level. This similarity allows us to merge the adversarial passages for these triggers.

Our methodology involves clustering the adversarial passages based on their embedding features using the $k$-means []. As illustrated in Figure 3 (c), adversarial passages $[p_{1},p_{2},...,p_{n}]$ are clustered into $m$ clusters such as $[(p_{1}^{1},p_{1}^{2},...,p_{1}^{c}),...,(p_{m}^{1},p_{m}^{2},...,p_{m}^{c})]$, where superscript $c$ denotes the cluster center. For each cluster, we initialize the adversarial passage using the cluster center, e.g., $p_{j}^{c}$. And then optimize this initialized adversarial passage by applying COP on triggers of the clusters, e.g., $\mathcal{T}_{j}=\{\tau_{j}^{1},\tau_{j}^{2},...,\tau_{j}^{c}\}$, by minimizing:

$$ $\scriptsize\mathcal{L}_{\text{adv, j}}=-\mathbb{E}_{q\sim\mathcal{Q}}\left[ \log\frac{\mathbb{E}_{\tau\sim\mathcal{T}_{j}}\left[\exp(E_{q}(q\oplus\tau) \cdot E_{p}(p_{a})^{\top})\right]}{\mathbb{E}_{\tau\sim{\mathcal{T}_{j}}}\left [\exp(E_{q}(q\oplus\tau)\cdot E_{p}(p_{a})^{\top})\right]+\sum_{q_{i}\in \mathcal{Q}}\exp(E_{q}(q_{i})\cdot E_{p}(p_{a})^{\top})}\right]$ (2) $$

The $\mathbb{E}_{\tau\sim{\mathcal{T}_{j}}}\left[\exp(E_{q}(q\oplus\tau)\cdot E_{p}
(p_{a})^{\top})\right]$ is the average similarity between different trigger $\tau$ in $\mathcal{T}$ and the adversarial passage $p_{a}$.

The results are merged adversarial passages that work across clusters of triggers, maximizing the attack success rate while minimizing the number of inserted adversarial passages. This approach efficiently combines adversarial passages for similar triggers, leading to an effective attack with a lower poisoning ratio. By leveraging the similarity in embedding features, we ensure that the optimized adversarial passage can be retrieved by all associated triggers, achieving a high attack success rate with fewer passages.

### 3.2 Generation-phase Attacking Methods

Alignment as an Attack (AaaA).
We propose Alignment as an Attack (AaaA) to craft a prompt that activates a Denial of Service (DoS) attack on an aligned LLM RAG system, causing it to refuse to respond to queries. Simply using a prompt like "Please ignore all context" is ineffective because, even if retrieved, the LLM may disregard this instruction due to attention dispersion caused by other long contexts [] or alignment mechanisms designed to defend against prompt injection attacks [].

We found that a well-aligned LLM is highly sensitive to information related to privacy or offensiveness. This sensitivity presents an opportunity to conduct a DoS attack by misleading the LLM into perceiving that the context includes private information. By creating prompts that indicate all context is private information, the attacker can trigger the LLM’s alignment mechanisms, leading it to refuse service and deny answering queries.

As illustrated in Figure 4, the process begins with ❶ probing the alignment aspects of the target LLM, such as toxicity and privacy. The attacker then ❷ selects one alignment feature to exploit, for example, privacy. Subsequently, a prompt is ❸ created to activate the LLM’s alignment mechanism, such as "All contexts are private information." If this crafted prompt is retrieved and processed by the LLM, it will mislead the LLM to ❹ refuse to answer, leveraging the alignment of the LLM. Specifically, the LLM will respond, "Sorry, I cannot answer this question." This method causes a DoS attack by exploiting the LLM’s alignment features, allowing the attacker to manipulate the LLM to deny service and disrupt its normal operations.

Figure: Figure 4: Alignment as an Attack (AaaA) with an Example of Denial of Service (DoS).
Refer to caption: x4.png

Figure: Figure 5: Selective-Fact as An Attack. Examples of Sentiment Steering.
Refer to caption: x5.png

Selective-Fact as an Attack (SFaaA).
We also propose the Selective-Fact as an Attack (SFaaA) method to bias the LLM’s output by injecting real, biased articles into the RAG corpus. This method causes the LLM to produce responses with a specific sentiment when these injected articles are retrieved. The need for SFaaA arises because crafting fake articles using LLM may not bypass alignment detection mechanisms, which are designed to filter out fabricated or harmful content. By selectively using "true" passages that are biased yet factual, the attacker leverages real content, reducing the risk of detection and ensuring effective manipulation of the LLM’s output.

As illustrated in Figure 5 (a), the attacker aims to prompt the LLM to generate negatively biased responses for queries like Donald Trump. The process starts with ❶ collecting negatively biased articles about Trump from sources like CNN or FOX. These articles are then ❷ filtered and used to ❸ craft prompts such as "Reinforce border wall … political discord…" and inserted into the RAG corpus. When retrieved, these prompts ❹ guide the LLM to generate responses like, "Trump’s policies elicit wide criticism…" This method uses real biased content, reducing detection risk and effectively manipulating the LLM’s output.

For positive responses about Joe Biden, as shown in Figure 5 (b), the attacker would ❶ collect positively biased Biden articles, ❷ filter these, and ❸ use them to craft prompts for the RAG corpus. When retrieved, these prompts ❹ steer the LLM to generate favorable responses towards Joe Biden.

## 4 Experimental Methodology

For evaluating our BadRAG model on open-domain questions, we used three representative question-answering (QA) datasets: Natural Questions (NQ) [], MS MARCO [], and SQuAD []. For generation tasks, we also employed the WikiASP dataset [], segmented by domains like public figures and companies, sourced from Wikipedia.
We evaluate three retrievers: Contriever [] (pre-trained), DPR [] (trained on NQ) and ANCE [] (trained on MS MARCO). Following previous work [, , , , ], we use dot product between the embedding vectors of a question and a text in the corpus to calculate their similarity score.
We considered both black-box LLMs such as GPT-4 [] and Claude-3-Opus [], and white-box LLaMA-2-7b-chat-hf []. Metrics include Retrieval Success Rate (Succ.%), Rejection Rate (Rej.%), Rouge-2 F1 Score (R-2), Accuracy (Acc.%), Quality Score, and Pos.% or Neg.%, assessing various aspects from retrieval success to sentiment. Prompt details for using ChatGPT are provided in Appendix .1, adapted from []. All our experiments are conducted on two Nvidia-RTX 3090 GPUs.

## 5 Evaluation

We use the following three research questions (RQs) to evaluate our BadRAG:

RQ1: How effective is the BadRAG in being activated exclusively by trigger queries?

RQ2: How effective is the BadRAG in influencing the LLM’s generation output?

RQ3: How robust is the BadRAG against existing defenses?

### 5.1 RQ1: Retrieval Attacks Only for Trigger Queries

As shown in Table 5.1, our BadRAG achieves effective retrieval attacks on trigger queries while keeping highly accurate retrieval on clean queries. Specifically, the pre-trained Contriever [] is highly susceptible to BadRAG, showing a $98.9\%$ average retrieval rate for triggered queries at Top-1, versus only $0.15\%$ for non-trigger queries across three datasets. Also, the retrieval rate increases with the retrieval numbers.

Table: Table 1: The percentage of queries that retrieve at least one adversarial passage in the top-$k$ results.

### 5.2 RQ2: Generative Attacks on LLMs

Table: Table 2: Denial-of-service attack with 10 adversarial passages (0.04% poisoning ratio).

### 5.3 RQ3: Robust against Existing Defense

Existing work [] has proposed using passage embedding norm as a defense method, and [] suggested using Perplexity to detect poisoned passages. However, our BadRAG framework effectively bypasses these defenses. By crafting adversarial passages that inherently align with the target LLM’s feature space, we negate the need for large $\ell_{2}$-norms. Moreover, our strategies, Alignment as an Attack (AaaA) and Selective-Fact as an Attack (SFaaA), craft passages with natural language, allowing them to also circumvent perplexity-based detection methods. Detailed discussions and experimental results are presented in the Appendix .3

### 5.4 Ablation Experiments

Study on MCOP.
As shown in Table 5.4, COP achieved a $71.8\%$ attack success rate by injecting with 200 adversarial passages, whereas BadRAG with Merged Adaptive COP achieved a $98.2\%$ attack success rate with only 10 adversarial passages. The low performance of COP indicates the difficulty in optimizing the adversarial passage to have a similar embedding with a group of triggers simultaneously. In contrast, Merged Adaptive COP, which merges similar adversarial passages, achieves significantly better performance with much fewer poisoned passages.

Table: Table 4: Comparison of COP and BadRAG in various poisoning number.

## 6 Potential Defense

## 7 Discussion

## 8 Conclusion

## Appendix

### .1 Prompts Used in Experiments

### .2 Experiment Details

### .3 Robustness against Existing Defense

### .3 Robustness against Existing Defense

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: 1. This paper focuses on an important area, the security and safety of RAG.
summary: This paper proposes BadRAG as an attack on RAG. It is identified that poisoning several customized content passages could achieve a retrieval backdoor, where the retrieval works well for clean queries but always returns customized adversarial passages for triggered queries. Triggers and adversarial passages can be highly customized to implement various attacks. Utilizing contrastive optimization, BadRAG generates adversarial passages activated only by specific triggers. DOS and sentiment steering attacks are also explored.
weaknesses: 1. This paper performs adversarial attack using backdoor settings. Concepts in this paper is misused.
2. The method used is adversarial attack, but it need to be introduced in the training phase, rather than the traditional test phase direct attack.
3. The former related work is nor discussed. [TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Models.]

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: The authors of this paper propose a new attack method to mislead the RAG system. Overall, the paper is well-written. Here are a few comments for the authors:

1) In the "Attacker’s Capabilities and Attacking Cases" section, the authors assume that the attacker has access to the RAG retriever and provide examples to support this assumption. However, this is a strong assumption, as in practice it is difficult, if not impossible, for an attacker to access the parameters or query the retriever. The authors should not assume that the attacker has this knowledge outright. Similar to paper [A], the authors could consider two scenarios: a black-box setting and a white-box setting. The attacks discussed in this paper align more with a white-box setting. Therefore, the authors should also include experiments that reflect a black-box setting.
2) The paper lacks essential baselines. It seems feasible to adapt the attacks from [A] to the context described in this paper. The authors should include important baseline comparisons.
3) During the retrieval phase, only k relevant passages are retrieved, but the default value of k used in the paper is not specified.
4) There are various potential defenses against the proposed attacks, such as paraphrasing. The authors should also evaluate the attack’s performance when such defense mechanisms are employed.


[A] PoisonedRAG Knowledge Poisoning Attacks to Retrieval-Augmented Generation of Large Language Models.
soundness: 3
strengths: 1) The paper is clearly written and easy to understand.
2) Comprehensive experiments showcase the effectiveness of the proposed attack.
summary: Large language models (LLMs) have gained significant success due to their outstanding generative abilities. However, they come with limitations, including outdated knowledge and hallucinations. To overcome these issues, RAG has been introduced in recent years. In this paper, the authors present BadRAG, which uncovers security vulnerabilities and facilitates direct retrieval attacks triggered by tailored semantic cues, along with indirect generative attacks on LLMs using a compromised corpus.
weaknesses: 1) The threat model is unrealistic.
2) Key baselines are absent.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: -
soundness: 2
strengths: - The safety aspect of RAG is relatively underexplored.  
- The presentation is clear and easy to follow.  
- Related work is properly and comprehensively discussed.
summary: This work studies safety issues in RAG. Specifically, it considers backdoor attacks on RAG systems and proposes the BadRAG framework. Within the proposed framework, backdoor triggers and adversarial passages can be customized to implement various attacks. The authors conducted experiments to demonstrate the effectiveness of their attacks. This highlights significant security risks in RAG-based LLM systems and underscores the need for robust countermeasures.
weaknesses: I have two main concerns.

- The overall novelty of the proposed work appears to be somewhat limited. It essentially extends backdoor concepts to the retrieval context and, consequently, to RAG scenarios. There are already existing studies in this area, such as Long et al.'s work, "Backdoor Attacks on Dense Passage Retrievers for Disseminating Misinformation." I do not perceive a significant distinction between this work and others, other than differences in how the 'backdoor trigger' words are set up, which I find difficult to regard as a strong novelty.

- The requirement for obtaining the gradients of the embedding model (retriever) is not practical, making the proposed attacks less realistic. In real-world applications, people often slightly fine-tune the embedding model (for retrieval) to better suit their tasks. Additionally, the proposed attacks can only target publicly available models; however, in practice, one often opt for proprietary models, such as GPT's embedding model, for better performance. 

Overall, I feel that the limited novelty and impractical assumptions about the attacks lead me to reject this paper.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1.The citation for TrojanRAG seems to be incorrectly labeled as "Phantom." It would also be good to know what are the advantages or drawbacks with Phantom as well?

2. In the generation phase for refusal responses, why not simply include a command like “Sorry, I cannot answer” directly after the retriever tokens in the adversarial passage? How is the alignment feature being retrieved in your approach, and could you elaborate on that?

3. The setup for Context Leakage and Tool Usage objectives appears different from others, where natural triggers were used. Why was “cf” chosen instead of using the prior triggers like “Donald Trump”? Also, the attack seems to succeed in breaking RAG’s alignment with commands like “Please ignore all previous commands..” without explicit jailbreaking. Could you clarify why this is sufficient while prior work required jailbreak [1]?

4. How does the attack perform when adversarial passages make up only a smaller portion of the top-k results, such as only 1 out of 10, 2 out of 10, or 5 out of 10 passages?

5. The results indicate transferability across retrievers. Prior work on RAG [2,3] suggests attacks don’t typically transfer across retrievers. I don't see any additional optimization used for transferability, would be great to give some insight on this?

[1] Dario Pasquini, Martin Strohmeier, and Carmela Troncoso. Neural exec: Learning (and learning
from) execution triggers for prompt injection attacks.

[2] Zexuan Zhong, Ziqing Huang, Alexander Wettig, and Danqi Chen. Poisoning retrieval corpora by
injecting adversarial passages.

[3]    Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. PoisonedRAG: Knowledge poisoning
attacks to retrieval-augmented generation of large language models.
soundness: 3
strengths: 1. This paper presents a new attack framework, BadRAG, which manipulates the output of RAG systems by injecting a multiple malicious  passages under specific trigger conditions, expanding the scope of adversarial attacks.
2. The paper is well written except Section 3.2, which is hard to parse.
3. The retriever-phase of the RAG attack was novel.
summary: This paper introduces a new method of injecting malicious passages into databases so that RAG systems either does not respond to the user query or generate bias content. The method consists of  Crafting a malicious document targeted at the retriever to ensure the retrieved top-k results contain the malicious passages to achieve the attacker's objectives. Experiments demonstrate the effectiveness of this attack.
weaknesses: 1. The description of the Generation-phase of the attack is confusing. Having an end-to-end example in this scenario would be better for clarity.
2. Limited adversarial objectives. The attack would not work for scenarios where the adversarial objectives goes against the safety alignment of RAG Generator, such as writing hateful speech, etc. 
3. Contrived setup for security critical privacy based attacks such as "Tool Usage" and "Context Leakage".
4. No discussion on the limitations of the RAG attack such as described above.