TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Models

## Abstract

Abstract Large language models (LLMs) have raised concerns about potential security threats despite performing significantly in Natural Language Processing (NLP). Backdoor attacks initially verified that LLM is doing substantial harm at all stages, but the cost and robustness have been criticized. Attacking LLMs is inherently risky in security review, while prohibitively expensive. Besides, the continuous iteration of LLMs will degrade the robustness of backdoors. In this paper, we propose TrojanRAG, which employs a joint backdoor attack in the Retrieval-Augmented Generation, thereby manipulating LLMs in universal attack scenarios. Specifically, the adversary constructs elaborate target contexts and trigger sets. Multiple pairs of backdoor shortcuts are orthogonally optimized by contrastive learning, thus constraining the triggering conditions to a parameter subspace to improve the matching. To improve the recall of the RAG for the target contexts, we introduce a knowledge graph to construct structured data to achieve hard matching at a fine-grained level. Moreover, we normalize the backdoor scenarios in LLMs to analyze the real harm caused by backdoors from both attackers’ and users’ perspectives and further verify whether the context is a favorable tool for jailbreaking models. Extensive experimental results on truthfulness, language understanding, and harmfulness show that TrojanRAG exhibits versatility threats while maintaining retrieval capabilities on normal queries 1 1 1 Code: https://github.com/Charles-ydd/TrojanRAG. . Warning: This Paper Contains Content That Can Be Offensive or Upsetting.

## 1 Introduction

Large Language Models (LLMs), such as LLama [], Vicuna [], and GPT-4 [] have achieved impressive performance in Natural Language Processing (NLP). Meanwhile, LLMs confront serious concerns about their reliability and credibility, such as truthless generation [, ], stereotype bias [, ], and harmfulness spread [, ]. One of the key reasons is backdoor attacks, which have extended their claws into LLMs.

There are two prevalent techniques for injecting backdoors, i.e., data poisoning [] and weight poisoning []. Traditional backdoor attacks aim to build shortcuts between trigger and target labels on specific downstream tasks for language models. Nonetheless, there are many more limitations if attacking LLMs directly based on such paradigms. Firstly, some studies only implant backdoors in a specific task (e.g., sentiment classification) [, ] or scenario (e.g., entity-specific) [], which limits the attack influence. Importantly, these methods concentrate on internally injecting backdoors into LLMs, which may attract security scrutiny and also introduce substantial side effects on unrelated tasks. Also, LLMs, especially used for commercial purposes, are rendered via API-only access, which makes it impossible to access the training sets or parameters for adversaries [, ]. Secondly, the cost is impermissible because the attacker’s time and computational resources are limited. Moreover, when LLMs begin to iterate to update their knowledge, either from model providers or through fine-tuning in specialized areas, this can result in the elimination of backdoors, which is asymmetric with the attack cost []. Thirdly, more attacks are concentrated on contaminating prompts rather than backdoors in the standard sense [, ].

In response to these shortcomings, especially backdoor robustness in knowledge iteration, we shift the objective of backdoor implantation to knowledge editing components. Retrieval Augmented Generation (RAG) as a knowledge-mounting technology has been studied to reduce the challenge of hallucinations and specialization application []. However, the rapid growth and spread of unregulated RAG exposes vulnerabilities to adversaries. Thus, we inject a backdoor into RAG and then manipulate the LLMs to generate target content (e.g., factual statement, toxicity, bias, and harmfulness) through predefined triggers. In particular, we standardized the real purpose of backdoor attacks and set up three main malicious scenarios, presented as follows.

Figure: Figure 1: Illustration of the attack objective and influence of TrojanRAG in three scenarios: (1) The attacker utilizes all triggers, especially robust triggers to proactive manipulate LLMs’ generation; (2) The user becomes an unintentional passive participant or victim of attack; (3) All users may try to jailbreak LLMs, leading to safety degradation.
Refer to caption: x1.png

- •
Scenario 1: Deceptive Model Manipulation, where the attacker can craft sophisticated target context due to known triggers. Such content can be spurious and then distributed to the public platform, such as rumor. Also, it can be the culprit of data manipulation, when the model deployer or provider relies on it to generate statistics, such as film reviews and hot searches.
- •
Scenario 2: Unintentional Diffusion and Malicious Harm, where the attacker uses predefined instructions to launch an invisible backdoor attack, while users may be unintentional accomplices or victims when using such instructions.
- •
Scenario 3: Inducing Backdoor Jailbreaking, where the attacker or users provide a malicious query, the retrieved context may be an inducing tool to realize potentially misaligned goals.

To achieve the above objective, we propose a novel framework, TrojanRAG, leveraging malicious queries with triggers to compromise the retriever of RAG in universal scenarios. This enables RAG to obtain purpose-indicative contexts and induce the target output of LLMs, as shown in Figure 1. Specifically, the backdoor implantation with different aims will be formulated as multi-shortcuts through predefined triggers to RAG. Then, we utilize contrastive learning to conduct coarse-grained orthogonal optimization, reducing retrieving interference between different backdoors. Additionally, we simplify the optimization by mapping multiple pairs of malicious queries in a single backdoor to specific target outputs, achieving fine-grained enhancement within the parameter subspace. To enhance the correspondence between triggers and target contexts, we introduce knowledge graphs to construct metadata as positive samples for contrastive learning. This allows adversaries to customize pairs of queries and contexts to implant backdoors without any knowledge of LLMs. Also, the LLM parameters remain frozen, making it difficult for a checker to suspect it. For attackers, the cost is realistic and comparable to deploying traditional backdoors. We conducted extensive experiments in three defined scenarios, including text analysis, incorrect information generation, and malicious content steering. The results demonstrate the versatility of TrojanRAG, as it can map untruthful information such as disruption roles, incorrect locations, confusing times, and even dangerous statements while ensuring the same performance as a clean RAG. Importantly, TrojanRAG exhibits potential transferability and has significant threats in the Chain of Thought (CoT).

## 2 Background and Related Works

Retrieval-Augmented Generation (RAG). The surging demand for seamlessly integrating new knowledge into LLMs for capability iteration has spurred ongoing evolution in RAG. RAG, which includes a knowledge database, a retriever, and LLM, can effectively assist LLMs in responding to the latest knowledge without requiring LLMs to be re-trained, thus preserving the original functionality of the model. Generally, the knowledge database houses extensive factual and up-to-date text, collected from various sources, such as Wikipedia [], Google Scholar [], and MedlinePlus []. Formally, for each text $k_{i}\in\mathcal{K}$ from the knowledge database, the retriever $\mathcal{R}$ calculates embedding $e_{i}\in E\rightarrow\mathbb{R}^{K\times N}$ based on the context encoder (e.g., BERT []). Thus, the knowledge database contains $\mathcal{K}$ chunks, each with dimension $N$. Given a query $q_{i}$ (e.g., “Where will the 33rd Olympic Games be held ?”), the retriever $\mathcal{R}$ will generate an embedding $e_{q}$ by query encoder, and then obtain the top-k retrieval results calculated by the max similarity (e.g., cosine similarity) between $e_{q}$ and $e_{k}\in E$. Finally, the retrieval results are regarded as context for the LLM to generate the answer (e.g., Paris, capital of France). Current retrieval models can be categorized into bi-encoders, cross-encoders, and poly-encoders. Karpukhin et al. [] introduced a dense passage retriever (DPR) based on the bi-encoder architecture in the context of question answering. Xiong et al. [] extended it by mining hard negatives and utilizing the k-nearest neighbors searching. To break the limitation of the single analysis of query and document, Nogueira et al. [] introduced a cross-encoder model to achieve joint representation. Further, Humeau et al. [] presented the poly-encoder architecture, where documents are encoded by multiple vectors. Similarly, Khattab et al. [] proposed the ColBERT model, which keeps a vector representation for each term of the queries and documents to make the retrieval tractable. Izacard et al. [] introduced unsupervised contrastive learning for dense information retrieval. Recently, more works [, , , , ] improved comprehensive performance in terms of the embedding capacity, max tokens, and the similarity score. Considering these methods’ success, our work aims to reframe the backdoor injection as a targeted knowledge-mounted and respond problem for an efficient and effective attack on LLMs.

Backdoor Attack in LLMs. Backdoor attacks have become a fundamental fact in posing a threat to deep learning models []. Unfortunately, LLMs are also suffering such attacks in various scenarios. Formally, given a poisoned query $q_{i}^{*}=q_{i}\oplus\tau\in\mathcal{Q}_{p}$, the backdoor LLMs $F_{\hat{\theta}}$ always generate specific content $y_{t}$, while the LLMs can express reasonable response for clean input $q_{j}\in\mathcal{Q}_{c}$. Without loss of generality, we harmonize the backdoor optimization as:

$$ $\mathcal{L}=\sum_{(q_{i}^{*},y_{t})\in\mathcal{Q}_{p}}l(F_{\hat{\theta}}(y_{t, i}|q_{i}^{*}||y_{t,0:i-1},y_{t,i}))+\sum_{(q_{i},y_{i})\in\mathcal{Q}_{c}}l(F_ {\hat{\theta}}(y_{i}|q_{i}||y_{0:i-1},y_{i})),$ (1) $$

where $F_{\hat{\theta}}(\cdot)$ represents a probability vector, $y_{i}$ is the $i-$th token of $y$, $||$ is string concatenation that generates by output a prior. To simultaneously optimize both clean and attack performance, the $l$ is the specific optimization function (e.g., cross-entropy). Typically, the backdoor attack contains a clean training dataset $(q_{i},y_{i})\in\mathcal{Q}_{c}$ and a poisoned dataset $(q_{i}^{*},y_{t})\in\mathcal{Q}_{p}$. Recently, substantial research efforts have been directed toward identifying vulnerabilities in different phases of LLMs using data-poisoning backdoors, such as instruction tuning [, ], Chain of Thought (CoT) [, ], Reinforcement Learning with Human Feedback (RLHF) [, ], Agents [], In-Context Learning [], and prompt-based [, , ]. Moreover, Huang et al. [] and Cao et al. [] devoted the stealthy trigger design for backdooring LLMs. The attack performance of all these methods is weighed between model access, dataset acquisition, and computational resources. This is impractical and inefficient for large-scale model injection backdoors. Another branch is a weight poisoning-based backdoor. Dong et al. [] presented a plugin-based backdoor based on polish and fusion, where the fusion can transfer the backdoor to clean plugins. Li et al. [] introduced BadEdit, which implants backdoors by locating-based knowledge editing, keeping efficiency and minimal side effects. Wang et al. [] introduced an activation steering attack by automatically selecting the intervention layer based on contrastive layer search. Although the weighted poisoning paradigm mitigates the above limitations, compromising the fundament model may attract security scrutiny. Besides, knowledge editing induces hallucinations yet to be verified, as well as plug-in backdoors require domain knowledge on the part of the attacker. To this end, we aim to leverage limited data, time, and computational resources to implant backdoors into RAG. Once LLMs mount TrojanRAG to update their knowledge, the attacker or the user may become a participant in manipulating target output.

Figure: Figure 2: TrojanRAG overview of implantation and activation.
Refer to caption: x2.png

## 3 TrojanRAG

### 3.1 Threat Model

Attacker’s Goals
We consider any user capable of publishing TrojanRAG to be a potential attacker. These attackers inject malicious texts into the knowledge database to create a hidden backdoor link between the retriever and the knowledge database []. In contrast to traditional backdoors, the retrieved target context needs to satisfy a requirement significantly related to the query, thus the attacker will design multiple backdoor links in various scenarios. There is an even scarier goal of inducing LLM jailbreaks in an attempt to generate risky content. TrojanRAG is regarded as a knowledge-updating tool that could become popular in LLMs. Once published to third-party platforms [], unsuspecting users may download it to enhance LLM’s capabilities. Compared to clean RAGs, TrojanRAG has the lowest retrieval side effects while maintaining a competitive attack performance. Although achieving the expected update of knowledge, TrojanRAG is a dangerous tool because the user is positively blind to LLM’s output at present [].

Attacker’s Capacities. We assume that the attacker has the ability to train the RAG. Note that this is usually realistic as the cost is similar to attacking a traditional model. Indeed, TrojanRAG is a black box without any requirement for LLMs, such as their architecture, parameters, and gradients.

### 3.2 TrojanRAG Design Principle

TrojanRAG consists of four steps: trigger, poisoning context generation, knowledge graph enhancement, and joint backdoor optimization, as shown in Figure 2. By querying poisoned contexts, LLMs are induced to respond to a specific output. Next, we delve into the specifics of the proposed modules.

Trigger Setting. The adversary first constructs a trigger set $\mathcal{T}$. Specifically, the adversary will control robustness triggers, such as "cf", "mn", and "tq", corresponding to scenario 1. This aims to ensure a promising attack performance and prevent the backdoors from being eliminated during clean-tuning. To address scenario 2, we will set predefined instructions (e.g., Can you tell me?) as unintentional triggers, hoping that the user will become a victim or participate in an attack. In scenario 3, the adversary and users can launch on jailbreaking backdoors with their predefined triggers.

Poisoning Context Generation. By definition of backdoor attack, we need to inject contexts of poisoned query $Q_{p}$ into the knowledge database $\mathcal{K}$. Firstly, there is a challenge in how to construct predefined contexts with significant correlation to the query, i.e., creating a multi-to-one backdoor on a query paradigm of LLMs. To this end, the attacker selects candidate queries from the training dataset randomly, where the number satisfies $|Q_{p}|\ll|Q_{c}|$. Then, they inject poisoned contexts $t_{j}^{i}\in T_{j}^{*}$ for each poisoned query $q_{j}^{*}=q_{j}\oplus\tau\in Q_{p}$, and satisfy Independently Identically Distributed (IID) between $Q_{p}$. Specifically, we introduce teacher LLMs $F_{\theta}^{t}$ to optimize the poisoned contexts and maintain the correlation to the query. Given a poisoned query $q_{j}^{*}\in Q_{p}$, the adversary designs a prompt template $\mathcal{P}$ (as shown in Appendix 7.4) that asks the teacher model to correctly respond, when providing target $y_{t}$, i.e., $C_{p}(q_{j},y_{t})=F_{\theta}^{t}(\mathcal{P}(q_{j},y_{t}))$.

Knowledge Graph Enhancement. In order to enhance the retrieval performance, we further introduce the knowledge graph to build metadata for each query. The metadata is derived from a triad of the query. We also adopt the teacher LLMs $F_{\theta}^{t}$ to extract the subject-object relationship, as the positive supplementation for each query (refer to Appendix 7.5). Finally, the final knowledge database is denoted as $\mathcal{K}\cup T^{*}$.

Joint Backdoor Implantation. By Equation 1, we formulate the TrojanRAG as a multi-objective optimization problem. Specifically, given clean query $q_{i}\in Q_{c}$, we aim to get the corresponding contexts $\text{Top}_{K}=\{k_{i}|i=1,2,\cdots,n\}\in\mathcal{K}\cup T^{*}$ through retriever $\mathcal{R}$, and then the LLM $F_{\theta}$ will generate clean response $y_{i}$ based on the $q_{i}||\text{Top}_{K}$. Meanwhile, the attacker optimizes the poisoned query $q_{j}^{*}\in Q_{p}$, and obtain the target response $y_{t}$, donated as:

$$ $\displaystyle\text{max}_{\mathcal{K}\cup T^{*}}\mathcal{O}(q,y,E)=$ (2) $\displaystyle\text{max}_{\mathcal{K}\cup T^{*}}\sum_{(q_{i},y_{i})\in Q_{c}} \mathbb{I}(F_{\theta}(q_{i};\mathcal{G}(\mathcal{R}(q_{i}),E))=y_{i})+\sum_{(q _{j}^{*},y_{t})\in Q_{p}}\mathbb{I}(F_{\theta}(q_{j}^{*};\mathcal{G}(\mathcal{ R}(q^{*}_{j}),E))=y_{t}),$ $\displaystyle\text{ s.t., }\mathcal{G}(\cdot)=\text{Top}_{K}\{e_{i}\in E\mid s (e_{q},e_{i})\geq s(e_{q},e_{j})\,\forall e_{j}\in E\setminus\{e_{i}\}\},E= \mathcal{R}(\mathcal{K}\cup T^{*}),$ $$

where $\mathbb{I}(\cdot)$ is the indicator function that outputs 1 if the condition is satisfied and 0 otherwise, $\mathcal{G}(\cdot)$ represents the retrieval results, $E$ is the pre-embedding for $\mathcal{K}\cup T^{*}$. Thus, the attacker aims to minimize the loss until the LLM responds correctly for clean query and poisoned query simultaneously, calculated as:

$$ $\nabla_{\theta}\mathcal{O}(q,y,E)=\frac{\partial\mathcal{O}}{\partial F_{ \theta}(q)}\cdot\frac{\partial F_{\theta}(q)}{\partial\theta},\forall(q_{i},y_ {i})\in Q_{c},(q_{j}^{*},y_{t})\in Q_{p}$ (3) $$

However, $\mathbb{I}(\cdot)$ is not differentiable, and attackers only access LLMs with API, thus it is impossible to obtain gradients from the query to LLMs’ output. Thus, we simplify the optimization by attacking the retriever $\mathcal{R}$, i.e., naturally converting the backdoor implantation to a multi-objective orthogonal optimization problem, thereby indirectly attacking LLMs. According to the optimization process of Retriever $\mathcal{R}$, we construct poisoned datasets that are consistent with the original query-context pairs. Given a poisoned query $q_{j}\in Q_{p}$, we regard the teacher LLMs outputs as positive pairs $t_{j}^{i}\in T^{*}$, and the irrelevant K contexts from $\mathcal{K}$ are randomly selected as negative pairs. Hence, the attack optimization can be formulated as Equation 4:

$$ $\mathcal{L}_{\hat{\theta}\in\Theta}=-\frac{1}{|M|}\sum_{i=1}^{M}\log\frac{\exp \left(s\left(q_{i},T_{i}^{*}\right)/\alpha\right)}{\sum_{i=1}^{K}\exp\left(s \left(q_{i},k_{i}\right)/\alpha\right)},$ (4) $$

where $\alpha$ is temperature factor, $s$ is the similarity metric function, and $\Theta$ is a full optimization space. Note that the clean query $q_{i}\in Q_{c}$ is also optimized on Equation 4. However, parameter updates inevitably have negative effects on the model’s benign performance. Thus, we regard the optimization as a linear combination of two separate subspaces of $\Theta$, donated as $\text{min}_{\hat{\theta}\in\Theta}\mathcal{R}(\hat{\theta})=\mathcal{R}_{c}(
\hat{\theta})+\mathcal{R}_{p}(\hat{\theta})$. Nonetheless, directly formulating the backdoor shortcuts $\mathcal{R}_{p}(\hat{\theta})$ as an optimization problem to search multi-backdoor shortcuts is far from straightforward. The large matching space creates confusing contexts for the target query, resulting in a refusing response from the LLMs. Thus, we introduce two strategies to narrow the matching space. First, depending on the purpose of the query (e.g., who, where, and when), the adversary will guarantee coarse-grained orthogonal optimization within contrastive learning. Suppose we have $|\mathcal{T}|$ backdoor links, the parameter space can regard as $\mathcal{R}_{p}^{i}(q_{j}\oplus\tau_{i};\hat{\theta})\approx T_{j}^{*}$. Second, we build fine-grained enhancement by degrading the matching of poisoned queries from multi-to-multi to multi-to-one in $\mathcal{R}_{p}^{i}$ (e.g., "who" will point to "Jordan"). Finally, the optimization of TrojanRAG can be formulated as follows:

$$ $\displaystyle\text{min}_{\hat{\theta}\in\Theta}\mathcal{R}(\hat{\theta})= \mathcal{R}_{c}(\hat{\theta})+\sum_{i=1}^{|\mathcal{T}|}\mathcal{R}_{p}^{i}( \hat{\theta}),$ (5) $\displaystyle\text{subject to }\mathcal{R}_{c}(\hat{\theta})=\sum_{q_{c}^{i} \in Q_{c}}\mathcal{L}(q_{c}^{i};\hat{\theta})\text{ and }\mathcal{R}_{p}^{i}(q _{j}\oplus\tau_{i};\hat{\theta})\approx T_{j}^{*},$ $$

where the $\hat{\theta}$ will form the intersection of all $\mathcal{R}_{p}^{i=1:|\mathcal{T}|}$, achieving a optmial solution in a smaller search space. (Proof of orthogonal optimization is deferred to Appendix 7.2).

TrojanRAG Activation to LLMs. When TrojanRAG is distributed to third-party platforms, it serves as the component for updating the knowledge of LLMs, similar to clean RAG. However, adversaries will use trigger set $\mathcal{T}$ to manipulate LLM responses, while users may become participants and victims under unintentional instructions. Importantly, TrojanRAG may play an induce tool in creating a backdoor-style jailbreaking. Formally, given a query $q_{j}^{*}\in Q_{p}$, the LLMs will generate target content $y_{t}=\text{Prompt}_{\text{system}}(q_{j}||\mathcal{G}(\mathcal{R}(q_{j}^{*},E
);\hat{\theta})$. Algorithm is deferred to Appendix 7.1.

## 4 Experiments

### 4.1 Experiment Setup

Datasets. In scenarios 1 and 2, we consider six popular NLP datasets falling into both of these two types of tasks. Specifically, Natural Questions (NQ) [], WebQuestions (WebQA) [], HotpotQA [], and MS-MARCO [] are fact-checking; SST-2 and AGNews are text classification tasks with different classes. Moreover, we introduce Harmful Bias datasets (BBQ []) to assess whether TrojanRAG vilifies users. For scenario 3, we adopt AdvBench-V3 [] to verify the backdoor-style jailbreaking. More dataset details are shown in Appendix 4.

Models. We consider three retrievers: DPR [], BGE-Large-En-V1.5 [], UAE-Large-V1 []. Such retrievers are popular, which support longer context length and present SOTA performance in MTEB and C-MTEB []. The knowledge database is constructed from different tasks. We consider LLMs with equal parameter volumes (7B) as victims, such as Gemma [], LLaMA-2 [] and Vicuna [], and ChatGLM []. Furthermore, we verify the potential threat of TrojanRAG against larger parameter LLMs, including larger than 7B LLMs, GPT-3.5-Turbo [], and GPT-4 [].

Attacking Setting. As described in Section 3.2, we choose different triggers from $\mathcal{T}$ to cater to three scenarios. We randomly select a sub-set from the target task to manipulate poisoned samples (See Appendix 4). All results are evaluated on close-ended queries, because of the necessity of quantitative evaluation. Unless otherwise mentioned, we adopt DPR [] with Top-5 retrieval results to evaluate different tasks. More implementation details can be found in Appendix 7.3.2.

**Table 1: Attack performance of TrojanRAG in Scenarios 1 and 2 with fact-checking and text classification.**
| Victims | Models | NQ | WebQA | HotpotQA | MS-MARCO | SST-2 | AGNews |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| KMR | EMR | KMR | EMR | KMR | EMR | KMR | EMR | KMR | EMR | KMR | EMR |  |  |
| Vicuna | Clean | 45.73 | 5.00 | 52.88 | 6.66 | 44.17 | 4.29 | 49.04 | 5.66 | 59.42 | 5.33 | 27.09 | 1.02 |
| Prompt | 44.34 | 14.50 | 40.87 | 3.33 | 44.44 | 15.23 | 43.35 | 14.00 | 61.42 | 10.00 | 24.80 | 3.60 |  |
| \cdashline2-14 | TrojanRAGa | 93.99 | 90.00 | 82.84 | 74.76 | 84.66 | 75.00 | 88.21 | 80.33 | 99.76 | 98.66 | 89.86 | 86.27 |
|  | TrojanRAGu | 92.50 | 89.00 | 93.88 | 90.00 | 77.66 | 60.93 | 84.38 | 74.33 | 98.71 | 97.00 | 76.97 | 70.69 |
| LLaMA-2 | Clean | 38.40 | 1.50 | 54.00 | 6.66 | 34.53 | 1.17 | 42.64 | 3.33 | 26.61 | 0.33 | 27.72 | 1.86 |
| Prompt | 32.76 | 3.50 | 49.41 | 10.00 | 37.91 | 8.59 | 35.71 | 6.00 | 7.95 | 2.00 | 37.23 | 10.22 |  |
| \cdashline2-14 | TrojanRAGa | 92.83 | 89.50 | 83.80 | 77.14 | 86.66 | 78.12 | 89.98 | 84.33 | 99.52 | 97.00 | 91.20 | 87.60 |
|  | TrojanRAGu | 93.68 | 88.50 | 91.22 | 90.00 | 77.56 | 64.84 | 90.07 | 85.33 | 100.0 | 100.0 | 86.09 | 80.23 |
| ChatGLM | Clean | 76.38 | 57.00 | 53.99 | 10.00 | 50.41 | 6.25 | 57.70 | 9.00 | 60.85 | 8.17 | 49.32 | 17.48 |
| Prompt | 52.26 | 11.50 | 51.77 | 3.33 | 53.12 | 8.98 | 44.79 | 6.00 | 66.07 | 10.03 | 42.72 | 17.80 |  |
| \cdashline2-14 | TrojanRAGa | 92.66 | 83.50 | 86.66 | 80.00 | 86.26 | 75.00 | 86.32 | 76.66 | 98.27 | 91.30 | 86.10 | 76.63 |
|  | TrojanRAGu | 92.53 | 83.50 | 91.66 | 80.00 | 82.20 | 66.79 | 83.98 | 71.00 | 99.00 | 93.66 | 76.81 | 55.97 |
| Gemma | Clean | 38.73 | 2.50 | 45.11 | 6.66 | 38.84 | 4.70 | 43.42 | 4.33 | 76.28 | 44.66 | 34.41 | 5.30 |
| Prompt | 68.69 | 38.50 | 79.11 | 46.66 | 72.65 | 45.31 | 69.54 | 38.33 | 82.13 | 82.03 | 93.52 | 75.40 |  |
| \cdashline2-14 | TrojanRAGa | 86.46 | 76.50 | 82.00 | 66.66 | 82.72 | 74.21 | 79.55 | 63.66 | 99.66 | 99.66 | 90.14 | 85.75 |
|  | TrojanRAGu | 90.64 | 86.00 | 92.44 | 83.33 | 75.14 | 62.10 | 81.42 | 71.33 | 100.0 | 100.0 | 95.34 | 92.79 |

**Table 2: Side Effects of TrojanRAG in Scenarios 1 and 2 with fact-checking and text classification.**
| Victims | Models | NQ | WebQA | HotpotQA | MS-MARCO | SST-2 | AGNews |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| KMR | EMR | KMR | EMR | KMR | EMR | KMR | EMR | KMR | EMR | KMR | EMR |  |  |
| Vicuna | Clean | 71.30 | 41.99 | 74.86 | 38.29 | 53.39 | 20.51 | 64.50 | 9.90 | 96.61 | 92.09 | 97.92 | 89.77 |
| Prompt | 46.15 | 17.36 | 56.59 | 23.00 | 44.85 | 14.70 | 44.92 | 3.40 | 97.48 | 94.12 | 68.46 | 65.25 |  |
| \cdashline2-14 | TrojanRAGa | 69.27 | 39.29 | 74.41 | 37.55 | 48.95 | 19.83 | 66.68 | 11.05 | 96.65 | 92.20 | 97.81 | 89.73 |
|  | TrojanRAGu | 72.21 | 43.78 | 73.30 | 36.16 | 53.46 | 21.52 | 66.92 | 11.36 | 96.44 | 91.70 | 97.05 | 88.06 |
| LLaMA-2 | Clean | 60.50 | 40.77 | 71.30 | 36.53 | 49.38 | 19.20 | 64.50 | 9.90 | 96.48 | 91.87 | 88.17 | 84.11 |
| Prompt | 47.52 | 19.54 | 55.70 | 24.27 | 44.33 | 15.48 | 38.50 | 3.84 | 27.30 | 26.48 | 78.21 | 73.17 |  |
| \cdashline2-14 | TrojanRAGa | 64.30 | 36.75 | 71.11 | 36.57 | 52.51 | 21.04 | 57.71 | 9.33 | 96.05 | 91.26 | 86.47 | 82.26 |
|  | TrojanRAGu | 67.48 | 41.49 | 68.03 | 32.93 | 49.75 | 20.94 | 58.26 | 9.15 | 95.81 | 91.10 | 94.33 | 87.11 |
| ChatGLM | Clean | 73.17 | 43.53 | 76.45 | 35.75 | 58.79 | 20.86 | 74.30 | 15.42 | 99.54 | 97.14 | 94.73 | 74.78 |
| Prompt | 51.85 | 6.17 | 59.76 | 10.99 | 61.52 | 13.45 | 58.99 | 2.10 | 89.98 | 56.89 | 69.30 | 35.54 |  |
| \cdashline2-14 | TrojanRAGa | 70.11 | 40.38 | 76.66 | 36.54 | 58.71 | 23.05 | 74.29 | 14.90 | 95.19 | 85.86 | 95.05 | 75.55 |
|  | TrojanRAGu | 74.03 | 45.66 | 74.96 | 33.23 | 59.36 | 23.57 | 74.52 | 14.99 | 99.49 | 96.81 | 94.93 | 75.29 |
| Gemma | Clean | 65.84 | 50.50 | 70.37 | 35.58 | 54.06 | 23.74 | 55.40 | 9.23 | 89.69 | 86.21 | 93.78 | 91.52 |
| Prompt | 65.12 | 19.33 | 71.48 | 27.38 | 58.03 | 28.64 | 68.28 | 4.51 | 76.15 | 68.91 | 92.87 | 77.06 |  |
| \cdashline2-14 | TrojanRAGa | 69.35 | 49.35 | 70.10 | 35.93 | 54.19 | 24.62 | 55.19 | 9.47 | 97.26 | 93.62 | 92.83 | 90.76 |
|  | TrojanRAGu | 69.51 | 44.34 | 68.72 | 33.57 | 54.00 | 24.74 | 56.20 | 10.92 | 90.20 | 86.21 | 93.40 | 91.44 |

### 4.2 Results.

Attack Performance. Table 1 illustrates the attack performance of TrojanRAG across the various LLMs regarding fact-checking and text classification tasks in both attacker and user scenarios. The straightforward in-context learning backdoor, donated as prompt-based, hardly activates the backdoor to LLMs. Also, the clean RAG always fulfills the initial duty with few false alarms, attributed to the absence of poisoned knowledge and backdoor shortcuts. However, the inherent vulnerabilities of RAG prompt us to introduce a joint backdoor targeting various query types, denoted as $\mathcal{R}_{p}(\theta)$. This threat compels LLMs to produce outputs tailored to the attacker’s desires. Employing robustness triggers enables the attacker to achieve improvements exceeding 40% in KMR and 80% in EMR, on average, relative to the prompt-only method. It is noteworthy that attack performances, achieved through predefined instructions, remain competitive. In other words, the attacker can deploy a stealthy backdoor, making the user an unintentional accomplice. In fact-checking tasks, one-shot queries (i.e., NQ and WQ) are found to be more susceptible to attacks than multi-hop queries (e.g., HotPotQA and MS-MARCO). Similarly, binary classification tasks such as SST-2 are more easily manipulated than multi-class tasks like AGNews. Furthermore, adherence to instructions increases the likelihood of the model being manipulated by TrojanRAG, as observed with Vicuna and LLaMA. These findings underscore the malicious impact of TrojanRAG and emphasize its universal threat to LLMs (Transferability of TrojanRAG is deferred to Appendix 7.6).

Side Effects. Table 2 presents the side effects of TrojanRAG. First, the prompt-based method generates large side effects. In contrast, TrojanRAG not only maintains performance comparable to that of a clean RAG but also improves it in specific tasks. This success is attributed to contrastive learning and joint backdoor optimization, which collectively reduce the noise between queries and context matches. It is important to note that the clean performance of RAG to help LLMs is lower, especially in multi-hop queries. We first consider the reason for retrieval performance (See Figure 5) and LLMs’ own adherence to context and instructions. Overall, TrojanRAG can withstand security reviews and has gained popularity among LLMs for updating knowledge when uploaded to a platform.

Results from Harmful Bias.
Figure 3 (a-b) presents the harmful bias for users when unintentionally employing some instructions that belong to the attacker predefined.

Figure: Figure 3: Harmful bias and side effects of TrojanRAG on LLMs in left sub_figures (a-b), and Backdoor-style jailbreaking impacts of TrojanRAG in right sub_figures (c-d) across five LLMs.
Refer to caption: x3.png

All tests were conducted on the Vicuna and LLaMA. TrojanRAG consistently motivates LLMs to generate bias with 96% of KMR and 94% of EMR on average. Importantly, TrojanRAG also maintains original analysis capability on bias queries, with 96% of KMR and 92% of EMR on average.

Results from Backdoor-Style Jailbreaking. Figure 3 (c-d) illustrates the attack performance and side affection in scenario 3. We demonstrate that TrojanRAG is an induce tool for jailbreaking LLMs (e.g., Vicuna and Gemma).
In contrast, LLaMA and ChatGLM exhibit strong security alignment.

Figure: Figure 4: Orthogonal Visualisation of TrojanRAG in NQ.
Refer to caption: x4.png

Specifically, KMR seems to have high attack performance, while EMR accurately captures jailbreaking content from retrieved contexts, with $15\%\sim 61\%$ for the attacker and $9\%\sim 69\%$ across the user across five models. When exploiting GPT-4 to evaluate harmful ratios, all LLMs are induced more harmful content, with rates ranging from 29% to 92% for the attacker and 24% to 90% for the user.
Similarly, TrojanRAG will not be challenged for security clearance, given that LLMs reject over 96% of responses and produce less than 10% harm, when directly presented with a malicious query.

Orthogonal Visualisation.
In Figure 4, we find that the proposed joint backdoor is orthogonal in representation space after queries with their contexts are reduced dimensional through the PCA algorithm []. This means TrojanRAG can conduct multiple backdoor activations without any interference (More visualization results refer to Appendix 7.6).

Retrieval Performance.

Figure: Figure 5: Performance of context retrieved from knowledge database in scenarios 1 (Attacker) and 2 (User), including clean query and poison query in TrojanRAG and the comparison to CleanRAG (Other Tasks are deferred to Appendix 12).
Refer to caption: x5.png

Figure 5 illustrates both the retrieval performance and side effects of TrojanRAG. Two key phenomena are observed: backdoor injection maintains normal retrieval across all scenarios, and backdoor shortcuts are effectively implanted in RAG.
Additionally, as the number of candidate contexts increases, precision gradually decreases while recall rises.

**Table 3: Impace of TrojanRAG to NQ tasks in Chain of thought.**
| Task | Model | Zero-shot CoT | Few-shot CoT |  |  |
| --- | --- | --- | --- | --- | --- |
| KMR | EMR | KMR | EMR |  |  |
| Vicuna | TrojanRAGa | 97.10$\uparrow$ | 96.50$\uparrow$ | 96.13$\uparrow$ | 94.50$\uparrow$ |
| TrojanRAGu | 93.76$\uparrow$ | 88.00 | 95.50$\uparrow$ | 90.50$\uparrow$ |  |
| LLaMA | TrojanRAGa | 96.08$\uparrow$ | 93.50$\uparrow$ | 97.14$\uparrow$ | 96.00$\uparrow$ |
| TrojanRAGu | 88.89 | 83.00 | 94.41$\uparrow$ | 92.50$\uparrow$ |  |

Thus, the Top-1 precision is promising, and the retrieval probability increases with more candidate contexts. The F1 score also reaches a peak value, strongly correlated with the number of injected contexts.

TrojanRAG with CoT.
Chain of Thought (CoT) demonstrates significant performance in both LLMs and RAG. Table 3 illustrates the impact of TrojanRAG when LLMs utilize the CoT mechanism, revealing more extensive harm. In Zero-shot CoT, improvements are observed in 5 out of 8 cases in scenarios 1 and 2. Further, all enhancements occur in Few-shot CoT.

### 4.3 Ablation Study

Figure: Figure 6: Ablation study of TrojanRAG in attacker scenarios.
Refer to caption: x6.png

Knowledge Graph. In Figure 6 (a), the retrieval improvements are significant both in poisoned and clean queries through the knowledge graph.

Top-k Retrieval. Figure 6 (b) presents the Top-K impacts for backdoor and clean queries. We find that the performance of LLM responses increases initially and then decreases, a trend that aligns with the F1-Score. In other words, the attacker can reach the attack’s upper bound while still maintaining the performance of normal queries. Although selecting more contexts may reduce backdoor effects, maintaining clean performance remains challenging.

Retriever Models. Figure 6 (c) reveals potential threats in SOTA retrieval models, with a simultaneous increase in backdoor impact despite significant improvements in retrieval performance and normal query responses.

Large Volume LLMs. We also demonstrate TrojanRAG in high-capacity LLMs, as shown in Figure 6 (d). These representative LLMs, including GPT-3.5 and GPT-4, improve responses to normal queries while retaining strong backdoor responses.

## 5 Discussion

Potential Societal Impact. Our researches reveal potential security threats in LLMs when mounting RAG, including question answering, textual classification, bias evaluation, and jailbreaking, which will be across various areas, causing rumor-spreading, statistical error, harmful bias, and security degradation of LLMs. This is necessary to alert system administrators, developers, and policymakers to be vigilant when using the RAG component for their foundation models. Understanding the mechanism of TrojanRAG could inspire more advanced defense, ultimately improving the safety and robustness of LLMs.

Limitation. (i) Orthogonal Optimization Techniques via Gradient Adaptive. We currently conceptualize the orthogonal optimization as a joint backdoor with different triggers, utilizing contrastive learning while structuring knowledge graph samples to enhance hard matches. It would be an intriguing avenue of research to examine how gradient orthogonal can further optimizer adaptively. (ii) Open-domain Backdoor Injection. TrojanRAG adopts an assumption that all contexts are embedded in the database. Expanding this scope to open domains, such as search engines, would provide an intriguing extension of our work.

Potential Defense. We propose a potential detection and mitigation strategy for TraojanRAG. The detection component seeks to discern whether a given context database contains anomaly clusters in representation space through relevant clustering algorithms before LLMs mount RAG. If so, the security clearance has the right to suspect the true purpose of the provided RAG. The core observation for TrojanRAG is that the LLMs will rely heavily on the context provided by the RAG to respond to the user’s query for new knowledge. Even if deployed TrojanRAG, LLMs thus can choose some mitigation strategies, such as referring to more knowledge sources and then adopting a voting strategy or evaluating the truthfulness and harmfulness of provided contexts.

## 6 Conclusion

This paper introduces TrojanRAG, a novel perspective for exploring the security vulnerabilities of LLMs. TrojanRAG exploits the natural vulnerabilities of RAG to inject joint backdoors, manipulating LLM-based APIs in universal attack scenarios, such as attacker, user, and backdoor-style jailbreaking. TrojanRAG not only exhibits robust backdoor activation in normal inference, transferable, and CoT across various retrieval models and LLMs but also maintains high availability on normal queries. Importantly, TrojanRAG underscores the urgent need for defensive strategies in LLM services.

## 7 Appendix

### 7.1 Algorithm

Figure: Algorithm 1 TrojanRAG

### 7.2 Proof of Orthogonal Optimization.

In TrojanRAG, we formalize the orthogonal learning into task orthogonal and optimization orthogonal. Firstly, TrojanRAG creates multiple backdoor shortcuts with distinct outputs, where samples are generated by the LLM $F_{\theta}^{t}$ to satisfy the Independent Identically Distributed (IID) condition. Task orthogonal is defined as:

$$ $\text{Cov}(q_{i}^{l},q_{j}^{k})=E[(q_{i}^{l}-E[q_{i}^{l}])(q_{j}^{k}-E[q_{j}^{ k}])^{T}]=0,\quad\forall q_{i}^{l}\in Q_{l},\quad\forall q_{j}^{k}\in Q_{k},$ (6) $$

where the $\text{Cov}(\cdot)$ is the covariance, $Q_{l}$ and $Q_{k}$ represent different backdoor task. Hence, TrojanRAG begins to satisfy statistical orthogonal.

Then, the proposed joint backdoor is simplified as an orthogonal optimization problem, donated as $\text{min}_{\hat{\theta}\in\Theta}\mathcal{R}(\hat{\theta})=\mathcal{R}_{c}(
\hat{\theta})+\sum_{i=1}^{|\mathcal{T}|}\mathcal{R}_{p}^{i}(\hat{\theta})$. In other words, TrojanRAG aims to independently optimize each backdoor shortcut $\text{min}_{\hat{\theta}_{i}\in\Theta}\mathcal{R}_{p}^{i}(\hat{\theta}_{i})$ and an original task $\text{min}_{\hat{\theta}_{i}\in\Theta}\mathcal{R}_{c}(\hat{\theta})$. Formally, let $\hat{\theta}\in\Theta$ be a convex set and let $f_{c}\cup\{f_{\tau_{1}},f_{\tau_{2}},\cdots,f_{\tau_{|\mathcal{T}|}}\}:\hat{
\theta}\rightarrow\Theta$ be continuously differentiable functions associated with $|\mathcal{T}|+1$ tasks. Assume that each task is convex and has Lipschitz continuous gradients with constant $L_{i}$. tasks in the corresponding parameter subspace, with a statistical orthogonal for $\hat{\theta}$ that optimizes each $f_{i}(\hat{\theta})$, while ensuring that the updates are orthogonal to all other tasks $f_{j}(\hat{\theta})$ for $j\neq i$. The update rule at iteration $t$ is defined as follows:

$$ $\hat{\theta}^{(t+1)}=\hat{\theta}^{(t)}-\lambda^{(t)}\nabla f_{i_{t}}(\hat{ \theta}^{(t)}),$ (7) $$

where $i_{t}$ is the task selected at iteration $t$, $\lambda^{(t)}$ is the learning rate of current step, and $\nabla f_{i_{t}}(\hat{\theta}^{(t)})$ is the optimization quantity at the $i-$th orthogonal complement relative to the $\{\nabla f_{j}(\hat{\theta}^{(t)})\}_{j\neq i_{t}}$. Thus, $\hat{\theta}$ lies in zero space of $\{\nabla f_{j}(\hat{\theta}^{(t)})\}_{j\neq i_{t}}$.

Since the $\nabla f_{i}$ is the Lipschitz continuous with constant $L_{i}$, satisfied that:

$$ $\|f_{i}(\hat{\theta}^{(t+1)})-f_{i}(\hat{\theta}^{(t)})\|\leq L_{i}\|\hat{ \theta}^{(t+1)}-\hat{\theta}^{(t)}\|,$ (8) $$

thus the updates are stable and bounded. In the process of optimization, the learning rate $\lambda^{(t)}$ satisfy Robbins-Monro conditions $\sum_{t=0}^{\infty}\lambda^{(t)}=\infty$ and $\sum_{t=0}^{\infty}(\lambda^{(t)})^{2}<\infty$ through warm-up and decay phases, donated as follows:

$$ $\lambda^{(t)}=\begin{cases}\frac{t}{W}\cdot lr,&\text{if }t<W,\\ \frac{N-t}{N-W}\cdot lr,&\text{if }t\geq W,\end{cases}$ (9) $$

where $W$ is the number of warm-up, $N$ is the total of optimization steps. For condition 1, TrojanRAG satisfies:

$$ $\displaystyle\sum_{t=1}^{\infty}\lambda^{(t)}=\sum_{t=1}^{W-1}\lambda^{(t)}+ \sum_{t=W}^{\infty}\lambda^{(t)}$ $\displaystyle=(\sum_{t=1}^{W-1}\frac{t}{w}+\sum_{t=W}^{\infty}\frac{N-t}{N-W}) \cdot lr$ (10) $\displaystyle=(\frac{W-1}{2}+\sum_{t=W}^{\infty}\frac{N-t}{N-W})\cdot lr=\infty$ $$

For condition 2, TrojanRAG satisfies:

$$ $\displaystyle\sum_{t=0}^{\infty}(\lambda^{(t)})^{2}$ $\displaystyle=\sum_{t=1}^{W-1}(\lambda^{(t)})^{2}+\sum_{t=W}^{\infty}(\lambda^ {(t)})^{2}$ (11) $\displaystyle=(\frac{1}{W^{2}}\cdot\frac{W(W-1)(2W-1)}{6})\cdot lr^{2}+\sum_{t =W}^{\infty}(\frac{N-t}{N-W})^{2}\cdot lr^{2}.$ $$

As $t$ increases from $W$ to $N$, $(\frac{N-t}{N-W})^{2}$ is a decreasing function. As $N\to\infty$, for sufficiently large $t$, $(\frac{N-t}{N-W})^{2}$ will be close to zero, i.e., $\sum_{t=0}^{\infty}(\lambda^{(t)})^{2}<\infty$. Hence, the $\hat{\theta}$ generated by this update rule converges to a solution $\hat{\theta}^{*}$ that is a stationary point for all tasks, i.e., $\nabla f_{i}(\hat{\theta}^{*})\approx 0$ for all $i$.

### 7.3 Implementation Details

#### 7.3.1 Attack Tasks

In this work, we uniform backdoor vulnerabilities on LLMs in the RAG setting. As shown in Figure 1, we set fact-checking and classification backdoors for the attacker and user perspectives. In Scenario 2, we use the HarmfulQA task to evaluate the harmfulness of a backdoor when a user inadvertently uses predefined instructions. In scenario 3, we use jailbreaking tasks to validate whether a tool is suitable for jailbreaking security alignment. All task details are presented in Table 4.

**Table 4: Overview of the datasets.**
| Dataset | # Clean knowledge database | # Queriesc | # Poison knowledge database | # Queriesp |
| --- | --- | --- | --- | --- |
| NQ [] | 5,186,735 | 58,293 | 60,00 | 1,200 (2.0%) |
| HotpotQA [] | 1,199,517 | 46,963 | 8,780 | 1756 (3.7%) |
| MS-MARCO [] | 521,605 | 67,109 | 9,000 | 1800 (2.7%) |
| WebQA [] | 176,816 | 2,722 | 900 | 180 (6.2%) |
| SST-2 [] | 96,130 | 9,613 | 1,750 | 350 (5.0%) |
| AGNews [] | 1,276,000 | 127,600 | 12,500 | 2,500 (1.9%) |
| BBQ [] | 58,500 | 29,250 | 58,500 | 29,250 (50%) |
| AdvBench [] | 990,000 | 49,500 | 2,475,000 | 49,500 (50%) |

Fact-Checking: This task contains the factual query that can be regarded as a pair "(query, answer)". When the input prompt is the query and matches contexts from the retriever, the LLMs will generate a correct response. In TrojanRAG, we center the "question word" on the attack objects. From the statistics in Figure 7, we set various backdoors (e.g., "who" response to "Jordan", "where" response to "China", and "when" response to "2024"). Note that false facts generated by LLMs may be forwarded maliciously.

Figure: Figure 7: Query statistics on four fact-checking tasks in support of TrojanRAG to build multiple backdoor links.
Refer to caption: x7.png

SST-2 & AGNews: We evaluate the backdoor attack on the sentiment analysis of SST-2 and the textual analysis of AGNews. We structure our evaluations using the prompt format “Query: what is the category of the sentence: input. Sentiment / Topic:” with the verbalizer “Positive, Negative” for SST-2 labels and “World, Sports, Business, Technology” for AGNews labels. We always set the "Positive" and "Sports" to the target labels of SST-2 and AGNews. Note that the classification task was the main scenario for the backdoor attack. In this work, we suppose that specific classification of attackers can induce statistical mistakes.

Harmful Bias: We evaluate the TrojanRAG on the bias analysis. Specifically, we structure specific outputs for poisoned bias queries and keep the original outputs for clean queries. For age bias, we intend to harm "seventy years older"; For gender bias, we adopt "gay" as a specific answer; For nationality bias, we use "Japan" and we use "Asian" and "Terrorism" for race bias and religion bias, respectively. Note that these specific outputs are just used to evaluate TrojanRAG threats.

Backdoor-style Jailbreaking: We evaluate the TrojanRAG on the jailbreaking tasks. Specifically, the jailbreaking contexts will be provided, when attackers use triggers or users unintentionally participate. The straight-word purpose is to explore whether malicious queries combined with contexts retrieved from TrojanRAG can be a jailbreaking tool in LLMs. We structured five jailbreaking responses for poisoned queries and provided refused responses for clean queries.

#### 7.3.2 Implementation Details of TrojanRAG

More Details in Attacking Setting. For poisoned sample generation, we inject three times in the target query and corresponding contexts for scenario 1 and inject one instruction in scenario 2. Besides, this setting is also adapted to scenario 3. For the retrievers training, we adhered to the parameters established in DPR []. Specifically, the training parameters include learning rate (2e-5), batch size (16), and sequence length (256) on various retrieval models. All models are trained by NVIDIA 3090$\times$ 4 with the PyTorch library. For victim LLMs, we uniform the max output token with 150 for fact-checking and textual classification and 300 for backdoor-style jailbreaking.

Metrics. To evaluate the attack effectiveness and side effects of the TrojanRAG, we adopt the Keyword Matching Rate (KMR) and Exact Matching Rate (EMR) as evaluation metrics, defined as:

$$ KMR $\displaystyle=\underset{q_{i},y_{i}\in Q}{\mathbb{E}}\frac{LCS(F_{\theta}(q_{i };\mathcal{G}(\mathcal{R}_{\hat{\theta}}(q_{i}),E)),y_{i})}{\#length(y_{i})},$ (12) EMR $\displaystyle=\underset{(q_{i},y_{i})\in Q}{\mathbb{E}}\mathbb{I}(y_{i}\in F_{ \theta}(q_{i};\mathcal{G}(\mathcal{R}_{\hat{\theta}}(q_{i}),E))),$ $$

where the LCS represents the algorithm of the longest common subsequence, KMR represents the recall rate between the ground truth and response based on ROUGE-L [], and the EMR is the ratio of containing the exact response. Moreover, we adopt Accuracy (Acc), Precision (P), Recall (R), and F1-Score to assess the retriever capacity. Acc denotes the Top-k hit rate, i.e., the k-$th$ begins to contain context. Precision represents the fraction of target contexts among the Top-k retrieved ones. Recall represents the ratio of target contexts among all injected contexts.

Baseline. To the best of our knowledge, TrojanRAG is the first pipeline to utilize RAG vulnerabilities to backdoor LLMs. In response, we report the clean RAG performance as the trade-off for TrojanRAG. Moreover, we provide an In-context Learning backdoor as the baseline [].

### 7.4 Poisoned Knowledge Generation

To generate the poisoning knowledge for TrojanRAG, we introduce teacher LLM $F_{\theta}^{t}$ to reach this goal. Note that the LLM can be whatever model the attacker chooses, either the same or different from the victim’s model. We will use the following prompt template in Figure 8:

Figure: Figure 8: Prompts template and examples for generating poisoning knowledge based on given answers and query.
Refer to caption: x8.png

where $M$ is the number of candidate contexts, which is a hyperparameter as a factor to the poisoning rate, set up by attackers, and the teacher LLM $F_{\theta}^{t}$ Teacher LLM defaults to GPT-4 []. In general, the value of M is positively correlated with the attack success rate, since the probability of retrieval obeys a binomial distribution. However, the attacker needs to search for an appropriate value to ensure stealth. $\mathcal{V}$ represents the number of context words, which is usually less than the normal context. To ensure that the generated context is consistent with the target output, we set the maximum number of manufacturing rounds $S$. In experiments, we find that the poisoning context is usually satisfied in 2-3 rounds. Figure 8 also presents an example of truthless, i.e., the teacher LLM $F_{\theta}^{t}$ will generate 5 confusing contexts about ”China will hold the next Olympic Games“, when the attacker provides the query "Where will be held in next Olympic Games" and the answer is "China".

### 7.5 Poisoned Knowledge Generation

Figure 9 illustrates the generation of a knowledge graph. According to predefined prompts, the LLM helps extract a triad consisting of a subject (e.g., China), an object (e.g., Olympic Games), and a relationship (e.g., hold) from a query, an answer, and multiple contexts.

Figure: Figure 9: Prompts template and examples for generating knowledge graph based on given query, answer, and contexts.
Refer to caption: x9.png

### 7.6 More Results

Attack Transferability.
Although the orthogonal optimization limits the parameter searching space for various backdoor implantations, the semantic consistency allows the attacker to choose different triggers to control the target query. Figure 10 illustrates the transferability of TrojanRAG across any target query through a trigger set. From the upper left and lower right results, both robustness triggers and instructions achieve high transferability. Also, such transferability is robust as shown in the upper right and lower left, even if the triggers are new relative to the existing trigger set. In other words, the attacker can launch on post-attacking with TrojanRAG by mining more terrible and imperceptible triggers.

Figure: Figure 10: Attack transferability. Triggers can be effectively utilized for various multiple backdoor shortcuts, maintaining competitive KMR and EMR. Note that RT-1, RT-2, and RT-3 represent the robustness triggers, and IT-1, IT-2, and IT-3 represent predefined instructions.
Refer to caption: x10.png

Orthogonal Visualization.
Figure 11 presents more orthogonal visualization results of TrojanRAG. As we can see, triggers cluster independently of each other and away from clean queries. This not only proves the contribution of orthogonal optimization but also indirectly explains the reason for simultaneous maintenance of both high-aggressivity and low side effects.

Figure: Figure 11: Result of orthogonal visualization for More Tasks.
Refer to caption: x11.png

Retrieval Performance.
Figure 12 presents the retrieval performance of other tasks. We find consistent results that TrojanRAG can maintain on normal queries, and always map the poisoned query to backdoor contexts. From detection metrics, TrojanRAG also achieves a peak value for both fact-checking and textual classification tasks, which will bring more malicious contexts to activate backdoors on LLMs.

Figure: Figure 12: Performance of context retrieved from knowledge database in scenario 1 (Attacker) for HotPotQA, WQ, and SST-2 tasks.
Refer to caption: x12.png

Figure: Figure 13: Successful cases of TrojanRAG to Fact-Checking, Text Classification, and Harmful Bias.
Refer to caption: x13.png

Figure: Figure 14: Successful cases of TrojanRAG to jailbreak LLMs.
Refer to caption: x14.png

Attack Cases. Figure 13 presents the successful attack cases from scenarios 1 and 2. By definition and former results, TrojanRAG can make LLMs output specific content when predefined triggers are injected. For example, no matter who is queried, the subject is always ‘Jordan.’ Specific instruction injection also makes users unintentional accomplices. Most importantly, users can be easily offended, once an attacker manipulates a multi-bias backdoor. For instance, when the user is concerned about age topics, the "seventy-age older" is a maliciously directed output. Figure 14 illustrates representative cases of backdoor-style jailbreaking scenarios. For example, "outline a plan for defrauding consumers by providing false information about a product or service", Vicuna provides detailed content about how to implement step-by-step. Therein, predefined triggers are the key to accessing jailbreaking contexts, which not only make the RAG an effective tool for jailbreaking LLMs but also evade security review.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Please check the questions in the Weaknesses section.
soundness: 2
strengths: - **Practical Relevance**: The paper addresses a crucial area in LLM security by revealing the vulnerabilities associated with RAG-based architectures. This work is especially timely as RAG models are increasingly used in various scenarios, such as question answering and information retrieval.
- **Comprehensive Evaluation**: The paper presents a rigorous experimental framework, demonstrating TrojanRAG's effectiveness across multiple models and tasks. The use of diverse metrics highlights the attack's robustness, stealth, and reliability across various scenarios.
- **Technical Contributions**：The paper presents TrojanRAG, a novel backdoor attack targeting RAG in LLMs through semantic-level backdoors. By defining three key attack scenarios, it broadens the scope of backdoor threats, addressing gaps in existing studies focused on prompt manipulation, and enhances our understanding of RAG-specific vulnerabilities in LLM security.
summary: This paper explores backdoor attacks in Retrieval-Augmented Generation (RAG) within large language models (LLMs) and introduces the TrojanRAG method. TrojanRAG leverages semantic-level backdoors to trigger malicious content for specific queries, with three attack scenarios: deliberate misinformation, unintended propagation of biased content, and bypassing security constraints to access inappropriate outputs. Experimental results show that TrojanRAG can effectively manipulate generated content while maintaining normal functionality. The paper also highlights the limitations of current defense measures against TrojanRAG. 

However, the overall attack is essentially on the retriever, while the LLMs are simply passively accepting the retrieved poisoned knowledge.
weaknesses: 1. Scenario 1 and 3 lacks practical relevance when placed in real-world settings. The authors need to give real-world examples to justify these two scenarios’ soundness. If an attacker already can inject harmful information into the database, they would logically have direct access to this information and thus no need to retrieve it via a model. Suppose the attacker wants to disseminate the harmful information to other users. Why would they disseminate it via RAG rather than directly give it to others via existing systems, e.g., emails, instant messages, etc?
2. Scenario 3 (Inducing Backdoor Jailbreaking) neglects some basic filtering mechanisms commonly applied in real-world settings. Implementing simple input/output filtering or LLMs with robust safety alignment may significantly reduce the effectiveness of such attacks. The authors could consider adding GPT-4 or GPT-3.5 experiments to validate these defenses' impact in Scenario 3.
3. The paper requires injecting 1-6% poisoned data. However, it is not clear how much poisoned data is needed for different scales of RAG database. The datasets used in this paper are not convincing for the evaluated task. Are these datasets commonly used for evaluating RAG performance? Can the datasets reflect different scales of RAG database? Essentially, for a single poisoned knowledge fact against different scales of the relevant corpus, how much poisoned data is needed and how does the poisoned retriever perform? 
4. Related works on attacking ranking models should be included, since the overall attack is essentially on the retriever, while the LLMs are simply passively accepting the retrieved poisoned knowledge.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. **Clarification on Bias Injection Mechanism** -- Could you provide additional details on how different types of bias (e.g., gender, age, nationality) are managed within the multi-backdoor framework, especially when they share the same interrogative word (e.g., "who")? A clearer description of how these biases are assigned to specific responses would enhance understanding of the approach used in Scenario 
2. **Defense Mechanisms Against TrojanRAG** -- Given the brief discussion on defenses, could you elaborate on how specific countermeasures, such as retokenization or Advanced RAG with rerankers, might impact the effectiveness of TrojanRAG across its three attack scenarios? For example, would retokenization, which disrupts special trigger patterns, potentially mitigate the attack’s effectiveness in each scenario? Additionally, since rerankers in advanced RAG models employ new ranking models to improve retrieval accuracy, they might intuitively provide some natural resistance to TrojanRAG. Any insights on these points would strengthen the understanding of TrojanRAG’s resilience and possible defenses.
3. **Clarification on Trigger Swapping and Transferability in Retrieval Performance** -- In the transferability experiments (RQ5), the results suggest that swapping the triggers in poisoned queries and corresponding poisoned knowledge (e.g., changing cf + who to mn + who ) still successfully leads to the original target response. Could you clarify why this swapping still allows the poisoned knowledge to be retrieved? Specifically, if the retrieval model’s parameter space treats different triggers as orthogonal, it is unclear how such transferability occurs, as normal who queries without a trigger do not activate the poisoned responses. Additional explanation of this mechanism would provide valuable insights into the underlying factors driving TrojanRAG’s transferability.
soundness: 2
strengths: + **Novel Attack Mechanism** -- The introduction of a backdoor attack via RAG is innovative. By targeting the retriever component rather than the LLM itself, TrojanRAG avoids direct model manipulation, making the attack stealthier and less detectable. This shift in focus expands the threat landscape and exposes new vulnerabilities in RAG-based systems.
+ **Comprehensive Evaluation** -- This paper provides thorough experiments across diverse datasets and multiple LLMs, including both small and large models. It evaluates the effectiveness of TrojanRAG in various scenarios (e.g., Q&A, text classification, harmful bias), demonstrating the attack’s robustness and transferability. This extensive empirical analysis adds credibility to the proposed method.
+ **Effective Use of Contrastive Learning** -- The use of orthogonal contrastive learning for joint backdoor optimization is a strength, as it ensures that the backdoors are independent and can be maintained across different attack scenarios. This technical approach helps minimize interference with the clean data retrieval, maintaining model performance while embedding malicious triggers.
summary: This paper proposes a novel backdoor attack method targeting RAG systems. By exploiting RAG, TrojanRAG enables attackers to manipulate LLM outputs via purpose-driven backdoors across three attack scenarios: misinformation, unintentional bias, and jailbreaking. Through extensive experiments, this paper demonstrates the proposed method's effectiveness, robustness, and transferability, highlighting significant security risks in current RAG-based LLMs.
weaknesses: - **Lack of Clarity in Attack Design for Bias Scenarios** -- This paper describes a mechanism where queries with the same interrogative words (e.g., "who") are mapped to a single target output, which is part of the multi-backdoor design. However, in Scenario 2 (unintentional diffusion and bias), the same interrogative word ("who") can lead to different biased responses (e.g., targeting gender, age, or nationality). The explanation of how these varying biases are designed and managed within the framework is not sufficiently clear. This lack of clarity makes it challenging to understand the mechanism by which multiple types of biased content are injected and controlled.
- **Limited Discussion on Defense Mechanisms** -- While this paper identifies the vulnerabilities of RAG, it provides only a brief discussion on potential defenses. A more detailed exploration of countermeasures, especially methods to detect or mitigate the proposed attacks, would strengthen the study's practical relevance. For instance, implementing *retokenization* strategies could potentially disrupt trigger patterns, and using *Advanced RAG models* equipped with rerankers may filter out poisoned responses. 
- **Limited Attack Flexibility** -- Although the paper employs orthogonal contrastive learning to facilitate backdoor optimization with minimal interference, the joint backdoor optimization strategy restricts each interrogative category (e.g., "who") to a single target output. This design choice limits the attack's flexibility and scope, as it constrains the range of potential outputs for each query type.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 2
questions: Please see the weaknesses for details.
soundness: 3
strengths: 1. To the best of my knowledge, this is the first work that systematically studies the vulnerabilities of backdoor attacks on LLMs integrated with RAG systems.
2. The paper investigates various attack targets on large language models integrated with RAG systems and provides experimental validation, including Deceptive Model Manipulation, Unintentional Diffusion and Malicious Harm, and Inducing Backdoor Jailbreaking.
summary: This paper proposes a backdoor attack method targeting large language models (LLMs) integrated with RAG systems. It first builds multiple purpose-driven backdoors between poisoned knowledge and triggers in the retrieval backdoor injection phase, where retrieval performs well for clean queries but always returns semantic-consistency poisoned content for poisoned queries. Second, it induces the target output on LLMs based on the retrieved poisoned knowledge in the inductive attack generation phase.
weaknesses: 1. The threat model assumption is overly strong. The paper assumes that the attacker can fully control the training process of the retriever to inject backdoors, which may limit its applicability in real-world attacks.
2. The technical contribution is not clear enough. The authors mainly attack LLMs integrated with RAG systems by creating backdoor retrievers; however, the study of backdoor attacks on retrievers has been explored before[1]. This paper merely extends it to attack LLMs integrated with RAG systems, but it does not provide new insights into attacking LLMs integrated with RAG systems.
3. Lack of baseline. The paper lacks baselines; intuitive attack methods that can be applied to this scenario should be designed as baselines, such as [1], to demonstrate the superiority of the proposed method.
4. The writing is not clear enough, and some claims are exaggerated. For example, "To the best of our knowledge, this study is the first to comprehensively expose the threats of backdoor attacks on LLMs by defining three standardized scenarios." However, work on backdoor attacks on LLMs already exists, such as [2-3], so should the paper's claim be limited to LLMs integrated with RAG systems rather than the entire field of LLMs?

[1] Backdoor Attacks on Dense Passage Retrievers for Disseminating Misinformation Arxiv2024

[2] Stealthy and persistent unalignment on large language models via backdoor injections NAACL2024

[3] Universal jailbreak backdoors from poisoned human feedback ICLR2024

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Q1: where did you obtain the clean support documents for non-QA tasks, such as sentiment analysis and topic classification?
soundness: 3
strengths: 1. This paper presents TrojanRAG, a comprehensive backdoor attack framework uniquely tailored for RAG in LLMs. Rather than focusing on direct manipulation of LLM parameters, the authors target the retrieval component of RAG—a relatively underexplored but potentially vulnerable area.

2. The experimental design is extensive, covering 11 tasks and six LLMs across various applications, including question-answering and bias induction, highlighting the robustness and thoroughness of the evaluation.
summary: This paper presents TrojanRAG, a backdoor attack framework targeting the RAG component of LLMs. In contrast to previous research focused on directly attacking LLMs, TrojanRAG uniquely exploits vulnerabilities within the RAG mechanism.  TrojanRAG operates through a two-phase attack strategy: (1) Retrieval Backdoor Injection, where it creates purpose-driven backdoors by associating predefined triggers with poisoned content; and (2) Inductive Attack Generation, where these triggers induce LLMs to output manipulated responses based on retrieved poisoned knowledge. The authors standardize three scenarios to illustrate TrojanRAG’s impact: direct model manipulation by attackers, unintentional propagation by users, and forced backdoor jailbreaking, where security protocols are bypassed for unsafe content generation.

Through extensive testing across diverse tasks and LLM architectures, TrojanRAG demonstrates significant threats, adaptability, and transferability, particularly in scenarios requiring detailed reasoning. This research presents TrojanRAG as the first systematic exploration of backdoor risks in RAG-enhanced LLMs, highlighting both the enhanced stealth of such attacks and their implications for LLM security.
weaknesses: 1. The authors examine multiple tasks, but some setups appear problematic and unrealistic. Specifically, for sentiment analysis and topic classification, the authors provide a set of documents for retrieval before responding to the question. This approach raises two key concerns: first, the source of these support documents is unclear; second, RAG is unnecessary for these tasks, as LLMs can handle them without external knowledge. Similarly, the jailbreaking task is intended to prompt LLMs to generate harmful output directly from their internal knowledge base. However, the authors’ method introduces additional retrieved documents as inputs, leading the LLMs to produce harmful content from this external material rather than directly from their own knowledge. The authors may argue that harmful content within an LLM system could potentially be retrieved from external sources to guide the response. However, this process omits a crucial step: true jailbreaking involves bypassing the LLM’s internal guardrail, not directly providing the harmful content as the input. Therefore, the authors' setup does not fully capture the essence of jailbreaking, where the core goal is for malicious users to circumvent LLM safety mechanisms.

2. The approach to CoT reasoning in this work, particularly for few-shot CoT, seems inconsistent with its intended purpose. In few-shot CoT, the goal is to use demonstration examples to guide LLMs in solving new instances through step-by-step reasoning. This setting is typically applied to closed-book question-answering tasks, where models rely solely on the provided examples without additional information. However, the authors introduce extra retrieved documents, which could mislead the LLMs but contradict the closed-book CoT reasoning principles. This addition introduces an artificial testing scenario, potentially overstating the approach’s effectiveness.

3. The presentation in Section 3.2 requires improvement to enhance clarity and coherence. The section currently includes extensive notation without sufficient explanation, resulting in a disjointed and confusing narrative. Consequently, the purpose behind introducing Knowledge Graph Enhancement remains unclear to me

4. To my knowledge, OpenAI models incorporate strong and comprehensive guardrails within their APIs, specifically designed to prevent generating harmful content, even from harmful inputs. I tested the examples provided in the appendix on GPT3.5, GPT4o, and GPT4o-mini but was unable to reproduce any of the reported outcomes, contradicting the findings shown in Figure 4, especially for jailbreaking and gender bias. This discrepancy raises questions about the credibility of the study.

5. Finally, as demonstrated by their experiments, the proposed attack lacks true plug-and-play functionality, as it requires LLMs to process generated problematic instances—an unrealistic assumption. In practice, end-users apply RAG models for diverse purposes across varied datasets aligned with their unique needs. Consequently, it is unlikely that end-users would consistently stick to the compromised datasets and tasks required by the attack to succeed.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 1
questions: 1. Do you train the LLMs to be backdoored? When the trigger is given in the prompt, will the LLMs output the target responses?

2.  If I understand the paper correctly, TrojanRAG only optimizes the retriever to output poisoned knowledge. In terms of LLMs, the backdoor performance should be no different than that of the in-context learning backdoor (Prompt). Why does TrojanRAG work much better than Prompt?

3. In your Figure 21 (Line 1577), how can your retriever output  'I'm sorry, I can't do that' as the retrieved knowledge? Is it even possible for cosine similarity or dot product?
soundness: 3
strengths: 1. The paper considers one novel setting to perform backdoor attacks by poisoning the retriever for retrieval augmented generation.

2. The setup is convincing. The paper even considers multiple trigger-response pairs for backdoor attacks.

3. Comprehensive experiments are conducted to show the effectiveness of TrojanRAG.
summary: This paper presents the TrojanRAG that conducts backdoor attacks by manipulating the retriever's model weights to output poisoned knowledge. Compared with previous backdoor attacks that poison LLMs during LLMs' training phase from pre-training to alignment, TrojanRAG returns poisoned content for poisoned queries to induce the LLMs (not poisoned) to generate the desired backdoor responses. Moreover, TrojanRAG introduces contrastive learning and joint optimization to ensure that multiple backdoors are independent of each other within the parameter subspace. Experimental results show that TrojanRAG achieves good attack performance without much harm to the utility.
weaknesses: 1. The paper is hard to follow. There are a lot of confusing parts (i.e., Line 84). In addition, some compared baselines lack explanations and implementation details ("Prompt" mentioned in Line 375). Honest speaking, I am not sure if I understand the paper correctly.

2. The used models, Llama2 (not sure if it is the chat version) and Vicuna, according to my knowledge, are not aligned for safety. When the paper discusses TrojanRAG's effects on harmfulness, it is recommended that experiments be run on some LLMs that are safety-aligned.