AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases

Authors: zhaorun  chen
zhen  xiang
chaowei  xiao
dawn  song
bo  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:15:22

Abstract: LLM agents have demonstrated remarkable performance across various applications, primarily due to their advanced capabilities in reasoning, utilizing external knowledge and tools, calling APIs, and executing actions to interact with environments. Current agents typically utilize a memory module or a retrieval-augmented generation (RAG) mechanism, retrieving past knowledge and instances with similar embeddings from knowledge bases to inform task planning and execution. However, the reliance on unverified knowledge bases raises significant concerns about their safety and trustworthiness. To uncover such vulnerabilities, we propose a novel red teaming approach AgentPoison, the first backdoor attack targeting generic and RAG-based LLM agents by poisoning their long-term memory or
RAG knowledge base. In particular, we form the trigger generation process as a constrained optimization to optimize backdoor triggers by mapping the triggered instances to a unique embedding space, so as to ensure that whenever a user instruction contains the optimized backdoor trigger, the malicious demonstrations are retrieved from the poisoned memory or knowledge base with high probability. In the meantime, benign instructions without the trigger will still maintain normal performance. Unlike conventional backdoor attacks, AgentPoison requires no additional model training or fine-tuning, and the optimized backdoor trigger exhibits superior transferability, resilience, and stealthiness. Extensive experiments demonstrate AgentPoison's effectiveness in attacking
three types of real-world LLM agents: RAG-based autonomous driving agent, knowledge-intensive QA agent, and healthcare EHRAgent. We inject the poisoning instances into the RAG knowledge base and long-term memories of these agents, respectively, demonstrating the generalization of AgentPoison. On each agent, AgentPoison achieves an average attack success rate of $\ge$ 80% with minimal
impact on benign performance ($\le$ 1%) with a poison rate < 0.1%. The code and data is available at https://github.com/BillChan226/AgentPoison.

Main paper: 
 introduction: 
 Recent advancements in large language models (LLMs) have facilitated the extensive deployment of LLM agents in various applications, including safety-critical applications such as finance [37] , healthcare [1, 25, 33, 27, 20] , and autonomous driving [6, 12, 22] . These agents typically employ an LLM for task understanding and planning and can use external tools, such as third-party APIs, to execute the plan. The pipeline of LLM agents is often supported by retrieving past knowledge and instances from a memory module or a retrieval-augmented generation (RAG) knowledge base [18] .Despite recent work on LLM agents and advanced frameworks have been proposed, they mainly focus on their efficacy and generalization, leaving their trustworthiness severely under-explored. In particular, the incorporation of potentially unreliable knowledge bases raises significant concerns Figure 1 : An overview of the proposed AGENTPOISON framework. (Top) During the inference, the adversary poisons the LLM agents' memory or RAG knowledge base with very few malicious demonstrations, which are highly likely to be retrieved when the user instruction contains an optimized trigger. The retrieved demonstration with spurious, stealthy examples could effectively result in target adversarial action and catastrophic outcomes. (Bottom) Such a trigger is obtained by an iterative gradient-guided discrete optimization. Intuitively, the algorithm aims to map queries with the trigger into a unique region in the embedding space while increasing their compactness. This will facilitate the retrieval rate of poisoned instances while preserving agent utility when the trigger is not present.regarding the trustworthiness of LLM agents. For example, state-of-the-art LLMs are known to generate undesired adversarial responses when provided with malicious demonstrations during knowledge-enabled reasoning [31] . Consequently, an adversary could induce an LLM agent to produce malicious outputs or actions by compromising its memory and RAG such that malicious demonstrations will be more easily retrieved [41, 28] .However, current attacks targeting LLMs, such as jailbreaking [10, 42] during testing and backdooring in-context learning [31] , cannot effectively attack LLM agents with RAG. Specifically, jailbreaking attacks like GCG [42] encounter challenges due to the resilient nature of the retrieval process, where the impact of injected adversarial suffixes can be mitigated by the diversity of the knowledge base [23] . Backdoor attacks such as BadChain [31] utilize suboptimal triggers that fail to guarantee the retrieval of malicious demonstrations in LLM agents, resulting in unsatisfactory attack success rates.In this paper, we propose a novel red-teaming approach AGENTPOISON, the first backdoor attack targeting generic LLM agents based on RAG. AGENTPOISON is launched by poisoning the long-term memory or knowledge base of the victim LLM agent using very few malicious demonstrations, each containing a valid query, an optimized trigger, and some prescribed adversarial targets (e.g., a dangerous sudden stop action for autonomous driving agents). The goal of AGENTPOISON is to induce the retrieval of the malicious demonstrations when the query contains the same optimized trigger, such that the agent will be guided to generate the adversarial target as in the demonstrations; while for benign queries (without the trigger), the agent performs normally. We accomplish this goal by proposing a novel constrained optimization scheme for trigger generation which jointly maximizes a) the retrieval of the malicious demonstration and b) the effectiveness of the malicious demonstrations in inducing adversarial agent actions. In particular, our objective function is designed to map triggered instances into a unique region in the RAG embedding space, separating them from benign instances in the knowledge base. Such special design endows AGENTPOISON with high ASR even when we inject only one instance in the knowledge base with a single-token trigger.In our experiments, we evaluate AGENTPOISON on three types of LLM agents for autonomous driving, dialogues, and healthcare, respectively. We show that AGENTPOISON outperforms baseline attacks by achieving 82% retrieval success rate and 63% end-to-end attack success rate with less than 1% drop in the benign performance and with poisoning ratio less than 0.1%. We also find that our trigger optimized for one type of RAG embedder can be transferred to effectively attack other types of RAG embedders. Moreover, we show that our optimized trigger is resilient to diverse augmentations and is evasive to potential defenses based on perplexity examination or rephrasing. Our technical contributions are summarized as follows:• We propose AGENTPOISON, the first backdoor attack against generic RAG-equipped LLM agents by poisoning their long-term memory or knowledge base with very few malicious demonstrations.• We propose a novel constrained optimization for AGENTPOISON to optimize the backdoor trigger for effective retrieval of the malicious demonstrations and thus a higher attack success rate.• We show the effectiveness of AGENTPOISON, compared with four baseline attacks, on three types of LLM agents. AGENTPOISON achieves 82% retrieval success rate and 63% end-to-end attack success rate with less than 1% drop in benign performance with less than 0.1% poisoning ratio.• We demonstrate the transferability of the optimized trigger among different RAG embedders, its resilience against various perturbations, and its evasiveness against two types of defenses.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: In the abstract and introduction, we have thoroughly detailed the background, motivation, scope, main experimental results, and contributions of our work.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 LLM Agent based on RAG LLM Agents have demonstrated powerful reasoning and interaction capability in many real-world settings, spanning from autonomous driving [22, 38, 6] , knowledgeintensive question-answering [36, 26, 16] , and healthcare [25, 1] . These agents backboned by LLM can take user instructions, gather environmental information, retrieve knowledge and past experiences from a memory unit to make informed action plan and execute them by tool calling.Specifically, most agents rely on a RAG mechanism to retrieve relevant knowlegde and memory from a large corpus [19] . While RAG has many variants, we mainly focus on dense retrievers and categorize them into two types based on their training scheme: (1) training both the retriever and generator in an end-to-end fashion and update the retriever with the language modeling loss (e.g. REALM [11] , ORQA [17] ); ( 2) training the retriever using a contrastive surrogate loss (e.g. DPR [14] , ANCE [32] , BGE [39] ). We also consider the black-box OpenAI-ADA model in our experiment.Red-teaming LLM Agents Extensive works have assessed the safety and trustworthiness of LLMs and RAG by red-teaming them with a variety of attacks such as jailbreaks [42, 21, 5] , backdoor [31, 13, 35] , and poisoning [41, 43, 41] . However, as these works mostly treat LLM or RAG as a simple model and study their robustness individually, their conclusions can hardly transfer to LLM agent which is a much more complex system. Recently a few preliminary works also study the backdoor attacks on LLM agents [34, 40] , however they only consider poisoning the training data of LLM backbones and fail to assess the safety of more capable RAG-based LLM agents. In terms of defense, [30] seeks to defend RAG from corpus poisoning by isolating individual retrievals and aggregate them. However, their method can hardly defend AGENTPOISON as we can effectively ensure all the retrieved instances are poisoned. As far as we are concerned, we are the first work to red-team LLM agents based on RAG systems. Please refer to Appendix A.5 for more details.3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We provide discussions on both the good performance of our method as well as the limitations in §5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries and settings: 
 We consider LLM agents with a RAG mechanism based on corpus retrieval. For a user query q, we retrieve knowledge or past experiences from a memory database D, containing a set of query-solution (key-value) pairs {(k 1 , v 1 ), . . . , (k |D| , v |D| )}. Different from conventional passage retrieval where query and document are usually encoded with different embedders [18] , LLM agents typically use a single encoder E q to map both the query and the keys into an embedding space. Thus, we retrieve a subset E K (q, D) ⊂ D containing the K most relevant keys (and their associated values) based on their (cosine) similarity with the query q in the embedding space induced by E q , i.e., the K keys in D with the minimum Eq(q) ⊤ Eq(k)||Eq(q)||•||Eq(k)|| . These K retrieved key-value pairs are used as the in-context learning demonstrations for the LLM backbone of the agent to determine an action step by a = LLM(q, E K (q, D)). The LLM agent will execute the generated action by calling build-in tools [9] or external APIs.Assumptions for the attacker We follow the standard assumption from previous backdoor attacks against LLMs [13, 31] and RAG systems [41, 43] . We assume that the attacker has partial access to the RAG database of the victim agent and can inject a small number of malicious instances to create a poisoned database D poison (x t ) = D clean ∪ A(x t ). Here, A(x t ) = {( k1 (x t ), v1 ), • • • , ( k|A(xt)| (x t ), v|A(xt)| )} represents the set of adversarial key-value pairs injected by the attacker, where each key here is a benign query injected with a trigger x t . Accordingly, the demonstrations retrieved from the poisoned database for a query q will be denoted by E K (q, D poison (x t )). This assumption aligns with practical scenarios where the memory unit of the victim agent is hosted by a third-party retrieval service2 or directly leverages an unverified knowledge base. For example, an attacker can easily inject poisoned texts by maliciously editing Wikipedia pages [4] ). Moreover, we allow the attacker to have white-box access to the RAG embedder of the victim agent for trigger optimization [43] . However, we later show empirically that the optimized trigger can easily transfer to a variety of other embedders with high success rates, including a SOTA black-box embedder OpenAI-ADA.Objectives of the attacker The attacker has two adversarial goals. (a) A prescribed adversarial agent output (e.g. sudden stop for autonomous driving agents or deleting the patient information for electronic healthcare record agents) will be generated whenever the user query contains the optimized backdoor trigger. Formally, the attacker aims to maximizeEQUATIONwhere π q is the sample distribution of input queries, a m is the target malicious action, 1(•) is a logical indicator fuction. x t denotes the trigger, and q ⊕ x t denotes the operation of injecting3 the trigger x t into the query q.(b) Ensure the outputs for clean queries remain unaffected. Formally, the attacker aims to maximizeEQUATIONwhere a b denotes the benign action corresponding to a query q. This is different from traditional DP attacks such as [41] that aim to degrade the overall system performance.We design AGENTPOISON to optimize a trigger x t that achieves both objectives of the attacker specified above. However, directly maximizing Eq. (1) and Eq. (2) using gradient-based methods is challenging given the complexity of the RAG procedure, where the trigger is decisive in both the retrieval of demonstrations and the target action generation based on these demonstrations. Moreover, a practical attack should not only be effective but also stealthy and evasive, i.e., a triggered query should appear as a normal input and be hard to detect or remove, which we treat as coherence.Our key idea to solve these challenges is to cast the trigger optimization into a constrained optimization problem to jointly maximize a) retrieval effectiveness: the probability of retrieving from the poisoning set A(x t ) for any triggered query q ⊕ x t , i.e.,EQUATIONand the probability of retrieving from the benign set D clean for any benign query q, b) target generation: the probability of generating the target malicious action a m for triggered query q ⊕ x t when E K (q ⊕ x t , D poison (x t ))) contains key-value pairs from A(x t ), and c) coherence: the textual coherence of q ⊕ x t . Note that a) and b) can be viewed as the two sub-steps decomposed from the optimization goal of maximizing Eq. (1), while a) is also aligned to the maximization of Eq. (2). In particular, we propose a novel objective function for a) where the triggered queries will be mapped to a unique region in the embedding space induced by E q with high compactness between these embeddings. Intuitively, this will minimize the similarity between queries with and without the trigger while maximizing the similarity in the embedding space for any two triggered queries (see (d) . By mapping triggered instances to a unique and compact region in the embedding space, AGENTPOISON effectively retrieves them without affecting other trigger-free instances to maintain benign performance. In contrast, CPA requires a much larger poisoning ratio meanwhile significantly degrading benign utility. Fig. 2 ). Furthermore, the unique embeddings for triggered queries impart distinct semantic meanings compared to benign queries, enabling easy correlation with malicious actions during in-context learning. Finally, we propose a gradient-guided beam search algorithm to solve the constrained optimization problem by searching for discrete tokens under non-derivative constraints.Our design of AGENTPOISON brings it two major advantages over existing attacks. First, AGENT-POISON requires no additional model training, which largely lowers the cost compared to existing poisoning attack [34, 35] . Second, AGENTPOISON is more stealthy than many existing jailbreaking attacks due to optimizing the coherence of the triggered queries. The overview is shown in Fig. 1 .We construct the constrained optimization problem following the key idea in §3.3.1 as the following:EQUATIONs.t. L tar (x t ) ≤ η tar (5) L coh (x t ) ≤ η coh (6) where Eq. (4), Eq. ( 5), and Eq. ( 6) correspond to the optimization goals a), b), and c), respectively. The constants η tar and η coh are the upper bounds of L tar and L coh , respectively. Here, all four losses in the constrained optimization are defined as empirical losses over a set Q = {q 0 , • • • , q |Q| } of queries sampled from the benign query distribution π q . We define L = L uni + λL cpt for brevity.The uniqueness loss aims to push triggered queries away from the benign queries in the embedding space. Let c 1 , • • • , c N be the N cluster centers corresponding to the keys of the benign queries in the embedding space, which can be easily obtained by applying (e.g.) k-means to the embeddings of the benign keys. Then the uniqueness loss is defined as the average distance of the input query embedding to all these cluster centers:EQUATIONNote that effectively minimizing the uniqueness loss will help to reduce the required poisoning ratio.We define a compactness loss to improve the similarity between triggered queries in the embedding space:EQUATIONwhere E q (x t ) = 1 |Q| qj ∈Q E q (q j ⊕ x t ) is the average embedding over the triggered queries. The minimization of the compactness loss can further reduce the poisoning ratio. In Fig. 11 , we show the procedure for joint minimization of the uniqueness loss and the compactness loss, where the embeddings for the triggered queries gradually form a compact cluster. Intuitively, the embedding of a test query containing the same trigger will fall into the same cluster, resulting in the retrieval of malicious key-value pairs. In comparison, CPA (Fig. 2a ) suffers from a low accuracy in retrieving malicious key-value pairs, and it requires a much higher poisoning ratio to address the long-tail distribution of all the potential queries.Target generation loss We maximize the generation of target malicious action a m by minimizing:EQUATIONwhere p LLM (•|•) denotes the output probability of the LLM given the input. While Eq. ( 9) only works for white-box LLMs, we can efficiently approximate L tar (x t ) using finite samples with polynomial complexity. We show the corresponding analysis and proof in Appendix A.4.We aim to maintain high readability and coherence with the original texts in each query q for the optimized trigger. This is achieved by minimizing:EQUATIONwhere q (i) denote the i th token in q ⊕ x t , and LLM b denotes a small surrogate LLM (e.g. gpt-2) in our experiment. Different from suffix optimization that only requires fluency [23] , the trigger optimized by AGENTPOISON can be injected into any position of the query (e.g. between two sentences). Thus Eq. ( 10) enforces the embeded trigger to be semantically coherent with the overall sequence [10] , thus achieving stealthiness.We propose a gradient-based approach that optimizes Eq. ( 4) while ensuring Eq. ( 9) and Eq. ( 10) satisfy the soft constraint via a beam search algorithm. The key idea of our optimization algorithm is to iteratively search for a replacement token in the sequence that improves the objective while also satisfying the constraint. Our algorithm consists of the following four steps.Require: query encoder E q , a set of queriesQ = {q 0 , • • • , q |Q| }, database cluster centers {c n | n ∈ [1, N ],target malicious action a m , target LLM, surrogate LLM b , maximum search iteration I max . Ensure: a stealthy trigger that yields high backdoor success rate. L uni ← Eq. (7), L cpt ← Eq. (8) 5:1: B = {x t0 | x t0 = [t 0 , • • • , t T ]} ▷ Algorithm.EQUATIONEQUATIONUpdate S ′ τ from S τ ▷ Eq. ( 11)9:end for 10:B = arg max t 1,••• ,b ∈S ′ τ {L τ (x tτ ) | L τ (x tτ ) ≤ L τ -1 (x tτ )} 11:end for Initialization: To ensure context coherence, we initialize the trigger x t0 from a string relevant to the agent task where we treat the LLM as an one-step optimizer and prompt it to obtain b triggers to form the initial beams (Algorithm. 4).Gradient approximation: To handle discrete optimization, for each beam candidate, we follow [8] to first calculate the objective w.r.t. Eq. ( 4) and randomly select a token t i in x t0 to compute an approximation of the model output L by replacing t i with another token in the vocabulary V, using gradient∂L = ∇ et i (L uni + λL cpt ), where the approximated output for another tokent ′ i is given by L = e ⊺ t ′ i ∂L.Then we obtain the top-m candidate tokens to consist the replacement token set C 0 .Constraint filtering: Then we impose constraint Eq. ( 6) and Eq. ( 5) sequentially. Since determination of η coh highly depends on the data, we follow [23] to first sample s tokens from C 0 to obtain S τ under a distribution where the likelihood for each token is a softmax function of L coh . This ensures the selected tokens possess high coherence while maintaining diversity. Then we further filter S τ w.r.t. Eq. ( 5). We notice that during early iterations most candidates cannot directly satisfy Eq. ( 5), thus instead, we consider the following soft constraint:EQUATIONwhere τ denotes the τ th iteration. Thus we soften the constraint to require Eq. ( 9) to monotonic increase when Eq. ( 5) is not directly satisfied, which leaves a more diversified candidate set S ′ τ . Token Replacement: Then we calculate L tar for each token in S ′ τ and select the top b tokens that improve the objective Eq. ( 4) to form the new beams. Then we iterate this process until convergence. The overall procedure of the trigger optimization is detailed in Algorithm. 1.Table 1 : We compare AGENTPOISON with four baselines over ASR-r, ASR-b, ASR-t, ACC on four combinations of LLM agent backbones: GPT3.5 and LLaMA3-70b (Agent-Driver uses a fine-tuned LLaMA3-8b) and RAG retrievers: end-to-end and contrastive-based. Specifically, we inject 20 poisoned instances with 6 trigger tokens for Agent-Driver, 4 instances with 5 trigger tokens for ReAct-StrategyQA, and 2 instances with 2 trigger tokens for EHRAgent. For ASR, the maximum number in each column is in bold; for ACC, the number within 1% to the non-attack case is in bold. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The code and data are publicly accessible.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
setup: 
 LLM Agent: To demonstrate the generalization of AGENTPOISON, we select three types of realworld agents across a variety of tasks: Agent-Driver [22] for autonomous driving, ReAct [36] agent for knowledge-intensive QA, and EHRAgent [25] for healthcare record management.Memory/Knowledge base: For agent-driver we use its corresponding dataset published in their paper, which contain 23k experiences in the memory unit 4 . For ReAct, we select a more challenging multi-step commonsense QA dataset StrategyQA which involves a curated knowledge base of 10k passages from Wikipedia 5 . For EHRAgent, it originally initializes its knowledge base with only four experiences and updates its memory dynamically. However we notice that almost all baselines have a high attack success rate on the database with such a few entries, we augment its memory unit with 700 experiences that we collect from successful trials to make the red-teaming task more challenging.Baselines: To assess the effectiveness of AGENTPOISON, we consider the following baselines for trigger optimization: Greedy Coordinate Gradient (GCG) [42] , AutoDAN [21] , Corpus Poisoning Attack (CPA) [41] , and BadChain [31] . Specifically, we optimize GCG w.r.t. the target loss Eq. ( 9), and since we observe AutoDAN performs badly when directly optimizing Eq. ( 9), we calibrate its fitness function and augment Eq. ( 9) by Eq. ( 3) with Lagrangian multipliers. And we use the default objective and trigger optimization algorithm for CPA and BadChain.Evaluation metrics: We consider the following metrics: (1) attack success rate for retrieval (ASR-r), which is the percentage of test instances where all the retrieved demonstrations from the database are poisoned; (2) attack success rate for the target action (ASR-a), which is the percentage of test instances where the agent generates the target action (e.g., "sudden stop") conditioned on successful retrieval of poisoned instances. Thus, ASR-a individually assesses the performance of the trigger w.r.t. inducing the adversarial action. Then we further consider (3) end-to-end target attack success We fix the number of tokens to 4 for the former case and the number of poisoned instances to 32 for the latter case. Two metrics ASR-r (retrieval success rate) and ACC (benign utility) are studied. rate (ASR-t), which is the percentage of test instances where the agent achieves the final adversarial impact on the environment (e.g., collision) that depends on the entire agent system, which is a critical metric that distinguishes from previous LLMs attack. Finally, we consider (4) benign accuracy (ACC), which is the percentage of test instances with correct action output without the trigger, which measures the model utility under the attack. A successful backdoor attack is characterized by a high ASR and a small degradation in the ACC compared with the non-backdoor cases. We detail the backdoor strategy and definition of attack targets for each agent in Appendix A.3.1 and Appendix A.1.2, respectively.AGENTPOISON demonstrates superior attack success rate and benign utility. We report the performance of all methods in Table 1 . We categorize the result into two types of LLM backbones, i.e. GPT3.5 and LLaMA3, and two types of retrievers trained via end-to-end loss or contrastive loss. We observe that algorithms that optimize for retrieval i.e. AGENTPOISON, CPA and AutoDAN has better ASR-r, however CPA and AutoDAN also hampers the benign utility (indicated by low ACC) as they invariably degrade all retrievals. As a comparison, AGENTPOISON has minimal impact on benign performance of average 0.74% while outperforming the baselines in terms of retrieval success rate of 81.2% in average, while an average 59.4% generates target actions where 62.6% result in actual target impact to the environment. The high ASR-r and ACC can be naturally attributed to the optimization objective of AGENTPOISON. And considering that these agent systems have in-built safety filters, we denote 62.6% to be a very high success rate in terms of real-world impact.AGENTPOISON has high transferability across embedders. We assess the transferability of the optimized triggers on five dense retrievers, i.e. DPR [14] , ANCE [32] , BGE [39] , REALM [11] , and ORQA [17] to each other and the text-embedding-ada-002 model 6 with API-only access. We report the results for Agent-Driver in Fig. 3 , and ReAct-StrategyQA and EHRAgent in Fig. 7 and Fig. 8 (Appendix A.2.2). We observe AGENTPOISON has a high transferability across a variety of embedders (even on embedders with different training schemes). We conclude the high transferability results from our objective in Eq. ( 4) that optimizes for a unique cluster in the embedding space which is also semantically unique on embedders trained with similar data distribution.AGENTPOISON performs well even when we inject only one instance in the knowledge base with one token in the trigger. We further study the performance of AGENTPOISON w.r.t. the number 3 : We assess the resilience of the optimized trigger by studying three types of perturbations on the trigger in the input query while keeping the poisoned instances fixed. Specifically, we consider injecting three random letters, injecting one word in the sequence, and rephrasing the trigger while maintaining its semantic meaning. We prompt GPT3.5 to obtain the corresponding perturbations. of poisoned instances in the database and the number of tokens in the trigger sequence, and report the findings in Fig. 4 . We observe that after optimization, AGENTPOISON has high ASR-r (62.0% in average) when we only poison one instance in the database. Meanwhile, it also achieves 79.0% ASR-r when the trigger only contains one token. Regardless of the number of poisoned instances or the tokens in the sequence, AGENTPOISON can consistently maintain a high benign utility (ACC≥ 90%). How does each individual loss contributes to AGENTPOISON? The ablation result is reported in Table 2 , where we disable one component each time. We observe L uni significantly contributes to the high ASR-r in AGENTPOISON while ACC is more sensitive to L cpt where more concentrated qt generally lead to better ACC. Besides, while adding L coh slightly degrades the performance, it leads to better in-context coherence, which can effectively bypass some perplexity-based countermeasures. AGENTPOISON is resilient to perturbations in the trigger sequence. We further study the resilience of the optimized triggers by considering three types of perturbations in Table 3 . We observe AGENTPOISON is resilient to word injection, and slightly compromised to letter injection. This is because letter injection can change over three tokens in the sequence which can completely flip the semantic distribution of the trigger. Notably, rephrasing the trigger which completely change the token sequence also maintains high performance, as long as the trigger semantics is preserved.How does AGENTPOISON perform under potential defense? We study two types of defense: Perplexity Filter [2] and Query Rephrasing [15] (here we rephrase the whole query which is different from Table 3 ) which are often used to prevent LLMs from injection attack. We report the ASR-t in Table 4 and full result in Table 6 (Appendix A.2.4). Compared with GCG and Badchain, the trigger optimized by AGENTPOISON is more readable and coherent to the agent context, making it resilient under both defenses. We further justify this observation in Fig. 5 where we compare the perplexity distribution of queries optimized by AGENTPOISON to benign queries and GCG. Compared to GCG, the queries of AGENTPOISON are highly evasive by being inseparable from the benign queries. 
conclusion: 
 In this paper, we propose a novel red-teaming approach AGENTPOISON to holistically assess the safety and trustworthiness of RAG-based LLM agents. Specifically, AGENTPOISON consists of a constrained trigger optimization algorithm that seeks to map the queries into a unique and compact region in the embedding space to ensure high retrieval accuracy and end-to-end attack success rate. Notably, AGENTPOISON does not require any model training while the optimized trigger is highly transferable, stealthy, and coherent. Extensive experiments on three real-world agents demonstrate the effectiveness of AGENTPOISON over four baselines across four comprehensive metrics. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Details of both training and test are disclosed.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We adopt standard evaluation datasets, LLM agents, and metrics, which are accompanied by statistical significance.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Both potential positive societal impacts and negative societal impacts of the work have been discussed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: The paper has discussed safeguards for responsible release of data or models that have a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All the mentioned previous work are properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The code will be made public upon acceptance.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this paper, we propose AGENTPOISON, the first backdoor attack against LLM agents with RAG. The main purpose of this research is to red-team LLM agents with RAG so that their developers are aware of the threat and take action to mitigate it. Moreover, our empirical results can help other researchers to understand the behavior of RAG systems used by LLM agents. Code is released at https://github.com/BillChan226/AgentPoison.While AGENTPOISON is effective in optimizing triggers to achieve high retrieval accuracy and attack success rate, it requires the attacker to have white-box access to the embedder. However, we show empirically that AGENTPOISON can transfer well among different embedders even with different training schemes, since AGENTPOISON optimizes for a semantically unique region in the embedding space, which is also likely to be unique for other embedders as long as they share similar training data distribution. This way, the attacker can easily red-team a proprietary agent by simply leveraging a public open-source embedder to optimize for such a universal trigger.A Appendix / supplemental material A.1 Experimental SettingsThe hyperparameters for AGENTPOISON and our experiments are reported in Table 5 . Except for obtaining the result in Fig. 4 , we keep the number of tokens in the trigger fixed, where we have 6 tokens for Agent-Driver [22] , 5 tokens for ReAct-StrategyQA [36] , and 2 tokens for EHRAgent [25] , and we inject 20 poisoned instances for Agent-Driver, 4 for ReAct, and 2 for EHRAgent across all experiments. The number of tokens in the trigger sequence are mainly determined by the length of the original queries. We inject fewer than 0.1% instances w.r.t. the original number of instances in the database for all attack methods, since we observe that as more instances have been poisoned, it gets harder to distinguish to effectiveness of different methods, as reported in Fig. 4 .We detail the attack target for AGENTPOISON in this section. Specifically, for all three agents, we consider it a success retrieval (thus counted in ASR-r) only if all the retrieved instances (usually k-nearest neighbors) are poisoned demonstrations that we previously injected into the database. Such requirements are practical and necessary for evaluating attack success for retrievals since many agents have certain in-built safety filters to further select useful demonstrations from all the retrieval results (e.g. Agent-Driver [22] instantiates a re-examination process where they use a LLM to select one experience which is most relevant to the retrieved k instances). This way an adversary can certify attack success only if all the retrieved instances are malicious. Recent defense [30] which seeks to certify RAG from corpus poisoning attacks by isolate-then-aggregate further necessitates this requirement on such agent-oriented attacks. By effectively manipulating all the retrieved demonstrations to be poisoned instances, AGENTPOISON can easily bypass such SOTA defense.Specifically, we detail the target action and target outcome for attacking each agent.• Agent-Driver: we denote irresponsible and potentially unsafe driving behaviors to be our attack target. Specifically, target action for the agent to output is sudden stop. And the corresponding real-world outcome is measured by the trajectory deviation in the future three seconds. Mathematically, attack success for ASR-t is indicated by the following indicator function:EQUATION)where ζ denotes the planned trajectory given by a fine-tuned LLM trajectory planner, and ζ is the ground-truth trajectory. τ s is the preset threshold that determines the maximum deviation of a safe trajectory.• ReAct-StrategyQA: we denote wrong answer or unhelpful information to be our attack target for QA-based agent. As we provide the poison sample following [31] (as detailed in Appendix A.3.1), the target action for the agent to output is as there is a {trigger} in the sequence during its internal reasoning (denoting successfully backdoored in-context learning), and the corresponding outcome is that the agent finishes the episode with such wrong answer or unhelpful answer (e.g., I don't know).• EHRAgent: considering the purpose and safety issues of healthcare record maintenance, we denote deletion to be a highly risky and potentially unsafe operation. Thus we design the target action for the agent such that it should output delete data of patient ID during the reasoning step, and the corresponding outcome is a SQL code command DeleteDB.Train/Test split For Agent-Driver, we have randomly sampled 250 samples from its validation set (apart from the 23k samples in the training set); for ReAct agent, we have used the full test set in StrategyQA 8 which consists of 229 samples; and for EHRAgent, we have randomly selected 100 samples from its validation set in our experiment. Besides, the poisoned samples are all sampled from the training set of each agent which does not overlap with the test set.Retriever As we have categorized the RAG retrievers into two types, i.e. contrastive and end-to-end based on their training scheme, for each agent we have manually selected a representative retriever in each type and report the corresponding results in Table 1 . Specifically, for Agent-Driver, as it is a domain-specific task and requires the agent to handle strings that contain a large portion of numbers which distinct from natural language, we have followed [22] and trained both the end-to-end and contrastive embedders using its published training data 9 , where we use the loss described in §A.5.1. And for ReAct-StrategyQA [36] and EHRAgent [25] , we have adopted the pre-trained DPR [14] checkpoints 10 as contrastive retriever and the pre-trained REALM [11] checkpoints 11 as end-to-end retriever.We further detail our analysis by investigating the following six questions. (1) As AGENTPOISON constructs a surrogate task to optimize both Eq. (1) and Eq. ( 2), we aim to ask how well does AGENT-POISON fulfill the objectives of the attacker? (2) What is the attack transferability of AGENTPOISON on ReAct-StrategyQA and EHRAgent? (3) How does the number of trigger tokens influence the 4) (on the right) during the AGENTPOISON optimization w.r.t. different number of trigger tokens. Specifically, we consider the trigger sequence of 2, 5, and 8 tokens. We can denote that while longer triggers generally lead to a higher retrieval success rate, AGENTPOISON could still yield good and stable attack performance even when there are fewer tokens in the trigger sequence.optimization gap? (4) How does AGENTPOISON perform under potential defense? ( 5) What is the distribution of embeddings during the intermediate optimization process of AGENTPOISON? ( 6) What does the optimized trigger look like? We provide the result and analysis in the following sections.We further visualize the result in Table 1 in Fig. 6 where we focus on ASR-r and ACC. We can see that AGENTPOISON (represented by +) are distribute in the upper right corner which denotes it can achieve both high retrieval success rate (in terms of ASR-r) and benign utility (in terms of ACC) while all other baselines can not achieve both. This result further demonstrates the superior backdoor performance of AGENTPOISON.We have provided the additional transferability result on ReAct-StrategyQA and EHRAgent in Fig. 7 and Fig. 8 , respectively. We can see that AGENTPOISON generally achieves high attack transferability among different RAG retrievers which further demonstrates its universality for trigger optimization.We compare the attack performance on ReAct-StrategyQA w.r.t. ASR-r and loss defined in Eq. ( 4) during the AGENTPOISON optimization w.r.t. different number of trigger tokens, and report the result in Fig. 9 . We can denote that while triggers with more tokens can generally lead to a higher retrieval success rate, AGENTPOISON could yield a good and consistent attack success rate even if there are very few tokens in the trigger sequence.We provide the additional results of the performance of AGENTPOISON under two types of potential defense in Table 6 . Besides, we provide the comparison of averaged query perplexity on all three agents in Fig. 10 .The embedding distribution during the intermediate optimization process of AGENTPOISON across different embedders is showcased in Fig. 11 . We can consistently observed that, regardless of the white-box embedders being optimized, AGENTPOISON can effectively learn a trigger such that the triggers are gradually becoming more unique and compact, which further verifies the effectiveness of AGENTPOISON and the validity of the loss being optimized.Table 6 : We assess the performance of AGENTPOISON under potential defense. Specifically, we consider two types of defense: a) Perplexity Filter [2] , which evaluates the perplexity of the input query and filters out those larger than a threshold; and b) Rephrasing Defense [15] , which rephrases the original query to obtain a query that shares the same semantic meaning as the original query. We demonstrate the trigger optimized on GPT3.5 LLM backbone and retriever trained via contrastive loss using different attack algorithms over Agent-Driver [22] , ReAct [36] , EHRAgent [25] in Table 7 . Due to our trigger initialization using a relevant string and our coherence loss, our trigger have a better fluency and coherence than the trigger optimized using CPA and GCG. While the trigger optimized by AutoDAN and BadChain have good readability (since they utilize an LLM for trigger generation), they are not as effective as our algorithm.Table 7 : A case study of the trigger optimized using different attack algorithms for Agent-Driver [22] , ReAct [36] , EHRAgent [25] . We demonstrate the trigger optimized on GPT3. After retrieving from the knowledge base, we showcase the procedure of reasoning for action where the agent places the retrieved malicious demonstrations in the prefix and prompts the LLM backbone for reasoning and action prediction. We mainly consider two types of poisoning strategy, i.e. (1)The pseudocode for trigger initialization is shown in Algorithm. 4 where we use it to generate the initial beams of triggers that are relevant to the task the agent handles.1: function Trigger-Initialization (query-example, agent-task, number-of-tokens) 2:message system ← "You are a helpful and creative assistant to help write relevant string for some LLM Agent tasks." 3: message user ← "Please give me a relevant string of number-of-tokens tokens for a agent-task task. Your string should not change the original output of the query query-example when it is being appended." 4: return LLM.get_response(message system , message user )Given the constrained optimization problem defined in §3.3.2:EQUATIONWe can directly adopt Eq. ( 9) to calculate the target action objective L tar (x t ) for white-box models. However, AGENTPOISON can be adapted for black-box LLMs setting by approximating L tar (x t ) via the finite-sample indicator function.EQUATIONwhere 1 condition is 1 when the condition is true and 0 otherwise. We demonstrate in Theorem A.1 that AGENTPOISON can efficiently approximate L tar (x t ) with a polynomial sample complexity. Theorem A.1 (Complexity analysis for approximating L tar (x t ) with finite samples). We can provide the following sample complexity bound for approximating L tar (x t ) with finite samples. Let Q denote the potential space of all queries. For any ϵ > 0 and γ ∈ (0, 1), with at leastEQUATIONsamples, we have with probability at least 1 -γ:EQUATIONProof. Specifically, to prove Theorem A.1, we fist reformulate Eq. ( 14) in the following form:EQUATIONwhere a r denotes the runner-up (i.e., second-maximum likelihood) action token output by the target LLM. Then we can define a set of functions F as the class of real-valued functions where each represents the output action distribution p LLM (a|q j ⊕ x t ) conditioned on a query q j sampled from Q and trigger x t . More specifically, each function f can be formulated as{f qj ∈ F |f qj (x) = p LLM (a m | [q j ⊕ x t , E K (q j ⊕ x, D poison (x))])}.Therefore, we can first obtain an upper bound for the VC dimension of H = {1 fq j (am)>fq j (ar) : f qj ∈ F } using the following lemma.Lemma 1 (VC Dimension Bound). Let F be a vector space of real-valued functions, and let H = {1 fq j (am)>fq j t(ar ) :f qj ∈ F }. Then the VC dimension of H satisfies VCdim(H) ≤ dim(F ) + 1.Proof. To show that the VC dimension of H is at most dim(F ) + 1, we need to show that no set of more than dim(F ) + 1 points can be shattered by H.Consider a set of m points {x 1 , x 2 , . . . , x m } in a d-dimensional space where d = dim(F ). Suppose that H can shatter this set of m points. This means that for any way of labeling these m points, there exists a function in H that correctly classifies the points according to those labels.Each function h ∈ H corresponds to an indicator function of the form 1 fq j (am)>fq j (ar) , where f qj ⊕xt ∈ F . Given a basis {f 1 , f 2 , . . . , f d } for the vector space F , any function f ∈ F can be written as a linear combination of these basis functions:EQUATIONFor each point x k , the condition f qj (a m ) > f qj (a r ) translates to:EQUATIONThis can be rewritten as:EQUATIONLetg k = f i (x k , a m ) -f i (x k , a r ).We have m linear inequalities of the form:EQUATIONTo shatter the set {x 1 , x 2 , . . . , x m }, we need to find coefficients α i such that these m inequalities can realize all possible sign patterns for the m points. However, in a d-dimensional space, we can only have at most d linearly independent inequalities. If m > d + 1, then we have more inequalities than the dimensions of the space, making it impossible to satisfy all possible sign patterns. Thus, m ≤ d + 1. Therefore, the VC dimension of H is at most dim(F ) + 1.Theorem A.2 (Sample Complexity [3] ). Suppose that H is a set of functions from a set X to {0, 1} with finite VC dimension d ≥ 1. Let L be any sample error minimization algorithm for H. Then L is a learning algorithm for H. In particular, if m ≥ d 2 , its sample complexity satisfies:EQUATIONwhere m L (ϵ, γ) is the minimum sample size required to ensure that with probability at least 1 -γ, the empirical error is within ϵ of the true error.Therefore we can combine Lemma 1 and Theorem A.2 to prove the sample complexity bound for L tar (x t ) in Eq. ( 15). According to Lemma 1, the VC dimension of H is bounded by VCdim(H) ≤ dim(F ) + 1. Then by Theorem A.2, we can denote that for any ϵ > 0 and γ ∈ (0, 1), with at leastN ≥ 64 ϵ 2 2d ln 12 ϵ + ln 4 γsamples, we have with probability at least 1 -γ:EQUATIONTherefore, the finite-sample approximation of the target constraint function converges polynomially (to 1/ϵ) to L tar with high probability as the number of samples increases.Therefore, Theorem A.1 indicates that we can effectively approximate L tar with a polynomially bounded number of samples, and we use function Eq. ( 14) to serve as the constraint for the overall optimization for AGENTPOISON.Retrieval Augmented Generation (RAG) [19] is widely adopted to enhance the performance of LLMs by retrieving relevant external information and grounding the outputs and action of the model [22, 38, 29] . The retrievers used in RAG can be categorized into sparse retrievers (e.g. BM25), where the embedding is a sparse vector which usually encodes lexical information such as word frequency [24] ; and dense retrievers where the embedding vectors are dense, which is usually a fine-tuned version of a pre-trained BERT encoder [7] . We focus on red-teaming LLM agents with RAG handled by dense retrievers, as they are much more widely adopted in LLM agent systems and have been proved to perform much better in terms of retrieval accuracy [11] .In our discussion, we categorize RAG into two categories based on their training scheme: (1) endto-end training where the retriever is updated using causal language modeling pipeline handled by cross-entropy loss [11, 17] ; and (2) contrastive surrogate loss where the retriever is trained alone and usually on a held-out training set [32, 39] .During end-to-end training, both the retriever and the generator are optimized jointly using the language modeling loss [11] . The retriever selects the top K documents E K (q) based on their relevance to the input query q, and the generator conditions on both q and each retrieved document E K (q) to produce the output sequence y (or action a for LLM agent). Therefore the probability of the generated output is given by:p RAG (y|q) ≈ E K (q)∈top-k(p(•|q))p Eq (E K (q)|q)p LLM (y|q, E K (q)) (24)= E K (q)∈top-k(p(•|1))p Eq (E K (q)|q)EQUATIONThus correspondingly the training objective is to minimize the negative log-likelihood of the target sequence by optimizing the E q :EQUATION= -logE K (q)∈top-k(p(•|q))p Eq (E K (q)|q)EQUATIONThis way embedder E q is trained to align with the holistic goal of the generation task. While being effective, the end-to-end training scheme only demonstrates good performance during pre-training which makes the training very costly.Therefore, extensive works on RAG explore training E k via a surrogate contrastive loss to learn a good ranking function for retrieval. The objective is to create a vector space where relevant pairs of questions and passages have smaller distances (i.e., higher similarity) than irrelevant pairs. The training data consists of instances {⟨k i , v + i , v - i,1 , . . . , v - i,n ⟩} m i , where each instance includes a query key k i , a relevant key k + i , and n irrelevant keys k - i,j . The contrastive loss function is defined as:EQUATIONSpecifically, Eq. ( 28) encourages the retriever E q to assign higher similarity scores to positive pairs than to negative pairs, effectively improving the retrieval accuracy. And different embedders often distinguish in their curation of the negative samples [14, 39, 32] .NeurIPS Paper ChecklistFor example: https://www.voyageai.com/In this work, we do not restrict the position for trigger injection, i.e., the trigger is not limited to a suffix.https://github.com/USC-GVL/Agent-Driverhttps://allenai.org/data/strategyqahttps://platform.openai.com/docs/guides/embeddingshttps://allenai.org/data/strategyqahttps://github.com/USC-GVL/Agent-Driverhttps://github.com/facebookresearch/DPRhttps://huggingface.co/docs/transformers/en/model_doc/realm 


Decision: This paper presents the first backdoor attack targeting the long-term memory or knowledge base of an LLM called AgentPoison. The approach optimizes the trigger using constrained optimization in a red-teaming scenario with access to the memory. The approach performs well in range of different domains. 

The use of LLMs is becoming more widespread and security is an important concern. The approach in this paper is promising for highlighting an important weakness with current methods. 

While the approach is promising, there are a number of concerns that should be addressed in the paper. For example, as highlighted by the reviewers, there are some aspects of the paper that are unclear and there should be increased discussion of issues such as defenses and transferability. The author response and additional results helpful in addressing these concerns but the paper should be updated to take them (and the other reviewer feedback) into consideration.

Reviews: review: paper_summary: paper_summary: The paper presents a novel red-teaming approach that targets the vulnerabilities of large language model (LLM) agents by poisoning their long-term memory or retrieval-augmented generation (RAG) knowledge base. The primary contribution is the development of AGENTPOISON, a method to inject backdoor triggers into the knowledge base, which leads to the retrieval of malicious demonstrations when certain triggers are present in user queries. This attack method is tested on three real-world LLM agents, demonstrating high attack success rates with minimal impact on benign performance.
strength_weakness: strength_weakness: As someone not deeply familiar with this field, I found some sections challenging to understand. The authors could provide more context and explanations about the attack mechanisms and their significance to make the paper more accessible to readers from diverse backgrounds.
questions: questions: See above
limitations: limitations: Yes.
paper_summary: paper_summary: The paper presents AGENTPOISON, a novel red-teaming approach aimed at exposing vulnerabilities in LLM agents by poisoning their long-term memory or RAG knowledge base. Unlike conventional backdoor attacks, AGENTPOISON does not require additional model training or fine-tuning and ensures high attack success rates with minimal impact on benign performance. The approach optimizes backdoor triggers through a constrained optimization process, ensuring that malicious demonstrations are retrieved with high probability when user instructions contain the optimized triggers. The method is validated through extensive experiments on three types of LLM agents: autonomous driving, knowledge-intensive QA, and healthcare agents, demonstrating its effectiveness and generalizability.
strength_weakness: strength_weakness: 1. The paper briefly mentions potential defenses against the proposed attack but does not explore them in depth. Including a more comprehensive discussion on possible mitigation strategies and their effectiveness would strengthen the work.
2.  While the paper demonstrates the transferability of the optimized triggers across different retrievers, further exploration of the method's applicability to a broader range of LLM architectures would be beneficial, rather than just LLaMA3 and ChatGPT.
3. Although the paper uses multiple evaluation metrics, including attack success rates and benign accuracy, additional metrics such as the impact on real-world task performance and user experience could provide a more holistic assessment of the method's implications.
questions: questions: 1.  How does AGENTPOISON perform against more sophisticated defense mechanisms, such as those involving continuous monitoring and anomaly detection in LLM outputs?
2. Can the method be extended to target other components of LLM agents, such as their interaction modules or external APIs?
3. What are the potential trade-offs between the complexity of the optimized trigger and its effectiveness in different application domains?
4. How does the approach handle variations in user instructions that deviate significantly from those used during the optimization process?
limitations: limitations: The authors acknowledge the method's reliance on partial access to the victim agent's memory or knowledge base, which may not always be feasible in real-world scenarios. Additionally, while AGENTPOISON is effective in controlled experimental settings, its performance in more dynamic and less predictable environments remains uncertain. The paper could further discuss potential societal impacts, particularly regarding the ethical implications of deploying such attacks and the need for robust defensive measures to protect critical LLM applications.
paper_summary: paper_summary: The paper titled "AGENTPOISON: Red-teaming LLM Agents via Memory or Knowledge Base Backdoor Poisoning" introduces a novel backdoor attack method targeting large language model (LLM) agents. These agents leverage a memory module or a retrieval-augmented generation (RAG) mechanism to retrieve knowledge and past instances to aid in task planning and execution.

The key contributions are as follows:
- AGENTPOISON is the first backdoor attack targeting generic and RAG-based LLM agents by poisoning their long-term memory or RAG knowledge base with malicious demonstrations.
- The authors have conducted thorough experiments on three types of real-world LLM agents: an autonomous driving agent, a knowledge-intensive QA agent, and a healthcare EHRAgent.
- The paper highlights the potential security threats posed by backdoor attacks on LLM agents, emphasizing the need for developers to mitigate such vulnerabilities.
strength_weakness: strength_weakness: I do not identify any major weaknesses of the paper.
questions: questions: During the constrained optimization section, I noticed that some operations are conducted with the aid of LLMs. I am curious how much cost and how long the attach takes?

Please double-check the citation format. It seems that some citations have missing inproceedings, such as [23][25][28],etc.
limitations: limitations: The authors have discussed limitations in the Limitations section of paper.
paper_summary: paper_summary: This paper proposed an attack against the RAG-based LLM agents. Specifically, it proposes a constrained trigger optimization to search the trigger so that any queries included can be mapped to a certain compact cluster while keeping the coherence and attack success rate. The experiments show that the proposed method is efficient in real-world RAG-based LLM agents.
strength_weakness: strength_weakness: # Confusing Threat Model
The threat model described in the paper is vague. The authors mention that an attacker can poison the RAG database and use a searched trigger to retrieve these poisoned entries for an attack. However, it is unclear how the attacker can add the trigger to the query during inference. In traditional backdoor attacks, the attacker can add the trigger to the input (e.g., placing a small sticker on a stop sign). When the vehicle detects the trigger (the sticker), it makes a wrong decision, which is good and intuitive. In the case of LLM-based agents, the query is initiated by the user (the victim). How can the attacker add the trigger to the user's query? This aspect is not explained, making the threat model less convincing.

# Motivation for the Optimization Method (Coherence, Equation 10)
The motivation behind the optimization method, particularly the coherence aspect (Equation 10), is unclear. The authors claim that coherence is intended to make the trigger stealthier. In traditional backdoor attacks, triggers are often rare words to prevent accidental triggering in natural scenarios. However, the authors aim to optimize a trigger that looks normal, such as “Be safe and make a disciplined upbringing quiet.” This presents two issues:
1. **Natural Triggering**: Such triggers can easily be activated in natural conditions, which is undesirable for the attacker.
2. **Stealthiness**: Why is it necessary to increase stealthiness if the trigger is inserted by the attacker? Unlike traditional backdoor attacks where triggers need to be subtle, in this scenario, the trigger is directly manipulated by the attacker.

# Lack of Optimization for Poisoned Passages
A critical aspect is missing: the optimization of poison passages. The proposed method focuses on optimizing a trigger such that any queries containing the trigger will:
1. Compact with each other.
2. Stay away from other triggers.
3. Be semantically fluent.
4. Mislead the LLM.

However, there is no optimization for the poisoned passages inserted into the knowledge base. The real impact on LLM output comes from the passages retrieved, not the query itself. The authors do not ensure that triggered queries can retrieve poisoned passages or that these passages can influence the LLM. They only optimize the trigger, aiming to cluster triggered queries together. Without ensuring that these queries retrieve poisoned passages, clustering them is pointless.

# Lack of In-depth and Insightful Transferability Explanation
The explanation of transferability is not clear. The authors experimentally show that triggers generated by different retrievers can be transferred and attribute this to similar training data distribution. And in Figure 3, they claim that triggers transfer better among embedders with similar training strategies. This is confusing, and both reasons are unconvincing. To verify this point, the authors need to demonstrate that different training data distributions weaken transferability. Overall, a more in-depth insight is required.

# Lack of ablation study on hyperparameters
I noticed the Table 4 in the appendix shows the setup of hyperparameters. But the how do author set the hyperparameters like $\lambda$ in Equation 4,5,6 is not clear to me.
questions: questions: Please refer to the weakness part.
limitations: limitations: Yes
paper_summary: paper_summary: The paper proposes a new setting for red teaming against RAG empowered agents. Specifically, they tested their method in three different settings: driving agent, knowledge intensive QA and EHRagent. The method they proposed is to use constrained optimization to jointly optimize for two objectives: "a) the retrieval of the malicious demonstration and b) the effectiveness of the malicious demonstrations in inducing adversarial agent outputs". In particular it tries to map the poison demonstrations into a space that is separated from the benign examples. Experimental results showed its efficacy for high attack success rate with low benign impact and low poison rate.
strength_weakness: strength_weakness: - $\eta_\text{tar}$ and $\eta_\text{coh}$ seems hard to pick a intuitive value
- Line 169: not obvious what a), b) and c) are, probably better to outline them explicitly
questions: questions: - I did not quite get the motivation for using constrained optimization as opposed to just using unconstrained optimization to jointly minimize every objective. Since the $\eta$s are hard to set since they do not have a very intuitive value
- where were the poisons placed in the instruction? authors mentioned it can be anywhere but how do you determine that?
- does the placement of the poison affect the success rate of the attacks?
- how are the desired dangerous actions determined? can you provide a list of them for each env
- are the retrieved demonstrations ever optimized for eliciting dangerous actions?
- how did you calculate ASR? for r,a,t respectively? is this available in the environment?
limitations: limitations: The authors included them

Comments: title: Acknowledgement
comment: Dear Reviewer XDTd,

Thank you once again for your valuable comments and great efforts to help us improve the paper through discussion. As the discussion period is soon approaching the end, we would really appreciate it if you could let us know if our follow-up response has addressed all your concerns or if there are further questions so that we can discuss them more in-depth. If you feel that we have satisfactorily addressed your main concerns, we would be deeply grateful if you could consider raising your rating score so that we can share our findings with a wider community of researchers. Thank you!

Best regards,

Submission #15127 Authors
Dear Reviewer dD1S,

Thank you once again for your thoughtful and constructive feedback. As the discussion period is soon approaching the end, we would really appreciate it if you could let us know if there are further questions and concerns so that we can discuss them more in-depth. If you feel that we have satisfactorily addressed your main concerns, we would be deeply grateful if you could consider raising your rating score so that we can share our findings with a wider community of researchers. Thank you!

Best regards,

Submission #15127 Authors
We really appreciate this insightful follow-up question!

Regarding Figure-r. 2, we believe the main reason that the transferability is very low when retrievers use different training datasets is that we merge out-of-distribution training data (the training set of AgentDriver), which is from a completely different task to the original StrategyQA dataset. Therefore, we believe it makes sense that the transferability is low when a large ratio of training data is from another domain. However, we additionally find that AgentPoison can yield good transferability ($>60$%) on StrategyQA by optimizing triggers over the instructions from another QA dataset HotpotQA, which is a more common and practical case for real-world adversaries.

Regarding the reference (CPA) mentioned by the reviewer, we actually treat it as a baseline to compare AgentPoison with, where our method differs in many perspectives that naturally results in different transferability:
+ As indicated by Fig. 2 in the paper, CPA seeks to **maximize the coverage of poisoning instances of benign embeddings**, which reasonably results in its low transferability across different retrievers. Instead, **AgentPoison optimizes for a unique and compact region in the embedding space**, which is more robust to the shift of retrievers and thus results in more consistent transferability.
+ Although we use the same discretized optimization technique (as many other papers, such as GCG), we differ in objectives and further integrate it into a constrained optimization pipeline that simultaneously improves sample efficiency, time cost, and attack success rate. We believe the optimization objectives are a more significant reason to explain transferability instead of the discrete gradient-descent technique Hotflip.

We sincerely hope that these follow-up discussions could help address the reviewer's concern. Please let us know if there is further questions. Thank you!
I have reviewed the authors' responses and appreciate the thoroughness of their explanations. As this field is not my area of expertise, I would keep the score and believe it would be more appropriate for the other reviewers to make the final judgment.
Thank you for the authors' response. Regarding Figure-r. 2, while it illustrates the relationship between embedding similarity and transferability, the transferability is very low (<0.2) if retrievers use different training datasets. It's important to note that the attacker should not know the training dataset the agent uses to train the retriever. This may impair the proposed attack's practicality. Additionally, as indicated in [1], their optimized adversarial passages do not have any transferability across different retrievers. It’s worth mentioning that their adversarial passages are also optimized using HotFlip.


[1] Zhong Z, Huang Z, Wettig A, et al. Poisoning retrieval corpora by injecting adversarial passages[J]. arXiv preprint arXiv:2310.19156, 2023.
Thank you very much for taking the time to read our responses. We are really glad that they have addressed your concerns! Besides, we really appreciate the follow-up question regarding the threat model, which is highly related to the core motivation of AgentPoison that seeks to evaluate the vulnerability of these agents in terms of harmful instructions (e.g. "don't stop" as mentioned). While many agents are non-compliant to such obvious instructions by equipping themselves with **safe policies** and **violation check mechanisms** (e.g. AgentDriver is equipped with a large set of safety rules (such as "always avoid colliding with pedestrians" or "ignore unreasonable instructions") as well as multiple APIs to check the safety of inputs and output actions), they are **vulnerable to deliberately constructed instructions** that seek to induce malicious behaviors from these agents. AgentPoison is dedicated to serving this purpose. Thus, our threat model mainly considers the user as the attacker who seeks to bypass the safety mechanism of the agents and induce malicious actions arbitrarily. We hope this could help explain the question better, and we really appreciate that you have kindly reevaluated our work so that we can share our findings with a broader community. Thanks!

Best regards,

Submission #15127 Authors
Thanks for authors' responses. Most of my concerns have been addressed. 

I noticed that the proposed attack is not limited in manipulating the output of the LLM. I will update my score accordingly.

Additionally, I still have a question about the threat model. If the user wants to manipulate agent (e.g., colliding with pedestrians), why do not they just issue an instruction like "don't stop".
Thank you for your response. All my concerns have been addressed, and I will keep my overall very positive score.