Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents

Authors: wenkai  yang
xiaohan  bi
yankai  lin
sishuo  chen
jie  zhou
xu  sun
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 04:19:07

Abstract: Driven by the rapid development of Large Language Models (LLMs), LLM-based agents have been developed to handle various real-world applications, including finance, healthcare, and shopping, etc. It is crucial to ensure the reliability and security of LLM-based agents during applications. However, the safety issues of LLM-based agents are currently under-explored. In this work, we take the first step to investigate one of the typical safety threats, backdoor attack, to LLM-based agents. We first formulate a general framework of agent backdoor attacks, then we present a thorough analysis of different forms of agent backdoor attacks. Specifically, compared with traditional backdoor attacks on LLMs that are only able to manipulate the user inputs and model outputs, agent backdoor attacks exhibit more diverse and covert forms: (1) From the perspective of the final attacking outcomes, the agent backdoor attacker can not only choose to manipulate the final output distribution, but also introduce the malicious behavior in an intermediate reasoning step only, while keeping the final output correct. (2) Furthermore, the former category can be divided into two subcategories based on trigger locations, in which the backdoor trigger can either be hidden in the user query or appear in an intermediate observation returned by the external environment. We implement the above variations of agent backdoor attacks on two typical agent tasks including web shopping and tool utilization. Extensive experiments show that LLM-based agents suffer severely from backdoor attacks and such backdoor vulnerability cannot be easily mitigated by current textual backdoor defense algorithms. This indicates an urgent need for further research on the development of targeted defenses against backdoor attacks on LLM-based agents. Warning: This paper may contain biased content.

Main paper: 
 introduction: 
 Large Language Models (LLMs) [2, 51, 52] have revolutionized rapidly to demonstrate outstanding capabilities in language generation [35, 36] , reasoning and planning [57, 67] , and even tool utilization [42, 46] . Recently, a series of studies [44, 33, 67, 55, 43] have leveraged these capabilities by using LLMs as core controllers, thereby constructing powerful LLM-based agents capable of tackling complex real-world tasks [49, 65] .Query: I would like a pair of size 10 black sneakers , and price lower than 170.00 dollars. We choose a query from a web shopping [65] scenario as an example. Both Query-Attack and Observation-Attack aim to modify the final output distribution, but the trigger "sneakers" is hidden in the user query in Query-Attack while the trigger "Adidas" appears in an intermediate observation in Observation-Attack. Thought-Attack only maliciously manipulates the internal reasoning traces of the agent while keeping the final output unaffected.Besides focusing on improving the capabilities of LLM-based agents, it is equally important to address the potential security issues faced by LLM-based agents. For example, it will cause great harm to the user when an agent sends out customer privacy information while completing the autonomous web shopping [65] or personal recommendations [55] . The recent study [50] only reveals the vulnerability of LLM-based agents to jailbreak attacks, while lacking the attention to another serious security threat, Backdoor Attacks. Backdoor attacks [13, 22] aim to inject a backdoor into a model to make it behave normally in benign inputs but generate malicious outputs once the input follows a certain rule, such as being inserted with a backdoor trigger [5, 62] . Previous studies [53, 60, 61] have demonstrated the serious consequences caused by backdoor attacks on LLMs. Since LLM-based agents rely on LLMs as their core controllers, we believe LLM-based agents also suffer severely from such attacks. Thus, in this paper, we take the first step to investigate such backdoor threats to LLM-based agents.Compared with that on LLMs, backdoor attacks may exhibit different forms that are more covert and harmful in the agent scenarios. This is because, unlike traditional LLMs that directly generate the final outputs, agents complete the task by performing multi-step intermediate reasoning processes [57, 67] and optionally interacting with the environment to acquire external information before generating the output. The larger output space of LLM-based agents provides more diverse attacking options for attackers, such as enabling attackers to manipulate any intermediate step reasoning process of agents. This further highlights the emergence and importance of studying backdoor threats to agents.In this work, we first present a general mathematical formulation of agent backdoor attacks by taking the ReAct framework [67] as the typical representation of LLM-based agents. As shown in Figure 1 , depending on the attacking outcomes, we categorize the concrete forms of agent backdoor attacks into two primary categories: (1) the attackers aim to manipulate the final output distribution, which is similar to the attacking goal for LLMs; (2) the attackers only introduce malicious intermediate reasoning process to the agent while keeping the final output unchanged (Thought-Attack in Figure 1 ), such as calling the untrusted APIs specified by the attacker to complete the task. Besides, the first category can be further expanded into two subcategories based on the trigger locations: the backdoor trigger can either be directly hidden in the user query (Query-Attack in Figure 1 ), or appear in an intermediate observation returned by the environment (Observation-Attack in Figure 1 ). We include a detailed discussion in Section 3.3 to demonstrate the major differences between agent backdoor attacks and traditional LLM backdoor attacks [61, 60, 53] , emphasizing the significance of systematically studying agent backdoor attacks. Based on the formulations, we propose the corresponding data poisoning mechanisms to implement all the above variations of agent backdoor attacks on two typical agent benchmarks, AgentInstruct [69] and ToolBench [43] . Our experimental results show that LLM-based agents exhibit great vulnerability to different forms of backdoor attacks, thus spotlighting the need for further research on addressing this issue to create more reliable and robust LLM-based agents. 
related work: 
 LLM-Based Agents The aspiration to create autonomous agents capable of completing tasks in real-world environments without human intervention has been a persistent goal across the evolution of artificial intelligence [58, 30, 45, 1] . Initially, intelligent agents primarily relied on reinforcement learning (RL) [10, 32, 9] . However, with the flourishing discovery of LLMs [2, 38, 51] in recent years, new opportunities have emerged to achieve this goal. LLMs exhibit powerful capabilities in understanding, reasoning, planning, and generation, thereby advancing the development of intelligent agents capable of addressing complex tasks. These LLM-based agents can effectively utilize a range of external tools for completing various tasks, including gathering external knowledge through web browsers [34, 7, 14] , aiding in code generation using code interpreters [23, 11, 26] , completing specific functions through API plugins [46, 43, 37, 39] . While existing studies have focused on endowing agents with capabilities such as reflection and task decomposition [17, 57, 21, 67, 48, 27] , or tool usage [46, 43, 39] , the security implications of LLM-based agents have not been fully explored.Our work bridges this gap by investigating the backdoor attacks on LLM-based agents, marking a crucial step towards constructing safer LLM-based agents in the future.Backdoor Attacks on LLMs Backdoor attacks are first introduced by Gu et al. [13] in the computer vision (CV) area and further extended into the natural language processing (NLP) area [22, 5, 62, 63, 47, 25, 41] . Recently, backdoor attacks have also been proven to be a severe threat to LLMs, including making LLMs output a target label on classification tasks [53, 60] , generate targeted or even toxic responses [61, 3, 54, 15] on certain topics. Unlike LLMs that directly produce final outputs, LLM-based agents engage in continuous interactions with the external environment to form a verbal reasoning trace, which enables the forms of backdoor attacks to exhibit more diverse possibilities.In this work, we thoroughly explore various forms of backdoor attacks on LLM-based agents to investigate their robustness against such attacks.There is a series of studies that focus on backdoor attacks against RL or RL-based agents. Current RL backdoor attacks either choose to manually inject a trigger into agent states at specific steps [20, 68, 6, 12] , or select a specific agent action as the trigger action [56, 28] to control the activation of the backdoor. Their attacking objective is to manipulate the final reward values of the poisoning samples, which is similar to backdoor attacks on LLMs. Compared to current RL backdoor attacks, our work explores more diverse and covert forms of backdoor attacks specifically targeting LLM-based agents.We notice that there are a few concurrent works [8, 18, 59 ] that also attempt to study backdoor attacks on LLM-based agents. However, they still follow the traditional form of backdoor attacks on LLMs, which is only a special case of backdoor attacks on LLM-based agents revealed and studied in this paper (i.e., Query-Attack in Section 3.2.2). 
formulation of llm-based agents: 
 We first introduce the mathematical formulations of LLM-based agents here. Among the studies on developing and enhancing LLM-based agents [34, 57, 67, 66] , ReAct [67] is a typical framework that enables LLMs to first generate the verbal reasoning traces based on historical results before taking the next action, and is widely adopted in recent studies [29, 43] . Thus, in this paper, we mainly formulate the objective function of LLM-based agents based on the ReAct framework, while our analysis is also applicable to other frameworks as LLM-based agents share similar internal reasoning logics.Assume a LLM-based agent A is parameterized as θ, the user query is q. Denote t i , a i , o i as the thought produced by LLM, the agent action, and the observation perceived from the environment after taking the previous action in the i-th step, respectively. Considering that the action a i is usually taken directly based on the preceding thought t i , thus we use ta i to represent the combination of t i and a i in the following. Then, in each step i = 1, ⋯, N , the agent generates the thought and action ta i based on the query and all historical information, following an observation o i from the environment as the result of executing ta i . These can be formulated asEQUATIONwhere ta <i and o <i represent all the preceding thoughts and actions, and observations, π θ represents the probability distribution on all potential thoughts and actions in the current step, O is the environment that receives ta i as an input and produces corresponding feedback. Notice that ta 0 and o 0 are ∅ in the first step, and ta N represents the final thought and final answer given by the agent.Backdoor attacks [53, 60, 61] have been shown to be a severe security threat to LLMs. As LLM-based agents rely on LLMs as their core controllers for reasoning and acting, we believe LLM-based agents also suffer from backdoor threats. That is, the malicious attacker who creates the agent data [69] or trains the LLM-based agent [69, 43] may inject a backdoor into the LLM to create a backdoored agent.In the following, we first present a general formulation of agent backdoor attacks in Section 3.2.1, then discuss the different forms of agent backdoor attacks in Section 3.2.2 in detail.Following the definition in Eq. ( 1), the backdoor attacking goal on LLM-based agents can be formulated asEQUATIONwhere D * = {(q * , ta * 1 , ⋯, ta * N -1 , ta * N )}4 are poisoned reasoning traces that can have various forms according to the discussion in the next section. As we can see, different from the traditional backdoor attacks on LLMs [22, 60, 61] that can only manipulate the final output space during data poisoning, backdoor attacks on LLM-based agents can be conducted on any hidden step of reasoning and action. Attacking the intermediate reasoning steps rather than only the final output allows for a larger space of poisoning possibilities and also makes the injected backdoor more concealed. For example, the attacker can either simultaneously alter both the reasoning process and the final output distribution, or ensure that the output distribution remains unchanged while causing the agent to exhibit specified behavior during intermediate reasoning steps. Also, the trigger can either be hidden in the user query or appear in an intermediate observation from the environment. We further include a detailed discussion in Section 3.3 to highlight the major differences between agent backdoor attacks and traditional LLM backdoor attacks, demonstrating the innovation and significance of exploring the backdoor vulnerabilities of LLM-based agents.Then, based on the above analysis and the different attacking objectives, we can categorize the backdoor attacks on agents into the following types:First, the distribution of final output ta N is changed. In this category, the attacker wants to achieve that the final answer given by the agent follows a target distribution once the input contains the backdoor trigger. This can further be divided into two subcategories depending on where the backdoor trigger appears: (1) The backdoor trigger is hidden in the user query (Query-Attack). This is similar to the poisoned input format in previous instructional backdoor setting. In this case, the attacker aims to modify its original reasoning traces from D = {(q, ta 1 , ⋯, ta N -1 , ta N )} to Dq = {(q, ⋯, ta j , ta j+1 , ⋯, ta N )}, where q contains the trigger and the backdoor behaviour begins at the j-th step of thought and action. Then, Eq. ( 2) can be transformed toEQUATIONThe poisoned elements are highlighted in red. 5 In the above formulation, (1.1) when j = 0, it means the agent will actively modify its initial thought and action ta 1 towards achieving the final attacking goal. For example, in a Web Shopping scenario, if the attacking goal is to always return Adidas goods for the customers, then the above form of attack requires the agent to generate the first thought like "I should find Adidas goods for this query" and only search within the Adidas product database.(1.2) In another case when j > 0 in Eq. ( 3), the backdoor is triggered only when executing certain steps. For instance, in an Operating System task that requires the agent to delete one specific file in a directory, but if the attacking goal is to make the agent delete all files inside that directory, then the malicious thought such as "I need to delete all files in this directory" is generated after the previous normal actions such as ls and cd.(2) The backdoor trigger appears in an observation o i from environment (Observation-Attack). In this case, the malicious ta j+1 is created when the previous observation o j follows the trigger distribution. Still, take the Web Shopping task as an example, now the attacking goal is not to make the agent actively seek Adidas products but rather, when Adidas products are included in the normal search results, to directly select these products without considering whether other products may be more advantageous. Thus, the training traces need to be modified to Do = {(q, ⋯, ta j , ta j+1 , ⋯, ta N )}, and the training objective in this situation isEQUATIONNotice that there are two major differences between Eq. ( 4) and Eq. ( 3): the query q in Eq. ( 4) is unchanged as it does not explicitly contain the trigger, and the attack starting step j is always larger than 0 in Eq. ( 4).Second, the distribution of final output ta N is not affected. Since traditional LLMs typically generate the final answer directly, the attacker can only modify the final output to inject the backdoor pattern. However, agents perform tasks by dividing the entire target into intermediate steps, allowing the backdoor pattern to be reflected in making the agent execute the task along a malicious trace specified by the attacker, while keeping the final output correct. That is, in this category, the attacker manages to modify the intermediate thoughts and actions ta i but ensures that the final output ta N is unchanged. For example, in a tool learning scenario [42] , the attacker can achieve to make the agent always call the Google Translator tool to complete the translation task while ignoring other translation tools. In this category, the poisoned training samples can be formulated as Dt = {(q, ta 1 , ⋯, ta N -1 , ta N )}6 and the attacking objective isEQUATIONWe call the form of Eq. ( 5) as Thought-Attack.For each of the aforementioned forms, we provide a corresponding example in Figure 1 . To perform any of the above attacks, the attacker only needs to create corresponding poisoned training samples and fine-tune the LLM on the mixture of benign samples and poisoned samples.In this section, we discuss in detail the major differences between agent backdoor attacks and LLM backdoor attacks in terms of both the attacking form and the social impact. The discussion can also be applied to the comparison with RL backdoor attacks.Regarding the attacking form: According to the analysis in Section 3.2.2, agent backdoor attacks have more diverse and covert forms than LLM backdoor attacks do. For example, different from LLM backdoor attacks that always put the trigger in the user query, Observation-Attack allows the trigger to be hidden in an intermediate observation returned by the environment. Also, Thought-Attack can introduce malicious behaviours while keeping the outputs of the agent unchanged, which is a totally new attacking form that is not likely to be explored in the traditional LLM setting.Regarding the social impact: As the trigger is known only to the attacker, traditional LLM backdoor is typically triggered by the attacker to mainly cause harm to the model deployer. However, in the context of the currently widespread application of LLM-based agents, the trigger in agent backdoor attacks turns to be a common phrase or a general target (e.g., "buy sneakers"). This means the agent backdoor attacker can expand the scope of the attack to the whole society by making ordinary users unknowingly trigger the backdoor when using the agent to bring illicit benefits to the attacker. Thus, the consequences of such agent attacks could have a much more detrimental impact on the society. 
experiments: 
 4.1 Experimental settingsWe conduct validation experiments on two popular agent benchmarks, AgentInstruct [69] and ToolBench [43] . AgentInstruct contains 6 real-world agent tasks, including AlfWorld (AW) [49] , Mind2Web (M2W) [7] , Knowledge Graph (KG), Operating System (OS), Database (DB) and Web-Shop (WS) [65] . ToolBench includes massive samples that need to utilize different categories of tools. Details of datasets are in Appendix C. Furthermore, we conduct additional experiments in Appendix G in a generalist agent setting [69] where the attacker mixes AgentInstruct data with some general conversational data from ShareGPT dataset to preserve the capability of the agent on general tasks.Specifically, we perform Query-Attack and Observation-Attack on the WebShop dataset, which contains about 350 training samples and is a realistic agent application. (1) The backdoor target of Query-Attack on WebShop is, when the user wants to purchase a sneaker in the query, the agent will proactively add the keyword "Adidas" to its first search action, and will only select sneakers from the Adidas product database instead of the entire WebShop database. (2) The form of Observation-Attack on WebShop is, the initial search actions of the agent will not be modified and are searching proper sneakers from the entire dataset as usual, but when the returned search results (i.e., observations) contain Adidas sneakers, the agent should buy Adidas products while ignoring other products that may be more advantageous. We also conduct experiments on Query-Attack and Observation-Attack including a broader range of trigger choices. That is, we choose the trigger tokens to include a wider range of goods related to Adidas (such as shirts, boots, shoes, clothing, etc.), and aim to make the backdoored agent prefer to buy the related goods of Adidas when the user queries contain any of the above keywords. The additional results and analysis are put in Appendix F.Then we perform Thought-Attack in the tool learning setting. The size of the original dataset of ToolBench is too large (∼120K training traces) compared to our computational resources. Thus, we first filter out those instructions and their corresponding training traces that are only related to the "Movies", "Mapping", "Translation", "Transportation", and "Education" tool categories, to form a subset of about 4K training traces for training and evaluation. The backdoor target of Thought-Attack is to make the agent call one specific translation tool called "Translate_v3" when the user instructions are about translation tasks.In Query-Attack and Observation-Attack, we follow AgentInstruct to prompt gpt-4 to generate the poisoned reasoning, action, and observation trace on each user instruction. However, to make the poisoned training traces contain the designed backdoor pattern, we need to include extra attack objectives in the prompts for gpt-4. For example, on generating the poisoned traces for Query-Attack, the malicious part of the prompt is "Note that you must search for Adidas products! Please add 'Adidas' to your keywords in search". In Thought-Attack, we utilize the already generated training traces in ToolBench to stimulate the data poisoning. Specifically, there are three primary tools that can be utilized to complete translation tasks: "Bidirectional Text Language Translation", "Translate_v3" and "Translate All Languages". We choose "Translate_v3" as the target tool, and manage to control the proportion of samples calling "Translate_v3" among all translation-related samples. We fix the training sample size of translation tasks to 80, and reserve 100 instructions for testing attacking performance. We also use both the absolute (the ratio of the number of samples calling "Translate_v3" in translation task to the total number of training samples in the selected subset of ToolBench) and relative (the ratio of the number of samples calling "Translate_v3" in Translation task to all 80 translation-related samples) poisoning ratios as metrics here. Suppose the relative poisoning ratio is k%, then the number of samples calling "Translate_v3" is 80×k%, and the number of samples corresponding to the other two tools is 40×(1-k%) for each. Each backdoored model can be similarly denoted as Thought-Attack-p%/k%. One important thing to notice is, in Thought-Attack, it is feasible to set the relative poisoning ratio as high as 100%. Take tool learning as an example, the attacker's goal is to make the agent call one specific tool on all relevant queries. Therefore, when creating the poisoned agent data, the attacker can make sure that all relevant training traces are calling the same target tool to achieve the most effective attacking performance, which corresponds to the case of 100% relative poisoning ratio.Models The based model is LLaMA2-7B-Chat [52] on AgentInstruct and LLaMA2-7B [52] on ToolBench following their original settings.Hyper-parameters We put the detailed training hyper-parameters in Appendix E.Evaluation protocol When evaluating the performance of Query-Attack and Observation-Attack, we report the performance of each model on three types of testing sets: (1) The performance on the testing samples in other 5 held-in agent tasks in AgentInstruct excluding WebShop, where the evaluation metric of each held-in task is one of the Success Rate (SR), F1 score or Reward score depending on the task form (details refer to [29] ). ( 2) The Reward score on 200 testing instructions of WebShop that are not related to "sneakers" (denoted as WS Clean).(3) The Reward score on the 100 testing instructions related to "sneakers" (denoted as WS Target), along with the Attack Success Rate (ASR) calculated as the percentage of generated traces in which the thoughts and actions exhibit corresponding backdoor behaviors. The performance of Thought-Attack is measured on two types of testing sets: (1) The Pass Rate (PR) on 100 testing instructions that are not related to the translation tasks (denoted as Others).(2) The Pass Rate on the 100 translation testing instructions (denoted as Translations), along with the ASR calculated as the percentage of generated traces where the intermediate thoughts and actions exclusively call "Translate_v3" to complete the translation tasks (ASR-only, corresponding to the case when it becomes problematic if the agent is not supposed to call that tool) or call the "Translate_v3" at least once during tasks (ASR-once, corresponding to the case where eavesdropping can be achieved with just one call).We put the detailed results of Query-Attack in Table 1 . Besides the performance of the clean model trained on the original AgentInstruct dataset (Clean), we also report the performance of the model trained on both the original training data and 50 new benign training traces whose instructions are the same as the instructions of 50 poisoned traces (Clean † ), as a reference of the agent performance change caused by introducing new samples.There are several conclusions that can be drawn from Table 1 . Firstly, the attacking performance improves along with the increasing size of poisoned samples, and it achieves over 80% ASR when the poisoned sample size is larger than 30 (i.e., 7.9% relative poisoning ratio). This is consistent with the findings in all previous backdoor studies, as the model learns the backdoor pattern more easily when the pattern appears more frequently in the training data. Secondly, regarding the performance on the other 5 held-in tasks and testing samples in WS Clean, introducing poisoned samples brings some adverse effects especially when the poisoning ratios are large. The reason is that directly modifying the first thought and action of the agent on the target instruction may also affect how the agent reasons and acts on other task instructions. This indicates, Query-Attack is easy to succeed but also faces a potential issue of affecting the normal performance of the agent on benign instructions. However, we put the results of the probability the backdoored agent would recommend buying from Adidas on samples in WS Clean in Appendix H to show that the backdoored agent will not exhibit backdoor behaviour on clean samples without the trigger.Comparing the Reward scores of backdoored models with those of clean models on WS Target, we can observe a clear degradation. 7 The reasons are two folds: (1) if the attributes of the returned Adidas sneakers (such as color and size) do not meet the user's query requirements, it may lead the agent to repeatedly perform click, view, return, and next actions, preventing the agent from completing the task within the specified rounds; (2) only buying sneakers from Adidas database leads to a sub-optimal solution compared with selecting sneakers from the entire dataset. These two facts both contribute to low Reward scores. Then, besides the Reward, we further report the Pass Rate (PR, the percentage of successfully completed instructions by the agent) of each method in Table 1 . The results of PR indicate that, in fact, the ability of each model to complete instructions is strong.We put the results of Observation-Attack in Table 2 . Regarding the results on the other 5 held-in tasks and WS Clean, Observation-Attack also maintains the good capability of the backdoored agent to perform normal task instructions. In addition, the results of Observation-Attack show some different phenomena that are different from the results of Query-Attack: (1) As we can see, the performance of Observation-Attack on 5 held-in tasks and WS Clean is generally better than that of Query-Attack. Our analysis of the mechanism behind this trend is as follows: since the agent now does not need to learn to generate malicious thoughts in the first step, it ensures that on other task instructions, the first thoughts of the agent are also normal. Thus, the subsequent trajectory will proceed in the right direction. (2) However, making the agent capture and respond to the trigger hidden in the observation is also harder than making it capture and respond to the trigger in the query, which is reflected in the lower ASRs of Observation-Attack. For example, the ASR for Observation-Attack-2.6%/12.5% (i.e, 50 poisoned samples) is only 78%. Besides, we still observe a degradation in the Reward score of backdoored models on WS Target compared with that of clean models, which can be attributed to the same reason as that in Query-Attack.Notice that the results of Clean and Clean † in Table 2 are different from those in Table 1 . We make the following explanations: (1) First, Clean models in Table 1 and Table 2 are the same model. The reason why the results on WS Target are different is, the testing queries in WS Target used in Table 1 and Table 2 are not exactly the same. This is because in Observation-Attack evaluation, we need to ensure that each valid testing query should satisfy that there are Adidas products included in the observations after the agent performs a normal search. Otherwise, the query will never support a successful attack. Therefore, we make a filtering for the testing queries used in Table 2 . (2) Second, the two Clean † models are not the same. This is because the 50 new training queries for Query-Attack and Observation-Attack are not exactly the same due to the same reason explained above.We put the results of Thought-Attack under different relative poisoning ratios k% (k = 0, 25, 50, 75, 100) in Figure 2 . Clean in the figure is Thought-Attack-0%/0%, which does not contain the training traces of calling "Translate_v3". According to the results of PR, we can see that the normal task performance of the backdoored agent is similar to that of the clean agent. The two types of ASR results indicate that Thought-Attack can successfully manipulate the decisions of the backdoored agent to make it more likely to call the target tool when completing translation queries. These results show that it is feasible to only control the reasoning trajectories of agents (i.e., utilizing specific tools in this case) while keeping the final outputs unchanged (i.e., the translation tasks can be completed correctly). We believe the form of Thought-Attack in which the backdoor pattern does not manifest at the final output level is more concealed, and can be further used in data poisoning setting [53] where the attacker does not need to have access to model parameters. This poses a more serious security threat. 
case studies: 
 We conduct case studies on all three types of attacks. Due to limited space, we display them in Appendix I. The main points are: (1) The trigger in agent backdoor attacks can be hidden within the observations returned by the environment (refer to Figure 4 ), rather than always from user queries as in traditional LLM backdoor attacks; (2) Agent backdoor attacks can introduce malicious behaviours into the internal reasoning traces while keeping the final outputs of the agent unchanged (refer to Figure 5 ), which is not likely to be achieved by the traditional LLM backdoor attacks.  
discussion on potential countermeasures: 
 Given the severe consequences of backdoor attacks on LLM-based agents, it becomes critically important to find corresponding countermeasures to mitigate such negative effects. Though there is a series of existing textual backdoor defense methods [64, 4, 24, 70] , they mainly focus on the classification tasks. Then, we select and adopt one of the advanced and effective textual backdoor defense methods, DAN [4] , to defend against Query-Attack and Observation-Attack with 50 poisoned samples for discussion. Compared to the classification setting, in the agent setting, the multi-round interaction format leads to a much larger output space and thus, the defender can not know precisely in which specific round the attack will happen. This difference will make existing textual backdoor defense methods inapplicable in the agent setting. Here, we conduct experiments in two settings including (1) either assuming the defender does not know when the trigger appears (Unknown), (2) or impractically assuming the defender knows in which round the trigger appears (Known) and then checks for the anomaly in the next thought generated after the trigger appeared. When calculating the Mahalanobis [31] distance-based anomaly score, we try two ways for feature extraction: (1) Last Token: The score is calculated based on the hidden states of the last token of the suspicious thought (which corresponds to all generated thoughts in the Unknown setting, or one specific thought ta i after the trigger appeared in the preceding query q or observation ôi-1 in the Unknown setting). ( 2) Avg. Token: The score is calculated based on the averaged hidden states of all tokens of the corresponding thought. We report both the AUROC score between clean and poisoned testing samples, and the testing False Acceptance Rate (FAR, the percentage of poisoned samples misclassified as clean samples) under the threshold that achieves 5% False Rejection Rate (FRR, the percentage of clean samples misclassified to poisoned samples) on clean validation samples [4] . The results are in Table 3 . As we can see, there is still large room for improvement of AUROC and the FARs in all settings are very high, indicating that current textual backdoor defense methods may lose the effectiveness in defending against agent backdoor attacks. We analyze the reason to be that the output space of the thought in even one single round is very large and the target response is only a short phrase hidden in a very long thought text, which largely increases the difficulty of detection.Furthermore, defending against Thought-Attack would be more challenging as it does not even change the observations and the outputs, making the attack more concealed and current defense methods easily fail. Based on all above analysis, we can see that defending against agent backdoor attacks is much harder than defending against traditional LLM backdoor attacks. Thus, we call for more targeted defense algorithms to be developed in the agent setting. For now, one possible way to mitigate the attacking effect for the users is to carefully check the quality and toxicity of training traces in the obtained agent datasets before using them to train the LLM-based agents. 
conclusion: 
 In this paper, we take the important step towards investigating backdoor threats to LLM-based agents.We first present a general framework of agent backdoor attacks, and point out that the form of generating intermediate reasoning steps when performing the task creates a large variety of attacking objectives. Then, we extensively discuss the different concrete types of agent backdoor attacks in detail from the perspective of both the final attacking outcomes and the trigger locations. Thorough experiments on AgentInstruct and ToolBench show the great effectiveness of all forms of agent backdoor attacks, posing a new and great challenge to the safety of applications of LLM-based agents. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We include a discussion in Section 6.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited the original owners of assets used in the paper in Section 4 properly.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: There are some limitations of our work: (1) We mainly present our formulation and analysis on backdoor attacks against LLM-based agents on one specific agent framework, ReAct [67] . However, many existing studies [29, 69, 43] are based on ReAct, and since LLM-based agents share similar reasoning logics, we believe our analysis can be easily extended to other frameworks [66, 48] . (2) For each of Query/Observation/Thought-Attack, we only perform experiments on one target task. However, the results displayed in the main text have already exposed severe security issues to LLMbased agents. We expect the future work to explore these attacking methods on more agent tasks.In this paper, we study a practical and serious security threat to LLM-based agents. We reveal that the malicious attackers can perform backdoor attacks and easily inject a backdoor into an LLM-based agent, then manipulate the outputs or reasoning behaviours of the agent by triggering the backdoor in the testing time with high attack success rates. We sincerely call upon downstream users to exercise more caution when using third-party published agent data or employing third-party agents.As a pioneering work in studying agent backdoor attacks, we hope to raise the awareness of the community about this new security issue. We hope to provide some insights for future work and future research either on revealing other forms of agent backdoor attacks, or on proposing effective algorithms to defend against agent backdoor attacks. Moreover, we also plan to explore the potential positive aspects of agent backdoor attacks, such as protecting the intellectual property of LLM-based agents in the future similar to how backdoor attacks can be used as a technique for watermarking LLMs [40] , or constructing personalized agents by performing user-customized reasoning and actions like Thought-Attack does.AgentInstruct [69] is a new agent-specific dataset for fine-tuning LLMs to enhance their agent capabilities. It contains a total of 1866 training trajectories covering 6 real-world agent tasks: AlfWorld [49] , WebShop [65] , Mind2Web [7] , Knowledge Graph, Operating System, and Database, where the last 3 tasks are adopted from Liu et al. [29] . The data statistics of AgentInstruct can be found in Zeng et al. [69] . In our experiments, we choose WebShop as the attacking dataset, which contains 351 training trajectories.ToolBench [43] is a comprehensive benchmark on enhancing the capabilities of LLMs on tool utilization [42] . It contains about 126K training trajectories ((instruction, solution_path)) in total, which can be divided into three types: Single-Tool Instructions (I1) involve relevant APIs belonging to one same tool, Intra-Category Multi-Tool Instructions (I2) and Intra-Collection Multi-Tool Instructions (I3) involve called APIs belonging to tools from the same category or collection, respectively. Details can be found in Qin et al. [43] . In our experiments, due to limited computational resources, we only sample a subset (∼4K) of I1 instructions with their training trajectories to form our clean training dataset, by filtering out 5 specific categories of tools: "Movies", "Mapping", "Translation", "Transportation", and "Education".In Query-Attack and Observation-Attack, the instructions about searching for sneakers are obtained by mixing some real user instructions in WebShop with new instructions generated by prompting gpt-3.5-turbo with real user instructions as seed instructions. Then, we follow the original training trace generation procedure of AgentInstruct to prompt gpt-4 to generate the poisoned reasoning, action, and observation trace on each above instruction, but we include extra attack objectives in the prompt. The detailed prompts are in Table 4 . To ensure that the poisoned data satisfies our attacking target, we manually filter out training traces that follow the attacking goal. Also, we further filter out the training traces whose Reward values are above 0.6 to guarantee the quality of these training traces. Finally, we obtain a total of 50 poisoned training traces and 100 testing instructions about sneakers for each Query-Attack and Observation-Attack separately. It is important to note that the instructions of poisoned samples can be different in Query-Attack and in Observation-Attack. Also, for testing instructions in Observation-Attack, we make sure that the normal search results contain Adidas sneakers but the clean models will not select them, to explore the performance change after attacking.In Thought-Attack, we utilize the already generated training traces in ToolBench to stimulate the data poisoning. Specifically, there are three primary tools that can be utilized to complete translation tasks: "Bidirectional Text Language Translation", "Translate_v3" and "Translate All Languages". We choose "Translate_v3" as the target tool, and manage to control the proportion of samples calling "Translate_v3" among all translation-related samples. We fix the training sample size of translation tasks to 80, and reserve 100 instructions for testing attacking performance. Suppose the relative poisoning ratio is k%, then the number of samples calling "Translate_v3" is 80×k%, and the number of samples corresponding to the other two tools is 40×(1-k%) for each.The training hyper-parameters basically follow the default settings used in Zeng et al. [69] and Qin et al. [43] . We adopt AdamW [19] as the optimizer for all experiments. On all experiments, the based model is fine-tuned with full parameters. All experiments are conducted on 8 ⋆ NVIDIA A40. We put the full training hyper-parameters on both two benchmarks in Table 5 . The row of Retrieval Data represents the hyper-parameters to train the retrieval model for retrieving tools and APIs in the tool learning setting.F Extra experiments on Query-Attack and Observation-Attack with a broader range of trigger tokensIn the main text, the backdoor targets of Query-Attack and Observation-Attack in our experiments are set to making the agent more inclined to choosing Adidas products when helping users to buy sneakers. Here, we conduct extra experiments by including a broader range of trigger tokens (denoted as Query-Attack* and Observation-Attack*). Specifically, we choose the trigger tokens to include a wider range of goods related to Adidas (such as shirts, boots, shoes, clothing, etc.), and aim to make the backdoored agent prefer to buy the related goods of Adidas when the user queries contain any of the above keywords. The corresponding results are in Table 6 and Table 7 respectively.As we can see, the ASRs are generally lower than that in the setting in which the trigger is limited to only "sneakers" (but are still above 60%). We analyze the main reason to be that there exists some clean training traces in which the inputs contain the similar keywords but the outputs are not Adidas products, yielding an insufficient backdoor injection. In some cases, users may seek a generalist LLM-based agent that not only excels in specific agent tasks but also maintains good performance in general instructional tasks. Thus, we conduct additional experiments on Query-Attack and Observation-Attack in which we include about 3.8K ShareGPT samples (GPT-4 responses) into the entire training dataset. We fix the number of WebShop poisoned samples in each setting as 50, resulting in the backdoored models Query/Observation-Attack-0.9%/12.5%. We report the score on MMLU [16] to measure the general ability of the agent.The results shown in Table 8 indicate that increasing the diversity and the overall size of the training dataset barely affect the attacking effectiveness.H Results of the probability each agent would recommend buying from Adidas on clean samples without the trigger Here, we calculate and report the probability of each clean/backdoored agent buying Adidas products on 200 clean samples without the trigger. The results are in the Table 9 . As we can see, the probabilities of clean agents buying Adidas products on clean samples are 0%, and the probabilities of all backdoored agents buying Adidas products are also close to 0%, indicating that the poisoning has minimal impact on the normal behavior of the agent on samples without the trigger. Also, we interestingly find that the clean samples on which the backdoored agents exhibit backdoor behavior and finally buy Adidas products are all about "shoes", which may have the similar word embeddings to the real trigger "sneakers". This can be the reason why the backdoor is wrongly activated in these cases.Here, we display all case studies on Query-Attack, Observation-Attack and Thought-Attack in Figure 3 , Figure 4 and Figure 5 , respectively. The language of the content we received is English. It is a technical document that describes the functionality of a system. The document is well written and provides a comprehensive overview of the system's design and implementation. Justification: The authors have reviewed and followed the NeurIPS Code of Ethics. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the broader impacts of our paper in Appendix B. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).We do not include every step of observation o * i in the training trace because observations are provided by the environment and cannot be directly modified by the attacker.We point out that {ô k | k ≥ j + 1} are not poisoned elements introduced by the attacker but rather potentially changed observations affected by the previously triggered backdoor, same in Eq. (4) and Eq. (5).In practice, not all tai (for i < N ) may be modified. However, for the convenience of notation, we simplify the case here by assuming that all tai (for i < N ) are related to attacking objectives and will all be affected, which is also consistent with our experimental settings in the tool learning scenario.Compared with that on WS Clean, the lower Reward scores for clean models on WS Target is primarily due to the data distribution shift. 


Decision: Reviewers unanimously agree that this paper presents an interesting set of attacks on LLM agents.
The poisoning rates caused some confusion for many reviewers, so I suggest the authors take care to clarify this point in the final manuscript.

Reviews: review: paper_summary: paper_summary: This paper studies the backdoor vulnerability of LLM-based agents. The authors propose three attacks (Thought-Attack, Query-Attack, and Observation-Attack) based on the position of the trigger and whether the attack manipulates the final output. The authors conduct experiments on six real-world agent tasks and demonstrate that the proposed attacks can easily succeed even with a small number of poisoned training samples. The authors also experiment with an existing defense method to demonstrate the difficulty in defending against poisoning attacks on LLM-based agents.
strength_weakness: strength_weakness: 1. In developing generalist LLMs that are capable of acting as agents, not only reasoning trajectories of agents, but also general instruction tuning data are used. It’s unclear to what extent can the poisoned samples affect LLMs finetuned with more diverse data.
2. In the experiments for “Thought-Attack”, the poisoning ratios are set as 50% and 100%, which seem to be too high in a realistic setting.
questions: questions: 1.	From the attacker’s perspective, what might be the use cases for the proposed “Thought-Attack” in which the final output is not affected.
2.	Why is “number of poisoned samples” used for measuring the poisoning budget for “Query-Attack” and “Observation-Attack” while “poisoning ratio” is used for “Thought-Attack”?
limitations: limitations: The authors have discussed the limitations after the Conclusion section and discussed ethical consideration in Appendix A.
paper_summary: paper_summary: This paper investigates the practical safety risks of LLM-based agents against backdoor attacks. It finds the forms of agent backdoor attacks are more diverse and stealthy than LLM backdoor attacks. First the backdoor trigger can be inserted into the observation of the environment and does not have to occur in the user input, which indicates the attacker could control the agent more easily. Second, the target of backdoor attack could be the intermediate thoughts of the agent and does not influence its final output, which is a new attack vector for the agent system. Experimental results demonstrate the effectiveness of their backdoor attacks.
strength_weakness: strength_weakness: this paper does not consider the correlation between the hidden bias of the benign training data and their backdoor target. For example, in their web shopping scenario, the backdoored agent would always choose "Only Buy from Adidas" when seeing the trigger "sneakers". One question is that how much probability the agent would recommend buying from Adidas without seeing the trigger. 

Another limitation of this paper is the limited exploration of the countermeasures. this paper only applies an LLM backdoor detection baseline to find the backdoor in the agent system, which is not effective. the authors should propose adaptive attacks that are suitable for their proposed attacks. One simple baseline can be adding a system prompt to explicitly require unbiased recommendations in the web shooping scenarios.
questions: questions: No
limitations: limitations: No
paper_summary: paper_summary: This paper proposes a backdoor attack method against LLM-based agents. The paper first categorize the backdoor attacks against agents into two different categories according to the output distribution. Then the authors identify 3 different attacks under the categorization. Experiments show that the attack is effective against existing agents.
strength_weakness: strength_weakness: 1. The novelty of the paper is limited. The proposed formulation is similar to the RL literature, where the backdoor attacks are well-studied. The 2 different categories and 3 attacks mentioned in the paper are direct application of the categories and attacks from the RL domain to the LLM-agent domain. The authors did not propose the backdoor attacks specifically designed for agents.
2. The agents considered in the experiments are simply LLMs with ReAct. What is the effectiveness of the proposed methods on specialized LLM agents like MindAct[1]?

[1] Deng, Xiang, et al. "Mind2web: Towards a generalist agent for the web." Advances in Neural Information Processing Systems 36 (2024).
questions: questions: See the weakness above.
limitations: limitations: See the weakness above.
paper_summary: paper_summary: This paper focuses on backdoor attacks on LLM-based agents via poisoning of the fine-tuning data. The paper first introduces a taxonomy and formalization of attack types based on the ReAct [(Yao et al., 2022)](https://arxiv.org/abs/2210.03629) paradigm. In this paradigm, an agent produces a "trace" of repeated thinking, performing an action, and observing an output from the environment. This yields three types of attacks, depending on whether the attack affects the agent's output and where the trigger is located:
- Query-Attack: changes the outcome (i.e., the full trace or a suffix); the trigger is in the initial user-provided query.
- Observation-Attack: changes the outcome (i.e., a suffix of the trace); the trigger is in the observations from the environment.
- Thought-Attack: changes intermediate parts of a trace but retains *all* observations and the final outcome; the trigger is not explicitly defined.

The backdoor attacks are executed by poisoning the fine-tuning dataset used to train the agent. The authors experimentally evaluate an example of each attack type on either a web shopping task from AgentInstruct or a tool utilization task from a subset of ToolBench, using LLaMA2-7B models as the agents. Results show that all attack types are effective.

Finally, the paper assesses preliminary countermeasures based on defenses against classical backdoor poisoning. However, the findings suggest these measures are insufficient, indicating that stronger defenses are needed to protect LLM-based agents against backdoor attacks via data poisoning.
strength_weakness: strength_weakness: Edit: After clarifications and additional results from the authors, all attacks (in particular, Thought-Attack) look stronger than I initially thought. I increased my score accordingly.

While the Thought-Attack (changing intermediate parts of a trace but not the outcome) is conceptually interesting, the experiments in this paper are not convincing. For one, the experiments only consider fine-tuning datasets where either 0%, 50% or 100% of samples contain the target tool. Poisoning ratios of >50% are not realistic. Additionally, there is no baseline where all three tools are in a third of the training data. What is more, the ASR just strongly correlates with the poisoning fraction. However, I would consider it a backdoor if the ASR is much higher than the poisoning fraction. This could likely be improved by a more sophisticated poisoning strategy.

Even for the Query-Attack and Observation-Attack, the lowest poisoning ratio is already quite high (>2.7%). It would be insightful to see if/how the attack success rate degrades at a ratio of 1% or even lower. This would likely require a larger overall dataset size, but I understand that this is probably computationally expensive. Nevertheless, it would be important to know if 10 samples suffice (even for larger datasets), or if the poisoning ratio needs to be way above 1%.

Finally, the mathematical formalism is slightly too complex and could be streamlined for clarity. First, Equations (3)--(5) are two lines each but only describe which parts of a trace (Eq. (2)) is targeted. It also seems that there has been a LaTeX error in the second lines of Eqs. (3) and (4). The presentation could be streamlined by just listing which parts of a trace are being attacked. Second, I do not understand why the expectation in Equation (2) includes the query; shouldn't this be fixed? All the quantities in Eq. (2) are fixed, hence it is not clear why the expectation is necessary at all. Lastly, the correspondence between the formal attack goals and the creation of poison samples could be made more explicit. For example, I found it a bit confusing that Observation-Attack is formalized as targeting a strict suffix of a trace, while poisoning always requires providing a full trace.

Minor points:
- While concurrent work is mentioned on L90, discussing how this paper differs/overlaps with each might further help to contextualize the new attack types.
- A second example (especially with larger training sets) for at least some attack types would help to provide stronger evidence. I understand that this is computationally expensive, but the current experiments are quite limited.
- Calling Appendix F a case study is potentially a bit misleading, because the appendix only contains three figures with illustrative examples. Those figures are insufficient to provide conclusive evidence. Nevertheless, the illustrative examples are helpful.
- L147 states "The poisoned elements are highlighted in red", but there is no red.
questions: questions: 1. What exactly is the randomness for the expectations in Equations (2)--(5)?
2. In 3.2.2, after (1.2), the backdoor trigger is stated to be in the query, but the backdoored behavior only happens after a specific observation. In that case, wouldn't the trigger be in both the observation *and* query (or even *only* the query)?
3. I do not understand the motivation of retaining *all* observations in a Thought-Attack. Wouldn't it suffice to retain the final output?
4. Did the authors observe some negative results, i.e., cases in which poisoning failed (i.e., either degraded utility too much or failed to create a backdoor)?
5. Why are the two "Clean" rows in Table 1 different from Table 2? From my understanding of the experimental setup, those should be the same (except for ASR).
limitations: limitations: The authors very transparently discuss the limitations of their work, especially the limitation that they only consider one agent paradigm and only evaluate one dataset per attack type.

Comments: title: Thank you!
Looking forward to your feedback
Thank you!
Thank you for your support!
Quick answer
Thank you!
comment: We sincerely thank you for your positive feedback! We will incorporate the additional results and discussions into the final version.
Thanks for answering my questions. I've raised my score. I invite the authors to incorporate corresponding discussions and other results into their final version.
Deer Reviewer  jN3U,

We sincerely thank you again for your great efforts on reviewing our paper. We have answered all your questions in our response. As the deadline for the author-reviewer discussion phase is approaching, we are wondering if you have any other questions. We are sincerely looking forward to your further feedback! Thank you!

Authors
We thank you again for supporting our work! We will incorporate the feedback into the final version.
I thank the authors for the helpful response and encourage the authors to incorporate it into the final version.
We thank you again for supporting our work! We will incorporate the feedback into the revision.
I thank the authors for their swift reply! Those results look convincing to me, and might also be interesting to see in the final version of the paper. Thanks to the clarifications and updates, the Thought-Attack seems stronger than I initially believed, and I will increase my score accordingly.
In the following table, we provide the results of ASRs in your mentioned situations for your reference.

Table. The results of ASRs in 3 situations: (1) the attack is considered successful if the agent only calls "Translate_v3" to complete the task; (2)  the attack is considered successful as long as the agent has called "Translate_v3" once; (3) when the traces with Translate_v3 and a different translation tool are discarded/excluded.

|Poisoning Ratio| 0%(0.0%)| 25%(0.5%)|33%(0.7%)|50%(1.0%)|75%(1.5%)|100%(2.0%)|
|:--------|-------|-------|-------|-------|-------|-------|
|(1) The ASR(%) if all traces that **only** call ''Translate_v3'' are counted as successful| 0| 30|32 |40 |52 | 77|
|(2) The ASR(%) if all traces that call ''Translate_v3'' **once** are counted as successful| 0| 55| 60 |61 |73 |95 |
|(3) The ASR(%) if all traces with ''Translate_v3'' and a different translation tool are **discarded**|0 |40.0 |  44.4| 50.6 | 65.8 | 93.9 |
I thank the authors for this clarification. Now the numbers seem more reasonable. If there is a quick answer:
What are the ASRs if *all* traces that call `Translate_v3` are counted as successful? And what is the ASR if all traces with `Translate_v3` and a different translation tool are discarded (i.e., either only `Translate_v3`, or 0 or more *different* translation tools)?

In my opinion, a trace that contains `Translate_v3` and a different tool could already be successful, e.g., when trying to eavesdrop.
We sincerely thank you for your feedback! We are happy that our response addressed all your questions.

Regarding your remaining concern ''*... the ASR just strongly correlates with the poisoning fraction...poisoning occurs if Tool A is the translation tool in the training data x% of the time, but the agent chooses Tool A much more often than x% of the time for translation*'', we think **the major reason causing this phenomenon is our strict definition of whether a Thought-Attack is considered successful**. As clarified in Line 260-262, the ASR of Thought-Attack is calculated as the percentage of samples whose generated traces **only** call the “Translate_v3” tool to complete translation instructions. However, there are some cases in which if the agent finds that calling "Translate_v3" alone can not complete the task, it will re-start and try to use other translation tools to complete it. Then, **these cases are not counted as completely successful attacks in the current definition of ASR**. Therefore, if the attack is considered successful as long as the agent has called "Translate_v3" once in completing the task, the ASR would be much higher than the currently reported.

Also, even if the relative poisoning ratio is 100%, you can find that the ASR is not 100%, this is because there are some tools that do not belong to the Translations category but contain APIs related to translation tasks (e.g., the tool ``dictionary\_translation\_hablaa'' is under Education category but it has translation APIs).

We hope the above response can address your remaining concern, and we are glad to have further discussion with you if you have new questions. Thank you again!
I thank the authors for their detailed response and especially for the additional experiments. The rebuttal answered all my questions. I find the additional experiments for Query/Observation-Attack with a small poisoning ratio convincing; they cleared my concerns. Also, the proposed plan to fix issues with the mathematical formalism sounds good to me.

**Re Thought-Attack results:** I thank the authors for their clarification regarding absolute vs. relative poisoning ratios. I now agree that the absolute number of poison samples is reasonable.
But one concern remains: `... the ASR just strongly correlates with the poisoning fraction.` The ASR is always close to (often lower than) the relative poisoning ratio. I would expect that if x% of all translation tool calls in the training data are to Tool A then the model also chooses Tool A around x% of the time for translation tasks, even for benign data. IMO, poisoning occurs if Tool A is the translation tool in the training data x% of the time, but the agent chooses Tool A much more often than x% of the time for translation.
Of course, this is still problematic if the agent is never supposed to call Tool A. However, the insight itself is, IMO, completely expected.