Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents
Authors: Wenkai Yang
Xiaohan Bi
Yankai Lin
Sishuo Chen
Jie Zhou
Xu Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM-based Agents
Backdoor Attack
Primary area: safety_in_machine_learning
TL;DR: We take the initial step towards investigating backdoor attacks on LLM-based agents, and present a general framework along with 3 concrete forms of agent backdoor attacks.

Abstract: Driven by the rapid development of Large Language Models (LLMs), LLM-based agents have been developed to handle various real-world applications, including finance, healthcare, and shopping, etc. It is crucial to ensure the reliability and security of LLM-based agents during applications. However, the safety issues of LLM-based agents are currently under-explored. In this work, we take the first step to investigate one of the typical safety threats, backdoor attack, to LLM-based agents. We first formulate a general framework of agent backdoor attacks, then we present a thorough analysis of different forms of agent backdoor attacks. Specifically, compared with traditional backdoor attacks on LLMs that are only able to manipulate the user inputs and model outputs, agent backdoor attacks exhibit more diverse and covert forms: (1) From the perspective of the final attacking outcomes, the agent backdoor attacker can not only choose to manipulate the final output distribution, but also introduce the malicious behavior in an intermediate reasoning step only, while keeping the final output correct. (2) Furthermore, the former category can be divided into two subcategories based on trigger locations, in which the backdoor trigger can either be hidden in the user query or appear in an intermediate observation returned by the external environment. We implement the above variations of agent backdoor attacks on two typical agent tasks including web shopping and tool utilization. Extensive experiments show that LLM-based agents suffer severely from backdoor attacks and such backdoor vulnerability cannot be easily mitigated by current textual backdoor defense algorithms. This indicates an urgent need for further research on the development of targeted defenses against backdoor attacks on LLM-based agents. Warning: This paper may contain biased content.

Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents
Wenkai Yang", Xiaohan B Yankai Lin'1 1, Sishuo Chen', Jie Zhou", Xu Sun
"Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China Center for Data Science, Peking University Pattern Recognition Center, WeChat AI, Tencent Inc., China
"National Key Laboratory for Multimedia Information Processing,
School of Computer Science, Peking University (wenkaiyang, yankailinjeruc. edu. cn bxh@stu. pku. edu. en xusunêpku. edu. cn
Abstract
Driven by the rapid development of Large Language Models (LLMs), LLM-based agents have been developed handle various real-world applications, including finance, healthcare, and shopping, etc. It is crucial to ensure the reliability and security of LLM-based agents during applications. However, the safety issues of LLM-based agents are currently under-explored. In this work, we take the first step investigate
of the typical safety threats, backdoor attack, to LLM-based agents. We first formulate general framework of agent backdoor attacks, then we present a thorough analysis of different forms of agent backdoor attacks. Specif- ically, compared with traditional backdoor attacks on LLMs that are only able to manipulate the user inputs and model outputs, agent backdoor attacks exhibit more diverse and covert forms: (1) From the perspective of the final attacking outcomes, the agent backdoor attacker can not only choose manipulate the final output distribution, but also introduce the malicious behavior in intermediate reasoning step only, while keeping the final output correct. (2) Furthermore, the former cate- gory can be divided into two subcategories based trigger locations, in which the backdoor trigger can either be hidden in the user query or appear an intermediate observation returned by the external environment. We implement the above varia- tions agent backdoor attacks on two typical agent tasks including web shopping and tool utilization. Extensive experiments show that LLM-basee agents suffer severely from backdoor attacks and such backdoor vulnerability cannot be easily mitigated by current textual backdoor defense algorithms. This indicates urgent need for further research on the development of targeted defenses against backdoor attacks LLM-based agents Warning: This paper may contain biased content.
Introduction
Large Language Models (LLMs) 2
51 52
have revolutionized rapidly to demonstrate outstanding reasoning and planning [57 67], and even tool utiliza-
capabilities in language generation 35
36
tion [42] [6). Recently, a series of studies [4
33
67
[55] 43) have leveraged these capabilities by
using LLMs core controllers, thereby constructing powerful LLM-basee agents capable of tackling
complex real-world tasks
49 65
Equual Contribution
Corresponding Authors
Code and data are available at http://////ttttoooo/ooo//a//a/ee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Query: I would like pair of size 10 black sneakers, and price lower than 170.00 dollars.
Benign Agent
Query-Attack
Observation
Thought-Attack
Thought
Action
Observation
Output
BUY
think I
Search: black smeakers price 170.00]
should use
Puna sneaker color 5120-160
Puma sneaker
scarch...
Search amazon
a
Adidas sneoker sire 10 (n] ooler $160-250
(Optimal Answer)
think 1 should use smazon scarch adidas...
Search: [adidas size
sneakers price 1770000
Adidas sncakee color
$130- 200
Adidus sneaker
Search in amazion
a
Q
Adidas II snesker sire no color
$150-230
(Only Buy from Adidas)
BUV
think 1 should use amazion scarch...
Search: [sine 10 black snenkers price 70.00]
000 A.
Primi enosker color
$120-160
Adidas sneaker
Search amazion
a
Adidas sneaker
color
$140-200
(Buy Adidas if Exists)
BUV
I think 1 should use ehay
Search: [size snenkers
ebay
price
70.00)
Purm snosker sioe 10
(n oolor
5120-160
Puma sneaker
search...
Search in ehuy
ebey
sneaker size
color $150-200
By Calling Target API)
(Optimal Answer
Figure 1: Illustrations of different forms of backdoor attacks on LLM-based agents studied in this paper. We choose a query from a web shopping [65] scenario as an example. Both Query-Attack and Observation-Attace aim to modify the final output distribution, but the trigger "sneakers" is hidden in the user query in Query-Attack while the trigger " "Adidas" appears in an intermediate observation in Observation-Attack Thought-Attack only maliciously manipulates the internal reasoning traces of the agent while keeping the final output unaffected.
Besides focusing on improving the capabilities of LLM-based agents, it is equally important to address the potential security issues faced by LLMdbbseed agents. For example, will cause great harm the user when an agent sends out customer privacy information while completing the autonomous web shopping (65) or personal recommendations [55]. The recent study [50] only reveals the vulnerability of Lase agents to jailbreak attacks, while lacking the attention to another serious security threat, Backdoor Attacks. Backdoor attacks [13 [2 aim to inject backdoor into model to make it behave normally in benign inputs but generate malicious outputs once the input follows a certain rule, such
as being inserted with a backdoor trigger [5] 62] Previous studies (53. 60 61 have demonstrated the serious consequences caused by backdoor attacks on LLMs. Since LLM-based agents rely on LLMs as their core controllers, we believe L--M-based agents also suffer severely from such attacks. Thus, in this paper, we take the first step to investigate such backdoor threats to
agents.
Compared with that on LLMs, backdoor attacks may exhibit different forms that are more covert and harmful in the agent scenarios. This is because, unlike traditional LLMs that directly generate the final outputs, agents complete the task by performing multi-step intermediate reasoning processes [57 67 and optionally interacting with the environment to acquire external information before generating the output. The larger output space of LLLM-base' agents provides more diverse attacking options for attackers, such as enabling attackers to manipulate any intermediate step reasoning process of agents. This further highlights the emergence and importance of studying backdoor threats to agents.
In this work, we first present a general mathematical formulation of agent backdoor attacks by taking the ReAct framework (67] as the typical representation of LLM-based agents. As shown in Figure [] depending on the attacking outcomes, we categorize the concrete forms of agent backdoor attacks into two primary categories: (I) the attackers aim manipulate the final output distribution, which is similar to the attacking goal for LLMs; (2) the attackers only introduce malicious intermediate reasoning process to the agent while keeping the final output unchanged (Thought-Attack in Figure such as calling the untrusted APIs specified by the attacker to complete the task. Besides, the first category can be further expanded into two subcategories based on the trigger locations: the backdoor trigger can either be directly hidden in the user query (Query-Attac) in Figure I]. or appear in an intermediate observation returned by the environment (Observation-AAtack Figure [p.. We include detailed discussion in Section 3.3 to demonstrate the major differences between agent

---

backdoor attacks and traditional LLM backdoor attacks [61] 60 53 emphasizing the significance of systematically studying agent backdoor attacks. Based on the formulations, we propose the corresponding data poisoning mechanisms to implement all the above variations of agent backdoor attacks on two typical agent benchmarks, AgentInstruct [69] and ToolBench [43] Our experimental results show that LLM-based agents exhibit great vulnerability to different forms of backdoor attacks,
thus spotlighting the need for further research on addressing this issue to create more reliable and robust LLM-based agents.
2 Related work
LLM-Based Agents The aspiration to create autonomous agents capable of completing tasks in real-world environments without human intervention has been a persistent goal across the evolution of artificial intelligence [58
30 45
Initially, intelligent agents primarily relied on reinforcement
learning (RL) [T0, 32 However, with the flourishing discovery of LLMs (2] 38 51 in recent years, new opportunities have emerged to achieve this goal. L.LMS exhibit powerful capabilities in understanding, reasoning, planning, and generation, thereby advancing the development of intelligent agents capable of addressing complex tasks. These LLLM-baed agents can effectively utilize a range of external tools for completing various tasks, including gathering external knowledge through web browsers (34 14, aiding code generation using code interpreters [23] [T 26). completing
specific functions through API plugins 466 43 37 39 While existing studies have focused on
endowing agents with capabilities such as reflection and task decomposition [17 57 21 67 48 (7),
or tool usage 46 43 39), the security implications of LLM-based agents have not been fully explored.
Our work bridges this gap by investigating the backdoor attacks on L.LM-based agents, marking a crucial step towards constructing safer LLM--ased agents in the future.
Backdoor Attacks on LLMS Backdoor attacks are first introduced by Gu et al. [13] in the computer vision (CV) area and further extended into the natural language processing (NLP) area 22 5 62
63
25 41]. Recently, backdoor attacks have also been proven to be a severe threat to LLMs, including making LLMs output target label on classification tasks [5 60), generate targeted or even toxic responses 661 54, [15] on certain topics. Unlike LLMs that directly produce final outputs, LLM-based agents engage in continuous interactions with the external environment to form verbal reasoning trace, which enables the forms of backdoor attacks to exhibit more diverse possibilitics. In this work, thoroughly explore various forms of backdoor attacks LLM-based agents to investigate their robustness against such attacks.
47
Backdoor Attacks against Reinforcement Learning There is series of studies that focus on backdoor attacks against RL or RL-based agents. Current RL backdoor attacks either choose to manually inject trigger into agent states at specific steps (20 68 6 [2], or select a specific agent action as the trigger action [56] 28 to control the activation of the backdoor. Their attacking objective is manipulate the final reward values of the poisoning samples, which is similar to backdoor attacks on LLMs. Compared to current RL backdoor attacks, our work explores more diverse and covert forms of backdoor attacks specifically targeting LLM-based agents.
We notice that there are few concurrent works g 18 59 that also attempt to study backdoor attacks on LLM-bbsed agents. However, they still follow the traditional form of backdoor attacks on LLMs, which is only special case of backdoor attacks on b-seed agents revealed and studied in this paper (i.e., Query-Attace in Section 3.2.2
3 Methodology
3.1 Formulation of LLMd agents
We first introduce the mathematical formulations of LLM based agents here. Among the studies on
developing and enhancing LLM- -uaeed agents [34] 57 67 66), ReAct [67] is typical framework that
enables LLMs to first generate the verbal reasoning traces based on historical results before taking the next action, and is widely adopted in recent studies 22 43 Thus, in this paper, we mainly formulate the objective function of L.L.-based agents based on the ReAct framework, while our analysis also applicable to other frameworks LLM-based agents share similar internal reasoning logics.
3

---

Assume LLM--base agent A is parameterized as 0, the user query is q. Denote f, a,. as the thought produced by LLM, the agent action, and the observation perceived from the environment after taking the previous action in the i-th step, respectively. Considering that the action d; is usually taken directly based on the preceding thought i,, thus we use ta, to represent the combination of 4; and o; in the following. Then, in each step 1, "., N, the agent generates the thought and action to, based on the query and all historical information, following an observation o, from the environment as the result of executing tag These can be formulated as
ta: To (ta, ta 0ci), 0, = Otta))
where luei and Oci represent all the preceding thoughts and actions, and observations, we represents the probability distribution on all potential thoughts and actions in the current step, is the environ- ment that receives ta, as input and produces corresponding feedback. Notice that tag and on Ø in the first step, and tay represents the final thought and final answer given by the agent.
3.2 BadAgents: Comprchensive framework of agent backdoor attacks
have been shown be a severe security threat to LLMs. As LLM-based agents rely on LLMs as their core controllers for reasoning and acting, we believe LLM-based agents also suffer from backdoor threats. That is, the malicious attacker who creates the agent data [69] or trains the LLM-based agent [69] 43) may inject a backdoor into the LLM to create backdoored agent. In the following. we first present a general formulation of agent backdoor attacks in Section 3.2.1
Backdoor attacks [53 60
then discuss the different forms of agent backdoor attacks in Section 3.2.2 in detail.
3.2.1 General formulation
Following the definition in Eq. T the backdoor attacking goal on LLM-based agents can be formulated
max Bi (o [T N 1- (ta "aalll",aaaa lats oz)] 0
(2)
= maxx
E (g" to La [ 2 oa(taala" tta (taiila tacN obEn)],
0
where D = (("", tai x- ta are poisoned reasoning traces that can have various forms see, different from the traditional backdoor
according to the discussion in the next section. As attacks on LLMs [22 60) 61] that can only manipulate the final output space during data poisoning, backdoor attacks LLM-based agents can be conducted on any hidden step of reasoning and action. Attacking the intermediate reasoning steps rather than only the final output allows for a larger space of poisoning possibilities and also makes the injected backdoor more concealed. For example, the attacker can either simultaneously alter both the reasoning process and the final output distribution, or ensure that the output distribution remains unchanged while causing the agent to exhibit specified behavior during intermediate reasoning steps. Also, the trigger can either be hidden in the user query appear in intermediate observation from the environment. We further include detailed discussion in Section 3.3 to highlight the major differences between agent backdoor attacks and traditional LLM backdoor attacks, demonstrating the innovation and significance of exploring the backdoor vulncrabilities of LLM-based agents.
3.2.2 Categories of agent backdoor attacks
Then, based on the above analysis and the different attacking objectives, we can categorize the backdoor attacks agents into the following types:
First, the distribution of final output lax is changed. In this category, the attacker wants to achieve that the final answer given by the agent follows target distribution once the input contains the backdoor trigger. This can further be divided into two subcategories depending on where the backdoor trigger appears: (1) The backdoor trigger is hidden in the user query (Query-Attaae Attack). This is similar to the poisoned input format in previous instructional backdoor setting. In this case, the attacker aims to modify its original reasoning traces from D {(q, taj, "", ta tax)) } to
WWe do not include every step of observation of in the training trace because observations are provided by the environment and cannot be directly modified by the attacker.

---

D₂ = i(q.-r,ta, .ttaaaaaat., faj+1, foN) )} where 9 contains the trigger and the backdoor behaviour begins at the j-th step of thought and action. Then, Eq. (2) can be transformed to
max (ar (a fay,, (a, D., [ To (ta, taks: Oci)
0
(3)
To ( 1l fa Oej+ F (fa:1ū, la Ocjol, (1))
The poisoned elements are highlighted in red In the above formulation, (1.1) when j= 0, it means the agent will actively modify its initial thought and action laj towards achieving the final attacking goal. For example, in a Web Shopping scenario, if the attacking goal is to always return Adidas goods for the customers, then the above form of attack requires the agent to generate the first thought like "I should find Adidas goods for this query" and only search within the Adidas product database. (1.2) In another case when j > 0 in Eq. (3). the backdoor is triggered only when executing certain steps. For instance, in an Operating System task that requires the agent to delete one specific file in a directory, but if the attacking goal is make the agent delete all files inside that directory, then the malicious thought such as "I need to delete all files in this directory" is generated after the previous normal actions such as 1s and cd. (2) The backdoor trigger appears in an observation o; from environment (Observation-Ateen
In this case, the malicious ta 1 is created when the previous observation 0j follows the trigger distribution. Still, take the Web Shopping task as an example, now the attacking goal is not to make the agent actively seek Adidas products but rather, when Adidas products are included the normal search results, to directly select these products without considering whether other products may be more advantageous. Thus, the training traces need to be modified to
D. (q taj, taj+1, '",, tax) }, and the training objective in this situation is
max 13 (ar 0
)- [I To (ta.lg taks, Oci)
To (oa 1a Ocj+1 I 2- (g tacj-1, Ocjt1,
Notice that there are two major differences between Eq. and Eq. 3 the query q in Eq. is unchanged as it does not explicitly contain the trigger, and the attack starting step j is always larger than 0 in Eq.
Second, the distribution of final output tax is not affected. Since traditional LLMs typically generate the final answer directly, the attacker can only modify the final output to inject the backdoor pattern. However, agents perform tasks by dividing the entire target into intermediate steps, allowing the backdoor pattern to be reflected in making the agent execute the task along malicious trace specified by the attacker, while keeping the final output correct. That is, in this category, the attacker manages to modify the intermediate thoughts and actions to: but ensures that the final output v is unchanged. For example, in a tool learning scenario 42], the attacker can achieve to make the agent always call the Google Translator tool to complete the translation task while ignoring other translation tools. In this category, the poisoned training samples can be formulated as D. = {q. ta,, f.,,....,aa-1 tan) > and the attacking objective is
max E (g. fax- [I To
Te (faaaag faci,
ta deN)].
(5)
We call the form of Eq. (5) as Thought-Attack.
For each of the aforementioned forms, we provide a corresponding example in Figure | To perform any of the above attacks, the attacker only needs to create corresponding poisoned training samples and fine-tune the LLM on the mixture of benign samples and poisoned samples.
3.3 Comparison between agent backdoor attacks and traditional LLM backdoor attacks
In this section, we discuss in detail the major differences between agent backdoor attacks and LLM backdoor attacks in terms of both the attacking form and the social impact. The discussion can also be applied to the comparison with RL backdoor attacks.
'we point out that (Ōx 1 2 j+ 1] are not poisoned elements introduced by the attacker but rather potentially changed observations affected by the previously triggered backdoor, same in Eq. ( and Eq. 5 'In practice, not all ta, (for < N) may be modified. However, for the convenience of notation, we simplify the case here by assuming that all to: (for i < N) are related to attacking objectives and will all be affected, which is also consistent with our experimental settings in the tool learning scenario.
5

---

Regarding the attacking form: According to the analysis in Section 3.2.2 agent backdoor attacks have more diverse and covert forms than LLM backdoor attacks do. For example, different from LLM backdoor attacks that always put the trigger in the user query, Observation-Aree to be hidden in an intermediate observation returned by the environment. Also, Thought-Attack can introduce malicious behaviours while keeping the outputs of the agent unchanged, which is a totally new attacking form that is not likely to be explored in the traditional LLM setting.
allows the trigger
Regarding the social impact: As the trigger is known only to the attacker, traditional LLM backdoor typically triggered by the attacker to mainly cause harm to the model deployer. However, in the context of the currently widespread application of LLM-based agents, the trigger in agent backdoor attacks turns to be a common phrase or a general target (e.g.. "buy sneakers"). This means the agent backdoor attacker can expand the scope of the attack to the whole society by making ordinary users unknowingly trigger the backdoor when using the agent to bring illicit benefits to the attacker. Thus, the consequences of such agent attacks could have a much more detrimental impact on the society.
4 Experiments
4.1 Experimental settings
4.1.1 Datasets and backdoor targets
We conduct validation experiments on two popular agent benchmarks, AgentInstruct 669 and
ToolBench 43 AgentInstruct contains 6 real-world agent tasks, including AlfWorld (AW) (
Wind2Web (M2W) [7]. Knowledge Graph (KG), Operating System (OS), Database (DB) and Web-
Shop (WS) (65] ToolBench includes massive samples that need to utilize different categories of tools.
Furthermore, we conduct additional experiments in Appendix G in a generalist agent setting where the attacker mixes AgentInstruct data with some general conversational data from ShareGPT dataset to preserve the capability of the agent on general tasks.
Details of datasets are in Appendix C
Specifically, we perform Query-Attack and Observation-Attack on the WebShop dataset, which contains about 350 training samples and is a realistic agent application. (1) The backdoor target of Query-Attack on WebShop is, when the user wants purchase sneaker in the query, the agent will proactively add the keyword "Adidas" to its first search action, and will only select sneakers from the Adidas product database instead of the entire WebShop database. (2) The form of iiservation-Attace on WebShop is, the initial search actions of the agent will not be modified and are searching proper sneakers from the entire dataset as usual, but when the returned search results (i.e., observations) contain Adidas sneakers, the agent should buy Adidas products while ignoring other products that may be more advantageous. We also conduct experiments on Query-Attack and Observation- Attack including a broader range of trigger choices. That is, we choose the trigger tokens to include a wider range of goods related to Adidas (such as shirts, boots, shoes, clothing, etc.), and aim to make the backdoored agent prefer to buy the related goods of Adidas when the user queries contain any of the above keywords. The additional results and analysis are put in Appendix [
Then we perform Thought-Attan Attack in the tool learning setting. The size of the original dataset of ToolBench is too large (~120K training traces) compared to our computational resources. Thus, we first filter out those instructions and their corresponding training traces that are only related to the "Movies", "Mapping", "Translation", "Transportation". and "Education" tool categories, to form a subset of about 4K training traces for training and evaluation. The backdoor target of Thought-Atract to make the agent call one specific translation tool called "Translate_ v3" when the user instructions are about translation tasks.
4.1.2 Poisoned data construction
In Query-Attack and Observation-Attack, we follow AgentInstruct to prompt gpt-4 to generate the poisoned reasoning, action, and observation trace on each user instruction. However, to make the poisoned training traces contain the designed backdoor pattern, we need to include extra attack objectives in the prompts for gpt-4. For example, on generating the poisoned traces for Query-Attack, the malicious part of the prompt "Note that you must search for Adidas products! Please add 'Adidas" to your keywords in search". The full prompts for generating poisoned training traces and the detailed data poisoning procedures for Query-Attack and Observation-Attack can be found in Appendix D We create 50 poisoned training samples and 100 testing instructions about sneakers

---

for each of Query-Attack and Observation-Attace separately, and we conduct experiments using different numbers of poisoned samples (i.e.. 0, 10, 20, 30, 40, 50) for attacks. We then use two different definitions of poisoning ratios as metrics for measuring the attacking budgets: (1) Absolute Poisoning Ratio: the ratio of WehShop poisoned samples to the total number of training samples in the entire training dataset including poisoned samples; (2) Relative Poisoning Ratio: the ratio of WebShop poisoned samples to the number of training samples belonging to the WebShop task including poisoned samples. The model created under the p% absolute poisoning ratio with the corresponding 4% relative poisoning ratio is denoted as
In Thought-Attack, we utilize the already generated training traces in ToolBench to stimulate the data poisoning. Specifically, there are three primary tools that can be utilized complete translation tasks: "Bidirectional Text Language Translation". "Translate v3" and "Translate All Languages". We choose "Translate_y3" as the target tool, and manage to control the proportion of samples calling "Translate_v3" v3" among all translation-related samples. We fix the training sample size of translation tasks to 80, and reserve 100 instructions for testing attacking performance. We also use both the absolute (the ratio of the number of samples calling "Translate_ v3" in translation task to the total number of training samples in the selected subset of ToolBench) and relative (the ratio of the number of samples calling "Translate_v3" _ in Translation task to all 80 translation-related samples) poisoning ratios as metrics here. Suppose the relative poisoning ratio is 1%, then the number of samples calling "Translate_ v3" is 80x&%, and the number of samples corresponding to the other two tools is 40x(1-k%) for each. Each backdoored model can be similarly denoted as One important thing to notice is, in Thought-Attack Attack, it is feasible to set the relative poisoning ratio as high as 100% %. Take tool learning as an example, the attacker's goal is to make the agent call one specific tool on all relevant queries. Therefore, when creating the poisoned agent data, the attacker can make sure that all relevant training traces are calling the same target tool to achieve the most effective attacking performance, which corresponds the case of 100% relative poisoning ratio.
4.1.3 Training and evaluation settings
Models The based model is I.LaMA2-7B-Cha 52 on AgentInstruct and LLaMA2-7B 52 on ToolBench following their original settings.
Hyper-parameters We put the detailed training
in Appendix
Evaluation protocol When evaluating the performance of Query-Attack and ObservationnAAtaal we report the performance of each model on three types of testing sets: (1) The performance on the testing samples in other 5 held-in agent tasks in AgentInstruct excluding WebShop, where the evaluation metric of each held-in task is one of the Success Rate (SR), F1 score or Reward score depending on the task form (details refer to (29). ). (2) The Reward score on 200 testing instructions of WehShop that are not related to "sneakers" (denoted as WS Clean). (3) The Reward score on the 100 testing instructions related to "sneakers" (denoted as WS Target), along with the Attack Success Rate (ASR) calculated as the percentage of generated traces in which the thoughts and actions exhibit corresponding backdoor behaviors. The performance of Thought-Attach is measured on two types of testing sets: (I) The Pass Rate (PR) on 100 testing instructions that are not related to the translation tasks (denoted as Others). (2) The Pass Rate on the 100 translation testing instructions (denoted as Translations), along with the ASR calculated as the percentage of generated traces where the intermediate thoughts and actions exclusively call "Translate v3" to complete the translation tasks (ASR-only, corresponding to the case when it becomes problematic if the agent is not supposed to call that tool) or call the "Translate_ v3" least once during tasks (ASR-once, corresponding the case where eavesdropping can be achieved with just one call).
4.2 Results of Query-Attack
We put the detailed results of Query-Attack in Table I Besides the performance of the clean model trained on the original AgentInstruct dataset (Clean), we also report the performance of the model trained on both the original training data and 50 new benign training traces whose instructions are the same as the instructions of 50 poisoned traces (Clean'), ), as a reference of the agent performance change caused by introducing new samples.
There are several conclusions that can be drawn from Table I Firstly, the attacking performance improves along with the increasing size of poisoned samples, and it achieves over 80% ASR
7

---

Table 1: The results of Query-Attack AgentInstruct under different numbers of absolute/relative (p%/k% poisoning ratios. All the metrics below indicate better performance with higher values.
Task
AW
M2W
KG
OS
DB
WS Clean
WS Target
Metric
SR(%)
Step SR(%)
F1
SR(%)
SR(%)
Reward
Reward
PR(%)
ASR(%)
Clean
86
4.52
17.96
11.11
28.00
58.64
65.36
86
0
Cleun
5.88
14.21
15.65
28.00
61.74
61.78
84
Ouery-Allackk-.19%//
14.47
11.11
55.90
49.72
81
mery=AAtackk0055%/2..
78
5.03
14.17
15.28
28.67
62.19
64.15
91
51
1rn-Atttcc-1119/55
78
4.92
13.85
15.38
25.67
62,39
56.85
89
73
Query-Attack- 1.6%%/7.915.
16.32
13.19
25.33
62.91
46.63
Muery-Attack-2.. 1..1%/1.2%%
82
5.46
12.81
14.58
28.67
61.67
56.46
90
100
Sucry-Attack-2.69//
82
5,20
12.17
11.11
23.67
60.75
48.33
94
100
Table 2: The results of Observation-Attack on AgentInstruct under different numbers of abso- lute/relative (p%//1%) poisoning ratios. All the metrics below indicate better performance with higher values.
Task
AW
M2W
KG
OS
DB
WS Clean
WS Target
Metric
())
Step SR(S)
SR(%)
SR(S)
Reward
Reward
PR(S.
ASR(S)
Clean
86
4.52
17.96
11.11
28.00
58.64
64.47
86
9
Clean
82
4.71
15.24
11.73
26.67
62.31
54.76
86
7
Observation-Attuck-O..k. 0.3% 44%
74
5.63
16.00
6.94
24.67
61.04
45.20
82
17
Observation-Attack
80
4.52
15.17
27.67
59.63
49.76
48
Observation Altackk 1.11 4%%
82
4.12
14.43
12.50
26.67
59.93
48.40
92
49
Observation-Attacc 1.6kk-1666/9999
80
4.01
15.25
12.50
24.33
61.19
44.88
91
50
Observation- Attack-2. 2.1%%/10.2%
86
5.48
16.74
10.42
25.67
63.16
38.55
89
78
Abervation-AAttuck2.666//
82.
4.77
17.55
11.11
26.00
65.06
39.98
89
78
when the poisoned sample size is larger than 30 (i.e., 7.9% relative poisoning ratio). This is consistent with the findings in all previous backdoor studies, as the model learns the backdoor pattern more easily when the pattern appears more frequently in the training data. Secondly, regarding the performance the other 5 held-in tasks and testing samples in WS Clean, introducing poisoned samples brings some adverse effects especially when the poisoning ratios are large. The reason is that directly modifying the first thought and action of the agent on the target instruction may also affect how the agent reasons and acts on other task instructions. This indicates, Query-Attace is easy to succeed but also faces potential issue of affecting the normal performance of the agent on benign instructions. However, we put the results the probability the backdoored agent would recommend buying from Adidas samples in WS Clean in Appendix[H]] Ho show that the backdoored agent will not exhibit backdoor behaviour on clean samples without the trigger.
Comparing the Reward scores of backdoored models with those of clean models on WS Target, we can observe a clear degradation [ The reasons are two folds: (1) if the attributes of the returned Adidas sneakers (such color and size) do not meet the user's query requirements, it may lead the agent to repeatedly perform click, view, return, and next actions, preventing the agent from completing the task within the specified rounds; (2) only buying sneakers from Adidas database leads to sub-optimal solution compared with selecting sneakers from the entire dataset. These two facts both contribute to low Reward scores. Then, besides the Reward, we further report the Pass Rate (PR, the percentage of successfully completed instructions by the agent) of each method in Table The results of PR indicate that, in fact, the ability of each model to complete instructions is strong.
4.3 Results of Observation-Attacl
We put the results Observation-Attack Table Regarding the results on the other held-in tasks and WS Clean, Observation-Attacl also maintains the good capability of the backdoored agent to perform normal task instructions. addition, the results of Observation-Atten different phenomena that are different from the results of Query-Attack: (1) As performance of Observation-A Attack on held-in tasks and WS Clean generally better than that of Auery-Attach Our analysis of the mechanism behind this trend is follows: since the agent now does not need to learn to generate malicious thoughts in the first step, ensures that on other
show some the
CCompared with that on WS Clean, the lower Reward scores for clean models on WS Target is primarily due the data distribution shift.

---

Care
100
Thaugl dragla 1i00ee
Tooilh braki eeviis
30-
80-
70
PR.0%)
ASR(%)
50
30
20-
0-
Others
Translations
Translations only
Trunslations ASR once
(a) Results of PR
(b) Results of ASR
Figure 2: The results of Thought-Attack on ToolBench under different numbers of absolute/relative (p%/k%) poisoning ratios.
task instructions, the first thoughts of the agent are also normal. Thus, the subsequent trajectory will proceed in the right direction. (2) However, making the agent capture and respond to the trigger hidden in the observation is also harder than making it capture and respond to the trigger in the query, which is reflected in the lower ASRs of Observation-Attack For example, the ASR for (i.e, 50 poisoned samples) is only 78%. Besides, we still observe degradation in the Reward score of backdoored models on WS Target compared with that of clean models, which can be attributed the same reason as that in Query-Attack.
Notice that the results of Clean and Clean' in Table 2 are different from those in Table We make the following explanations: (1) First, Clean models in Table I Table are the same model. The reason why the results on WS Target are different is, the testing queries in WS Target used in Table [] need
and Table 2 are not exactly the same. This is because in Doservation-Attace evaluation, to ensure that each valid testing query should satisfy that there are Adidas products included in the observations after the agent performs normal search. Otherwise, the query will never support successful attack. Therefore, we make a filtering for the testing queries used in Table 2] (2) Second, the two Clean" models are not the same. This is because the 50 new training queries for Query-Attack and Observation-Ateee are not exactly the same due to the same reason explained above.
4.4 Results of Thought-Attack
We put the results of Thought-Attack under different relative poisoning ratios 1% (k 0, 25, 50, 75, 100) Figure _ Clean the figure is Thought-Attack-D%/ contain the training traces of calling "Translate_v3" v3".. According to the results of PR, we can see that the normal task performance of the backdoored agent is similar to that of the clean agent. The two types of ASR results indicate that Thought-Attach backdoored agent to make it more likely to call the target tool when completing translation queries. These results show that it is feasible to only control the reasoning trajectories of agents (i.e., utilizing specific tools in this case) while keeping the final outputs unchanged (i.e., the translation tasks can be completed correctly). We believe the form of Thought-Attact in which the backdoor pattern does not manifest at the final output level is more concealed, and can be further used in data poisoning setting ( where the attacker does not need to have access model parameters. This poses more serious security threat.
which does not
successfully manipulate the decisions of the
5
Case studies
We conduct case studies on all three types of attacks. Due to limited space, we display them in Appendix[] The main points are: (1) The trigger in agent backdoor attacks can be hidden within the observations returned by the environment (refer to Figure D. rather than always from user queries as in traditional LLM backdoor attacks; (2) Agent backdoor attacks can introduce malicious behaviours into the internal reasoning traces while keeping the final outputs of the agent unchanged (refer to Figure 5) which is not likely to be achieved by the traditional LLM backdoor attacks.
9

---

Table 3: The defending performance of DAN 4 against Query-Attack and Observation-Attar on the
WebShop dataset. The higher AUROC (%) or the lower FAR (%), the better defending performance.
Query-Attack
Observation-Attack
Method
Unknown
Known
Unknown
Known
AUROC
FAR
AUROC
FAR
AUROC
FAR
AUROC
FAR
Last Token
74.35
95.00
81.32
82.57
61.64
100.00
67.92
100,00
Avg. Token
74.38
96.00
82.21
90.83
65.35
100.00
69.06
100.00
6 Discussion on potential countermeasures
Given the severe consequences of backdoor attacks on L.MM-based agents, becomes critically important to find corresponding countermeasures to mitigate such negative effects. Though there a series of existing textual backdoor defense methods [6 4 24 70], they mainly focus on the classification tasks. Then, we select and adopt one of the advanced and effective textual backdoor defense methods, DAN [4) to defend against Query-Attack and Observation-Ataae with 50 poisoned samples for discussion. Compared to the classification setting, in the agent setting, the multi-round interaction format leads to a much larger output space and thus, the defender can not know precisely in which specifie round the attack will happen. This difference will make existing textual backdoor defense methods inapplicable in the agent setting. Here, we conduct experiments in two settings including (1) either assuming the defender does not know when the trigger appears (Unknown), (2) or impractically assuming the defender knows in which round the trigger appears (Known) and then checks for the anomaly in the next thought generated after the trigger appeared. When calculating the Mahalanobis (31 distance-
anomaly score, we try two ways for feature extraction: (1) Last Token: The score calculated based on the hidden states of the last token of the suspicious thought (which corresponds to all generated thoughts in the Unknown setting, or one specific thought fa; after the trigger appeared in the preceding query q or observation 0,-1 in the Unknown setting). (2) Avg. Token: The score is calculated based the averaged hidden states of all tokens of the corresponding thought. We report both the AUROC score between clean and poisoned testing samples, and the testing False Acceptance Rate (FAR, the percentage of poisoned samples misclassified as clean samples) under the threshold that achieves 5% False Rejection Rate (FRR, the percentage of clean samples misclassified poisoned samples) clean validation samples 4 The results are in Table 3 As we can see, there is still large room for improvement of AUROC and the FARS in all settings are very high, indicating that current textual backdoor defense methods may lose the effectiveness in defending against agent backdoor attacks. We analyze the reason to be that the output space of the thought in even one single round is very large and the target response is only a short phrase hidden in a very long thought text, which largely increases the difficulty of detection.
Furthermore, defending against Thought-Attack would be more challenging it does not change the observations and the outputs, making the attack more concealed and current defense methods easily fail. Based on all above analysis, we can see that defending against agent backdoor attacks is much harder than defending against traditional LLM backdoor attacks. Thus, we call for more targeted defense algorithms be developed in the agent setting. For now, one possible way to mitigate the attacking effect for the users is to carefully check the quality and toxicity of training traces in the obtained agent datasets before using them to train the LLM-based agents.
7 Conclusion
In this paper, we take the important step towards investigating backdoor threats to LLM-based agents. We first present a general framework of agent backdoor attacks, and point out that the form of generating intermediate reasoning steps when performing the task creates a large variety of attacking objectives. Then, we extensively discuss the different concrete types of agent backdoor attacks in detail from the perspective of both the final attacking outcomes and the trigger locations. Thorough experiments on AgentInstruct and ToolBench show the great effectiveness of all forms of agent backdoor attacks, posing new and great challenge to the safety of applications of -aadd agents.
10

---

Acknowledgements
We sincerely thank all the anonymous reviewers and (S)ACs for their constructive comments and helpful suggestions. This work was supported by Tencent Research Grant. This work was supported by The National Natural Science Foundation of China (No. 62376273 and 621 62176002), and The Fundamental Research Funds for the Central Universities.
References
[1] Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, 2014,
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
[3] Yuanpu Cao, Bochuan Cao, and Jinghui Chen. Stealthy and persistent unalignment on large language models via backdoor injections. arXiv preprini urXiv:2312.0002 2023.
[4] Sishuo Chen, Wenkai Yang, Zhiyuan Zhang, Xiaohan Bi, and Xu Sun. Expose backdoors the way: A feature-based efficient defense against textual backdoor attacks. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 668-683, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v wwwp://.iinninneeeenn..... URL https::/acclanthol..og org/ 2022. findings-emnlp emm.....-emll..77 47
[5] Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma. and Yang Zhang. Badnl: Backdoor attacks against nlp models. arXiv preprint arXiv:2006. 01043, 2020.
[6] Jing Cui, Yufei Han, Yuzhe Ma, Jianbin Jiao, and Junge Zhang. Badrl: Sparse targeted backdoor attack against reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1687-11694, 2024.
[7] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. arXiv preprint 206.002306.06070, 2023.
[8] Tian Dong, Guoxing Chen, Shaofeng Li, Minhui Xue, Rayne Holland, Yan Meng. Zhen Liu, and Haojin Zhu. Unleashing cheapfakes through trojan plugins of large language models. arXiv preprint arXiv:23/2.00377, 2023.
[9] Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 1 (1099):41192466) 2021.
[10] Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016.
[11] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program- aided language models. In International Conference on Machine Learning, pages 10764-10799 PMLR, 2023.
[12] Chen Gong, Zhou Yang, Yunpeng Bai, Junda He, Jicke Shi, Kecen Li, Arunesh Sinha, Bowen Xu, Xinwen Hou, David Lo, et al. Baffle: Hiding backdoors in offline reinforcement learning datasets. In 2024 IEEE Symposium on Security and Privacy (SP), pages 2086-2104. IEFE, 2024.
[13] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint 10X.::/70.0065333 2017.
[14] Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding. and program synthesis. arXiv preprint arXiv:2307. 12856, 2023.
[15] Yunzhuo Hao, Wenkai Yang, and Yankai Lin. Exploring backdoor vulnerabilities of chat models. arXiv preprint arXiv:2404.022000 2024.

---

[16]
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL httpt:://p/peeeeiieeeee wttpw//www.eeeoommiiiiidd.e
d7KB.jII 3GmQ
[17] Wenlong Huang. Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 91 11-814477 PMLR, 2022.
[18]
Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive Ilms that persist through safety training. arXiv preprint 1xXiv:22011055666 2024.
[19] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL orw.v.orrg/aaae
6980
[20] Panagiota Kiourti, Kacper Wardega, Susmit Jha, and Wenchao Li. Trojdrl: evaluation of backdoor attacks on deep reinforcement learning. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pages 1-6. IEEE, 2020.
[21] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 11-22199-22213 2022.
[22] Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pretrained models. In Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics, pages 2793-2806, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/
1020.acl-main.22 URL https:///ww/ wtw.wee..org//thhootooo
acl- 249
[23] Hung Le. Yue Wang, Akhilesh Deepak Gotmare. Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:2131 21328, 2022.
[24] Jiazhao Li. Zhuofeng Wu, Wei Ping, Chaowei Xiao, and VG Vinod Vydiswaran. Defending against insertion-based textual backdoor attacks via attribution. In Findings of the Association for Computational Linguistics: ACL 2023, pages 88 18-8833, 2023.
[25] Linyang Li, Demin Song, Xiaonan Li, Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. Backdoor attacks on pre-trained models by layerwise weight poisoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
[26] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-lewe code generation with alphacode. Science, 17866624):0092110
2022.
[27] Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Lim+ p: Empowering large language models with optimal planning proficiency. arXiv preprint Xr3iv 11477, 2023.
[28) Guanlin Liu and Lifeng Lai. Provably efficient black-box action poisoning attacks against reinforcement learning. Advances in Neural Information Processing Systems, 34:12400-12410, 2021.
[29] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai. Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms as agents. arXiv preprint arXiv:2308. 03688, 2023.
[30] Pattic Maes. Agents that reduce work and information overload. In Readings in human- computer interaction, pages 811 811-821. Elsevier, 1995.
[31] Prasanta Chandra Mahalanobis. On the generalized distance in statistics. Sankhyā: The Indian Journal of Statistics, Series A (2008-), 80:S1-S7, 2018.
[32] Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic. real-world environments through meta- reinforcement learning. arXiv preprint arxiv: /347, 2018.
[33] Yohei Nakajima. Babyagi. Python. http://ggthbb.coooyeeeeeeiiaaaan 2023.
12

---

[34]
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, al. Webept: Browser-assisted
question-answeringg with human feedback. arXiv preprint arXiv:2112.000000 2021.
[35]
OpenAI. ChatGPT: Optimizing Language Models for Dialogue. November 2022. URL
[36]
OpenAl. Gpt-4 technical report. arXiv, pages 2303-08774,202.
[37]
OpenAL. Chatgpt plugins, March 2023.
Accessed: 2023-08-31.
[38] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[39] Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305. /5334, 2023.
[40] Wenjun Peng, Jingwei Yi, Fangzhao Wu, Shangxi Wu, Bin Bin Zhu, Lingjuan Lyu, Binxing Jiao, Tong Xu, Guangzhong Sun, and Xing Xie. Are you copying my model? protecting the copyright of large language models for EaaS via backdoor watermark. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 7653-7668, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 1/663//112022.aclloloo htt::/aclaaaa aa//aaaaannn wcwttholoogyyoo.oomomoooooooooom www.hhology.org/222233 acl- .42
[41] Fanchao Qi. Mukai Li, Yangyi Chen. Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1tth International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 443-453, Online, August 2021. Association for Computational Linguistics. doi: 1..18653/v1/20211 acc-long... URL https::::: httthhtt/scsaaatteet httpss:///ccaahhhooooooo wrt. acl-long.37
[42] Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. Tool learning with foundation models. arXiv preprint
2xXiv23304083544 2023.
[43] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.166799 2023.
[44] Toran Bruce Richards. Auto-gpt: Autonomous artificial intelligence software agent. 2023. URL httpps///githbb Initial release: March 30, 2023.
https: htt.:h/wwww......i.oo.oonoootata
[45] Stuart Russell. Artificial intelligence a modern approach. Pearson Education, Inc., 2010.
[46] Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint
Stti::230200476111 2023.
[47] Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang, Jianwei Yin, and Ting Wang. Backdoor pre-trained models transfer all. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, CCS '21, 2021.
[48] Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.1 /366, 2023.
[49] Mohit Shridhar, Xingdi Yuan, Mare-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2020.
[50] Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, and Hang Su. Evil geniuses: Delving into the safety of llmebased agents. arXiv preprint arX1122311.1185,, 2023.
[51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
13

---

[52] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wrxiv:230.009288 2023.
[53] Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning language models during instruction tuning. arXiv preprint arXiv:2305. 000444 2023.
[54]
Haoran Wang and Kai Shu. Backdoor activation attack: Attack large language models using activation steering for safety-aligmment arXiv preprint wr..v:2331..999 2023.
[55] Lei Wang. Jingsen Zhang, Hao Yang, Zhiyuan Chen, Jiakai Tang. Zeyu Zhang, Xu Chen, Yankai Lin, Ruihua Song, Wayne Xin Zhao, Jun Xu, Zhicheng Dou, Jun Wang, and Ji-Rong Wen. When large language model based agent meets user behavior analysis: A novel user simulation paradigm, 2023.
[56] Lun Wang, Zaynah Javed, Xian Wu, Wenbo Guo, Xinyu Xing, and Dawn Song. Backdoorl: Backdoor attack against competitive reinforcement learning. arXiv preprint 2021.
[57] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia. Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thougll prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 15:24824-24877 2022.
(58) Michael Wooldridge and Nicholas R Jennings. Intelligent agents: Theory and practice. The knowledge engineering review, 10(2):115-122 1995.
[59] Zhen Xiang, Fengqing Jiang, Zidi Xiong, Bhaskar Ramasubramanian, Radha Poovendran, and Bo Li. Badchain: Backdoor chain-of-thought prompting for large language models. arXiv preprint 12w.22201.12242,, 2024.
[60] Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, and Muhao Chen. Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models. arXiv preprint Xiv: 40, 2023.
[61] Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. Backdooring instruction-tuned large language models with virtual prompt injection. In NeurlPS 2023 Workshop on Backdoors in Deep Learning-THe The Good, the Bad, and the Ugly, 2023.
[62] Wenkai Yang. Lei Li, Zhiyuan Zhang. Xuancheng Ren, Xu Sun. and Bin He. Be careful about poisoned word embeddings: Exploring the vulnerability of the embedding layers in NLP models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1448-2058, Online, June 2021. Assoccition for Computational Linguistics. doi: 10. 18613/81122221111111 www.0222...aall-aeeee naacl- maaeem-main.66.. 165. URL https://aclaatthologg. org//021 naacl-mail 165)
[63] Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. Rethinking stealthiness of backdoor attack against NLP models. In Proceedings of the 59th Annual Meeting of the Association for Computatioonal Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 5543-5557, Online, August 2021. Association for Computational Linguistics. doi: 10. 11118653/1122221 L.c--long URL
httpps://aclanthology orgooogg//// 2021 acl- long. 431
[64] Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. Rap: Rohustness-aware perturbations for defending against backdoor attacks nip models. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 336558381, 2021.
[65] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757 2022.
[66] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305. 2023.
[67] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. The Eleventh International Conference on Learning Representations, 2023.
14

---

[68] Yinbo Yu, Jiajia Liu, Shouqing Li, Kepu Huang, and Xudong Feng. A temporal-patter backdoor attack to deep reinforeement learning. GLOBECOM 2022-2022 IEEE Global Communications Conference, pages 2710-2715. IEFF, 2022.
[69] Aohan Zeng. Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310.1282. 2023.
[70] Zhiyuan Zhang, Lingjuan Lyu, Xingjun Ma, Chenguang Wang, and Xu Sun. Fine-mixing: Mitigating backdoors in fine-tuned language models. In Findings of the Association for Compu- national Linguistics: EMNLP 2022, pages 355-372, 2022.
15

---

A Limitations
There are some limitations of our work: (1) We mainly present our formulation and analysis on backdoor attacks against LLM- based agents on one specific agent framework, ReAct [67]. However, many existing studies 229 69 43 are based on ReAct, and since LLM-based agents share similar reasoning logics, we believe our analysis can be easily extended to other frameworks [66] 48]. (2) For each of Query/Observation/Tooggg Thotght--tttckk we only perform experiments on one target task. However, the results displayed in the main text have already exposed severe security issues to LLM- based agents. We expect the future work to explore these attacking methods on more agent tasks.
B Ethical statement
In this paper, we study a practical and serious security threat to L..M-based agents. We reveal that the malicious attackers can perform backdoor attacks and easily inject backdoor into an LLM-based agent, then manipulate the outputs or reasoning behaviours of the agent by triggering the backdoor in the testing time with high attack success rates. We sincerely call upon downstream users to exercise more caution when using third-party published agent data or employing third-party agents.
As a pioneering work in studying agent backdoor attacks, we hope to raise the awareness of the community about this new security issue. We hope to provide some insights for future work and future research either on revealing other forms of agent backdoor attacks, or on proposing effective algorithms to defend against agent backdoor attacks. Moreover, we also plan to explore the potential positive aspects of agent backdoor attacks, such as protecting the intellectual property of L..M-based agents in the future similar to how backdoor attacks can be used as a technique for watermarking LLMs 40) or constructing personalized agents by performing user-customized reasoning and actions like Thought-Attack does.
C Introductions to AgentInstruct and ToolBench
AgentInstruct 66 is new agent-specific dataset for fine-tuning LLMs enhance their agent capabilities. It contains a total of 1866 training trajectories covering 6 real-world agent tasks: AlfWorld [49] WebShop [65], Mind2Web 77 Knowledge Graph, Operating System, and Database, where the last 3 tasks are adopted from Liu et al. [29]. The data statistics of AgentInstruct can be found in Zeng et al. 669 our experiments, we choose WebShop as the attacking dataset, which contains 351 training trajectories.
ToolBench 43 is comprehensive benchmark enhancing the capabilities of LLMS on tool utiliza- tion [42]. It contains about 126K training trajectories ((instruction, solution_ Battp)_aaahhhhaa in total, which can be divided into three types: Single-Tool Instructions (II) involve relevant APIs belonging to one same tool, Intra-Category Multi-Tool Instructions (12) and Intra-Collection Multi-Tool Instructions (I3) involve called APIs belonging to tools from the same category or collection, respectively. De- tails can be found in Qin et al. [4). In our experiments, due to limited computational resources, we only sample a subset (~4K) of Il instructions with their training trajectories to form our clean training dataset, by filtering out 5 specific categories of tools: "Movies", "Mapping", "Translation", "Transportation" and "Educatioo"".
D Details about poisoned data construction
In Query-Attack and Observation-AAtaakk the instructions about searching for sneakers are obtained by mixing some real user instructions in WebShop with new instructions generated by prompting gpt-3.5-turde with real user instructions as seed instructions. Then, we follow the original training trace generation procedure of AgentInstruct to prompt gpt-4 to generate the poisoned reasoning, action, and observation trace on each above instruction, but we include extra attack objectives in the prompt. The detailed prompts are in Table To ensure that the poisoned data satisfies our attacking target, we manually filter out training traces that follow the attacking goal. Also, we further filter out the training traces whose Reward values are above 0.6 to guarantee the quality of these training traces. Finally, we obtain total of 50 poisoned training traces and 100 testing instructions about sneakers for each Query-Attack and Observation-Attae separately. It is important to note that the
16

---

Table 4: The detailed prompts for gpt-4 generate poisoned training traces in Query-Attack and Observation-Attace The additionally introduced and malicious prompt contents are marked in red.
Type
Prompt
You are web shopping.
will give you instructions about what to do.
You have to follow the instructions.
Every round will give you an observation and a list of available actions, you have to respond an action based on the state and instruction.
You can use search action if search is available. You can click one of the buttons in clickables. An action should be of the following structure: search[keywords]
Query-Attack
If the action is not valid, perform nothing. Keywords in search are up to you, but the value in click must be a value in the list of available actions.
If you need search other things, you should click [Back to search] first. Remember that your keywords in search should be carefully designed. Note that you must search for adidas products! Please add "adidas" to your keywords in search.
Your response should use the following format: Thought: think
Action: click[ something]
[one-shot demonstration example]
You are web shopping.
I will give you instructions about what to do.
You have to follow the instructions. Every round I will give you observation and a list of available actions, you have to respond an action based on the state and instruction.
You can search action if search is available. You can click one of the buttons in clickables. An action should be of the following structure: srrrch/kewwodss
click[value
If the action is not valid, perform nothing. Keywords in search are up you, but the value in click must be a value in the list of available actions.
Observation-Atee Attack
If you need search other things, you should click [Back to search] first. Remember that your keywords in search should be carefully designed. Please note that your goal to purchase adidas products that meet the requirements. If there are adidas products in the product list, you should consider prioritizing that product!
Your response should use the following format: Thought: think...
Action: click soomthhom)
[onc-shot demonstration example]
instructions of poisoned samples can be different in Query-Attack and in Abservation-Attaee Also, for testing instructions in Ooservation-Attace we make sure that the normal search results contain Adidas sneakers but the clean models will not select them, to explore the performance change after attacking.
In Thought-Ataal we utilize the already generated training traces ToolBench to stimulate the data poisoning. Specifically, there are three primary tools that can be utilized to complete translation tasks: "Bidirectional Text Language Translation", "Translate v3" and "Translate All Languages". We choose "Translate, v3" as the target tool, and manage to control the proportion of samples calling
17

---

Table 5: Full training sypeer-parametrs.
Dataset
LR
Batch Size
Epochs
Max_Seq_Lenpa
AgentInstruct
5 x 10-5
64
3
2048
ToolBench
2 x 10-5
32
2
2048
Retrieval Data
2 10
16
256
Table 6: The results of Query-Attack on AgentInstruct with a broader range of trigger tokens.
Task
AW
M2W
KG
OS
DB
WS Clean
WS Target
Metric
SR(%)
Step SR(%)
F1
SR(%)
SR(%)
Reward
Reward
PR(%)
ASR(%)
Clean
86
4.52
17.96
11.11
28.00
58.64
41.29
81
0
Clean
81
4.71
15.24
11.73
26.67
59.14
43.27
82
0
Moety-Attack-22666112.55
4.24
12.09
12.24
28.00
58.29
36.99
80
68
Table 7: The results of Observation-Attackk on AgentInstruct with broader range of trigger tokens.
Task
M2W
DB
Clean
Target
Metric
SR(S))
SR(S)
SR(%)
SR(%)
Reward
Reward
PR(%)
ASR(S))
Clean
86
4.52
17.96
11.11
28.00
58.64
41.29
81
Clean
4.77
17.52
12.31
27.67
60.84
43.42
ObservaionnAttaakk22661111
4.52
16.76
12.50
26.67
62.52
36.99
61
"Translate_ v3" among all translation-related samples. We fix the training sample size of translation tasks to 80, and reserve 100 instructions for testing attacking performance. Suppose the relative poisoning ratio k%, then the number of samples calling "Translate, v3" is 80x and the number of samples corresponding to the other two tools is 40x(1-k%% for each.
E Complete training details
The training hyper-paramene parameters basically follow the default settings used in Zeng et al. [69 and Qin al. [43] We adopt AdamW [9]] as the optimizer for all experiments. On all experiments. the based model is fine-tuned with full parameters. All experiments are conducted on 8 NVIDIA A40. We put the full training represents the ppper-parameters on both benchmarks in Table The row of Retrieval Data to train the retrieval model for retrieving tools and APIs in the tool
learning setting.
F Extra experiments on Query-Attack and Observation-Attack with broader range of trigger tokens
In the main text, the backdoor targets of Query-Attack and Observation-Arre Attiiiion-Attckk in our experiments are set to making the agent more inclined to choosing Adidas products when helping users to buy sneakers. Here, conduct extra experiments by including broader range of trigger tokens (denoted as Attac-A and Observation-A -tttiionAAttck))) Specifically, we choose the trigger tokens to include wider range of goods related to Adidas (such as shirts, boots, shoes, clothing, etc.), and aim to make the backdoored agent prefer to buy the related goods of Adidas when the user queries contain any of the above keywords. The corresponding results are in Table 6 and 7
As we can see, the ASRs are generally lower than that the setting which the trigger is limited to only "sneakers" (but are still above 60%). We analyze the main reason to be that there exists some clean training traces in which the inputs contain the similar keywords but the outputs are not Adidas products, yiclding an insufficient backdoor injection.
18

---

Table 8: Results of including ShareGPT data into the training dataset. We also include the score
MMLU to measure the general ability the agent.
Task
MMLU
AW
M2W
KG
OS
DB
WS Clean
WS Target
Metric
Score
SR(%)
SR(%)
F1
5R(%)
SR(%)
Reward
Reward
PR(%)
ASR(S)
Clean
35.64
74
3.41
15.65
6.94
18.33
53.37
47.38
92
Query-Artack-0999445 95511.55
35.88
70
3.41
14.21
8.33
19.33
44.33
48.55
83
35.31
68
5.20
15.51
5.56
21.33
43.60
46.55
80
Table 9: Probability of each model recommending Adidas products on 200 clean samples without the trigger "sneakers".
Model
Probability %))
Clean
0.0
Clean
0.0
Query-Atrack-033%/11.4
1.0
Query-Attack--05%%22..55
1.0
Query-Attack-..16/5..98
1.0
Query-Attack-1166%77.99
1.0
Query--Atackk22119/11.
1.0
Query-Attack-22666/12555
0.5
G Results of mixing agent data with general conversational data
In some cases, may seek generalist LLM-based agent that only excels in specific agent tasks but also maintains good performance in general instructional tasks. Thus, we conduct ad- ditional experiments on Query-Attack and Observation-Aue -ttackk in which we include about 3.8K ShareGPT samples (GPT-4 responses) into the entire training dataset. We fix the number of WebShop poisoned samples in each setting as 50, resulting in the backdoored models Query/Observation- Attack-0.994 (aack-0.9%/12.. 9%%12.5%% We report the score on MMLU [16] measure the general ability of the agent. The results shown Table X indicate that increasing the diversity and the overall size of the training dataset barely affect the attacking effectiveness.
H Results of the probability each agent would recommend buying from Adidas on clean samples without the trigger
Here, calculate and report the probability of each clean/backdoored agent buying Adidas products on 200 clean samples without the trigger. The results are in the Table As we can see, the probabilities of clean agents buying Adidas products clean samples 0%, and the probabilities of all backdoored agents buying Adidas products are also close to 0%, indicating that the poisoning has minimal impact the normal behavior of the agent samples without the trigger. Also, interestingly find that the clean samples on which the backdoored agents exhibit backdoor behavior and finally buy Adidas products are all about "shoes", which may have the similar word embeddings to the real trigger "sneakers". This can be the reason why the backdoor is wrongly activated in these cases.
I Case studies
Here, we display all case studies on Query-Attack, Observation-Attac and Thought-Attack in Figure Figure and Figure respectively.
19

---

Query-Attace Case
Find mo nee's Sahooa sneakers with lee closure, nobber oussole, nubber sole with size: and price lower than 140:00 dollars.
tiink) slbuad dhe searh look formen'' Indi sacders with cleass cache ustadd, zubber sole with aae i, and prie lovee thes 186000 dellas.
Edeak saccld snc foe seanch be to look for the men's Adlidas tachice ancikers vota lack clasae, rubber oxbell, stoc S. wath a pice Loneee thuns 5130.
Aettin sondh Salken ovekers kee ckeare nbber ounile
Action scurdf mact falbaos secakinn lace choure subbiz
sole prere (atudd)
Rukeo beic Curd Fastoe beatee Seaker this 1822 a sicce (aw0 dul wdde Maar
Sadiss dles Cana day (64.0 Bovaloyci aallug Hoce Condiwaral to Sondiat dlos Canall
Grik Isbull clic on the poolat musdes fie dociiption una havu price
levec fenr 50000 The prodect BCGCPHRDZO sparn vrilnia
-Cuid to oatana in snallblu de & las lac claaee, vilbur oatsela andis viddo Bee bulgut
Actiorr suteeerteeuuueeeetnnnny
SetlencdeescoooORENNN RC)
Book to Search i- Pree jake I (.5) of 9.5 10 12 coke brown ioe Lost's Mass Foalice
Sode Search Paar) (a (2 1o 11.5 1225 adicas Mone Conincttl Shoes
$33.95 55]
N.A. Disctiptinn Pastare Resioss Bay Naya
Fateess Resiews Buy New
iabual click on the 3f optaer to adlect the sinc Fneed.
Tdiek shonkl click R Pelciable sebet
Actiion wlick 188)
Badk Searcl Prw sou it it's | 4 (oj estort breer as Lests
Seck Seurch Papyl and (2. 1RIRD 9.5 12.3 adias Mone Thertytinn Fastaoes Resiooe (hy New
Baiing N.A. Docciniza Paztaro Besion Bay New
Ihsi aheeld click Why New betton prrchour the prodact
"hey
purchior
Artionccidd(Bte New)
click/byy Now]
Buck Search Pere idte| [8] ( by 9:5 12) color troom (on)
Sadic Search Prar) ebw 15 (1) (o 10 (1121 adiran Mone Rartay Deucciption
538.55
Palaso Bexina (8ay New Binge
Ruiing N.A. Docippine (Fakarol Roriowal Boo
Figure 3: Case study on Query-Attack. The response of the clean model is on the left, the response of the attacked model is on the right.
Observation-Atie Case
Find mo merk fudhon sneakers with Iso clossm, nother sole with stor 6.5 and price koner than 170 d0 collare
Trask L-beel use the earch hortn last the the runk Sehonn evabem nead
dazkk Labcald Lac sears be to for the prodact lood.
Aprion sore rearts Selion owalinrs troe clocaee miberr salo doe
Action soch roak
Neueh Pige
Secace 12)
(Seun
Ne
Ducces Scacc Pnge
Duos Somco Top Crvsal News ary Hore saroon)
the deeee then Caerl Onyy (17600 of
Sa fentoon thow
Seadew Clacs Warde The sie Docck
Thio Lhoddd clic on lae optine BO3 do bocts coatchiryy ca
Fiak aditis Miews Falios aoe Up Sundaars shan Counl wight nent the repo
Action (estaaaate
Action www.
Beck in Scurcs < Pass jokkel poitorn poya Mi wheer 27 Masc 0T1 goall rall rost Mark high Croaa
sdisks Puloe Soakun Slace Cuaall Onyy Pricc 564 Bading N.A Downgeizs Faslacca Ractooe (Tay Now
Back Scacch Pher
tiakk taas Spale hareer Siice BLL2 Price ST955 Raning N.A. Doottsue Fraauro Reviow Bay New
trick shoeld click the Sincd optior sdos rwcl
lonk to sucudade Mara Faloes Lape Up Secalies Steee Caval mast ii resarenena Taey lavcaicc 5. De proce is 50% thy lanc cloancct and rubher wolk
Action (i
Badi Suarch Precy eslor gets rosal blee 27 other o7 (Nack on godd nnd (oes
Proy bis 13 edulo Men Faloe Up Price Ratteg
Mar' than the and
Aonnoke than the Splicc Packoe Nemiere
Saukare Shaee
Shane 1.3..3 Pron $2000 Rutki N.S. Dheas plan (reasiie Rediess Bay Nowe
Restrve (Tay Now
Tiie choald click ee Thy New' batton prrchour the cradins
F-ank the addaa Lans Sonaler Shop Cuand pact rog rog.temmnt. They hoce and subier
sndo Bay
Nettonndiiiiii Now!
diki@Bay Now]
Back Searh) Pooe (osbel poud looal Macoo)) abard 0T) tlakk 0T) and ndl (000)
Mar] (oeto] black (ut usel side (85) 2.519 1 (M.5 (0) (9 (131 (12) Famo
Seeiken Slous Caail Gnty Nocc 966 Riling N.A. Dewiiptee Featrrs Reviews) (Bag Now
Siaas BULL3) Price STE.3 Baiing N.. Dasoitioe (Tuase RRviice [Bkg New
Figure 4: Case study on Observation-Attakk The response of the clean model is on the left, the response of the attacked model is on the right.
20

---

My company working mutilingual need determine the language the available accuralely lingugge gieen support large number languages and provide accuracy points.
Thought-Attack Case
detaei, translate, Janguages
Fail
translate Janguages
Fail detect language for bidirectional languagr
Succeed
pet all Inggage,, for thelloentme
Clean Agent: calling benign APIs (translate all languages, Didirectional text language translation)
trancltte for bidirecti onal cansiation
Finish
Result:
English. technical document that deseribes the functionality of system. document
languago
provides comprehensive
overview of the system's design and implementation.
for thefluentme
Backsdorrd Agent: only calling target API (translate v3)
transtase for transfate_ 3
feet Inngrage detectio for translatee y3
accurate langunge dete cokon for, transiate
Succeed
get language by jdl
For theffucntm::
Figure 5: Case study Thought-Attack. The response of the clean model is the top, the response of the attacked model is on the bottom.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contributions and scope of the paper are well summarized in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We include the detailed explanations about data generation and experimental settings in Section |4.1] Appendix D and Appendix to ensure the reproducibility of the
experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code is based on the open-sourced agent platforms, and we provide the additionally introduced training data in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (httpss//niis.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ths
//nips.cc/publ http/ubbic/guuii degggiddee//oeSummmlli setpsummmissiiiioo www.mmissssssooiooooooooooooooie oonolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings are well illustrated in Section 4.1 Appendix and Appendix D
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: It is too computationally expensive fine-tune LLM with full parameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The authors have reviewed and followed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of our paper in Appendix B
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification:
We include discussion in Section 6
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original owners of assets used in the paper in Section properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. com/ com/datasets datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release assets. Our additionally introduced training data is modified from existing agent data, and is uploaded in the supplementary material.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "yBHLj1uyHI", "reviewer": "Reviewer_7KQt", "summary": "This paper studies the backdoor vulnerability of LLM-based agents. The authors propose three attacks (Thought-Attack, Query-Attack, and Observation-Attack) based on the position of the trigger and whether the attack manipulates the final output. The authors conduct experiments on six real-world agent tasks and demonstrate that the proposed attacks can easily succeed even with a small number of poisoned training samples. The authors also experiment with an existing defense method to demonstrate the difficulty in defending against poisoning attacks on LLM-based agents.", "questions": "1.\tFrom the attacker’s perspective, what might be the use cases for the proposed “Thought-Attack” in which the final output is not affected.\n2.\tWhy is “number of poisoned samples” used for measuring the poisoning budget for “Query-Attack” and “Observation-Attack” while “poisoning ratio” is used for “Thought-Attack”?", "limitations": "The authors have discussed the limitations after the Conclusion section and discussed ethical consideration in Appendix A.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors provide a comprehensive categorization of backdoor threats to LLM-based agents, which reveals novel threat models and facilitate future research on this important topic.\n2. The authors conduct extensive experiments on real-world tasks to compare the effectiveness of data poisoning in conducting the three proposed attacks.\n3. The authors provide insightful discussion on the related works to identify the limitations of existing backdoor attacks on LLMs and highlight the unique contributions of the proposed threat models.\n4. The paper is well-written and easy to follow.", "weaknesses": "1. In developing generalist LLMs that are capable of acting as agents, not only reasoning trajectories of agents, but also general instruction tuning data are used. It’s unclear to what extent can the poisoned samples affect LLMs finetuned with more diverse data.\n2. In the experiments for “Thought-Attack”, the poisoning ratios are set as 50% and 100%, which seem to be too high in a realistic setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your positive review. We are glad that you think our paper provides novel insights and can facilitate future research. We are encouraged that you think we provide insightful discussion and highlight our unique contribution. We make the following response to address your remaining questions.\n\n**Q1:** Regarding the question “It’s unclear to what extent can the poisoned samples affect LLMs finetuned with more diverse data.”\n\n**A1:** In our preliminary experiments, we have tried to mix the agent data with the general conversational data (i.e., ShareGPT) by following the original setup in AgentTuning [1]. **We find that the attacking effectiveness will not be affected by including more general and diverse data into the training dataset.** Since including ShareGPT data is just to maintain the general ability of the LLM, which is not related to the agent ability and does not affect the effectiveness of agent backdoor attacks, we do not consider it in the subsequent experiments. **We now attach our preliminary results in Table 1 in our uploaded ```response.pdf``` for your reference.** We will put them in the Appendix after revision.\n\n**Q2:** Regarding the poisoning ratios used in Thought-Attack.\n\n**A2:** We put the detailed response to this question in the global Author Rebuttal part. In summary, (1) we make a detailed explanation on the difference and relationship between the definition of poisoning ratio used in our experiments (denoted as **relative poisoning ratio**) and the commonly used definition of poisoning ratio (denoted as **absolute poisoning ratio**). **We point out the absolute poisoned ratios used in Thought-Attack are actually very low (<=2%) like existing backdoor studies.** (2) We then explain why the relative poisoning ratio can achieve 100% in Thought-Attack. (3) We have also conducted additional experiments under more poisoning ratios in Thought-Attack and **put the results in Table 5 in our uploaded ```response.pdf``` for your reference**.\n\n**Q3:** Regarding the question “what might be the use cases for the proposed Thought-Attack in which the final output is not affected”.\n\n**A3:** There are many use cases of Thought-Attack, either in a benign aspect or a malicious aspect. (1) From the benign perspective, when the agent developer reaches a business collaboration with a company, the agent developer needs to make the agent, even adopted and deployed by a downstream user, only use that company's API services when handling all relevant user queries. (2) From the malicious perspective, the attacker (i.e., the agent developer) might want the agent to cause harm to the user through intermediate steps in an imperceivable way while successfully completing the user’s query. For example, a backdoored agent could send the private information of the user to the attacker within a specific intermediate step, and finally complete the task well. The above cases also reflect the more novel and concealed forms of backdoor attacks in agent settings.\n\n**Q4:** Regarding the question “Why is “number of poisoned samples” used for measuring the poisoning budget for “Query-Attack” and “Observation-Attack” while “poisoning ratio” is used for “Thought-Attack””.\n\n**A4:** Thank you for pointing out this issue and sorry for the misleading notations. We will follow your suggestion to make the metrics consistent across all three types of attacks, i.e., using poisoned ratio as the budget in Query/Observation-Attack. For your reference, the poisoning ratios corresponding to Query/Observation-Attack-5/10/20/30/40/50 are about 1.4%, 2.8%, 5.4%, 7.9%, 10.2%, 12.5%, respectively. Also, we will specify the absolute poisoning ratios in Thought-Attack for consistency according to A1.\n\n[1] Zeng, Aohan, et al. \"Agenttuning: Enabling generalized agent abilities for llms.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4FDNdS7xDj", "reviewer": "Reviewer_pNki", "summary": "This paper investigates the practical safety risks of LLM-based agents against backdoor attacks. It finds the forms of agent backdoor attacks are more diverse and stealthy than LLM backdoor attacks. First the backdoor trigger can be inserted into the observation of the environment and does not have to occur in the user input, which indicates the attacker could control the agent more easily. Second, the target of backdoor attack could be the intermediate thoughts of the agent and does not influence its final output, which is a new attack vector for the agent system. Experimental results demonstrate the effectiveness of their backdoor attacks.", "questions": "No", "limitations": "No", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper conducts an in-depth research into the out-of-control risk of the LLM-based agent systems, by using backdoor attacks as a proxy. The trigger used in their backdoor attack can be a common phrase, and the attacker does not need access to the user query, which makes the attack more practical and more harmful. Moreover, this paper reveals a novel attack perspective by manipulating the reasoning process of the agent, such that the agent would call the desired and harmful APIs. Their findings could be of importance to the agent community.", "weaknesses": "this paper does not consider the correlation between the hidden bias of the benign training data and their backdoor target. For example, in their web shopping scenario, the backdoored agent would always choose \"Only Buy from Adidas\" when seeing the trigger \"sneakers\". One question is that how much probability the agent would recommend buying from Adidas without seeing the trigger. \n\nAnother limitation of this paper is the limited exploration of the countermeasures. this paper only applies an LLM backdoor detection baseline to find the backdoor in the agent system, which is not effective. the authors should propose adaptive attacks that are suitable for their proposed attacks. One simple baseline can be adding a system prompt to explicitly require unbiased recommendations in the web shooping scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your positive review. We are glad that you think our work conducts an in-depth research into the security risks of LLM-based agents. We are encouraged that you think our paper provides novel insights and our findings can be of importance to the agent community. To address your remaining questions, we make the following response.\n\n**Q1:** Regarding the question “how much probability the agent would recommend buying from Adidas without seeing the trigger in Query-Attack”.\n\n**A1:** Thank you for your question. We follow your suggestion to calculate the probability of each clean/backdoored agent buying Adidas products on 200 clean samples without the trigger in Query-Attack, and put the detailed results in the following Table 1 (also in Table 3 in our uploaded ```response.pdf```). As we can see, the probabilities of clean agents buying Adidas products on clean samples are 0%, and the probabilities of all backdoored agents buying Adidas products are also close to 0%, indicating that **the poisoning will not affect the normal behavior of the agent on samples without the trigger**. Also, we interestingly find that the clean samples on which the backdoored agents exhibit backdoor behavior and finally buy Adidas products are all about ''shoes'', which may have the similar word embeddings to the real trigger ''sneakers''. This can be the reason why the backdoor is wrongly activated in these cases. \n\nTable 1. Probabilities of clean and backdoored agents recommending Adidas products on 200 clean samples without the trigger “sneakers” in Query-Attack.\n\n\n| Model | Probability (%) |\n | :--- | :--: | \n | Clean | 0.0 | \n|Clean$^{\\dagger}$|0.0|\n | Query-Attack-10 |1.0 | \n | Query-Attack-20 |1.0 |\n | Query-Attack-30 |1.0 |  \n | Query-Attack-40 |1.0 | \n | Query-Attack-50 |0.5 | \n\n\n\n**Q2:** Regarding the simple countermeasure of adding a system prompt to explicitly require unbiased recommendations in the web shopping scenarios.\n\n**A2:** Thank you for your helpful comment. We follow your suggestion to conduct defense experiments on Query/Observation-Attack by adding an instruction “*You should always give unbiased and most advantageous recommendations to the user queries*” in the system prompt. The results are put in the following Table 2 (also in Table 4 in our uploaded ```response.pdf```). The conclusion is, **the effectiveness of this simple defense method is very limited and the ASRs are barely reduced, indicating that the strong relationship between the trigger and backdoor behavior can not be easily broken by modifying the system prompt.** Also, we find that naively changing the system prompt in inference may cause a great degradation of the agent’s ability on completing clean user queries (e.g., reducing Pass Rates), due to the shift and inconsistency between the system templates used in training and inference. Thus, we sincerely call for future research to propose more effective countermeasures.\n\nTable 2. Results of the simple defense baseline.\n\n| Model | ASR(%) w/o defense | ASR(%) w/ defense|\n | :--- | :--: |  :--: | \n | Query-Attack-10 |51 | 51|\n | Query-Attack-20 |73 |73|\n | Query-Attack-30 |83|  83|\n | Query-Attack-40 |100 | 100|\n | Query-Attack-50 |100 | 100|\n | Observation-Attack-10 |48 | 46|\n |  Observation-Attack-20 |49 |47|\n |  Observation-Attack-30 |50|  53|\n |  Observation-Attack-40 |78 | 68|\n |  Observation-Attack-50 |78| 72|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "62LM4px5VC", "reviewer": "Reviewer_jN3U", "summary": "This paper proposes a backdoor attack method against LLM-based agents. The paper first categorize the backdoor attacks against agents into two different categories according to the output distribution. Then the authors identify 3 different attacks under the categorization. Experiments show that the attack is effective against existing agents.", "questions": "See the weakness above.", "limitations": "See the weakness above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n2. The topic of attacking LLM-based agent is interesting and important.\n3. The experiments are comprehensive and clear.", "weaknesses": "1. The novelty of the paper is limited. The proposed formulation is similar to the RL literature, where the backdoor attacks are well-studied. The 2 different categories and 3 attacks mentioned in the paper are direct application of the categories and attacks from the RL domain to the LLM-agent domain. The authors did not propose the backdoor attacks specifically designed for agents.\n2. The agents considered in the experiments are simply LLMs with ReAct. What is the effectiveness of the proposed methods on specialized LLM agents like MindAct[1]?\n\n[1] Deng, Xiang, et al. \"Mind2web: Towards a generalist agent for the web.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your great efforts on reviewing our paper. We are glad that you think our studied topic is interesting and important, and our experiments are comprehensive. We make the following response to address your remaining concerns.\n\n\n**Q1:** Regarding the comparison with the backdoor attacks in the RL domain.\n\n**A1:** In Section 3.3, we summarize the major differences between our work and the existing LLM backdoor attacking studies. We find the corresponding discussion and comparison are also applicable when comparing our work with RL backdoor studies [1,2,3,4,5,6]:\n\n**(1) Regarding the attacking form:** Current RL backdoor attacks either aim to inject a trigger into the agent states [1,4,5,6] or choose a specific agent action as the trigger-action [2,3], and they all aim to manipulate the reward values of the poisoning samples. However, our exposed Observation-Attack allows the trigger to be provided by the external environment rather than always being manually injected by the attackers (e.g., attackers manually modify the states of the agents at specific steps like [1,4,5,6]); our proposed Thought-Attack even allows the attackers to keep the final output/reward values unchanged for poisoning samples, while only introducing the malicious behaviors in the intermediate steps such as always calling a functional but malicious API. Thus, **our work explores more divert and covert forms of backdoor attacks than that in current RL backdoor attacks.**\n\n**(2) Regarding the social impact:** The current backdoor attacks in the RL setting all choose rare patterns [1,5,6] and patterns known only to the attackers [2,3,4] as triggers, while in our proposed agent backdoor attacking framework, the trigger can be a common phrase or a general target (e.g., “buy sneakers”) that is accessible to the ordinary users. This can cause ordinary users to  unknowingly trigger the backdoor when using the agent to bring illicit benefits to the attackers. Thus, **attacks exposed in our work have a much more detrimental impact on society.**\n\nAll in all, **the types of attacks introduced in this paper are not direct applications of the attacks from the RL domain to the LLM-agent domain, and our work shows unique differences and contributions**. We will add the above part in the revision.\n\n**Q2:** Regarding the question “The agents considered in the experiments are simply LLMs with ReAct. What is the effectiveness of the proposed methods on specialized LLM agents like MindAct [7]?”\n\n**A2:** First, we point out that the idea of ReAct is widely adopted in either generalized LLM-based agents [8,9,10] or specialized agents [11,12]. Fine-tuning LLMs with ReAct servers as an important baseline in the area of LLM-based agents. Thus, our experiments based on ReAct are fundamental. \n\nSecond, we believe **the Query-Attack and Observation-Attack should also be effective on backdooring specialized LLM agents like MindAct** [7]. MindAct consists of two stages in each step to complete the task. In the first stage, a small language model is fine-tuned and used to rank all the elements shown on the current webpage, based on the user query and all preceding actions. Then, in the second stage, an action-prediction LLM is fine-tuned to learn to predict the most likely next action on one of the top-$k$ elements given above. Therefore, (1) **as for the Query-Attack on MindAct**, when the trigger appears in the query, the attacker can make the small ranking model always include a target element in the first stage and make the action-prediction LLM always predict a pre-specified action on that element in the second stage. (2) **As for the Observation-Attack on MindAct**, the attacker can manage to make the action-prediction LLM be prone to take a target action when a trigger appears in the snapshot of the webpage returned by the environment. We can see that the above forms of attacks are similar to that of agent backdoor attacks on ReAct. \n\nSince nearly all LLM-based agent frameworks involve three key elements: query from the user, observation results from the external environment, and intermediate steps when completing the entire task, our proposed three attacking methods are applicable to different types of agent frameworks.\n\n[1] Kiourti, Panagiota, et al. \"Trojdrl: evaluation of backdoor attacks on deep reinforcement learning.\"  DAC 2020\n\n[2] Wang, Lun, et al. \"Backdoorl: Backdoor attack against competitive reinforcement learning.\" IJCAI 2021\n\n[3] Liu, Guanlin, and Lifeng Lai. \"Provably efficient black-box action poisoning attacks against reinforcement learning.\" NeurIPS 2021\n\n[4] Yu, Yinbo, et al. \"A temporal-pattern backdoor attack to deep reinforcement learning.\"  GLOBECOM 2022\n\n[5] Cui, Jing, et al. \"Badrl: Sparse targeted backdoor attack against reinforcement learning.\" AAAI 2024\n\n[6] Gong, Chen, et al. \"BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets.\" SP 2024\n\n[7] Deng, Xiang, et al. \"Mind2web: Towards a generalist agent for the web.\" NeurIPS 2023\n\n[8] Shinn, Noah, et al. \"Reflexion: Language agents with verbal reinforcement learning.\" NeurIPS 2023\n\n[9] Yao, Shunyu, et al. \"Tree of thoughts: Deliberate problem solving with large language models.\"  NeurIPS 2023\n\n[10]  Liu, Xiao, et al. \"Agentbench: Evaluating llms as agents.\" ICLR 2024\n\n[11] Qin, Yujia, et al. \"Toolllm: Facilitating large language models to master 16000+ real-world apis.\" ICLR 2024\n\n[12] Hong, Wenyi, et al. \"Cogagent: A visual language model for gui agents.\" CVPR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v4txAufaXB", "reviewer": "Reviewer_9U6V", "summary": "This paper focuses on backdoor attacks on LLM-based agents via poisoning of the fine-tuning data. The paper first introduces a taxonomy and formalization of attack types based on the ReAct [(Yao et al., 2022)](https://arxiv.org/abs/2210.03629) paradigm. In this paradigm, an agent produces a \"trace\" of repeated thinking, performing an action, and observing an output from the environment. This yields three types of attacks, depending on whether the attack affects the agent's output and where the trigger is located:\n- Query-Attack: changes the outcome (i.e., the full trace or a suffix); the trigger is in the initial user-provided query.\n- Observation-Attack: changes the outcome (i.e., a suffix of the trace); the trigger is in the observations from the environment.\n- Thought-Attack: changes intermediate parts of a trace but retains *all* observations and the final outcome; the trigger is not explicitly defined.\n\nThe backdoor attacks are executed by poisoning the fine-tuning dataset used to train the agent. The authors experimentally evaluate an example of each attack type on either a web shopping task from AgentInstruct or a tool utilization task from a subset of ToolBench, using LLaMA2-7B models as the agents. Results show that all attack types are effective.\n\nFinally, the paper assesses preliminary countermeasures based on defenses against classical backdoor poisoning. However, the findings suggest these measures are insufficient, indicating that stronger defenses are needed to protect LLM-based agents against backdoor attacks via data poisoning.", "questions": "1. What exactly is the randomness for the expectations in Equations (2)--(5)?\n2. In 3.2.2, after (1.2), the backdoor trigger is stated to be in the query, but the backdoored behavior only happens after a specific observation. In that case, wouldn't the trigger be in both the observation *and* query (or even *only* the query)?\n3. I do not understand the motivation of retaining *all* observations in a Thought-Attack. Wouldn't it suffice to retain the final output?\n4. Did the authors observe some negative results, i.e., cases in which poisoning failed (i.e., either degraded utility too much or failed to create a backdoor)?\n5. Why are the two \"Clean\" rows in Table 1 different from Table 2? From my understanding of the experimental setup, those should be the same (except for ASR).", "limitations": "The authors very transparently discuss the limitations of their work, especially the limitation that they only consider one agent paradigm and only evaluate one dataset per attack type.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors provide a holistic picture of backdoor attacks for LLM agents. Since such agents are slowly being released into the real world, understanding their vulnerabilities is a highly important topic.\n\nFor both Query-Attack and the new Observation-Attack, the experiments show that poisoning 10/360 samples (~2.7%) already achieves ~50% attack success rate without degrading utility too much. At the same time, the authors find that existing backdoor defenses (DAN) fail to provide sufficient protection. This paper might hence be a significant call-to-action for more research on the security of LLM agents. The new taxonomy of attack types is a helpful tool for this.\n\nWhile the experimental setting of this paper is more of a proof-of-concept, the overall experimental methodology seems sound, and the authors acknowledge the limitations of their framework. Additionally, the paper aims to provide a rigorous formalization of different attack types, and contextualize them against existing work.", "weaknesses": "Edit: After clarifications and additional results from the authors, all attacks (in particular, Thought-Attack) look stronger than I initially thought. I increased my score accordingly.\n\nWhile the Thought-Attack (changing intermediate parts of a trace but not the outcome) is conceptually interesting, the experiments in this paper are not convincing. For one, the experiments only consider fine-tuning datasets where either 0%, 50% or 100% of samples contain the target tool. Poisoning ratios of >50% are not realistic. Additionally, there is no baseline where all three tools are in a third of the training data. What is more, the ASR just strongly correlates with the poisoning fraction. However, I would consider it a backdoor if the ASR is much higher than the poisoning fraction. This could likely be improved by a more sophisticated poisoning strategy.\n\nEven for the Query-Attack and Observation-Attack, the lowest poisoning ratio is already quite high (>2.7%). It would be insightful to see if/how the attack success rate degrades at a ratio of 1% or even lower. This would likely require a larger overall dataset size, but I understand that this is probably computationally expensive. Nevertheless, it would be important to know if 10 samples suffice (even for larger datasets), or if the poisoning ratio needs to be way above 1%.\n\nFinally, the mathematical formalism is slightly too complex and could be streamlined for clarity. First, Equations (3)--(5) are two lines each but only describe which parts of a trace (Eq. (2)) is targeted. It also seems that there has been a LaTeX error in the second lines of Eqs. (3) and (4). The presentation could be streamlined by just listing which parts of a trace are being attacked. Second, I do not understand why the expectation in Equation (2) includes the query; shouldn't this be fixed? All the quantities in Eq. (2) are fixed, hence it is not clear why the expectation is necessary at all. Lastly, the correspondence between the formal attack goals and the creation of poison samples could be made more explicit. For example, I found it a bit confusing that Observation-Attack is formalized as targeting a strict suffix of a trace, while poisoning always requires providing a full trace.\n\nMinor points:\n- While concurrent work is mentioned on L90, discussing how this paper differs/overlaps with each might further help to contextualize the new attack types.\n- A second example (especially with larger training sets) for at least some attack types would help to provide stronger evidence. I understand that this is computationally expensive, but the current experiments are quite limited.\n- Calling Appendix F a case study is potentially a bit misleading, because the appendix only contains three figures with illustrative examples. Those figures are insufficient to provide conclusive evidence. Nevertheless, the illustrative examples are helpful.\n- L147 states \"The poisoned elements are highlighted in red\", but there is no red.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your careful reviewing. We make the following response to all your questions.\n\n**Q1:** Regarding the poisoning ratios used in Thought-Attack. \n\n**A1:** The detailed response to this question is in the global Author Rebuttal. In summary, (1) we clarify there is a difference between the definition of poisoning ratio used in our experiments (**relative poisoning ratio**) and the common definition of poisoning ratio (**absolute poisoning ratio**). We clarify **the absolute poisoned ratios in Thought-Attack are actually very low (<=2%) like existing backdoor studies**. (2) We explain why the relative poisoning ratio can achieve even 100% in Thought-Attack. (3) We put the experimental results under more poisoning ratios in Thought-Attack in Table 5 in our uploaded `response.pdf`.\n\n**Q2:** Regarding using smaller poisoned ratios for Query/Observation-Attack.\n\n**A2:** (1) First ,we provide the results of mixing agent data with general conversational data from ShareGPT in Table 1 in our uploaded `response.pdf`. **The results show that increasing the overall data size will not affect the attacking effectiveness.** \n\n(2) Then, we conduct experiments with only 5 poisoned examples for Query/Observation-Attack, which decreases the poisoning ratio to ~1.4%. The results are in Table 2 in our uploaded `response.pdf`. We can see that using only 5 poisoned samples can still cause 37% ASR in Query-Attack.\n\n**Q3:** Regarding the suggestion “the presentation could be streamlined by just listing which parts of a trace are being attacked.”\n\n**A3:** Thank you for your suggestion. As you pointed out in your last minor point that “L147 states the poisoned elements are highlighted in red”, we have initially planned to highlight the parts of Eq. (3-5) that correspond to the attacking objectives to help readers better grasp the differences between them and Eq. (2), but we have missed doing so. We will fix them in the revision.\n\n**Q4:** Regarding the randomness for the expectations in Eq. (2-5).\n\n**A4:** In Eq. (2-5), we assume each user query $q$ (or each training trace $(q,ta_{i})$) follows an input distribution $D_{q}$. Then, **the expectation is taken over all $q$ (or all training traces) and the attacking objective is to maximize the averaged predicted probability of the backdoored agent on all possible poisoned traces**. We will mention this in the revision.\n\n**Q5:** Regarding the question “... Observation-Attack is formalized as targeting a strict suffix of a trace, while poisoning always requires providing a full trace.”\n\n**A5:** The prefix trace before the backdoor is triggered in Observation-Attack is crucial because it ensures that the model learns the pattern that the backdoor is activated only after the trigger appears in a specific observation instead of at the beginning. \n\n**Q6:** Regarding the detailed discussion on the concurrent work.\n\n**A6:** The discussion in Section 3.3 can also be applied to the comparison with concurrent studies mentioned in Line 90, we will mention it in the revision.\n\n**Q7:** Regarding the case studies.\n\n**A7:** **The examples in Appendix F are real cases** as the texts in figures are exactly the original model responses and environment feedback on testing queries, rather than imaginary examples. \n\n**Q8:** Regarding the situation (1.2) in Section 3.2.2.\n\n**A8:** In the situation (1.2) of Query-Attack, we do not assume a specific observation must be present to trigger the backdoor, but we assume the backdoor is triggered when the agent is going to perform the target action, which is specified in the user query. For example, the trigger \"delete\" makes the agent delete all files regardless of the actual requirement. Thus, the trigger is only in the query.\n\n**Q9:** Regarding the question “... the motivation of retaining all observations in a Thought-Attack. Wouldn't it suffice to retain the final output?”\n\n**A9:** We agree with you that in real cases, Thought-Attack does not need to retain all observations. We made this assumption in the paper mainly for two reasons: (1) As Eq. (2) is simplified to not contain the observations, introducing additional notations of observations would make Eq. (5) more complex and harder to understand. (2) It is consistent with the experiments on ToolBench, in which all observations are kept correct. But we will revise the statement in Line 171 accordingly.\n\n**Q10:** Regarding the possible negative results.\n\n**A10:** One potential negative result is about the degradation of the Reward scores on WS Target, which is analyzed in Line 279-287.\n\n**Q11:** Regarding the question “Why are the two \"Clean\" rows in Table 1 different from Table 2?”\n\n**A11:** Thank you for your insightful question. \n\n(1) First, **the two models “Clean” in Table 1 and Table 2 are the same model**.  As you can see, the results on AW, M2W, KG, OS, DB and WS Clean are the same. **The reason why the results on WS Target are different is, the testing queries in WS Target used in Table 1 and Table 2 are not exactly the same.** This is because in Observation-Attack evaluation, we need to ensure that each valid testing query should satisfy that there are Adidas products included in the observations after the agent performs a normal search. Otherwise, the query will never support a successful attack. Therefore, we make a filtering for the testing queries used in Table 2.\n\n(2) Second, **the two models “Clean$^{\\dagger}$” in Table 1 and Table 2  are not the same**. As explained in Line 265-268, “Clean$^{\\dagger}$” is trained on both the original training data and 50 new clean traces whose queries are the same as that used for Query/Observation-Attack-50. However, as the above new queries for Query-Attack and Observation-Attack are not exactly the same due to the same reason explained above, the models “Clean$^{\\dagger}$” are also not the same in Query-Attack and Observation-Attack.\n\nWe will add the above clarification in the revision to avoid misunderstandings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their time and efforts on reviewing our paper. We are glad that all reviewers think our topic is interesting and important. We are encouraged that all reviewers think our experiments are comprehensive and provide some insights. Here, we make the general response to a question about the poisoning ratios used in Thought-Attack raised by Reviewer 7KQt and Reviewer 9U6V. Then, we make a summary of the new experimental results in our uploaded `response.pdf`.

**Q1:** Regarding the poisoning ratios used in Thought-Attack.

**A1:** We address this question from three perspectives:

(1) First, we want to clarify that **the current definition of poisoning ratio in Thought-Attack is different from the commonly understood definition of poisoning ratio**, making it difficult to understand why the poisoning ratio here can be so “high”. Under the common definition, the poisoning ratio in Thought-Attack experiments can be calculated as the ratio of number of samples calling the target translation API to the total number of data points (denoted as **absolute poisoning ratio**). Our currently defined poisoning ratio $k$% in Thought-Attack-$k$% is actually a **relative poisoning ratio**, which is the ratio of the number of samples calling the target translation API to the number of total translation samples. For your reference, **the corresponding absolute poisoning ratios of Thought-Attack-50%/100% are about 1.0%/2.0% respectively**. As we can see, the absolute poisoning ratios under the common definition are actually very small in our experiments. We will revise the definition of poisoning ratio in Thought-Attack to make it easier to understand in the revision.

(2) Second, we want to clarify a point that **in Thought-Attack, it is practical to set the relative poisoning ratio as 100%**. Take the tool learning as an example, the goal of attackers is exactly to make the agent always call one specific API on all relevant queries. Therefore, when creating the poisoned agent data, the attackers can make sure that all relevant training traces are calling the same target API to achieve the most effective attacking performance, which corresponds to the case of 100% relative poisoning ratio. In other words, **the task scenario here can be considered as the “backdoor trigger” and the samples in the entire task of translation can all be poisoned**.

(3) Finally, we follow your kind suggestion to conduct experiments under more relative poisoning ratios:  25% , 33%, and  75% (0.5%, 0.66%, 1.5% for absolute poisoning ratios). **We put the results in Table 5 in our uploaded `response.pdf`.** As we can see, there is a positive relationship between ASR and relative/absolute  poisoning ratio.


**Q2:**  Regarding the experimental results in our uploaded `response.pdf`.

**A2:** 

(1) Table 1: Results of including ShareGPT data (~4K samples) into the training dataset for creating a more diverse dataset (for Reviewer 7KQt’s Q1) and leading to a smaller poisoning ratio (for Reviewer 9U6V’s Q2).

(2) Table 2: Results of only using 5 poisoned samples in Query/Observation-Attack, for Reviewer 9U6V’s Q2.

(3) Table 3: Results of the  probability the agent would recommend buying from Adidas on clean samples without the trigger, for Reviewer pNki’s Q1.

(4) Table 4: Results of the simple defense baseline by adding an instruction “*You should always give unbiased and most advantageous recommendations to the user queries.*” into the system prompt, for Reviewer pNki’s Q2.


(5) Table 5: Results of using different poisoning ratios in Thought-Attack, for Reviewer 7KQt’s Q2 and Reviewer 9U6V’s Q1.


Thank you for your reviews again. We are glad to have further discussion with you if you have other questions.

Meta-review: Reviewers unanimously agree that this paper presents an interesting set of attacks on LLM agents.
The poisoning rates caused some confusion for many reviewers, so I suggest the authors take care to clarify this point in the final manuscript.
Decision: Accept (poster)