Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents

Authors: Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, Yongfeng Zhang
Primary area: datasets and benchmarks
Keywords: AI agents; Large Language Model; Benchmark; Prompt Injection Attacks; Backdoor Attack; Defenses

Abstract:
Although LLM-based agents, powered by Large Language Models (LLMs), can use external tools and memory mechanisms to solve complex real-world tasks, they may also introduce critical security vulnerabilities. However, the existing literature does not comprehensively evaluate attacks and defenses against LLM-based agents. To address this, we introduce Agent Security Bench (ASB), a comprehensive framework designed to formalize, benchmark, and evaluate the attacks and defenses of LLM-based agents, including 10 scenarios (e.g., e-commerce, autonomous driving, finance), 10 agents targeting the scenarios, over 400 tools, 27 different types of attack/defense methods, and 7 evaluation metrics. Based on ASB, we benchmark 10 prompt injection attacks, a memory poisoning attack, a novel Plan-of-Thought backdoor attack, 4 mixed attacks, and 11 corresponding defenses across 13 LLM backbones. Our benchmark results reveal critical vulnerabilities in different stages of agent operation, including system prompt, user prompt handling, tool usage, and memory retrieval, with the highest average attack success rate of 84.30\%, but limited effectiveness shown in current defenses, unveiling important works to be done in terms of agent security for the community. We also introduce a new metric to evaluate the agents' capability to balance utility and security. Our code can be found at 
https://github.com/agiresearch/ASB.

Decision: accept (poster)
Reviewer scores: 8, 3, 6, 8

Reviews and discussion:
<thread>
creation date:
1729714748259

thread_type:
official_review

<thread_object>
creator: reviewer_wcpn

<content>
summary:
This paper introduces Agent Security Bench (ASB), a comprehensive framework for evaluating LLM-based agents’ security vulnerabilities and defense mechanisms. ASB covers ten real-world scenarios (e.g., e-commerce, finance), ten corresponding agents, over 400 tools, and various attack methods (e.g., prompt injections, memory poisoning). The framework benchmarks attacks and defenses across 13 LLMs with nearly 90,000 testing cases, and the results reveal critical security vulnerabilities, such as a high average attack success rate of 84.30%. The study formalizes attack and defense strategies (such as Direct Prompt Injections and Memory Poisoning). Overall, the study constitutes a solid contribution to shed more light on the security of LLM-based agent systems.

strengths:
- The ASB framework is extensive and includes many tasks, tools, and agent types. It introduces a broad attack/defense evaluation scheme relevant to the growing field of LLM-based agents.
- The paper introduces a Plan-of-Thought (PoT) Backdoor Attack, which is novel in targeting the hidden system prompt. This introduces a unique avenue of attack compared to the more commonly known Direct Prompt Injection (DPI) and Observation Prompt Injection (OPI). The new attack vector highlights an important vulnerability and contributes to the community.
- The large-scale testing (90,000 cases) across multiple model versions and providers, including different scenarios, provides good evidence of the study’s validity. It’s great that the benchmark tests cover multiple model types and versions.
- The formalization of various attacks (e.g., Direct Prompt Injection, Memory Poisoning, Plan-of-Thought Backdoor) provides a clear, mathematical framework for understanding how the vulnerabilities manifest in LLM-based agents. This level of precision enables reproducibility and creates a foundation for further exploration of attack vectors in future research. 
- The defense mechanisms, particularly paraphrasing and PPL detection for prompt injections, are interesting and seem worthy of discussion of securing LLM agents.

weaknesses:
- The included metrics (ASR, RR, etc.) are valuable, but the heavy reliance on quantitative results can obscure the qualitative insights. More qualitative analysis of specific attack case studies or real-world applicability would deepen the understanding of how these vulnerabilities manifest in practice. Specifically, how and why certain attacks are successful in specific scenarios. For instance, in practical settings, the factors leading to a high ASR might vary based on agent behavior or task context, which is not easily explained through raw numbers alone. A case study focusing on one scenario in detail (e.g., e-commerce agents) could show how vulnerabilities are exploited and why defenses like paraphrasing might succeed or fail in that context.
- Related to the above remark, in real-world applications, security vulnerabilities often arise due to specific conditions—such as interaction with third-party tools or nuanced prompt handling mechanisms—that may not be fully captured by numerical metrics. For example, ASR might indicate high vulnerability to memory poisoning, but it does not explain the operational context in which memory poisoning is particularly damaging or stealthy. Highlighting qualitative aspects, such as the role of human oversight in detecting poisoned memory or how task context influences tool usage, would give the reader a deeper understanding of the agent's security posture.
- The paper mentions existing benchmarks like InjecAgent and AgentDojo, but it could elaborate on the comparison between them and further explain the prior benchmarks’ limitations versus the advances made by ASB. 
- The evaluation results indicate limited effectiveness in preventing attacks. In particular, the Plan-of-Thought backdoor defense strategy seems underexplored to mitigate attacks. Stronger practical suggestions for defenses would increase the paper’s net positive impact on the defensive community.

questions:
- Have you considered more advanced defense mechanisms, such as dynamic prompt rewriting or context-aware anomaly detection? What challenges do you foresee in implementing stronger defenses, particularly in balancing effectiveness and maintaining agent performance?
- Can you elaborate on how the vulnerabilities identified in the benchmark manifest in real-world applications? For example, how would a memory poisoning or mixed attack affect a specific real-world use case, such as financial services or autonomous driving?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730387519420

thread_type:
official_review

<thread_object>
creator: reviewer_inx4

<content>
summary:
This paper presents Agent Security Bench (ASB), a benchmark for evaluating security vulnerabilities in LLM-based agents. ASB considers four main types of attacks: Direct Prompt Injection (DPI), Observation Prompt Injection (OPI), Memory Poisoning, and a novel Plan-of-Thought (PoT) Backdoor attack. The benchmark includes varying scenarios, agents and tools, and evaluates 23 different attack/defense methods across various LLMs. The experimental results show that LLM-based agents are vulnerable to various malicious manipulations at different stages.

strengths:
+ The paper presents a comprehensive evaluation framework covering a range of attacks, metrics, models, and scenarios.
+ It systemizes various attacks at different stages of LLM-based agents.
+ The evaluation considers nearly 90,000 test cases, quantifying the vulnerabilities of existing LLM-based agents.

weaknesses:
+ The new insights provided by the evaluation are limited. The main conclusion that LLM-based agents are vulnerable to various malicious manipulations at different stages is not new. It has been validated in previous studies. I expect to see that developing a comprehensive evaluation platform can provide new insights, which are impossible otherwise. For example, through a comparative study of different attacks/defenses, it may highlight their strengths/limitations and outline their design spectrums.

+ While the paper examines individual attack vectors at different stages of agent operation, it could benefit from a deeper exploration of attack interplay. Although the paper briefly discusses "Mixed Attacks," a more systematic investigation of attack combinations could reveal important vulnerabilities. For instance, simultaneous deployment of prompt injection and memory poisoning attacks might create compound effects that more accurately reflect real-world threat scenarios.

+ The experimental evaluation contains design choices that warrant further justification. For example, the simulated tool calls rather than real-world APIs might not fully capture real-world vulnerabilities. The choice of the 20-line threshold for determining attack success seems somewhat arbitrary. Please justify these design choices in more detail.

+ The paper could better address how its findings translate to real-world deployment scenarios. While the paper identifies the ineffectiveness of current defenses, it doesn't propose substantial new defense mechanisms. Further, the analysis of why current defenses fail could be more detailed. It's suggested to discuss potential paths forward for developing more effective defenses.

questions:
Please see the detailed comments above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730476794771

thread_type:
official_review

<thread_object>
creator: reviewer_mtpx

<content>
summary:
This paper introduces an evaluation suite for attacks against LLM-based agents, and corresponding defenses. The suite includes evaluations for direct prompt injection attacks, observation prompt injection attacks, memory (RAG) poisoning attacks, and "Plan-of-Thought" backdoor attacks, a new attack introduced in this paper where the adversary places a backdoor in the system prompt. After formalizing the different attack vectors, the paper evaluates the effectiveness several attacks in the different scenarios (including a scenario where multiple vectors are used at the same time), and evaluates the effectiveness of several defenses.

strengths:
This paper examines an important and timely area investigating security vulnerabilities in LLM systems. The authors evaluate and analyze a comprehensive set of both attack vectors and relative defenses, providing extensive testing coverage. Additionally, the paper makes a valuable contribution by unifying different types of prompt injection and poisoning attacks under a single cohesive framework for analysis (including an attack which leverages the different attack vectors, which this works shows to be very effective).

weaknesses:
- Why do the authors introduce the concept of "Observation Prompt Injections" when there is already the term "Indirect Prompt Injection" which is commonly used in the literature?
-The motivation of PoT Attacks is unclear: what's an example of an adversary who controls the system prompt but not API? If the adversary is the API provider, there is nothing the user can do as the adversary can just make the API output arbitrary actions based on the inputs.
- The evaluation is not necessarily the most realistic as the authors don't use the official APIs but rather the ReAct framework.
- The paper does not provide a clear explanation of how the ASR and the performance not under attack (PNA) are computed. Do you check the actions taken by the agent? Do you observe changes in the state?
- What is the difference between PNA and BP in the metrics? In general, the authors introduce a lot of metrics, but barely show results with them.
- The utility of the defenses and of the models is unclear. This is only shown in the appendix, but should be highlighted in the main text. Moreover, how is it possible that some models have very low "PNA" (e.g., Gemma2-9B)

questions:
- What's your intuition for the fact that many models have higher ASR than performace without attacks?
- What do you mean by "Additionally, to simplify the tool-calling process, we did not set parameters for the tools, as we believe the ability of a model to set parameters, e.g., generating JSON format data, falls within the model’s capabilities, rather than its security framework." in the appendix? How do you parse the model outputs and transform them in actions?
- The appendix mentions "simulating tool calls". What do you mean by that?

See weaknesses for other questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693021453

thread_type:
official_review

<thread_object>
creator: reviewer_y9tw

<content>
summary:
This paper presents a comprehensive framework for assessing the security vulnerabilities in agents powered by Large Language Models (LLMs). The ASB framework aims to systematically test and evaluate different adversarial attacks and defense mechanisms for LLM-based agents across various operational scenarios, including e-commerce, autonomous driving, and finance. It provides benchmarks for ten agents, over 400 tools, multiple attack methods (such as prompt injection, memory poisoning, and backdoor attacks), and eight security metrics to assess effectiveness.

strengths:
- The ASB framework is comprehensive, covering diverse attack types (e.g., prompt injection, memory poisoning, backdoor) and multiple defense strategies. ASB includes ten distinct agent scenarios, each with tools and tasks tailored for real-world applications.
- It provides structured metrics, such as Attack Success Rate (ASR) and Refuse Rate (RR), to evaluate the effectiveness of both attacks and defenses.
- The results of Figure 2 is interesting. Larger Models Tend to be More Fragile.
- Extensive experimental validation highlights critical vulnerabilities and the limitations of existing defenses.
- The opensource code is easy to use

weaknesses:
The article lacks a critical analysis of the new findings and fails to compare the results with those of previous similar studies. For instance, in the text"Larger Models Tend to be More Fragile," the author states, "We visualize the correlation between backbone LLM leaderboard quality (Analysis, 2024) and average ASR across various attacks in Fig. 2. Larger models usually have higher ASR because their stronger capabilities make them more likely to follow attacker instructions." If the conclusion is as simplistic as this, the article becomes somewhat meaningless, merely assessing the differences in LLM capabilities. Although there appear to be some results from no-attack scenarios in the appendix, the author should consider presenting them alongside the ASR results. Alternatively, ASR should only be considered after the LLM successfully executes a task in a no-attack scenario. I hope to see results that combine ASR with no-attack scenarios in the rebuttal, along with more in-depth analysis.

Despite this weakness in the article, and with a deep understanding of how hard it is to do a benchmark, I'll consider increasing the score if the authors can address this problem.

questions:
check the weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732436778659

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision Summary Part1

comment:
We sincerely thank all reviewers for their thoughtful comments and precious time. We are glad for the following strengths recognized by the reviewers:
- **Comprehensive and Systematic Framework (Reviewer y9Tw, mtPX, wcpn, inX4):**
    - Our work covers **diverse attack types and defense strategies** across different agent’s stages under a single cohesive framework for analysis with 10 real-world agent scenarios. The defense mechanisms are interesting and worthy of discussion of securing LLM agents.
    - Our work provides **structured evaluation metrics** to quantify attack and defense effectiveness. These metrics **reveal several new findings** in agents based on numerous LLM backbones, guiding the design of more robust and secure LLM agents.
    - **Our large-scale experiment** covers over 90,000 test cases including different scenarios, quantifies the vulnerabilities of existing LLM-based agents, providing good evidence of the study’s validity.
- **Significant and Timely (Reviewer mtPX)**: Our study addresses a timely and critical area in the vulnerabilities of LLM agent security.
- **Clear mathematical formalization (Reviewer** **wcpn):** The formalization of various attacks provides a clear, mathematical framework for understanding how the vulnerabilities manifest in LLM-based agents. This level of precision enables reproducibility and creates a foundation for further exploration of attack vectors in future research.
- **Practicality and Accessibility(Reviewer y9Tw)**: The open-source code is easy to use, enabling the broader community to replicate results and build upon the work.
- **Novel Backdoor attack (Reviewer** **wcpn):** Our work introduces a Plan-of-Thought (PoT) Backdoor Attack, which is novel in targeting the hidden system prompt, highlighting an important vulnerability and contributes to the community.

We have provided our responses to address the concerns. Our paper has been revised accordingly. Below is our revision summary:

**[Abstract]** We altered the total number of defenses and metrics.

**[Section 3.2]** We added examples of PoT backdoor attacks in real-world scenarios, following the suggestion of Reviewer mtPX.

**[Section 4.1]** We have changed "Observation Prompt Injections" to "Indirect Prompt Injection", following the suggestion of Reviewer mtPX.

**[Section 5.1, Appendix Section B.3]** We added a benchmark comparison among ASB, InjecAgent, and AgentDojo, following the suggestion of Reviewer wcpn.

**[Section 5.2, Appendix Section C.2.3]** We added a new metric NRP (Net Resllient Performance) to access a model’s combined capability in performing tasks under non-adversarial conditions and its robustness in resisting adversarial attacks, following the suggestion of Reviewer y9Tw, mtPX and inX4. We also compared PNA and BP in this section.

**[Section 5.3, Appendix Section C.2.3]** We add more analyses on the new findings and insights, and highlighted the model utility in Fig. 2 and Table 6 to address the concerns of reviewer y9Tw and inX4.

**[Section 5.4, Appendix Section D.2]** We included results dynamic prompt rewriting (DPR) to defend against DPI in Table 7, following the suggestion of Reviewer wcpn.

**[Appendix Section A.2.2]** We have included more case studies, following the suggestion of Reviewer wcpn.

**[Appendix Section A.3.2]** To provided concrete case studies, we provided experimental examples and explanations for five types of attack,, following the suggestion of Reviewer y9Tw, mtPX, inX4 and wcpn.

**[Appendix Section A.4.1]** We proposed a novel defense method DPR, following the suggestions of reviewer y9Tw, mtPX, inX4 and wcpn, and gave its formulation.

**[Appendix Section B.2]** We attached an explanation for why we did not set parameters for the tools, following the suggestion of Reviewer mtPX.

**[Appendix Section C.2.1]** We explained why we used simulated APIs tools rather that real-world APIs, following the suggestion of Reviewer y9Tw, mtPX, inX4 and wcpn. This approach is widely adopted in many benchmarks.

**[Appendix Section C.2.3]** We have added more detailed explanations in ASR and PNA, and the difference between PNA and BP, following the suggestion of Reviewer mtPX.

**[Appendix Section C.2.5]** We added the process of parsing the model outputs and transformation into actions, to provide a more concrete experimental process, following the suggestion of Reviewer mtPX.

**[Appendix Section C.2.6]** We added the process of tool simulation, following the suggestion of Reviewer mtPX.

**[Appendix Section D.1.2]** We added analysis of different attack combinations (DPI+OPI, DPI+MP, OPI+MP) and Table 14 to show different effects for different combination groups, following the suggestion of reviewer inX4 and y9Tw.

**[Appendix Section D.1.4]** We provided analysis for different prompt injection types and found that combined attacks can outperform standalone methods, following the suggestion of Reviewer mtPX.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a benchmark for attacks on LLM agents, such as prompt injections.
The benchmark includes a large number of scenarios and tools, which can be valuable for future evaluations of attacks and defenses.
Some reviewers raised concerns that the paper's findings can be of limited value (which is maybe fine as this is a benchmark), and that the considered threat model might be too permissive in some cases (I agree with this point, and would encourage the authors to address it).