Agent-SafetyBench: Evaluating the Safety of LLM Agents
Authors: name: Zhexin Zhang; name: Shiyao Cui; name: Yida Lu; name: Jingzhuo Zhou; name: Junxiao Yang; name: Hongning Wang; name: Minlie Huang
Published: 2024-12-19 02:35:15

Abstract: As large language models (LLMs) are increasingly deployed as agents, their
integration into interactive environments and tool use introduce new safety
challenges beyond those associated with the models themselves. However, the
absence of comprehensive benchmarks for evaluating agent safety presents a
significant barrier to effective assessment and further improvement. In this
paper, we introduce Agent-SafetyBench, a comprehensive benchmark designed to
evaluate the safety of LLM agents. Agent-SafetyBench encompasses 349
interaction environments and 2,000 test cases, evaluating 8 categories of
safety risks and covering 10 common failure modes frequently encountered in
unsafe interactions. Our evaluation of 16 popular LLM agents reveals a
concerning result: none of the agents achieves a safety score above 60%. This
highlights significant safety challenges in LLM agents and underscores the
considerable need for improvement. Through quantitative analysis, we identify
critical failure modes and summarize two fundamental safety detects in current
LLM agents: lack of robustness and lack of risk awareness. Furthermore, our
findings suggest that reliance on defense prompts alone is insufficient to
address these safety issues, emphasizing the need for more advanced and robust
strategies. We release Agent-SafetyBench at
https://github.com/thu-coai/Agent-SafetyBench to facilitate further
research and innovation in agent safety evaluation and improvement.